@HeMaN You're correct that no changes have been published yet.

We were under the impression that we had found an undocumented requirement, but I was reminded that giving each host the same certificate is not the best practice. xcp-ng should be able to handle each host having its own certificate as long as their respective certificate authorities are trusted.

In any case, I need to do some more testing to narrow down the exact cause of the issues that I was seeing. I have been working a systemd service and timer with a few supporting scripts that automatically renew certificates by making ACME requests to my local, private CA, specifically adding support for additional SANs (previous iterations just used the system's FQDN).

Specifically, I want to test each server with SANs that correspond to each of it's IP addresses and FQDNs, deploy them using xe host-server-certificate-install, then perform packet captures as needed to determine why the Xapi#getResource /rrd_updates (on xcp-ng-1) 0% task is getting stuck.

So far, life has gotten a bit in the way, so I haven't dedicated the time to testing this, but I hope to get back to this soon.

@florent the CR job was completed with health checks. The issue appears to be fixed in the fix_cr_healthcheck branch.

@Andrew @Danp - I discovered my boneheaded oversight

I have been developing on a one-shot systemd service with a timer for automatically renewing my short-lived XO certificate daily between a prescribed window of time using my local Step CA instance's ACME provisioner. It has been a fun project and I now have mostly automatic renewal of short-live certificates using what is essentially a private letsencrypt server that uses the CA that I generated.

I just looked at the logs and realized that I completely overlooked the fact that the the step-renewer.service is being run starting at midnight + random number of seconds between 0 and 5 minutes. This service will get a new certificate and then it restarts xo-server.service and that is what has been interrupting my backups.

Facepalm

With mixed feelings, I can confidently report that this issue is entirely self-induced. Thanks for the assistance narrowing down the cause!

@stormi @olivierlambert Please see my submitted PR and please provide feedback.

@irtaza9 happy to help!

@olivierlambert Thank you for bringing this to the attention of other folks

@andriy.sultanov great, thank you for taking that on!

I must have skimmed over that part of the docs too quickly to notice the recommendation to run usb_scan.py.

I think it would be helpful to explain more verbosely how the order and specificity impact the final outcomes as it's not clear to me. It seems that if I make a more specific rule for my Yubikey before a more general rule that would block devices of the same class, it seems to work, but perhaps not in reverse?

I'm reminded of the apache Order directive where it can be AllowDeny or DenyAllow — my dyslexic brain has a hard time keeping track of binaries.

tl;dr - empty line(s) in /etc/xensource/usb-policy.conf crashes /opt/xensource/libexec/usb_scan.py

I did a bit of scanning through the xapi source, in particular https://github.com/xapi-project/xen-api/blob/master/python3/libexec/usb_scan.py

I'm not a python expert, so I could generally follow the flow of things, but I wasn't totally sure what was happening at a detailed level. Then I did some googling and found this xenserver help doc regarding troubleshooting usb passthrough: https://support.citrix.com/external/article/235040/how-to-troubleshoot-xenserver-usb-passth.html

This article suggested running /opt/xensource/libexec/usb_scan.py with the -d parameter for additional details and that lead me to discover that the script fails when it encounters an empty line in usb-policy.conf

[23:46 xcp-ng-4 ~]# /opt/xensource/libexec/usb_scan.py -d
Traceback (most recent call last):
  File "/opt/xensource/libexec/usb_scan.py", line 681, in <module>
    pusbs = make_pusbs_list(devices, interfaces)
  File "/opt/xensource/libexec/usb_scan.py", line 660, in make_pusbs_list
    policy = Policy()
  File "/opt/xensource/libexec/usb_scan.py", line 384, in __init__
    self.parse_line(line)
  File "/opt/xensource/libexec/usb_scan.py", line 444, in parse_line
    if action.lower() == "allow":
UnboundLocalError: local variable 'action' referenced before assignment

After removing all empty lines, usb_scan.py scanned output properly, but it was giving me an empty array

[23:53 xcp-ng-4 ~]# /opt/xensource/libexec/usb_scan.py -d
[]

Since it as no longer crashing, I decided to go back to the default usb-policy.conf then try only adding my single allow rule without any extra lines and then test.

cp /etc/xensource/usb-policy.conf.default /etc/xensource/usb-policy.conf

I inserted a new line at line 10 just above the first rule and added my allow rule:

ALLOW:vid=1050 pid=0407

and that was all that was needed! Now I can see the device after running usb_scan.py

[23:53 xcp-ng-4 ~]# /opt/xensource/libexec/usb_scan.py -d
[{"path": "2-1.1", "version": "2.00", "vendor-id": "1050", "product-id": "0407", "vendor-desc": "Yubico.com", "product-desc": "Yubikey 4 OTP+U2F+CCID", "speed": "12", "serial": "", "description": "Yubico.com_Yubikey 4 OTP+U2F+CCID"}]

I also learned that the last good output of xe usb-scan seems to be cached somewhere and is quietly returned without hesitation when usb_policy.py fails. Maybe it's logged somewhere, but I don't know.

In any case, it was as simple as an empty line -- don't take anything for granted!

Lastly, I did a bit of testing to confirm that for my Yubikey to be detected and allowed, the allow rule must be BEFORE the rule that denies HID Boot Keyboards.

This results in detection:

# When you change this file, run 'xe pusb-scan' to confirm
# the file can be parsed correctly.
#
# Syntax is an ordered list of case insensitive rules where # is line comment
#  and each rule is (ALLOW | DENY) : ( match )*
#  and each match is (class|subclass|prot|vid|pid|rel) = hex-number
# Maximum hex value for class/subclass/prot is FF, and for vid/pid/rel is FFFF
#
# USB Hubs (class 09) are always denied, independently of the rules in this file
DENY: vid=17e9 # All DisplayLink USB displays
DENY: class=02 # Communications and CDC-Control
ALLOW:vid=056a pid=0315 class=03 # Wacom Intuos tablet
ALLOW:vid=056a pid=0314 class=03 # Wacom Intuos tablet
ALLOW:vid=056a pid=00fb class=03 # Wacom DTU tablet
# @jeff - allow passthrough of Yubikey 5 FIPS, "Yubikey 4 OTP+U2F+CCID"
ALLOW:vid=1050 pid=0407
DENY: class=03 subclass=01 prot=01 # HID Boot keyboards
DENY: class=03 subclass=01 prot=02 # HID Boot mice
DENY: class=0a # CDC-Data
DENY: class=0b # Smartcard
DENY: class=e0 # Wireless controller
DENY: class=ef subclass=04 # Miscellaneous network devices
ALLOW: # Otherwise allow everything else

This does not:

# When you change this file, run 'xe pusb-scan' to confirm
# the file can be parsed correctly.
#
# Syntax is an ordered list of case insensitive rules where # is line comment
#  and each rule is (ALLOW | DENY) : ( match )*
#  and each match is (class|subclass|prot|vid|pid|rel) = hex-number
# Maximum hex value for class/subclass/prot is FF, and for vid/pid/rel is FFFF
#
# USB Hubs (class 09) are always denied, independently of the rules in this file
DENY: vid=17e9 # All DisplayLink USB displays
DENY: class=02 # Communications and CDC-Control
ALLOW:vid=056a pid=0315 class=03 # Wacom Intuos tablet
ALLOW:vid=056a pid=0314 class=03 # Wacom Intuos tablet
ALLOW:vid=056a pid=00fb class=03 # Wacom DTU tablet
DENY: class=03 subclass=01 prot=01 # HID Boot keyboards
# @jeff - allow passthrough of Yubikey 5 FIPS, "Yubikey 4 OTP+U2F+CCID"
ALLOW:vid=1050 pid=0407
DENY: class=03 subclass=01 prot=02 # HID Boot mice
DENY: class=0a # CDC-Data
DENY: class=0b # Smartcard
DENY: class=e0 # Wireless controller
DENY: class=ef subclass=04 # Miscellaneous network devices
ALLOW: # Otherwise allow everything else

Thanks to everyone who took a look. Hopefully you don't get caught by this same gotcha!

@olivierlambert Thank you and your team again for your commitment to this fantastic FOSS tool and for allowing me to build it myself!

I very much appreciate the personal touch of my issues being triaged by the CEO and Co-Founder. It's refreshing to see an executive officer stay in touch with their customer base.

@olivierlambert as pointed out by @psafont on my PR #216,

I believe there is no such technical requirement, when following a redirect the
new request should be done against a different IP/host and the TLS connection renegotiated with that host, meaning none of the hosts' certs should have identifying information from the other one.

I think I need to dive deeper and hope to find an a log message related to the lingering Xapi#getResource /rrd_updates tasks.

@olivierlambert Thank you for bringing this to the attention of other folks

@andriy.sultanov great, thank you for taking that on!

I must have skimmed over that part of the docs too quickly to notice the recommendation to run usb_scan.py.

I think it would be helpful to explain more verbosely how the order and specificity impact the final outcomes as it's not clear to me. It seems that if I make a more specific rule for my Yubikey before a more general rule that would block devices of the same class, it seems to work, but perhaps not in reverse?

I'm reminded of the apache Order directive where it can be AllowDeny or DenyAllow — my dyslexic brain has a hard time keeping track of binaries.

tl;dr - empty line(s) in /etc/xensource/usb-policy.conf crashes /opt/xensource/libexec/usb_scan.py

I did a bit of scanning through the xapi source, in particular https://github.com/xapi-project/xen-api/blob/master/python3/libexec/usb_scan.py

I'm not a python expert, so I could generally follow the flow of things, but I wasn't totally sure what was happening at a detailed level. Then I did some googling and found this xenserver help doc regarding troubleshooting usb passthrough: https://support.citrix.com/external/article/235040/how-to-troubleshoot-xenserver-usb-passth.html

This article suggested running /opt/xensource/libexec/usb_scan.py with the -d parameter for additional details and that lead me to discover that the script fails when it encounters an empty line in usb-policy.conf

[23:46 xcp-ng-4 ~]# /opt/xensource/libexec/usb_scan.py -d
Traceback (most recent call last):
  File "/opt/xensource/libexec/usb_scan.py", line 681, in <module>
    pusbs = make_pusbs_list(devices, interfaces)
  File "/opt/xensource/libexec/usb_scan.py", line 660, in make_pusbs_list
    policy = Policy()
  File "/opt/xensource/libexec/usb_scan.py", line 384, in __init__
    self.parse_line(line)
  File "/opt/xensource/libexec/usb_scan.py", line 444, in parse_line
    if action.lower() == "allow":
UnboundLocalError: local variable 'action' referenced before assignment

After removing all empty lines, usb_scan.py scanned output properly, but it was giving me an empty array

[23:53 xcp-ng-4 ~]# /opt/xensource/libexec/usb_scan.py -d
[]

Since it as no longer crashing, I decided to go back to the default usb-policy.conf then try only adding my single allow rule without any extra lines and then test.

cp /etc/xensource/usb-policy.conf.default /etc/xensource/usb-policy.conf

I inserted a new line at line 10 just above the first rule and added my allow rule:

ALLOW:vid=1050 pid=0407

and that was all that was needed! Now I can see the device after running usb_scan.py

[23:53 xcp-ng-4 ~]# /opt/xensource/libexec/usb_scan.py -d
[{"path": "2-1.1", "version": "2.00", "vendor-id": "1050", "product-id": "0407", "vendor-desc": "Yubico.com", "product-desc": "Yubikey 4 OTP+U2F+CCID", "speed": "12", "serial": "", "description": "Yubico.com_Yubikey 4 OTP+U2F+CCID"}]

I also learned that the last good output of xe usb-scan seems to be cached somewhere and is quietly returned without hesitation when usb_policy.py fails. Maybe it's logged somewhere, but I don't know.

In any case, it was as simple as an empty line -- don't take anything for granted!

Lastly, I did a bit of testing to confirm that for my Yubikey to be detected and allowed, the allow rule must be BEFORE the rule that denies HID Boot Keyboards.

This results in detection:

# When you change this file, run 'xe pusb-scan' to confirm
# the file can be parsed correctly.
#
# Syntax is an ordered list of case insensitive rules where # is line comment
#  and each rule is (ALLOW | DENY) : ( match )*
#  and each match is (class|subclass|prot|vid|pid|rel) = hex-number
# Maximum hex value for class/subclass/prot is FF, and for vid/pid/rel is FFFF
#
# USB Hubs (class 09) are always denied, independently of the rules in this file
DENY: vid=17e9 # All DisplayLink USB displays
DENY: class=02 # Communications and CDC-Control
ALLOW:vid=056a pid=0315 class=03 # Wacom Intuos tablet
ALLOW:vid=056a pid=0314 class=03 # Wacom Intuos tablet
ALLOW:vid=056a pid=00fb class=03 # Wacom DTU tablet
# @jeff - allow passthrough of Yubikey 5 FIPS, "Yubikey 4 OTP+U2F+CCID"
ALLOW:vid=1050 pid=0407
DENY: class=03 subclass=01 prot=01 # HID Boot keyboards
DENY: class=03 subclass=01 prot=02 # HID Boot mice
DENY: class=0a # CDC-Data
DENY: class=0b # Smartcard
DENY: class=e0 # Wireless controller
DENY: class=ef subclass=04 # Miscellaneous network devices
ALLOW: # Otherwise allow everything else

This does not:

# When you change this file, run 'xe pusb-scan' to confirm
# the file can be parsed correctly.
#
# Syntax is an ordered list of case insensitive rules where # is line comment
#  and each rule is (ALLOW | DENY) : ( match )*
#  and each match is (class|subclass|prot|vid|pid|rel) = hex-number
# Maximum hex value for class/subclass/prot is FF, and for vid/pid/rel is FFFF
#
# USB Hubs (class 09) are always denied, independently of the rules in this file
DENY: vid=17e9 # All DisplayLink USB displays
DENY: class=02 # Communications and CDC-Control
ALLOW:vid=056a pid=0315 class=03 # Wacom Intuos tablet
ALLOW:vid=056a pid=0314 class=03 # Wacom Intuos tablet
ALLOW:vid=056a pid=00fb class=03 # Wacom DTU tablet
DENY: class=03 subclass=01 prot=01 # HID Boot keyboards
# @jeff - allow passthrough of Yubikey 5 FIPS, "Yubikey 4 OTP+U2F+CCID"
ALLOW:vid=1050 pid=0407
DENY: class=03 subclass=01 prot=02 # HID Boot mice
DENY: class=0a # CDC-Data
DENY: class=0b # Smartcard
DENY: class=e0 # Wireless controller
DENY: class=ef subclass=04 # Miscellaneous network devices
ALLOW: # Otherwise allow everything else

Thanks to everyone who took a look. Hopefully you don't get caught by this same gotcha!

For what it's worth, since you're building from sources, you might look into using a community tool that does most of the hard work for you:

• https://github.com/Jarli01/xenorchestra_installer
• https://github.com/ronivay/XenOrchestraInstallerUpdater (forked from above. I use this one in my homelab, personally)

These tools are given no official support from Vates as they are 3rd party tools, and as per Vates, you ought not be using the compiled from sources version in production, so YMMV..

IIRC Jarli01 is an active user on this forum but they have a different username here that I can't recall.

If nothing else, scripts like these make the process consistent which makes troubleshooting MUCH easier when things do go wrong.

Just a bit of food for thought.
Cheers

Does anyone have any suggestions for troubleshooting or investigating this further? I didn't really change my process that has worked with several updates but now things are behaving differently..

@knightjoel Thanks for the suggestion. Since my original message, I've tried moving my allow rule to the top, before any Deny rules, after any deny rules, I even tried experimenting with commenting all of the deny rules to see if any of those would make a difference; unfortunately none of them made a difference.

I've tried simply saving the file then initiating a xe pusb-scan on the host, I also tried rebooting to see if that would have an effect, but it doesn't seem to.

Hello,

As always, I am grateful for such an awesome open-source solution! Thank you all!

Asking regarding my home lab -- I have a VM whose disk is located on my master host's local storage and I've been using USB passthrough to pass a Yubikey to the VM.

I know that in the past, after an update, I often had to update /etc/xensource/usb-policy.conf to allow my YubiKey, reboot the host, then run xe pusb-scan host-uuid=<host-uuid> before I could configure passthrough of my yubikey.

I installed updates the other day, but as the weather was hot and I wasn't using my homelab, I shut everything down before any of my TLS certificates expired, so I didn't notice that the usb-policy.conf file had been overwritten by the update (I know that this is to be expected).

Today, I booted up my system and noticed the usb-policy.conf file had overwritten during the update, so I backed up /etc/xensource/usb-policy.conf, then added the line plus comments that has historically allowed my YubiKey to be passed through.

Original /etc/xensource/usb-policy.conf:

# When you change this file, run 'xe pusb-scan' to confirm
# the file can be parsed correctly.
#
# Syntax is an ordered list of case insensitive rules where # is line comment
#  and each rule is (ALLOW | DENY) : ( match )*
#  and each match is (class|subclass|prot|vid|pid|rel) = hex-number
# Maximum hex value for class/subclass/prot is FF, and for vid/pid/rel is FFFF
#
# USB Hubs (class 09) are always denied, independently of the rules in this file
DENY: vid=17e9 # All DisplayLink USB displays
DENY: class=02 # Communications and CDC-Control
ALLOW:vid=056a pid=0315 class=03 # Wacom Intuos tablet
ALLOW:vid=056a pid=0314 class=03 # Wacom Intuos tablet
ALLOW:vid=056a pid=00fb class=03 # Wacom DTU tablet
DENY: class=03 subclass=01 prot=01 # HID Boot keyboards
DENY: class=03 subclass=01 prot=02 # HID Boot mice
DENY: class=0a # CDC-Data
DENY: class=0b # Smartcard
DENY: class=e0 # Wireless controller
DENY: class=ef subclass=04 # Miscellaneous network devices
ALLOW: # Otherwise allow everything else

Updated /etc/xensource/usb-policy.conf

# When you change this file, run 'xe pusb-scan' to confirm
# the file can be parsed correctly.
#
# Syntax is an ordered list of case insensitive rules where # is line comment
#  and each rule is (ALLOW | DENY) : ( match )*
#  and each match is (class|subclass|prot|vid|pid|rel) = hex-number
# Maximum hex value for class/subclass/prot is FF, and for vid/pid/rel is FFFF
#
# USB Hubs (class 09) are always denied, independently of the rules in this file
DENY: vid=17e9 # All DisplayLink USB displays
DENY: class=02 # Communications and CDC-Control
ALLOW:vid=056a pid=0315 class=03 # Wacom Intuos tablet
ALLOW:vid=056a pid=0314 class=03 # Wacom Intuos tablet
ALLOW:vid=056a pid=00fb class=03 # Wacom DTU tablet
DENY: class=03 subclass=01 prot=01 # HID Boot keyboards
DENY: class=03 subclass=01 prot=02 # HID Boot mice
DENY: class=0a # CDC-Data
DENY: class=0b # Smartcard
DENY: class=e0 # Wireless controller
DENY: class=ef subclass=04 # Miscellaneous network devices

### Jeff
# YubiKey 5 FIPS Series PID 0x0407 - YubiKey OTP+FIDO+CCID
ALLOW: VID=1050 PID=0407

ALLOW: # Otherwise allow everything else

Today, however, the YubiKey won't show up as a PUSB device when viewing the host's advanced tab, nor is it in the list of available devices when I attempt to create a VUSB for my VM.

I have rebooted the system, I have run xe pusb-scan host-uuid=... for the appropriate host, I have physically disconnected and reconnected the YubiKey, I have powered down the host, then powered back on, but running xe pusb-list doesn't show the yubikey and I can't select it for passthrough.

When I run lsusb I do see the YubiKey listed (thoigh it detects it as a Yubikey 4 series instead of 5 series. Can't recall whether that's consistent with past behavior)

[14:03 xcp-ng-4 ~]# lsusb
Bus 002 Device 004: ID 1050:0407 Yubico.com Yubikey 4 OTP+U2F+CCID
Bus 002 Device 005: ID 413c:2113 Dell Computer Corp. 
Bus 002 Device 003: ID 0557:8021 ATEN International Co., Ltd Hub
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0557:2221 ATEN International Co., Ltd Winbond Hermon
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
[14:03 xcp-ng-4 ~]# 

Is there anything I'm missing? Any suggestions for where to look or what to check?

Thank you!

@olivierlambert - I'm not sure who would be the best person at Vates to ping or whether there is another channel I should be using to request enhancements. I'm happy to be directed to the correct place if that's not here.

Despite the fact that I brought this upon myself... I do think that it would be nice if Xen Orchestra could improve the error handling/messaging for situations where a task fails due to an invalid object UUID. It seems like the UI is already making a simple XAPI call to lookup the name-label of the SR, which, upon failure results in the schedule where an invalid/unknown UUID is configured displaying the invalid/unknown UUID in Red text with a red triangle.

@irtaza9 happy to help!

It's also not hard to "copy" a template from one pool to another. So if you create your "golden image" template, you can just copy that template to another pool.

You can see the template Intangible Debian Bookworm 12 (Cloud Init)_2023-09-26T21:48:00.318Z that I originally created in my "performance" pool, then later when I set up my "efficiency" pool, I simply copied to an SR in my "efficiency" pool.

In order for a pool to utilize a template, the template needs to be within one of the shared SRs within that pool. Once it has been copied to an SR in the destination pool, that pool can now create new VMs using that template.