USB Passthrough Override Script - to ensure usb-policy.conf consistency

gb.123

First of all, I would like to thank you for your efforts in developing XCP-ng and my reply is not directed towards any objections / offence etc.
I am only trying to help everyone.
My following reply is also just a clarification of my understanding in the hopes that it will ultimately help the whole community.

Somes files are not meant to be modified. usb-policy.conf was one of them. It still isn't.

Well this is a bit confusing. I am sorry but I was referring to your documentation here :
Passing through Keyboards & Mice
Where is is mentioned:
"xcp-ng host uses usb-policy.conf at /etc/xensource/usb-policy.conf ..... You can edit this file to allow these devices (and any other ones similarly)."

Per my understanding of the Filesystem Hierarchy Standard, /opt/ is for files installed by users for the whole system outside the package management system, but user configuration related to this belongs to /etc/opt/.

You are right. But in my case the script itself is the "files installed by users for the whole system outside the package management system" and since my script is whole in itself and does not have any other configuration, I thought it was best to put in /opt as enumerated in your link, since I am treating my script as a whole package which includes uninstaller of the script also (which does not have configuration of the package (to put in /etc/opt), so to speak.)

And the best part is - You could simply remove my script and leave no 'crumbs' and have your system exactly like it was before installing my script simple by running my script with --remove command.

We unfortunately get too many support tickets due to unwanted dom0 customizations for me to be thrilled by the perspective that many users that using a workaround script that is completely outside our control and (as far as I understood) is susceptible to overriding any future changes we make to the file. Or be used on the wrong version of XCP-ng in a few years. There are really many things to take into account as a maintainer.

I understand and very much appreciate your concern. Trust me I have gone through this.
I can therefore offer you this :

I will mention the XC-ng version this script is tested on in the first post.
In case you guys change/plan to change anything in the usb-policy.config, just post the link to the new file and I'll change my script accordingly and post the new version alongwith the old one mentioning the version(s).

Another option:
If you include the script with your XCP-ng iso/installer, I can fork your repo in github and directly send you pull requests whenever you change the usb-policy.config so that the users would not need to install the wrong script even by mistake. And if you want further security, you could integrate this with XO so that the user can only edit the 'filter' portion of the file and not the header and footer. (Basically once XO builds and pushes the file, it sticks. Incase of change in base file, both XO & XCP-ng are changed together. Version mismatch can also be incorporated in the script to check that wrong versions are not over-written and user is given a warning so that they know the config has reverted to 'original'. This can also be replicated in XO-Lite since code base will be the same in both cases and therefore would conform to your XO6 standard in future.
^
Please note that the above is ofcourse free of charge as my dedication and contribution to help this community grow.

We're not discussing the use case between ourselves, we're making a point towards upstream XAPI developers who didn't see the need I took the time to explain to them on behalf of our users.

I once again, very very much appreciate that you are doing this as usb is a critical part for others and me as well. I know this is used more in home-labs than in corporate environments but I believe there is enough userbase for this to make sense. So I very much appreciate that you are trying to make the upstream XAPI developers so that we can have a tighter and error free integration in future. However, as per my experience, I don't think that happening very soon as it is generally a herculean task to convince a big company. The smaller ones are more open to suggestion and development since they have a stronger drive to grow.

The first step before changing Xen Orchestra is providing the appropriate API at the XAPI level, where XAPI will manage the USB configuration changes asked by users. Adding it to Xen Orchestra, which is a XAPI client, would then likely be rather simple.

I very much agree with you and thats why I wrote 'in addition' to your suggestion. But as I earlier said, out of experience, I don't see that this happening very soon even though the best thing would be to incorporate this in XAPI as you suggested since it will be consistent with the whole software in general.

gb.123

@stormi said in USB Passthrough Override Script - to ensure usb-policy.conf consistency:

Then I have no doubts that they will use it anyway (and that, someday, this will consume time from our support or developers because they forgot they've done it, and they upgrade to a newer XCP-ng, which erases the changes in /opt, and then they open forum thread or support ticket because the new release is "broken").

My intention was to help and not over-burden you guys. If you feel that this is going to create a havoc of 'tickets', I am taking this script down until you guys decide to use it in which case I would be happy to help.

PS: Once again, I am NOT taking this as an offence, just as a gesture so as not to bother the developers with unwanted tickets.

stormi

@gb-123 said in USB Passthrough Override Script - to ensure usb-policy.conf consistency:

@stormi

Well this is a bit confusing. I am sorry but I was referring to your documentation here :
Passing through Keyboards & Mice
Where is is mentioned:
"xcp-ng host uses usb-policy.conf at /etc/xensource/usb-policy.conf ..... You can edit this file to allow these devices (and any other ones similarly)."

Well spotted. Parts of the documentation are user-contributed, and despite we review them, our knowledge when this was written wasn't as good as what it is now. I would have asked that users avoid it or at least be aware of the risks (file may get overwritten), were I to review such a contribution today.

It would be a valuable contribution to update this part of the doc. There's a link at the bottom of the page that you can use to contribute changes.

We unfortunately get too many support tickets due to unwanted dom0 customizations for me to be thrilled by the perspective that many users that using a workaround script that is completely outside our control and (as far as I understood) is susceptible to overriding any future changes we make to the file. Or be used on the wrong version of XCP-ng in a few years. There are really many things to take into account as a maintainer.

I understand and very much appreciate your concern. Trust me I have gone through this.
I can therefore offer you this :

I will mention the XC-ng version this script is tested on in the first post.

In case you guys change/plan to change anything in the usb-policy.config, just post the link to the new file and I'll change my script accordingly and post the new version alongwith the old one mentioning the version(s).

I can't promise we would remember that we need to notify you whenever it changes. This is rather something you could watch on your side. But anyway what is likely to happen is that users having installed the script will forget about it and not update it anyway.

Another option:
If you include the script with your XCP-ng iso/installer, I can fork your repo in github and directly send you pull requests whenever you change the usb-policy.config so that the users would not need to install the wrong script even by mistake. And if you want further security, you could integrate this with XO so that the user can only edit the 'filter' portion of the file and not the header and footer. (Basically once XO builds and pushes the file, it sticks. Incase of change in base file, both XO & XCP-ng are changed together. Version mismatch can also be incorporated in the script to check that wrong versions are not over-written and user is given a warning so that they know the config has reverted to 'original'. This can also be replicated in XO-Lite since code base will be the same in both cases and therefore would conform to your XO6 standard in future.
^
Please note that the above is ofcourse free of charge as my dedication and contribution to help this community grow.

I have considered it, but I prefer to not take detours and rather go directly in the right direction.

So, the topic of factory config vs user config is not a new one. I have put a lot of energy personnally in improving things regarding multipath.conf, for example, where now, if you look at it, you will see a warning plus informations on how to create your own configuration without overwriting the defaults.

I'd like something similar, but this requires changes in the code of XAPI, which adds more steps. It's that much work, it's just a matter or priority, in a context where time is rare and precious. I have no doubt we'll get there eventually. That's also why it's important to share the use cases in the github issue, to also convince developers from XenServer.

What we could do short term would be diverging from XenServer's packaging of usb-policy.conf in two ways:

stop overwriting the file when it has customizations
add a big warning on top of file stating that were you to modify it, you would not get updates of this file anymore lest you manually watch for usb-policy.conf.rpmnew files to appear after an update, manually merge changes,, and not get support regarding USB matters anymore

This would achieve what your script does, in a much simpler way, but still give us some level of control. Configuration would still be overwritten when you upgrade to a newer release of XCP-ng, which whould also be mentioned in the warning.

We're not discussing the use case between ourselves, we're making a point towards upstream XAPI developers who didn't see the need I took the time to explain to them on behalf of our users.

I once again, very very much appreciate that you are doing this as usb is a critical part for others and me as well. I know this is used more in home-labs than in corporate environments but I believe there is enough userbase for this to make sense. So I very much appreciate that you are trying to make the upstream XAPI developers so that we can have a tighter and error free integration in future. However, as per my experience, I don't think that happening very soon as it is generally a herculean task to convince a big company. The smaller ones are more open to suggestion and development since they have a stronger drive to grow.

It is true that it is sometimes hard to work with XenServer, but we also have successes, and good work relations with XAPI developers. It's not the component that is the harder to make evolve in a common direction.

john.c

@stormi I have added a use case to the GitHub issue on the XAPI repository. One which is very likely to be happening possibly often as VMs can be used in software development release cycle, in this case building and digitally signing software.

I also added a link to a specific item which in the use case, which would need to be able to be passed through.

gb.123

@stormi said in USB Passthrough Override Script - to ensure usb-policy.conf consistency:

What we could do short term would be diverging from XenServer's packaging of usb-policy.conf in two ways:

-stop overwriting the file when it has customizations
-add a big warning on top of file stating that were you to modify it, you would not get updates of this file anymore lest you manually watch for usb-policy.conf.rpmnew files to appear after an update, manually merge changes,, and not get support regarding USB matters anymore
This would achieve what your script does, in a much simpler way, but still give us some level of control. Configuration would still be overwritten when you upgrade to a newer release of XCP-ng, which whould also be mentioned in the warning.

IMHO, this is not exactly a solution. User Information will still get overwritten and you would need to manage to store it elsewhere and copy paste it on every update.

The above solution is messy and I believe will cause even more tickets as user might 'forget' to re-install his changes and/or update manually.

Better solution IMHO, would be :

Have a separate user config which probably says #Start User config and ends with #End user config.
Build usb-policy.conf by removing default filters and injecting the above user config file at the appropriate place every-time you guys update and also re-build/inject when the file changes.

In the above way, the user-config will be saved and there should be no tickets stemming from this.

gb.123

@stormi

I have another suggestion:

We can simply make a usb-policy.conf builder in XO; where the user can add/remove/edit the filters using GUI and the XO pushes the file after re-building it at every update ?

This should be simple enough and the base XCP-ng would not be touched at all (still be close to upstream Xen!

Of course the caveat is that the user has to update using XO; not directly run yum update since in that case XO would not know. (We can build a manual push-override in XO for that like you do for VxLAN certificates)

This is assuming pushing a custom file through XAPI is possible .....

stormi

@gb-123 said in USB Passthrough Override Script - to ensure usb-policy.conf consistency:

This is assuming pushing a custom file through XAPI is possible .....

That's the whole point : it isn't until we code support for it in XAPI.

zeropointer

I ran into the same issue (usb-policy.conf overwritten by update) and found this conversation. Sad to see that the script was removed, so I was not able to use/improve it and had to build something new.

You were saying something about cronjob not being ideal, this workaround is using systemd.path as trigger. The script also just re-inserts your change at the top of the file in case the VID/PID combination is missing, so possible security fixes are not un-fixed.

/root/modify-usb-policy.sh (or put it wherever seems reasonable for you):

#!/bin/bash 
USBPOL=/etc/xensource/usb-policy.conf
VID= # insert VendorId in hex
PID= # insert ProductId in hex
if ! grep -q "vid=${VID}.*pid=${PID}" "$USBPOL"; then
  sed -i "1s/^/ALLOW: vid=${VID} pid=${PID} # manually added\n/" "$USBPOL"
fi

/etc/systemd/system/modify-usb-policy.service

[Unit]
Description=Modify usb-policy.conf to restore custom changes

[Service]
Type=oneshot
ExecStart=/root/modify-usb-policy.sh

[Install]
WantedBy=multi-user.target

/etc/systemd/system/modify-usb-policy.path

[Path]
PathChanged=/etc/xensource/usb-policy.conf

[Install]
WantedBy=multi-user.target

Enable the path file:

systemctl enable modify-usb-policy.path
systemctl start modify-usb-policy.path

License CC0

john.c

@zeropointer said in USB Passthrough Override Script - to ensure usb-policy.conf consistency:

I ran into the same issue (usb-policy.conf overwritten by update) and found this conversation. Sad to see that the script was removed, so I was not able to use/improve it and had to build something new.

You were saying something about cronjob not being ideal, this workaround is using systemd.path as trigger. The script also just re-inserts your change at the top of the file in case the VID/PID combination is missing, so possible security fixes are not un-fixed.

/root/modify-usb-policy.sh (or put it wherever seems reasonable for you):
#!/bin/bash 
USBPOL=/etc/xensource/usb-policy.conf
VID= # insert VendorId in hex
PID= # insert ProductId in hex
if ! grep -q "vid=${VID}.*pid=${PID}" "$USBPOL"; then
  sed -i "1s/^/ALLOW: vid=${VID} pid=${PID} # manually added\n/" "$USBPOL"
fi
/etc/systemd/system/modify-usb-policy.service
[Unit]
Description=Modify usb-policy.conf to restore custom changes

[Service]
Type=oneshot
ExecStart=/root/modify-usb-policy.sh

[Install]
WantedBy=multi-user.target
/etc/systemd/system/modify-usb-policy.path
[Path]
PathChanged=/etc/xensource/usb-policy.conf

[Install]
WantedBy=multi-user.target
Enable the path file:
systemctl enable modify-usb-policy.path
systemctl start modify-usb-policy.path
License CC0

This is no longer needed if you have Xen Orchestra version 5.93 or later, alternatively if using in home lab and from source then any commit dated after 29th March 2024 (or current head on GitHub Main - Master). The reason being that Xen Orchestra from version 5.93 has official support for USB Passthrough and later from version 5.95 you can do PCI Passthrough.

https://xen-orchestra.com/blog/xen-orchestra-5-93/
https://xen-orchestra.com/blog/xen-orchestra-5-95/

zeropointer

@john-c Nice to see progress there, I did not update to that version yet. Did you already check if the shown USB devices are independent of the usb-policy.conf (so that you see really all devices there)? The blogpost did not get specific about this detail. As @stormi stated above, controlling the usb-policy is not available via XAPI for now.