USB Passthrough Override Script - to ensure usb-policy.conf consistency
-
Why don't you write your custom configuration somewhere in
/etc? It does not seem right to modify files under/opt, which is supposed to be untouched outside installation and update operations.As with any dom0 customizations, anyone using this script, make sure you remember that it is there, understand what it does, and be ready to debug it if doesn't work or breaks something. I don't really like it to see automated override of a configuration file by a cron job. What if we need to fix something really important in it, such as a security issue? Or it causes an issue that you don't immediately recognize as a consequence of the script, but then ask our support team to troubleshoot it?
A proper fix would be making XAPI include custom configuration from users, as I suggested to the XAPI project. You can share your use cases in the bug report, by the way: https://github.com/xapi-project/xen-api/issues/4935. Especially if they're in production setups.
-
Why don't you write your custom configuration somewhere in /etc?
Because updates seem to override my configurations. (I think we all are already aware of this issue)
It does not seem right to modify files under /opt, which is supposed to be untouched outside installation and update operations.
That's not true! The /opt folder in general in Linux is for user installed files(common files for all users)/ common applications which are generally not touched by OS updates so this is correctly placed as intended in this case.
Precisely the reason why this has been kept under /opt so that the updates don't over-write the files which are supposed to by already modified by the user !As with any dom0 customizations, anyone using this script, make sure you remember that it is there, understand what it does, and be ready to debug it if doesn't work or breaks something.
I know. This script's intension is to modify the config file in the usb-policy.conf under /etc/xensource folder. There is not much to remember, understand or debug as this a very very simple script which has been tested by me several times and I assure you there are 0 bugs ! This will hold true till the format of
/etc/xensource/usb-policy.confis not drastically changed (which seems very unlikely at this stage).What if we need to fix something really important in it, such as a security issue? Or it causes an issue that you don't immediately recognize as a consequence of the script, but then ask our support team to troubleshoot it?
Security issue ? Cause an issue that is not immediately recognized? It's almost impossible!
This particular file is just a list of allow & deny rights for USB. This script does NOTHING that the user himself/herself cannot do ! What would happen if the user changes the file manually and then comes to you for support ? You have already put this in your docs that the user should change this file if he wants pass-through of HID devices. And when this change is over-written without any warning, there is even more chance of the user approaching you for support. Atleast in this case you would know what exactly is being done!Infact I was hoping you guys would thank me for building this script so that you guys don't get support tickets due to wrong editing/ errors in editing
usb-policy.confmanually by the user!As far as my script is concerned, I have not come to you for support and you have my word that I will NEVER come to you for support for this particular script !
A proper fix would be making XAPI include custom configuration from users, as I suggested to the XAPI project. You can share your use cases in the bug report, by the way: https://github.com/xapi-project/xen-api/issues/4935. Especially if they're in production setups.
As I have already mentioned here (Please refer to the Disclaimer which is literally the first line of the post !)
And here also, that :
this is supposed to be a workaround until you guys fix this issue.Is this the best implementation ?
No ! Absolutely not ! But what can we do till you ponder and finalize it in this issue : https://github.com/xapi-project/xen-api/issues/4935
To be fair, its been about a year and still its being 'pondered' upon if it is needed !Right now you are just discussing the 'use-case' (which for some of us are already using), I am assuming this would be months and months of wait till you even reach to stage of bug-testing let alone the release. (which is ok, considering the work you guys are doing and I know you guys have even more priorities with other features so this would probably be taking a back-seat)
Till this is done, you really don't expect us to just not do anything at all ?
Please note that my intention with this post was to help people and contribute to XCP-ng in any and every way I can !
If I can help other users, then I don't see the reason not to... Everyone is free to post any questions/bugs they have regarding this here and I'll try my best to answer them !Now for the implementation part :
What you suggested is Absolutely the right way to go about it. I welcome your suggestion and to add to your suggestion, the best way should be that we create another 'Tab' in the Xen-Orchestra GUI with label as 'USB' in the Host section besides the 'Advanced' tab (or maybe besides the 'Storage' tab)
Inside the Tab, we can have a section called 'Filters' which allows the user to add/edit/delete the filters line by line and the
usb-policy.conffile gets generated and pushed by XAPI to the host. This file should be pushed at a place where is it not touched by updates. In case you want to update the base file due to any reason like security or update in APIs etc. , you can update the Xen Orchestra and the dom0 and then just re-build & re-push the user filters back using XAPI so that the user does not have to re-edit the filters every-time they update dom0. -
@gb-123 said in USB Passthrough Override Script - to ensure usb-policy.conf consistency:
Why don't you write your custom configuration somewhere in /etc?
Because updates seem to override my configurations. (I think we all are already aware of this issue)
First, it's not because some files can be overriden that any additional file you put into
/etcwill be.Then, XCP-ng is not your average Linux distribution open for full customization. Not all configuration files are meant to be modified by users. Whenever new needs arise, the philosophy is to expand the API and do changes through the API. Which makes them survive upgrades.
Somes files are not meant to be modified.
usb-policy.confwas one of them. It still isn't. We are inheriting decisions made initially by XAPI developers at XenServer, because it wasn't in their business need to offer the ability handle more USB devices. Temporarily, until a proper way to answer specific needs is developed, that your configuration is overriden is not a bug, it's a desirable feature for some users. Anything else is a workaround, and comes with its own limitations and risks.Your lenghty answer makes it sound like my warnings offended you after all the efforts you put in your script, and I can understand it, but it's my Release Manager / Lead Maintainer role to do so and discourage customizations or at least make it clear that they come with their set of disadvantages, notably regarding our ability to provide pro support.
It does not seem right to modify files under /opt, which is supposed to be untouched outside installation and update operations.
That's not true! The /opt folder in general in Linux is for user installed files(common files for all users)/ common applications which are generally not touched by OS updates so this is correctly placed as intended in this case.
Precisely the reason why this has been kept under /opt so that the updates don't over-write the files which are supposed to by already modified by the user !Per my understanding of the Filesystem Hierarchy Standard,
/opt/is for files installed by users for the whole system outside the package management system, but user configuration related to this belongs to/etc/opt/.As with any dom0 customizations, anyone using this script, make sure you remember that it is there, understand what it does, and be ready to debug it if doesn't work or breaks something.
I know. This script's intension is to modify the config file in the usb-policy.conf under /etc/xensource folder. There is not much to remember, understand or debug as this a very very simple script which has been tested by me several times and I assure you there are 0 bugs ! This will hold true till the format of
/etc/xensource/usb-policy.confis not drastically changed (which seems very unlikely at this stage).That there are bugs or not is not the point
What if we need to fix something really important in it, such as a security issue? Or it causes an issue that you don't immediately recognize as a consequence of the script, but then ask our support team to troubleshoot it?
Security issue ? Cause an issue that is not immediately recognized? It's almost impossible!
This particular file is just a list of allow & deny rights for USB. This script does NOTHING that the user himself/herself cannot do ! What would happen if the user changes the file manually and then comes to you for support ? You have already put this in your docs that the user should change this file if he wants pass-through of HID devices. And when this change is over-written without any warning, there is even more change of the user approaching you for support. Atleast in this case you would know what exactly is being done!I'm not saying your script is a security issue, I'm saying it's likely to override any security fixes we would do inside
usb-policy.conf. The probability is quite low, but it's not zero.Infact I was hoping you guys would thank me for building this script so that you guys don't get support tickets due to wrong editing/ errors in editing
usb-policy.confmanually by the user!We unfortunately get too many support tickets due to unwanted dom0 customizations for me to be thrilled by the perspective that many users that using a workaround script that is completely outside our control and (as far as I understood) is susceptible to overriding any future changes we make to the file. Or be used on the wrong version of XCP-ng in a few years. There are really many things to take into account as a maintainer.
As far as my script is concerned, I have not come to you for support and you have my word that I will NEVER come to you for support for this particular script !
I believe you. However my comments were targeted in priority towards users of your script, who don't master how it works as well as you do.
A proper fix would be making XAPI include custom configuration from users, as I suggested to the XAPI project. You can share your use cases in the bug report, by the way: https://github.com/xapi-project/xen-api/issues/4935. Especially if they're in production setups.
As I have already mentioned here (Please refer to the Disclaimer which is literally the first line of the post !)
And here also, that :
this is supposed to be a workaround until you guys fix this issue.I'm aware that you mentioned it and am grateful that you did. Still I had to play my maintainer role regarding other users.
Do you want an example of waster 2h30 min today for me? Someone applied something I myself put in a message of this forum while we were trying to debug an issue, even if I was very careful to mention something along "only do this for testing, because this will override system files". There's always one or more users who will try things, even when dangerous, because they have an issue and sometimes make things worse. I hope this helps understand my initial reaction, by which I stand.
Is this the best implementation ?
No ! Absolutely not ! But what can we do till you ponder and finalize it in this issue : https://github.com/xapi-project/xen-api/issues/4935
To be fair, its been about a year and still its being 'pondered' upon if it is needed !Right now you are just discussing the 'use-case' (which for some of us are already using), I am assuming this would be months and months of wait till you even reach to stage of bug-testing let alone the release. (which is ok, considering the work you guys are doing and I know you guys have even more priorities with other features so this would probably be taking a back-seat)
We're not discussing the use case between ourselves, we're making a point towards upstream XAPI developers who didn't see the need I took the time to explain to them on behalf of our users.
Yes, the feature is not present now. Yes, you have the right to write a workaround script. But as I said above, I must warn users again and again against such customizations. Then I have no doubts that they will use it anyway (and that, someday, this will consume time from our support or developers because they forgot they've done it, and they upgrade to a newer XCP-ng, which erases the changes in /opt, and then they open forum thread or support ticket because the new release is "broken").
Till this is done, you really don't expect us to just not do anything at all ?
Please note that my intention with this post was to help people and contribute to XCP-ng in any and every way I can !
If I can help other users, then I don't see the reason not to... Everyone is free to post any questions/bugs they have regarding this here and I'll try my best to answer them !Now for the implementation part :
What you suggested is Absolutely the right way to go about it. I welcome your suggestion and to add to your suggestion, the best way should be that we create another 'Tab' in the Xen-Orchestra GUI with label as 'USB' in the Host section besides the 'Advanced' tab (or maybe besides the 'Storage' tab)
Inside the Tab, we can have a section called 'Filters' which allows the user to add/edit/delete the filters line by line and the
usb-policy.conffile gets generated and pushed by XAPI to the host. This file should be pushed at a place where is it not touched by updates. In case you want to update the base file due to any reason like security or update in APIs etc. , you can update the Xen Orchestra and the dom0 and then just re-build & re-push the user filters back using XAPI so that the user does not have to re-edit the filters every-time they update dom0.The first step before changing Xen Orchestra is providing the appropriate API at the XAPI level, where XAPI will manage the USB configuration changes asked by users. Adding it to Xen Orchestra, which is a XAPI client, would then likely be rather simple.
-
First of all, I would like to thank you for your efforts in developing XCP-ng and my reply is not directed towards any objections / offence etc.
I am only trying to help everyone.
My following reply is also just a clarification of my understanding in the hopes that it will ultimately help the whole community.Somes files are not meant to be modified. usb-policy.conf was one of them. It still isn't.
Well this is a bit confusing. I am sorry but I was referring to your documentation here :
Passing through Keyboards & Mice
Where is is mentioned:
"xcp-ng host uses usb-policy.conf at /etc/xensource/usb-policy.conf ..... You can edit this file to allow these devices (and any other ones similarly)."Per my understanding of the Filesystem Hierarchy Standard, /opt/ is for files installed by users for the whole system outside the package management system, but user configuration related to this belongs to /etc/opt/.
You are right. But in my case the script itself is the "files installed by users for the whole system outside the package management system" and since my script is whole in itself and does not have any other configuration, I thought it was best to put in /opt as enumerated in your link, since I am treating my script as a whole package which includes uninstaller of the script also (which does not have configuration of the package (to put in /etc/opt), so to speak.)
And the best part is - You could simply remove my script and leave no 'crumbs' and have your system exactly like it was before installing my script simple by running my script with
--removecommand.We unfortunately get too many support tickets due to unwanted dom0 customizations for me to be thrilled by the perspective that many users that using a workaround script that is completely outside our control and (as far as I understood) is susceptible to overriding any future changes we make to the file. Or be used on the wrong version of XCP-ng in a few years. There are really many things to take into account as a maintainer.
I understand and very much appreciate your concern. Trust me I have gone through this.
I can therefore offer you this :- I will mention the XC-ng version this script is tested on in the first post.
- In case you guys change/plan to change anything in the usb-policy.config, just post the link to the new file and I'll change my script accordingly and post the new version alongwith the old one mentioning the version(s).
Another option:
If you include the script with your XCP-ng iso/installer, I can fork your repo in github and directly send you pull requests whenever you change the usb-policy.config so that the users would not need to install the wrong script even by mistake. And if you want further security, you could integrate this with XO so that the user can only edit the 'filter' portion of the file and not the header and footer. (Basically once XO builds and pushes the file, it sticks. Incase of change in base file, both XO & XCP-ng are changed together. Version mismatch can also be incorporated in the script to check that wrong versions are not over-written and user is given a warning so that they know the config has reverted to 'original'. This can also be replicated in XO-Lite since code base will be the same in both cases and therefore would conform to your XO6 standard in future.
^
Please note that the above is ofcourse free of charge as my dedication and contribution to help this community grow.We're not discussing the use case between ourselves, we're making a point towards upstream XAPI developers who didn't see the need I took the time to explain to them on behalf of our users.
I once again, very very much appreciate that you are doing this as usb is a critical part for others and me as well. I know this is used more in home-labs than in corporate environments but I believe there is enough userbase for this to make sense. So I very much appreciate that you are trying to make the upstream XAPI developers so that we can have a tighter and error free integration in future. However, as per my experience, I don't think that happening very soon as it is generally a herculean task to convince a big company. The smaller ones are more open to suggestion and development since they have a stronger drive to grow.
The first step before changing Xen Orchestra is providing the appropriate API at the XAPI level, where XAPI will manage the USB configuration changes asked by users. Adding it to Xen Orchestra, which is a XAPI client, would then likely be rather simple.
I very much agree with you and thats why I wrote 'in addition' to your suggestion. But as I earlier said, out of experience, I don't see that this happening very soon even though the best thing would be to incorporate this in XAPI as you suggested since it will be consistent with the whole software in general.
-
@stormi said in USB Passthrough Override Script - to ensure usb-policy.conf consistency:
Then I have no doubts that they will use it anyway (and that, someday, this will consume time from our support or developers because they forgot they've done it, and they upgrade to a newer XCP-ng, which erases the changes in /opt, and then they open forum thread or support ticket because the new release is "broken").
My intention was to help and not over-burden you guys. If you feel that this is going to create a havoc of 'tickets', I am taking this script down until you guys decide to use it in which case I would be happy to help.
PS: Once again, I am NOT taking this as an offence, just as a gesture so as not to bother the developers with unwanted tickets.
-
@gb-123 said in USB Passthrough Override Script - to ensure usb-policy.conf consistency:
Well this is a bit confusing. I am sorry but I was referring to your documentation here :
Passing through Keyboards & Mice
Where is is mentioned:
"xcp-ng host uses usb-policy.conf at /etc/xensource/usb-policy.conf ..... You can edit this file to allow these devices (and any other ones similarly)."Well spotted. Parts of the documentation are user-contributed, and despite we review them, our knowledge when this was written wasn't as good as what it is now. I would have asked that users avoid it or at least be aware of the risks (file may get overwritten), were I to review such a contribution today.
It would be a valuable contribution to update this part of the doc. There's a link at the bottom of the page that you can use to contribute changes.
We unfortunately get too many support tickets due to unwanted dom0 customizations for me to be thrilled by the perspective that many users that using a workaround script that is completely outside our control and (as far as I understood) is susceptible to overriding any future changes we make to the file. Or be used on the wrong version of XCP-ng in a few years. There are really many things to take into account as a maintainer.
I understand and very much appreciate your concern. Trust me I have gone through this.
I can therefore offer you this :- I will mention the XC-ng version this script is tested on in the first post.
- In case you guys change/plan to change anything in the usb-policy.config, just post the link to the new file and I'll change my script accordingly and post the new version alongwith the old one mentioning the version(s).
I can't promise we would remember that we need to notify you whenever it changes. This is rather something you could watch on your side. But anyway what is likely to happen is that users having installed the script will forget about it and not update it anyway.
Another option:
If you include the script with your XCP-ng iso/installer, I can fork your repo in github and directly send you pull requests whenever you change the usb-policy.config so that the users would not need to install the wrong script even by mistake. And if you want further security, you could integrate this with XO so that the user can only edit the 'filter' portion of the file and not the header and footer. (Basically once XO builds and pushes the file, it sticks. Incase of change in base file, both XO & XCP-ng are changed together. Version mismatch can also be incorporated in the script to check that wrong versions are not over-written and user is given a warning so that they know the config has reverted to 'original'. This can also be replicated in XO-Lite since code base will be the same in both cases and therefore would conform to your XO6 standard in future.
^
Please note that the above is ofcourse free of charge as my dedication and contribution to help this community grow.I have considered it, but I prefer to not take detours and rather go directly in the right direction.
So, the topic of factory config vs user config is not a new one. I have put a lot of energy personnally in improving things regarding
multipath.conf, for example, where now, if you look at it, you will see a warning plus informations on how to create your own configuration without overwriting the defaults.I'd like something similar, but this requires changes in the code of XAPI, which adds more steps. It's that much work, it's just a matter or priority, in a context where time is rare and precious. I have no doubt we'll get there eventually. That's also why it's important to share the use cases in the github issue, to also convince developers from XenServer.
What we could do short term would be diverging from XenServer's packaging of
usb-policy.confin two ways:- stop overwriting the file when it has customizations
- add a big warning on top of file stating that were you to modify it, you would not get updates of this file anymore lest you manually watch for
usb-policy.conf.rpmnewfiles to appear after an update, manually merge changes,, and not get support regarding USB matters anymore
This would achieve what your script does, in a much simpler way, but still give us some level of control. Configuration would still be overwritten when you upgrade to a newer release of XCP-ng, which whould also be mentioned in the warning.
We're not discussing the use case between ourselves, we're making a point towards upstream XAPI developers who didn't see the need I took the time to explain to them on behalf of our users.
I once again, very very much appreciate that you are doing this as usb is a critical part for others and me as well. I know this is used more in home-labs than in corporate environments but I believe there is enough userbase for this to make sense. So I very much appreciate that you are trying to make the upstream XAPI developers so that we can have a tighter and error free integration in future. However, as per my experience, I don't think that happening very soon as it is generally a herculean task to convince a big company. The smaller ones are more open to suggestion and development since they have a stronger drive to grow.
It is true that it is sometimes hard to work with XenServer, but we also have successes, and good work relations with XAPI developers. It's not the component that is the harder to make evolve in a common direction.
-
@stormi I have added a use case to the GitHub issue on the XAPI repository. One which is very likely to be happening possibly often as VMs can be used in software development release cycle, in this case building and digitally signing software.
I also added a link to a specific item which in the use case, which would need to be able to be passed through.
-
@stormi said in USB Passthrough Override Script - to ensure usb-policy.conf consistency:
What we could do short term would be diverging from XenServer's packaging of usb-policy.conf in two ways:
-stop overwriting the file when it has customizations
-add a big warning on top of file stating that were you to modify it, you would not get updates of this file anymore lest you manually watch for usb-policy.conf.rpmnew files to appear after an update, manually merge changes,, and not get support regarding USB matters anymore
This would achieve what your script does, in a much simpler way, but still give us some level of control. Configuration would still be overwritten when you upgrade to a newer release of XCP-ng, which whould also be mentioned in the warning.IMHO, this is not exactly a solution. User Information will still get overwritten and you would need to manage to store it elsewhere and copy paste it on every update.
The above solution is messy and I believe will cause even more tickets as user might 'forget' to re-install his changes and/or update manually.
Better solution IMHO, would be :
- Have a separate user config which probably says
#Start User configand ends with#End user config. - Build usb-policy.conf by removing default filters and injecting the above user config file at the appropriate place every-time you guys update and also re-build/inject when the file changes.
In the above way, the user-config will be saved and there should be no tickets stemming from this.
- Have a separate user config which probably says
-
I have another suggestion:
We can simply make a usb-policy.conf builder in XO; where the user can add/remove/edit the filters using GUI and the XO pushes the file after re-building it at every update ?
This should be simple enough and the base XCP-ng would not be touched at all (still be close to upstream Xen!
Of course the caveat is that the user has to update using XO; not directly run
yum updatesince in that case XO would not know. (We can build a manual push-override in XO for that like you do for VxLAN certificates)This is assuming pushing a custom file through XAPI is possible .....
-
@gb-123 said in USB Passthrough Override Script - to ensure usb-policy.conf consistency:
This is assuming pushing a custom file through XAPI is possible .....
That's the whole point : it isn't until we code support for it in XAPI.
-
I ran into the same issue (usb-policy.conf overwritten by update) and found this conversation. Sad to see that the script was removed, so I was not able to use/improve it and had to build something new.
You were saying something about cronjob not being ideal, this workaround is using systemd.path as trigger. The script also just re-inserts your change at the top of the file in case the VID/PID combination is missing, so possible security fixes are not un-fixed.
/root/modify-usb-policy.sh(or put it wherever seems reasonable for you):#!/bin/bash USBPOL=/etc/xensource/usb-policy.conf VID= # insert VendorId in hex PID= # insert ProductId in hex if ! grep -q "vid=${VID}.*pid=${PID}" "$USBPOL"; then sed -i "1s/^/ALLOW: vid=${VID} pid=${PID} # manually added\n/" "$USBPOL" fi/etc/systemd/system/modify-usb-policy.service[Unit] Description=Modify usb-policy.conf to restore custom changes [Service] Type=oneshot ExecStart=/root/modify-usb-policy.sh [Install] WantedBy=multi-user.target/etc/systemd/system/modify-usb-policy.path[Path] PathChanged=/etc/xensource/usb-policy.conf [Install] WantedBy=multi-user.targetEnable the path file:
systemctl enable modify-usb-policy.path systemctl start modify-usb-policy.pathLicense CC0
-
@zeropointer said in USB Passthrough Override Script - to ensure usb-policy.conf consistency:
I ran into the same issue (usb-policy.conf overwritten by update) and found this conversation. Sad to see that the script was removed, so I was not able to use/improve it and had to build something new.
You were saying something about cronjob not being ideal, this workaround is using systemd.path as trigger. The script also just re-inserts your change at the top of the file in case the VID/PID combination is missing, so possible security fixes are not un-fixed.
/root/modify-usb-policy.sh(or put it wherever seems reasonable for you):#!/bin/bash USBPOL=/etc/xensource/usb-policy.conf VID= # insert VendorId in hex PID= # insert ProductId in hex if ! grep -q "vid=${VID}.*pid=${PID}" "$USBPOL"; then sed -i "1s/^/ALLOW: vid=${VID} pid=${PID} # manually added\n/" "$USBPOL" fi/etc/systemd/system/modify-usb-policy.service[Unit] Description=Modify usb-policy.conf to restore custom changes [Service] Type=oneshot ExecStart=/root/modify-usb-policy.sh [Install] WantedBy=multi-user.target/etc/systemd/system/modify-usb-policy.path[Path] PathChanged=/etc/xensource/usb-policy.conf [Install] WantedBy=multi-user.targetEnable the path file:
systemctl enable modify-usb-policy.path systemctl start modify-usb-policy.pathLicense CC0
This is no longer needed if you have Xen Orchestra version 5.93 or later, alternatively if using in home lab and from source then any commit dated after 29th March 2024 (or current head on GitHub Main - Master). The reason being that Xen Orchestra from version 5.93 has official support for USB Passthrough and later from version 5.95 you can do PCI Passthrough.
https://xen-orchestra.com/blog/xen-orchestra-5-93/
https://xen-orchestra.com/blog/xen-orchestra-5-95/ -
@john-c Nice to see progress there, I did not update to that version yet. Did you already check if the shown USB devices are independent of the usb-policy.conf (so that you see really all devices there)? The blogpost did not get specific about this detail. As @stormi stated above, controlling the usb-policy is not available via XAPI for now.
