I haven't tried Duo on a host myself, so this comes from reading an 8.3 dom0 rather than from experience.
The first thing I'd watch is that XAPI checks the root password through PAM as well. /etc/pam.d/xapi just includes system-auth, so my guess is that a Duo module placed in system-auth would make every password login to XAPI (XO's connection to the pool, xe -pw, XO Lite) ask for a factor it can't give.
The physical console is less simple than it looks too: tty1 runs xsconsole, and on my 8.3 host it checks root against the password-auth PAM service by handing over just the password, so I'd expect a module that asks a second question to make that login fail rather than prompt.
Also worth knowing before an audit: software from outside the XCP-ng repos isn't supported in dom0 (https://docs.xcp-ng.org/installation/requirements), and as far as I can tell from rpm, /etc/pam.d/xapi isn't marked as a config file, so an update would put it back.
Anyone who has been through this for real will know more than me.