Some update:

After removing the ip-address from eth4 the following messages disappeared

 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=30408.
[info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 

By the way I forgot to mention that each time I enable HA the following warning appears in xha.log

Sep 29 13:31:56 CEST 2026 [warn] BM: cannot open bonding status file (/proc/net/bonding/bond0). (2)
Sep 29 13:31:56 CEST 2026 [info] BM: this is not bonded. Terminating bonding thread.

bond0 (eth2 + eth3) is the management interface and also used for the HA.

@bvitnik I think there is some confusing.
During the initial installation we have have used the eth4 (172.28.4.11) as the management interface. Later on we created a bond with eth2 and eth3 and changed the management interface to this bond using the xsconsole, and also used the same ip-address (172.28.4.11). We thought that the IP address had also been removed from eth4 during the reconfiguration, but we didn't check it afterwards. It turns out that this was not the case.
So now I removed the ip from eth4 using the xe pif-reconfigure-ip command.

Next step is to use the sysctl workaround @john.c mentioned in an earlier post and enable HA again.

(I also think it would be wise to use dedicated bond for the heartbeat.)

@john.c We are using a LCAP bond.
For my understanding, if I remove the IP settings from eth4, the problem will still not be solved because of the known issue with the bond that you explained in “Weak Host Model & HA Heartbeat Failures.”
(At the moment we have HA disabled.)

@john.c now I remember, during the installation we used the eth4 as the management interface.

[16:54 dacshyp002 ~]# cat /etc/firstboot.d/data/management.conf 
LABEL='eth4'
MODE='static'
IP='172.28.4.11'
NETMASK='255.255.252.0'
GATEWAY='172.28.4.1'
MODEV6='none'
DNS='8.8.8.8'

Afterwords we created a bond bond0 with eth2/eth3 and used this as the management interface.
The xe pif-list command shows the following:

device ( RO)                   : bond0
               management ( RO): true
    IP-configuration-mode ( RO): Static
                       IP ( RO): 172.28.4.11

device ( RO)                   : eth4
               management ( RO): false
    IP-configuration-mode ( RO): Static
                       IP ( RO): 172.28.4.11

Is it that easy to remove the IP address from the eth4 interface? Because it make no sense to have duplicate ip-addresses.
I think we've had duplicate IP addresses all along, but we only noticed the issue after enabling HA.
By the way what is the correct way to change the management interface? We used the xsconsole.

@tjkreidl

According to xsconsole the management interface is bond0

Current Management Interface          
                                              
Device           bond0                
MAC Address	 30:3e:a7:1d:b0:90    
DHCP/Static IP   Static               
IP address       172.28.4.11          
Netmask          255.255.252.0        
Gateway          172.28.4.1           
Hostname         dacshyp002  

According the output of xe pif-list

uuid ( RO)                  : 0761d887-268e-5cf0-401b-a08ad7c419aa
                device ( RO): bond0
                   MAC ( RO): 30:3e:a7:1d:b0:90
    currently-attached ( RO): true
                  VLAN ( RO): -1
          network-uuid ( RO): 28d51793-0226-cf3f-75d3-ed7c5a9d8b33
             host-uuid ( RO): 41a2a448-a5dc-44c6-be44-c07540d75c60

Let's check the network

# xe network-list uuid=28d51793-0226-cf3f-75d3-ed7c5a9d8b33
uuid ( RO)                : 28d51793-0226-cf3f-75d3-ed7c5a9d8b33
          name-label ( RW): mgmt-bond
    name-description ( RW): 
              bridge ( RO): xapi1

At linux level:

# ip addr show xapi1
15: xapi1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 30:3e:a7:1d:b0:90 brd ff:ff:ff:ff:ff:ff
    inet 172.28.4.11/22 brd 172.28.7.255 scope global xapi1
       valid_lft forever preferred_lft forever

xapi1 is a bond of et2 and eth3. So far so good.
But if I am running the xe-reset-networking command

Your network will be re-configured as follows:

Management interface:   eth4
Reset interface name rules: Yes
IP configuration mode:  dhcp
IPv6 configuration mode:none

If you want to change any of the above settings, type 'no' and re-run
the command with appropriate arguments (use --help for a list of options).

Type 'yes' to continue.
Type 'no' to cancel.
no

So for some reason xcp-ng thinks the eth4 is management interface, which also has the ip-address 172.28.4.11/22 .
By the way --reset-primary is not a valid option:

# xe-reset-networking --reset-primary
Usage: xe-reset-networking [options]

xe-reset-networking: error: no such option: --reset-primary

I did some further investigation and I think there is something wrong with the ip-settings of the management interface.

At the xsconsole I see the following:

Management Network Parameters    

Device        bond0         
address       172.28.4.11    
Netmask       255.255.252.0   
Gateway       172.28.4.1

So it's using the bond0 interface.

# xe pif-list host-uuid=41a2a448-a5dc-44c6-be44-c07540d75c60

uuid ( RO)                  : 0761d887-268e-5cf0-401b-a08ad7c419aa
                device ( RO): bond0
                   MAC ( RO): 30:3e:a7:1d:b0:90
    currently-attached ( RO): true
                  VLAN ( RO): -1
          network-uuid ( RO): 28d51793-0226-cf3f-75d3-ed7c5a9d8b33
             host-uuid ( RO): 41a2a448-a5dc-44c6-be44-c07540d75c60

This is part of the output of xe bond-list

# xe bond-list
uuid ( RO)      : 79c525c3-22f9-04ce-4a80-21d18ef65ebc
    master ( RO): 0761d887-268e-5cf0-401b-a08ad7c419aa
    slaves ( RO): 7ac18886-e0a5-057b-bac0-36b909588dba ; 2f18fc9a-8c1c-dfce-9416-fa8c657c5f63 

I already checked that:

7ac18886-e0a5-057b-bac0-36b909588dba --> eth3
2f18fc9a-8c1c-dfce-9416-fa8c657c5f63 --> eth2

And now the confusing part. This is part of the output of the ip a command:

2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 30:3e:a7:1d:b0:90 brd ff:ff:ff:ff:ff:ff
3: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 30:3e:a7:1d:b0:91 brd ff:ff:ff:ff:ff:ff
4: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 30:3e:a7:1d:b0:92 brd ff:ff:ff:ff:ff:ff


13: xenbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 30:3e:a7:1d:b0:92 brd ff:ff:ff:ff:ff:ff
    inet 172.28.4.11/22 brd 172.28.7.255 scope global xenbr4
       valid_lft forever preferred_lft forever

As you can see the xenbr4 has also the management ip-address and has the same mac-address as the eth4 interface.
OK check the following commands:

[17:21 dacshyp002 ~]# ip addr show xapi1
15: xapi1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 30:3e:a7:1d:b0:90 brd ff:ff:ff:ff:ff:ff
    inet 172.28.4.11/22 brd 172.28.7.255 scope global xapi1
       valid_lft forever preferred_lft forever
[17:21 dacshyp002 ~]# ip addr show xenbr4
13: xenbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 30:3e:a7:1d:b0:92 brd ff:ff:ff:ff:ff:ff
    inet 172.28.4.11/22 brd 172.28.7.255 scope global xenbr4
       valid_lft forever preferred_lft forever

And If I look at the routing table:

# ip route
default via 172.28.4.1 dev xapi1 
172.18.8.0/23 dev xenbr0 proto kernel scope link src 172.18.8.11 
172.18.10.0/23 dev xenbr7 proto kernel scope link src 172.18.10.11 
172.28.4.0/22 dev xenbr4 proto kernel scope link src 172.28.4.11 
172.28.4.0/22 dev xapi1 proto kernel scope link src 172.28.4.11 

According to me the node has 2 interfaces xenbr4 / xapi1 with the same ip-address and also 2 different routes for the same ip-range 172.28.4.0/22.
And that's why the heartbeat has some issues. Can someone confirm this ?
By the way I really don't know how this happened.

I also checked the xha.log on the dacshyp002:

Sep 21 11:39:49 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=30408.
Sep 21 11:40:09 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:40:29 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=31466.
Sep 21 11:40:50 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:46:30 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=35178.
Sep 21 11:46:50 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:47:10 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=33233.
Sep 21 11:47:30 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:55:51 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=40853.
Sep 21 11:56:11 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:56:31 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=32914.
Sep 21 11:56:51 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:59:31 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=33377.
Sep 21 11:59:51 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:01:51 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=38588.
Sep 21 12:02:12 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:10:12 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=38125.
Sep 21 12:10:12 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=38323.
Sep 21 12:10:32 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:19:33 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=43934.
Sep 21 12:19:53 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:20:13 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=32989.
Sep 21 12:20:33 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:21:13 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=42077.
Sep 21 12:21:33 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:21:53 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=31325.
Sep 21 12:22:13 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:25:54 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=34497.
Sep 21 12:26:14 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:26:34 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=32558.
Sep 21 12:26:54 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:27:14 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=30616.
Sep 21 12:27:34 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:31:14 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=33975.
Sep 21 12:31:34 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:38:15 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=34797.
Sep 21 12:38:35 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:53:56 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=37000.
Sep 21 12:53:56 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=40176.
Sep 21 12:54:16 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:56:16 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=36380.
Sep 21 12:56:36 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:56:57 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=31439.
Sep 21 12:57:17 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 

I can conclude the heartbeat communication is generally unstable in my HA pool. Tomorrow I check the logging of our switch/router.

dacshyp003 booted at

[Mon Sep 21 05:44:37 2026] Linux version 4.19.0+1 (mockbuild@1efe98ffea2245bcb4c11893890153ac) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)) #1 SMP Thu Aug 13 13:06:11 UTC 2026

This is the part of the xensource logging of the dacshyp001 before the reboot:

Sep 21 11:17:52 dacshyp001 xapi: [debug||17 ha_monitor|HA monitor D:58d1715e0726|xapi_ha] The node we think is the master is still alive and marked as master; this is OK
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

Same behavior on dacshyp003.
I also attached the xha.log from both nodes.

xha_log_001.txt xha_log_003.txt

Hi all,

I need some help regarding HA.
I have a 3 node cluster running xcp-ng 8.3 (002 is the master)

[13:55 dacshyp002 ~]# xe host-list
uuid ( RO)                : 6b99f1ab-6f4c-4a8d-b766-d16a8a942bdf
          name-label ( RW): dacshyp003
    name-description ( RW): Default install


uuid ( RO)                : d99e150c-079a-4092-8909-ad1a36e07dec
          name-label ( RW): dacshyp001
    name-description ( RW): Default install


uuid ( RO)                : 41a2a448-a5dc-44c6-be44-c07540d75c60
          name-label ( RW): dacshyp002
    name-description ( RW): Default install

Last Thursday I enabled HA on the pool and also on some om the VM's. On Friday I also did an "rolling pool update"
Today I had a "spontaneous" reboot of the dacshyp001 and dacshyp003.
Boot time of dacshyp001:

[Mon Sep 21 11:21:31 2026] Linux version 4.19.0+1 (mockbuild@1efe98ffea2245bcb4c11893890153ac) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)) #1 SMP Thu Aug 13 13:06:11 UTC 2026

Around that time I also see the following message in the xensource.log:

Sep 21 11:21:08 dacshyp002 xapi: [debug||20398 ha_monitor|HA monitor D:7b5068fa6bfc|xapi_ha_vm_failover] Setting host dacshyp001 to dead
Sep 21 11:21:28 dacshyp002 xapi: [debug||20398 ha_monitor|HA monitor D:7b5068fa6bfc|xapi_ha_vm_failover] Setting host dacshyp001 to dead
Sep 21 11:21:48 dacshyp002 xapi: [debug||20398 ha_monitor|HA monitor D:7b5068fa6bfc|xapi_ha_vm_failover] Setting host dacshyp001 to dead

I will attached all logging around this time.
I am trying to understand what triggered the reboot and how I can troubleshoot this.

I am running XO community commit 6a441 . I am aware this is not the latest version. Before upgrading I want to know the cause of the reboot.

Also ignore the error:

/var/lib/xcp/xapi|dispatch:VDI.get_by_uuid D:1f5b4c8d329e|backtrace] VDI.get_by_uuid D:9df5c0255224 failed with exception Db_exn.Read_missing_uuid("VDI", "", "9219113f-65e4-4368-b42d-1b8bdc7614a9")

I will create another ticket for this.
HA-problem.txt
Thanks in advance.
Carlo

@poddingue https://github.com/vatesfr/xen-orchestra/issues/9779

Is there some update? Can this issue be reproduced?

@ataxyanetwork I managed to install XOA (Current version: 6.2.2 - XOA build: 20251219) and did some more testing.

I am experiencing the same issue when trying to restore from an existing backup.

When I create a new backup job, I can perform a filelevel restore without any problems. However, when I rerun the job, I am no longer able to restore from the most recent backup. It only works with the initial backup.

Can you also reproduce this ?

@AtaxyaNetwork This is not going to work because our nodes don't have access to the internet.
If it works using XOA, what conclusion can we draw? Migrate to XOA ?

@AtaxyaNetwork Thank you for the quick response.
"Are you running an XO source with LVM for the OS ?" --> no

You did the restore using XOA instead of XO from source ?
To Install XOA I have to install a new instance and import the config of my current XO ?
I will test this tomorrow.

@AtaxyaNetwork Sorry forgot to add this to my topic.

XOA - commit 3ed8c
VM's are based on Debain13
UEFI
Backup mode: Delta backup

~# lsblk -f 
NAME              FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
xvda                                                                                               
├─xvda1           vfat        FAT32          0E7D-17B9                               965.3M     1% /boot/efi
├─xvda2           ext4        1.0            bf2a2e01-b9ff-4b93-a824-ba7562461c15      1.3G    20% /boot
├─xvda3           LVM2_member LVM2 001       v2o7yJ-51Hc-sO1m-ubrj-YCvP-7Zo2-zOjxzZ                
│ ├─vgdata-lvroot ext4        1.0            d0e71b5a-5404-4b18-9249-eb222a6d346d     22.6G     6% /
│ ├─vgdata-lvhome ext4        1.0            5aa25f3d-708d-4708-8405-a97391e2e989      4.3G     0% /home
│ └─vgdata-lvvar  ext4        1.0            e82f5584-810b-40e2-a6e6-ed05805146db      3.5G    30% /var
└─xvda4           swap        1              6124f717-58cf-47a8-b268-4c99dcd495d0                  [SWAP]

Hi all,

First of all, I am aware that there are still issues with file-level restores and LVM. However, for some unknown reason, I am now encountering a new error when selecting the disk during the restore procedure.- Error while scanning disk

Apr 16 11:08:34 dacsvm-xoa xo-server[3124716]: 2026-04-16T09:08:34.366Z xo:api WARN admin | backupNg.listPartitions(...) [214ms] =!> Error: Command failed: vgchange -an vgdata
Apr 16 11:08:34 dacsvm-xoa xo-server[3124716]: File descriptor 23 (/var/lib/xo-server/data/leveldb/LOG) leaked on vgchange invocation. Parent PID 3124716: node
Apr 16 11:08:34 dacsvm-xoa xo-server[3124716]: File descriptor 24 (/var/lib/xo-server/data/leveldb/LOCK) leaked on vgchange invocation. Parent PID 3124716: node
Apr 16 11:08:34 dacsvm-xoa xo-server[3124716]: File descriptor 25 (/var/lib/xo-server/data/leveldb/000379.log) leaked on vgchange invocation. Parent PID 3124716: node
Apr 16 11:08:34 dacsvm-xoa xo-server[3124716]: File descriptor 28 (/var/lib/xo-server/data/leveldb/MANIFEST-000377) leaked on vgchange invocation. Parent PID 3124716: node
Apr 16 11:08:34 dacsvm-xoa xo-server[3124716]: File descriptor 37 (/dev/fuse) leaked on vgchange invocation. Parent PID 3124716: node
Apr 16 11:08:34 dacsvm-xoa xo-server[3124716]:   WARNING: Not using device /dev/loop0 for PV v2o7yJ-51Hc-sO1m-ubrj-YCvP-7Zo2-zOjxzZ.
Apr 16 11:08:34 dacsvm-xoa xo-server[3124716]:   WARNING: PV v2o7yJ-51Hc-sO1m-ubrj-YCvP-7Zo2-zOjxzZ prefers device /dev/xvda3 because device is used by LV.
Apr 16 11:08:34 dacsvm-xoa xo-server[3124716]:   Logical volume vgdata/lvroot contains a filesystem in use.
Apr 16 11:08:34 dacsvm-xoa xo-server[3124716]:   Can't deactivate volume group "vgdata" with 3 open logical volume(s)

Is this also a known issue ?
Do you have an estimate for when LVM support will be available? This is currently a deal breaker for us, as our infrastructure (running on XCP-ng) is becoming increasingly business-critical.
If this is not supported in the near future, we may need to consider alternative backup solutions.

XOA - commit 3ed8c
VM's are based on Debain13
Backup mode: Delta backup

Regards,
Carlo

Thanks for the reply's! This really helps.

I added the

console.log('OIDC profile:', JSON.stringify(profile, null, 2))

and then tried with the following:

username field: email
scope: email

logging:

mrt 30 12:34:57 vm-xoa xo-server[2747568]: OIDC profile: {
mrt 30 12:34:57 vm-xoa xo-server[2747568]:   "id": "38882f04f015223135313da0b919cb3d67bf4fbc@sram.surf.nl"
mrt 30 12:34:57 vm-xoa xo-server[2747568]: }
mrt 30 12:34:57 vm-xoa xo-server[2747568]: Cannot read properties of undefined (reading '0')

username field: uid
scope: uid

logging:

mrt 30 12:35:54 vm-xoa xo-server[2747568]: OIDC profile: {
mrt 30 12:35:54 vm-xoa xo-server[2747568]:   "id": "38882f04f015223135313da0b919cb3d67bf4fbc@sram.surf.nl"
mrt 30 12:35:54 vm-xoa xo-server[2747568]: }
mrt 30 12:35:54 vm-xoa xo-server[2747568]: Expected values to be strictly equal:
mrt 30 12:35:54 vm-xoa xo-server[2747568]: + actual - expected
mrt 30 12:35:54 vm-xoa xo-server[2747568]: + 'undefined'
mrt 30 12:35:54 vm-xoa xo-server[2747568]: - 'string'

So it seems that in both cases we only receive the "sub" from the scope openid from surf. Which is here named "id". Is this translated by xo?

Then I applied the patch from @olivierlambert . Therafter we were able to login by using "id" as Username field, "sub" returned an error. The user 38882f04f015223135313da0b919cb3d67bf4fbc@sram.surf.nl was then created.

If I use email or uid we get now better logging:

mrt 30 12:41:14 vm-xoa xo-server[2747760]: Could not find username: field "uid" is missing from the OIDC profile. Ensure the required scopes are configured and granted by your identity provider.
mrt 30 12:42:27 vm-xoa xo-server[2747760]: Could not find username: field "email" is missing from the OIDC profile. Ensure the required scopes are configured and granted by your identity provider.

We will check if we have to enable/allow additional claims from the authentication provider to be available and let you now.

@HeMaN

We would like to know that too.

It seems that xo is receiving something, but not what it is expecting. But unfortunately we cannot see what it receives.

With a saml tracer plugin I can see data that my browser exchanges with Surf and there are the field that are used/needed, like the uid and the mail.

But from my understanding xo requests its own data from surf based on token from the web session.

We are running Xen Orchestra with commit c3dcb and the auth-oidc (v0.4.2) plugin.

The users that login are unique and not yet present as local users.

The OICD provider is SURF with SRAM: https://www.surf.nl/en/services/identity-access-management/surf-research-access-management

They support the following attributes/scopes: https://servicedesk.surf.nl/wiki/spaces/IAM/pages/74226142/Attributes+in+SRAM

There are some IPs that need to be accessable: https://servicedesk.surf.nl/wiki/spaces/IAM/pages/74226067/IP+addresses#IPaddresses-OIDC . Outgoing traffic from the server to port 443 is open and works.

We did try several settings for the Username field and scopes.

For example the following:

This should create a user R123456789

The logging shows the following:

mrt 27 09:29:48 vm-xoa xo-server[2641104]: Expected values to be strictly equal:
mrt 27 09:29:48 vm-xoa xo-server[2641104]: + actual - expected
mrt 27 09:29:48 vm-xoa xo-server[2641104]: + 'undefined'
mrt 27 09:29:48 vm-xoa xo-server[2641104]: - 'string'

If we change it to:

The following shows up:

mrt 27 09:32:21 vm-xoa xo-server[2641104]: Cannot read properties of undefined (reading '0')

I hope this helps to understand the problem. Thanks.

We are trying to use the OIDC auth plugin to enable login to our Xen Orchestra without local accounts.

We registered a client with our identity provider and got a client id, client secret and the auto-discovery url. That we used to configure the plugin.

However, if we login we get redirected back from the identity provider to the XO callback url and receive then an "Internal Server Error"

The callback URL is as follow:

https://xoa.domain.com/signin/oidc/callback?state=STRING&scope=profile+openid&code=STRING&iss=https://identity-provider.url&client_id=XXXXX

In the log file we see then the following 4 lines:

mrt 25 12:29:25 vm-xoa xo-server[2618522]: Expected values to be strictly equal:
mrt 25 12:29:25 vm-xoa xo-server[2618522]: + actual - expected
mrt 25 12:29:25 vm-xoa xo-server[2618522]: + 'undefined'
mrt 25 12:29:25 vm-xoa xo-server[2618522]: - 'string'

If we change both the username field and the scope to email, we get the same Internal Server Error, but with a different single log line:

mrt 25 13:18:04 vm-xoa xo-server[2618522]: Cannot read properties of undefined (reading '0')

Because we are getting redirected back from our identity provider to Xen Orchestra we guess that the issue is not there. We also get in the browser a SAML response with the userdata.

Running a wireshark on the server shows also traffic between Xen Orchestra and the identity provider, but unfortunately we cannot look in the contents of that traffic stream.

Setting the log level to debug does unfortunately not produce more (error) output.

We are running Xen Orchestra with commit c3dcb and the auth-oidc (v0.4.2) plugin

Is there an other way to figure out what is going wrong?