Team - XO Backend group | XCP-ng and XO forum

Bastien Nollet

Thanks for your contribution and sorry for not giving an answer earlier.

We initially planned to review your PR while doing some additional work on the audit plugin, but our planning evolved and that work was postponed.

We'll review your PR in the following days.

mpiton

@acebmxer thanks, the trace explains it. After the Finalize, the new RPU patched Host 1 and rebooted it at 11:42 your time, then waited for it to come back. When you disabled and re-enabled the pool in XO (I assume Settings > Servers?), XO dropped the connection the RPU was using and opened a new one. The running RPU kept watching the old connection, so it never saw Host 1 come back, and it failed on the 20 minute timeout at 12:02. That's also why it never started on Host 2. The stop you sent at 11:51 didn't cut that wait short either, which we'll look at on our side.

So next time, leave the pool connected during an RPU even if the master takes a few minutes to come back. XO reconnects on its own and gives each host 20 minutes.

The bugtool isn't needed for this run. Can you confirm two things: was "disable and re-enable" the toggle in Settings > Servers, and is Tilton Host 1 the pool master?

mpiton

@acebmxer good to hear the pool is up to date. Your guess makes sense: the banner shows the last RPU XO recorded on that pool, so if you hadn't touched it since, the timeout is from that earlier run.

One thing I'd like to understand: Finalize only closes the record, it doesn't install anything. Did you start a new RPU after that, and did it stop after Host 1? If so, could you post the error from that
"Rolling pool update" task in XO? Its trace files would help too, they're in /var/lib/xo-server/data/rpu-traces on XOA (the exact path is in the task's traceFile property). The rpu-<poolId>-<timestamp>.ndjson file for that run is enough.

mpiton

@acebmxer that banner comes from XO, not from the hosts. The previous RPU on Tilton Pool 1 stopped because Host 1 didn't come back within 20 minutes of its reboot (XO waits for the host to be enabled again with a restarted toolstack), and Host 2 is still pending. As long as an RPU is left incomplete, XO won't start a new one on that pool. That's why the button is greyed out.

What I'd do:

Check that Host 1 is up, enabled, and running the new packages.
Click Finalize. If the failed run left something it had changed (HA, WLB, load balancer, backup schedules, a disabled host, VMs moved or halted), XO refuses and lists it. Put those back, or force it if you're fine leaving them as they are. Forcing only lists them in the task, it doesn't change anything on the pool.
Run the RPU again to update Host 2. If the master is already up to date but the other host isn't, XO asks you to confirm first.

If that host is just slow to boot (long POST, RAID init, that kind of thing), you can raise restartHostTimeout in the [xapiOptions] section of the xo-server config. The default is 20 minutes.

If it never came back enabled at all, then the host logs lmolina asked for are still the way to go.

florent

@Velocity1192 If you can wait a little, we are atively working on the new XO6 backup job uI, that will come with some changes : one schedule per job, but with conditionnal actions

You will run an incremental job, and will be able to define that the schedule of sunday is a full,and even run a healthcheck on it

The ETA is on 6.11 ( next month release )

pierrebrunet

Hi,
This was fixed in Latest. But if you need to stay on Stable, you can open a ticket to have a patch.

pierrebrunet

@johnnezero You can use the docs directly in /rest/v0/docs. You just have to connect and fill in the uuid you need. It will work

mpiton

@jr-m4

Hi, your journalctl output has the answer: vectura: data plane: could not connect: timed out.

With a vCenter as source, the login and the CBT query go through the vCenter on port 443, but the disk itself is read directly from the ESXi host running the VM, on port 902. Your new XO reaches the vCenter fine, it just can't open a connection to that ESXi on 902. The stream has ended without data error is only the side effect of vectura exiting. CBT has nothing to do with it.

The old instance works because it can reach that host (VDDK needed the same port). I'd check firewall/VLAN rules or DNS on the new VM. From the new XO, using the ESXi name as vCenter shows it for that VM:

  getent hosts <esxi-host>
  nc -vz -w 5 <esxi-host> 902

and compare with the old instance.

~Mathieu

pierrebrunet

@johnnezero Hi,
Sorry, it seems it is only in Rest API for now:
You can check it here:

<xoa-url>/rest/v0/docs/#/backup-repositories/ReclaimSpaceBackupRepository

MathieuRA

Issue identified.

@spelon If you're using XO from source, you can try this mra-fix-undici-xen-api branch, otherwise, feel free to open a tunnel so I can fix your XOA.

Team - XO Backend

Posts

Member List