@MajorP93 It's being packaged at the moment. between this and the testing that needs to be done it might take a couple of weeks to be released, but I can't promise anything yet

@dinhngtu Xenopsd does not have a concept of snapshot. Because I understand that "freezing" the VM means suspending it, I think that the suspend hook will get triggered in any snapshot with memory

@dinhngtu It's possible to run hooks inside the control domain on different stages of VM lifecycle operations, but these need to be installed beforehand, see https://xapi-project.github.io/new-docs/xenopsd/design/hooks/index.html

I think the vm-pre-suspend is the name of the hook the does this, but I can't confirm it. The docuementation is missing the hook, but it exists in the source code: https://github.com/xapi-project/xen-api/blob/master/ocaml/xenopsd/lib/xenops_hooks.ml#L37

@olivierlambert This was fixed by xenserver last year: https://github.com/xapi-project/xen-api/pull/6328

For XCP-ng, the team decided they didn't like the solution, and there's work on a better solution, but it's not ready yet

GeraldEV opened this pull request in xapi-project/xen-api

closed CA-407370: Use remote.conf for customer rsyslog forwarding rules #6328

@olivierlambert Because Andriy did quite a bit of work before this merge, xcp-ng depends on xenserver releasing the package to the clients before we can integrate it into xcp-ng. I'd say we can release the package February at the earliest

@olivierlambert XCP-ng will include an option to keep hosts disabled after reboot. This will be part of the next update for xapi. (25.33.1) This was contributed upstream by Andriy and is used by adding an extra parameter to disable: xe host-disable uuid=$HOST auto-enable=false.

@olivierlambert said in XOCE - ISO upload is renamed after upload to ISO SR:

I think we discussed it internally, IIRC it's not XO doing any renaming, but the toolstack. Adding @andriy.sultanov or @psafont in the loop

I'm not sure about that, xapi is quite conscious it cannot rename these ISOs, in the few places I've seen it mentioned. I did not see any place in xapi that could have renamed these, nor reproduce it in my system.

I see at least on a couple of instances that the pool join is being forced. Please don't do this unless you understand the checks being ignored.

What's the result of a pool join without the force option?

Do both hosts have tls verification enabled?
If not, run xe pool-enable-tls-verification. This will set up the certificates for pool communication correctly for the hosts in that pool and turn on TLS verification for pool communication.
If yes, run xe host-list --minimal | xargs -I _ xe host-param-get uuid=_ param-name=name-label | xargs -I _ xe host-refresh-server-certificate host=_. This will reset the certificates for all the hosts in the pool.

Try running the normal join command after doing these steps, and report whether it went well, or report the error.

@jmannik ah, indeed. Do you know which server / interface holds the IP 192.168.30.13? I suspect is still VMHost13, but a different interface.

Until the members have configured their master as 192.168.30.13, you'll have this error. This can be done by a call, but since it's a delicate operation, it's better if there are no operations running on the pool. SSH into the VMHost13, and run

xe host-list name-label=VMHost13 --minimal | xargs -I _ xe pool-designate-new-master host-uuid=_

This should write the new IP to the files of all the pool members and stop blocking this issue from enabling HA

@jmannik The IPs match, and now I don't have an explanation on why is this happening, I'll take another look at the codepath, but that'll have to take a while, as work is piling up

@jmannik Could you collect the file contents of /etc/xensource/pool.conf from all the other hosts? The command is failing in one of them, not on the master host.

@olivierlambert

So we probably need to tell XO team the "right way" to enable HA because there's no way to know from "outside it's not meant to, xapi makes the call automatically.

I don't think so, it's xapi's responsibility to make that call

@olivierlambert The call is indeed hidden from the docs, and only callable from inside a pool... it's called as part as Pool.enable_ha

@jmannik
So the problem goes like this:

• HA uses a local-only database to avoid depending on the database
• This database contains a mapping from UUID to the IP host_address for all hosts in an HA cluster / pool. This information should be gathered right before HA is enabled, from the normal database.
• When trying to enable HA, the host fetches the coordinator's address from the filesystem. Then it uses the previous mapping and the coordinator address to find the coordinator's UUID. This step fails.

I'm not sure what has actually happening, but some scenarios come to mind:

• XO isn't calling the API function Host.preconfigure_ha, which means the local database is not created (unlikely)
• The coordinator's address has somehow changed between the local database being written and the HA being enabled

things to check out:

• inspect the values that the failing host has about the host_address of the coordinator / master host, both on:
  1. the normal database. You can SSH into the failing host and run the following command, replacinf POOL_UUID with the actual uuid, this can be done deleting POOL_UUID , placing the cursor after the = and pressing tab twice.

xe pool-param-get uuid=POOL_UUID param-name=master | xargs -I _ xe host-param-get uuid=_ param-name=address

2. and the pool role file, similar to the previous command, SSH in the failing host and run

cat /etc/xensource/pool.conf

Let us know how it goes. If the IPs don't match, there's a problem with the configuration of the member, and otherwise it's because the local database is outdated and should be refreshed before enabling HA. I don't know how XO handles it.

@jmannik I have a test build that you can test, it will hopefully provide better error messages by raising an internal error with a reason.

The code is based on the newest builds, so I recommend updating to the latest version of XCP beforehand:

yum update
reboot

Once that is done, the test packages can be installed by [deleted, packages removed from repository]

Note: the repository will only be available for a limited amount of time, after which I will repurpose it and delete the instructions so it's not used anymore by accident.

@jmannik said in Unable to enable High Availability - INTERNAL_ERROR(Not_found):

@andriy.sultanov @psafont
https://drive.google.com/file/d/1aJyCYSAuRIBb0X-23gJ6ORtrHSciYH8a/view?usp=sharing
Here is the log file

It's not crystal clear the condition that causes the exception, but I can see some unprotected exception being raised in that path host.ha_join_liveset when trying to recover the host uuid and it's not found. I'll investigate

@jmannik said in Unable to enable High Availability - INTERNAL_ERROR(Not_found):

@olivierlambert

[18:15 vmhost13 ~]# xe pool-ha-enable heartbeat-sr-uuids=381caeb2-5ad9-8924-365d-4b130c67c064
The server failed to handle your request, due to an internal error. The given message may give details useful for debugging the problem.
message: Not_found

That message is created by an exception. It's commonly raised by List.find and List.assoc, in this case the exception wasn't caught.

It's usually difficult to find out which one, since these functions are frequently used and catching the exception can happen in a caller of the function that uses it.

Could you provide the xenserver.log, as Andriy has asked? Otherwise I don't think we'll be able to find the exact cause.

@acebmxer I tried to force it by uploading the same iso twice, and couldn't reproduce the issue. XO shouldn't allow to upload an ISO to an SR with the same name

@olivierlambert Just tried it, because I don't know what XO does:

It starts with a vdi create on the ISO SR:

Oct  8 17:10:20 vega xapi: [debug||2506617 HTTPS XO_IP->:::80|VDI.create R:ceff14462188|audit] VDI.create: SR = 'ISOs' UUID'; name label = 'ubuntu-24.04.3-live-server-arm64.iso'

then it imports the iso into it:

Oct  8 17:10:20 vega xapi: [ info||2411746 HTTPS XO_IP->:::80|task.create D:355d9ec67664|taskhelper] task [XO] Importing content into VDI ubuntu-24.04.3-live-server-arm64.iso on SR ISOs R:0506cfe68373 (uuid:96ba5dea-ffc5-5131-571d-7545277a7083) created (trackid=df64863568c5db04d470bfd2c5872e20) by task D:355d9ec67664
Oct  8 17:10:21 vega xapi: [debug||2506627 HTTPS XO_IP->:::80|[XO] Importing content into VDI ubuntu-24.04.3-live-server-arm64.iso on SR ISOs R:0506cfe68373|import] import_raw_vdi task_id = OpaqueRef:0506cfe6-8373-7b22-505b-fcc07348bb7b vdi = OpaqueRef:81c1154f-17f4-2114-7c7a-bd74604aff6f; chunked = false; format = raw

I see SM managing a vdi with the name UUID.img, a strange error, but it ends up creating the VDI with the expected name. I'll wait to see if the same thing happens and it's later renamed to the .img. I've refreshed the SR just in case, but I can't reproduce the issue just yet

@bnerickson
About 1, That's something that I'd like to support in the future, but it will take some time to get there. About 2, currently LLAs are not allowed to be the used as the management IPs, and I wouldn't like them to be used as such: it can lead to hosts not being to connect to one another if they're not in the same physical network. They still could be configured for the interfaces and shown to the user, but not as the only IP for the interface, or the main one for that matter.