@bnerickson
About 1, That's something that I'd like to support in the future, but it will take some time to get there. About 2, currently LLAs are not allowed to be the used as the management IPs, and I wouldn't like them to be used as such: it can lead to hosts not being to connect to one another if they're not in the same physical network. They still could be configured for the interfaces and shown to the user, but not as the only IP for the interface, or the main one for that matter.

@Tristis-Oris said in What metadata restore really do?:

My pool died so i decied to restore it from backup.

• fresh xen install, retore metadata, everything looks fine, but i can't join any new hosts into this pool.
  https://paste.vates.tech/?ca81f43fd2d10456#Ay6yKRNTS8LDDSHTkin7eLppP33eAMS5grLrevuqf5rL

It looks like the metadata restored has a certificate with name 'sdn-controller-ca', but this certificate does not exist in the filesystem (/etc/stunnel/certs/sdn-controller-ca.pem). I believe this is installed by Xen Orchestra whenever the SDN controller is turned on.
To remove it from the database run xe pool-uninstall-ca-certificate name=sdn-controller-ca --force

The force flag should allow the certificate to be removed from the database even if the file is missing

@benapetr You're right. Unfortunately, there's no VDI revert that allows the revert to happen '. This is shown in the documentation: https://xapi-project.github.io/new-docs/toolstack/features/snapshots/index.html (see revert section)

There's an old proposal to do add this: https://xapi-project.github.io/new-docs/design/snapshot-revert/index.html

But the effort fizzed out because currently the imports do not set the snapshot_of correctly, and the operation needs to work even if the field is not set correctly, as it is now. (falling back to the current code seems sensible) https://github.com/xapi-project/xen-api/pull/2058

This needs some effort to get fixed, I'll set up some ticketing so it can be prioritized accordingly.

djs55 opened this pull request in xapi-project/xen-api

closed VDI.revert pull request + extra bits #2058

@benapetr This is driven by hacky logic from 16 years ago:

• on revert, unserialize the previous state, and update the VM record with its saved values. As we do not want to modify that each time we add a field in the datamodel, use some low-level database functions to iterate over the fields of a record. Not very nice as it makes some assumptions on the database layer, but seems to work allright and I don't think that database layer will change a lot in the future.

I think it might be a good idea to add a revert rpc call to the storage interface that xapi can call to, with a backup to use the current logic if necessary; xapi should be able to clean up the database afterwards. I'll ask other maintainers about this or possible alternatives, but since SMAPIv1 is considered deprecated, I doubt it will happen.

I have to say that SMAPIv3 was finally fixed upstream on June by Xenserver (migrations were finally done!) and XCP-ng should get the update that fixes it in the coming weeks. Given this, I would encourage you to take all the learnings you've acquired while doing the driver and porting it to SMAPIv3. SMAPIv1 just simply has too many problems, some of them are architectural, so in general xenserver and xcp-ng maintainers would like to see it finally go away.

for now I am still targetting XCP-ng 8.2 as that's what I use in production, and I haven't seen many SMAPIv3 drivers there.

8.2 is out of support for xenserver, and for xcp-ng yesterday was the last day it was supported, you really should update

@jmannik said in Unable to enable High Availability - INTERNAL_ERROR(Not_found):

@andriy.sultanov @psafont
https://drive.google.com/file/d/1aJyCYSAuRIBb0X-23gJ6ORtrHSciYH8a/view?usp=sharing
Here is the log file

It's not crystal clear the condition that causes the exception, but I can see some unprotected exception being raised in that path host.ha_join_liveset when trying to recover the host uuid and it's not found. I'll investigate

@rmaclachlan This looks awfully similar to https://github.com/xapi-project/xen-api/pull/5451

freddy77 opened this pull request in xapi-project/xen-api

closed CP-47754: Do not report errors attempting to read PCI vendor:product #5451

@stormi said in XCP-ng 8.3 beta :

@psafont Will a 8.2 to 8.3 upgrade (through the installation ISO) leave TLS verification disabled, or will it enable it by default?

It's not enabled by default, enabling it by default is not possible with the current update procedure where the pool coordinator is updated before the other pool member because these do not expose the new certificates to xmlrpc clients before upgrading, breaking communications.

In other words: must we expect any user who upgrades from 8.2 or lower and then later wants to add a new host to the pool to see this error (and likely ask for help, even if we document it properly - and we would of course)?

Yes

@gsrfan01 The error happens because the joining host has TLS certificate checking enabled for pool connections while the joined host don't.

This mismatch happens because on fresh installs TLS certificate checking is enabled, while for updates from previous versions is not.

To enable TLS certificate checking in a pool simply running xe pool-enable-tls-verification.

The emergency command is not needed in this case, it's useful to re-enable certificate checking in a single host after is has been disabled using the emergency disable

@olivierlambert Snapshot is in essence a VDI clone, I don't see any checks being done before the filtering for ignored VDIs is done. And that is done pretty early on, not sure why there are operations affecting virtual block devices from ignored VDIs: https://github.com/xapi-project/xen-api/blob/master/ocaml/xapi/xapi_vm_clone.ml#L416

@cocoon
The best thing we can do here is inspect the actual certificate:
Please run openssl x509 -text -noout -in /etc/xensource/xapi-ssl.pem

xenserver has generated host certificates with 2048-bit RSA keys for years, these should be able to be loaded by stunnel (through openssl) just fine.

If the key is smaller that this then the fix is easy: generate a new certificate for that host: xe host-refresh-server-certificate host uuid=<>
Be mindful that clients that trusted the previous certificate will need to trust the new one in order for the TLS connections to be established

@jmannik said in Unable to enable High Availability - INTERNAL_ERROR(Not_found):

@andriy.sultanov @psafont
https://drive.google.com/file/d/1aJyCYSAuRIBb0X-23gJ6ORtrHSciYH8a/view?usp=sharing
Here is the log file

It's not crystal clear the condition that causes the exception, but I can see some unprotected exception being raised in that path host.ha_join_liveset when trying to recover the host uuid and it's not found. I'll investigate

@jmannik said in Unable to enable High Availability - INTERNAL_ERROR(Not_found):

@olivierlambert

[18:15 vmhost13 ~]# xe pool-ha-enable heartbeat-sr-uuids=381caeb2-5ad9-8924-365d-4b130c67c064
The server failed to handle your request, due to an internal error. The given message may give details useful for debugging the problem.
message: Not_found

That message is created by an exception. It's commonly raised by List.find and List.assoc, in this case the exception wasn't caught.

It's usually difficult to find out which one, since these functions are frequently used and catching the exception can happen in a caller of the function that uses it.

Could you provide the xenserver.log, as Andriy has asked? Otherwise I don't think we'll be able to find the exact cause.

@acebmxer I tried to force it by uploading the same iso twice, and couldn't reproduce the issue. XO shouldn't allow to upload an ISO to an SR with the same name

@olivierlambert Just tried it, because I don't know what XO does:

It starts with a vdi create on the ISO SR:

Oct  8 17:10:20 vega xapi: [debug||2506617 HTTPS XO_IP->:::80|VDI.create R:ceff14462188|audit] VDI.create: SR = 'ISOs' UUID'; name label = 'ubuntu-24.04.3-live-server-arm64.iso'

then it imports the iso into it:

Oct  8 17:10:20 vega xapi: [ info||2411746 HTTPS XO_IP->:::80|task.create D:355d9ec67664|taskhelper] task [XO] Importing content into VDI ubuntu-24.04.3-live-server-arm64.iso on SR ISOs R:0506cfe68373 (uuid:96ba5dea-ffc5-5131-571d-7545277a7083) created (trackid=df64863568c5db04d470bfd2c5872e20) by task D:355d9ec67664
Oct  8 17:10:21 vega xapi: [debug||2506627 HTTPS XO_IP->:::80|[XO] Importing content into VDI ubuntu-24.04.3-live-server-arm64.iso on SR ISOs R:0506cfe68373|import] import_raw_vdi task_id = OpaqueRef:0506cfe6-8373-7b22-505b-fcc07348bb7b vdi = OpaqueRef:81c1154f-17f4-2114-7c7a-bd74604aff6f; chunked = false; format = raw

I see SM managing a vdi with the name UUID.img, a strange error, but it ends up creating the VDI with the expected name. I'll wait to see if the same thing happens and it's later renamed to the .img. I've refreshed the SR just in case, but I can't reproduce the issue just yet

@bnerickson
About 1, That's something that I'd like to support in the future, but it will take some time to get there. About 2, currently LLAs are not allowed to be the used as the management IPs, and I wouldn't like them to be used as such: it can lead to hosts not being to connect to one another if they're not in the same physical network. They still could be configured for the interfaces and shown to the user, but not as the only IP for the interface, or the main one for that matter.

@olivierlambert Ideally XCP-ng (xapi) could add this to a queue, and wait for some time before cancelling the task because it took too long. This also needs some kind of feedback that can be given to the user / client, which I think currently is quite undercooked (how to report is waiting on other migrations to the same host when a client asks?). For the time I think XO being aware that it can retry the operation would be simpler, especially because it already has code to do it for other operations

@benapetr You're right. Unfortunately, there's no VDI revert that allows the revert to happen '. This is shown in the documentation: https://xapi-project.github.io/new-docs/toolstack/features/snapshots/index.html (see revert section)

There's an old proposal to do add this: https://xapi-project.github.io/new-docs/design/snapshot-revert/index.html

But the effort fizzed out because currently the imports do not set the snapshot_of correctly, and the operation needs to work even if the field is not set correctly, as it is now. (falling back to the current code seems sensible) https://github.com/xapi-project/xen-api/pull/2058

This needs some effort to get fixed, I'll set up some ticketing so it can be prioritized accordingly.

djs55 opened this pull request in xapi-project/xen-api

closed VDI.revert pull request + extra bits #2058

@benapetr This is driven by hacky logic from 16 years ago:

• on revert, unserialize the previous state, and update the VM record with its saved values. As we do not want to modify that each time we add a field in the datamodel, use some low-level database functions to iterate over the fields of a record. Not very nice as it makes some assumptions on the database layer, but seems to work allright and I don't think that database layer will change a lot in the future.

I think it might be a good idea to add a revert rpc call to the storage interface that xapi can call to, with a backup to use the current logic if necessary; xapi should be able to clean up the database afterwards. I'll ask other maintainers about this or possible alternatives, but since SMAPIv1 is considered deprecated, I doubt it will happen.

I have to say that SMAPIv3 was finally fixed upstream on June by Xenserver (migrations were finally done!) and XCP-ng should get the update that fixes it in the coming weeks. Given this, I would encourage you to take all the learnings you've acquired while doing the driver and porting it to SMAPIv3. SMAPIv1 just simply has too many problems, some of them are architectural, so in general xenserver and xcp-ng maintainers would like to see it finally go away.

for now I am still targetting XCP-ng 8.2 as that's what I use in production, and I haven't seen many SMAPIv3 drivers there.

8.2 is out of support for xenserver, and for xcp-ng yesterday was the last day it was supported, you really should update

@kubuntu-newbie I've looked at the code. The intel passthrough seems to not care about the value of the vga key in platform, and instead it expects the key igd_passthrough to be set to true.

https://github.com/xapi-project/xen-api/blob/9eb5f9f9f3c742c0c0b691098e1dafc02e40c856/ocaml/xapi/xapi_xenops.ml#L390

@Tristis-Oris said in What metadata restore really do?:

My pool died so i decied to restore it from backup.

• fresh xen install, retore metadata, everything looks fine, but i can't join any new hosts into this pool.
  https://paste.vates.tech/?ca81f43fd2d10456#Ay6yKRNTS8LDDSHTkin7eLppP33eAMS5grLrevuqf5rL

It looks like the metadata restored has a certificate with name 'sdn-controller-ca', but this certificate does not exist in the filesystem (/etc/stunnel/certs/sdn-controller-ca.pem). I believe this is installed by Xen Orchestra whenever the SDN controller is turned on.
To remove it from the database run xe pool-uninstall-ca-certificate name=sdn-controller-ca --force

The force flag should allow the certificate to be removed from the database even if the file is missing