Andrew

Andrew

@stormi Microcode updated on affected Gen11 i7. Running normally.

Andrew

@gduperrey I jumped in all the way by mistake... I updated a wrong host, so I just did them all. Older AMD, Intel E3/E5, NUC11, etc. So far, so good. Add/migrate/backup/etc VMs are working as usual. Good for guest tools too, but mine are mostly Debian 7-11. Stuff is as usual so far.

Andrew

I agree and I'll say it again, Thanks! It's not just Linux/Xen stuff. It is XCP-ng and XO that make everything work as a cohesive vertical open-source solution (some nice buzz words). Thanks to the Vates team and community that have built and support it. I look forward to ongoing continuous improvement and innovation!

Andrew

@bleader Installed and running.

Andrew

@bleader Updates running on several old and new intel machines (including microcode update). Working fine so far. Rolling Pool Reboot is a helpful feature.

Andrew

@bleader I installed it on a bunch of busy hosts. All are fine, but none used PCI passthrough. The Rolling Pool Reboot in XO was very helpful.

Andrew

@dinhngtu On a quick test, the ISO boots and installs. When the VM boots from the HD it crashes (and on reboots).

Updating XCP 8.3 with qemu-4.2.1-5.2.12.2~fixvwc1.1.xcpng8.3.x86_64 solves the issue and the installed windows image boots correctly without any additional changes.

Andrew

@abudef @olivierlambert @thomas-dkmt I agree. I read block the same way... how about disable or prevent. (French? empêcher)

Andrew

@julien-f @olivierlambert

Thanks for the ability to delete XO task logs feature! (XO commit f6e6e)

Andrew

@gduperrey Installed and running on Intel systems, and Zen3 system that sees the microcode update.

Andrew

@lmolina All updated. RPU successful. Systems running normally.

Andrew

@lmolina Ok, this time all is working (so far).

Andrew

@lmolina @olivierlambert Sorry, but this update got ugly fast. CR backups got stuck. Migrations got stuck. VMs would not start any more. After reverting all the patches and rebooting all hosts I had several VMs that would not start again (including primary XO) because the VDIs stuck marked as activating. It took some time to fix them but at least there was no data loss. Some backups lost their snapshots and had to revert to full backups again.

Andrew

@lmolina Updates ran fine, but I'm having CR backup problems too... Mostly ran fine, but I now have 3 stuck with VBD.unplug_force. I can't get them to clear. I'll have to try harder before I revert updates.

Andrew

@gleh Up and running on busy pools. Rolling pool reboot worked correctly.

Andrew

@pierrebrunet I ran the normal delta backups to S3 using fix_alias_vhd and it runs again (full and delta).

Andrew

@poddingue @Team-XO-Backend @pierrebrunet @florent I have the same issue in XO (delta backup to S3)... Caused by master commit adecd7306e2fa262ef77d71f83555d160f03aa03

{
  "data": {
    "mode": "delta",
    "reportWhen": "never"
  },
  "id": "1789353842351",
  "jobId": "d6c0a656-62c5-4c39-a57a-f246b39f1cef",
  "jobName": "minio-test",
  "message": "backup",
  "scheduleId": "bd4ef436-fd85-4f16-bf9e-71d1d0c8586f",
  "start": 1789353842351,
  "status": "failure",
  "tasks": [
    {
      "id": "0mu0n2une-6717eycsj3k",
      "start": 1789353844394,
      "status": "failure",
      "tasks": [
        {
          "id": "0mu0n2unj-32t56n3u8vn",
          "start": 1789353844399,
          "status": "success",
          "end": 1789353844854,
          "result": {
            "removedFiles": [],
            "changedFiles": [],
            "merge": false,
            "size": 0
          },
          "message": "clean-vm"
        },
        {
          "id": "0mu0n2vw5-jwlf68g7htp",
          "start": 1789353846005,
          "status": "success",
          "end": 1789353847486,
          "result": "6d4454f8-5735-0581-1eba-a66021f22a50",
          "message": "snapshot"
        },
        {
          "id": "0mu0n2x1b-qze2zs5ytaq",
          "start": 1789353847487,
          "status": "failure",
          "tasks": [
            {
              "id": "0mu0n2ylj-b5qu3cxprie",
              "start": 1789353849511,
              "status": "failure",
              "end": 1789353849570,
              "result": {
                "code": "NOT_SUPPORTED",
                "message": "Can't init vhd directory without using alias",
                "name": "Error",
                "stack": "Error: Can't init vhd directory without using alias\n    at RemoteVhdDisk.init (file:///opt/xo/xo-builds/xen-orchestra-202609132201/@xen-orchestra/backup-archive/dist/disks/RemoteVhdDisk.mjs:74:41)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async openDisposableDisk (file:///opt/xo/xo-builds/xen-orchestra-202609132201/@xen-orchestra/backup-archive/dist/disks/index.mjs:32:9)\n    at async Promise.all (index 0)\n    at async checkDisk (file:///opt/xo/xo-builds/xen-orchestra-202609132201/@xen-orchestra/backups/_runners/_writers/_checkDisk.mjs:5:3)\n    at async DiskConsumerVhdDirectory.write (file:///opt/xo/xo-builds/xen-orchestra-202609132201/packages/vhd-lib/disk-consumer/DiskConsumerVhdDirectory.mjs:98:7)\n    at async writeToVhdDirectory (file:///opt/xo/xo-builds/xen-orchestra-202609132201/packages/vhd-lib/disk-consumer/index.mjs:43:10)\n    at async RemoteAdapter.writeVhd (file:///opt/xo/xo-builds/xen-orchestra-202609132201/@xen-orchestra/backups/RemoteAdapter.mjs:487:14)\n    at async asyncEach.concurrency (file:///opt/xo/xo-builds/xen-orchestra-202609132201/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:276:31)"
              },
              "message": "transfer"
            }
          ],
          "end": 1789353849570,
          "result": {
            "code": "NOT_SUPPORTED",
            "message": "Can't init vhd directory without using alias",
            "name": "Error",
            "stack": "Error: Can't init vhd directory without using alias\n    at RemoteVhdDisk.init (file:///opt/xo/xo-builds/xen-orchestra-202609132201/@xen-orchestra/backup-archive/dist/disks/RemoteVhdDisk.mjs:74:41)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async openDisposableDisk (file:///opt/xo/xo-builds/xen-orchestra-202609132201/@xen-orchestra/backup-archive/dist/disks/index.mjs:32:9)\n    at async Promise.all (index 0)\n    at async checkDisk (file:///opt/xo/xo-builds/xen-orchestra-202609132201/@xen-orchestra/backups/_runners/_writers/_checkDisk.mjs:5:3)\n    at async DiskConsumerVhdDirectory.write (file:///opt/xo/xo-builds/xen-orchestra-202609132201/packages/vhd-lib/disk-consumer/DiskConsumerVhdDirectory.mjs:98:7)\n    at async writeToVhdDirectory (file:///opt/xo/xo-builds/xen-orchestra-202609132201/packages/vhd-lib/disk-consumer/index.mjs:43:10)\n    at async RemoteAdapter.writeVhd (file:///opt/xo/xo-builds/xen-orchestra-202609132201/@xen-orchestra/backups/RemoteAdapter.mjs:487:14)\n    at async asyncEach.concurrency (file:///opt/xo/xo-builds/xen-orchestra-202609132201/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:276:31)"
          },
          "message": "export",
          "data": {
            "id": "9890e0c4-ba3a-4810-8245-a49fdf16b16e",
            "isFull": false,
            "type": "remote"
          }
        }
      ],
      "infos": [
        {
          "message": "Transfer data using NBD"
        },
        {
          "message": "will delete snapshot data"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:59ead5f8-710e-0e75-969e-bc59be6fab7f"
          },
          "message": "Snapshot data has been deleted"
        }
      ],
      "end": 1789353852197,
      "result": {
        "code": "NOT_SUPPORTED",
        "message": "Can't init vhd directory without using alias",
        "name": "Error",
        "stack": "Error: Can't init vhd directory without using alias\n    at RemoteVhdDisk.init (file:///opt/xo/xo-builds/xen-orchestra-202609132201/@xen-orchestra/backup-archive/dist/disks/RemoteVhdDisk.mjs:74:41)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async openDisposableDisk (file:///opt/xo/xo-builds/xen-orchestra-202609132201/@xen-orchestra/backup-archive/dist/disks/index.mjs:32:9)\n    at async Promise.all (index 0)\n    at async checkDisk (file:///opt/xo/xo-builds/xen-orchestra-202609132201/@xen-orchestra/backups/_runners/_writers/_checkDisk.mjs:5:3)\n    at async DiskConsumerVhdDirectory.write (file:///opt/xo/xo-builds/xen-orchestra-202609132201/packages/vhd-lib/disk-consumer/DiskConsumerVhdDirectory.mjs:98:7)\n    at async writeToVhdDirectory (file:///opt/xo/xo-builds/xen-orchestra-202609132201/packages/vhd-lib/disk-consumer/index.mjs:43:10)\n    at async RemoteAdapter.writeVhd (file:///opt/xo/xo-builds/xen-orchestra-202609132201/@xen-orchestra/backups/RemoteAdapter.mjs:487:14)\n    at async asyncEach.concurrency (file:///opt/xo/xo-builds/xen-orchestra-202609132201/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:276:31)"
      },
      "message": "backup VM",
      "data": {
        "id": "a081f208-4e40-2ca5-c68e-86106af5a8ef",
        "type": "VM",
        "name_label": "Utest",
        "progress": 0
      }
    }
  ],
  "end": 1789353852211,
  "infos": [
    {
      "data": {
        "vms": [
          "a081f208-4e40-2ca5-c68e-86106af5a8ef"
        ]
      },
      "message": "vms"
    }
  ]
}```

Andrew

@gduperrey Installed and running. Rolling pool update worked.

Andrew

@gduperrey Rolling pool update worked with released production patches.

Andrew

@gleh After the update and a manual primary host reboot I was able to start new VMs and migrate VMs, but still could not run rolling pool update. I tried restarting tool stack, but it did not help.

pool.rollingReboot
{
  "pool": "40a08640-88dc-7ff9-9190-e13b89025b4d",
  "bypassBackupCheck": false,
  "shutdownPinnedVms": false
}
{
  "code": "CANNOT_EVACUATE_HOST",
  "params": [
    "NO_HOSTS_AVAILABLE,OpaqueRef:26968b78-860d-1842-a001-c93c8f07c895|NO_HOSTS_AVAILABLE,OpaqueRef:535b4eba-bc66-85a2-db6a-a857495bfafc|NO_HOSTS_AVAILABLE,OpaqueRef:49b9453d-5725-26ad-c1db-be85f5bf0cc4|NO_HOSTS_AVAILABLE,OpaqueRef:9c17b691-1b3d-3aee-138b-8e3a7ea7e58a|NO_HOSTS_AVAILABLE,OpaqueRef:c29f5630-a246-8413-8477-73f75cfb4c61|NO_HOSTS_AVAILABLE,OpaqueRef:5440c209-4e36-97e8-6d8c-74f20f82d710|NO_HOSTS_AVAILABLE,OpaqueRef:00581b21-7700-0803-9708-d0ffac2bf2f8"
  ],
  "call": {
    "duration": 4,
    "method": "host.assert_can_evacuate",
    "params": [
      "* session id *",
      "OpaqueRef:a8fb168d-bd82-5c1a-60fe-e965eefc43ef"
    ]
  },
  "message": "CANNOT_EVACUATE_HOST(NO_HOSTS_AVAILABLE,OpaqueRef:26968b78-860d-1842-a001-c93c8f07c895|NO_HOSTS_AVAILABLE,OpaqueRef:535b4eba-bc66-85a2-db6a-a857495bfafc|NO_HOSTS_AVAILABLE,OpaqueRef:49b9453d-5725-26ad-c1db-be85f5bf0cc4|NO_HOSTS_AVAILABLE,OpaqueRef:9c17b691-1b3d-3aee-138b-8e3a7ea7e58a|NO_HOSTS_AVAILABLE,OpaqueRef:c29f5630-a246-8413-8477-73f75cfb4c61|NO_HOSTS_AVAILABLE,OpaqueRef:5440c209-4e36-97e8-6d8c-74f20f82d710|NO_HOSTS_AVAILABLE,OpaqueRef:00581b21-7700-0803-9708-d0ffac2bf2f8)",
  "name": "XapiError",
  "stack": "XapiError: CANNOT_EVACUATE_HOST(NO_HOSTS_AVAILABLE,OpaqueRef:26968b78-860d-1842-a001-c93c8f07c895|NO_HOSTS_AVAILABLE,OpaqueRef:535b4eba-bc66-85a2-db6a-a857495bfafc|NO_HOSTS_AVAILABLE,OpaqueRef:49b9453d-5725-26ad-c1db-be85f5bf0cc4|NO_HOSTS_AVAILABLE,OpaqueRef:9c17b691-1b3d-3aee-138b-8e3a7ea7e58a|NO_HOSTS_AVAILABLE,OpaqueRef:c29f5630-a246-8413-8477-73f75cfb4c61|NO_HOSTS_AVAILABLE,OpaqueRef:5440c209-4e36-97e8-6d8c-74f20f82d710|NO_HOSTS_AVAILABLE,OpaqueRef:00581b21-7700-0803-9708-d0ffac2bf2f8)
    at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/_XapiError.mjs:16:12)
    at file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:65:5)
    at processImmediate (node:internal/timers:502:9)"
}

Andrew

@Andrew