@doogie06
register cli: xo-cli --register http://url login
then remove all tasks: xo-cli rest del tasks

Problem with broken package on rhel8-9 looks resolved now. Done multiple tests.

@DustinB said in New Rust Xen guest tools:

systemctl enable xe

yep, found them.
full steps for RHEL:

wget https://gitlab.com/xen-project/xen-guest-agent/-/jobs/6041608360/artifacts/raw/RPMS/x86_64/xen-guest-agent-0.4.0-0.fc37.x86_64.rpm
rpm -i xen-guest-agent*
yum remove -y xe-guest-utilities-latest
systemctl enable xen-guest-agent.service --now

@chrisfonte new tools will remove the old one during install.

@ckargitest that was fixed few weeks ago. Do you use latest commit?

@julien-f @olivierlambert
2371109b6fea26c15df28caed132be2108a0d88e
Fixed now, thanks you.

updated to fix_cbt branch.

CR NBD backup works.
Delta NBD backup works.
just once, so we can't be sure yet.

No broken tasks is generated.

Still confused why CBT toggle is enabled on some VMs.
2 similars vms on same pool, same storage, same ubuntu version. One is enabled automaticaly, other is not.

if you have no any dedicated storage, it's possible to use literally any PC with nfs\smb\external drive, windows share also.

Backups not very fast anyway, because of XAPI limitations. I see cap about 300-400Mbit at 10Gbit local network.

I agree, got a similar dilemma now.
Required to grant permissions to single host (gpu computing) and ability to create\remove VMs.

But it looks it no ACL for VM creation, it available only for global admin\pool admin accounts. So only way is setup standalone XO for each single host? This way i lose the pool storage and VM migration.

@stephane-m-dev
Device: \\1.2.3.4.:Port:/backupVM
Options: vers=4,nconnect=16,noauto,x-systemd.automount,x-systemd.device-timeout=10,x-systemd.idle-timeout=1min

commit 9939c, xen 8.2.1.
after latest update, one VM at one job failed with interesting error.

    {
      "data": {
        "type": "VM",
        "id": "6feb8e90-2595-3585-8338-24b6aae17f9a",
        "name_label": "name"
      },
      "id": "1732179827258",
      "message": "backup VM",
      "start": 1732179827258,
      "status": "failure",
      "tasks": [
        {
          "id": "1732179827332",
          "message": "clean-vm",
          "start": 1732179827332,
          "status": "failure",
          "warnings": [
            {
              "data": {
                "path": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241107T090318Z.alias.vhd",
                "error": {
                  "generatedMessage": true,
                  "code": "ERR_ASSERTION",
                  "actual": false,
                  "expected": true,
                  "operator": "=="
                }
              },
              "message": "VHD check error"
            },
            {
              "data": {
                "alias": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241107T090318Z.alias.vhd"
              },
              "message": "missing target of alias"
            }
          ],
          "end": 1732180079109,
          "result": {
            "code": "ERR_INVALID_ARG_TYPE",
            "message": "The \"paths[1]\" argument must be of type string. Received undefined",
            "name": "TypeError",
            "stack": "TypeError [ERR_INVALID_ARG_TYPE]: The \"paths[1]\" argument must be of type string. Received undefined\n    at resolve (node:path:1169:7)\n    at normalize (/opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/fs/dist/path.js:21:27)\n    at NfsHandler.__unlink (/opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/fs/dist/abstract.js:412:32)\n    at NfsHandler.unlink (/opt/xo/xo-builds/xen-orchestra-202411191133/node_modules/limit-concurrency-decorator/index.js:97:24)\n    at checkAliases (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:132:25)\n    at async Array.<anonymous> (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:284:5)\n    at async Promise.all (index 0)\n    at async RemoteAdapter.cleanVm (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:283:3)"
          }
        },
        {
          "id": "1732179827338",
          "message": "clean-vm",
          "start": 1732179827338,
          "status": "failure",
          "warnings": [
            {
              "data": {
                "path": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241107T090318Z.alias.vhd",
                "error": {
                  "generatedMessage": true,
                  "code": "ERR_ASSERTION",
                  "actual": false,
                  "expected": true,
                  "operator": "=="
                }
              },
              "message": "VHD check error"
            },
            {
              "data": {
                "alias": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241107T090318Z.alias.vhd"
              },
              "message": "missing target of alias"
            }
          ],
          "end": 1732180078116,
          "result": {
            "code": "ERR_INVALID_ARG_TYPE",
            "message": "The \"paths[1]\" argument must be of type string. Received undefined",
            "name": "TypeError",
            "stack": "TypeError [ERR_INVALID_ARG_TYPE]: The \"paths[1]\" argument must be of type string. Received undefined\n    at resolve (node:path:1169:7)\n    at normalize (/opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/fs/dist/path.js:21:27)\n    at NfsHandler.__unlink (/opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/fs/dist/abstract.js:412:32)\n    at NfsHandler.unlink (/opt/xo/xo-builds/xen-orchestra-202411191133/node_modules/limit-concurrency-decorator/index.js:97:24)\n    at checkAliases (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:132:25)\n    at async Array.<anonymous> (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:284:5)\n    at async Promise.all (index 5)\n    at async RemoteAdapter.cleanVm (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:283:3)"
          }
        },
        {
          "id": "1732180081190",
          "message": "snapshot",
          "start": 1732180081190,
          "status": "success",
          "end": 1732180083668,
          "result": "ab331405-2ae3-af13-1c6d-3482cfcf1f3a"
        },
        {
          "data": {
            "id": "4b6d24a3-0b1e-48d5-aac2-a06e3a8ee485",
            "isFull": false,
            "type": "remote"
          },
          "id": "1732180083668:0",
          "message": "export",
          "start": 1732180083668,
          "status": "success",
          "tasks": [
            {
              "id": "1732180087479",
              "message": "transfer",
              "start": 1732180087479,
              "status": "success",
              "end": 1732180165572,
              "result": {
                "size": 6172148736
              }
            },
            {
              "id": "1732180539007",
              "message": "clean-vm",
              "start": 1732180539007,
              "status": "success",
              "warnings": [
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241107T090318Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241107T170223Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241107T170223Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T010229Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T010229Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T090332Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T090332Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T170238Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T170238Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T010253Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T010253Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T090400Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T090400Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T170401Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T170401Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241110T010358Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241110T010358Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241110T090459Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241110T090459Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241110T170554Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241108T170238Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T170238Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241109T010253Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T010253Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241107T170223Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241107T170223Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241108T090332Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T090332Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241108T010229Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T010229Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241109T090400Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T090400Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241109T170401Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T170401Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241110T170554Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241110T170554Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241110T010358Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241110T010358Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241110T090459Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241110T090459Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                }
              ],
              "end": 1732180651483,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1732180651497
        },

that a make a sense, now. So no issue here.

@florent yes. it was recomended at NBD release) not required now?

@olivierlambert

NBD not enabled at task.

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1729706400074",
  "jobId": "2b83aac0-6d98-4969-9290-2605849683ed",
  "jobName": "no nbd",
  "message": "backup",
  "scheduleId": "b40fe09a-8d7c-4817-ae45-d2ca24321b28",
  "start": 1729706400074,
  "status": "success",
  "infos": [
    {
      "data": {
        "vms": [
          "d79c461b-3ace-8fb0-c055-300ac4b15301"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "d79c461b-3ace-8fb0-c055-300ac4b15301",
        "name_label": "name"
      },
      "id": "1729706402635",
      "message": "backup VM",
      "start": 1729706402635,
      "status": "success",
      "tasks": [
        {
          "id": "1729706402723",
          "message": "clean-vm",
          "start": 1729706402723,
          "status": "success",
          "end": 1729706403591,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1729706402811",
          "message": "clean-vm",
          "start": 1729706402811,
          "status": "success",
          "end": 1729706403925,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1729706404428",
          "message": "snapshot",
          "start": 1729706404428,
          "status": "success",
          "end": 1729706417536,
          "result": "331c7509-15cc-06ce-a824-f834dfa42f88"
        },
        {
          "data": {
            "id": "8da40b08-636f-450d-af15-3264b9692e1f",
            "isFull": false,
            "type": "remote"
          },
          "id": "1729706417539",
          "message": "export",
          "start": 1729706417539,
          "status": "success",
          "tasks": [
            {
              "id": "1729706423056",
              "message": "transfer",
              "start": 1729706423056,
              "status": "success",
              "end": 1729706453043,
              "result": {
                "size": 2158497792
              }
            },
            {
              "id": "1729706457390",
              "message": "clean-vm",
              "start": 1729706457390,
              "status": "success",
              "end": 1729706458077,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1729706458082
        },
        {
          "data": {
            "id": "4b6d24a3-0b1e-48d5-aac2-a06e3a8ee485",
            "isFull": false,
            "type": "remote"
          },
          "id": "1729706417538",
          "message": "export",
          "start": 1729706417538,
          "status": "success",
          "tasks": [
            {
              "id": "1729706423053",
              "message": "transfer",
              "start": 1729706423053,
              "status": "success",
              "end": 1729706452967,
              "result": {
                "size": 2158497792
              }
            },
            {
              "id": "1729706458083",
              "message": "clean-vm",
              "start": 1729706458083,
              "status": "success",
              "end": 1729706458368,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1729706458371
        }
      ],
      "infos": [
        {
          "message": "Transfer data using NBD"
        }
      ],
      "end": 1729706458371
    }
  ],
  "end": 1729706458373
}

i also removed one 8.3 pool from nbd backup, create the usual one. Looks it work fine.

@jhansen ah. maybe only this way.

@jhansen yum update not intended.

Already every VM at 8.3 pool stuck at control domain. Can't detach them without pool reboot. Any workaround?

vbd.delete
{
  "id": "7af4b39d-d6aa-2589-509f-f082a1f0f2bb

"
}
{
  "code": "OPERATION_NOT_ALLOWED",
  "params": [
    "VBD '7af4b39d-d6aa-2589-509f-f082a1f0f2bb

' still attached to 'e9e4e600-b180-481d-9a1b-a45be8fc192e

'"
  ],
  "call": {
    "method": "VBD.destroy",
    "params": [
      "OpaqueRef:65481d14-d3de-82d7-fa7d-95131f14a209

"
    ]
  },
  "message": "OPERATION_NOT_ALLOWED(VBD '7af4b39d-d6aa-2589-509f-f082a1f0f2bb

' still attached to 'e9e4e600-b180-481d-9a1b-a45be8fc192e

')",
  "name": "XapiError",
  "stack": "XapiError: OPERATION_NOT_ALLOWED(VBD '7af4b39d-d6aa-2589-509f-f082a1f0f2bb

' still attached to 'e9e4e600-b180-481d-9a1b-a45be8fc192e

')
    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202410181215/packages/xen-api/_XapiError.mjs:16:12)
    at file:///opt/xo/xo-builds/xen-orchestra-202410181215/packages/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:60:5)
    at processImmediate (node:internal/timers:454:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}