Posts made by Tristis Oris

Tristis Oris

@stephane-m-dev for now it single error.

Tristis Oris

@stephane-m-dev
Device: \\1.2.3.4.:Port:/backupVM
Options: vers=4,nconnect=16,noauto,x-systemd.automount,x-systemd.device-timeout=10,x-systemd.idle-timeout=1min

Tristis Oris

commit 9939c, xen 8.2.1.
after latest update, one VM at one job failed with interesting error.

    {
      "data": {
        "type": "VM",
        "id": "6feb8e90-2595-3585-8338-24b6aae17f9a",
        "name_label": "name"
      },
      "id": "1732179827258",
      "message": "backup VM",
      "start": 1732179827258,
      "status": "failure",
      "tasks": [
        {
          "id": "1732179827332",
          "message": "clean-vm",
          "start": 1732179827332,
          "status": "failure",
          "warnings": [
            {
              "data": {
                "path": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241107T090318Z.alias.vhd",
                "error": {
                  "generatedMessage": true,
                  "code": "ERR_ASSERTION",
                  "actual": false,
                  "expected": true,
                  "operator": "=="
                }
              },
              "message": "VHD check error"
            },
            {
              "data": {
                "alias": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241107T090318Z.alias.vhd"
              },
              "message": "missing target of alias"
            }
          ],
          "end": 1732180079109,
          "result": {
            "code": "ERR_INVALID_ARG_TYPE",
            "message": "The \"paths[1]\" argument must be of type string. Received undefined",
            "name": "TypeError",
            "stack": "TypeError [ERR_INVALID_ARG_TYPE]: The \"paths[1]\" argument must be of type string. Received undefined\n    at resolve (node:path:1169:7)\n    at normalize (/opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/fs/dist/path.js:21:27)\n    at NfsHandler.__unlink (/opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/fs/dist/abstract.js:412:32)\n    at NfsHandler.unlink (/opt/xo/xo-builds/xen-orchestra-202411191133/node_modules/limit-concurrency-decorator/index.js:97:24)\n    at checkAliases (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:132:25)\n    at async Array.<anonymous> (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:284:5)\n    at async Promise.all (index 0)\n    at async RemoteAdapter.cleanVm (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:283:3)"
          }
        },
        {
          "id": "1732179827338",
          "message": "clean-vm",
          "start": 1732179827338,
          "status": "failure",
          "warnings": [
            {
              "data": {
                "path": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241107T090318Z.alias.vhd",
                "error": {
                  "generatedMessage": true,
                  "code": "ERR_ASSERTION",
                  "actual": false,
                  "expected": true,
                  "operator": "=="
                }
              },
              "message": "VHD check error"
            },
            {
              "data": {
                "alias": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241107T090318Z.alias.vhd"
              },
              "message": "missing target of alias"
            }
          ],
          "end": 1732180078116,
          "result": {
            "code": "ERR_INVALID_ARG_TYPE",
            "message": "The \"paths[1]\" argument must be of type string. Received undefined",
            "name": "TypeError",
            "stack": "TypeError [ERR_INVALID_ARG_TYPE]: The \"paths[1]\" argument must be of type string. Received undefined\n    at resolve (node:path:1169:7)\n    at normalize (/opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/fs/dist/path.js:21:27)\n    at NfsHandler.__unlink (/opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/fs/dist/abstract.js:412:32)\n    at NfsHandler.unlink (/opt/xo/xo-builds/xen-orchestra-202411191133/node_modules/limit-concurrency-decorator/index.js:97:24)\n    at checkAliases (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:132:25)\n    at async Array.<anonymous> (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:284:5)\n    at async Promise.all (index 5)\n    at async RemoteAdapter.cleanVm (file:///opt/xo/xo-builds/xen-orchestra-202411191133/@xen-orchestra/backups/_cleanVm.mjs:283:3)"
          }
        },
        {
          "id": "1732180081190",
          "message": "snapshot",
          "start": 1732180081190,
          "status": "success",
          "end": 1732180083668,
          "result": "ab331405-2ae3-af13-1c6d-3482cfcf1f3a"
        },
        {
          "data": {
            "id": "4b6d24a3-0b1e-48d5-aac2-a06e3a8ee485",
            "isFull": false,
            "type": "remote"
          },
          "id": "1732180083668:0",
          "message": "export",
          "start": 1732180083668,
          "status": "success",
          "tasks": [
            {
              "id": "1732180087479",
              "message": "transfer",
              "start": 1732180087479,
              "status": "success",
              "end": 1732180165572,
              "result": {
                "size": 6172148736
              }
            },
            {
              "id": "1732180539007",
              "message": "clean-vm",
              "start": 1732180539007,
              "status": "success",
              "warnings": [
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241107T090318Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241107T170223Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241107T170223Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T010229Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T010229Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T090332Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T090332Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T170238Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T170238Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T010253Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T010253Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T090400Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T090400Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T170401Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T170401Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241110T010358Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241110T010358Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241110T090459Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "parent": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241110T090459Z.alias.vhd",
                    "child": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241110T170554Z.alias.vhd"
                  },
                  "message": "parent VHD is missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241108T170238Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T170238Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241109T010253Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T010253Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241107T170223Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241107T170223Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241108T090332Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T090332Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241108T010229Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241108T010229Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241109T090400Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T090400Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241109T170401Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241109T170401Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241110T170554Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241110T170554Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241110T010358Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241110T010358Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                },
                {
                  "data": {
                    "backup": "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/20241110T090459Z.json",
                    "missingVhds": [
                      "/xo-vm-backups/6feb8e90-2595-3585-8338-24b6aae17f9a/vdis/8b3f311e-90dd-495d-8626-819df32958f6/06681026-471a-4243-ac6b-ce6d173d687a/20241110T090459Z.alias.vhd"
                    ]
                  },
                  "message": "some VHDs linked to the backup are missing"
                }
              ],
              "end": 1732180651483,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1732180651497
        },

Tristis Oris

that a make a sense, now. So no issue here.

Tristis Oris

@florent yes. it was recomended at NBD release) not required now?

Tristis Oris

@olivierlambert

Tristis Oris

NBD not enabled at task.

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1729706400074",
  "jobId": "2b83aac0-6d98-4969-9290-2605849683ed",
  "jobName": "no nbd",
  "message": "backup",
  "scheduleId": "b40fe09a-8d7c-4817-ae45-d2ca24321b28",
  "start": 1729706400074,
  "status": "success",
  "infos": [
    {
      "data": {
        "vms": [
          "d79c461b-3ace-8fb0-c055-300ac4b15301"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "d79c461b-3ace-8fb0-c055-300ac4b15301",
        "name_label": "name"
      },
      "id": "1729706402635",
      "message": "backup VM",
      "start": 1729706402635,
      "status": "success",
      "tasks": [
        {
          "id": "1729706402723",
          "message": "clean-vm",
          "start": 1729706402723,
          "status": "success",
          "end": 1729706403591,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1729706402811",
          "message": "clean-vm",
          "start": 1729706402811,
          "status": "success",
          "end": 1729706403925,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1729706404428",
          "message": "snapshot",
          "start": 1729706404428,
          "status": "success",
          "end": 1729706417536,
          "result": "331c7509-15cc-06ce-a824-f834dfa42f88"
        },
        {
          "data": {
            "id": "8da40b08-636f-450d-af15-3264b9692e1f",
            "isFull": false,
            "type": "remote"
          },
          "id": "1729706417539",
          "message": "export",
          "start": 1729706417539,
          "status": "success",
          "tasks": [
            {
              "id": "1729706423056",
              "message": "transfer",
              "start": 1729706423056,
              "status": "success",
              "end": 1729706453043,
              "result": {
                "size": 2158497792
              }
            },
            {
              "id": "1729706457390",
              "message": "clean-vm",
              "start": 1729706457390,
              "status": "success",
              "end": 1729706458077,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1729706458082
        },
        {
          "data": {
            "id": "4b6d24a3-0b1e-48d5-aac2-a06e3a8ee485",
            "isFull": false,
            "type": "remote"
          },
          "id": "1729706417538",
          "message": "export",
          "start": 1729706417538,
          "status": "success",
          "tasks": [
            {
              "id": "1729706423053",
              "message": "transfer",
              "start": 1729706423053,
              "status": "success",
              "end": 1729706452967,
              "result": {
                "size": 2158497792
              }
            },
            {
              "id": "1729706458083",
              "message": "clean-vm",
              "start": 1729706458083,
              "status": "success",
              "end": 1729706458368,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1729706458371
        }
      ],
      "infos": [
        {
          "message": "Transfer data using NBD"
        }
      ],
      "end": 1729706458371
    }
  ],
  "end": 1729706458373
}

Tristis Oris

i also removed one 8.3 pool from nbd backup, create the usual one. Looks it work fine.

Tristis Oris

@jhansen ah. maybe only this way.

Tristis Oris

@jhansen yum update not intended.

Tristis Oris

Already every VM at 8.3 pool stuck at control domain. Can't detach them without pool reboot. Any workaround?

vbd.delete
{
  "id": "7af4b39d-d6aa-2589-509f-f082a1f0f2bb

"
}
{
  "code": "OPERATION_NOT_ALLOWED",
  "params": [
    "VBD '7af4b39d-d6aa-2589-509f-f082a1f0f2bb

' still attached to 'e9e4e600-b180-481d-9a1b-a45be8fc192e

'"
  ],
  "call": {
    "method": "VBD.destroy",
    "params": [
      "OpaqueRef:65481d14-d3de-82d7-fa7d-95131f14a209

"
    ]
  },
  "message": "OPERATION_NOT_ALLOWED(VBD '7af4b39d-d6aa-2589-509f-f082a1f0f2bb

' still attached to 'e9e4e600-b180-481d-9a1b-a45be8fc192e

')",
  "name": "XapiError",
  "stack": "XapiError: OPERATION_NOT_ALLOWED(VBD '7af4b39d-d6aa-2589-509f-f082a1f0f2bb

' still attached to 'e9e4e600-b180-481d-9a1b-a45be8fc192e

')
    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202410181215/packages/xen-api/_XapiError.mjs:16:12)
    at file:///opt/xo/xo-builds/xen-orchestra-202410181215/packages/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:60:5)
    at processImmediate (node:internal/timers:454:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}

Tristis Oris

@yann yep, already reinstall it.

Tristis Oris

last weeks i always get that problem. Only way to detach vdi from dom0 - reboot host(
xe vbd-unplug uuid= didn't work at that case.

Tristis Oris

@yann yay!

Tristis Oris

@yann so, i don't get indication about running master change and was able to run it again.

Tristis Oris

@Tristis-Oris after i fixed all dom0 stuck VDIs, backup succeed once and now stuck again.

Tristis Oris

on 8.3 pool more and more VMs became attached to Control Domain. Backups fail due VDI_IN_USE.

can't unplug vdi:
xe vbd-list vm-uuid=***
xe vbd-unplug uuid=***
The server failed to handle your request, due to an internal error. The given message may give details useful for debugging the problem.
message: Expected 0 or 1 VDI with datapath, had 5

can't migrate to 8.2 pool due incompatible versions.

and already got coalesce on VM without backup.

upd
removed all stuck vdi, but sr coalesce won't move.

Tristis Oris

@yann new one.

new part1 https://pastebin.com/ij0B7KHy
new part2 https://pastebin.com/0tQyvesY
old https://pastebin.com/dvAkEEbY

Tristis Oris

@yann

Tristis Oris

@yann pool>advanced>master.
Fresh xen installation, wait some time after joining last host, then changed master. Balancer disabled, no updates, few VMs runing on all hosts.

only one thing, master change don't work at 1st attempt - just nothing happens. Looks like task been canceled , so i repeat it again after some time.

i don't think it totaly broken, maybe something happens at same time. but never got that issue before.