Good morning, today i see a strange thing in my pool

I've not update ctx7..
I login to host (ctx7 and ctx6 for comparision), do a yum update and i see the same 9 packages: why xoa not see the patch for ctx7?

@olivierlambert said in Clarification of "VM Limits" in XO:

2. Static is the global range that can be modified only when the VM is halted. Dynamic is the range when the the VM memory can be changed while the VM is running. Obviously, dynamic range is included inside the static one.

Most of the time, except if you have a very good reason for it, do not use dynamic memory.

ok.. but if i set memory limit static 1gb-16gb
and dynamic 16gb-16gb XCP assign 16 gb to vm?
the static limit is only a "barrier" for dynamic?

@nikade
in VM (linux) with a free i see 94 gb of total memory

@bleader said in update via yum or via xoa?:

yes you're basically doing an RPU manually.
But it is indeed odd that the process is stuck at 0%, it should be fairly fast to do the install patches, no errors in the logs?

i've another install all patches and install all.
Now i'll use rpm update and see if speed is the same or not

@bleader said in update via yum or via xoa?:

It actually depends if you chose "rolling pool update" or "Install all pool patches", as you're talking about evacuate I assume you went with the first one.

Rolling pool update (RPU) is documented here and Pool updates here.

But to sum it up, "Install all pool patches" button will indeed run the yum update on all server, so similar as doing it manually, while RPU will do hosts one by one, moving VM to other hosts in the pool, install updates and reboot host. Therefore it can take way longer to complete, time will vary based on the number of VMs that have to be migrated around, network speed between hosts, etc…

RPU is the recommanded way as it allows hosts to restart and therefore take hypervisor, microcode and dom0 kernel updates into account right away with no service interruption. But if you don't really mind shutting down some of the VMs to restart hosts, or if there are no low level updates that requires a reboot, you could get away with just the yum update manually. But if the RPU is started already, I would not advise trying to do things manually at the same time.

ok, i've do a "install all patches" from xoa host page, i want some control form move vm to other hosts
But is the same thing at all (if i evacuate by hand the host, rpm update, reboot host), right?

Hi, i'm updating my poo; i evacuate the master, goes to patch, install all patch..
for now is at 0% (11 minutes from start)
Why is too slow?
It's the same if i go with console and do a yum update to all server?

Too many error for me..
Time for backup very very long, error at start (with retry) and at the end (with error on backup:

Couldn't deleted snapshot data
Couldn't deleted snapshot data
Retry the VM backup due to an error
the writer IncrementalRemoteWriter has failed the step writer.beforeBackup() with error Lock file is already being held. It won't be used anymore in this job execution.
Retry the VM backup due to an error
Transfer data using NBD
will delete snapshot data
Snapshot data has been deleted
Snapshot data has been deleted
Transfer data using NBD

    Clean VM directory
    cleanVm: incorrect backup size in metadata
    Start: 2024-07-11 20:23
    End: 2024-07-11 20:24
    Snapshot
    Start: 2024-07-11 20:24
    End: 2024-07-11 20:25
    Backup XEN OLD
    Start: 2024-07-11 20:25
    End: 2024-07-11 20:27
    Duration: 2 minutes
    Clean VM directory
    cleanVm: incorrect backup size in metadata
    Start: 2024-07-12 01:44
    End: 2024-07-12 01:44
    Snapshot
    Start: 2024-07-12 01:44
    End: 2024-07-12 01:46
    Backup XEN OLD
        transfer
        Start: 2024-07-12 01:46
        End: 2024-07-12 05:39
        Duration: 4 hours
        Size: 397.86 GiB
        Speed: 29.12 MiB/s
    Start: 2024-07-12 01:46
    End: 2024-07-12 05:39
    Duration: 4 hours

Start: 2024-07-11 20:23
End: 2024-07-12 05:39
Duration: 9 hours
Error: Disk is still attached to DOM0 VM
Type: delta

not good..

@rtjdamen said in CBT: the thread to centralize your feedback:

We have this error "stream has ended with not enough data (actual: 446, expected: 512)" on multiple vms in the last few days anyone seeing this issue?

the same for me.
Backup too long (delta for 18 hours, before nbd+cbt less than hour.. is a full, not delta) and at the end many many errors

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1720616400003",
  "jobId": "30159f11-3b54-48d1-ab8b-d32858991349",
  "jobName": "Delta FtPA",
  "message": "backup",
  "scheduleId": "b94e6227-c7b8-4a39-9bf1-b881422971df",
  "start": 1720616400003,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "be0a9812-fd14-be75-e2fa-40c31ce8875c"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "be0a9812-fd14-be75-e2fa-40c31ce8875c",
        "name_label": "FtPA"
      },
      "id": "1720616402582",
      "message": "backup VM",
      "start": 1720616402582,
      "status": "failure",
      "tasks": [
        {
          "id": "1720616402661",
          "message": "clean-vm",
          "start": 1720616402661,
          "status": "success",
          "warnings": [
            {
              "data": {
                "path": "/xo-vm-backups/be0a9812-fd14-be75-e2fa-40c31ce8875c/20240703T070203Z.json",
                "actual": 356950134784,
                "expected": 356950138368
              },
              "message": "cleanVm: incorrect backup size in metadata"
            }
          ],
          "end": 1720616408054,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1720616410224",
          "message": "snapshot",
          "start": 1720616410224,
          "status": "failure",
          "end": 1720616668284,
          "result": {
            "code": "HANDLE_INVALID",
            "params": [
              "VBD",
              "OpaqueRef:27874b37-4e3b-4d33-9a68-0d3dbaae7664"
            ],
            "task": {
              "uuid": "1823000a-df0a-970e-db2b-c12be53943fc",
              "name_label": "Async.VM.snapshot",
              "name_description": "",
              "allowed_operations": [],
              "current_operations": {},
              "created": "20240710T13:00:29Z",
              "finished": "20240710T13:04:28Z",
              "status": "failure",
              "resident_on": "OpaqueRef:4706cbe1-12ab-45d9-9001-cbe6ec1270ce",
              "progress": 1,
              "type": "<none/>",
              "result": "",
              "error_info": [
                "HANDLE_INVALID",
                "VBD",
                "OpaqueRef:27874b37-4e3b-4d33-9a68-0d3dbaae7664"
              ],
              "other_config": {},
              "subtask_of": "OpaqueRef:NULL",
              "subtasks": [],
              "backtrace": "(((process xapi)(filename ocaml/xapi-client/client.ml)(line 7))((process xapi)(filename ocaml/xapi-client/client.ml)(line 19))((process xapi)(filename ocaml/xapi-client/client.ml)(line 6016))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 134))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))"
            },
            "message": "HANDLE_INVALID(VBD, OpaqueRef:27874b37-4e3b-4d33-9a68-0d3dbaae7664)",
            "name": "XapiError",
            "stack": "XapiError: HANDLE_INVALID(VBD, OpaqueRef:27874b37-4e3b-4d33-9a68-0d3dbaae7664)\n    at XapiError.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)\n    at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:13:29)\n    at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1033:24)\n    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1067:14\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1057:12)\n    at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1230:14)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "id": "1720616668336",
          "message": "clean-vm",
          "start": 1720616668336,
          "status": "success",
          "end": 1720616674074,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1720616674189",
          "message": "clean-vm",
          "start": 1720616674189,
          "status": "success",
          "end": 1720616679904,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1720616681937",
          "message": "snapshot",
          "start": 1720616681937,
          "status": "success",
          "end": 1720616891256,
          "result": "4f5c9b99-d96e-4e1f-5f2e-0b9e7fa28952"
        },
        {
          "data": {
            "id": "601f8729-7602-4d6f-a018-d4cc525ca371",
            "isFull": false,
            "type": "remote"
          },
          "id": "1720616891257",
          "message": "export",
          "start": 1720616891257,
          "status": "success",
          "tasks": [
            {
              "id": "1720617038363",
              "message": "clean-vm",
              "start": 1720617038363,
              "status": "success",
              "end": 1720617044510,
              "result": {
                "merge": true
              }
            }
          ],
          "end": 1720617044698
        },
        {
          "id": "1720617044736",
          "message": "clean-vm",
          "start": 1720617044736,
          "status": "failure",
          "tasks": [
            {
              "id": "1720617048047",
              "message": "merge",
              "start": 1720617048047,
              "status": "failure",
              "end": 1720617656522,
              "result": {
                "errno": -2,
                "code": "ENOENT",
                "syscall": "open",
                "path": "/run/xo-server/mounts/601f8729-7602-4d6f-a018-d4cc525ca371/xo-vm-backups/be0a9812-fd14-be75-e2fa-40c31ce8875c/vdis/30159f11-3b54-48d1-ab8b-d32858991349/1f3803c8-0335-4470-8c26-297f98af442c/20240703T070203Z.vhd",
                "message": "ENOENT: no such file or directory, open '/run/xo-server/mounts/601f8729-7602-4d6f-a018-d4cc525ca371/xo-vm-backups/be0a9812-fd14-be75-e2fa-40c31ce8875c/vdis/30159f11-3b54-48d1-ab8b-d32858991349/1f3803c8-0335-4470-8c26-297f98af442c/20240703T070203Z.vhd'",
                "name": "Error",
                "stack": "Error: ENOENT: no such file or directory, open '/run/xo-server/mounts/601f8729-7602-4d6f-a018-d4cc525ca371/xo-vm-backups/be0a9812-fd14-be75-e2fa-40c31ce8875c/vdis/30159f11-3b54-48d1-ab8b-d32858991349/1f3803c8-0335-4470-8c26-297f98af442c/20240703T070203Z.vhd'\nFrom:\n    at NfsHandler.addSyncStackTrace (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/dist/local.js:21:26)\n    at NfsHandler._openFile (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/dist/local.js:154:35)\n    at NfsHandler.__openFile (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/dist/abstract.js:448:51)\n    at NfsHandler.openFile (/usr/local/lib/node_modules/xo-server/node_modules/limit-concurrency-decorator/index.js:97:24)\n    at VhdFile.open (/usr/local/lib/node_modules/xo-server/node_modules/vhd-lib/Vhd/VhdFile.js:86:30)\n    at openVhd (/usr/local/lib/node_modules/xo-server/node_modules/vhd-lib/openVhd.js:15:28)\n    at async #openVhds (/usr/local/lib/node_modules/xo-server/node_modules/vhd-lib/merge.js:118:23)\n    at async Disposable.<anonymous> (/usr/local/lib/node_modules/xo-server/node_modules/vhd-lib/merge.js:164:39)"
              }
            }
          ],
          "end": 1720617656522,
          "result": {
            "errno": -2,
            "code": "ENOENT",
            "syscall": "open",
            "path": "/run/xo-server/mounts/601f8729-7602-4d6f-a018-d4cc525ca371/xo-vm-backups/be0a9812-fd14-be75-e2fa-40c31ce8875c/vdis/30159f11-3b54-48d1-ab8b-d32858991349/1f3803c8-0335-4470-8c26-297f98af442c/20240703T070203Z.vhd",
            "message": "ENOENT: no such file or directory, open '/run/xo-server/mounts/601f8729-7602-4d6f-a018-d4cc525ca371/xo-vm-backups/be0a9812-fd14-be75-e2fa-40c31ce8875c/vdis/30159f11-3b54-48d1-ab8b-d32858991349/1f3803c8-0335-4470-8c26-297f98af442c/20240703T070203Z.vhd'",
            "name": "Error",
            "stack": "Error: ENOENT: no such file or directory, open '/run/xo-server/mounts/601f8729-7602-4d6f-a018-d4cc525ca371/xo-vm-backups/be0a9812-fd14-be75-e2fa-40c31ce8875c/vdis/30159f11-3b54-48d1-ab8b-d32858991349/1f3803c8-0335-4470-8c26-297f98af442c/20240703T070203Z.vhd'\nFrom:\n    at NfsHandler.addSyncStackTrace (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/dist/local.js:21:26)\n    at NfsHandler._openFile (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/dist/local.js:154:35)\n    at NfsHandler.__openFile (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/dist/abstract.js:448:51)\n    at NfsHandler.openFile (/usr/local/lib/node_modules/xo-server/node_modules/limit-concurrency-decorator/index.js:97:24)\n    at VhdFile.open (/usr/local/lib/node_modules/xo-server/node_modules/vhd-lib/Vhd/VhdFile.js:86:30)\n    at openVhd (/usr/local/lib/node_modules/xo-server/node_modules/vhd-lib/openVhd.js:15:28)\n    at async #openVhds (/usr/local/lib/node_modules/xo-server/node_modules/vhd-lib/merge.js:118:23)\n    at async Disposable.<anonymous> (/usr/local/lib/node_modules/xo-server/node_modules/vhd-lib/merge.js:164:39)"
          }
        },
        {
          "id": "1720617656648",
          "message": "snapshot",
          "start": 1720617656648,
          "status": "success",
          "end": 1720618073493,
          "result": "b5ae2b32-ef3c-aa99-4e1f-07d42835746a"
        },
        {
          "data": {
            "id": "601f8729-7602-4d6f-a018-d4cc525ca371",
            "isFull": true,
            "type": "remote"
          },
          "id": "1720618073494",
          "message": "export",
          "start": 1720618073494,
          "status": "success",
          "tasks": [
            {
              "id": "1720618090517",
              "message": "transfer",
              "start": 1720618090517,
              "status": "success",
              "end": 1720680654839,
              "result": {
                "size": 2496704791552
              }
            },
            {
              "id": "1720680657579",
              "message": "clean-vm",
              "start": 1720680657579,
              "status": "success",
              "warnings": [
                {
                  "data": {
                    "mergeStatePath": "/xo-vm-backups/be0a9812-fd14-be75-e2fa-40c31ce8875c/vdis/30159f11-3b54-48d1-ab8b-d32858991349/9becf507-abfc-47b4-9091-803ef2a1b47c/.20240703T070203Z.vhd.merge.json",
                    "missingVhdPath": "/xo-vm-backups/be0a9812-fd14-be75-e2fa-40c31ce8875c/vdis/30159f11-3b54-48d1-ab8b-d32858991349/9becf507-abfc-47b4-9091-803ef2a1b47c/20240703T070203Z.vhd"
                  },
                  "message": "orphan merge state"
                },
                {
                  "data": {
                    "path": "/xo-vm-backups/be0a9812-fd14-be75-e2fa-40c31ce8875c/20240703T090201Z.json",
                    "actual": 83366506496,
                    "expected": 360746910208
                  },
                  "message": "cleanVm: incorrect backup size in metadata"
                }
              ],
              "end": 1720680685535,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1720680685538
        }
      ],
      "warnings": [
        {
          "data": {
            "attempt": 1,
            "error": "HANDLE_INVALID(VBD, OpaqueRef:27874b37-4e3b-4d33-9a68-0d3dbaae7664)"
          },
          "message": "Retry the VM backup due to an error"
        },
        {
          "data": {
            "error": {
              "code": "VDI_IN_USE",
              "params": [
                "OpaqueRef:7462ea3f-8b99-444e-9007-07529868daf2",
                "data_destroy"
              ],
              "call": {
                "method": "VDI.data_destroy",
                "params": [
                  "OpaqueRef:7462ea3f-8b99-444e-9007-07529868daf2"
                ]
              }
            },
            "vdiRef": "OpaqueRef:7462ea3f-8b99-444e-9007-07529868daf2"
          },
          "message": "Couldn't deleted snapshot data"
        },
        {
          "data": {
            "error": {
              "code": "VDI_IN_USE",
              "params": [
                "OpaqueRef:1dfc4766-a3b1-4540-ba6a-8c0eab4dbaca",
                "data_destroy"
              ],
              "call": {
                "method": "VDI.data_destroy",
                "params": [
                  "OpaqueRef:1dfc4766-a3b1-4540-ba6a-8c0eab4dbaca"
                ]
              }
            },
            "vdiRef": "OpaqueRef:1dfc4766-a3b1-4540-ba6a-8c0eab4dbaca"
          },
          "message": "Couldn't deleted snapshot data"
        },
        {
          "data": {
            "attempt": 2,
            "error": "stream has ended with not enough data (actual: 446, expected: 512)"
          },
          "message": "Retry the VM backup due to an error"
        }
      ],
      "infos": [
        {
          "message": "will delete snapshot data"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:18043fd4-1a85-495a-b011-18ce047e46de"
          },
          "message": "Snapshot data has been deleted"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:e4f89ea5-626d-4131-a9ce-ed330d3b2aec"
          },
          "message": "Snapshot data has been deleted"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:1b61a681-598b-4d6e-92c3-05c384fa0164"
          },
          "message": "Snapshot data has been deleted"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:ec381ed4-9d2d-407a-b56b-203c8029fcee"
          },
          "message": "Snapshot data has been deleted"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:abc11faa-73b0-46c3-b3dc-ffe8752671a7"
          },
          "message": "Snapshot data has been deleted"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:a387b947-47bf-4339-92c8-0d749803115f"
          },
          "message": "Snapshot data has been deleted"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:2f054a5c-f1f6-4b1d-9d54-f2e8d74b5757"
          },
          "message": "Snapshot data has been deleted"
        },
        {
          "message": "Transfer data using NBD"
        }
      ],
      "end": 1720680685540,
      "result": {
        "generatedMessage": false,
        "code": "ERR_ASSERTION",
        "actual": true,
        "expected": false,
        "operator": "strictEqual",
        "message": "Disk is still attached to DOM0 VM",
        "name": "AssertionError",
        "stack": "AssertionError [ERR_ASSERTION]: Disk is still attached to DOM0 VM\n    at Array.<anonymous> (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:244:20)\n    at Function.from (<anonymous>)\n    at asyncMap (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/async-map/index.js:21:28)\n    at Array.<anonymous> (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:233:13)\n    at Function.from (<anonymous>)\n    at asyncMap (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/async-map/index.js:21:28)\n    at IncrementalXapiVmBackupRunner._removeUnusedSnapshots (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:219:11)\n    at IncrementalXapiVmBackupRunner.run (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:375:18)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_runners/VmsXapi.mjs:166:38"
      }
    }
  ],
  "end": 1720680685540
}

Hi, why this difference?

The backup joband the VM are the same

@olivierlambert no xostor, sr is on fiber channel san (two san with some volumes, HDD or SSD)
may i try the reactivation of both?

A little question: in my environment i've big problem with SR migration.
i've opened a ticket and the answer is:
There is currently a problem with NBD backup where the VDI are not correctly disconnected from the Dom0.
The team is investigating but for now if you want to avoid more being accumulated, it is advised to disable NBD backups.
I've disabled NBD and for small SR work fine, for big SR i can't try (three days estimated for migration..)

This problem is resolved?
Or i can't use CBT?

@olivierlambert how many time for us with precompiled XOA?

Hi, i've this situation in Task

What's this task?
Now i've ~50 of this call..

@olivierlambert now work fine, thank you

@robyt ok, i've drop and reinstall new XOA and work fine

Hi, ive deploy new xoa on my pool but i can't login.
I've use the default password, i've use the xo-server-recover-account with 2 email but nothing, i've invalid credentials
whats'up doc?

@CJ said in two separate process for backup?:

@robyt said in two separate process for backup?:

@CJ said in two separate process for backup?:

@DustinB What are you using for cloud sync?

is not a cloud sync, is a copy to a qnap connected via gigabit ethernet

You'll note that that was a reply to @DustinB who is using cloud sync, not you.

@robyt said in two separate process for backup?:

@CJ said in two separate process for backup?:

@robyt When you say 10/12 NBD, are you referring to the setting of NBD connections per disk? Or connexion as the UI says. How are you checking that NBD is being used?

Connexion in UI

What happens if you reduce the number? From my understanding your Qnap isn't all that powerful, so it sounds like you're overwhelming the poor thing. I'm using 1 per disk and seeing over 1Gbps speeds. My main limitation is the 2.5G connection to XO.

However, I will say that I'm not sure if I'm actually using NBD or not. I don't see anything in the UI to indicate whether it's being taken advantage of. I also have my backup concurrency set to 1 to avoid running multiple transfers through XO as the switching can eat into the overall bandwidth.

when i've incremented connexion i see a lot of http timeout.
In UI if i open the detail of a backup job i see "Transfer data using NBD"
I know that qnap is not a wonderful and very fast nas but i want buy another (and more expensive) hardware without more test
Some vm is very very slow (the transfer of mailserver's snapshot is.. slooooooowwww) but the configuration is the same (network, part of disk in SSD and part in mechanical HD etc)

@robyt ops....

@CJ said in two separate process for backup?:

@robyt When you say 10/12 NBD, are you referring to the setting of NBD connections per disk? Or connexion as the UI says. How are you checking that NBD is being used?

Connexion in UI

@CJ said in two separate process for backup?:

@DustinB What are you using for cloud sync?

is not a cloud sync, is a copy to a qnap connected via gigabit ethernet