snapshot with memory crashing debian VM

manilx

When I try to create a snapshot of a debian VM it crashes with the following error:

vm.snapshot
{
  "id": "0c8fd092-ab36-478e-df85-16b1a01e1ca8",
  "saveMemory": true
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "xenopsd internal error: Failure(\"Error from xenguesthelper: xenguest Invalid argument\")"
  ],
  "task": {
    "uuid": "9b776334-5257-13aa-b5be-6e7c7ddc1c72",
    "name_label": "Async.VM.checkpoint",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20250105T18:39:23Z",
    "finished": "20250105T18:40:34Z",
    "status": "failure",
    "resident_on": "OpaqueRef:a7d64d2c-7f91-8c95-4173-9d211e12a2d8",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "INTERNAL_ERROR",
      "xenopsd internal error: Failure(\"Error from xenguesthelper: xenguest Invalid argument\")"
    ],
    "other_config": {
      "debug_info:cancel_points_seen": "26"
    },
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi-client/client.ml)(line 7))((process xapi)(filename ocaml/xapi-client/client.ml)(line 19))((process xapi)(filename ocaml/xapi-client/client.ml)(line 5833))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 146))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1737))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))"
  },
  "message": "INTERNAL_ERROR(xenopsd internal error: Failure(\"Error from xenguesthelper: xenguest Invalid argument\"))",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(xenopsd internal error: Failure(\"Error from xenguesthelper: xenguest Invalid argument\"))
    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202412311421/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xo/xo-builds/xen-orchestra-202412311421/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202412311421/packages/xen-api/index.mjs:1068:24)
    at file:///opt/xo/xo-builds/xen-orchestra-202412311421/packages/xen-api/index.mjs:1102:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202412311421/packages/xen-api/index.mjs:1092:12)
    at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202412311421/packages/xen-api/index.mjs:1265:14)"
}

This is repeatable on this VM and I have observed this on others too (Linux/Windows). Sometimes it works but it often crashes.

I don't use memory snapshots frequently, mostly just normal ones. So I never reported this. But this has been the case for a long time.

olivierlambert

Hi,

Which Debian version and which kernel + which xen tools are you using (and version too!). Thanks

manilx

@olivierlambert Debian 12 fully patched, 6.1.0-28-amd64, XO latest, Rust guest tools

manilx

@manilx P.S: On Windows latest XenServer VM Tools

olivierlambert

@dinhngtu Does this message rings a bell to you?

    "error_info": [
      "INTERNAL_ERROR",
      "xenopsd internal error: Failure(\"Error from xenguesthelper: xenguest Invalid argument\")"
    ],
    "other_config": {
      "debug_info:cancel_points_seen": "26"
    },

dinhngtu

@manilx @olivierlambert Sorry, I'm not familiar with this error. Are there any errors coming from emu-manager in the syslog?

manilx

@dinhngtu Please provide me with exact commands to check whatever you need....
I'll do a snapshot (crash is repeatable) and report

manilx

@olivierlambert @dinhngtu As a test I uninstalled rust tools, installed the old tolls from the CD and tried a mem-snap:

vm.snapshot
{
  "id": "0c8fd092-ab36-478e-df85-16b1a01e1ca8",
  "saveMemory": true
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "xenopsd internal error: Failure(\"Error from xenguesthelper: xenguest Invalid argument\")"
  ],
  "task": {
    "uuid": "ecc2199c-f64a-2b1f-6344-a6ebee708975",
    "name_label": "Async.VM.checkpoint",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20250106T12:38:02Z",
    "finished": "20250106T12:39:03Z",
    "status": "failure",
    "resident_on": "OpaqueRef:a7d64d2c-7f91-8c95-4173-9d211e12a2d8",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "INTERNAL_ERROR",
      "xenopsd internal error: Failure(\"Error from xenguesthelper: xenguest Invalid argument\")"
    ],
    "other_config": {
      "debug_info:cancel_points_seen": "28"
    },
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi-client/client.ml)(line 7))((process xapi)(filename ocaml/xapi-client/client.ml)(line 19))((process xapi)(filename ocaml/xapi-client/client.ml)(line 5833))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 146))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1737))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))"
  },
  "message": "INTERNAL_ERROR(xenopsd internal error: Failure(\"Error from xenguesthelper: xenguest Invalid argument\"))",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(xenopsd internal error: Failure(\"Error from xenguesthelper: xenguest Invalid argument\"))
    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202501061032/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xo/xo-builds/xen-orchestra-202501061032/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202501061032/packages/xen-api/index.mjs:1068:24)
    at file:///opt/xo/xo-builds/xen-orchestra-202501061032/packages/xen-api/index.mjs:1102:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202501061032/packages/xen-api/index.mjs:1092:12)
    at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202501061032/packages/xen-api/index.mjs:1265:14)"
}

dinhngtu

@manilx Try grep emu-manager /var/log/daemon.log /var/log/xensource.log. If there's no output could you look into the entire file instead?

manilx

@dinhngtu I have none of those logs...

ScreenShot 2025-01-06 at 13.27.42.png

dinhngtu

@manilx Sorry, I meant in the host.

manilx

@dinhngtu

[13:43 host2 ~]# grep emu-manager /var/log/daemon.log /var/log/xensource.log
/var/log/daemon.log:Jan  6 12:38:03 host2 emu-manager-100[2915195]: Startup: xenopsd control fds (8, 7).
/var/log/daemon.log:Jan  6 12:38:03 host2 emu-manager-100[2915195]: Startup: domid 100.
/var/log/daemon.log:Jan  6 12:38:03 host2 emu-manager-100[2915195]: Startup: operation mode (hvm_save, non-live).
/var/log/daemon.log:Jan  6 12:38:03 host2 emu-manager-100[2915195]: Emu `xenguest` is enabled.
/var/log/daemon.log:Jan  6 12:38:03 host2 emu-manager-100[2915195]: Emu `qemu` is enabled.
/var/log/daemon.log:Jan  6 12:38:03 host2 emu-manager-100[2915195]: Starting `/usr/libexec/xen/bin/xenguest`...
/var/log/daemon.log:Jan  6 12:38:03 host2 emu-manager-100[2915195]: Connecting to `xenguest` (/run/xen/xenguest-control-100)...
/var/log/daemon.log:Jan  6 12:38:42 host2 emu-manager-100[2915195]: Emu `xenguest` is completed.
/var/log/daemon.log:Jan  6 12:38:42 host2 emu-manager-100[2915195]: Emu `xenguest` completed normally.
/var/log/daemon.log:Jan  6 12:38:49 host2 emu-manager-101[2916301]: Startup: xenopsd control fds (8, 7).
/var/log/daemon.log:Jan  6 12:38:49 host2 emu-manager-101[2916301]: Startup: domid 101.
/var/log/daemon.log:Jan  6 12:38:49 host2 emu-manager-101[2916301]: Startup: operation mode (hvm_restore, non-live).
/var/log/daemon.log:Jan  6 12:38:49 host2 emu-manager-101[2916301]: Emu `xenguest` is enabled.
/var/log/daemon.log:Jan  6 12:38:49 host2 emu-manager-101[2916301]: Emu `qemu` is enabled.
/var/log/daemon.log:Jan  6 12:38:49 host2 emu-manager-101[2916301]: Starting `/usr/libexec/xen/bin/xenguest`...
/var/log/daemon.log:Jan  6 12:38:49 host2 emu-manager-101[2916301]: Connecting to `xenguest` (/run/xen/xenguest-control-101)...
/var/log/daemon.log:Jan  6 12:39:01 host2 emu-manager-101[2916301]: poll failed because revents=0x11 for `xenguest`.
/var/log/daemon.log:Jan  6 12:39:01 host2 emu-manager-101[2916301]: Error for emu `xenguest`: wait_for_event => Invalid argument
/var/log/daemon.log:Jan  6 12:39:01 host2 emu-manager-101[2916301]: Error waiting for events: `Invalid argument`.
/var/log/daemon.log:Jan  6 12:39:01 host2 emu-manager-101[2916301]: Error sending message to emu client: `Broken pipe`.
/var/log/daemon.log:Jan  6 12:39:01 host2 emu-manager-101[2916301]: Emu `xenguest` exited with an error: 1.
/var/log/daemon.log:Jan  6 12:39:01 host2 emu-manager-101[2916301]: Reporting: `error:xenguest Invalid argument#012`...
/var/log/daemon.log:Jan  6 12:39:01 host2 forkexecd: [error||0 ||forkexecd] 2916301 (/usr/lib64/xen/bin/emu-manager -controloutfd 7 -controlinfd 8 -mode hvm_resto...) exited with code 1
/var/log/xensource.log:Jan  6 12:38:03 host2 xenopsd-xc: [debug||34 |Async.VM.checkpoint R:8437e831f497|xenops] looking for emu-manager in [ featureset : 1fcbfbff-f7fa3203-2c100800-00000121-0000000f-219c07ab-8040070c-00000000-00101000-bc004410-00000810-00000000-00000000-00000017-00000000-00000000-1408e163-00000000-00000000-00000000-00000000-00000000; timeoffset : 0; usb : true; usb_tablet : true; secureboot : false; device-model : qemu-upstream-compat; vga : std; videoram : 8; viridian : false; device_id : 0001; nx : true; acpi : 1; apic : true; pae : true; hpet : true ]
/var/log/xensource.log:Jan  6 12:38:03 host2 xenopsd-xc: [debug||34 |Async.VM.checkpoint R:8437e831f497|emu-manager] connect: args = [ -fd 76f7d8af-896c-3d07-fc08-fb08bae47391 -mode hvm_save -domid 100 -dm qemu ]
/var/log/xensource.log:Jan  6 12:38:03 host2 xenopsd-xc: [debug||34 |Async.VM.checkpoint R:8437e831f497|xenops] VM = 0c8fd092-ab36-478e-df85-16b1a01e1ca8; domid = 100; waiting for emu-manager...
/var/log/xensource.log:Jan  6 12:38:03 host2 xenopsd-xc: [debug||34 |Async.VM.checkpoint R:8437e831f497|xenops] VM = 0c8fd092-ab36-478e-df85-16b1a01e1ca8; domid = 100; message from emu-manager: suspend:
/var/log/xensource.log:Jan  6 12:38:03 host2 xenopsd-xc: [debug||34 |Async.VM.checkpoint R:8437e831f497|xenops] VM = 0c8fd092-ab36-478e-df85-16b1a01e1ca8; domid = 100; waiting for emu-manager...
/var/log/xensource.log:Jan  6 12:38:03 host2 xenopsd-xc: [debug||34 |Async.VM.checkpoint R:8437e831f497|xenops] VM = 0c8fd092-ab36-478e-df85-16b1a01e1ca8; domid = 100; message from emu-manager: prepare:xenguest
/var/log/xensource.log:Jan  6 12:38:03 host2 xenopsd-xc: [debug||34 |Async.VM.checkpoint R:8437e831f497|xenops] VM = 0c8fd092-ab36-478e-df85-16b1a01e1ca8; domid = 100; waiting for emu-manager...
/var/log/xensource.log:Jan  6 12:38:42 host2 xenopsd-xc: [debug||34 |Async.VM.checkpoint R:8437e831f497|xenops] VM = 0c8fd092-ab36-478e-df85-16b1a01e1ca8; domid = 100; message from emu-manager: result:0 0
/var/log/xensource.log:Jan  6 12:38:42 host2 xenopsd-xc: [debug||34 |Async.VM.checkpoint R:8437e831f497|xenops] VM = 0c8fd092-ab36-478e-df85-16b1a01e1ca8; domid = 100; emu-manager completed successfully
/var/log/xensource.log:Jan  6 12:38:49 host2 xenopsd-xc: [debug||11 |Async.VM.checkpoint R:8437e831f497|xenops] looking for emu-manager in [ featureset : 1fcbfbff-f7fa3203-2c100800-00000121-0000000f-219c07ab-8040070c-00000000-00101000-bc004410-00000810-00000000-00000000-00000017-00000000-00000000-1408e163-00000000-00000000-00000000-00000000-00000000; timeoffset : 0; usb : true; usb_tablet : true; secureboot : false; device-model : qemu-upstream-compat; vga : std; videoram : 8; viridian : false; device_id : 0001; nx : true; acpi : 1; apic : true; pae : true; hpet : true ]
/var/log/xensource.log:Jan  6 12:38:49 host2 xenopsd-xc: [debug||11 |Async.VM.checkpoint R:8437e831f497|emu-manager] connect: args = [ -mode hvm_restore -domid 101 -fd cb83bbec-05b1-75e6-1d5d-89401e13a4f4 -store_port 9 -console_port 10 -dm qemu ]
/var/log/xensource.log:Jan  6 12:38:49 host2 xenopsd-xc: [debug||11 |Async.VM.checkpoint R:8437e831f497|mig64] Waiting for response from emu-manager
/var/log/xensource.log:Jan  6 12:38:49 host2 xenopsd-xc: [debug||25900 |Async.VM.checkpoint R:8437e831f497|mig64] Sent restore:xenguest to emu-manager. Waiting for result...

dinhngtu

@manilx Could you post the entirety of /var/log/daemon.log and /var/log/xensource.log?

manilx

@dinhngtu Here it is (I've included the archived logs from yesterday also, when I did the snapshots in the afternoon):

https://www.dropbox.com/scl/fi/14mg7vsgupbjaan4bsn66/Archive.zip?rlkey=2qbcrphoo0ncgwmv1ydyv89s7&dl=0

dinhngtu

@manilx Could you gather the output of xl dmesg as well?

manilx

@dinhngtu Sure:

 __  __            _  _    _ _____ ____
 \ \/ /___ _ __   | || |  / |___  | ___|
  \  // _ \ '_ \  | || |_ | |  / /|___ \
  /  \  __/ | | | |__   _|| | / /_ ___) |
 /_/\_\___|_| |_|    |_|(_)_|/_/(_)____/

(XEN) [0000000d19af3089] Xen version 4.17.5-4 (mockbuild@[unknown]) (gcc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1)) debug=n Wed Nov 13 11:42:55 CET 2024
(XEN) [0000000d19af40cf] Latest ChangeSet: 430ce6cd9365, pq f09cf5633773
(XEN) [0000000d19af53b9] build-id: 0c6d91001e3befae6cbe6dbb635b6313c99e706c
(XEN) [0000000d19af584b] Bootloader: GRUB 2.06
(XEN) [0000000d19af5d37] Command line: watchdog ucode=scan dom0_max_vcpus=1-12 crashkernel=256M,below=4G console=vga vga=mode-0x0311 dom0_mem=8589934592B,max:8589934592B
(XEN) [0000000d19af659b] Xen image load base address: 0x6ca00000
(XEN) [0000000d19af69a3] Video information:
(XEN) [0000000d19af6eb9]  VGA is graphics mode 1920x1080, 32 bpp
(XEN) [0000000d19af72f7] Disc information:
(XEN) [0000000d19af771f]  Found 0 MBR signatures
(XEN) [0000000d19af7b0f]  Found 2 EDD information structures
(XEN) [0000000d19af9e31] CPU Vendor: Intel, Family 6 (0x6), Model 154 (0x9a), Stepping 4 (raw 000906a4)
(XEN) [0000000d19afbab1] Enabling Supervisor Shadow Stacks
(XEN) [0000000d19afc46b] Enabling Indirect Branch Tracking
(XEN) [0000000d19afc959]   - IBT disabled in UEFI Runtime Services
(XEN) [0000000d19aff263] EFI RAM map:
(XEN) [0000000d19aff9bd]  [0000000000000000, 000000000009dfff] (usable)
(XEN) [0000000d19b000fd]  [000000000009e000, 000000000009efff] (reserved)
(XEN) [0000000d19b006f3]  [000000000009f000, 000000000009ffff] (usable)
(XEN) [0000000d19b00c63]  [00000000000a0000, 00000000000fffff] (reserved)
(XEN) [0000000d19b012e5]  [0000000000100000, 0000000072054fff] (usable)
(XEN) [0000000d19b018c5]  [0000000072055000, 0000000075154fff] (reserved)
(XEN) [0000000d19b01e09]  [0000000075155000, 000000007526efff] (ACPI data)
(XEN) [0000000d19b023af]  [000000007526f000, 000000007534bfff] (ACPI NVS)
(XEN) [0000000d19b02929]  [000000007534c000, 0000000075ffefff] (reserved)
(XEN) [0000000d19b02e37]  [0000000075fff000, 0000000075ffffff] (usable)
(XEN) [0000000d19b0333f]  [0000000076000000, 0000000079ffffff] (reserved)
(XEN) [0000000d19b03849]  [000000007aa00000, 000000007abfffff] (reserved)
(XEN) [0000000d19b03d39]  [000000007b000000, 00000000803fffff] (reserved)
(XEN) [0000000d19b04221]  [00000000c0000000, 00000000cfffffff] (reserved)
(XEN) [0000000d19b04721]  [00000000fe000000, 00000000fe010fff] (reserved)
(XEN) [0000000d19b04c07]  [00000000fec00000, 00000000fec00fff] (reserved)
(XEN) [0000000d19b050f1]  [00000000fed00000, 00000000fed00fff] (reserved)
(XEN) [0000000d19b055d9]  [00000000fed20000, 00000000fed7ffff] (reserved)
(XEN) [0000000d19b05ad1]  [00000000fee00000, 00000000fee00fff] (reserved)
(XEN) [0000000d19b05fbd]  [00000000ff000000, 00000000ffffffff] (reserved)
(XEN) [0000000d19b065cb]  [0000000100000000, 000000187fbfffff] (usable)
(XEN) [0000000d1aa26f39] Kdump: 256MB (262144kB) at 0x5cc00000
(XEN) [0000000d1aa2d8ad] ACPI: RSDP 752E8014, 0024 (r2 ALASKA)
(XEN) [0000000d1aa2ec53] ACPI: XSDT 752E7728, 011C (r1 ALASKA   A M I   1072009 AMI   1000013)
(XEN) [0000000d1aa2fde9] ACPI: FACP 7526C000, 0114 (r6 ALASKA   A M I   1072009 AMI   1000013)
(XEN) [0000000d1aa31077] ACPI: DSDT 751D1000, 9A360 (r2 ALASKA   A M I   1072009 INTL 20200717)
(XEN) [0000000d1aa31b99] ACPI: FACS 7534B000, 0040
(XEN) [0000000d1aa323fb] ACPI: FIDT 751D0000, 009C (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) [0000000d1aa32eed] ACPI: SSDT 7526E000, 038C (r2 PmaxDv Pmax_Dev        1 INTL 20200717)
(XEN) [0000000d1aa339cf] ACPI: SSDT 751CA000, 5D34 (r2 CpuRef  CpuSsdt     3000 INTL 20200717)
(XEN) [0000000d1aa34491] ACPI: SSDT 751C7000, 2B7D (r2 SaSsdt  SaSsdt      3000 INTL 20200717)
(XEN) [0000000d1aa34e4b] ACPI: SSDT 751C3000, 33E9 (r2 INTEL  IgfxSsdt     3000 INTL 20200717)
(XEN) [0000000d1aa3583b] ACPI: SSDT 751B5000, D7C2 (r2 INTEL  TcssSsdt     1000 INTL 20200717)
(XEN) [0000000d1aa361e7] ACPI: HPET 7526D000, 0038 (r1 ALASKA   A M I   1072009 AMI   1000013)
(XEN) [0000000d1aa36c1b] ACPI: APIC 751B4000, 01DC (r5 ALASKA   A M I   1072009 AMI   1000013)
(XEN) [0000000d1aa375d3] ACPI: MCFG 751B3000, 003C (r1 ALASKA   A M I   1072009 AMI   1000013)
(XEN) [0000000d1aa37f81] ACPI: SSDT 751B1000, 1F1A (r2 ALASKA Ther_Rvp     1000 INTL 20200717)
(XEN) [0000000d1aa38967] ACPI: UEFI 7529E000, 0048 (r1 ALASKA   A M I   1072009 AMI   1000013)
(XEN) [0000000d1aa3931d] ACPI: SDEV 751B0000, 0046 (r1                        0             0)
(XEN) [0000000d1aa39deb] ACPI: NHLT 751AF000, 002D (r0 ALASKA   A M I   1072009 AMI   1000013)
(XEN) [0000000d1aa3a85d] ACPI: LPIT 751AE000, 00CC (r1 ALASKA   A M I   1072009 AMI   1000013)
(XEN) [0000000d1aa3b27b] ACPI: SSDT 751AA000, 2A83 (r2 ALASKA PtidDevc     1000 INTL 20200717)
(XEN) [0000000d1aa3bc77] ACPI: SSDT 751A1000, 8F27 (r2 ALASKA TbtTypeC        0 INTL 20200717)
(XEN) [0000000d1aa3c655] ACPI: DBGP 751A0000, 0034 (r1 ALASKA   A M I   1072009 AMI   1000013)
(XEN) [0000000d1aa3d011] ACPI: DBG2 7519F000, 0054 (r0 ALASKA   A M I   1072009 AMI   1000013)
(XEN) [0000000d1aa3d9f7] ACPI: SSDT 7519D000, 190A (r2 ALASKA UsbCTabl     1000 INTL 20200717)
(XEN) [0000000d1aa3e3ab] ACPI: DMAR 7519C000, 0088 (r2 INTEL  EDK2            2       1000013)
(XEN) [0000000d1aa3edab] ACPI: FPDT 7519B000, 0044 (r1 ALASKA   A M I   1072009 AMI   1000013)
(XEN) [0000000d1aa3f74d] ACPI: SSDT 7519A000, 07AE (r2  INTEL xh_adlLP        0 INTL 20200717)
(XEN) [0000000d1aa401bb] ACPI: SSDT 75196000, 3AEA (r2 SocGpe  SocGpe      3000 INTL 20200717)
(XEN) [0000000d1aa40bc5] ACPI: SSDT 75192000, 39DA (r2 SocCmn  SocCmn      3000 INTL 20200717)
(XEN) [0000000d1aa4159f] ACPI: SSDT 75191000, 0144 (r2 Intel  ADebTabl     1000 INTL 20200717)
(XEN) [0000000d1aa41f9b] ACPI: SPCR 75190000, 0050 (r2 ALASKA   A M I   1072009 AMI     5001B)
(XEN) [0000000d1aa42951] ACPI: BGRT 7518F000, 0038 (r1 ALASKA   A M I   1072009 AMI     10013)
(XEN) [0000000d1aa432eb] ACPI: TPM2 7518E000, 004C (r4 ALASKA   A M I         1 AMI         0)
(XEN) [0000000d1aa43c8d] ACPI: PHAT 7518D000, 05D1 (r1 ALASKA   A M I         5 MSFT  100000D)
(XEN) [0000000d1aa44663] ACPI: WSMT 751AD000, 0028 (r1 ALASKA   A M I   1072009 AMI     10013)
(XEN) [0000000d1aa75db3] System RAM: 98075MB (100429780kB)
(XEN) [0000000d21f12a89] No NUMA configuration found
(XEN) [0000000d21f133f5] Faking a node at 0000000000000000-000000187fc00000
(XEN) [0000000d2f3941a9] Domain heap initialised
(XEN) [0000000d2f7034a9] vesafb: framebuffer at 0x0000004000000000, mapped to 0xffff82c000201000, using 8128k, total 8128k
(XEN) [0000000d2f703ef9] vesafb: mode is 1920x1080x32, linelength=7680, font 8x16
(XEN) [0000000d2f70498f] vesafb: Truecolor: size=8:8:8:8, shift=24:16:8:0
(XEN) [0000000d38b4c44f] SMBIOS 3.5 present.
(XEN) [0000000d38b61bd3] XSM Framework v1.0.1 initialized
(XEN) [0000000d38b74e3b] Initialising XSM SILO mode
(XEN) [0000000d38ba8247] Using APIC driver default
(XEN) [0000000d38bc8155] ACPI: PM-Timer IO Port: 0x1808 (24 bits)
(XEN) [0000000d38bdcb2d] ACPI: v5 SLEEP INFO: control[0:0], status[0:0]
(XEN) [0000000d38bf441d] ACPI: SLEEP INFO: pm1x_cnt[1:1804,1:0], pm1x_evt[1:1800,1:0]
(XEN) [0000000d38c0bdc9] ACPI: 32/64X FACS address mismatch in FADT - 7534b000/0000000000000000, using 32
(XEN) [0000000d38c2948f] ACPI:             wakeup_vec[7534b00c], vec_size[20]
(XEN) [0000000d38d00d17] Overriding APIC driver with bigsmp
(XEN) [0000000d38d30d95] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) [0000000d38d537c7] IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-119
(XEN) [0000000d38d89d75] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) [0000000d38da1c77] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) [0000000d38dddf49] ACPI: HPET id: 0x8086a201 base: 0xfed00000
(XEN) [0000000d38e01c77] PCI: MCFG configuration 0: base c0000000 segment 0000 buses 00 - ff
(XEN) [0000000d38e1e8af] PCI: MCFG area at c0000000 reserved in E820
(XEN) [0000000d38e38347] PCI: Using MCFG for segment 0000 bus 00-ff
(XEN) [0000000d38e51897] ACPI: BGRT: invalidating v1 image at 0x6dc7b018
(XEN) [0000000d38e69889] Using ACPI (MADT) for SMP configuration information
(XEN) [0000000d38e7fee9] SMP: Allowing 12 CPUs (0 hotplug CPUs)
(XEN) [0000000d38ea9565] IRQ limits: 120 GSI, 2376 MSI/MSI-X
(XEN) [0000000d39a57ff5] Switched to APIC driver x2apic_mixed
(XEN) [0000000d39c23def] BSP microcode revision: 0x00000429
(XEN) [0000000d3a4be885] microcode: CPU0 updated from revision 0x429 to 0x435, date = 2024-06-03
(XEN) [0000000d3a4eb46f] CPU0: TSC: 38400000 Hz * 136 / 2 = 2611200000 Hz
(XEN) [0000000d3a5009b1] CPU0: bus: 100 MHz base: 2600 MHz max: 4700 MHz
(XEN) [0000000d3a52cc6b] CPU0: 400 ... 2600 MHz
(XEN) [0000000d3a549445] xstate: size: 0xa88 and states: 0x207
(XEN) [0000000d3a5946a1] CPU0: Intel machine check reporting enabled
(XEN) [0000000d3a5ac6e7] Unrecognised CPU model 0x9a - assuming vulnerable to LazyFPU
(XEN) [0000000d3a5c4ba3] Speculative mitigation facilities:
(XEN) [0000000d3a5d7d25]   Hardware hints: RDCL_NO EIBRS RRSBA SKIP_L1DFL MDS_NO TAA_NO SBDR_SSDP_NO FBSDP_NO PSDP_NO GDS_NO
(XEN) [0000000d3a5f6bed]   Hardware features: IBPB IBRS STIBP SSBD PSFD L1D_FLUSH MD_CLEAR RFDS_CLEAR
(XEN) [0000000d3a611315]   Compiled-in support: INDIRECT_THUNK HARDEN_ARRAY HARDEN_BRANCH HARDEN_GUEST_ACCESS HARDEN_LOCK
(XEN) [0000000d3a62ff33]   Xen settings: BTI-Thunk: JMP, SPEC_CTRL: IBRS+ STIBP+ SSBD- PSFD- BHI_DIS_S+, Other: IBPB-ctxt VERW BRANCH_HARDEN
(XEN) [0000000d3a652085]   Support for HVM VMs: MSR_SPEC_CTRL MSR_VIRT_SPEC_CTRL RSB EAGER_FPU VERW
(XEN) [0000000d3a66bad7]   Support for PV VMs: MSR_SPEC_CTRL EAGER_FPU VERW
(XEN) [0000000d3a681b49]   XPTI (64-bit PV only): Dom0 disabled, DomU disabled (with PCID)
(XEN) [0000000d3a69a063]   PV L1TF shadowing: Dom0 disabled, DomU disabled
(XEN) [0000000d3a6b07ff] Using scheduler: SMP Credit Scheduler (credit)
(XEN) [0000000d3a6eaaf7] Disabling HPET for being unreliable
(XEN) [0000000d423b5749] Platform timer is 3.580MHz ACPI PM Timer
(XEN) [    0.260587] Detected 2611.192 MHz processor.
(XEN) [    0.266407] EFI memory map:
(XEN) [    0.266440]  0000000000000-0000000003fff type=2 attr=000000000000000f
(XEN) [    0.266492]  0000000004000-000000008dfff type=7 attr=000000000000000f
(XEN) [    0.266544]  000000008e000-000000009dfff type=2 attr=000000000000000f
(XEN) [    0.266596]  000000009e000-000000009efff type=0 attr=000000000000000f
(XEN) [    0.266647]  000000009f000-000000009ffff type=4 attr=000000000000000f
(XEN) [    0.266699]  0000000100000-0000001b41fff type=2 attr=000000000000000f
(XEN) [    0.266751]  0000001b42000-0000047c1dfff type=7 attr=000000000000000f
(XEN) [    0.266803]  0000047c1e000-0000061c76fff type=1 attr=000000000000000f
(XEN) [    0.266855]  0000061c77000-0000061cf6fff type=4 attr=000000000000000f
(XEN) [    0.266909]  0000061cf7000-00000653fefff type=7 attr=000000000000000f
(XEN) [    0.266962]  00000653ff000-00000697fffff type=4 attr=000000000000000f
(XEN) [    0.267015]  0000069800000-00000698b1fff type=7 attr=000000000000000f
(XEN) [    0.267068]  00000698b2000-00000699c5fff type=1 attr=000000000000000f
(XEN) [    0.267121]  00000699c6000-0000069ac5fff type=4 attr=000000000000000f
(XEN) [    0.267174]  0000069ac6000-0000069ae0fff type=3 attr=000000000000000f
(XEN) [    0.267227]  0000069ae1000-0000069b0bfff type=4 attr=000000000000000f
(XEN) [    0.267280]  0000069b0c000-0000069b0dfff type=3 attr=000000000000000f
(XEN) [    0.267333]  0000069b0e000-0000069b12fff type=4 attr=000000000000000f
(XEN) [    0.267386]  0000069b13000-0000069b14fff type=3 attr=000000000000000f
(XEN) [    0.267439]  0000069b15000-0000069b19fff type=4 attr=000000000000000f
(XEN) [    0.267492]  0000069b1a000-0000069b1dfff type=3 attr=000000000000000f
(XEN) [    0.267545]  0000069b1e000-0000069b25fff type=4 attr=000000000000000f
(XEN) [    0.268954]  0000069b26000-0000069b27fff type=3 attr=000000000000000f
(XEN) [    0.270451]  0000069b28000-0000069b2dfff type=4 attr=000000000000000f
(XEN) [    0.271973]  0000069b2e000-0000069b2efff type=3 attr=000000000000000f
(XEN) [    0.273537]  0000069b2f000-0000069b45fff type=4 attr=000000000000000f
(XEN) [    0.275107]  0000069b46000-0000069b49fff type=3 attr=000000000000000f
(XEN) [    0.276685]  0000069b4a000-0000069b51fff type=4 attr=000000000000000f
(XEN) [    0.278261]  0000069b52000-0000069b59fff type=3 attr=000000000000000f
(XEN) [    0.279828]  0000069b5a000-0000069b71fff type=4 attr=000000000000000f
(XEN) [    0.281380]  0000069b72000-0000069b77fff type=3 attr=000000000000000f
(XEN) [    0.282944]  0000069b78000-0000069b87fff type=4 attr=000000000000000f
(XEN) [    0.284503]  0000069b88000-0000069bd3fff type=3 attr=000000000000000f
(XEN) [    0.286057]  0000069bd4000-0000069c16fff type=4 attr=000000000000000f
(XEN) [    0.287608]  0000069c17000-0000069c19fff type=3 attr=000000000000000f
(XEN) [    0.289155]  0000069c1a000-0000069c8bfff type=4 attr=000000000000000f
(XEN) [    0.290703]  0000069c8c000-0000069c8cfff type=7 attr=000000000000000f
(XEN) [    0.292285]  0000069c8d000-0000069ceefff type=4 attr=000000000000000f
(XEN) [    0.293862]  0000069cef000-0000069cf5fff type=3 attr=000000000000000f
(XEN) [    0.295441]  0000069cf6000-0000069d00fff type=4 attr=000000000000000f
(XEN) [    0.297028]  0000069d01000-0000069d05fff type=3 attr=000000000000000f
(XEN) [    0.298569]  0000069d06000-0000069d0efff type=4 attr=000000000000000f
(XEN) [    0.300151]  0000069d0f000-0000069d4bfff type=3 attr=000000000000000f
(XEN) [    0.301746]  0000069d4c000-000006ac37fff type=4 attr=000000000000000f
(XEN) [    0.303310]  000006ac38000-000006ac4efff type=3 attr=000000000000000f
(XEN) [    0.304873]  000006ac4f000-000006ac5efff type=4 attr=000000000000000f
(XEN) [    0.306465]  000006ac5f000-000006ac66fff type=3 attr=000000000000000f
(XEN) [    0.308055]  000006ac67000-000006ac6efff type=7 attr=000000000000000f
(XEN) [    0.309651]  000006ac6f000-000006ac6ffff type=2 attr=000000000000000f
(XEN) [    0.311224]  000006ac70000-000006cbfffff type=7 attr=000000000000000f
(XEN) [    0.312788]  000006cc00000-000006d5fffff type=2 attr=000000000000000f
(XEN) [    0.314396]  000006d600000-000006d7f9fff type=7 attr=000000000000000f
(XEN) [    0.316003]  000006d7fa000-000006dddafff type=4 attr=000000000000000f
(XEN) [    0.317571]  000006dddb000-000006ddeefff type=7 attr=000000000000000f
(XEN) [    0.319153]  000006ddef000-000006ddeffff type=4 attr=000000000000000f
(XEN) [    0.320736]  000006ddf0000-000006ddf1fff type=7 attr=000000000000000f
(XEN) [    0.322303]  000006ddf2000-0000071375fff type=4 attr=000000000000000f
(XEN) [    0.323878]  0000071376000-0000071608fff type=7 attr=000000000000000f
(XEN) [    0.325422]  0000071609000-0000072054fff type=3 attr=000000000000000f
(XEN) [    0.326958]  0000072055000-0000075154fff type=0 attr=000000000000000f
(XEN) [    0.328450]  0000075155000-000007526efff type=9 attr=000000000000000f
(XEN) [    0.329939]  000007526f000-000007534bfff type=10 attr=000000000000000f
(XEN) [    0.331469] Setting RUNTIME attribute for 000007526f000-000007534bfff
(XEN) [    0.333016]  000007534c000-0000075f65fff type=6 attr=800000000000000f
(XEN) [    0.334615]  0000075f66000-0000075ffefff type=5 attr=800000000000000f
(XEN) [    0.336202]  0000075fff000-0000075ffffff type=4 attr=000000000000000f
(XEN) [    0.337757]  0000100000000-000187fbfffff type=7 attr=000000000000000f
(XEN) [    0.339322]  00000000a0000-00000000fffff type=0 attr=0000000000000000
(XEN) [    0.340921]  0000076000000-0000079ffffff type=0 attr=000000000000000f
(XEN) [    0.342519]  000007aa00000-000007abfffff type=0 attr=000000000000000f
(XEN) [    0.344116]  000007b000000-000007bffffff type=0 attr=0000000000000009
(XEN) [    0.345698]  000007c000000-00000803fffff type=0 attr=0000000000000000
(XEN) [    0.347262]  00000c0000000-00000cfffffff type=11 attr=8000000000000001
(XEN) [    0.351195]  00000fe000000-00000fe010fff type=11 attr=8000000000000001
(XEN) [    0.352796]  00000fec00000-00000fec00fff type=11 attr=8000000000000001
(XEN) [    0.354389]  00000fed00000-00000fed00fff type=11 attr=8000000000000001
(XEN) [    0.355946]  00000fed20000-00000fed7ffff type=0 attr=0000000000000000
(XEN) [    0.357545]  00000fee00000-00000fee00fff type=11 attr=8000000000000001
(XEN) [    0.359147]  00000ff000000-00000ffffffff type=11 attr=800000000000100d
(XEN) [    0.360714] alt table ffff82d0406a36b8 -> ffff82d0406b2cc2
(XEN) [    0.388640] Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB
(XEN) [    0.389507] Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB
(XEN) [    0.390376] Intel VT-d Snoop Control not enabled.
(XEN) [    0.391743] Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) [    0.392840] Intel VT-d Queued Invalidation enabled.
(XEN) [    0.393798] Intel VT-d Interrupt Remapping enabled.
(XEN) [    0.394803] Intel VT-d Posted Interrupt not enabled.
(XEN) [    0.396016] Intel VT-d Shared EPT tables enabled.
(XEN) [    0.398206] I/O virtualisation enabled
(XEN) [    0.399322]  - Dom0 mode: Relaxed
(XEN) [    0.400659] Interrupt remapping enabled
(XEN) [    0.401705] Enabled directed EOI with ioapic_ack_old on!
(XEN) [    0.402750] Enabling APIC mode.  Using 1 I/O APICs
(XEN) [    0.414140] ENABLING IO-APIC IRQs
(XEN) [    0.414906]  -> Using old ACK method
(XEN) [    0.416107] ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) [    0.941444] Allocated console ring of 64 KiB.
(XEN) [    0.942223] VMX: Supported advanced features:
(XEN) [    0.942993]  - APIC MMIO access virtualisation
(XEN) [    0.943792]  - APIC TPR shadow
(XEN) [    0.944626]  - Extended Page Tables (EPT)
(XEN) [    0.945496]  - Virtual-Processor Identifiers (VPID)
(XEN) [    0.946408]  - Virtual NMI
(XEN) [    0.947357]  - MSR direct-access bitmap
(XEN) [    0.948331]  - Unrestricted Guest
(XEN) [    0.949483]  - VMCS shadowing
(XEN) [    0.950675]  - VM Functions
(XEN) [    0.951664]  - Virtualisation Exceptions
(XEN) [    0.952642]  - TSC Scaling
(XEN) [    0.953584] HVM: ASIDs enabled.
(XEN) [    0.954508] HVM: VMX enabled
(XEN) [    0.955450] HVM: Hardware Assisted Paging (HAP) detected
(XEN) [    0.956371] HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) [    0.957322] alt table ffff82d0406a36b8 -> ffff82d0406b2cc2
(XEN) [    0.958293] altcall: Optimised away 327 endbr64 instructions
(XEN) [0000000dafceac57] microcode: CPU2 updated from revision 0x429 to 0x435, date = 2024-06-03
(XEN) [0000000db05453ca] microcode: CPU4 updated from revision 0x429 to 0x435, date = 2024-06-03
(XEN) [0000000db0fd4daa] microcode: CPU8 updated from revision 0x429 to 0x435, date = 2024-06-03
(XEN) [    0.974337] Brought up 12 CPUs
(XEN) [    0.975090] Testing NMI watchdog on all CPUs:ok
(XEN) [    1.020096] Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) [    1.021153] mcheck_poll: Machine check polling timer started.
(XEN) [    1.042083] NX (Execute Disable) protection active
(XEN) [    1.042888] d0 has maximum 952 PIRQs
(XEN) [    1.043707] csched_alloc_domdata: setting dom 0 as the privileged domain
(XEN) [    1.044556] *** Building a PV Dom0 ***
(XEN) [    1.262842]  Xen  kernel: 64-bit, lsb
(XEN) [    1.263584]  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x302c000
(XEN) [    1.264543] PHYSICAL MEMORY ARRANGEMENT:
(XEN) [    1.265301]  Dom0 alloc.:   000000183c000000->0000001840000000 (2075731 pages to be allocated)
(XEN) [    1.266112]  Init. ramdisk: 000000187e853000->000000187fbfff34
(XEN) [    1.266936] VIRTUAL MEMORY ARRANGEMENT:
(XEN) [    1.267767]  Loaded kernel: ffffffff81000000->ffffffff8302c000
(XEN) [    1.268619]  Phys-Mach map: 0000008000000000->0000008001000000
(XEN) [    1.269513]  Start info:    ffffffff8302c000->ffffffff8302c4b8
(XEN) [    1.270423]  Page tables:   ffffffff8302d000->ffffffff8304a000
(XEN) [    1.271362]  Boot stack:    ffffffff8304a000->ffffffff8304b000
(XEN) [    1.272310]  TOTAL:         ffffffff80000000->ffffffff83400000
(XEN) [    1.273486]  ENTRY ADDRESS: ffffffff8242b180
(XEN) [    1.275170] Dom0 has maximum 12 VCPUs
(XEN) [    1.314728] Initial low memory virq threshold set at 0x4000 pages.
(XEN) [    1.315641] Scrubbing Free RAM in background
(XEN) [    1.316541] Std. Loglevel: Errors, warnings and info
(XEN) [    1.317465] Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) [    1.318790] Xen is relinquishing VGA console.
(XEN) [    1.326626] *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) [    1.326780] Freed 2048kB init memory

P.S: Hardware: Protectli VP6670 w/96GB RAM

dinhngtu

@manilx How much free memory do you have on the host at the time of snapshot? Does it happen on an empty host (excluding the VM being saved)?

manilx

@dinhngtu I have more than 50% of free mem on the hosts at all time.
ScreenShot 2025-01-06 at 15.21.03.png
I don't have an empty host to test. Would have to shift VM's between hosts to test this but this would break current snapshots for backup, so reluctant to do that.

manilx

@dinhngtu This is the VM I was trying to snapshot:
ScreenShot 2025-01-06 at 15.23.50.png

dinhngtu

@manilx Please follow the steps below on the host:

Run xl debug-keys + until xl dmesg says "standard log level: All"
Trigger the error again
Post the output of xl dmesg
Afterwards, you can run xl debug-keys - (that's a minus sign at the end) until the standard log level falls back down to "Errors and warnings"