Backup keep failing
-
Hi,
I added a S3 remote and created a backup job containing Delta + Continuous Replication.
After a while, this is what I get:
When I test the S3 remote I have a successful connection
Could anyone please tell me if there is a way to tweak the timeout or find out why the backup is not sending to S3? When I look at my S3 bucket, it is emptyHow do I resolved the
VDI_IO_ERROR(Device I/O errors)How do I clear coalesce file that stop the other snapshot backup to run?
Thank you
-
Ping @nraynaud
-
@olivierlambert @nraynaud are you able to assist please?
-
@fred974 Hello Fred, I'm working on removing the timeout error, in the mean time you can remove it by increasing node memory by at least 2Gb.
You can not stop the coalesce from happening, the only thing that can be done is planning the backup at a different time.
I don't know what to do of the VDI_IO_ERROR.
Nicolas.
-
@nraynaud said in Backup keep failing:
coalesce
Hi,
I have now removed all the backup jobs from XO and a few hours later, I can still see the same coalesce screen above.
My understanding is that they meant to clear after a while
-
@fred974 Could anyone please assist me in resolving this Job canceled to protect the VDI chain error ?
{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1608677679049", "jobId": "c2d527c4-6032-4924-9ce5-1d37a564784f", "jobName": "daily_delta", "message": "backup", "scheduleId": "0b87e9de-6052-4e61-88b6-71345faf35c0", "start": 1608677679049, "status": "skipped", "tasks": [ { "data": { "type": "VM", "id": "17a93294-e15c-485b-a3af-b7b3452e97c9" }, "id": "1608677679081", "message": "Starting backup of Axigen-Mail. (c2d527c4-6032-4924-9ce5-1d37a564784f)", "start": 1608677679081, "status": "skipped", "end": 1608677679196, "result": { "VDI": { "uuid": "28e6a067-db57-4246-bdc5-a5330ef37947", "name_label": "srv-tri-axigen", "name_description": "Axigen-Prod", "allowed_operations": [ "snapshot", "clone" ], "current_operations": {}, "SR": "OpaqueRef:24decb9d-aa2e-4615-b540-9279a4f2db26", "VBDs": [ "OpaqueRef:5adb53e0-7e8e-41ef-b29e-3949c7150c96" ], "crash_dumps": [], "virtual_size": 128849018880, "physical_utilisation": 129109065728, "type": "user", "sharable": false, "read_only": false, "other_config": {}, "storage_lock": false, "location": "28e6a067-db57-4246-bdc5-a5330ef37947", "managed": true, "missing": false, "parent": "OpaqueRef:NULL", "xenstore_data": {}, "sm_config": { "vdi_type": "vhd", "host_OpaqueRef:feda7d4a-e638-4f8d-b447-af1260c46964": "RW", "read-caching-enabled-on-badb72a0-10a2-4c0a-b18c-832db6900e2c": "false", "vhd-parent": "c65f53c3-2856-48ad-8636-fea72b9725b2", "read-caching-reason-badb72a0-10a2-4c0a-b18c-832db6900e2c": "SR_NOT_SUPPORTED" }, "is_a_snapshot": false, "snapshot_of": "OpaqueRef:NULL", "snapshots": [ "OpaqueRef:42c52287-3f4f-40b0-9d3a-65b1390c3453" ], "snapshot_time": "19700101T00:00:00Z", "tags": [], "allow_caching": false, "on_boot": "persist", "metadata_of_pool": "", "metadata_latest": false, "is_tools_iso": false, "cbt_enabled": false }, "VM": { "uuid": "17a93294-e15c-485b-a3af-b7b3452e97c9", "allowed_operations": [ "changing_dynamic_range", "migrate_send", "pool_migrate", "changing_VCPUs_live", "checkpoint", "snapshot" ], "current_operations": {}, "name_label": "Axigen-Mail", "name_description": "intIP 172.16.xxx.22 - extIP 193.12.xx.xx - 4vCPU, 8GB RAM, 120GB Disk", "power_state": "Running", "user_version": 1, "is_a_template": false, "is_default_template": false, "suspend_VDI": "OpaqueRef:NULL", "resident_on": "OpaqueRef:feda7d4a-e638-4f8d-b447-af1260c46964", "affinity": "OpaqueRef:NULL", "memory_overhead": 73400320, "memory_target": 4294967296, "memory_static_max": 8589934592, "memory_dynamic_max": 8589934592, "memory_dynamic_min": 4294967296, "memory_static_min": 2147483648, "VCPUs_params": {}, "VCPUs_max": 4, "VCPUs_at_startup": 4, "actions_after_shutdown": "destroy", "actions_after_reboot": "restart", "actions_after_crash": "restart", "consoles": [ "OpaqueRef:c1bb56ac-0b51-4f5b-9647-74e09b895cbd" ], "VIFs": [ "OpaqueRef:f35121db-3295-49d1-9806-3751a083a227" ], "VBDs": [ "OpaqueRef:b0171b1e-11cd-4cb2-b749-cdafbcc40658", "OpaqueRef:5adb53e0-7e8e-41ef-b29e-3949c7150c96" ], "VUSBs": [], "crash_dumps": [], "VTPMs": [], "PV_bootloader": "", "PV_kernel": "", "PV_ramdisk": "", "PV_args": "", "PV_bootloader_args": "", "PV_legacy_args": "", "HVM_boot_policy": "BIOS order", "HVM_boot_params": { "firmware": "bios", "order": "cdn" }, "HVM_shadow_multiplier": 1, "platform": { "timeoffset": "0", "videoram": "8", "hpet": "true", "secureboot": "false", "device-model": "qemu-upstream-compat", "apic": "true", "device_id": "0001", "vga": "std", "nx": "true", "pae": "true", "viridian": "false", "acpi": "1" }, "PCI_bus": "", "other_config": { "auto_poweron": "true", "base_template_name": "CentOS 8", "import_task": "OpaqueRef:52440cd1-8e98-4a5d-9eef-6fc36dbdaff1", "mac_seed": "06639988-0197-4394-b996-84cf1df88515", "install-methods": "cdrom,nfs,http,ftp", "linux_template": "true" }, "domid": 2, "domarch": "", "last_boot_CPU_flags": { "vendor": "GenuineIntel", "features": "1fcbfbff-97ba2223-2d93fbff-00000403-00000001-00000000-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000" }, "is_control_domain": false, "metrics": "OpaqueRef:71e4bc16-feda-4c9d-9b40-e913217d2f4e", "guest_metrics": "OpaqueRef:c8b03d4e-d91c-4dce-a72d-7183072c1f02", "last_booted_record": "", "recommendations": "<restrictions><restriction field=\"memory-static-max\" max=\"1649267441664\"/><restriction field=\"vcpus-max\" max=\"32\"/><restriction field=\"has-vendor-device\" value=\"false\"/><restriction field=\"allow-gpu-passthrough\" value=\"1\"/><restriction field=\"allow-vgpu\" value=\"1\"/><restriction field=\"allow-network-sriov\" value=\"1\"/><restriction field=\"supports-bios\" value=\"yes\"/><restriction field=\"supports-uefi\" value=\"no\"/><restriction field=\"supports-secure-boot\" value=\"no\"/><restriction max=\"255\" property=\"number-of-vbds\"/><restriction max=\"7\" property=\"number-of-vifs\"/></restrictions>", "xenstore_data": { "vm-data": "", "vm-data/mmio-hole-size": "268435456" }, "ha_always_run": false, "ha_restart_priority": "", "is_a_snapshot": false, "snapshot_of": "OpaqueRef:NULL", "snapshots": [ "OpaqueRef:c95fdceb-3295-452e-9012-ebb315165321" ], "snapshot_time": "19700101T00:00:00Z", "transportable_snapshot_id": "", "blobs": {}, "tags": [ "CentOS", "Production", "SSD" ], "blocked_operations": { "hard_shutdown": "true", "suspend": "true", "pause": "true", "clean_reboot": "true", "shutdown": "true", "hard_reboot": "true", "clean_shutdown": "true", "destroy": "true" }, "snapshot_info": {}, "snapshot_metadata": "", "parent": "OpaqueRef:c95fdceb-3295-452e-9012-ebb315165321", "children": [], "bios_strings": { "bios-vendor": "Xen", "bios-version": "", "system-manufacturer": "Xen", "system-product-name": "HVM domU", "system-version": "", "system-serial-number": "", "baseboard-manufacturer": "", "baseboard-product-name": "", "baseboard-version": "", "baseboard-serial-number": "", "baseboard-asset-tag": "", "baseboard-location-in-chassis": "", "enclosure-asset-tag": "", "hp-rombios": "", "oem-1": "Xen", "oem-2": "MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d" }, "protection_policy": "OpaqueRef:NULL", "is_snapshot_from_vmpp": false, "snapshot_schedule": "OpaqueRef:NULL", "is_vmss_snapshot": false, "appliance": "OpaqueRef:NULL", "start_delay": 0, "shutdown_delay": 0, "order": 0, "VGPUs": [], "attached_PCIs": [], "suspend_SR": "OpaqueRef:NULL", "version": 0, "generation_id": "", "hardware_platform_version": 0, "has_vendor_device": false, "requires_reboot": false, "reference_label": "centos-8", "domain_type": "hvm", "NVRAM": {} }, "message": "unhealthy VDI chain", "name": "Error", "stack": "Error: unhealthy VDI chain\n at Xapi._assertHealthyVdiChain (/opt/xo/xo-builds/xen-orchestra-202022122000/packages/xo-server/src/xapi/index.js:735:15)\n at Xapi._assertHealthyVdiChain (/opt/xo/xo-builds/xen-orchestra-202022122000/packages/xo-server/src/xapi/index.js:739:10)\n at Xapi._assertHealthyVdiChain (/opt/xo/xo-builds/xen-orchestra-202022122000/packages/xo-server/src/xapi/index.js:739:10)\n at Xapi._assertHealthyVdiChain (/opt/xo/xo-builds/xen-orchestra-202022122000/packages/xo-server/src/xapi/index.js:739:10)\n at /opt/xo/xo-builds/xen-orchestra-202022122000/packages/xo-server/src/xapi/index.js:746:14\n at arrayEach (/opt/xo/xo-builds/xen-orchestra-202022122000/node_modules/lodash/_arrayEach.js:15:9)\n at forEach (/opt/xo/xo-builds/xen-orchestra-202022122000/node_modules/lodash/forEach.js:38:10)\n at Xapi._assertHealthyVdiChains (/opt/xo/xo-builds/xen-orchestra-202022122000/packages/xo-server/src/xapi/index.js:744:5)\n at BackupNg.apply (/opt/xo/xo-builds/xen-orchestra-202022122000/packages/xo-server/src/xo-mixins/backups-ng/index.js:1119:14)\n at BackupNg._backupVm (/opt/xo/xo-builds/xen-orchestra-202022122000/node_modules/golike-defer/src/index.js:57:17)\n at iteratee (/opt/xo/xo-builds/xen-orchestra-202022122000/packages/xo-server/src/xo-mixins/backups-ng/index.js:693:28)\n at /opt/xo/xo-builds/xen-orchestra-202022122000/@xen-orchestra/async-map/src/index.js:32:17\n at Promise._execute (/opt/xo/xo-builds/xen-orchestra-202022122000/node_modules/bluebird/js/release/debuggability.js:384:9)\n at Promise._resolveFromExecutor (/opt/xo/xo-builds/xen-orchestra-202022122000/node_modules/bluebird/js/release/promise.js:518:18)\n at new Promise (/opt/xo/xo-builds/xen-orchestra-202022122000/node_modules/bluebird/js/release/promise.js:103:10)\n at /opt/xo/xo-builds/xen-orchestra-202022122000/@xen-orchestra/async-map/src/index.js:31:7\n at /opt/xo/xo-builds/xen-orchestra-202022122000/node_modules/lodash/_baseMap.js:17:23\n at /opt/xo/xo-builds/xen-orchestra-202022122000/node_modules/lodash/_createBaseFor.js:17:11\n at baseForOwn (/opt/xo/xo-builds/xen-orchestra-202022122000/node_modules/lodash/_baseForOwn.js:13:20)\n at /opt/xo/xo-builds/xen-orchestra-202022122000/node_modules/lodash/_createBaseEach.js:17:14\n at baseMap (/opt/xo/xo-builds/xen-orchestra-202022122000/node_modules/lodash/_baseMap.js:16:3)\n at map (/opt/xo/xo-builds/xen-orchestra-202022122000/node_modules/lodash/map.js:50:10)" } } ], "end": 1608677679197 } -
Protecting the VDI chain is not an error, but… a protection.
-
@olivierlambert said in Backup keep failing:
Protecting the VDI chain is not an error, but… a protection.
Hi,
Its a protection from what? How do I resolve this issue?
I am unable to perform any backup at all -
Have you looked at the link given when it's happening? This link will send you to our documentation.
Everything is explained here: https://xen-orchestra.com/docs/backup_troubleshooting.html#vdi-chain-protection
A good reflex is to always have the documentation near you, there's a search bar also. Really, it's the first step you might do always before.
-
@olivierlambert I have read it but struggled with some of the English (my problem). I'll try to read it again.
My SR was almost full so I guess it couldn't run it to protect to get any bigger. I am moving VM away to free some space and see if this fix it -
Not enough space means not enough space to coalesce, and therefore XO will prevent to continue and fill entirely your storage because of uncoalesced chains.
