Backup Error: 302 Found

[stucamp]

Problem Description:
So I'm fairly new to XCP-ng. I've been using it and the self compiled XO on a fairly basic setup (vms on local drive, no HA or anything fancy, etc) for about 6 months without any issue on some older servers in my homelab. I've been backing up to a truenas scale nfs share without issue the entire time. Recently I picked up some new lower power/noise hosts to use.

I installed XCP-ng on the 3 new hosts, added each to a new pool, and migrated the vm's off the old pool/hosts to the new ones. Everything seemed to be working fine, can migrate the vms between the new hosts etc. As such I decommissioned the old hosts and went on my merry way for about a week. Then I noticed that the weekly backup tasks show as failed. To be clear just the backups are the issue; snapshots, metadata, and configs are all working fine

After combing through the results, it seems that backups of any VM that isn't running on the master has an issue. I can backup any VM that's on the new master, but whenever I attempt a backup of a VM on either 2 of the other hosts, the backup fails with Error: 302 Found and I can't seem to figure out why/what's going on.

Efforts to solve so far:

• I migrated the XO vm to a different host to see if that changed anything, but still no success unless the backup target vm is on the pool master.
• I've run xe pool-emergency-reset-master master-address=x.x.x.x both with domain name and IP on the slave hosts to ensure it wasn't some sort of name resolution error
• Updated the remote back up storage address using IP and domain name to ensure name resolution there isn't an issue.
• Reset the toolstack, and even spun up a fresh XO vm all with the same results.
• Even spun up a new remote storage vdev to rule that out

I'm sure I've overlooked something fairly simple, but my google-fu is failing me here as I'm coming up with nearly zero results for Error: 302 Found pertaining to xcp-ng. Attaching log results from a failed attempt. Anything thoughts/help would be appreciated!

{
  "data": {
    "mode": "full",
    "reportWhen": "failure"
  },
  "id": "1676998844595",
  "jobId": "dd59ee54-881b-4cc8-bdf8-93e3098adf40",
  "jobName": "xo",
  "message": "backup",
  "scheduleId": "38719082-093c-4c9c-91b7-0a695d56662d",
  "start": 1676998844595,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "1303b645-be48-5517-ee41-c4a8623a8f49",
          "29a934ef-2c3b-2068-f93d-71828627704d"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "1303b645-be48-5517-ee41-c4a8623a8f49"
      },
      "id": "1676998845220",
      "message": "backup VM",
      "start": 1676998845220,
      "status": "success",
      "tasks": [
        {
          "id": "1676998845606",
          "message": "snapshot",
          "start": 1676998845606,
          "status": "success",
          "end": 1676998847679,
          "result": "cb42346d-51c1-d87b-35bd-23ce48216bd2"
        },
        {
          "data": {
            "id": "efc51928-59c0-41c2-b108-2552d65ab253",
            "type": "remote",
            "isFull": true
          },
          "id": "1676998847697",
          "message": "export",
          "start": 1676998847697,
          "status": "success",
          "tasks": [
            {
              "id": "1676998847702",
              "message": "transfer",
              "start": 1676998847702,
              "status": "success",
              "end": 1676998934832,
              "result": {
                "size": 6791107584
              }
            }
          ],
          "end": 1676998935302
        },
        {
          "id": "1676998935985",
          "message": "clean-vm",
          "start": 1676998935985,
          "status": "success",
          "end": 1676998936308,
          "result": {
            "merge": false
          }
        }
      ],
      "end": 1676998936392
    },
    {
      "data": {
        "type": "VM",
        "id": "29a934ef-2c3b-2068-f93d-71828627704d"
      },
      "id": "1676998845252",
      "message": "backup VM",
      "start": 1676998845252,
      "status": "failure",
      "tasks": [
        {
          "id": "1676998845613",
          "message": "snapshot",
          "start": 1676998845613,
          "status": "success",
          "end": 1676998848296,
          "result": "5fdb480d-c6fe-948b-b075-8f1b84b336da"
        },
        {
          "id": "1676998849110",
          "message": "clean-vm",
          "start": 1676998849110,
          "status": "success",
          "end": 1676998849118,
          "result": {
            "merge": false
          }
        }
      ],
      "end": 1676998849123,
      "result": {
        "originalUrl": "https://10.0.13.41/export/?ref=OpaqueRef%3A3958c079-c26c-4145-9639-0a70c846201e&use_compression=false&session_id=OpaqueRef%3A9a2d2e15-fa60-4e16-81d8-480bd6abe436&task_id=OpaqueRef%3A8fdc0e8c-d0c5-479a-b7d1-f6bb7e2792fb",
        "url": "https://10.0.13.41/export/?ref=OpaqueRef%3A3958c079-c26c-4145-9639-0a70c846201e&use_compression=false&session_id=OpaqueRef%3A9a2d2e15-fa60-4e16-81d8-480bd6abe436&task_id=OpaqueRef%3A8fdc0e8c-d0c5-479a-b7d1-f6bb7e2792fb",
        "pool_master": {
          "uuid": "c1ef5a9b-df57-4b3e-9a86-6914c9c22566",
          "name_label": "tc1",
          "name_description": "Default install",
          "memory_overhead": 736063488,
          "allowed_operations": [
            "vm_migrate",
            "provision",
            "vm_resume",
            "evacuate",
            "vm_start"
          ],
          "current_operations": {},
          "API_version_major": 2,
          "API_version_minor": 16,
          "API_version_vendor": "XenSource",
          "API_version_vendor_implementation": {},
          "enabled": true,
          "software_version": {
            "product_version": "8.2.1",
            "product_version_text": "8.2",
            "product_version_text_short": "8.2",
            "platform_name": "XCP",
            "platform_version": "3.2.1",
            "product_brand": "XCP-ng",
            "build_number": "release/yangtze/master/58",
            "hostname": "localhost",
            "date": "2022-10-12",
            "dbv": "0.0.1",
            "xapi": "1.20",
            "xen": "4.13.4-9.29.2",
            "linux": "4.19.0+1",
            "xencenter_min": "2.16",
            "xencenter_max": "2.16",
            "network_backend": "openvswitch",
            "db_schema": "5.602"
          },
          "other_config": {
            "agent_start_time": "1676965320.",
            "boot_time": "1676965267.",
            "rpm_patch_installation_time": "1676913670.263",
            "iscsi_iqn": "iqn.2023-02.com.someotherdomainname:e8e0d172"
          },
          "capabilities": [
            "xen-3.0-x86_64",
            "hvm-3.0-x86_32",
            "hvm-3.0-x86_32p",
            "hvm-3.0-x86_64",
            ""
          ],
          "cpu_configuration": {},
          "sched_policy": "credit",
          "supported_bootloaders": [
            "pygrub",
            "eliloader"
          ],
          "resident_VMs": [
            "OpaqueRef:04f8b7ba-b40b-4408-bcc3-01d80a523a03",
            "OpaqueRef:12163a49-9217-4d9c-9271-61708f2e7d68",
            "OpaqueRef:c88a9ff3-9bc4-4db2-8fb7-67cff79c3c12",
            "OpaqueRef:e03eeee3-1337-490c-8866-0c5f58c594c3",
            "OpaqueRef:23558b40-b332-496a-b5e1-84c0fb4a77bd"
          ],
          "logging": {},
          "PIFs": [
            "OpaqueRef:ffabdef9-ee08-494d-9d6c-43a04640389f",
            "OpaqueRef:22cd56cb-2fb1-4476-a946-225ba3232e7d",
            "OpaqueRef:1ef1b3f5-dca2-4422-9d23-d5c9e3a43fac",
            "OpaqueRef:018307b7-611a-491b-af7a-850715f9fd50"
          ],
          "suspend_image_sr": "OpaqueRef:67a25085-72d2-4823-8b50-864fb3f5fadc",
          "crash_dump_sr": "OpaqueRef:67a25085-72d2-4823-8b50-864fb3f5fadc",
          "crashdumps": [],
          "patches": [],
          "updates": [],
          "PBDs": [
            "OpaqueRef:e10a5251-2f76-448d-8237-2cb9a8fb78ca",
            "OpaqueRef:799acc20-fb22-4cdf-af05-46bcfcda70eb",
            "OpaqueRef:5e3a7920-72ea-4467-b405-a71486cb98c4",
            "OpaqueRef:11fb41cd-2a6a-4c63-a139-a2d30a63ac94"
          ],
          "host_CPUs": [
            "OpaqueRef:1f736157-f0d3-43ce-bf46-ca33d68b6b7e",
            "OpaqueRef:e75a599f-258a-45bd-bee1-234d5d769a07",
            "OpaqueRef:cac0414a-d608-45e8-b238-c3785cefac67",
            "OpaqueRef:26aeb086-3596-4e6b-b9fb-8ed86f56887c",
            "OpaqueRef:aeac32e8-4ee8-400a-b870-fce6fa68dc6e",
            "OpaqueRef:b32eec21-ec1e-4458-a9b3-9dc219e64c1d",
            "OpaqueRef:7a4562bd-53cd-445d-925e-5ba90f913491",
            "OpaqueRef:7392601e-b978-4011-8c02-ae156d0f9109",
            "OpaqueRef:ed7f3483-0a49-4b42-8942-0da90b8fa32d",
            "OpaqueRef:01cd294d-7ba3-4b61-9cdb-7d6ebf6b6de7",
            "OpaqueRef:0a55a809-7cd6-485d-a2b8-bf5b2e6f3742",
            "OpaqueRef:f0b6c78f-a050-493c-9821-001e333b27fd"
          ],
          "cpu_info": {
            "cpu_count": "12",
            "socket_count": "1",
            "vendor": "GenuineIntel",
            "speed": "2400.064",
            "modelname": "Intel(R) Core(TM) i7-8700T CPU @ 2.40GHz",
            "family": "6",
            "model": "158",
            "stepping": "10",
            "flags": "fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid pni pclmulqdq monitor est ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp fsgsbase bmi1 avx2 bmi2 erms rdseed adx clflushopt xsaveopt xsavec xgetbv1 arch_capabilities",
            "features_pv": "1fc9cbf5-f6f83203-2991cbf5-00000123-00000007-008c0b39-00000000-00000000-00001000-8c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000",
            "features_hvm": "1fcbfbff-f7fa3223-2d93fbff-00000523-0000000f-009c4fbb-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000",
            "features_hvm_host": "1fcbfbff-f7fa3223-2c100800-00000121-0000000f-009c47ab-00000000-00000000-00001000-9c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000",
            "features_pv_host": "1fc9cbf5-f6f83203-28100800-00000121-00000007-008c0329-00000000-00000000-00001000-8c000400-00000000-00000000-00000000-00000000-00000000-00000000-00000000-00000000"
          },
          "hostname": "tc1",
          "address": "10.0.13.41",
          "metrics": "OpaqueRef:43c049a4-bc2c-4966-b5bd-c689aedc2a4f",
          "license_params": {
            "restrict_vswitch_controller": "false",
            "restrict_lab": "false",
            "restrict_stage": "false",
            "restrict_storagelink": "false",
            "restrict_storagelink_site_recovery": "false",
            "restrict_web_selfservice": "false",
            "restrict_web_selfservice_manager": "false",
            "restrict_hotfix_apply": "false",
            "restrict_export_resource_data": "false",
            "restrict_read_caching": "false",
            "restrict_cifs": "false",
            "restrict_health_check": "false",
            "restrict_xcm": "false",
            "restrict_vm_memory_introspection": "false",
            "restrict_batch_hotfix_apply": "false",
            "restrict_management_on_vlan": "false",
            "restrict_ws_proxy": "false",
            "restrict_vlan": "false",
            "restrict_qos": "false",
            "restrict_pool_attached_storage": "false",
            "restrict_netapp": "false",
            "restrict_equalogic": "false",
            "restrict_pooling": "false",
            "enable_xha": "true",
            "restrict_marathon": "false",
            "restrict_email_alerting": "false",
            "restrict_historical_performance": "false",
            "restrict_wlb": "false",
            "restrict_rbac": "false",
            "restrict_dmc": "false",
            "restrict_checkpoint": "false",
            "restrict_cpu_masking": "false",
            "restrict_connection": "false",
            "platform_filter": "false",
            "regular_nag_dialog": "false",
            "restrict_vmpr": "false",
            "restrict_vmss": "false",
            "restrict_intellicache": "false",
            "restrict_gpu": "false",
            "restrict_dr": "false",
            "restrict_vif_locking": "false",
            "restrict_storage_xen_motion": "false",
            "restrict_vgpu": "false",
            "restrict_integrated_gpu_passthrough": "false",
            "restrict_vss": "false",
            "restrict_guest_agent_auto_update": "false",
            "restrict_pci_device_for_auto_update": "false",
            "restrict_xen_motion": "false",
            "restrict_guest_ip_setting": "false",
            "restrict_ad": "false",
            "restrict_nested_virt": "false",
            "restrict_live_patching": "false",
            "restrict_set_vcpus_number_live": "false",
            "restrict_pvs_proxy": "false",
            "restrict_igmp_snooping": "false",
            "restrict_rpu": "false",
            "restrict_pool_size": "false",
            "restrict_cbt": "false",
            "restrict_usb_passthrough": "false",
            "restrict_network_sriov": "false",
            "restrict_corosync": "true",
            "restrict_zstd_export": "false",
            "restrict_pool_secret_rotation": "false"
          },
          "ha_statefiles": [],
          "ha_network_peers": [],
          "blobs": {},
          "tags": [],
          "external_auth_type": "",
          "external_auth_service_name": "",
          "external_auth_configuration": {},
          "edition": "xcp-ng",
          "license_server": {
            "address": "localhost",
            "port": "27000"
          },
          "bios_strings": {
            "bios-vendor": "LENOVO",
            "bios-version": "M1UKT47A",
            "system-manufacturer": "LENOVO",
            "system-product-name": "10T7008CUS",
            "system-version": "ThinkCentre M720q",
            "system-serial-number": "MJ0A37YF",
            "baseboard-manufacturer": "LENOVO",
            "baseboard-product-name": "312D",
            "baseboard-version": "SDK0J40697 WIN 3305227205396",
            "baseboard-serial-number": "",
            "oem-1": "Xen",
            "oem-2": "MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d",
            "oem-3": "LENOVO ThinkCentre Embedded Controller -[M1UCT18A-1.18]-",
            "oem-4": "LENOVO ThinkCentre BIOS Boot Block Revision 1.47",
            "oem-5": "INVALID",
            "oem-6": "INVALID",
            "oem-7": "INVALID",
            "oem-8": "INVALID",
            "oem-9": "INVALID",
            "oem-10": "INVALID",
            "oem-11": "INVALID",
            "oem-12": "INVALID",
            "oem-13": "INVALID",
            "oem-14": "INVALID",
            "oem-15": "INVALID",
            "oem-16": "INVALID",
            "oem-17": "INVALID",
            "hp-rombios": ""
          },
          "power_on_mode": "",
          "power_on_config": {},
          "local_cache_sr": "OpaqueRef:NULL",
          "chipset_info": {
            "iommu": "true"
          },
          "PCIs": [
            "OpaqueRef:ed6fc133-3c71-4c27-ac44-43f796e80d96",
            "OpaqueRef:d79e6cd0-4db6-49a4-a557-e58b2cd547bc",
            "OpaqueRef:bbf831bf-e2a7-4e41-9180-54c647fb47f2",
            "OpaqueRef:a7de8fa2-5189-4d0f-b227-735194517b4a",
            "OpaqueRef:8cc71df2-ce4c-4197-9dec-6948ee330858",
            "OpaqueRef:849e649d-acbc-4210-85b4-a2bbfcdc6c3b",
            "OpaqueRef:47f8d61d-bd7e-4bf4-be00-7fe3b0885c9f",
            "OpaqueRef:3f7fd15f-f491-4ba8-94d8-bea6348e8518",
            "OpaqueRef:0b29d598-2c7d-43fc-835a-74c658b14cc9"
          ],
          "PGPUs": [
            "OpaqueRef:62a756b2-758e-48c8-91c6-f30f8aacfd7a"
          ],
          "PUSBs": [],
          "ssl_legacy": false,
          "guest_VCPUs_params": {},
          "display": "enabled",
          "virtual_hardware_platform_versions": [
            0,
            1,
            2
          ],
          "control_domain": "OpaqueRef:23558b40-b332-496a-b5e1-84c0fb4a77bd",
          "updates_requiring_reboot": [],
          "features": [],
          "iscsi_iqn": "iqn.2023-02.com.someotherdomainname:e8e0d172",
          "multipathing": false,
          "uefi_certificates": "",
          "certificates": [],
          "editions": [
            "xcp-ng"
          ]
        },
        "VM": "OpaqueRef:3958c079-c26c-4145-9639-0a70c846201e",
        "message": "302 Found",
        "name": "Error",
        "stack": "Error: 302 Found\n    at Object.assertSuccess (/opt/xo/xo-builds/xen-orchestra-202302210121/node_modules/http-request-plus/index.js:138:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async httpRequestPlus (/opt/xo/xo-builds/xen-orchestra-202302210121/node_modules/http-request-plus/index.js:205:22)\n    at async Xapi.getResource (/opt/xo/xo-builds/xen-orchestra-202302210121/packages/xen-api/dist/index.js:334:22)\n    at async Xapi.export (/opt/xo/xo-builds/xen-orchestra-202302210121/@xen-orchestra/xapi/vm.js:500:22)\n    at async VmBackup._copyFull (/opt/xo/xo-builds/xen-orchestra-202302210121/@xen-orchestra/backups/_VmBackup.js:288:20)\n    at async VmBackup.run (/opt/xo/xo-builds/xen-orchestra-202302210121/@xen-orchestra/backups/_VmBackup.js:479:9)"
      }
    }
  ],
  "end": 1676998936392
}```

[julien-f]

@stucamp I finally had time to investigate this, it should now be fixed

Thanks for your report!

https://github.com/vatesfr/xen-orchestra/commit/24cac9dcd5d83c3170175c9c874d0a3ae51f4e6d

0 julien-f committed to vatesfr/xen-orchestra

fix(xen-api/getResource): fix redirection handling

Introduced by ab96c549a

[Danp]

Check your configuration under Settings > Servers. Only the master host should be listed there.

[olivierlambert]

Also, as usual: https://xen-orchestra.com/docs/community.html#report-a-bug it's a good start

[stucamp]

@Danp Yep, just the master showing.

Currently detaching one of the hosts and reattaching to the pool to see if that sorts anything out!

[stucamp]

No dice. Moved all vms off host 3. Detached it from pool. Reattached it to the pool. Migrated a VM onto it. Backup of VM fails with Error: 302 Found. Migrated VM back to master... Back of VM succeeds without issue.

[stucamp]

@olivierlambert I hadn't explicitely stated as such, but this is with the latest master, all requirements checked, and a fully fresh rebuilt XO.

[olivierlambert]

Okay good I can't guess it's the case, so it's always better to state it

[stucamp]

Okay so interestingly. If I detach a host from the pool and have it in it's own pool. Migrate a VM to that pool/host. Backups go through without issue.

The only time I have an issue is when trying to backup a vm on a host that isn't classified as a master of my primary pool. Not familiar enough with the workings to really guess at what's going on, but any guesses? Maybe just kill the pool, reinstall on the current master and recreate the pool?

[stucamp]

Just to double check to see if it's something awry with the xcp-ng install, I did a full clean install of xcp-ng. Added the two other machines (tc2 and tc3) to the pool and I'm getting the exact same issue. When backing up any vm's on tc2 and tc3, if part of a pool, I get an Error: 302 Found. As soon as i migrate the vms off, detach tc2/3, and migrate the vms back onto those machines as their own pool... backups go through just fine.

[Danp]

Have you tried doing the following?

• Edit the backup job
• Remove the target remote
• Readd the target remote
• Save changes

[stucamp]

@Danp

Yep I've edited, removed, and readded the backup job.

I've nuked the xcp-ng install, resinstalled, xcp-ng, created a brand new debain 11 vm, compiled xo from source using latest master, added hosts to the pool, migrated vms to pool.

I've nuked the remote vdev, created a new vdev, added new vdev as new remote, setup a new backup job using new vdev.

I've detached all the hosts and left them in their own pool, used existing and new backup tasks.

The only time backups go through are when the vm is on a master, whether it be in one bit pool or three seperate pools (one for each host). If the vm is on a master it goes... if it isn't... Error: 302 Found

[olivierlambert]

There's a problem with the redirection on the slave host.

[stucamp]

@olivierlambert
Any suggestions to try? I had a similar suspicion, leading me to disconenct and reconnect the slave hosts using IP, https://<hostname>, and <hostname>... but all methods result in the same outcome a working pool, but unable to backup an VM on the slave hosts.

Not sure where else to go other than nuking everything and starting from scratch. Which means running backups on everything while the hosts are all masters and then playing 3 card monty with the vms in the process.

[olivierlambert]

First thing is to see if someone can reproduce the issue first

[stucamp]

@olivierlambert When I get a bit of time I'll do a bit of a more comprehensive write up outlining initial migration, actions taken, etc in a more cohesive manner. I'll also spin up a few non-critical spare hosts I have on hand and see if I can replicate it or not myself with different equipment.

As it stands now, all the hosts are on their own pool and in the process of running a comprehensive backups for everything to give me a bit of piece of mind. Once that's done, I'll get a bit more aggressive in my efforts!

[olivierlambert]

I was counting on the community to see if someone with a similar setup could report

[olivierlambert]

Okay we have other reports on this. I wonder if it's a recent change. @julien-f do you want to ask @stucamp to do a bisect to identify the culprit?

[julien-f]

@stucamp I finally had time to investigate this, it should now be fixed

Thanks for your report!

https://github.com/vatesfr/xen-orchestra/commit/24cac9dcd5d83c3170175c9c874d0a3ae51f4e6d

0 julien-f committed to vatesfr/xen-orchestra

fix(xen-api/getResource): fix redirection handling

Introduced by ab96c549a

[olivierlambert]

@stucamp you were just the first to discover the issue congrats and thanks for the report!

[stucamp]

@julien-f @olivierlambert
Heyo... good stuff guys. I have recompiled on latest and can happily confirm that the fix has resolved the issue!!

Thanks!