@tjkreidl I'm using the "Backup" feature of Xen Orchestra that was previously called Backup-ng, IIRC. A person creates a backup job that determines the type of backup, i.e. Delta, Continuous Replication, etc. (Those are the old names, though the terminology is going to be changing with XO6/XOLite), the destination "remote" for the backup, either a discrete list of VMs to backup or "smart mode" which is dynamic based on pools to in/exclude and VM tags to in/exclude, and lastly a schedule which has an option to perform a health check (XO restores the backed up VM to the SR of your choice, waits for it to boot successfully, then deletes the restored VM since it was only temporary and not needed). The schedule displays the equivalent cron job syntax, but I'm not sure whether that is implemented by cron or if it's just displayed like that as a convenience.

AFAIK, the backup tool is a higher-level abstraction built on top of XAPI, but with additional niceties, like health checks in this particular case.

My two overlapping jobs are both using "smart mode" to determine the list of VMs to backup based on the tags assigned to the VMs and they both perform health checks. The first is a Delta backup that starts at midnight and usually completes fairly quickly, but sometimes it runs later than 2am when my other backup job starts (continuous Replication to the local storage of one of my xcp-ng hosts).

The issue I'm encountering is that sometimes the second backup begins before the first is finished and sometimes a healthcheck VM is in the middle of booting which results in the second backup job including that healthcheck VM in the list of VMs that it needs to backup. Later, by the time the second backup gets around to actually backing up the healthcheck VM, that VM will have been deleted (the health check is complete), but the second backup job doesn't know that it was deleted, so when it starts making XAPI calls against that healthcheck VM's UUID, XAPI responds indicating that no VM exists with that UUID and XO reports the INVALID_UUID for that particular VM in the backup. Thankfully the backup job is smart enough to know that only that VM failed and it continues with the other VMs.

Thanks for the suggestion, @tjkreidl.

I'm not sure what commands I would run with this cron/systemd job/service.

I assume I would need to utilize the XO API calls to determine the list of running backups and then kill the second if the first is still running.. the issue I see with your suggestion is that my backup log would end up with many failures when I currently only get just one, if any.

While this home-lab thing is a hobby and platform for learning, I have a feeling that your suggestion would require that I invest time into learning how to and then building a Rube Goldberg machine that would results in me becoming dependent upon it, or I could let the seemingly amenable devs work on my low-hanging suggested improvement to their relatively new feature: backup health checks. I suppose I could also look into submitting a pull-request

Regardless, these backups don't hold anything critical per se; only the feeling of satisfaction I get from maintaining moderately resilient backups (I can't afford "3-2-1", but I can afford "2") and getting that sweet notification from my xcp-ng hosted internal mail server that the backup was successful. TBH, I could lose "everything" and not really lose anything because I still have the knowledge and experience and it would give me the excuse to practice settings things up again from scratch.

Also, solutions like adding/upgrading hardware to speed up backups are not options at this point in life due to financial, electrical, and space limitations. As it stands, all of my hardware is 5-10+ years old, second-hand (probably 3rd, 4th or more in some cases--several pieces were donated to GoodWill they were so poorly valued several years ago), and I have only a single 20A 120V circuit breaker powering all lights and outlets in the upstairs of the apartment -- the joys of being an American millennial that graduated high school with little familial wealth just before the great recession that has never managed to get a degree

The neat thing is that these computers give me computational power and learning potential while heating our apartment instead of turning on a heater which only consumes money. I really need to move the computational heaters downstairs for more effective heating.. one of these days!

TL;DR - After some consideration I don't think your suggestion fits my use case, but it did provide for a good thought experiment!

@djingo, FWIW, configuring with XO (from sources) has worked just fine for me. I actually just tested this the other day because I saw that my log server wasn't getting anything. When I looked at the pool settings I realized that I had made a typo. After I fixed it the logs started flowing.

TL;DR, I think just setting the remote syslog host is enough, but I could be wrong.

The host's logs might have some insights you can glean: https://xcp-ng.org/docs/troubleshooting.html#log-files

@florent thank you! Please let me know if you would any more information or further assistance from me. As of yet, the scenario I described is a just a theory as I wanted to get feedback about whether it is a reasonable hypothesis before attempting to conclusively replicate it.

Also, I realized the other day that I had a typo in my remote syslog host address (for who knows how long--apparently I don't check my logs often which I'm calling a sign of reliable tools and setup ) so I don't have logs beyond the backup report which doesn't give much more information than the UUID of a VM that doesn't exist anymore..

In any case, now that my logging is fixed, if I see this happen again, I'll try to gather more details and share them.

@pdonias thanks for the update! I took a look at the latest commit and apparently I was right even though it didn't feel right.

Thanks again to you all for this amazing set of FOSS tools!

Hello,

I have two backup jobs that I attempted to offset to prevent them from running at the same time, but sometimes they take longer than others and they end up overlapping which causes problems when using "smart mode" to match VMs to backup by their tags.

I've noticed that sometimes if health check VM from backup job A is being "restored" while the other is running then I will get the UUID_INVALID error for a single VM that doesn't exist and I suspect that backup job B is attempting to backup the healthcheck vm because it has matching tags, but then the healthcheck vm is deleted after the check is complete which triggers the error I'm seeing.

Obviously, I could make efforts to avoid the two backup jobs running at the same time, but I'm hoping that there may be some sort of tag applied to a healthcheck VM that indicates that it is being used for a health-check which would allow me to configure the "smart mode" to exclude those VMs.

If this isn't already feature, I would like to vote for it being added -- the tag could be something like xo-backup-healthcheck it would be fitting with similar tags.

Any other advice or suggestions are appreciated as well.

Thanks again!

Hello. I don't have a bug to report per se, but more of a curious observation. I'm looking to see if there is an obvious explanation for what I saw that I'm not familiar with. Also not sure if this belongs here or in the XO category.

I'm using XO built from sources (just updated to commit 3c047 this morning after the 5.87 Release).

I have 3 xcp-ng 8.2.1 hosts, xcp-ng-1, xcp-ng-2, xcp-ng-3 (master). To save power, I have been only using xcp-ng-1 and xcp-ng-3 while keeping xcp-ng-2 powered off. I have several VMs with host-affinity set to xcp-ng-3 because it has slower, lower TDP CPUs and I want to reserve xcp-ng-1 for vms running a game servers as it has faster CPUs.

Today I powered up xcp-ng-2 to perform a rolling pool update which went smoothly and was uneventful except that once the final host had updated, several VMs appeared to migrate from xcp-ng-3 to xcp-ng-3 (the same host, no typo) which seemed strange to me...

I checked in several places--list of running vms, the VMs themselves, list of hosts, tasks, etc.--but they all consistently showed that the VMs were being migrated from xcp-ng-3 to xpc-ng-3 and the only machine that was "busy" (yellow/amber) was xcp-ng-3 which is the pool master.

Perhaps the load balancer plugin decided that it should run after the rolling pool update to redistribute compute resources and some of the VMs it picked had host-affinity for the host they were already running on which resulted in them being migrated to the same host? This sounds silly to say haha -- I didn't think it was possible or that there would ever be a reason to migrate this way and I'm feeling a bit gaslit

All jokes aside, I'm "reporting" this behavior in case it might indicative to devs of some greater issue, but mostly I'm curious why this would happen. Has anyone seen something like this before?

Thanks in advance!

EDIT Here's a screenshot of my task history that shows what I saw:

@olivierlambert Thank you and your team again for your commitment to this fantastic FOSS tool and for allowing me to build it myself!

I very much appreciate the personal touch of my issues being triaged by the CEO and Co-Founder. It's refreshing to see an executive officer stay in touch with their customer base.

@florent the CR job was completed with health checks. The issue appears to be fixed in the fix_cr_healthcheck branch.

@florent I reviewed the change you made and got slightly embarrassed that my brain didn't notice the missing "l" when I had previously reviewed MixinXapiWriter.mjs

After checking out the fix_cr_healthcheck branch and rebuilding, I restarted the backup of just one VM in the backup job and it was successful!

Now, I will run the full backup job and report back for the sake of being thorough.

@florent thank you for your effort as well!

I will checkout that branch, attempt to restart the backup job for one of the small VMs, then I'll report back here, hopefully before the end of the day (UTC-7).

Hi, @florent. The CR job is configured for "Normal" snapshots, not with memory.

Also, for curiosity's sake, I decided to rebuild this same rule from scratch--the hypothesis was that the record of the original CR job could have been somehow "broken" or now incompatible. The backup job was rebuilt with exactly the same settings apart from "Rebuild of " being added to the beginning of the job and schedule names.

Because it's a new CR backup of ~20 VMs to really cheap spinning disks, the backup is still in progress after 8 hours (another delta-backup job also started ~30 minutes after I started the new job), but so far it has yielded the same results: apart from the 2 VMs that are in progress, all other VMs have failed with the same TypeError.

I'm currently running Xen Orchestra, commit abd0a built from sources using ronivay's XenOrchestraInstallerUpdater.

Thank you for your assistance, @florent! Please let me know if you need additional information or want me to perform tests.

@florent I just updated to Xen Orchestra, commit 6b936, then attempted to restart the failed backup (only a single VM, not the whole job) and I am still getting the error.

It looks like there was another commit about an hour ago, but it doesn't look to be related to this at all.

Here's the full JSON error log:

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1690565798795",
  "jobId": "7ee43819-d12c-416c-ad07-95dad15bc47d",
  "jobName": "Weekly continuous replication of core services and Work VMs to TrueNAS - NFS - HDDs-3.5in - VM Backups at 2am on Mondays",
  "message": "backup",
  "scheduleId": "d9076c80-5b4f-4f8e-ae59-9621ed2575c6",
  "start": 1690565798795,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "69ed2d54-b28a-4ec5-195f-dee283f730cf"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "69ed2d54-b28a-4ec5-195f-dee283f730cf",
        "name_label": "mgmt-win10.jeff.intangible.home.arpa"
      },
      "id": "1690565809617",
      "message": "backup VM",
      "start": 1690565809617,
      "status": "failure",
      "tasks": [
        {
          "id": "1690565811948",
          "message": "snapshot",
          "start": 1690565811948,
          "status": "success",
          "end": 1690565818268,
          "result": "97927cb2-10fc-e0ff-5c6a-0e3a0b3e405d"
        },
        {
          "data": {
            "id": "abbf0a1c-d358-3e0d-f697-94791671b9d9",
            "isFull": false,
            "name_label": "TrueNAS - NFS - HDDs-3.5in - VM Backups",
            "type": "SR"
          },
          "id": "1690565818269",
          "message": "export",
          "start": 1690565818269,
          "status": "failure",
          "tasks": [
            {
              "id": "1690565821451",
              "message": "transfer",
              "start": 1690565821451,
              "status": "success",
              "end": 1690565843248,
              "result": {
                "size": 312320
              }
            },
            {
              "id": "1690565868762",
              "message": "health check",
              "start": 1690565868762,
              "status": "failure",
              "end": 1690565868794,
              "result": {
                "message": "Cannot read properties of undefined (reading 'uuid')",
                "name": "TypeError",
                "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n    at #isAlreadyOnHealthCheckSr (file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17"
              }
            }
          ],
          "end": 1690565868794,
          "result": {
            "message": "Cannot read properties of undefined (reading 'uuid')",
            "name": "TypeError",
            "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n    at #isAlreadyOnHealthCheckSr (file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17"
          }
        }
      ],
      "end": 1690565868806,
      "result": {
        "message": "Cannot read properties of undefined (reading 'uuid')",
        "name": "TypeError",
        "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n    at #isAlreadyOnHealthCheckSr (file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:21:49)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202307280948/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.mjs:43:17"
      }
    }
  ],
  "end": 1690565868807
}

Hi, @florent. I'm just checking in to confirm that the SR I used for health checks is available and in use. When I disable the health check, the backup job works just fine. I configured the CR job with a different health check SR, but the job failed with the same error.

Please let me know if there is any other information that you would like from me or if there is anything else I can do to assist.

Hi, @florent. Thank you for your assistance!

The SR used for the health check is available and functional (most of my running VMs boot from this SR). This SR has only 225GB free, but the VM failing the health check has 1x 50GB disk and 1x 100GB disk, so that should be enough to perform a single VM health check. The destination SR for the CR job is not the same as the SR used for the health check. Do they need to be the same? I assume they do not need to be the same since I can choose which SR to use for the health check.

I have 3 hosts in my pool; one is powered down unless needed, so the SR shows an orange color dot because it is not connected to all hosts. Perhaps this could be the reason.

Thanks again for your assistance. FYI, I am about to go on a camping trip (after sending this message) and won't return until July 12. I'll check this thread when I return.

@olivierlambert, thanks again for your help!

I see that there are new commits since my post--I guess I got a bit excited to update before the official announcement was made

As requested, I have updated and rebuilt to commit f32742225, but I am still getting the same issue when I attempt to retry a single VM in that CR job.

Below is the error log:

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1688151228295",
  "jobId": "7ee43819-d12c-416c-ad07-95dad15bc47d",
  "jobName": "Weekly continuous replication of core services and work VMs to TrueNAS - NFS - HDDs-3.5in - VM Backups at 2am on Mondays",
  "message": "backup",
  "scheduleId": "d9076c80-5b4f-4f8e-ae59-9621ed2575c6",
  "start": 1688151228295,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "69ed2d54-b28a-4ec5-195f-dee283f730cf"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "69ed2d54-b28a-4ec5-195f-dee283f730cf",
        "name_label": "mgmt-win10.jeff.intangible.home.arpa"
      },
      "id": "1688151279573",
      "message": "backup VM",
      "start": 1688151279573,
      "status": "failure",
      "tasks": [
        {
          "id": "1688151282519",
          "message": "snapshot",
          "start": 1688151282519,
          "status": "success",
          "end": 1688151288614,
          "result": "d74cc31c-5287-81f3-176c-ffb3998c9694"
        },
        {
          "data": {
            "id": "abbf0a1c-d358-3e0d-f697-94791671b9d9",
            "isFull": false,
            "name_label": "TrueNAS - NFS - HDDs-3.5in - VM Backups",
            "type": "SR"
          },
          "id": "1688151288616",
          "message": "export",
          "start": 1688151288616,
          "status": "failure",
          "tasks": [
            {
              "id": "1688151292272",
              "message": "transfer",
              "start": 1688151292272,
              "status": "success",
              "end": 1688151313070,
              "result": {
                "size": 312320
              }
            },
            {
              "id": "1688151326976",
              "message": "health check",
              "start": 1688151326976,
              "status": "failure",
              "end": 1688151327016,
              "result": {
                "message": "Cannot read properties of undefined (reading 'uuid')",
                "name": "TypeError",
                "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n    at #isAlreadyOnHealthCheckSr (/opt/xo/xo-builds/xen-orchestra-202306301139/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:23:49)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202306301139/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:45:17"
              }
            }
          ],
          "end": 1688151327016,
          "result": {
            "message": "Cannot read properties of undefined (reading 'uuid')",
            "name": "TypeError",
            "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n    at #isAlreadyOnHealthCheckSr (/opt/xo/xo-builds/xen-orchestra-202306301139/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:23:49)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202306301139/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:45:17"
          }
        }
      ],
      "end": 1688151327029,
      "result": {
        "message": "Cannot read properties of undefined (reading 'uuid')",
        "name": "TypeError",
        "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n    at #isAlreadyOnHealthCheckSr (/opt/xo/xo-builds/xen-orchestra-202306301139/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:23:49)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202306301139/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:45:17"
      }
    }
  ],
  "end": 1688151327034
}

Hello,

Thanks again for this awesome open-source project!

I discovered that one of my continuous replication backup jobs is failing to perform health checks despite completing transfers. I have two other CR jobs that are still working just fine. I updated Xen Orchestra this morning to commit 5f71e. This XO community edition (built from sources) instance is running on Debian 10 which has all of its system updates installed.

Is this something that I can fix on my own or might this be a symptom of some of the recent changes to backup nomenclature?

Here's the log file for the last run of this job:

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1688145264192",
  "jobId": "7ee43819-d12c-416c-ad07-95dad15bc47d",
  "jobName": "Weekly continuous replication of core services and work VMs to TrueNAS - NFS - HDDs-3.5in - VM Backups at 2am on Mondays",
  "message": "backup",
  "scheduleId": "d9076c80-5b4f-4f8e-ae59-9621ed2575c6",
  "start": 1688145264192,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "69ed2d54-b28a-4ec5-195f-dee283f730cf"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "69ed2d54-b28a-4ec5-195f-dee283f730cf",
        "name_label": "mgmt-win10.jeff.intangible.home.arpa"
      },
      "id": "1688145336401",
      "message": "backup VM",
      "start": 1688145336401,
      "status": "failure",
      "tasks": [
        {
          "id": "1688145386549",
          "message": "snapshot",
          "start": 1688145386549,
          "status": "success",
          "end": 1688145393151,
          "result": "88a6830c-b17c-1076-e6b9-6240888b778a"
        },
        {
          "data": {
            "id": "abbf0a1c-d358-3e0d-f697-94791671b9d9",
            "isFull": false,
            "name_label": "TrueNAS - NFS - HDDs-3.5in - VM Backups",
            "type": "SR"
          },
          "id": "1688145393152",
          "message": "export",
          "start": 1688145393152,
          "status": "failure",
          "tasks": [
            {
              "id": "1688145396765",
              "message": "transfer",
              "start": 1688145396765,
              "status": "success",
              "end": 1688145408511,
              "result": {
                "size": 312320
              }
            },
            {
              "id": "1688145423393",
              "message": "health check",
              "start": 1688145423393,
              "status": "failure",
              "end": 1688145423434,
              "result": {
                "message": "Cannot read properties of undefined (reading 'uuid')",
                "name": "TypeError",
                "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n    at #isAlreadyOnHealthCheckSr (/opt/xo/xo-builds/xen-orchestra-202306300957/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:23:49)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202306300957/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:45:17"
              }
            }
          ],
          "end": 1688145423434,
          "result": {
            "message": "Cannot read properties of undefined (reading 'uuid')",
            "name": "TypeError",
            "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n    at #isAlreadyOnHealthCheckSr (/opt/xo/xo-builds/xen-orchestra-202306300957/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:23:49)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202306300957/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:45:17"
          }
        }
      ],
      "end": 1688145423447,
      "result": {
        "message": "Cannot read properties of undefined (reading 'uuid')",
        "name": "TypeError",
        "stack": "TypeError: Cannot read properties of undefined (reading 'uuid')\n    at #isAlreadyOnHealthCheckSr (/opt/xo/xo-builds/xen-orchestra-202306300957/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:23:49)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202306300957/@xen-orchestra/backups/_runners/_writers/_MixinXapiWriter.js:45:17"
      }
    }
  ],
  "end": 1688145423449
}

@HeMaN You're correct that no changes have been published yet.

We were under the impression that we had found an undocumented requirement, but I was reminded that giving each host the same certificate is not the best practice. xcp-ng should be able to handle each host having its own certificate as long as their respective certificate authorities are trusted.

In any case, I need to do some more testing to narrow down the exact cause of the issues that I was seeing. I have been working a systemd service and timer with a few supporting scripts that automatically renew certificates by making ACME requests to my local, private CA, specifically adding support for additional SANs (previous iterations just used the system's FQDN).

Specifically, I want to test each server with SANs that correspond to each of it's IP addresses and FQDNs, deploy them using xe host-server-certificate-install, then perform packet captures as needed to determine why the Xapi#getResource /rrd_updates (on xcp-ng-1) 0% task is getting stuck.

So far, life has gotten a bit in the way, so I haven't dedicated the time to testing this, but I hope to get back to this soon.

@Andrew of course -- this is how FOSS thrives. I am now back to my original configuration where I am running my CR jobs at 2am and I have removed the tag that excluded my XO vm from the mix, because I definitely want backups of that!

To be clear, it is entirely valid to use XO to backup itself and this makes me happy