Latest commit smoked NFS remotes

fataugie

Not sure who needs to hear this, but I applied last nights updates (7 or so commits from 12/16) and overnight my backups to NFS shares failed. I rolled back to 4784b and was able to re-enable the shares.
The error message was remotes were disabled. When I tried to re-enable, the log message was the same....remote disabled.

remote.test
{
  "id": "7ec02e68-7d51-4ea0-afdd-532747e33034"
}
{
  "message": "remote is disabled",
  "name": "Error",
  "stack": "Error: remote is disabled
    at _class2.getRemoteWithCredentials (file:///opt/xo/xo-builds/xen-orchestra-202312190034/packages/xo-server/src/xo-mixins/remotes.mjs:191:13)
    at _class2.testRemote (file:///opt/xo/xo-builds/xen-orchestra-202312190034/packages/xo-server/src/xo-mixins/remotes.mjs:105:20)
    at Api.#callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202312190034/packages/xo-server/src/xo-mixins/api.mjs:445:20)"
}

olivierlambert

Hey Thanks for the feedback. So right now you are on the last commit on master, right? (I mean, when you have the issue)

fataugie

@olivierlambert Yeah, sorry I didn't get the one I was actually on when it died. I believe about told me I was 7 behind the one I rolled back to this morning when I ran the update last night.

olivierlambert

Ping @florent maybe

julien-f

Thanks, I'm investigating.

Ref: https://github.com/vatesfr/xen-orchestra/issues/7246

andrew64k created this issue in vatesfr/xen-orchestra

closed S3 Remotes Broken (Disabled) #7246

julien-f

Should be fixed, let me know if that's not the case.

https://github.com/vatesfr/xen-orchestra/commit/5a0cfd86c7cb0be80631427b9678ea687eccbf63

0

julien-f committed to vatesfr/xen-orchestra

fix(xo-server/remote#_unserialize): handle boolean enabled

Introduced by 32afd5c46

Fixes #7246
Fixes https://xcp-ng.org/forum/post/68575

Andrew

@julien-f
Fixed. Thanks.

fataugie

@julien-f Cool, I'll run the update and let you know.

Andrew

@julien-f One step forward, two steps back....

While the remote backup target (S3 or NFS) can now be enabled and backups work, restore shows nothing. So there's a new related issue.

olivierlambert

I suppose you clicked on "Refresh backup list"? Any error somewhere?

Andrew

@olivierlambert

refresh... refresh... refresh...

Dec 19 13:07:33 xo2 xo-server[109910]: 2023-12-19T18:05:42.578Z xo:xo-mixins:backups-ng WARN listVmBackups for remote 9890e0c4-ba3a-4810-8245-a49fdf16b16e: {
Dec 19 13:07:33 xo2 xo-server[109910]:   error: TypeError: Cannot convert undefined or null to object
Dec 19 13:07:33 xo2 xo-server[109910]:       at Function.values (<anonymous>)
Dec 19 13:07:33 xo2 xo-server[109910]:       at formatVmBackup (file:///opt/xo/xo-builds/xen-orchestra-202312191146/@xen-orchestra/backups/formatVmBackups.mjs:28:30)
Dec 19 13:07:33 xo2 xo-server[109910]:       at Array.map (<anonymous>)
Dec 19 13:07:33 xo2 xo-server[109910]:       at file:///opt/xo/xo-builds/xen-orchestra-202312191146/@xen-orchestra/backups/formatVmBackups.mjs:35:52
Dec 19 13:07:33 xo2 xo-server[109910]:       at /opt/xo/xo-builds/xen-orchestra-202312191146/node_modules/lodash/mapValues.js:38:34
Dec 19 13:07:33 xo2 xo-server[109910]:       at /opt/xo/xo-builds/xen-orchestra-202312191146/node_modules/lodash/_createBaseFor.js:17:11
Dec 19 13:07:33 xo2 xo-server[109910]:       at baseForOwn (/opt/xo/xo-builds/xen-orchestra-202312191146/node_modules/lodash/_baseForOwn.js:13:20)
Dec 19 13:07:33 xo2 xo-server[109910]:       at mapValues (/opt/xo/xo-builds/xen-orchestra-202312191146/node_modules/lodash/mapValues.js:37:3)
Dec 19 13:07:33 xo2 xo-server[109910]:       at formatVmBackups (file:///opt/xo/xo-builds/xen-orchestra-202312191146/@xen-orchestra/backups/formatVmBackups.mjs:35:10)
Dec 19 13:07:33 xo2 xo-server[109910]:       at Function.<anonymous> (file:///opt/xo/xo-builds/xen-orchestra-202312191146/packages/xo-server/src/xo-mixins/backups-ng/index.mjs:587:11)
Dec 19 13:07:33 xo2 xo-server[109910]: }
Dec 19 13:07:33 xo2 xo-server[109910]: 2023-12-19T18:07:33.588Z xo:xo-mixins:backups-ng WARN listVmBackups for remote ac27c736-d5da-41b0-8cb7-32e8719958ef: {
Dec 19 13:07:33 xo2 xo-server[109910]:   error: Error: remote is disabled
Dec 19 13:07:33 xo2 xo-server[109910]:       at _class2.getRemoteWithCredentials (file:///opt/xo/xo-builds/xen-orchestra-202312191146/packages/xo-server/src/xo-mixins/remotes.mjs:191:13)
Dec 19 13:07:33 xo2 xo-server[109910]:       at BackupNg._listVmBackupsOnRemote (file:///opt/xo/xo-builds/xen-orchestra-202312191146/packages/xo-server/src/xo-mixins/backups-ng/index.mjs:573:22)
Dec 19 13:07:33 xo2 xo-server[109910]: }

julien-f

@Andrew It appears to be related to a PR we just merged which introduces a lot of changes.

We'll fix it tomorrow, feel free to go back to a previous commit in the meantime. 5a0cfd86c7 appears to work fine.

Andrew

@julien-f Working again. Thanks.

fataugie

@Andrew
So on release 5cf5d on the master, there was a strange issue of when you create a backup job, fill everything in and hit save, the first time it would not save the schedule. Edit the job, and the schedule was not there....so I recreate the schedule and hit save again. This time, the schedule was saved, but the retention count reverts to either null or zero (nothing is shown in the gui), and if you try to run it would complain that you can't have a blank retention in the schedule section. Edit the job again and change the retention and then....if you hit save, is everything finally committed properly.

In the meantime, I see four more commits have taken place on the Xen Orchestra so I'm updating and will test once that's complete and report my results.

Thanks for all the hard work guys!

UPDATE: No joy, same progression through the failures to create a backup job.

olivierlambert

@fataugie I can confirm and reproduce this bug, thanks for the report!

julien-f

@fataugie Investigating.

fataugie

@olivierlambert said in Latest commit smoked NFS remotes:

@fataugie I can confirm and reproduce this bug, thanks for the report!

No problem....If I had any idea how to submit via Github, I would have there....but I'm a novice at best with Git

olivierlambert

Don't worry, since it's easy to trigger, your report is very valuable even if on this thread and not on Github. Thanks again!

julien-f

@fataugie Should be fixed, let me know if there are any issues left!

https://github.com/vatesfr/xen-orchestra/commit/194db8d0dd28ccfd058c97e8c16b32d853b26cf8

0

julien-f committed to vatesfr/xen-orchestra

fix(xo-server/Redis#{add,update}): return complete record

Introduced by 32afd5c46

Fixes https://xcp-ng.org/forum/post/68664

The returned record should not be serialized and should contain the identifier.

fataugie

@julien-f said in Latest commit smoked NFS remotes:

@fataugie Should be fixed, let me know if there are any issues left!

https://github.com/vatesfr/xen-orchestra/commit/194db8d0dd28ccfd058c97e8c16b32d853b26cf8

Will do! Thanks for the quick attention.
I'm currently remote and my network has decided to not let me in so it will be probably 10+ hours until I can get the update installed and tested. I'll report back one way or the other asap.
Thanks