Backup started to fail

fred974

Hi,

We are using XO community to run our backup and we had no issue for a while. Recently our Pool metadata + XO config to backblaze b2 started to fail with the following error message:

Error: more than 1000 objects, unsupported in this implementation

--- Backup settings ---

--- Remote settings ---

Could anyone please tell me if this is an issue with Backblaze b2 or xcp-ng. Given the error message I believe it is Backblaze but I just wanted to make sure.

Thank you

olivierlambert

Let me ping @florent or @julien-f

julien-f

@fred974 Please show me the full log (I need the stacktrace of this error), there is a button at the top of the log modal.

fred974

@julien-f Find bellow the the full log file.
2022-02-07T00_00_00.026Z - backup NG.txt

Thank you

julien-f

@fred974 Thank you. Unfortunately there isn't much more info, can you take a look at your remote and tell me how many files you have in /xo-config-backups/4866b683-59a1-42ba-93d8-b704b5bbbbe6 and what do they look like?

fred974

@julien-f this is what I see inside /xo-config-backups/4866b683-59a1-42ba-93d8-b704b5bbbbe6

There are 48 folders in there and here the content of the last folder.

Thank you

fred974

According to backblaze support, there is over 5 million files in the bucket. Is that not too much for my setting of daily backup?

olivierlambert

5M blocks of 2MiB, means near 10TiB of data. Is that the expected volume of your backups?

julien-f

@fred974 What's important for XO is that there is no more than 1000 files per directories.

If this condition is respected, then maybe it's an issue between our implementation and Blackblaze (particularly Delimiter support for ListObjectsV2).

Our implementation has been tested against AWS S3 and MinIO.

florent

Hi fred,

I looked into your problem and did not reproduced it on our test lab. Have you got a xo support subscription ? That would allow us to open a support tunnel and search a solution more efficiently

Regards

Florent

fred974

@florent said in Backup started to fail:

Have you got a xo support subscription  ?

No I don't. We use the community edition alonside XOA..

stevewest15

XO 5.67.0

Hi,

@florent @julien-f I just updated XO via the web browser from an older version to the latest version 5.67.0 and now I'm getting the same issue as @fred974 :

Mar 07 08:20:29 xoa xo-server[13189]:       at S3Handler._list (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/src/s3.js:199:21)
Mar 07 08:20:29 xoa xo-server[13189]:       at runMicrotasks (<anonymous>)
Mar 07 08:20:29 xoa xo-server[13189]:       at processTicksAndRejections (node:internal/process/task_queues:96:5) {
Mar 07 08:20:29 xoa xo-server[13189]:     dir: '/xo-vm-backups'
Mar 07 08:20:29 xoa xo-server[13189]:   }
Mar 07 08:20:29 xoa xo-server[13189]: }
Mar 07 08:34:12 xoa xo-server[13189]: 2022-03-07T13:34:12.720Z xo:xo-mixins:backups-ng WARN listVmBackups for remote 53745311-c2b2-41d9-bf98-67d9789b16a3: {
Mar 07 08:34:12 xoa xo-server[13189]:   error: Error: more than 1000 objects, unsupported in this implementation
Mar 07 08:34:12 xoa xo-server[13189]:       at S3Handler._list (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/src/s3.js:199:21)
Mar 07 08:34:12 xoa xo-server[13189]:       at runMicrotasks (<anonymous>)
Mar 07 08:34:12 xoa xo-server[13189]:       at processTicksAndRejections (node:internal/process/task_queues:96:5) {
Mar 07 08:34:12 xoa xo-server[13189]:     dir: '/xo-vm-backups'
Mar 07 08:34:12 xoa xo-server[13189]:   }
Mar 07 08:34:12 xoa xo-server[13189]: }

Backblaze Bucket:

After doing the XO update, I can't see the list of backups when I go to Backup > Restore:

Prior to updating XO to the latest version, I was able to see the backups on Backblaze when going to Backup > Restore.

Here is the prior version of XOA installed which didn't have this issue:

Mar  7 06:52:13 xoa xoa-updater[479]: stable channel selected
Mar  7 06:52:13 xoa xoa-updater[479]: Following packages will be removed:
Mar  7 06:52:13 xoa xoa-updater[479]:   xen-orchestra-upload-ova 0.1.4
Mar  7 06:52:13 xoa xoa-updater[479]:   xen-orchestra 5.65.3
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-server-telemetry 0.5.0
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-server-xoa 0.12.0
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-server 5.85.1
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-web-free 5.90.0
Mar  7 06:52:13 xoa xoa-updater[479]:   xoa-cli 0.28.0
Mar  7 06:52:13 xoa xoa-updater[479]:   xoa-updater 0.39.0
Mar  7 06:52:13 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:13 xoa xoa-updater[479]:   node 16.13.2 (14.17.5 installed)
Mar  7 06:52:13 xoa xoa-updater[479]:   npm 8.4.0 (6.14.13 installed)
Mar  7 06:52:13 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:13 xoa xoa-updater[479]:   xoa-updater 0.39.0 (0.39.0 installed)
Mar  7 06:52:13 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:13 xoa xoa-updater[479]:   xen-orchestra 5.67.0 (5.65.3 installed)
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-server 5.87.0 (5.85.1 installed)
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-web-free 5.92.0 (5.90.0 installed)
Mar  7 06:52:13 xoa xoa-updater[479]:   xoa-cli 0.29.0 (0.28.0 installed)
Mar  7 06:52:16 xoa xoa-updater[479]: stable channel selected
Mar  7 06:52:16 xoa xoa-updater[479]: Following packages will be removed:
Mar  7 06:52:16 xoa xoa-updater[479]:   xen-orchestra-upload-ova 0.1.4
Mar  7 06:52:16 xoa xoa-updater[479]:   xen-orchestra 5.65.3
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-server-telemetry 0.5.0
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-server-xoa 0.12.0
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-server 5.85.1
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-web-free 5.90.0
Mar  7 06:52:16 xoa xoa-updater[479]:   xoa-cli 0.28.0
Mar  7 06:52:16 xoa xoa-updater[479]:   xoa-updater 0.39.0
Mar  7 06:52:16 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:16 xoa xoa-updater[479]:   node 16.13.2 (14.17.5 installed)
Mar  7 06:52:16 xoa xoa-updater[479]:   npm 8.4.0 (6.14.13 installed)
Mar  7 06:52:16 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:16 xoa xoa-updater[479]:   xoa-updater 0.39.0 (0.39.0 installed)
Mar  7 06:52:16 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:16 xoa xoa-updater[479]:   xen-orchestra 5.67.0 (5.65.3 installed)
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-server 5.87.0 (5.85.1 installed)
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-web-free 5.92.0 (5.90.0 installed)
Mar  7 06:52:16 xoa xoa-updater[479]:   xoa-cli 0.29.0 (0.28.0 installed)

florent

hi @stevewest15 , can you create a support ticket on https://help.vates.fr/ ? Then we will open a tunnel support to look into this in depth

Regards,
Florent

stevewest15

@florent Thank you but can't seem to login to the support ticket system:

Login failed. Have you double-checked your credentials and completed the email verification step?

Not sure how to get the email verification or to reset password as I don't see that option on the login page.

Darkbeldin

@stevewest15
Hi,

It's the same login you used just before on the website, it's not a new account you just have to confirm your password.

stevewest15

@Darkbeldin Thank you! I was able to get in after 1st logging to the xen-orchestra.com website, updating my profile, and then clicking on support from that portal.

Here is the Ticket#775709.

Look forward to your update!

Best Regards,

SW

Darkbeldin

@stevewest15 Happy to help