Old thread but I'd like to add a follow-up on this.
The issues continued with the new Minio instance but probably the storage is quite slow which I would guess caused the issues with listing and caused timeouts.

After a lot of troubleshooting it seems to have been improved (so far) by increasing the socketTimeout for S3 from 10 minutes (600000) to 2 hours (7200000) in '@xen-orchestra/fs/dist/s3.js'

I thought it could be good to know, possibly add a config option to adjust the timeout if needed if it doesn't exist already. I could not find one at least.

That sounds interesting, the backups are on a new remote on a new Minio-instance so the issue is handled for now. I guess we will see over time if the backup times increases.

One thing I've notices is that if I do a trace in Minio I see no S3-requests but suddenly it shows a couple of s3.list_objects_v2 witch gives status 499 and the duration is just about 10 minutes. But this gives no timeoutin the backuplog.

Are you thinking about test a public provider such as AWS?
I'm not sure we are able to do so. One thing I'm about to do is to test a newer version of Minio in docker on another server running Ubuntu 24.04
As of now it's an older version running native on a OmniOS server ontop of ZFS-storage.

Is there a way to get more verbose logging from the backup job?

Hi,

I have problems with delta backup which take a long time to finish.
The remote is in a local Minio instance to buckets with a retention-policy set to get immutable backups.
I has been working fine for about 4 month but it's after that it got bad.

First of all, I'm not sure it's related to the long backup times but I get some of these in the logs and my guess it's because with retention enabled Minio keeps empty directories/prefixes until the deleted files exceeds than the retention period:

no alias references VHD

Transfer and Merge seems quite ok and I think it is "Clean VM Directory" which is the culprit, it can go on for hours.
At first I thought the long backup times could be related to performance on Minio but I can't see any S3 requests at that time.
During my troubleshooting I start to think it's the backup-process which is idle or waiting and will continue after a timeout but I don't know how to debug this further.
There have been a few timeout logged:

            "message": "Connection timed out after 600000 ms",
            "stack": "TimeoutError: Connection timed out after 600000 ms\n    at ClientRequest.<anonymous> (/opt/xen-orchestra/node_modules/@aws-sdk/node-http-handler/node_modules/@smithy/node-http-handler/dist-cjs/set-socket-timeout.js:7:30)\n    at Object.onceWrapper (node:events:632:28)\n    at ClientRequest.emit (node:events:518:28)\n    at ClientRequest.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)\n    at TLSSocket.emitRequestTimeout (node:_http_client:863:9)\n    at Object.onceWrapper (node:events:632:28)\n    at TLSSocket.emit (node:events:530:35)\n    at TLSSocket.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)\n    at Socket._onTimeout (node:net:604:8)\n    at listOnTimeout (node:internal/timers:588:17)"

I have adjusted Concurrency both up and down, with a lower value the timeout seems to be avoided, but even without there errors the backup times is just as large.

Some assistance would be much appreciated.

I just posted this in omnios-discuss but I'll try here as well, not sure if I'm picked the correct category.

OmniOS guest fails to boot in xcp-ng on version 8.1 and 8.2.

Installation is fine: create a new vm, set viridian=false and installation progress smoothly. I've noticed that installation seem to lack PV-drivers.

When booting from freshly installed disk it panics with:
panic[cpu0]/thread=ffffffffffbc490c0: unable to configure /xpvd nexus

I've tried several version of OmniOS (and latest OpenIndiana), tried disabling apix with apix_enable=0 in /etc/system and all without success.
The same setup in xcp-ng 8.0 works like a charm with working PV-drivers (but no tools).

Anyone with an idea of a workaround or even better, a resolution to this.

Best regards
Niclas