@Danp Thanks, that makes sense. Perhaps is should show "Premium Feature" instead of "netdata plugin is necessary" when using the free versions.

Since the update I have run full backup that have taken over an hour to complete due to their size and many smaller delta backups and the error has not come up again. I assume whatever was causing the error was fixed in the xo-server 5.84.2 update.

Since the update I have run full backup that have taken over an hour to complete due to their size and many smaller delta backups and the error has not come up again. I assume whatever was causing the error was fixed in the xo-server 5.84.2 update.

Just noticed there was another update that makes a few changes to backup, I did not see the same error message that I am getting but I updated to the latest version and I will do some more testing to see if the error message comes back.
https://github.com/vatesfr/xen-orchestra/blob/master/CHANGELOG.md

I am running xo-server 5.84.1 xo-web 5.90.0 built from source and since the latest update I am sometimes getting a Error: HTTP connection has timed out. I never had this error come up prior to the upgrade and It's not consistent and running the same backup a few more times does not cause it to fail each time. Not sure if it might be the system getting a bit overloaded when running a while running a larger the backup, it's never occurred on small VM's or delta backups. The XO vm is running on a Xeon E5-2670 0 @ 2.60GHz with 16 cores assigned & 4GB of memory.

{
  "data": {
    "mode": "full",
    "reportWhen": "failure"
  },
  "id": "1638444000006",
  "jobId": "d1abe537-771e-4b62-8c3c-05d08c6854e8",
  "jobName": "Graylog",
  "message": "backup",
  "scheduleId": "6ea649f1-a45f-4665-af52-71ccad7edefe",
  "start": 1638444000006,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "62570b52-e56b-a79b-ca5f-88803038532f"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "62570b52-e56b-a79b-ca5f-88803038532f"
      },
      "id": "1638444000875",
      "message": "backup VM",
      "start": 1638444000875,
      "status": "failure",
      "tasks": [
        {
          "id": "1638444015767",
          "message": "snapshot",
          "start": 1638444015767,
          "status": "success",
          "end": 1638444017382,
          "result": "ada208e5-cb2f-5e4f-f6d1-19c64e6c8026"
        },
        {
          "data": {
            "id": "b0323f4d-1828-4ad1-b9bd-550f38ff6cfa",
            "type": "remote",
            "isFull": true
          },
          "id": "1638444017480",
          "message": "export",
          "start": 1638444017480,
          "status": "failure",
          "tasks": [
            {
              "id": "1638444017488",
              "message": "transfer",
              "start": 1638444017488,
              "status": "failure",
              "end": 1638450296391,
              "result": {
                "canceled": false,
                "method": "GET",
                "url": "https://192.168.3.92/export/?ref=OpaqueRef%3A0ca54512-7a8c-452b-af08-bb18c83a0138&use_compression=zstd&session_id=OpaqueRef%3A09c8ef17-0f39-464a-a2c2-e3d9c0a51499&task_id=OpaqueRef%3Ae16c33e8-8360-4848-8a29-20ae6027e103",
                "timeout": true,
                "message": "HTTP connection has timed out",
                "name": "Error",
                "stack": "Error: HTTP connection has timed out\n    at IncomingMessage.emitAbortedError (/etc/xo/xo-builds/xen-orchestra-202112010617/node_modules/http-request-plus/index.js:83:19)\n    at Object.onceWrapper (events.js:519:28)\n    at IncomingMessage.emit (events.js:400:28)\n    at IncomingMessage.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n    at IncomingMessage.emit (domain.js:475:12)\n    at TLSSocket.socketCloseListener (_http_client.js:432:11)\n    at TLSSocket.emit (events.js:412:35)\n    at TLSSocket.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n    at TLSSocket.emit (domain.js:475:12)\n    at net.js:686:12"
              }
            }
          ],
          "end": 1638450296392,
          "result": {
            "canceled": false,
            "method": "GET",
            "url": "https://192.168.3.92/export/?ref=OpaqueRef%3A0ca54512-7a8c-452b-af08-bb18c83a0138&use_compression=zstd&session_id=OpaqueRef%3A09c8ef17-0f39-464a-a2c2-e3d9c0a51499&task_id=OpaqueRef%3Ae16c33e8-8360-4848-8a29-20ae6027e103",
            "timeout": true,
            "message": "HTTP connection has timed out",
            "name": "Error",
            "stack": "Error: HTTP connection has timed out\n    at IncomingMessage.emitAbortedError (/etc/xo/xo-builds/xen-orchestra-202112010617/node_modules/http-request-plus/index.js:83:19)\n    at Object.onceWrapper (events.js:519:28)\n    at IncomingMessage.emit (events.js:400:28)\n    at IncomingMessage.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n    at IncomingMessage.emit (domain.js:475:12)\n    at TLSSocket.socketCloseListener (_http_client.js:432:11)\n    at TLSSocket.emit (events.js:412:35)\n    at TLSSocket.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n    at TLSSocket.emit (domain.js:475:12)\n    at net.js:686:12"
          }
        }
      ],
      "end": 1638450311244,
      "result": {
        "canceled": false,
        "method": "GET",
        "url": "https://192.168.3.92/export/?ref=OpaqueRef%3A0ca54512-7a8c-452b-af08-bb18c83a0138&use_compression=zstd&session_id=OpaqueRef%3A09c8ef17-0f39-464a-a2c2-e3d9c0a51499&task_id=OpaqueRef%3Ae16c33e8-8360-4848-8a29-20ae6027e103",
        "timeout": true,
        "message": "HTTP connection has timed out",
        "name": "Error",
        "stack": "Error: HTTP connection has timed out\n    at IncomingMessage.emitAbortedError (/etc/xo/xo-builds/xen-orchestra-202112010617/node_modules/http-request-plus/index.js:83:19)\n    at Object.onceWrapper (events.js:519:28)\n    at IncomingMessage.emit (events.js:400:28)\n    at IncomingMessage.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n    at IncomingMessage.emit (domain.js:475:12)\n    at TLSSocket.socketCloseListener (_http_client.js:432:11)\n    at TLSSocket.emit (events.js:412:35)\n    at TLSSocket.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n    at TLSSocket.emit (domain.js:475:12)\n    at net.js:686:12"
      }
    }
  ],
  "end": 1638450311245
}

@julien-f

Awesome, thanks!

One other question, and what sent me learning about the xo-cli, how would I filter for running VM's that are using a particular shared storage? I have a multiple shared storage systems attached to the pool. I need to update and reboot one of those storage units so I would like to move or stop the VM's that are using it. I know I can see which ones are running by going and see which attached drives are green but I figured there was an object I could filter for withing XO in the search.

Is there a command line option for xo-cli to use a self signed certificate? Or is answer to have the server also listening on http as well?

Running this command

xo-cli --register https://localhost/api/admin@admin.net admin

gives this error

"type": "error",
  "message": "self signed certificate",
  "error": {
  "code": "DEPTH_ZERO_SELF_SIGNED_CERT"

@olivierlambert That sounds like enough to me, maybe @sheridancompute or @tekwendell have a few other thoughts on it.

@olivierlambert I don't think it really has to do anything as advanced as checking any other part of the system. Tools such as the Datto backup uses "Screenshot Backup Verification" https://www.datto.com/technologies/screenshot-verification which shows the VM boots but nothing else to verify any other application status within the VM. I think the tool verification and starting the system with an isolated network interface to avoid IP conflict issues or the sever reaching out to anything would be enough. I like the idea as this being part of a backup job. For example I currently have delta jobs running during the week and I have a separate full back job that does the offline snapshot that runs on the weekends and that is where I would also like the "Restore Test" type of feature.

@olivierlambert

That would be a simple method. Boot the system with a host only / isolated network, confirm tools start, append that to the backup log, done.

@fred974 said in S3 Remote:

HAproxy in pfSense

I was hoping to avoid this method as my pfsense is not connected at 10G. Thank you for the clarification.

@fred974 I am not really clear on how you solved it. I am still getting "Error calling AWS.S3.putObject: self signed certificate" using xo-server 5.74.1 / xo-web 5.77.0. and the S3 target system is TrueNAS 12.0 U2