XO task watcher issue/CR broken

[julien-f]

My bad, there is no remote with CR

I have no idea what's going though…

[Andrew]

@julien-f I was just going to say it's another host local storage, not a remote. Desination host is not in the same pool but is on the same 10GB LAN. All hosts are 8.2.1.

[julien-f]

@Andrew I'm unable to reproduce on my end

If you can reproduce with an official XOA, open a support tunnel and I'll investigate further.

[Andrew]

@julien-f I loaded XOA (5.109.0) but it's not new enough to include to problematic code that cause problems in XO source.

[julien-f]

@Andrew If you open a support tunnel, I can deploy a sources XO in your appliance.

[Andrew]

@julien-f ok.

[julien-f]

@Andrew Please test the branch cr-issue and tell me if that helps.

[Andrew]

@julien-f cr-issue branch commit 27d81 resolved this new CR problem for me. Thanks!

[julien-f]

@Andrew I need to understand why it's working now

Which version of XCP-ng/XenServer are you using as source and as the target of the CR?

[Gheppy]

This commit 27d81 is working for me to.
XCP-ng is 8.2.1, up to date

[julien-f]

@Andrew & @Gheppy: I've just pushed a new commit in the cr-issue branch which adds some debug logs, it adds something like that:

putResource(b1imtu1j0y): taskRef: OpaqueRef:10eb89fd-d31e-495b-955d-672c61a4ea48
putResource(b1imtu1j0y): useHack: false
putResource(b1imtu1j0y): body#end
putResource(b1imtu1j0y): request#unpipe
putResource(b1imtu1j0y): request#prefinish
putResource(b1imtu1j0y): request#finish
putResource(b1imtu1j0y): response#resume
putResource(b1imtu1j0y): body#close
putResource(b1imtu1j0y): response#readable
putResource(b1imtu1j0y): response#end
putResource(b1imtu1j0y): response#close
putResource(b1imtu1j0y): request#close

If you could try it and show me the output, that would help me understand what's going on

[olivierlambert]

Eager to read the output, it's weird we can't reproduce here, there's something

[Gheppy]

@julien-f
with command -- journalctl -u xo-server -f -n 70

Feb 10 13:33:31 tb-xoce-e-19 xo-server[32712]: putResource(ty99546973n): body#drain
Feb 10 13:33:31 tb-xoce-e-19 xo-server[32712]: putResource(vcybyae8pn): body#drain
Feb 10 13:33:31 tb-xoce-e-19 xo-server[32712]: putResource(ty99546973n): body#drain
Feb 10 13:33:31 tb-xoce-e-19 xo-server[32712]: putResource(vcybyae8pn): body#drain
Feb 10 13:33:31 tb-xoce-e-19 xo-server[32712]: putResource(ty99546973n): body#drain
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(ty99546973n): response#readable
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(ty99546973n): request#close
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(ty99546973n): request#unpipe
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(ty99546973n): body#error
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(ty99546973n): body#unpipe
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(ty99546973n): body#close
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(ty99546973n): response#resume
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(ty99546973n): response#end
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(ty99546973n): response#close
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(vcybyae8pn): response#readable
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(vcybyae8pn): request#close
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(vcybyae8pn): request#unpipe
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(vcybyae8pn): body#error
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(vcybyae8pn): body#unpipe
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(vcybyae8pn): body#close
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(vcybyae8pn): response#resume
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(vcybyae8pn): response#end
Feb 10 13:33:32 tb-xoce-e-19 xo-server[32712]: putResource(vcybyae8pn): response#close
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]: [WARN] plugin transport-email: Error: No recipients defined
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at SMTPConnection._formatError (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:790:19)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at SMTPConnection._setEnvelope (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:1007:34)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at SMTPConnection.send (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:621:14)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at sendMessage (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-transport/index.js:228:28)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at /opt/xen-orchestra/node_modules/nodemailer/lib/smtp-transport/index.js:289:21
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at SMTPConnection.<anonymous> (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:213:17)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at Object.onceWrapper (node:events:627:28)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at SMTPConnection.emit (node:events:513:28)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at SMTPConnection.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at SMTPConnection._actionEHLO (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:1331:14)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at SMTPConnection._processResponse (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:953:20)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at SMTPConnection._onData (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:755:14)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at Socket.SMTPConnection._onSocketData (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:193:44)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at Socket.emit (node:events:513:28)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at Socket.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at addChunk (node:internal/streams/readable:324:12)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at readableAddChunk (node:internal/streams/readable:297:9)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at Socket.Readable.push (node:internal/streams/readable:234:10)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]: 2023-02-10T11:34:05.207Z xo:xo-server WARN possibly unhandled rejection {
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:   error: Error: No recipients defined
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at SMTPConnection._formatError (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:790:19)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at SMTPConnection._setEnvelope (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:1007:34)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at SMTPConnection.send (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:621:14)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at sendMessage (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-transport/index.js:228:28)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at /opt/xen-orchestra/node_modules/nodemailer/lib/smtp-transport/index.js:289:21
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at SMTPConnection.<anonymous> (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:213:17)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at Object.onceWrapper (node:events:627:28)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at SMTPConnection.emit (node:events:513:28)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at SMTPConnection.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at SMTPConnection._actionEHLO (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:1331:14)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at SMTPConnection._processResponse (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:953:20)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at SMTPConnection._onData (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:755:14)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at Socket.SMTPConnection._onSocketData (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:193:44)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at Socket.emit (node:events:513:28)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at Socket.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at addChunk (node:internal/streams/readable:324:12)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at readableAddChunk (node:internal/streams/readable:297:9)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at Socket.Readable.push (node:internal/streams/readable:234:10)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:       at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     code: 'EENVELOPE',
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:     command: 'API'
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]:   }
Feb 10 13:34:05 tb-xoce-e-19 xo-server[32617]: }

On backup log I have this

{
  "data": {
    "mode": "delta",
    "reportWhen": "always"
  },
  "id": "1676028066913",
  "jobId": "dca46bf4-dca6-4e07-8dab-dc644d6f47c1",
  "jobName": "productie-Sincronizare-Activa-NU-STERGE",
  "message": "backup",
  "scheduleId": "118e0756-3588-4d70-97d8-f6705bc3785f",
  "start": 1676028066913,
  "status": "success",
  "infos": [
    {
      "data": {
        "vms": [
          "629bdfeb-7700-561c-74ac-e151068721c2"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "629bdfeb-7700-561c-74ac-e151068721c2"
      },
      "id": "1676028068026:0",
      "message": "backup VM",
      "start": 1676028068026,
      "status": "success",
      "tasks": [
        {
          "id": "1676028068072",
          "message": "snapshot",
          "start": 1676028068072,
          "status": "success",
          "end": 1676028071566,
          "result": "6f52cc17-328e-589a-82a8-c217ac34d6f1"
        },
        {
          "data": {
            "id": "5148208b-acf6-13f3-4f36-5817e3eb6ab1",
            "isFull": false,
            "type": "SR"
          },
          "id": "1676028071568",
          "message": "export",
          "start": 1676028071568,
          "status": "success",
          "tasks": [
            {
              "id": "1676028071596",
              "message": "transfer",
              "start": 1676028071596,
              "status": "success",
              "end": 1676028094729,
              "result": {
                "size": 162336256
              }
            }
          ],
          "end": 1676028107179
        },
        {
          "data": {
            "id": "7a4829c0-e59a-726c-34bc-1a1052b0334f",
            "isFull": false,
            "type": "SR"
          },
          "id": "1676028071567",
          "message": "export",
          "start": 1676028071567,
          "status": "success",
          "tasks": [
            {
              "id": "1676028071592",
              "message": "transfer",
              "start": 1676028071592,
              "status": "success",
              "end": 1676028094954,
              "result": {
                "size": 162336256
              }
            }
          ],
          "end": 1676028110020
        }
      ],
      "end": 1676028129516
    }
  ],
  "end": 1676028129517
}

[julien-f]

@Gheppy Could you please include all the logs up to the taskRef entry?

[Andrew]

@julien-f Hosts are XCP 8.2.1 and updated.

Here's the logs... It did suppress 373874 messages... only 417267 left...

rename output.txt as output.txt.gz and uncompress.

output.txt

[julien-f]

@Andrew Thank you, I just pushed a new commit in which I suppressed the drain event

[julien-f]

Once filtered, here's what I see:

putResource(hhprvsf5i): taskRef: OpaqueRef:bb9bbc77-8c88-4621-82a7-745197306c3b
putResource(hhprvsf5i): useHack: false
putResource(hhprvsf5i): response#readable
putResource(hhprvsf5i): request#close
putResource(hhprvsf5i): request#unpipe
putResource(hhprvsf5i): body#error
putResource(hhprvsf5i): body#unpipe
putResource(hhprvsf5i): body#close
putResource(hhprvsf5i): response#resume
putResource(hhprvsf5i): response#end
putResource(hhprvsf5i): response#close

I don't understand why there is not a request#finish event like in my test

[julien-f]

It's probably related to the body#error event.

I will add more debug and will keep you posted when the branch is ready.

[Andrew]

@julien-f I'll do another run after the next debug update.

[Gheppy]

@julien-f
Logs with commit dd1edd2519c593b06372ddec9041bd525f40ba05

Feb 10 17:58:17 tb-xoce-e-19 systemd[1]: Stopping XO Server...
Feb 10 17:58:17 tb-xoce-e-19 xo-server[9598]: 2023-02-10T15:58:17.143Z xo:main INFO SIGTERM caught, closing…
Feb 10 17:58:17 tb-xoce-e-19 xo-server[9598]: 2023-02-10T15:58:17.181Z xo:main INFO bye :-)
Feb 10 17:58:17 tb-xoce-e-19 systemd[1]: xo-server.service: Deactivated successfully.
Feb 10 17:58:17 tb-xoce-e-19 systemd[1]: Stopped XO Server.
Feb 10 17:58:17 tb-xoce-e-19 systemd[1]: xo-server.service: Consumed 19.958s CPU time.
-- Boot 077eda8b3bde4af6b29a23add0da503a --
Feb 10 17:58:41 tb-xoce-e-19 systemd[1]: Started XO Server.
Feb 10 17:59:02 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:02.632Z xo:main INFO Configuration loaded.
Feb 10 17:59:02 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:02.711Z xo:main INFO Web server listening on https://0.0.0.0:443
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:03.263Z xo:mixins:hooks WARN start failure {
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:   error: Error: Command failed with exit code 5: vgchange -an
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:     Logical volume ubuntu-vg/ubuntu-lv contains a filesystem in use.
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:     Can't deactivate volume group "ubuntu-vg" with 1 open logical volume(s)
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:       at makeError (file:///opt/xen-orchestra/packages/xo-server/node_modules/execa/lib/error.js:59:11)
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:       at handlePromise (file:///opt/xen-orchestra/packages/xo-server/node_modules/execa/index.js:119:26) {
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:     shortMessage: 'Command failed with exit code 5: vgchange -an',
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:     command: 'vgchange -an',
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:     escapedCommand: 'vgchange -an',
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:     exitCode: 5,
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:     signal: undefined,
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:     signalDescription: undefined,
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:     stdout: '',
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:     stderr: '  Logical volume ubuntu-vg/ubuntu-lv contains a filesystem in use.\n' +
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:       `  Can't deactivate volume group "ubuntu-vg" with 1 open logical volume(s)`,
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:     failed: true,
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:     timedOut: false,
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:     isCanceled: false,
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:     killed: false
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]:   }
Feb 10 17:59:03 tb-xoce-e-19 xo-server[837]: }
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.300Z xo:main INFO Setting up / → /opt/xen-orchestra/packages/xo-web/dist
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.315Z xo:plugin INFO register audit
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.331Z xo:plugin INFO register auth-github
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.332Z xo:plugin INFO register auth-google
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.341Z xo:plugin INFO register auth-ldap
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.343Z xo:plugin INFO register auth-saml
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.355Z xo:plugin INFO register backup-reports
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.367Z xo:plugin INFO register load-balancer
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.368Z xo:plugin INFO register netbox
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.370Z xo:plugin INFO register perf-alert
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.379Z xo:plugin INFO register sdn-controller
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.380Z xo:plugin INFO register test-plugin
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.381Z xo:plugin INFO register transport-email
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.395Z xo:plugin INFO register transport-icinga2
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.401Z xo:plugin INFO register transport-nagios
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.403Z xo:plugin INFO register transport-slack
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.408Z xo:plugin INFO register transport-xmpp
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.413Z xo:plugin INFO register usage-report
Feb 10 17:59:05 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:05.414Z xo:plugin INFO register web-hooks
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.945Z xo:plugin INFO successfully register auth-github
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.945Z xo:plugin INFO successfully register auth-google
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.945Z xo:plugin INFO successfully register auth-ldap
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.945Z xo:plugin INFO successfully register auth-saml
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.945Z xo:plugin INFO successfully register netbox
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.945Z xo:plugin INFO successfully register test-plugin
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.945Z xo:plugin INFO successfully register transport-icinga2
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.945Z xo:plugin INFO successfully register transport-nagios
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.945Z xo:plugin INFO successfully register transport-slack
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.945Z xo:plugin INFO successfully register transport-xmpp
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.945Z xo:plugin INFO successfully register web-hooks
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.945Z xo:plugin INFO successfully register usage-report
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.945Z xo:plugin INFO successfully register backup-reports
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.945Z xo:plugin INFO successfully register transport-email
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.956Z xo:plugin INFO successfully register audit
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.957Z xo:plugin INFO successfully register load-balancer
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.957Z xo:plugin INFO successfully register perf-alert
Feb 10 17:59:07 tb-xoce-e-19 xo-server[837]: 2023-02-10T15:59:07.966Z xo:plugin INFO successfully register sdn-controller
Feb 10 17:59:37 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): taskRef: OpaqueRef:655531f1-ba0b-4583-95a2-f376bf85f526
Feb 10 17:59:37 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): useHack: false
Feb 10 17:59:37 tb-xoce-e-19 xo-server[1181]: putResource(7x5rgk2ksq4): taskRef: OpaqueRef:1b623504-9dd2-4d4a-96a8-fca88adbb66e
Feb 10 17:59:37 tb-xoce-e-19 xo-server[1181]: putResource(7x5rgk2ksq4): useHack: false
Feb 10 17:59:37 tb-xoce-e-19 xo-server[1181]: putResource(3n9ugoikq5v): taskRef: OpaqueRef:0323510a-df09-4072-8e43-c9a34449c396
Feb 10 17:59:37 tb-xoce-e-19 xo-server[1181]: putResource(3n9ugoikq5v): useHack: false
Feb 10 17:59:37 tb-xoce-e-19 xo-server[1181]: putResource(q3eg2gp4h0m): taskRef: OpaqueRef:72eed850-b1e3-438c-987e-469cd1331845
Feb 10 17:59:37 tb-xoce-e-19 xo-server[1181]: putResource(q3eg2gp4h0m): useHack: false
Feb 10 17:59:37 tb-xoce-e-19 xo-server[1181]: putResource(3n9ugoikq5v): waiting for request to finish
Feb 10 17:59:37 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): waiting for request to finish
Feb 10 17:59:37 tb-xoce-e-19 xo-server[1181]: putResource(q3eg2gp4h0m): waiting for request to finish
Feb 10 17:59:37 tb-xoce-e-19 xo-server[1181]: putResource(7x5rgk2ksq4): waiting for request to finish
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(3n9ugoikq5v): response#readable
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(3n9ugoikq5v): request#close
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(3n9ugoikq5v): request#unpipe
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(3n9ugoikq5v): body#error
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(3n9ugoikq5v): body#unpipe
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(3n9ugoikq5v): body#close
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(3n9ugoikq5v): waiting for response to finish
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(3n9ugoikq5v): response#resume
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(3n9ugoikq5v): response#end
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(3n9ugoikq5v): response#close
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(3n9ugoikq5v): returning task result promise
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): body#prefinish
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): body#end
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): request#unpipe
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): request#prefinish
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): body#finish
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): body#unpipe
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): request#finish
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): body#close
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): waiting for response to finish
Feb 10 17:59:38 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): response#resume
Feb 10 17:59:39 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): response#readable
Feb 10 17:59:39 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): response#end
Feb 10 17:59:39 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): response#close
Feb 10 17:59:39 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): returning task result promise
Feb 10 17:59:39 tb-xoce-e-19 xo-server[1181]: putResource(rhnuq5h5ix): request#close
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(q3eg2gp4h0m): response#readable
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(q3eg2gp4h0m): request#close
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(q3eg2gp4h0m): request#unpipe
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(q3eg2gp4h0m): body#error
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(q3eg2gp4h0m): body#unpipe
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(q3eg2gp4h0m): body#close
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(q3eg2gp4h0m): waiting for response to finish
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(q3eg2gp4h0m): response#resume
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(q3eg2gp4h0m): response#end
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(q3eg2gp4h0m): response#close
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(q3eg2gp4h0m): returning task result promise
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(7x5rgk2ksq4): response#readable
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(7x5rgk2ksq4): request#close
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(7x5rgk2ksq4): request#unpipe
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(7x5rgk2ksq4): body#error
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(7x5rgk2ksq4): body#unpipe
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(7x5rgk2ksq4): body#close
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(7x5rgk2ksq4): waiting for response to finish
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(7x5rgk2ksq4): response#resume
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(7x5rgk2ksq4): response#end
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(7x5rgk2ksq4): response#close
Feb 10 17:59:47 tb-xoce-e-19 xo-server[1181]: putResource(7x5rgk2ksq4): returning task result promise
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]: [WARN] plugin transport-email: Error: No recipients defined
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at SMTPConnection._formatError (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:790:19)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at SMTPConnection._setEnvelope (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:1007:34)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at SMTPConnection.send (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:621:14)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at sendMessage (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-transport/index.js:228:28)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at /opt/xen-orchestra/node_modules/nodemailer/lib/smtp-transport/index.js:289:21
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at SMTPConnection.<anonymous> (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:213:17)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at Object.onceWrapper (node:events:627:28)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at SMTPConnection.emit (node:events:513:28)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at SMTPConnection.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at SMTPConnection._actionEHLO (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:1331:14)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at SMTPConnection._processResponse (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:953:20)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at SMTPConnection._onData (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:755:14)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at Socket.SMTPConnection._onSocketData (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:193:44)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at Socket.emit (node:events:513:28)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at Socket.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at addChunk (node:internal/streams/readable:324:12)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at readableAddChunk (node:internal/streams/readable:297:9)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at Socket.Readable.push (node:internal/streams/readable:234:10)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]: 2023-02-10T16:00:23.194Z xo:xo-server WARN possibly unhandled rejection {
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:   error: Error: No recipients defined
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at SMTPConnection._formatError (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:790:19)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at SMTPConnection._setEnvelope (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:1007:34)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at SMTPConnection.send (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:621:14)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at sendMessage (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-transport/index.js:228:28)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at /opt/xen-orchestra/node_modules/nodemailer/lib/smtp-transport/index.js:289:21
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at SMTPConnection.<anonymous> (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:213:17)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at Object.onceWrapper (node:events:627:28)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at SMTPConnection.emit (node:events:513:28)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at SMTPConnection.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at SMTPConnection._actionEHLO (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:1331:14)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at SMTPConnection._processResponse (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:953:20)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at SMTPConnection._onData (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:755:14)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at Socket.SMTPConnection._onSocketData (/opt/xen-orchestra/node_modules/nodemailer/lib/smtp-connection/index.js:193:44)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at Socket.emit (node:events:513:28)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at Socket.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at addChunk (node:internal/streams/readable:324:12)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at readableAddChunk (node:internal/streams/readable:297:9)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at Socket.Readable.push (node:internal/streams/readable:234:10)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at TCP.onStreamRead (node:internal/stream_base_commons:190:23)
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:       at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     code: 'EENVELOPE',
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:     command: 'API'
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]:   }
Feb 10 18:00:23 tb-xoce-e-19 xo-server[837]: }
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(uljxc697y2l): taskRef: OpaqueRef:f3ef509e-4121-437b-b140-282419c869ba
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(uljxc697y2l): useHack: false
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(uljxc697y2l): waiting for request to finish
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(7dnih4mm1qf): taskRef: OpaqueRef:0b25fc40-83f1-44b3-86cf-59f00bbbc272
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(7dnih4mm1qf): useHack: false
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(n0ygkko6hs): taskRef: OpaqueRef:099a9a90-719d-4b5e-839a-7bdacfccf50d
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(n0ygkko6hs): useHack: false
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(n0ygkko6hs): waiting for request to finish
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(7dnih4mm1qf): waiting for request to finish
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(oabtodmaeda): taskRef: OpaqueRef:028adad2-d651-418d-b3bb-ca767daf3801
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(oabtodmaeda): useHack: false
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(30gsda5k9w): taskRef: OpaqueRef:fd44c094-7dda-4a96-8ff4-2c76e4d75c80
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(30gsda5k9w): useHack: false
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(oabtodmaeda): waiting for request to finish
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(30gsda5k9w): waiting for request to finish
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(v44nlq1wrs8): taskRef: OpaqueRef:4fe09fe7-63d7-4318-aa94-65362ce00cbd
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(v44nlq1wrs8): useHack: false
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(v44nlq1wrs8): waiting for request to finish
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(ytgd25jz6wi): taskRef: OpaqueRef:2b216564-d923-46b0-a377-08806e4d28a1
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(ytgd25jz6wi): useHack: false
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): taskRef: OpaqueRef:9ad749aa-baed-4ce1-8ded-27ee463bbd87
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): useHack: false
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): waiting for request to finish
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(ytgd25jz6wi): waiting for request to finish
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(zwad8yuqlz): taskRef: OpaqueRef:40ac30f8-240b-4278-93a1-5cb6b3d2e681
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(zwad8yuqlz): useHack: false
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(e9in7oo6sxp): taskRef: OpaqueRef:948bec59-35a8-41fb-849e-3c3c33214fc2
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(e9in7oo6sxp): useHack: false
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(zwad8yuqlz): waiting for request to finish
Feb 10 18:00:30 tb-xoce-e-19 xo-server[1209]: putResource(e9in7oo6sxp): waiting for request to finish
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(ytgd25jz6wi): response#readable
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(ytgd25jz6wi): request#close
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(ytgd25jz6wi): request#unpipe
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(ytgd25jz6wi): body#error
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(ytgd25jz6wi): body#unpipe
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(ytgd25jz6wi): body#close
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(ytgd25jz6wi): waiting for response to finish
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(ytgd25jz6wi): response#resume
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(ytgd25jz6wi): response#end
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(ytgd25jz6wi): response#close
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(ytgd25jz6wi): returning task result promise
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): body#prefinish
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): body#end
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): request#unpipe
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): request#prefinish
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): body#finish
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): body#unpipe
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): request#finish
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): body#close
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): waiting for response to finish
Feb 10 18:00:32 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): response#resume
Feb 10 18:00:33 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): response#readable
Feb 10 18:00:33 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): response#end
Feb 10 18:00:33 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): response#close
Feb 10 18:00:33 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): returning task result promise
Feb 10 18:00:33 tb-xoce-e-19 xo-server[1209]: putResource(eqjv69wmqff): request#close
Feb 10 18:07:19 tb-xoce-e-19 xo-server[1209]: putResource(n0ygkko6hs): response#readable
Feb 10 18:07:19 tb-xoce-e-19 xo-server[1209]: putResource(n0ygkko6hs): request#close
Feb 10 18:07:19 tb-xoce-e-19 xo-server[1209]: putResource(n0ygkko6hs): request#unpipe
Feb 10 18:07:19 tb-xoce-e-19 xo-server[1209]: putResource(n0ygkko6hs): body#error
Feb 10 18:07:19 tb-xoce-e-19 xo-server[1209]: putResource(n0ygkko6hs): body#unpipe
Feb 10 18:07:19 tb-xoce-e-19 xo-server[1209]: putResource(n0ygkko6hs): body#close
Feb 10 18:07:19 tb-xoce-e-19 xo-server[1209]: putResource(n0ygkko6hs): waiting for response to finish
Feb 10 18:07:19 tb-xoce-e-19 xo-server[1209]: putResource(n0ygkko6hs): response#resume
Feb 10 18:07:19 tb-xoce-e-19 xo-server[1209]: putResource(n0ygkko6hs): response#end
Feb 10 18:07:19 tb-xoce-e-19 xo-server[1209]: putResource(n0ygkko6hs): response#close
Feb 10 18:07:19 tb-xoce-e-19 xo-server[1209]: putResource(n0ygkko6hs): returning task result promise
Feb 10 18:07:20 tb-xoce-e-19 xo-server[1209]: putResource(7dnih4mm1qf): response#readable
Feb 10 18:07:20 tb-xoce-e-19 xo-server[1209]: putResource(7dnih4mm1qf): request#close
Feb 10 18:07:20 tb-xoce-e-19 xo-server[1209]: putResource(7dnih4mm1qf): request#unpipe
Feb 10 18:07:20 tb-xoce-e-19 xo-server[1209]: putResource(7dnih4mm1qf): body#error
Feb 10 18:07:20 tb-xoce-e-19 xo-server[1209]: putResource(7dnih4mm1qf): body#unpipe
Feb 10 18:07:20 tb-xoce-e-19 xo-server[1209]: putResource(7dnih4mm1qf): body#close
Feb 10 18:07:20 tb-xoce-e-19 xo-server[1209]: putResource(7dnih4mm1qf): waiting for response to finish
Feb 10 18:07:20 tb-xoce-e-19 xo-server[1209]: putResource(7dnih4mm1qf): response#resume
Feb 10 18:07:20 tb-xoce-e-19 xo-server[1209]: putResource(7dnih4mm1qf): response#end
Feb 10 18:07:20 tb-xoce-e-19 xo-server[1209]: putResource(7dnih4mm1qf): response#close
Feb 10 18:07:20 tb-xoce-e-19 xo-server[1209]: putResource(7dnih4mm1qf): returning task result promise
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(v44nlq1wrs8): response#readable
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(v44nlq1wrs8): request#close
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(v44nlq1wrs8): request#unpipe
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(v44nlq1wrs8): body#error
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(v44nlq1wrs8): body#unpipe
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(v44nlq1wrs8): body#close
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(v44nlq1wrs8): waiting for response to finish
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(v44nlq1wrs8): response#resume
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(v44nlq1wrs8): response#end
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(v44nlq1wrs8): response#close
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(v44nlq1wrs8): returning task result promise
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(uljxc697y2l): response#readable
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(uljxc697y2l): request#close
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(uljxc697y2l): request#unpipe
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(uljxc697y2l): body#error
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(uljxc697y2l): body#unpipe
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(uljxc697y2l): body#close
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(uljxc697y2l): waiting for response to finish
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(uljxc697y2l): response#resume
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(uljxc697y2l): response#end
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(uljxc697y2l): response#close
Feb 10 18:09:05 tb-xoce-e-19 xo-server[1209]: putResource(uljxc697y2l): returning task result promise
Feb 10 18:28:03 tb-xoce-e-19 xo-server[1209]: putResource(zwad8yuqlz): response#readable
Feb 10 18:28:03 tb-xoce-e-19 xo-server[1209]: putResource(zwad8yuqlz): request#close
Feb 10 18:28:03 tb-xoce-e-19 xo-server[1209]: putResource(zwad8yuqlz): request#unpipe
Feb 10 18:28:03 tb-xoce-e-19 xo-server[1209]: putResource(zwad8yuqlz): body#error
Feb 10 18:28:03 tb-xoce-e-19 xo-server[1209]: putResource(zwad8yuqlz): body#unpipe
Feb 10 18:28:03 tb-xoce-e-19 xo-server[1209]: putResource(zwad8yuqlz): body#close
Feb 10 18:28:03 tb-xoce-e-19 xo-server[1209]: putResource(zwad8yuqlz): waiting for response to finish
Feb 10 18:28:03 tb-xoce-e-19 xo-server[1209]: putResource(zwad8yuqlz): response#resume
Feb 10 18:28:03 tb-xoce-e-19 xo-server[1209]: putResource(zwad8yuqlz): response#end
Feb 10 18:28:03 tb-xoce-e-19 xo-server[1209]: putResource(zwad8yuqlz): response#close
Feb 10 18:28:03 tb-xoce-e-19 xo-server[1209]: putResource(zwad8yuqlz): returning task result promise
Feb 10 18:28:04 tb-xoce-e-19 xo-server[1209]: putResource(e9in7oo6sxp): response#readable
Feb 10 18:28:04 tb-xoce-e-19 xo-server[1209]: putResource(e9in7oo6sxp): request#close
Feb 10 18:28:04 tb-xoce-e-19 xo-server[1209]: putResource(e9in7oo6sxp): request#unpipe
Feb 10 18:28:04 tb-xoce-e-19 xo-server[1209]: putResource(e9in7oo6sxp): body#error
Feb 10 18:28:04 tb-xoce-e-19 xo-server[1209]: putResource(e9in7oo6sxp): body#unpipe
Feb 10 18:28:04 tb-xoce-e-19 xo-server[1209]: putResource(e9in7oo6sxp): body#close
Feb 10 18:28:04 tb-xoce-e-19 xo-server[1209]: putResource(e9in7oo6sxp): waiting for response to finish
Feb 10 18:28:04 tb-xoce-e-19 xo-server[1209]: putResource(e9in7oo6sxp): response#resume
Feb 10 18:28:04 tb-xoce-e-19 xo-server[1209]: putResource(e9in7oo6sxp): response#end
Feb 10 18:28:04 tb-xoce-e-19 xo-server[1209]: putResource(e9in7oo6sxp): response#close
Feb 10 18:28:04 tb-xoce-e-19 xo-server[1209]: putResource(e9in7oo6sxp): returning task result promise
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(30gsda5k9w): response#readable
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(30gsda5k9w): request#close
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(30gsda5k9w): request#unpipe
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(30gsda5k9w): body#error
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(30gsda5k9w): body#unpipe
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(30gsda5k9w): body#close
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(30gsda5k9w): waiting for response to finish
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(30gsda5k9w): response#resume
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(30gsda5k9w): response#end
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(30gsda5k9w): response#close
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(30gsda5k9w): returning task result promise
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(oabtodmaeda): response#readable
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(oabtodmaeda): request#close
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(oabtodmaeda): request#unpipe
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(oabtodmaeda): body#error
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(oabtodmaeda): body#unpipe
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(oabtodmaeda): body#close
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(oabtodmaeda): waiting for response to finish
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(oabtodmaeda): response#resume
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(oabtodmaeda): response#end
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(oabtodmaeda): response#close
Feb 10 18:28:55 tb-xoce-e-19 xo-server[1209]: putResource(oabtodmaeda): returning task result promise