@ph7 This update only covers the security issue described above. Fix for the stats issue will roll out later.

@jivanpal We do not currently have any plans to support elliptic curve keys - this is a very sensitive topic given different governmental security requirements around the world.

Note that Let's Encrypt recommends a dual setup for this exact reason: "Our recommendation is to serve a dual-cert config, offering an RSA certificate by default, and a (much smaller) ECDSA certificate to those clients that indicate support." (https://letsencrypt.org/docs/integration-guide/)

@olivierlambert The only discussion about USB3 that I'm aware of is about making USB3 passthrough faster (https://github.com/xapi-project/xen-api/issues/6389). Not being able to see them in the guest OS at all would indicate a different issue...

stormi created this issue in xapi-project/xen-api

open USB 3 for vUSB #6389

@Greg_E Thanks, but that will not be necessary - I think I've figured out where the problem lies now. Good luck with the move

@olivierlambert @hitechhillbilly

I don't see any indications that it's capped at 16... it's at least 16 with a VGPU, but otherwise the value is just passed through to QEMU (which might do some capping of its own ...)

You should be able to see how much your VM booted with:

xe vm-param-list uuid=VM_UUID | grep video_mib

And set it to a bigger value with:

xe vm-param-set uuid=VM_UUID platform:videoram=32

@olivierlambert I think this has already been fixed upstream (https://github.com/xapi-project/xen-api/pull/6458) - I will backport it for the release after the LTS and see if it fixes the issue for people in this thread.

BengangY opened this pull request in xapi-project/xen-api

closed CA-409482: Using computed delay for RRD loop #6458

Could you please provide the output of (for the master and the candidate host):

xe host-list params=software-version

It does seem odd....

I've been able to reproduce this on 8.3 without XO involved (just by querying the /host_logs_download endpoint).

The only error I can see in xensource.log is at the end of a successful download call:

xapi: [debug||6246 HTTPS 10.1.0.100->:::80|get_host_logs_download|xapi_http] Leaving RBAC-handler in xapi_http after: HANDLE_INVALID: [ task; OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d ]
xapi: [error||6246 :::80||backtrace] get_host_logs_download failed with exception Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d")
xapi: [error||6246 :::80||backtrace] Raised Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d")
xapi: [error||6246 :::80||backtrace] 1/19 xapi Raised at file ocaml/database/db_rpc_client_v1.ml, line 33
xapi: [error||6246 :::80||backtrace] 2/19 xapi Called from file ocaml/xapi/db_actions.ml, line 2577
xapi: [error||6246 :::80||backtrace] 3/19 xapi Called from file ocaml/xapi/taskHelper.ml, line 272
xapi: [error||6246 :::80||backtrace] 4/19 xapi Called from file ocaml/xapi/context.ml, line 515
 xapi: [error||6246 :::80||backtrace] 5/19 xapi Called from file ocaml/xapi/context.ml, line 522
xapi: [error||6246 :::80||backtrace] 6/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6246 :::80||backtrace] 7/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6246 :::80||backtrace] 8/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 108
xapi: [error||6246 :::80||backtrace] 9/19 xapi Called from file ocaml/xapi/server_helpers.ml, line 69
xapi: [error||6246 :::80||backtrace] 10/19 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
xapi: [error||6246 :::80||backtrace] 11/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6246 :::80||backtrace] 12/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6246 :::80||backtrace] 13/19 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6246 :::80||backtrace] 14/19 xapi Called from file ocaml/libs/log/debug.ml, line 267
xapi: [error||6246 :::80||backtrace] 15/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6246 :::80||backtrace] 16/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6246 :::80||backtrace] 17/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 364
xapi: [error||6246 :::80||backtrace] 18/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 370
xapi: [error||6246 :::80||backtrace] 19/19 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6246 :::80||backtrace]
xapi: [error||6246 :::80||http_internal_errors] Responding with 500 Internal Error due to Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d")
xapi: [debug||6246 :::80||http_internal_errors] Raised at Debug.with_thread_associated in file \\"ocaml/libs/log/debug.ml\\", line 267, characters 6-15\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 509, characters 4-32\\nCalled from Http_svr.let@ in file \\"ocaml/libs/http-lib/http_svr.ml\\" (inlined), line 46, characters 19-22\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 508, characters 4-195\\n
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, lineApr  3 15:09:04 xcp-gtn-ip12 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 5/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 6/6 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace]
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] handler:http/get_host_logs_download D:4e4d887163d8 failed with exception Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] Raised Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 1/13 xapi Raised at file ocaml/libs/log/debug.ml, line 267
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 2/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 3/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 4/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 5/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 6/13 xapi Called from file ocaml/xapi/xapi_http.ml, line 257
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 7/13 xapi Called from file ocaml/xapi/rbac.ml, line 188
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 8/13 xapi Called from file ocaml/xapi/rbac.ml, line 197
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 9/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 69
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 10/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 11/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 12/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 13/13 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 5/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 6/6 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace]
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] handler:http/get_host_logs_download D:4e4d887163d8 failed with exception Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] Raised Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 1/13 xapi Raised at file ocaml/libs/log/debug.ml, line 267
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 2/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 3/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 4/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 5/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 6/13 xapi Called from file ocaml/xapi/xapi_http.ml, line 257
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 7/13 xapi Called from file ocaml/xapi/rbac.ml, line 188
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 8/13 xapi Called from file ocaml/xapi/rbac.ml, line 197
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 9/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 69
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 10/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 11/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 12/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 13/13 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace]
xapi: [ info||6372 :::80|xapi_http_session_check D:7cc5eb59b433|taskhelper] task Xapi#getResource /host_logs_download R:83146129fe1c forwarded
xapi: [debug||6372 :::80|xapi_http_session_check D:7cc5eb59b433|taskhelper] the status of R:83146129fe1c is failure; cannot set it to `failure
xapi: [debug||6372 HTTPS 10.1.0.100->:::80|get_host_logs_download|xapi_http] Leaving RBAC-handler in xapi_http after: INTERNAL_ERROR: [ Subprocess exited with unexpected code 1 ]
xapi: [error||6372 :::80||backtrace] get_host_logs_download failed with exception Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80||backtrace] Raised Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80||backtrace] 1/14 xapi Raised at file ocaml/libs/log/debug.ml, line 267
xapi: [error||6372 :::80||backtrace] 2/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80||backtrace] 3/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 114
xapi: [error||6372 :::80||backtrace] 4/14 xapi Called from file ocaml/xapi/server_helpers.ml, line 69
xapi: [error||6372 :::80||backtrace] 5/14 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
xapi: [error||6372 :::80||backtrace] 6/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80||backtrace] 7/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80||backtrace] 8/14 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80||backtrace] 9/14 xapi Called from file ocaml/libs/log/debug.ml, line 267
xapi: [error||6372 :::80||backtrace] 10/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80||backtrace] 11/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80||backtrace] 12/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 364
xapi: [error||6372 :::80||backtrace] 13/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 370
xapi: [error||6372 :::80||backtrace] 14/14 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80||backtrace]
xapi: [error||6372 :::80||http_internal_errors] Responding with 500 Internal Error due to Forkhelpers.Subprocess_failed(1)
xapi: [debug||6372 :::80||http_internal_errors] Raised at Debug.with_thread_associated in file \\"ocaml/libs/log/debug.ml\\", line 267, characters 6-15\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 509, characters 4-32\\nCalled from Http_svr.let@ in file \\"ocaml/libs/http-lib/http_svr.ml\\" (inlined), line 46, characters 19-22\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 508, characters 4-195\\n

And then the subsequent download stalls without any indication. I will investigate this further but just to confirm, can any of you see these or similar errors in your logs? Do you have occasional successful downloads followed by failures?

@jivanpal We do not currently have any plans to support elliptic curve keys - this is a very sensitive topic given different governmental security requirements around the world.

Note that Let's Encrypt recommends a dual setup for this exact reason: "Our recommendation is to serve a dual-cert config, offering an RSA certificate by default, and a (much smaller) ECDSA certificate to those clients that indicate support." (https://letsencrypt.org/docs/integration-guide/)

@TITUS-MAXIMUS You are correct. Sorry, the command to run is /opt/xensource/libexec/xen-cmdline --get-dom0 xen-pciback.hide - does this return anything? what's the return code of the command?

@TITUS-MAXIMUS --get-dom0 being empty means no PCI devices were hidden from dom0 either, did you follow this step of the guide? https://docs.xcp-ng.org/compute/#2-tell-xcp-ng-not-to-use-this-device-id-for-dom0

@TITUS-MAXIMUS Could you please attach /var/log/xensource.log from the time of the error? Would be very useful to have a backtrace from where the error occurs

@olivierlambert @hitechhillbilly

I don't see any indications that it's capped at 16... it's at least 16 with a VGPU, but otherwise the value is just passed through to QEMU (which might do some capping of its own ...)

You should be able to see how much your VM booted with:

xe vm-param-list uuid=VM_UUID | grep video_mib

And set it to a bigger value with:

xe vm-param-set uuid=VM_UUID platform:videoram=32

@olivierlambert I think this has already been fixed upstream (https://github.com/xapi-project/xen-api/pull/6458) - I will backport it for the release after the LTS and see if it fixes the issue for people in this thread.

BengangY opened this pull request in xapi-project/xen-api

closed CA-409482: Using computed delay for RRD loop #6458

@olivierlambert The only discussion about USB3 that I'm aware of is about making USB3 passthrough faster (https://github.com/xapi-project/xen-api/issues/6389). Not being able to see them in the guest OS at all would indicate a different issue...

stormi created this issue in xapi-project/xen-api

open USB 3 for vUSB #6389

@ph7 This update only covers the security issue described above. Fix for the stats issue will roll out later.

@Greg_E Thanks, but that will not be necessary - I think I've figured out where the problem lies now. Good luck with the move

@bufanda Thanks!
Hmm, can't see anything suspicious either in your logs or @abudef's

Since you said it reproduces if you reboot the host again, I'd really appreciate if you could send the result of rrd2csv running for 30 seconds or so on a system that doesn't have the stats working - so before any toolstack restarts.