@AtaxyaNetwork Yes, this worked. Thanks.

@AtaxyaNetwork Yes, I do.

I've added two new disks to the server and created md0 raid1 array, as described here: https://xcp-ng.org/docs/guides.html#software-raid-storage-repository. It works. Now I have to move VMs from the main, single disk of the server. But when I click migrate arrow on this server's vm, it does not show the same server as the destination, only other servers.

Yes, I can migrate VMs to another server and back, selecting new raid storage. But it's kinda awkward to haul hefty data between local disks via external machine...

@olivierlambert No, it does not. Still, sometimes badge with patches blink... funny.

@olivierlambert Well, after erasing that file it appears again with the same "check_update" contents.

@burbilog

It seems to return

{"error": "The updater plugin is busy (current operation: check_update)"}

all the time on that server.

@olivierlambert said in Orange exclamation mark without errors:

plugin=updater.py fn=check_update

Where can I run this command? On the host with missing patches?

@olivierlambert Anyway, I don't understand why XOA does not detect them at all

@olivierlambert

64 bytes from 172.16.1.46 (172.16.1.46): icmp_seq=1 ttl=62 time=3.88 ms
64 bytes from 172.16.1.46 (172.16.1.46): icmp_seq=2 ttl=62 time=4.40 ms
64 bytes from 172.16.1.46 (172.16.1.46): icmp_seq=3 ttl=62 time=4.67 ms

@olivierlambert "Missing patches"

Funny thing, one of these hw servers is kinda old with 8.2.0 xcp-ng and there ARE missing patches. But no such notifications neither in XOA or XOCE right now.

Whoa. Just noticed. Red badge with number of patches appear in XOCE! But only after a while, if I leave that browser window for a while. But XOA shows no notification at all.

@olivierlambert Yes, I connect all four machines. No complaints in XOA.

@olivierlambert I did not connect XOA to other hw servers. Is it safe to run both xoa and xoce in parallel on all hw servers?

XOCE shows yellow exclamation mark on Pools menu entry, but there are no errors in host list...

Xen Orchestra, commit 07bd9
xo-server 5.93.1
xo-web 5.96.0

@olivierlambert Installed xoa appliance, connected it to that server and it seems to work, xoa creates new VM just fine. So, it's certainly a problem of xoce.

@olivierlambert Ugh.

I erased xoce (rm -rf /opt/xen-orchestra && apt purge redis-server), then reinstalled both xoce and single server afresh. Then I connected brand new, shiny xoce to brand new, shiny server. Only. I did not connect other servers.

And I've got the same problem. Added NFS SR, tried to install VM, pink square, error:

vm.create
{
  "clone": true,
  "existingDisks": {},
  "installation": {
    "method": "cdrom",
    "repository": "6b15b403-a94a-4f59-b4bc-4da75bffe4a7"
  },
  "name_label": "x",
  "template": "b8896401-be53-cbc1-58f3-ac0bd696ec55-4b6425be-728a-4a2c-8333-0aef209a348b",
  "VDIs": [
    {
      "bootable": true,
      "device": "0",
      "size": 10737418240,
      "type": "system",
      "SR": "aeaa681d-bf4a-5070-f553-ffe6debb0b54",
      "name_description": "Created by XO",
      "name_label": "CentOS 8_erofa"
    }
  ],
  "VIFs": [
    {
      "network": "b4fd13cc-7b32-0794-d617-2ecd39139dc1",
      "allowedIpv4Addresses": [],
      "allowedIpv6Addresses": []
    }
  ],
  "CPUs": 1,
  "cpusMax": 1,
  "cpuWeight": null,
  "cpuCap": null,
  "name_description": "x",
  "memory": 4294967296,
  "bootAfterCreate": true,
  "copyHostBiosStrings": false,
  "secureBoot": false,
  "share": false,
  "coreOs": false,
  "tags": [],
  "hvmBootFirmware": "bios"
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "(Failure \"Expected string, got 'N'\")"
  ],
  "call": {
    "method": "VM.get_power_state",
    "params": [
      null
    ]
  },
  "message": "INTERNAL_ERROR((Failure \"Expected string, got 'N'\"))",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR((Failure \"Expected string, got 'N'\"))
    at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)
    at /opt/xen-orchestra/packages/xen-api/src/transports/json-rpc.js:37:27
    at AsyncResource.runInAsyncScope (node:async_hooks:202:9)
    at cb (/opt/xen-orchestra/node_modules/bluebird/js/release/util.js:355:42)
    at tryCatcher (/opt/xen-orchestra/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/opt/xen-orchestra/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/opt/xen-orchestra/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/opt/xen-orchestra/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/opt/xen-orchestra/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (node:internal/timers:466:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}

May 13 16:49:42 xen xapi: [error||1712 HTTPS 10.1.0.188->:::80|VM.add_to_platform D:4a241a51b855|sql] Duplicate key in set or map: table VM; field platform; ref OpaqueRef:0766c0ce-6a4d-4c81-b8ab-9956ed2702f1; key secureboot
May 13 16:49:42 xen xapi: [error||1714 HTTPS 10.1.0.188->:::80|VM.add_to_HVM_boot_params D:0d9aca8b9b24|sql] Duplicate key in set or map: table VM; field HVM__boot_params; ref OpaqueRef:0766c0ce-6a4d-4c81-b8ab-9956ed2702f1; key firmware
May 13 16:49:42 xen xapi: [error||1712 :::80||backtrace] VM.add_to_platform D:4a241a51b855 failed with exception Db_exn.Duplicate_key("VM", "platform", "OpaqueRef:0766c0ce-6a4d-4c81-b8ab-9956ed2702f1", "secureboot")
May 13 16:49:42 xen xapi: [error||1712 :::80||backtrace] Raised Db_exn.Duplicate_key("VM", "platform", "OpaqueRef:0766c0ce-6a4d-4c81-b8ab-9956ed2702f1", "secureboot")
May 13 16:49:42 xen xapi: [error||1712 :::80||backtrace] 1/8 xapi Raised at file ocaml/database/db_cache_impl.ml, line 310
May 13 16:49:42 xen xapi: [error||1712 :::80||backtrace] 2/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
May 13 16:49:42 xen xapi: [error||1712 :::80||backtrace] 3/8 xapi Called from file ocaml/xapi/rbac.ml, line 231
May 13 16:49:42 xen xapi: [debug||1711 :::80||api_effect] VM.set_name_description
May 13 16:49:42 xen xapi: [debug||1715 :::80||api_effect] VM.add_to_platform
May 13 16:49:42 xen xapi: [error||1714 :::80||backtrace] VM.add_to_HVM_boot_params D:0d9aca8b9b24 failed with exception Db_exn.Duplicate_key("VM", "HVM__boot_params", "OpaqueRef:0766c0ce-6a4d-4c81-b8ab-9956ed2702f1", "firmware")
May 13 16:49:42 xen xapi: [error||1712 :::80||backtrace] 4/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 100
May 13 16:49:42 xen xapi: [error||1715 HTTPS 10.1.0.188->:::80|VM.add_to_platform D:e5850b8e2aca|sql] Duplicate key in set or map: table VM; field platform; ref OpaqueRef:0766c0ce-6a4d-4c81-b8ab-9956ed2702f1; key device-model
May 13 16:49:42 xen xapi: [error||1714 :::80||backtrace] Raised Db_exn.Duplicate_key("VM", "HVM__boot_params", "OpaqueRef:0766c0ce-6a4d-4c81-b8ab-9956ed2702f1", "firmware")
May 13 16:49:42 xen xapi: [error||1712 :::80||backtrace] 5/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 121
May 13 16:49:42 xen xapi: [error||1714 :::80||backtrace] 1/8 xapi Raised at file ocaml/database/db_cache_impl.ml, line 310
May 13 16:49:42 xen xapi: [error||1715 :::80||backtrace] VM.add_to_platform D:e5850b8e2aca failed with exception Db_exn.Duplicate_key("VM", "platform", "OpaqueRef:0766c0ce-6a4d-4c81-b8ab-9956ed2702f1", "device-model")
May 13 16:49:42 xen xapi: [error||1712 :::80||backtrace] 6/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
May 13 16:49:42 xen xapi: [error||1714 :::80||backtrace] 2/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
May 13 16:49:42 xen xapi: [error||1715 :::80||backtrace] Raised Db_exn.Duplicate_key("VM", "platform", "OpaqueRef:0766c0ce-6a4d-4c81-b8ab-9956ed2702f1", "device-model")
May 13 16:49:42 xen xapi: [error||1714 :::80||backtrace] 3/8 xapi Called from file ocaml/xapi/rbac.ml, line 231
May 13 16:49:42 xen xapi: [error||1715 :::80||backtrace] 1/8 xapi Raised at file ocaml/database/db_cache_impl.ml, line 310
May 13 16:49:42 xen xapi: [error||1712 :::80||backtrace] 7/8 xapi Called from file map.ml, line 135
May 13 16:49:42 xen xapi: [error||1715 :::80||backtrace] 2/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
May 13 16:49:42 xen xapi: [error||1714 :::80||backtrace] 4/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 100
May 13 16:49:42 xen xapi: [error||1712 :::80||backtrace] 8/8 xapi Called from file src/sexp_conv.ml, line 156
May 13 16:49:42 xen xapi: [error||1714 :::80||backtrace] 5/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 121
May 13 16:49:42 xen xapi: [error||1715 :::80||backtrace] 3/8 xapi Called from file ocaml/xapi/rbac.ml, line 231
May 13 16:49:42 xen xapi: [error||1712 :::80||backtrace]
May 13 16:49:42 xen xapi: [error||1714 :::80||backtrace] 6/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
May 13 16:49:42 xen xapi: [error||1715 :::80||backtrace] 4/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 100
May 13 16:49:42 xen xapi: [error||1714 :::80||backtrace] 7/8 xapi Called from file map.ml, line 135
May 13 16:49:42 xen xapi: [error||1715 :::80||backtrace] 5/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 121
May 13 16:49:42 xen xapi: [error||1714 :::80||backtrace] 8/8 xapi Called from file src/sexp_conv.ml, line 156
May 13 16:49:42 xen xapi: [error||1715 :::80||backtrace] 6/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
May 13 16:49:42 xen xapi: [error||1714 :::80||backtrace]
May 13 16:49:42 xen xapi: [error||1715 :::80||backtrace] 7/8 xapi Called from file map.ml, line 135
May 13 16:49:42 xen xapi: [error||1715 :::80||backtrace] 8/8 xapi Called from file src/sexp_conv.ml, line 156
May 13 16:49:42 xen xapi: [error||1715 :::80||backtrace]

Perhaps, it's a problem with some latest changes to xoce after all?

@burbilog ah, no, there are some repeated, i had to sort it first. but there is a lot of uuids that are repeated.

@olivierlambert installed xo-cli, connected and ran xo-cli --list-objects , then grepped all uuids, stirpped everything else and tried to find dupes -- none. all uuids are unique there.

@olivierlambert I ran xe sr-list and xe-vm-list on all hosts, compiled a list of uuids, sorted them and all of them are unique. How to search for duplicate keys then?

@olivierlambert So, how to look up these keys? I ran xe vm-list and xe sr-list on all hosts, it seems that all UUIDs are different.

I even reinstalled one of the hosts, set up 8.2.1 anew and the problem here persists.

@olivierlambert I did not do anything on these hosts, except occasional yum update. Frankly, i did not touch this setup for a while and it worked a month or two ago ok.

I'm kinda surprised that all 3 servers do the same. One of them wasn't touched at all except yum update since it worked ok.