XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Bitten by XSA-474

    Scheduled Pinned Locked Moved Development
    10 Posts 3 Posters 461 Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • V Offline
      vague
      last edited by

      Hi,
      I seem to have hit XSA-474 and trying to figure out how I can patch xapi to get the node up and running again. This appears to be when the issue started but not quite sure which part of the /var/log/xensource.log is relevant
      Portion of xensource.log

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Hi,

        Keep your XCP-ng up to date and you are protected. Cf https://docs.vates.tech/security/advisories/2025/vates-sa-2025-002

        V 1 Reply Last reply Reply Quote 0
        • V Offline
          vague @olivierlambert
          last edited by vague

          @olivierlambert
          from what I can tell I should be up to date and I've run the repair steps in the advisory also but still I see the below error. I was behind on updates but xapi crashed and before a reboot I thought an update was in order.
          I've reboot the node several times and latest was just before I ran the below commands

          [15:01 expo ~]# xe vm-list
Error: Connection refused (calling connect )
[15:01 expo ~]# rpm -qa | egrep 'xen|xapi'
xapi-xe-25.33.1-2.1.xcpng8.3.x86_64
xenopsd-25.33.1-2.1.xcpng8.3.x86_64
xenserver-status-report-2.0.15-1.xcpng8.3.noarch
xenopsd-cli-25.33.1-2.1.xcpng8.3.x86_64
xapi-tests-25.33.1-2.1.xcpng8.3.x86_64
python2-xapi-storage-24.19.2-1.10.xcpng8.3.x86_64
xapi-rrd2csv-25.33.1-2.1.xcpng8.3.x86_64
xapi-core-25.33.1-2.1.xcpng8.3.x86_64
xenserver-dracut-10-2.xcpng8.3.noarch
xenopsd-xc-25.33.1-2.1.xcpng8.3.x86_64
xengt-userspace-4.0.0-1.xcpng8.3.noarch
mellanox-mlnxen-5.9_0.5.5.0-3.1.xcpng8.3.x86_64
xapi-nbd-25.33.1-2.1.xcpng8.3.x86_64
xcp-ng-xapi-plugins-1.15.0-1.xcpng8.3.noarch
xenserver-hwdata-20240411-1.xcpng8.3.noarch
xen-hypervisor-4.17.5-23.1.xcpng8.3.x86_64
xen-tools-4.17.5-23.1.xcpng8.3.x86_64
xen-dom0-tools-4.17.5-23.1.xcpng8.3.x86_64
xen-crashdump-analyser-2.6.1-1.xcpng8.3.x86_64
xapi-storage-script-25.33.1-2.1.xcpng8.3.x86_64
xen-libs-4.17.5-23.1.xcpng8.3.x86_64
xen-dom0-libs-4.17.5-23.1.xcpng8.3.x86_64
          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            Let me add @team-os-platform-release

            V 1 Reply Last reply Reply Quote 0
            • V Offline
              vague @olivierlambert
              last edited by

              @olivierlambert thanks 🙂
              Just to limit the amount of data here is the db init part from xensource.log

              Jan  2 15:21:46 expo xapi: [debug||0 |Setup DB configuration D:888c979e4a5e|parse_db_conf]
Jan  2 15:21:46 expo xapi: [debug||0 |server_init D:02d6bb70f07f|startup] task [starting up database engine]
Jan  2 15:21:46 expo xapi: [debug||0 |server_init D:02d6bb70f07f|dummytaskhelper] task starting up database engine D:2a59ed50ccb5 created by task D:02d6bb70f07f
Jan  2 15:21:46 expo xapi: [debug||0 |starting up database engine D:2a59ed50ccb5|xapi] Attempting to populate database from one of these locations: [/var/lib/xcp/state.db]
Jan  2 15:21:46 expo xapi: [debug||0 |starting up database engine D:2a59ed50ccb5|xapi] Dbconf contains: /var/lib/xcp/state.db (generation 46799956)
Jan  2 15:21:46 expo xapi: [debug||0 |starting up database engine D:2a59ed50ccb5|xapi] Most recent db is /var/lib/xcp/state.db (generation 46799956)
Jan  2 15:21:46 expo xapi: [debug||0 |starting up database engine D:2a59ed50ccb5|sql] attempting to restore database from /var/lib/xcp/state.db
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] starting up database engine D:2a59ed50ccb5 failed with exception Xmlm.Error(2:317874, "malformed character stream")
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] Raised Xmlm.Error(2:317874, "malformed character stream")
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] 1/15 xapi Raised at file src/xmlm.ml, line 330
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] 2/15 xapi Called from file ocaml/database/db_xml.ml, line 201
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] 3/15 xapi Called from file ocaml/database/db_xml.ml, line 233
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] 4/15 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] 5/15 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] 6/15 xapi Called from file ocaml/database/backend_xml.ml, line 43
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] 7/15 xapi Called from file ocaml/database/db_cache_impl.ml, line 415
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] 8/15 xapi Called from file ocaml/database/db_cache_impl.ml, line 542
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] 9/15 xapi Called from file ocaml/xapi/xapi.ml, line 85
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] 10/15 xapi Called from file ocaml/xapi/xapi.ml, line 99
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] 11/15 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] 12/15 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] 13/15 xapi Called from file ocaml/xapi/server_helpers.ml, line 75
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] 14/15 xapi Called from file ocaml/xapi/server_helpers.ml, line 97
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace] 15/15 xapi Called from file ocaml/libs/log/debug.ml, line 250
Jan  2 15:21:46 expo xapi: [error||0 |server_init D:02d6bb70f07f|backtrace]
Jan  2 15:21:46 expo xapi: [ warn||0 |server_init D:02d6bb70f07f|startup] task [starting up database engine] exception: Xmlm.Error(2:317874, "malformed character stream")
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace] server_init D:02d6bb70f07f failed with exception Xmlm.Error(2:317874, "malformed character stream")
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace] Raised Xmlm.Error(2:317874, "malformed character stream")
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace] 1/14 xapi Raised at file ocaml/libs/log/debug.ml, line 271
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace] 2/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace] 3/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace] 4/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace] 5/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace] 6/14 xapi Called from file ocaml/xapi/startup.ml, line 95
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace] 7/14 xapi Called from file ocaml/xapi/startup.ml, line 103
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace] 8/14 xapi Called from file list.ml, line 110
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace] 9/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace] 10/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace] 11/14 xapi Called from file ocaml/xapi/xapi.ml, line 1071
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace] 12/14 xapi Called from file ocaml/xapi/server_helpers.ml, line 75
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace] 13/14 xapi Called from file ocaml/xapi/server_helpers.ml, line 97
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace] 14/14 xapi Called from file ocaml/libs/log/debug.ml, line 250
Jan  2 15:21:46 expo xapi: [error||0 |Xapi.watchdog|backtrace]
Jan  2 15:21:46 expo xapi: [debug||0 |Xapi.watchdog|xapi] xapi top-level caught exception: INTERNAL_ERROR: [ Xmlm.Error(2:317874, "malformed character stream") ]
Jan  2 15:21:46 expo xapi: [error||0 ||backtrace] Xapi.watchdog failed with exception Xmlm.Error(2:317874, "malformed character stream")
Jan  2 15:21:46 expo xapi: [error||0 ||backtrace] Raised Xmlm.Error(2:317874, "malformed character stream")
Jan  2 15:21:46 expo xapi: [error||0 ||backtrace] 1/11 xapi Raised at file ocaml/libs/log/debug.ml, line 271
Jan  2 15:21:46 expo xapi: [error||0 ||backtrace] 2/11 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Jan  2 15:21:46 expo xapi: [error||0 ||backtrace] 3/11 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
Jan  2 15:21:46 expo xapi: [error||0 ||backtrace] 4/11 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Jan  2 15:21:46 expo xapi: [error||0 ||backtrace] 5/11 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
Jan  2 15:21:46 expo xapi: [error||0 ||backtrace] 6/11 xapi Called from file ocaml/xapi/xapi.ml, line 1070
Jan  2 15:21:46 expo xapi: [error||0 ||backtrace] 7/11 xapi Called from file ocaml/xapi/xapi.ml, line 1526
Jan  2 15:21:46 expo xapi: [error||0 ||backtrace] 8/11 xapi Called from file ocaml/xapi/xapi.ml, line 1534
Jan  2 15:21:46 expo xapi: [error||0 ||backtrace] 9/11 xapi Called from file ocaml/xapi/xapi.ml, line 1540
Jan  2 15:21:46 expo xapi: [error||0 ||backtrace] 10/11 xapi Called from file ocaml/xapi/xapi.ml, line 1545
Jan  2 15:21:46 expo xapi: [error||0 ||backtrace] 11/11 xapi Called from file ocaml/libs/log/debug.ml, line 250
Jan  2 15:21:46 expo xapi: [error||0 ||backtrace]
              V 1 Reply Last reply Reply Quote 0
              • V Offline
                vague @vague
                last edited by

                This file appears to have been created when the issue started, can I remove it?

                -rw-r--r-- 1 root root       8 31 dec 19.15 state.db.generation
                V 1 Reply Last reply Reply Quote 0
                • V Offline
                  vague @vague
                  last edited by vague

                  I moved it to an .old file but didn't make a difference even after reboot

                  1 Reply Last reply Reply Quote 0
                  • nikadeN Offline
                    nikade Top contributor
                    last edited by

                    @vague said in Bitten by XSA-474:

                    starting up database engine D:2a59ed50ccb5 failed with exception Xmlm.Error(2:317874, "malformed character stream")

                    My guess is that your xapi database has been corrupted, if you dont have a backup this will be a rough one.

                    To add some content I'll summarize what you wrote on IRC:

                    1. You got an error, it was not resolved.
                    2. You then decided to start upgrading your xcp installation
                    3. Now the database is corrupt, not sure it it was already or if it happend in step 2.
                    V 1 Reply Last reply Reply Quote 0
                    • V Offline
                      vague @nikade
                      last edited by

                      I've restored my vms on the other node from backups but before I reinstall xcp-ng on the failing node I'd like to know if there is anything I can do to recover the database

                      1 Reply Last reply Reply Quote 0
                      • nikadeN Offline
                        nikade Top contributor
                        last edited by

                        I have no clue, I usually back it up with XO, its the one called "XO config & pool metadata backup" and only had to restore it once.

                        1 Reply Last reply Reply Quote 0

                        Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                        Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                        With your input, this post could be even better 💗

                        Register Login
                        • First post
                          Last post