Every virtual machine I restart doesn't boot.

Pilow

@ohthisis you confirm that your 2 VMs Grafana and Tor are paused in web UI ?
if not, try

xe vm-reset-powerstate uuid=<uuid_of_vm> force=true

and do a restart toolstack afterward

ohthisis

@Pilow
Doesn't work. It just show VM.start: 33%
I'm afraid to restart the server too because the XO will not run also.

ohthisis

I don't know what happened.

Pilow

@ohthisis the --psr- state of Grafana is suspicious...
0.0 in Time column for Tor too

did you have a brutal reboot / power loss of this server ?
is your storage OK ?

there is some commands to brutally kill all VM in hybrid states like that, but could corrupt data inside (like a forced hard shutdown would do)

do you have backups of your VMs ?

# xl destroy 114

This command would kill the domain runtime (hard shutdown) of Grafana VM
it would then disappear of xl list
followed by a toolstack restart, should present you an halted VM in web UI that potentially could be started normally if no other issue on storage exists

but :

i do not recommend this action if your storage currently have issues
i do not recommend this action if you do not have backups
i do not recommend this action, it could corrupt data inside the VM (it is paused.. but is it really ?!)

just noticed you do not have same vCPUs on VMs with each command.
there is a real desync in your system

ohthisis

@Pilow
Please take a look at these:

[09:08 xcp-ng ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        2.8G   20K  2.8G   1% /dev
tmpfs           2.8G  160K  2.8G   1% /dev/shm
tmpfs           2.8G   11M  2.8G   1% /run
tmpfs           2.8G     0  2.8G   0% /sys/fs/cgroup
/dev/sda1        18G  2.0G   15G  12% /
xenstore        2.8G     0  2.8G   0% /var/lib/xenstored
/dev/sda5       3.9G  836M  2.8G  23% /var/log
tmpfs           571M     0  571M   0% /run/user/0
[09:17 xcp-ng ~]# mount | grep -E "(\/var|\/opt)"
xenstore on /var/lib/xenstored type tmpfs (rw,relatime,mode=755)
/dev/sda5 on /var/log type ext3 (rw,relatime)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
[09:18 xcp-ng ~]# dmesg | grep -iE "(error|fail|timeout|scsi|sd|hba)" | tail -20
[81734324.210688] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.217399] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.223994] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.230647] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.237867] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.244529] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.251239] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.257781] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.264466] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.270941] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.277574] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.284149] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.290576] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.297118] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.306270] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.314061] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.322632] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.330717] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.337242] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[81734324.344686] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[09:18 xcp-ng ~]# pvs
  PV         VG                                                 Fmt  Attr PSize  PFree  
  /dev/sda3  VG_XenStorage-00f82a18-a9f6-f7bc-9ca1-f42698d46b5f lvm2 a--  95.18g <95.18g
  /dev/sdb   VG_XenStorage-c5129868-a590-68ca-e587-db708ad61f38 lvm2 a--  <3.64t   3.09t
[09:18 xcp-ng ~]# vgs
  VG                                                 #PV #LV #SN Attr   VSize  VFree  
  VG_XenStorage-00f82a18-a9f6-f7bc-9ca1-f42698d46b5f   1   1   0 wz--n- 95.18g <95.18g
  VG_XenStorage-c5129868-a590-68ca-e587-db708ad61f38   1   8   0 wz--n- <3.64t   3.09t
[09:18 xcp-ng ~]# lvs -o lv_name,vg_name,lv_size,lv_attr
  LV                                       VG                                                 LSize    Attr      
  MGT                                      VG_XenStorage-00f82a18-a9f6-f7bc-9ca1-f42698d46b5f    4.00m -wi-a-----
  MGT                                      VG_XenStorage-c5129868-a590-68ca-e587-db708ad61f38    4.00m -wi-a-----
  VHD-1461c885-89c6-4e0e-8ee1-7d5be059f3dc VG_XenStorage-c5129868-a590-68ca-e587-db708ad61f38  <30.07g -wi-------
  VHD-2aaa4501-1c9b-48d6-8532-961ab8a3e627 VG_XenStorage-c5129868-a590-68ca-e587-db708ad61f38  <30.07g -wi-ao----
  VHD-4de5831d-5a4d-4d2d-9f0a-ce4d1c2d8ef5 VG_XenStorage-c5129868-a590-68ca-e587-db708ad61f38  100.20g -wi-a-----
  VHD-6b1ea821-d677-4426-99e0-43314ef3c536 VG_XenStorage-c5129868-a590-68ca-e587-db708ad61f38 <250.50g -wi-ao----
  VHD-6c08ae7f-71a7-4f97-a553-3c067dbbe243 VG_XenStorage-c5129868-a590-68ca-e587-db708ad61f38  <50.11g -wi-a-----
  VHD-bc8dd3e4-ea0e-4006-a918-817b18d65456 VG_XenStorage-c5129868-a590-68ca-e587-db708ad61f38  <50.11g -wi-ao----
  VHD-ccaaabb0-b5ae-4e29-ab8d-c895af000550 VG_XenStorage-c5129868-a590-68ca-e587-db708ad61f38  <50.11g -wi-a-----
[09:18 xcp-ng ~]# lvdisplay /dev/VG_XenStorage-c5129868-a590-68ca-e587-db708ad61f38/VHD-ccaaabb0-b5ae-4e29-ab8d-c895af000550
  --- Logical volume ---
  LV Path                /dev/VG_XenStorage-c5129868-a590-68ca-e587-db708ad61f38/VHD-ccaaabb0-b5ae-4e29-ab8d-c895af000550
  LV Name                VHD-ccaaabb0-b5ae-4e29-ab8d-c895af000550
  VG Name                VG_XenStorage-c5129868-a590-68ca-e587-db708ad61f38
  LV UUID                TggCle-7H7d-BN1o-KU5U-8oME-lckS-z0puvZ
  LV Write Access        read/write
  LV Creation host, time xcp-ng, 2023-07-11 14:19:40 +0330
  LV Status              available
  # open                 0
  LV Size                <50.11 GiB
  Current LE             12827
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

Pilow

@ohthisis I see nothing outstanding
you have two SRs, thick provisionned
one small 95Gb that is empty probably created on the install, and one big 3.64Tb on /dev/sdb

VMs are on the big SR, same sized VDIs could indicate existing snapshots.

is your /dev/sdb a RAID5 array or a standlone disk ?

Can you create a new test VM, that is running normally on this SR ?

ohthisis

@Pilow
I guess my server using RAID5.
I created a VM with PXE as boot, but it is VM.start: 50%.

Pilow

@ohthisis I really don't understand what is happening to your server
all basic tests show something is wrong but I don't know what

hope that someone else could hop in on this case
@Danp ?

ohthisis

I rebooted the server and now I got:
Screenshot at 2026-02-11 12-34-45.png

Pilow

@ohthisis you have no bootable disk on this VM, or boot order is wrong.
could you screen the DISKS tab and the ADVANCED tab ?

Danp

It is difficult to know the source of the problem without more details from your logs. Can you run the following commands on your XCP-ng host and then attach the generated files here?

grep -A 15 -B 15 -i error /var/log/xensource.log/xensource > xensource.txt
grep -A 15 -B 15 -i exception /var/log/SMlog > smlog.txt
dmesg -T|grep -Eiv 'guest|capacity|promiscuous' > dmesg.txt

ohthisis

Hello,
This issue was related to XPC-ng and caused me to delete and recreate and configure one of the most important virtual machines I have. As I said, every virtual machine I create is only 33% complete, and when I try to turn on virtual machines that are already turned off, I get the message "VM state is halted but should be running".
I had to turn the server off and back on to fix the problem. To be honest, I'm afraid to test again to see if the problem is fixed, because I have a VPN server that has a number of users connected to it, and I also have a monitoring server.
If it is a hardware problem, the green lights on the hard drive should change to orange or red, but all the hard drive lights on the server are green. I think the problem is related to the stability of XCP-ng.

olivierlambert

XCP-ng is rock stable. We have (and Citrix does too) many many automated test for such basic operations. So you clearly have an issue somewhere triggering that problem.

ohthisis

@olivierlambert Where? How to find it? As you can see, I only have 4 virtual machines on the XCP-ng host.

olivierlambert

Starting with the documentation is a good thing: https://docs.xcp-ng.org/troubleshooting/

This will give you (and us) hints on what's going on

nikade

@ohthisis Go to the Disks tab of the VM, what does it look like?

DustinB

A few things can be implied from his last screenshot and comments.

His boot order is set to PXE boot first, disk next.
He has no bootable disk attached to the VM.

It's possible that the disk is "detached" from the VM. IE it says disconnected or whatever under disks.

My Example

He earlier mentioned that he "created a VM with PXE as boot" but what is his PXE server doing? Why not just download the ISO or import the image directly to XCP-ng and build a VM from that as a template.

nikade

@DustinB yeah im guessing the VDI isn't attached to the VM for some reason, based on the screenshot.

Im also wondering if he ever rebooted the VM's after installing them with PXE

DustinB

@nikade said in Every virtual machine I restart doesn't boot.:

@DustinB yeah im guessing the VDI isn't attached to the VM for some reason, based on the screenshot.

Im also wondering if he ever rebooted the VM's after installing them with PXE

Right, it's a likely answer... but even then I would've expected his PXE server to just restart the installation process all over again... assuming that the disk is attached to the VM etc and that PXE boot isn't disabled automatically like it is with an ISO after first boot.

haha