Booting to Dracut (I trusted ChatGPT)

[nuentes]

Well, I've gotten myself into trouble using ChatGPT. I'm sure this will read like a horror novel for you. This started with my newer USB HDD enclosure. Things were working fine for a while, and then I started to see random disconnects. All the disks would drop from xcp-ng simultaneously after x mins/hours. Here is a dmesg from xcp-ng displaying what happened:

[09:58 xcp-ng-3 ~]# dmesg -T | grep -Ei "uas|reset|error|fail|disconnect" 
[Mon Nov 17 09:35:23 2025] nvme nvme0: failed to allocate host memory buffer. 
[Mon Nov 17 09:35:23 2025] GPT: Use GNU Parted to correct GPT errors. 
[Mon Nov 17 09:35:24 2025] GPT: Use GNU Parted to correct GPT errors. 
[Mon Nov 17 09:35:24 2025] scsi host1: uas 
[Mon Nov 17 09:35:24 2025] usbcore: registered new interface driver uas 
[Mon Nov 17 09:35:25 2025] scsi host2: uas 
[Mon Nov 17 09:35:25 2025] scsi host3: uas 
[Mon Nov 17 09:35:25 2025] scsi host4: uas 
[Mon Nov 17 09:35:25 2025] scsi host5: uas 
[Mon Nov 17 09:35:25 2025] scsi host6: uas 
[Mon Nov 17 09:36:22 2025] CIFS VFS: Error connecting to socket. Aborting operation. 
[Mon Nov 17 09:36:22 2025] CIFS VFS: cifs_mount failed w/return code = -113 
[Mon Nov 17 09:36:31 2025] CIFS VFS: Error connecting to socket. Aborting operation. 
[Mon Nov 17 09:36:31 2025] CIFS VFS: cifs_mount failed w/return code = -113 
[Mon Nov 17 10:05:23 2025] sd 2:0:0:0: [sdc] tag#0 data cmplt err -71 uas-tag 1 inflight: CMD 
[Mon Nov 17 10:05:54 2025] sd 2:0:0:0: [sdc] tag#10 uas_eh_abort_handler 0 uas-tag 11 inflight: CMD IN 
[Mon Nov 17 10:05:54 2025] sd 2:0:0:0: [sdc] tag#9 uas_eh_abort_handler 0 uas-tag 10 inflight: CMD OUT 
[Mon Nov 17 10:05:54 2025] sd 2:0:0:0: [sdc] tag#8 uas_eh_abort_handler 0 uas-tag 9 inflight: CMD OUT 
[Mon Nov 17 10:05:54 2025] sd 2:0:0:0: [sdc] tag#7 uas_eh_abort_handler 0 uas-tag 8 inflight: CMD OUT 
[Mon Nov 17 10:05:54 2025] sd 2:0:0:0: [sdc] tag#6 uas_eh_abort_handler 0 uas-tag 7 inflight: CMD OUT 
[Mon Nov 17 10:05:54 2025] sd 2:0:0:0: [sdc] tag#5 uas_eh_abort_handler 0 uas-tag 6 inflight: CMD OUT 
[Mon Nov 17 10:05:54 2025] sd 2:0:0:0: [sdc] tag#4 uas_eh_abort_handler 0 uas-tag 5 inflight: CMD OUT 
[Mon Nov 17 10:05:54 2025] sd 2:0:0:0: [sdc] tag#3 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD OUT 
[Mon Nov 17 10:05:54 2025] sd 2:0:0:0: [sdc] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT 
[Mon Nov 17 10:05:54 2025] sd 2:0:0:0: [sdc] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT 
[Mon Nov 17 10:05:54 2025] sd 2:0:0:0: [sdc] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD 
[Mon Nov 17 10:06:39 2025] sd 2:0:0:0: [sdc] tag#11 uas_eh_abort_handler 0 uas-tag 12 inflight: CMD 
[Mon Nov 17 10:06:39 2025] scsi host2: uas_eh_device_reset_handler start 
[Mon Nov 17 10:06:39 2025] usb 4-1.1: reset SuperSpeedPlus Gen 2 USB device number 4 using xhci_hcd 
[Mon Nov 17 10:06:39 2025] scsi host2: uas_eh_device_reset_handler success 
[Mon Nov 17 10:06:39 2025] sd 2:0:0:0: [sdc] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT 
[Mon Nov 17 10:12:41 2025] usb 1-2: reset full-speed USB device number 2 using xhci_hcd 
[Mon Nov 17 10:34:59 2025] sd 2:0:0:0: [sdc] tag#2 data cmplt err -71 uas-tag 3 inflight: CMD 
[Mon Nov 17 10:35:31 2025] sd 2:0:0:0: [sdc] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD 
[Mon Nov 17 10:35:31 2025] scsi host2: uas_eh_device_reset_handler start 
[Mon Nov 17 10:35:31 2025] usb 4-1.1: reset SuperSpeedPlus Gen 2 USB device number 4 using xhci_hcd 
[Mon Nov 17 10:35:31 2025] scsi host2: uas_eh_device_reset_handler success 
[Mon Nov 17 10:36:29 2025] sd 2:0:0:0: [sdc] tag#0 data cmplt err -71 uas-tag 1 inflight: CMD 
[Mon Nov 17 10:37:01 2025] sd 2:0:0:0: [sdc] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT 
[Mon Nov 17 10:37:01 2025] sd 2:0:0:0: [sdc] tag#3 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD OUT 
[Mon Nov 17 10:37:01 2025] sd 2:0:0:0: [sdc] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT 
[Mon Nov 17 10:37:01 2025] sd 2:0:0:0: [sdc] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD 
[Mon Nov 17 10:37:01 2025] scsi host2: uas_eh_device_reset_handler start 
[Mon Nov 17 10:37:01 2025] usb 4-1.1: reset SuperSpeedPlus Gen 2 USB device number 4 using xhci_hcd 
[Mon Nov 17 10:37:01 2025] scsi host2: uas_eh_device_reset_handler success 
[Mon Nov 17 10:37:12 2025] sd 6:0:0:0: [sdg] tag#0 data cmplt err -71 uas-tag 1 inflight: CMD 
[Mon Nov 17 10:37:44 2025] sd 6:0:0:0: [sdg] tag#3 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD OUT 
[Mon Nov 17 10:37:44 2025] sd 6:0:0:0: [sdg] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT 
[Mon Nov 17 10:37:44 2025] sd 6:0:0:0: [sdg] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT 
[Mon Nov 17 10:37:44 2025] sd 6:0:0:0: [sdg] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD 
[Mon Nov 17 10:37:44 2025] scsi host6: uas_eh_device_reset_handler start 
[Mon Nov 17 10:37:44 2025] usb 4-1.4: reset SuperSpeedPlus Gen 2 USB device number 8 using xhci_hcd 
[Mon Nov 17 10:37:44 2025] scsi host6: uas_eh_device_reset_handler success 
[Mon Nov 17 12:06:18 2025] sd 3:0:0:0: [sdd] tag#2 data cmplt err -71 uas-tag 3 inflight: CMD 
[Mon Nov 17 12:06:49 2025] sd 3:0:0:0: [sdd] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD 
[Mon Nov 17 12:06:49 2025] scsi host3: uas_eh_device_reset_handler start 
[Mon Nov 17 12:06:49 2025] usb 4-1.2: reset SuperSpeedPlus Gen 2 USB device number 5 using xhci_hcd 
[Mon Nov 17 12:06:49 2025] scsi host3: uas_eh_device_reset_handler success 
[Mon Nov 17 12:08:15 2025] sd 3:0:0:0: [sdd] tag#0 data cmplt err -71 uas-tag 1 inflight: CMD 
[Mon Nov 17 12:08:48 2025] sd 3:0:0:0: [sdd] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT 
[Mon Nov 17 12:08:48 2025] sd 3:0:0:0: [sdd] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT 
[Mon Nov 17 12:08:48 2025] sd 3:0:0:0: [sdd] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD 
[Mon Nov 17 12:08:48 2025] scsi host3: uas_eh_device_reset_handler start 
[Mon Nov 17 12:08:48 2025] usb 4-1.2: reset SuperSpeedPlus Gen 2 USB device number 5 using xhci_hcd 
[Mon Nov 17 12:08:48 2025] scsi host3: uas_eh_device_reset_handler success 
[Wed Nov 19 03:05:32 2025] sd 2:0:0:0: [sdc] tag#6 data cmplt err -71 uas-tag 7 inflight: CMD 
[Wed Nov 19 03:06:04 2025] sd 2:0:0:0: [sdc] tag#9 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT 
[Wed Nov 19 03:06:04 2025] sd 2:0:0:0: [sdc] tag#8 uas_eh_abort_handler 0 uas-tag 9 inflight: CMD OUT 
[Wed Nov 19 03:06:04 2025] sd 2:0:0:0: [sdc] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT 
[Wed Nov 19 03:06:04 2025] sd 2:0:0:0: [sdc] tag#7 uas_eh_abort_handler 0 uas-tag 8 inflight: CMD OUT 
[Wed Nov 19 03:06:04 2025] sd 2:0:0:0: [sdc] tag#6 uas_eh_abort_handler 0 uas-tag 7 inflight: CMD 
[Wed Nov 19 03:06:04 2025] scsi host2: uas_eh_device_reset_handler start 
[Wed Nov 19 03:06:04 2025] usb 4-1.1: reset SuperSpeedPlus Gen 2 USB device number 4 using xhci_hcd 
[Wed Nov 19 03:06:04 2025] scsi host2: uas_eh_device_reset_handler success 
[Wed Nov 19 03:07:20 2025] sd 2:0:0:0: [sdc] tag#7 uas_eh_abort_handler 0 uas-tag 8 inflight: CMD OUT 
[Wed Nov 19 03:07:20 2025] sd 2:0:0:0: [sdc] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD OUT 
[Wed Nov 19 03:07:20 2025] sd 2:0:0:0: [sdc] tag#3 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD OUT 
[Wed Nov 19 03:07:20 2025] scsi host2: uas_eh_device_reset_handler start 
[Wed Nov 19 03:07:20 2025] usb 4-1.1: reset SuperSpeedPlus Gen 2 USB device number 4 using xhci_hcd 
[Wed Nov 19 03:07:20 2025] scsi host2: uas_eh_device_reset_handler success 
[Wed Nov 19 03:14:01 2025] sd 2:0:0:0: [sdc] tag#1 data cmplt err -71 uas-tag 2 inflight: CMD 
[Wed Nov 19 03:14:34 2025] sd 2:0:0:0: [sdc] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD OUT 
[Wed Nov 19 03:14:34 2025] sd 2:0:0:0: [sdc] tag#4 uas_eh_abort_handler 0 uas-tag 5 inflight: CMD OUT 
[Wed Nov 19 03:14:34 2025] sd 2:0:0:0: [sdc] tag#3 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD OUT 
[Wed Nov 19 03:14:34 2025] sd 2:0:0:0: [sdc] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT 
[Wed Nov 19 03:14:34 2025] sd 2:0:0:0: [sdc] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD 
[Wed Nov 19 03:14:34 2025] scsi host2: uas_eh_device_reset_handler start 
[Wed Nov 19 03:14:34 2025] usb 4-1.1: reset SuperSpeedPlus Gen 2 USB device number 4 using xhci_hcd 
[Wed Nov 19 03:14:34 2025] scsi host2: uas_eh_device_reset_handler success 
[Wed Nov 19 03:14:58 2025] sd 2:0:0:0: [sdc] tag#11 data cmplt err -71 uas-tag 12 inflight: CMD 
[Wed Nov 19 03:15:29 2025] sd 2:0:0:0: [sdc] tag#3 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD OUT 
[Wed Nov 19 03:15:29 2025] sd 2:0:0:0: [sdc] tag#15 uas_eh_abort_handler 0 uas-tag 16 inflight: CMD OUT 
[Wed Nov 19 03:15:29 2025] sd 2:0:0:0: [sdc] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT 
[Wed Nov 19 03:15:29 2025] sd 2:0:0:0: [sdc] tag#14 uas_eh_abort_handler 0 uas-tag 15 inflight: CMD OUT 
[Wed Nov 19 03:15:29 2025] sd 2:0:0:0: [sdc] tag#13 uas_eh_abort_handler 0 uas-tag 14 inflight: CMD OUT 
[Wed Nov 19 03:15:29 2025] sd 2:0:0:0: [sdc] tag#12 uas_eh_abort_handler 0 uas-tag 13 inflight: CMD OUT 
[Wed Nov 19 03:15:29 2025] sd 2:0:0:0: [sdc] tag#11 uas_eh_abort_handler 0 uas-tag 12 inflight: CMD 
[Wed Nov 19 03:15:29 2025] scsi host2: uas_eh_device_reset_handler start 
[Wed Nov 19 03:15:29 2025] usb 4-1.1: reset SuperSpeedPlus Gen 2 USB device number 4 using xhci_hcd 
[Wed Nov 19 03:15:29 2025] scsi host2: uas_eh_device_reset_handler success 
[Wed Nov 19 03:20:30 2025] sd 2:0:0:0: [sdc] tag#1 data cmplt err -71 uas-tag 2 inflight: CMD 
[Wed Nov 19 03:21:01 2025] sd 2:0:0:0: [sdc] tag#15 uas_eh_abort_handler 0 uas-tag 16 inflight: CMD OUT 
[Wed Nov 19 03:21:01 2025] sd 2:0:0:0: [sdc] tag#14 uas_eh_abort_handler 0 uas-tag 15 inflight: CMD OUT 
[Wed Nov 19 03:21:01 2025] sd 2:0:0:0: [sdc] tag#13 uas_eh_abort_handler 0 uas-tag 14 inflight: CMD OUT 
[Wed Nov 19 03:21:01 2025] sd 2:0:0:0: [sdc] tag#12 uas_eh_abort_handler 0 uas-tag 13 inflight: CMD OUT 
[Wed Nov 19 03:21:01 2025] sd 2:0:0:0: [sdc] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD 
[Wed Nov 19 03:21:01 2025] scsi host2: uas_eh_device_reset_handler start 
[Wed Nov 19 03:21:01 2025] usb 4-1.1: reset SuperSpeedPlus Gen 2 USB device number 4 using xhci_hcd 
[Wed Nov 19 03:21:01 2025] scsi host2: uas_eh_device_reset_handler success 
[Wed Nov 19 03:26:43 2025] sd 2:0:0:0: [sdc] tag#0 data cmplt err -71 uas-tag 1 inflight: CMD 
[Wed Nov 19 03:27:14 2025] sd 2:0:0:0: [sdc] tag#7 uas_eh_abort_handler 0 uas-tag 8 inflight: CMD OUT 
[Wed Nov 19 03:27:14 2025] sd 2:0:0:0: [sdc] tag#6 uas_eh_abort_handler 0 uas-tag 7 inflight: CMD OUT 
[Wed Nov 19 03:27:14 2025] sd 2:0:0:0: [sdc] tag#5 uas_eh_abort_handler 0 uas-tag 6 inflight: CMD OUT 
[Wed Nov 19 03:27:14 2025] sd 2:0:0:0: [sdc] tag#4 uas_eh_abort_handler 0 uas-tag 5 inflight: CMD OUT 
[Wed Nov 19 03:27:14 2025] sd 2:0:0:0: [sdc] tag#3 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD OUT 
[Wed Nov 19 03:27:14 2025] sd 2:0:0:0: [sdc] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT 
[Wed Nov 19 03:27:14 2025] sd 2:0:0:0: [sdc] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT 
[Wed Nov 19 03:27:14 2025] sd 2:0:0:0: [sdc] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD 
[Wed Nov 19 03:27:17 2025] sd 2:0:0:0: [sdc] tag#14 uas_eh_abort_handler 0 uas-tag 15 inflight: CMD IN 
[Wed Nov 19 03:27:17 2025] sd 2:0:0:0: [sdc] tag#13 uas_eh_abort_handler 0 uas-tag 14 inflight: CMD IN 
[Wed Nov 19 03:27:17 2025] sd 2:0:0:0: [sdc] tag#12 uas_eh_abort_handler 0 uas-tag 13 inflight: CMD IN 
[Wed Nov 19 03:27:17 2025] sd 2:0:0:0: [sdc] tag#11 uas_eh_abort_handler 0 uas-tag 12 inflight: CMD IN 
[Wed Nov 19 03:27:17 2025] sd 2:0:0:0: [sdc] tag#10 uas_eh_abort_handler 0 uas-tag 11 inflight: CMD IN 
[Wed Nov 19 03:27:17 2025] sd 2:0:0:0: [sdc] tag#9 uas_eh_abort_handler 0 uas-tag 10 inflight: CMD IN 
[Wed Nov 19 03:27:17 2025] sd 2:0:0:0: [sdc] tag#8 uas_eh_abort_handler 0 uas-tag 9 inflight: CMD IN 
[Wed Nov 19 03:27:38 2025] sd 2:0:0:0: [sdc] tag#18 uas_eh_abort_handler 0 uas-tag 19 inflight: CMD IN 
[Wed Nov 19 03:27:38 2025] sd 2:0:0:0: [sdc] tag#17 uas_eh_abort_handler 0 uas-tag 18 inflight: CMD IN 
[Wed Nov 19 03:27:38 2025] sd 2:0:0:0: [sdc] tag#16 uas_eh_abort_handler 0 uas-tag 17 inflight: CMD IN 
[Wed Nov 19 03:27:38 2025] sd 2:0:0:0: [sdc] tag#15 uas_eh_abort_handler 0 uas-tag 16 inflight: CMD IN 
[Wed Nov 19 03:27:38 2025] scsi host2: uas_eh_device_reset_handler start 
[Wed Nov 19 03:27:38 2025] usb 4-1.1: reset SuperSpeedPlus Gen 2 USB device number 4 using xhci_hcd 
[Wed Nov 19 03:27:38 2025] scsi host2: uas_eh_device_reset_handler success 
[Wed Nov 19 04:01:27 2025] sd 3:0:0:0: [sdd] tag#1 data cmplt err -71 uas-tag 2 inflight: CMD 
[Wed Nov 19 04:01:58 2025] sd 3:0:0:0: [sdd] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT 
[Wed Nov 19 04:01:58 2025] sd 3:0:0:0: [sdd] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD 
[Wed Nov 19 04:01:58 2025] scsi host3: uas_eh_device_reset_handler start 
[Wed Nov 19 04:01:58 2025] usb 4-1.2: reset SuperSpeedPlus Gen 2 USB device number 5 using xhci_hcd 
[Wed Nov 19 04:01:58 2025] scsi host3: uas_eh_device_reset_handler success 
[Wed Nov 19 04:02:49 2025] sd 3:0:0:0: [sdd] tag#1 data cmplt err -71 uas-tag 2 inflight: CMD 
[Wed Nov 19 04:03:20 2025] sd 3:0:0:0: [sdd] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT 
[Wed Nov 19 04:03:20 2025] sd 3:0:0:0: [sdd] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD 
[Wed Nov 19 04:03:20 2025] scsi host3: uas_eh_device_reset_handler start 
[Wed Nov 19 04:03:20 2025] usb 4-1.2: reset SuperSpeedPlus Gen 2 USB device number 5 using xhci_hcd 
[Wed Nov 19 04:03:20 2025] scsi host3: uas_eh_device_reset_handler success 
[Wed Nov 19 04:04:31 2025] sd 3:0:0:0: [sdd] tag#0 data cmplt err -71 uas-tag 1 inflight: CMD 
[Wed Nov 19 04:05:03 2025] sd 3:0:0:0: [sdd] tag#2 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT 
[Wed Nov 19 04:05:03 2025] sd 3:0:0:0: [sdd] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT 
[Wed Nov 19 04:05:03 2025] sd 3:0:0:0: [sdd] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD 
[Wed Nov 19 04:05:03 2025] scsi host3: uas_eh_device_reset_handler start 
[Wed Nov 19 04:05:03 2025] usb 4-1.2: reset SuperSpeedPlus Gen 2 USB device number 5 using xhci_hcd 
[Wed Nov 19 04:05:03 2025] scsi host3: uas_eh_device_reset_handler success 
[Wed Nov 19 04:09:18 2025] sd 3:0:0:0: [sdd] tag#0 data cmplt err -71 uas-tag 1 inflight: CMD 
[Wed Nov 19 04:09:18 2025] usb 4-1.1: USB disconnect, device number 4 
[Wed Nov 19 04:09:18 2025] sd 2:0:0:0: [sdc] tag#0 uas_zap_pending 0 uas-tag 1 inflight: CMD 
[Wed Nov 19 04:09:18 2025] sd 2:0:0:0: [sdc] tag#4 uas_zap_pending 0 uas-tag 5 inflight: CMD 
[Wed Nov 19 04:09:18 2025] sd 2:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK 
[Wed Nov 19 04:09:18 2025] print_req_error: I/O error, dev sdc, sector 3173539592 
[Wed Nov 19 04:09:18 2025] sd 2:0:0:0: [sdc] tag#4 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK 
[Wed Nov 19 04:09:18 2025] print_req_error: I/O error, dev sdc, sector 3173539512 
[Wed Nov 19 04:09:18 2025] sd 2:0:0:0: [sdc] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK 
[Wed Nov 19 04:09:18 2025] usb 4-1.2: USB disconnect, device number 5 
[Wed Nov 19 04:09:18 2025] sd 3:0:0:0: [sdd] tag#0 uas_zap_pending 0 uas-tag 1 inflight: CMD 
[Wed Nov 19 04:09:18 2025] sd 3:0:0:0: [sdd] tag#1 uas_zap_pending 0 uas-tag 2 inflight: CMD 
[Wed Nov 19 04:09:18 2025] sd 3:0:0:0: [sdd] tag#2 uas_zap_pending 0 uas-tag 3 inflight: CMD 
[Wed Nov 19 04:09:18 2025] sd 3:0:0:0: [sdd] tag#0 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK 
[Wed Nov 19 04:09:18 2025] print_req_error: I/O error, dev sdd, sector 981465279 
[Wed Nov 19 04:09:18 2025] sd 3:0:0:0: [sdd] tag#1 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK 
[Wed Nov 19 04:09:18 2025] print_req_error: I/O error, dev sdd, sector 981465895 
[Wed Nov 19 04:09:18 2025] sd 3:0:0:0: [sdd] tag#2 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK 
[Wed Nov 19 04:09:18 2025] print_req_error: I/O error, dev sdd, sector 981466599 
[Wed Nov 19 04:09:18 2025] sd 3:0:0:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK 
[Wed Nov 19 04:09:18 2025] usb 4-1.3: USB disconnect, device number 7 
[Wed Nov 19 04:09:18 2025] usb 4-1.4: USB disconnect, device number 8 
[Wed Nov 19 04:09:19 2025] sd 6:0:0:0: [sdg] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK 
[Wed Nov 19 04:09:19 2025] usb 4-1: reset SuperSpeedPlus Gen 2 USB device number 2 using xhci_hcd 
[Wed Nov 19 04:09:20 2025] scsi host7: uas 
[Wed Nov 19 04:09:20 2025] scsi host8: uas 
[Wed Nov 19 04:09:20 2025] scsi host9: uas 
[Wed Nov 19 04:09:20 2025] scsi host10: uas

So my first steps were to disconnect other disks, install updates, and reboot. I left it like that for about 48 hours and then the disks disconnected again.

ChatGPT then had me make a quirk entry in XCP-ng disabling UAS
nano /etc/modprobe.d/disable_uas.conf

options usb-storage quirks=174c:55aa:u

dracut -f
Reboot xcp-ng

[13:19 xcp-ng-3 ~]# dmesg | grep -i uas 
[ 2.307857] scsi host1: uas 
[ 2.307940] usbcore: registered new interface driver uas 
[ 3.989685] scsi host2: uas 
[ 4.538192] scsi host3: uas 
[ 4.657229] scsi host4: uas 
[ 4.769897] scsi host5: uas 
[ 4.885542] scsi host6: uas 
[13:19 xcp-ng-3 ~]# dmesg | grep -Ei "usb|storage" 
[ 0.822881] ACPI: bus type USB registered 
[ 0.822881] usbcore: registered new interface driver usbfs 
[ 0.822883] usbcore: registered new interface driver hub 
[ 0.822907] usbcore: registered new device driver usb 
[ 1.666549] xhci_hcd 0000:04:00.3: new USB bus registered, assigned bus number 1 
[ 1.667073] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.19 
[ 1.667075] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 
[ 1.667076] usb usb1: Product: xHCI Host Controller 
[ 1.667077] usb usb1: Manufacturer: Linux 4.19.0+1 xhci-hcd 
[ 1.667078] usb usb1: SerialNumber: 0000:04:00.3 
[ 1.667159] hub 1-0:1.0: USB hub found 
[ 1.667385] xhci_hcd 0000:04:00.3: new USB bus registered, assigned bus number 2 
[ 1.667387] xhci_hcd 0000:04:00.3: Host supports USB 3.10 Enhanced SuperSpeed 
[ 1.667403] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM. 
[ 1.667427] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 4.19 
[ 1.667428] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 
[ 1.667429] usb usb2: Product: xHCI Host Controller 
[ 1.667430] usb usb2: Manufacturer: Linux 4.19.0+1 xhci-hcd 
[ 1.667431] usb usb2: SerialNumber: 0000:04:00.3 
[ 1.667516] hub 2-0:1.0: USB hub found 
[ 1.667877] xhci_hcd 0000:04:00.4: new USB bus registered, assigned bus number 3 
[ 1.668369] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.19 
[ 1.668370] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 
[ 1.668370] usb usb3: Product: xHCI Host Controller 
[ 1.668371] usb usb3: Manufacturer: Linux 4.19.0+1 xhci-hcd 
[ 1.668372] usb usb3: SerialNumber: 0000:04:00.4 
[ 1.668427] hub 3-0:1.0: USB hub found 
[ 1.668609] xhci_hcd 0000:04:00.4: new USB bus registered, assigned bus number 4 
[ 1.668612] xhci_hcd 0000:04:00.4: Host supports USB 3.10 Enhanced SuperSpeed 
[ 1.668644] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM. 
[ 1.668669] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 4.19 
[ 1.668670] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 
[ 1.668671] usb usb4: Product: xHCI Host Controller 
[ 1.668672] usb usb4: Manufacturer: Linux 4.19.0+1 xhci-hcd 
[ 1.668673] usb usb4: SerialNumber: 0000:04:00.4 
[ 1.668781] hub 4-0:1.0: USB hub found 
[ 1.998932] usb 1-2: new full-speed USB device number 2 using xhci_hcd 
[ 1.998936] usb 3-2: new high-speed USB device number 2 using xhci_hcd 
[ 2.152287] usb 3-2: New USB device found, idVendor=2109, idProduct=2822, bcdDevice= 7.04 
[ 2.152291] usb 3-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 
[ 2.152293] usb 3-2: Product: USB2.0 Hub 
[ 2.152295] usb 3-2: Manufacturer: VIA Labs, Inc. 
[ 2.152296] usb 3-2: SerialNumber: 000000001 
[ 2.172438] usb 1-2: New USB device found, idVendor=0764, idProduct=0501, bcdDevice= 2.00 
[ 2.172442] usb 1-2: New USB device strings: Mfr=3, Product=1, SerialNumber=2 
[ 2.172443] usb 1-2: Product: ABMT1500 
[ 2.172445] usb 1-2: Manufacturer: CPS 
[ 2.172446] usb 1-2: SerialNumber: FCEMU2000539 
[ 2.197744] usbcore: registered new interface driver usbhid 
[ 2.197745] usbhid: USB HID core driver 
[ 2.200290] hid-generic 0003:0764:0501.0001: hiddev96,hidraw0: USB HID v1.10 Device [CPS ABMT1500] on usb-0000:04:00.3-2/input0 
[ 2.213064] hub 3-2:1.0: USB hub found 
[ 2.278937] usb 2-1: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd 
[ 2.299468] usb 2-1: New USB device found, idVendor=0781, idProduct=5583, bcdDevice= 1.00 
[ 2.299470] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 
[ 2.299471] usb 2-1: Product: SanDisk 3.2Gen1 
[ 2.299472] usb 2-1: Manufacturer: USB 
[ 2.299472] usb 2-1: SerialNumber: 03050119061825030101 
[ 2.304099] usbcore: registered new interface driver usb-storage 
[ 2.307940] usbcore: registered new interface driver uas 
[ 2.308321] usb 4-2: new SuperSpeedPlus Gen 2 USB device number 2 using xhci_hcd 
[ 2.309318] scsi 1:0:0:0: Direct-Access USB SanDisk 3.2Gen1 1.00 PQ: 0 ANSI: 7 
[ 2.742973] usb 4-2: New USB device found, idVendor=2109, idProduct=0822, bcdDevice= 7.04 
[ 2.742975] usb 4-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 
[ 2.742976] usb 4-2: Product: USB3.1 Hub 
[ 2.742977] usb 4-2: Manufacturer: VIA Labs, Inc. 
[ 2.742978] usb 4-2: SerialNumber: 000000001 
[ 2.772856] hub 4-2:1.0: USB hub found 
[ 2.866841] usb 3-3: new full-speed USB device number 3 using xhci_hcd 
[ 3.042673] usb 3-3: New USB device found, idVendor=8087, idProduct=0029, bcdDevice= 0.01 
[ 3.042677] usb 3-3: New USB device strings: Mfr=0, Product=0, SerialNumber=0 
[ 3.343380] usb 3-2.3: new high-speed USB device number 4 using xhci_hcd 
[ 3.499069] usb 3-2.3: New USB device found, idVendor=2109, idProduct=2822, bcdDevice= 8.b3 
[ 3.499073] usb 3-2.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0 
[ 3.499076] usb 3-2.3: Product: USB2.0 Hub 
[ 3.499078] usb 3-2.3: Manufacturer: VIA Labs, Inc. 
[ 3.556951] hub 3-2.3:1.0: USB hub found 
[ 3.588683] usb 4-2.3: new SuperSpeedPlus Gen 2 USB device number 3 using xhci_hcd 
[ 3.756958] usb 4-2.3: New USB device found, idVendor=2109, idProduct=0822, bcdDevice= 8.b3 
[ 3.756960] usb 4-2.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0 
[ 3.756961] usb 4-2.3: Product: USB3.1 Hub 
[ 3.756961] usb 4-2.3: Manufacturer: VIA Labs, Inc. 
[ 3.780931] hub 4-2.3:1.0: USB hub found 
[ 3.962994] usb 4-2.4: new SuperSpeed Gen 1 USB device number 4 using xhci_hcd 
[ 3.987919] usb 4-2.4: New USB device found, idVendor=0bc2, idProduct=331a, bcdDevice= 9.11 
[ 3.987923] usb 4-2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3 
[ 3.987926] usb 4-2.4: Product: Expansion Desk 
[ 3.987928] usb 4-2.4: Manufacturer: Seagate 
[ 3.987930] usb 4-2.4: SerialNumber: NAAA33R1 
[ 4.327289] usb 3-2.3.5: new high-speed USB device number 5 using xhci_hcd 
[ 4.430265] usb 3-2.3.5: New USB device found, idVendor=2109, idProduct=8822, bcdDevice= 0.01 
[ 4.430269] usb 3-2.3.5: New USB device strings: Mfr=1, Product=2, SerialNumber=3 
[ 4.430271] usb 3-2.3.5: Product: USB Billboard Device 
[ 4.430272] usb 3-2.3.5: Manufacturer: VIA Labs, Inc. 
[ 4.430274] usb 3-2.3.5: SerialNumber: 0000000000000001 
[ 4.510914] usb 4-2.3.1: new SuperSpeedPlus Gen 2 USB device number 5 using xhci_hcd 
[ 4.536008] usb 4-2.3.1: New USB device found, idVendor=174c, idProduct=55aa, bcdDevice= 1.00 
[ 4.536014] usb 4-2.3.1: New USB device strings: Mfr=2, Product=3, SerialNumber=1 
[ 4.536017] usb 4-2.3.1: Product: ASM235CM 
[ 4.536021] usb 4-2.3.1: Manufacturer: ASMedia 
[ 4.536023] usb 4-2.3.1: SerialNumber: AAAABBBB0002 
[ 4.626914] usb 4-2.3.2: new SuperSpeedPlus Gen 2 USB device number 6 using xhci_hcd 
[ 4.651614] usb 4-2.3.2: New USB device found, idVendor=174c, idProduct=55aa, bcdDevice= 1.00 
[ 4.651615] usb 4-2.3.2: New USB device strings: Mfr=2, Product=3, SerialNumber=1 
[ 4.651616] usb 4-2.3.2: Product: ASM235CM 
[ 4.651617] usb 4-2.3.2: Manufacturer: ASMedia 
[ 4.651618] usb 4-2.3.2: SerialNumber: AAAABBBB0002 
[ 4.743229] usb 4-2.3.3: new SuperSpeedPlus Gen 2 USB device number 7 using xhci_hcd 
[ 4.763405] usb 4-2.3.3: New USB device found, idVendor=174c, idProduct=55aa, bcdDevice= 1.00 
[ 4.763407] usb 4-2.3.3: New USB device strings: Mfr=2, Product=3, SerialNumber=1 
[ 4.763408] usb 4-2.3.3: Product: ASM235CM 
[ 4.763409] usb 4-2.3.3: Manufacturer: ASMedia 
[ 4.763409] usb 4-2.3.3: SerialNumber: AAAABBBB0002 
[ 4.858858] usb 4-2.3.4: new SuperSpeedPlus Gen 2 USB device number 8 using xhci_hcd 
[ 4.883456] usb 4-2.3.4: New USB device found, idVendor=174c, idProduct=55aa, bcdDevice= 1.00 
[ 4.883457] usb 4-2.3.4: New USB device strings: Mfr=2, Product=3, SerialNumber=1 
[ 4.883458] usb 4-2.3.4: Product: ASM235CM 
[ 4.883459] usb 4-2.3.4: Manufacturer: ASMedia 
[ 4.883460] usb 4-2.3.4: SerialNumber: AAAABBBB0002 
[ 77.523054] usb 1-2: reset full-speed USB device number 2 using xhci_hcd

ChatGPT then told me:

• On XCP-ng (CentOS-based, older kernel), this syntax is not enough:
• It will only apply if usb-storage loads before uas, and on your system uas loads earlier, so the quirk is ignored.
• We need to explicitly force usb-storage to claim the device before uas is even allowed to load.

nano /etc/modprobe.d/blacklist_uas_174c_55aa.conf

blocklist uas

dracut -f
reboot host

This again did nothing.

ChatGPT then said:

• Correct Fix for XCP-ng: Update the bootloader's initrd

[13:36 xcp-ng-3 ~]# cat /boot/efi/EFI/xenserver/grub.cfg | grep initrd
module2 /boot/initrd-4.19-xen.img
module2 /boot/initrd-4.19-xen.img
module2 /boot/initrd-4.19-xen.img
module2 /boot/initrd-fallback.img
module2 /boot/initrd-fallback.img

and then:

dracut -f /boot/initrd-4.19-xen.img 4.19-xen

After this, upon reboot I was booting to a dracut command prompt. The issue here was that apparently USB had been fully disabled, so my keyboard is fully disabled.

I probably should have posted on here then, but I didn't. At ChatGPT's suggested, I booted the device with the install media and ran shell. After a bit of work, here's what we did that seemed to be working:

mkdir -p /mnt/root
mount /dev/md127p1 /mnt/root
mount /dev/md127p4 /mnt/root/boot/efi
mount --bind /proc /mnt/root/proc
mount --bind /sys /mnt/root/sys
mount --bind /dev /mnt/root/dev
chroot /mnt/root

From chroot, I used the following to remove any offending modprobe.d files

ls -l /etc/modprobe.d

Followed by

dracut --regenerate-all --force

I then umounted everything, exited chroot, and rebooted.

After a reboot the issue persisted. So we tried dracut with the verbose flag (NOTE: this is OCR from a photo, so there are typos and weirdness):

sh-4.2# dracut -force -verbose /boot/initranfs-S(uname -r).img $(uname -r)
Executing: /usr/sbin/dracut -force -verbose /boot/initramfs-4.19.0+1.ing 4.19.0+1
dracut module 'btris' will not be installed, because connand 'btrfs' could not be foundf
dracut module 'dmraid' will not be installed, because command 'dnraid' could not be foundf
dracut module 'ansquash-live-ntfs' will not be installed, because command 'ntfs-3g' could not be foundf
dracut module 'nultipath' will not be installed, because it's in the list to be onitted!
dracut module 'fcoe-uefi' will not be installed, because it's in the list to be onittedt
dracut module 'btris' will not be installed, because command 'btrfs' could not be found!
dracut module 'dnraid' will not be installed, because command 'dmraid' could not be found!
dracut module 'dnsquash-live-ntfs' will not be installed, because command 'ntfs-3g' could not be found!
*** Including module: bash #nw
*** Including wodule: nss-softokn *n*
*** Including module: 118n ***
w Including modulc: network NwN
*** Including module: xsnctwork ***
*** Including nodule: ifcfg *w*
** Including module: drm NNN
M Including modulc: plynouth ***
*** Including module: kernel-nodules ***
.*** Including nodule: ndraid NNM
:Skipping udeu rule: 64-nd-raid.rules
N Including modulc: scsi-dh *w*
NN Including module: rootfs-block ***
*** Including module: terminfo ***
*** Including module: udcu-rules *n*
Skipping udeu rulc: 40-redhat-cpu-hotplug.rules
:Skipping udeu rule: 91-permissions.rules
N* Including module: biosdcuname ***
Skipping udev rule: 71-biosdeunane.rules
*** Including modulc: systend ***
** Including module: usrmount ***
*** Including module: basc ***
*M* Including module: fs-1ib ***
*** Including module: shutdown # **
*** Including nodules done ***
*** Installing kernel nodule dependencies and firmware *nw
** Installing kernel nodule dependencies and firmware done ***
*** Resoluing exccutable dependencies ***
. *** Resolving executable dependencies donewww
*** Hardlinking files ***
*** Hardlinking files done ***
*** Stripping files ***
w* Stripping files donc ***
*** Generating early-microcode cpio image contents ** #
. *** Constructing AuthenticAMD,bin ****
. *** Store current conmand line paraneters ***
** Creating image file # **
. *N* Creating nicrocode section ***
. *** Created microcode section www
*** Creating image file done # **
** Creating initramfs image file '/boot/initramfs-4.19.0+1.ing' done #w*

ChatGPT's solutions are escalating, and getting downright dangerous at this point. It would be great to have a (preferably sympathetic) human help me out.

So my issue is still that:

• I boot to Dracut with a disabled keyboard
• I can boot install media to shell, but something isn't working right

Other things that I didn't mention yet that may or may not be relevent:

• the enclosure is connected with a USB-C cable to a USB-C input
• the enclosure hosts 4 disks
• Other disks/enclosures were not experiencing the disconnect issue
• The enclosure was actually working fine until a few days after I troubleshot/resolved an issue with one disk having slow transfer speeds. I switched the enclosure from USB-A to USB-C and also disabled spindown for the affected disk in the enclosure. Disabling spindown was done at the VM level, so I didn't mention it above.

[dthenot]

@nuentes Hello,

Following an IA seem to be dangerous already, no need for Skynet

There is a documentation part about regenerating the initrd: https://docs.xcp-ng.org/troubleshooting/common-problems/#initrd-is-missing-after-an-update

You can likely used what you did above to mount the XCP-ng FS and then regenerate the initrd using this command.
It's not an initramfs that you need to generate but a initrd

[nuentes]

@dthenot Thanks for getting back to me.

Yes, it seems we still have time to prepare for the robot uprising

I did boot from the initrd fallback before, and ChatGPT walked me through hosing that one as well.

I ran the command from that doc as verbose.

I ran the exact command a 2nd time as:

dracut -f --verbose /boot/initrd-4.19-xen.img 4.19-xen

No change. Boot to dracut with the keyboard not working. I've tried multiple kernels.

[nuentes]

Does anybody have any ideas?

Am I mounting the correct partition from shell? p1 and p2 look very similar, but not identical.

Why does my dracut error have so many modules that "will not be installed because x could not be found!"?

I also have metadata backups. Would these be helpful?

[bvitnik]

@nuentes said in Booting to Dracut (I trusted ChatGPT):

Other things that I didn't mention yet that may or may not be relevent:

• the enclosure is connected with a USB-C cable to a USB-C input
• the enclosure hosts 4 disks
• Other disks/enclosures were not experiencing the disconnect issue
• The enclosure was actually working fine until a few days after I troubleshot/resolved an issue with one disk having slow transfer speeds. I switched the enclosure from USB-A to USB-C and also disabled spindown for the affected disk in the enclosure. Disabling spindown was done at the VM level, so I didn't mention it above.

there is your problem

USB is very unreliable for any kind of serious data transfer. Disconnecting devices, data transfer errors, corruptions are just the tip of the iceberg. For anything reliable you have to go for network attached storage or eSATA.

I'm that type of a person that verifies md5 sums of all files copied to the USB flash, disk, enclosure etc. I've spotted data corruptions sooooo many times regardless of the OS, version, HW, USB type, storage device type... The only common thing was USB. eSATA and ethernet never produced such corruptions. As far as I'm concerned, USB is for mice and keyboards... and maybe a lamp or fan

[nuentes]

@bvitnik I'm having a hard time figuring out if you're trying to be helpful, so I'm going to assume you are.

All of my USB disks are data disks. Xcp-ng can boot just fine without them attached. My OS runs on an NVMe and a hardwired SSD with software RAID 1 (through xcp-ng).

So yes, this issue initially began as a USB issue, but that's definitely not related to why I'm unable to boot now. In fact, the USB disks have been fully disconnected during all of my troubleshooting so as not to disturb them (or their data) accidentally.

[Pilow]

@nuentes where you're at, would not be possible to simply wipe/reinstall/restore metadata ?
or rejoin pool if this is multi host pool ?

is this a single host with all vms currently sitting in the inaccessible USB ?

[nuentes]

@Pilow I run a single host, so I'm fully offline for a few days now. I know it can be recovered, I'm just not sure the right next steps. I'm just looking for the best way forward right now.

I do have a metadata backup on one of the USB hard drives. Would restoring from metadata backup actually resolve this?

[Pilow]

@nuentes i GUESS if you make a clean reinstall
then restore the metadata... you should be up & running again

BEWARE before reinstalling, be sure you can restore said metadatas ! I don't want you to be in any more troubles

[bvitnik]

@nuentes No. My intention was to rise awareness of USB (un)reliability, especially the reliability of USB attached storage. Also, either I'm blind or there is no mention of your system not being installed on USB storage.

You said everything yourself. Your problems started with USB which you assumed can be fixed by flicking some kernel parameters. In the process of "fixing", you destroyed your system. Unfortunately, I believe that the system is now beyond repair via interactive forum session because no one knows what really happened. Backup is your best friend.

[nuentes]

@bvitnik You're right. I added a lot of details, but neglected to mention that I'm not booting from USB.

I'm really not convinced I've destroyed my system. I truly think that's an over-reaction. I think I ruined my initrd and initramfs files, yes. But that should be recoverable. I haven't done nearly as much as you think I have. The reason I haven't succeeded in that yet is because I'm not really convinced I've been doing it the right way.

Since my my disks run in RAID, my system has like 6 partitions.

md127p1
md127p2
md127p3
md127p4
md127p5
md127p6

From memory, p1 and p2 are very similar. However p1 doesn't include grub (/boot/efi/EFI). P4 is grub. P2 looks very similar to p1, but it includes grub. P3 is my VHDs. P5 is maybe swap, and I can't remember what the other one is.

My point is that I don't believe that I've mounted everything correctly through the shell in order to be able to successfully chroot into the device and be able to run the dracut commands successfully. When I run the dracut commands, I see failures for applications that I can see in the sbin folder.

So there is something that I'm missing in mounting these disks in the shell that is preventing me from solving this issue. This is why I'm here. I'm not here for lectures about the dangers of USB.

Alternatively, I could boot the install media and simply perform a metadata/pool restore from backup, but I just want someone to tell me that's an actual viable option.

I'm not going to simply re-install the OS. If I do, I'll clone it first, and then boot the clone and test a metadata restore. But that's a lot of work for it to fail.

[bvitnik]

@nuentes Oh no, no. Your system is not destroyed beyond repair. It can be repaired. It's just that it is almost impossible or too much of a hustle for anyone to try to help you over forum. Someone has to sit in front of your machine to do it.

My only guess is that ChatGPT instructed you to make changes based on a CentOS system but XCP-ng and Xen virtualization in general is much different than regular CentOS. It has two stage boot process. First the Xen kernel boots and then a special virtual machine called Dom0 is booted. What you are accessing and reconfiguring is in fact this VM, not the underlying "system". So it's like a two layer system and some configuration must be done on Xen layer, some on Dom0 layer. I'm unfortunately unfamiliar with exact specifics on kernel and initrd image generation for this case so I can't spot where thing have gone wrong.

In short terms. Instead of going back and forth and trying a lot of different things, it's more time saving and simpler to reinstall the system and restore metadata if you already have a backup.

[Greg_E]

@bvitnik

Let's say you don't have the metadata backed up... Is there a way to boot from a rescue disk and save the metadata?

[bvitnik]

@Greg_E Theoretically yes. I've never been in such situation so I would have to learn, experiment and improvise along the way. If you can mount the file system of the host, you can find XAPI database in this location:

/var/lib/xcp/state.db

This is an XML formated file. I don't know if this is the same format as metadata backup (I think it's JSON instead). So you could possibly restore this file to proper location and restart the host but you would probably need to change some references in it, like UUIDs of local file systems on a freshly installed system. Other possibility is that the state file could be converted to metadata backup and imported but I'm not aware of there being any such conversion tool. It would have to be improvised.

There is always alot you can do. It just depends how deep you want to go, how complex you can go and how much time you can spare.

[nuentes]

@bvitnik I'm still unclear what metadata backs up. I suppose I don't need to know specifically, but I'm assuming it doesn't include initramfs and initrd? So a simple metadata restore would not resolve my issue, presumably?

I'm worried about doing a reinstall - I'm assuming that would wipe the disk (although I know sometimes that's not the case with Linux reinstalls). I would prefer to preserve my VMs if possible. If I'm expecting to need to wipe the disk, then I'm not 100% certain that I have backups of all of my VMs, so I'd like to clone the disk. I ran into some issues trying to run clonezilla the other night, so I might just pull the SSD and clone it to another SSD bit for bit with a disk cloning enclosure.

Just to point out - yes, my system has been offline for approaching a week now. But I've recently moved, so I have a lot of other things going on in life right now. I'd like to get it back online asap of course, but life keeps getting in the way. I feel that you are over-estimating how much I've done and mucked about. Anything I've actually performed is pretty full documented in this thread

[Pilow]

@nuentes I explored my pool metadata backups, and I see a "part.1" file, that is a big XML of all the configuration of the pool

so yeah I don't think there is something else in here like your OS parameters, only pool config.

I could be wrong, but it's all I can see in here

[AtaxyaNetwork]

@nuentes You can reinstall without wiping the SR. Just plug the ISO, it will ask you if you want a clean install or a reinstall

[Pilow]

@AtaxyaNetwork yeah but if he doesn't have metadata backups, he will have SRs full of disconnected VDIs ? and the need to recreate each VM ?

[AtaxyaNetwork]

If I was in your place, I would do:

• Disconnect all USB used as SR
• Reinstall XCP-ng
• if you have only a few VMs, my option would be just to redeploy XOA, REATTACH the SR (not create, REATTACH, be careful, a create will wipe your disk)
  • Recreate VMs, attach the VDI, boot from here
• if you have to many VM, directly after deploying XOA restore your metadata backup, it should be repluging the SR and create PDB at least (but it's been a while since I didn't use the metadata)

In any case, backup everything, especially the VDI (.vhd). If you have the VDI, you can work around with the rest

[bvitnik]

@nuentes metadata in this context is just XAPI database. In other words, it only contains information about your VMs, SRs, networks, pools etc. It does not contain anything system level. It is not a backup of the host system.

As far as I know, but someone from Vates can confirm, metadata backup functionality in XO is based on XAPI pool-dump-database command:

xe pool-dump-database file-name=dump.xml

There is some info about it here:

https://docs.xenserver.com/en-us/xenserver/8/dr/backup.html

P.S. I guess metadata backup is also XML just like XAPI state file (database). I don't know why JSON came to my mind regarding metadata backup.