@olivierlambert,
I ended up copying the old application data from the crashed VM via a live CD and deployed a new VM (running on HVM.) Everything seems to be working fine now.

@olivierlambert,
Thanks for the assistance. I'm exporting the data from the VM via Live CD and will build a new VM and import applications from old VM.

@olivierlambert,
Is there a simple way to start the VM with PVshim? After googling around I couldn't find anything that would explain how to do this for the non-specialized person.

@olivierlambert,
This is an x86_64 VM. I originally made the VM on XenServer back when it was Citrix owned. Then I actually migrated it over to KVM about 7 years ago. And then about 3 years ago, I migrated the VM back to XCP-ng using a qcow conversion routine that included running a dracut regenerate command.

dracut --regenerate-all -f && grub2-mkconfig -o /boot/grub2/grub.cfg

I wonder if I could somehow start in recovery mode and use that same command to regenerate the VM?

@olivierlambert , thank you for your reply.

I did switch to HVM, however that's when I get the grub bootloader screen with the Error 13: Invalid or unsupported executable format.

So, PVShim might work in this case? I will give it a shot.

@Danp ,

The crazy thing is is these are old centos 5.5 vms. They haven't had any patches for ages. Neither have the hosts been patched for quite a while...

That's what I would think too but seems unlikely in this case...

Greetings,

After a clean shutdown yesterday, two older CentOS 5 VM that have been running fine for many years refused to come back up. I'm running XCP 7.6.

The VM CPU spikes to 100%, and I get "Error 13: Invalid or unsupported executable format" in the console.

I can't seem to restore from a recent (3 day old) snapshot either, same problem.

After doing some searching and trying launching from Grub, I'm stuck.

Can anyone shed some light on this?

Thanks for your insight.

@d_j , man that's weird.

It hasn't happened again, but it happened 3 months in a row. Dec. - Feb.

Super weird.

@danp and @olivierlambert ,

Thanks for the response. There are no logs in the guest VM, just a cutoff in /var/log/messages.

I am using XO, and using Delta Backups, but they were not happening on the 24th consistently for the last 3 months, only last month did a backup happen on the 24th. They're scheduled on Mon., Wed., and Fri.

Yes, this is a very odd thing. I've been running Xen VMs since 2007 and have never seen this kind of thing.

Greetings,

We have a VM that has started to hard shutdown on the 24th of every month for the last 3 months. There's nothing in the crontab (CentOS 7), and the xen logs show things like below. Can anyone give us some advice on where to start looking for solutions here? Thank you for any assistance you can render.

Feb 24 23:59:42 monota xenopsd-xc: [debug|monota|7 ||xenops] EPOLL error on domain-17, close QMP socket
Feb 24 23:59:43 monota xenopsd-xc: [debug|monota|17 |Parallel:task=3310620.atoms=2.(VBD.unplug vm=6e91bf5a-6f02-4597-6e37-30698b4e539c)|xenops] Device.Generic.hard_shutdown about to blow away backend and error paths
Feb 24 23:59:43 monota xenopsd-xc: [debug|monota|15 |Parallel:task=3310620.atoms=2.(VBD.unplug vm=6e91bf5a-6f02-4597-6e37-30698b4e539c)|xenops] Device.Generic.hard_shutdown about to blow away backend and error paths
Feb 24 23:59:43 monota xenopsd-xc: [debug|monota|17 |Parallel:task=3310620.atoms=2.(VBD.unplug vm=6e91bf5a-6f02-4597-6e37-30698b4e539c)|xenops] xenstore-rm /local/domain/0/error/backend/vbd3/17
Feb 24 23:59:43 monota xenopsd-xc: [debug|monota|17 |Parallel:task=3310620.atoms=2.(VBD.unplug vm=6e91bf5a-6f02-4597-6e37-30698b4e539c)|xenops] xenstore-rm /local/domain/17/error/device/vbd/768
Feb 24 23:59:43 monota xenopsd-xc: [debug|monota|15 |Parallel:task=3310620.atoms=2.(VBD.unplug vm=6e91bf5a-6f02-4597-6e37-30698b4e539c)|xenops] xenstore-rm /local/domain/0/error/backend/vbd3/17
Feb 24 23:59:43 monota xenopsd-xc: [debug|monota|15 |Parallel:task=3310620.atoms=2.(VBD.unplug vm=6e91bf5a-6f02-4597-6e37-30698b4e539c)|xenops] xenstore-rm /local/domain/17/error/device/vbd/5696
Feb 24 23:59:44 monota xenopsd-xc: [debug|monota|10 |events|xenops] Device.Generic.hard_shutdown about to blow away backend and error paths
Feb 24 23:59:44 monota xenopsd-xc: [debug|monota|10 |events|xenops] xenstore-rm /local/domain/0/error/backend/vif/17
Feb 24 23:59:44 monota xenopsd-xc: [debug|monota|10 |events|xenops] xenstore-rm /local/domain/17/error/device/vif/0
Feb 24 23:59:45 monota xapi: [debug|monota|561 |xapi events D:34fe87c9d9cc|helpers] Helpers.call_api_functions failed to logout: Server_error(SESSION_INVALID, [ OpaqueRef:ce31d273-1eaf-440b-8632-69a6282b278c ]) (ignoring)

@julien-f

Thanks for looking at this:

2019-09-11T08_00_00.003Z - backup NG.txt

@julien-f , here is the error message from the Backup NG report

Snapshot 
Start: Sep 11, 2019, 01:00:00 AM
End: Sep 11, 2019, 01:00:06 AM
 XenBackupsUnify 
 transfer 
Start: Sep 11, 2019, 01:00:07 AM
End: Sep 11, 2019, 01:07:53 AM
Duration: 8 minutes
Size: 2.38 GiB
Speed: 5.22 MiB/s
 merge 
Start: Sep 11, 2019, 01:07:53 AM
End: Sep 11, 2019, 01:11:03 AM
Duration: 3 minutes
Error: "value" argument is out of bounds
Start: Sep 11, 2019, 01:00:07 AM
End: Sep 11, 2019, 01:11:03 AM
Duration: 11 minutes
Error: "value" argument is out of bounds

@julien-f , if it's helpful, I recently increased the size of the VMs disk by 40 GB.

@julien-f , what do you need from me to provide more info? Which logs can I get that info from?

Pending your response, I will start poking around the logs myself, and see if I can turn up anything useful.

@olivierlambert , I should add that these backups have been running for weeks, 3 times a week, with no problems, this is the first time we've seen any indication that there's an issue.

Thank you, @olivierlambert

I am using this github script to install XOA https://github.com/Jarli01/xenorchestra_installer

And my XO version is: 5.44.0

Greetings,

I got an alert this morning that a Backup NG scheduled backup on a VM had failed with the error: "value" argument is out of bounds.

I am unable to find anything on-line related to this error. Can anyone shed light on this or where to start looking for answers? Thank you.

Asai

Thank you, @olivierlambert, for all you do here.

OK, thanks, I'm thinking about using 2 remotes for redundancy.

Greetings,

I just had a quick question about using multiple remotes on Backup NG. Basically, if you choose to use multiple remotes, does it simply make a copy of the backups to each remote?

Thanks,
Asai