After installing updates: 0 bytes free, Control domain memory = 0B
-
I'm not sure to get it, this cluster running on 8.3 beta is a production cluster?
-
@olivierlambert said in After installing updates: 0 bytes free, Control domain memory = 0B:
I'm not sure to get it, this cluster running on 8.3 beta is a production cluster?
Yeah seems like.
@Dataslak do you have a host that you can reboot without any big impact?
If yes, please do so to see if we can reproduce this issue, might be a bug in the 8.3 or just with the host that you rebooted - time will tell
-
@nikade
I do not have any more hosts I can reboot without impact (as in 10 to 20 hours of work + annoyed customers). The only one I had (008) has been used to re-construct the VMs on the failed 005 host. 005 has now become the backup-host.If I know I can get help within (a couple of) hours then I will absolutely reboot another host to see if this reproduces the issue; When do you recommend I do so? (Guesstimating what time periods are least busy for you?)
-
@olivierlambert wrote "I'm not sure to get it, this cluster running on 8.3 beta is a production cluster?
"The first two hosts I built ran 8.2 great. Then I got more customers and built three more hosts with newer components (due to shortage of the ones I used initially). But on these newer machines the installation of 8.2 failed: Black screen midway through the installation. In the forums I read about others having the same problem, solving it by using 8.3.
I do not remember what I read or thought about this being a beta, or if I believed it to be a newer version having arrived. I tried 8.3 and it has worked beautifully.Would it be an idea to inform/remind about 8.3 being in beta like this?
Stormi has recently given me an 8.2.1 image that I installed on one of the newer hosts (008) and it worked. I then spent two full days with ~2hrs sleep to re-create the VMs from 005 on it.
I will start converting all hosts to 8.2.1 as soon as I learn what level of XO I need to buy and what tool I must use in order to be able to migrate VMs from 8.3 to 8.2.
If this is not possible then I will be very happy to hear your recommendation on what to do / not do. Re-creating all VMs will be a task I hope to avoid. Would it then be best to keep running 8.3, but use a test-server to install any updates and test function before rollout?(As I get more income to this business and can pay down debts then I plan on procuring a test environment to do things on before implementing on production. If Vates would grant customers rebated licenses on such non-production servers then that would lower the threshold.)
I am very grateful and happy for your help - as well as your colleagues and contributors.
-
@Dataslak good to hear its starting to work out, always a pain when something like this happends in production.
Are those hosts in the same datacenter and/or network? If yes, you can probably migrate the VM's between your hosts to avoid downtime and having to re-create all the VM's when you want to rebuild a host.We do that at work all the time and it works pretty good, if the VM is larger than 500Gb we try to avoid it but anything below that seems to be fine. Just remember a VM migration may not go on for longer than 24h, so hence my question if the hosts are in the same network
-
-
@nikade
Thank you for your kind words!
Yes, the hosts are located physically in the same room. But due to a mix of issues with slow performance from the 1Gbps network cards (cheap non-intel cards with poor driver availability) and me not having the money to purchase XO license initially + other things and worries I have yet to gather the hosts into a pool. I will get to that when I find room to breatheThank you for the tip about VM migration can not go on >24hrs. Luckily I run small VMs of 35GB.
May I ask if you have any experience migrating hosts between 8.2.1 and 8.3?
-
@yann
With you having to rummage through lots of forum threads reading large amounts of text of high intricacy written by very different people struggling to explain their problems/questions I would be amazed if you caught 100% of intent and meaning at all times. You are human and not AI ?
I usually say that the sender has the greatest responsibility in a conversation since they know their question/issue the best. If you think you misread something then it is most likely me who failed to explain well enough. -
@Dataslak said in After installing updates: 0 bytes free, Control domain memory = 0B:
If these are possible behaviors after updates are applied then I will add this to my experience and not be so worried next time.
It's definitely possible behaviour that an update requires a reboot immediately after applying. But it's rare, limited to major upgrades, and here it happened because it is still a beta release.
-
So, to summarize:
- The root cause of the symptoms on host 005 is grub booting from an unsynced disk. Why exactly this happened is still a mystery, unless I've skipped important data when I catched up with the messages above.
- There's no reason to think this would happen to the other hosts, since we established that the disks are correctly synced. A reboot should be safe and not cause the same issues as what happened on 005.
- To limit the risks, it would be good to migrate VMs away from one host before rebooting it. However, with the major change in version in the Xen component, I can't promise migration will really work as expected (VM crashing, VM refusing to move in one direction or the other...).
The safest is probably to fix host 005 first. I assume removing the bad disk and then booting should be enough?
-
@Dataslak said in After installing updates: 0 bytes free, Control domain memory = 0B:
@nikade
Thank you for your kind words!
Yes, the hosts are located physically in the same room. But due to a mix of issues with slow performance from the 1Gbps network cards (cheap non-intel cards with poor driver availability) and me not having the money to purchase XO license initially + other things and worries I have yet to gather the hosts into a pool. I will get to that when I find room to breatheThank you for the tip about VM migration can not go on >24hrs. Luckily I run small VMs of 35GB.
May I ask if you have any experience migrating hosts between 8.2.1 and 8.3?
From 8.2.1 should be fine, from 8.3 to 8.2.1 might be a problem.
I think that if your VM's are only 35Gb it should be fine to migrate them between hosts even on 1Gbps network, also you can do it without having them in a pool so I think you should try with a test VM and see how long it takes. -
@nikade @olivierlambert @stormi @Danp @yann
Just wanted to say to you all:
Thank you for your contributions and kind helpful assistance which has helped me through this crisis.
I would have been in deep trouble without you. I respect your expertise, and appreciate deeply that you are working so hard to help us dumb users. I have learned a lot, and hope one day to become skilled enough to at least help other new users on this forum.
Best wishes
Aslak -
@Dataslak said in After installing updates: 0 bytes free, Control domain memory = 0B:
@nikade @olivierlambert @stormi @Danp @yann
Just wanted to say to you all:
Thank you for your contributions and kind helpful assistance which has helped me through this crisis.
I would have been in deep trouble without you. I respect your expertise, and appreciate deeply that you are working so hard to help us dumb users. I have learned a lot, and hope one day to become skilled enough to at least help other new users on this forum.
Best wishes
AslakHappy everything worked out, this is what this community is all about.
I've gotten a lot of help and given some too, it's all about helping out with the things that you can.
With time you'll be able to help out more and more and more
