XO task watcher issue/CR broken

julien-f

Thanks to you both, I have what I need to work on this for the moment, I'll keep you posted when I have something more to test

Andrew

@julien-f Any new thoughts on this issue?

julien-f

@Andrew Yes, I'm working on this and should have something to test tomorrow

julien-f

@Andrew @Gheppy, I have something new test, it's an important change concerning a low-level component of XO, hopefully I did not break anything

The cr-issue branch has been rebased, make sure to reset it properly, re-install dependencies and rebuild:

cd xen-orchestra/
git checkout cr-issue
git fetch
git reset --hard origin/cr-issue
yarn
yarn build

Let me know if you have any issues

Gheppy

@julien-f
I'll tested in a short time, I am in the middle of CR for next 6 hours

Andrew

@julien-f Broken. Job failed. Lots of Error: Premature close. Data not replicated.

Job report: job.txt (text format)

Journal XO output: out.txt.gz (GZ format).

Gheppy

same error

CR log
CR-log.txt

XOCE log
Srv-log.txt

julien-f

I still don't understand exactly the issue, not sure if it comes from XO or XCP-ng/XenServer, but latest version does integrate a work-around, if you encounter Premature close error during CR, you can add the following to your xo-server's configuration file (usually /etc/xo-server/config.toml) :

[xapiOptions]
ignorePrematureClose = true

It's not enabled by default until completely understand the root cause and it's properly fixed.

@Gheppy Thanks a lot for your tests and feedbacks

@Andrew Thank you very much for the test appliance, it was an invaluable help investigating this. If you can keep it online for the time being I'll probably have further tests to do with it next week

Andrew

@julien-f I updated XO Source to current master and added the new ignorePrematureClose=true option. Backup ran the CR correctly again.

Yes, I can leave the XOA test tunnel up for testing. I'm happy to help you help me!

olivierlambert

@Andrew do you still have the issue without the ignorePrematureClose?

Andrew

@olivierlambert Yes. Still problems on the new code without the option set. 90% of the VMs fail 10% finish correctly on CR.

olivierlambert

Thanks for your precious feedback

Gheppy

Just for info
I installed the latest version of XOCE, commit 083db67df9e1645a2f8fe2fac564b3aecf30d55e
CR is ok and can be used in case of disaster, with ignorePrematureClose = true.
I managed to start the VM without problems, the VM itself ( that is the CR copy ) .
At the moment I have a very slow copy problem of the VM that is created with CR on the seccond server ( 18Mb max ), the problem is that the same connection goes with 400Mb for a CR copy. I want to make a copy of VM-CR and start it

magicker

@Gheppy weird.. DR is now working for me but CR is still never ending

Gheppy

@magicker
You need to add this on config file.
Location for me is /opt/xen-orchestra/packages/xo-server/.xo-server.toml

[xapiOptions]
ignorePrematureClose = true

magicker

@Gheppy Yes.. putting that in place seemed to fix DR.. but not CR.

olivierlambert

Any log or something @magicker ? Are you sure it's a CR started after modifying the config and restarting xo-server?

magicker

@olivierlambert actually.. just tried CR again... and this time it is working.. will test with a few more vms.

magicker

@olivierlambert spoke too soon.. CR works in one direction but not the other
https://images.dx3webs.com/kgSC4E.png

julien-f

@magicker Any differences between the two hosts? XCP-ng versions maybe?

Solved XO task watcher issue/CR broken