daily Continuous Replication backup in XO
-
-
We've just merged a potential fix, if you could try it that would be great
-
@julien-f Unfortunately, my experience with this latest fix has not been a positive one --
- GUI is unresponsive (spinning graphic only)
- Backup job kicks off improperly (scheduled for midnight)
- Weekly XO report has been generated multiple times
- Timeout show in logs
Oct 25 12:48:31 ubuntutest xo-server[93776]: 2019-10-25T12:48:31.583Z xo:perf INFO blocked for 28525ms Oct 25 12:49:01 ubuntutest xo-server[93776]: _watchEvents { TimeoutError: operation timed out Oct 25 12:49:01 ubuntutest xo-server[93776]: at Promise.timeout (/opt/xen-orchestra/node_modules/promise-toolbox/timeout.js:13:16 ) Oct 25 12:49:01 ubuntutest xo-server[93776]: at /opt/xen-orchestra/packages/xen-api/src/index.js:620:36 Oct 25 12:49:01 ubuntutest xo-server[93776]: at Generator.next (<anonymous>) Oct 25 12:49:01 ubuntutest xo-server[93776]: at asyncGeneratorStep (/opt/xen-orchestra/packages/xen-api/dist/index.js:58:103) Oct 25 12:49:01 ubuntutest xo-server[93776]: at _next (/opt/xen-orchestra/packages/xen-api/dist/index.js:60:194) Oct 25 12:49:01 ubuntutest xo-server[93776]: at /opt/xen-orchestra/packages/xen-api/dist/index.js:60:364 Oct 25 12:49:01 ubuntutest xo-server[93776]: at Promise._execute (/opt/xen-orchestra/node_modules/bluebird/js/release/debuggabili ty.js:427:9) Oct 25 12:49:01 ubuntutest xo-server[93776]: at Promise._resolveFromExecutor (/opt/xen-orchestra/node_modules/bluebird/js/release /promise.js:518:18) Oct 25 12:49:01 ubuntutest xo-server[93776]: at new Promise (/opt/xen-orchestra/node_modules/bluebird/js/release/promise.js:103:1 0) Oct 25 12:49:01 ubuntutest xo-server[93776]: at /opt/xen-orchestra/packages/xen-api/dist/index.js:60:97 Oct 25 12:49:01 ubuntutest xo-server[93776]: at Xapi._call (/opt/xen-orchestra/packages/xen-api/src/index.js:617:71) Oct 25 12:49:01 ubuntutest xo-server[93776]: at /opt/xen-orchestra/packages/xen-api/src/index.js:958:30 Oct 25 12:49:01 ubuntutest xo-server[93776]: at Generator.next (<anonymous>) Oct 25 12:49:01 ubuntutest xo-server[93776]: at asyncGeneratorStep (/opt/xen-orchestra/packages/xen-api/dist/index.js:58:103) Oct 25 12:49:01 ubuntutest xo-server[93776]: at _next (/opt/xen-orchestra/packages/xen-api/dist/index.js:60:194) Oct 25 12:49:01 ubuntutest xo-server[93776]: at tryCatcher (/opt/xen-orchestra/node_modules/bluebird/js/release/util.js:16:23) Oct 25 12:49:01 ubuntutest xo-server[93776]: at Promise._settlePromiseFromHandler (/opt/xen-orchestra/node_modules/bluebird/js/re lease/promise.js:547:31) Oct 25 12:49:01 ubuntutest xo-server[93776]: at Promise._settlePromise (/opt/xen-orchestra/node_modules/bluebird/js/release/promi se.js:604:18) Oct 25 12:49:01 ubuntutest xo-server[93776]: at Promise._settlePromiseCtx (/opt/xen-orchestra/node_modules/bluebird/js/release/pr omise.js:641:10) Oct 25 12:49:01 ubuntutest xo-server[93776]: at _drainQueueStep (/opt/xen-orchestra/node_modules/bluebird/js/release/async.js:97: 12) Oct 25 12:49:01 ubuntutest xo-server[93776]: at _drainQueue (/opt/xen-orchestra/node_modules/bluebird/js/release/async.js:86:9) Oct 25 12:49:01 ubuntutest xo-server[93776]: at Async._drainQueues (/opt/xen-orchestra/node_modules/bluebird/js/release/async.js: 102:5) Oct 25 12:49:01 ubuntutest xo-server[93776]: at Immediate.Async.drainQueues [as _onImmediate] (/opt/xen-orchestra/node_modules/bl uebird/js/release/async.js:15:14) Oct 25 12:49:01 ubuntutest xo-server[93776]: at runCallback (timers.js:810:20) Oct 25 12:49:01 ubuntutest xo-server[93776]: at tryOnImmediate (timers.js:768:5) Oct 25 12:49:01 ubuntutest xo-server[93776]: at processImmediate [as _immediateCallback] (timers.js:745:5) Oct 25 12:49:01 ubuntutest xo-server[93776]: call: Oct 25 12:49:01 ubuntutest xo-server[93776]: { method: 'event.from', Oct 25 12:49:01 ubuntutest xo-server[93776]: params: [ [Array], '00000000000088155376,00000000000088111756', 60.1 ] }, Oct 25 12:49:01 ubuntutest xo-server[93776]: [Symbol(originalCallSite)]: Oct 25 12:49:01 ubuntutest xo-server[93776]: [ CallSite {}, Oct 25 12:49:01 ubuntutest xo-server[93776]: CallSite {}, Oct 25 12:49:01 ubuntutest xo-server[93776]: CallSite {} ], Oct 25 12:49:01 ubuntutest xo-server[93776]: [Symbol(mutatedCallSite)]: Oct 25 12:49:01 ubuntutest xo-server[93776]: [ CallSite {}, Oct 25 12:49:01 ubuntutest xo-server[93776]: CallSite { Oct 25 12:49:01 ubuntutest xo-server[93776]: constructor: [Function: CallSite], Oct 25 12:49:01 ubuntutest xo-server[93776]: getColumnNumber: [Function], Oct 25 12:49:01 ubuntutest xo-server[93776]: getEvalOrigin: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getFileName: [Function], Oct 25 12:49:01 ubuntutest xo-server[93776]: getFunction: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getFunctionName: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getLineNumber: [Function], Oct 25 12:49:01 ubuntutest xo-server[93776]: getMethodName: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getPosition: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getScriptNameOrSourceURL: [Function], Oct 25 12:49:01 ubuntutest xo-server[93776]: getThis: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getTypeName: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: isConstructor: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: isEval: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: isNative: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: isToplevel: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: toString: [Function: CallSiteToString] }, Oct 25 12:49:01 ubuntutest xo-server[93776]: CallSite {}, Oct 25 12:49:01 ubuntutest xo-server[93776]: CallSite { Oct 25 12:49:01 ubuntutest xo-server[93776]: constructor: [Function: CallSite], Oct 25 12:49:01 ubuntutest xo-server[93776]: getColumnNumber: [Function], Oct 25 12:49:01 ubuntutest xo-server[93776]: getEvalOrigin: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getFileName: [Function], Oct 25 12:49:01 ubuntutest xo-server[93776]: getFunction: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getFunctionName: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getLineNumber: [Function], Oct 25 12:49:01 ubuntutest xo-server[93776]: getMethodName: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getPosition: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getScriptNameOrSourceURL: [Function], Oct 25 12:49:01 ubuntutest xo-server[93776]: getThis: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getTypeName: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: isConstructor: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: isEval: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: isNative: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: isToplevel: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: toString: [Function: CallSiteToString] }, Oct 25 12:49:01 ubuntutest xo-server[93776]: CallSite { Oct 25 12:49:01 ubuntutest xo-server[93776]: constructor: [Function: CallSite], Oct 25 12:49:01 ubuntutest xo-server[93776]: getColumnNumber: [Function], Oct 25 12:49:01 ubuntutest xo-server[93776]: getEvalOrigin: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getFileName: [Function], Oct 25 12:49:01 ubuntutest xo-server[93776]: getFunction: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getFunctionName: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getLineNumber: [Function], Oct 25 12:49:01 ubuntutest xo-server[93776]: getMethodName: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getPosition: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getScriptNameOrSourceURL: [Function], Oct 25 12:49:01 ubuntutest xo-server[93776]: getThis: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: getTypeName: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: isConstructor: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: isEval: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: isNative: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: isToplevel: [Object], Oct 25 12:49:01 ubuntutest xo-server[93776]: toString: [Function: CallSiteToString] }, Oct 25 12:49:01 ubuntutest xo-server[93776]: CallSite {}, Oct 25 12:49:01 ubuntutest xo-server[93776]: CallSite {} ] } Oct 25 12:49:01 ubuntutest xo-server[93776]: 2019-10-25T12:49:01.999Z xo:perf INFO blocked for 30316ms- System crashes / restarts
Oct 25 12:50:57 ubuntutest xo-server[93776]: <--- Last few GCs ---> Oct 25 12:50:57 ubuntutest xo-server[93776]: [93776:0x2d80740] 454655 ms: Mark-sweep 1367.1 (1460.1) -> 1367.0 (1460.1) MB, 3439.6 / 0.1 ms allocation failure GC in old space requested Oct 25 12:50:57 ubuntutest xo-server[93776]: [93776:0x2d80740] 457817 ms: Mark-sweep 1367.0 (1460.1) -> 1367.0 (1427.1) MB, 3161.9 / 0.9 ms last resort GC in old space requested Oct 25 12:50:57 ubuntutest xo-server[93776]: [93776:0x2d80740] 459992 ms: Mark-sweep 1367.0 (1427.1) -> 1367.0 (1427.1) MB, 2175.0 / 0.0 ms last resort GC in old space requested Oct 25 12:50:57 ubuntutest xo-server[93776]: <--- JS stacktrace ---> Oct 25 12:50:57 ubuntutest xo-server[93776]: ==== JS stack trace ========================================= Oct 25 12:50:57 ubuntutest xo-server[93776]: Security context: 0x305f21fa5891 <JSObject> Oct 25 12:50:57 ubuntutest xo-server[93776]: 1: /* anonymous */ [/opt/xen-orchestra/packages/xo-server/dist/xo-mixins/logs/index.js:~63] [pc=0x148d528b15e5](this=0x38730f1c8101 <Stream map = 0x2677818c9d71>,data=0x161e745a92a9 <Object map = 0x32a34867fbe9>) Oct 25 12:50:57 ubuntutest xo-server[93776]: 2: emitOne(aka emitOne) [events.js:~114] [pc=0x148d528c5f98](this=0x956b0d822d1 <undefined>,handler=0x38730f1ded49 <JSFunction logger.createReadStream.on.data (sfi = 0x5c6aef025... Oct 25 12:50:57 ubuntutest xo-server[93776]: FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory Oct 25 12:50:57 ubuntutest xo-server[93776]: 1: node::Abort() [/usr/local/bin/node] Oct 25 12:50:57 ubuntutest xo-server[93776]: 2: 0x8cd14c [/usr/local/bin/node] Oct 25 12:50:57 ubuntutest xo-server[93776]: 3: v8::Utils::ReportOOMFailure(char const*, bool) [/usr/local/bin/node] Oct 25 12:50:57 ubuntutest xo-server[93776]: 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node] Oct 25 12:50:57 ubuntutest xo-server[93776]: 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [/usr/local/bin/node] Oct 25 12:50:57 ubuntutest xo-server[93776]: 6: v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::New(v8::internal::Isolate*, int, v8::internal::PretenureFlag, v8::internal::MinimumCapacity) [/usr/local/bin/node] Oct 25 12:50:57 ubuntutest xo-server[93776]: 7: v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::EnsureCapacity(v8::internal::Handle<v8::internal::NameDictionary>, int, v8::internal::PretenureFlag) [/usr/local/bin/node] Oct 25 12:50:57 ubuntutest xo-server[93776]: 8: v8::internal::BaseNameDictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::EnsureCapacity(v8::internal::Handle<v8::internal::NameDictionary>, int) [/usr/local/bin/node] Oct 25 12:50:57 ubuntutest xo-server[93776]: 9: v8::internal::BaseNameDictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape>::Add(v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, int*) [/usr/local/bin/node] Oct 25 12:50:57 ubuntutest xo-server[93776]: 10: v8::internal::LookupIterator::ApplyTransitionToDataProperty(v8::internal::Handle<v8::internal::JSObject>) [/usr/local/bin/node] Oct 25 12:50:57 ubuntutest xo-server[93776]: 11: v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node] Oct 25 12:50:57 ubuntutest xo-server[93776]: 12: v8::internal::Object::SetProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::LanguageMode, v8::internal::Object::StoreFromKeyed) [/usr/local/bin/node] Oct 25 12:50:57 ubuntutest xo-server[93776]: 13: v8::internal::Runtime_SetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node] Oct 25 12:50:57 ubuntutest xo-server[93776]: 14: 0x148d525842fd Oct 25 12:50:58 ubuntutest systemd[1]: xo-server.service: Main process exited, code=dumped, status=6/ABRT Oct 25 12:50:58 ubuntutest systemd[1]: xo-server.service: Failed with result 'core-dump'. Oct 25 12:50:58 ubuntutest systemd[1]: xo-server.service: Service RestartSec=100ms expired, scheduling restart. Oct 25 12:50:58 ubuntutest systemd[1]: xo-server.service: Scheduled restart job, restart counter is at 1. Oct 25 12:50:58 ubuntutest systemd[1]: Stopped XO Server. Oct 25 12:50:58 ubuntutest systemd[1]: Started XO Server. Oct 25 12:51:08 ubuntutest xo-server[93989]: 2019-10-25T12:51:08.660Z app-conf /opt/xen-orchestra/packages/xo-server/config.toml Oct 25 12:51:08 ubuntutest xo-server[93989]: 2019-10-25T12:51:08.692Z xo:main INFO Configuration loaded. -
Thanks for your feedback, I'm not sure it's related to the fix itself but we are investigating
-
Indeed, there is a major issue with the fix, I'm on it.
Sorry for this
-
The fix has been merged
@Danp Sorry for the inconvenience and thank you for your feedback.
-
@julien-f Much better, thanks!
Only minor issue I observed is that there appears to be a minute delay between the time the job starts and the UI is updated to reflect that the job is active.
-
@Danp Probably unrelated, do you mind creating an issue on GitHub for this? We'll investigate.
-
@Danp Are you talking abouth Legacy backups or Backup NG?
-
@julien-f Backup NG
-
Hi
I had exatly the same problem when I upgrade one of my four server from xenserver 7.1 to xcp-ng 8.0, one backup failed with error: job already running and the other finish normaly later!
I upgrade Xo from 5.51 to 5.54 and it's Ok for this problem BUT I have a new problem, the cold backup to NFS and the Continuous replication to a Xcp-ng 8 server are REALLY slow !! One exemple:
Start time: Thursday, December 19th 2019, 7:20:00 am
End time: Thursday, December 19th 2019, 7:58:54 am
Duration: 39 minutes
Snapshot
Start time: Thursday, December 19th 2019, 7:20:00 am End time: Thursday, December 19th 2019, 7:20:05 am Duration: a few secondsSRs
Local storage (a70a93c4-4951-9bec-54ab-d45bb5039b44) ✔ Start time: Thursday, December 19th 2019, 7:20:05 am End time: Thursday, December 19th 2019, 7:58:52 am Duration: 39 minutes transfer ✔ Start time: Thursday, December 19th 2019, 7:20:06 am End time: Thursday, December 19th 2019, 7:58:50 am Duration: 39 minutes Size: 1.34 GiB Speed: 605.55 KiB/sI try to upgrade debian OS, upgrade VM (4cpus 3Go ram) but same problem for the moment
Thanks in advance
Edit: I do a manual test by copy an iso of 2G locally on the virtual machine Xen orchestra, to the same connected NFS for backup, and the performance is normal, 10 seconds for 2Go, it seems to be a problem with Xen orchestra system backup?
