Delta backup fails on SMB remote

[johnarvid]

Hi,
I had an issue when I tried to run a delta backup in my environment. (First time, so when it does a full backup)

I'm running from the sources.
xo-server 5.78.2
xo-web 5.80.0
node v14.15.5

The backup job takes a snapshot, moves the vdi to the remote location but then it fails. The snapshot is still present, and the vdi file looks complete.

When I did a "normal" backup instead of delta it completed successfully.

Error message: The expression evaluated to a falsy value: assert(offset > 0)

Should I create a issue on github for this?

Screenshot of the remote:

Here is the log from the backup job:
2021-04-08T09_49_51.216Z - backup NG.txt

[olivierlambert]

Are you on the latest master commit?

[johnarvid]

@olivierlambert Yeah, I just read through some forum posts and saw that the version numbers did not mean much anymore. I have to test some more in the container were it is running.

[johnarvid]

Did another test now with the latest master.

Now I got another error message: EINVAL: invalid argument, open '/run/xo-server/mounts/93d6810c-711c-4c13-b5ba-9009f3d75933/xo-vm-backups/f8d7e336-57ac-5d98-7825-98c722ef927c/vdis/7e734662-59e8-45cd-a5e0-3abedeecf0dc/621d81e1-0ea2-4155-a7d4-0966b607fb60/20210409T130439Z.vhd'

Log file: 2021-04-09T13_04_30.972Z - backup NG.txt

FYI: While I did this test I also ran a normal backup from another xen-orchestra instance. The VM that I ran a delta backup on was not currently worked with by the normal backup.

[olivierlambert]

Are you sure you can access the VHD file reported by the error?

[johnarvid]

I will try to import it and check.
But does it matter? Normal backup works, delta backup does not.

[johnarvid]

But yes, the VDI that was mentioned in the error is working as expected.
Let me know if there is anything else I should try.

[olivierlambert]

From XO, what's the result of:

ls -lah '/run/xo-server/mounts/93d6810c-711c-4c13-b5ba-9009f3d75933/xo-vm-backups/f8d7e336-57ac-5d98-7825-98c722ef927c/vdis/7e734662-59e8-45cd-a5e0-3abedeecf0dc/621d81e1-0ea2-4155-a7d4-0966b607fb60/20210409T130439Z.vhd`

[johnarvid]

@olivierlambert Do you mean from xcp-ng?
ls: cannot access .... No such file or directory
There is no xo-server folder in /run/

[olivierlambert]

Does it ring any bell @julien-f ?

[julien-f]

Maybe your remote is not mounted, you can try to disable/enable the remote.

[johnarvid]

@julien-f Do you mean that the remote is not mounted when a normal backup completes successfully but a Delta backup fails?
How can I go forward?

[julien-f]

@johnarvid I was answering the ls problem, not the initial message of the thread

Backup issues should be self correcting, does the next run not fix the situation for you?

If not, you can take a look at the integrity of your backup files: https://xcp-ng.org/forum/topic/2994/check-backup-integrity/5

[johnarvid]

@julien-f Ok, understand:)
I missunderstood @olivierlambert "From XO" description. I ran that command from the xcp-ng server, and not from the XO server.

I ran it now and the ls command is successfull.

A next run does not fix the situasion.

vhd-cli check returns OK.
Do you want me to run the xo-backups clean-vms also?

[DeOccultist]

@johnarvid That error looks like the error that I've been getting... https://xcp-ng.org/forum/topic/4353/delta-backup-failed-invalid-argument

I've switched back to full backups for now.
I'm still investigating and troubleshooting though so no solutions yet...

[johnarvid]

@deoccultist Thanks for the reply.
I have tried all the steps that @julien-f suggested to you in the last post in your topic.
Guess I have to wait until they have something new I should try.

[johnarvid]

I have tested with a clean install of xcp-ng with the latest patches. (Just to make sure it was nothing there causing a problem)
But the issue is still there.

[johnarvid]

I can add information that using a share on windows 10 as remote causes this error.
But using samba on linux with this config is ok:

[global]
  workgroup = BERG
  server string = ember
  security = user
  guest ok = no
  map to guest = Bad Password
  log file = /var/log/samba/%m.log
  max log size = 50
  printcap name = /dev/null
  load printers = no
  min protocol = SMB2
  ntlm auth = yes

[localNetwork]
  comment = Local Network Storage
  path = /mnt/storage/localNetwork
  browsable = yes
  read only = no
  guest ok = no
  valid users = local,john-arvid
  force group = samba
  force user = local

[olivierlambert]

So could it be permission related on Windows share?

[DeOccultist]

@olivierlambert I don't think it's a permission error since the same Windows Server 2019 SMB share works fine with full back ups.