XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Backup Fail: Trying to add data in unsupported state

    Scheduled Pinned Locked Moved Backup
    38 Posts 7 Posters 4.6k Views 7 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N Offline
      nmadunich
      last edited by

      I am having the same issue with backups. I get the same error Trying to add data in unsupported state and I see the same log entry that @nvoss posted. I think this is related to the backup encryption.

      I did some testing after seeing this post. I created a new NFS remote and my VM backed up just fine without encryption. I had never successfully backed up this VM before. I deleted out the backup then cleared out the NFS file directory then added encryption. My VM failed to backup after adding the encryption.

      My VM has only a single drive and it's Windows 10 and I have another Win10 VM that backs up fine under the same job. I don't think it's drive numbers or OS related.

      I have some more testing tomorrow and will update my post.

      Thanks
      Nick

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        So it seems to be something very specific with encryption, right?

        nvossN 1 Reply Last reply Reply Quote 0
        • nvossN Offline
          nvoss @olivierlambert
          last edited by

          @olivierlambert that's my experience as well, but it's inconsistent in that I have like 6 other VMs on the same backup that work fine to the encrypted repos. In this case I have two -- one that's a local synology and one that's a remote s3 compatible Wasabi. Both show the same failure, but only with these 2 VMs.

          1 Reply Last reply Reply Quote 0
          • N Offline
            nmadunich
            last edited by

            @olivierlambert I can confidently say that it's an issue with encrypting the VM during backups. I was using 2 Windows 10 VMs for testing. 1 would fail every time I enabled encryption on the backup and 1 would backup with and without encryption enabled. I was able to reproduce this multiple times. I am still trying to figure out what the difference between these 2 VMs is that would cause this issue.

            Both VMs are using BIOS boot firmware and I am running full backups. Both of these VMs were imported from VMWare and have the guest tools installed. We are using NetApp NFS storage and AWS S3 for our remotes so I don't think it's storage vendor related.

            I updated XOA this morning from 5.95.1 to 5.95.2 and the backup still failed.

            1 Reply Last reply Reply Quote 0
            • olivierlambertO Offline
              olivierlambert Vates 🪐 Co-Founder CEO
              last edited by

              Do you have the same behavior on XOA on latest release channel?

              nvossN 1 Reply Last reply Reply Quote 0
              • nvossN Offline
                nvoss @olivierlambert
                last edited by

                @olivierlambert I'm using XO built from sources. Current commit level: 88c7c. I pulled those updates down ~2 days ago I think.

                1 Reply Last reply Reply Quote 0
                • N Offline
                  nmadunich
                  last edited by

                  I updated XOA to the latest channel 5.97.0 and I still have the same issue. I can backup the VM without encryption and then it fails when encryption is enabled. We did purchase licensing from VATES so it's not an issue with being built from source.

                  olivierlambertO 1 Reply Last reply Reply Quote 0
                  • D Offline
                    Delgado
                    last edited by

                    I can confirm I am seeing the same thing with full a mirror full backup job to Backblaze B2 utilizing encryption. I am on Master, commit 732ca in my homelab. I have attached the backup log as well.2024-08-28T22_32_37.271Z - backup NG.json.txt

                    1 Reply Last reply Reply Quote 0
                    • olivierlambertO Offline
                      olivierlambert Vates 🪐 Co-Founder CEO
                      last edited by

                      Thanks for the feedback, let me ping @florent internally

                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO @nmadunich
                        last edited by

                        @nmadunich and same on XOA stable?

                        1 Reply Last reply Reply Quote 0
                        • florentF Online
                          florent Vates 🪐 XO Team
                          last edited by florent

                          Hi,

                          what are the size of the failing VMs ? is there anything in the syslog before having the cipher error message ?

                          To be fair, uploading full backup ( > 50GB) , without knowing the size first is full of hurdles. And the xapi dont tell the size of the exports. Incremental backup with block storage completly circumvent this, by uploading separate blocks of known size

                          Regards

                          1 Reply Last reply Reply Quote 0
                          • N Offline
                            nmadunich
                            last edited by nmadunich

                            @olivierlambert Yes I get the same result on stable vs latest channel.

                            @florent The ones that fail do seem to be some of my larger VMs the Windows 10 VM that I have been testing with is about 88.3 GB used according to the OS.

                            All of my VMs are thin provisioned and our NetApp storage is using de-duplication so the size of the VHD on my storage is significantly less in this case it was about 3 GBs.

                            As a test I created a new storage volume without thin provisioning and de-duplication. I migrated the Disk to the new volume and the VHD is 103 GB. I also removed de-duplication and compression on my remote. I tried the backup again and it failed with the same error.

                            I do see some errors from the xensource.log around the time it fails and I attached those here.

                            xensource errors.txt

                            I am editing my post after looking at the log file @Delgado posted mine are slightly different. I added mine for comparison. At some point during my testing the error also changed slightly and started stating VDI must be free or attached to exactly one VM. It appears after a failed backup it's not cleaning up the snapshots.

                            2024-08-29T21_32_42.161Z - backup NG.json.txt

                            2024-08-28T15_05_12.613Z - backup NG.json.txt

                            1 Reply Last reply Reply Quote 0
                            • D Offline
                              Delgado
                              last edited by

                              Hello,

                              My vms are about 150G each. I was using compression when I backed up the vm to the remote before mirroring it to the s3 bucket. I did end up changing to delta backups and the error did go away but I can create another normal backup and mirror it to the bucket again to see if I get the same results.

                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO Offline
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                That's interesting 🤔 So it's only with full backup (XVAs) then.

                                nvossN 1 Reply Last reply Reply Quote 0
                                • nvossN Offline
                                  nvoss @olivierlambert
                                  last edited by nvoss

                                  @olivierlambert yeah my experience is also that deltas run without error. Though what they're backing up exactly w/o a full in the remote is pretty questionable. I assume its a delta off of the snapshot full, where the snapshot is completed without issue and it's just the copy to encrypted remote that's failing.

                                  These are definitely my larger VMs -- >100gb total disk.

                                  1 Reply Last reply Reply Quote 0
                                  • D Offline
                                    daniel.grimm
                                    last edited by

                                    Hi,

                                    Same problem here.
                                    Its an encrypted S3 remote to Backblaze.
                                    Full mirror backup with selected VMs.
                                    Small VM like Xen-Orchestra works. As soon as a large VM is added (approx. 500GB), the error occurs after about 3 hours.
                                    Tried several times.

                                    xen-orchestra build from source

                                    1 Reply Last reply Reply Quote 0
                                    • olivierlambertO Offline
                                      olivierlambert Vates 🪐 Co-Founder CEO
                                      last edited by

                                      And with or without backup compression?

                                      1 Reply Last reply Reply Quote 0
                                      • D Offline
                                        daniel.grimm
                                        last edited by

                                        sorry... i forgot...
                                        with zstd compression

                                        1 Reply Last reply Reply Quote 0
                                        • olivierlambertO Offline
                                          olivierlambert Vates 🪐 Co-Founder CEO
                                          last edited by

                                          Can you try without it and report?

                                          1 Reply Last reply Reply Quote 0
                                          • D Offline
                                            daniel.grimm
                                            last edited by

                                            Yes.

                                            iam making a non compressed backup now.
                                            And then i try to mirror it with the mirroring job.

                                            Report follows... But uncompressed backup and upload will need some time

                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post