@Delgado said in Health Checks Failing:
@florent can confirm that health checks are working again! Thanks for the prompt reply!
thanks for the precise bug report, that is half of the job done
@Delgado said in Health Checks Failing:
@florent can confirm that health checks are working again! Thanks for the prompt reply!
thanks for the precise bug report, that is half of the job done
@Delgado said in Health Checks Failing:
_MixinXapiWriter
Hi,
This is probably linked to the PR that makes healthcheck timeout configurable, I am looking into it right now
Regards
@erlicthemad we are doing some check atthe connection to the remote to detect Object Lock, since enabling Content-md5 is only mandatory then. And computing it means keeping a lot of more data in memory (code here https://github.com/vatesfr/xen-orchestra/blob/master/%40xen-orchestra/fs/src/s3.js#L452 )
let me look into the dock if we can detect it also for backblaze
edit : can you try to disable the remote, enable content lock and reenable the remote ?
have you got any message like 'Object lock is not available or not configured, don't add the content MD5 header' or 's3 user doesnt have enough privilege to check for Object Lock, enable content MD5 header' in your logs ?
transfer successfull
There was a misconception the NFS mount should be a remote ( in XO side ) ,not a SR ( in XCP-ng side)
@CodeMercenary thanks
these little correction improve the process for everyone
@erlicthemad in theory object lock is honored by the backup process.
Can you post the full error log ?
@Carello The ticket will probably reach my desk. And for now I have only one user still stuck (and he's migrating to xenserver so I can't really count this as a miss). Hopefully we'll find a solution for you too , that will improve our solution for every one else
(BTW, I work on Paris time )
@Tristis-Oris said in NBD setup - No such item:
@florent lets a brief summary.
nbd work only via management interface. If nbd enabled on any another interface and specified as pool backup interface, it don't work (the issue with blocked packets at iptables). That was a main reason of my problem.
For now i made it work at all my pools. Max single thread speed increased 80-90 > 140-150. At usual backup task (when daily delta very small) it show same 10-40Mb, because it done too fast. Average task time decreased fo about twice.
That is really strange, but it is also useful. We'll try to see why , and update the doc accordingly
@CodeMercenary That are some nice volume to transfer
Agent/file based migration is on the table, but there are a lot of hurdles to stay platform independant. Depending on the plarform you can use a combiation of rsync or robocopy /vss to transfer the file while both the VM are running, "freeze" the source VM, make a last sync and provide the services from the destination (xcp-ng) VM
the right size check is updated and the fix will reach master in a few days : https://github.com/vatesfr/xen-orchestra/pull/7554/commits/6e899287c6a1a26cb95099cbecc5d080c91c76f0
Please note that what is limited is the capactiy of the drive, not the data present on it
@Carello Hi, I am working on the migration tool
If you have a XOA, could you open a support ticket and then a support tunnel ? We'll be able to connect and see the logs directly, because most of the test you've done should have work.
if it's something else, we'll do our best to make it work, v2v is an important part of our platform
@frank-s said in s3 remote download speeds:
@florent Hi, Have you been able to make any progress with this?
Thanks.
not for now, but it's in our backlog , and I have some major work to do in the next quarter to the S3 code
@CodeMercenary the name shouldn't create problem since we use the disk refs
the ERR_STREAM_PREMATURE_CLOSE is probably because the import is limited to 2TB minus 8MB of data reserved for metadata , and one disk of your may be exactly 2TB
The import till 5.91 ( decemebre) was doiing 2 passes, but now it should work in one pass
@Tristis-Oris said in NBD setup - No such item:
next weird thing.
i enabled NBD on all manage interfaces at all pools.
it working with CR backup, but shouldn't, at least because it no option to enable NBD.
it working with half of other delta backup tasks, but not for all. Main backup is still without nbd.
only idea, it probably because of 1 small pool with few vm, where i forget to enable NBD. so 1 VM without nbd, force it to work in usual mode?
that is really puzzling. I don't forget this issue, but I don't have much idea of how to fix this
Did you add the preferNbd flag to the config files ?
@wttw said in OVA import support?:
@olivierlambert said in OVA import support?:
- Yes it's supposed to work. Sadly, there's as much as different OVAs than VMs and VMware versions. Every time we are fixing new stuff when it's reported. Please provide the OVA somewhere so we can find why and fix it
https://tupid.org/debian11.ova - it's about 850MB.
I just created this on ESXi 7.0U3, then exported it with
./ovftool --noSSLVerify --diskMode=thin --targetType=ova --compress=9 "vi://skull/debian11" .
(VMware ovftool 4.6.2 (build-22220919))
- Import from VMware should work if you have indeed the XCP-ng machine in extra for it.
The hardware is on order, but I thought I'd start testing this weekend.
But we could fix the OVA already, at least as soon we can check what's wrong with the XML.
Here's debian11.ovf, for anyone who wants to look without downloading the ova.
<?xml version="1.0" encoding="UTF-8"?> <!--Generated by VMware ESX Server, User: root, UTC time: 2024-04-07T15:42:54.40766Z--> <Envelope vmw:buildId="build-19482537" xmlns="http://schemas.dmtf.org/ovf/envelope/1" xmlns:cim="http://schemas.dmtf.org/wbem/wscim/1/common" xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" xmlns:rasd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData" xmlns:vmw="http://www.vmware.com/schema/ovf" xmlns:vssd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <References> <File ovf:compression="gzip" ovf:href="debian11-disk1.vmdk.gz" ovf:id="file1" ovf:size="856235944"/> <File ovf:compression="gzip" ovf:href="debian11-file1.nvram.gz" ovf:id="file2" ovf:size="1821"/> </References> <DiskSection> <Info>Virtual disk information</Info> <Disk ovf:capacity="16" ovf:capacityAllocationUnits="byte * 2^30" ovf:diskId="vmdisk1" ovf:fileRef="file1" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" ovf:populatedSize="2324692992"/> </DiskSection> <NetworkSection> <Info>The list of logical networks</Info> <Network ovf:name="VM Network"> <Description>The VM Network network</Description> </Network> </NetworkSection> <VirtualSystem ovf:id="debian11"> <Info>A virtual machine</Info> <Name>debian11</Name> <OperatingSystemSection ovf:id="1" vmw:osType="debian11Guest"> <Info>The kind of installed guest operating system</Info> </OperatingSystemSection> <VirtualHardwareSection> <Info>Virtual hardware requirements</Info> <System> <vssd:ElementName>Virtual Hardware Family</vssd:ElementName> <vssd:InstanceID>0</vssd:InstanceID> <vssd:VirtualSystemIdentifier>debian11</vssd:VirtualSystemIdentifier> <vssd:VirtualSystemType>vmx-19</vssd:VirtualSystemType> </System> <Item> <rasd:AllocationUnits>hertz * 10^6</rasd:AllocationUnits> <rasd:Description>Number of Virtual CPUs</rasd:Description> <rasd:ElementName>1 virtual CPU(s)</rasd:ElementName> <rasd:InstanceID>1</rasd:InstanceID> <rasd:ResourceType>3</rasd:ResourceType> <rasd:VirtualQuantity>1</rasd:VirtualQuantity> </Item> <Item> <rasd:AllocationUnits>byte * 2^20</rasd:AllocationUnits> <rasd:Description>Memory Size</rasd:Description> <rasd:ElementName>2048MB of memory</rasd:ElementName> <rasd:InstanceID>2</rasd:InstanceID> <rasd:ResourceType>4</rasd:ResourceType> <rasd:VirtualQuantity>2048</rasd:VirtualQuantity> </Item> <Item> <rasd:Address>0</rasd:Address> <rasd:Description>SATA Controller</rasd:Description> <rasd:ElementName>SATA Controller 0</rasd:ElementName> <rasd:InstanceID>3</rasd:InstanceID> <rasd:ResourceSubType>vmware.sata.ahci</rasd:ResourceSubType> <rasd:ResourceType>20</rasd:ResourceType> <vmw:Config ovf:required="false" vmw:key="slotInfo.pciSlotNumber" vmw:value="34"/> </Item> <Item> <rasd:Address>0</rasd:Address> <rasd:Description>SCSI Controller</rasd:Description> <rasd:ElementName>SCSI Controller 0</rasd:ElementName> <rasd:InstanceID>4</rasd:InstanceID> <rasd:ResourceSubType>VirtualSCSI</rasd:ResourceSubType> <rasd:ResourceType>6</rasd:ResourceType> <vmw:Config ovf:required="false" vmw:key="slotInfo.pciSlotNumber" vmw:value="160"/> </Item> <Item ovf:required="false"> <rasd:Address>0</rasd:Address> <rasd:Description>USB Controller (EHCI)</rasd:Description> <rasd:ElementName>USB Controller</rasd:ElementName> <rasd:InstanceID>5</rasd:InstanceID> <rasd:ResourceSubType>vmware.usb.ehci</rasd:ResourceSubType> <rasd:ResourceType>23</rasd:ResourceType> <vmw:Config ovf:required="false" vmw:key="slotInfo.pciSlotNumber" vmw:value="32"/> <vmw:Config ovf:required="false" vmw:key="slotInfo.ehciPciSlotNumber" vmw:value="33"/> <vmw:Config ovf:required="false" vmw:key="autoConnectDevices" vmw:value="false"/> <vmw:Config ovf:required="false" vmw:key="ehciEnabled" vmw:value="true"/> </Item> <Item> <rasd:Address>1</rasd:Address> <rasd:Description>IDE Controller</rasd:Description> <rasd:ElementName>VirtualIDEController 1</rasd:ElementName> <rasd:InstanceID>6</rasd:InstanceID> <rasd:ResourceType>5</rasd:ResourceType> </Item> <Item> <rasd:Address>0</rasd:Address> <rasd:Description>IDE Controller</rasd:Description> <rasd:ElementName>VirtualIDEController 0</rasd:ElementName> <rasd:InstanceID>7</rasd:InstanceID> <rasd:ResourceType>5</rasd:ResourceType> </Item> <Item ovf:required="false"> <rasd:AutomaticAllocation>false</rasd:AutomaticAllocation> <rasd:ElementName>VirtualVideoCard</rasd:ElementName> <rasd:InstanceID>8</rasd:InstanceID> <rasd:ResourceType>24</rasd:ResourceType> <vmw:Config ovf:required="false" vmw:key="useAutoDetect" vmw:value="true"/> <vmw:Config ovf:required="false" vmw:key="videoRamSizeInKB" vmw:value="4096"/> <vmw:Config ovf:required="false" vmw:key="enable3DSupport" vmw:value="false"/> <vmw:Config ovf:required="false" vmw:key="use3dRenderer" vmw:value="automatic"/> <vmw:Config ovf:required="false" vmw:key="graphicsMemorySizeInKB" vmw:value="262144"/> </Item> <Item ovf:required="false"> <rasd:AutomaticAllocation>false</rasd:AutomaticAllocation> <rasd:ElementName>VirtualVMCIDevice</rasd:ElementName> <rasd:InstanceID>9</rasd:InstanceID> <rasd:ResourceSubType>vmware.vmci</rasd:ResourceSubType> <rasd:ResourceType>1</rasd:ResourceType> <vmw:Config ovf:required="false" vmw:key="allowUnrestrictedCommunication" vmw:value="false"/> </Item> <Item ovf:required="false"> <rasd:AddressOnParent>0</rasd:AddressOnParent> <rasd:AutomaticAllocation>false</rasd:AutomaticAllocation> <rasd:ElementName>CD-ROM 1</rasd:ElementName> <rasd:InstanceID>10</rasd:InstanceID> <rasd:Parent>3</rasd:Parent> <rasd:ResourceSubType>vmware.cdrom.atapi</rasd:ResourceSubType> <rasd:ResourceType>15</rasd:ResourceType> <vmw:Config ovf:required="false" vmw:key="connectable.allowGuestControl" vmw:value="true"/> </Item> <Item> <rasd:AddressOnParent>0</rasd:AddressOnParent> <rasd:ElementName>Hard Disk 1</rasd:ElementName> <rasd:HostResource>ovf:/disk/vmdisk1</rasd:HostResource> <rasd:InstanceID>11</rasd:InstanceID> <rasd:Parent>4</rasd:Parent> <rasd:ResourceType>17</rasd:ResourceType> <vmw:Config ovf:required="false" vmw:key="backing.writeThrough" vmw:value="false"/> </Item> <Item> <rasd:AddressOnParent>7</rasd:AddressOnParent> <rasd:AutomaticAllocation>true</rasd:AutomaticAllocation> <rasd:Connection>VM Network</rasd:Connection> <rasd:Description>VmxNet3 ethernet adapter on "VM Network"</rasd:Description> <rasd:ElementName>Ethernet 1</rasd:ElementName> <rasd:InstanceID>12</rasd:InstanceID> <rasd:ResourceSubType>VmxNet3</rasd:ResourceSubType> <rasd:ResourceType>10</rasd:ResourceType> <vmw:Config ovf:required="false" vmw:key="slotInfo.pciSlotNumber" vmw:value="192"/> <vmw:Config ovf:required="false" vmw:key="wakeOnLanEnabled" vmw:value="false"/> <vmw:Config ovf:required="false" vmw:key="connectable.allowGuestControl" vmw:value="true"/> </Item> <vmw:Config ovf:required="false" vmw:key="cpuHotAddEnabled" vmw:value="false"/> <vmw:Config ovf:required="false" vmw:key="cpuHotRemoveEnabled" vmw:value="false"/> <vmw:Config ovf:required="false" vmw:key="memoryHotAddEnabled" vmw:value="false"/> <vmw:Config ovf:required="false" vmw:key="firmware" vmw:value="bios"/> <vmw:Config ovf:required="false" vmw:key="cpuAllocation.shares.shares" vmw:value="1000"/> <vmw:Config ovf:required="false" vmw:key="cpuAllocation.shares.level" vmw:value="normal"/> <vmw:Config ovf:required="false" vmw:key="tools.syncTimeWithHost" vmw:value="false"/> <vmw:Config ovf:required="false" vmw:key="tools.syncTimeWithHostAllowed" vmw:value="true"/> <vmw:Config ovf:required="false" vmw:key="tools.afterPowerOn" vmw:value="true"/> <vmw:Config ovf:required="false" vmw:key="tools.afterResume" vmw:value="true"/> <vmw:Config ovf:required="false" vmw:key="tools.beforeGuestShutdown" vmw:value="true"/> <vmw:Config ovf:required="false" vmw:key="tools.beforeGuestStandby" vmw:value="true"/> <vmw:Config ovf:required="false" vmw:key="tools.toolsUpgradePolicy" vmw:value="manual"/> <vmw:Config ovf:required="false" vmw:key="powerOpInfo.powerOffType" vmw:value="soft"/> <vmw:Config ovf:required="false" vmw:key="powerOpInfo.resetType" vmw:value="soft"/> <vmw:Config ovf:required="false" vmw:key="powerOpInfo.suspendType" vmw:value="soft"/> <vmw:Config ovf:required="false" vmw:key="nestedHVEnabled" vmw:value="false"/> <vmw:Config ovf:required="false" vmw:key="vPMCEnabled" vmw:value="false"/> <vmw:Config ovf:required="false" vmw:key="virtualICH7MPresent" vmw:value="false"/> <vmw:Config ovf:required="false" vmw:key="virtualSMCPresent" vmw:value="false"/> <vmw:Config ovf:required="false" vmw:key="flags.vvtdEnabled" vmw:value="false"/> <vmw:Config ovf:required="false" vmw:key="flags.vbsEnabled" vmw:value="false"/> <vmw:Config ovf:required="false" vmw:key="bootOptions.efiSecureBootEnabled" vmw:value="false"/> <vmw:Config ovf:required="false" vmw:key="powerOpInfo.standbyAction" vmw:value="checkpoint"/> <vmw:ExtraConfig ovf:required="false" vmw:key="nvram" vmw:value="ovf:/file/file2"/> <vmw:ExtraConfig ovf:required="false" vmw:key="svga.autodetect" vmw:value="TRUE"/> </VirtualHardwareSection> </VirtualSystem> </Envelope>
this is an ova with vmdk disk compressed. They are not supported for now (since vmdk are already compressed per block). TBF, we're putting much of the import/export effort into the v2v tool
@manilx said in Epyc VM to VM networking slow:
@florent Florent, yes I do use NBD for all backups. And checking the backup log of the completed jobs I see that NBD is being used.
could you test disabling NBD ?
@manilx do you use NBD for delta backups ?
in the advanced settings
@manilx said in Epyc VM to VM networking slow:
@florent Hi,
Both storages are NFS, all connections 10G.
On both cases XO/XOA is running on the master.
thank you for the test. At least it removed the easy fixes
@manilx hi, I am working on the backup side, that is a very interesting finding. I have some question to rule out some hypothesis :
What storage do you use on both side ? iscsi / nfs ?
Is XO running on the master ?
my test LVM VM works
@michmoor0725 or @Andrew , do you have any open tunnel I can use to test on your VM ?
@Andrew said in File restore error on LVMs:
@florent Still not working.
Here is the partition list for a standard Ubuntu 20.04.3 install.
Partition 3 is the main OS root as a single LVM (19G).
Trying to restore a file from it makes XO confused.xvda 202:0 0 20G 0 disk ├─xvda1 202:1 0 1M 0 part ├─xvda2 202:2 0 1G 0 part /boot └─xvda3 202:3 0 19G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 19G 0 lvm /
it see the partition but fail to m
@Andrew said in File restore error on LVMs:
@florent Still not working.
Here is the partition list for a standard Ubuntu 20.04.3 install.
Partition 3 is the main OS root as a single LVM (19G).
Trying to restore a file from it makes XO confused.xvda 202:0 0 20G 0 disk ├─xvda1 202:1 0 1M 0 part ├─xvda2 202:2 0 1G 0 part /boot └─xvda3 202:3 0 19G 0 part └─ubuntu--vg-ubuntu--lv 253:0 0 19G 0 lvm /
@michmoor0725 the lvm parittion with one disk should work. Is it the case for you ?