@webminster said in Windows10 boot: SYSTEM THREAD EXCEPTION:

@john-c No. It's a basic no-frills standalone machine.

Have you made sure it's not a hidden file as Windows hides files by default! So if the "%SystemRoot%\Memory.dmp" has the Hidden and/or System bit set it will be not visible on default settings.

One sure fire way to tell is to enable Windows to show the hidden files, then have a look or using the command line terminal to run a listing and/or search in the %SystemRoot% directory for the file.

Linux is case sensitive don't forget, but Windows isn't so if the case for the file name doesn't match then it won't be found.

Also if the system crashes too early and/or quickly when doing BSOD it may not have the filesystem components and drivers, loaded yet for the "%SystemRoot%\Memory.dmp" file creation.

@webminster said in Windows10 boot: SYSTEM THREAD EXCEPTION:

@john-c It's not something I set explicitly. The Windows 10 Pro install was a clean install from ISO, I did not change settings beyond enabling RDP and patching.

Not GP or such. The machine on boot looped between a restart after the BSOD (restarted itself) and a self-triggered automatic repair and reboot (which BSOD'd after).

Definitely not interrupting the boot cycle.
-Alan

Is the VM part of a Domain Login (via Windows Server or Samba) if it is did you define it in the GPO when setting up the Domain's Group policy?

Cause this is propagated to all domain member computers.

@webminster You may have jumped in with the reboot and boot disk before Windows had the opportunity to generate the file, or may have the memory dump functionality disabled via Group Policy or in the System Properties. You could have had the file erased with "Disk Clean-up" or as part of "Storage Sense".

Thus you will likely not be able to find that file your looking for.

@stormi said in XCP-NG 8.3 xsconsole backup, restore pool metadata:

@Prilly said in XCP-NG 8.3 xsconsole backup, restore pool metadata:

Snicky this one from vates

If you mean sneaky, I'm not sure I appreciate what this word implies.

This is absolutely something to write about in the release note of 8.3.

Agreed, this should be in the release notes. I'll add it.

when is it expected back?

I had a look, and it was fixed on XenServer side. So it will be back in XCP-ng 8.3 once we rebase on their latest updates. Hopefully before the end of the year.

How do you restore this metadata backup in xcp-ng 8.3 without the use of XO?

If you really can't use XO, you can use the same API that XO uses to implement the feature in its UI: XAPI. I don't know myself the exact end points that must be used for this back-up feature.

I think they may have actually meant to say sticky this one from Vates, they may have been hit by autocorrect, or are using sliding typing and didn't notice the mistake.

@manilx said in Epyc VM to VM networking slow:

@john-c Hi. Ordered from Amazon that day and after more than 2 weeks order was cancelled without notice from supplier. Reordered from another one and I'm still waiting....
Not easy to get one.

Thanks for your reply. I hope it goes well this time, anyway if it still proves difficult then you can go for another quad port 10Gbe NIC which is compatible to do the LACP 2 bond with.

If the selected quad port 10Gbe NIC is available on general sale, then you can get it through the supplier who provided you with your HPE Care Packs.

@manilx I've been waiting for your ping back with the report. Following you saying the first week in November 2024, now in the beginning of the 2nd week in November 2024.

I'm wondering how's it going please, anything holding it up?

@TheNorthernLight said in XO Lite: building an embedded UI in XCP-ng:

Just upgraded my home network to 8.3.0 and I must say, it went flawlessly. Also my first time taking a look at XO-Lite. Also fantastic! Kudo's to the whole vates dev and testing team. This is a huge milestone.

The only real feature for me that is missing from XO-Lite is the ability to shutdown/restart the host itself from the menu (I'm assuming this is still being added).

Currently, I cant see a way to restart the host from XO-Lite. My XCPNG host runs headless, so when I needed to restart the host, I loaded XCPNG Center and shutdown the XOA VM, and then rebooted the host (only needed when applying host patches). This method wont work now with 8.3. I was forced to use the "force reboot" from XOA to complete the final patch/reboot steps.

Without the control to restart/shutdown the host itself, the only other option is to enable SSH (when the host is headless). I would think that XO-Lite would "in theory" be more secure then leaving SSH open only for this scenario.

Separately, I also run XCPNG/XOA at work (licensed), and currently we are waiting for XOSTOR to come out of beta before we upgrade/test it in our environment. Any ETA on XOSTOR going full version in 8.3 ?

Looking forward to the future of this version and product.

Leaving SSH open won't work as eventually its port will be found, then exploited (used maliciously to gain access to host). Also it would bypass the lock down feature, modelled after VMware ESXi lockdown.

Anyway there's RPU & RPR both of which are available in the appropriate edition of XOA (Xen Orchestra Appliance) and also Xen Orchestra from Sources. The RPU will reboot each host in the pool after updating them. The RPR will reboot each host in the pool in sequence, before moving onto the next.

@m-mirzayev said in ISCSI mount - SR_BACKEND_FAILURE_432:

@john-c
Would you mind to post your custom.conf so i have a reference?

@m-mirzayev I don’t use multipath personally however, though managed to get Microsoft Copilot to put what you gave above into a valid structure. I also remember people in the past having trouble with multipath, so Vates employees implemented this /etc/multipath/conf.d/custom.conf to fix these issues.

# /etc/multipath/conf.d/custom.conf
defaults {
    user_friendly_names yes
}

multipaths {
    multipath {
        wwid                    "your_device_wwid_here"
        alias                   "truenas_iscsi"
        path_selector           "round-robin 0"
        path_grouping_policy    multibus
        path_checker            tur
        prio                    const
        failback                immediate
    }
}

devices {
    device {
        vendor                  "TrueNAS"
        product                 "iSCSI"
        path_selector           "round-robin 0"
        path_grouping_policy    multibus
        hardware_handler        "0"
        prio                    const
        failback                immediate
    }
}

Please replace "your_device_wwid_here" with the WWID of the device on your network.

@m-mirzayev said in ISCSI mount - SR_BACKEND_FAILURE_432:

@john-c
As i mentioned i am not a pro in regard of iscsi and multipath.conf. If you are, can you just look over the settings and give your opinion on it?

And do i just put this into the custom.conf:

        device {
                vendor                  "TrueNAS"
                product                 "iSCSI"
                path_selector           "round-robin 0"
                path_grouping_policy    multibus
                hardware_handler        "0"
                prio                    "const"
                failback                immediate
        }

without the outer devices {}

The structure of custom.conf must be a valid structure configuration file for multipath. As even though its overriding the multipath.conf file it still needs to be valid so that the software, can parse it correctly.

@m-mirzayev said in ISCSI mount - SR_BACKEND_FAILURE_432:

@john-c
Thanks for this information. I will put it into the custom.conf

Having it in this file will also enable it to be placed as part of automated installation processes and infrastructure as code.

@m-mirzayev said in ISCSI mount - SR_BACKEND_FAILURE_432:

I had a similar problem with a TrueNAS Server. On the host multipath was enabled and connecting to Synology iSCSI target was not a problem at all. Connecting to the TrueNAS target was not working with an error:

sr.probeIscsiExists
{
  "host": "81e4f97f-bc48-4b11-9ece-97f395c7e24c",
  "target": "172.16.1.21",
  "targetIqn": "iqn.2024-11.com.truenas:target0",
  "scsiId": "36589cfc0000000de53d4a06991332938",
  "port": 3260
}
{
  "code": "SR_BACKEND_FAILURE_432",
  "params": [
    "",
    "Device Mapper path missing [opterr=Device mapper path /dev/mapper/36589cfc0000000de53d4a06991332938 not found]",
    ""
  ],
  "call": {
    "duration": 10810,
    "method": "SR.probe",
    "params": [
      "OpaqueRef:1cdb7e37-eb90-4644-8ac1-37fddfd7e58c",
      {
        "target": "172.16.1.21",
        "targetIQN": "iqn.2024-11.com.truenas:target0",
        "SCSIid": "36589cfc0000000de53d4a06991332938",
        "port": "3260"
      },
      "lvmoiscsi",
      {}
    ]
  },
  "message": "SR_BACKEND_FAILURE_432(, Device Mapper path missing [opterr=Device mapper path /dev/mapper/36589cfc0000000de53d4a06991332938 not found], )",
  "name": "XapiError",
  "stack": "XapiError: SR_BACKEND_FAILURE_432(, Device Mapper path missing [opterr=Device mapper path /dev/mapper/36589cfc0000000de53d4a06991332938 not found], )
    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202411020002/packages/xen-api/_XapiError.mjs:16:12)
    at file:///opt/xo/xo-builds/xen-orchestra-202411020002/packages/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:60:5)
    at processImmediate (node:internal/timers:454:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}
After researching 2 days and working with chatgpt the solution was to change ```

/etc/multipath.conf

. At the end of the file there was a section with an device for TrueNAS.
I changed it to:

    device {
            vendor                  "TrueNAS"
            product                 "iSCSI"
            path_selector           "round-robin 0"
            path_grouping_policy    multibus
            hardware_handler        "0"
            prio                    "const"
            failback                immediate
    }

I am not an expert on the configured settings. It was what chatgpt suggested. So maybe if someone can view this settings and maybe we can then add it to the standard configuration which comes with XCP-ng. Maybe @olivierlambert can view this and we can put it on github.

That alteration to the /etc/multipath.conf file will be lost when XCP-ng has that file restored as part of updates or upgrades. Make the changes to /etc/multipath/conf.d/custom.conf instead.

Yes it does! Just make sure your using the most up to date instance of XCP-ng or the latest XCP-ng 8.2.1 (renewed installer) or 8.3.

@olivierlambert said in Encrypt Server Passwords:

I'm not saying it's not possible, I'm saying this will be at the huge expense of usability.

One such software implementation of a KMIP protocol server, which is in software form is an open source fork of HashiCorp Vault. This open source fork is called OpenBao (https://openbao.org/).

@olivierlambert @Forza @jeffmetal @bettysweetss During deployment of XOA it could allow for the capacity to enable such a feature. As well as a configuration option in XOA for afterwards, if not enabled during deployment. Then as part of the same section there would be the option for the user to choose for where the encryption key for the credentials can be stored:-

• TPM 2.0
• KMIP protocol server
• Tang server (https://github.com/latchset/tang)
• AWS Secrets Manager
• HashiCorp Vault (or any open source forks)
• Google Secrets Manager
• Azure Key Vault
• IBM Cloud Secrets Manager
• CyberArk Conjur

As well other options, but any way being able to have the encryption key go to one of these, then accessed from there when required. Also through an option the encrypted SR could also gain the option to have the capacity to get the keys going to the users choice of one of the list above.

The KMIP protocol server can come in two forms hardware and software. The software form comes as a piece of software which is installed on a device of your choosing or as a virtual appliance. The hardware form is a physical appliance, which you setup, connect to your network, configure and administer.

An example of an open source fork of HashiCorp Vault is the OpenBao (https://openbao.org/). It's a place where secrets can be sent using compatible software.

Some of the above will work in an airgap network environment, as they don't require an Internet connection to work. There's also the option to use an online third party service which is provided by companies which may have higher security than you, on your network; or if the other options of having it on your network isn't viable.

@McHenry @nick-lloyd With todays release of Xen Orchestra (XOA) version 5.100 there's now support for custom ACLs based around tags. These will prove really useful when your setting up the XOA with its ACL for your clients to access self service, while ensuring that they can't access the other's pools and VMs etc.

@nick-lloyd said in Pool Management:

@McHenry If I'm understanding this correctly, Client Site A, B, n... will each have a server and all share multiple instances at OVH for DR. First off, I hope you give the person that's doing the networking for this a raise .

Second, are you and/or a team the sole manager of the infrastructure? If clients have their own IT team that assists/uses XO(A), this will be more difficult. For the sake of this solution, I'm going to assume that the client can use XO(A) to start VMs in case of a DR scenario.

If I were to build it out, I would build an XO host in OVH and utilize the XO-proxy and the Self Service feature. This way, you can register the XO Proxy to all of the client sites, and manage who can see what VMs, networking/VLANS, and storage.

If I'm wrong about an assumption or solution, let me know!

@McHenry @nick-lloyd If you use an up to date stable (or latest) release channel in XOA you can use tags and maintain a list of clients and a customer number. Then you can use the custom tags (scoped tags) feature of Xen Orchestra to tag each client's pool(s) with a specific customer number tag (relevant to their list entry).

This feature was introduced in Xen Orchestra version 5.90, so is present in every version since then.

The tags will be applied to the VMs so each VM created self service by the client would be tagged with a scoped tag for their customer (client) number. The appropriate pools and thus their hosts would have an affinity for their customer number and anti affinity for other customer number.

@andsmith said in 8.3 USB Passthrough - Win Server 2019 BSOD:

@john-c said in 8.3 USB Passthrough - Win Server 2019 BSOD:

winget install Microsoft.WinDbg

Following information from this here

I've run the following commands in PowerShell:

Install-Script -Name winget-install
winget-install

These ran without issue, however when I ran the command:

winget install Microsoft.WinDbg

The installer starts, but fails with the error:

InstallFlowReturnCodeBlockedByPolicy
InstallerFailedWithCode

That's a reference to GroupPolicy settings applied which affect the software installation of Appx Packages, as well as possibly other relevant policies. As well as any Conditional Access policies applied which affected the installation, these policies can also come from services like Microsoft Entra etc.

@olivierlambert said in 8.3 USB Passthrough - Win Server 2019 BSOD:

Agreed

Oh also this includes Windows Server 2016 and also if not present then Windows 10 and 11.

Anyway just thought I would help out the user with the method of installing WinDbg on their server VM.

While waiting to be pinged back later by a poster with the Epyc network bug, worth checking out the latest responses to that bug on the forums.

@andsmith said in 8.3 USB Passthrough - Win Server 2019 BSOD:

@dinhngtu Sorry, I failed to mention this is a Windows Server 2019 VM, it appears that Windbg is a Windows 10/11 application. I've googled installing on Server 2019, but haven't had any luck getting it installed.

Update the AppX Package Deployment software via Microsoft Store or its equivalent in Windows Server 2019. Then once winget is updated (or installed) run the following command:

winget install Microsoft.WinDbg

@olivierlambert If this works for andsmith then it may be worth documenting this, for future users of Windows Server 2019, 2022 and possibly if required the upcoming Windows Server 2025.

@Ajmind-0 said in XCP-ng 8.3 betas and RCs feedback :

@john-c

Yes, Sandisk USB Stick (Ventoy) in UEFI boot mode. I had also created a standard DVD ROM in ourder to exclude any differences in such a boot construct.

Did you verify that the checksums matched for the ISO you downloaded, before preparing the media?

The reason being any corruption in the downloaded image can affect the resulting media and their contents. So unexpected things can happen if the media or source media is corrupt, such as an in ability to complete the installation of OS software even at the final stage - boot manager installation.

@Ajmind-0 said in XCP-ng 8.3 betas and RCs feedback :

@john-c
PXE boot ROM for network boot. (Never used)

So SanDisk HD is the portable drive used or is it something else?