@DustinB

Hmm - just got done running mem86+ - 4 passes -- all 14 tests. No RAM errors. I wonder the what would cause this error? I'll probably just save config and reinstall. So strange.

@DustinB I have time to run a ram check (luckily), and if not I'll just recreate VM. Luckily pfsense makes it pretty easy to reinstall from scratch by saving to a configuration file which can be imported to new version. I'll work on it later today or tomorrow and report back.

I'm running pfsense version 2.8.0 virtualized on xcp-ng 8.3.0.

The host has 64 GB RAM (two 32 GB dual channel DDR4 modules) with a 1Tb Nvme drive and 1tb samsung SSD.

I have 5 virtualized systems on the xcp-ng host consuming a total of 30Gb of the 64 GB.

One of my virtualized hosts is pfSense 2.8.0. Recently the pfSense host is giving me consistent error messages such as the following:

Aug 25 07:46:28	kernel		vm_fault: pager read error, pid 24443 (rrdtool)
Aug 25 07:46:28	kernel		vm_fault: pager read error, pid 24443 (rrdtool)
Aug 25 07:46:28	kernel		vm_fault: pager read error, pid 24443 (rrdtool)
Aug 25 07:46:28	kernel		vm_fault: pager read error, pid 24443 (rrdtool)
Aug 25 07:46:28	kernel		vm_fault: pager read error, pid 24443 (rrdtool)
Aug 25 07:46:28	kernel		vm_fault: pager read error, pid 24443 (rrdtool)
Aug 25 07:46:28	kernel		vm_fault: pager read error, pid 24443 (rrdtool)
Aug 25 07:46:28	kernel		vm_fault: pager read error, pid 24443 (rrdtool)
Aug 25 07:46:28	kernel		vm_fault: pager read error, pid 24443 (rrdtool)
Aug 25 07:46:28	kernel		vm_fault: pager read error, pid 24443 (rrdtool)

Doing some reading on the error (https://www.reddit.com/r/opnsense/comments/1jlfshn/vm_fault_pager_read_error_pid_76098_rrdtool/), it could indicate a bad RAM module?? I'm just not sure. Is RAM presented to the pfsense host as virtual memory? Other than doing a mem86+ test on the RAM modules of the xcp-ng host, is there any other way to debug this error? I don't see errors from any other of the virtualized linux operating systems, however is there something I should be checking on the other virtualized systems or the host that would confirm a possible memory error as error (assuming its a memory error)?

Thanks for any input.

So I managed to build with Cargo on arch (which also required clang as a dependency). Moved xen-guest-agent/target/debug/xen-guest-agent to /usr/sbin and also copied the basic xen-guest-agent/startup/xen-guest-agent.service file to /etc/systemd/system and enable/started the service.

And ahh -- success --

I guess my project for week is to figure out how to write a PKGBUILD file for this particular project. We'll see how that goes

I'm assuming since building from git repository (https://gitlab.com/xen-project/xen-guest-agent) there aren't going to be any file signatures to check against. I'm looking at cargo, clang, python-setuptools and xen as dependencies?

When using this toolset, what network interface names does this match against? For example it will match against interface names starting with eth and I think enpS. I looked in the source code within the main branch but couldn't find the file where this search occurs.

@gskger Why don't you install acme.sh or something similar on XO host and deploy from there. No need to use pfSense.

@yann Yes I tried compiling with cargo. Got along some of the way until I reached this:

  Failed to locate xenstore library:
  pkg-config exited with status code 1
  > PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 pkg-config --libs --cflags xenstore

  The system library `xenstore` required by crate `xenstore-sys` was not found.
  The file `xenstore.pc` needs to be installed and the PKG_CONFIG_PATH environment variable must contain its parent directory.
  The PKG_CONFIG_PATH environment variable is not set.

  HINT: if you have installed the library, try setting PKG_CONFIG_PATH to the directory containing `xenstore.pc`.

Where do I get the xenstore library? I've searched the AUR and pacman official archives and I can't seem to find.

Hey do you have actual instructions on how to compile from source? I trying to work with someone creating an arch linux AUR package and was looking for a little more input.

@olivierlambert Hey thanks a lot for that tip.

@Andrew Your link doesn't really address the question. xoa would be the username and the password is the same as what was setup during xoa creation -- or as your link helpfully pointed out -- you could reset the xoa system user password and go from there.

I need to become the root user in xoa. What is the root password? I've tried root/root but it doesn't work and it isn't in the documentation. I need to change the network interface for xoa.

@Danp Aww. see that box. Let me look into this problem a bit more. Learning some new things here.

@DustinB Didn't know about the health check drivers to be honest. My VM not working is a pfSense VM. I looked at the link posted and didn't see BSD drivers. Looking my XO installation for pfSense I dont even see anything in the GUI regarding healthchecks. Weird. Not sure how this option was even turned on in the first place.

Been using XO with delta backups for a while however one of my VMs is now receiving an error and I'm not sure how to rectify. I'll post what I have below. Looks like there might be two errors -- VHD check error and WaitObjectState Error. Not sure how to actually fix any of these errors.

Thanks for help.

     Clean VM directory
    VHD check error
        path
        "/xo-vm-backups/a8a20272-aaef-0bc9-82c8-d5ba96d1e725/vdis/fed92d10-c8a0-4587-a018-8a6b6c050453/60e7144d-ea34-4cf7-b798-c0bfd17134c2/.20240204T075759Z.vhd"
        error
        {"generatedMessage":false,"code":"ERR_ASSERTION","actual":false,"expected":true,"operator":"=="}
    Start: Mar 4, 2024, 01:00:02 AM
    End: Mar 4, 2024, 01:00:02 AM
    Snapshot
    Start: Mar 4, 2024, 01:00:02 AM
    End: Mar 4, 2024, 01:00:04 AM
    Backups for XO
        transfer
        Start: Mar 4, 2024, 01:00:06 AM
        End: Mar 4, 2024, 01:18:53 AM
        Duration: 19 minutes
        Size: 4.34 GiB
        Speed: 3.94 MiB/s
        health check
            transfer
            Start: Mar 4, 2024, 01:19:00 AM
            End: Mar 4, 2024, 05:34:33 AM
            Duration: 4 hours
            Size: 9.3 GiB
            Speed: 635.79 KiB/s
            vmstart
            Start: Mar 4, 2024, 05:34:33 AM
            End: Mar 4, 2024, 05:44:33 AM
            Error: waitObjectState: timeout reached before OpaqueRef:214f572e-ed48-4f80-a9c7-6f984129a2f7 in expected state
        Start: Mar 4, 2024, 01:19:00 AM
        End: Mar 4, 2024, 05:44:37 AM
        Error: waitObjectState: timeout reached before OpaqueRef:214f572e-ed48-4f80-a9c7-6f984129a2f7 in expected state
    Start: Mar 4, 2024, 01:00:04 AM
    End: Mar 4, 2024, 05:44:37 AM
    Duration: 5 hours
    Error: waitObjectState: timeout reached before OpaqueRef:214f572e-ed48-4f80-a9c7-6f984129a2f7 in expected state

Start: Mar 4, 2024, 01:00:02 AM
End: Mar 4, 2024, 05:44:37 AM
Duration: 5 hours
Error: waitObjectState: timeout reached before OpaqueRef:214f572e-ed48-4f80-a9c7-6f984129a2f7 in expected state
Type: delta
code_text

@julien-f Thanks for explanation. Thank you

@olivierlambert

Ok I looked at the hostname directive and changed it to the specific IP address. However just a few thoughts since honestly I've never thought about it before.

If I had two physical or virtual NICs assigned to a xcp-ng VM -- say eth0 and eth1 -- how does the program by default decide on which NIC its going to bind it's ports by default? Is it always the card assigned to eth0 (since eth0 can be manipulated by systemd network setting so it may not necessarily represent the first actual card brought up on the bus architecture)?

I also thought hostname was used in XO's acme plugin which would could be used to generate automatic acme LE certs. If you change the hostname to an actual IP address, isn't this process going to be altered?

Thanks for your insights. I don't mind disaster recovery since no matter how many times you practice or simulate things, it seems I learn the most when the actual S**T hits the fan.

@olivierlambert Hey thanks for the suggestion. I'm pretty sure it's probably a problem with the underlying lvm hardware, but its funny, taking a look at Dom0, I don't see anything mentioning any disk related problem.

Sample of dmesg log below

[805638.417594] block tde: sector-size: 512/512 capacity: 419430400
[805641.656976] vif vif-30-1 vif30.1: Guest Rx ready
[805649.655582] vif vif-30-1 vif30.1: Guest Rx stalled
[805651.179897] device vif32.0 entered promiscuous mode
[805655.092408] device tap32.0 entered promiscuous mode
[805658.494363] device tap32.0 left promiscuous mode
[805659.661403] vif vif-30-1 vif30.1: Guest Rx ready
[805692.482092] device vif32.0 left promiscuous mode
[805702.199871] vif vif-30-1 vif30.1: Guest Rx stalled
[805711.543432] vif vif-30-1 vif30.1: Guest Rx ready
[805719.664229] vif vif-30-1 vif30.1: Guest Rx stalled
[805729.659247] vif vif-30-1 vif30.1: Guest Rx ready
[805745.392833] block tde: sector-size: 512/512 capacity: 419430400
[805749.905041] device vif30.1 left promiscuous mode
[805752.496307] device vif33.0 entered promiscuous mode
[805755.222166] device vif30.0 left promiscuous mode
[805756.847363] device tap33.0 entered promiscuous mode
[805799.443948] device tap33.0 left promiscuous mode
[805804.852848] vif vif-33-0 vif33.0: Guest Rx ready
[830198.036797] device vif33.0 left promiscuous mode

In terms of backups --- kind of a sticky issue. Yes I have delta backups on a FreeNAS partition. Is there documentation on how to actually restore these backups if starting from scratch? By scratch I mean lets say no hardware disks with a new XO installation?

Here is my backup directory structure BTW in case things aren't exactly clear:

freenas% pwd
/mnt/tank/backups/Xen
freenas% ls
1632582667671.test			encryption.json
1633705775069.test			metadata.json
1668267335259.test			xo-config-backups
1f5adaf3-7631-d478-3c74-468c48079177	xo-pool-metadata-backups
66efa31e-5595-dda6-5ce9-dc2a1bb26cb9	xo-vm-backups
c514822f-74bb-bfde-77d8-8f2b0c0b844b

Not sure where to start here.

I'm running xcp-ng 8.2.1 on a Proctetli box.

My actual partitions on the hardware are as follows:

NAME                       MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sdb                          8:16   0 931.5G  0 disk
└─XSLocalEXT--99f0de78--37cb--ead5--0c56--bd5e341416aa-99f0de78--37cb--ead5--0c56--bd5e341416aa
                           253:0    0   1.8T  0 lvm  /run/sr-mount/99f0de78-37cb-ead5-0c56-bd5e341416aa
tdc                        254:2    0    10G  0 disk
tda                        254:0    0   100G  0 disk
sda                          8:0    0 931.5G  0 disk
├─sda4                       8:4    0   512M  0 part
├─sda2                       8:2    0    18G  0 part
├─sda5                       8:5    0     4G  0 part /var/log
├─sda3                       8:3    0   890G  0 part
│ └─XSLocalEXT--99f0de78--37cb--ead5--0c56--bd5e341416aa-99f0de78--37cb--ead5--0c56--bd5e341416aa
                           253:0    0   1.8T  0 lvm  /run/sr-mount/99f0de78-37cb-ead5-0c56-bd5e341416aa
├─sda1                       8:1    0    18G  0 part /
└─sda6                       8:6    0     1G  0 part [SWAP]
tdd                        254:3    0 834.2M  1 disk
tdb                        254:1    0   721M  1 disk

Within the actual xcp-ng host I'm using local storage which is the LVM 1.8T partition.
I have a number of VMs on the host, however at the most I had either 4/5 running.
VMs on the host are either Arch Linux, Ubuntu Linux or pfsense. Currently I'm having a problem with all the Ubuntu and Arch VMs.

I believe most of the VMs that were created were created with partition scheme of /dev/xvda1 --> boot partition, /dev/xvda2 ---> root partition, /dev/xvda3 ---> swap partition.

When attempting to boot the Arch or Ubuntu VM's, I'm getting i/o errors when trying to mount the /dev/xvda2 or the root partition.

Although I haven't troubleshooted every VM, I've tried the following:

1. VM boots to recovery or busybox shell, try fsck /dev/xvda2 however process doesn't work
2. Boot a rescue CD (such as an Arch Install Disk), and then try fsck /dev/xvda2.
   When trying such an approach I'm seeing the following:

# lsblk                                                                                                  :(
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0     7:0    0 607.1M  1 loop /run/archiso/sfs/airootfs
sr0      11:0    1   721M  0 rom  /run/archiso/bootmnt
xvda    202:0    0   100G  0 disk
├─xvda1 202:1    0     1M  0 part
└─xvda2 202:2    0   100G  0 part

# fsck -yv /dev/xvda2
fsck from util-linux 2.34
e2fsck 1.45.4 (23-Sep-2019)
/dev/xvda2: recovering journal
Superblock needs_recovery flag is clear, but journal has data.
Run journal anyway? yes

fsck.ext4: Input/output error while recovering journal of /dev/xvda2
fsck.ext4: unable to set superblock flags on /dev/xvda2


/dev/xvda2: ********** WARNING: Filesystem still has errors **********

I've seen similar error when working with physical disk, however xvda represents virtual partitions.
I'm I just totally hosed here in terms of recovery?? I'm a little stumped how to recover.

I’m running Xen Orchestra on an Ubuntu VM with two virtual NICs attached to VM. The two NICs are designated eth0 and wg1. wg1 is a wireguard NIC and eth0 is a non-wireguard NIC.

Within the xen-orchestra configuration xo-server.toml, all I see is the option to bind to a specific port

[[http.listen]]
port = 443

When starting the xo-server.service, xo-server listens on port 443 however this port seems bound to the eth0 interface, not the wg1 interface.

I’d actually like xo-server to listen on wg1 and not on eth0. I’m unfortunately not a networking expert nor expert using xen-orchestra. Is it possible to configure xo-server to listen on a specific NIC?

If looking at server I have xo server bound to IP address of the eth0 NIC:

# netstat -tulpn
...
tcp        0      0 142.xx.xxx.xxx:443      0.0.0.0:*               LISTEN      2258771/node
...

Current NICs on the system:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 1a:0b:43:61:c3:70 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 142.xxx.xxx.xxx/20 brd 142.93.127.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.10.0.5/16 brd 10.10.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::180b:43ff:fe61:c370/64 scope link
       valid_lft forever preferred_lft forever
3: wg1: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none
    inet 10.x.xxx.x/24 scope global wg1
       valid_lft forever preferred_lft forever

Hi currently running xcp-ng 8.2 and within my small form factor I have 4 VMs. One of the VMs is currently virtualized pfsense. When xcp-ng boots I'd actually like the pfsense VM to start first. Is it possible to assign in XO or through the xcp-ng CLI itself the proposed boot order of the various VMs?

@tjkreidl

So here is the dilemma I'm running into, the host I added with the incorrect network ports and such. It was reachable via the internet until I upgraded some pool patches, now its extremely unreachable. I can boot the installation, however I can't reach the slave to eject the host.

I tried emergency resetting network, however that didn't work either.

lshw -C network reveals a lot of unclaimed ethernet controllers which are the Intel 225-V version. I looked up various internet resources to see what "unclaimed" meant, and these sources suggest there isn't an appropriate driver -- which is weird since prior to the pool patches applied this installation was up and running (but misconfigured).

Sooo -- I either have two options at this point, and I don't really care. I'd like to eject this pool member -- destroy it -- however it seems I can't since it's unreachable, or I need to reinstall the I225V network driver.

Any suggestions?