Mark C

Mark C

@maverick

Fab. Glad to see you got there and it builds on Oracle Linux ARM64 / aarch64 too! It'd be good to hear if you come across any functions that don't work on the aarch64 build.

I'm also sorry for the missing dependencies. I guess I had installed those early on in my attempts to get this compiled in August and missed them from the instructions. I'll go back and edit my post above to document them.

Cheers,

Mark

Mark C

@maverick

I don't have an oracle 8.8 build environment to check directly, but this error seems to imply node-gyp can't be found / started, or it is looking in the wrong place / there's a path difference between Oracle and Debian build environments.

code: 'ENOENT',
  syscall: 'spawn node-gyp',
  path: 'node-gyp',
  spawnargs: [ 'rebuild' ]

Try 'npm install -g node-gyp' to make sure it is present? The same for node-gyp-build.

Do you also have the pkg-config rpm installed? And the native fuse libs rpm? The sagemathinc package will need these.

Cheers,

Mark

PS. I did run a 'yarn build' in my Debian VM and it built successfully after I gave it 3GB of RAM.

EDIT 24/Oct/2023: the -devel fuse libraries are needed too.

Mark C

Sorry for the bad paste of the diff. Quoted properly here.

xo_test@xo2:~/build$ diff -u fuse-native.orig fuse-native
Common subdirectories: fuse-native.orig/.git and fuse-native/.git
diff -u fuse-native.orig/package.json fuse-native/package.json
--- fuse-native.orig/package.json	2023-10-23 14:25:18.067638984 +0100
+++ fuse-native/package.json	2023-10-23 14:29:19.703990676 +0100
@@ -1,6 +1,6 @@
 {
-  "name": "@cocalc/fuse-native",
-  "version": "2.4.0",
+  "name": "fuse-native",
+  "version": "2.2.6",
   "description": "Node.js bindings for the FUSE api (Filesystem in Userspace)",
   "main": "index.js",
   "bin": {
@@ -28,7 +28,7 @@
   },
   "repository": {
     "type": "git",
-    "url": "https://github.com/sagemathinc/fuse-native.git"
+    "url": "https://github.com/fuse-friends/fuse-native.git"
   },
   "author": {
     "name": "William Stein (SageMath, Inc.)",
@@ -44,7 +44,7 @@
   "author": " ()",
   "license": "MIT",
   "bugs": {
-    "url": "https://github.com/sagemathinc/fuse-native/issues"
+    "url": "https://github.com/fuse-friends/fuse-native/issues"
   },
-  "homepage": "https://github.com/sagemathinc/fuse-native"
+  "homepage": "https://github.com/fuse-friends/fuse-native"
 }
Common subdirectories: fuse-native.orig/test and fuse-native/test

Mark C

@maverick Hi Maverick,

I just tried a test build prep and it seems to work for me. I created a new user, xo_test and swapped to it, then:


xo_test@xo2:~$ mkdir build
xo_test@xo2:~$ cd build/
xo_test@xo2:~/build$ git clone https://github.com/sagemathinc/fuse-native
Cloning into 'fuse-native'...
remote: Enumerating objects: 409, done.
remote: Counting objects: 100% (409/409), done.
remote: Compressing objects: 100% (160/160), done.
remote: Total 409 (delta 259), reused 396 (delta 246), pack-reused 0
Receiving objects: 100% (409/409), 150.36 KiB | 2.64 MiB/s, done.
Resolving deltas: 100% (259/259), done.

xo_test@xo2:~/build$ git clone https://github.com/vatesfr/xen-orchestra.git
Cloning into 'xen-orchestra'...
remote: Enumerating objects: 119983, done.
remote: Counting objects: 100% (3095/3095), done.
remote: Compressing objects: 100% (1400/1400), done.
remote: Total 119983 (delta 2143), reused 2278 (delta 1654), pack-reused 116888
Receiving objects: 100% (119983/119983), 64.38 MiB | 9.78 MiB/s, done.
Resolving deltas: 100% (86309/86309), done.

xo_test@xo2:~/build$ cp -a fuse-native fuse-native.orig
xo_test@xo2:~/build/fuse-native$ nano -w fuse-native/package.json 

xo_test@xo2:~/build$ diff -u fuse-native.orig fuse-native
Common subdirectories: fuse-native.orig/.git and fuse-native/.git
diff -u fuse-native.orig/package.json fuse-native/package.json
--- fuse-native.orig/package.json	2023-10-23 14:25:18.067638984 +0100
+++ fuse-native/package.json	2023-10-23 14:29:19.703990676 +0100
@@ -1,6 +1,6 @@
 {
-  "name": "@cocalc/fuse-native",
-  "version": "2.4.0",
+  "name": "fuse-native",
+  "version": "2.2.6",
   "description": "Node.js bindings for the FUSE api (Filesystem in Userspace)",
   "main": "index.js",
   "bin": {
@@ -28,7 +28,7 @@
   },
   "repository": {
     "type": "git",
-    "url": "https://github.com/sagemathinc/fuse-native.git"
+    "url": "https://github.com/fuse-friends/fuse-native.git"
   },
   "author": {
     "name": "William Stein (SageMath, Inc.)",
@@ -44,7 +44,7 @@
   "author": " ()",
   "license": "MIT",
   "bugs": {
-    "url": "https://github.com/sagemathinc/fuse-native/issues"
+    "url": "https://github.com/fuse-friends/fuse-native/issues"
   },
-  "homepage": "https://github.com/sagemathinc/fuse-native"
+  "homepage": "https://github.com/fuse-friends/fuse-native"
 }
Common subdirectories: fuse-native.orig/test and fuse-native/test

xo_test@xo2:~/build$ cp -a fuse-native xen-orchestra
xo_test@xo2:~/build$ cd xen-orchestra/
xo_test@xo2:~/build/xen-orchestra$ cd fuse-native/
xo_test@xo2:~/build/xen-orchestra/fuse-native$ yarn link
yarn link v1.22.19
success Registered "fuse-native".
info You can now run `yarn link "fuse-native"` in the projects where you want to use this package and it will be used instead.
Done in 0.15s.

xo_test@xo2:~/build/xen-orchestra/fuse-native$ cd ..
xo_test@xo2:~/build/xen-orchestra$ yarn link fuse-native
yarn link v1.22.19
success Using linked package for "fuse-native".
Done in 0.13s.

xo_test@xo2:~/build/xen-orchestra$ yarn
yarn install v1.22.19
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
warning url-loader@1.1.2: Invalid bin field for "url-loader".
[4/5] Linking dependencies...
warning "workspace-aggregator-6dab12fa-fabf-4759-8627-2058ff251bdf > @vates/node-vsphere-soap > soap@1.0.0" has incorrect peer dependency "axios@^0.27.2".
warning Workspaces can only be enabled in private projects.
[5/5] Building fresh packages...
$ husky install
husky - Git hooks installed
Done in 425.65s.

I haven't kicked off the yarn build, but it was the yarn stage that failed for you.

Does the above help? [ As I said back in August, manually editing the package.json for sagemathinc's fuse-native to match the fuse-friends one that is called out in the xen-orchestra git tree is a completely horrible hack, but it does seem to work for me. ]

xo_test@xo2:~/build$ cat /etc/debian_version 
12.2

Cheers,

Mark

EDIT 24/Oct/2023: The build needs node-gyp and node-gyp-build packages installing with npm, and the fuse libraries and devel packages and pkg-config for the operating system, along with the pre-reqs noted in the XO from the Sources build instructions in the wiki. The build host needs a decent amount of RAM - it built for me in 3GB but failed with 1GB. Once built, it runs fine for me with 1GB.

Mark C

@florent said in libfuse.so: error adding symbols: File in wrong format:

@Mark-C I can't test easily for now, but I think there may be an easier path after reviewing your solution

fuse-native use fuse-shared-library 1.0.2
from version 1.1 , fuse-shared-library should support ARM , maybe it is possible to bump it in the dependencies of fuse-native ?

This way , it should still work without libfuse on the system, but also build it on arm

Hi @florent ,

I am building on aarch64 / ARM64 rather than just arm.

I did look at fuse-shared-library-linux-arm library - the embedded libfuse.so in that is a 32 bit version rather than 64 bit from what I could see:

root@xo-aarch64:/home/xo/xoi/xo-builds/xen-orchestra-202308250931/node_modules/fuse-shared-library-linux-arm/libfuse/lib# file libfuse.so
libfuse.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=6ed4a4b321ab69af9aeef3db322ba67396f4d52b, stripped

That's in contrast to the system library:

/usr/lib/aarch64-linux-gnu/libfuse.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=29ce785f4418d869ead1b31501c4a82446de0cfe, stripped

@olivierlambert - I do appreciate the support issue for existing platforms and users. This is very much me looking at using a turing pi 2 as a low power setup in my home lab and I appreciate it is a niche case.

The more supportable fix would be to add ARM64 support into the fuse-friends version of fuse-native, but it seems that node's current policies don't allow adoption of orphaned/abandoned modules to avoid supply-chain attacks. [ https://docs.npmjs.com/policies/disputes - When not to use this process ].

Mark C

The typical hypervisor node was provisioned with at least 4 network ports, but most had 8 to allow trunking/resilience for every connection:

Management + live migration (Isolated)
World (Trunk)
iSCSI-A
iSCSI-B

The DotHills were 4 ports per controller, and the two controllers were configured as:

Management
iSCSI-A
iSCSI-B
Spare

The iSCSI-A and iSCSI-B were simple web-managed procurves with 24 ports each.

We had this scaled to 12 hypervisor nodes in 3 pools, all running off the same underlying dothill (dual redundant controller) hardware. We had provisioned some local SSD storage in each hypervisor for any nodes that were struggling for IOPs but ultimately never needed that.

Mark C

@splastunov The DotHill certainly wasn't great for scaling out, because of the age of the firmware and the old-school treatment of the disk arrays.

We did deploy several disk expansion trays that were part populated at purchase as some level of expansion, and operated most of the array as a single underlying pool, which was then split up into iSCSI presented volumes.

So the limitations of the DotHill controllers and firmware we were using didn't address the storage pool scaling issue that CEPH / GPFS / etal are designed to.

However, some of the newer filer heads that treat their entire available disk array as a consolidated pool and support iSCSI would have that scalability, whilst keeping the hypervisor side relatively straight-forward.

CEPH etal are also in a much more mature place than they were back in the early 2010s too, so if I was doing it again, maybe we would go for a solution architecture based around that. Or XOSTOR, of course!

My intent was to present an iSCSI setup instance as something we found relatively bomb-proof, and post-initial configuration, relatively easy to scale. We provisioned 24 port A and B side network switches with jumbo frame support to give us the scalability to add further pools of hypervisors or storage, as required.

Mark C

@splastunov / all.

Just on multipath iSCSI. I had this running for years with separate switches running the 'A side' and 'B side' multipath networks between the iSCSI dot hill's resilient and redundant controllers.

Once you have it configured, it is incredibly solid for what is a very budget solution. It just takes a bit of careful planning up front to make sure you wire up in a way that is properly resilient. Plus reliable monitoring to make sure that you notice when one of the resilient elements fails.

We had a number of switch power supply failures over the 10 year period that were completely transparent to the services running on the connected XCP-NG and XenServer hosts.

Similarly we were able to do a full DotHill controller replacement without any downtime after one of the two controllers failed.

We also replaced the hypervisor hardware a number of times over the lifetime of the platform.

Mark C

@lover

See /etc/yum.repos.d/xcp-ng.repo and update all references from http:/ to https:/

Mark C

@maverick

Fab. Glad to see you got there and it builds on Oracle Linux ARM64 / aarch64 too! It'd be good to hear if you come across any functions that don't work on the aarch64 build.

I'm also sorry for the missing dependencies. I guess I had installed those early on in my attempts to get this compiled in August and missed them from the instructions. I'll go back and edit my post above to document them.

Cheers,

Mark

Mark C

@maverick

I don't have an oracle 8.8 build environment to check directly, but this error seems to imply node-gyp can't be found / started, or it is looking in the wrong place / there's a path difference between Oracle and Debian build environments.

code: 'ENOENT',
  syscall: 'spawn node-gyp',
  path: 'node-gyp',
  spawnargs: [ 'rebuild' ]

Try 'npm install -g node-gyp' to make sure it is present? The same for node-gyp-build.

Do you also have the pkg-config rpm installed? And the native fuse libs rpm? The sagemathinc package will need these.

Cheers,

Mark

PS. I did run a 'yarn build' in my Debian VM and it built successfully after I gave it 3GB of RAM.

EDIT 24/Oct/2023: the -devel fuse libraries are needed too.

Mark C

Sorry for the bad paste of the diff. Quoted properly here.

xo_test@xo2:~/build$ diff -u fuse-native.orig fuse-native
Common subdirectories: fuse-native.orig/.git and fuse-native/.git
diff -u fuse-native.orig/package.json fuse-native/package.json
--- fuse-native.orig/package.json	2023-10-23 14:25:18.067638984 +0100
+++ fuse-native/package.json	2023-10-23 14:29:19.703990676 +0100
@@ -1,6 +1,6 @@
 {
-  "name": "@cocalc/fuse-native",
-  "version": "2.4.0",
+  "name": "fuse-native",
+  "version": "2.2.6",
   "description": "Node.js bindings for the FUSE api (Filesystem in Userspace)",
   "main": "index.js",
   "bin": {
@@ -28,7 +28,7 @@
   },
   "repository": {
     "type": "git",
-    "url": "https://github.com/sagemathinc/fuse-native.git"
+    "url": "https://github.com/fuse-friends/fuse-native.git"
   },
   "author": {
     "name": "William Stein (SageMath, Inc.)",
@@ -44,7 +44,7 @@
   "author": " ()",
   "license": "MIT",
   "bugs": {
-    "url": "https://github.com/sagemathinc/fuse-native/issues"
+    "url": "https://github.com/fuse-friends/fuse-native/issues"
   },
-  "homepage": "https://github.com/sagemathinc/fuse-native"
+  "homepage": "https://github.com/fuse-friends/fuse-native"
 }
Common subdirectories: fuse-native.orig/test and fuse-native/test

Mark C

@maverick Hi Maverick,

I just tried a test build prep and it seems to work for me. I created a new user, xo_test and swapped to it, then:


xo_test@xo2:~$ mkdir build
xo_test@xo2:~$ cd build/
xo_test@xo2:~/build$ git clone https://github.com/sagemathinc/fuse-native
Cloning into 'fuse-native'...
remote: Enumerating objects: 409, done.
remote: Counting objects: 100% (409/409), done.
remote: Compressing objects: 100% (160/160), done.
remote: Total 409 (delta 259), reused 396 (delta 246), pack-reused 0
Receiving objects: 100% (409/409), 150.36 KiB | 2.64 MiB/s, done.
Resolving deltas: 100% (259/259), done.

xo_test@xo2:~/build$ git clone https://github.com/vatesfr/xen-orchestra.git
Cloning into 'xen-orchestra'...
remote: Enumerating objects: 119983, done.
remote: Counting objects: 100% (3095/3095), done.
remote: Compressing objects: 100% (1400/1400), done.
remote: Total 119983 (delta 2143), reused 2278 (delta 1654), pack-reused 116888
Receiving objects: 100% (119983/119983), 64.38 MiB | 9.78 MiB/s, done.
Resolving deltas: 100% (86309/86309), done.

xo_test@xo2:~/build$ cp -a fuse-native fuse-native.orig
xo_test@xo2:~/build/fuse-native$ nano -w fuse-native/package.json 

xo_test@xo2:~/build$ diff -u fuse-native.orig fuse-native
Common subdirectories: fuse-native.orig/.git and fuse-native/.git
diff -u fuse-native.orig/package.json fuse-native/package.json
--- fuse-native.orig/package.json	2023-10-23 14:25:18.067638984 +0100
+++ fuse-native/package.json	2023-10-23 14:29:19.703990676 +0100
@@ -1,6 +1,6 @@
 {
-  "name": "@cocalc/fuse-native",
-  "version": "2.4.0",
+  "name": "fuse-native",
+  "version": "2.2.6",
   "description": "Node.js bindings for the FUSE api (Filesystem in Userspace)",
   "main": "index.js",
   "bin": {
@@ -28,7 +28,7 @@
   },
   "repository": {
     "type": "git",
-    "url": "https://github.com/sagemathinc/fuse-native.git"
+    "url": "https://github.com/fuse-friends/fuse-native.git"
   },
   "author": {
     "name": "William Stein (SageMath, Inc.)",
@@ -44,7 +44,7 @@
   "author": " ()",
   "license": "MIT",
   "bugs": {
-    "url": "https://github.com/sagemathinc/fuse-native/issues"
+    "url": "https://github.com/fuse-friends/fuse-native/issues"
   },
-  "homepage": "https://github.com/sagemathinc/fuse-native"
+  "homepage": "https://github.com/fuse-friends/fuse-native"
 }
Common subdirectories: fuse-native.orig/test and fuse-native/test

xo_test@xo2:~/build$ cp -a fuse-native xen-orchestra
xo_test@xo2:~/build$ cd xen-orchestra/
xo_test@xo2:~/build/xen-orchestra$ cd fuse-native/
xo_test@xo2:~/build/xen-orchestra/fuse-native$ yarn link
yarn link v1.22.19
success Registered "fuse-native".
info You can now run `yarn link "fuse-native"` in the projects where you want to use this package and it will be used instead.
Done in 0.15s.

xo_test@xo2:~/build/xen-orchestra/fuse-native$ cd ..
xo_test@xo2:~/build/xen-orchestra$ yarn link fuse-native
yarn link v1.22.19
success Using linked package for "fuse-native".
Done in 0.13s.

xo_test@xo2:~/build/xen-orchestra$ yarn
yarn install v1.22.19
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
warning url-loader@1.1.2: Invalid bin field for "url-loader".
[4/5] Linking dependencies...
warning "workspace-aggregator-6dab12fa-fabf-4759-8627-2058ff251bdf > @vates/node-vsphere-soap > soap@1.0.0" has incorrect peer dependency "axios@^0.27.2".
warning Workspaces can only be enabled in private projects.
[5/5] Building fresh packages...
$ husky install
husky - Git hooks installed
Done in 425.65s.

I haven't kicked off the yarn build, but it was the yarn stage that failed for you.

Does the above help? [ As I said back in August, manually editing the package.json for sagemathinc's fuse-native to match the fuse-friends one that is called out in the xen-orchestra git tree is a completely horrible hack, but it does seem to work for me. ]

xo_test@xo2:~/build$ cat /etc/debian_version 
12.2

Cheers,

Mark

EDIT 24/Oct/2023: The build needs node-gyp and node-gyp-build packages installing with npm, and the fuse libraries and devel packages and pkg-config for the operating system, along with the pre-reqs noted in the XO from the Sources build instructions in the wiki. The build host needs a decent amount of RAM - it built for me in 3GB but failed with 1GB. Once built, it runs fine for me with 1GB.

Mark C

Hi @maverick,

I've not done much with this since my 'bodge' to build an ARM64 version back in August.

The version I have running on a Raspberry Pi CM4 from back then does appear to be functioning as expected for everything I have tried. ( Basic start / stop / backup / restore / create / destroy for a small XCP-NG cluster running on x64. ). That satisfies my use case to have XO from the sources running on something that isn't the cluster it is managing.

I do understand the support issues @olivierlambert flagged up in August, and I don't have any good solutions to resolve those for existing deployments of XOA doing upgrades rather than fresh installs. That's something the real maintainers will have to consider around the (minor) benefit of building for ARM64 vs any related support overheads of existing XOA upgrade issues.

Sincerely,

Mark

Mark C

@florent said in libfuse.so: error adding symbols: File in wrong format:

@Mark-C I can't test easily for now, but I think there may be an easier path after reviewing your solution

fuse-native use fuse-shared-library 1.0.2
from version 1.1 , fuse-shared-library should support ARM , maybe it is possible to bump it in the dependencies of fuse-native ?

This way , it should still work without libfuse on the system, but also build it on arm

Hi @florent ,

I am building on aarch64 / ARM64 rather than just arm.

I did look at fuse-shared-library-linux-arm library - the embedded libfuse.so in that is a 32 bit version rather than 64 bit from what I could see:

root@xo-aarch64:/home/xo/xoi/xo-builds/xen-orchestra-202308250931/node_modules/fuse-shared-library-linux-arm/libfuse/lib# file libfuse.so
libfuse.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=6ed4a4b321ab69af9aeef3db322ba67396f4d52b, stripped

That's in contrast to the system library:

/usr/lib/aarch64-linux-gnu/libfuse.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=29ce785f4418d869ead1b31501c4a82446de0cfe, stripped

@olivierlambert - I do appreciate the support issue for existing platforms and users. This is very much me looking at using a turing pi 2 as a low power setup in my home lab and I appreciate it is a niche case.

The more supportable fix would be to add ARM64 support into the fuse-friends version of fuse-native, but it seems that node's current policies don't allow adoption of orphaned/abandoned modules to avoid supply-chain attacks. [ https://docs.npmjs.com/policies/disputes - When not to use this process ].

Mark C

@Mark C

Best posts made by Mark C

Latest posts made by Mark C