Move build to use npm and Turbo

julien-f

Hello everyone,

We have been using Yarn 1 in XO because it had support for workspaces, which npm lacked at the time.
But now that npm has caught up, and Yarn 1 is frozen, I think it's time to get back to using it.

Also, we are thinking about using Turborepo instead of our own custom scripts to build the repository. This would, among others, bring caching, i.e. packages are no longer rebuilt if there were no changes.

I hope these changes will not introduce any problems (I'm thinking about https://xcp-ng.org/forum/post/54567).

I you want to test, please take a look at this PR: https://github.com/vatesfr/xen-orchestra/pull/6559

Keep me posted if you have encounter any issues

julien-f opened this pull request in vatesfr/xen-orchestra

draft chore: use npm #6559

Danp

Seems to work fine here.

On a separate issue, what's the plan to address the existing vulnerabilities (see below)? Will any of these be addressed prior to the move to 6.x?

81 vulnerabilities (12 low, 14 moderate, 49 high, 6 critical)

Andrew

@julien-f Branch npm-turbo builds for me too. Seemed to take about 25% longer than before.

julien-f

@Andrew My test results:

install time: Yarn 17.58s → npm 50.80s
build time: custom 71.68s → Turborepo 76.13s

I agree the install time is not very good with npm, I'll try to understand why and if we can improve the situation, still we can probably stay compatible with both, and maybe even try to be compatible with pnpm.

With Turborepo and the cache system, the build time is greatly improved in following runs (~10s in my second one), so I think that's a plus

julien-f

@Danp I don't think so, from what I can tell the concerned packages are used by xo-web build system. Their (theorical) vulnerabilities should have no effect in prod and upgrading them would probably be too hard.

Small targeted PRs fixing some of these issues are welcome though.

julien-f

@Danp And of course if there is a real risk/issue, keep us posted and we'll take care of it

julien-f

Turborepo integrated: https://github.com/vatesfr/xen-orchestra/commit/3bfd6c6979a16e7a22b8bea1cb30529b1a0fb3c2

Let me know if you have any issues.

0

julien-f committed to vatesfr/xen-orchestra

chore: use Turborepo to build

Why?

- ordering: build dependencies before dependents
- cache: don't rebuild if no changes in files or dependencies
- possibility to restrict to specific scopes

Changes:

- `yarn build` now only build `xo-server` and `xo-web` (and dependencies)
- `yarn build:xo-lite` build `@xen-orchestra/lite\ (and dependencies)

zizzithefox

@julien-f I guess that rules out freebsd as a platform for building xen orchestra from sources.

I have been using it in a Tuenas core jail, but it's no use because I have found that this stuff is not supported:

turborepo
fuse-shared-library (https://xcp-ng.org/forum/topic/6564/xen-orchestra-unsupported-on-freebsd-because-of-fuse-shared-library?_=1673445212328)
who knows what else

Not ranting about it, I understand if you need better tools. Maybe you should remove freebsd from the build instructions? Just saying...

Best regards.

julien-f

@zizzithefox Thanks for your feedback.

Turborepo should work on FreeBSD, what issues do you have?

The FreeBSD instructions has been contributed by the community, I think it's nice to keep it as long as not completely broken, I've added a disclaimer expliciting it's not officially supported (it will be visible soon)

zizzithefox

@julien-f said in Move build to use npm and Turbo:

should work on FreeBSD, what issues do you have?
The FreeBSD instructions has been contributed by the comm

It seems to me like they have explicitly excluded any platform that is neither windows nor linux (nor darwin):

root@xo:/srv/xo # yarn
yarn install v1.22.18
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
[4/5] Linking dependencies...
warning "workspace-aggregator-9be715c1-80fc-4c5d-b807-b5414e3a3afa > @xen-orchestra/fs > @aws-sdk/lib-storage@3.241.0" has unmet peer dependency "@aws-sdk/abort-controller@^3.0.0".
warning Workspaces can only be enabled in private projects.
[5/5] Building fresh packages...
[1/20] ⢀ turbo
[6/20] ⢀ @fortawesome/fontawesome-common-types
[3/20] ⢀ highlight.js
[7/20] ⢀ vue-demi
error /srv/xo/node_modules/turbo: Command failed.
Exit code: 1
Command: node install.js
Arguments: 
Directory: /srv/xo/node_modules/turbo
Output:
/srv/xo/node_modules/turbo/node-platform.js:38
    throw new Error(`Unsupported platform: ${platformKey}`);
          ^

Error: Unsupported platform: freebsd x64 LE
    at pkgAndSubpathForCurrentPlatform (/srv/xo/node_modules/turbo/node-platform.js:38:11)
    at checkAndPreparePackage (/srv/xo/node_modules/turbo/install.js:252:28)
    at Object.<anonymous> (/srv/xo/node_modules/turbo/install.js:302:1)
    at Module._compile (node:internal/modules/cjs/loader:1155:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1209:10)
    at Module.load (node:internal/modules/cjs/loader:1033:32)

I have found this (seems it's the correct project):
https://github.com/vercel/turbo/pull/2408
and this:
https://github.com/vercel/turbo/issues/2339
I guess I could modify the sources, compile from sources with patches and make it to work somehow... but it's probably better to just move to linux.

yonas created this issue in vercel/turbo

closed [turborepo] Failing to build on FreeBSD 13 #2339

yonas opened this pull request in vercel/turbo

closed [turborepo] Fix build on FreeBSD #2408

julien-f

@zizzithefox Indeed, this is probably easier, thanks for the info