Move build to use npm and Turbo
-
Hello everyone,
We have been using Yarn 1 in XO because it had support for workspaces, which
npm
lacked at the time.
But now thatnpm
has caught up, and Yarn 1 is frozen, I think it's time to get back to using it.Also, we are thinking about using Turborepo instead of our own custom scripts to build the repository. This would, among others, bring caching, i.e. packages are no longer rebuilt if there were no changes.
I hope these changes will not introduce any problems (I'm thinking about https://xcp-ng.org/forum/post/54567).
I you want to test, please take a look at this PR: https://github.com/vatesfr/xen-orchestra/pull/6559
Keep me posted if you have encounter any issues
-
Seems to work fine here.
On a separate issue, what's the plan to address the existing vulnerabilities (see below)? Will any of these be addressed prior to the move to 6.x?
81 vulnerabilities (12 low, 14 moderate, 49 high, 6 critical)
-
@julien-f Branch
npm-turbo
builds for me too. Seemed to take about 25% longer than before. -
@Andrew My test results:
- install time: Yarn 17.58s → npm 50.80s
- build time: custom 71.68s → Turborepo 76.13s
I agree the install time is not very good with npm, I'll try to understand why and if we can improve the situation, still we can probably stay compatible with both, and maybe even try to be compatible with pnpm.
With Turborepo and the cache system, the build time is greatly improved in following runs (~10s in my second one), so I think that's a plus
-
@Danp I don't think so, from what I can tell the concerned packages are used by xo-web build system. Their (theorical) vulnerabilities should have no effect in prod and upgrading them would probably be too hard.
Small targeted PRs fixing some of these issues are welcome though.
-
@Danp And of course if there is a real risk/issue, keep us posted and we'll take care of it
- about a month later
-
Turborepo integrated: https://github.com/vatesfr/xen-orchestra/commit/3bfd6c6979a16e7a22b8bea1cb30529b1a0fb3c2
Let me know if you have any issues.
-
@julien-f I guess that rules out freebsd as a platform for building xen orchestra from sources.
I have been using it in a Tuenas core jail, but it's no use because I have found that this stuff is not supported:
- turborepo
- fuse-shared-library (https://xcp-ng.org/forum/topic/6564/xen-orchestra-unsupported-on-freebsd-because-of-fuse-shared-library?_=1673445212328)
- who knows what else
Not ranting about it, I understand if you need better tools. Maybe you should remove freebsd from the build instructions? Just saying...
Best regards.
-
@zizzithefox Thanks for your feedback.
Turborepo should work on FreeBSD, what issues do you have?
The FreeBSD instructions has been contributed by the community, I think it's nice to keep it as long as not completely broken, I've added a disclaimer expliciting it's not officially supported (it will be visible soon)
-
@julien-f said in Move build to use npm and Turbo:
should work on FreeBSD, what issues do you have?
The FreeBSD instructions has been contributed by the commIt seems to me like they have explicitly excluded any platform that is neither windows nor linux (nor darwin):
root@xo:/srv/xo # yarn yarn install v1.22.18 [1/5] Validating package.json... [2/5] Resolving packages... [3/5] Fetching packages... [4/5] Linking dependencies... warning "workspace-aggregator-9be715c1-80fc-4c5d-b807-b5414e3a3afa > @xen-orchestra/fs > @aws-sdk/lib-storage@3.241.0" has unmet peer dependency "@aws-sdk/abort-controller@^3.0.0". warning Workspaces can only be enabled in private projects. [5/5] Building fresh packages... [1/20] ⢀ turbo [6/20] ⢀ @fortawesome/fontawesome-common-types [3/20] ⢀ highlight.js [7/20] ⢀ vue-demi error /srv/xo/node_modules/turbo: Command failed. Exit code: 1 Command: node install.js Arguments: Directory: /srv/xo/node_modules/turbo Output: /srv/xo/node_modules/turbo/node-platform.js:38 throw new Error(`Unsupported platform: ${platformKey}`); ^ Error: Unsupported platform: freebsd x64 LE at pkgAndSubpathForCurrentPlatform (/srv/xo/node_modules/turbo/node-platform.js:38:11) at checkAndPreparePackage (/srv/xo/node_modules/turbo/install.js:252:28) at Object.<anonymous> (/srv/xo/node_modules/turbo/install.js:302:1) at Module._compile (node:internal/modules/cjs/loader:1155:14) at Object.Module._extensions..js (node:internal/modules/cjs/loader:1209:10) at Module.load (node:internal/modules/cjs/loader:1033:32)
I have found this (seems it's the correct project):
https://github.com/vercel/turbo/pull/2408
and this:
https://github.com/vercel/turbo/issues/2339
I guess I could modify the sources, compile from sources with patches and make it to work somehow... but it's probably better to just move to linux. -
@zizzithefox Indeed, this is probably easier, thanks for the info
-