[RRFC] Add libc fields to select optionalDependencies should be installed or skipped

[Brooooooklyn]

Motivation ("The Why")

Example

For my packages published as prebuilt native addons, I config the native packages cross-platform as optionalDependencies to let npm choose the suitable one to download. And I can avoid written postinstall scripts to download them in this way.
Like canvas package:

The @napi-rs/canvas package is published for users depending on it directly:

{
  "name": "@napi-rs/canvas",
  "version": "0.1.6",
  "optionalDependencies": {
    "@napi-rs/canvas-linux-x64-gnu": "0.1.6",
    "@napi-rs/canvas-linux-x64-musl": "0.1.6",
    "@napi-rs/canvas-darwin-x64": "0.1.6",
    "@napi-rs/canvas-win32-x64": "0.1.6",
    "@napi-rs/canvas-linux-aarch64-gnu": "0.1.6",
    "@napi-rs/canvas-linux-aarch64-musl": "0.1.6"
  }
}

And the packages include prebuilt native addons:

{
  "name": "@napi-rs/canvas-linux-x64-musl",
  "version": "0.1.6",
  "os": ["linux"],
  "cpu": ["x64"],
  "main": "skia.linux-x64-musl.node",
  "files": ["skia.linux-x64-musl.node"]
}

{
  "name": "@napi-rs/canvas-linux-x64-gnu",
  "version": "0.1.6",
  "os": ["linux"],
  "cpu": ["x64"],
  "main": "skia.linux-x64-gnu.node",
  "files": ["skia.linux-x64-gnu.node"]
}

In the macOS/Windows/FreeBSD this mechanism works fine, but on the Linux systems, npm will download two native binding packages, both gnu and musl. It will significantly increase the install size. The install size of @napi-rs/canvas on linux-x64 systems is 37mb which could be the half of it.

References

• Related: Runtime information about libc version and type nodejs/node#39877

[hyj1991]

Also as process.versions.modules, npm should also add modules field to support downloading the appropriate package of the current nodejs version, such as:

{
  "name": "example",
  "version": "0.0.1",
  "optionalDependencies": {
    // for node-v8.x
    "example-linux-x64-v57-gnu": "0.0.1",
    // for node-v10.x
    "example-linux-x64-v64-gnu": "0.0.1",
    // for node-v12.x
    "example-linux-x64-v72-gnu": "0.0.1",  
    // for node-v14.x
    "example-linux-x64-v83-gnu": "0.0.1",  
    // for node-v16.x
    "example-linux-x64-v93-gnu": "0.0.1",  
}

[isaacs]

If node.js put more of that info in process.versions (as suggested in nodejs/node#39877) then it would be a relatively straightforward thing to provide a way to select based on it.

The other workaround is to use a preinstall script in each optional dep that checks the environment and exits with a non-zero status code if it's not suitable. This might be a good idea anyway, just to prevent someone using it in a place where it'll never work. If an optional dependency fails its install scripts, npm will clean it up. (Ie, remove it along with any dependencies that are no longer needed as a result of its removal.)

For selecting based on node version, you can use "engines": {"node":"<semver range>"} and npm will avoid even attempting to install optional deps with mismatched engine requirements.

(For non-optional deps, a failing install script will fail and roll back the entire install, and in the case of a mismatched engines.node range, it'll warn but continue, unless the user has specified --engines-strict.)

[Brooooooklyn]

The other workaround is to use a preinstall script in each optional dep that checks the environment and exits with a non-zero status code if it's not suitable. This might be a good idea anyway, just to prevent someone using it in a place where it'll never work.

It won't be a suitable solution if #488 landed

[Brooooooklyn]

@styfle here is glibcVersionCompiler and glibcVersionRuntime in process.report.getReport().header. Could we approach this feature via the glibcVersionRuntime filed?

[styfle]

@Brooooooklyn Yeah that sounds like it could work 👍

I could imagine a new package.json field with "glibcVersionRuntime": ">=2 <3" kind of like engines and that would only install the glibc version of the package.

However, I'm not sure how this would work for the musl version. How would you negate it? "glibcVersionRuntime": "<=0"?

I'm almost thinking we need to expose family instead like detect-libc does. That way each optional dependency can match on the correct family.

[Brooooooklyn]

In my opinion, libc runtime is part of the system, not Node.js engine. So I prefer a filed libc: ['glibc', 'musl', 'msvc'] like os and cpu.

And I don't like the idea that introduce version restrict to the libc runtime. The version of libc is just like the version of OS. They do not follow the semver, and lack documentation to list the compatible matrix.

[styfle]

I like the libc: ['glibc', 'musl', 'msvc'] idea 👍

Now the question is, does it make sense to implement this in Node.js core or should this be implemented in npm?

The advantage of Node.js core is that other package managers could easily implement the feature (yarn, pnpm, etc).

The disadvantage of Node.js core is that the user must update both node and npm in order to use this feature.

[lovell]

Hello, I help maintain the detect-libc module mentioned here, which is now a dependency of quite a few install-time tools. It has not been updated in over 4 years and I'd like to simplify/modernise it, especially if doing so would help get this useful feature added to npm itself.

There's a list of possible improvements in lovell/detect-libc#14 that might be of interest.

[arcanis]

While a little off-topic, I'm open to add support for a libc field in Yarn.

Main issue I'd have is the difficulty around detecting it (detect-libc is a good start, but it's currently sync; overall I'd prefer if this information came from Node, since it should be possible to get it at build-time), and decide whether the field should also expose individual majors (ie libc: ['glibc2']).

[Brooooooklyn]

/cc @zkochan

[Brooooooklyn]

overall I'd prefer if this information came from Node, since it should be possible to get it at build-time), and decide whether the field should also expose individual majors (ie libc: ['glibc2']).

Node.js already has the process.report.getReport() API to detect glibc versions: https://nodejs.org/api/report.html#diagnostic-report

{
  "header": {
    "reportVersion": 1,
    "nodejsVersion": "v12.0.0-pre",
    "glibcVersionRuntime": "2.17",
    "glibcVersionCompiler": "2.17",
    "wordSize": "64 bit",
    "arch": "x64",
    "platform": "linux",
    "componentVersions": {
      "node": "12.0.0-pre",
      "v8": "7.1.302.28-node.5",
      "uv": "1.24.1",
      "zlib": "1.2.11",
      "ares": "1.15.0",
      "modules": "68",
      "nghttp2": "1.34.0",
      "napi": "3",
      "llhttp": "1.0.1",
      "openssl": "1.1.0j"
    }
  }
}

But only for glibc systems. While I think it's enough for the native addons:

• Reduce install size for linux glibc/musl Brooooooklyn/canvas#401

The postinstall parts in this pull request should be implemented in package managers.

[arcanis]

Right, the implementation of the skip mechanism will be up to each package manager, I'm not too worried about that.

My point is that knowing whether the system uses the glibc or musl should ideally come from either process.versions or process.report.getReport(), not from an external package (or at least, not long term). Spawning external process just for that seems a waste, considering that Node is supposed to have been built against the libc and should thus have the information available already.