v20.19.0 proposal by marco-ippolito · Pull Request #57349 · nodejs/node
Instead of using an async function wrapper, just try compiling code with unknown module format as SourceTextModule when it cannot be compiled as CJS and the error message indicates that it's worth a retry. If it can be parsed as SourceTextModule then it's considered ESM. Also, move shouldRetryAsESM() to C++ completely so that we can reuse it in the CJS module loader for require(esm). Drive-by: move methods that don't belong to ContextifyContext out as static methods and move GetHostDefinedOptions to ModuleWrap. PR-URL: #52413 Backport-PR-URL: #56927 Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Jacob Smith <jacob@frende.me> Refs: #52697
This patch: 1. Adds ESM syntax detection to compileFunctionForCJSLoader() for --experimental-detect-module and allow it to emit the warning for how to load ESM when it's used to parse ESM as CJS but detection is not enabled. 2. Moves the ESM detection of --experimental-detect-module for the entrypoint from executeUserEntryPoint() into Module.prototype._compile() and handle it directly in the CJS loader so that the errors thrown during compilation *and execution* during the loading of the entrypoint does not need to be bubbled all the way up. If the entrypoint doesn't parse as CJS, and detection is enabled, the CJS loader will re-load the entrypoint as ESM on the spot asynchronously using runEntryPointWithESMLoader() and cascadedLoader.import(). This is fine for the entrypoint because unlike require(ESM) we don't the namespace of the entrypoint synchronously, and can just ignore the returned value. In this case process.mainModule is reset to undefined as they are not available for ESM entrypoints. 3. Supports --experimental-detect-module for require(esm). PR-URL: #52047 Backport-PR-URL: #56927 Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Refs: #52697
So that if there are circular dependencies in the synchronous module graph, they could be resolved using the cached jobs. In case linking fails and the error gets caught, reset the cache right after linking. If it succeeds, the caller will cache it again. Otherwise the error bubbles up to users, and since we unset the cache for the unlinkable module the next attempt would still fail. PR-URL: #52868 Backport-PR-URL: #56927 Fixes: #52864 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Vinícius Lourenço Claro Cardoso <contact@viniciusl.com.br> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com> Refs: #52697
Unifies the CJS and ESM source map cache map with SourceMapCacheMap and allows the CJS cache entries to be queried more efficiently with a source url without iteration on an IterableWeakMap. Add a test to verify that the CJS source map cache entry can be reclaimed. PR-URL: #51711 Backport-PR-URL: #56927 Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Refs: #52697
This patch: 1. Refactor the routines used to compile and run an embedder entrypoint. In JS land special handling for SEA is done directly in main/embedding.js instead of clobbering the CJS loader. Add warnings to remind users that currently the require() in SEA bundled scripts only supports loading builtins. 2. Don't use the bundled SEA code cache when compiling CJS loaded from disk, since in that case we are certainly not compiling the code bundled into the SEA. Use a is_sea_main flag in CompileFunctionForCJSLoader() (which replaces an unused argument) to pass this into the C++ land - the code cache is still read directly from C++ to avoid the overhead of ArrayBuffer creation. 3. Move SEA loading code into MaybeLoadSingleExecutableApplication() which calls LoadEnvironment() with its own StartExecutionCallback(). This avoids more hidden switches in StartExecution() and make them explicit. Also add some TODOs about how to support ESM in embedded applications. 4. Add more comments PR-URL: #53573 Backport-PR-URL: #56927 Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Refs: #52697
Tooling in the ecosystem have been using the __esModule property to
recognize transpiled ESM in consuming code. For example, a 'log'
package written in ESM:
export function log(val) { console.log(val); }
Can be transpiled as:
exports.__esModule = true;
exports.default = function log(val) { console.log(val); }
The consuming code may be written like this in ESM:
import log from 'log'
Which gets transpiled to:
const _mod = require('log');
const log = _mod.__esModule ? _mod.default : _mod;
So to allow transpiled consuming code to recognize require()'d real ESM
as ESM and pick up the default exports, we add a __esModule property by
building a source text module facade for any module that has a default
export and add .__esModule = true to the exports. We don't do this to
modules that don't have default exports to avoid the unnecessary
overhead. This maintains the enumerability of the re-exported names
and the live binding of the exports.
The source of the facade is defined as a constant per-isolate property
required_module_facade_source_string, which looks like this
export * from 'original';
export { default } from 'original';
export const __esModule = true;
And the 'original' module request is always resolved by
createRequiredModuleFacade() to wrap which is a ModuleWrap wrapping
over the original module.
PR-URL: #52166
Backport-PR-URL: #56927
Refs: #52134
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
Reviewed-By: Guy Bedford <guybedford@gmail.com>
Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>
Refs: #52697
It was intended that warnings should only be emitted for an existing package.json without a type. This fixes a confusing warning telling users to update /package.json when there are no package.json on the lookup path at all, like this: [MODULE_TYPELESS_PACKAGE_JSON] Warning: ... parsed as an ES module because module syntax was detected; to avoid the performance penalty of syntax detection, add "type": "module" to /package.json Drive-by: update the warning message to be clear about reparsing and make it clear what's actionable. PR-URL: #54045 Backport-PR-URL: #56927 Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Refs: #52697
The synchronous CJS translator can handle entrypoints now, this can be hit when --import is used, so lift the bogus assertions and added tests. PR-URL: #54592 Backport-PR-URL: #56927 Fixes: #54577 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Refs: #52697
This patch implements a "module-sync" exports condition
for packages to supply a sycnrhonous ES module to the
Node.js module loader, no matter it's being required
or imported. This is similar to the "module" condition
that bundlers have been using to support `require(esm)`
in Node.js, and allows dual-package authors to opt into
ESM-first only newer versions of Node.js that supports
require(esm) while avoiding the dual-package hazard.
```json
{
"type": "module",
"exports": {
"node": {
// On new version of Node.js, both require() and import get
// the ESM version
"module-sync": "./index.js",
// On older version of Node.js, where "module" and
// require(esm) are not supported, use the transpiled CJS version
// to avoid dual-package hazard. Library authors can decide
// to drop support for older versions of Node.js when they think
// it's time.
"default": "./dist/index.cjs"
},
// On any other environment, use the ESM version.
"default": "./index.js"
}
}
```
We end up implementing a condition with a different name
instead of reusing "module", because existing code in the
ecosystem using the "module" condition sometimes also expect
the module resolution for these ESM files to work in CJS
style, which is supported by bundlers, but the native
Node.js loader has intentionally made ESM resolution
different from CJS resolution (e.g. forbidding `import
'./noext'` or `import './directory'`), so it would be
semver-major to implement a `"module"` condition
without implementing the forbidden ESM resolution rules.
For now, this just implments a new condition as semver-minor
so it can be backported to older LTS.
Refs: https://webpack.js.org/guides/package-exports/#target-environment-independent-packages
PR-URL: #54648
Backport-PR-URL: #56927
Fixes: #52173
Refs: https://github.com/joyeecheung/test-module-condition
Refs: #52697
Reviewed-By: Jacob Smith <jacob@frende.me>
Reviewed-By: Jan Krems <jan.krems@gmail.com>
Reviewed-By: Chengzhong Wu <legendecas@gmail.com>
This lays the foundation for supporting synchronous hooks proposed in nodejs/loaders#198 for ESM. - Corrects and adds several JSDoc comments for internal functions of the ESM loader, as well as explaining how require() for import CJS work in the special resolve/load paths. This doesn't consolidate it with import in require(esm) yet due to caching differences, which is left as a TODO. - The moduleProvider passed into ModuleJob is replaced as moduleOrModulePromise, we call the translators directly in the ESM loader and verify it right after loading for clarity. - Reuse a few refactored out helpers for require(esm) in getModuleJobForRequire(). PR-URL: #54769 Backport-PR-URL: #56927 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Stephen Belanger <admin@stephenbelanger.com> Reviewed-By: James M Snell <jasnell@gmail.com> Refs: #52697
This unflags --experimental-require-module so require(esm) can be used without the flag. For now, when require() actually encounters an ESM, it will still emit an experimental warning. To opt out of the feature, --no-experimental-require-module can be used. There are some tests specifically testing ERR_REQUIRE_ESM. Some of them are repurposed to test --no-experimental-require-module. Some of them are modified to just expect loading require(esm) to work, when it's appropriate. PR-URL: #55085 Backport-PR-URL: #56927 Refs: #52697 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com> Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: LiviaMedeiros <livia@cirno.name> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Filip Skokan <panva.ip@gmail.com> Reviewed-By: Michael Dawson <midawson@redhat.com> Reviewed-By: Richard Lau <rlau@redhat.com>
This is faster and more consistent with other places using the regular expression to detect node_modules. PR-URL: #55243 Backport-PR-URL: #56927 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com> Refs: #52697
This previously compiles a script and run it in a new context to avoid global pollution, which is more complex than necessary and can be too slow for it to be reused in other cases. The new implementation just checks the frames in C++ which is safe from global pollution, faster and simpler. The previous implementation also had a bug when the call site is in a ESM, because ESM have URLs as their script names, which don't start with '/' or '\' and will be skipped. The new implementation removes the skipping to fix it for ESM. PR-URL: #55286 Backport-PR-URL: #56927 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com> Refs: #52697
Previously we assumed if `--experimental-detect-module` is true, then `--experimental-require-module` is true, which isn't the case, as the two can be enabled/disabled separately. This patch fixes the checks so `--no-experimental-require-module` is still effective when `--experimental-detect-module` is enabled. Drive-by: make the assertion messages more informative and remove obsolete TODO about allowing TLA in entrypoints handled by require(esm). PR-URL: #55250 Backport-PR-URL: #56927 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Rafael Gonzaga <rafael.nunu@hotmail.com> Refs: #52697
When emitting the experimental warning for `require(esm)`, include information about the parent module and the module being require()-d to help users locate and update them. PR-URL: #55397 Backport-PR-URL: #56927 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Stephen Belanger <admin@stephenbelanger.com> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Chengzhong Wu <legendecas@gmail.com> Refs: #52697
When a ESM module cannot be loaded by require due to the presence of TLA, its module status would be stopped at kInstantiated. In this case, when it's imported again, we should allow it to be evaluated asynchronously, as it's also a common pattern for users to retry with dynamic import when require fails. PR-URL: #55502 Backport-PR-URL: #56927 Fixes: #55500 Refs: #52697 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
Trim off irrelevant internal stack frames for require(esm) warnings so it's easier to locate where the call comes from when --trace-warnings is used. PR-URL: #55496 Backport-PR-URL: #56927 Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com> Reviewed-By: Paolo Insogna <paolo@cowtech.it> Refs: #52697