build,tools: make addons tests work with GN#50737

zcbenz

This PR refactors the install and build_addons scripts so they can be reused by node-ci and electron projects to build the addons tests.

In detail, this PR does following things:

Minor modifications of the GN configurations to fix loading native modules.
Add some flags to tools/install.py script to allow customize locations of config.gypi, v8, etc. With some refactoring to remove global variables.
Rewrite the tools/build_addons.py script:
1. Rewrite it from js to python, because implementing cmd args handling with vanilla Node.js would take much longer time than a simple rewrite.
2. Add a few more flags to make it easier to use for embedders.
Modify the Makefile to generate headers in out dir first, and then calls build_addons.py --headers-dir=out_dir/headers (which calls node-gyp --nodedir=out_dir/headers) to build addons. Previously the build system was just calling node-gyp --nodedir=., which relied on quirky behavior of node-gyp and did not really test whether the generated headers work.

Users that rely on make or vcbuild.bat to work on Node are not affected.

nodejs-github-bot

Review requested:

@nodejs/tsc

zcbenz

I have separated some changes out of this PR so review should be easier.

/cc @nodejs/gyp @joyeecheung @richardlau for reviewing changes on tools.

nodejs-github-bot

CI: https://ci.nodejs.org/job/node-test-pull-request/55698/

zcbenz

Ping @nodejs/gyp @joyeecheung @anonrig for review after the holiday.

joyeecheung

LGTM % a correction but I think ideally we should have someone from @nodejs/build to confirm that it works with our releases.

nodejs-github-bot

CI: https://ci.nodejs.org/job/node-test-pull-request/56066/

PR-URL: #50737 Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>

batrla

Sorry to report it, but this PR caused regression, build in --shared mode now fails:

11:02:41: stdout installing /some_dir/build/prototype/i386/some_bindir/node
11:02:41: stderr Traceback (most recent call last):
11:02:41: stderr   File "/some_dir/build/amd64/tools/install.py", line 419, in <module>
11:02:41: stderr     run(parse_options(sys.argv[1:]))
11:02:41: stderr   File "/some_dir/build/amd64/tools/install.py", line 372, in run
11:02:41: stderr     files(options, install)
11:02:41: stderr   File "/some_dir/build/amd64/tools/install.py", line 179, in files
11:02:41: stderr     action(options, [os.path.join(output_prefix, output_lib)],
11:02:41: stderr                                   ^^^^^^^^^^^^^
11:02:42: stderr UnboundLocalError: cannot access local variable 'output_prefix' where it is not associated with a value

The problem in source code is obvious, the variable 'output_prefix' is only defined in some branches.
Later in code the variable is used in a branch where it was not defined before.

zcbenz

Sorry for breaking it! I'll create a fix soon.

Fix a bug caused by nodejs#50737 that, make install fails for --shared mode.

Fix a bug caused by #50737 that, make install fails for --shared mode. PR-URL: #51910 Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>

- Use node in-tree gn configs when possible. Some in-tree configs (deps/openssl) were modified in our mirror until they are fixed in the upstream node repo [1]. - Trigger landmine to clean out directory (required due to changed targets with new build files) - Update to node-ci-2024-01-17 - Add fp16 to update_deps.py [1] nodejs/node#50737. Bug: v8:14572 Change-Id: I86cc19d23151b2a64d7173cf8f0aa6adc417c091 Reviewed-on: https://chromium-review.googlesource.com/c/v8/node-ci/+/5203209 Commit-Queue: Patrick Thier <pthier@chromium.org> Reviewed-by: Victor Gomes <victorgomes@chromium.org>

PR-URL: #50737 Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>

Fix a bug caused by #50737 that, make install fails for --shared mode. PR-URL: #51910 Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>

PR-URL: #50737 Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>

Fix a bug caused by #50737 that, make install fails for --shared mode. PR-URL: #51910 Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>

PR-URL: nodejs#50737 Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>

Fix a bug caused by nodejs#50737 that, make install fails for --shared mode. PR-URL: nodejs#51910 Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>

nodejs/node#50737

* chore: bump Node.js to v22.9.0 * build: drop base64 dep in GN build nodejs/node#52856 * build,tools: make addons tests work with GN nodejs/node#50737 * fs: add fast api for InternalModuleStat nodejs/node#51344 * src: move package_json_reader cache to c++ nodejs/node#50322 * crypto: disable PKCS#1 padding for privateDecrypt nodejs-private/node-private#525 * src: move more crypto code to ncrypto nodejs/node#54320 * crypto: ensure valid point on elliptic curve in SubtleCrypto.importKey nodejs/node#50234 * src: shift more crypto impl details to ncrypto nodejs/node#54028 * src: switch crypto APIs to use Maybe<void> nodejs/node#54775 * crypto: remove DEFAULT_ENCODING nodejs/node#47182 * deps: update libuv to 1.47.0 nodejs/node#50650 * build: fix conflict gyp configs nodejs/node#53605 * lib,src: drop --experimental-network-imports nodejs/node#53822 * esm: align sync and async load implementations nodejs/node#49152 * esm: remove unnecessary toNamespacedPath calls nodejs/node#53656 * module: detect ESM syntax by trying to recompile as SourceTextModule nodejs/node#52413 * test: adapt debugger tests to V8 11.4 nodejs/node#49639 * lib: update usage of always on Atomics API nodejs/node#49639 * test: adapt test-fs-write to V8 internal changes nodejs/node#49639 * test: adapt to new V8 trusted memory spaces nodejs/node#50115 * deps: update libuv to 1.47.0 nodejs/node#50650 * src: use non-deprecated v8::Uint8Array::kMaxLength nodejs/node#50115 * src: update default V8 platform to override functions with location nodejs/node#51362 * src: add missing TryCatch nodejs/node#51362 * lib,test: handle new Iterator global nodejs/node#51362 * src: use non-deprecated version of CreateSyntheticModule nodejs/node#50115 * src: remove calls to recently deprecated V8 APIs nodejs/node#52996 * src: use new V8 API to define stream accessor nodejs/node#53084 * src: do not use deprecated V8 API nodejs/node#53084 * src: do not use soon-to-be-deprecated V8 API nodejs/node#53174 * src: migrate to new V8 interceptors API nodejs/node#52745 * src: use supported API to get stalled TLA messages nodejs/node#51362 * module: print location of unsettled top-level await in entry points nodejs/node#51999 * test: make snapshot comparison more flexible nodejs/node#54375 * test: do not set concurrency on parallelized runs nodejs/node#52177 * src: move FromNamespacedPath to path.cc nodejs/node#53540 * test: adapt to new V8 trusted memory spaces nodejs/node#50115 * build: add option to enable clang-cl on Windows nodejs/node#52870 * chore: fixup patch indices * chore: add/remove changed files * esm: drop support for import assertions nodejs/node#54890 * build: compile with C++20 support nodejs/node#52838 * deps: update nghttp2 to 1.62.1 nodejs/node#52966 * src: parse inspector profiles with simdjson nodejs/node#51783 * build: add GN build files nodejs/node#47637 * deps,lib,src: add experimental web storage nodejs/node#52435 * build: add missing BoringSSL dep * src: rewrite task runner in c++ nodejs/node#52609 * fixup! build: add GN build files * src: stop using deprecated fields of v8::FastApiCallbackOptions nodejs/node#54077 * fix: shadow variable * build: add back incorrectly removed SetAccessor patch * fixup! fixup! build: add GN build files * crypto: fix integer comparison in crypto for BoringSSL * src,lib: reducing C++ calls of esm legacy main resolve nodejs/node#48325 * src: move more crypto_dh.cc code to ncrypto nodejs/node#54459 * chore: fixup GN files for previous commit * src: move more crypto code to ncrypto nodejs/node#54320 * Fixup Perfetto ifdef guards * fix: missing electron_natives dep * fix: node_use_node_platform = false * fix: include src/node_snapshot_stub.cc in libnode * 5507047: [import-attributes] Remove support for import assertions https://chromium-review.googlesource.com/c/v8/v8/+/5507047 * fix: restore v8-sandbox.h in filenames.json * fix: re-add original-fs generation logic * fix: ngtcp2 openssl dep * test: try removing NAPI_VERSION undef * chore(deps): bump @types/node * src: move more crypto_dh.cc code to ncrypto nodejs/node#54459 * esm: remove unnecessary toNamespacedPath calls nodejs/node#53656 * buffer: fix out of range for toString nodejs/node#54553 * lib: rewrite AsyncLocalStorage without async_hooks nodejs/node#48528 * module: print amount of load time of a cjs module nodejs/node#52213 * test: skip reproducible snapshot test on 32-bit nodejs/node#53592 * fixup! src: move more crypto_dh.cc code to ncrypto * test: adjust emittedUntil return type * chore: remove redundant wpt streams patch * fixup! chore(deps): bump @types/node * fix: gn executable name on Windows * fix: build on Windows * fix: rename conflicting win32 symbols in //third_party/sqlite On Windows otherwise we get: lld-link: error: duplicate symbol: sqlite3_win32_write_debug >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:47987 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj lld-link: error: duplicate symbol: sqlite3_win32_sleep >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48042 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj lld-link: error: duplicate symbol: sqlite3_win32_is_nt >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48113 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj lld-link: error: duplicate symbol: sqlite3_win32_utf8_to_unicode >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48470 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj lld-link: error: duplicate symbol: sqlite3_win32_unicode_to_utf8 >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48486 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj lld-link: error: duplicate symbol: sqlite3_win32_mbcs_to_utf8 >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48502 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj lld-link: error: duplicate symbol: sqlite3_win32_mbcs_to_utf8_v2 >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48518 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj lld-link: error: duplicate symbol: sqlite3_win32_utf8_to_mbcs >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48534 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj lld-link: error: duplicate symbol: sqlite3_win32_utf8_to_mbcs_v2 >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48550 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj * docs: remove unnecessary ts-expect-error after types bump * src: move package resolver to c++ nodejs/node#50322 * build: set ASAN detect_container_overflow=0 nodejs/node#55584 * chore: fixup rebase * test: disable failing ASAN test * win: almost fix race detecting ESRCH in uv_kill libuv/libuv#4341

zcbenz force-pushed the build-addons-gn branch 2 times, most recently from 507f7ab to 353afc9 Compare November 15, 2023 11:06

zcbenz mentioned this pull request Nov 15, 2023

Things blocking a full-featured GN build of upstream Node.js photoionization/node_with_gn#1

Closed

27 tasks

zcbenz force-pushed the build-addons-gn branch 2 times, most recently from 0940864 to 13f87fc Compare November 16, 2023 07:32

joyeecheung reviewed Nov 16, 2023

View reviewed changes

gengjiawen added the request-ci label Nov 17, 2023

github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Nov 17, 2023

joyeecheung approved these changes Nov 28, 2023

View reviewed changes

zcbenz force-pushed the build-addons-gn branch from 118f738 to 5348591 Compare November 29, 2023 00:09

benjamingr approved these changes Nov 29, 2023

View reviewed changes

zcbenz mentioned this pull request Dec 1, 2023

Create a branch on latest main v8/node#172

Closed

gengjiawen added the request-ci Add this label to start a Jenkins CI on a PR. label Dec 4, 2023

github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 4, 2023

zcbenz force-pushed the build-addons-gn branch 3 times, most recently from 974401b to 3be1111 Compare January 4, 2024 02:28

zcbenz deleted the build-addons-gn branch February 23, 2024 07:17

zcbenz added a commit to zcbenz/node that referenced this pull request Feb 28, 2024

tools: fix installing node with shared mode …

a0e52cb

Fix a bug caused by nodejs#50737 that, make install fails for --shared mode.

zcbenz mentioned this pull request Feb 28, 2024

tools: fix installing node with shared mode #51910

Merged

marco-ippolito mentioned this pull request Mar 1, 2024

v21.7.0 proposal #51932

Merged

richardlau mentioned this pull request Mar 25, 2024

v20.12.0 proposal #52212

Merged

vmoroz mentioned this pull request Apr 22, 2024

build,tools,node-api: fix building node-api tests for Windows Debug #52632

Merged

codebytere mentioned this pull request Jun 6, 2024

build: fix generate_gn_filenames_json.py electron/electron#42389

Merged

3 tasks

codebytere added a commit to electron/electron that referenced this pull request Oct 16, 2024

build,tools: make addons tests work with GN …

5ee874b

nodejs/node#50737

codebytere added a commit to electron/electron that referenced this pull request Oct 21, 2024

build,tools: make addons tests work with GN …

e4038c5

nodejs/node#50737

codebytere added a commit to electron/electron that referenced this pull request Oct 21, 2024

build,tools: make addons tests work with GN …

527b190

nodejs/node#50737

codebytere added a commit to electron/electron that referenced this pull request Oct 22, 2024

build,tools: make addons tests work with GN …

2cea186

nodejs/node#50737

codebytere added a commit to electron/electron that referenced this pull request Oct 28, 2024

build,tools: make addons tests work with GN …

3785fdc

nodejs/node#50737

codebytere added a commit to electron/electron that referenced this pull request Oct 29, 2024

build,tools: make addons tests work with GN …

4d357a8

nodejs/node#50737

codebytere added a commit to electron/electron that referenced this pull request Oct 31, 2024

build,tools: make addons tests work with GN …

7ae4c77

nodejs/node#50737

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build,tools: make addons tests work with GN#50737

build,tools: make addons tests work with GN#50737
zcbenz wants to merge 3 commits into
nodejs:mainfrom
zcbenz:build-addons-gn

zcbenz commented Nov 15, 2023 •

edited

Loading

Uh oh!

nodejs-github-bot commented Nov 15, 2023

Uh oh!

zcbenz commented Nov 16, 2023

Uh oh!

nodejs-github-bot commented Nov 17, 2023

Uh oh!

zcbenz commented Nov 28, 2023

Uh oh!

joyeecheung left a comment •

edited

Loading

Uh oh!

nodejs-github-bot commented Dec 4, 2023

Uh oh!

batrla commented Feb 28, 2024 •

edited

Loading

Uh oh!

zcbenz commented Feb 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

zcbenz commented Nov 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nodejs-github-bot commented Nov 15, 2023

Uh oh!

zcbenz commented Nov 16, 2023

Uh oh!

nodejs-github-bot commented Nov 17, 2023

Uh oh!

zcbenz commented Nov 28, 2023

Uh oh!

joyeecheung left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nodejs-github-bot commented Dec 4, 2023

Uh oh!

batrla commented Feb 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zcbenz commented Feb 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

zcbenz commented Nov 15, 2023 •

edited

Loading

joyeecheung left a comment •

edited

Loading

batrla commented Feb 28, 2024 •

edited

Loading