Compile in-process on OCaml domains in OCaml rewatch

[cknitt]

Stacked on #8780.

This PR moves compilation into the OCaml rewatch process. Instead of starting bsc for every parse and compile, rescript links the compiler and runs requests on a bounded pool of OCaml 5 domains. That removes process start-up and pipe handling, which make the subprocess port about 6% slower than Rust on clean builds. It also opens the door to sharing compiler state between requests later.

Changes

• The compiler's process-wide mutable state is made request-scoped. Clflags, Js_config, warnings, environment and CRC caches, predefined types, Ident stamps, printers, diagnostics and backend caches move into domain-local state. Almost all of it lives in Request_local keys: every request starts with fresh values, as a new bsc process would, and the previous values are restored afterwards. Only Ident stamps, the request context, output capture and the order-sensitive predefined types keep explicit handling. The driver moves from rescript_compiler_main.ml to rescript_compiler_driver.ml, so bsc and rewatch share it.
• Each in-process request's compiler output goes to request-owned buffers, written in text mode so that Windows output keeps bsc's CRLF line endings. Output written before an escaping exception is kept and reported with the failed module. The standalone bsc still writes directly to its stdout and stderr.
• Rewatch schedules independent parse and compile requests on worker domains. By default that's at most 8, and one fewer than the CPU count; REWATCH_COMPILER_DOMAINS overrides it. Artifact publication stays on the scheduling domain. PPXs and hooks still run as external processes.
• rescript format also runs on the embedded compiler. PPXs, js-post-build, --after-build and the lock-owner check are the only remaining subprocesses. RESCRIPT_BSC_EXE now only affects rescript-rust, and CONTRIBUTING.md says how to test a local compiler instead.
• A PPX's temporary AST files are removed when the PPX fails or the build is interrupted, not only on success.
• lib/bs/compiler-info.json still records the bsc.exe shipped next to rescript.exe. The VS Code extension reads bsc_path to find bsc.exe and the other platform binaries.

Known issues before this can leave draft

• An earlier domain build once produced a differing output that was never explained. Review since found and fixed shared or leaked state:

  • Js_analyzer's free-variable accumulator was shared between domains, so Js_shake could drop a live function definition; this reproduced once in CI;
  • Parmatch.Conv's name counter, Ctype's abbreviation memo and levels, and the parser hooks for error messages.

  An audit of top-level mutable records, visitors and tables found no other live case. Regression tests cover the Js_analyzer race and the leaked Ctype memo.

• The first Ctrl-C waits for modules that are compiling in-process to finish; a second one releases the build lock and exits at once. The focused suite tests this with one large module.

ThreadSanitizer

An OCaml 5.5.0 ThreadSanitizer build of rescript reported no data races in compiler or rewatch code on clean builds of rewatch/testrepo (4 domains), tests/tests (2, 8 and 12 domains), the gentype example and the React PPX tests, or in the rewatch unit tests. Two reports appeared in some tests/tests runs, both inside the runtime's parallel minor GC (oldify_one against pool_allocate). With the shared Js_analyzer accumulator restored, the same tests/tests build reports 91 races, all in Js_analyzer, so the tool does catch this class of bug.

Tests

tests/rewatch_ounit_tests/compiler_driver_tests.ml checks that each kind of request-scoped state is isolated between consecutive and concurrent requests. The focused suite tests with one large module that a second build waits for the build lock of an in-process build, and that a failing PPX leaves no temporary files. A unit test checks that an exception escaping the compiler fails only its own module. rewatch-ocaml/bench compares clean-build artifacts across domain counts; it does not run in CI.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 80.08037% with 347 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (rewatch-ocaml-port@9e4fb4b). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
compiler/bsc/rescript_compiler_driver.ml	73.26%	73 Missing ⚠️
compiler/ml/ctype.ml	81.99%	47 Missing ⚠️
compiler/ml/printtyp.ml	60.34%	46 Missing ⚠️
compiler/gentype/debug.ml	51.21%	20 Missing ⚠️
compiler/ml/env.ml	83.47%	19 Missing ⚠️
compiler/ext/compiler_request_output.ml	72.00%	14 Missing ⚠️
compiler/ml/typecore.ml	91.73%	10 Missing ⚠️
compiler/core/js_implementation.ml	79.06%	9 Missing ⚠️
compiler/ml/cmi_format.ml	50.00%	8 Missing ⚠️
compiler/ml/ast_mapper.ml	53.33%	7 Missing ⚠️
... and 51 more

Additional details and impacted files

@@                  Coverage Diff                  @@
##             rewatch-ocaml-port    #8781   +/-   ##
=====================================================
  Coverage                      ?   80.48%           
=====================================================
  Files                         ?      452           
  Lines                         ?    61778           
  Branches                      ?        0           
=====================================================
  Hits                          ?    49723           
  Misses                        ?    12055           
  Partials                      ?        0           

Files with missing lines	Coverage Δ
analysis/src/module_resolution.ml	18.18% <100.00%> (ø)
compiler/common/js_config.ml	81.81% <100.00%> (ø)
compiler/core/bs_conditional_initial.ml	100.00% <ø> (ø)
compiler/core/config_util.ml	93.75% <100.00%> (ø)
compiler/core/ir_diagnostics.ml	92.59% <100.00%> (ø)
compiler/core/js_analyzer.ml	82.91% <100.00%> (ø)
compiler/core/js_cmj_format.ml	94.64% <100.00%> (ø)
compiler/core/js_dump.ml	92.55% <100.00%> (ø)
compiler/core/js_dump_program.ml	98.46% <100.00%> (ø)
compiler/core/lam_beta_reduce_util.ml	96.87% <100.00%> (ø)
... and 103 more

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pkg-pr-new]

Open in StackBlitz

rescript

npm i https://pkg.pr.new/rescript@8781

@rescript/belt

npm i https://pkg.pr.new/@rescript/belt@8781

@rescript/darwin-arm64

npm i https://pkg.pr.new/@rescript/darwin-arm64@8781

@rescript/darwin-x64

npm i https://pkg.pr.new/@rescript/darwin-x64@8781

@rescript/linux-arm64

npm i https://pkg.pr.new/@rescript/linux-arm64@8781

@rescript/linux-x64

npm i https://pkg.pr.new/@rescript/linux-x64@8781

@rescript/runtime

npm i https://pkg.pr.new/@rescript/runtime@8781

@rescript/win32-x64

npm i https://pkg.pr.new/@rescript/win32-x64@8781

commit: 20777a1