Skip to content

fs: stop stat()ing every entry in recursive readdir - #65487

Open
codebytere wants to merge 1 commit into
nodejs:mainfrom
codebytere:perf/fs-readdir-recursive-dirent-types
Open

fs: stop stat()ing every entry in recursive readdir#65487
codebytere wants to merge 1 commit into
nodejs:mainfrom
codebytere:perf/fs-readdir-recursive-dirent-types

Conversation

@codebytere

@codebytere codebytere commented Aug 22, 2026

Copy link
Copy Markdown
Member

Makes fs.readdir(), fs.readdirSync() and fsPromises.readdir() with { recursive: true } use the entry types the binding already returns instead of a stat() per entry, and build relative names by prefix instead of path.join() + path.relative() per entry. Results, their order and the symlink-following behavior are unchanged.

benchmark/fs/bench-readdir.js / bench-readdirSync.js (gain a recursive option), recursive=true, 30 runs:

lib/ (473 entries) test/parallel/ (~4 000 entries)
readdirSync +272 % +676 %
readdirSync, withFileTypes +124 % +402 %
readdir +93 % +582 %
readdir, withFileTypes +47 % +344 %

The recursive variants asked the binding for names only and then called internalModuleStat() on every entry to find the directories to descend into; with withFileTypes they built the Dirents and still stat()ed every entry that was not already a directory (to follow symbolic links, #52663). On a 13 500-entry tree that is 13 500 extra stat() calls, and the sync variant was ~4.5x slower than a hand-written readdirSync(dir, { withFileTypes: true }) walk.

Now the binding is always asked for types, directories are descended into directly, and only symbolic links and entries of unknown type (the ones that could still lead to a directory) are stat()ed. Output was compared entry-for-entry against main on test/ and on a tree with file/directory/broken symlinks and a FIFO. The known_issues test for Buffer paths (#58892) now asserts success: the ERR_INVALID_ARG_TYPE reaches the callback instead of being thrown from the completion handler, so with a bare mustCall() it would have started passing.

Tests: test-fs-readdir-recursive*, test-fs-readdir-types*, test-fs-opendir and fs.cp tests pass.


Disclosure: the code, measurements and this description were written by Claude Code, directed and reviewed by @codebytere.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/performance

@nodejs-github-bot nodejs-github-bot added fs Issues and PRs related to the fs subsystem / file system. needs-ci PRs that need a full CI run. labels Aug 22, 2026
@codebytere
codebytere force-pushed the perf/fs-readdir-recursive-dirent-types branch from 0b1854c to 441770b Compare August 22, 2026 15:42
Comment thread lib/fs.js
Comment thread lib/fs.js
readdir({ recursive: true }) asked the binding for names only and then
called internalModuleStat() on every entry to find the directories to
descend into; with withFileTypes it built the Dirents and still stat()ed
every entry that was not already a directory. Both variants also ran
path.join() and path.relative() per entry to build the relative result.

Ask the binding for file types in all cases, descend into directories
directly, and only stat() symbolic links and entries of unknown type
(which is what could point to a directory). The relative name is the
parent's prefix plus the entry name. Results, their order and the
symlink-following behavior are unchanged for fs.readdirSync, fs.readdir
and fs.promises.readdir.

The known_issues test for Buffer paths (nodejs#58892) called back without
checking the error; the error now reaches the callback instead of being
thrown from the completion handler, so the test asserts success to keep
expressing the issue.

Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
@codebytere
codebytere force-pushed the perf/fs-readdir-recursive-dirent-types branch from 441770b to 5858205 Compare August 22, 2026 17:21
@codecov

codecov Bot commented Aug 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.13%. Comparing base (21f0f27) to head (5858205).
⚠️ Report is 38 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff            @@
##             main   #65487    +/-   ##
========================================
  Coverage   90.12%   90.13%            
========================================
  Files         752      751     -1     
  Lines      252315   252458   +143     
  Branches    47444    47509    +65     
========================================
+ Hits       227395   227545   +150     
- Misses      16217    16220     +3     
+ Partials     8703     8693    -10     
Files with missing lines Coverage Δ
lib/fs.js 98.38% <100.00%> (-0.01%) ⬇️
lib/internal/fs/promises.js 92.41% <100.00%> (-0.11%) ⬇️

... and 58 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bjohansebas bjohansebas added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 22, 2026
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 22, 2026
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fs Issues and PRs related to the fs subsystem / file system. needs-ci PRs that need a full CI run.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants