Skip to content

fix(core): warn when the watcher cannot read a new directory - #1686

Closed
sammywachtel wants to merge 1 commit into
basicmachines-co:mainfrom
sammywachtel:fix/warn-unreadable-new-directory
Closed

sammywachtel wants to merge 1 commit into
basicmachines-co:mainfrom
sammywachtel:fix/warn-unreadable-new-directory

Conversation

@sammywachtel

Copy link
Copy Markdown
Contributor

Why

This is the smaller change suggested in the review of #1664.

On Linux the watcher adds a watch on each new directory when it appears. If the directory cannot be read at that moment, adding the watch fails and notify discards the error. Files written into that directory then produce no events and are never indexed, and nothing is logged. The fix is to correct the permissions and run bm project index <name>, but nothing told anyone that was needed.

The directory's own creation event still arrives, because its parent's watch is fine. That is the one place the watch service can see the problem.

What changed

src/basic_memory/index/watch_service.py: new warn_unreadable_new_directories(), called at the start of handle_changes. For each added path in the batch that is a directory (not a symlink), it tries to list it. If that fails, it logs one warning:

New directory cannot be read, so the file watcher cannot watch it and files written into it will not be indexed: <path> (Permission denied). Once its permissions are fixed, run `bm project index <project>` (or `bm reindex`).

Nothing else changes. The check is one scandir for each new directory in a batch, and none for files.

What it detects, and what it does not

  • Detected: a new directory that is still unreadable when its batch is handled, one debounce interval (index_delay, 1 s by default) after it appeared. For example, a directory created by another user, or by root, with permissions the server's user lacks.
  • Not detected: a directory that was unreadable only for an instant. install -d -o <user> run as root creates the directory owned by root and closed, then hands it over a moment later. The watch can fail inside that moment, but by the time the batch is handled the directory is readable, so there is nothing left to see. The watch service has no other signal for it, because notify reports no error.
  • Subdirectories inside a new directory are not checked. Only directories the batch reports are checked.

Testing

tests/index/test_watch_service.py:

  • test_handle_changes_warns_when_a_new_directory_cannot_be_read (skipped as root): a mode-000 directory passed through handle_changes produces exactly one warning, naming the directory and bm project index <project>. Fails without the call in handle_changes.
  • test_warn_unreadable_new_directories_ignores_readable_directories_and_files: a readable directory, a file, a path that no longer exists, and a modified event produce no warning.

Runs:

  • tests/index on macOS, SQLite, Python 3.13: 188 passed, 1 failed (test_local_watcher_embeds_indexed_file, which fails the same way on main here).
  • tests/index/test_watch_service.py on Linux (Debian, Python 3.12, non-root user): passed.
  • A real watcher (WatchService.run) on macOS and on Linux, with a mode-000 directory created in a watched project: the warning was logged within the debounce interval on both. This was a throwaway test, not committed.
  • ruff check, ruff format --check: clean. ty check src: clean apart from the pymilvus imports, which are not installed here.

On Linux the watcher adds a watch on a new directory when it appears.
If the directory cannot be read at that moment, adding the watch fails
and notify discards the error. Files written into the directory then
produce no events and are never indexed, and nothing is logged.

The directory's own creation event still arrives, because the parent's
watch is fine. When a batch reports a new directory that cannot be
listed, log a warning that names it and says to run
`bm project index <name>` (or `bm reindex`) once its permissions are
fixed.

Signed-off-by: sammywachtel <subp@wachtel.us>
@phernandez

Copy link
Copy Markdown
Member

Thank you, @sammywachtel. This is the right-sized fix. Your commit is cherry-picked unchanged into #1692, with your authorship kept, so the full CI suite runs; fork PRs skip it. #1692 also adds a changelog entry. It lands in v0.24.0.

@phernandez phernandez closed this Oct 9, 2026
phernandez added a commit that referenced this pull request Oct 9, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01APFUk2bjEwMptMQqpRhjea
Signed-off-by: phernandez <paul@basicmachines.co>
phernandez added a commit that referenced this pull request Oct 9, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01APFUk2bjEwMptMQqpRhjea
Signed-off-by: phernandez <paul@basicmachines.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants