Skip to content

fix(sync): make the index delete check case-exact - #1699

Merged
phernandez merged 2 commits into
mainfrom
fix/1627-case-exact-delete-check
Oct 9, 2026
Merged

phernandez merged 2 commits into
mainfrom
fix/1627-case-exact-delete-check

Conversation

@phernandez

Copy link
Copy Markdown
Member

Part of #1627.

What was wrong

After a case-only rename with the watcher on, a ghost entity for the old lowercase path can appear (case/config.md next to the real case/Config.md). The ghost never went away. bm project index planned its delete, then LocalProjectIndexDeletePathVerifier.confirm_deleted_paths checked existence with (base_path / path).stat(). On a case-insensitive filesystem (APFS, NTFS) that stat succeeds for any case, so the verifier reported the path as present and the maintenance runner logged "Skipping planned index deletes for paths present in storage again" on every pass.

Fix

path_spelled_exactly_on_disk in src/basic_memory/index/local_project.py runs after a successful stat. It checks every path component against its parent directory's listing, which returns the stored spelling. Listings are cached per directory for the batch, so each parent is listed once. Names are compared in NFC because APFS is also normalization-insensitive and only case is in question here. Listing errors go through the verifier's existing OSError handling, so a failed probe still skips the delete instead of confirming it.

The old spelling now counts as absent, so the next index pass deletes the ghost. This is derived state converging through the existing index pass. No locks or extra machinery.

Not fixed here (follow-up)

The watcher race that creates the ghost in the first place (the stale lowercase path indexed as a new file after the move, and the brief -1 permalink rewrite of the file bytes) is not addressed. That needs separate work in the watcher's new-file handling. In the index-pass test below, the renamed file is created before the old row is deleted in the same pass, so it can still be assigned a -1 permalink; that is also part of the follow-up.

Tests

tests/index/test_local_project_index.py:

  • path_spelled_exactly_on_disk with Case/Config.md on disk: the exact spelling is present; Case/config.md and case/Config.md are absent. On a case-sensitive filesystem those paths simply do not exist, which also reports absent. One listing per directory.
  • NFC and NFD spellings of the same name both count as present.
  • The verifier confirms case/config.md absent and keeps case/Config.md.
  • An index pass after replacing case/config.md with case/Config.md (different content, so no move detection) leaves exactly one entity, case/Config.md.

On macOS (APFS) the two verifier/index tests fail without the fix and pass with it. On a case-sensitive filesystem they pass either way.

Verification

  • just fast-check: passes.
  • BASIC_MEMORY_TESTMON_SELECT_FLAGS="--import-mode=importlib" just fast-test tests/index tests/indexing test-int/test_project_index_delete_materialization_race.py: 793 passed, 3 skipped.

🤖 Generated with Claude Code

https://claude.ai/code/session_01APFUk2bjEwMptMQqpRhjea

The project-index delete verifier confirmed absence with stat(). On a
case-insensitive filesystem (APFS, NTFS) stat finds `case/config.md`
when only `case/Config.md` exists, so after a case-only rename the old
spelling was always "present in storage again" and its ghost entity was
never deleted.

Check each path component against its parent's directory listing, one
listing per directory per batch. The old spelling now counts as absent,
and the next index pass deletes the ghost row.

Part of #1627. The watcher race that creates the ghost is not fixed here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01APFUk2bjEwMptMQqpRhjea
Signed-off-by: phernandez <paul@basicmachines.co>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-09T02:37:34.137281Z 0971ee0 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 72a06a4a89

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/basic_memory/index/local_project.py
The verifier cached each directory listing for the batch. A file
created after its directory was listed passed stat() but was missing
from the stale listing, so it was confirmed deleted. Deleting its
entity and NoteContent loses its external_id, which a later index does
not restore. A cached listing now only proves presence; a miss is
re-listed before it counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01APFUk2bjEwMptMQqpRhjea
Signed-off-by: phernandez <paul@basicmachines.co>
@phernandez
phernandez merged commit 60e6e8a into main Oct 9, 2026
34 checks passed
@phernandez
phernandez deleted the fix/1627-case-exact-delete-check branch October 9, 2026 03:08
phernandez added a commit that referenced this pull request Oct 9, 2026
Entries for #1697, #1698, #1699, #1700, #1701, #1702, #1704 and #1705,
which landed without changelog edits so the parallel PRs would not
conflict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01APFUk2bjEwMptMQqpRhjea
Signed-off-by: phernandez <paul@basicmachines.co>
phernandez added a commit that referenced this pull request Oct 9, 2026
Entries for #1697, #1698, #1699, #1700, #1701, #1702, #1704 and #1705,
which landed without changelog edits so the parallel PRs would not
conflict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01APFUk2bjEwMptMQqpRhjea
Signed-off-by: phernandez <paul@basicmachines.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant