Skip to content

Require full unique indexes for lookup tables - #915

Open
workstonedai-collab wants to merge 2 commits into
simonw:mainfrom
workstonedai-collab:fix/lookup-full-unique-index
Open

workstonedai-collab wants to merge 2 commits into
simonw:mainfrom
workstonedai-collab:fix/lookup-full-unique-index

Conversation

@workstonedai-collab

@workstonedai-collab workstonedai-collab commented Oct 24, 2026 •

Copy link
Copy Markdown

Fixes #914.

Problem

Table.lookup() checked only whether an index covered the lookup columns. A non-unique index could therefore be mistaken for the required uniqueness constraint, and a partial unique index could be treated as equivalent to a full unique index. This could allow duplicate lookup rows.

Behavior

  • Before: an ordinary or partial index could prevent lookup() from creating the required full unique index.
  • After: lookup() reuses an existing full unique index on exactly the lookup columns, or creates one when needed.
  • Existing indexes are preserved.
  • If existing data violates the required uniqueness constraint, the unique-index creation fails with sqlite3.IntegrityError before any rows are changed.

This change is limited to Table.lookup() unique-index detection and creation. It makes no broad performance or indexing claims and is independent of #913; both are based on the same upstream commit.

Expression index review follow-up

Following the review by @Muguett-DBY, ignore unique indexes containing expressions when matching an exact lookup-column constraint. This also covers indexes combining ordinary columns with expressions. This is a pre-existing issue, not a regression introduced by this PR.

The follow-up was implemented and validated by Codex: both new regression cases failed before the fix and pass after it; the lookup suite passed (17 tests), and the full suite passed (1504 passed, 16 skipped). Black, flake8 and mypy passed.

Validation

Validation was run by Codex:

  • Existing lookup tests, new regression tests, and three standalone probes: 18 passed.
  • Repository lookup tests: 15 passed after the fix (4 failed before the fix).
  • Full suite: 1502 passed, 16 skipped, 33 warnings.
  • mypy: 10 source files passed.
  • Black, flake8, and diff checks passed.

Implementation and tests: MiniMax. Review and validation: Codex.


📚 Documentation preview 📚: https://sqlite-utils--915.org.readthedocs.build/en/915/

@Muguett-DBY Muguett-DBY left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ordinary-index duplicate case and partial-index case behave as described in my checks. I also tried a full expression index, which still makes lookup() fail:

db = sqlite_utils.Database(memory=True)
db.execute("CREATE TABLE species(id INTEGER PRIMARY KEY, name TEXT)")
db.execute("CREATE UNIQUE INDEX species_lower ON species(lower(name))")
db["species"].lookup({"name": "Palm"})
# AttributeError: 'NoneType' object has no attribute 'translate'

SQLite's PRAGMA index_info reports None for the expression's column name, and the new comprehension passes that to fold_identifier_case(). This also happens on the base, so it isn't a regression from this PR. A non-unique expression index and a partial unique expression index now work because they're filtered out before the comprehension.

Would it be reasonable to skip indexes containing expressions when looking for an exact column constraint, and add a test? I checked the exact base and PR lookup() methods with in-memory SQLite and the existing sqlite-utils support modules; I didn't run the full branch suite.

Skip indexes containing expressions when comparing column constraints.
Add regression tests for expression-only and mixed indexes.

Reported-by: Muguett-DBY
@Muguett-DBY

Copy link
Copy Markdown

Thanks for covering the expression indexes. I rechecked the original case and a mixed column/expression index with the updated lookup(): both work now, retain the existing index, and reuse the same row on repeated lookup. The duplicate-data case still raises IntegrityError.

I ran the exact method against in-memory SQLite using the existing sqlite-utils support modules; I haven't run the full branch suite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

lookup() treats ordinary and partial indexes as unique lookup constraints

2 participants