Build indexes and rebuild joint rollup history in the background

[koculu]

Summary

Opening the engine no longer waits for index builds, repairs or joint rollup history reconstruction. It decides what work is needed and returns; the work then runs in the background while the engine ingests events and answers queries.

Until an index is ready, queries fall back to raw events or primary rollups. Results are the same, and the query plan includes a notice saying which index is still being built.

Changes

• Readiness: hosts can wait for the background work with WaitForIndexes / WaitForIndexesAsync. If an index can't be built, the wait throws AnalyticsStorageException. The remaining work still runs, and reopening the engine resumes the failed build.
• Visibility:
  • QueryPlan.Notices, and plan.notices in HTTP responses, report background work. Notices don't make a result incomplete.
  • GetStatistics().IndexMaintenance names the job that is running.
• Correctness while ingesting:
  • Background work covers only the events stored before the engine opened; ingestion handles the rest, so every event counts once.
  • Erasure can't be undone by a build.
  • Rebuilt history becomes visible only once it is complete.
  • Raw retention waits for a reconstruction to finish.
• Recovery: interrupted builds resume from their checkpoints after a restart or a crash, including a first build.
• Shutdown logs: each stage that shutdown waits for is logged with its duration.
• Dashboard: the benchmark waits for index work before it measures queries.

Tests

• New: tests cover building while ingesting and querying, resuming, repairs, failed builds, and shutdown logs. Each regression test was checked to fail without its fix.
• Full suite: 889 of 889 pass.

Known limitations

• Crash right after a checkpoint: with the default asynchronous write-ahead log, the most recent rebuilt contributions can be lost for good, while their raw events remain. This is accepted and documented.
• A job that keeps failing for a reason other than a failed copy holds up the jobs after it, until the engine reopens.