Build indexes and rebuild joint rollup history in the background

README.md

@@ -233,7 +233,7 @@ Use `analytics.GetStatistics()` and host logging to monitor the engine. See [dep
 | [Product analytics][product] | [Secondary indexes][indexes] |
 | [Service telemetry][telemetry] | [HTTP API reference][http-api] |
 | [Sites and tenants][tenants] | [Extensibility][extensibility] |
-| [Deployment and lifecycle][deployment] | [Storage format][storage-format] |
+| [Deployment and lifecycle][deployment] | |
 
 ## Build and contribute
 
@@ -292,7 +292,6 @@ Released under the [MIT license](https://github.com/ZoneTree/ZoneTree.Analytics/
 [accuracy]: https://analytics.zonetree.dev/docs/concepts/time-and-accuracy/
 [architecture]: https://analytics.zonetree.dev/docs/concepts/architecture/
 [configuration]: https://analytics.zonetree.dev/docs/reference/configuration/
-[storage-format]: https://analytics.zonetree.dev/docs/reference/storage-format/
 [web-reports]: https://analytics.zonetree.dev/docs/guides/web-reports/
 [rollups]: https://analytics.zonetree.dev/docs/performance/rollups/
 [product]: https://analytics.zonetree.dev/docs/guides/product-analytics/

apps/ZoneTree.Analytics.Dashboard/server/SampleBenchmark.cs

@@ -89,6 +89,8 @@ async Task<SampleBenchmarkResult> MeasureAsync(string directory, AnalyticsEvent[
       compacting = true;
       progress(new("compacting", events.Length, events.Length, statistics.EventsProcessed,
           stopwatch.Elapsed.TotalMilliseconds, statistics.EventsProcessed / Math.Max(0.000001, stopwatch.Elapsed.TotalSeconds)));
+      // Reports measure queries that use the indexes.
+      await engine.WaitForIndexesAsync(applicationStopping);
       await engine.CompactAsync(applicationStopping);
       if (engine.GetStatistics(includePartitions: false).Errors > 0)
         throw new InvalidOperationException("The benchmark dataset could not be finalized successfully.");

apps/ZoneTree.Analytics.Dashboard/ui/package.json

@@ -10,13 +10,13 @@
     "publish": "purestack publish --content ./content"
   },
   "dependencies": {
-    "@purestack/ts-components": "^1.1.12",
-    "@purestack/ts-css": "^1.1.12",
-    "@purestack/ts-html": "^1.1.12",
-    "@purestack/ts-style": "^1.1.12",
-    "@purestack/ts-svg-icons": "^1.1.12",
-    "@purestack/ts-util": "^1.1.12",
-    "purestack": "^1.1.11",
+    "@purestack/ts-components": "^1.1.14",
+    "@purestack/ts-css": "^1.1.14",
+    "@purestack/ts-html": "^1.1.14",
+    "@purestack/ts-style": "^1.1.14",
+    "@purestack/ts-svg-icons": "^1.1.14",
+    "@purestack/ts-util": "^1.1.14",
+    "purestack": "^1.1.14",
     "regor": "^1.7.5"
   },
   "devDependencies": {

docs-site/content/docs/operations/deployment/deployment.mdx

@@ -31,9 +31,15 @@ Metadata, event history, aggregates, sessions, and logs belong to the dataset. C
 
 ## Account for startup
 
-Startup checks configuration and index coverage. Clean reuse can be quick; adding indexes to a large retained dataset can require substantial backfill work.
+Startup checks configuration and index coverage. Adding indexes to a large retained dataset can require substantial backfill work, which runs in the background: reports stay available meanwhile, and use the new indexes once it completes.
 
-Use initialization logs to explain startup duration and set readiness budgets accordingly. The dashboard prepares retained sample datasets before serving their reports.
+Use initialization logs to follow the backfill. To delay readiness until background index work finishes, await:
+
+<import-codeblock src="./wait-for-indexes.cs"/>
+
+The wait includes index builds, repairs, and reconstruction of indexed rollup history. It raises `AnalyticsStorageException` if an index cannot be built in this engine instance; queries continue using the available sources, and reopening the engine resumes the build. Canceling the token stops waiting without stopping background work.
+
+The dashboard prepares retained sample datasets before serving their reports.
 
 ## Close cleanly
 

docs-site/content/docs/operations/deployment/wait-for-indexes.cs

@@ -0,0 +1 @@
+await analytics.WaitForIndexesAsync(cancellationToken);

docs-site/content/docs/operations/observability/observability.mdx

@@ -18,7 +18,7 @@ Use engine statistics, initialization logs, and query plans to understand collec
 | Live visitors decrease | Identities leaving the recent sliding window | [Time and freshness](../../concepts/time-and-accuracy/time-and-accuracy.mdx#report-freshness) |
 | A filtered report is much slower | Plan sources and required joint dimensions | [Explain a slow report](#explain-a-slow-report) |
 | Country rows are missing | Geolocation adapter or trusted country dimensions | [Website context](../../concepts/data-model/data-model.mdx#standard-website-context) |
-| Startup takes longer after configuration changes | Index necessity checks and backfill progress | [Explain startup duration](#explain-startup-duration) |
+| Reports are slower after an index change | Plan notices and backfill progress | [Explain index work](#explain-index-work) |
 | Reporting returns 403 | Host policy and AuthorizeSite for the normalized site | [Sites and tenants](../../guides/sites-and-tenants/sites-and-tenants.mdx) |
 | Reporting returns 503 | Error body, concurrency, timeout, and response-size limits | [HTTP limits](../../reference/http-api/http-api.mdx#limits-and-authorization) |
 
@@ -38,10 +38,11 @@ Statistics cover the current engine instance. Lightweight polling with `includeP
 | EventsFiltered grows | Collection policy is excluding events |
 | Errors grows | Extension, processing, storage, or maintenance failures |
 | Flushes stops advancing during activity | Aggregate-saving progress or storage health |
+| IndexMaintenance has a value | An index build, repair, or rollup history reconstruction is running |
 
 Accepted and processed counters can differ while background work is active. Bot counters distinguish excluded and separately reported automated traffic.
 
-## Explain startup duration
+## Explain index work
 
 Enable Information logging for `ZoneTree.Analytics` in an ASP.NET Core host to see initialization detail. The standalone core defaults to warning-level console output.
 
@@ -55,9 +56,15 @@ For a standalone engine, set its logger before opening it:
 
 An engine opened outside dependency injection in an ASP.NET Core host can also use the integration's `UseLogger` extension to forward engine and storage messages to a host logger.
 
-Backfill checks report whether stores are disabled, reusable, need repair of affected days, or require rebuilding. Work logs report durable checkpoints, resumed event or row counts, completion duration, and throughput. Each identifies the dataset directory.
+Backfill checks report whether stores are disabled, reusable, need repair of affected days, or require rebuilding. Work logs report saved checkpoints, resumed event or row counts, completion duration, and throughput. Each identifies the dataset directory.
 
-These logs distinguish several engines in one host and explain why a restart can take longer after changing index configuration.
+These logs distinguish several engines in one host. `Plan.Notices` identifies temporary query fallbacks during background index work; the [index guide](../../performance/secondary-indexes/secondary-indexes.mdx#plan-backfill-work) explains how to interpret them.
+
+`IndexMaintenance` is null when no job is running, including after a failed build. Use `WaitForIndexesAsync` to confirm successful completion. A failed build raises `AnalyticsStorageException` through that wait and is logged; the remaining jobs still run. Queries keep using available sources until the engine reopens and resumes the failed build.
+
+## Explain shutdown duration
+
+Disposal logs each stage it waits for, such as draining queued events and closing each store, with its duration. Use them to choose the host's shutdown timeout.
 
 ## Explain a slow report
 
@@ -69,7 +76,7 @@ These logs distinguish several engines in one host and explain why a restart can
 
 The number of rows displayed is not the number of records read. A filtered card can do far more work than the same unfiltered card.
 
-The [rollup guide](../../performance/rollups/rollups.mdx#follow-a-reports-plan) includes code for inspecting sources, index dimensions, and warnings. Capture the exact site, range, filter values, metrics, and whether the read was cold or repeated with the timing.
+The [rollup guide](../../performance/rollups/rollups.mdx#follow-a-reports-plan) includes code for inspecting sources, index dimensions, warnings, and notices. Capture the exact site, range, filter values, metrics, and whether the read was cold or repeated with the timing.
 
 If the plan reads raw events, first check whether the requested joint rollup is missing. If it already reads rollups but visits many irrelevant aggregate rows, consider an equality-leading [rollup index](../../performance/secondary-indexes/secondary-indexes.mdx). Increasing raw-scan parallelism does not address an aggregate-read bottleneck.
 

docs-site/content/docs/performance/rollups/query-plan.cs

@@ -8,3 +8,6 @@
 
 foreach (var warning in result.Plan.Warnings)
     Console.WriteLine($"Warning: {warning}");
+
+foreach (var notice in result.Plan.Notices)
+    Console.WriteLine($"Notice: {notice}");

docs-site/content/docs/performance/rollups/rollups.mdx

@@ -40,6 +40,7 @@ It can read daily summaries for aligned days, hourly or minute summaries for fin
 | UsedDimensionIndexes | An ordered copy narrowed raw reads |
 | UsedRollupIndexes | A covering equality index read aggregate rows |
 | Warnings | A condition affects coverage or interpretation |
+| Notices | Temporary source fallback during background index work |
 
 Each step names an event, a time range, and a source. Indexed steps also identify the equality dimension.
 

docs-site/content/docs/performance/secondary-indexes/query-plan.cs

@@ -8,3 +8,6 @@
 
 foreach (var warning in result.Plan.Warnings)
     Console.WriteLine($"Warning: {warning}");
+
+foreach (var notice in result.Plan.Notices)
+    Console.WriteLine($"Notice: {notice}");

docs-site/content/docs/performance/secondary-indexes/secondary-indexes.mdx

@@ -56,13 +56,19 @@ The default ten-million-row partition target counts combined copies. An event ca
 
 A new partition does not hide earlier data. Queries read every overlapping partition needed for their selected range.
 
-## Plan startup work
+## Plan backfill work
 
 Opening the engine checks whether indexes can be reused. Missing or incomplete dimension indexes rebuild from retained raw events. Index-managed joint combinations can rebuild eligible aggregate history, then populate their ordered rollup stores.
 
-Backfills complete before normal ingestion starts. Completed coverage is persisted immediately, so killing an idle process does not invalidate its indexes. After interrupted writes, startup repairs the affected days from authoritative data, including missing copies and orphaned copies from interrupted deletions.
+Backfills run in the background after the engine opens, while it ingests events and answers queries. Queries use raw events or primary rollups while an index or its rollup history is being built. `Plan.Notices` explains the temporary fallback; notices do not affect `Plan.IsComplete`. Continue to check `Plan.Warnings` for coverage or interpretation concerns.
 
-Interrupted backfills resume from durable checkpoints. Copy backfills checkpoint every 100,000 input records and at partition boundaries. Historical joint-rollup reconstruction also preserves active sessions and replays absolute replacements, so resuming does not count events twice.
+Await `WaitForIndexesAsync` before measuring query performance, or to delay readiness until the work finishes. See [deployment and lifecycle](../../operations/deployment/deployment.mdx#account-for-startup) for an example and failure behavior.
+
+Completed coverage is recorded for reuse after reopening. After interrupted writes, the engine repairs the affected days from authoritative data, including missing copies and orphaned copies from interrupted deletions.
+
+Interrupted backfills resume from saved checkpoints. Copy backfills checkpoint every 100,000 input records and at partition boundaries. Historical joint-rollup reconstruction also preserves active sessions and replays absolute contributions to avoid counting the same chunk twice.
+
+Recovery follows the configured write-ahead log (WAL) mode. The default `AsyncCompressed` mode can lose recent writes in a crash, including index copies or reconstructed aggregate contributions. Raw events can remain stored while their reconstructed aggregates miss those contributions. Use synchronous WAL for the stores involved when checkpoint recovery must retain every completed write.
 
 Information logs report reuse, repair, or rebuild decisions, checkpoint progress, resumed counts, duration, and throughput. Configuration changes, missing retained partitions, and stores configured without a write-ahead log can still require a full rebuild.
 

docs-site/content/docs/reference/configuration/configuration.mdx

@@ -13,7 +13,7 @@ Use this page to locate a setting, check its default, and understand its depende
 
 Set group members and add to collections in place. Groups are created by `AnalyticsOptions`; they are not replaced wholesale. Options are cloned when an engine opens, so edits afterward do not reconfigure it.
 
-Reporting time zone, sketch precision, and the chosen separate write-ahead-log directory are persisted dataset choices. Reopening incompatible settings fails. Index changes can trigger startup backfills; ordinary rollup and aggregate changes can introduce activation boundaries.
+Reporting time zone, sketch precision, and the chosen separate write-ahead-log directory are persisted dataset choices. Reopening incompatible settings fails. Index changes can trigger background backfills; ordinary rollup and aggregate changes can introduce activation boundaries.
 
 ## Find a setting
 

docs-site/content/docs/reference/engine/engine.mdx

@@ -61,26 +61,33 @@ Storage-reading asynchronous forms run the work on a thread-pool thread. They do
 | Compact / CompactAsync | Flush and merge open stores toward disk |
 | ApplyRetention / ApplyRetentionAsync | Run physical retention maintenance |
 | EraseVisitor / EraseVisitorAsync | Remove an explicit visitor's retained raw events, index copies, and session |
+| WaitForIndexes / WaitForIndexesAsync | Wait for index builds, repairs, and reconstruction of indexed rollup history |
 | Dispose / DisposeAsync | Stop ingestion, drain accepted work, flush, and release ownership |
 
 Erasure returns deleted raw-event count and does not subtract already aggregated rollup history. Compaction is useful after bulk imports but is separate from ordinary periodic flushing.
 
+Index work runs in the background after the engine opens. Queries use the available sources meanwhile; `Plan.Notices` explains temporary fallbacks. See [secondary indexes](../../performance/secondary-indexes/secondary-indexes.mdx#plan-backfill-work) for query and recovery behavior.
+
+`WaitForIndexesAsync` completes immediately when there is no work. It raises `AnalyticsStorageException` if an index cannot be built in this engine instance, after the remaining jobs have run. Reopening resumes the failed build. Canceling the caller's token cancels only the wait; disposing the engine cancels unfinished background work.
+
 ## Statistics
 
 `GetStatistics()` includes partition information. `GetStatistics(includePartitions:false)` gives lightweight counters without inspecting those stores.
 
 | Group | Fields |
 | --- | --- |
 | Ingestion | EventsAccepted, EventsProcessed, EventsDropped, EventsRejected, EventsFiltered |
-| Health | QueueLength, Errors, Flushes |
+| Health | QueueLength, Errors, Flushes, IndexMaintenance |
 | Web activity | SessionsStarted, BotsExcluded, BotsTracked |
 | Storage | OpenPartitions, PartitionsDropped, Partitions |
 
 Partition record counts are approximate and may be null for partitions not opened since startup. Counters start again with a new engine instance.
 
+`IndexMaintenance` names the current job: `"dimension indexes"`, `"joint rollup history"`, or `"rollup indexes"`. Null means no job is running; use `WaitForIndexesAsync` to distinguish successful completion from a failed build.
+
 ## Errors and cancellation
 
-`AnalyticsException` is the base for engine-specific errors. `AnalyticsConfigurationException` reports incompatible options or dataset configuration. `AnalyticsQueryException` reports unanswerable or bounded queries; `AnalyticsQueryTimeoutException` is its timeout subclass. `AnalyticsStorageException` reports a flush whose rollup rows could not be written; its inner exception is the storage failure, and the rows are retried by later flushes.
+`AnalyticsException` is the base for engine-specific errors. `AnalyticsConfigurationException` reports incompatible options or dataset configuration. `AnalyticsQueryException` reports unanswerable or bounded queries; `AnalyticsQueryTimeoutException` is its timeout subclass. `AnalyticsStorageException` reports data that could not be written to storage: a flush whose rollup rows are retried by later flushes, or, from `WaitForIndexesAsync`, an index build that requires reopening to resume. Its inner exception carries the underlying storage failure when available.
 
 Invalid arguments use standard argument exceptions. Caller cancellation uses the cancellation token; configured query timeout additionally protects expensive report execution.
 
@@ -132,8 +139,6 @@ than ZoneTree and is not exposed.
 
 For shared indexes, restrict iterators using the returned logical `KeyPrefix`. Each accessor exposes a logical namespace within the shared physical store.
 
-See the [storage format reference](../storage-format/storage-format.mdx) for directory layouts, exact key and payload encodings, metadata generations, and recovery behavior.
-
 ## Related pages
 
 - [First events and reports](../../getting-started/first-events/first-events.mdx)

docs-site/content/docs/reference/http-api/http-api.mdx

@@ -139,10 +139,13 @@ A general query returns metric-keyed objects. Here is an example response for a
 | totalRows | Result-row count before global pagination |
 | previous | Comparison object, when one is available |
 | plan.complete / warnings | Historical coverage and interpretation concerns |
+| plan.notices | Temporary source fallback during background index work |
 
 Numeric rates such as bounce rate and conversion rate are fractions: `0.25` means 25%. Built-in session durations are seconds. Custom numeric values retain their producer's unit, such as `duration_ms` or `revenue_usd`.
 
-Plans include source steps, warnings, completeness, and rollup suggestions. Use them when historical coverage matters.
+Plans include source steps, warnings, notices, completeness, and rollup suggestions. Use them when historical coverage matters.
+
+Notices do not affect `plan.complete`; warnings still describe coverage and interpretation concerns.
 
 HTTP plans expose `steps[].source` and optional `steps[].indexDimension`. The .NET-only convenience flags such as `UsedRollupIndexes` are not additional fields in the JSON contract.
 

docs-site/content/docs/reference/http-api/query-response.json

@@ -18,6 +18,7 @@
   "plan": {
     "complete": true,
     "warnings": [],
+    "notices": [],
     "steps": [
       {
         "event": "purchase",

docs-site/content/docs/reference/index.mdx

@@ -11,7 +11,7 @@ Look up exact API contracts and defaults. These pages complement the worked exam
 
 ## Find the contract
 
-Use the [engine API](./engine/engine.mdx) for entry points, [query model](./query-model/query-model.mdx) for calculation semantics, [configuration](./configuration/configuration.mdx) for settings, [HTTP API](./http-api/http-api.mdx) for integration, and [storage format](./storage-format/storage-format.mdx) for persisted layouts and recovery contracts.
+Use the [engine API](./engine/engine.mdx) for entry points, [query model](./query-model/query-model.mdx) for calculation semantics, [configuration](./configuration/configuration.mdx) for settings, and [HTTP API](./http-api/http-api.mdx) for integration.
 
 ## In this section
 
@@ -20,7 +20,6 @@ Use the [engine API](./engine/engine.mdx) for entry points, [query model](./quer
   <DocCard number="02" title="Query model reference" summary="Query fields, metrics, filters, time ranges, and execution-plan results." href="/docs/reference/query-model/" icon="tabler:code"/>
   <DocCard number="03" title="Configuration reference" summary="Look up engine, storage, retention, privacy, and web integration settings." href="/docs/reference/configuration/" icon="tabler:code"/>
   <DocCard number="04" title="HTTP API reference" summary="Send collection and reporting requests, interpret JSON results, and handle API limits." href="/docs/reference/http-api/" icon="tabler:code"/>
-  <DocCard number="05" title="Storage format" summary="Directory layout, binary keys and values, metadata, partition routing, and recovery contracts." href="/docs/reference/storage-format/" icon="tabler:database"/>
 </Grid>
 
 ## Continue exploring