Skip to content

Commit 40f87fc

Browse files
committed
Optimize gated replica term reads and checkpoint all current work
1 parent 1353482 commit 40f87fc

30 files changed

Lines changed: 5206 additions & 219 deletions

‎README.md‎

Lines changed: 16 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -12,13 +12,14 @@ The product is one database server for AI agents with linked documents, typed re
1212

1313
## Development status
1414

15-
The latest [eeef54b source qualification](docs/implementation/isolated-source-qualification-37107911987.json)
16-
passes118 analyzer and63 RF3 cases, but its full solution build fails29 Site and
17-
Recovery diagnostics. Normal/scalar/recovery and native27/270 did not execute;
18-
Pages rejected the incomplete cohort. The [reviewed039 source repair](docs/implementation/isolated-source-repairs-039.json)
19-
adds the explicit50,000-slot TimeSeries raw writer and fixes Site test keys;
20-
scoped project builds/formatting pass, while exact-source GitHub and native family
21-
host/copy/coverage/publication qualification remain open.
15+
The latest [1353482 source qualification](docs/implementation/isolated-source-qualification-37108640954.json)
16+
passes118 analyzer and63 RF3 cases, but its full solution build fails8 Recovery
17+
diagnostics. Prior21 Site errors are resolved; normal/scalar/recovery, the8 raw
18+
writer cases and native27/270 did not execute. The [reviewed039 source](docs/implementation/isolated-source-repairs-039.json)
19+
retains all50,000 TimeSeries attempt slots. The [native identity input join](docs/implementation/isolated-timeseries-identity-source-040.json)
20+
binds actual cluster incarnation and voters for1/2/3-node models; scoped development
21+
build/formatting passes. Genuine native family host/copies/coverage and complete
22+
cohort/site publication remain open.
2223

2324
This is an early implementation of the clustered kernel. The default server topology has three persistent voting nodes and requires a majority for writes and strong reads. It needs no external database, Redis or message broker.
2425

@@ -319,11 +320,14 @@ and891ops/s forRF3, at16 closed-loop clients and five10000-operation repetitions
319320
Each group runs on one Docker host; this is not multi-host scaling or a complete
320321
competitor comparison. Server CPU/allocations and phase timings are unavailable.
321322
The [first preserving repair](docs/Features/ClusterReplication/ReplicaTermMetadata.md)
322-
targets repeated full-entry decoding during term checks, retaining both public
323-
authorized quorum barriers. New regressions, implementation and measured benefit
324-
remain pending. [Current source ownership evidence](docs/implementation/image-http-owner-source-r115.json)
325-
closes the Node-helper source review; the full R113 development build still fails
326-
21 other SiteTests diagnostics, and delivered-SHA native tests remain required.
323+
now avoids repeated full-entry decoding during term checks at an identical
324+
verified store cut, retaining both public authorized quorum barriers. Its
325+
[source receipt](docs/implementation/database-term-metadata-source-r119.json)
326+
records independent review,18 authored real-store cases and full development
327+
build/formatter passes. Earlier Node-helper, SiteTests and term-test compilation
328+
failures are repaired in current source. Delivered-SHA native tests, server
329+
profiles and measured benefit remain required; this source change establishes
330+
no throughput increase or competitor ranking.
327331

328332

329333
The delivered `2f374fc34` [Linux run37093197474](https://github.com/managedcode/KeyLoad/actions/runs/37093197474) now passes the complete source gates,118 analyzer cases,1483 normal and1483 scalar unit cases,164 process-recovery cases and63 Docker RF3 cases without skips. The [original-source receipt](docs/implementation/isolated-source-qualification-37093197474.json) retains exact report/artifact hashes. Comparative image and diagnostic gates pass;27 native preflights finish16 job successes and11 failures, including two explicit unavailable Neo4j topologies among the successes. Complete270 performance and site publication remain unqualified. The separate TimeSeries pure30 cases pass in both modes, while its bounded runner has36 new declared TUnit cases prepared in source and native adapters/all30 cells remain pending.

‎benchmarks/KeyLoad.ComparisonHost/Features/BenchmarkComparisons/ComparisonExecutionIdentity.cs‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
using System.Globalization;
22
using KeyLoad.Comparisons;
3+
using KeyLoad.Comparisons.Features.BenchmarkComparisons.TimeSeries.Intensive;
34
using Microsoft.Extensions.Configuration;
45

56
namespace KeyLoad.ComparisonHost.Features.BenchmarkComparisons;
@@ -65,8 +66,13 @@ internal sealed record ComparisonExecutionIdentity(
6566
ComparisonTopology topology)
6667
=> ReadIdentity(configuration, sourceRevision, topology, isTimeSeries: false, isIsolated: true);
6768

69+
/// <summary>Requires the separate intensive TimeSeries family identity.</summary>
70+
internal static ComparisonExecutionIdentity ReadTimeSeriesIntensive(IConfiguration configuration, string sourceRevision)
71+
=> ReadIdentity(configuration, sourceRevision, topology: null, isTimeSeries: false, isIntensiveTimeSeries: true)
72+
?? throw InvalidIdentity();
73+
6874
private static ComparisonExecutionIdentity? ReadIdentity(IConfiguration configuration, string? sourceRevision,
69-
ComparisonTopology? topology, bool isTimeSeries, bool isIsolated = false)
75+
ComparisonTopology? topology, bool isTimeSeries, bool isIsolated = false, bool isIntensiveTimeSeries = false)
7076
{
7177
ArgumentNullException.ThrowIfNull(configuration);
7278
var loadGeneratorImage = configuration[LoadGeneratorImageSetting];
@@ -83,17 +89,16 @@ internal sealed record ComparisonExecutionIdentity(
8389
var workflow = Nonempty(Required(configuration, WorkflowSetting));
8490
var gitHubSha = Required(configuration, GitHubShaSetting);
8591
var configuredRevision = Required(configuration, SourceRevisionSetting);
86-
var profile = Required(configuration, EvidenceProfileSetting);
92+
var profile = Required(configuration, isIntensiveTimeSeries
93+
? TimeSeriesIntensiveSelection.EvidenceProfileSetting : EvidenceProfileSetting);
8794

8895
if (!ComparisonExecutionIdentityImageReference.IsValid(keyLoadImage)
8996
|| !ComparisonExecutionIdentityImageReference.IsValid(loadGeneratorImage)
9097
|| !IsRevision(sourceRevision)
9198
|| !string.Equals(sourceRevision, configuredRevision, StringComparison.Ordinal)
9299
|| !IsRevision(gitHubSha)
93100
|| !string.Equals(sourceRevision, gitHubSha, StringComparison.Ordinal)
94-
|| !(isIsolated ? topology is { } selected && Enum.IsDefined(selected)
95-
&& profile == IsolatedComparisonContract.Current.Profile
96-
: IsAcceptedProfile(profile, topology, isTimeSeries)))
101+
|| !IsAcceptedExecutionProfile(profile, topology, isTimeSeries, isIsolated, isIntensiveTimeSeries))
97102
{
98103
throw InvalidIdentity();
99104
}
@@ -102,6 +107,14 @@ internal sealed record ComparisonExecutionIdentity(
102107
keyLoadImage, loadGeneratorImage);
103108
}
104109

110+
private static bool IsAcceptedExecutionProfile(string profile, ComparisonTopology? topology,
111+
bool isTimeSeries, bool isIsolated, bool isIntensiveTimeSeries)
112+
=> isIntensiveTimeSeries
113+
? profile == TimeSeriesIntensiveFamilyContract.Current.EvidenceProfile
114+
: isIsolated ? topology is { } selected && Enum.IsDefined(selected)
115+
&& profile == IsolatedComparisonContract.Current.Profile
116+
: IsAcceptedProfile(profile, topology, isTimeSeries);
117+
105118
private static string Required(IConfiguration configuration, string key)
106119
=> configuration[key] ?? throw new InvalidOperationException(MissingSettingPrefix + key);
107120

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
using System.Runtime.CompilerServices;
2+
3+
[assembly: InternalsVisibleTo(KeyLoad.ComparisonHost.Features.BenchmarkComparisons.ComparisonHostTestFriend.AssemblyName)]
4+
5+
namespace KeyLoad.ComparisonHost.Features.BenchmarkComparisons;
6+
7+
internal static class ComparisonHostTestFriend
8+
{
9+
internal const string AssemblyName = "KeyLoad.UnitTests";
10+
}

‎docs/ADR/ADR-059-isolated-intensive-timeseries.md‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,108 @@
11
# ADR-059: separate native intensive TimeSeries family
22

3+
## TS009H staged executable input contract
4+
5+
Accepted TH009001..004, REQ-BC059/061/064 and TSI001/003/007/008. Root owns all
6+
existing files, shared identity method, friend/project reference, dispatches,
7+
control protocol/native copy and durable/Git/workflow joins. Two disjoint workers
8+
own temporary NEW files only; no repository, Git, build/test/native/provider or
9+
package action. Stop on missing signatures, shared overlap or design expansion.
10+
11+
H-A: IsolatedTimeSeriesBenchmarkResources.Add(IDistributedApplicationBuilder)
12+
under AppHost BenchmarkComparisons. Selection.Read then explicit Enabled, Current
13+
contract/FamilyPlan unique matching Selection; require private full-qualified
14+
root/output if explicitly supplied (missing uses fresh private root/reports).
15+
If configured CellId/ContractSha256 are supplied, reject contradictory values
16+
before allocation; absence uses exact computed identities. Require Root/native
17+
to be absent (file or directory) before runner/directory allocation; reject and
18+
preserve existing native inputs. Parent-owned reports/control may already exist.
19+
Bind selection keys replacing ':' with '__', omit Preflight Scenario, and NEW
20+
Benchmarks:TimeSeries:CellId/ContractSha256. Benchmarks:Storage equals exactly "Fresh TimeSeries cell-owned native directories; no shared database"
21+
(a description, not a path). Reuse BenchmarkRunnerContainer.Create and existing
22+
selected TimeSeries resources/context. No dispatch/host/native qualification yet.
23+
Root later installs routing which rejects any partial intensive selection before
24+
legacy/default allocation; that is a separate reviewed join.
25+
26+
H-S: NEW TimeSeriesIntensiveHostSettings(Selection, Cell, ContractSha256, Identity,
27+
JobId, Image, OutputDirectory, Storage, RunId, Native), internal immutable record.
28+
Read(IConfiguration) uses Current/FamilyPlan to resolve unique Cell, requires
29+
configured CellId/ContractSha256 exact equality, new shared
30+
ComparisonExecutionIdentity.ReadTimeSeriesIntensive(configuration, sourceRevision)
31+
(nonnullable), positive canonical KEYLOAD_COMPARISON_JOB_ID and actual native image
32+
(valid immutable; KeyLoad equals Identity.KeyLoadImage). Private RunId is fresh
33+
Guid N, separate from actual GitHub RunId and workload hash. Root adds identity
34+
entry to existing validator using the selection EvidenceProfile key and Current
35+
family profile; missing runner identity rejects. Legacy methods unchanged.
36+
37+
NEW TimeSeriesIntensiveHostNativeSettings(Image, ImmutableArray<Uri> Endpoints,
38+
ImmutableArray<string> VoterIds, Guid? Incarnation, string? AdminKey,
39+
string? ConnectionString). Read(configuration, selection, image). Exact contiguous
40+
0..N-1 arrays with no scalar/extra/duplicate/alias keys, URI authority restrictions
41+
from TH009003. Incarnation canonical D GUID, nonempty; secret fields private
42+
nonempty/no CRLF. Timescale Npgsql parser checks one Host (no comma), valid port,
43+
nonempty Username/Password without connecting; malformed connection produces the
44+
same safe code, never private parse message. Opposite-engine credentials reject.
45+
Read arrays from the merged IConfiguration view; reject nested indexed children,
46+
while pre-merge duplicate/case-equivalent keys cannot be observed or certified.
47+
Storage must equal the exact H-A description; only output/root are paths.
48+
All required/invalid settings throw InvalidOperationException with fixed
49+
TimeSeriesIntensiveHostSettingsInvalid; null IConfiguration ArgumentNullException.
50+
Both positional records override ToString to the fixed type name, preventing
51+
auto-generated private credential diagnostic echoes. Normalize expected selection,
52+
shared identity and parser input exceptions to the fixed settings code without
53+
private inner exceptions. Test safe ToString and both normalized boundaries.
54+
Use named constants/helpers,400/200/50/depth3 limits and genuine TUnit inputs.
55+
56+
Stages: tests first temporary candidates; root every diff integration; development
57+
build/scoped formatter/governance; exact-source GitHub normal/scalar/models; later
58+
original native host/control/copy/SDK/MCP/6/30 joins. Root creates host internal
59+
UnitTests friend and enables ReferenceOutputAssembly on its already existing
60+
UnitTests ProjectReference; workers edit neither. Every source stage remains
61+
pending native qualification. No feature/interface/data/production migration;
62+
rollback additive source and metadata only. ADR remains Accepted.
63+
64+
```mermaid
65+
flowchart LR
66+
Config[Exact selected family configuration] --> Plan[Canonical six and thirty cells]
67+
Plan --> Compose[One runner and selected native group]
68+
Plan --> Settings[Typed private host settings]
69+
Provenance[Shared actual source and image identity] --> Settings
70+
Compose --> Later[Pending native control copy and evidence joins]
71+
Settings --> Later
72+
```
73+
74+
## TS009B native KeyLoad identity configuration join
75+
76+
Accepted REQ-BC059/061/063, AC-TSI001/003/006 and AC-TB009-001. Existing
77+
ClusterResources.Origin(string node) becomes internal with identical body, so
78+
the comparison composition consumes the canonical owner of voter addresses.
79+
IsolatedTimeSeriesKeyLoadResources.Add binds Native Incarnation to the existing
80+
incarnation ParameterResource created by ClusterResources, and binds indexed
81+
Native VoterIds__i to ClusterResources.Origin(nodes[i].Resource.Name). Existing
82+
HTTP endpoints/private admin, image, exact fixed group, waits/storage and default
83+
RF3 remain. Incarnation is not synthesized from a resource name or run identity.
84+
85+
Root owns the existing ClusterResources visibility and shared docs/Git joins.
86+
timeseries_resources_worker owns only temporary revised copies of existing
87+
IsolatedTimeSeriesKeyLoadResources.cs and its matching resource tests under
88+
/private/tmp/keyload-ts009b-candidate. Tests first extend the original three cases
89+
with independent address values and actual node/runner incarnation reference
90+
equality; retain every assertion. Source limits/constants apply. No repository,
91+
other file, package, local test/native/Git or provider action. Root reviews every
92+
diff, integrates and builds/formats/static-checks; exact-SHA GitHub models and
93+
later genuine host/preflight/copy/ACK joins qualify behaviour. Rollback removes
94+
only new bindings and restores private visibility; no data or public migration.
95+
Stop on an unspecified signature or source ownership overlap.
96+
97+
```mermaid
98+
flowchart LR
99+
Cluster[Actual cluster composition] --> Incarnation[Existing incarnation parameter]
100+
Cluster --> Origins[Canonical fixed voter origins]
101+
Incarnation --> Runner[One selected TimeSeries runner]
102+
Origins --> Runner
103+
Runner --> Later[Later actual membership and copy qualification]
104+
```
105+
3106
## TS009J explicit settled-run raw payload contract
4107

5108
TJ009003 source refinement: full pre-output validation also rejects undefined

‎docs/ADR/ADR-061-bounded-replica-term-metadata.md‎

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# ADR-061: bounded node-owned replica term metadata
22

3-
Status: Accepted; implementation and exact-SHA GitHub qualification pending.
3+
Status: Accepted; source implementation/review and development checks complete;
4+
exact-SHA GitHub and measured performance qualification pending.
45
Owner: KeyLoad lead. Related [ClusterReplication](../Features/ClusterReplication.md),
56
[term feature contract](../Features/ClusterReplication/ReplicaTermMetadata.md),
67
[ADR-007](ADR-007-replica-consensus-bootstrap.md),
@@ -10,7 +11,7 @@ Owner: KeyLoad lead. Related [ClusterReplication](../Features/ClusterReplication
1011
## Decision and preserved boundaries
1112

1213
REQ-REP-052 / AC-DBHP-001..008 extend REQ-MP-001/002/005 and
13-
AC-MP-001/004/006/011/012. Current TermAt fully copies and strictly decodes the
14+
AC-MP-001/004/006/011/012. Before this repair TermAt fully copied and strictly decoded the
1415
last retained entry to read its scalar term. Application authentication and
1516
operation each retain their own authorized quorum round; do not merge or cache
1617
those boundaries here. The node-wide write pump, shared leader round gate,
@@ -42,6 +43,21 @@ actual fences rather than trusting an ungated property observation. Alternative
4243
providers that cannot honor this contract require a separate decision; no fake
4344
provider is accepted as proof.
4445

46+
Required provider preconditions are explicit for this DurableReplicaLog use:
47+
inside its Read callback, Position and the extracted Identity fields describe
48+
the same protected cut as the view; every effective record-changing publication
49+
advances Position before returning success; replacement that can reuse a local
50+
position advances a nonreused ReadGeneration or changes incarnation/NodeId before
51+
another read can observe it; failed or uncertain live journal/tree publication
52+
or live replacement rejects subsequent reads until recovery. Pure validation,
53+
compile or snapshot verification rejection before live publication preserves the
54+
unchanged healthy cut. These are required provider semantics, not inferred from an interface
55+
name. Root documents them in the existing shared storage contract and the log's
56+
store parameter; signatures/wire/data do not change. Actual ZoneTree is the only
57+
current inspected provider; real fence-test/native qualification is pending. An unknown/nonhonoring
58+
provider is not silently qualified or enabled by this optimization and requires
59+
a separate decision and real provider regressions before DurableReplicaLog use.
60+
4561
```mermaid
4662
flowchart LR
4763
Request[Authorized quorum protocol] --> Log[Physical log monitor]
@@ -76,6 +92,10 @@ flowchart LR
7692
4. Strong reviewer reads every joined product/test diff and exact current hash;
7793
root integrates code, fixes actionable findings and owns shared docs/config,
7894
Git and gates. No worker may run local tests or alter diagnostics/limits.
95+
Root alone owns the semantic documentation of the existing shared
96+
`src/KeyLoad.Abstractions/Storage/StorageContracts.cs` provider fields/read gate;
97+
the worker documents the accepted store preconditions in its already owned
98+
DurableReplicaLog constructor parameter without new signatures or interfaces.
7999
5. Root runs enabled full development build, formatter/governance/static checks,
80100
commits ALL eligible current scope on main and ordinary pushes. Authenticate
81101
exact-SHA native CI artifacts for complete unit/scalar, process recovery and

0 commit comments

Comments
 (0)