Repository navigation
Tracking: Flaky compaction E2E tests skipped in 3 of 4 SDKs with no tracking issue #1227
Description
Summary
Compaction E2E tests are skipped in 3 of 4 SDKs with the comment "Compaction tests are skipped due to flakiness — re-enable once stabilized", but no tracking issue was filed.
This was discovered while running the Java SDK port's CompactionTest on Windows vs macOS — the test produced different results on each platform. Investigation revealed that the reference implementation had already skipped these tests in Node.js, Python, and Go, but left .NET (and by extension the Java port) still running them.
Current status by SDK
| SDK | Status | File | Skip mechanism |
|---|---|---|---|
| Node.js | ❌ Skipped | nodejs/test/e2e/compaction.e2e.test.ts |
describe.skip(...) |
| Python | ❌ Skipped | python/e2e/test_compaction_e2e.py |
pytest.mark.skip(reason=...) |
| Go | ❌ Skipped | go/internal/e2e/compaction_e2e_test.go |
t.Skip(...) |
| .NET | ✅ Active | dotnet/test/E2E/CompactionE2ETests.cs |
[Fact] (no skip) |
Root cause
The compaction snapshot replay is non-deterministic — after compaction, the CLI reconstructs user messages with a summary whose exact content can vary between runs. This causes the replaying proxy to fail to match post-compaction messages against the recorded snapshot.
Impact
- The Java SDK port (
github/copilot-sdk-java) inherited the .NET test and experienced intermittent CI failures onmain(e.g.,CompactionTest.testShouldTriggerCompactionWithLowThresholdAndEmitEvents). - PR Merge reference implementation SDK changes (2026-05-06) copilot-sdk-java#166 worked around this by adding
rerunFailingTestsCount=2to Maven Surefire, which masks the flakiness rather than fixing it.
Request
- File this as the tracking issue so the skip comments can reference it.
- Stabilize the compaction test snapshot or make the assertion resilient to non-deterministic compaction output.
- Re-enable the tests across all SDKs once stabilized.
- Decide whether .NET should also skip the test for consistency until this is resolved.
Activity
github-actions commentedon May 7, 2026
Bug Investigation Findings
I verified the claims in this issue by inspecting the relevant test files directly.
What I found
All four SDK compaction test files were checked:
nodejs/test/e2e/compaction.e2e.test.tsdescribe.skip(...)with flakiness comment ✅python/e2e/test_compaction_e2e.pypytest.mark.skip(reason="Compaction tests are skipped due to flakiness...")✅go/internal/e2e/compaction_e2e_test.got.Skip("Compaction tests are skipped due to flakiness...")✅dotnet/test/E2E/CompactionE2ETests.cs[Fact]— no skip, tests run unconditionally ❌The .NET file has two active
[Fact]tests (Should_Trigger_Compaction_With_Low_Threshold_And_Emit_EventsandShould_Not_Emit_Compaction_Events_When_Infinite_Sessions_Disabled) with no corresponding skip.Why this is a bug
The skip comments in the other three SDKs are explicit: the compaction replay is non-deterministic because post-compaction summaries vary between runs, causing the replaying proxy to fail to match recorded snapshots. The .NET tests use the same snapshot-replay harness and are subject to the same non-determinism, but were never marked as skipped. This is an accidental omission — the intent was clearly to skip them consistently across all SDKs until the underlying issue is stabilized.
Recommended next steps (as described in the issue)
Should_Trigger_Compaction_With_Low_Threshold_And_Emit_Eventstest (and evaluateShould_Not_Emit_Compaction_Events_When_Infinite_Sessions_Disabled) referencing this issue number.Labelling as bug — the .NET test suite is inconsistent with the other three SDKs due to a missed skip, causing intermittent CI failures.
On branch edburns/resolve-fake-test-time-token-failures
On branch edburns/resolve-fake-test-time-token-failures
stephentoub commentedon May 16, 2026
Fixed by #1314
Preserve Java commit authorship from standalone repo (#1561)
Preserve Java commit authorship from standalone repo (#1561)