[repository-quality] Repository Quality: Test Isolation & Hermeticity

[github-actions[bot]]

🎯 Repository Quality Improvement Report - Test Isolation & Hermeticity

Analysis Date: 2026-10-08
Focus Area: Test isolation & hermeticity (process-global state in tests)
Strategy Type: Custom
Custom Area: Yes — the test suite is ~2x source LOC (613k vs 328k), so process-global mutations (cwd, env) are the main flakiness and parallelism risk.

Executive Summary

The suite has 1,893 test files. Only 575 use t.Parallel(), and 1,318 files do not. Process-global state is still widely mutated: 677 os.Chdir occurrences across 94 files, 195 os.Setenv calls in 24 files, and only 33 t.Chdir uses. t.Setenv is already common (450 uses), so the migration path is established.

Several files combine os.Chdir with t.Parallel() (e.g. pkg/cli/commands_test.go, list_workflows_command_test.go, add_no_args_test.go, git_test.go, forecast_test.go). This is a latent race condition. os.Setenv with manual restore (e.g. pkg/cli/security_regression_test.go:365, engine_secrets_test.go:550) is fragile on early t.Fatal or panics, and it blocks parallelism. There are 222 t.Skip calls (132 without an obvious short-mode/platform reason) and 29 time.Sleep calls in tests.

Full Analysis Report

Current State Assessment

Metric	Value	Status
Test files	1,893	✅
Files with t.Parallel()	575 (30%)	⚠️
os.Chdir occurrences (94 files)	677	❌
t.Chdir uses	33	⚠️
os.Setenv in tests (24 files)	195	⚠️
t.Setenv uses	450	✅
t.Skip calls (no clear reason)	222 (132)	⚠️
time.Sleep in tests	29	⚠️

Findings

Strengths

• Large t.Setenv adoption.
• No hard-coded /tmp file writes in tests.

Areas for Improvement

• High: Chdir combined with Parallel in at least 5 files.
• Medium: os.Setenv + manual restore.
• Medium: Many t.Skip calls lack a documented reason.
• Low: time.Sleep (7 in pkg/console/spinner_test.go).

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: Split the following tasks into individual work items.

Improvement Tasks

Task 1: Remove Chdir/Parallel conflicts

Priority: High
Estimated Effort: Medium
Focus Area: Test isolation

Description: Tests that call os.Chdir must not run in parallel. Convert them to t.Chdir (Go 1.24+, which panics under t.Parallel) and drop t.Parallel() where cwd is needed, or pass explicit directories to the code under test.

Acceptance Criteria:

• No test file mixes os.Chdir and t.Parallel()
• go test -race ./pkg/cli/... passes

Code Region: pkg/cli/commands_test.go, list_workflows_command_test.go, add_no_args_test.go, git_test.go, forecast_test.go

Find test files in pkg/ that use both os.Chdir and t.Parallel(). Replace os.Chdir (and its defer-restore) with t.Chdir(dir) and remove t.Parallel() from those tests, or refactor to pass paths explicitly. Run make fmt and the affected package tests.

---

Task 2: Migrate remaining os.Chdir to t.Chdir

Priority: Medium
Estimated Effort: Large
Focus Area: Test isolation

Description: Mechanically replace the os.Chdir + defer-restore pattern in the other ~90 files.

Acceptance Criteria:

• os.Chdir in *_test.go reduced by >80%
• Tests pass

Code Region: pkg/cli/*_test.go, pkg/workflow/*_test.go

Replace the os.Getwd/os.Chdir/defer os.Chdir pattern in *_test.go with t.Chdir(dir). Do it in batches per package and run tests after each.

---

Task 3: Replace os.Setenv with t.Setenv

Priority: Medium
Estimated Effort: Small
Focus Area: Test hermeticity

Description: 24 files use os.Setenv with manual restore.

Acceptance Criteria:

• No os.Setenv in tests except inside TestMain
• Tests still pass

Code Region: pkg/cli/security_regression_test.go:365, pkg/cli/engine_secrets_test.go:550

Replace os.Setenv/os.Unsetenv with t.Setenv in test files, removing manual restore code. Use os.Unsetenv after t.Setenv("X","") when the variable must be unset.

---

Task 4: Audit t.Skip reasons and Sleep

Priority: Low
Estimated Effort: Small
Focus Area: Test reliability

Description: Give every t.Skip a clear reason (condition + message), delete stale skips, and replace time.Sleep with synchronization where feasible (start with pkg/console/spinner_test.go).

Acceptance Criteria:

• Unconditional skips are removed or linked to an issue
• Sleeps in spinner tests use channels or polling

Code Region: pkg/**/*_test.go, pkg/console/spinner_test.go

List unconditional t.Skip calls in pkg/. Remove, fix, or annotate each with an issue link. Replace time.Sleep in spinner_test.go with require.Eventually or channel sync.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2026-10-05	Error-handling contract consistency	Custom	Y	4 tasks
2026-10-06	Context propagation & cancellation	Custom	Y	5 tasks
2026-10-07	Function size & file cohesion	Custom	Y	4 tasks

---

🎯 Recommendations

Immediate Actions (This Week)

1. Fix Chdir+Parallel files — Priority: High

Short-term Actions (This Month)

1. Migrate Setenv and Chdir — Priority: Medium

Long-term Actions (This Quarter)

1. Raise parallel adoption above 50% — Priority: Low

---

📈 Success Metrics

• os.Chdir in tests: 677 → <100
• os.Setenv in tests: 195 → 0

---

Next Steps

1. Review and prioritise the tasks above
2. Assign tasks to Copilot coding agent via planner agent
3. Track progress on improvement items
4. Re-evaluate this focus area in 4 weeks

---

Generated by Repository Quality Improvement Agent
Next analysis: 2026-10-09 — Focus area selected by diversity algorithm

Generated by ⚡ Repository Quality Improvement Agent · copilot · auto · 17.2 AIC · ⌖ 0.658 AIC · ⊞ 9.8K · ◷

• expires on Oct 9, 2026, 5:08 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Repository Quality Improvement Agent.

A newer discussion is available at Discussion #67208.