[experiments] Daily Experiment Report — 2026-10-08

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-08

52 experiments analysed across 49 workflows. No experiment has decision-quality evidence: 0 PROMOTE, 0 REJECT. 38 are EXTEND and 14 are INCONCLUSIVE (unsupported_multi_variant). No interaction safety holds apply, because no core decision is PROMOTE.

⚡ Quick Stats

Metric	Value
Active experiments	52
Ready for analysis (sample gate)	27
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 38 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0

🔎 Key findings

• 26 of 38 EXTEND decisions are insufficient_observations, so the metric pipeline is not recording outcomes for these experiments. 23 of those already have READY sample counts.
• 3 experiments (cicoach, dailyfact, dailysecurityredteam) are READY but blocked by guardrail_unsupported.
• 14 experiments are INCONCLUSIVE with unsupported_multi_variant, because core analysis only compares control vs one candidate. These are mostly 3+ variant output_format, model_size and tone_variant tests.
• 9 are insufficient_samples and need more runs.

Note: per-run outcome charts (success rate and duration) were not generated in this run. The experiments analyze output is used as the canonical source. Descriptive per-run outcome statistics are omitted.

📊 Full Experiments Table

Workflow	Experiment	Readiness	Samples	Metric	Core decision	Reason code
agentperformanceanalyzer	prompt_compression	🟢 READY	{"caveman":76,"verbose":63} (min 20)	-	EXTEND	insufficient_observations
agentpersonaexplorer	sub_agent_strategy	🟢 READY	{"batch":74,"per_scenario":65} (min 20)	-	EXTEND	insufficient_observations
architectureguardian	sub_agent_strategy	🟡 COLLECTING	{"single_agent":17,"sub_agents":21} (min 20)	-	EXTEND	insufficient_samples
auditworkflows	audit_decomposition	🟡 COLLECTING	{"phased_sub_agents":54,"single_agent":39} (min 264)	run_success_rate	EXTEND	insufficient_samples
awfailureinvestigator	tone_variant	🟢 READY	{"assertive":210,"clinical":199,"narrative":194} (min 20)	-	INCONCLUSIVE	unsupported_multi_variant
blogauditor	prompt_style	🟡 COLLECTING	{"concise":4,"detailed":9} (min 20)	-	EXTEND	insufficient_samples
breakingchangechecker	tone_variant	🟢 READY	{"neutral":32,"urgent":27} (min 20)	-	EXTEND	insufficient_observations
cicoach	prompt_style	🟢 READY	{"concise":43,"detailed":61} (min 20)	token_count_per_run	EXTEND	guardrail_unsupported
copilotagentanalysis	output_format	🟡 COLLECTING	{"prose":35,"ste":4,"structured":43} (min 30)	ai_credits_used	INCONCLUSIVE	unsupported_multi_variant
copilotprnlpanalysis	nlp_prompt_style	🟡 COLLECTING	{"concise":0,"structured":0} (min 30)	eval:insights_report_produced	EXTEND	insufficient_samples
dailyagentrxtraceoptimizer	sub_agent_strategy	🟢 READY	{"single_agent":47,"sub_agents":46} (min 20)	-	EXTEND	insufficient_observations
dailyarchitecturediagram	detail_level	🟡 COLLECTING	{"brief":12,"comprehensive":8} (min 20)	-	EXTEND	insufficient_samples
dailyastrostylelitemarkdownspellcheck	prompt_style	🟢 READY	{"concise":82,"detailed":74} (min 20)	-	EXTEND	insufficient_observations
dailycachestrategyanalyzer	model_size	🟡 COLLECTING	{"gpt-5.3-codex":20,"gpt-5.3-codex-spark":12} (min 20)	ai_credits_total	EXTEND	insufficient_samples
dailycavemanoptimizer	model_size	🟡 COLLECTING	{"agent":1,"claude-haiku-4.5":75,"claude-sonnet-4.6":34,"claude-sonnet-5":12,"small-agent":2} (min 20)	-	INCONCLUSIVE	unsupported_multi_variant
dailycodemetrics	output_format	🟡 COLLECTING	{"executive_summary":54,"full_detail":58,"ste":16} (min 20)	-	INCONCLUSIVE	unsupported_multi_variant
dailycommunityattribution	prompt_style	🟢 READY	{"concise":71,"verbose":72} (min 20)	-	EXTEND	insufficient_observations
dailycompilerquality	output_format	🟢 READY	{"concise":56,"detailed":66,"ste":22} (min 20)	-	INCONCLUSIVE	unsupported_multi_variant
dailydochealer	model_size	🟡 COLLECTING	{"agent":2,"claude-haiku-4.5":66,"claude-sonnet-4.6":43,"claude-sonnet-5":14,"small-agent":1} (min 20)	-	INCONCLUSIVE	unsupported_multi_variant
dailydocupdater	model_size	🟡 COLLECTING	{"agent":1,"claude-haiku-4.5":51,"claude-sonnet-4.6":47,"claude-sonnet-5":16,"small-agent":2} (min 20)	-	INCONCLUSIVE	unsupported_multi_variant
dailyfact	reasoning_depth	🟢 READY	{"multi_candidate":43,"single_pass":61} (min 30)	discussion_reaction_count	EXTEND	guardrail_unsupported
dailyfunctionnamer	model_size	🟡 COLLECTING	null (min 20)	-	EXTEND	insufficient_observations
dailyissuesreport	output_format	🟢 READY	{"collapsible":67,"inline":64,"ste":23} (min 20)	-	INCONCLUSIVE	unsupported_multi_variant
dailynews	prompt_style	🟢 READY	{"concise":65,"detailed":55} (min 20)	-	EXTEND	insufficient_observations
dailyrenderingscriptsverifier	remove_redundant_context_v1	🟡 COLLECTING	null (min 20)	-	EXTEND	insufficient_observations
dailysafeoutputoptimizer	log_fetch_strategy	🟡 COLLECTING	null (min 20)	run_duration_ms	EXTEND	insufficient_observations
dailysecurityredteam	reasoning_depth	🟢 READY	{"iterative":85,"single_pass":63} (min 30)	false_positive_rate	EXTEND	guardrail_unsupported
dailysemgrepscan	semgrep_output_format	🟡 COLLECTING	{"bullet_list":28,"prose":27,"structured_sections":22} (min 30)	alert_creation_rate	INCONCLUSIVE	unsupported_multi_variant
dailysubagentoptimizer	timeout_setting	🟡 COLLECTING	{"default":2,"relaxed":1,"tight":1} (min 20)	-	INCONCLUSIVE	unsupported_multi_variant
dataflowprdiscussiondataset	caveman_mode	🟡 COLLECTING	{"no":6,"yes":11} (min 20)	-	EXTEND	insufficient_samples
deepreport	output_format	🟢 READY	{"annotated_brief":70,"executive_brief":90,"full_briefing":67,"ste":53} (min 20)	-	INCONCLUSIVE	unsupported_multi_variant
dependabotcampaign	summary_detail	🟡 COLLECTING	{"brief":6,"detailed":6} (min 20)	-	EXTEND	insufficient_samples
dependabotgochecker	prompt_style	🟡 COLLECTING	{"concise":20,"detailed":19,"step_by_step":18} (min 20)	-	INCONCLUSIVE	unsupported_multi_variant
gpclean	tool_verbosity	🟢 READY	{"full_bash":65,"minimal_toolset":71} (min 20)	-	EXTEND	insufficient_observations
issuearborist	prompt_style	🟢 READY	{"concise":51,"detailed":44} (min 20)	-	EXTEND	insufficient_observations
plan	reasoning_depth	🟡 COLLECTING	{"baseline":4,"deep":2,"shallow":1} (min 20)	-	INCONCLUSIVE	unsupported_multi_variant
prsouschef	remove_redundant_context_v1	🟡 COLLECTING	null (min 20)	-	EXTEND	insufficient_observations
smokeantigravity	sub_agent_strategy	🟢 READY	{"single_agent":99,"sub_agents":109} (min 20)	-	EXTEND	insufficient_observations
smokecopilot	caveman	🟢 READY	{"no":193,"yes":192} (min 20)	-	EXTEND	insufficient_observations
smokecopilot	subagent_model	🟢 READY	{"large":182,"small":183} (min 20)	-	EXTEND	insufficient_observations
smokecopilotaoaiapikey	caveman	🟢 READY	{"no":99,"yes":99} (min 20)	-	EXTEND	insufficient_observations
smokecopilotaoaiapikey	subagent_model	🟢 READY	{"large":99,"small":99} (min 20)	-	EXTEND	insufficient_observations
smokecopilotaoaientra	caveman	🟢 READY	{"no":85,"yes":86} (min 20)	-	EXTEND	insufficient_observations
smokecopilotaoaientra	subagent_model	🟢 READY	{"large":85,"small":86} (min 20)	-	EXTEND	insufficient_observations
smokecopilotsubagents	sub_agent_strategy	🟡 COLLECTING	{"delegated_sequential":17,"inline_strict":17,"single_agent_control":20} (min 30)	pass_rate	INCONCLUSIVE	unsupported_multi_variant
smokegemini	sub_agent_strategy	🟢 READY	{"single_agent":134,"sub_agents":147} (min 20)	-	EXTEND	insufficient_observations
smokepi	sub_agent_decomposition	🟢 READY	{"parallel_sub_agents":42,"single_agent":34} (min 20)	-	EXTEND	insufficient_observations
smokeproject	prompt_style_test	🟡 COLLECTING	null (min 20)	-	EXTEND	insufficient_observations
smoketemporaryid	sub_agent_strategy	🟡 COLLECTING	null (min 20)	aic	EXTEND	insufficient_observations
testqualitysentinel	model_size	🟢 READY	{"claude-haiku-4.5":149,"claude-sonnet-5":140} (min 20)	-	EXTEND	insufficient_observations
typist	tone_style	🟢 READY	{"conversational":54,"formal":43} (min 20)	-	EXTEND	insufficient_observations
weeklyblogpostwriter	prefetch_strategy	🟡 COLLECTING	{"eager":7,"lazy":10} (min 20)	-	EXTEND	insufficient_samples

🔁 Self-Tuning Continuation Plan

Top 3 experiment actions

1. Wire outcome observations (insufficient_observations) for the 23 READY experiments, starting with smokecopilot*, smokegemini and smokeantigravity, which have 100+ runs each.
2. Resolve the guardrail_unsupported metrics in cicoach (run_success_rate, empty_output_rate), dailyfact and dailysecurityredteam (false_positive_rate) by mapping them to supported native or eval metrics.
3. Collapse multi-variant experiments to control vs candidate, or add core support for them. The worst cases are deepreport and awfailureinvestigator, which have 200+ runs per variant and no decision.

Top 3 eval actions

1. Add an eval for insights_report_produced for copilotprnlpanalysis, which has 0 samples.
2. Add output-quality graders for the output_format experiments (dailycompilerquality, dailyissuesreport, deepreport), using length and STE readability.
3. Add a proposal-quality grader for cicoach to back the hypothesis of no degradation.

Decision-pipeline gaps for the next PR

• Native metrics (ai_credits_used, run_duration_ms, token_count_per_run, discussion_reaction_count) are not yet emitted as observations.
• Multi-variant comparison, and pairwise interaction diagnostics (smokecopilot* caveman × subagent_model) are not yet normalized core signals.

Run: 37746907657

Generated by 🧪 daily-experiment-report · copilot · auto · 21.3 AIC · ⌖ 0.943 AIC · ⊞ 16.3K · ◷

• expires on Oct 11, 2026, 12:05 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #67152.