RubricBasedEvaluator _normalize_text too basic — fails on judge model markdown output

Description

_normalize_text() in rubric_based_evaluator.py only does text.lower().strip(). When LLM judge models return rubric verdicts with markdown formatting (bullets, smart quotes, extra whitespace, non-ASCII characters), the exact-match rubric lookup fails silently, producing incorrect scores.

Reproduction

Judge model returns: "• The response correctly identifies the tool"
Expected rubric: "the response correctly identifies the tool"
_normalize_text() produces: "• the response correctly identifies the tool"
Result: No match → score defaults to 0 or lowest rubric

Common Patterns That Fail

Leading bullets: •, *, -
Smart quotes: "...", '...'
Non-ASCII: accented characters, em-dashes
Multi-space: "the response" vs "the response"
Trailing whitespace/newlines

Suggested Fix

Enhanced normalization:

def _normalize_text(text: str) -> str:
    if not isinstance(text, str):
        return ""
    text = re.sub(r'^[\s*•\-]+', '', text)   # Strip leading bullets
    text = re.sub(r'[\s*•\-]+$', '', text)   # Strip trailing
    text = re.sub(r'\s+', ' ', text)          # Collapse whitespace
    text = text.encode('ascii', 'ignore').decode()  # Remove non-ASCII
    return text.lower().strip()

Additionally, a substring fallback when exact match fails would prevent silent scoring failures:

# If exact match fails, try substring match
for rubric_text, score in rubric_map.items():
    if normalized_response in rubric_text or rubric_text in normalized_response:
        return score

Impact

Without this fix, GEPA optimization produces unreliable rubric-based scores (rubric_based_final_response_quality_v1, rubric_based_tool_use_quality_v1), leading to suboptimal prompt evolution.

Environment

google-adk 2.2.0
Judge models: gemini-2.5-pro, gemini-3.5-flash

@tottenjordan

@tottenjordan Thanks for the clear writeup — reproduced this on 2.2.0 with a quick offline test: every pattern you listed (bullets, bold, double spaces, smart quotes) misses the exact-match lookup and the rubric just gets dropped from the average with only a warning log.

Your normalization direction works, though I'd suggest two tweaks: swap the ascii-ignore step for NFKC normalization plus a smart-quote/dash mapping so non-English rubrics don't get mangled, and add a uniqueness guard to the substring fallback so two similar rubrics can't cross-match.

Since _normalize_text is applied to both sides of the comparison, strengthening it is fairly low-risk.

Could you try that variant as a workaround and verify it fixes the scores on your GEPA runs?

If it holds up after thorough testing, it would make a solid PR — and a raw failing judge transcript from your runs would be a great regression fixture to include!

@surajksharma07

Thanks @surajksharma07 — tried both tweaks, they work well together.

1. NFKC + smart-char mapping instead of ascii-ignore:

Replaced .encode('ascii', 'ignore').decode('ascii') with unicodedata.normalize('NFKC', text) plus an explicit translation table for smart quotes/dashes:

import unicodedata, re

_SMART_CHARS = {
    0x2018: "'", 0x2019: "'",  # smart single quotes
    0x201C: '"', 0x201D: '"',  # smart double quotes
    0x2013: "-", 0x2014: "-",  # en/em dashes
    0x2026: "...",             # ellipsis
}

def _fuzzy_normalize(text):
    text = unicodedata.normalize('NFKC', text)
    text = text.translate(_SMART_CHARS)
    text = re.sub(r'^[\s*•\-"\']+', '', text)
    text = re.sub(r'[\s*•\-"\']+$', '', text)
    text = re.sub(r'\s+', ' ', text)
    return text.lower().strip()

This preserves accented characters (e.g. réponse stays réponse instead of rponse) while still normalizing all the judge garbling patterns we've seen:

Input	Normalized	Match
`- The response correctly uses tools`	`the response correctly uses tools`	✅
`* The response correctly uses tools`	`the response correctly uses tools`	✅
`“The response correctly uses tools”`	`the response correctly uses tools`	✅
`— The response correctly uses tools`	`the response correctly uses tools`	✅
`The response correctly uses tools`	`the response correctly uses tools`	✅

2. Uniqueness guard on substring fallback:

Changed the substring fallback from first-match-wins to only-if-unique:

# Before (fragile — first match wins, could cross-match similar rubrics)
for config_text, r in normalized_map.items():
    if config_text in norm_text or norm_text in config_text:
        rubric = r
        break

# After (only accept if exactly one rubric matches)
candidates = [r for ct, r in normalized_map.items()
              if ct in norm_text or norm_text in ct]
if len(candidates) == 1:
    rubric = candidates[0]

Verified: when two rubrics share a common substring (e.g., "uses tools correctly" vs "uses tools efficiently"), a judge response of just "uses tools" correctly gets rejected instead of arbitrarily matching the first one.

Test results: 107 tests pass on our production codebase with both changes applied. We'll run this through our next GEPA optimization pipeline to validate on real judge transcripts. Happy to include a failing transcript as a regression fixture once we capture one.

Follow-up: End-to-end pipeline validation complete.

Ran a full GEPA optimization pipeline (deploy → eval_before → optimize → redeploy → eval_after → analysis) with the NFKC normalization + uniqueness guard applied:

Pipeline: Vertex AI Pipeline gepa-run-8fb68a8f52-20260611-115752
Model: claude-sonnet-4-6 on 5 eval cases, gemini-2.5-pro as judge
GEPA config: 15 max metric calls, 4 rubric-based criteria (hallucinations_v1, safety_v1, rubric_based_final_response_quality_v1, rubric_based_tool_use_quality_v1)
Result: Zero "Rubric not matched" warnings across all generations. All rubric scores populated correctly.

Unit tests (29 total, all pass):

We added TestFuzzyNormalize (12 parametrized cases) and TestSubstringUniquenessGuard (2 cases) covering:

Markdown bullets (- text, * **text**, • text)
Smart quotes ("text", mid-word ')
Dashes (— text, – text)
Whitespace collapse (double spaces, leading whitespace)
NFKC normalization (ellipsis … → ..., accented chars preserved)
Non-string/empty input edge cases
Ambiguous substring rejection (2+ candidates → no match)

We didn't capture a raw failing judge transcript from this run since the normalization resolved everything cleanly — no failures to capture. We can intentionally disable the patch and re-run to generate one for the regression fixture if that would be useful for the PR.

fix: NFKC rubric normalization and substring uniqueness guard

@tottenjordan

@tottenjordan went through #6080 and it's exactly the right shape: NFKC plus the smart-char map keeps non-English rubrics intact, the uniqueness guard behaves correctly in the ambiguous-substring test, and a zero-warning GEPA run on 4 rubric criteria is solid validation.

One edge case to close before review: if the judge emits a blank Property: line, _normalize_text returns "", which is a substring of every rubric — so with a single configured rubric the fallback matches it as the lone candidate.
Guarding with if not rubric and normalized_rubric_text: (plus a small test) fixes that.

Minor: a logger.debug when the fallback rescues a match would keep judge drift visible in eval logs.

No need to re-run with the patch disabled — the parametrized cases in TestNormalizeText already pin the regression, so the synthetic transcript wouldn't add much!

Team is looking more into it.
Thanks!

fix: guard empty rubric text in substring fallback + debug logging

fix(eval): improve rubric text normalization for judge-garbled output

fix: guard empty rubric text in substring fallback + debug logging

fix(eval): improve rubric text normalization for judge-garbled output

fix: guard empty rubric text in substring fallback + debug logging

fix: strip markdown and typographic decoration from rubric text

Navigation Menu

Repository navigation

RubricBasedEvaluator _normalize_text too basic — fails on judge model markdown output #6072

Description

Reproduction

Common Patterns That Fail

Suggested Fix

Impact

Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

RubricBasedEvaluator _normalize_text too basic — fails on judge model markdown output #6072

Description

Description

Reproduction

Common Patterns That Fail

Suggested Fix

Impact

Environment

Activity

surajksharma07 commented on Jun 11, 2026

tottenjordan commented on Jun 11, 2026

tottenjordan commented on Jun 11, 2026

surajksharma07 commented on Jun 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions