Repository navigation
Eval fails for non-English languages #3111
Description
Describe the bug
When evaluating text in other languages (e.g., Thai, etc.), the eval logic incorrectly returns mismatches (Match score: 0)— even when the evaluated expression should be valid or identical in meaning.
To Reproduce
root_agent = Agent(
name="sample_agent",
model="gemini-2.5-flash",
description="Greeting agent",
instruction="""
You are an assistant that will reply to the user with only the word 'สวัสดี'
""",
)
Expected behavior
Eval should correctly handle input in other languages.
Screenshots
Desktop (please complete the following information):
- OS: macOS
- Python : 3.12.8
- ADK version: 1.15.1
Model Information:
- Are you using LiteLLM: No
- Which model is being used : gemini-2.5-flash
Additional context
Add any other context about the problem here.
Activity
hangfei commentedon Nov 4, 2025
Convert this to a feature request for more language support.
G26karthik commentedon Nov 12, 2025
I've submitted a PR to fix this issue: #3503
The fix enhances the
FinalResponseMatchV2Evaluatorprompt to explicitly instruct the LLM judge to handle non-English languages including Thai, Chinese, Japanese, Korean, Arabic, and other non-Latin scripts.Changes:
This also addresses #3162 (Chinese string evaluation issue) with the same fix.
Testing can be done by reproducing the original Thai "สวัสดี" test case - it should now return
score=1.0instead ofscore=0.0.fix(eval): Support non-English languages in response_match_score
6 remaining items
Yashas1600 commentedon Jul 22, 2026
Put together a fix in #6441.
guptaishaan commentedon Jul 22, 2026
Opened #6443 with a fix for this.
guptaishaan commentedon Jul 22, 2026
Put together a fix in #6444.
guptaishaan commentedon Jul 22, 2026
Put together a fix in #6446.
guptaishaan commentedon Jul 23, 2026
PR up at #6456.
fix(evaluation): Support non-English responses in ROUGE-1 matching
guptaishaan commentedon Jul 24, 2026
Took a look, #6463 should address it.
guptaishaan commentedon Jul 25, 2026
Put together a fix in #6476.
guptaishaan commentedon Jul 26, 2026
Put together a fix in #6483.
ftnext commentedon Jul 26, 2026
@guptaishaan It looks like the maintainers(@i-yliu) have decided to consolidate the fix in #6292, and several alternative PRs have already been closed in its favor. To avoid further duplication, it would be best to focus follow-up work and review on #6292.
guptaishaan commentedon Jul 27, 2026
Took a look, #6494 should address it.
fix(evaluation): support non-English responses in ROUGE-1 matching
fix(evaluation): support non-English responses in ROUGE-1 matching