Repository navigation
response_match_score (ROUGE-1) is not effectively in Japanese (with manual tokenization) #4122
Description
Describe the bug
response_match_score uses rouge-score's English-oriented tokenizer.
Japanese text without whitespace is treated as a single token, so scores are incorrect.
Because we have no built-in Japanese tokenization, we had to manually pre-tokenize the Japanese input and pass it as space-separated text.
This is a workaround but there is a bug..
To Reproduce
Minimal example (using pre-tokenized Japanese as a workaround):
from google.adk.evaluation.final_response_match_v1 import _calculate_rouge_1_scores
candidate = "これ は テスト 候補 の 応答"
reference = "これ は テスト の 正解"
score = _calculate_rouge_1_scores(candidate, reference)
print(score)Score(precision=0.0, recall=0.0, fmeasure=0.0)
Even in this case, the score depends on manual tokenization and does not reflect native Japanese text behavior.
Expected behavior
Japanese should be supported without requiring users to manually insert whitespace.
Ideally response_match_score should accept a tokenizer option or provide language-aware tokenization for ROUGE.
Score(precision=0.6666666666666666, recall=0.8, fmeasure=0.7272727272727272)
Desktop (please complete the following information):
- OS: macOS
- Python version(python -V): Python 3.13.8
- ADK version(pip show google-adk): v1.22.0
Model Information:
- Are you using LiteLLM: No
- Which model is being used(e.g. gemini-2.5-pro): N/A (offline metric)
Additional context
- The limitation comes from rouge-score default tokenization.
- We had to pass pre-tokenized Japanese to get any meaningful ROUGE-1 result, whichis not ideal.
Activity
feat(evaluation): Add CJK tokenizer support for ROUGE-1 evaluation
maru0804 commentedon Jan 13, 2026
I've submitted a PR (#4143) to fix this issue.
The fix introduces a
CJKTokenizerclass that provides character-based tokenization for CJK languages. You can enable it by specifyingRougeScoreCriterion(tokenizer='cjk')in your evaluation metric configuration.This allows ROUGE-1 scores to be calculated correctly for Chinese, Japanese, and Korean text.
surajksharma07 commentedon Jan 23, 2026
@ftnext The ROUGE-1 scorer in
final_response_match_v1.py:113uses the default English tokenizer which splits on whitespace, causing Japanese text to be treated as a single token and returning 0.0 scores.PR #4143 addresses this by adding a CJK tokenizer option.
Once merged, you can use
RougeScoreCriterion(tokenizer='cjk')in your eval config for character-based tokenization.For now, you can apply the PR patch manually if this is blocking your work.
Team is looking more into it.
Thanks!
surajksharma07 commentedon Mar 26, 2026
@ankursharmas Please have a look into it once.
i-yliu commentedon Jul 28, 2026
Hi @ftnext , thanks for raising the issue, we have merged PR #6292, it should resolve the issue with Japanese Characters in ROUGE-1
ftnext commentedon Jul 29, 2026
@i-yliu Thank you for working on this issue.
I tested the implementation merged from #6292 against the original reproduction in this issue.
Both the Japanese text without whitespace and the manually tokenized text produce the same character-level tokens and score:
This confirms that #6292 fixes the problem where Japanese text always received a score of
0.0. Japanese text without whitespace can now be evaluated using character-level ROUGE-1.However, manually supplied word boundaries are not preserved. The whitespace-tokenized example from this issue is still split into individual characters instead of the supplied words. Therefore, it does not produce the expected word-level result:
Also, there does not appear to be a public option for providing or selecting a tokenizer.
Would it be possible to reopen this issue? I think #6292 resolves the “all Japanese scores are zero” problem, but it does not fully resolve the behavior reported here for manually pre-tokenized Japanese input.