Repository navigation
Parallelize LLM as Judge #3958
Description
Is your feature request related to a problem? Please describe.
Right now, LLM as judge is run in series. This is N * M LLM calls in series, where N is # samples and M is # eval cases. This part can take a long time.
Describe the solution you'd like
I would like it to be possible for these to run in parallel, either by default or via a flag that can be passed into the agent evaluator.
Additional context
I've done a monkey patch of this in my own project. Before it took close to 5 minutes for me to eval one test case with 5 samples & 2 different rubric numbers with gemini 3 pro. After the patch it was down to 1 minute.
Activity
maheshwaritanay commentedon Dec 18, 2025
Could you share the monkey patch please
llalitkumarrr commentedon Mar 24, 2026
Hello @allenli873 ,
Thank you for reporting this issue. Would you be willing to submit a PR for it? We always appreciate contributions from the community.
yugborana commentedon Mar 27, 2026
@llalitkumarrr i would like to work on this issue
llalitkumarrr commentedon Mar 30, 2026
Hello @yugborana ,
Thank you for your interest. We always welcome community contributions so please feel free to open a PR.
yugborana commentedon Mar 30, 2026
@llalitkumarrr for time being please check this pr #5040
GWeale commentedon Aug 28, 2026
fixed in v2.8.0, llm_as_judge gathers the samples under asyncio.gather bounded by judge_model_options.parallelism_limit. heads up it still defaults to 1, so you get no speedup until you set it. v2 only, not on 1.x. reopen if you want the default itself changed.