A forkable template repository with example Auto-Judge implementations for building custom judges.
The TREC AutoJudge Participant HowTo is the canonical guide, one page per activity:
| # | Activity | In Claude Code |
|---|---|---|
| 1 | Set up your dev environment | /autojudge-setup |
| 2 | Configure your LLM endpoint | /autojudge-setup |
| 3 | Develop an AutoJudge | /autojudge-develop |
| 4 | Run workflows | /autojudge-develop |
| 5 | Prompt cache | /autojudge-develop |
| 6 | Meta-evaluation | /autojudge-develop |
| 7 | Submit to TIRA | /autojudge-submit |
The three Claude Code skills, shipped in this repo under .claude/skills/, walk you through every activity interactively — the HowTo pages cover the same ground for manual use.
In short, getting started means: clone this repo into your own repository (see the HowTo's setup page), uv venv && source .venv/bin/activate && uv pip install -e '.[all]', verify with bash run_kiddie.sh, and build your judge under judges/ — with the details in the HowTo pages above.
This repository stays a bare-bones template: to build on it you must make it your own — change the project name in pyproject.toml, replace this README with a description of your judge, and create your own judges/<yourjudge>/ entry (deleting the examples before submission).
TREC Auto-Judge offers the first rigorous, cross-task benchmark for Large-Language-Model judges.
Large-Language-Model judges have emerged as a pragmatic solution when manual relevance assessment is costly or infeasible. However, recent studies reveal wide variation in accuracy across tasks, prompts, and model sizes.
Currently, shared task organizers choose an LLM judge per track ad hoc, risking inconsistent baselines and hidden biases.
Auto-Judge provides a test bed for comparing different LLM judge ideas across several tasks and correlating results against manually created relevance judgments. AutoJudge provides a testbed to study emerging evaluation approaches, as well as vulnerabilities of LLM judges, and the efficacy of safeguards for those vulnerabilities.
This project provides a means to evaluate AutoJudge approaches and provide a system ranking / leaderboard.
It will be used by TREC AutoJudge coordinators to score submissions. We encourage prospective participants to run this locally for method development.
This code will handle obtaining data sets (akin to ir_datasets), input/output and format conversions, and evaluation measures.
A fully-documented example demonstrating all three protocols:
ExampleNuggetCreator: Creates nugget questions for topicsExampleQrelsCreator: Creates relevance judgmentsExampleLeaderboardJudge: Scores responses and produces leaderboard
No LLM calls - all logic is deterministic. Use this as a reference for building judges that use nuggets and qrels.
A simple baseline judge that scores based on:
- Response text length
- Deterministic random score (for baseline comparison)
A minimal LLM-based judge with prompt caching — the smallest realistic template for an LLM judge.
A small synthetic dataset for development and testing:
- 5 topics with simple queries
- 4 runs of varying quality
- Useful for validating workflow configurations and quick iteration
# Run your judge against kiddie
auto-judge run \
--workflow judges/naive/workflow.yml \
--rag-responses data/kiddie/runs/repgen/ \
--rag-topics data/kiddie/topics/kiddie-topics.jsonl \
--out-dir ./output-kiddie/Or run the included smoke test script which also does meta-evaluation: bash run_kiddie.sh
auto-judge-starterkit/
├── pyproject.toml # Dependencies and package config
├── README.md # This file
├── run_kiddie.sh # End-to-end smoke test on kiddie
├── run_all_datasets.py # Batch driver: one run per dataset in datasets.yml
├── judges/
│ ├── complete_example/ # Full protocol example (nuggets, qrels, leaderboard)
│ ├── naive/ # Simple baseline judge
│ ├── tinyjudge/ # Minimal LLM judge example
├── data/
│ └── kiddie/ # Synthetic test dataset
├── .claude/skills/ # /autojudge-setup and /autojudge-submit walkthroughs
└── tests/
└── test_examples.py # Smoke tests
MIT