Screen Test

The Multimodal Society's open benchmark for generative video models, judged on craft.

One creative brief. Every major video model. Same protocol, same seeds, blind judging by working filmmakers and engineers on direction, continuity, physics, and prompt fidelity. Cost and generation time published alongside. No automated similarity scores standing in for taste.

Results: https://multimodalsociety.com/evals Society: https://multimodalsociety.com

How a run works

1. A brief (briefs/*.yaml) defines the creative task: prompt protocol, duration, aspect ratio, and what "good" means for this brief.
2. The runner executes the brief across every registered model, N seeds each, and writes videos + a manifest (timings, parameters) to results/<brief>/.
3. At an Eval Club mixer, judges vote on blind A/B pairs in the judging app. They never see model names.
4. The report generator turns votes into Elo rankings, per-criterion counts, and a cost table, published as JSON + Markdown.

Quick start

pip install -e .
export FAL_KEY=...             # provider key for fal-hosted models

# dry run: fakes generations so you can test the full pipeline free
python -m screen_test.runner briefs/001-first-light.yaml --dry-run

# real run
python -m screen_test.runner briefs/001-first-light.yaml

# blind judging (open http://localhost:8930 in the room)
python judging/app.py results/001-first-light

# rankings
python -m screen_test.report results/001-first-light

Run the tests with pip install -e ".[dev]" && python -m pytest.

Seeds and resolution

Every model gets the same brief, seeds, and spec — but not every endpoint honors them. Some ignore a caller-supplied seed and sample their own noise; some ignore a resolution field and pick their own. We don't pretend otherwise: each model declares honors_seed and honors_resolution in the registry, and those flags are recorded into every manifest entry. Read them when comparing runs.

Adding a model

Add an entry to MODELS in screen_test/providers.py. Fal-hosted models need an endpoint id and a parameter mapping; set honors_seed / honors_resolution honestly. Models with their own APIs implement a generate(prompt, seed, spec, out_path) -> dict function. See CONTRIBUTING.md. PRs welcome, including from vendors, with one rule: nobody touches the briefs, the judging, or the results.

Neutrality

Sponsors and vendors never influence briefs, judging, or rankings. Judges are society members; model identities are hidden until votes are locked. The harness, briefs, and results are public so every run can be reproduced and challenged.

License

Apache-2.0. See LICENSE.