dgm-runner

A Python library for evolving any file with a verifiable eval using Darwin Gödel Machine methodology.

You bring:

• a set of files that can be edited (prompts, configs, code, anything)
• a way to evaluate the system (any function or shell command that returns per-task pass/fail)
• an "evolver" (any function that edits files — call an LLM, run a heuristic, whatever)

The library handles:

• Running eval N times per generation, treating results as a distribution
• Aggregating to LCB (median - spread/2) — rewards stable prompts over flaky high-mean ones
• Hold-out split for unbiased scoring (evolver never sees holdout failures)
• PROTECTED files via git checkout (catches "evolver edited the grader")
• Stale-result detection (crashed eval → treated as worst score → triggers rollback)
• Promote / rollback / observe decisions per generation
• Backup and restore on syntax errors or LCB drops

Why this exists

LLM-driven self-improvement papers ("the agent edits its own prompt") often have:

• Optimistic point estimates (one eval run, ignoring variance)
• No held-out set (overfits to whatever the optimizer can see)
• No defense against cheating (the agent could just edit the scorer)
• Silent failure modes (eval crashes → driver reads stale results → bad change is kept)

This library bakes those concerns into the loop so you don't have to.

Install

pip install dgm-runner          # not on PyPI yet, see "from source" below

From source:

git clone https://github.com/Crows12138/dgm-runner
cd dgm-runner
pip install -e .

Quick start

from pathlib import Path
from dgm_runner import (
    DGMRunner, ShellEvalRunner, CallableEvolver, TaskResult,
)

def my_evolver(failures: list[TaskResult], evolvable_files: list[Path]) -> bool:
    """Edit one or more files based on failures. Return True if edited."""
    # Your code here: call an LLM, apply a heuristic, etc.
    prompt = evolvable_files[0]
    new_prompt = call_llm("Improve this prompt given failures: ...", prompt.read_text())
    prompt.write_text(new_prompt)
    return True

runner = DGMRunner(
    evolvable_files=[Path("prompt.md")],
    holdout_tasks={"task_3", "task_7", "task_11"},     # ~30% of total
    protected_files={"eval/scoring.py"},                # auto-revert if touched
    eval_runner=ShellEvalRunner(
        cmd=["python", "eval/run_eval.py", "--output", "result.json"],
        result_glob="result_*.json",
        result_dir=Path("eval"),
    ),
    evolver=CallableEvolver(my_evolver),
    n_per_eval=5,            # higher N = more stable LCB, more compute
    on_event=print,          # stream progress
)
result = runner.run(generations=3)
print(f"Best holdout LCB: {result.best_hold_lcb:.1f}")
runner.save_history(Path("evolution.json"), result)

Decision logic per generation

hold_LCB at start of gen N  =  L_old
evolver edits files
hold_LCB after evolution    =  L_new
noise_floor                 =  max(old_spread, new_spread) / 2

L_new > best_seen_so_far     ──>  PROMOTE  (keep edit, update best snapshot)
L_new < L_old - noise_floor  ──>  ROLLBACK (restore best snapshot)
otherwise                    ──>  OBSERVE  (keep edit, but best stays put)

Early stop: if train_median == 100% AND hold_median == 100% AND spreads == 0.

Key abstractions

Class	Purpose
DGMRunner	The main loop.
EvalRunner (abstract)	"How to run my eval." Implement run() -> list[TaskResult].
ShellEvalRunner	Built-in implementation: shell command + JSON result file.
CallableEvalRunner	Wraps any () -> list[TaskResult] Python function.
Evolver (abstract)	"How to edit files given failures." Implement evolve(failures, files) -> bool.
CallableEvolver	Wraps any (failures, files) -> bool Python function.
TaskResult	One task's outcome: name, passed, duration, verify_output.
AggregatedMetrics	After N runs: median, spread, LCB for train / hold / all.

What's not in this library

• No LLM client — the library doesn't talk to OpenAI / Anthropic / Ollama. You write the Evolver and decide how to call your model.
• No prompt templates — your Evolver decides what to prompt the LLM with.
• No agent runtime — the library has no notion of "agent loop" or "tools". See evolving-coding-agent for a full coding-agent reference implementation that uses this library.

Examples

• examples/toy_prompt_evolution.py — minimal runnable demo, no LLM, shows the full API in ~50 lines.

Status

Alpha. API may change. The core math (LCB, holdout, protected) is solid; the surface API is being shaped by real use.

License

Apache License 2.0. See LICENSE and NOTICE.

Acknowledgments

• Methodology refined in evolving-coding-agent.
• DGM concept from Sakana AI's paper: "Darwin Gödel Machine".