This repository lets you reproduce the LLM baseline from Resonant IQ's churn-prediction benchmark. It includes the synthetic account histories, held-out labels, prompt builder, and scoring code used in our internal evaluation. It does not include Resonant IQ's churn-signal engine; that is the product being evaluated. The purpose of this pack is to let anyone rerun the competing "give the full account history to an LLM" approach against another model or prompt and compare the results. Our engine scored 0.94 precision, 0.53 recall, and 0.93 ROC-AUC on 178 scorable accounts. None of the six LLM baselines we tested came close. The corpus is synthetic, the sample is modest, and this is not an exhaustive test of every possible prompting strategy.
node run-benchmark.mjs \
--api-base https://openrouter.ai/api/v1 \
--model deepseek/deepseek-chat \
--api-key-env OPENROUTER_API_KEY \
--concurrency 8
Requires Node >= 20 (no npm dependencies) and an API key for an
OpenAI-compatible endpoint, set in the env var named by --api-key-env
(default OPENROUTER_API_KEY). We used OpenRouter
internally; any OpenAI-compatible /chat/completions API works.
Cost: a full run (178 scored accounts) costs roughly $3-8 depending
on model choice and pricing; a --max-accounts 10 smoke run costs cents.
Responses are disk-cached under ./.cache/ (keyed by sha256(model + prompt)), so a re-run against the same model and data costs nothing.
| Arm | P | R | AUC |
|---|---|---|---|
Resonant IQ engine (E-css) |
0.94 | 0.53 | 0.93 |
gpt-5 |
0.21 | 0.97 | 0.75 |
claude-opus-4-8 |
0.18 | 1.00 | 0.78 |
gemini-2.5-pro |
0.18 | 1.00 | 0.64 |
deepseek-chat |
0.23 | 1.00 | 0.70 |
kimi-k2 |
0.19 | 1.00 | 0.63 |
glm-4.6 |
0.18 | 1.00 | 0.61 |
n=178 scored accounts (32 positive) for every row. Full table with run
metadata: PUBLISHED-RESULTS.md.
Think you can beat the engine's P 0.94 @ R ≥ 0.5 with a different model
or a better prompt strategy, using only this baseline arm? Generate your
predictions from data/conversations.jsonl and data/events.jsonl alone;
only load labels/labels-t90.json to score a run you've already made, not
to inform it. Post your results publicly and tag @resonantiq, open an
issue on this repository, or email us with your model, method, and
metrics; we may re-run submitted prompts before featuring results.
data/conversations.jsonl: 3,071 pre-scrubbed synthetic support conversations across 200 accounts.data/events.jsonl: 10,691 pre-filtered observable lifecycle/billing/ support events.labels/labels-t90.json: held-out T90 outcome labels (32 churned, 146 retained, 22 excluded).run-benchmark.mjs: the zero-dependency runner (prompt builder + scorer).PREREGISTRATION.md: the frozen protocol this data was generated under.PUBLISHED-RESULTS.md: full results table and run metadata.LICENSE.md: usage terms.
- The corpus is synthetic, generated by the same simulator family (ResonantForge) used to tune the engine during development.
- The sample is case-enriched, so these precision/recall percentages don't transfer to natural prevalence.
- Live model outputs drift with provider updates, so re-run results may not match the published numbers exactly.
METHODOLOGY.md: prompt construction, therecord_predictiontool-wording note, known deviations, and reproducibility caveats.LEAKAGE.md: the leakage guard, data provenance, scrubbing, and what was stripped and why.PREREGISTRATION.md: the frozen protocol.PUBLISHED-RESULTS.md: the full results table.LICENSE.md: usage terms.- We benchmarked six frontier LLMs on churn prediction: the blog post.
Resonant IQ is a churn-signal detection engine for B2B SaaS teams. Learn more at resonantiq.app.