A TypeScript metrics face for @edgeproc/avow, pinned to Python by shared vectors

[hseshadr]

The claim this PR touches

"A metric computed in the browser is the number the server would have printed."

@edgeproc/avow exported zero metric functions. Every TypeScript metric in the
portfolio was therefore hand-rolled by construction — there was nothing to adopt.
aml-filter's release gate reads a recall number produced by 29 lines of bespoke
arithmetic in its own repo, and it is the only shipped recall figure anywhere in the
portfolio. Nothing checked it against a reference.

What was ported

Face	Shipped	Mirrors
Ranking	precisionAtK, recallAtK, f1AtK, mrr, binaryJudgments	assay.ranking
Classification	confusionCounts (TP/FP/TN/FN), binaryRates (accuracy, precision, recall, F1, FPR, FNR), ratesFromCounts	assay.metrics
Refusals	AssayError, InvalidRankingRequest, EmptyRelevantSet, InvalidScoreRequest — same assay.* codes	assay.errors

Semantics are mirrored exactly, refusals included: empty ranked list, duplicate
document, fractional or negative gain, non-positive k, nothing judged relevant,
length mismatch, empty input, single-class labels.

What was deliberately NOT ported, and why

• nDCG@k and average precision / MAP. Both are trec_eval's, whose engine is a
  C++ binary with no npm binding. Both carry conventions — how the ideal ranking is built
  over documents the ranker never returned, how graded gains collapse to binary, where
  truncation applies — that belong to that implementation rather than to any textbook. A
  version written from the definition would print a number that looks like Python's and
  is not, which is worse than not shipping it.
• PR-AUC and ROC-AUC. They integrate over every threshold with scikit-learn's own
  tie-handling; they are the two metrics on binary_scores that are not a ratio of
  confusion cells.
• ranking_report, bootstrap intervals, Estimate. A seeded resampling procedure
  cannot be made bit-identical across languages.

Python remains the only implementation of all of these. Stated in the module docstrings
and in ts/README.md under "What is deliberately missing".

Kept anyway: the two-class refusal. It exists in Python because the AUCs are
undefined on one class, and this module has no AUCs — but dropping it would mean an input
that scores in the browser and refuses on the server, which is the divergence the contract
exists to prevent. It also earns its keep: with both classes present, recall, FPR and FNR
can never divide by zero.

Wrap, don't rebuild — the audit

No JavaScript package clears provenance + currency + canonicity:

Candidate	Verdict
ir_measures / trec_eval	the reference — but Python + a C++ binary, no npm binding
ml-confusion-matrix (mljs)	last published 2023-01; a matrix container, not a metric reference
node-dcg	one maintainer, 2023
@mukundakatta/ragmetric-mcp, @ongravy/agent-kit	single-author 0.1.0 packages

Wrapping any of them would buy a dependency and no correctness. So these are written
against their definitions — which is safe only because of the vectors below.
Zero new dependencies are added by this PR.

The cross-language pin

testdata/vectors/metrics.json holds 23 hand-computed cases — 7 ranking, 7 ranking
refusals, 5 classification, 4 classification refusals — replayed by both
tests/test_metric_vectors.py and ts/src/metricVectors.test.ts. Python reaches its
answers through trec_eval and scikit-learn; TypeScript counts them out against the
definitions. A divergence fails CI in both languages.

It is deliberately not generated. canonical.json holds bytes nobody could author by
hand, so a generator is the only way to write it. A generated metric vector is a
transcript of whatever the code currently returns — green through the exact bug it exists
to catch. Every number was computed from the definition (each case carries its arithmetic
in a hand field) and then checked against Python.

How the confusion cells are pinned without a Python confusion-count function. The
actual-positive count is countable straight off y_true; scikit-learn's recall and
accuracy then determine all four cells uniquely. The Python replay re-derives them and
requires the result to equal the cells the TypeScript suite asserts.

One honest asymmetry, recorded in the vector itself: a non-binary label is refused in
both languages with different classes — TypeScript raises the coded
InvalidScoreRequest, Python lets scikit-learn raise an uncoded ValueError. The
accept/reject boundary is identical; that row pins the refusal and deliberately does not
pin a shared code it does not have.

The vectors immediately earned their keep: a real divergence, found and fixed

A document id of __proto__ scored 1.0 in Python and 0 in the browser.

binaryJudgments accumulated into a plain object, and plain["__proto__"] = 1 does not
create a property — it invokes Object.prototype's __proto__ setter, which ignores a
non-object value. The document silently vanished from the judgments. Python has no such
rule and keeps the key.

No refusal fired in either language. No rounding difference. Two confident, different
answers — precisely the failure this PR exists to prevent, present in the PR that
introduces the prevention. Found by probing the new face, not by a test, which is the
honest way to say it.

Fixed with a null-prototype accumulator, and pinned three ways: the
document_id_named_proto shared vector (both suites), a unit test on binaryJudgments
itself, and the ts-ranking-keeps-a-document-called-proto mutation. Watched red run:
expected [ 'ok' ] to deeply equal [ '__proto__', 'ok' ]. Both languages now print 1.0.

Northstar: watched red runs

uv run poe mutants — 37/37 guards fired, 18 of them new, 17 breaking ts/src under
vitest. Every mutation is read back off disk before its verdict is trusted.

Two mutations exist purely to prove the cross-language pin bites — their only guard is
the shared-vector suite:

Mutation	Break	Guard
ts-cells-match-pythons-sklearn	transpose a confusion cell	vector one_error_of_each_kind: the four confusion cells match
ts-ranking-matches-pythons-trec-eval	drift the recall denominator	vector graded_gains_and_a_zero_gain_judgment: every ranking metric matches

Both go red. The vectors are load-bearing, not decoration.

Vitest's exit code is not a verdict

vitest run -t 'no-such-test' exits 0, counting every test in the file as "total"
while running none of them. Read by exit code, a guard that no longer exists reports a
green baseline. The runner therefore reads numPassedTests / numFailedTests from the
JSON reporter, and a reporter that writes no file at all is a harness error rather than a
verdict.

Found on the way: the harness was scoring a guard on stale bytecode

Running poe mutants repeatedly at an unchanged commit, ranking-k-reaches-trec-eval
was scored SURVIVED on two of three runs.

CPython invalidates a .pyc on the source's mtime and size, and three existing
mutations are one character for one character (P @ k -> P @ 1, R @ k -> P @ k,
AP) -> RR)). The size never changes, so a write inside the same mtime tick as the
cached .pyc left the next interpreter loading unmutated bytecode — the guard ran
against code that was never broken. The restore path was worse: the .pyc written during
a mutated run could shadow the restored source, so the next mutation's baseline ran
mutated code.

Both writes now drop the cached .pyc. 1/3 runs clean before, 4/4 runs at 36/36 with
zero anomalies after. This is the harness's own instance of the defect it exists to
catch: it reported a verdict it had not measured.

Also pinned len(MUTATIONS) to the literal the README states — that number had already
drifted (README said 18, harness carried 19) and nothing was counting. Red run:
assert 35 == 36.

Evidence

Claim	Proof
Python gate green	uv run poe gate — 228 passed, 100% statement + branch coverage (803 stmts, 68 branches, none missed)
TS gate green	pnpm -r --include-workspace-root gate — 112 in @edgeproc/avow (100%: 121 stmts, 52 branches, 34 funcs), 31 in @edgeproc/receipt-ui (100%)
Guards can fail	uv run poe mutants — 37/37 fired, 0 anomalies; deterministic across repeated runs
The built artifact works	imported ts/dist/index.js as a consumer: all 23 vector cases replay, 11 refusals refuse with the right codes, and the README quickstart prints the exact numbers it documents (recallAtK 0.5, mrr 0.5, falseNegativeRate 0.5)
Hand-computed, not read back	every expected value carries its arithmetic in a # hand: / // hand: comment or a hand field; all were independently checked against Python before the vectors were written
No gate weakened	mutation-gate gained a Node/pnpm toolchain; no threshold, floor or check was lowered. Zero new dependencies.

Publishing — NOT done here

This PR does not publish. ts/package.json and src/avow/_version.py both move to
0.4.0 because test_packaging pins them to each other: one v* tag fans out to PyPI
and npm, and bumping only one would push an already-published version to the other
registry.

To publish after merge: tag v0.4.0 on main and let .github/workflows/publish.yml
fan out. Verify per the house rule — npm view @edgeproc/avow version returning 0.4.0
and PyPI serving avow 0.4.0 with provenance at /integrity. A green Publish run is not
evidence; the registry serving the version is.

Note on PR #23

#23 was open and unmerged when this branched, so this is cut from main and does not
touch #23's branch. Both set version 0.4.0 — #23 bumps it for the Python agreement face,
this one for the npm metrics face; they are the same release. On rebase, keep 0.4.0 and
merge the two CHANGELOG Added blocks. #23 also adds a Python ConfusionCounts; once it
lands, tests/test_metric_vectors.py can assert the vector's cells against it directly
instead of re-deriving them from scikit-learn's recall and accuracy.

🤖 Generated with Claude Code

https://claude.ai/code/session_019mNrrQ5dDEr6ZwLntu7XtF

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.