feat(billing): operational controls, re-rating, and a measured scale run - #231
Merged
saurabh6790 merged 9 commits intoJul 31, 2026
Merged
Conversation
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three workstreams off the improvement scope: operational metrics and controls,
re-rating/backfill, and a synthetic scale test. Branched off the transition
authority work, which is now in
develop.The run says what it did, and someone gets told when it goes wrong
We had three standing controls — daily reconciliation, the daily invariant audit,
and webhook failure recording. All three worked. None of them told a human
anything; they wrote to an error log somebody had to think to go and read.
platform/metrics.pyemits one parseable JSON line per counter. Each phase of themonthly run reports
drafted / skipped / failed / settledplus its duration, andthe collect tick emits the whole run status before fanning out.
Billing Runis a durable row per period (teams, drafted, pending, collected,failures, timings). It is re-derived from the tables on every refresh, never
accumulated — same reason
billing_run_statusreads from the data: a counterstarts lying the moment a worker dies, and the run you most need to read is the one
that only half happened.
platform/alerts.pyruns hourly and mails operators about the three things thesweeps detect but cannot fix: an invariant that no longer holds, a webhook
Failedfor over an hour, and an attempt still in flight a day after reconciliation should
have answered it. It sends a digest rather than one mail per row, and an unchanged
digest stays quiet for six hours — an unfixed problem must not mail sixty times, or
the real alerts get filtered. Clearing the trouble re-arms it immediately.
Four internal views on data we already had: Webhook Lag (receipt → processed,
median/p95/worst per gateway per day), Dunning Recovery (of the invoices that
went overdue, how many were eventually collected), Involuntary Churn (teams
suspended for non-payment vs recovered), and a
Monthgroup-by on the existinggateway success-ratio report so auth rate reads as a trend.
The last three read off the Billing Event stream — the first real use of it as a
✢ Accomplishing… (44s · ↓ 2.0k tokens)
The last three read off the Billing Event stream — the first real use of it as a
read model. They therefore only see transitions since the stream began, which the
docstrings say plainly rather than implying deeper history.
Correcting a price after the month is billed
A usage rollup carries the rate and allowance locked at first receipt; that
grandfathering is what makes a metered charge auditable months later, so editing it
in place would destroy the property it exists for.
metering.override_termswrites a new version of the row with the correctedterms and points the old one at it via
superseded_by._metered_linesfilterssuperseded rows so the same usage is never billed twice, and
ingest_rollupfollowsthe chain so a later usage report for a re-priced period lands on the row that is
actually billed rather than the retired one.
revenue/rerating.pyis the operator flow:preview()returns every affectedinvoice with its current total, what it would be rated at today, and the delta —
changing nothing.
apply()re-issues through the existing cancel+reissue path, oneinvoice at a time committed as it goes, so a failure half way through leaves the
ones it already fixed correct. The whole thing is recorded as a
Rerating Run(what, why, who, the preview, and what happened to each invoice).
Paid invoices are excluded: money already taken is a refund's problem, and silently
rewriting a bill someone has paid is not a correction.
Measuring instead of extrapolating
Every throughput number in the scope doc was arithmetic.
tests/test_scale.pyseedsN teams with real subscriptions and asserts three things: drafting stays inside a
per-team wall-clock budget, no money invariant breaks over a full run, and a run
killed half way through and restarted produces exactly one invoice per team.
NisSCALE_TEAMS, so the same test is a fast guard by default and a load run before arelease.
Measured at 1000 teams: ~14ms per team to draft, single process — against the 2s
per invoice the scope doc assumed. The budget in the test sits far above that on
purpose; it exists to catch a regression back into a per-team query, not to
benchmark the machine.