pcc-eval-toolkit

English | 简体中文

Honest rate-distortion evaluation for compression research: Bjontegaard-Delta metrics with hard validity gates, a defensible single-point metric, and publication-style RD overlay plots.

Built while reproducing a learned point-cloud attribute compression paper, where I kept needing three things the usual snippets don't give you: BD numbers that refuse to exist when the data can't support them, a legitimate way to compare a fixed-rate model (one operating point) against a curve, and plots where every legend entry carries its own verdict.

Why gates instead of graceful fallbacks

BD-rate integrates the horizontal distance between two interpolated curves over their shared quality range. Two failure modes silently produce plausible-looking garbage:

1. Too few points. A cubic through 2–3 points is a guess, not a curve. This toolkit requires ≥ 4 points per side — otherwise you get GateFailure("test has 1 points, need >= 4"), printed verbatim into the plot legend.
2. No overlap. If the test codec's quality range doesn't intersect the anchor's, any number requires extrapolation. Refused.

The same philosophy applies to the single-point metric: matched_quality_ratio interpolates only on the reference curve (no assumption about the test codec's curve shape) and refuses when the test point lies outside the reference range. It reports "at this quality, X% more rate" — a statement about one quality level, clearly not a BD-rate.

Quick start

pip install -e .
python -m pcc_eval.selftest       # analytic sanity suite - must print SELFTEST PASS
python examples/demo.py           # renders the figure above from synthetic data

from pcc_eval.bdrate import bd_rate
from pcc_eval.matched import matched_quality_ratio

res = bd_rate(rate_anchor, psnr_anchor, rate_test, psnr_test)
if res.ok:
    print(f"BD-rate {res.value:+.2f}% over quality {res.quality_lo:.1f}-{res.quality_hi:.1f} dB")
else:
    print(res)   # e.g. "no BD: no quality overlap (anchor 30.1-38.2 vs test 39.0-41.5)"

mq = matched_quality_ratio(rate_ref, psnr_ref, test_point=(1.24, 38.4))

What's inside

Module	What it does
pcc_eval.bdrate	BD-rate & BD-quality, Akima interpolation in the (quality, log-rate) plane, gate-checked
pcc_eval.matched	Matched-quality rate ratio for single operating points (interpolation-only, refuses extrapolation)
pcc_eval.plot	Multi-panel RD overlays: per-sequence + average, BD verdicts (or refusal reasons) in the legend
pcc_eval.roundtrip	Codec-agnostic round-trip, determinism and bit-exactness checks; failures report the first differing byte offset or point index
pcc_eval.selftest	Analytic ground-truth suite: identical curves → exactly 0.00%, rate ×1.1 → exactly +10%, gate refusals, and each round-trip check pointed at a codec built to fail it

Before the numbers: is the codec self-consistent?

An RD point from a codec that cannot reproduce its own bitstream is not a result. pcc_eval.roundtrip takes an encode/decode pair of callables and checks the things that have to hold first:

from pcc_eval.roundtrip import verify_roundtrip

report = verify_roundtrip(encode, decode, cloud)   # encode(cloud)->bytes, decode(bytes)->cloud
assert report.ok, report.summary()

Check	What must hold
encoder_determinism	the same cloud encoded twice is byte identical
decoder_determinism	the same bitstream decoded twice gives the same cloud
reencode_bit_exactness	encode(decode(bits)) == bits, byte for byte
reconstruction_fidelity	the decoded cloud matches the input, exactly or within a stated tolerance
context_agreement	two encoders you supply -- CPU and GPU, two seeds, two builds -- produce the same bitstream

Three things follow the same philosophy as the BD gates:

• reencode_bit_exactness applies to lossy codecs too. The reconstruction may differ from the input; re-encoding it may not change the bitstream. When it does, encoder and decoder disagree about internal state and drift further apart over a sequence.
• Nothing claimed, nothing checked. A lossy codec with no stated tolerance gets reconstruction_fidelity skipped, not quietly passed.
• Failures carry a location. A boolean tells you a bug exists, which you already knew:

toy codec: FAIL (300 points, 3361 bytes)
  PASS encoder_determinism: 3 encodes agreed, 3361 bytes
  PASS decoder_determinism: 3 decodes agreed
  FAIL reencode_bit_exactness [at 18]: re-encoded stream differs at byte 18
      original:   45 98 09 d8 8f 65 16 c6  (from offset 14)
      re-encoded: 45 98 09 d8 97 63 16 c6  (from offset 14)
  FAIL reconstruction_fidelity [at 137]: geometry differs at point 137: [900, 993, 734] vs [900, 994, 734]

The case worth knowing about: a codec whose header carries a field the decoder discards reconstructs every point correctly and still fails to reproduce its bitstream. A reconstruction-only test calls that codec correct. The selftest contains exactly that codec, plus one with an unseeded nonce, one that loses a single point, and a quantiser -- each pointed at the check it is built to break, because a harness that cannot catch a codec written to defeat it is decoration.

Practices this encodes

• Self-calibrate before you measure. anchor-vs-anchor must be exactly 0.00% — the selftest makes that a one-command proof, and it should run in CI (it does here).
• A refused metric is a result. Point-count tables and refusal reasons tell you which experiments are still missing — printing pts=1, need >= 4 next to a sequence is a roadmap, not an error.
• Self-consistency comes before accuracy. Determinism and a bit-exact round-trip are cheap to check and are preconditions for every number downstream. Establishing them is not the same as conformance to a standard, and this repository does not claim the latter.
• Single-point comparisons are allowed but labelled. Fixed-rate learned codecs are common; comparing them honestly against curves is possible without slope assumptions — as long as the metric never masquerades as a BD-rate.

License

MIT