test(fsst): golden test for the paper's Figure 2 worked example#296
Merged
Conversation
Add PaperFigure2Test reproducing the FSST paper's Figure 2 worked example
(Boncz/Neumann/Leis, PVLDB 13(11) 2020, p. 2653): corpus "tumcwitumvldb",
maximum symbol length 3, maximum table size 5. After the paper's fourth
iteration the symbol table is {tum, cwi, vld, b} and the figure's
"Compressed" row shows tum|cwi|tum|vld|b, len = 5.
The test hand-builds that exact table via Compressor.of(List<Symbol>) rather
than CompressorBuilder.train(...) — training would re-derive a table through
this codebase's own heuristics, which is circular. A golden test needs an
external, human-verifiable fixed point, so the symbols and codes come straight
off the figure (gain order tum=0, cwi=1, vld=2, b=3). It asserts the greedy
compressor reproduces both the code sequence [tum,cwi,tum,vld,b] and length 5
(paper fidelity), and that the result round-trips to the original 13 bytes
(primary correctness gate). The javadoc cites Figure 2 so a maintainer can
re-verify against the primary source; the figure was re-fetched and confirmed
from the authors' cwida/fsst reference repo (fsstcompression.pdf, p. 2653).
Also tighten FsstEncodingEncoderTest
.encode_repeatedText_learnsMultiByteSymbols_compressesBelowRaw from
maxSymbolLength > 2 to > 4. On that test's input ("the quick brown fox jumps
over the lazy dog" x100) the paper-faithful trainer deterministically learns
ten length-8 symbols and one length-3 symbol (max 8) with the fixed 0x5EED
encoder seed, so a > 4 floor is a stronger regression guard with ample headroom
against a training change that quietly stopped growing symbols past a trigram.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 6 of #287 — test-only, no production code touched.
PaperFigure2Test: hand-builds the exact symbol table from the FSST paper's Figure 2 (corpus"tumcwitumvldb", table{tum, cwi, vld, b}) — not trained, since that would be circular — and asserts the greedy compressor reproduces the figure's own compressed sequence[tum,cwi,tum,vld,b](length 5) plus a full round-trip. The figure's exact contents were re-verified against the authors' own reference-repo copy of the paper (the vldb.org mirror truncates before Figure 2).FsstEncodingEncoderTest'smaxSymbolLength > 2assertion to> 4— the paper-faithful trainer deterministically learns ten length-8 symbols on that test's fixed input, so this is a meaningfully stronger regression guard with ample headroom.integrationsuite (not scoped to FSST), and a clean root reactor build.Test plan
./mvnw test— all modules green./mvnw verify -pl integration -am— 294 tests, 0 failures (Raincloud self-excluded by default)./mvnw verify -DskipTestsat root — full reactor SUCCESSfsst/writertest suites and hand-traced the golden test's greedy parse myself🤖 Generated with Claude Code