feat(fsst): branch-free matching (ShortCodeTable + LossyPerfectHashTable + Matcher)#291
Merged
Merged
Conversation
…ble + Matcher)
Implements the FSST paper's Algorithm 4 ("lossy perfect hashing", §5.1) as the
matching core for the extracted fsst module (issue #287, PR 2 of the plan in
goofy-roaming-moler.md).
WHY: the current writer's FsstEncodingEncoder.SymbolTable.longestMatch probes
symbol lengths 8 down to 1 in a per-position loop — up to eight sequential
open-addressing hash lookups per input byte. That variable-length loop is
exactly the non-uniform hot-loop body CLAUDE.md forbids (it blocks C2
auto-vectorization). This replaces it with an O(1) branch-free structure:
- ShortCodeTable: a 65536-entry array direct-indexed on the first two input
bytes, resolving 0/1/2-byte matches in one array read.
- LossyPerfectHashTable: 2048 slots (matching the spiraldb/fsst Rust reference's
L1D-cache-line-split reasoning and the vortex-jni benchmark target, not the
paper's literal 4096), keyed on the first three bytes, resolving 3-8 byte
matches with one hash lookup plus one masked 8-byte compare — no probing.
- Matcher: composes the two branch-free, no loop over candidate lengths.
Collision-ordering invariant (load-bearing): the hash table inserts in a single
forward pass with first-writer-wins. The caller must pass symbols in
descending-gain order, so on a 3-byte-prefix hash collision the higher-gain
symbol (seen first) keeps the slot and the lower-gain one is rejected, never
overwritten — the paper's rule that the more valuable symbol wins a lossy
collision. These classes do not re-sort their input. Tests cover a real
collision (higher gain wins, lower gain correctly misses) and an adversarial
hash-collision-but-byte-mismatch case proving the masked compare, not the hash
alone, gates a hit.
Purely additive: writer/reader untouched; these are wired into training in PR 3.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 2 of the #287 FSST rewrite sequence. Implements the paper's Algorithm 4 ("lossy perfect hashing") as an O(1) branch-free replacement for the current encoder's up-to-8-sequential-hash-probe matching loop — the single biggest perf gap identified against the paper/Rust reference.
ShortCodeTable— 65536-entry direct array resolving 0/1/2-byte matches in one read.LossyPerfectHashTable— 2048 slots (matching thespiraldb/fsstreference's L1D-cache reasoning, our actualvortex-jnicomparison target, not the paper's 4096) resolving 3-8 byte matches via one hash lookup + one masked compare, no probing.Matcher— composes both, no loop over candidate lengths.Collision-ordering invariant (load-bearing): the hash table's single-forward-pass first-writer-wins insertion requires the caller to pass symbols in descending-gain order. Tested with both a real collision (higher gain wins the slot) and an adversarial hash-collision-but-byte-mismatch case (masked compare, not the hash alone, gates a hit).
Purely additive —
writer/readeruntouched. Wired into training in PR 3.Test plan
./mvnw test -pl fsst -am— 28 tests, 0 failures./mvnw verify -pl fsst -am— checkstyle passes./mvnw javadoc:javadoc -pl fsst— zero output./mvnw verify -DskipTestsat root — full 16-module reactor SUCCESS🤖 Generated with Claude Code