feat(fsst): module skeleton, Symbol, Decompressor#290
Merged
Conversation
Introduce a standalone `vortex-fsst` module as the first step of #287 ("improve FSST implementation"). The current inline FSST encoder/decoder is a simplified first pass; comparing against vortex-jni surfaced real gaps in matching, training, and decode. Following the paper (Boncz/Neumann/Leis, VLDB 2020) and the Rust reference's split of a pure algorithm crate (spiraldb/fsst) from the thin wire adapter, and since research found no zero-dependency Java FSST library to reuse, the algorithm is isolated into its own module so it can be iterated on and benchmarked without the Vortex wire-format shell in the way. This PR is purely additive — no existing code is touched. It ports the lowest-risk piece first: the `Symbol` record (LSB-first byte packing, the same convention as the writer's existing SymbolCandidate) and a byte[]-based `Decompressor` implementing the paper's Algorithm 1 (escape-or-symbol decode loop). The MemorySegment-native hot path, branch-free matching, training, and the adapter rewire that makes writer/reader depend on this module all land in later PRs of the sequence. Unlike the build-tooling generators, this module ships to Maven Central (writer/reader will depend on it at runtime), so its POM omits `maven.deploy.skip`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 1 of an 8-PR sequence implementing #287 ("improve FSST implementation"). Full plan: paper-faithful rewrite of FSST (branch-free matching, adaptive-sample training with reference refinements, unconditional-8-byte-store decode) isolated into a standalone
vortex-fsstmodule — no zero-dep Java FSST library exists to reuse, and the Rust reference itself splits a pure algorithm crate (spiraldb/fsst) from the thin wire adapter (vortex-fsst), which is exactly what this mirrors.This PR is purely additive:
fsstmodule (vortex-fsst), modeled onfbs-gen's pom but publishable (nomaven.deploy.skip) sincewriter/readerdepend on it starting in PR 5.Symbolrecord — LSB-first byte packing, same convention as the writer's existingSymbolCandidate.Decompressor— byte[]-based, implements the paper's Algorithm 1. MemorySegment-native hot path is PR 4's scope.No existing code touched —
writer/readerunaffected.Test plan
./mvnw verify -pl fsst -am— 13 tests, checkstyle 0 violations./mvnw verify -DskipTestsat root — full 16-module reactor SUCCESS./mvnw javadoc:javadoc -pl fsst— zero output (build-enforced gate)🤖 Generated with Claude Code