Add get_live_bytes() to report the current heap footprint of theta, tuple and kll sketches - #513
Draft
stojkomilos wants to merge 1 commit into
Draft
Add get_live_bytes() to report the current heap footprint of theta, tuple and kll sketches#513stojkomilos wants to merge 1 commit into
stojkomilos wants to merge 1 commit into
Conversation
…uple and kll sketches The sketch classes expose get_serialized_size_bytes() (an upper bound on the serialized form) but no accessor for the heap they are actually holding right now. get_live_bytes() reports that exact live footprint in O(1), which is useful for memory accounting when many mutable sketches are kept in memory at once. The accessor lives on theta_update_sketch_base (the hash table shared by theta and tuple update sketches, unions and intersections): it returns (1 << lg_cur_size_) * sizeof(Entry), or 0 when the table is unallocated. The public update_theta_sketch/tuple update sketch, theta/tuple union and theta/tuple intersection forward to it; tuple's union and intersection reuse the theta base, so no separate implementation is needed there. kll is not a hash table, so kll_sketch::get_live_bytes() sums the items_ buffer (items_size_ elements) and the levels_ vector, mirroring the sketch's allocate() sites. Adds test cases (theta update sketch, theta union, theta intersection, tuple update sketch, kll) asserting the footprint is a power-of-two entry count where applicable, is 0 for an unallocated intersection, and only grows as entries are inserted. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a public
get_live_bytes()accessor that reports, in O(1), the exact heap a sketch'smutable internal state is holding right now. The sketches already expose
get_serialized_size_bytes(), but that is an upper bound on the serialized form; there iscurrently no way to ask how much memory a live, mutable sketch is actually retaining. This is
useful for memory accounting when many mutable sketches are held in memory at once (e.g. one
sketch per group/window in a streaming aggregation).
Where
theta_update_sketch_base, the hash table shared bytheta and tuple update sketches, unions and intersections. It returns
(1 << lg_cur_size_) * sizeof(Entry), or0when the table is unallocated (lg_cur_size_ == 0leaves
entries_ == nullptr).update_theta_sketch,theta_union,theta_intersection, thetuple update sketch,
tuple_unionandtuple_intersectionforward to it. Tuple's union andintersection reuse the theta base as their
State, so no separate implementation is neededthere.
kll_sketch::get_live_bytes()sums theitems_buffer(
items_size_elements) and thelevels_vector, mirroring the sketch'sallocate()sites.Unlike
get_serialized_size_bytes(), this touches no data and does no iteration; it just reportsthe current allocation size.
Tests
Adds Catch2 cases in
theta_sketch_test,theta_union_test,theta_intersection_test,tuple_sketch_testandkll_sketch_testthat:0,the initial allocation.
All theta / tuple / kll suites pass locally.
Notes
Scope is theta, tuple and kll.
hllandcpccould get an equivalent accessor in a follow-up ifmaintainers want the whole family consistent.