Skip to content

[To dev/1.3] clone partial columns of aligned tvlist for query - #18394

Open
shuwenwei wants to merge 12 commits into
dev/1.3from
partial_tvlist_clone_dev_1.3
Open

[To dev/1.3] clone partial columns of aligned tvlist for query#18394
shuwenwei wants to merge 12 commits into
dev/1.3from
partial_tvlist_clone_dev_1.3

Conversation

@shuwenwei

Copy link
Copy Markdown
Member

Description

For wide aligned devices, a query that only touches a few columns currently
clones the entire aligned TVList (all columns) in the working memtable, which
is very expensive in both memory and time. This PR makes the query-side clone
copy only the accessed columns and move the remaining uncloned columns by
reference into the clone, so partial-column queries no longer duplicate the
whole aligned device.

Changes

  • AlignedTVList: add clone(Set<Integer>), cloneColumnDataTo and
    moveUnclonedColumnsTo so the clone keeps only the accessed columns and the
    uncloned columns are transferred by reference into the new working list.
  • ResourceByPathUtils.prepareTvListMapForQuery: pass the accessed column
    index list into the clone path; re-fetch and re-verify the working TVList
    under the memChunk lock after acquiring its queryListLock, and perform
    clone + column move + working-list swap in the same memChunk critical
    section so a concurrent query never observes a detached list whose columns
    were already moved away.
  • FragmentInstanceContext: track column-level access for AlignedTVList.

Tests

  • AlignedTVListTest: partial clone keeps only the accessed columns and moves
    the uncloned ones by reference.
  • FragmentInstanceExecutionTest: querying partial columns of an aligned
    device returns correct results.

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
  • added unit tests or modified existing tests to cover new code paths
  • added integration tests.
  • been tested in a test IoTDB cluster.

shizy818 and others added 12 commits August 4, 2026 18:11
# Conflicts:
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/AlignedTVList.java
…l clone

Re-fetch and re-verify the working TVList under the memChunk lock after
acquiring its queryListLock. The clone, column move and working-list swap
now happen in the same memChunk critical section, so a concurrent query can
never observe a working TVList whose uncloned columns were already moved
away, and the fast paths never read a detached list.
… calc

refreshArrayMemCostWithoutIndex() was invoked before the values field was
initialized, but alignedTvListArrayMemCost(Set) now dereferences
values.get(column) to skip moved columns, so construction of any aligned
TVList threw NPE. Initialize values before computing the array mem cost.
@codecov

codecov Bot commented Aug 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 69.23077% with 68 lines in your changes missing coverage. Please review.
✅ Project coverage is 44.30%. Comparing base (49d6dad) to head (256cece).
⚠️ Report is 1 commits behind head on dev/1.3.

Files with missing lines Patch % Lines
...he/iotdb/db/utils/datastructure/AlignedTVList.java 74.80% 32 Missing ⚠️
...engine/schemaregion/utils/ResourceByPathUtils.java 71.64% 19 Missing ⚠️
...ne/execution/fragment/FragmentInstanceContext.java 18.75% 13 Missing ⚠️
...e/dataregion/memtable/AlignedReadOnlyMemChunk.java 0.00% 2 Missing ⚠️
.../dataregion/memtable/AbstractWritableMemChunk.java 83.33% 1 Missing ⚠️
...geengine/dataregion/memtable/ReadOnlyMemChunk.java 50.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             dev/1.3   #18394    +/-   ##
===========================================
  Coverage      44.29%   44.30%            
  Complexity       198      198            
===========================================
  Files           3634     3634            
  Lines         245511   245718   +207     
  Branches       30045    30118    +73     
===========================================
+ Hits          108746   108859   +113     
- Misses        136765   136859    +94     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants