fix: ParquetImporter detects duplicate column names before VortexWriter#288
Merged
Merged
Conversation
Previously the first signal a caller got was VortexWriter's internal
uniqueness check, several stack frames removed from the real cause and
with no mention of which import or source file triggered it:
java.lang.IllegalArgumentException: duplicate field name: A
at io.github.dfa1.vortex.writer.VortexWriter.<init>(VortexWriter.java:164)
at io.github.dfa1.vortex.parquet.ParquetImporter.importParquet(...)
checkNoDuplicateNames now runs right after the schema is built from the
Parquet columns, throwing a ParquetImporter-specific message naming the
duplicate(s) and the source file. Root cause is always the source
Parquet file's own schema (DType.Struct legitimately permits duplicate
names in memory; VortexWriter correctly enforces the file format's
uniqueness requirement) — this only improves error quality at the
import boundary, no auto-disambiguation.
Verified against the real repro (Raincloud's uk-price-paid slug, whose
source CSV lacked a header row so the Parquet conversion tool used the
first data row's values as column names):
IllegalArgumentException: Parquet schema has duplicate column
name(s): [A]; source file: .../uk-price-paid.parquet
Fixes #280.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ParquetImporternow checks the source Parquet schema for duplicate column names right after building it, and throws a clear, actionableIllegalArgumentExceptionnaming the duplicate(s) and the source file — instead of surfacingVortexWriter's internal uniqueness-invariant error several stack frames removed from the real cause.DType.Structlegitimately permits duplicate names in memory andVortexWritercorrectly enforces the file format's uniqueness requirement. This only improves error quality at the import boundary — no auto-disambiguation (would mask a real data-quality problem).Test plan
ParquetImporterTest$DuplicateNamesunit tests (no-dup passthrough + duplicate-throws with name/path in message)ParquetImporterTest: 26/26 passuk-price-paidslug, hydrated locally): now throwsIllegalArgumentException: Parquet schema has duplicate column name(s): [A]; source file: .../uk-price-paid.parquet🤖 Generated with Claude Code