fix(fs): create parent directories before opening Jindo writers#430
Open
Zouxxyy wants to merge 3 commits into
Open
fix(fs): create parent directories before opening Jindo writers#430Zouxxyy wants to merge 3 commits into
Zouxxyy wants to merge 3 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Jindo filesystem adapter so that Create() proactively creates the target file’s parent directory before opening a writer, improving compatibility with Jindo-backed object stores that require directory markers to exist.
Changes:
- Create parent directories (via
Mkdirs) inJindoFileSystem::Create()before opening a writer. - Refactor writer-opening into an overridable
OpenWriter()helper to enable focused unit testing. - Add unit tests for the parent-directory behavior and wire them into the Jindo FS test target.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/paimon/fs/jindo/jindo_file_system.h | Adds a protected OpenWriter() hook for Create() to call. |
| src/paimon/fs/jindo/jindo_file_system.cpp | Creates parent directory before writer open; factors writer open into OpenWriter(). |
| src/paimon/fs/jindo/jindo_file_system_unit_test.cpp | Adds unit tests verifying parent-dir creation ordering and error propagation. |
| src/paimon/CMakeLists.txt | Registers the new Jindo filesystem unit test source when Jindo is enabled. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+72
to
+75
| const std::string parent_path = PathUtil::GetParentDirPath(path); | ||
| if (!parent_path.empty()) { | ||
| PAIMON_RETURN_NOT_OK(Mkdirs(parent_path)); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: N/A
Ensure the Jindo file-system adapter creates a file's parent directory before opening its writer. Some Jindo-backed object stores require bucket or partition directories to exist before creating the first data file.
This prevents native writers from failing during flush or prepare-commit when writing the first file under a new directory.
Tests
cmake --build build --target paimon-fs-test -j 8./build/debug/paimon-fs-test(65 tests passed)uvx pre-commit run --files src/paimon/CMakeLists.txt src/paimon/fs/jindo/jindo_file_system.cpp src/paimon/fs/jindo/jindo_file_system.h src/paimon/fs/jindo/jindo_file_system_unit_test.cppAPI and Format
No public API, storage format, or protocol changes.
Documentation
No documentation changes. This fixes existing Jindo file creation behavior.
Generative AI tooling
Generated-by: OpenAI Codex (GPT-5)