feat(csv): probe for a dialect instead of checking for one

[andiwand]

🤖 Generated with Claude Code

Second of five stacked PRs. Based on #665 — review that first. Plan: src/odr/internal/csv/PLAN.md.

What was wrong

check_csv_file conflated two jobs — is this a csv and can this be parsed — and answered both by reading the whole file. open_strategy.cpp:281 called it speculatively for every unrecognised text file, so classifying one large .txt cost a full pass over it. It also hard-coded , and ", and refused any file whose records disagreed on a field count.

The split

RecordReader reads records out of decoded UTF-8 text for a given dialect and judges nothing. Ragged records come out ragged. An unterminated quote still yields its field and sets a flag. Whether either is a reason to refuse the file is a question for detection, not parsing.

probe scores a bounded sample and returns the dialect it resolved plus a verdict:

• tries ,, ;, tab, |; takes the field count most records carry; picks the separator explaining the most records
• Excel's sep=; opening line wins outright
• verdict wants ≥ 2 columns and no dangling quote in a complete file

Both verdict rules are heuristics, not validity rules — one column is every line of prose ever written, and a dangling quote is good evidence of not-csv. Neither says a one-column csv is illegitimate. Once a caller declares a file to be csv (next PR), it goes straight to RecordReader and is never asked to pass this.

A sample cut mid-record says nothing about the record it cut, so the last record is dropped unless the sample is the whole file. That is why a_dangling_quote_in_a_complete_file_is_evidence_against and a_dangling_quote_in_a_sample_is_not are separate tests.

The sample is decoded before scanning, which is what makes a UTF-16 csv detectable at all — Excel's "Unicode Text" export is UTF-16LE and tab-separated, so the separator is invisible in the raw bytes.

Tests

The old csv_file_test assertions move to the probe side, where they belong. a_quoted_field_must_be_terminated carried a comment recording a real misclassification; that rule survives, on the detection side only. New coverage for separator scoring, the sep= directive, the complete-vs-sample distinction, and RecordReader's new tolerance.

Full suite green (559 unit, 234 reference-output, no drift).

[chatgpt-codex-connector]

💡 Codex Review

https://github.com/opendocument-app/OpenDocument.core/blob/528cb2f887ef63098dbf28734392ff4ff557b523/output/odr-private/output/pdf/mova-viax-500-user-manual-de.pdf/document.html#L1
Remove generated private render output

This adds 624 generated artifacts under output/odr-private, output/odr-public, and media_* (about 459 MB of HTML/media) to the source tree. In the checked test workflow these are transient translate/test outputs that should stay outside the repo tree; keeping them in this feature commit will massively bloat every clone and risks publishing private corpus render output, so please drop these files from the commit.

AGENTS.md reference: AGENTS.md:L91-L91

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".