feat(encoding): name and decode text encodings

[andiwand]

🤖 Generated with Claude Code

First of five stacked PRs driving the CSV implementation toward a spreadsheet interface. The plan lives in src/odr/internal/csv/PLAN.md, added here.

No csv in this one. It builds the substrate the rest stands on, and closes a live bug on its own.

What was wrong

• TextFile::charset() was a stub returning {}.
• The charset uchardet guessed was used nowhere — html/text_file.cpp piped raw bytes into a document declaring <meta charset="UTF-8"> (// TODO charset), so every non-UTF-8 text file rendered as mojibake.
• guess_charset drained the whole stream, so classifying one unrecognised large file cost a full pass over it — twice over, since the csv probe did the same.

What this adds

internal/encoding, a package of its own rather than a corner of internal/text, because it is what pdf would later share.

• Public TextEncoding enum with a name table mirroring file_type_table: canonical name first, aliases after, one row per encoding, and a test that fails when an alias is claimed twice. Matching ignores case and -/_/space, so windows-1252, WINDOWS_1252 and cp1252 land on the same row. uchardet hands back a name; this is what maps it home.
• to_utf8 covering UTF-8/16/32 and the WHATWG single-byte set. Tables generated from Python's own codecs by tools/encoding/generate_encoding_data.py, in the output shape tools/pdf already uses.
• Bounded detection (64 KiB), with a NUL byte as the binary test. uchardet answers which encoding, never whether it is text — and that gate is what open_strategy.cpp:276 leans on, so text::TextFile now throws NoTextFile where it threw UnknownCharset.

Malformed input becomes U+FFFD rather than an error: a decoder that throws half way through a document leaves the caller with nothing, and a wrong guess is the expected failure here — that is what the override in the next PRs is for.

Multi-byte legacy encodings (Shift-JIS, GBK, Big5, EUC-KR, …) are named but not decoded. That is enough to render them as text with their own charset in the html header and let the browser decode. Only the spreadsheet path, which must hand UTF-8 to the bindings, is closed to them. PLAN.md records the two routes for closing that gap and their costs.

API

• TextFile::encoding() — new, and the first working answer.
• TextFile::charset() — kept, [[deprecated]], implemented via encoding(). Bindings moved off it so the attribute stays quiet.
• TextFile::text() — now decodes to UTF-8 where it can, raw bytes where it cannot.
• UnknownCharset — deprecated; nothing throws it any more.

Tests

22 new, covering the table's uniqueness and round-tripping, single-byte/UTF-16/UTF-32 decoding, U+FFFD on broken input, BOM handling (a mark is only stripped by its own encoding), and the binary gate. Full suite green, including all 234 reference-output tests — no output drift.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c462066d7c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".