Fix bytes serializer dropping non-UTF-8 values to null - #1378
Open
gaoflow wants to merge 1 commit into
Open
Conversation
Serializer.do_bytes returned None on the first encoding that failed to decode, so the latin-1/utf-16/utf-16le fallbacks in ENCODINGS_TO_TRY never ran. Any non-UTF-8 bytes attribute was therefore serialized to null by to_json/to_csv/to_dict -- silent data loss (e.g. a UTF-16BE annotation Contents value). Continue to the next encoding instead, restoring the behavior from before the type-annotation refactor (9587cc7); latin-1 decodes any byte string, so values survive serialization.
gaoflow
force-pushed
the
fix-bytes-serializer-encoding
branch
from
July 31, 2026 18:39
8e7f232 to
c0efcf9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Serializer.do_bytesserializes any non-UTF-8bytesattribute tonull, so values silently disappear fromto_json/to_csv/to_dictoutput:End-to-end, on the shipped
tests/pdfs/issue-463-example.pdf(an annotation whoseContentsis UTF-16BE):Root cause
The
exceptreturnsNoneon the first encoding that fails, so thelatin-1/utf-16/utf-16lefallbacks never run.git blameshows the type-annotation refactor in9587cc7changed the originalpass(continue the loop) toreturn None. Becauselatin-1decodes any byte string, the intended loop never returnsNoneand never reaches the trailing re-raise — both of which became dead code (the# pragma: no covereven encodes the now-false assumption that theexceptnever fires).The fix
Restore the loop:
continueto the next encoding instead of returningNone(and drop the now-reachable# pragma: no cover). One line; the trailing re-raise is left untouched.I deliberately do not change the encoding order —
latin-1precedesutf-16, so a UTF-16 value comes back as latin-1 text rather than its "ideal" decoding. That ordering is a separate, pre-existing design choice; restoring the loop exactly matches the original behaviour and fixes the data loss without scope creep.Tests (
tests/test_convert.py)test_serialize_bytes_encoding_fallbackpins the contract:utf-8passthrough is unchanged, and each of several non-UTF-8 byte strings serializes to itslatin-1string rather thanNone.test_json_non_utf8_bytes_preservedexercises the publicto_jsonAPI on the shippedissue-463-example.pdffixture, asserting both the top-levelcontentsand the serializeddata.Contentssurvive as strings.Both fail on the current code with the value dropped to
None/nulland pass with the fix;tests/test_convert.py+tests/test_basics.pystay green (39 passed);black/isort/flake8clean. CHANGELOG updated.