Skip to content

Fix bytes serializer dropping non-UTF-8 values to null - #1378

Open
gaoflow wants to merge 1 commit into
jsvine:developfrom
gaoflow:fix-bytes-serializer-encoding
Open

Fix bytes serializer dropping non-UTF-8 values to null#1378
gaoflow wants to merge 1 commit into
jsvine:developfrom
gaoflow:fix-bytes-serializer-encoding

Conversation

@gaoflow

@gaoflow gaoflow commented Jun 30, 2026

Copy link
Copy Markdown

The bug

Serializer.do_bytes serializes any non-UTF-8 bytes attribute to null, so values silently disappear from to_json / to_csv / to_dict output:

from pdfplumber.convert import Serializer, ENCODINGS_TO_TRY
print(ENCODINGS_TO_TRY)                    # ['utf-8', 'latin-1', 'utf-16', 'utf-16le']
print(Serializer().serialize(b"caf\xe9"))  # -> None   (expected 'café')

End-to-end, on the shipped tests/pdfs/issue-463-example.pdf (an annotation whose Contents is UTF-16BE):

import json, pdfplumber
pdf = pdfplumber.open("tests/pdfs/issue-463-example.pdf")
a = json.loads(pdf.to_json(object_types=["annot"]))["pages"][0]["annots"][0]
print(a["contents"], "|", a["data"]["Contents"])   # '日本語' | None   <- data loss

Root cause

def do_bytes(self, obj: bytes) -> Optional[str]:
    for e in ENCODINGS_TO_TRY:
        try:
            return obj.decode(e)
        except UnicodeDecodeError:  # pragma: no cover
            return None             # <-- aborts the loop on the first failure

The except returns None on the first encoding that fails, so the latin-1 / utf-16 / utf-16le fallbacks never run. git blame shows the type-annotation refactor in 9587cc7 changed the original pass (continue the loop) to return None. Because latin-1 decodes any byte string, the intended loop never returns None and never reaches the trailing re-raise — both of which became dead code (the # pragma: no cover even encodes the now-false assumption that the except never fires).

The fix

Restore the loop: continue to the next encoding instead of returning None (and drop the now-reachable # pragma: no cover). One line; the trailing re-raise is left untouched.

I deliberately do not change the encoding orderlatin-1 precedes utf-16, so a UTF-16 value comes back as latin-1 text rather than its "ideal" decoding. That ordering is a separate, pre-existing design choice; restoring the loop exactly matches the original behaviour and fixes the data loss without scope creep.

Tests (tests/test_convert.py)

  • test_serialize_bytes_encoding_fallback pins the contract: utf-8 passthrough is unchanged, and each of several non-UTF-8 byte strings serializes to its latin-1 string rather than None.
  • test_json_non_utf8_bytes_preserved exercises the public to_json API on the shipped issue-463-example.pdf fixture, asserting both the top-level contents and the serialized data.Contents survive as strings.

Both fail on the current code with the value dropped to None/null and pass with the fix; tests/test_convert.py + tests/test_basics.py stay green (39 passed); black/isort/flake8 clean. CHANGELOG updated.

Serializer.do_bytes returned None on the first encoding that failed to
decode, so the latin-1/utf-16/utf-16le fallbacks in ENCODINGS_TO_TRY never
ran. Any non-UTF-8 bytes attribute was therefore serialized to null by
to_json/to_csv/to_dict -- silent data loss (e.g. a UTF-16BE annotation
Contents value). Continue to the next encoding instead, restoring the
behavior from before the type-annotation refactor (9587cc7); latin-1
decodes any byte string, so values survive serialization.
@gaoflow
gaoflow force-pushed the fix-bytes-serializer-encoding branch from 8e7f232 to c0efcf9 Compare July 31, 2026 18:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant