Fix JsonCodec decoder accepting malformed JSON with trailing characters - #1136
Fix JsonCodec decoder accepting malformed JSON with trailing characters#1136chengdazhi wants to merge 1 commit into
Conversation
Reject non-whitespace input remaining after a valid JSON value in the single-value decode entry points (jsonDecoder(...).decodeJson, JsonDecoder.decode, schemaBasedBinaryCodec.decode, zioJsonBinaryCodec.decode), while keeping stream decoding lenient. Closes zio#712
|
|
|
Note on the failing |
|
Note on CI: the single failing job, |
/claim 712
Closes #712
Problem
JsonCodec's decoders stop after the first valid JSON value and silently ignore any remaining input, so malformed documents like{}},{"a":1}}or"foo""decode successfully (see the Scastie reproducer in the issue).Fix
The strictness cannot be added by overriding
decodeJson, because zio-json declares itfinal; it funnels throughunsafeDecodewith aFastStringReader, while the streaming entry points (decodeJsonPipeline,decodeJsonStream, ...) use otherRetractReaderimplementations. This PR therefore:StrictJsonDecoderwrapper whoseunsafeDecoderejects any non-whitespace remainder, but only when the reader is aFastStringReader, so single-value decoding becomes strict while streaming stays lenient;42would otherwise be wrongly rejected);jsonDecoder(schema)and uses it inJsonDecoder.decode,schemaBasedBinaryCodec.decodeandzioJsonBinaryCodec.decode;schemaBasedBinaryCodec.streamDecoderandzioJsonBinaryCodec.streamDecoderlenient (viaJsonDecoder.decodeLenientand unwrapping ofStrictJsonDecoder), preserving existing behavior such as1 2, 3;;; 4x5decoding as a stream of five ints and partialFallbackdecodes ([30,"hello"]asFallback.Left(30)).Tests
New
trailing characters after a valid JSON valuesuite inJsonCodecSpec, covering the issue reproducers ({}},{"a":1}},"foo""), extra characters after arrays/booleans/numbers, top-level numbers (the retract quirk), acceptance of leading/trailing whitespace, strictness ofschemaBasedBinaryCodec.decodeandzioJsonBinaryCodec.decode, and unchanged newline-delimited stream decoding.Verified locally:
zioSchemaJsonJVM/teston Scala 2.12.21, 2.13.18 and 3.3.8 (312 tests),zioSchemaJsonJS/test(307 tests),zioSchemaJsonNative/Test/compile,fmtCheck,fixCheckandmimaReportBinaryIssues.Comparison with #724
#724 applies the trailing check inside
schemaDecoderitself, so it also runs after every nested value (e.g. right after a field value, where the object's closing}follows), which breaks decoding of nested structures; it also reformats the wholeschemaDecodermatch and currently has merge conflicts withmain. This PR applies the check only at the top-level decode entry points, leaves nested decoding untouched, and additionally covers theBinaryCodecentry points and the number-retract edge case described above.This PR was prepared with the assistance of an AI coding agent; every change was verified by running the test suites and lint checks listed above.