Repository navigation
fix(uri_utils): unquote percent-encoded base64 payload in data URIs - #2682
Reginald Alfret V (reginaldalfret) wants to merge 2 commits into
Conversation
…icrosoft#2481) Signed-off-by: reginaldalfret <reginaldalfret@gmail.com>
PRABHU KIRAN VANDRANKI (VANDRANKI)
left a comment
There was a problem hiding this comment.
Community review (not a maintainer, does not clear the merge gate). I read the full diff (the one-line change in _uri_utils.parse_data_uri and the two new assertions in tests/test_module_misc.py) and ran it on Windows with Python 3.13.
What I ran, calling parse_data_uri directly on this PR head and then on origin/main (4cc9fa1) with only _uri_utils.py swapped:
data:text/plain;base64,SGVsbG8%3D: PR givesb'Hello'. Main raises "Invalid base64-encoded string: number of data characters (9) cannot be 1 more than a multiple of 4".data:application/octet-stream;base64,%2B/8=: PR givesb'\xfb\xff', which is the correct decode of+/8=. Main returnsb'\xd8\x1f\xfc', i.e. silent corruption: the%is dropped by the lenientb64decodeand the2Bis read as base64 digits. This second case is the more serious one, since there is no error.+/8=with no percent-encoding still givesb'\xfb\xff', so a literal+is not turned into a space (unquote_to_bytesdoes not do that). Also unchanged and still fine:%0Aand%20inside the payload (whitespace is dropped as before), and a payload with trailing=%20.- tests/test_module_misc.py on the PR head: 169 passed, 1 failed, 1 skipped. The failure is
test_speech_transcription(a markitdown exception); the same test fails on main in my environment, so I treat it as environmental and unrelated.
Notes, none blocking:
- Behaviour for malformed escapes is the same as before and still lenient:
...base64,SGVsbG8%3(truncated escape) returnsb'Hello7'on both main and the PR, becauseunquote_to_bytesleaves a bad escape alone andb64decodediscards the%. If you want strictness,validate=Truewould be a separate change. parse_data_urihas one caller in this repo (MarkItDown.convert_uri, _markitdown.py line 497) by grep, so the blast radius is small. I did not check whether the HTML or other converters decode inlinedata:images through a different path; they do not call this function.- Optional test addition: a case mixing a literal
+and%2Bin one payload would pin the "no plus-to-space" behaviour.
The fix is correct and minimal: unquote first, then base64-decode, as RFC 2397 requires for the data part of a URI.
…plus (microsoft#2481) Signed-off-by: reginaldalfret <reginaldalfret@gmail.com>
|
Added the suggested test case in packages/markitdown/tests/test_module_misc.py with mixed literal + and percent-encoded %2B in data:application/octet-stream;base64,%2B+8= asserting �'\xfb\xef' to pin the no plus-to-space behavior. Verified tests pass cleanly. |
Fixes #2481
Problem
According to RFC 2397 Section 3, data URIs permit percent-encoded octets (
uric) within the data payload. For base64-encoded payloads, characters like padding (=as%3D), plus signs (+as%2B), or slashes (/as%2F) may be percent-encoded.Previously,
parse_data_uripassed the raw data slice directly tobase64.b64decode(data)without unquoting percent-escapes first:SGVsbG8%3D) causedbase64.b64decodeto raisebinascii.Error: Invalid base64-encoded string.%2B/8=) were decoded incorrectly (interpreting%,2,Bas data characters), producing corrupted outputb'\xd8\x1f\xfc'instead ofb'\xfb\xff'.Solution
_uri_utils.py, applyunquote_to_bytes(data)prior tobase64.b64decode.tests/test_module_misc.pytesting RFC 2397 percent-encoded base64 payloads with both padding (%3D) and characters (%2B).Verification
pytest packages/markitdown/tests/test_module_misc.py -k test_data_uris: passed cleanly.