Skip to content

fix(uri_utils): unquote percent-encoded base64 payload in data URIs - #2682

Open
Reginald Alfret V (reginaldalfret) wants to merge 2 commits into
microsoft:mainfrom
reginaldalfret:fix/issue-2481-percent-encoded-data-uri
Open

Reginald Alfret V (reginaldalfret) wants to merge 2 commits into
microsoft:mainfrom
reginaldalfret:fix/issue-2481-percent-encoded-data-uri

Conversation

@reginaldalfret

Copy link
Copy Markdown

Fixes #2481

Problem

According to RFC 2397 Section 3, data URIs permit percent-encoded octets (uric) within the data payload. For base64-encoded payloads, characters like padding (= as %3D), plus signs (+ as %2B), or slashes (/ as %2F) may be percent-encoded.

Previously, parse_data_uri passed the raw data slice directly to base64.b64decode(data) without unquoting percent-escapes first:

  • Percent-encoded padding (e.g. SGVsbG8%3D) caused base64.b64decode to raise binascii.Error: Invalid base64-encoded string.
  • Percent-encoded bytes (e.g. %2B/8=) were decoded incorrectly (interpreting %, 2, B as data characters), producing corrupted output b'\xd8\x1f\xfc' instead of b'\xfb\xff'.

Solution

  • In _uri_utils.py, apply unquote_to_bytes(data) prior to base64.b64decode.
  • Added test cases in tests/test_module_misc.py testing RFC 2397 percent-encoded base64 payloads with both padding (%3D) and characters (%2B).

Verification

  • Tested with pytest packages/markitdown/tests/test_module_misc.py -k test_data_uris: passed cleanly.

…icrosoft#2481)

Signed-off-by: reginaldalfret <reginaldalfret@gmail.com>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Community review (not a maintainer, does not clear the merge gate). I read the full diff (the one-line change in _uri_utils.parse_data_uri and the two new assertions in tests/test_module_misc.py) and ran it on Windows with Python 3.13.

What I ran, calling parse_data_uri directly on this PR head and then on origin/main (4cc9fa1) with only _uri_utils.py swapped:

  • data:text/plain;base64,SGVsbG8%3D: PR gives b'Hello'. Main raises "Invalid base64-encoded string: number of data characters (9) cannot be 1 more than a multiple of 4".
  • data:application/octet-stream;base64,%2B/8=: PR gives b'\xfb\xff', which is the correct decode of +/8=. Main returns b'\xd8\x1f\xfc', i.e. silent corruption: the % is dropped by the lenient b64decode and the 2B is read as base64 digits. This second case is the more serious one, since there is no error.
  • +/8= with no percent-encoding still gives b'\xfb\xff', so a literal + is not turned into a space (unquote_to_bytes does not do that). Also unchanged and still fine: %0A and %20 inside the payload (whitespace is dropped as before), and a payload with trailing =%20.
  • tests/test_module_misc.py on the PR head: 169 passed, 1 failed, 1 skipped. The failure is test_speech_transcription (a markitdown exception); the same test fails on main in my environment, so I treat it as environmental and unrelated.

Notes, none blocking:

  1. Behaviour for malformed escapes is the same as before and still lenient: ...base64,SGVsbG8%3 (truncated escape) returns b'Hello7' on both main and the PR, because unquote_to_bytes leaves a bad escape alone and b64decode discards the %. If you want strictness, validate=True would be a separate change.
  2. parse_data_uri has one caller in this repo (MarkItDown.convert_uri, _markitdown.py line 497) by grep, so the blast radius is small. I did not check whether the HTML or other converters decode inline data: images through a different path; they do not call this function.
  3. Optional test addition: a case mixing a literal + and %2B in one payload would pin the "no plus-to-space" behaviour.

The fix is correct and minimal: unquote first, then base64-decode, as RFC 2397 requires for the data part of a URI.

…plus (microsoft#2481)

Signed-off-by: reginaldalfret <reginaldalfret@gmail.com>
@reginaldalfret

Copy link
Copy Markdown
Author

Added the suggested test case in packages/markitdown/tests/test_module_misc.py with mixed literal + and percent-encoded %2B in data:application/octet-stream;base64,%2B+8= asserting �'\xfb\xef' to pin the no plus-to-space behavior. Verified tests pass cleanly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Base64 data URIs fail or decode incorrect bytes when their payload is percent-encoded

2 participants