Skip to content

feat(sdk): add isValidTdf with Go lax-schema validation, and tdf validate (DSPX-5352) - #428

Draft
dmihalcik-virtru wants to merge 2 commits into
DSPX-5103-harden-java-zipreader-against-hostile-central-direfrom
DSPX-5352-java-isvalidtdf
Draft

dmihalcik-virtru wants to merge 2 commits into
DSPX-5103-harden-java-zipreader-against-hostile-central-direfrom
DSPX-5352-java-isvalidtdf

Conversation

@dmihalcik-virtru

@dmihalcik-virtru dmihalcik-virtru commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

What and why

DSPX-5352 (epic DSPX-5348, TDF Validity). java-sdk had no equivalent of Go's sdk.IsValidTdf. SDK.isTDF only looks at entry names and never reads the manifest. The cross-SDK validity suite (DSPX-5349) needs a real check in Java, and a CLI hook to drive it.

Stacked on #418 (DSPX-5103, ZipReader hardening). Without it, hostile central directories can hang the reader or throw unchecked exceptions, so a validity check can't give a reliable answer. Review only the top two commits; the base branch is DSPX-5103-harden-java-zipreader-against-hostile-central-dire.

SDK: SDK.isValidTdf and SDK.validateTdf

public static boolean isValidTdf(SeekableByteChannel channel) throws IOException
public static void validateTdf(SeekableByteChannel channel) throws IOException // throws MalformedTDFException(reason)

The pair mirrors attributeExists / validateAttributes. validateTdf gives the reason (the first problem found, on one line), which the CLI prints. Malformed archive or manifest: false / MalformedTDFException. I/O failure: IOException, including a read failure part way through the manifest. No platform connection or credentials are needed.

A TDF is valid when all of these hold:

  1. Archive. It opens through TDFReader.open, the same container checks as SDK.readManifest and loadTDF, so "valid" never names a file that Java's own reader refuses. No entry name may appear twice, and there must be a 0.payload entry.
  2. Manifest choice. It uses manifest.json if present, otherwise 0.manifest.json, the same as Go and web-sdk. When the spec name is present but fails, there is no fallback.
  3. Strict JSON. The manifest must be exactly one RFC 8259 value: Gson in Strictness.STRICT, no BOM (Gson would skip one silently, so it is checked explicitly), nothing after the value, and nesting at most 10,000 deep, which is Go's encoding/json limit. Comments, single quotes, unquoted names, trailing commas, NaN, TRUE, \', raw control characters in strings, and the )]}' prefix are all refused.
  4. Lax schema. The manifest must satisfy manifest-lax.schema.json, vendored byte for byte from Go SDK sdk/v0.35.0 (sdk/schema/manifest-lax.schema.json, SHA-256 5e361720…f87d). The source, version and hash are recorded in the TDFValidator Javadoc. A test pins the hash of the classpath resource, which also catches the sdk module's resource filtering ever altering it.

It does not decrypt or verify anything: not the payload, not integrity, not the policy binding, not assertion signatures. The lax schema omits fields that decryption needs, so true does not promise loadTDF succeeds. The Javadoc says so.

Dependency: dev.harrel:json-schema 1.9.1

Nothing in the tree validated JSON Schema. I checked mvn dependency:tree: Gson, json-smart via nimbus, moshi via connect-kotlin, and no validator. Why this one:

  • License and size. MIT, about 210 KB, Java 8 bytecode. Every dependency in its pom is optional, so it adds nothing transitively.
  • Gson provider. It validates the Gson JsonElement we already parse strictly, so there is no second JSON parser with different leniency rules.
  • Compliance. It supports draft 2020-12 (the schema's $schema) and is tested against the official JSON Schema test suite through Bowtie. The schema only uses type/properties/required/enum/oneOf/items, but it also has quirks a validator has to get right. For example, the keyAccess required sits on the array rather than on items, so it constrains nothing. A spec-compliant library honors such quirks the way gojsonschema does, with no case-by-case reasoning.
  • No network. Meta-schemas are bundled, and there is no URL fetching in the jar (checked with javap).
  • Maintenance. It is active (last release Feb 2026, last push Oct 2026), but it has a single maintainer and a small star count.
  • Alternative rejected. networknt json-schema-validator, the popular choice, pulls in jackson-databind, jackson-dataformat-yaml and itu for an SDK that otherwise uses Gson.

pom.xml is code-owned by @opentdf/security, so this needs their review.

Memory: please weigh this. The manifest is materialized, as it is in Go (gojsonschema) and in web-sdk. Measured on JDK 21:

Manifest Size Heap needed to validate Time
100K segments 9.3 MB ~200 MB ~0.2 s
1M segments (≈2 TiB payload at 2 MiB) 93 MB ~2 GB ~1 s
3M empty {} segment objects (adversarial) 9 MB 1–2 GB ~0.6 s

That is roughly 20× for real manifests and about 150× for one built to inflate. Gson's tree is about 7×; harrel's node wrappers and per-node annotations make up the rest. The Javadoc tells callers screening untrusted files to cap the input size. Once #424 lands, its deflate budget lets a ~10 KB archive declare a 10 MiB manifest, so this matters more there. A streaming validator for this six-keyword subset would make memory O(depth) and drop the dependency, at the cost of hand-rolling JSON Schema semantics. I chose a maintained, spec-tested library over hand-rolled schema semantics. A streaming validator is a reasonable follow-up if reviewers prefer its memory profile.

SDK.isTDF: kept, not deprecated

isTDF answers a different, cheaper question: does this look like a ZTDF, from the central directory alone, with a yes/no contract that never throws. That suits format sniffing and routing. Deprecating it would push those callers into a full manifest parse and schema check, and #418 has only just aligned it with the reader. Its Javadoc now says it never reads the manifest and points to isValidTdf for that. The two checks nest: isValidTdf(x) implies isTDF(x), but not the reverse.

cmdline: tdf validate <file>

This is the shared contract for the xtest suite (DSPX-5349) and the Go/JS CLIs being built in parallel:

  • stdout: one JSON line, {"valid":true} or {"valid":false,"reason":"..."}.
  • Exit codes: 0 valid; 3 invalid (the check completed); any other nonzero means the check could not run. That covers a missing or unreadable file (1), usage errors (2), and unexpected exceptions (1). Nothing goes to stdout in that case. This replaces the ticket's "exit 1 = invalid", because picocli already uses 1 and 2.
  • No platform options or credentials. It doesn't touch buildSDK().
  • tdf supports tdf_validity now exits 0, so xtest/sdk/java/cli.sh supports tdf_validity can delegate to the jar the way dpop does. I didn't touch opentdf/tests; the DSPX-5349 PR owns cli.sh.
# java -jar cmdline/target/cmdline.jar (shaded jar), exit codes appended
$ tdf validate sample.txt.tdf          # written by this SDK
{"valid":true}                                                        exit=0
$ tdf validate schema-fail.tdf
{"valid":false,"reason":"manifest does not match the lax manifest schema at /payload/isEncrypted: Value is [string] but should be [boolean]"}   exit=3
$ tdf validate bom.tdf
{"valid":false,"reason":"manifest starts with a byte order mark"}     exit=3
$ tdf validate trailing.tdf
{"valid":false,"reason":"manifest has data after its JSON value, starting at line 1 column 115"}   exit=3
$ tdf validate duplicate.tdf
{"valid":false,"reason":"tdf is not a valid zip container: tdf contains more than one entry named 0.manifest.json"}   exit=3
$ tdf validate empty.tdf
{"valid":false,"reason":"tdf is not a valid zip container: Didn't find the end of central directory in the last 0 bytes of this 0 byte archive"}   exit=3
$ tdf validate missing.tdf             # stdout empty; stderr: missing.tdf
exit=1
$ tdf validate                         # stderr: Missing required parameter: '<file>'
exit=2
$ tdf supports tdf_validity
exit=0

Cross-SDK divergences (for the epic's decide-and-align tickets)

Case Go v0.35.0 web-sdk #1104 Java (this PR) Note
Same entry name listed twice last wins last wins invalid Matches Java's reader (#418) and DSPX-4924, which wants rejection: two readers picking different copies read different policies
No 0.payload entry valid valid invalid Matches Java's reader and isTDF. Corpus authors: include a 0.payload in schema cases, or Java reports "no payload" for every one of them
Data after the manifest's JSON value valid invalid invalid Strict parse, as in web-sdk. Go's decoder reads one value and ignores the rest
BOM before the manifest invalid invalid invalid
Nesting > 10,000 invalid valid (at 20,000) invalid Same limit as Go, which this matches exactly (10,000 accepted, 10,001 refused)
Repeated JSON member name last wins last wins last wins Pinned by a test
Invalid UTF-8 inside a string U+FFFD not checked U+FFFD Pinned by a test
Number of >10,000 digits, or exponent ≥10,000 valid not checked invalid Gson refuses to convert it; reported as invalid, never thrown
mlkem-wrapped key access invalid invalid invalid The schema's enum lacks it (DSPX-4388), so Java's own ML-KEM output (sdk-pqc-bc) is "invalid". Pinned by a test that should flip when the schema is refreshed
ZIP archive comment invalid valid valid Java's reader tolerates comments; pinned by a test

The deflate question (not resolved here)

TDF writers emit STORED entries. Go is moving to reject non-STORED entries in IsValidTdf (DSPX-5354, per the DSPX-4924 profile), while #424 makes Java's reader accept method 8. Because this check opens archives through TDFReader, it inherits whatever the reader does:

  • On this base, ZipReader ignores the method. Real deflated bytes are read as if stored and fail as JSON (invalid). Method 8 declared over stored bytes reads fine (valid, like Go v0.35.0 today).
  • After feat(sdk): read deflate-compressed ZIP entries in TDF archives #424, deflated entries validate. Unsupported methods, corrupt deflate data and over-budget sizes fail as invalid; the InvalidZipException path in TDFValidator is written for that.

If the epic decides validity should require STORED, as Go is heading, that needs an explicit method check here, independent of what the reader can inflate. I'm flagging it here rather than picking a side in this PR.

#405 (opt-in manifest.json writer) needs no change here; both names are read. It and this PR both touch Command.java/CommandTest.java, in separate places.

How to test

export JAVA_HOME="$(brew --prefix openjdk@21)/libexec/openjdk.jdk/Contents/Home"
mvn --batch-mode -pl sdk,cmdline -am test
mvn --batch-mode -pl cmdline -am package -DskipTests && java -jar cmdline/target/cmdline.jar validate <file>

Results (Homebrew OpenJDK 21, Maven, macOS):

mvn --batch-mode clean test -pl sdk,cmdline -am     # includes buf codegen
  sdk          Tests run: 476, Failures: 0, Errors: 0, Skipped: 9   (TDFValidatorTest: 105)
  sdk-pqc-bc   Tests run: 28,  Failures: 0, Errors: 0, Skipped: 0
  cmdline      Tests run: 21,  Failures: 0, Errors: 0, Skipped: 0
  BUILD SUCCESS

mvn --batch-mode -pl sdk -am test -Dmaven.antrun.skip -Dtest='Fuzzing' -Dsurefire.failIfNoSpecifiedTests=false
  Fuzzing      Tests run: 46, Failures: 0 (regression replay of every target's seeds, incl. fuzzIsValidTdf)

JAZZER_FUZZ=1 mvn --batch-mode -pl sdk -am test -Dmaven.antrun.skip -Denforcer.skip -Dtest='Fuzzing#fuzzIsValidTdf'
  600 s, ~5.6M executions, cov 288 / ft 799, no findings

The fuzz run also wrote two slow-unit-* inputs. Replayed without the test log4j config's DEBUG logging (which logs every rejected end-of-central-directory candidate), each takes ≤2 ms and is rejected at the zip64 locator. That is logging and instrumentation overhead, not a hang, so they are not committed. Not run: the coverage profile, the FIPS profile, Sonar.

What the tests cover

  • TDFValidatorTest (105 tests).
    • Round trips through this SDK's writer, each asserting the key access type it claims: RSA, EC, two-KAS split, assertions plus system metadata plus MIME type, many segments, and pre-4.3.0 target mode. Also sample.txt.tdf.
    • All 42 lax-schema cases from web-sdk#1104 (lax-manifest-cases.json), exported unchanged from manifest-cases.ts with provenance. They are a snapshot, to be cut to a smoke test once DSPX-5349 owns the corpus.
    • Manifest choice, including no fallback. Duplicate manifest entries under both names in both orders, and a duplicate payload. A missing manifest, a missing payload, an extra entry.
    • BOM under both names, trailing data, lenient-JSON constructs, empty or truncated JSON, the nesting limit at 10,000/10,001, repeated member names, invalid UTF-8, huge numbers.
    • Empty input, non-zip input, truncation at every length, an entry claiming more bytes than it has, an archive comment.
    • All fix(sdk): harden ZipReader central directory parsing against hostile fields (DSPX-5103) #418 hostile central-directory shapes (5 s timeout each), and every checked-in fuzzTDF/fuzzZipRead input. Each gets an answer with no exception and no hang.
    • I/O failures throw: the channel fails, the channel fails inside the manifest, the channel stalls.
    • The schema resource hash.
  • Fuzzing#fuzzIsValidTdf. A new Jazzer target, with seeds, added to the fuzz.yaml matrix. Any exception from an in-memory channel is a finding, including IOException. Surefire's default includes don't select Fuzzing, so on PRs the fuzzTDF/fuzzZipRead corpora are replayed through isValidTdf by TDFValidatorTest.
  • CommandTest. Valid file → {"valid":true} with exit 0. Schema failure, non-zip and empty file → exit 3 with a one-line JSON reason. A missing file → exit 1 and no stdout. No argument → exit 2. supports tdf_validity.

🤖 Generated with Claude Code

SDK.isValidTdf answers the question that Go's IsValidTdf and web-sdk's
isValidTdf answer. SDK.validateTdf makes the same check and reports why a
TDF is not valid, as a MalformedTDFException. A malformed archive or
manifest is invalid; an I/O failure throws.

- open the archive through TDFReader, so the check never calls valid a
  file that Java's own reader refuses: a name listed twice, or a missing
  0.payload, is invalid, and manifest.json is preferred over
  0.manifest.json with no fallback
- parse the manifest strictly: exactly one RFC 8259 value, no byte order
  mark, nothing after the value, nested at most 10,000 deep as in Go
- validate it against manifest-lax.schema.json, vendored unchanged from
  Go SDK v0.35.0, using dev.harrel:json-schema (MIT, no required
  dependencies) over the Gson tree
- keep isTDF, a cheap check of the central directory that never throws,
  and point its Javadoc at the new method instead of deprecating it
- add a fuzzIsValidTdf Jazzer target and run it in the fuzz workflow

Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>
tdf validate <file> prints one line of JSON, {"valid":true} or
{"valid":false,"reason":"..."}. It exits 0 if the file is valid, 3 if it
is not, and with any other code if the check could not run. This is the
contract shared with the Go and JS CLIs for the cross-SDK validity suite
(DSPX-5349); 3 because picocli already uses 1 and 2. It needs no platform
connection or credentials, and tdf supports tdf_validity reports it.

Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>
@coderabbitai

coderabbitai Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant