Repository navigation
feat(sdk): add isValidTdf with Go lax-schema validation, and tdf validate (DSPX-5352) - #428
Draft
dmihalcik-virtru wants to merge 2 commits into
Conversation
SDK.isValidTdf answers the question that Go's IsValidTdf and web-sdk's isValidTdf answer. SDK.validateTdf makes the same check and reports why a TDF is not valid, as a MalformedTDFException. A malformed archive or manifest is invalid; an I/O failure throws. - open the archive through TDFReader, so the check never calls valid a file that Java's own reader refuses: a name listed twice, or a missing 0.payload, is invalid, and manifest.json is preferred over 0.manifest.json with no fallback - parse the manifest strictly: exactly one RFC 8259 value, no byte order mark, nothing after the value, nested at most 10,000 deep as in Go - validate it against manifest-lax.schema.json, vendored unchanged from Go SDK v0.35.0, using dev.harrel:json-schema (MIT, no required dependencies) over the Gson tree - keep isTDF, a cheap check of the central directory that never throws, and point its Javadoc at the new method instead of deprecating it - add a fuzzIsValidTdf Jazzer target and run it in the fuzz workflow Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>
tdf validate <file> prints one line of JSON, {"valid":true} or
{"valid":false,"reason":"..."}. It exits 0 if the file is valid, 3 if it
is not, and with any other code if the check could not run. This is the
contract shared with the Go and JS CLIs for the cross-SDK validity suite
(DSPX-5349); 3 because picocli already uses 1 and 2. It needs no platform
connection or credentials, and tdf supports tdf_validity reports it.
Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
DSPX-5352 (epic DSPX-5348, TDF Validity). java-sdk had no equivalent of Go's
sdk.IsValidTdf.SDK.isTDFonly looks at entry names and never reads the manifest. The cross-SDK validity suite (DSPX-5349) needs a real check in Java, and a CLI hook to drive it.Stacked on #418 (DSPX-5103, ZipReader hardening). Without it, hostile central directories can hang the reader or throw unchecked exceptions, so a validity check can't give a reliable answer. Review only the top two commits; the base branch is
DSPX-5103-harden-java-zipreader-against-hostile-central-dire.SDK:
SDK.isValidTdfandSDK.validateTdfThe pair mirrors
attributeExists/validateAttributes.validateTdfgives the reason (the first problem found, on one line), which the CLI prints. Malformed archive or manifest:false/MalformedTDFException. I/O failure:IOException, including a read failure part way through the manifest. No platform connection or credentials are needed.A TDF is valid when all of these hold:
TDFReader.open, the same container checks asSDK.readManifestandloadTDF, so "valid" never names a file that Java's own reader refuses. No entry name may appear twice, and there must be a0.payloadentry.manifest.jsonif present, otherwise0.manifest.json, the same as Go and web-sdk. When the spec name is present but fails, there is no fallback.Strictness.STRICT, no BOM (Gson would skip one silently, so it is checked explicitly), nothing after the value, and nesting at most 10,000 deep, which is Go'sencoding/jsonlimit. Comments, single quotes, unquoted names, trailing commas,NaN,TRUE,\', raw control characters in strings, and the)]}'prefix are all refused.manifest-lax.schema.json, vendored byte for byte from Go SDKsdk/v0.35.0(sdk/schema/manifest-lax.schema.json, SHA-2565e361720…f87d). The source, version and hash are recorded in theTDFValidatorJavadoc. A test pins the hash of the classpath resource, which also catches the sdk module's resource filtering ever altering it.It does not decrypt or verify anything: not the payload, not integrity, not the policy binding, not assertion signatures. The lax schema omits fields that decryption needs, so
truedoes not promiseloadTDFsucceeds. The Javadoc says so.Dependency:
dev.harrel:json-schema1.9.1Nothing in the tree validated JSON Schema. I checked
mvn dependency:tree: Gson, json-smart via nimbus, moshi via connect-kotlin, and no validator. Why this one:optional, so it adds nothing transitively.JsonElementwe already parse strictly, so there is no second JSON parser with different leniency rules.$schema) and is tested against the official JSON Schema test suite through Bowtie. The schema only usestype/properties/required/enum/oneOf/items, but it also has quirks a validator has to get right. For example, thekeyAccessrequiredsits on the array rather than onitems, so it constrains nothing. A spec-compliant library honors such quirks the way gojsonschema does, with no case-by-case reasoning.javap).json-schema-validator, the popular choice, pulls in jackson-databind, jackson-dataformat-yaml and itu for an SDK that otherwise uses Gson.pom.xmlis code-owned by @opentdf/security, so this needs their review.Memory: please weigh this. The manifest is materialized, as it is in Go (gojsonschema) and in web-sdk. Measured on JDK 21:
{}segment objects (adversarial)That is roughly 20× for real manifests and about 150× for one built to inflate. Gson's tree is about 7×; harrel's node wrappers and per-node annotations make up the rest. The Javadoc tells callers screening untrusted files to cap the input size. Once #424 lands, its deflate budget lets a ~10 KB archive declare a 10 MiB manifest, so this matters more there. A streaming validator for this six-keyword subset would make memory O(depth) and drop the dependency, at the cost of hand-rolling JSON Schema semantics. I chose a maintained, spec-tested library over hand-rolled schema semantics. A streaming validator is a reasonable follow-up if reviewers prefer its memory profile.
SDK.isTDF: kept, not deprecatedisTDFanswers a different, cheaper question: does this look like a ZTDF, from the central directory alone, with a yes/no contract that never throws. That suits format sniffing and routing. Deprecating it would push those callers into a full manifest parse and schema check, and #418 has only just aligned it with the reader. Its Javadoc now says it never reads the manifest and points toisValidTdffor that. The two checks nest:isValidTdf(x)impliesisTDF(x), but not the reverse.cmdline:
tdf validate <file>This is the shared contract for the xtest suite (DSPX-5349) and the Go/JS CLIs being built in parallel:
{"valid":true}or{"valid":false,"reason":"..."}.0valid;3invalid (the check completed); any other nonzero means the check could not run. That covers a missing or unreadable file (1), usage errors (2), and unexpected exceptions (1). Nothing goes to stdout in that case. This replaces the ticket's "exit 1 = invalid", because picocli already uses 1 and 2.buildSDK().tdf supports tdf_validitynow exits 0, soxtest/sdk/java/cli.sh supports tdf_validitycan delegate to the jar the waydpopdoes. I didn't touch opentdf/tests; the DSPX-5349 PR ownscli.sh.Cross-SDK divergences (for the epic's decide-and-align tickets)
0.payloadentryisTDF. Corpus authors: include a0.payloadin schema cases, or Java reports "no payload" for every one of themmlkem-wrappedkey accessThe deflate question (not resolved here)
TDF writers emit STORED entries. Go is moving to reject non-STORED entries in
IsValidTdf(DSPX-5354, per the DSPX-4924 profile), while #424 makes Java's reader accept method 8. Because this check opens archives throughTDFReader, it inherits whatever the reader does:ZipReaderignores the method. Real deflated bytes are read as if stored and fail as JSON (invalid). Method 8 declared over stored bytes reads fine (valid, like Go v0.35.0 today).InvalidZipExceptionpath inTDFValidatoris written for that.If the epic decides validity should require STORED, as Go is heading, that needs an explicit method check here, independent of what the reader can inflate. I'm flagging it here rather than picking a side in this PR.
#405 (opt-in
manifest.jsonwriter) needs no change here; both names are read. It and this PR both touchCommand.java/CommandTest.java, in separate places.How to test
Results (Homebrew OpenJDK 21, Maven, macOS):
The fuzz run also wrote two
slow-unit-*inputs. Replayed without the test log4j config's DEBUG logging (which logs every rejected end-of-central-directory candidate), each takes ≤2 ms and is rejected at the zip64 locator. That is logging and instrumentation overhead, not a hang, so they are not committed. Not run: the coverage profile, the FIPS profile, Sonar.What the tests cover
TDFValidatorTest(105 tests).sample.txt.tdf.lax-manifest-cases.json), exported unchanged frommanifest-cases.tswith provenance. They are a snapshot, to be cut to a smoke test once DSPX-5349 owns the corpus.fuzzTDF/fuzzZipReadinput. Each gets an answer with no exception and no hang.Fuzzing#fuzzIsValidTdf. A new Jazzer target, with seeds, added to thefuzz.yamlmatrix. Any exception from an in-memory channel is a finding, includingIOException. Surefire's default includes don't selectFuzzing, so on PRs thefuzzTDF/fuzzZipReadcorpora are replayed throughisValidTdfbyTDFValidatorTest.CommandTest. Valid file →{"valid":true}with exit 0. Schema failure, non-zip and empty file → exit 3 with a one-line JSON reason. A missing file → exit 1 and no stdout. No argument → exit 2.supports tdf_validity.🤖 Generated with Claude Code