Repository navigation
Conversation
… OOM Updating a very large metadata datasource (~100k+ entries) caused OutOfMemoryError because the response, which echoes the whole datasource, was buffered as a String and then materialized as two object trees. - Add opt-in `skip_response_body` option: the body is drained, not parsed, and an empty response is returned (status and rate-limit headers kept). - Add `updateMetadataFieldDatasource` and `restoreDatasourceEntries` overloads that accept options. Existing signatures are unchanged. - Parse successful responses directly from the stream (UTF-8) instead of buffering a String, and close the response. - Network failures while reading the body still raise GeneralError. - A non-200 response without an `error` key now maps to the proper exception instead of an NPE. - Add offline tests against a local HTTP server and a `memoryTest` Gradle task running them under a 256m heap. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Updating a very large metadata datasource (~100k+ entries, e.g. Best Buy's
upcfield, SNI-8550) withupdateMetadataFieldDatasourcecausedOutOfMemoryError. The API echoes the whole datasource in the response, and the SDK held it in memory several times over: body String ->JSONObjecttree ->Maptree. A plain HTTP client that ignores the body does not have this problem.Changes
skip_response_body: on a 200 the body is drained, not parsed, and an empty response is returned. Status and rate-limit headers are still available, and errors still throw as before.Api.updateMetadataFieldDatasource(id, entries, options)andApi.restoreDatasourceEntries(id, ids, options). The existing signatures are unchanged and delegate to them.GeneralError, as before.errorkey now maps to the proper exception instead of an NPE.Compatibility
No public signature changes. The only behavior difference is the non-200-without-
error-key case above. The default path still builds a full object tree, so very large datasources should useskip_response_body. A backendno_entries_in_responseparam would be the full fix and is out of scope.Tests
DatasourceLargeResponseTest(offline, local JDK HttpServer): large response, large request, skip option, UTF-8 round trip, error mapping, missingerrorkey, malformed JSON, empty body, non-JSON error body, dropped connection mid-body.-Ddatasource.test.entries). Run under a small heap (e.g.-Xmx256m): on the old code 3 tests fail, including the OOM; on this branch all pass.🤖 Generated with Claude Code