Skip to content

fix(bigquery-jdbc): limit Storage Read API Arrow batch queue to 32 batches - #14709

Open
logachev wants to merge 1 commit into
mainfrom
kirl/htapi_arrow_buffer_bound
Open

logachev wants to merge 1 commit into
mainfrom
kirl/htapi_arrow_buffer_bound

Conversation

@logachev

Copy link
Copy Markdown
Contributor

We were using the same buffer size constant for Arrow and Json data retrieval. The problem was that while for JSON buffer represented rows, for Arrow it represented batches.

As a result, when using ReadAPI we allowed 20k batches which will be causing OOM or memory pressure issues when consumption is slower than fetching.

b/575989190

@logachev
logachev requested review from a team as code owners October 21, 2026 03:42

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a separate buffer size configuration for Arrow results in BigQueryStatement, replacing the use of getBufferSize() with a new getArrowBufferSize() method initialized to a default capacity of 32. A corresponding unit test was added to verify the bounded queue capacity. The feedback suggests changing the visibility of getArrowBufferSize() to private to improve encapsulation, as it is only used internally.

Comment on lines +1516 to +1518
int getArrowBufferSize() {
return DEFAULT_ARROW_BUFFER_SIZE;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The getArrowBufferSize() method is only used internally within the BigQueryStatement class. To adhere to encapsulation principles and minimize the class's API surface, its visibility should be restricted to private.

Suggested change
int getArrowBufferSize() {
return DEFAULT_ARROW_BUFFER_SIZE;
}
private int getArrowBufferSize() {
return DEFAULT_ARROW_BUFFER_SIZE;
}

@logachev
logachev enabled auto-merge (squash) October 21, 2026 03:44

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant