Antalya 26.6: Forward auth tokens to data lake catalogs by zvonand · Pull Request #2329 · Altinity/ClickHouse

zvonand · 2026-09-08T16:36:15Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Forward auth tokens to data lake catalogs

CI/CD Options

Exclude tests:

Regression jobs to run:

github-actions · 2026-09-09T06:50:08Z

Workflow [PR], commit [7a09b52]

hpcnt-stewart · 2026-09-10T11:28:52Z

cool

A `DataLakeCatalog` database contacted its catalog as one shared service identity, so the catalog could not see, authorize or audit the human behind a query, and every ClickHouse user got identical catalog and storage access. With `oauth_forward_user_token = 1` the catalog is contacted as the user running the query. It requires the server-level `enable_token_forwarding` setting -- off by default, and re-read on every forwarded request, so turning it off stops already-authenticated sessions from forwarding -- and users who authenticated with a token (`Authorization: Bearer`, or `--jwt` on the native protocol). The token is carried on the session and copied into the query context. There is no service session to fall back to: a query without a token fails with `CATALOG_USER_TOKEN_NOT_AVAILABLE` instead of reaching the catalog as the service principal. How the token is used depends on the catalog type; the setting is rejected for every other type rather than silently ignored: - An Iceberg REST catalog takes bearer tokens directly, optionally after an RFC 8693 exchange at `oauth_token_exchange_uri`, with `oauth_subject_token_type` / `oauth_requested_token_type` describing the token and `oauth_forward_actor_token` sending the service principal's own token as the `actor_token` for delegation. Exchanged session tokens and the credentials the catalog vends are cached per user, bounded by `oauth_user_token_cache_ttl` and `vended_credentials_cache_ttl`. - Glue authenticates with AWS SigV4 and takes no bearer token, so the token goes to AWS STS instead: `AssumeRoleWithWebIdentity` against `aws_role_arn` returns temporary credentials that sign every Glue request and the S3 reads that follow, with the ClickHouse user name as the `RoleSessionName`. The Glue client and its credentials provider are built per user and cached on the token fingerprint, so one user is never handed another's session. `CREATE DATABASE` rejects the combinations that could not be honoured: forwarding without `aws_role_arn`, with static `aws_access_key_id` / `aws_secret_access_key`, or with `oauth_token_exchange_uri`. Rotating credentials with `ALTER DATABASE ... MODIFY SETTING catalog_credential` is carried out as the calling user when forwarding is on, and an `auth_generation` counter scopes every cached token and vended credential to the credentials it was minted under, so a request that authenticated before the ALTER cannot put a rotated-away artifact back into the caches. A lazy `/v1/config` load racing with the same ALTER is dropped rather than republishing the credentials it started with. An asynchronous insert keeps the token verified when it was pushed and flushes under it; the token's fingerprint takes part in the queue key, so inserts authenticated with different tokens never share a batch. Tests: the `test_datalake_token_forwarding`, `test_datalake_glue_token_forwarding` and `test_datalake_sso_lakekeeper` (Keycloak with Lakekeeper) integration suites; `gtest_rest_catalog_token_forwarding` and `gtest_sts_assume_role_with_web_identity` unit tests; the `05027_datalake_token_forwarding_masking`, `05028_datalake_token_forwarding_fail_closed` and `05053_glue_token_forwarding_validation` stateless tests.

Remove commentary that restates the code and keep concise explanations of credential lifetime, publication ordering, and test synchronization. Consolidate exchange and STS assertions, remove duplicate integration cases, and drop checks that do not exercise token expiry or rotation. Retain user isolation, credential-rotation races, async token lifetime, and fail-closed coverage. Related: #2329

Drop the three added stateless tests and their references. Runtime refusal and log-safety checks overlap retained unit and integration coverage, while masking uses existing behavior. The remaining cases restate straightforward settings-validation branches rather than cover distinct regressions. Related: #2329

Reduce the token-forwarding unit suite to five cases covering cache invalidation, failed preparation, and in-flight credential/config races. Remove unused forwarding helpers and simplify shared request recording. Related: #2329

Shorten documentation and comments, remove redundant code, and replace unnecessary integration setup with direct forwarding checks. Pass the query token to `updateMetadata` during Iceberg compaction. Validation: full unit-test build; 19 affected gtests and all 7 remaining integration cases passed. Related: #2329

Simplify request serialization, settings validation, and uncached token exchange. Remove unused state, instrumentation, fixture settings, and the STS rejection gtest already covered by integration testing. Trim repeated documentation and restore unrelated client OAuth encoding to the release-branch implementation. Validation: rebuilt ClickHouse and unit tests; affected gtests and all seven integration cases passed. After removing the redundant gtest, both remaining STS mock tests and both Glue integration tests passed. Related: #2329

The endpoint is checked against `remote_url_allow_hosts`, so forwarded user tokens cannot be sent to arbitrary hosts via `AssumeRoleWithWebIdentity`.

glue.py signs in to Entra with MSAL and passes the access token to clickhouse-connect. ClickHouse verifies it and forwards it to STS AssumeRoleWithWebIdentity for a DataLakeCatalog Glue database (Altinity/ClickHouse#2329), so Glue is called with per-user credentials.

Replace `TokenRequest` with `Poco::URI::QueryParameters`, simplify settings validation, and remove unnecessary test-server ownership wrappers. Validated with 32 catalog, STS, and async-insert unit tests. Related: #2329

zvonand · 2026-09-30T08:21:43Z

@blau-ai

zvonand · 2026-09-30T08:21:50Z

@codex review

chatgpt-codex-connector · 2026-09-30T08:22:08Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-07T11:47:00.034486Z	`fb3889b`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fc378c0429

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Throw `BAD_ARGUMENTS` when `oauth_forward_actor_token = 1` is set without `oauth_forward_user_token`, on a Glue catalog, or on a REST catalog without `oauth_token_exchange_uri`, instead of silently falling back to plain passthrough. #2329 (comment)

zvonand · 2026-10-07T11:28:07Z

@codex review

zvonand · 2026-10-07T11:28:16Z

@blau-ai

zvonand · 2026-10-07T11:31:17Z

Manually tested against "Real" AWS Glue (token exchanged for credentials), with Azure as external OIDC provider

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fb3889b13f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-10-07T11:46:56Z

+    return [this, storage_id, credentials_provider]() -> std::shared_ptr<IStorageCredentials>
    {
        LOG_DEBUG(log, "Refreshing AWS credentials for {} after expired token", storage_id.getNameForLogs());
        auto credentials = credentials_provider->GetAWSCredentials();


Recheck forwarding before refreshing Glue credentials

When a long-running Glue-backed S3 read receives ExpiredToken, this callback invokes the captured web-identity provider without rechecking the hot-reloadable enable_token_forwarding switch. If an administrator disables forwarding after the storage was created, the provider can still send the retained user token to STS and mint new credentials, whereas normal catalog calls stop in validateForwardedToken; recheck the switch when the callback runs and fail without contacting STS when it is disabled.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-10-07T11:46:57Z

    auto [value, _] = metadata_objects.getOrSet(metadata_uri, [&]()
    {
-        auto [object_storage, bucket_name, metadata_path] = createObjectStorageForEarlyTableAccess(metadata_uri, table_metadata);
+        auto [object_storage, bucket_name, metadata_path] = createObjectStorageForEarlyTableAccess(client, metadata_uri, table_metadata);


Partition cached Glue metadata by forwarded identity

In a token-forwarding Glue catalog, the S3 client used to fetch Iceberg metadata is user-specific, but metadata_objects remains keyed only by metadata_uri. If user A warms an entry and user B can access the Glue table but lacks permission to read that S3 metadata object, B receives A's cached parsed object without the lambda—and therefore B's S3 authorization—ever running, exposing metadata-derived schema and snapshot information. Include the forwarded-token identity in this cache key or avoid sharing these entries across users.

Useful? React with 👍 / 👎.

… cache by token The `GlueCatalog` credentials refresh callback now calls `getClient` on each refresh, so it stops contacting STS once `enable_token_forwarding` is disabled. `metadata_objects` is keyed by the forwarded token fingerprint and the metadata URI, so a user never gets Iceberg metadata fetched with another user's credentials. Addresses review comments in #2329

…ature/antalya-26.6/aouth-forward-to-datalake

…ature/antalya-26.6/aouth-forward-to-datalake # Conflicts: # src/Databases/DataLake/Common.h # src/Databases/DataLake/DatabaseDataLake.cpp # src/Databases/DataLake/DatabaseDataLake.h # src/Databases/DataLake/GlueCatalog.cpp # src/Databases/DataLake/GlueCatalog.h # src/Databases/DataLake/HiveCatalog.cpp # src/Databases/DataLake/HiveCatalog.h # src/Databases/DataLake/ICatalog.cpp # src/Databases/DataLake/ICatalog.h # src/Databases/DataLake/PaimonRestCatalog.cpp # src/Databases/DataLake/PaimonRestCatalog.h # src/Databases/DataLake/RestCatalog.cpp # src/Databases/DataLake/RestCatalog.h # src/Databases/DataLake/S3TablesCatalog.cpp # src/Databases/DataLake/S3TablesCatalog.h # src/Databases/DataLake/UnityCatalog.h # src/Interpreters/AsynchronousInsertQueue.h # src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp # src/Storages/ObjectStorage/StorageObjectStorage.cpp

…alogs Audit: #2329 (comment) - M-1: `validateCreateTableEngine` now loads the REST catalog config with the caller's forwarded token before checking the catalog storage type, so the engine-vs-catalog backend check is not skipped on a cold catalog. - M-2: remove `config_mutex`; `/v1/config` is fetched without holding a lock and published under `auth_publish_mutex`. - L-1: `StorageObjectStorage` captures the forwarded token only when it has a catalog. - L-2: clamp `expires_in` before computing token expiry. - L-3: document that catalog and token-exchange endpoints should use HTTPS. - L-5: append a hash of the full principal to the Glue `RoleSessionName` when the principal is not a valid session name as is. - L-6: remove `DataLakeCatalogUserTokenCacheBytes` and `DataLakeCatalogUserClientCacheBytes`, which counted entries, not bytes.

…o-datalake

`AccessControl::setExternalAuthenticatorsConfig` re-reads `enable_token_forwarding` on every config reload, but the setting was not listed in `collectChangeableServerSettings`, so `system.server_settings` reported `changeable_without_restart = No`. #2626

…closing the connection If resolving a table in `TCPHandler::processTablesStatusRequest` throws, the exception is now logged and sent back to the client as `Exception`, keeping the connection open. Before interserver-secret authentication, the exception is still rethrown and the connection is closed.

zvonand added antalya-26.6 port-antalya PRs to be ported to all new Antalya releases labels Sep 8, 2026

zvonand force-pushed the feature/antalya-26.6/aouth-forward-to-datalake branch from 5d6ced1 to 33ba90f Compare September 9, 2026 06:48

zvonand force-pushed the feature/antalya-26.6/aouth-forward-to-datalake branch from 4acc349 to b5f1b30 Compare September 9, 2026 20:59

zvonand force-pushed the feature/antalya-26.6/aouth-forward-to-datalake branch from dadc31f to 37a108b Compare September 21, 2026 12:15

zvonand force-pushed the feature/antalya-26.6/aouth-forward-to-datalake branch from 37a108b to 1072cc3 Compare September 21, 2026 12:21

zvonand added 5 commits September 21, 2026 14:32

zvonand mentioned this pull request Sep 24, 2026

Token-based authentication and authorization (OAuth) tracking #2428

Open

Add aws_sts_endpoint setting for Glue catalog token forwarding

a7cdb88

The endpoint is checked against `remote_url_allow_hosts`, so forwarded user tokens cannot be sent to arbitrary hosts via `AssumeRoleWithWebIdentity`.

Simplify data lake token forwarding and test setup

fc378c0

Replace `TokenRequest` with `Poco::URI::QueryParameters`, simplify settings validation, and remove unnecessary test-server ownership wrappers. Validated with 32 catalog, STS, and async-insert unit tests. Related: #2329

This comment was marked as outdated.

Sign in to view

chatgpt-codex-connector Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread src/Databases/DataLake/RestCatalog.cpp Outdated

Comment thread src/Databases/DataLake/DatabaseDataLake.cpp

svb-alt added the antalya label Oct 1, 2026

This comment was marked as outdated.

Sign in to view

chatgpt-codex-connector Bot reviewed Oct 7, 2026

View reviewed changes

zvonand added 3 commits October 7, 2026 14:58

Merge branch 'antalya-26.6' of github.com:Altinity/ClickHouse into fe…

d2f3549

…ature/antalya-26.6/aouth-forward-to-datalake

This comment was marked as outdated.

Sign in to view

zvonand added 2 commits October 20, 2026 21:41

Merge branch 'antalya-26.6' into feature/antalya-26.6/aouth-forward-t…

0187f60

…o-datalake

This was referenced Oct 21, 2026

A remote node drops the connection instead of reporting the missing token #2625

Open

system.server_settings says enable_token_forwarding needs a restart #2626

Open

zvonand added 2 commits October 21, 2026 18:46

Conversation

zvonand commented Sep 8, 2026

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

github-actions Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hpcnt-stewart commented Sep 10, 2026

Uh oh!

zvonand commented Sep 30, 2026

Uh oh!

zvonand commented Sep 30, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

This comment was marked as outdated.

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

zvonand commented Oct 7, 2026

Uh oh!

zvonand commented Oct 7, 2026

Uh oh!

zvonand commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

github-actions Bot commented Sep 9, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 30, 2026 •

edited

Loading

zvonand commented Oct 7, 2026 •

edited

Loading