Repository navigation
docs(auth): link ComputeEngineCredentials Javadoc to bound token section - #14627
Open
macastelaz wants to merge 15 commits into
Open
macastelaz wants to merge 15 commits into
macastelaz wants to merge 15 commits into
Conversation
Co-authored-by: Kirill Logachev <kirl@google.com> Co-authored-by: Min Zhu <zhumin@google.com> Co-authored-by: Lawrence Qiu <lawrenceqiu@google.com> Co-authored-by: yangyzs <171981480+yangyzs@users.noreply.github.com> Co-authored-by: cloud-java-bot <122572305+cloud-java-bot@users.noreply.github.com> Co-authored-by: Keshav Dandeva <keshavdandeva@google.com> Co-authored-by: Mattie Fu <mattiefu@google.com> Co-authored-by: Will Howes <whowes@google.com> Co-authored-by: Jin Seop Kim <jinseop@google.com> Co-authored-by: Jiang Zhu <jiangzzhu@google.com> Co-authored-by: Knut Olav Løite <koloite@gmail.com> Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: rahul2393 <irahul@google.com> Co-authored-by: Sakthivel Subramanian <179120858+sakthivelmanii@users.noreply.github.com> Co-authored-by: Gaole Meng <gaole@google.com>
This PR introduces a feature which enables the auth library to acquire bound access-tokens and bound id-tokens in Agentic Environments. 1) We detect certs in default paths and check if they match the SPIFFE format for agents. 2) If 1. is a yes then we call the MDS endpoint in a POST request with the certificate in the body. Note this PR was based on googleapis#13169 --- ## Manual Testing & End-to-End Verification We verified this feature end-to-end across both a **Live Cloud Run Agent Identity environment** (testing against the live Google Metadata Server, Security Token Service, and Vertex AI with the **Java Agent Development Kit (`com.google.adk:google-adk:1.9.0`)**) and a **10-Scenario Local Mock MDS Simulation Harness** (testing exact HTTP request payloads, true cryptographic certificate/key rotation, combined bundle private-key stripping, well-known directory discovery, non-agent SPIFFE fallback, environment variable precedence, non-atomic rotation retries, and asynchronous container startup polling). ### 1. Live Cloud Run Agent Identity Verification (`<PROJECT_ID>`, `us-central1`) We deployed a containerized Java test application built against this branch (`google-auth-library-oauth2-http:1.50.0-SNAPSHOT` @ `27776f6d62a`) + Java ADK (`com.google.adk:google-adk:1.9.0`) to Cloud Run with Agent Identity enabled (`--functional-type=agent --identity-type=agent-identity`). #### Execution A: Default Bound Token Acquisition (`GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN` unset / default `true`) - **Cloud Run Job Executions**: `agent-bound-token-java-test-cjrgp` (latest run with Step 4D) & `agent-bound-token-java-test-dd9gm` - **Workload Certificate Discovery**: - Detected `/var/run/secrets/workload-spiffe-credentials/credentials.json` - Extracted Leaf Certificate SAN URI: `spiffe://agents.global.org-<ORG_NUMBER>.system.id.goog/resources/run/projects/<PROJECT_NUMBER>/locations/us-central1/jobs/agent-bound-token-java-test` - Leaf Certificate SHA-256 (`x5t#S256`): `5Ggf2ofOIkHkvBAwfku6eKQVI5pYzW50whzv-VwSAM8` - **Bound Access Token (`ComputeEngineCredentials`)**: - Successfully acquired bound OAuth2 Access Token via `POST` to Metadata Server (`ya29.d.c0AZ4bNp...`). - **Bound ID Token (`IdTokenCredentials`) & Cryptographic Binding Verification (RFC 8705 § 3.1)**: - Acquired bound ID Token for target audience `https://example-target-service.run.app`. - Decoded JWT payload inline (`textPayload`) and verified live Google STS embedded the `cnf` (Confirmation) claim containing the SHA-256 thumbprint (`x5t#S256`) of the workload's leaf X.509 certificate: ```json "cnf": { "x5t#S256": "5Ggf2ofOIkHkvBAwfku6eKQVI5pYzW50whzv-VwSAM8" } ``` - Verified an **exact byte-for-byte match** (`[PASS] JWT x5t#S256 thumbprint EXACTLY matches local leaf certificate SHA-256!`). - **Java ADK (`com.google.adk:google-adk:1.9.0`) + `google-genai` & mTLS Proof-of-Possession Verification (Steps 4A, 4B, 4C, 4D)**: - **Step 4A (ADK `HttpClientFactory` Non-mTLS Transport)**: Inspected ADK's shared `OkHttpClient` (`sun.security.ssl.SSLSocketFactoryImpl`, no client cert). Calling Google APIs over this non-mTLS channel with the bound token is rejected at the auth layer with **`HTTP 401 UNAUTHENTICATED`**. - **Step 4B (Live ADK `LlmAgent` + `Gemini` Turn on Vertex AI)**: Invoking `InMemoryRunner.runAsync(...)` with `Gemini` (`gemini-2.5-flash`) fails on turn 1 with `com.google.genai.errors.ClientException: 401 . Request had invalid authentication credentials`, confirming the known limitation where `google-genai` sends bound tokens over a non-mTLS channel. - **Step 4C (True mTLS Contrast Test — Matching Workload Certificate)**: Presenting the **exact same bound access token** over a true mTLS `OkHttpClient` (configured with `/var/run/secrets/workload-spiffe-credentials/certificates.pem` + `private_key.pem`, `x5t#S256 = 5Ggf2ofOIkHkvBAwfku6eKQVI5pYzW50whzv-VwSAM8`) to `https://cloudresourcemanager.mtls.googleapis.com/v1/projects/<PROJECT_ID>` **passes authentication** (`HTTP 403 PERMISSION_DENIED` IAM check instead of `401 UNAUTHENTICATED`), proving Google API Frontend verified the token binding against the TLS client certificate handshake. - **Step 4D (Sender-Constraining / Proof-of-Possession Negative Test — Mismatched Client Certificate)**: Presenting the **exact same bound access token** to `https://cloudresourcemanager.mtls.googleapis.com/v1/projects/<PROJECT_ID>` over an mTLS `OkHttpClient` configured with a **different** X.509 client certificate (`x5t#S256 = -57ZjYVm89oczfdO02Hf3Sz-FaYQIVH1DD3c9zaBIKA`) is **rejected at the auth layer with `HTTP 401 UNAUTHENTICATED`**, confirming that Google API Frontend enforces cryptographic thumbprint matching (`cnf.x5t#S256 == SHA256(TLS client cert)`). #### Execution B: Opt-Out Unbound Token Acquisition (`GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false`) - **Cloud Run Job Execution**: `agent-bound-token-java-test-t7b98` - Updated job environment variable `--set-env-vars="GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false,GOOGLE_CLOUD_LOCATION=global,GOOGLE_GENAI_USE_VERTEXAI=true"`. - Verified `ComputeEngineCredentials` and `IdTokenCredentials` fell back to standard `HTTP GET` requests against MDS and issued standard unbound tokens (decoded JWT payload confirmed absence of the `cnf` claim). - **Java ADK Live Agent Turn (`Step 4B`)**: With `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false`, the unbound token works over `google-genai`'s non-mTLS channel and the live ADK `LlmAgent` + `Gemini` turn **SUCCEEDS**: `[ADK Event] author=bound-token-verify-agent, content=Hello! Yes, as an agent designed to test bound token behavior with the Java ADK, I am expected to receive and process bound tokens.` --- ### 2. Local End-to-End Simulation & Wire Verification (10 Scenarios) To verify internal wire-level, discovery, rotation, environment-variable, and error-handling behavior between the client and MDS, we executed our local simulation suite (`LocalSimulationRunner.java`) spinning up a local Mock MDS (`HttpServer`) across **10 end-to-end scenarios**: 1. **Bound Access Token (`ComputeEngineCredentials` + `GOOGLE_API_CERTIFICATE_CONFIG`)**: Verified `HTTP POST` to `/computeMetadata/v1/instance/service-accounts/default/token?scopes=https://www.googleapis.com/auth/cloud-platform`, verified JSON body `{"certificate_chain": "-----BEGIN CERTIFICATE-----\n..."}` (serialized as a single PEM string), and verified no extra fields are included in the JSON payload. 2. **Bound ID Token (`IdTokenCredentials` + `GOOGLE_API_CERTIFICATE_CONFIG`)**: Verified `HTTP POST` to `/computeMetadata/v1/instance/service-accounts/default/identity?audience=https://target.run.app` (with `audience` passed as a URL query parameter) and JSON body `{"certificate_chain": "-----BEGIN CERTIFICATE-----\n..."}`. 3. **True Live Cryptographic Certificate & Key Rotation on Disk (`Gen-1` $\rightarrow$ `Gen-2`)**: Generated a new X.509 SPIFFE certificate and matching 2048-bit RSA key pair on disk, updated file `mtime`, and verified `AgentIdentityUtils.getAgentIdentityCertInfo()` invalidated its cache, verified the new key pair, and transmitted the rotated `Gen-2` certificate chain (`!req3.body.equals(req1.body)`). 4. **Opt-Out Override (`GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false`)**: Verified setting `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` switches requests back to `HTTP GET` with an empty request body. 5. **Well-Known Directory Combined Bundle (`credentialbundle.pem`) + Private Key Stripping**: Unset `GOOGLE_API_CERTIFICATE_CONFIG`, wrote a combined `credentialbundle.pem` containing **both** `-----BEGIN CERTIFICATE-----` and `-----BEGIN PRIVATE KEY-----` in the well-known directory, and verified that `AgentIdentityUtils` resolved `certPath == keyPath`, verified the key pair, and **stripped the `PRIVATE KEY` block** from the transmitted `POST` payload. 6. **Well-Known Directory Separate Files (`certificates.pem` + `private_key.pem`)**: Unset `GOOGLE_API_CERTIFICATE_CONFIG`, removed `credentialbundle.pem`, placed separate `certificates.pem` and `private_key.pem` in the well-known directory, and verified bound `HTTP POST` acquisition. 7. **Non-Agent SPIFFE Trust Domain Fallback (`shouldRequestBoundToken == false`)**: Configured a valid X.509 certificate with a standard GKE Workload Identity SAN (`spiffe://my-standard-gke-project.svc.id.goog/ns/default/sa/my-ksa`); verified `AgentIdentityUtils` did not throw, cached `shouldRequestBoundToken = false`, and fell back to standard `HTTP GET`. 8. **Environment Variable Precedence & `GOOGLE_API_USE_CLIENT_CERTIFICATE` Matrix**: - **8a**: Legacy `GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES=false` (with `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN` unset) $\rightarrow$ falls back to `HTTP GET`. - **8b**: Modern `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=true` overrides legacy `GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES=false` $\rightarrow$ sends bound `HTTP POST`. - **8c**: `GOOGLE_API_USE_CLIENT_CERTIFICATE=false` with valid agent certs on disk $\rightarrow$ disables bound token (`HTTP GET`) without startup polling. - **8d**: `GOOGLE_API_USE_CLIENT_CERTIFICATE=true` with missing cert files $\rightarrow$ fails closed with `IOException` after retries instead of silently downgrading to an unbound token. 9. **Non-Atomic Rotation / Transient Key-Pair Mismatch Recovery (`CERT_KEY_MATCH_RETRIES`)**: Wrote a new `Gen-3` non-prod SPIFFE certificate (`spiffe://agents-nonprod.global.org-54321.system.id.goog/...`) first while delaying the matching `private_key.pem` update on a background thread; verified `loadAndVerifyCredentials()` retried cleanly via `CERT_KEY_MATCH_RETRIES` and transmitted `Gen-3`. 10. **Asynchronous Container Startup Credential Delivery Polling (`TOTAL_POLL_CYCLES`)**: Started with an empty well-known directory on initial startup (`GOOGLE_API_USE_CLIENT_CERTIFICATE=true`), delivered `certificates.pem` + `private_key.pem` asynchronously from a background thread after ~180ms, and verified the initial `refreshAccessToken()` polled until the files arrived and succeeded with a bound `HTTP POST`. --- ### 3. Reproducible Test Artifacts & Execution Logs (`gpaste` - Internal Corp Access Only) | Artifact / Log | Description | Link | | :--- | :--- | :--- | | **Combined Verification Summary & Logs** | Complete report & console outputs from Live Cloud Run (Bound `cjrgp` & Opt-Out `t7b98`) + Java ADK + Local 10-Scenario Simulation | https://paste.googleplex.com/4671448266964992 | | **`CloudRunAgentVerifyApp.java`** | Live Cloud Run + Java ADK verification app (validates SPIFFE cert, ADC access token, ID token `cnf.x5t#S256` match, and ADK Steps 4A/4B/4C/4D) | https://paste.googleplex.com/5431546950057984 | | **`LocalSimulationRunner.java`** | Self-contained local E2E test harness with Mock MDS (`HttpServer`) testing all 10 scenarios | https://paste.googleplex.com/4541159570014208 | | **`Dockerfile`** | Multi-stage Docker build resolving Java ADK 1.9.0 + `google-genai` and overlaying our local PR JAR | https://paste.googleplex.com/6569302979903488 | | **`deploy_cloud_run_job.sh`** | Turnkey script to build via Cloud Build, deploy Cloud Run Job with `--identity-type=agent-identity`, and execute | https://paste.googleplex.com/4908078735163392 | | **`build_and_run_local.sh`** | Turnkey script to compile and run the local simulation | https://paste.googleplex.com/6449866457350144 | | **Raw Cloud Logging — Bound Run (`cjrgp`)** | Plain-text Cloud Run logs from `gcloud logging read` for default Bound Token execution (with inline decoded JWT & ADK Steps 4A/4B/4C/4D) | https://paste.googleplex.com/5404054864396288 | | **Raw Cloud Logging — Opt-Out Run (`t7b98`)** | Plain-text Cloud Run logs from `gcloud logging read` for `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` execution (showing live ADK Gemini response) | https://paste.googleplex.com/5990735250325504 | #### Quick Reproduction Steps To replicate the live Cloud Run test in any GCP project with Agent Identity enabled: ```bash # 1. Place CloudRunAgentVerifyApp.java, LocalSimulationRunner.java, Dockerfile, and deploy_cloud_run_job.sh # in a directory alongside your local google-cloud-java checkout. chmod +x deploy_cloud_run_job.sh build_and_run_local.sh # 2. Run the local 10-scenario simulation suite: ./build_and_run_local.sh # 3. Deploy and execute the Cloud Run Job with Agent Identity enabled (Default Bound Token mode): ./deploy_cloud_run_job.sh <PROJECT_ID> us-central1 # 4. To test opt-out behavior (unblocks ADK non-mTLS Gemini calls): gcloud alpha run jobs update agent-bound-token-java-test \ --project=<PROJECT_ID> --region=us-central1 \ --set-env-vars="GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false,GOOGLE_CLOUD_LOCATION=global,GOOGLE_GENAI_USE_VERTEXAI=true" gcloud alpha run jobs execute agent-bound-token-java-test \ --project=<PROJECT_ID> --region=us-central1 --wait ``` --------- Co-authored-by: Pranav Iyer <pjiyer@google.com>
…ons (googleapis#13995) ## Description This PR adds transparent retries for mTLS workload certificate rotation to the gRPC and HTTP/JSON transports. When a request fails with `UNAUTHENTICATED` and the workload certificate on disk has changed, the transport is rebuilt with the new certificate and the request is retried once, without interrupting in-flight RPCs or streams. ### 🚀 Core Features & Architectural Updates • Rotation detection: `CertificateRotationTracker` compares the SHA-256 fingerprint of the workload certificate on disk with the certificate the transport was built with. The check only runs after an auth failure (never on the request path), and positive results are cached for at most 1 second so a burst of failures doesn't trigger a burst of file reads. Empty or mid-write files are ignored. • gRPC: `ChannelPool` replaces all of its channels when a rotation is detected. Calls already in flight finish on their old channels, which are shut down once idle. During a rotation refresh, a channel that can't be recreated is dropped rather than kept, so no traffic keeps using the old certificate; a statically sized pool is refilled in the background. • HTTP/JSON: `RefreshingHttpJsonChannel` swaps the channel's `HttpTransport` for one built with the new certificate. Calls already created keep using the transport they started with, so in-flight requests aren't interrupted. • Retries: `AttemptCallable` (unary) and `ServerStreamingAttemptCallable` (server streaming) refresh the transport after an `UNAUTHENTICATED` failure. If the transport moved to a new certificate during or after the attempt, the failure is flagged for a single immediate retry on the refreshed channel; its configured `isRetryable()` value is unchanged. • `ApiResultRetryAlgorithm` gives that retry once, immediately, without using a regular attempt. • If the free retry also fails after another rotation, the call stops instead of falling back to backoff retries. • For server streams, the free retry becomes available again once the stream has made progress, and the stream resumes through the existing resumption strategy. • Client and bidi streams refresh the transport on `UNAUTHENTICATED` but don't retry; the next stream uses the new certificate. • Plumbing: `TransportChannel` gains `shouldRefresh()`, `refresh()` and `getGeneration()`, which default to no-ops. `ApiCallContext.getTransportChannel()` gives retrying callables access to the channel, and `GrpcCallContext` / `HttpJsonCallContext` carry it through `merge()` and `withChannel()`. ### 🔒 System Hardening & Bug Fixes • Outstanding RPC leak (`ChannelPool.ReleasingClientCall`): if a call was cancelled before `start()`, `start()` threw without releasing the channel entry, leaving the channel with a permanently outstanding RPC count so it could never be cleaned up after a refresh. The entry is now released on that path and on cancellation before start. • HTTP/JSON refresh vs. shutdown: `shutdown()` / `shutdownNow()` are serialized with `refresh()` so a transport swap can't race with teardown. ###⚠️ Behavioral & Security Boundary Changes * mTLS is enabled automatically when a workload certificate config is present (`MtlsUtils`, auth library): * When `GOOGLE_API_CERTIFICATE_CONFIG`, or the default gcloud `certificate_config.json`, contains a `workload` certificate, client certificates are used even if `GOOGLE_API_USE_CLIENT_CERTIFICATE` is unset. * `GOOGLE_API_USE_CLIENT_CERTIFICATE=false` still turns mTLS off. * mTLS misconfiguration now fails closed (`MtlsUtils`): * Previously, an invalid explicit config was swallowed and the client silently connected without mTLS. * Now client creation throws `IllegalStateException` in these cases: * the explicit `GOOGLE_API_CERTIFICATE_CONFIG` file is missing, unreadable or malformed; * the default config file exists but is unreadable or malformed; * a configured certificate or key file can't be read. * When mTLS is required but the transport can't load the client certificate, gRPC and HTTP/JSON channel creation fails with an `IOException` instead of falling back to a non-mTLS connection. ### 🧪 Testing #### Automated Testing • Added and updated comprehensive unit-tests reflecting the thread-safety fixes inside ChannelPoolTest.java and RefreshingHttpJsonChannelTest.java. • Corrected edge case test configurations to leverage realistic mocked X.509 certificates to properly exercise deep WorkloadCertificateUtils. getCertificateFingerprint() filesystem caching mechanisms. #### Manual Testing End-to-end tests with real generated GAPIC clients against local mTLS servers, run on this PR's head. Each scenario runs in its own JVM and checks the exact sequence of client certificates and outcomes the server saw. * **Setup** * Clients: `KeyManagementServiceClient` over gRPC and HTTP/JSON, and `GrpcBigQueryReadStub` for server-streaming `ReadRows` with offset-based resumption. * Servers: local gRPC and HTTPS servers that require a client certificate and return `UNAUTHENTICATED` / 401 for "revoked" certificates (by CN). * Rotation: the workload certificate under `GOOGLE_API_CERTIFICATE_CONFIG` is replaced by atomic rename, key first, then cert. * **gRPC unary:** * basic rotation: one rejected attempt, then a free retry on the new cert, and no extra attempts afterwards; * the server rejects every cert: one free retry, then stop; * a second rotation during the free retry: stops after 2 attempts, with no backoff retry; * UNAVAILABLE, then rotation; * corrupt cert on disk: no crash, the call fails, and the client recovers once a valid cert is written; * 20-call bursts across 2 rotations less than 1s apart: all succeed with exactly 1 refresh each; * a 3-channel pool: no stale channel after refresh; * 8 threads across 4 rotations, then `close()` with calls in flight: 70,624/70,624 calls succeeded, no hangs. * **gRPC server streaming:** * rotation before the stream starts; * rotation mid-stream: resumes at the right offset with no gaps or duplicates; * rotation retry re-armed after the stream makes progress, following an earlier UNAVAILABLE retry; * bounded: a second rotation without progress stops the stream, with no third attempt. * **HTTP/JSON:** basic rotation; reject-all; double rotation; a slow call started before rotation completes on the old cert while new calls use the new cert; corrupt cert, then restore; 8-thread stress with 4 rotations, then `close()` (49,561/49,561 calls succeeded, no hangs). The socket factory was verified for both Conscrypt and plain JDK TLS. * **No mTLS** (`GOOGLE_API_USE_CLIENT_CERTIFICATE=false`, or no certificate config): a single attempt with no client cert and no refresh, over both gRPC and HTTP/JSON, even when the cert files on disk change. * **Results:** * JDK 26 (JDK TLS): all 22 scenarios pass. * JDK 21 (Conscrypt): all gRPC, streaming and no-mTLS scenarios pass. * HTTP/JSON with mTLS and Conscrypt on TLS 1.3 fails with an existing `Unknown authType: GENERIC` handshake error, which googleapis#14556 fixes. With googleapis#14556 applied on top of this PR, all 22 scenarios pass on JDK 21 with Conscrypt active. * Harness source: https://paste.googleplex.com/6120902048219136 * Logs: https://paste.googleplex.com/5467233782988800 (this PR on JDK 26 and JDK 21), https://paste.googleplex.com/5563956459077632 (this PR + googleapis#14556 on JDK 21)
…igquery pom (googleapis#14587) ## Why Every PR into `agentic-identities-bound-token` is currently failing CI (for example googleapis#14559 and googleapis#14557, about 35 jobs each) with: ``` [ERROR] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: com.google.cloud:google-cloud-storage:jar -> duplicate declaration of version (?) [ERROR] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: com.google.cloud:google-cloud-datacatalog:jar -> duplicate declaration of version 1.102.0-SNAPSHOT [ERROR] The build could not read 1 project ``` `java-bigquery/google-cloud-bigquery/pom.xml` declares `google-cloud-storage` and `google-cloud-datacatalog` twice in `<dependencies>`. This has been the case for a while, but Maven 3.9 only logged a `[WARNING]`. The GitHub `ubuntu-24.04` runner image `20261004` upgraded Maven from 3.9.16 to 3.10.0, which treats it as an error, so any job that loads the reactor now fails immediately. ## Change Remove the second declaration of each dependency (11 lines, one file): - `google-cloud-storage`: drops the later test-scoped entry. This is the same change as googleapis#14586 on `main`. - `google-cloud-datacatalog`: drops the second, identical test-scoped entry. `main` no longer has this dependency in this pom, so googleapis#14586 doesn't need it. ## Verification - `mvn help:effective-pom` on the module before and after: the effective `<dependencies>` section is identical. `google-cloud-storage` stays test-scoped because the parent's `dependencyManagement` sets `<scope>test</scope>`, and the two datacatalog entries were identical. - Each artifact now appears once in `<dependencies>`; the pom parses as valid XML.
Adds a README section and ComputeEngineCredentials class Javadoc covering: - bound tokens are the default when an agent identity workload certificate is present, and require mTLS with the same certificate - when a bound token is requested (cert discovery, agent SPIFFE trust domain) - opt-out env vars and precedence: GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN over the legacy GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES, and GOOGLE_API_USE_CLIENT_CERTIFICATE=false also disabling binding - the process-wide scope of the opt-out - requesting unbound ID tokens for specific targets with IdTokenProvider.Option.BIND_ID_TOKEN_FALSE - the ADK for Java known limitation and its workaround
…ext (googleapis#14556) > [!IMPORTANT] > **This must merge before the `agentic-identities-bound-token` feature branch is merged to `main`.** Without it, HTTP/JSON clients fail by default wherever mTLS is enabled automatically (see below). ## Problem When mTLS is active, `InstantiatingHttpJsonChannelProvider.configureMtls()` builds a Conscrypt `SSLContext` but initializes it with the JDK (SunJSSE) PKIX `TrustManagerFactory`. On TLS 1.3, Conscrypt passes authType `"GENERIC"` to the trust manager. SunJSSE's end-entity checks reject that authType for server certificates that are CA-issued and carry a KeyUsage extension, which includes Google front ends. So every mTLS HTTP/JSON handshake fails with: ``` javax.net.ssl.SSLHandshakeException: Unknown authType: GENERIC Caused by: java.security.cert.CertificateException: Unknown authType: GENERIC ``` The bug is already on `main`, but there it only triggers when `GOOGLE_API_USE_CLIENT_CERTIFICATE=true` is set explicitly. On this feature branch, googleapis#13995 enables mTLS automatically whenever a workload certificate config is present, so on Cloud Run (agent identity) **every HTTP/JSON client fails by default**. gRPC isn't affected. The non-mTLS path isn't affected either, because google-http-client already pairs Conscrypt with a provider-matched trust manager there. A second provider mismatch shows up on JDK 26: the mTLS `SSLContext` also used the JDK's default (`SunX509`) `KeyManagerFactory`. Since JDK 26 ([JDK-8359956](https://bugs.openjdk.org/browse/JDK-8359956)), that key manager applies algorithm constraints to the client certificate. Called from a Conscrypt handshake, it rejects valid certificates (e.g. `SHA256withRSA`), so the client sends an empty certificate chain and the server aborts with `certificate_required`. ## Fix Use Conscrypt's own PKIX `TrustManagerFactory` (`TrustManagerFactory.getInstance("PKIX", conscryptProvider)`) for the mTLS `SSLContext`. I called the JDK API directly rather than `SslUtils.getPkixTrustManagerFactory(Provider)`, which only exists in google-http-client 2.2.0+. Likewise, use Conscrypt's own `KeyManagerFactory` (`KeyManagerFactory.getInstance("PKIX", conscryptProvider)`), so the trust manager and key manager both come from the same provider as the `SSLContext`. Conscrypt's trust manager loads the same default trust store as the JDK. Verified on JDK 21: 174 anchors in both, the same set, and both honor `-Djavax.net.ssl.trustStore` overrides identically. ## Testing - New `InstantiatingHttpJsonChannelProviderTls13Test`: a local JDK TLS 1.3 server presents a CA-issued leaf with KeyUsage and requires a client certificate. The test asserts that the transport completes the request and presents its client certificate. - Without the fix: fails with `Unknown authType: GENERIC`. - With the fix: passes. - Skips when Conscrypt native or TLS 1.3 is unavailable. - Full `gax-httpjson` suite on the current `agentic-identities-bound-token` (with googleapis#13995 merged): 194/194 pass on JDK 8, 11, 17, 21, 25 and 26. On JDK 26, the Tls13 test fails without the `KeyManagerFactory` change: the client sends an empty certificate chain and times out. - End-to-end with googleapis#13995's certificate-rotation support, on JDK 21 with Conscrypt active: 22 scenarios with real KMS and BigQuery Storage GAPIC clients against local mTLS servers, where the certificate rotates on disk. - With this fix: all 22 pass. That includes HTTP/JSON rotation, in-flight calls during refresh, and stress with `close()`. The refreshed transport still uses Conscrypt's socket factory. - Without this fix: every HTTP/JSON mTLS scenario fails with `Unknown authType: GENERIC`. Limiting the test server to TLS 1.2 makes the error go away, which confirms that the TLS 1.3 authType is the trigger. - Logs: https://paste.googleplex.com/5563956459077632 (with fix), https://paste.googleplex.com/5467233782988800 (without fix, JDK 21 set) - Live on Cloud Run (agent identity), combined googleapis#13873 + googleapis#13995 build, default env: | | gRPC | HTTP/JSON | |---|---|---| | Before | ✅ | ❌ `Unknown authType: GENERIC` | | After | ✅ | ✅ authenticated over mTLS with a bound token |
…te-bound ID tokens (googleapis#14559) > [!IMPORTANT] > **This must merge before the `agentic-identities-bound-token` feature branch is merged to `main`.** Docs PR googleapis#14557 documents this option. ## Why When an agent identity certificate is available, `ComputeEngineCredentials` requests certificate-bound ID tokens by default. A bound ID token is only accepted by targets that authenticate the caller over mTLS with the same certificate. Targets reached over standard HTTPS, such as a Cloud Run service's `*.run.app` URL or a custom domain, reject it with `401 Unauthorized`. Per the [ID token binding 1-pager](https://docs.google.com/document/d/1yL_xdDimlUlUyid6HDtqZUIMQjX9VznwzXn1SBABjZc/edit?pli=1&resourcekey=0-GxSXGdfH5Ni_nSI9f7xkQg&tab=t.0#heading=h.94fnt3dk2gx) (Sep 30 decision), ID tokens stay bound by default, and callers get a per-target `bind_id_token` setting to turn binding off where needed. The process-wide opt-out (`GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false`) is unchanged. ## What changes - New `IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN`. Callers pass it per target, the same way as `FORMAT_FULL` and `LICENSES_TRUE`: ```java IdTokenCredentials.newBuilder() .setIdTokenProvider((IdTokenProvider) credentials) .setTargetAudience("https://my-service-12345.us-central1.run.app") .setOptions(Arrays.asList(IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN)) .build(); ``` - `ComputeEngineCredentials.idTokenWithAudience()`: with the option, the certificate lookup is skipped and the identity endpoint is called with a plain GET. Without it, behavior is unchanged (bound whenever a certificate is available and binding isn't turned off by the env var). - Other credential types (`ServiceAccountCredentials`, `ImpersonatedCredentials`, `UserCredentials`) already ignore options they don't use, so the same code works outside agent identity environments. - Access tokens are unaffected. ### API shape `Option` is a set of on/off flags rather than a value, so this adds only the "off" flag (`DISABLE_BOUND_ID_TOKEN`). Leaving it unset means "auto", which currently means bound. This matches the 1-pager's tri-state (unset = auto) without adding an "enable" flag that would do nothing today; one can be added later without breaking callers if "auto" starts deciding per target. Python exposes the same control as `bind_id_token: Optional[bool]` (googleapis/google-cloud-python#18559). ## Tests ### Unit tests New `ComputeEngineCredentialsTest` cases, all with an agent identity certificate present and binding enabled: - `idTokenWithAudience_disableBoundIdToken_requestsUnboundToken`: GET, no request body, no forced `format=full`. - `idTokenWithAudience_disableBoundIdTokenWithFormatFull_requestsUnboundFullToken`: still GET; `format=full` is kept when the caller asks for it. - `idTokenWithAudience_disableBoundIdToken_skipsCertificateLookup`: with a certificate config pointing to a missing file, the default request fails, but the opted-out request succeeds, which shows the certificate is never read. - `idTokenCredentials_withDisableBoundIdTokenOption_requestsUnboundToken`: the option set on `IdTokenCredentials` reaches `ComputeEngineCredentials`. The existing bound-ID-token tests are unchanged and still pass. Full `oauth2_http` suite: 1064 tests, 0 failures. `fmt` is clean. ### Live test on GKE Run on a GKE cluster with agent identity enabled, using a test-only local merge of this PR with googleapis#14595 (GKE support) and googleapis#14556 (already on the base branch). A single process fetches ID tokens through `IdTokenCredentials` from `ComputeEngineCredentials` (ADC) and sends them to an agent-to-agent receiver pod. The receiver accepts only bound tokens over mTLS (whose `cnf` must match the client certificate), and only unbound tokens over plain HTTP. | Check | Defaults | `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` | |---|---|---| | ID token with `DISABLE_BOUND_ID_TOKEN` | **unbound** (no `cnf`); receiver: plain HTTP 200, mTLS 401 | unbound | | ID token for another target without the option, same credentials, fetched afterwards | **still bound** (`cnf` = certificate leaf); receiver: mTLS 200 | unbound | | Access token after the ID-token opt-out | **still bound** (401 on a plain, non-mTLS endpoint) | unbound | The option unbinds only the ID token it's set on. Other targets and access tokens are unaffected, and it's harmless when binding is already off. **Internal links (Googlers only):** [results](https://paste.googleplex.com/5926225961418752) · [harness](https://paste.googleplex.com/5713819712749568) · cluster [setup](https://paste.googleplex.com/5635555912712192) and [manifests](https://paste.googleplex.com/6258309762514944) (shared with googleapis#14595)
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request updates the Javadoc for the ComputeEngineCredentials class to document that certificate-bound access tokens and ID tokens are requested by default when a workload certificate for an agent identity is available. It also adds references to IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN and the Google Auth Library guide. I have no feedback to provide.
…oogleapis#14557) > [!IMPORTANT] > **Merge after googleapis#14559**, which adds `IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN` referenced here. Both must merge before the `agentic-identities-bound-token` feature branch is merged to `main`. ## What this adds Docs only; no code changes. - **`ComputeEngineCredentials` class Javadoc**: adds a brief summary noting that when a workload certificate for an agent identity is available, `ComputeEngineCredentials` requests certificate-bound access tokens and ID tokens by default, with pointers to `IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN` and the [Google Auth Library guide](https://cloud.google.com/java/getting-started/getting-started-with-google-auth-library). - The full "Certificate-bound tokens for agent identities" user guide section (covering environment variables, per-target ID token opt-out, and the ADK for Java limitation) is being added to the DevSite `getting-started-with-google-auth-library` page (`cl/996749889`) alongside the Cloud Run documentation (`cl/990396526`). ## Release notes `google-auth-library-java/CHANGELOG.md` is generated, so release notes come from commit messages. Suggested text for the `BEGIN_COMMIT_OVERRIDE` block of the feature-branch → `main` merge PR: ``` feat(auth): request certificate-bound tokens for agent identities by default. Set GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false to opt out. feat(auth): add IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN to request unbound ID tokens for specific targets. docs(auth): known limitation: ADK for Java doesn't support mTLS yet; agents that use it must set GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false. ``` ## Testing - `google-java-format` and `javadoc:javadoc` pass.
macastelaz
changed the base branch from
agentic-identities-bound-token
to
main
October 19, 2026 19:13
lsirac
approved these changes
Oct 20, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Merge after
cl/996749889publishes on DevSite (or retarget tomainifagentic-identities-bound-tokenmerges first).Summary
Follow-up to #14557 (comment): updates the
ComputeEngineCredentialsclass Javadoc link to point directly to the new#certificate-bound-tokens-for-agent-identitiessection anchor introduced incl/996749889: