fix(auth): present the full certificate chain for mTLS by macastelaz · Pull Request #14621 · googleapis/google-cloud-java

macastelaz · 2026-10-09T20:27:12Z

Why

X509Provider and SecureConnectProvider build their mTLS key store with google-http-client's SecurityUtils.createMtlsKeyStore. That method keeps only the first CERTIFICATE in the PEM input, so Java presents only the leaf certificate in the TLS handshake. This is unchanged in the latest google-http-client release (2.2.0).

Google APIs over *.mtls.googleapis.com are unaffected, because Google's front ends already know the intermediates.
Peers that trust only a root CA can't verify the leaf without the intermediate, so they reject the handshake.
- A common case is agent-to-agent mTLS, where the receiving agent trusts the SPIFFE trust bundle.
- Agent identity certificates on GKE (and Cloud Run) are issued by an intermediate CA, so a Java agent that reuses the SDK's mTLS key store can't call such a peer.
- Python sends the whole chain (load_cert_chain).

This bug predates the agent identity work; the agent identity testing caught it.

What changes

New package-private MtlsKeyStoreUtils.createMtlsKeyStore(InputStream). It replaces the upstream method.

The change: the key entry's chain holds every distinct certificate, in input order. The first certificate is still the leaf.
Same as upstream: key store type (JKS), alias, empty password, accepted key format (PKCS#8 PRIVATE KEY), which private key is chosen when there are several, and error messages.
Also different from upstream: the input is decoded as UTF-8 instead of the platform default charset, and it's read to the end. Upstream stopped once it had a certificate and a key.

Used by:

X509Provider.getKeyStore(), on both the certificate-configuration path (Cloud Run, WIF X.509) and the GKE credential-bundle path;
SecureConnectProvider.getKeyStore() (ECP), which drains the bytes already buffered on the helper's stdout after the process exits so a background child process holding stdout open cannot block waiting for EOF.

Inputs that worked before still work. Because more of the input is now parsed, once a certificate and a private key have been found (the point where upstream stopped reading):

a malformed later PEM section ends the input instead of failing;
an extra certificate that can't be parsed is left out of the chain (and logged at FINE).

Unchanged:

exception wrapping in the providers;
the token side: the cnf binding is still the leaf thumbprint, read from the certificate file;
the public API (no clirr impact).

Known limitations (unchanged from before):

GAX's deprecated com.google.api.gax.rpc.mtls.MtlsProvider (ECP-only) still uses the upstream method. GAX's transports don't use it; they use com.google.auth.mtls.MtlsProvider through DefaultMtlsProviderFactory.
There's no check that the private key matches the first certificate (same as upstream).

Tests

Unit tests

MtlsKeyStoreUtilsTest (new), using a generated test chain (root → intermediate → leaf, EC P-256, long validity) in testresources/mtls/test_chain_{cert,key}.pem:
- upstream SecurityUtils.createMtlsKeyStore keeps only the leaf for the same input. This pins down the bug, and the test fails if upstream ever fixes it;
- leaf + intermediate (and 3-certificate chains) in order, with the certificates first or the key first, and duplicate certificates deduplicated;
- the private key matches the leaf (sign/verify);
- a single certificate gives the same alias, type, certificate and key as SecurityUtils.createMtlsKeyStore;
- non-certificate sections are ignored;
- with several private keys, the same key is chosen as upstream: the last one before the first certificate, or the first one after it. Each of these tests runs upstream on the same input and compares the encoded key bytes;
- a malformed trailing section after a certificate and key is ignored, while a malformed section after only a certificate or only a key throws IllegalArgumentException, and an IOException mid-stream propagates;
- an unparsable or empty extra certificate is left out;
- an unparsable leaf, a missing certificate, or a missing key throws, with the same messages as before.
X509ProviderTest: a GKE bundle with an intermediate (certificates first, and key first as GKE writes it), and a certificate config whose cert_path holds a chain, all give leaf + intermediate.
SecureConnectProviderTest: helper output with a chain gives leaf + intermediate, and only buffered stdout bytes are read after the process exits.

The provider tests fail without the fix (expected: <2> but was: <1>). Full oauth2_http suite passes. fmt is clean.

Live tests

Environment	Check	Before	After
GKE agent identity	Agent-to-agent mTLS using the SDK's `DefaultMtlsProviderFactory` key store, against a receiver that trusts only the SPIFFE root	`Broken pipe` (handshake rejected)	200
GKE agent identity	GCS over `storage.mtls.googleapis.com`, GAX clients over gRPC and HTTP/JSON, bound tokens	pass	pass (unchanged)
Workload identity federation X.509 (STS mTLS)	Token exchange with a leaf-only `cert_path`	success, leaf sent	success, leaf sent
Workload identity federation X.509 (STS mTLS)	Token exchange with a leaf + intermediate `cert_path`	success, leaf only sent	success, leaf + intermediate sent

The certificates sent were confirmed from the TLS handshake debug log. The live tests ran on the first commit (b2876a4d3a9). Subsequent commits only refine how unusual or malformed inputs are handled, plus tests, so well-formed GKE and WIF files behave the same.

Internal links (Googlers only): live test results · GKE setup, manifests and harness (shared with #14595)

…kens and mTLS On GKE, agent identity credentials are delivered as a single combined PEM file (certificate chain + PKCS#8 private key) at /var/run/secrets/workload-spiffe-credentials/x509.credential-bundle.private-key.pem. When no certificate configuration exists (GOOGLE_API_CERTIFICATE_CONFIG unset and no ~/.config/gcloud/certificate_config.json), the presence of a readable, non-empty bundle now: - auto-enables mTLS on the client transport (MtlsUtils.useMtlsClientCertificate / getWorkloadCertPath, and the X509Provider created by DefaultMtlsProviderFactory), and - lets ComputeEngineCredentials request certificate-bound tokens with the same certificate (AgentIdentityUtils). GOOGLE_API_USE_CLIENT_CERTIFICATE=false and GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false still opt out. The bundle is never polled for. The GKE fallback is limited to the transport factory path, so workload identity federation X.509 credentials are unchanged. This mirrors googleapis/google-cloud-python#18559.

…nfig exists

X509Provider and SecureConnectProvider built their key stores with google-http-client's SecurityUtils.createMtlsKeyStore, which keeps only the first certificate. Java therefore presented only the leaf during the TLS handshake. Google front ends accept that, but peers that trust only the root (for example, agent-to-agent mTLS with a SPIFFE trust bundle) cannot verify the leaf without the intermediate and reject the handshake. Build the key store with every certificate in the PEM input, keeping the first certificate as the leaf, as before.

gemini-code-assist

Code Review

This pull request introduces support for GKE credential bundles (Pod Certificates) as a fallback client certificate source for mTLS when no explicit certificate configuration is present. It adds MtlsKeyStoreUtils to build a KeyStore that preserves the full certificate chain, and updates MtlsUtils, X509Provider, and AgentIdentityUtils to handle the GKE credential bundle path and establish mTLS intent accordingly. The review feedback suggests two improvements in MtlsKeyStoreUtils: normalizing the "ECDSA" public key algorithm name to "EC" to prevent NoSuchAlgorithmException on standard JDKs, and explicitly calling keyStore.load(null, null) instead of keyStore.load(null) to avoid relying on implicit JDK fallback behavior.

@throws

- Choose the private key the same way as SecurityUtils.createMtlsKeyStore: a later key replaces an earlier one until the first certificate is seen. - Once a certificate and key are found, also treat a read error as the end of input, and leave out an extra certificate that fails with a runtime exception, so inputs the upstream method accepted keep working. - Javadoc: describe the differences from upstream and complete @throws; X509Provider.getKeyStore now returns the certificate chain. - Tests: compare key selection with upstream for keys before and after the certificates, pin that upstream keeps only the leaf, cover a key-first GKE bundle with an intermediate, and make a test independent of line endings.

…into HEAD

…full-cert-chain

…ll-cert-chain

lsirac · 2026-10-15T16:53:29Z


    // Create mTLS key store with the input certificates from shell command.
-    return SecurityUtils.createMtlsKeyStore(process.getInputStream());
+    return MtlsKeyStoreUtils.createMtlsKeyStore(process.getInputStream());


Reading stdout to EOF after waitFor returns can hang getKeyStore without bound if a helper leaves a child process holding stdout open, whereas upstream stopped after the first cert and key. Can we read only the buffered bytes here, or move this item out of the unchanged limitations section in the PR description?

Done in 8245ef3d657 — drained only the available() bytes buffered on process.getInputStream() after waitFor returns (and removed the corresponding limitation note from the PR description). Added testGetKeyStore_stdoutHeldOpenAfterBufferedBytes_doesNotReadToEof in SecureConnectProviderTest.

lsirac · 2026-10-15T16:53:30Z

+      PemReader.Section section;
+      try {
+        section = reader.readNextSection();
+      } catch (IllegalArgumentException | IOException e) {


Catching IOException here once a key and leaf are read swallows real mid-stream read errors on key-first bundles like GKE's and returns a leaf-only chain. Can we catch only IllegalArgumentException here so genuine I/O failures still surface?

Done in 8245ef3d657 — narrowed the catch block to IllegalArgumentException (and updated the Javadoc accordingly) so mid-stream IOExceptions propagate.

lsirac · 2026-10-15T16:53:30Z

+
+  private static X509Certificate parseCertificate(CertificateFactory certFactory, byte[] der)
+      throws CertificateException {
+    return (X509Certificate) certFactory.generateCertificate(new ByteArrayInputStream(der));


Should we null-check certFactory.generateCertificate here? BouncyCastle returns null for an empty CERTIFICATE block instead of throwing CertificateException, which puts null into chain and throws an NPE in KeyManagerFactory.init.

Done in 8245ef3d657 — added a null check in parseCertificate that throws CertificateException("Empty certificate"), and added an empty CERTIFICATE block to createMtlsKeyStore_unparsableExtraCertificate_leftOut.

lsirac · 2026-10-15T16:53:30Z

+  }
+
+  @Test
+  void createMtlsKeyStore_malformedTrailingSection_ignored() throws Exception {


This is the only test that reaches the read-loop catch, and dropping IOException or either half of the && check still passes the suite. Can we add cases for an IOException after the cert and key, and for a malformed section after only one of them?

Done in 8245ef3d657 — added createMtlsKeyStore_malformedSectionAfterOnlyCertificate_throws, createMtlsKeyStore_malformedSectionAfterOnlyKey_throws, and createMtlsKeyStore_ioExceptionAfterCertAndKey_throws (asserting that an IOException after the cert and key propagates now that only IllegalArgumentException is caught).

lsirac · 2026-10-15T16:53:30Z

+    KeyStore keyStore =
+        MtlsKeyStoreUtils.createMtlsKeyStore(
+            stream(
+                read(CHAIN_CERT_PATH), read(CHAIN_KEY_PATH), "-----BEGIN CERTIFICATE-----\nAAAA"));


Putting both certificates before the key here means this test still passes even if reading stops as soon as a cert and key are seen. Can we order this input as leaf, key, intermediate, and trailing junk so it covers reading a certificate after the key?

Done in 8245ef3d657 — ordered the input in createMtlsKeyStore_malformedTrailingSection_ignored as leaf, key, intermediate, trailing junk and asserted assertLeafThenIntermediate(keyStore).

lsirac · 2026-10-15T16:53:30Z

+    // Documents why MtlsKeyStoreUtils exists. If google-http-client starts keeping the full chain,
+    // this fails and the helper can be replaced by the upstream method again.
+    KeyStore keyStore =
+        com.google.api.client.util.SecurityUtils.createMtlsKeyStore(


Nit: Can we import com.google.api.client.util.SecurityUtils rather than repeating the fully qualified name five times?

Done in 8245ef3d657 — imported SecurityUtils and replaced the fully qualified references.

lsirac · 2026-10-15T16:53:30Z

+    KeyStore keyStore = MtlsKeyStoreUtils.createMtlsKeyStore(stream(input));
+    KeyStore upstream = com.google.api.client.util.SecurityUtils.createMtlsKeyStore(stream(input));
+
+    assertEquals("EC", keyStore.getKey(onlyAlias(keyStore), new char[] {}).getAlgorithm());


Nit: Since createMtlsKeyStore always builds the key with the leaf certificate's algorithm, getAlgorithm is guaranteed to return EC once the call succeeds. Comparing getEncoded with the expected key bytes would pin which key was selected.

Done in 8245ef3d657 — replaced the getAlgorithm() assertions with assertArrayEquals comparing getKey(...).getEncoded() against the expected key bytes in both createMtlsKeyStore_keysBeforeCertificate_usesLastKeyLikeUpstream and createMtlsKeyStore_keysAfterCertificate_usesFirstKeyLikeUpstream.

lsirac · 2026-10-15T16:53:30Z

+        String.join("\n", read(CHAIN_KEY_PATH), read(SINGLE_KEY_PATH), read(CHAIN_CERT_PATH));
+
+    assertThrows(
+        GeneralSecurityException.class, () -> MtlsKeyStoreUtils.createMtlsKeyStore(stream(input)));


Nit: Asserting InvalidKeySpecException.class instead of GeneralSecurityException.class here would pin that the failure comes from the mismatched key spec rather than certificate parsing.

Done in 8245ef3d657 — asserted InvalidKeySpecException.class in createMtlsKeyStore_keysBeforeCertificate_lastKeyDoesNotMatch_throwsLikeUpstream.

lsirac · 2026-10-15T16:53:30Z

+  void x509Provider_gkeCredentialBundleWithIntermediate_keepsFullChain(@TempDir Path tempDir)
+      throws Exception {
+    Path bundle = tempDir.resolve("x509.credential-bundle.private-key.pem");
+    Files.write(


Nit: This and the next test repeat the GKE bundle setup from writeGkeBundleAt. Can we add an overload taking the cert and key paths so both tests reuse it?

Done in 8245ef3d657 — added writeGkeBundle / writeGkeBundleAt overloads taking (certPath, keyPath, keyFirst) and reused them in both intermediate GKE bundle tests.

lsirac · 2026-10-15T16:53:30Z

+
+    KeyStore store =
+        SecureConnectProvider.getKeyStore(
+            new ByteArrayInputStream("{}".getBytes(StandardCharsets.UTF_8)), processProvider);


Nit: TestUtils.stringToInputStream already converts a string to a UTF-8 InputStream, which would let us drop the ByteArrayInputStream and StandardCharsets imports here.

Done in 8245ef3d657 — used TestUtils.stringToInputStream and removed the ByteArrayInputStream and StandardCharsets imports.

…r agent identities (#14668) ## Summary Merges the `agentic-identities-bound-token` feature branch into `main`, adding end-to-end support for certificate-bound tokens and automatic mTLS with transparent certificate rotation for agent identities on Cloud Run and GKE. ### `google-auth-library-java` (`oauth2_http`) - **Default certificate-bound tokens (`#13873`, `#14595`, `#14658`):** When an agent identity SPIFFE workload certificate is present (either via `GOOGLE_API_CERTIFICATE_CONFIG` on Cloud Run or the GKE credential bundle at `/var/run/secrets/workload-spiffe-credentials/x509.credential-bundle.private-key.pem`), `ComputeEngineCredentials` requests certificate-bound access tokens and ID tokens via `POST` to the Metadata Server with the leaf certificate chain. - **Opt-out controls (`#13873`, `#14559`, `#14557`):** - Process-wide: `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` (or legacy `GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES=false`) disables token binding. - Per-target ID tokens: `IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN` requests an unbound ID token for specific targets (such as plain HTTPS `*.run.app` URLs) while keeping other ID tokens and access tokens bound. - **Automatic mTLS & strict configuration (`#13995`, `#14595`, `#14658`):** - `MtlsUtils.useMtlsClientCertificate()` enables mTLS automatically when a workload certificate config or GKE credential bundle is present, unless `GOOGLE_API_USE_CLIENT_CERTIFICATE` is set to a non-`"true"` value. - Missing, unreadable, or malformed explicit `GOOGLE_API_CERTIFICATE_CONFIG` files fail closed with `IllegalStateException`. - **Full mTLS certificate chain (`#14621`):** `X509Provider` and `SecureConnectProvider` now populate the mTLS key store with the full certificate chain (leaf + intermediates) via `MtlsKeyStoreUtils.createMtlsKeyStore` rather than only the leaf certificate. ### `sdk-platform-java/gax-java` (`gax`, `gax-grpc`, `gax-httpjson`) - **Transparent retries on mTLS certificate rotation (`#13995`):** - `CertificateRotationTracker` checks the SHA-256 fingerprint of the workload certificate on disk after an `UNAUTHENTICATED` failure. - `ChannelPool` (gRPC) and `RefreshingHttpJsonChannel` (HTTP/JSON) rebuild transport channels with the rotated certificate without interrupting in-flight RPCs. - `AttemptCallable` (unary) and `ServerStreamingAttemptCallable` (server streaming) issue one immediate free retry on the refreshed channel when a certificate rotation is detected. - **Fail-closed default mTLS provider loading (`#14658`):** When `useMtlsClientCertificate()` is `true`, `EndpointContext`, `InstantiatingGrpcChannelProvider`, and `InstantiatingHttpJsonChannelProvider` propagate `IOException`s (while still falling back cleanly on `CertificateSourceUnavailableException` when no certificate source exists on disk). - **Conscrypt TLS 1.3 mTLS fix (`#14556`):** `InstantiatingHttpJsonChannelProvider` uses Conscrypt's own `TrustManagerFactory` and `KeyManagerFactory` for the mTLS `SSLContext`, fixing `Unknown authType: GENERIC` on TLS 1.3 and key-manager algorithm constraint mismatches on JDK 26+. ## Sub-PRs Included on `agentic-identities-bound-token` | PR | Description | | :--- | :--- | | `#13873` | `feat: Enable Bound Token for Agentic Identities` | | `#13995` | `feat(gax): support transparent retries during mTLS certificate rotations` | | `#14556` | `fix(gax-httpjson): use Conscrypt TrustManagerFactory for mTLS SSLContext` | | `#14559` | `feat(auth): add DISABLE_BOUND_ID_TOKEN option to opt out of certificate-bound ID tokens` | | `#14557` | `docs(auth): document certificate-bound tokens for agent identities` | | `#14595` | `feat(auth): support GKE agent identity credential bundle for bound tokens and mTLS` | | `#14626` | `fix(gax): drop jsr305 annotations left over after syncing with main` | | `#14621` | `fix(auth): present the full certificate chain for mTLS` | | `#14658` | `fix(auth,gax): align Java mTLS and agent identity certificate handling with Python parity` | ## Testing - **Automated tests:** Full unit test suites in `google-auth-library-java` (`credentials`, `oauth2_http`) and `sdk-platform-java/gax-java` (`gax`, `gax-grpc`, `gax-httpjson`) pass across JDK 8, 11, 17, 21, 25, and 26, including new unit tests for certificate discovery, bound token requests, startup polling, rotation retries, Conscrypt TLS 1.3 mTLS handshakes, and full certificate chains. - **Integrated manual testing (`45/45` scenarios passed):** Built all 9 changed artifacts (`google-auth-library-credentials`, `google-auth-library-oauth2-http`, `api-common`, `gax`, `gax-grpc`, `gax-httpjson`, `google-cloud-core`, `google-cloud-core-http`, `google-cloud-core-grpc`) together from the combined branch and ran 22 live GAPIC mTLS rotation scenarios, 14 local bound-token and GAX parity checks, 3 live Cloud Run Agent Identity job executions, and 6 live GKE Agent Identity pod scenarios. <details> <summary>Full manual test matrix (45 scenarios)</summary> ### 1. Live GAPIC mTLS Rotation (`22/22` Scenarios, `125/125` Checks) Executed real `KeyManagementServiceClient` (gRPC and HTTP/JSON) and `BigQueryReadClient` (server-streaming `ReadRows`) against live `.mtls.googleapis.com` endpoints (`cloudkms.mtls.googleapis.com`, `bigquerystorage.mtls.googleapis.com`) while rotating client certificates on disk. | Group | Scenarios | What Was Verified | Result | | :--- | :--- | :--- | :--- | | **Unary gRPC (`G1`–`G7`)** | `atomic-mv`, `inplace`, `symlink-swap`, `corrupt-recover`, `missing-recover`, `idle-gap`, `burst-25-rpcs` | `KeyManagementServiceClient` gRPC channel seamlessly picks up rotated certs across atomic rename, in-place write, Kubernetes-style symlink swap, corrupt/missing file recovery, idle gaps, and concurrent bursts. | **7/7 Passed** | | **Server-Streaming gRPC (`S1`–`S4`)** | `mid-stream`, `between-streams`, `reconnect-after-rotate`, `corrupt-mid-stream` | `BigQueryReadClient.readRowsCallable()` active streams continue uninterrupted during rotation; subsequent streams and reconnects use the new certificate. | **4/4 Passed** | | **Unary HTTP/JSON (`H1`–`H5`)** | `atomic-mv`, `inplace`, `corrupt-recover`, `missing-recover`, `burst-20-rpcs` | `KeyManagementServiceClient` HTTP/JSON transport reloads rotated certs and recovers from transient corrupt/missing files without dropping requests. | **5/5 Passed** | | **Startup & Providers (`N1a`, `N1b`, `C1`)** | `N1a`, `N1b`, `C1-conscrypt`, `C1-jdk` | Missing cert config handling at startup; full rotation works under both Conscrypt and JDK (`SunJSSE`) TLS providers. | **4/4 Passed** | | **Combined Assertions** | `G1`–`S4` assertions | Verified `125/125` individual assertions across 22 isolated JVMs. | **22/22 Passed** | ### 2. Local Bound-Token & GAX Parity (`14/14` Checks) | Check | PR / Area | What Was Verified | Result | | :--- | :--- | :--- | :--- | | **Scenarios 1–7** | `#13873` / `#14595` | Default bound token (`cnf.x5t#S256` matches leaf), `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false`, `GOOGLE_API_USE_CLIENT_CERTIFICATE=false`, cert rotation updates `x5t#S256`, and `GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES` behavior. | **Passed** | | **Scenario 8d** | `#14658` | `GOOGLE_API_USE_CLIENT_CERTIFICATE=true` with no certs falls back immediately (`43ms` GET) with no 30s polling delay. | **Passed** | | **Scenario 8e** | `#14658` | Invalid explicit `GOOGLE_API_CERTIFICATE_CONFIG` fails closed with `IOException` instead of silently downgrading. | **Passed** | | **Scenario 10** | `#14658` | 0-byte startup cert file is polled until populated rather than failing immediately. | **Passed** | | **Scenario 11** | `#14658` | Kubernetes `..data` symlinked parent directory resolves to the real directory for file watching. | **Passed** | | **Scenario 12a** | `#14559` | `IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN` returns an unbound ID token without leaking state to subsequent default calls. | **Passed** | | **Scenario 12b** | `#14621` | `X509Provider` preserves the full 2-certificate chain (`leaf + intermediate`) in the `KeyStore`. | **Passed** | | **GAX `fail-closed`** | `#14658` | Missing explicit `GOOGLE_API_CERTIFICATE_CONFIG` throws `IllegalStateException` in both `InstantiatingGrpcChannelProvider` and `InstantiatingHttpJsonChannelProvider`. | **Passed** | | **GAX `zero-ms-fallback`** | `#14658` | `GOOGLE_API_USE_CLIENT_CERTIFICATE=true` with no certs builds plain gRPC and HTTP/JSON channels immediately (`0ms` polling delay). | **Passed** | ### 3. Live Cloud Run Agent Identity (`3/3` Scenarios) Tested on a Cloud Run Agent Identity job (`--functional-type=agent --identity-type=agent-identity`). | Scenario | Env Vars | Key Results | | :--- | :--- | :--- | | **Default (Bound)** | *(unset)* | `DefaultMtlsProviderFactory` chain length = `2` (`#14621`). Access token acquired in `414ms`. ID token `cnf.x5t#S256` matched leaf cert. `DISABLE_BOUND_ID_TOKEN` (`#14559`) returned `cnf: false` and next default call restored `cnf: true`. GAX `ProjectsClient` (gRPC & HTTP/JSON) resolved `cloudresourcemanager.mtls.googleapis.com:443` and passed mTLS + bound-token auth (`PERMISSION_DENIED` IAM check). Non-mTLS transport rejected bound token (`401`), true mTLS passed auth (`403`), and mismatched client cert was rejected (`401`). | | **Runtime Bound Opt-Out** | `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` | Unbound access token and ID token (`cnf` absent). Non-mTLS endpoints passed auth (`403` IAM) and live Java ADK `LlmAgent` + `Gemini` call on Vertex AI succeeded (`200 OK`). | | **Client Cert Opt-Out** | `GOOGLE_API_USE_CLIENT_CERTIFICATE=false` | Unbound access token (`450ms`) and ID token (`cnf` absent). GAX `ProjectsClient` (gRPC & HTTP/JSON) resolved plain `cloudresourcemanager.googleapis.com:443` and passed auth (`PERMISSION_DENIED`). Live Java ADK `LlmAgent` + `Gemini` call on Vertex AI succeeded (`200 OK`). | ### 4. Live GKE Agent Identity (`6/6` Scenarios) Tested on a GKE Agent Identity cluster against live GCP APIs (`cloudresourcemanager`, `storage`) and an in-cluster SPIFFE mTLS receiver configured to trust **only** the SPIFFE root CA (requiring the full leaf + intermediate chain from `#14621`). | Scenario | Pod Annotations | Env Vars | Key Results | | :--- | :--- | :--- | :--- | | **`s1-default`** | `inject-podcertificates=true` | *(unset)* | `bundle exists=true`, first token fetch `1430ms`, SDK KeyStore chain length = `2`, mTLS receiver (SDK KeyStore) = `200 OK`, mTLS GCS passed auth (`404`), plain GCS rejected bound token (`401`), gRPC & HTTP/JSON resolved `.mtls.googleapis.com:443` and passed auth, `DISABLE_BOUND_ID_TOKEN` -> plain `200` / mTLS `401` / default after opt-out -> mTLS `200` / access token still bound -> plain GCS `401`. | | **`s2-runtime-bound-false`** | `inject-podcertificates=true` | `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` | Unbound tokens (`cnf=null`), mTLS receiver rejected unbound token (`401`), plain receiver accepted (`200`), raw GCS and Storage SDK (HTTP & gRPC) passed auth (`404`), gRPC & HTTP/JSON resolved `.mtls.googleapis.com:443` and passed auth. | | **`s3-use-cert-false`** | `inject-podcertificates=true` | `GOOGLE_API_USE_CLIENT_CERTIFICATE=false` | Unbound tokens (`cnf=null`), gRPC & HTTP/JSON resolved plain `cloudresourcemanager.googleapis.com:443` and passed auth, plain receiver = `200`, plain GCS passed auth (`404`). | | **`s4-use-cert-true`** | `inject-podcertificates=true` | `GOOGLE_API_USE_CLIENT_CERTIFICATE=true` | Identical bound behavior to `s1-default`: SDK KeyStore mTLS receiver = `200`, `.mtls.googleapis.com:443` resolved and passed auth, all `DISABLE_BOUND_ID_TOKEN` checks passed. | | **`s5-no-annotation`** | *(none)* | *(unset)* | `bundle exists=false`, first token fetch `646ms` (no 30s hang), resolved plain `cloudresourcemanager.googleapis.com:443`, passed auth (`PERMISSION_DENIED` / `403`). | | **`s6-use-cert-true-no-annotation`** | *(none)* | `GOOGLE_API_USE_CLIENT_CERTIFICATE=true` | Verified `#14658` on real GKE: `bundle exists=false`, first token fetch `609ms` (no 30s polling delay), resolved plain `cloudresourcemanager.googleapis.com:443`, passed auth (`PERMISSION_DENIED` / `403`). | </details> BEGIN_COMMIT_OVERRIDE feat(auth): request certificate-bound tokens for agent identities by default. Set GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false to opt out. feat(auth): add IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN to request unbound ID tokens for specific targets. feat(auth): support GKE agent identity credential bundle for bound tokens and mTLS. fix(auth): present the full certificate chain for mTLS. docs(auth): known limitation: ADK for Java doesn't support mTLS yet; agents that use it must set GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false. feat(gax): enable mTLS automatically when a workload certificate is present and support transparent retries during mTLS certificate rotations. fix(gax-httpjson): use Conscrypt TrustManagerFactory and KeyManagerFactory for mTLS SSLContext. END_COMMIT_OVERRIDE --------- Co-authored-by: Kirill Logachev <kirl@google.com> Co-authored-by: Min Zhu <zhumin@google.com> Co-authored-by: Lawrence Qiu <lawrenceqiu@google.com> Co-authored-by: yangyzs <171981480+yangyzs@users.noreply.github.com> Co-authored-by: cloud-java-bot <122572305+cloud-java-bot@users.noreply.github.com> Co-authored-by: Keshav Dandeva <keshavdandeva@google.com> Co-authored-by: Mattie Fu <mattiefu@google.com> Co-authored-by: Will Howes <whowes@google.com> Co-authored-by: Jin Seop Kim <jinseop@google.com> Co-authored-by: Jiang Zhu <jiangzzhu@google.com> Co-authored-by: Knut Olav Løite <koloite@gmail.com> Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: rahul2393 <irahul@google.com> Co-authored-by: Sakthivel Subramanian <179120858+sakthivelmanii@users.noreply.github.com> Co-authored-by: Gaole Meng <gaole@google.com> Co-authored-by: Pranav Iyer <pjiyer@google.com>

macastelaz added 3 commits October 7, 2026 02:26

fix(auth): do not reuse a cached GKE credential once a certificate co…

fb43614

…nfig exists

gemini-code-assist Bot reviewed Oct 9, 2026

View reviewed changes

Comment thread google-auth-library-java/oauth2_http/java/com/google/auth/mtls/MtlsKeyStoreUtils.java

Comment thread google-auth-library-java/oauth2_http/java/com/google/auth/mtls/MtlsKeyStoreUtils.java

macastelaz added 4 commits October 9, 2026 21:32

Merge remote-tracking branch 'origin/agentic-identities-bound-token' …

6b1f7c7

…into HEAD

Merge remote-tracking branch 'fork/gke-agent-bound-tokens' into mtls-…

f68c9c7

…full-cert-chain

chore(auth): merge origin/agentic-identities-bound-token into mtls-fu…

2dae679

…ll-cert-chain

macastelaz marked this pull request as ready for review October 14, 2026 13:38

macastelaz requested review from a team as code owners October 14, 2026 13:38

macastelaz requested review from lsirac and nbayati and removed request for a team and lsirac October 14, 2026 13:38

chore(auth): merge origin/agentic-identities-bound-token into mtls-fu…

f581f96

…ll-cert-chain

nbayati reviewed Oct 15, 2026

View reviewed changes

Comment thread google-auth-library-java/oauth2_http/javatests/com/google/auth/mtls/X509ProviderTest.java

Comment thread google-auth-library-java/oauth2_http/java/com/google/auth/mtls/MtlsKeyStoreUtils.java

test(auth): use forward-slash resource paths in X509ProviderTest

228aa05

macastelaz requested a review from nbayati October 15, 2026 16:03

lsirac requested changes Oct 15, 2026

View reviewed changes

fix(auth): address PR review feedback on mTLS certificate chain handling

8245ef3

macastelaz requested a review from lsirac October 15, 2026 20:15

nbayati approved these changes Oct 16, 2026

View reviewed changes

macastelaz merged commit 215fabb into googleapis:agentic-identities-bound-token Oct 16, 2026
263 checks passed

Conversation

macastelaz commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What changes

Tests

Unit tests

Live tests

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

macastelaz commented Oct 9, 2026 •

edited

Loading