Skip to content

Sandbox nested persona template rendering in evaluation prompts - #5266

Closed
petrmarinec wants to merge 1 commit into
google:mainfrom
petrmarinec:fix-jinja-ssti-sandbox
Closed

petrmarinec wants to merge 1 commit into
google:mainfrom
petrmarinec:fix-jinja-ssti-sandbox

Conversation

@petrmarinec

Copy link
Copy Markdown
Contributor

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

2. Or, if no issue exists, describe the change:

Problem:
Nested persona behavior strings were rendered through render_string_filter, which created a fresh Jinja template from persona-controlled content instead of consistently reusing a sandboxed environment. That allowed nested persona templates to execute outside the intended sandbox boundary in evaluation prompt construction.

Affected files:

  • src/google/adk/evaluation/simulation/llm_backed_user_simulator_prompts.py
  • src/google/adk/evaluation/simulation/per_turn_user_simulator_quality_prompts.py

Solution:
Render nested persona strings through SandboxedEnvironment and use SandboxedEnvironment for the per-turn evaluator prompt builder as well. This keeps supported nested placeholders such as {{ stop_signal }} working while blocking unsafe nested template access. Added regression tests for both the safe interpolation path and blocked unsafe attribute traversal.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

Passed in clean Linux Docker (python:3.11-bookworm):

  • uv sync --all-extras
  • pytest tests/unittests/evaluation/simulation
  • Result: 70 passed

Additional repo-wide validation:

  • pytest tests/unittests
  • Result on patched branch: 5326 passed, 1 skipped, 5 failed
  • The same 5 failures reproduce on unmodified origin/main
  • Those failures are unrelated tests/unittests/tools/test_skill_toolset.py integration timeouts

Manual End-to-End (E2E) Tests:

  • Ran a live adk web regression test against the eval API in Linux Docker using a local non-LLM root agent.
  • Malicious nested persona template {{ ''.__class__.__mro__ }} was blocked during prompt construction with jinja2.exceptions.SecurityError.
  • A safe persona using nested {{ stop_signal }} placeholders did not raise TemplateSyntaxError or SecurityError and progressed beyond prompt rendering into a real Gemini model call.
  • The safe run did not fully complete because the test key hit 429 RESOURCE_EXHAUSTED, but the absence of template errors and the subsequent model call confirm the sandboxed nested rendering path is functioning as intended.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

This is a small, focused fix intended to close the nested-template sandbox bypass path without changing the supported nested placeholder behavior used by existing personas.

@adk-bot adk-bot added the eval [Component] This issue is related to evaluation label Apr 10, 2026
@petrmarinec

Copy link
Copy Markdown
Contributor Author

For clarity, I also reran the remaining failing full-suite tests on unmodified origin/main in the same Docker-based setup.

The only full-suite failures I saw were these 5 test_skill_toolset integration tests:

  • tests/unittests/tools/test_skill_toolset.py::test_integration_python_stdout
  • tests/unittests/tools/test_skill_toolset.py::test_integration_python_sys_exit_zero
  • tests/unittests/tools/test_skill_toolset.py::test_integration_shell_stdout_and_stderr
  • tests/unittests/tools/test_skill_toolset.py::test_integration_shell_stderr_only
  • tests/unittests/tools/test_skill_toolset.py::test_integration_shell_nonzero_exit

Those same 5 failures reproduce on unmodified origin/main, so they are not introduced by this PR.

@rohityan rohityan self-assigned this Apr 13, 2026
@rohityan rohityan added the needs review [Status] The PR/issue is awaiting review from the maintainer label Apr 13, 2026
@rohityan

Copy link
Copy Markdown
Collaborator

Hi @petrmarinec , Thank you for your contribution! We appreciate you taking the time to submit this pull request. Your PR has been received by the team and is currently under review. We will provide feedback as soon as we have an update to share.

@rohityan

Copy link
Copy Markdown
Collaborator

Hi @wukath , can you please review this.

@petrmarinec
petrmarinec force-pushed the fix-jinja-ssti-sandbox branch from c4e30e5 to edc6364 Compare June 10, 2026 09:04
copybara-service Bot pushed a commit that referenced this pull request Jun 17, 2026
Merge #5266

### Link to Issue or Description of Change

**1. Link to an existing issue (if applicable):**

- Related: #5265

**2. Or, if no issue exists, describe the change:**

**Problem:**
Nested persona behavior strings were rendered through `render_string_filter`, which created a fresh Jinja template from persona-controlled content instead of consistently reusing a sandboxed environment. That allowed nested persona templates to execute outside the intended sandbox boundary in evaluation prompt construction.

Affected files:
- `src/google/adk/evaluation/simulation/llm_backed_user_simulator_prompts.py`
- `src/google/adk/evaluation/simulation/per_turn_user_simulator_quality_prompts.py`

**Solution:**
Render nested persona strings through `SandboxedEnvironment` and use `SandboxedEnvironment` for the per-turn evaluator prompt builder as well. This keeps supported nested placeholders such as `{{ stop_signal }}` working while blocking unsafe nested template access. Added regression tests for both the safe interpolation path and blocked unsafe attribute traversal.

### Testing Plan

**Unit Tests:**

- [x] I have added or updated unit tests for my change.
- [ ] All unit tests pass locally.

Passed in clean Linux Docker (`python:3.11-bookworm`):
- `uv sync --all-extras`
- `pytest tests/unittests/evaluation/simulation`
- Result: `70 passed`

Additional repo-wide validation:
- `pytest tests/unittests`
- Result on patched branch: `5326 passed, 1 skipped, 5 failed`
- The same 5 failures reproduce on unmodified `origin/main`
- Those failures are unrelated `tests/unittests/tools/test_skill_toolset.py` integration timeouts

**Manual End-to-End (E2E) Tests:**

- [x] Ran a live `adk web` regression test against the eval API in Linux Docker using a local non-LLM root agent.
- [x] Malicious nested persona template `{{ ''.__class__.__mro__ }}` was blocked during prompt construction with `jinja2.exceptions.SecurityError`.
- [x] A safe persona using nested `{{ stop_signal }}` placeholders did not raise `TemplateSyntaxError` or `SecurityError` and progressed beyond prompt rendering into a real Gemini model call.
- [x] The safe run did not fully complete because the test key hit `429 RESOURCE_EXHAUSTED`, but the absence of template errors and the subsequent model call confirm the sandboxed nested rendering path is functioning as intended.

### Checklist

- [x] I have read the [CONTRIBUTING.md](https://github.com/google/adk-python/blob/main/CONTRIBUTING.md) document.
- [x] I have performed a self-review of my own code.
- [ ] I have commented my code, particularly in hard-to-understand areas.
- [x] I have added tests that prove my fix is effective or that my feature works.
- [ ] New and existing unit tests pass locally with my changes.
- [x] I have manually tested my changes end-to-end.
- [x] Any dependent changes have been merged and published in downstream modules.

### Additional context

This is a small, focused fix intended to close the nested-template sandbox bypass path without changing the supported nested placeholder behavior used by existing personas.

Co-authored-by: George Weale <gweale@google.com>
COPYBARA_INTEGRATE_REVIEW=#5266 from petrmarinec:fix-jinja-ssti-sandbox edc6364
PiperOrigin-RevId: 933966100
@adk-bot

adk-bot commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator

Thank you @petrmarinec for your contribution! 🎉

Your changes have been successfully imported and merged via Copybara in commit 30493ba.

Closing this PR as the changes are now in the main branch.

@adk-bot adk-bot added the merged [Status] This PR is merged label Jun 17, 2026
@adk-bot adk-bot closed this Jun 17, 2026
@petrmarinec

Copy link
Copy Markdown
Contributor Author

Google OSS VRP #498357322

FrigaZzz pushed a commit to FrigaZzz/adk-python that referenced this pull request Aug 11, 2026
Merge google#5266

### Link to Issue or Description of Change

**1. Link to an existing issue (if applicable):**

- Related: google#5265

**2. Or, if no issue exists, describe the change:**

**Problem:**
Nested persona behavior strings were rendered through `render_string_filter`, which created a fresh Jinja template from persona-controlled content instead of consistently reusing a sandboxed environment. That allowed nested persona templates to execute outside the intended sandbox boundary in evaluation prompt construction.

Affected files:
- `src/google/adk/evaluation/simulation/llm_backed_user_simulator_prompts.py`
- `src/google/adk/evaluation/simulation/per_turn_user_simulator_quality_prompts.py`

**Solution:**
Render nested persona strings through `SandboxedEnvironment` and use `SandboxedEnvironment` for the per-turn evaluator prompt builder as well. This keeps supported nested placeholders such as `{{ stop_signal }}` working while blocking unsafe nested template access. Added regression tests for both the safe interpolation path and blocked unsafe attribute traversal.

### Testing Plan

**Unit Tests:**

- [x] I have added or updated unit tests for my change.
- [ ] All unit tests pass locally.

Passed in clean Linux Docker (`python:3.11-bookworm`):
- `uv sync --all-extras`
- `pytest tests/unittests/evaluation/simulation`
- Result: `70 passed`

Additional repo-wide validation:
- `pytest tests/unittests`
- Result on patched branch: `5326 passed, 1 skipped, 5 failed`
- The same 5 failures reproduce on unmodified `origin/main`
- Those failures are unrelated `tests/unittests/tools/test_skill_toolset.py` integration timeouts

**Manual End-to-End (E2E) Tests:**

- [x] Ran a live `adk web` regression test against the eval API in Linux Docker using a local non-LLM root agent.
- [x] Malicious nested persona template `{{ ''.__class__.__mro__ }}` was blocked during prompt construction with `jinja2.exceptions.SecurityError`.
- [x] A safe persona using nested `{{ stop_signal }}` placeholders did not raise `TemplateSyntaxError` or `SecurityError` and progressed beyond prompt rendering into a real Gemini model call.
- [x] The safe run did not fully complete because the test key hit `429 RESOURCE_EXHAUSTED`, but the absence of template errors and the subsequent model call confirm the sandboxed nested rendering path is functioning as intended.

### Checklist

- [x] I have read the [CONTRIBUTING.md](https://github.com/google/adk-python/blob/main/CONTRIBUTING.md) document.
- [x] I have performed a self-review of my own code.
- [ ] I have commented my code, particularly in hard-to-understand areas.
- [x] I have added tests that prove my fix is effective or that my feature works.
- [ ] New and existing unit tests pass locally with my changes.
- [x] I have manually tested my changes end-to-end.
- [x] Any dependent changes have been merged and published in downstream modules.

### Additional context

This is a small, focused fix intended to close the nested-template sandbox bypass path without changing the supported nested placeholder behavior used by existing personas.

Co-authored-by: George Weale <gweale@google.com>
COPYBARA_INTEGRATE_REVIEW=google#5266 from petrmarinec:fix-jinja-ssti-sandbox edc6364
PiperOrigin-RevId: 933966100
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

eval [Component] This issue is related to evaluation merged [Status] This PR is merged needs review [Status] The PR/issue is awaiting review from the maintainer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants