feat: trace Codex, Worker, retrieval and session persistence

README.md

@@ -182,3 +182,11 @@ Join our discussion group by scanning the QR code below:
 ## License
 
 This project is licensed under the [Apache 2.0 License](./LICENSE).
+
+## Codex 性能诊断
+
+The Codex path retains `call_llm` as a whole-turn compatibility record for session indexing and evaluation. Each Responses Shim backend `litellm.aresponses` call has a separate `codex.model.request` Span under the originating Agent context, including library-internal retries; the reasoning compatibility retry creates another Span. This is not a per-HTTP-attempt or TTFT measurement: the backend returns a complete non-streaming response. Model Spans do not duplicate whole-turn usage or record bodies, endpoints or raw exceptions.
+
+验证使用 `tests/runtime/codex/test_model_tracing.py` 与现有 `test_codex_tracing.py`、`test_codex_shim_rounds.py`；实际观测接收仍需按部署环境验证。
+
+The remote CodeEnv Worker client traces readiness, session creation, turn start/status/cancel and each SSE connection as CLIENT Spans, propagating W3C context. Retries, backoff, authentication, idempotency keys and event cursors preserve their existing semantics. Stream Spans are detached from ambient context across generator yields. Spans exclude endpoint, routing IDs, bodies and credentials. Subscription time includes caller consumption and must not be interpreted as remote execution time. Remote queue/start timing and cross-turn recovery still require receiver instrumentation.

docs/content/docs/framework/tracing.en.mdx

@@ -163,3 +163,7 @@ In a multi-agent scenario, spans are linked through `parent_span_id`, showing th
 <Callout type="info" title="OpenTelemetry Compatibility">
   VeADK's field naming follows the OpenTelemetry generative-AI conventions, so you can import tracing data directly into any OpenTelemetry-compatible system for analysis and visualization.
 </Callout>
+
+### Session event persistence duration
+
+VeADK's built-in SQLite, MySQL and PostgreSQL session backends emit `veadk.session.append_event` for actual non-partial event append calls, helping identify storage delays within execution. Streaming partial events are not counted as database saves, and custom session services are not automatically instrumented. Existing export settings apply; event bodies, session state and database URLs are not recorded. The duration covers the ADK storage operation, not an individual SQL statement.

docs/content/docs/framework/tracing.mdx

@@ -161,3 +161,7 @@ Tracing 数据由一系列 **Span** 组成，每个 Span 记录一次操作（
 <Callout type="info" title="与 OpenTelemetry 的兼容性">
   VeADK 的字段命名遵循 OpenTelemetry 生成式 AI 规范，你可以直接将 Tracing 数据导入到 OpenTelemetry 兼容的系统中进行分析和可视化。
 </Callout>
+
+### 会话事件保存耗时
+
+VeADK 内置 SQLite、MySQL、PostgreSQL 会话后端为实际保存事件的 `append_event` 调用生成 `veadk.session.append_event` Span，便于判断会话保存是否拖慢执行。流式 partial 事件不作为数据库保存计时；自定义会话后端不自动增加此埋点。复用现有上报配置，不记录事件正文、会话状态或数据库地址。该耗时覆盖 ADK 存储调用，不等于某条 SQL 的执行时间。

prd-spec/features/execution-diagnostics/2026-10-10-execution-diagnostics.md

@@ -0,0 +1,28 @@
+# Execution diagnostics
+
+[中文](2026-10-10-execution-diagnostics.zh.md)
+
+Change ID: execution-diagnostics; date: 2026-10-10; status: in-review.
+Related contract: [execution tracing](../../../specs/execution-tracing/README.md).
+
+## Evidence and scope
+The existing Codex `call_llm` Span covers a whole execution. The Responses Shim performs non-streaming `litellm.aresponses` requests. At the design baseline, Worker HTTP requests were traced, while the SSE subscription was not. This design records the scope already authorized by the performance-diagnostics goal; it does not assert a new user approval or deployed capability.
+
+## Requirements and design
+FR-1: Preserve whole-turn compatibility while recording each model backend call separately under the originating Agent context. No duplicate whole-turn usage or synthetic TTFT.
+FR-2: Record Worker requests and each SSE connection, propagate only W3C tracing context, and preserve authentication, idempotency, retry limits, event validation and cursors.
+FR-3: Stream Spans must not remain current across an async-generator yield. End on completion, failure, cancellation or generator close. Record no endpoint, routing IDs, bodies or raw errors.
+FR-4: Missing optional telemetry must preserve execution. Reuse the existing Provider/exporter.
+
+## Tasks and acceptance
+T-1 / AC-1: Model Span tests prove parentage, retries, failures and cancellation; existing Codex indexing remains compatible.
+T-2 / AC-2: Worker HTTP tests prove W3C propagation, redaction and unchanged keys.
+T-3 / AC-3: SSE tests prove per-connection parentage, cursor continuity, cancellation/close and no current-context leak.
+T-4 / AC-4: Synchronize bilingual contracts and user documentation, run affected tests and pre-commit, and record actual results.
+
+## Review and risks
+Direct review: no public signature, protocol, authentication, persistence, concurrency ownership or deployment configuration changes. Existing retry semantics remain authoritative. Async-generator context leakage is a blocking risk addressed by detached stream Spans. Client subscription time is not remote queue/start/execution time. Real Worker receiver propagation, APMPlus reception and deployed versions remain unverified. SSE implementation and affected-test acceptance are complete in draft PR #1161; HTTP/model evidence is also recorded there. This record reconciles existing work and precedes the SSE extension.
+
+Verification (2026-10-10): 10 Worker tracing tests pass; 17 existing remote Sandbox tests pass (the initial combined run before adding new tests had 22 passes). SSE completion, reconnect/cursor, early close, cancellation and HTTP failure are covered. Full repository regression, live Codex smoke and APMPlus reception are not_run: isolated full-smoke dependencies and deployed receiver evidence are unavailable. Process deviation: SSE tests were added after the first implementation edit instead of first run failing; direct behavior assertions are now present. No deployed capability is claimed.
+
+Final checks: 27 affected tests pass after formatting; all-files pre-commit (Ruff and both secret scans) pass. Requirement scope is authorized by the existing performance-diagnostics goal. Direct design review found no remaining blocker for this extension; server-side verification remains separate.

prd-spec/features/execution-diagnostics/2026-10-10-execution-diagnostics.zh.md

@@ -0,0 +1,28 @@
+# 执行诊断
+
+[English](2026-10-10-execution-diagnostics.md)
+
+变更 ID：execution-diagnostics；日期：2026-10-10；状态：in-review。
+相关契约：[执行链路](../../../specs/execution-tracing/README.zh.md)。
+
+## 依据与范围
+现有 Codex call_llm Span 覆盖整个执行；Responses Shim 使用非流式 litellm.aresponses。设计基线时 Worker HTTP 请求已有记录，SSE 订阅尚未记录。本设计记录性能诊断 goal 已授权的范围，不宣称新增用户审批或已部署能力。
+
+## 要求与设计
+FR-1：保留整轮兼容记录，单独记录各次模型后端请求并关联原 Agent；不重复记录整轮用量，不计算合成 TTFT。
+FR-2：记录 Worker 请求与每次 SSE 连接，仅传递 W3C 链路上下文；认证、幂等、重试上限、事件校验和游标保持原行为。
+FR-3：流 Span 不得在异步生成器 yield 期间保持为当前 Span；完成、失败、取消和关闭生成器时结束。不记录 Endpoint、路由标识、正文和原始异常。
+FR-4：缺少可选观测依赖时保持执行；复用既有 Provider 和导出器。
+
+## 任务与验收
+T-1 / AC-1：模型测试证明父子关联、重试、失败与取消，Codex 索引兼容。
+T-2 / AC-2：Worker HTTP 测试证明 W3C 传递、脱敏与 Key 不变。
+T-3 / AC-3：SSE 测试证明各次连接父子关系、游标连续、取消与关闭，以及当前上下文不泄漏。
+T-4 / AC-4：同步双语契约及使用文档，运行受影响测试和 pre-commit，记录实际结果。
+
+## 评审与风险
+直接评审：不改公开签名、协议、认证、持久化、并发职责或部署配置。重试遵循现有语义。异步生成器上下文泄漏是必须解决的问题，使用不设为当前上下文的流 Span。客户端订阅时间不是远端排队、启动或执行耗时。Worker 接收端传递、APMPlus 接收与部署版本尚未验证。SSE 实现及受影响测试验收已在草稿 PR #1161 中完成，HTTP / 模型测试证据也记录在该 PR。本记录补齐已有改动依据，并先于 SSE 扩展建立。
+
+验证（2026-10-10）：10 项 Worker 链路测试通过，17 项现有远端 Sandbox 测试通过（新增用例前的初次合并运行共 22 项通过）。覆盖 SSE 完成、重连游标、提前关闭、取消和 HTTP 失败。全仓回归、真实 Codex smoke 和 APMPlus 接收为 not_run：缺少隔离的完整 smoke 依赖与部署接收证据。流程偏差：SSE 测试在首次实现编辑后添加，没有先执行失败测试；现已补充直接行为断言。不宣称已部署能力。
+
+最终检查：格式化后 27 项受影响测试通过；全文件 pre-commit（Ruff 和两项敏感信息扫描）通过。需求范围由既有性能诊断 goal 授权。直接设计评审未发现本扩展剩余阻断项；服务端验证仍单独进行。

prd-spec/features/retrieval-diagnostics/2026-10-10-retrieval-diagnostics.md

@@ -0,0 +1,64 @@
+# Retrieval diagnostics
+
+[中文](2026-10-10-retrieval-diagnostics.zh.md)
+
+Change ID: retrieval-diagnostics; revision: 2026-10-10; status: approved.
+
+## Background and scope
+
+`veadk/knowledgebase/knowledgebase.py:Knowledgebase.search` calls backend search and normalizes results. `veadk/memory/long_term_memory.py:LongTermMemory.search_memory` catches backend errors and returns empty memories; OpenViking calls run through `asyncio.to_thread`. `add_session_to_memory` filters events and saves them. These boundaries currently have no independent retrieval/write Span. The existing tool Span cannot distinguish retrieval from surrounding work.
+
+The user authorized the overall performance-diagnostics implementation, local verification and draft PR delivery. This change implements those already agreed retrieval boundaries. It does not authorize production deployment.
+
+## Goals and non-goals
+
+Measure actual knowledge search, memory search and memory save boundaries, inheriting the existing OTel context and Provider. Preserve signatures, results, error propagation and cancellation. Do not introduce exporters, sampling configuration, storage, dashboards or backend-internal embedding/reranking stages. Existing observability owns receiving, export, storage and queries; diagnostics owns the operation boundaries and evidence used to identify slow operations.
+
+## Requirements and scenarios
+
+- FR-1: Knowledge search creates `veadk.knowledge.search`, ending after result normalization; records backend type, effective top_k and result count. Backend failure retains its exception and records ERROR.
+- FR-2: Memory search creates `veadk.memory.search`; backend failure remains an empty response under the existing API, but Span status is ERROR with safe error type. Empty successful results remain successful.
+- FR-3: Memory save creates `veadk.memory.save`, covering filtering and backend save; records backend type, session ID and filtered event count. Errors still propagate.
+- FR-4: All operations inherit parent context, finish on success/error/cancellation, and restore the caller context. No raw exception events, query, memory content, credentials, user ID, index, URL or arbitrary kwargs are exported. A telemetry failure must not block or repeat business execution.
+
+Given a tool calling knowledge search through `asyncio.to_thread`, the retrieval Span is a child of the existing tool Span. Given two concurrent memory requests, each remains under its own parent. Given a cancelled OpenViking await, the caller Span ends with cancellation; the underlying synchronous thread may continue, so the Span must not claim backend completion.
+
+## Design and contract impact
+
+Use a small internal helper under `veadk/tracing/` for fail-open fixed-name spans with explicit safe outcome handling and no exporter setup. Reuse `trace.get_tracer`; do not change global Provider. The helper activates a span for the operation and restores context in finally, disables automatic exception recording, and records only error class/status. Avoid catching business exceptions in the helper as success. Backend attributes come from configured backend type, never full configuration. Apply result counts only after successful normalization. Memory search marks its handled backend exception before returning the established empty response. `asyncio.to_thread` already copies context; add no extra worker propagation protocol.
+
+Public signatures, result types, backend arguments, auth, retry behavior and memory isolation remain unchanged. There is no new persistent state or deployment configuration. Extend the bilingual execution-tracing spec with the three boundaries and their failure semantics.
+
+## Tasks and acceptance
+
+| Requirement | Task | Acceptance | Verification | Status |
+| --- | --- | --- | --- | --- |
+| FR-1 | T-1 knowledge boundary | AC-1 parent/name/effective top_k/count and unchanged errors | Targeted knowledge tests with local SpanExporter | pass (local) |
+| FR-2 | T-2 memory search | AC-2 handled failure distinguished from successful empty response | Memory tests: success, empty, failure, concurrent parents, OpenViking cancellation | pass (local) |
+| FR-3 | T-3 memory save | AC-3 event count/session, propagation and unchanged save arguments | Memory save tests: filtering, backend failure and cancellation | pass (local) |
+| FR-4 | T-4 protections | AC-4 no sensitive content; telemetry failures do not alter business calls/context | Helper tests with failing tracer/span plus integration parent tests | pass (local) |
+| FR-1–FR-4 | T-5 delivery | AC-5 bilingual docs and affected regression gates, actual PR recorded | Repository-local targeted pytest; all-files pre-commit; synchronize base before commit | pass (draft PR) |
+
+Test entry points: `tests/test_knowledgebase.py`, `tests/test_long_term_memory.py`, `tests/test_openviking_long_term_memory.py`, `tests/tools/builtin_tools/test_load_knowledgebase.py`; add colocated tracing-helper tests where existing tests organize tracing. Execute through repository `.venv` using `uv run --extra dev pytest`. No real services or credentials in fixtures. Use a controlled blocking backend for thread-cancellation semantics and clean it up before test teardown.
+
+## Risks and review
+
+Backend internals and deployed APMPlus reception remain unverified. A cancelled thread-backed call may still run remotely; measure caller boundary only and mark cancelled, never backend success. Global telemetry setup can be absent: no-op tracing must leave execution unchanged. Do not add duplicate tokens/cost or infer unknown internal stages.
+
+Design review: pass (2026-10-10). Verified actual public boundaries, context propagation through to_thread, existing handled-error semantics, bounded attributes and matched bilingual requirements. No blockers; thread completion remains explicitly outside caller-cancellation evidence. Production code/tests begin with TDD after this review. Approval scope comes from the user's existing active goal; no new deployment permission is inferred.
+
+## Local implementation evidence (2026-10-10)
+
+| Check | Result and scope |
+| --- | --- |
+| TDD | Helper test initially failed because the module did not exist. Initial boundary failures were fixture setup errors, not valid missing-Span evidence; corrected isolated backend fixtures before asserting the public methods. |
+| Retrieval/Worker subset | 76 passed, 1 optional-dependency skip; helper 100% lines and 98% combined branch coverage. Includes concurrent parents, thread cancellation with explicit drain, telemetry mutation/end failures, attribute whitelist and actual tool-to-retrieval context. |
+| Initial broad regression | fail: 6607 passed, 16 failed, 59 skipped, 4 xfailed, 2 collection errors (422.59s). All failures/errors reported missing llama_index, openai_codex or anthropic. No tests removed or reclassified. |
+| Environment recovery | Installed declared dev/extensions/codex/sandbox extras. Codex binary download first timed out at 30s; retry with command-scoped UV_HTTP_TIMEOUT=180 exited 0. No global configuration changed. |
+| Affected regression after recovery | pass: 199 passed, no skips/errors, seven existing dependency deprecation warnings (46.95s). Covers the failed Harness/Codex suites, both Sandbox suites, retrieval/memory/tool context and Worker requests. |
+| Final source/security review | pass: backend arguments, filtering, successful empty responses, errors and cancellation preserved. No new exporter/configuration or sensitive content. |
+| Base and hooks | Fresh origin/main remains 171d8d86, already rebased and an ancestor of HEAD. All-files pre-commit passed Ruff and both secret scanners with complete extras. |
+| Full regression | pass: 6698 passed, 50 skipped, 4 xfailed, 88 dependency warnings, 427.41s. Skips do not prove behavior. First interrupted run had no result; restarted only after confirming absence. Local metrics export reported localhost:8000 unavailable, so remote observability reception is not proved. |
+| Delivery/deployment | draft PR delivered: [#1161](https://github.com/volcengine/veadk-python/pull/1161), code commit e2c1e887. Remote PR head verified after push. No merge or deployment; APMPlus reception and real performance remain unverified. |
+
+Evidence: `/tmp/codex-retrieval-save-tests.log`, `/tmp/codex-veadk-retrieval-broad.log`, `/tmp/codex-veadk-complete-env-retry.log`, `/tmp/codex-veadk-complete-env-retest.log`, `/tmp/codex-retrieval-extras-final-hooks.log`, `/tmp/codex-veadk-complete-broad-resumed.log`.