fix(tracing): bound local span retention and share provider cache

docs/content/docs/framework/tracing.en.mdx

@@ -56,6 +56,30 @@ Once enabled, the IDs of the related tracing data are printed in the logs:
 
 For the full integration steps per platform, see [Observe on Volcengine](/en/docs/framework/ve-tracing).
 
+## Limit local retention
+
+The following settings ship with this fix and are unavailable in the published 1.1.14 release and earlier versions
+
+`OpentelemetryTracer` retains up to 1024 recent spans with an estimated data budget of 16 MiB by default. It evicts the oldest spans when either limit is reached. A span exceeding the byte limit is skipped locally and still passed to cloud exporters according to the existing configuration
+
+Set these environment variables before starting the application to adjust local retention:
+
+| Environment variable | Type | Default | Description |
+| --- | --- | --- | --- |
+| `OBSERVABILITY_OPENTELEMETRY_INMEMORY_ENABLED` | Boolean | `true` | Whether to retain local span history |
+| `OBSERVABILITY_OPENTELEMETRY_INMEMORY_MAX_SPANS` | Nonnegative integer | `1024` | Maximum retained span count; `0` disables retention |
+| `OBSERVABILITY_OPENTELEMETRY_INMEMORY_MAX_BYTES` | Nonnegative integer | `16777216` | Estimated data budget in bytes; `0` disables retention |
+
+For applications that only need cloud observability, set this variable in the terminal used to start the application:
+
+```bash
+export OBSERVABILITY_OPENTELEMETRY_INMEMORY_ENABLED=false
+```
+
+Disabling local retention preserves span annotations, trace IDs in logs, and configured cloud exports; `tracer.dump()` writes an empty array. With retention enabled, dumps contain only retained session history. If a span used to associate a trace with a session is evicted, that trace may also disappear from local dumps. Query your cloud observability platform for complete history
+
+Tracers using the same TracerProvider share these limits. Configuration takes effect when the cache is first created; restart the application or create a new provider after changing it. The byte budget estimates cached data and does not limit total process memory
+
 ## Why Tracing
 
 With tracing in place, you can:

docs/content/docs/framework/tracing.mdx

@@ -54,6 +54,30 @@ Tracing（链路追踪）对智能体执行过程进行**全链路记录**，是
 
 针对不同平台的完整接入方式，参见[在火山引擎观测](/cn/docs/framework/ve-tracing)。
 
+## 限制本地缓存
+
+以下配置随本次修复提供，已发布的 1.1.14 及更早版本不支持这些配置
+
+`OpentelemetryTracer` 默认保留最近的 1024 个 Span，估算数据总量最多为 16 MiB。达到任一上限时会淘汰最早的 Span；单个 Span 超过字节上限时不写入本地缓存，仍按原有配置交给云端 Exporter
+
+在启动应用前，可通过以下环境变量调整本地缓存：
+
+| 环境变量 | 类型 | 默认值 | 说明 |
+| --- | --- | --- | --- |
+| `OBSERVABILITY_OPENTELEMETRY_INMEMORY_ENABLED` | 布尔值 | `true` | 是否保留本地 Span 历史 |
+| `OBSERVABILITY_OPENTELEMETRY_INMEMORY_MAX_SPANS` | 非负整数 | `1024` | 最多保留的 Span 数量，`0` 表示不保留 |
+| `OBSERVABILITY_OPENTELEMETRY_INMEMORY_MAX_BYTES` | 非负整数 | `16777216` | 估算数据量上限，单位为字节，`0` 表示不保留 |
+
+仅需云端观测时，在启动应用的终端设置：
+
+```bash
+export OBSERVABILITY_OPENTELEMETRY_INMEMORY_ENABLED=false
+```
+
+关闭本地缓存后，Span 标注、日志中的 Trace ID 和已配置的云端上报保持可用，`tracer.dump()` 写入空数组。启用缓存时，`dump()` 只包含仍保留的会话历史；用于关联会话的 Span 被淘汰后，对应链路也可能不再出现在本地导出中，完整历史应在云端观测平台查询
+
+同一个 TracerProvider 下的 tracer 共享上述额度。配置在首次创建缓存时生效，修改后需重启应用或创建新的 Provider；字节数是缓存数据的估算值，不代表整个进程的内存上限
+
 ## Tracing 的价值
 
 引入 Tracing 后，你可以：

prd-spec/bugfixes/tracing-memory/2026-10-13-bounded-cache.md

@@ -0,0 +1,73 @@
+# Bound the local tracing cache
+
+[中文版](2026-10-13-bounded-cache.zh.md)
+
+## Metadata
+
+- Change ID: `tracing-memory`
+- Created/revised: 2026-10-13
+- Status: implemented
+- Component: [tracing cache](../../../specs/tracing-cache/README.md)
+
+## Background, goals, and scope
+
+Published versions 1.0.9 and 1.1.14 append every sampled span to a process-lifetime list. Each tracer also registers another collector with the global provider. The session index grows independently, and `clear()` does not clear it. Large responses attached to repeatedly polled requests amplify retention. Source inspection confirms the defect; the proportion of any production process's RSS attributable to it is not yet measured.
+
+The fix bounds local retention without disabling cloud exporters or changing span annotation, sampling, or instrumentation suppression. It preserves recent session dumps and the latest trace ID. It does not change AgentKit SDK request payloads, frontend behavior, production configuration, package versions, or deployment.
+
+## Scenarios and requirements
+
+- FR-1: Given sustained exports, retain at most 1,024 spans and 16,777,216 estimated bytes by default; evict oldest spans until both limits hold. Reject a single oversized span from the local cache without discarding earlier valid entries or altering cloud export.
+- FR-2: Given local caching disabled, retain no spans or session index, but keep the annotation processor, latest trace ID, and cloud pipeline operational.
+- FR-3: Given eviction, clear, or shutdown, do not retain stale session mappings. Retrieval returns a consistent snapshot of retained spans; concurrent export/clear/read must be safe.
+- FR-4: Given several tracers using one provider, register exactly one shared local collector at the front of its processor pipeline. Each provider has its own cache, with configuration captured when its first collector is created.
+- FR-5: Publish validated configuration and document that local dumps contain only retained history; cloud retention is owned by the cloud exporter/service.
+
+## Design and contract impact
+
+Use a lock-protected FIFO deque with parallel per-span size accounting. Estimate the known span fields, attribute values, resource, events, links, and instrumentation metadata using Python object sizes without serializing or copying large payloads. This estimate is a cache budget, not an exact process RSS limit; count bounds metadata overhead as well. Skip oversized entries locally. Derive the session-to-trace index from retained `call_llm` spans on read, avoiding an independently growing index. A trace whose session-bearing span was evicted may no longer appear in a local session dump.
+
+Expose `inmemory_enabled`, `inmemory_max_spans`, and `inmemory_max_bytes` in `OpenTelemetryConfig`, with corresponding `OBSERVABILITY_OPENTELEMETRY_INMEMORY_*` variables. Defaults are true, 1024, and 16777216; nonnegative limits are required and either zero disables storage. Configuration follows existing startup settings/config.yaml handling; changes require a new provider/process.
+
+The provider owns a shared exporter, initialized under a registration lock. Keep the existing annotation processor even when retention is disabled. Export, read, clear, and shutdown share a cache lock; shutdown is terminal. `force_flush()` does not clear history. No background threads, timers, new dependencies, new network calls, or permission changes are introduced.
+
+Alternatives rejected: periodic clear leaves peak retention dependent on traffic and loses whole batches; count-only bounds do not adequately handle large responses; removing the processor would remove annotation and suppression behavior. Public tracer/exporter constructor signatures remain compatible. Local history changes from unlimited to bounded by design; no cloud span is truncated or filtered by this cache.
+
+## Tasks and acceptance
+
+| Requirement | Task | Acceptance | Verification |
+| --- | --- | --- | --- |
+| FR-1, FR-3 | T-1: add failing regression tests, then implement bounded cache in `inmemory_exporter.py` | AC-1: count/byte eviction, oversized payload, session cleanup, shutdown, and concurrent operations pass | `uv run --no-sync --extra dev pytest tests/test_inmemory_exporter.py -q` |
+| FR-2, FR-4 | T-2: share the collector in `opentelemetry_tracer.py` | AC-2: multiple tracers share one collector; cloud spans, annotations, and trace IDs survive disabled/limited retention | `uv run --no-sync --extra dev pytest tests/test_tracing.py -q` |
+| FR-5 | T-3: add config validation and bilingual user docs/spec | AC-3: defaults, overrides, zero, and invalid values are verified | `uv run --no-sync --extra dev pytest tests/config/test_tracing_config.py -q` |
+| All | T-4: inspect diff, run focused tracing regressions and Ruff, reconcile both languages | AC-4: checks pass and limitations are recorded | See delivery record |
+
+## Risks and review
+
+Finite local history may truncate long session dumps. A shared cache covers the entire provider, so all tracers compete for the same budget and observe the same recent history. Python object estimates exclude allocator fragmentation and memory owned elsewhere; this does not promise an RSS ceiling. No production memory plateau or cloud behavior may be claimed from local tests alone.
+
+Design reviewed directly: ownership, errors, concurrency, compatibility, configuration, bilingual equivalence, and verification coverage have no open blockers. User approval: request on 2026-10-13 to fix the confirmed leak following the proposed bounded-cache approach. Complex tests and cloud operations require follow-up; neither is needed to implement the scoped regression fix.
+
+## Delivery record
+
+- Baseline: `b8038fb16`; verification date: 2026-10-13
+- Scope: bounded exporter, provider registration, configuration, regression tests, and bilingual cache documentation; T-1 through T-4 and AC-1 through AC-4 complete
+- Regression before implementation: fail as expected, 20 failed and 3 passed for cache/config tests; includes old-span retention and stale session state
+- Core checks: pass, 46 tests via `uv run --no-sync --extra dev pytest tests/test_inmemory_exporter.py tests/config/test_tracing_config.py tests/test_tracing.py -q`
+- Focused compatibility checks: pass, 75 tests total including the core checks, content collection, LLM attribute extraction, runtime tracing, and Studio telemetry; one pre-existing invalid-escape warning
+- Existing exporter fixtures also attempted metric delivery to `localhost:8000` during shutdown and logged `UNAVAILABLE`; the test process exited 0. New cache/cloud integration tests use a local in-memory stand-in and no cloud service
+- Ruff format/check and `git diff --check`: pass for changed Python files and the patch
+- Documentation: pass, both tracing MDX pages compile with `@mdx-js/mdx`; bilingual contract/configuration consistency and relative document links reviewed
+- Pre-submission checks: pass, reran 75 focused tests after synchronizing `origin/main`; `uv run --no-sync --extra dev pre-commit run --all-files` passed, including Ruff and both secret scanners
+- Full regression and documentation site build/type checks: not_run; this pass is limited to focused checks under the requested workflow, with no frontend application change
+- Production/cloud validation: not_run, no production access or changes requested
+- UI validation: not_applicable, no frontend behavior changes
+- Commit and PR: authorized; package publication and production deployment: not_run, outside this scope
+
+## CI follow-up
+
+The initial Python 3.12 CI run had 7023 passing tests and two failures caused by runtime tests slicing the private `_spans` list. Update those two consumers to read session snapshots through `get_finished_spans(session_id)`, avoiding assumptions about the queue type or offsets that change after eviction. Preserve the existing span and session assertions. This follow-up changes no production code or approved contract
+
+- Failed cases: reproduced both failures locally before the update; both pass afterward
+- Extended check: `uv run --no-sync --extra dev pytest tests/runtime/differential tests/runtime/piagent/test_piagent_turn_contract.py -q` finishes with 84 passed, 2 xfailed, and 1 failed after the update
+- Remaining failure: `test_model_fallbacks_warn_for_external_runtime` sees empty captured logs; it also fails on the original PR commit `9671acd7`, where the same check finishes with 82 passed, 2 xfailed, and 3 failed. It is outside the cache fix and remains unchanged

prd-spec/bugfixes/tracing-memory/2026-10-13-bounded-cache.zh.md

@@ -0,0 +1,73 @@
+# 限制本地追踪缓存
+
+[English](2026-10-13-bounded-cache.md)
+
+## 元数据
+
+- 变更 ID：`tracing-memory`
+- 创建及修订日期：2026-10-13
+- 状态：implemented
+- 组件：[追踪缓存](../../../specs/tracing-cache/README.zh.md)
+
+## 背景、目标与范围
+
+已发布的 1.0.9 和 1.1.14 会将每个被采样的 span 追加到进程生命周期内的列表中，每个追踪器还会向全局 provider 注册另一个收集器。会话索引独立增长，`clear()` 不会清理它。反复轮询请求携带的大响应会放大内存占用。源码检查已确认缺陷，尚未测量它在具体生产进程 RSS 中的占比
+
+本次修复限制本地保留量，保留云端 exporter、span 字段补充、采样及埋点抑制行为，继续支持近期会话导出和最新 trace ID。不修改 AgentKit SDK 请求内容、前端行为、生产配置、包版本或部署
+
+## 场景与需求
+
+- FR-1：持续导出时，默认最多保留 1,024 个 span 和 16,777,216 估算字节，淘汰最旧 span 直到同时满足两个限制。单个 span 超限时仅跳过本地缓存，不丢弃之前的有效记录，也不改变云端上报
+- FR-2：关闭本地缓存时，不保留 span 或会话索引，仍保留字段补充处理器、最新 trace ID 和云端链路
+- FR-3：淘汰、清空或关闭后，不保留过期会话映射。查询返回保留数据的一致快照，并发导出、清理及读取安全
+- FR-4：同一 provider 的多个追踪器只在处理流水线首部注册一个共享收集器。不同 provider 拥有各自的缓存，首次创建收集器时确定配置
+- FR-5：提供经过校验的配置，明确本地导出只包含仍被保留的历史，云端保留策略由云端 exporter 或服务负责
+
+## 设计与契约影响
+
+使用加锁的 FIFO 双端队列和对应的逐 span 大小记录。通过 Python 对象大小估算已知 span 字段、属性值、资源、事件、链接和埋点元数据，不序列化或复制大块内容。该估算是缓存预算，并非进程 RSS 精确限制；数量限制同时约束元数据开销。超大记录仅跳过本地保存。读取时从仍被保留的 `call_llm` span 生成会话到 trace 的索引，避免独立索引持续增长。若包含会话信息的 span 已被淘汰，对应 trace 可能不再出现在本地会话导出中
+
+在 `OpenTelemetryConfig` 中提供 `inmemory_enabled`、`inmemory_max_spans`、`inmemory_max_bytes`，对应 `OBSERVABILITY_OPENTELEMETRY_INMEMORY_*` 环境变量。默认值为 true、1024、16777216；限制值必须非负，任一限制为零即关闭保存。沿用已有启动 settings/config.yaml 处理方式，修改后需新建 provider 或重启进程
+
+provider 拥有共享 exporter，通过注册锁初始化。即使关闭保存，仍保留现有字段补充处理器。导出、读取、清空和关闭共用缓存锁，关闭为终态。`force_flush()` 不清理历史。不引入后台线程、定时任务、新依赖、额外网络调用或权限变更
+
+未选择的方案：定时清空的峰值仍由流量决定且会整批丢失历史；只限制数量不足以控制大响应；删除处理器会破坏字段补充与埋点抑制。公开 tracer/exporter 构造签名兼容。按设计将本地历史从无限保留改为有限保留，缓存不会截断或过滤任何云端 span
+
+## 任务与验收
+
+| 需求 | 任务 | 验收 | 验证 |
+| --- | --- | --- | --- |
+| FR-1、FR-3 | T-1：先增加失败回归，再在 `inmemory_exporter.py` 实现有界缓存 | AC-1：数量及字节淘汰、超大内容、会话清理、关闭和并发操作通过 | `uv run --no-sync --extra dev pytest tests/test_inmemory_exporter.py -q` |
+| FR-2、FR-4 | T-2：在 `opentelemetry_tracer.py` 共享收集器 | AC-2：多追踪器共用一个收集器，关闭或限制缓存仍保留云端 span、字段和 trace ID | `uv run --no-sync --extra dev pytest tests/test_tracing.py -q` |
+| FR-5 | T-3：增加配置校验和双语用户文档及规范 | AC-3：验证默认值、覆盖、零值和非法值 | `uv run --no-sync --extra dev pytest tests/config/test_tracing_config.py -q` |
+| 全部 | T-4：检查差异，运行针对性追踪回归及 Ruff，同步双语记录 | AC-4：检查通过，限制被如实记录 | 见交付记录 |
+
+## 风险与审查
+
+有限本地历史可能导致长会话导出不完整。共享缓存覆盖整个 provider，各追踪器竞争同一预算并观察同一近期历史。Python 对象估算不包含分配器碎片和其他组件拥有的内存，不承诺 RSS 上限。不能用本地测试声称生产内存已稳定或云端行为已验证
+
+已直接审查归属、错误、并发、兼容性、配置、双语一致性及验证覆盖，无待解决阻塞项。用户于 2026-10-13 在提出有界缓存方向后要求修复已确认的泄漏。复杂测试及云端操作需后续确认，本次范围内的回归修复不依赖二者
+
+## 交付记录
+
+- 基线：`b8038fb16`；验证日期：2026-10-13
+- 范围：有界 exporter、provider 注册、配置、回归测试和双语缓存文档；T-1 至 T-4、AC-1 至 AC-4 已完成
+- 实现前回归：按预期 fail，缓存及配置测试 20 项失败、3 项通过，包含旧 span 引用保留和会话状态残留
+- 核心检查：pass，`uv run --no-sync --extra dev pytest tests/test_inmemory_exporter.py tests/config/test_tracing_config.py tests/test_tracing.py -q` 共 46 项通过
+- 针对性兼容检查：pass，连同核心检查共 75 项通过，覆盖内容采集、LLM 属性提取、运行时 tracing 和 Studio 观测；出现一条原有非法转义警告
+- 原有 exporter fixture 在退出时尝试向 `localhost:8000` 发送指标并记录 `UNAVAILABLE`，测试进程退出码为 0；新增缓存及云端导出集成测试使用本地内存替身，不访问云服务
+- Ruff 格式及检查、`git diff --check`：pass，覆盖修改的 Python 文件和差异
+- 文档：pass，两份 tracing MDX 均通过 `@mdx-js/mdx` 编译，已审查双语契约及配置一致性和相对文档链接
+- 提交前检查：pass，同步 `origin/main` 后再次运行 75 项针对性测试，并通过 `uv run --no-sync --extra dev pre-commit run --all-files`，包含 Ruff 和两项密钥扫描
+- 全量回归、文档站构建及类型检查：not_run，按当前流程仅执行针对性检查，未修改前端应用
+- 生产及云端验证：not_run，未要求访问或修改生产环境
+- UI 验证：not_applicable，无前端行为变更
+- 提交及 PR：已获授权；软件包发布及生产部署：not_run，不在本次范围内
+
+## CI 后续修正
+
+Python 3.12 的首次 CI 有 7023 项通过、2 项失败；失败来自两个运行时测试对私有 `_spans` 列表的切片假设。将这两个使用方改为通过 `get_finished_spans(session_id)` 读取会话快照，避免依赖队列类型或会因淘汰而变化的列表偏移，保留原有 span 及会话断言。本次修正不修改生产代码或已批准契约
+
+- 失败用例：修正前本地复现 2 项失败，修正后 2 项通过
+- 扩展检查：`uv run --no-sync --extra dev pytest tests/runtime/differential tests/runtime/piagent/test_piagent_turn_contract.py -q` 在修正后为 84 passed、2 xfailed、1 failed
+- 剩余失败：`test_model_fallbacks_warn_for_external_runtime` 的日志捕获断言为空；原始 PR 提交 `9671acd7` 下同样复现，该基线为 82 passed、2 xfailed、3 failed。此项不在本次缓存修复范围，未修改