[Feature]: Add Python LMI cloud tests for lifecycle, concurrency and OpenTelemetry #742

What would you like?

Add an end-to-end cloud test suite that deploys the Python Durable Execution SDK to Lambda Managed Instances (LMI) and validates real service/runtime behavior, including multiple concurrent invocations in one execution environment, invocation deadlines, thread cleanup, checkpoint/replay, OpenTelemetry state cleanup, and X-Ray trace propagation.

This should provide cloud regression coverage for #741 (missing invocation deadline handling and unbounded branch cleanup), as well as a reusable LMI test deployment and runner. Local mocks cannot establish whether a timed-out invocation releases its runtime worker, whether an old attempt continues producing side effects, or whether other workers in the same environment remain healthy.

Observability validation scope: This issue also owns Python-specific OTel and X-Ray acceptance on deployed LMI, alongside the lifecycle coverage proposed in PR #743. Coordinate with JS #1017 and Java #727 under umbrella Java #798. Python's official LMI runtime handles one request at a time in each worker process; concurrent requests use separate processes. Validate sequential warm-process reuse and cross-process attribution rather than requiring simultaneous native invocations inside one Python process. The JS/Java shared-instance and invocation-local carrier defects are not assumed to apply to Python.

The repository already has:

• A cloud example workflow that builds the checkout, deploys with SAM, runs cloud-mode pytest, and produces JUnit summaries.
• A cloud test runner supporting real invocations, asynchronous starts, callbacks, and execution-history inspection.
• Example test fixtures that select cloud mode and resolve deployed function names.

The deployment/workflow inspected for the original report had no LMI capacity-provider configuration or LMI-specific assertions. PR #743 now proposes the dedicated lifecycle fixtures and deployment support; the observability acceptance added below requires explicit plugin-enabled scenarios and evidence.

Possible Implementation

1. Add an explicit LMI deployment target.

• Deploy functions with durability enabled, an LMI capacity provider, explicit PerExecutionEnvironmentMaxConcurrency, and a published version/qualified invocation target. Build and package the core SDK from the exact checkout being tested.
• Use Python 3.14 as the single LMI test runtime. Test environment concurrency 1 as a baseline and at least one explicit value greater than 1 (for example 2 or 4). Vary SDK map/parallel concurrency independently; it is a separate limit.
• Accept region, capacity-provider configuration, networking, runtime, and concurrency through the deployment/test configuration. Verify the resulting function version is active and actually uses LMI before running assertions.
• Make prerequisites explicit: the selected account/region/runtime must support the required LMI + durability configuration. Unsupported provisioning or unavailable capacity must be reported distinctly; never fall back silently to standard Lambda or count an unexecuted LMI scenario as passing.
• Use one persistent deployment for the complete Python 3.14 suite, following Java PR #728. Minimize functions by deploying only two shared handlers: one with native environment concurrency 1 and one with native environment concurrency 2. Reuse each handler for ordinary, deadline, replay, and cleanup cases with a common 60-second invocation timeout; wait for the actual service timeout in deadline tests. Do not duplicate functions per scenario or deploy separately per concurrency setting. Keep each function limited to one environment and avoid extra published versions. Use fixed stack/function/bucket names and create them only when absent; later runs update the same functions. Use per-run prefixes for data and serialize deployment/testing across runs. Before an update, verify persistent ownership tags and wait for earlier durable executions to finish. Failed stacks are retained for diagnosis; do not delete and recreate them automatically. Keep content-addressed code artifacts for updates and rollback.

2. Add the following scenarios.

Include the relevant lifecycle/progress guards from Java PR #728 using Python public APIs: root finally must complete before PENDING; two concurrent invocations of a shared decorated handler must progress independently; nested child/map/parallel operations must progress with single-worker branch pools; normal return and failure must settle in-flight work; and an abandoned child must reject a subsequent durable operation before executing its body. Python has no public shared-executor injection or stepAsync API, so use idiomatic equivalents and document that distinction.

For the #741 scenarios, land the infrastructure and failing regressions with explicit linkage to the fix. Any temporary expected failure must have a narrow condition, an issue reference, and strict unexpected-pass handling; it must remain visible in the report and be removed when the fix lands.

3. Make placement, timing, and side effects observable.

• Collect structured records containing run ID, execution ARN, request ID, environment identifier, per-process initialization identifier, PID, operation name, attempt, timestamp, and lifecycle phase. Use a documented environment identifier where available, or a test-only environment marker with safe cross-process initialization; PID alone is not globally unique.
• Keep infrastructure diagnostics outside the durable orchestration decision path. Use stable operation names. Perform test I/O and external barrier operations inside steps so the handlers still exercise the real public SDK model and remain replay-correct.
• Coordinate overlap and blocked I/O using a test-owned control endpoint or external store with explicit release signals and bounded emergency timeouts. Do not use arbitrary sleeps as the sole evidence of concurrency, and do not replace durable waits with blocking sleeps in ordinary workflow scenarios.
• Use before/after-wrapper diagnostics to distinguish user handler completion from SDK resource cleanup. For a wrapper that never returns, collect evidence from the driver/control endpoint and worker health probes. Post-timeout diagnostics must not depend solely on the timed-out invocation's response.
• Inspect real service execution history and checkpoint outcomes. Combine this with external side-effect records and runtime logs; absence of a late checkpoint is not proof that an old step stopped executing.
• Account for log/metric propagation delays with bounded polling. A run that cannot establish same-environment overlap or worker identity is inconclusive and must not pass its placement-dependent assertions.

4. Separate the three different timeouts.

Configure and report these independently:

• Lambda invocation timeout, which exercises LMI's non-terminating timeout behavior.
• Durable execution timeout, covering the logical execution across invocations.
• Test-driver/polling deadline, bounding how long CI waits.

The current cloud runner's result-wait timeout is a client-side polling limit, not a way to configure the deployed Lambda invocation timeout. Use actual function configuration and service/runtime evidence for timeout assertions. Give the driver enough additional time to observe cleanup/recovery after the invocation deadline without allowing an unbounded test run.

5. Integrate with CI and retain actionable artifacts.

• Add a dedicated LMI workflow/job using the existing repository's OIDC role, test-account verification, Hatch build/test commands, and cloud function mapping conventions. Preserve the current restrictions on privileged tests for forked PRs and Dependabot.
• Run the complete LMI e2e runtime/concurrency matrix automatically whenever a same-repository PR is opened, updated, or reopened (including Draft PRs), and on every push to main, including every merged change. Do not require labels, ready-for-review status, or path filters. Retain workflow_dispatch for manual reruns; do not add scheduled jobs. Preserve the existing privileged-test restrictions for forked PRs and Dependabot, while running the unprivileged harness checks on those PRs.
• Give every triggered commit isolated invocation, event, and control data while sharing the persistent functions. Serialize all LMI cloud jobs across PRs, main pushes, and manual runs with one repository-wide concurrency group that covers deployment, testing, diagnostics, and cleanup. Use cancel-in-progress: false and queue: max so pending jobs queue instead of replacing another commit's job; GitHub retains up to 100 pending jobs and cancels additional arrivals once full. Keep harness jobs independent. Preserve the existing provider capacity; do not increase MaxVCpuCount. This group only coordinates jobs in this repository, so manual local deployments and other repositories using the provider still need capacity coordination.
• Run the existing 28 lifecycle cases and the added observability cases in one serialized cloud job against that deployment, preserving both native concurrency configurations. The lifecycle case count does not establish OTel coverage. After each case's assertions, release its controls, stop remaining logical executions, and verify all observed wrappers exit before reusing its function. Cleanup must not turn a deadline/recovery regression into a pass. If case retirement fails, the next case must retire the outstanding work before invoking the shared function.
• Upload JUnit results and a concise summary, deployed SDK commit/version, runtime and LMI configuration, qualified function targets, execution histories, lifecycle/placement records, side-effect ledger, and relevant CloudWatch logs/metrics. Failure output should identify the scenario and affected requests/workers.
• Bound each scenario and the overall job. Run cleanup releases only that run's blocked fixtures, stops its remaining test executions, and waits for observed wrappers to exit; stopping a durable execution alone is not proof that LMI code stopped. Keep the persistent functions, stack, bucket, and code artifacts after successful, failed, or canceled tests. Use isolated runs/RUN_ID/events/ and runs/RUN_ID/control/ prefixes with one-day expiry; code artifacts do not expire automatically. Infrastructure retirement remains under the test-account owner's explicit lifecycle policy.
• Ensure failures and cancellations still produce diagnostics and run-scoped cleanup. Remove the automatic infrastructure janitor. After an interrupted run, wait for prior executions before updating; if they do not settle within the budget, fail setup without changing code. Re-running the suite must reuse the same two functions and avoid cross-run test-state collisions.

6. Validate Python OpenTelemetry and X-Ray behavior on deployed LMI.

Exercise both Invocation and Execution OTel views through the real public Python durable-handler decorator and supported plugin registration, with an actual OTel provider/exporter. Build and install compatible core/OTel wheels from the intended revisions and record their versions and checksums. Reuse the two persistent Python 3.14 functions at native concurrency 1 and 2, adding dedicated observability scenarios and assertions. Keep the plugin-free lifecycle controls for failure attribution.

• Warm-process reuse and cleanup. Prove that sequential invocations reuse the same process using environment identity, PID and a per-process initialization identifier. Run distinct executions with the same durable operation names/sequences and different execution/trace identities. After success, failure, suspension/resume and retry, verify correct span attribution, parentage, duration/status, context cleanup and documented replay behavior. A later invocation must not inherit the previous invocation's execution ARN, trace context, span maps or ended state. Observe wrapper boundaries and actual export records; receiving a handler result alone is insufficient.
• Concurrent workers and late activity. At concurrency 2, prove overlapping requests in distinct Python worker processes within the same execution environment. Check that exported spans remain attributable to the correct request/execution and that healthy-peer telemetry remains usable while a victim times out or enters slow cleanup. Track delayed SDK work and exports by the owning request; they must not relabel another invocation or corrupt subsequent warm reuse. A blocked worker may never become reusable, so report its retained/replaced process explicitly instead of assuming a later request used the same process.
• Runtime X-Ray carrier and sampling. Python LMI supplies the process-local _X_AMZN_TRACE_ID environment carrier consumed by its OTel integration. On deployed LMI, correlate the genuine runtime header at invocation entry with the documented Root, Parent and Sampled=0/1 behavior in both views, at concurrency 1 and 2 and across real resume/retry. Reused processes must use the current invocation's carrier rather than a cached prior header. Verify each view's documented trace/replay semantics; do not require the JS/Java InvocationInfo.xRayTraceId API or mutate the environment header in native-runtime tests.
• Carrier compatibility controls. Cover missing, empty and malformed headers, supported custom extractors, configured fallback sampling and ordinary-Lambda compatibility using public-wrapper/installed-wheel fixtures. Label injected carrier cases separately from genuine deployed-runtime cases. A missing span is not automatically a pass: distinguish an expected unsampled decision from failed export or incomplete collection.
• Plugin, provider and exporter failures. Reuse process-lifetime providers/exporters according to the supported Python registration model and verify that invocation cleanup leaves shared resources usable by the next invocation. Exercise bounded plugin failures, exporter errors/stalls, flush waits and actual invocation timeout. Check execution/trace attribution and the documented failure, cleanup and retry behavior. Keep [Bug]: Missing invocation deadlines and unbounded branch cleanup can pin LMI workers #741's non-cooperative user I/O, synchronous service-call and blocked-hook limitations visible; passing telemetry checks cannot establish worker reclamation. Do not assume forced thread termination, unconditional delivery after timeout or exactly-once export.
• Reproducible evidence. Retain tested commits and installed artifact identities, function/version/configuration readbacks, environment/process/request/execution IDs, runtime carrier observations, exported spans or trace-query results, lifecycle ledgers, JUnit, durable histories and run links. Use bounded collection and report provisioning/placement, SDK/plugin assertions, sampling, export failure and missing evidence distinctly. For behaviors with a known failing baseline, retain a pinned negative control; deterministic bad-attribution/stale-context fixtures can validate assertion sensitivity without claiming a previously unreported production defect.

Use the real public SDK operation paths and retain checkpointed successes/failures, stable operation identities and expected side-effect counts through replay. Required cloud cases remain unverified until their genuine-runtime evidence passes; local stubs, ordinary-Lambda OTel conformance, or the existing 28 lifecycle cases alone cannot satisfy this added scope. Keep this issue open until the original lifecycle requirements and the observability acceptance below are complete.

Acceptance criteria:

• The full LMI cloud matrix runs automatically on every same-repository PR update (including Draft PRs) and every push to main, without labels or path filters; fork/Dependabot credential restrictions remain explicit and another commit cannot replace a pending run.
• At most one LMI cloud job from this repository holds provider capacity at a time, through cleanup; pending jobs use the GitHub maximum queue without changing provider capacity.
• One persistent deployment with only two shared LMI functions runs every scenario at native concurrency 1 and 2. Consecutive runs update the same names/ARNs; tests never delete the functions, stack, bucket, or code artifacts.
• CI can deploy the checkout as durable functions on a verified LMI target and invoke a published version.
• The suite uses Python 3.14 only, with a concurrency-1 baseline and concurrency > 1; unsupported provisioning is explicitly reported.
• A test proves overlapping invocations in different Python processes within the same execution environment.
• Deadline and early-completion scenarios reproduce the failures tracked by [Bug]: Missing invocation deadlines and unbounded branch cleanup can pin LMI workers #741 and become passing regression tests when its cancellation/cleanup contract is implemented.
• Worker recovery and unaffected concurrent work are verified without autoscaling or unrelated placement masking the result.
• Success, failure, suspension, retry, replay, and nested branch cleanup preserve checkpoint semantics and have observable resource-lifetime assertions.
• Reports distinguish invocation timeout, execution timeout, driver timeout, assertion failure, and provisioning/placement failure.
• No test can remain blocked indefinitely; CI failure/cancellation run cleanup, persistent create/update, prior-execution waiting, and cross-run isolation are exercised.
• A documented command allows maintainers to run the suite against the configured test account and inspect retained artifacts.
• Both OTel views pass plugin-enabled deployed-LMI tests using compatible installed Python core/OTel artifacts, with the exact versions and hashes recorded.
• Sequential warm-process reuse is proven by process identity, and success/failure/suspension/resume/retry leave no stale execution or trace state in later invocations.
• Concurrency-2 tests prove distinct worker processes in the same environment and preserve attribution and healthy-peer telemetry during victim timeout/slow cleanup and delayed activity.
• Genuine runtime _X_AMZN_TRACE_ID values preserve the documented Root/Parent/Sampled behavior at concurrency 1 and 2 and across resume/retry; missing/empty/malformed and fallback cases are identified as native or controlled fixtures.
• Plugin failure, process-lifetime provider/exporter ownership, bounded flush/error scenarios and actual timeout are covered under the supported Python contract, with [Bug]: Missing invocation deadlines and unbounded branch cleanup can pin LMI workers #741 worker-reclamation limits reported separately.
• Passing observability run links and correlated span/trace, process and deployment evidence are attached here; collection gaps, expected unsampled output, export failures and regressions are distinguishable.

Is this a breaking change?

No. This adds test infrastructure, fixtures, and CI coverage. Any core SDK behavior changes belong to #741 or separate implementation issues.

Does this require an RFC?

No for the test harness itself. Any new runtime cancellation/worker-retirement contract discovered while implementing #741 should be designed and reviewed separately; these tests should validate that agreed contract.

Additional Context

• Related core SDK bug: [Bug]: Missing invocation deadlines and unbounded branch cleanup can pin LMI workers #741
• Python runtime for Lambda Managed Instances: each worker process handles one invocation at a time; process concurrency differs from SDK branch-thread concurrency.
• LMI execution-environment lifecycle: no freeze between invocations and no forced code termination at invocation timeout.

The initial evidence behind #741 was local source inspection and reproducible mocked-service tests. PR #743 now records the real-LMI validation, including the newly discovered late-operation defect added to #741. Runtime-specific tests should initially live in this Python repository. Any new language-neutral conformance requirements or shared runner changes should be coordinated with the conformance repository separately.