[XiaoHongbo-Hope]

Purpose

Decode independent video payloads in parallel within LeRobot batches, preserving frame deduplication and output order.

• read.video.decode-parallelism defaults to 16, shared across all cameras in each Dataset worker. Set it to 1 for serial decoding.
• Applies to video windows without image_transforms. Per-camera caches retain active decoders until tasks finish; worker-local pools are reused across batches and rebuilt after fork/spawn.
• Public Dataset interfaces are unchanged.

Performance

Three-run medians on public LeRobot datasets, comparing shared-pool head 1b6f1b04ee with master 8c09824c06:

Dataset	Master	Shared 1	Shared 2	Shared 4	Shared 8
ALOHA cups_open	204.6	84.4	155.8	193.4	196.8
PushT	3,888.1	3,762.2	3,936.8	3,935.9	3,936.2
Koch pick_place_5_lego	183.3	102.6	199.3	252.7	273.7

Samples/s; local warm-cache CPU decoding on 64 vCPUs, batch 32, 8 workers, 8-frame windows, uint8, exact TorchCodec 0.10, Native reads, no transforms/model. Each run warms 10 batches, then times 100 batches for ALOHA/Koch or 500 for PushT. Configurations 1/2/4 rotate order; master and 8 were measured afterward on the same machine.

A separate 4/8/16 comparison on the same decoder implementation rotated configurations across three seeds:

Dataset	Local 4 / 8 / 16 (samples/s)	Direct OSS 4 / 8 / 16 (samples/s)
ALOHA	209.7 / 213.5 / 212.9	135.7 / 134.9 / 134.3
PushT	4,012.5 / 3,955.8 / 3,980.2	473.4 / 488.9 / 521.2
Koch	274.1 / 296.7 / 298.9	14.17 / 15.38 / 15.34

Local results and OSS ALOHA/PushT are three-run medians; OSS Koch has one diagnostic run per setting. Default 16 prioritizes throughput, with gains depending on the dataset. This also matches the 16-thread decode pool in the official lerobot-lancedb integration, shared across cameras within each Dataset worker. Local Koch uses 34.3 CPU cores and 10.55 GiB PSS at 16, versus 26.6 cores and 8.16 GiB at 4. Lower the option for resource-constrained workloads.

OSS first-batch waits at 4/8/16 are 37.3/38.5/35.2 s (ALOHA), 5.3/5.3/5.0 s (PushT), and 360.7/378.4/375.3 s (Koch). These measure the first next() only, excluding Dataset/iterator construction. Log-reconstructed times including warmup are approximately 68.2/70.6/67.7 s for 3,520 ALOHA samples, 45.3/45.5/42.5 s for 16,320 PushT samples, and 756.2/747.3/732.2 s for 3,520 Koch samples; initial monitor startup lag is excluded. More threads do not resolve slow OSS video initialization.

All comparisons force TorchCodec exact mode, not the default indexed/PyAV path; OSS/server caches are uncontrolled. The earlier ALOHA regression against master remains unresolved. These results do not establish a universally optimal default.

Tests

• Default-16 validation: 76 tests and 84 subtests passed; 29 optional-dependency tests skipped. Flake8 and git diff --check passed.
• Multi-camera limits, single-payload cameras, eviction, failure cleanup, fork/spawn, output isolation and Torch modes covered.
• All 45 benchmark runs completed; ordered IDs and sampled images match. Separate 16-batch full-tensor checks match across all five configurations for each dataset.

[JingsongLi]

Two issues found during review; please see the inline comments.

[JingsongLi]

[P2] Honor the decode limit on the fallback path

The new option is passed only to the optimized window path. With two video cameras, no image_transforms, num_workers=0, and torch.get_num_threads() > 1, that path returns {}, and _decode_video_rows still decodes cameras concurrently. I reproduced read.video.decode-parallelism=1 yielding two simultaneous decoder calls through to_dataloader(batch_size=1, num_workers=0). This violates the documented serial setting. Please apply the same decode budget to the fallback path and add a regression test for this case.

[JingsongLi]

[P2] Skip this test when the optional Torch dependency is missing

This unconditional import makes the new test fail with ModuleNotFoundError in environments without Torch. The python-tests dependency group does not include Torch, and the neighboring tests already skip when it is unavailable. Running this file without Torch gives 1 failed, 6 passed, 2 skipped. Please use the same missing-dependency skip guard here.