From c3ec3385cc81fb46f2486cc2f783a44fd48e21b0 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 17:10:38 +0000 Subject: [PATCH 01/33] docs(queue): refresh transactional outbox implementation plan Reconcile the outbox design with the current queue, database, coroutine, Horizon and telemetry implementations while preserving opt-in queue APIs and the unchanged ordinary publication path. Specify exact transaction-session ownership separately from relay source coverage, writer-owned staging, and post-insert rollback snapshots. Add framework prerequisites for context-safe dispatch-lock release, SQS bulk lifecycle cleanup and single-pass Horizon preparation. Require owner approval before changing role-qualified transaction-record compatibility. Use bigint epoch deadlines and available_at lease expiry, bounded retry-bucket updates, existing Redis bulk Lua publication and normal queue forwarding. Keep concurrent relay resource scopes and operational commands while removing the configuration holder, redundant validators and speculative diagnostics. Refresh schema, integration boundaries, documentation, test coverage and benchmark requirements. Keep private integrations outside the public plan and avoid duplicate tests or fixed optimizer-plan assertions. Validation: source-backed design review and full plan self-review completed; staged whitespace checks passed. Documentation only; no implementation tests or benchmarks run. --- ...pervel-transactional-queue-outbox-codex.md | 563 ++++++++++++++++++ 1 file changed, 563 insertions(+) create mode 100644 docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md diff --git a/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md new file mode 100644 index 000000000..52d8f55eb --- /dev/null +++ b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md @@ -0,0 +1,563 @@ +# Hypervel Transactional Queue Outbox + +## Status and outcome + +Implement an optional transactional outbox in `hypervel/queue`. A job explicitly or conventionally routed through the outbox is serialized once, stored with the business write on the authoritative database transaction, and relayed to its existing queue connection after commit. This closes the database-commit / broker-publish crash window without replacing Queue, Bus, Horizon, queue drivers, or application job APIs. + +The completed feature must provide: + +- Laravel-style `viaOutbox()` / `withoutOutbox()` ergonomics plus a `ShouldQueueViaOutbox` marker and per-connection default; +- correct behavior when applications switch among Redis, SQS, Beanstalkd, database, sync, background, deferred, null, fake, and pooled connections; +- set-based outbox inserts for `Bus::bulk()` / batch work, using only the chunks required by documented database binding limits; +- horizontally scalable, lease-based relay workers with database-engine-specific claim paths; +- a staging/relay pipeline that preserves the original payload UUID, delay, SQS FIFO options, SQS overflow behavior, normal dispatch lifecycle, and Horizon metadata/events; +- an explicit at-least-once guarantee, including safe treatment of ambiguous transport outcomes; +- no outbox database work, service resolution, payload re-encoding, or additional outbox event allocation when the feature is not selected; +- production operations for monitoring, retrying, and deliberately forgetting pending rows; +- no dependency on Horizon and no third-party dependency. + +The result is a queue feature, not a generic event bus, inbox/deduplication system, workflow engine, CDC pipeline, or completed-message archive. Those concerns must not be smuggled into the implementation through aggregate metadata, status histories, a second dead-letter table, container swapping, or transport-specific application APIs. + +## Verified baseline + +### Hypervel + +- `Queue::enqueueUsing()` is the common single-job seam. `enqueueNow()` guards payload finalization, `JobQueueing`, and the driver callback as one attempt; accepts dispatch locks only after storage/transport acceptance; and raises `JobQueued` afterwards. Non-cancellation attempt failures raise `JobQueueingFailed`, while `PendingDispatch` owns final cleanup before a queue accepts the job. +- `DispatchLockContext` is the existing object-keyed `WeakMap` for dispatch-owned unique/debounce lock provenance. It carries the live cache repository needed for exact cleanup only while the job object remains live, exposes Laravel-compatible scalar unique metadata to payload hooks, and is cleared on acceptance/release. `Queueable::$uniqueLockOwner` is initialized to Laravel's empty-string default and owner-capable locks release through `UniqueLock::releaseOwned()`. +- Database, Redis, and SQS have optimized `bulk()` implementations; the base implementation loops through `push()` / `later()`. `Bus::bulk()` groups jobs by queue connection and queue name. `Batch::add()` already calls queue `bulk()` inside the batch repository transaction. Database and Redis bulk paths already provide bounded, atomic storage/publication. +- `DatabaseTransactionsManager::callbackApplicableTransactions()` exposes coroutine-local transaction records. Its current base-name indexing conflates distinct role connections; complete the exact-owner prerequisite below before using these records to select outbox storage. Matching host/database configuration never proves atomicity. +- `PdoConnection::lockForPopping()` and `maxBindings()` own live database capabilities and cache them only for the current physical session. MySQL/MariaDB/Vitess/PlanetScale, PostgreSQL, and SQLite supply their correct version-dependent values. `DatabaseQueue` already consumes both, and `Concerns\InsertsDatabaseRows` provides binding-safe, transactionally atomic chunked inserts. +- A database queue insert already participates atomically when its queue database connection is the same active connection as the business transaction. Adding a second outbox table in that case would add latency without improving the guarantee. +- `Queueable` already carries tri-state `afterCommit`, connection, queue, delay, chain, SQS message-group/deduplication, unique, and debounce state. `PendingDispatch`, `PendingBatch`, and `PendingChain` are the expected fluent surfaces. A valid batched job is only required to use `Batchable`, so batch-wide outbox state cannot assume that every job also uses `Queueable`. +- Final queue payloads already contain a stable job UUID. Hidden payload context carries the unique-job cache store, key, and owner; debounce ownership remains on the serialized job. Existing rollback cleanup depends on an unaccepted dispatch record; accepted transactional storage needs an independent release snapshot. +- `QueuePoolProxy` reacquires a pooled queue for after-commit dispatch rather than retaining a borrowed network client across a transaction. Raw outbox publication must likewise borrow once per batch and release immediately. +- `ConnectionResolver::releaseIdleConnections()` can return idle sessions early, but preserves pinned and certain multi-server write sessions and does not cover all custom drivers or listener-owned resources. Keep separate short-lived relay phase coroutines for guaranteed cleanup; committing a transaction alone does not provide that guarantee. +- SQS bulk delivery already chunks at 10 entries and the request byte cap, handles explicit per-entry success/failure, preserves FIFO options, and retains overflow pointers for ambiguous SDK failures. Reuse those rules rather than building a parallel SQS implementation. +- `JobQueued::$id` already matches the Queue contract's `mixed` result, and raw Sync/Deferred/Background publication executes through each queue's intended timing behavior. +- Horizon already separates `JobPayload::prepareForQueueing()` from `markAsPushed()` while retaining `prepare()` as their composition. Raw retry preserves stored classification, and delayed publication stamps `pushedAt` at the actual push. Combined preparation currently encodes twice; remove that redundancy in the prerequisite below. +- Generic Worker lifecycle callbacks and deferred signal callbacks already run through short-lived coroutine contexts, and worker delays use fakeable `Sleep`. The outbox worker must follow the same resource-ownership pattern. +- Queue, Bus, and Horizon READMEs already follow the required order, and the application skeleton's queue configuration matches Foundation. Reuse the existing foundation instead of repeating completed cleanup. + +### Reference findings + +- Reference checkouts live under `/home/binaryfire/workspace/references/`. `laravel-outbox` demonstrates bulk storage and `SKIP LOCKED`, but its request-wide mutable collector and temporary replacement of container `events` / `queue` bindings are unsafe under concurrent Swoole coroutines. Its aggregate IDs, payload HMAC, status/history fields, completed-row retention, second dead-letter table, and processor-job bootstrap are unnecessary for a native queue outbox. +- Laravel framework (`laravel/framework`) and Horizon (`laravel/horizon`) provide the public queue/batch/chain naming baseline; Hypervel's optimized bulk and transaction behavior is authoritative where it differs. +- Current Laravel's `Queueable`, `UniqueLock`, and hidden unique-job context carry the cache lock owner. Reusing that upstream behavior is safer and simpler than inventing outbox-only ownership state. +- MassTransit and NServiceBus persist the finalized outgoing transport operation and relay it later. Both confirm that the broker-send / outbox-delete boundary remains at-least-once and receivers must be idempotent. +- Watermill Forwarder / SQL confirms the small two-part shape: a transactional SQL publisher plus a daemon that forwards the stored envelope; its SQL publisher accepts multiple messages in one insert. +- Gruelbox Transaction Outbox validates bounded batches, retry scheduling, and skip-locked claiming. Eventuate validates CDC as an alternative deployment architecture, but CDC/Kafka infrastructure is unjustified for a framework queue primitive. +- TickerQ does support batch additions; it is not evidence against bulk APIs. Its scheduler concerns do not belong in this feature. +- PostgreSQL and MySQL both explicitly document `SKIP LOCKED` as suitable for multiple consumers of queue-like tables. SQLite remains a correct serialized fallback. + +Primary upstream reading: + +- +- +- +- +- +- +- +- +- +- +- +- +- + +## Framework changes + +### Independently useful prerequisites + +Implement these at their owning boundary, before outbox integration. If already completed when implementation starts, verify their contracts and reuse them rather than repeating the work. + +1. **Exact transaction identity.** Use the resolved connection object's `getNameWithReadWriteType()` consistently for transaction begin/commit/rollback/reset, applicable-transaction checks and connection-bound callbacks. Do not change `getName()` or derive identity from the caller's requested suffix: shared in-memory SQLite aliases return one owner object. Update `ManagesTransactions`, `Connection::resetTransactionState()`, Foundation testing's wrapping-transaction lookup and `RequestContextSynchronizer`, Pennant's transaction-record matching, and other consumers comparing record names. Distinct base/`::write` sessions must not consume each other's records or callbacks. **Owner approval before implementation:** role-qualified transaction-record names change values observable by extensions compared with Laravel's base-name records. Explain the correction and required exact-owner comparison in database compatibility documentation; preserve public signatures. Plan review does not replace approval for this or any other Laravel API departure. +2. **Context-safe dispatch-lock cleanup.** Add the supported boot registration method `DispatchLockContext::captureReleaseContextUsing(?callable $callback): void`, normalized to closures; registration appends, `null` clears. Its callbacks return a release wrapper or null, capturing only the context they need. Internal `captureReleaseContext(): ?Closure` composes wrappers in registration order, first outermost. Call it in `UniqueLock::acquireForDispatch()` and `DebounceLock::acquireForDispatch()` **before acquiring anything**, then pass its result into their internal registration methods. A throwing capture callback leaves no acquired lock/token. Unique and debounce acquisition are mutually exclusive, so capture once per acquisition. Records and snapshots retain the composed wrapper; release runs both existing owner-aware cleanup operations inside it, preserving existing error handling. Wrappers invoke the operation once and restore their prior context in `finally`. Without registrations return null immediately; without a captured wrapper run existing release code without allocating a release closure. Keep all record/capture operations internal, document the registration method's boot lifetime and callback contract in PHPDoc, and clear callbacks in the existing `flushState()`/test-subscriber path. Do not add a contextual-store interface, wrapper traversal, full coroutine-context snapshot, or cache implementation changes. +3. **Transactional storage rollback.** Immediately after a successful database-queue/outbox insert and before accepting dispatch locks, register an independent `DispatchLockContext::snapshot()` on the exact storage connection if its `transactionLevel() > 0`. Rollback calls `releaseSnapshot()` even after acceptance erases the live record. Apply to applicable single/bulk paths; after chunk-insertion savepoints complete, attach to the remaining enclosing transaction that can still undo the rows. Failed inserts and already-durable no-transaction staging register nothing. Existing deferred ownership remains intact; never release broker-accepted locks because a business transaction rolls back. Complete identity and capture-context fixes first. +4. **SQS batch lifecycle.** Move payload finalization and `JobQueueing` into the guarded unaccepted-attempt lifecycle in `sendBatchedMessages()`. Finish producer state for affected unaccepted messages on non-cancellation failure; preserve already confirmed chunks and never turn a later `JobQueued` listener error into rejection of accepted work. Follow the established Redis/database batch and cancellation behavior. +5. **Horizon combined preparation.** Make `JobPayload::prepare()` perform one encoding operation for classification plus `pushedAt`, preserving `prepareForQueueing()`, `markAsPushed()`, stored classification on raw retries, and supported public/protected customization. Do not alter ordinary publication timing. + +### Outbox integration + +Keep the new mechanism under `Hypervel\Queue\Outbox`; use small integration points in existing packages: + +- Bus/Foundation: add the marker, shared fluent state, pending-dispatch/batch methods, and wrapper-job propagation described below. +- Queue: add selection and payload/option hooks, outbox services and commands, config, and the migration generator. Stage through the existing guarded enqueue lifecycle and reuse `DispatchLockContext`, `PdoConnection` capabilities, and `InsertsDatabaseRows`. +- Queue drivers/proxies: implement the fast paths and raw-batch capability below; select/partition outbox work before SQS's direct FIFO-delay validation. Preserve direct validation, existing raw publication APIs, and direct overflow retention; isolate outbox overflow pointers by claim attempt. +- Horizon: classify staged payloads through the outbox-only hook and stamp them at relay publication through the existing split `JobPayload` lifecycle. The outbox path must not use Horizon's direct-push `CoroutineContext` handoff. +- OpenTelemetry: consume guarded outbox delivery events without adding work to ordinary dispatch. +- Foundation/application skeleton/docs: add matching outbox configuration and migrations, then add the canonical guide and concise links without revisiting completed baseline cleanup. +- Doctor remains unchanged: do not copy runtime selection rules into a second setup detector or add relay-heartbeat machinery. Explicit setup, migrations, native storage failures, and outbox commands provide the required operating surface. + +## Decisions + +### Ownership and guarantee + +- Implement under `src/queue/src/Outbox`; the migration stub/command live with Queue, while the shipped application defaults live at their existing owner, `src/foundation/config/queue.php`. Horizon only adapts actual Redis publication. The feature works with Redis whether Horizon is installed or not. +- Outbox atomicity covers only writes on the selected database transaction. It cannot atomically cover another database, an email, filesystem/object-store write, HTTP call, or other external effect. +- Relay is at-least-once. If the target accepts a job and the relay crashes or cannot delete the row, the same job payload can be published again. Stable job UUIDs are retained; application handlers that cannot tolerate duplicates must be idempotent. +- Dispatch order is best-effort: concurrent claims, delays and retries can reorder publication. SQS FIFO preserves broker-arrival order within a message group, not the order application transactions staged their jobs. Do not add per-group sequencing machinery. +- An outbox is not an inbox. Do not claim exactly-once processing or add receiver deduplication to this work. +- Successful rows are deleted. Failed/ambiguous rows remain in the same table for retry. Do not retain completed rows, introduce statuses that duplicate nullable lease state, or add an automatic terminal dead-letter cutoff that can silently abandon a committed business action. + +### Selection and precedence + +Add the marker as an ordinary queued-job contract. Put the fluent state in a small `InteractsWithQueueOutbox` trait used by both `Queueable` and `Batchable`, because Laravel-compatible batches do not require `Queueable`: + +```php +interface ShouldQueueViaOutbox extends ShouldQueue +{ +} + +// InteractsWithQueueOutbox; intentionally uninitialized until explicitly set. +public ?bool $viaOutbox; +public ?string $outboxDatabase; + +public function viaOutbox(): static; +public function withoutOutbox(): static; +public function onOutboxDatabase(UnitEnum|string $connection): static; +``` + +Framework code reads explicit state with `$job->viaOutbox ?? null` and `$job->outboxDatabase ?? null`. PHP serialization omits uninitialized typed properties, so ordinary jobs gain no payload fields or bytes. `onOutboxDatabase()` requires a non-empty resolved enum/string and also selects outbox delivery; `withoutOutbox()` unsets any selected database. Keep this state in `InteractsWithQueueOutbox`, used by `Queueable` and `Batchable`, so Laravel-compatible batch jobs that use only `Batchable` can participate. Do not add chain-specific outbox properties or `all*` methods: follow-up chain dispatch happens after the current handler completes, outside the business transaction, and the current queued job already supplies durable retry if that dispatch fails. Explicit Laravel-shaped properties are easier for jobs, wrappers, serialization, and agents to understand than a framework-only associative state bag. + +Selection precedence is: + +1. explicit `withoutOutbox()` / `viaOutbox()` on the job; +2. `ShouldQueueViaOutbox`; +3. the target queue connection's `outbox` boolean; +4. direct existing queue behavior. + +`afterCommit()` / `beforeCommit()` and outbox selection remain independent. They control when a direct queue publish occurs; an outbox-selected dispatch inserts its durable row immediately inside the owning transaction, and delivery necessarily follows commit. `withoutOutbox()` is the explicit opt-out when a marker or connection default would select the outbox. No combination throws merely because an application changed queue drivers or commit defaults. + +Mirror the current-job fluent methods on `PendingDispatch`. Add `viaOutbox()`, `withoutOutbox()`, and `onOutboxDatabase()` to `PendingBatch`; these are aggregate defaults and never overwrite a job whose corresponding property is initialized. Persist only the exact `viaOutbox` / `outboxDatabase` option keys on a batch. `Batch::add()` applies them to every job it is about to pass to `bulk()`, including jobs added after initial dispatch. Do not add outbox methods to `PendingChain`, propagate outbox state to later chain members, or add a second `bulkViaOutbox()` API. Normal `Bus::bulk()` honors job/batch/connection selection and is the optimized path. + +Examples: + +```php +ChargeInvoice::dispatch($invoice)->viaOutbox(); + +Bus::batch($jobs) + ->onConnection('redis') + ->viaOutbox() + ->dispatch(); +``` + +The dispatch return value for a staged single job is the finalized payload's job UUID, also indexed on the outbox row for operator lookup. Existing direct drivers retain their existing return values; bulk remains `null`. + +### Laravel-compatible surface + +- Do not change signatures or ordinary semantics of `dispatch`, `push`, `later`, `bulk`, `afterCommit`, `beforeCommit`, batch, chain, Queue/Bus fakes, or the `Hypervel\Contracts\Queue\Queue` contract. `pushRaw()` remains a direct, non-staging escape hatch even when a connection default enables outbox; the relay uses the separate raw-batch SPI when available and `pushRaw()` as its generic fallback, avoiding recursion and preserving existing raw-payload behavior. +- Add fluent methods to the same traits/builders that own their neighboring Laravel APIs; do not add an Outbox facade, transaction wrapper, alternate dispatcher, or environment-specific call site. +- Propagate outbox selection and the selected database from queued listeners, mailables, notifications, and broadcasts into their existing queue wrapper jobs with one assignment each beside the independent `afterCommit` propagation. Copy initialized tri-state state exactly; otherwise promote `ShouldQueueViaOutbox` to wrapper `viaOutbox = true`. Do not add producer-specific outbox APIs. +- Keep application-visible names short and intention-revealing: `viaOutbox`, `withoutOutbox`, `onOutboxDatabase`, `ShouldQueueViaOutbox`, `queue:relay`, and the `queue:outbox` operator command family. +- Add protected enqueue/destination/options hooks on the base queue and an optional raw-batch contract for optimized durable drivers. Database/local/failover queues override the smallest cohesive hook for their fast path; null already bypasses enqueueing. Do not build a central driver-name/type switch. Custom drivers retain existing behavior while base-lifecycle drivers gain staging and may opt into optimized publication. +- Preserve Horizon's existing public `JobPayload::prepare($job)` method as the composition of its classification and actual-push operations. Use the narrower existing operations for outbox staging/publication without breaking the Laravel-compatible method. + +### Driver behavior + +| Target at dispatch time | Selected outbox behavior | +|---|---| +| Redis, SQS, Beanstalkd, custom durable driver using the base enqueue lifecycle | Stage on the owning SQL database, then relay | +| Database queue, no active business transaction | Push directly; the jobs table is already durable | +| Database queue on the sole or explicitly selected active owning connection/session | Push directly inside that transaction; already atomic | +| Database queue on another connection while a business transaction is active | Stage on the business database, then relay to the database queue | +| Sync, background, deferred | Preserve local semantics; execute after successful commit when one exists, otherwise normally; an explicit database selects that exact active transaction; document that no crash durability is possible | +| Null | Preserve the no-op behavior and release only this dispatch's owner-token-matched unique/debounce locks | +| Queue/Bus fake | Record the job in the fake only; never touch an outbox table | +| Failover | Preserve normal failover, but defer the whole attempt until applicable transactions commit when outbox is selected; document that this degradation cannot provide crash durability | + +Failover and outbox solve different policies, and failover may cross heterogeneous drivers whose wire payload/options are created only in the selected child. Do not invent a multi-destination outbox envelope. Instead, selected outbox behavior degrades to failover's existing all-applicable-transactions after-commit path, then performs ordinary child attempts. `FailoverQueue` always invokes children inside a coroutine-scoped outbox-suppression context, so a child's own connection default cannot stage successfully and prevent later fallbacks from being tried. The failover connection's explicit/marker/config selection is authoritative; child outbox defaults still apply when those children are used directly. The base selection path consults this context only after it would otherwise select outbox, so ordinary unselected queue dispatch pays no context lookup. + +Driver switching never causes an exception merely because the selected driver is local, failover, or already durable. There are no environment checks; behavior follows runtime capability. Local/failover drivers register against the explicitly selected active transaction when supplied and reject only a false claim of atomic ownership (an explicit owner while transactions exist only on other connections); without an explicit owner they preserve their existing transaction-selection rules. + +A selected outbox keeps an SQS FIFO job staged until `deliver_at`, then publishes it without the forbidden per-message `DelaySeconds`. In `later()` and `bulk()`, select/partition outbox work before applying the existing FIFO checks to direct work. Capture `getQueueableOptions()` without a delay argument at staging; preserve its direct-call validation. No-transaction staging has the same delayed-publication behavior. + +A custom durable driver participates when it routes `push()` / `later()` / `bulk()` through the base enqueue hooks. The relay uses the optional raw-batch publication contract below when available and otherwise publishes due rows sequentially through the existing `pushRaw()` contract. Existing custom queues that override the public enqueue methods and bypass the base lifecycle retain their existing direct behavior; intercepting arbitrary overrides would require a permanent decorator and hot-path overhead for every queue operation. Document this extension contract instead of adding that machinery. + +Evaluate fake/local/failover and already-atomic database fast paths before resolving outbox services or touching the outbox table. These paths need no outbox migration merely because a job carries the marker; local/failover drivers consult active transactions for their documented after-commit fallback. `NullQueue` already bypasses enqueueing and `PendingDispatch` releases its unaccepted locks; add no null-driver hook. + +### Database ownership + +Configure each database source whose outbox table the relay drains; use configured connection names, not both a base name and its `::write` alias: + +```php +'outbox' => [ + 'databases' => [ + env('QUEUE_OUTBOX_DATABASE', env('DB_CONNECTION', 'sqlite')), + ], + 'table' => env('QUEUE_OUTBOX_TABLE', 'job_outbox'), + 'retry_after' => (int) env('QUEUE_OUTBOX_RETRY_AFTER', 300), +], +``` + +Every shipped connection record includes `'outbox' => false`. Base `Queue::setConfig()` caches the optional, already-cast value in a typed flag, using `false` when an existing custom record omits it; `QueueManager` continues passing configuration once. On lazy service resolution, inject typed config values into their owning services through the provider, following the existing batch/failed-job providers; no `OutboxConfiguration` class. Validate a non-empty database-source list and positive `retry_after`. Native failures cover invalid table/connection configuration; do not add duplicate-name validation. Relay batch size is a worker option, not duplicate config. Retry delays are fixed at `OutboxRelay::RETRY_DELAYS = [1, 5, 15, 30, 60, 300, 600]`; staging chunks follow database binding limits, not another knob. + +Resolve the owning database as follows: + +- one applicable active transaction: use that connection; an explicit different database is an error; +- multiple active transaction connections: require `onOutboxDatabase()` naming one of them; otherwise throw an exception that lists the active connections and names the required method. Document that only the explicitly selected transaction is atomic with the job; +- no active transaction: use the explicit database or the first configured database; +- whenever SQL staging is required, its source must be covered by `databases`, so the relay cannot silently ignore the rows. For base/`::write` transactions, compare the resolved owner's `getName()` with the configured source names, while inserting and registering rollback on the exact `getNameWithReadWriteType()` session. + +Nested records on one resolved connection count as one owner. Explicit selection must identify an applicable owner when transactions exist; otherwise it selects configured storage. Resolve transaction identities from objects, not requested suffixes or host/database comparisons. `onOutboxDatabase('mysql::write')` selects that exact active session while `databases => ['mysql']` covers draining its table. Do not list both as sources: that would duplicate polls/counts and make UUID lookup falsely ambiguous. Shared in-memory SQLite aliases remain one owner. Do not borrow an unrelated base connection merely to inspect an alias. Install the migration on each configured source; avoid listing different names for the same physical outbox table rather than adding topology detection. Workers visit sources round-robin or pin one source through their database argument. + +SQL outbox staging is writer-owned. During preflight, before finalization or insertion, reject a resolved owner for which `ConnectionName::parse($owner->getNameWithReadWriteType())->isRead()` is true, including explicit no-transaction selection. Name that connection in the error and direct the caller to its base/`::write` connection; existing pre-acceptance cleanup releases dispatch locks. Do not probe endpoints or exempt read aliases without separate read configuration. Shared in-memory SQLite resolves to its base owner and remains supported. This rule governs SQL staging, not local after-commit fallback or ordinary unselected queue APIs. + +Do not add a dynamic shard/source registry. Configured database connections are the complete, explicit set of sources the relay drains. + +## Storage model + +Generate with `make:queue-outbox-table` (alias `queue:outbox-table`) using the configured table name. `OutboxTableCommand` extends `MigrationGeneratorCommand` with normal duplicate-table detection; no combined migration needs an override. Commit the same schema in `src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php` and `/home/binaryfire/workspace/contrib/hypervel/hypervel/database/migrations/0001_01_01_000009_create_job_outbox_table.php`; do not maintain a third test-only migration copy. + +```php +Schema::create('{{table}}', function (Blueprint $table) { + $table->id(); + $table->uuid('uuid')->unique(); + $table->string('connection'); + $table->string('queue')->nullable(); + $table->longText('payload'); + $table->text('options')->nullable(); + $table->unsignedBigInteger('deliver_at')->nullable(); + $table->unsignedInteger('attempts')->default(0); + $table->unsignedBigInteger('available_at'); + $table->uuid('lease_token')->nullable(); + $table->text('last_error')->nullable(); + $table->unsignedBigInteger('created_at'); + + $table->index(['available_at', 'id']); +}); +``` + +- Bigint `id` keeps the hot index and range/order operations compact. `uuid` repeats the finalized payload's existing job UUID for indexed logs and operator commands; do not generate a second outbox identity. A fresh `lease_token` identifies only one claim attempt. +- All three time columns are integer Unix seconds supplied from the framework clock, following queue/cache operational deadline conventions. Bigint avoids PostgreSQL's signed 32-bit epoch limit. No database-clock default or timezone-dependent parsing: SQL session timezone and later `app.timezone` changes cannot move a staged instant. Keep message fields as `int`/`?int`, converting only for display. Preserve `InteractsWithTime::availableAt()`'s future ceil-to-second; already-due input becomes `deliver_at = null` with initial eligibility now. +- `deliver_at` is immutable intended delivery; `available_at` is next eligibility, including lease expiry and retry backoff. Optimized drivers compute non-negative remaining delay immediately before each transport chunk. Redis/database initially qualify now; standard SQS at `max(now, deliver_at - 900)`, with `DelaySeconds = min(900, max(0, remaining))`; FIFO and generic raw fallback qualify at `deliver_at ?? now`. Claim moves `available_at` to its lease deadline; retain the pre-claim value in the in-memory message as `eligibleAt` for ordering/latency. No separate reservation timestamp. +- `queue` stores the unforwarded logical name with the connection's default expanded. Publication applies queue-name forwarding once through the normal storage path. Changing the default cannot reroute the row; changing a queue-name forward can. Connection forwarding is already resolved at staging and is not rerun. A queue-less custom driver may store null. Drain before changing captured connection routing or forwarding between incompatible FIFO/standard destinations. +- `payload` is the finalized opaque envelope after payload hooks and `JobPayloadFinalizing`, not a job to redispatch through Bus. `longText` preserves its bytes; a database JSON type could normalize them. Adapters may add only delivery-time metadata such as Horizon's `pushedAt` or an SQS overflow pointer; SQL bytes, UUID and serialized command stay unchanged. Opaque `options` likewise uses text and is decoded only for publication. +- `options` is null when empty, otherwise JSON containing only namespaced transport options needed after the object is gone, notably `{"transport":{"sqs":{}}}` for precomputed group/deduplication. Do not duplicate payload lock metadata or add an unused format version; migrate in-flight rows if a future format changes. Adapters use their own option members; native transport errors retain/back off rows. Do not add a driver-kind compatibility registry or business fields. +- No tenant or shard column: this is an operational table like `jobs`, `failed_jobs`, and `job_batches`; application context already travels inside the finalized payload. +- No `updated_at`, `status`, `processed_at`, history, payload hash, or stack trace. Eligibility is exactly `available_at <= now`; a live lease has a non-null token and `available_at > now`. Full exceptions are reported once; `last_error` is bounded to 4,096 characters. + +Use Query Builder/raw SQL, never Eloquent, for this hot operational table. Raw claim SQL wraps configured identifiers through the active grammar and binds every value. A one-statement bulk relies on that statement's atomicity; a bulk requiring multiple binding-safe insert chunks runs inside `Connection::transaction()` (a savepoint when already nested), so even a caller that catches the exception cannot commit a partial owner group. Reuse `InsertsDatabaseRows`; do not add a second chunk calculation or expose another configuration knob. Claim and settlement sizes may be lower than the requested worker batch when an engine's binding/expression limit requires it. + +## Internal architecture + +Keep public surface small and group the cohesive implementation under `Hypervel\Queue\Outbox`: + +```text +src/contracts/src/Queue/PublishesOutboxMessages.php +src/contracts/src/Queue/ShouldQueueViaOutbox.php +src/bus/src/InteractsWithQueueOutbox.php +src/bus/src/Queueable.php +src/bus/src/Batchable.php +src/bus/src/Batch.php +src/bus/src/PendingBatch.php +src/foundation/src/Bus/PendingDispatch.php +src/queue/src/Concerns/InsertsDatabaseRows.php +src/queue/src/Outbox/ + Concerns/PublishesOutboxMessagesSequentially.php + DatabaseOutboxRepository.php + OutboxBatchResult.php + OutboxDispatcher.php + OutboxMessage.php + OutboxRelay.php + OutboxWorker.php + OutboxWorkerOptions.php +src/queue/src/Console/ + ForgetOutboxCommand.php + ListOutboxCommand.php + MonitorOutboxCommand.php + OutboxTableCommand.php + RelayCommand.php + RetryOutboxCommand.php + stubs/outbox.stub +src/queue/src/Events/ + OutboxBatchDelivered.php + OutboxBatchDeliveryFailed.php + OutboxBusy.php +``` + +The tree lists the new mechanism and primary API surfaces, not every modified file. Integration also touches the existing queue drivers, `FailoverQueue`, pool proxies, Events `Dispatcher`/listener wrappers, `SendQueuedMailable`, `SendQueuedNotifications`, `BroadcastEvent`, Horizon `RedisQueue`/`JobPayload`, OpenTelemetry `QueueInstrumentation`, and the prerequisite files identified above. + +Do not create a listed type if it would only forward one call. Do not combine routing, SQL claims, network publication and daemon lifecycle into one god class. Responsibilities are: + +- base `Queue` plus small protected hooks: selection precedence, local/database fast paths, normal lifecycle events, default-expanded logical destination and driver option capture; +- `InsertsDatabaseRows`: reuse the existing binding-safe, transactionally atomic chunk operation and its existing queued-job insertion error; +- `OutboxDispatcher`: transaction/database ownership, preflight validation, staged single/bulk coordination, and rollback registration without containing SQL claim or broker publication logic; +- `DatabaseOutboxRepository`: multi-row insert, engine-specific claim, token-guarded delete/release, retry/forget/list/monitor queries; +- `PublishesOutboxMessagesSequentially`: the one shared generic fallback used by the relay and pooled proxy, partitioning future rows and returning accepted-prefix/deferred/failure outcomes without duplicating the algorithm; +- `OutboxRelay`: group claimed rows by target connection, choose optimized raw-batch publication or the shared generic raw fallback, and settle only explicit outcomes; +- `OutboxWorker`: stateless daemon loop, bounded task admission, round-robin database fairness, restart/signal/memory/time/empty rules, and ownership of the short-lived child-coroutine boundaries described below; +- command classes: input/output only. + +Container-cached outbox services remain immutable/stateless. Database capability detection stays on the live connection that owns the capability, without another service or cache. `OutboxMessage`, `OutboxBatchResult`, and `OutboxWorkerOptions` are immutable per-operation values created directly rather than container-resolved. Loop counters, current claims, stop reasons, and per-delivery data stay method-local. The outer concurrency task may retain only scalar/immutable values and result data: never a database connection, query builder, resolved queue instance, pool lease, or transport client. + +Resource ownership is a hard invariant: **no relay database or transport connection remains borrowed while the relay performs work that does not require it**. Explicit idle release does not cover retained write sessions, every custom driver, or arbitrary callbacks. Use fresh child-coroutine boundaries for (1) source claim, (2) each target-connection publication group, and (3) that group's settlement. `Waiter(-1)` leaves phase duration to configured database/transport timeouts and daemon cancellation, not its unrelated ten-second default. Each child uses a fresh context, returns materialized values and ends before the next phase. Request-side staging instead uses the exact already-borrowed business connection for atomicity. + +Reuse the prerequisite `DispatchLockContext` capture/snapshot lifecycle and existing hidden scalar unique metadata. Never persist live cache provenance or captured release closures, or create an outbox-owned lock registry. + +### Queue integration seam + +Refactor the single and bulk enqueue paths so the cheap outbox-selection decision happens before ordinary after-commit deferral. Read the connection default from the typed boolean cached when the queue receives its immutable config; when neither job nor connection selects outbox, do not resolve an outbox service. Once selected, complete these steps inside the existing guarded enqueue attempt: + +1. resolve/validate selection, database ownership, driver fast-path behavior, and logical destination before finalization; +2. retain dispatch ownership until storage succeeds; any preparatory snapshot is data only, not a registered rollback callback; +3. finalize the payload exactly once while the original job object exists, inside the guarded enqueue attempt; +4. run the narrow outbox-payload hook, decode and validate the finalized envelope and its UUID string consistently on every database engine, and compute absolute `deliver_at` plus target-specific opaque options in that same guarded attempt; +5. raise normal `JobQueueing`; +6. insert on the selected connection; immediately after success, register independent snapshot rollback cleanup only if that exact connection still has an active transaction; +7. let the existing enqueue lifecycle accept the dispatch locks and raise `JobQueued` with the job UUID after insertion, or raise `JobQueueingFailed` for every non-cancellation exception within the guarded attempt. + +The existing `DispatchLockContext` / `PendingDispatch` lifecycle owns failures before queue acceptance, including payload serialization and queue resolution. Reuse it rather than adding an outbox cleanup path. A failure before a valid payload exists rethrows without fabricating a queue lifecycle event; a failure inside the guarded enqueue attempt follows the existing `JobQueueingFailed` and cancellation rules. Do not report an accepted job as failed when a later `JobQueued` listener throws. + +`JobQueued` means the configured logical queue accepted the job into its durable path; like a database-queue insert inside a caller transaction, that acceptance can still roll back with the transaction. It must not be delayed to another process. This preserves the original job object for listeners and completes OpenTelemetry producer state in the dispatching coroutine. Relay must not emit a second normal enqueue lifecycle. + +Provide additive protected queue hooks such as: + +```php +protected function resolveOutboxQueue(?string $queue): ?string; +protected function preparePayloadForOutbox( + object|string $job, + string $payload, +): string; +protected function prepareOutboxOptions( + object|string $job, + ?string $queue, + string $payload, +): array; + +protected function availableAtForOutbox( + ?string $queue, + ?int $deliverAt, +): int; +``` + +First-party drivers expand missing queues to their configured logical default without forwarding or deriving physical keys/URLs. The base options hook returns an empty array; initial availability is `deliver_at ?? now` because raw publication has no cross-driver delay contract. Redis/database become eligible immediately. SQS captures group/deduplication while the object exists, without relative `DelaySeconds`, and computes initial eligibility using the effective destination's standard/FIFO behavior. These inspections do not replace the stored logical name. At publication, normal driver routing resolves that name once. Register independent storage-rollback snapshots through the existing connection-aware transaction callbacks; do not reuse the live-record lookup callback after acceptance. + +`Queue::setConfig()` caches the optional connection `outbox` boolean alongside the already retained immutable configuration; `QueueManager` already calls this once for every resolved concrete connection. Do not add another constructor argument to every driver/connector. Missing `outbox` remains `false` for existing custom configuration. Selection computes explicit state, marker, and the cached flag first; only a positive result checks the failover suppression context. `FailoverQueue` scopes that context with exception-safe restore around all child calls, including raw calls, so nested failover queues remain correct. + +Base `Queue::bulk()` must preflight and partition outbox-selected jobs itself, stage its outbox partition with set-based owner-group inserts, then loop only the direct remainder. Beanstalkd and custom durable drivers that inherit the base method therefore get bulk staging without a driver override. Database, Redis, and SQS keep their optimized overrides but apply the same partition before their direct paths. + +Every implementation preserves order within each partition and groups staged rows by owning database. This prevents a later configuration error from partially staging an otherwise valid group. There is deliberately no ordering or atomicity guarantee between direct and outbox partitions, just as existing cross-connection bulk has none; callers requiring one atomic unit use one outbox owner. Finalize every owner group before its first insert, emit existing per-job queue lifecycle events, and use set-based storage. Each owner group is atomic across its insert chunks; distinct owner databases necessarily commit independently. A `PendingBatch` repository transaction provides one owner and rejects conflicting per-job choices. Preserve current `Bus::bulk()` semantics: it does not suddenly acquire unique/debounce locks or run `PreparesForDispatch`. + +### Raw-batch publication SPI + +`PublishesOutboxMessages` is a narrow optional capability in `Hypervel\Contracts\Queue`, beside `ClearableQueue` and `IndexAwareQueue`, not a new application API. Its signature may reference the queue-owned `OutboxMessage` and `OutboxBatchResult` value types under the repository's lazy contract-signature dependency convention. + +```php +interface PublishesOutboxMessages +{ + /** @param list $messages */ + public function publishOutboxMessages(array $messages): OutboxBatchResult; +} +``` + +`OutboxMessage` carries row ID/job UUID/lease token/attempt, finalized payload, unforwarded logical queue, integer `deliverAt` (nullable), `eligibleAt` (pre-claim availability), `createdAt`, and opaque options. `OutboxBatchResult` contains unique lists of accepted and deferred row IDs plus an optional failure. Accepted and deferred IDs must be disjoint subsets of the input; a clean result accounts for every input through their union. Explicit transport rejects return confirmed IDs plus one aggregate exception containing the reject count, first reason and full response context; a later-chunk exception returns earlier confirmed IDs plus that exception. Cancellation is rethrown. Optimized implementations return no deferred IDs; sequential fallback uses them for future rows. Failed/unattempted rows are input minus accepted minus deferred and receive the same bounded failure text. Foreign/overlapping IDs or unaccounted-for input without failure are a broken contract: report a `LogicException` and settle nothing from that call. Transport response IDs stay inside the driver. + +Use typed message/result boundaries and existing driver option handling; do not add a second transport validator. The generic fallback passes stored options through `pushRaw()`. Native publication errors and partial/ambiguous outcomes use the result contract above. + +Implement the SPI where it provides a real batch/resource benefit: + +- Redis: reuse `LuaScripts::bulk()` and its exact stored-count confirmation once per resolved destination, combining immediate/delayed entries and retaining existing Cluster-safe keys/script handling; preserve confirmed earlier destinations if a later call fails; +- database: one target-database transaction containing parameter-safe multi-row jobs-table inserts across logical queues, with each remaining delay converted to `available_at`; report success only after commit; +- SQS: reuse the 10-entry and 1,048,576-byte request limits, FIFO fields, SDK retry assumptions, and per-entry response mapping; derive overflow pointers from job UUID plus lease token so separate publication attempts never share cleanup ownership; +- pooled proxy: borrow once for the whole invocation, delegate when the borrowed queue implements the SPI, and otherwise invoke the shared sequential fallback on that same borrowed queue; never retain the lease; +- Horizon Redis: decorate the Redis batch path with `JobPending` before attempts and `JobPushed` only for accepted payloads. + +Do not add raw-delay methods to the general `Queue` contract. Custom durable drivers opt into staging through the documented base enqueue hooks; implementing the SPI is an optional publication optimization, and their connector remains otherwise unchanged. + +## Relay algorithm and failure handling + +### Claim + +- The concurrency-owning relay task calls claim through a fresh `Waiter` child with no copied request/transaction context. That child alone resolves the source database, commits the claim, returns immutable `OutboxMessage` values, and ends so `ConnectionResolver` releases the database lease before any broker work starts. +- Claim at most the validated worker batch limit eligible rows ordered by `available_at`, then `id`. +- Eligibility is `available_at <= now`. A claim sets one fresh UUID lease token, `available_at = now + retry_after`, and increments attempts in a short transaction. This moves live leases out of the hot eligibility range without a second timestamp/filter. Never hold SQL locks during broker calls. +- PostgreSQL: one CTE using `FOR UPDATE SKIP LOCKED` plus `UPDATE ... RETURNING`, returning the selected pre-update availability as `eligibleAt` with the updated token/attempt. Updating the indexed timestamp trades HOT-update eligibility for avoiding repeated live-lease scans; validate this tradeoff in the approved benchmark. +- Supported MySQL/MariaDB/Vitess: `SELECT ... FOR UPDATE SKIP LOCKED`, one guarded bulk update, and return the already selected rows with token/time/attempt fields reflected in memory rather than issuing another select. +- SQLite/engines without `SKIP LOCKED`: select an ordered, limited list of candidate IDs and their availability, perform one atomic eligibility-guarded token/expiry update restricted to those IDs, then load full rows restricted to those IDs bearing that token. Preserve original eligibility from the candidates. The second read distinguishes rows actually won; another claimant can cause an empty/partial result, not a double claim. SQLite serializes writers, so additional relays improve availability rather than write throughput. +- Normalize every returned claim to its pre-claim `(eligibleAt, id)` order; `UPDATE ... RETURNING` and `WHERE IN` do not promise order, and the newly shared lease deadline is not the original eligibility. +- Never return a connection, builder, cursor, lazy collection, or pool-backed object from the claim child. Materialize and validate the bounded result before it ends. +- Reuse `PdoConnection::lockForPopping()` and `maxBindings()` for PDO-backed connections. Keep base connection types driver-neutral and follow DatabaseQueue's non-PDO lock/binding fallback; custom drivers are not required to expose PDO methods. Existing live-session capabilities cover engine variants; do not add another detector, contract or cache. + +### Publish and settle + +- Group a claim by target connection, preserving order within each group. The driver partitions for existing Redis destination scripts, SQS URL/count/bytes, or database binding limits. This permits multi-row database inserts across queues and one pooled Beanstalk lease per connection batch without replacing existing driver mechanisms. +- Publish each group in its own fresh child. Only that child resolves the queue, borrows its transport and returns a materialized result. Use the optional raw-batch contract or shared sequential fallback. Like `queue:retry`, stored rows honor the recorded connection's current configuration, including an operator deliberately changing it to sync/null; add no driver-kind rejection. Document draining before configuration changes that discard durability or invalidate stored transport options. Native send errors retain/back off affected rows. +- The shared fallback moves rows whose `deliver_at` is still in the future into the result's deferred set without calling `pushRaw()`, then publishes the due rows in order and returns the accepted prefix plus any exception that stopped it. `QueuePoolProxy` invokes this same operation while its one borrowed queue is live when that queue lacks the SPI. Settlement releases deferred rows token-safely with `available_at = deliver_at`, restores the claim's attempt increment, and leaves `last_error` untouched. Deferral is not a publication failure: it does not occupy the task's failure-sleep slot or emit `OutboxBatchDeliveryFailed`. +- The driver's existing publish result is the acknowledgment—Redis/database replies, SQS success/failure entries, and Beanstalk IDs. Add no probe or extra broker round trip. Validate the returned accepted row IDs before settlement; an invalid result retains every row in that call and reports the driver-contract failure. +- Settle each completed group immediately in a fresh child and short source transaction: delete accepted rows using `id IN (...)` plus `lease_token`; clear deferred tokens, restore availability to `deliver_at`, undo the claim's attempt increment and preserve `last_error`; release failed/unattempted rows with one token-guarded update per binding-safe chunk. Choose one jittered deadline per capped retry bucket using `RETRY_DELAYS`, random 50%-100% jitter and a minimum one-second delay. A fixed `CASE WHEN attempts <= 1 THEN ? ... ELSE ? END` binds seven deadlines, not per-row cases; clear the token and bind one bounded error text. Only ID lists require size-dependent chunking. If fewer rows match, retain successful settlement and leave newer owners untouched; report ownership loss once. End the child before the next group. +- A failed connection group does not prevent publication/settlement of the other groups in the same claim. Collect and report their errors after all non-cancelled groups have been attempted, identifying affected row IDs, job UUIDs, and payload display names; decode display names only on failure so successful relay batches pay no diagnostic parsing cost. Cancellation stops further publication and leaves unsettled leases to expire. +- After any result failure, keep that relay task's concurrency slot occupied for its positive `--sleep` interval. Row backoff alone cannot protect a misconfigured/unavailable target from a large backlog of first-attempt rows; bounded occupied slots throttle further claims without distributed circuit-breaker state. +- If broker acceptance succeeds but delete fails or the process crashes, lease expiry makes the row retryable; a duplicate is correct. If the network outcome is ambiguous, retain and retry. +- Guard all settlement by lease token so a stale worker cannot delete/release rows reclaimed by another worker. +- Require `retry_after` to exceed the maximum expected lifetime of one claimed batch, including every connection-group publication and SDK retry. Do not add a heartbeat coroutine or callback machinery; an exceptionally slow cycle may produce a duplicate after lease expiry but cannot lose the row. +- On graceful stop, stop claiming new work and let active completed calls settle before joining them. Cancellation is never converted to an ordinary failure; an unsettled lease expires naturally. +- Parent cancellation while waiting for a phase propagates as `CanceledException`; a child that independently cancels while its parent remains active follows `Waiter`'s existing `ChildCancellationException` contract and is reported as a phase failure. Do not catch either merely to log and continue claiming. +- Continue retries indefinitely. Backoff prevents poison rows or outages from hot-looping, ordering plus `SKIP LOCKED` prevents one row from blocking later work, and monitor alerts surface permanent failures. + +### Lock cleanup + +Transactional rollback uses the prerequisite independent snapshot, including its captured release context. Existing deferred-dispatch callbacks retain their normal ownership lifecycle. Successful publication leaves locks for normal job processing. + +`queue:outbox-forget` reads existing hidden cache-store/key/owner scalars with object deserialization disabled; never hydrate arbitrary context or unserialize the queued command. Delete the non-live row first, then release the owner-matched unique lock. Keep this cleanup in a protected command method receiving the stored envelope so integrations can establish a context-dependent cache namespace without copying SQL or changing the relay. Commands already resolve through the container; no new registry is needed. Cache and SQL cannot commit atomically: malformed metadata/post-delete cache failure is reported and may require manual cleanup, but never restores or republishes deliberately forgotten work. Debounce tokens have bounded TTL and need no explicit forget cleanup. No outbox lock abstraction or persisted release closure. + +## Worker and operations + +Register these commands in `QueueServiceProvider`: + +- `queue:relay {database?}` — long-running relay plus `--once`, `--stop-when-empty`, `--batch=100`, `--sleep=1`, `--concurrency=1`, `--max-jobs=0`, `--max-time=0`, and `--memory=128`; validate batch 1–1,000 and positive sleep/concurrency. Sleep accepts fractions like `queue:work`. `--once` makes one synchronous round-robin pass, at most one bounded claim per configured/pinned database, with no empty/outage sleep. `--stop-when-empty` joins active tasks and exits after a complete source pass finds no eligible row; delayed/backoff rows do not keep it alive. `--max-jobs` counts claimed rows regardless of result; cap every final claim to the remaining allowance; +- `queue:outbox {database?}` — pending-row diagnostics by configured database, ID, UUID, job display name read from the bounded payload result, target, attempts, lease state, age, and last error, ordered by ID and bounded by validated `--limit=100` (maximum 1,000), with `--json` matching `queue:failed`; otherwise visit all configured sources; never render payloads, transport options, context, or lock-owner tokens; +- `queue:outbox-monitor {database?}` — total, eligible, delayed, leased, and retrying counts plus the oldest currently eligible row's age, with explicit `--max` / `--age` alert thresholds and `--json` output for one/all configured databases; use one portable conditional-aggregate query plus one indexed `(available_at, id)` head lookup per database, and emit `OutboxBusy` like the existing `queue:monitor` command without another index or payload scan. Eligibility uses the claim predicate, and age starts at `available_at`, so intentionally delayed jobs do not create false alerts; +- `queue:outbox-retry {uuid?*}` — targets only rows with `attempts > 0`, clears errors/leases, and makes selected rows (or explicit `all`) immediately available without erasing attempt history; accept `--database=` to pin a configured source, otherwise visit all sources. `all` snapshots each source's maximum ID among retryable rows and updates through that ceiling in bounded primary-key cursor chunks; an explicit UUID found on multiple unpinned sources fails rather than updating an ambiguous match. A never-attempted row waiting for its intended delivery time is not retryable; a previously failed row remains retryable regardless of its current backoff. Always reject live leases; +- `queue:outbox-forget {uuid}` — confirmation, live-lease rejection, token-safe delete, and owner-matched unique-lock cleanup from the payload metadata; accept `--database=` to pin a configured source and fail rather than guess if an unpinned UUID exists on more than one source. There is no force path because an in-flight publication cannot be recalled; +- `make:queue-outbox-table` / `queue:outbox-table` — migration generation matching existing queue table commands. + +The worker follows the relevant `queue:work` operational conventions: `queue:restart`, bounded coroutine concurrency, POSIX signals, memory/max-time exits, exception reporting, and no swallowed cancellation. It does not consult maintenance mode: relaying only moves committed jobs to their queue, while normal workers already control whether those jobs execute. One coordinator admits work in configured-database round-robin order only when `WaitConcurrent` has a free slot, so counters, source fairness, stop decisions, and the remaining `--max-jobs` allowance stay local and exact. + +Retry/forget load mutation targets from the writer using existing query-builder support, not replica reads or a separately borrowed connection. Recheck token/live-lease predicates in the modifying statement so a concurrent claim cannot be reset/deleted. A live lease is `lease_token IS NOT NULL AND available_at > now`; expired tokens do not prevent recovery. Monitor counts distinguish those live leases from tokenless scheduled/backoff rows and use `available_at <= now` for eligibility. + +Each admitted task owns no pooled resource. It uses a fresh child for claim, each publication and settlement, passing only immutable messages/results. Guarded delivery events/reporting also run in a short-lived child because application callbacks may resolve clients. A listener failure is reported once; a reporter failure propagates. Neither alters settled rows, emits the opposite delivery event or republishes accepted work. Use fakeable `Sleep::usleep()` for fractional empty/failure delays and the existing capacity-wait primitive instead of polling busy slots. Graceful stop ceases admission and joins tasks so completed publication can settle. Supervise this dedicated process; scheduled `--once` is also supported. Size source/transport pools for relay concurrency and any other users, allowing a short-lived extra slot for restart-cache/event/reporting work on the same pool. Never make the relay a queued job dependent on the broker it must recover. + +Horizon continues supervising and visualizing Redis jobs only after relay publishes them. Do not add relational outbox polling, migrations, or supervisor machinery to Horizon. + +Expose one guarded event per completed connection publication batch, with source database, target connection, accepted/failed counts, publish duration, oldest eligible age, and an optional failure, but no payload bodies: `OutboxBatchDelivered` only when every attempted row was published and settled, otherwise `OutboxBatchDeliveryFailed` for a result failure, partial acceptance, invalid result, lease-ownership loss, or settlement failure. Rows deferred before the generic fallback are excluded; a group containing only future rows emits no delivery event. The OpenTelemetry component consumes these events in the same change and records delivery count, failure count, eligible-to-publish latency, and publish duration. Rely on the monitor command for backlog/eligible-age gauges so ordinary dispatch does not issue count queries. + +## Horizon outbox integration + +Use the existing `JobPayload` lifecycle through its two semantic operations: + +- `prepareForQueueing($job)` while the object exists: type, tags, and silenced state; +- `markAsPushed()` at actual Redis publication: `pushedAt` only. + +Preserve `JobPayload::prepare($job)` as a single-pass combination of both operations. Direct immediate/delayed Horizon dispatch calls it once inside the actual publish callback, so after-commit timing is correct and the direct path gains no second JSON decode/encode. External `pushRaw()` uses the same single-pass method with `null`, preserving already-stored classification and refreshing only `pushedAt`. + +Add a narrow protected `preparePayloadForOutbox()` hook to base Queue that is called only after generic payload finalization on selected outbox paths and is otherwise a no-op. Horizon Redis overrides it to call `prepareForQueueing($job)` once before storage. Its raw-batch publication calls `markAsPushed()` once per payload immediately before Redis publication. Preserve job `uuid` / Horizon `id`; emit `JobPending` before each attempted publication and `JobPushed` only for explicit successes. The outbox path receives the original job through this hook; it must not reuse Horizon's direct-push coroutine handoff or classify with `null`. + +## Documentation cleanup + +Update in the same change so no stale behavior survives: + +- `src/docs/queue-outbox.md`: canonical setup, configuration, API precedence, transaction ownership, driver/custom-driver matrix, bulk/batches, `dispatchSync`/after-response limitations, delays, UUIDs, connection capture versus publication-time queue forwarding, deliberate driver changes, at-least-once/idempotency and best-effort dispatch ordering, SQS overflow attempt ownership/orphans, worker deployment/pool sizing, operations, Horizon and database fast paths; +- `src/docs/documentation.md`: add the new guide to navigation; +- `src/docs/queues.md`: add only a concise transactional-outbox introduction, when it is useful, one basic `viaOutbox()` example, and a link to `queue-outbox.md`; do not duplicate its setup, guarantees, driver matrix, or operations reference; +- `src/docs/events.md`, `mail.md`, `notifications.md`, and `broadcasting.md`: beside their existing after-commit guidance, add one concise example/cross-link showing that the producer's marker or fluent selection is propagated to its wrapper job; link to `queue-outbox.md` for guarantees rather than duplicating them; +- `src/docs/porting-from-laravel.md` and database README: explain exact resolved transaction identity for extensions observing role-qualified records. Do not add outbox catalogue entries: its opt-in APIs require no Laravel adaptation. Preserve Queue/Bus/Horizon README structure without adding additive-feature differences; +- `/home/binaryfire/workspace/contrib/hypervel/hypervel`: match Foundation's outbox configuration and the Testbench schema in the next application migration; +- configuration comments and migration command help: exact guarantees and required worker/table setup. + +Never imply exactly-once delivery, broker participation in the SQL transaction, or durable behavior for local drivers. Remove a comment only when the outbox implementation makes it stale; final comments explain only non-obvious invariants. + +## Testing plan + +### Framework prerequisites + +- Distinct base/`::write` sessions with simultaneous transactions and independent commit/rollback callbacks; nested savepoints, shared in-memory SQLite aliases, Foundation wrapping/context synchronization and Pennant rollback invalidation use resolved ownership. Do not use shared in-memory SQLite as proof of separate-session behavior. +- Context-dependent cache fixture backed by `WorkerArrayStore`: acquire in context A, release record/snapshot in B, remove A's key and preserve B's same logical key. Cover unique and debounce separately, with a second registered wrapper proving composition order and restoration. A throwing capture callback acquires neither lock nor token. One real CoroutineQueue closing-timer regression covers release outside the acquiring coroutine. No repeated tests per caller or cache wrapper: the hook does not branch on wrapper types. +- DatabaseQueue storage rollback releases its snapshot after acceptance, including nested rollback/deadlock retry; broker-accepted jobs are not released by unrelated rollback. Preserve existing deferred ownership coverage. +- SQS later-finalizer/queueing-listener failures finish unaccepted producer state without falsifying confirmed chunks; preserve cancellation. Reuse/extend affected existing tests. +- Horizon combined preparation encodes once and preserves classification/pushed timing/customization. Run the existing direct/retry/delayed tests; do not duplicate them under outbox names. + +### Contract and routing tests + +- `Queueable`/`Batchable` shared fluent state, pending dispatch/batch APIs, jobs that use only `Batchable`, jobs added to an existing batch, explicit per-job overrides, and the absence of chain-specific propagation/API state; +- precedence across explicit true/false, marker, and connection default; `beforeCommit`/`afterCommit` remain independent direct-publish controls; wrapper jobs preserve marker/explicit outbox state; and `dispatchSync`/`dispatchAfterResponse` retain their existing local/deferred semantics rather than pretending to provide pre-deferral durability; +- a custom durable transport without the raw-batch SPI stages normally and relays through the sequential raw fallback; failover selected by explicit state, marker, or its own connection default degrades to its existing after-commit attempt without schema access, child defaults are suppressed so fallbacks still run, nested suppression restores correctly, and a custom queue that bypasses the base lifecycle remains direct and is documented as not outbox-capable; preflight failure releases only locks acquired by the current dispatch; +- `pushRaw()` remains direct even on a connection whose default enables outbox; +- local/null/fake/database fast paths never query the outbox; exact local transaction selection is honored; expanded defaults stay fixed and queue-name forwarding runs once, including `A → B`, `B → C` and a changed forward before publication; +- staged single dispatch returns the same job UUID indexed on its row; direct driver return values remain unchanged. + +### Transaction tests + +- no transaction, commit, rollback, nested savepoint commit/rollback, and rollback lock release; +- exactly one active connection, nested records on it, multiple active owners with/without explicit selection, and unconfigured-source rejection; a `::write` owner covered by its configured base stages on the exact alias session and is drained/countable once through the base source; +- a read-role owner is rejected before insertion with no staged row and existing dispatch-lock cleanup; cover active ownership and explicit no-transaction selection within the owner tests, not a separate suite; +- same database queue connection inserts directly and rolls back atomically; a differently named/session connection stages and relays; +- bulk is all-or-nothing across insert chunks within each owner group both with no outer transaction and when an outer caller catches the exception after nested-savepoint rollback; separate owners and direct/outbox partitions are independently atomic and preserve only internal order; batch counters and outbox rows commit/roll back together; +- outbox preflight, payload-finalizer, option, and storage failures follow the existing lifecycle: every unaccepted single/bulk member raises `JobQueueingFailed`, dispatch-owned locks remain exact-owner safe, cancellation remains cancellation, and an accepted job is never reported as failed because a later `JobQueued` listener throws; +- rollback snapshots register on the selected owner and release only that dispatch's locks even after acceptance removes its live record; successful publication preserves the normal queued-job lock lifetime. + +### Payload and API integration tests + +- queued jobs, closures, listeners, mailables, notifications, broadcasts, encrypted jobs, serialized models, custom payload hooks, OpenTelemetry propagation, unique jobs, debounced jobs, delayed jobs, batches, nested batched chains, and ordinary chains; wrapper producers propagate marker/explicit selection and database without changing their public APIs, while absent source state leaves the wrapper's typed fields uninitialized; +- payload creation/finalization occurs once; malformed final envelopes or missing/invalid UUIDs fail uniformly before storage on every database engine; an outbox bulk finalizer failure leaves no earlier producer state open; opaque payload bytes remain unchanged in SQL and UUID/command bytes remain stable over relay retries apart from documented delivery-time transport metadata; +- SQS FIFO group/deduplication callback is evaluated once at staging; standard delays longer than 900 seconds wait until the broker window, FIFO delays wait until due, relay latency/backoff never rewrites immutable `deliver_at`, and publication after that instant uses zero additional delay; +- `Bus::bulk()` uses bounded multi-row inserts and preserves its existing lack of unique/debounce/preparation side effects; Beanstalkd and a custom durable driver inheriting base `Queue::bulk()` each stage one set-based insert per owner group rather than one insert per job, while their direct partitions retain the base loop; +- fake assertions work without schema setup or outbox service resolution; an unselected job does not serialize the uninitialized outbox state. + +### Persistence and concurrency tests + +- schema/migration command/config validation, byte-preserving payloads, opaque nullable text options, public UUID lookup and the hot composite index; staged time fields equal the frozen framework clock's epoch values, with one post-2038 fixture rather than a timezone/type matrix; normalize past dates and never truncate a future fractional due time early; +- indexed eligibility, pre-claim order/latency, attempt increment, lease expiry, token-guarded stale settlement, capped-bucket jitter bounds and last-error truncation; +- two/many concurrent relays do not claim the same live row; a killed claimant is recoverable after `retry_after`; +- instrumented tiny database/queue pools prove claim connection release before transport blocking, transport release before settlement, settlement release before another target or failure sleep, and bounded forward progress under concurrency; phase boundaries return no pool-backed objects and the long-lived coordinator/concurrency task retains none; +- PostgreSQL CTE/returning, MySQL/MariaDB skip-locked claim and competing fallback claims/full-row reload, consuming existing connection capabilities rather than retesting their engine-version detection; custom non-PDO paths do not assume PDO methods; +- query-count and bounded-fetch assertions cover one-statement PostgreSQL, two-statement MySQL/MariaDB and three-statement fallback claims, excluding ordinary transaction control. Verify limits in the SQL path, not just a truncated returned array. Do not assert a specific optimizer plan or pretend a mock proves live Vitess behavior; inspect representative query plans during approved benchmarks. + +Place engine-specific cases under `tests/Integration/Queue/Database/{Postgres|MySql|MariaDb|Sqlite}` so the existing database workflow discovers them. Live Redis cases stay under `tests/Integration/Queue/Redis` and use `InteractsWithRedis`; do not pretend SQLite or a mock validates PostgreSQL/MySQL locking or Redis behavior. + +### Transport tests + +- Redis immediate/delayed Lua batching, exact confirmation and ambiguous failure after earlier confirmed destinations; +- database queue multi-row insertion across logical queues and remaining-delay conversion; +- SQS 10-entry/1,048,576-byte chunking, standard/FIFO outbox delay behavior, group/deduplication, aggregate explicit-reject failure, SDK exception after confirmed earlier chunks, and partial accepted-ID results; overflow storage reuses one pointer inside SDK retries but uses independent job-UUID/lease-token pointers across outbox attempts, cleanup for a later explicitly rejected attempt cannot remove an earlier ambiguous attempt's payload, confirmed unpublished pointers are cleaned up, and ambiguous attempts are retained; +- recorded connections honor current driver configuration and native transport failures retain rows; invalid/overlapping result IDs retain the whole call, sibling groups continue independently, and stale settlement leaves newer owners untouched; SPI-to-non-SPI replacement before `deliver_at` defers without publication, failure telemetry, attempt inflation or error mutation; +- shared generic raw fallback accepted-prefix/deferred/ambiguous-failure results; Beanstalk uses it with one pooled lease per connection batch and switches tubes correctly; +- only confirmed accepted IDs are deleted; every failed/unattempted row is released with the aggregate failure, while future fallback rows follow the non-failure deferral path; acceptance followed by delete failure produces a duplicate with the same job UUID and no lost row; +- confirm settlement adds no broker probe beyond the normal publish call/result. + +### Worker, Horizon, and telemetry tests + +- round-robin sources, positive fractional empty/outage sleeps, backlog drain, once/stop/max-jobs/memory/restart/signal/cancellation behavior, maintenance-mode independence and exception reporting; +- list bounds/display names/source selection; monitor counts distinguish live leases from future tokenless rows and alert on eligible age; retry/forget use writer reads and atomic live-lease guards, bounded `all`, confirmation/source pinning and cross-source UUID disambiguation; forget releases exact scalar unique ownership, exposes its narrow cleanup override and reports malformed metadata. Retain bounded debounce expiry without adding cleanup machinery; +- Horizon outbox paths preserve type/tags/silenced, refresh only `pushedAt` and emit Pending/Pushed correctly on partial results; prerequisite coverage owns shared direct/retry/delay regressions; +- OpenTelemetry producer state terminates during staging; relay batch instruments do not leak state or duplicate normal enqueue metrics; +- delivery-listener and exception-reporter failures release their child-coroutine resources and cannot alter an already-settled row, emit the opposite delivery event, or cause a republish. + +### Performance gates + +- Pin direct dispatch before/after query count, outbox service-resolution count, serialized fields, and payload transformations in automated tests: all remain unchanged when outbox is unselected. Use a repeatable local benchmark to measure the minimal cached job/config selection checks and investigate any material CPU or allocation regression; do not place unstable wall-time thresholds in CI. +- Obtain the owner's confirmation that the machine is idle before any benchmark. Benchmark single and 1k/10k staging, lease claims, Redis Lua batches, database inserts and SQS request construction; compare query/network counts and allocation as well as throughput. +- Verify one insert per normal bulk chunk, one short claim transaction, publication using existing destination/protocol batches, and at most one accepted delete, deferred release and failed-row release per non-empty binding-safe settlement subset. Record peak borrowed source/transport connections and prove no lease spans unrelated work or sleep. +- Profile PostgreSQL/MySQL plans with representative pending/live-leased/retrying backlogs and concurrency. Compare the indexed availability/lease update against the prior separate-reservation predicate, including PostgreSQL index-write/HOT costs. Confirm the chosen design with evidence; resolve a substantive contrary result before changing the plan. Do not add indexes or tune engine settings speculatively. + +Run focused files immediately as they are added, then affected package suites, cross-package integration suites, analysis, and the repository checkpoint: + +```bash +cd /home/binaryfire/workspace/contrib/hypervel/components-queue-outbox +composer test:parallel -- tests/Queue --filter Outbox +composer test:parallel -- tests/Bus --filter Outbox +composer test:parallel -- tests/Horizon --filter Outbox +composer fix + +cd /home/binaryfire/workspace/contrib/hypervel/hypervel +composer cs-fix +composer analyse +composer test + +``` + +Run each new or changed test file immediately. The filters are only a coherent-slice inner loop. `composer fix` includes `composer test:testbench`, satisfying the Testbench migration-change check without a duplicate run, plus formatting, analysis, parallel tests and dogfood suites. When verifying the Testbench change independently, run `composer test:testbench` directly. + +If implementation uses a components worktree, copy `.env` from the main components checkout and run `composer install` in the worktree; never symlink `vendor` or Composer state. + +## Implementation order and acceptance + +1. Complete and verify the independently useful framework prerequisites in dependency order, then add marker/shared fluent state and selection tests. +2. Add config, migration command/schema, immutable value objects, `DatabaseOutboxRepository`, and transaction/multi-database tests. +3. Integrate single/bulk staging, existing fast paths, SQS delay selection, forwarding and the prerequisite snapshot lifecycle. +4. Add the raw-batch SPI and first-party driver implementations, including pooled and Horizon paths. +5. Add claim/publication/settlement resource scopes, relay/daemon and operator commands, guarded events, and OpenTelemetry listeners. +6. Complete canonical docs and the exact-owner compatibility note, approved benchmarks, focused suites, static analysis and `composer fix`. + +Before completion, re-read every changed file and confirm: + +- no mutable cross-coroutine state, pool lease retained beyond its owning phase or established queued-job terminal lifecycle, container swap, hidden environment branch, or network call under a SQL lock; +- source claim, each target publication, settlement, and arbitrary listener/reporter execution end their borrowing coroutine before unrelated work, another target, or throttling sleep begins; +- every crash boundary retains a retryable row or has a confirmed target acceptance, with duplicates preferred over loss; +- ordinary queue, batch, chain, Horizon, and OpenTelemetry behavior remains coherent; +- disabled/local/database-fast-path dispatch performs no outbox I/O; +- Foundation and application-skeleton queue defaults and outbox migrations agree; +- `queue-outbox.md` is the single detailed outbox reference and shorter queue/producer mentions link to it without duplicating guarantees; +- all configured owners are drainable, exact transaction identity is preserved, queue-name forwarding occurs once, current connection configuration is honored, and all SQL settlement is lease-token guarded; +- public names read like first-party Laravel queue APIs while internals remain Hypervel-native; +- no dead status/history/dead-letter/CDC/inbox abstraction, stale comment, duplicate README line, or superseded documentation remains. From 0e726ce4cfa9633b03cc0897a20a623a35267235 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 19:41:05 +0000 Subject: [PATCH 02/33] Clarify targeted verification after failures Choose rerun scope from the correction's impact rather than repeating the original broad command. Finish outstanding verification and reserve another full test-suite run for fixes with framework-wide impact. --- AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index c839ea1db..5bab5ab18 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -119,7 +119,7 @@ During implementation, run new or changed test files immediately. After completi Use checks that match the change. For isolated changes, run `composer lint:fix`, `composer analyse`, and the affected tests. Run a single affected test file with PHPUnit. Use ParaTest when the affected tests span multiple files. Only run `composer fix` when changes could affect code beyond the affected tests; it already runs formatting, analysis, and all test suites, so do not run those checks separately first. -If a check fails, use targeted checks while correcting the issue, then run the failed check and each remaining check. Rerun an earlier check only if the correction could affect it. +After a failure, fix the cause and rerun only affected tests, formatting or static analysis. Then finish any verification not yet run. Repeat the full test suite only if the fix itself has framework-wide impact. ## Development Conventions From b645321229dd3da78eed03eec207c8d1f2b3737d Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 20:46:48 +0000 Subject: [PATCH 03/33] fix(database): track transactions by resolved connection identity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Use the resolved base/read/write connection identity for transaction lifecycle records and connection-bound callbacks. Independent role sessions must not consume each other’s commit or rollback work, while shared in-memory SQLite aliases retain their single owner. Align Foundation wrapping transactions, request context synchronization and Pennant invalidation with exact ownership. Add role-session and rollback regressions and document the extension-facing identity convention. --- src/database/README.md | 1 + .../src/Concerns/ManagesTransactions.php | 14 +-- src/database/src/Connection.php | 2 +- .../src/DatabaseTransactionsManager.php | 4 +- .../Testing/DatabaseTransactionsManager.php | 2 +- .../Testing/RequestContextSynchronizer.php | 2 +- src/pennant/src/Drivers/Decorator.php | 5 +- .../DatabaseTransactionIdentityTest.php | 113 ++++++++++++++++++ .../DatabaseTransactionsManagerTest.php | 32 +++++ .../Testing/RefreshDatabaseTest.php | 4 +- .../RequestContextSynchronizerTest.php | 34 ++++++ tests/Pennant/TransactionTest.php | 39 ++++++ 12 files changed, 235 insertions(+), 17 deletions(-) create mode 100644 tests/Database/DatabaseTransactionIdentityTest.php diff --git a/src/database/README.md b/src/database/README.md index 447c39322..05fa854a3 100644 --- a/src/database/README.md +++ b/src/database/README.md @@ -10,6 +10,7 @@ Documentation: https://hypervel.org/docs/database - PDO connections require exception mode so failed statements cannot silently continue. Warning and silent options are rejected when connecting; custom connectors and supplied handles must honor the same requirement. See [PDO error mode](https://hypervel.org/docs/database#pdo-error-mode). - Default retry and lost-connection detection recognizes PDO failures, not arbitrary exception text. Non-PDO drivers provide their own policies through the existing detector bindings; see [detecting database failures](https://hypervel.org/docs/database#detecting-database-failures). - `DB::transaction()` throws if its closure manually ends the transaction it owns. Let `transaction()` manage completion, or use the manual transaction API separately. Balanced nested transactions remain supported; see [database transactions](https://hypervel.org/docs/database#database-transactions). +- Transaction records use the resolved connection's `getNameWithReadWriteType()`, not its base `getName()`, so distinct base, read and write sessions own separate callbacks. Extensions inspecting transaction records must compare against the resolved owner's full name. - SQLite requires 3.32.0 or later rather than Laravel's 3.26.0 minimum, so framework code assumes the newer default limit of 32,766 bindings per statement. - Outgoing framework HTTP requests return idle database sessions to their pools. Code using temporary tables, session locks or retained raw PDOs across these calls must use `withPinnedSession()`; ordinary queries and active transactions keep their behavior. See [releasing and pinning connections](https://hypervel.org/docs/database#releasing-and-pinning-connections). - MySQL and MariaDB connections require `strict` or `modes` in their configuration instead of implicitly inheriting the server's SQL mode, so SQL literals can be escaped correctly. diff --git a/src/database/src/Concerns/ManagesTransactions.php b/src/database/src/Concerns/ManagesTransactions.php index b300b7ef6..5543abbdc 100644 --- a/src/database/src/Concerns/ManagesTransactions.php +++ b/src/database/src/Concerns/ManagesTransactions.php @@ -88,7 +88,7 @@ public function transaction(Closure $callback, int $attempts = 1): mixed try { $this->transactionsManager?->commit( - $this->getName() ?? '', + $this->getNameWithReadWriteType() ?? '', $levelBeingCommitted, $this->transactions ); @@ -182,7 +182,7 @@ public function beginTransaction(): void try { $this->transactionsManager?->begin( - $this->getName() ?? '', + $this->getNameWithReadWriteType() ?? '', $this->transactions ); @@ -259,7 +259,7 @@ public function commit(): void try { $this->transactionsManager?->commit( - $this->getName() ?? '', + $this->getNameWithReadWriteType() ?? '', $levelBeingCommitted, $this->transactions ); @@ -369,7 +369,7 @@ public function rollBack(?int $toLevel = null): void try { $this->transactionsManager?->rollback( - $this->getName() ?? '', + $this->getNameWithReadWriteType() ?? '', $this->transactions ); } catch (CanceledException $cancellation) { @@ -447,7 +447,7 @@ public function hasApplicableTransaction(): bool return false; } - return $this->transactionsManager?->hasApplicableTransaction($this->getName() ?? '') ?? true; + return $this->transactionsManager?->hasApplicableTransaction($this->getNameWithReadWriteType() ?? '') ?? true; } /** @@ -460,7 +460,7 @@ public function hasApplicableTransaction(): bool public function afterCommit(callable $callback): void { if ($this->transactionsManager) { - $this->transactionsManager->addCallback($callback, $this->getName() ?? ''); + $this->transactionsManager->addCallback($callback, $this->getNameWithReadWriteType() ?? ''); return; } @@ -492,7 +492,7 @@ public function afterCommitOrNow(callable $callback): void public function afterRollBack(callable $callback): void { if ($this->transactionsManager) { - $this->transactionsManager->addCallbackForRollback($callback, $this->getName() ?? ''); + $this->transactionsManager->addCallbackForRollback($callback, $this->getNameWithReadWriteType() ?? ''); return; } diff --git a/src/database/src/Connection.php b/src/database/src/Connection.php index 83d295716..9553d63a9 100755 --- a/src/database/src/Connection.php +++ b/src/database/src/Connection.php @@ -1003,7 +1003,7 @@ protected function resetTransactionState(): void { $this->transactions = 0; - $this->transactionsManager?->rollback($this->getName() ?? '', 0); + $this->transactionsManager?->rollback($this->getNameWithReadWriteType() ?? '', 0); } /** diff --git a/src/database/src/DatabaseTransactionsManager.php b/src/database/src/DatabaseTransactionsManager.php index a52c9e76e..3e96b4603 100755 --- a/src/database/src/DatabaseTransactionsManager.php +++ b/src/database/src/DatabaseTransactionsManager.php @@ -288,7 +288,7 @@ protected function executeRollbackCallbacks(Collection $transactions): void * connection. The callback runs when that transaction and its enclosing stack on the * same connection commit. * - * @param null|string $connection base name from Connection::getName() + * @param null|string $connection resolved owner name from Connection::getNameWithReadWriteType() */ public function addCallback(callable $callback, ?string $connection = null): void { @@ -307,7 +307,7 @@ public function addCallback(callable $callback, ?string $connection = null): voi * connection. The callback runs when that transaction and its enclosing stack on the * same connection roll back. * - * @param null|string $connection base name from Connection::getName() + * @param null|string $connection resolved owner name from Connection::getNameWithReadWriteType() */ public function addCallbackForRollback(callable $callback, ?string $connection = null): void { diff --git a/src/foundation/src/Testing/DatabaseTransactionsManager.php b/src/foundation/src/Testing/DatabaseTransactionsManager.php index 246e10736..c9c7112db 100644 --- a/src/foundation/src/Testing/DatabaseTransactionsManager.php +++ b/src/foundation/src/Testing/DatabaseTransactionsManager.php @@ -57,7 +57,7 @@ public function beginTransactions(DatabaseManager $database): void try { $connection->beginTransaction(); - $this->wrappingTransactions[] = $this->getCurrentTransactionForConnection($connection->getName() ?? ''); + $this->wrappingTransactions[] = $this->getCurrentTransactionForConnection($connection->getNameWithReadWriteType() ?? ''); } finally { if ($dispatcher !== null) { $connection->setEventDispatcher($dispatcher); diff --git a/src/foundation/src/Testing/RequestContextSynchronizer.php b/src/foundation/src/Testing/RequestContextSynchronizer.php index 1e79f64b2..4e163f66e 100644 --- a/src/foundation/src/Testing/RequestContextSynchronizer.php +++ b/src/foundation/src/Testing/RequestContextSynchronizer.php @@ -26,7 +26,7 @@ public function syncTransactionState( $names = []; foreach ($connections as $connection) { - $names[$connection->getName() ?? ''] = true; + $names[$connection->getNameWithReadWriteType() ?? ''] = true; } foreach (['pending', 'committed'] as $property) { diff --git a/src/pennant/src/Drivers/Decorator.php b/src/pennant/src/Drivers/Decorator.php index 7de16f1eb..2e85772dc 100644 --- a/src/pennant/src/Drivers/Decorator.php +++ b/src/pennant/src/Drivers/Decorator.php @@ -6,7 +6,6 @@ use Closure; use Hypervel\Contracts\Container\Container; -use Hypervel\Database\ConnectionName; use Hypervel\Database\DatabaseManager; use Hypervel\Database\DatabaseTransactionRecord; use Hypervel\Database\DatabaseTransactionsManager; @@ -975,13 +974,13 @@ protected function trackTransaction(PennantState $state, string $connection): vo return; } - $base = ConnectionName::parse($connection)->base; + $owner = $db->connection($connection)->getNameWithReadWriteType(); /** @var DatabaseTransactionsManager $transactions */ $transactions = $this->container->make('db.transactions'); $state->forgetOnRollback( - $transactions->callbackApplicableTransactions()->last(fn (DatabaseTransactionRecord $record) => $record->connection === $base), + $transactions->callbackApplicableTransactions()->last(fn (DatabaseTransactionRecord $record) => $record->connection === $owner), $connection, ); } diff --git a/tests/Database/DatabaseTransactionIdentityTest.php b/tests/Database/DatabaseTransactionIdentityTest.php new file mode 100644 index 000000000..938ad14b4 --- /dev/null +++ b/tests/Database/DatabaseTransactionIdentityTest.php @@ -0,0 +1,113 @@ +connections(); + $callbacks = []; + + $base->beginTransaction(); + $writer->beginTransaction(); + $base->afterCommit(function () use (&$callbacks): void { + $callbacks[] = 'base committed'; + }); + $writer->afterRollBack(function () use (&$callbacks): void { + $callbacks[] = 'writer rolled back'; + }); + + $writer->transaction(function () use ($writer, &$callbacks): void { + $writer->afterCommit(function () use (&$callbacks): void { + $callbacks[] = 'writer committed'; + }); + }); + + $base->commit(); + + $this->assertSame(['base committed'], $callbacks); + $this->assertTrue($writer->hasApplicableTransaction()); + $this->assertFalse($base->hasApplicableTransaction()); + + $writer->rollBack(); + + $this->assertSame(['base committed', 'writer rolled back'], $callbacks); + $this->assertTrue($writer->getTransactionManager()->getPendingTransactions()->isEmpty()); + $this->assertTrue($writer->getTransactionManager()->getCommittedTransactions()->isEmpty()); + } + + public function testRollingBackASavepointOnlyReleasesItsOwnersCallbacks(): void + { + [$base, $writer] = $this->connections(); + $callbacks = []; + + $base->beginTransaction(); + $writer->beginTransaction(); + $writer->beginTransaction(); + $writer->afterRollBack(function () use (&$callbacks): void { + $callbacks[] = 'savepoint rolled back'; + }); + $base->afterCommit(function () use (&$callbacks): void { + $callbacks[] = 'base committed'; + }); + $writer->rollBack(); + + $this->assertSame(['savepoint rolled back'], $callbacks); + + $writer->afterCommit(function () use (&$callbacks): void { + $callbacks[] = 'writer committed'; + }); + $writer->commit(); + $this->assertSame(['savepoint rolled back', 'writer committed'], $callbacks); + + $base->commit(); + $this->assertSame(['savepoint rolled back', 'writer committed', 'base committed'], $callbacks); + } + + public function testDisconnectOnlyResetsItsOwnersTransactionRecords(): void + { + [$base, $writer] = $this->connections(); + $callbacks = []; + + $base->beginTransaction(); + $writer->beginTransaction(); + $base->afterRollBack(function () use (&$callbacks): void { + $callbacks[] = 'base rolled back'; + }); + $writer->afterRollBack(function () use (&$callbacks): void { + $callbacks[] = 'writer rolled back'; + }); + + $writer->disconnect(); + $this->assertSame(['writer rolled back'], $callbacks); + $this->assertTrue($base->hasApplicableTransaction()); + + $base->rollBack(); + $this->assertSame(['writer rolled back', 'base rolled back'], $callbacks); + } + + /** + * Create independent sessions for the same configured connection name. + * + * @return array{SQLiteConnection, SQLiteConnection} + */ + protected function connections(): array + { + $manager = new DatabaseTransactionsManager; + $base = new SQLiteConnection(new PDO('sqlite::memory:'), config: ['name' => 'testing']); + $writer = new SQLiteConnection(new PDO('sqlite::memory:'), config: ['name' => 'testing']); + $writer->setReadWriteType('write'); + $base->setTransactionManager($manager); + $writer->setTransactionManager($manager); + + return [$base, $writer]; + } +} diff --git a/tests/Foundation/Testing/DatabaseTransactionsManagerTest.php b/tests/Foundation/Testing/DatabaseTransactionsManagerTest.php index 5fbfdd004..e008e17ed 100644 --- a/tests/Foundation/Testing/DatabaseTransactionsManagerTest.php +++ b/tests/Foundation/Testing/DatabaseTransactionsManagerTest.php @@ -138,6 +138,38 @@ public function testPooledChildTransactionsDoNotCountAsTheirParentsWrapper(): vo $this->assertFalse($manager->hasApplicableTransaction('foo')); } + public function testRoleWrappersExcludeOnlyTheirResolvedOwnersTransaction(): void + { + $manager = new DatabaseTransactionsManager(['foo::write']); + $database = m::mock(DatabaseManager::class); + $writer = new SQLiteConnection(new PDO('sqlite::memory:'), config: ['name' => 'foo']); + $writer->setReadWriteType('write'); + $database->shouldReceive('connection')->once()->with('foo::write')->andReturn($writer); + $manager->beginTransactions($database); + + $this->assertFalse($writer->hasApplicableTransaction()); + + $base = new SQLiteConnection(new PDO('sqlite::memory:'), config: ['name' => 'foo']); + $base->setTransactionManager($manager); + $base->beginTransaction(); + $this->assertTrue($base->hasApplicableTransaction()); + + $called = false; + $writer->transaction(function () use ($writer, &$called): void { + $writer->afterCommit(function () use (&$called): void { + $called = true; + }); + }); + + $this->assertTrue($called); + $this->assertTrue($base->hasApplicableTransaction()); + $this->assertFalse($writer->hasApplicableTransaction()); + + $base->rollBack(); + $writer->rollBack(); + $this->assertTrue($manager->getPendingTransactions()->isEmpty()); + } + /** * Begin the real wrappers while keeping these manager tests independent of an application. */ diff --git a/tests/Foundation/Testing/RefreshDatabaseTest.php b/tests/Foundation/Testing/RefreshDatabaseTest.php index d091677b8..571cb06be 100644 --- a/tests/Foundation/Testing/RefreshDatabaseTest.php +++ b/tests/Foundation/Testing/RefreshDatabaseTest.php @@ -251,7 +251,7 @@ public function testRefreshDatabaseDoesNotPublishMigratedWhenTheAfterHookFails() $eventDispatcher = m::mock(Dispatcher::class); $connection = m::mock(PdoConnection::class); $connection->expects('getPdo')->andReturn($pdo); - $connection->expects('getName')->andReturn('default'); + $connection->expects('getNameWithReadWriteType')->andReturn('default'); $connection->expects('setTransactionManager'); $connection->expects('getEventDispatcher')->twice()->andReturn($eventDispatcher); $connection->expects('unsetEventDispatcher')->twice(); @@ -661,7 +661,7 @@ public function testRollbackRestoresTheDispatcherAndKeepsMigrationBookkeepingWhe protected function getMockedDatabase(): DatabaseManager { $connection = m::mock(ConnectionInterface::class); - $connection->expects('getName')->andReturn('default'); + $connection->expects('getNameWithReadWriteType')->andReturn('default'); $eventDispatcher = m::mock(Dispatcher::class); $connection->expects('getEventDispatcher') ->twice() diff --git a/tests/Foundation/Testing/RequestContextSynchronizerTest.php b/tests/Foundation/Testing/RequestContextSynchronizerTest.php index df54dd6aa..6717bdf7b 100644 --- a/tests/Foundation/Testing/RequestContextSynchronizerTest.php +++ b/tests/Foundation/Testing/RequestContextSynchronizerTest.php @@ -7,11 +7,45 @@ use ArrayObject; use Hypervel\Context\CoroutineContext; use Hypervel\Coroutine\Waiter; +use Hypervel\Database\DatabaseTransactionRecord; +use Hypervel\Database\DatabaseTransactionState; +use Hypervel\Database\SQLiteConnection; use Hypervel\Foundation\Testing\RequestContextSynchronizer; use Hypervel\Tests\TestCase; +use PDO; class RequestContextSynchronizerTest extends TestCase { + public function testTransactionSyncOnlyReplacesTheRetainedRoleOwnersRecords(): void + { + $base = new DatabaseTransactionRecord('testing', 1); + $writer = new DatabaseTransactionRecord('testing::write', 1); + $savepoint = new DatabaseTransactionRecord('testing::write', 2, $writer); + $source = new DatabaseTransactionState; + $source->pending->push($writer); + $source->committed->push($savepoint); + $source->current['testing::write'] = $writer; + + $target = new DatabaseTransactionState; + $target->pending->push($base, $writer, $savepoint); + $target->current = ['testing' => $base, 'testing::write' => $savepoint]; + + $connection = new SQLiteConnection(new PDO('sqlite::memory:'), config: ['name' => 'testing']); + $connection->setReadWriteType('write'); + $synchronizer = new RequestContextSynchronizer; + $synchronizer->syncTransactionState($source, $target, ['testing::write' => $connection]); + + $this->assertSame([$base, $writer], $target->pending->all()); + $this->assertSame([$savepoint], $target->committed->all()); + $this->assertSame(['testing' => $base, 'testing::write' => $writer], $target->current); + + $synchronizer->syncTransactionState(new DatabaseTransactionState, $target, ['testing::write' => $connection]); + + $this->assertSame([$base], $target->pending->all()); + $this->assertTrue($target->committed->isEmpty()); + $this->assertSame(['testing' => $base], $target->current); + } + public function testSyncContextKeysToParentCopiesPresentKeysAndForgetsMissingKeys(): void { CoroutineContext::set('foundation.testing.present', 'old'); diff --git a/tests/Pennant/TransactionTest.php b/tests/Pennant/TransactionTest.php index 651f1f63d..362222fde 100644 --- a/tests/Pennant/TransactionTest.php +++ b/tests/Pennant/TransactionTest.php @@ -6,6 +6,7 @@ use Hypervel\Contracts\Foundation\Application; use Hypervel\Database\Events\QueryExecuted; +use Hypervel\Database\SimpleConnectionResolver; use Hypervel\Foundation\Testing\RefreshDatabase; use Hypervel\Pennant\Drivers\DatabaseDriver; use Hypervel\Pennant\Feature; @@ -32,6 +33,9 @@ protected function defineEnvironment(Application $app): void 'driver' => 'database', 'cache' => ['store' => 'array'], ]); + $config->set('database.connections.independent', ['driver' => 'sqlite', 'database' => ':memory:']); + $config->set('pennant.stores.independent', ['driver' => 'database', 'connection' => 'independent']); + $config->set('pennant.stores.independent-writer', ['driver' => 'database', 'connection' => 'independent::write']); } protected function setUpInCoroutine(): void @@ -106,6 +110,41 @@ public function testCommittingKeepsValuesMemoizedInsideTheTransaction(): void $this->assertSame(0, $queries->count); } + public function testRoleRollbackForgetsOnlyValuesTrackedByThatSession(): void + { + $resolver = $this->app->make('db.resolver'); + $this->app->instance('db.resolver', new SimpleConnectionResolver($this->app->make('db'))); + $base = DB::connection('independent'); + $writer = DB::connection('independent::write'); + $this->assertNotSame($base, $writer); + + $this->app->instance(TransactionFeatureWithBeforeHook::class, $feature = new TransactionFeatureWithBeforeHook); + $feature->value = 'inside'; + + try { + $base->beginTransaction(); + $writer->beginTransaction(); + + foreach (['independent', 'independent-writer'] as $store) { + Feature::store($store)->define(TransactionFeatureWithBeforeHook::class); + $this->assertSame('inside', Feature::store($store)->for('taylor')->value(TransactionFeatureWithBeforeHook::class)); + } + + $writer->rollBack(); + $feature->value = 'after'; + + $this->assertSame('after', Feature::store('independent-writer')->for('taylor')->value(TransactionFeatureWithBeforeHook::class)); + $this->assertSame('inside', Feature::store('independent')->for('taylor')->value(TransactionFeatureWithBeforeHook::class)); + + $base->rollBack(); + $this->assertSame('after', Feature::store('independent')->for('taylor')->value(TransactionFeatureWithBeforeHook::class)); + } finally { + $base->rollBack(0); + $writer->rollBack(0); + $this->app->instance('db.resolver', $resolver); + } + } + public function testFlushingAStoreKeepsOtherStoresValuesTrackedForRollback(): void { Feature::store('secondary')->define('foo', fn () => 'resolved'); From fca61fbdeda6b229f9809070272bf6ecce15cddd Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 20:47:02 +0000 Subject: [PATCH 04/33] feat(bus): capture context for dispatch lock cleanup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Capture release context before unique or debounce lock acquisition and retain the composed wrapper with live ownership records and independent rollback snapshots. Release through the acquisition context while restoring the caller’s context afterwards. Keep registration boot-scoped, avoid closure allocation when no wrapper is installed, and clear registrations with existing framework test cleanup. Cover exact-owner release, composition, capture failures and delayed coroutine cleanup without introducing a separate lock registry. --- src/bus/src/DebounceLock.php | 3 +- src/bus/src/DispatchLockContext.php | 122 ++++++++- src/bus/src/UniqueLock.php | 3 +- tests/Bus/DispatchLockReleaseContextTest.php | 248 +++++++++++++++++++ 4 files changed, 369 insertions(+), 7 deletions(-) create mode 100644 tests/Bus/DispatchLockReleaseContextTest.php diff --git a/src/bus/src/DebounceLock.php b/src/bus/src/DebounceLock.php index 71c078a6c..5d4822e3f 100644 --- a/src/bus/src/DebounceLock.php +++ b/src/bus/src/DebounceLock.php @@ -45,13 +45,14 @@ public function acquire(mixed $job, ?int $debounceFor = null, ?int $maxWait = nu public function acquireForDispatch(object $job, ?int $debounceFor = null, ?int $maxWait = null): array { [$cache, $key, $ttl, $maxWait] = $this->resolveLockValues($job, $debounceFor, $maxWait); + $releaseContext = DispatchLockContext::captureReleaseContext(); $result = $this->acquireResolvedLock($cache, $key, $ttl, $maxWait); if (isset(class_uses_recursive($job)[Queueable::class])) { $job->debounceOwner = $result['owner']; } - DispatchLockContext::registerDebounce($job, $cache, $key, $result['owner']); + DispatchLockContext::registerDebounce($job, $cache, $key, $result['owner'], $releaseContext); return $result; } diff --git a/src/bus/src/DispatchLockContext.php b/src/bus/src/DispatchLockContext.php index ec1b101d4..d8e28d5aa 100644 --- a/src/bus/src/DispatchLockContext.php +++ b/src/bus/src/DispatchLockContext.php @@ -4,6 +4,7 @@ namespace Hypervel\Bus; +use Closure; use Hypervel\Contracts\Cache\Repository as Cache; use Throwable; use WeakMap; @@ -11,8 +12,6 @@ /** * Track dispatch lock ownership until a queue accepts the job. * - * @internal - * * @phpstan-type UniqueMetadata array{ * laravel_unique_job_cache_store: null|string, * laravel_unique_job_key: string, @@ -23,11 +22,13 @@ * metadata: null|UniqueMetadata, * unique: null|LockProvenance, * debounce: null|LockProvenance, + * releaseContext: null|Closure(Closure(): void): void, * delegated: bool * } * @phpstan-type LockSnapshot array{ * unique: null|LockProvenance, - * debounce: null|LockProvenance + * debounce: null|LockProvenance, + * releaseContext: null|Closure(Closure(): void): void * } */ class DispatchLockContext @@ -39,8 +40,73 @@ class DispatchLockContext */ protected static ?WeakMap $records = null; + /** + * The callbacks that capture context before dispatch lock acquisition. + * + * @var list + */ + protected static array $releaseContextCallbacks = []; + + /** + * Register a callback that captures the context needed for deferred lock release. + * + * Boot-only. Callbacks persist for the worker lifetime and affect subsequent dispatches. + * Return a wrapper that invokes the release operation once and restores the previous + * context in finally, or null when no context is needed. The first wrapper is outermost. + * Passing null clears registrations; capture runs before acquiring a lock. + * + * @param null|callable(): (null|Closure(Closure(): void): void) $callback + */ + public static function captureReleaseContextUsing(?callable $callback): void + { + if ($callback === null) { + static::$releaseContextCallbacks = []; + + return; + } + + static::$releaseContextCallbacks[] = $callback(...); + } + + /** + * Capture and compose the current release context. + * + * @internal + * + * @return null|Closure(Closure(): void): void + */ + public static function captureReleaseContext(): ?Closure + { + $context = null; + + foreach (static::$releaseContextCallbacks as $callback) { + if (($wrapper = $callback()) === null) { + continue; + } + + if ($context === null) { + $context = $wrapper; + + continue; + } + + $outer = $context; + $context = static function (Closure $release) use ($outer, $wrapper): void { + $outer(static function () use ($wrapper, $release): void { + $wrapper($release); + }); + }; + } + + return $context; + } + /** * Register an acquired unique lock. + * + * @internal + * + * @param null|Closure(Closure(): void): void $releaseContext */ public static function registerUnique( object $job, @@ -48,6 +114,7 @@ public static function registerUnique( ?string $cacheStore, string $key, string $owner, + ?Closure $releaseContext = null, ): void { /** @var WeakMap $records */ // @phpstan-ignore assign.propertyType (PHPStan rejects an empty WeakMap for this invariant closed-shape value.) @@ -61,29 +128,38 @@ public static function registerUnique( 'laravel_unique_job_lock_owner' => $owner, ]; $record['unique'] = compact('cache', 'key', 'owner'); + $record['releaseContext'] = $releaseContext; $records[$job] = $record; } /** * Register an acquired debounce lock. + * + * @internal + * + * @param null|Closure(Closure(): void): void $releaseContext */ public static function registerDebounce( object $job, Cache $cache, string $key, string $owner, + ?Closure $releaseContext = null, ): void { /** @var WeakMap $records */ // @phpstan-ignore assign.propertyType (PHPStan rejects an empty WeakMap for this invariant closed-shape value.) $records = static::$records ??= new WeakMap; $record = $records[$job] ?? static::newRecord(); $record['debounce'] = compact('cache', 'key', 'owner'); + $record['releaseContext'] = $releaseContext; $records[$job] = $record; } /** * Peek at hidden unique-job payload metadata. * + * @internal + * * @return null|UniqueMetadata */ public static function peekPayloadMetadata(object $job): ?array @@ -93,6 +169,8 @@ public static function peekPayloadMetadata(object $job): ?array /** * Determine whether a dispatch owns any locks. + * + * @internal */ public static function has(object $job): bool { @@ -101,6 +179,8 @@ public static function has(object $job): bool /** * Delegate a dispatch's lock ownership to a registered callback. + * + * @internal */ public static function delegate(object $job): bool { @@ -109,6 +189,8 @@ public static function delegate(object $job): bool /** * Claim lock ownership in a delegated callback. + * + * @internal */ public static function claim(object $job): bool { @@ -117,6 +199,8 @@ public static function claim(object $job): bool /** * Mark a dispatch as accepted by a queue. + * + * @internal */ public static function accept(object $job): void { @@ -129,6 +213,8 @@ public static function accept(object $job): void /** * Release locks still owned by a dispatch. + * + * @internal */ public static function release(object $job): void { @@ -149,11 +235,14 @@ public static function release(object $job): void static::releaseSnapshot([ 'unique' => $record['unique'], 'debounce' => $record['debounce'], + 'releaseContext' => $record['releaseContext'], ]); } /** - * Take a release snapshot for work accepted by an in-memory timer. + * Take an independent release snapshot before accepting a dispatch. + * + * @internal * * @return null|LockSnapshot */ @@ -168,15 +257,36 @@ public static function snapshot(object $job): ?array return [ 'unique' => $record['unique'], 'debounce' => $record['debounce'], + 'releaseContext' => $record['releaseContext'], ]; } /** - * Release locks from an in-memory timer snapshot. + * Release locks from an independent dispatch snapshot. + * + * @internal * * @param LockSnapshot $snapshot */ public static function releaseSnapshot(array $snapshot): void + { + if ($snapshot['releaseContext'] !== null) { + $snapshot['releaseContext'](static function () use ($snapshot): void { + static::releaseLocks($snapshot); + }); + + return; + } + + static::releaseLocks($snapshot); + } + + /** + * Release both owned locks even if one cleanup fails. + * + * @param LockSnapshot $snapshot + */ + protected static function releaseLocks(array $snapshot): void { $exception = null; @@ -252,6 +362,7 @@ protected static function newRecord(): array 'metadata' => null, 'unique' => null, 'debounce' => null, + 'releaseContext' => null, 'delegated' => false, ]; } @@ -262,5 +373,6 @@ protected static function newRecord(): array public static function flushState(): void { static::$records = null; + static::$releaseContextCallbacks = []; } } diff --git a/src/bus/src/UniqueLock.php b/src/bus/src/UniqueLock.php index c3c235c10..03fcbd5c3 100644 --- a/src/bus/src/UniqueLock.php +++ b/src/bus/src/UniqueLock.php @@ -45,6 +45,7 @@ public function acquireForDispatch(object $job): bool { [$uniqueFor, $cache, $key, $supportsOwnership] = $this->resolveLockValues($job); $cacheStore = $cache instanceof Repository ? $cache->getName() : null; + $releaseContext = DispatchLockContext::captureReleaseContext(); $owner = $this->acquireResolvedLock( $job, @@ -59,7 +60,7 @@ public function acquireForDispatch(object $job): bool return false; } - DispatchLockContext::registerUnique($job, $cache, $cacheStore, $key, $owner); + DispatchLockContext::registerUnique($job, $cache, $cacheStore, $key, $owner, $releaseContext); return true; } diff --git a/tests/Bus/DispatchLockReleaseContextTest.php b/tests/Bus/DispatchLockReleaseContextTest.php new file mode 100644 index 000000000..c69a275a8 --- /dev/null +++ b/tests/Bus/DispatchLockReleaseContextTest.php @@ -0,0 +1,248 @@ +registerContextCapture(); + $order = []; + DispatchLockContext::captureReleaseContextUsing(static function () use (&$order): Closure { + return static function (Closure $release) use (&$order): void { + $order[] = CoroutineContext::get(ReleaseContextCacheStore::CONTEXT_KEY); + $release(); + }; + }); + $cache = new Repository(new ReleaseContextCacheStore); + $lock = $kind === 'unique' ? new UniqueLock($cache) : new DebounceLock($cache); + $job = new ReleaseContextJob; + + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, 'first:'); + $lock->acquireForDispatch($job); + $key = $lock::getKey($job); + $captured = DispatchLockContext::snapshot($job); + + if ($snapshot) { + DispatchLockContext::accept($job); + } + + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, 'second:'); + $other = new ReleaseContextJob; + $lock->acquireForDispatch($other); + + if ($snapshot) { + DispatchLockContext::releaseSnapshot($captured); + } else { + DispatchLockContext::release($job); + } + + $this->assertSame(['first:'], $order); + $this->assertSame('second:', CoroutineContext::get(ReleaseContextCacheStore::CONTEXT_KEY)); + $this->assertTrue($kind === 'unique' ? $cache->lock($key)->isLocked() : $cache->has($key)); + + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, 'first:'); + $this->assertFalse($kind === 'unique' ? $cache->lock($key)->isLocked() : $cache->has($key)); + DispatchLockContext::release($other); + $this->assertSame('first:', CoroutineContext::get(ReleaseContextCacheStore::CONTEXT_KEY)); + } + + /** + * Provide each lock's live-record and accepted-snapshot release paths. + */ + public static function releasePaths(): array + { + return [ + 'unique live record' => ['unique', false], + 'unique accepted snapshot' => ['unique', true], + 'debounce live record' => ['debounce', false], + 'debounce accepted snapshot' => ['debounce', true], + ]; + } + + public function testCaptureFailureLeavesNeitherLockNorTokenAcquired(): void + { + $exception = new RuntimeException('Cannot capture context.'); + DispatchLockContext::captureReleaseContextUsing(static fn () => throw $exception); + $cache = new Repository(new WorkerArrayStore); + $job = new ReleaseContextJob; + + foreach ([new UniqueLock($cache), new DebounceLock($cache)] as $lock) { + try { + $lock->acquireForDispatch($job); + $this->fail('Context capture should have failed before acquisition.'); + } catch (RuntimeException $caught) { + $this->assertSame($exception, $caught); + } + } + + $this->assertFalse($cache->lock(UniqueLock::getKey($job))->isLocked()); + $this->assertFalse($cache->has(DebounceLock::getKey($job))); + $this->assertFalse(DispatchLockContext::has($job)); + } + + public function testReleaseFailureRestoresTheCallingContext(): void + { + $this->registerContextCapture(); + $exception = new RuntimeException('Cleanup failed.'); + DispatchLockContext::captureReleaseContextUsing(static fn () => static fn () => throw $exception); + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, 'first:'); + $job = new ReleaseContextJob; + (new UniqueLock(new Repository(new ReleaseContextCacheStore)))->acquireForDispatch($job); + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, 'second:'); + + try { + DispatchLockContext::release($job); + $this->fail('The release wrapper should have failed.'); + } catch (RuntimeException $caught) { + $this->assertSame($exception, $caught); + } + + $this->assertSame('second:', CoroutineContext::get(ReleaseContextCacheStore::CONTEXT_KEY)); + $this->assertFalse(DispatchLockContext::has($job)); + } + + public function testClosingTimerReleasesTheLockOutsideItsAcquiringCoroutine(): void + { + $this->registerContextCapture(); + $released = new Channel(1); + DispatchLockContext::captureReleaseContextUsing(static function () use ($released): Closure { + return static function (Closure $release) use ($released): void { + $release(); + $released->push(true); + }; + }); + + $container = new Container; + Container::setInstance($container); + $container->instance(Dispatcher::class, new EventsDispatcher($container)); + $timer = new Timer; + $queue = new BackgroundQueue(timer: $timer); + $queue->setContainer($container); + $queue->setConnectionName('background'); + $cache = new Repository(new ReleaseContextCacheStore); + $job = new ReleaseContextJob; + + try { + (new Waiter)->wait(function () use ($queue, $cache, $job): void { + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, 'first:'); + $this->assertTrue((new UniqueLock($cache))->acquireForDispatch($job)); + $queue->later(60, $job); + $this->assertFalse(DispatchLockContext::has($job)); + }); + + CoordinatorManager::until(Constants::WORKER_EXIT)->resume(); + $this->assertTrue($released->pop(2)); + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, 'first:'); + $this->assertFalse($cache->lock(UniqueLock::getKey($job))->isLocked()); + } finally { + $timer->clearAll(); + $released->close(); + } + } + + public function testNullCaptureAndClearingRegistrationsDoNotWrapRelease(): void + { + DispatchLockContext::captureReleaseContextUsing(static fn () => null); + $this->assertNull(DispatchLockContext::captureReleaseContext()); + DispatchLockContext::captureReleaseContextUsing(static fn () => throw new RuntimeException); + DispatchLockContext::captureReleaseContextUsing(null); + $this->assertNull(DispatchLockContext::captureReleaseContext()); + } + + /** + * Capture the cache prefix without retaining an entire coroutine context. + */ + protected function registerContextCapture(): void + { + DispatchLockContext::captureReleaseContextUsing(static function (): Closure { + $prefix = CoroutineContext::get(ReleaseContextCacheStore::CONTEXT_KEY); + + return static function (Closure $release) use ($prefix): void { + $previous = CoroutineContext::get(ReleaseContextCacheStore::CONTEXT_KEY); + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, $prefix); + + try { + $release(); + } finally { + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, $previous); + } + }; + }); + } +} + +class ReleaseContextJob implements ShouldBeUnique +{ + public int $debounceFor = 10; +} + +class ReleaseContextCacheStore extends WorkerArrayStore +{ + public const string CONTEXT_KEY = '__tests.dispatch_lock_prefix'; + + /** + * Resolve the namespace from the current execution. + */ + public function getPrefix(): string + { + return CoroutineContext::get(self::CONTEXT_KEY, ''); + } + + /** + * Create a lock in the current namespace. + */ + public function lock(string $name, int $seconds = 0, ?string $owner = null): ArrayLock + { + return parent::lock($this->getPrefix() . $name, $seconds, $owner); + } + + /** + * Read a value from the current namespace. + */ + public function get(string $key): mixed + { + return parent::get($this->getPrefix() . $key); + } + + /** + * Store a value in the current namespace. + */ + public function put(string $key, mixed $value, int $seconds): bool + { + return parent::put($this->getPrefix() . $key, $value, $seconds); + } + + /** + * Remove a value from the current namespace. + */ + public function forget(string $key): bool + { + return parent::forget($this->getPrefix() . $key); + } +} From 0c977ac81ef0f8c542b3dba25029b1ed0518c617 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 20:47:13 +0000 Subject: [PATCH 05/33] feat(bus): add opt-in transactional outbox dispatch options Add the ShouldQueueViaOutbox marker and fluent viaOutbox, withoutOutbox and onOutboxDatabase options to the existing job and pending-dispatch surfaces. Keep nullable state uninitialized until selected so ordinary serialized jobs gain no fields. Apply batch defaults without overwriting initialized job choices, including jobs added later and first jobs in batched chains. Propagate effective selection through queued listeners, mail, notifications and broadcasts without introducing producer-specific APIs. Preserve explicit false over markers and null fallthrough to markers. Skip uninitialized broadcast properties while retaining initialized nulls and existing payload formatting. Cover public APIs, batch precedence and wrapper propagation. --- src/broadcasting/src/BroadcastEvent.php | 16 ++ src/bus/src/Batch.php | 18 +- src/bus/src/Batchable.php | 2 + src/bus/src/InteractsWithQueueOutbox.php | 63 +++++++ src/bus/src/PendingBatch.php | 40 +++++ src/bus/src/Queueable.php | 2 + .../src/Queue/ShouldQueueViaOutbox.php | 9 + src/events/src/Dispatcher.php | 11 ++ src/foundation/src/Bus/PendingDispatch.php | 30 ++++ src/mail/src/SendQueuedMailable.php | 11 ++ .../src/SendQueuedNotifications.php | 11 ++ tests/Broadcasting/BroadcastEventTest.php | 22 +++ tests/Bus/BusBatchTest.php | 46 +++++ tests/Bus/OutboxOptionsTest.php | 107 +++++++++++ tests/Queue/Outbox/OutboxWrapperTest.php | 167 ++++++++++++++++++ 15 files changed, 552 insertions(+), 3 deletions(-) create mode 100644 src/bus/src/InteractsWithQueueOutbox.php create mode 100644 src/contracts/src/Queue/ShouldQueueViaOutbox.php create mode 100644 tests/Bus/OutboxOptionsTest.php create mode 100644 tests/Queue/Outbox/OutboxWrapperTest.php diff --git a/src/broadcasting/src/BroadcastEvent.php b/src/broadcasting/src/BroadcastEvent.php index 8a0552c80..e434a3d0b 100644 --- a/src/broadcasting/src/BroadcastEvent.php +++ b/src/broadcasting/src/BroadcastEvent.php @@ -7,6 +7,7 @@ use Hypervel\Bus\Queueable; use Hypervel\Contracts\Broadcasting\Factory as BroadcastingFactory; use Hypervel\Contracts\Queue\ShouldQueue; +use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; use Hypervel\Contracts\Support\Arrayable; use Hypervel\Queue\Attributes\Backoff; use Hypervel\Queue\Attributes\DeleteWhenMissingModels; @@ -67,6 +68,17 @@ public function __construct(mixed $event) $this->timeout = $this->getAttributeValue($event, Timeout::class, 'timeout'); $this->backoff = $this->getAttributeValue($event, Backoff::class, 'backoff'); $this->afterCommit = property_exists($event, 'afterCommit') ? $event->afterCommit : null; + + if (isset($event->viaOutbox)) { + $this->viaOutbox = $event->viaOutbox; + } elseif ($event instanceof ShouldQueueViaOutbox) { + $this->viaOutbox = true; + } + + if (isset($event->outboxDatabase)) { + $this->outboxDatabase = $event->outboxDatabase; + } + $this->maxExceptions = $this->getAttributeValue($event, MaxExceptions::class, 'maxExceptions'); $this->deleteWhenMissingModels = $this->getAttributeValue($event, DeleteWhenMissingModels::class, 'deleteWhenMissingModels', $this->deleteWhenMissingModels); } @@ -115,6 +127,10 @@ protected function getPayloadFromEvent(mixed $event): array $payload = []; foreach ((new ReflectionClass($event))->getProperties(ReflectionProperty::IS_PUBLIC) as $property) { + if (! $property->isInitialized($event)) { + continue; + } + $payload[$property->getName()] = $this->formatProperty($property->getValue($event)); } diff --git a/src/bus/src/Batch.php b/src/bus/src/Batch.php index 79735c2b0..0096dd3ed 100644 --- a/src/bus/src/Batch.php +++ b/src/bus/src/Batch.php @@ -76,11 +76,23 @@ public function add(array|object $jobs): ?Batch $first->allOnConnection($this->options['connection']); } - return $first->chain($chain->slice(1)->values()->all()); + $job = $first->chain($chain->slice(1)->values()->all()); + } else { + $job->withBatchId($this->id); + + ++$count; } - $job->withBatchId($this->id); - ++$count; + if (array_key_exists('viaOutbox', $this->options) || array_key_exists('outboxDatabase', $this->options)) { + // Preserve initialized nulls as well as explicit values on each job. + $properties = get_object_vars($job); + + foreach (['viaOutbox', 'outboxDatabase'] as $property) { + if (array_key_exists($property, $this->options) && ! array_key_exists($property, $properties)) { + $job->{$property} = $this->options[$property]; + } + } + } return $job; }); diff --git a/src/bus/src/Batchable.php b/src/bus/src/Batchable.php index bb90b12f0..4c5ba4e10 100644 --- a/src/bus/src/Batchable.php +++ b/src/bus/src/Batchable.php @@ -11,6 +11,8 @@ trait Batchable { + use InteractsWithQueueOutbox; + /** * The batch ID (if applicable). */ diff --git a/src/bus/src/InteractsWithQueueOutbox.php b/src/bus/src/InteractsWithQueueOutbox.php new file mode 100644 index 000000000..234426ed1 --- /dev/null +++ b/src/bus/src/InteractsWithQueueOutbox.php @@ -0,0 +1,63 @@ +viaOutbox = true; + + return $this; + } + + /** + * Dispatch the job without the transactional outbox. + */ + public function withoutOutbox(): static + { + $this->viaOutbox = false; + $this->outboxDatabase = null; + + return $this; + } + + /** + * Stage the job on the given database connection. + */ + public function onOutboxDatabase(UnitEnum|string $connection): static + { + $connection = (string) enum_value($connection); + + if ($connection === '') { + throw new InvalidArgumentException('The outbox database connection must not be empty.'); + } + + $this->outboxDatabase = $connection; + $this->viaOutbox = true; + + return $this; + } +} diff --git a/src/bus/src/PendingBatch.php b/src/bus/src/PendingBatch.php index ab42b5277..dd00e2d17 100644 --- a/src/bus/src/PendingBatch.php +++ b/src/bus/src/PendingBatch.php @@ -12,6 +12,7 @@ use Hypervel\Support\Arr; use Hypervel\Support\Collection; use Hypervel\Support\Traits\Conditionable; +use InvalidArgumentException; use Laravel\SerializableClosure\SerializableClosure; use RuntimeException; use Throwable; @@ -283,6 +284,45 @@ public function queue(): ?string return $this->options['queue'] ?? null; } + /** + * Dispatch batched jobs through the transactional outbox by default. + */ + public function viaOutbox(): static + { + $this->options['viaOutbox'] = true; + + return $this; + } + + /** + * Dispatch batched jobs without the transactional outbox by default. + */ + public function withoutOutbox(): static + { + $this->options['viaOutbox'] = false; + + unset($this->options['outboxDatabase']); + + return $this; + } + + /** + * Stage batched jobs on the given database connection by default. + */ + public function onOutboxDatabase(UnitEnum|string $connection): static + { + $connection = (string) enum_value($connection); + + if ($connection === '') { + throw new InvalidArgumentException('The outbox database connection must not be empty.'); + } + + $this->options['outboxDatabase'] = $connection; + $this->options['viaOutbox'] = true; + + return $this; + } + /** * Add additional data into the batch's options array. */ diff --git a/src/bus/src/Queueable.php b/src/bus/src/Queueable.php index 9a8d5f5b8..15678c77f 100644 --- a/src/bus/src/Queueable.php +++ b/src/bus/src/Queueable.php @@ -20,6 +20,8 @@ trait Queueable { + use InteractsWithQueueOutbox; + /** * The name of the connection the job should be sent to. */ diff --git a/src/contracts/src/Queue/ShouldQueueViaOutbox.php b/src/contracts/src/Queue/ShouldQueueViaOutbox.php new file mode 100644 index 000000000..881bd3b40 --- /dev/null +++ b/src/contracts/src/Queue/ShouldQueueViaOutbox.php @@ -0,0 +1,9 @@ +afterCommit = property_exists($listener, 'afterCommit') ? $listener->afterCommit : null; } + if (isset($listener->viaOutbox)) { + $job->viaOutbox = $listener->viaOutbox; + } elseif ($listener instanceof ShouldQueueViaOutbox) { + $job->viaOutbox = true; + } + + if (isset($listener->outboxDatabase)) { + $job->outboxDatabase = $listener->outboxDatabase; + } + $job->backoff = method_exists($listener, 'backoff') ? $listener->backoff(...$data) : $this->getAttributeValue($listener, Backoff::class, 'backoff'); $job->maxExceptions = $this->getAttributeValue($listener, MaxExceptions::class, 'maxExceptions'); $job->retryUntil = method_exists($listener, 'retryUntil') ? $listener->retryUntil(...$data) : null; diff --git a/src/foundation/src/Bus/PendingDispatch.php b/src/foundation/src/Bus/PendingDispatch.php index ace4fe525..1f0bf3d27 100644 --- a/src/foundation/src/Bus/PendingDispatch.php +++ b/src/foundation/src/Bus/PendingDispatch.php @@ -146,6 +146,36 @@ public function beforeCommit(): static return $this; } + /** + * Dispatch the job through the transactional outbox. + */ + public function viaOutbox(): static + { + $this->job->viaOutbox(); + + return $this; + } + + /** + * Dispatch the job without the transactional outbox. + */ + public function withoutOutbox(): static + { + $this->job->withoutOutbox(); + + return $this; + } + + /** + * Stage the job on the given database connection. + */ + public function onOutboxDatabase(UnitEnum|string $connection): static + { + $this->job->onOutboxDatabase($connection); + + return $this; + } + /** * Set the jobs that should run if this job is successful. */ diff --git a/src/mail/src/SendQueuedMailable.php b/src/mail/src/SendQueuedMailable.php index eef662a46..29ec7d74c 100644 --- a/src/mail/src/SendQueuedMailable.php +++ b/src/mail/src/SendQueuedMailable.php @@ -10,6 +10,7 @@ use Hypervel\Contracts\Mail\Mailable as MailableContract; use Hypervel\Contracts\Queue\ShouldBeEncrypted; use Hypervel\Contracts\Queue\ShouldQueueAfterCommit; +use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; use Hypervel\Queue\Attributes\Backoff; use Hypervel\Queue\Attributes\Connection; use Hypervel\Queue\Attributes\MaxExceptions; @@ -60,6 +61,16 @@ public function __construct( $this->afterCommit = property_exists($mailable, 'afterCommit') ? $mailable->afterCommit : null; } + if (isset($mailable->viaOutbox)) { + $this->viaOutbox = $mailable->viaOutbox; + } elseif ($mailable instanceof ShouldQueueViaOutbox) { + $this->viaOutbox = true; + } + + if (isset($mailable->outboxDatabase)) { + $this->outboxDatabase = $mailable->outboxDatabase; + } + $this->connection = $this->getAttributeValue($mailable, Connection::class, 'connection'); $this->maxExceptions = $this->getAttributeValue($mailable, MaxExceptions::class, 'maxExceptions'); $this->queue = $this->getAttributeValue($mailable, QueueAttribute::class, 'queue'); diff --git a/src/notifications/src/SendQueuedNotifications.php b/src/notifications/src/SendQueuedNotifications.php index 566ce6d2e..e3555df6c 100644 --- a/src/notifications/src/SendQueuedNotifications.php +++ b/src/notifications/src/SendQueuedNotifications.php @@ -9,6 +9,7 @@ use Hypervel\Contracts\Queue\ShouldBeEncrypted; use Hypervel\Contracts\Queue\ShouldQueue; use Hypervel\Contracts\Queue\ShouldQueueAfterCommit; +use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; use Hypervel\Database\Eloquent\Collection as EloquentCollection; use Hypervel\Database\Eloquent\Model; use Hypervel\Queue\Attributes\Backoff; @@ -95,6 +96,16 @@ public function __construct(mixed $notifiables, mixed $notification, ?array $cha $this->afterCommit = property_exists($notification, 'afterCommit') ? $notification->afterCommit : null; } + if (isset($notification->viaOutbox)) { + $this->viaOutbox = $notification->viaOutbox; + } elseif ($notification instanceof ShouldQueueViaOutbox) { + $this->viaOutbox = true; + } + + if (isset($notification->outboxDatabase)) { + $this->outboxDatabase = $notification->outboxDatabase; + } + $this->shouldBeEncrypted = $notification instanceof ShouldBeEncrypted; } diff --git a/tests/Broadcasting/BroadcastEventTest.php b/tests/Broadcasting/BroadcastEventTest.php index f5617e3d9..1546ff959 100644 --- a/tests/Broadcasting/BroadcastEventTest.php +++ b/tests/Broadcasting/BroadcastEventTest.php @@ -7,6 +7,7 @@ use Exception; use Hypervel\Broadcasting\BroadcastEvent; use Hypervel\Broadcasting\InteractsWithBroadcasting; +use Hypervel\Bus\Queueable; use Hypervel\Contracts\Broadcasting\Broadcaster; use Hypervel\Contracts\Broadcasting\Factory as BroadcastingFactory; use Hypervel\Contracts\Broadcasting\ShouldBroadcast; @@ -56,6 +57,27 @@ public function testManualParameterSpecification(): void (new BroadcastEvent($event))->handle($manager); } + public function testUninitializedPublicPropertiesAreOmittedFromThePayload(): void + { + $event = new class extends TestBroadcastEvent { + use Queueable; + + public string $uninitialized; + + public ?string $optional = null; + }; + + $payload = get_object_vars($event); + $payload['collection'] = ['foo' => 'bar']; + + $broadcaster = m::mock(Broadcaster::class); + $broadcaster->expects('broadcast')->with(['test-channel'], $event::class, $payload); + $manager = m::mock(BroadcastingFactory::class); + $manager->expects('connection')->with(null)->andReturn($broadcaster); + + (new BroadcastEvent($event))->handle($manager); + } + public function testSpecificBroadcasterGiven(): void { $broadcaster = m::mock(Broadcaster::class); diff --git a/tests/Bus/BusBatchTest.php b/tests/Bus/BusBatchTest.php index a1d0d27d5..9f2547b34 100644 --- a/tests/Bus/BusBatchTest.php +++ b/tests/Bus/BusBatchTest.php @@ -180,6 +180,52 @@ public function testJobsCanBeAddedToPendingBatch(): void $this->assertCount(2, $batch->jobs); } + public function testStoredOutboxDefaultsApplyToNewJobsWithoutReplacingExplicitChoices(): void + { + Queue::fake(); + + $defaultJob = new class { + use Batchable; + }; + $directJob = (new TestBatchJob)->onOutboxDatabase('previous')->withoutOutbox(); + $otherOwnerJob = (new TestBatchJob)->onOutboxDatabase('other'); + $nullJob = new TestBatchJob; + $nullJob->viaOutbox = null; + $nullJob->outboxDatabase = null; + + $batch = Bus::batch([$defaultJob, $directJob, $otherOwnerJob, $nullJob]) + ->onOutboxDatabase('billing') + ->dispatch(); + + $this->assertTrue($defaultJob->viaOutbox); + $this->assertSame('billing', $defaultJob->outboxDatabase); + $this->assertFalse($directJob->viaOutbox); + $this->assertNull($directJob->outboxDatabase); + $this->assertSame('other', $otherOwnerJob->outboxDatabase); + $this->assertNull($nullJob->viaOutbox); + $this->assertNull($nullJob->outboxDatabase); + + $added = new TestBatchJob; + $batch->fresh()->add([$added]); + + $this->assertTrue($added->viaOutbox); + $this->assertSame('billing', $added->outboxDatabase); + $this->assertSame(5, $batch->fresh()->totalJobs); + } + + public function testBatchOutboxDefaultsDoNotPropagateToLaterChainJobs(): void + { + Queue::fake(); + + $first = new TestBatchJob; + $second = new TestBatchJob; + + Bus::batch([[$first, $second]])->viaOutbox()->dispatch(); + + $this->assertTrue($first->viaOutbox); + $this->assertArrayNotHasKey('viaOutbox', get_object_vars(unserialize($first->chained[0]))); + } + public function testJobsCanBeAddedToThePendingBatchFromIterable(): void { $batch = new PendingBatch($this->app, collect()); diff --git a/tests/Bus/OutboxOptionsTest.php b/tests/Bus/OutboxOptionsTest.php new file mode 100644 index 000000000..637308076 --- /dev/null +++ b/tests/Bus/OutboxOptionsTest.php @@ -0,0 +1,107 @@ +assertStringNotContainsString('viaOutbox', serialize($job)); + $this->assertStringNotContainsString('outboxDatabase', serialize($job)); + + $this->assertSame($job, $job->onOutboxDatabase(OutboxDatabase::Billing)); + $this->assertTrue($job->viaOutbox); + $this->assertSame('billing', $job->outboxDatabase); + + $restored = unserialize(serialize($job)); + $this->assertTrue($restored->viaOutbox); + $this->assertSame('billing', $restored->outboxDatabase); + + $this->assertSame($job, $job->withoutOutbox()); + $this->assertFalse($job->viaOutbox); + $this->assertNull($job->outboxDatabase); + $this->assertSame($job, $job->viaOutbox()); + $this->assertTrue($job->viaOutbox); + $this->assertNull($job->outboxDatabase); + } + } + + public function testPendingDispatchForwardsOutboxOptions(): void + { + $job = new OutboxQueueableJob; + $dispatcher = m::mock(Dispatcher::class); + Container::getInstance()->instance(Dispatcher::class, $dispatcher); + $dispatcher->expects('dispatch')->with($job)->andReturnNull(); + + $pending = new PendingDispatch($job); + $this->assertSame($pending, $pending->onOutboxDatabase('billing')); + $this->assertSame('billing', $job->outboxDatabase); + $this->assertSame($pending, $pending->withoutOutbox()); + $this->assertFalse($job->viaOutbox); + $this->assertNull($job->outboxDatabase); + $this->assertSame($pending, $pending->viaOutbox()); + $this->assertTrue($job->viaOutbox); + + unset($pending); + } + + public function testPendingBatchStoresOnlyOutboxDefaults(): void + { + $batch = new PendingBatch(Container::getInstance(), new Collection); + $this->assertSame($batch, $batch->onOutboxDatabase(OutboxDatabase::Billing)); + $this->assertSame(['outboxDatabase' => 'billing', 'viaOutbox' => true], $batch->options); + $this->assertSame($batch, $batch->withoutOutbox()); + $this->assertSame(['viaOutbox' => false], $batch->options); + $this->assertSame($batch, $batch->viaOutbox()); + $this->assertSame(['viaOutbox' => true], $batch->options); + } + + public function testEmptyDatabaseIsRejectedWithoutChangingSelection(): void + { + foreach ([new OutboxQueueableJob, new PendingBatch(Container::getInstance(), new Collection)] as $target) { + try { + $target->onOutboxDatabase(''); + $this->fail('An empty owner must not select outbox delivery.'); + } catch (InvalidArgumentException $exception) { + $this->assertSame('The outbox database connection must not be empty.', $exception->getMessage()); + } + + $this->assertFalse(isset($target->viaOutbox)); + } + } +} + +class OutboxQueueableJob +{ + use Queueable; +} + +class OutboxBatchableJob +{ + use Batchable; +} + +class OutboxCombinedJob +{ + use Batchable; + use Queueable; +} + +enum OutboxDatabase: string +{ + case Billing = 'billing'; +} diff --git a/tests/Queue/Outbox/OutboxWrapperTest.php b/tests/Queue/Outbox/OutboxWrapperTest.php new file mode 100644 index 000000000..9945a8777 --- /dev/null +++ b/tests/Queue/Outbox/OutboxWrapperTest.php @@ -0,0 +1,167 @@ +wrap($source, $wrapper); + $this->assertArrayNotHasKey('viaOutbox', get_object_vars($job)); + $this->assertArrayNotHasKey('outboxDatabase', get_object_vars($job)); + + $source->onOutboxDatabase('billing'); + $job = $this->wrap($source, $wrapper); + $this->assertTrue($job->viaOutbox); + $this->assertSame('billing', $job->outboxDatabase); + + $source = new $marked; + $this->assertTrue($this->wrap($source, $wrapper)->viaOutbox); + + $source->viaOutbox = null; + $this->assertTrue($this->wrap($source, $wrapper)->viaOutbox); + + $source->withoutOutbox(); + $job = $this->wrap($source, $wrapper); + $this->assertFalse($job->viaOutbox); + $this->assertArrayNotHasKey('outboxDatabase', get_object_vars($job)); + } + + /** + * Provide the independent producer wrapper boundaries. + */ + public static function producerTypes(): array + { + return [ + 'mail' => [OutboxMailable::class, MarkedOutboxMailable::class, SendQueuedMailable::class], + 'notification' => [OutboxNotification::class, MarkedOutboxNotification::class, SendQueuedNotifications::class], + 'broadcast' => [OutboxBroadcast::class, MarkedOutboxBroadcast::class, BroadcastEvent::class], + ]; + } + + #[DataProvider('listenerTypes')] + public function testListenerDefaultsPropagateToQueuedJobs(string $listener, ?bool $selected, ?string $database): void + { + $queue = new QueueFake(new Container); + $dispatcher = new Dispatcher; + $dispatcher->setQueueResolver(fn () => $queue); + $dispatcher->listen('order.created', $listener); + + $dispatcher->dispatch('order.created'); + + $queue->assertPushed(CallQueuedListener::class, function (CallQueuedListener $job) use ($selected, $database): bool { + $this->assertSame($selected, $job->viaOutbox ?? null); + $this->assertSame($database, $job->outboxDatabase ?? null); + + if ($selected === null) { + $this->assertArrayNotHasKey('viaOutbox', get_object_vars($job)); + } + + return true; + }); + } + + /** + * Provide listener defaults read without invoking constructors. + */ + public static function listenerTypes(): array + { + return [ + 'absent' => [OutboxListener::class, null, null], + 'marker with uninitialized state' => [MarkedOutboxListener::class, true, null], + 'null with marker' => [NullMarkedOutboxListener::class, true, 'billing'], + 'explicit false overrides marker' => [DirectMarkedOutboxListener::class, false, null], + 'explicit true' => [SelectedOutboxListener::class, true, 'billing'], + ]; + } + + /** + * Construct the wrapper through its normal producer boundary. + */ + protected function wrap(object $source, string $wrapper): object + { + return $wrapper === SendQueuedNotifications::class + ? new SendQueuedNotifications([], $source) + : new $wrapper($source); + } +} + +class OutboxMailable extends Mailable +{ + use Queueable; +} + +class MarkedOutboxMailable extends OutboxMailable implements ShouldQueueViaOutbox +{ +} + +class OutboxNotification extends Notification +{ + use Queueable; +} + +class MarkedOutboxNotification extends OutboxNotification implements ShouldQueueViaOutbox +{ +} + +class OutboxBroadcast +{ + use Queueable; +} + +class MarkedOutboxBroadcast extends OutboxBroadcast implements ShouldQueueViaOutbox +{ +} + +class OutboxListener implements ShouldQueue +{ + /** + * Handle the queued event. + */ + public function handle(): void + { + } +} + +class MarkedOutboxListener extends OutboxListener implements ShouldQueueViaOutbox +{ + use Queueable; +} + +class NullMarkedOutboxListener extends OutboxListener implements ShouldQueueViaOutbox +{ + public ?bool $viaOutbox = null; + + public string $outboxDatabase = 'billing'; +} + +class DirectMarkedOutboxListener extends OutboxListener implements ShouldQueueViaOutbox +{ + public bool $viaOutbox = false; +} + +class SelectedOutboxListener extends OutboxListener +{ + public bool $viaOutbox = true; + + public string $outboxDatabase = 'billing'; +} From 2e3b848d116b815cfacc7217f2a722c3c61f11fe Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 20:47:37 +0000 Subject: [PATCH 06/33] feat(queue): define raw outbox publication contracts Introduce immutable stored-message and batch-result values plus the optional PublishesOutboxMessages capability. Keep accepted, deferred and failed outcomes separate so confirmed publications are not retried when later work fails. Provide one sequential fallback for non-batching transports and pooled adapters. Defer future messages without publishing, retain accepted prefixes on ordinary failure, and propagate coroutine cancellation unchanged. Preserve the existing Queue contract and pushRaw API. --- .../src/Queue/PublishesOutboxMessages.php | 18 +++++++ .../PublishesOutboxMessagesSequentially.php | 51 +++++++++++++++++++ src/queue/src/Outbox/OutboxBatchResult.php | 23 +++++++++ src/queue/src/Outbox/OutboxMessage.php | 26 ++++++++++ 4 files changed, 118 insertions(+) create mode 100644 src/contracts/src/Queue/PublishesOutboxMessages.php create mode 100644 src/queue/src/Outbox/Concerns/PublishesOutboxMessagesSequentially.php create mode 100644 src/queue/src/Outbox/OutboxBatchResult.php create mode 100644 src/queue/src/Outbox/OutboxMessage.php diff --git a/src/contracts/src/Queue/PublishesOutboxMessages.php b/src/contracts/src/Queue/PublishesOutboxMessages.php new file mode 100644 index 000000000..5b4c63157 --- /dev/null +++ b/src/contracts/src/Queue/PublishesOutboxMessages.php @@ -0,0 +1,18 @@ + $messages + */ + public function publishOutboxMessages(array $messages): OutboxBatchResult; +} diff --git a/src/queue/src/Outbox/Concerns/PublishesOutboxMessagesSequentially.php b/src/queue/src/Outbox/Concerns/PublishesOutboxMessagesSequentially.php new file mode 100644 index 000000000..be8fa02f7 --- /dev/null +++ b/src/queue/src/Outbox/Concerns/PublishesOutboxMessagesSequentially.php @@ -0,0 +1,51 @@ + $messages + */ + protected function publishOutboxMessagesSequentially(Queue $queue, array $messages): OutboxBatchResult + { + $due = []; + $deferred = []; + $accepted = []; + $now = $this->currentTime(); + + foreach ($messages as $message) { + if ($message->deliverAt !== null && $message->deliverAt > $now) { + $deferred[] = $message->id; + } else { + $due[] = $message; + } + } + + try { + foreach ($due as $message) { + $queue->pushRaw($message->payload, $message->queue, $message->options); + $accepted[] = $message->id; + } + } catch (CanceledException $exception) { + throw $exception; + } catch (Throwable $exception) { + return new OutboxBatchResult($accepted, $deferred, $exception); + } + + return new OutboxBatchResult($accepted, $deferred); + } +} diff --git a/src/queue/src/Outbox/OutboxBatchResult.php b/src/queue/src/Outbox/OutboxBatchResult.php new file mode 100644 index 000000000..6f63f3e53 --- /dev/null +++ b/src/queue/src/Outbox/OutboxBatchResult.php @@ -0,0 +1,23 @@ + $accepted + * @param list $deferred + */ + public function __construct( + public array $accepted = [], + public array $deferred = [], + public ?Throwable $failure = null, + ) { + } +} diff --git a/src/queue/src/Outbox/OutboxMessage.php b/src/queue/src/Outbox/OutboxMessage.php new file mode 100644 index 000000000..e760ac87d --- /dev/null +++ b/src/queue/src/Outbox/OutboxMessage.php @@ -0,0 +1,26 @@ + Date: Thu, 22 Oct 2026 20:48:02 +0000 Subject: [PATCH 07/33] feat(queue): stage transactional outbox jobs and publish raw batches Integrate opt-in staging into the existing single and bulk queue lifecycle. Resolve exact transaction owners before writing, reject ambiguous or read-role ownership, preserve finalized payload bytes and UUIDs, and store owner groups with binding-safe atomic inserts. Reuse direct database and local-driver fast paths without outbox storage access. Add the outbox schema, migration generator, lazy configuration and repository operations. Use atomic PostgreSQL claims, short locking-select transactions where required, conditional claims elsewhere, lease-token settlement and bounded retry operations. Read monitoring counts and oldest eligibility in one aggregate query. Publish Redis batches with confirmed Lua counts, database batches through shared binding-aware insertion, and SQS batches with protocol size limits and explicit partial results. Preserve queue-default versus requested delays, FIFO options and claim-specific overflow ownership. Pooled publication borrows once per batch; generic lifecycle events retain their finalized payload correlation. Accept confirmed dispatch locks before terminal observers and retain independent rollback snapshots for transactional storage. Keep failover forwarding job-scoped, satisfy explicit-owner callbacks without reentering unrelated transactions, and preserve independent dispatches from sync handlers. Inline after-commit work retains caller ownership and avoids a second pooled borrow. Cover staging precedence, owner selection, atomic bulk rollback, driver fast paths, partial acceptance, cancellation, payload identity, failover regressions and schema generation. This checkpoint supplies staging and transport foundations; relay orchestration, operator commands and end-to-end runtime coverage remain in the active implementation plan. --- src/foundation/config/queue.php | 32 ++ src/queue/src/BeanstalkdQueue.php | 16 + .../src/Concerns/InsertsDatabaseRows.php | 14 +- src/queue/src/Concerns/StagesOutboxJobs.php | 271 ++++++++++++ src/queue/src/Connectors/SqsConnector.php | 1 + src/queue/src/Console/OutboxTableCommand.php | 45 ++ src/queue/src/Console/stubs/outbox.stub | 41 ++ src/queue/src/DatabaseQueue.php | 177 ++++++-- src/queue/src/FailoverQueue.php | 50 ++- src/queue/src/NullQueue.php | 8 + .../src/Outbox/DatabaseOutboxRepository.php | 403 ++++++++++++++++++ src/queue/src/Outbox/OutboxDispatcher.php | 87 ++++ src/queue/src/Queue.php | 139 +++++- src/queue/src/QueuePoolProxy.php | 24 +- src/queue/src/QueueServiceProvider.php | 21 + src/queue/src/RedisQueue.php | 143 +++++-- src/queue/src/SqsQueue.php | 384 ++++++++++++----- src/queue/src/SyncQueue.php | 40 +- ...0009_testbench_create_job_outbox_table.php | 40 ++ tests/Bus/DispatchLockContextTest.php | 1 + .../QueueOutboxTableCommandTest.php | 29 ++ .../Integration/Queue/QueueConnectionTest.php | 4 + tests/Queue/DatabaseQueueRollbackTest.php | 176 ++++++++ tests/Queue/FailoverQueueTest.php | 53 +++ .../Outbox/DatabaseOutboxPublicationTest.php | 101 +++++ .../Outbox/DatabaseOutboxRepositoryTest.php | 227 ++++++++++ tests/Queue/Outbox/OutboxDispatchTest.php | 342 +++++++++++++++ tests/Queue/Outbox/OutboxDispatcherTest.php | 95 +++++ tests/Queue/Outbox/OutboxFastPathTest.php | 357 ++++++++++++++++ tests/Queue/Outbox/OutboxTransactionTest.php | 180 ++++++++ .../Outbox/PooledOutboxPublicationTest.php | 128 ++++++ .../Outbox/RedisOutboxPublicationTest.php | 158 +++++++ .../Queue/Outbox/SqsOutboxPublicationTest.php | 187 ++++++++ tests/Queue/QueueBackgroundQueueTest.php | 11 + tests/Queue/QueueDatabaseQueueUnitTest.php | 7 + tests/Queue/QueueDeferredQueueTest.php | 10 + tests/Queue/QueueRedisQueueTest.php | 5 + tests/Queue/QueueSyncQueueTest.php | 6 + tests/Queue/SqsBatchLifecycleTest.php | 192 +++++++++ tests/Testbench/CommanderTest.php | 1 + 40 files changed, 4009 insertions(+), 197 deletions(-) create mode 100644 src/queue/src/Concerns/StagesOutboxJobs.php create mode 100644 src/queue/src/Console/OutboxTableCommand.php create mode 100644 src/queue/src/Console/stubs/outbox.stub create mode 100644 src/queue/src/Outbox/DatabaseOutboxRepository.php create mode 100644 src/queue/src/Outbox/OutboxDispatcher.php create mode 100644 src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php create mode 100644 tests/Integration/Generators/QueueOutboxTableCommandTest.php create mode 100644 tests/Queue/DatabaseQueueRollbackTest.php create mode 100644 tests/Queue/Outbox/DatabaseOutboxPublicationTest.php create mode 100644 tests/Queue/Outbox/DatabaseOutboxRepositoryTest.php create mode 100644 tests/Queue/Outbox/OutboxDispatchTest.php create mode 100644 tests/Queue/Outbox/OutboxDispatcherTest.php create mode 100644 tests/Queue/Outbox/OutboxFastPathTest.php create mode 100644 tests/Queue/Outbox/OutboxTransactionTest.php create mode 100644 tests/Queue/Outbox/PooledOutboxPublicationTest.php create mode 100644 tests/Queue/Outbox/RedisOutboxPublicationTest.php create mode 100644 tests/Queue/Outbox/SqsOutboxPublicationTest.php create mode 100644 tests/Queue/SqsBatchLifecycleTest.php diff --git a/src/foundation/config/queue.php b/src/foundation/config/queue.php index 962ff713e..fd4987948 100644 --- a/src/foundation/config/queue.php +++ b/src/foundation/config/queue.php @@ -59,16 +59,19 @@ 'sync' => [ 'driver' => 'sync', 'after_commit' => false, + 'outbox' => false, ], 'background' => [ 'driver' => 'background', 'after_commit' => true, + 'outbox' => false, ], 'deferred' => [ 'driver' => 'deferred', 'after_commit' => true, + 'outbox' => false, ], 'database' => [ @@ -78,6 +81,7 @@ 'queue' => env('DB_QUEUE', 'default'), 'retry_after' => (int) env('DB_QUEUE_RETRY_AFTER', 90), 'after_commit' => false, + 'outbox' => false, ], 'beanstalkd' => [ @@ -88,6 +92,7 @@ 'retry_after' => (int) env('BEANSTALKD_QUEUE_RETRY_AFTER', 90), 'block_for' => 0, 'after_commit' => true, + 'outbox' => false, 'pool' => [ 'min_retained_objects' => 1, 'max_objects' => 10, @@ -108,6 +113,7 @@ 'suffix' => env('SQS_SUFFIX'), 'region' => env('AWS_DEFAULT_REGION', 'us-east-1'), 'after_commit' => true, + 'outbox' => false, 'overflow' => [ 'enabled' => (bool) env('SQS_OVERFLOW_ENABLED', false), 'store' => env('SQS_OVERFLOW_STORE'), @@ -133,6 +139,7 @@ // A value of 0 waits indefinitely and may delay worker shutdown. 'block_for' => null, 'after_commit' => true, + 'outbox' => false, ], 'failover' => [ @@ -142,9 +149,34 @@ 'deferred', ], 'after_commit' => true, + 'outbox' => false, ], ], + /* + |-------------------------------------------------------------------------- + | Transactional Outbox + |-------------------------------------------------------------------------- + | + | Outbox delivery stores jobs with your business transaction before sending + | them to a durable queue. Create the table on every listed database and + | run queue:relay. The retry timeout must exceed a full delivery batch, + | including transport timeouts and SDK retries, to avoid early redelivery. + | + | Jobs opt in with viaOutbox(), ShouldQueueViaOutbox, or their connection's + | outbox option. Local and failover queues retain after-commit behavior + | without crash durability; an already-atomic database queue inserts directly. + | + */ + + 'outbox' => [ + 'databases' => [ + env('QUEUE_OUTBOX_DATABASE', env('DB_CONNECTION', 'sqlite')), + ], + 'table' => env('QUEUE_OUTBOX_TABLE', 'job_outbox'), + 'retry_after' => (int) env('QUEUE_OUTBOX_RETRY_AFTER', 300), + ], + /* |-------------------------------------------------------------------------- | Job Batching diff --git a/src/queue/src/BeanstalkdQueue.php b/src/queue/src/BeanstalkdQueue.php index d84d90ffb..27773944c 100644 --- a/src/queue/src/BeanstalkdQueue.php +++ b/src/queue/src/BeanstalkdQueue.php @@ -300,6 +300,22 @@ public function getQueue(UnitEnum|string|null $queue): string return $this->resolveQueue($queue === null || $queue === '' ? $this->default : $queue); } + /** + * Capture the logical destination before publication-time forwarding. + */ + protected function resolveOutboxQueue(?string $queue): ?string + { + return $queue === null || $queue === '' ? $this->default : $queue; + } + + /** + * Prepare bulk payloads with the same hook destination as individual pushes. + */ + protected function prepareBatchJobs(array $jobs, mixed $data, ?string $queue): array + { + return parent::prepareBatchJobs($jobs, $data, $this->getQueue($queue)); + } + /** * Get the underlying Pheanstalk instance. * diff --git a/src/queue/src/Concerns/InsertsDatabaseRows.php b/src/queue/src/Concerns/InsertsDatabaseRows.php index 9ab34c031..3ba8967e4 100644 --- a/src/queue/src/Concerns/InsertsDatabaseRows.php +++ b/src/queue/src/Concerns/InsertsDatabaseRows.php @@ -5,6 +5,7 @@ namespace Hypervel\Queue\Concerns; use Hypervel\Database\ConnectionInterface; +use Hypervel\Database\PdoConnection; use RuntimeException; trait InsertsDatabaseRows @@ -18,9 +19,8 @@ protected function insertDatabaseRows( ConnectionInterface $connection, string $table, array $rows, - int $maxBindings, ): void { - $rowsPerInsert = intdiv($maxBindings, count(reset($rows))); + $rowsPerInsert = intdiv($this->maxBindings($connection), count(reset($rows))); if (count($rows) <= $rowsPerInsert) { $this->insertDatabaseRowChunk($connection, $table, $rows); @@ -46,4 +46,14 @@ protected function insertDatabaseRowChunk(ConnectionInterface $connection, strin throw new RuntimeException('Unable to insert queued jobs into the database.'); } } + + /** + * Get the statement binding limit without requiring a PDO-backed custom driver. + */ + protected function maxBindings(ConnectionInterface $connection): int + { + return $connection instanceof PdoConnection + ? $connection->maxBindings() + : PdoConnection::DEFAULT_MAX_BINDINGS; + } } diff --git a/src/queue/src/Concerns/StagesOutboxJobs.php b/src/queue/src/Concerns/StagesOutboxJobs.php new file mode 100644 index 000000000..35c2406fb --- /dev/null +++ b/src/queue/src/Concerns/StagesOutboxJobs.php @@ -0,0 +1,271 @@ +viaOutbox ?? ($job instanceof ShouldQueueViaOutbox || $this->dispatchViaOutbox); + + return $selected && (CoroutineContext::get(self::OUTBOX_FORWARDED_CONTEXT_KEY)['job'] ?? null) !== $job; + } + + /** + * Run a child attempt or handler with its own forwarding context. + * + * @param null|array{job: object|string, afterCommitHandled: bool} $forwarded + */ + protected function withOutboxForwarding(?array $forwarded, callable $callback): mixed + { + $previous = CoroutineContext::get(self::OUTBOX_FORWARDED_CONTEXT_KEY); + + if ($previous === $forwarded) { + return $callback(); + } + + $forwarded === null + ? CoroutineContext::forget(self::OUTBOX_FORWARDED_CONTEXT_KEY) + : CoroutineContext::set(self::OUTBOX_FORWARDED_CONTEXT_KEY, $forwarded); + + try { + return $callback(); + } finally { + $previous === null + ? CoroutineContext::forget(self::OUTBOX_FORWARDED_CONTEXT_KEY) + : CoroutineContext::set(self::OUTBOX_FORWARDED_CONTEXT_KEY, $previous); + } + } + + /** + * Determine whether this driver needs SQL staging for the selected job. + */ + protected function shouldStageViaOutbox(object|string $job): bool + { + return $this->shouldDispatchViaOutbox($job); + } + + /** + * Select an explicit transaction for a local or failover after-commit fallback. + */ + protected function resolveOutboxCommitConnection(DatabaseTransactionsManager $transactions, object|string $job): ?string + { + if (! $this->shouldDispatchViaOutbox($job) || ($database = $job->outboxDatabase ?? null) === null) { + return null; + } + + /** @var ConnectionResolverInterface $resolver */ + $resolver = $this->container->make('db'); + $owner = $resolver->connection($database)->getNameWithReadWriteType() ?? ''; + $active = $transactions->callbackApplicableTransactions(); + + if ($active->isNotEmpty() && ! $transactions->hasApplicableTransaction($owner)) { + throw new LogicException("Outbox database [{$database}] does not own an active transaction."); + } + + return $owner; + } + + /** + * Stage a single job through its normal guarded enqueue lifecycle. + */ + protected function enqueueViaOutbox(object|string $job, string $payload, ?string $queue, DateInterval|DateTimeInterface|int|null $delay): string + { + try { + $dispatcher = $this->container->make(OutboxDispatcher::class); + $connection = $dispatcher->databaseFor($job); + $payload = $this->finalizePayloadForQueueing($queue, $job, $payload, $delay); + $row = $this->prepareOutboxRow($job, $payload, $queue, $delay); + $this->raiseJobQueueingEvent($queue, $job, $payload, $delay); + $dispatcher->store($connection, [$row], [$job]); + } catch (CanceledException $exception) { + throw $exception; + } catch (Throwable $exception) { + $this->raiseJobQueueingFailedEvent($queue, $job, $payload, $delay, $exception); + + throw $exception; + } + + $this->acceptDispatchLocks($job); + $this->raiseJobQueuedEvent($queue, $row['uuid'], $job, $payload, $delay); + + return $row['uuid']; + } + + /** + * Stage selected jobs in owner groups and return the direct remainder. + */ + protected function stageOutboxJobs(array $jobs, mixed $data, ?string $queue): array + { + $direct = $jobs; + $selected = []; + + foreach ($jobs as $index => $job) { + if ($this->shouldStageViaOutbox($job)) { + $selected[] = $job; + unset($direct[$index]); + } + } + + if ($selected === []) { + return $jobs; + } + + $dispatcher = $this->container->make(OutboxDispatcher::class); + $prepared = $this->prepareBatchJobs($selected, $data, $queue); + $groups = []; + + try { + // Resolve every owner before finalization or insertion of any group. + foreach ($prepared as $index => $member) { + $connection = $dispatcher->databaseFor($member['job']); + $name = $connection->getNameWithReadWriteType() ?? ''; + $groups[$name]['connection'] = $connection; + $groups[$name]['members'][] = $index; + } + + foreach ($groups as $group) { + $rows = []; + $groupJobs = []; + + foreach ($group['members'] as $index) { + $member = &$prepared[$index]; + $member['payload'] = $this->finalizePayloadForQueueing($queue, $member['job'], $member['payload'], $member['delay']); + $rows[] = $this->prepareOutboxRow($member['job'], $member['payload'], $queue, $member['delay']); + $groupJobs[] = $member['job']; + unset($member); + } + + foreach ($group['members'] as $index) { + $member = $prepared[$index]; + $this->raiseJobQueueingEvent($queue, $member['job'], $member['payload'], $member['delay']); + } + + $dispatcher->store($group['connection'], $rows, $groupJobs); + $accepted = []; + + foreach ($group['members'] as $position => $index) { + $member = $prepared[$index]; + $this->acceptDispatchLocks($member['job']); + $member['uuid'] = $rows[$position]['uuid']; + $accepted[] = $member; + unset($prepared[$index]); + } + + $failure = $this->notifyBatchMembers($accepted, function (array $member) use ($queue): void { + $this->raiseJobQueuedEvent($queue, $member['uuid'], $member['job'], $member['payload'], $member['delay']); + }); + + if ($failure !== null) { + throw $failure; + } + } + } catch (CanceledException $exception) { + throw $exception; + } catch (Throwable $exception) { + $failure = $this->notifyBatchMembers($prepared, function (array $member) use ($queue, $exception): void { + $this->raiseJobQueueingFailedEvent($queue, $member['job'], $member['payload'], $member['delay'], $exception); + }); + + throw $failure ?? $exception; + } + + return array_values($direct); + } + + /** + * Build a byte-preserving row from a finalized payload. + */ + protected function prepareOutboxRow(object|string $job, string $payload, ?string $queue, DateInterval|DateTimeInterface|int|null $delay): array + { + $storedPayload = $this->preparePayloadForOutbox($job, $payload); + + try { + $decoded = json_decode($storedPayload, true, flags: JSON_THROW_ON_ERROR); + } catch (JsonException $exception) { + throw new InvalidPayloadException('Unable to decode the outbox payload: ' . $exception->getMessage(), $storedPayload); + } + + if (! is_array($decoded) || ! is_string($decoded['uuid'] ?? null) || ! Str::isUuid($decoded['uuid'])) { + throw new InvalidPayloadException('The outbox payload must contain a valid job UUID.', $storedPayload); + } + + $queue = $this->resolveOutboxQueue($queue); + $now = $this->currentTime(); + // Even an already-due timer overrides SQS's queue-level delay. + $deliverAt = empty($delay) ? null : max($this->availableAt($delay), $now); + $options = $this->prepareOutboxOptions($job, $queue, $storedPayload); + + return [ + 'uuid' => $decoded['uuid'], + 'connection' => $this->connectionName, + 'queue' => $queue, + 'payload' => $storedPayload, + 'options' => $options === [] ? null : json_encode($options, JSON_THROW_ON_ERROR), + 'deliver_at' => $deliverAt, + 'attempts' => 0, + 'available_at' => $this->availableAtForOutbox($queue, $deliverAt), + 'lease_token' => null, + 'last_error' => null, + 'created_at' => $now, + ]; + } + + /** + * Expand the logical destination without applying publication-time forwarding. + */ + protected function resolveOutboxQueue(?string $queue): ?string + { + return $queue; + } + + /** + * Add metadata required while the original job is still available. + */ + protected function preparePayloadForOutbox(object|string $job, string $payload): string + { + return $payload; + } + + /** + * Capture transport options that cannot be recovered from the stored envelope. + */ + protected function prepareOutboxOptions(object|string $job, ?string $queue, string $payload): array + { + return []; + } + + /** + * Determine when a stored message may first be published. + */ + protected function availableAtForOutbox(?string $queue, ?int $deliverAt): int + { + return $deliverAt ?? $this->currentTime(); + } +} diff --git a/src/queue/src/Connectors/SqsConnector.php b/src/queue/src/Connectors/SqsConnector.php index 2603d73d6..ceb872a6f 100644 --- a/src/queue/src/Connectors/SqsConnector.php +++ b/src/queue/src/Connectors/SqsConnector.php @@ -90,6 +90,7 @@ protected function clientConfiguration(array $config): array 'prefix', 'suffix', 'after_commit', + 'outbox', 'key', 'secret', 'token', diff --git a/src/queue/src/Console/OutboxTableCommand.php b/src/queue/src/Console/OutboxTableCommand.php new file mode 100644 index 000000000..06b4c32bd --- /dev/null +++ b/src/queue/src/Console/OutboxTableCommand.php @@ -0,0 +1,45 @@ +hypervel->make('config')->string('queue.outbox.table'); + } + + /** + * Get the path to the migration stub file. + */ + protected function migrationStubFile(): string + { + return __DIR__ . '/stubs/outbox.stub'; + } +} diff --git a/src/queue/src/Console/stubs/outbox.stub b/src/queue/src/Console/stubs/outbox.stub new file mode 100644 index 000000000..bc2782d04 --- /dev/null +++ b/src/queue/src/Console/stubs/outbox.stub @@ -0,0 +1,41 @@ +id(); + $table->uuid('uuid')->unique(); + $table->string('connection'); + $table->string('queue')->nullable(); + $table->longText('payload'); + $table->text('options')->nullable(); + $table->unsignedBigInteger('deliver_at')->nullable(); + $table->unsignedInteger('attempts')->default(0); + $table->unsignedBigInteger('available_at'); + $table->uuid('lease_token')->nullable(); + $table->text('last_error')->nullable(); + $table->unsignedBigInteger('created_at'); + + $table->index(['available_at', 'id']); + }); + } + + /** + * Reverse the migrations. + */ + public function down(): void + { + Schema::dropIfExists('{{table}}'); + } +}; diff --git a/src/queue/src/DatabaseQueue.php b/src/queue/src/DatabaseQueue.php index ae7b78121..b533b6334 100644 --- a/src/queue/src/DatabaseQueue.php +++ b/src/queue/src/DatabaseQueue.php @@ -6,8 +6,10 @@ use DateInterval; use DateTimeInterface; +use Hypervel\Bus\DispatchLockContext; use Hypervel\Contracts\Queue\ClearableQueue; use Hypervel\Contracts\Queue\Job; +use Hypervel\Contracts\Queue\PublishesOutboxMessages; use Hypervel\Contracts\Queue\Queue as QueueContract; use Hypervel\Database\ConnectionInterface; use Hypervel\Database\ConnectionResolverInterface; @@ -21,13 +23,15 @@ use Hypervel\Queue\Jobs\DatabaseJob; use Hypervel\Queue\Jobs\DatabaseJobRecord; use Hypervel\Queue\Jobs\InspectedJob; +use Hypervel\Queue\Outbox\OutboxBatchResult; +use Hypervel\Queue\Outbox\OutboxMessage; use Hypervel\Support\CarbonImmutable; use Hypervel\Support\Collection; use Swoole\Coroutine\CanceledException; use Throwable; use UnitEnum; -class DatabaseQueue extends Queue implements QueueContract, ClearableQueue +class DatabaseQueue extends Queue implements QueueContract, ClearableQueue, PublishesOutboxMessages { use DetectsConcurrencyErrors; use DetectsLostConnections; @@ -279,8 +283,11 @@ public function push(object|string $job, mixed $data = '', UnitEnum|string|null $this->createPayload($job, $this->getQueue($queue), $data), $queue, null, - static function (DatabaseQueue $owner, string $payload, ?string $queue) { - return $owner->pushToDatabase($queue, $payload); + static function (DatabaseQueue $owner, string $payload, ?string $queue) use ($job) { + $id = $owner->pushToDatabase($queue, $payload); + $owner->addStoredJobRollbackCallback($job); + + return $id; } ); } @@ -310,8 +317,11 @@ static function ( string $payload, ?string $queue, DateInterval|DateTimeInterface|int $delay - ) { - return $owner->pushToDatabase($queue, $payload, $delay); + ) use ($job) { + $id = $owner->pushToDatabase($queue, $payload, $delay); + $owner->addStoredJobRollbackCallback($job); + + return $id; } ); } @@ -326,7 +336,7 @@ static function ( public function bulk(array $jobs, mixed $data = '', UnitEnum|string|null $queue = null): mixed { $queue = $this->normalizeQueue($queue); - $jobs = array_values($jobs); + $jobs = array_values($this->stageOutboxJobs($jobs, $data, $queue)); if ($jobs === []) { return null; @@ -372,17 +382,7 @@ static function (Queue $owner) use ($preparedJobs, $queue): mixed { */ protected function prepareBatchJobs(array $jobs, mixed $data, ?string $queue): array { - return (new Collection($jobs)) - ->map(function (object|string $job) use ($data, $queue): array { - $delay = $this->getJobDelay($job); - - return [ - 'job' => $job, - 'delay' => $delay, - 'payload' => $this->createPayload($job, $this->getQueue($queue), $data, $delay), - ]; - }) - ->all(); + return parent::prepareBatchJobs($jobs, $data, $this->getQueue($queue)); } /** @@ -402,15 +402,13 @@ protected function enqueueBatch(array $jobs, ?string $queue): mixed // Every payload must be final before any batch member begins its // existing JobQueueing lifecycle or the database write. - foreach ($jobs as $job) { + foreach ($jobs as $index => $job) { $this->raiseJobQueueingEvent($queue, $job['job'], $job['payload'], $job['delay']); + $jobs[$index]['storedPayload'] = $this->preparePayloadForPush($job['job'], $job['payload'], $queue); } $now = $this->availableAt(); $connection = $this->getDatabase(); - $maxBindings = $connection instanceof PdoConnection - ? $connection->maxBindings() - : PdoConnection::DEFAULT_MAX_BINDINGS; $this->insertDatabaseRows( $connection, @@ -418,33 +416,99 @@ protected function enqueueBatch(array $jobs, ?string $queue): mixed (new Collection($jobs)) ->map(fn (array $job): array => $this->buildDatabaseRecord( $this->getQueue($queue), - $job['payload'], + $job['storedPayload'], $job['delay'] !== null ? $this->availableAt($job['delay']) : $now, ))->all(), - $maxBindings, ); - $result = true; - } catch (CanceledException $exception) { - throw $exception; - } catch (Throwable $exception) { + foreach ($jobs as $job) { - $this->raiseJobQueueingFailedEvent($queue, $job['job'], $job['payload'], $job['delay'], $exception); + $this->addStoredJobRollbackCallback($job['job'], $connection); } + $result = true; + } catch (CanceledException $exception) { throw $exception; + } catch (Throwable $exception) { + throw $this->notifyBatchMembers( + $jobs, + fn (array $job) => $this->raiseJobQueueingFailedEvent($queue, $job['job'], $job['payload'], $job['delay'], $exception), + ) ?? $exception; } foreach ($jobs as $job) { $this->acceptDispatchLocks($job['job']); } - foreach ($jobs as $job) { - $this->raiseJobQueuedEvent($queue, null, $job['job'], $job['payload'], $job['delay']); + $exception = $this->notifyBatchMembers( + $jobs, + fn (array $job) => $this->handlePayloadPushed($job['storedPayload'], $queue), + fn (array $job) => $this->raiseJobQueuedEvent($queue, null, $job['job'], $job['payload'], $job['delay']), + ); + + if ($exception !== null) { + throw $exception; } return $result; } + /** + * Release accepted dispatch locks if their stored job is rolled back. + */ + protected function addStoredJobRollbackCallback(object|string $job, ?ConnectionInterface $connection = null): void + { + $snapshot = is_object($job) ? DispatchLockContext::snapshot($job) : null; + + if ($snapshot === null) { + return; + } + + $connection ??= $this->getDatabase(); + + if ($connection->transactionLevel() > 0) { + /** @var DatabaseTransactionsManager $transactions */ + $transactions = $this->container->make('db.transactions'); + $transactions->addCallbackForRollback( + static fn () => DispatchLockContext::releaseSnapshot($snapshot), + $connection->getNameWithReadWriteType() ?? '', + ); + } + } + + /** + * Publish a raw outbox batch atomically across its logical queues. + * + * @param list $messages + */ + public function publishOutboxMessages(array $messages): OutboxBatchResult + { + if ($messages === []) { + return new OutboxBatchResult; + } + + try { + $connection = $this->getDatabase(); + $now = $this->currentTime(); + $rows = []; + + foreach ($messages as $message) { + $rows[] = $this->buildDatabaseRecord( + $this->getQueue($message->queue), + $message->payload, + max($now, $message->deliverAt ?? $now), + ); + } + + $this->insertDatabaseRows($connection, $this->table, $rows); + } catch (CanceledException $exception) { + throw $exception; + } catch (Throwable $exception) { + return new OutboxBatchResult(failure: $exception); + } + + return new OutboxBatchResult(array_column($messages, 'id')); + } + /** * Release a reserved job back onto the queue after (n) seconds. */ @@ -691,6 +755,59 @@ public function getQueue(UnitEnum|string|null $queue): string return $this->resolveQueue($queue === null || $queue === '' ? $this->default : $queue); } + /** + * Avoid a second durable table when the jobs insert already belongs to its owner. + */ + protected function shouldStageViaOutbox(object|string $job): bool + { + if (! $this->shouldDispatchViaOutbox($job) || ! $this->container->bound('db.transactions')) { + return false; + } + + /** @var DatabaseTransactionsManager $transactions */ + $transactions = $this->container->make('db.transactions'); + $active = []; + + foreach ($transactions->callbackApplicableTransactions() as $transaction) { + $active[$transaction->connection] = true; + } + + if ($active === []) { + return false; + } + + $target = $this->getDatabase()->getNameWithReadWriteType() ?? ''; + $explicit = $job->outboxDatabase ?? null; + + return ! (isset($active[$target]) && ($explicit === null + ? count($active) === 1 + : $this->resolver->connection($explicit)->getNameWithReadWriteType() === $target)); + } + + /** + * Keep an already-atomic outbox-selected insert inside its transaction. + */ + protected function shouldDispatchAfterCommit(object|string $job): bool + { + return ! $this->shouldDispatchViaOutbox($job) && parent::shouldDispatchAfterCommit($job); + } + + /** + * Capture the logical destination before publication-time forwarding. + */ + protected function resolveOutboxQueue(?string $queue): ?string + { + return $queue === null || $queue === '' ? $this->default : $queue; + } + + /** + * Publish delayed database jobs immediately with their future availability. + */ + protected function availableAtForOutbox(?string $queue, ?int $deliverAt): int + { + return $this->currentTime(); + } + /** * Get the underlying database connection. */ diff --git a/src/queue/src/FailoverQueue.php b/src/queue/src/FailoverQueue.php index cb4836a49..f0ccad5d8 100644 --- a/src/queue/src/FailoverQueue.php +++ b/src/queue/src/FailoverQueue.php @@ -228,6 +228,22 @@ protected function resolveForwardedQueue(UnitEnum|string|null $queue): ?string return $queue === null ? null : $this->queueRoutes()->forwardedQueueForConnection($queue, $this->connectionName ?? null); } + /** + * Preserve failover attempts instead of capturing an undecided destination. + */ + protected function shouldStageViaOutbox(object|string $job): bool + { + return false; + } + + /** + * Degrade selected outbox delivery to an after-commit failover attempt. + */ + protected function shouldDispatchAfterCommit(object|string $job): bool + { + return $this->shouldDispatchViaOutbox($job) || parent::shouldDispatchAfterCommit($job); + } + /** * Attempt the given method on all connections. * @@ -248,16 +264,38 @@ protected function attemptOnAllConnections(string $method, array $arguments, obj } } + return $this->attemptOnConnections($method, $arguments, $job); + } + + /** + * Forward an operation to the first available child connection. + */ + protected function attemptOnConnections( + string $method, + array $arguments, + object|string|null $job, + bool $afterCommitHandled = false, + ): mixed { $contextKey = self::FAILING_QUEUES_CONTEXT_PREFIX . spl_object_id($this); $state = CoroutineContext::get($contextKey); $failingQueues = $state !== null && $state[0]->get() === $this ? $state[1] : []; [$lastException, $failedQueues] = [null, []]; + $forwarded = CoroutineContext::get(self::OUTBOX_FORWARDED_CONTEXT_KEY); + + if ($job !== null && ($forwarded['job'] ?? null) !== $job) { + $forwarded = ['job' => $job, 'afterCommitHandled' => $afterCommitHandled]; + } try { foreach ($this->connections as $connection) { try { - return $this->manager->connection($connection)->{$method}(...$arguments); + return $job === null + ? $this->manager->connection($connection)->{$method}(...$arguments) + : $this->withOutboxForwarding( + $forwarded, + fn () => $this->manager->connection($connection)->{$method}(...$arguments), + ); } catch (CanceledException $exception) { throw $exception; } catch (Throwable $e) { @@ -303,6 +341,10 @@ protected function deferUntilAllTransactionsCommit( return false; } + if (($selectedConnection = $this->resolveOutboxCommitConnection($transactions, $job)) !== null) { + $connections = [$selectedConnection]; + } + $context = ContextRepository::hasInstance() ? ContextRepository::getInstance()->replicate() : null; $pendingConnections = array_fill_keys($connections, true); $settled = false; @@ -325,7 +367,7 @@ static function () use (&$settled, $releaseLocks): void { foreach ($connections as $connection) { $transactions->addCallback( - function () use (&$pendingConnections, &$settled, $connection, $method, $arguments, $job, $context): void { + function () use (&$pendingConnections, &$settled, $connection, $method, $arguments, $job, $context, $selectedConnection): void { if ($settled) { return; } @@ -344,7 +386,9 @@ function () use (&$pendingConnections, &$settled, $connection, $method, $argumen // Child payload hooks run after commit but must see the dispatch-time log context. try { - $this->withLogContext($context, fn () => $this->attemptOnAllConnections($method, $arguments, $job)); + $this->withLogContext($context, fn () => $selectedConnection !== null + ? $this->attemptOnConnections($method, $arguments, $job, true) + : $this->attemptOnAllConnections($method, $arguments, $job)); } finally { if (is_object($job)) { DispatchLockContext::release($job); diff --git a/src/queue/src/NullQueue.php b/src/queue/src/NullQueue.php index b015c1eeb..b86173b56 100644 --- a/src/queue/src/NullQueue.php +++ b/src/queue/src/NullQueue.php @@ -157,6 +157,14 @@ public function later(DateInterval|DateTimeInterface|int $delay, object|string $ return null; } + /** + * Keep inherited bulk dispatch a no-op even when jobs select the outbox. + */ + protected function shouldStageViaOutbox(object|string $job): bool + { + return false; + } + /** * Pop the next job off of the queue. */ diff --git a/src/queue/src/Outbox/DatabaseOutboxRepository.php b/src/queue/src/Outbox/DatabaseOutboxRepository.php new file mode 100644 index 000000000..82c8e18c7 --- /dev/null +++ b/src/queue/src/Outbox/DatabaseOutboxRepository.php @@ -0,0 +1,403 @@ + $databases + */ + public function __construct( + protected ConnectionResolverInterface $resolver, + protected array $databases, + protected string $table, + protected int $retryAfter, + ) { + if ($databases === []) { + throw new InvalidArgumentException('At least one outbox database must be configured.'); + } + + if ($retryAfter <= 0) { + throw new InvalidArgumentException('The outbox retry_after must be greater than zero.'); + } + } + + /** + * Get the configured outbox sources. + * + * @return list + */ + public function getDatabases(): array + { + return $this->databases; + } + + /** + * Resolve a configured outbox source in the current coroutine. + */ + public function getConnection(string $database): ConnectionInterface + { + if (! in_array($database, $this->databases, true)) { + throw new InvalidArgumentException("Outbox database [{$database}] is not configured in queue.outbox.databases."); + } + + return $this->resolver->connection($database); + } + + /** + * Insert finalized rows on the connection that owns the business transaction. + * + * @param non-empty-list> $rows + */ + public function insert(ConnectionInterface $connection, array $rows): void + { + $this->insertDatabaseRows($connection, $this->table, $rows); + } + + /** + * Claim eligible messages without retaining a database resource in the result. + * + * @return list + */ + public function claim(string $database, int $limit): array + { + $connection = $this->getConnection($database); + $now = $this->currentTime(); + $token = (string) Str::uuid(); + $expiresAt = $now + $this->retryAfter; + $lock = $connection instanceof PdoConnection ? $connection->lockForPopping() : true; + $limit = min($limit, $this->maxBindings($connection) - 3); + + if ($lock === 'FOR UPDATE SKIP LOCKED' && $connection->getDriverName() === 'pgsql') { + $rows = $this->claimOnPostgres($connection, $limit, $now, $expiresAt, $token); + } elseif ($lock === 'FOR UPDATE SKIP LOCKED') { + $rows = $this->claimWithSkipLocked($connection, $limit, $now, $expiresAt, $token); + } else { + $rows = $this->claimWithConditionalUpdate($connection, $limit, $now, $expiresAt, $token); + } + + usort($rows, static fn (object $first, object $second): int => [(int) $first->eligible_at, (int) $first->id] <=> [(int) $second->eligible_at, (int) $second->id]); + + return array_map(static fn (object $row): OutboxMessage => new OutboxMessage( + (int) $row->id, + $row->uuid, + $row->lease_token, + (int) $row->attempts, + $row->connection, + $row->queue, + $row->payload, + $row->options === null ? [] : json_decode($row->options, true, flags: JSON_THROW_ON_ERROR), + $row->deliver_at === null ? null : (int) $row->deliver_at, + (int) $row->eligible_at, + (int) $row->created_at, + ), $rows); + } + + /** + * Claim a PostgreSQL batch in one statement, retaining its original eligibility. + * + * @return list + */ + protected function claimOnPostgres(ConnectionInterface $connection, int $limit, int $now, int $expiresAt, string $token): array + { + $table = $connection->getQueryGrammar()->wrapTable($this->table); + + return $connection->selectFromWriteConnection( + "WITH candidates AS ( + SELECT id, available_at FROM {$table} + WHERE available_at <= ? ORDER BY available_at, id + LIMIT ? FOR UPDATE SKIP LOCKED + ) + UPDATE {$table} AS outgoing + SET lease_token = ?, available_at = ?, attempts = outgoing.attempts + 1 + FROM candidates WHERE outgoing.id = candidates.id + RETURNING outgoing.*, candidates.available_at AS eligible_at", + [$now, $limit, $token, $expiresAt], + ); + } + + /** + * Claim rows on engines that support skipping another relay's locked rows. + * + * @return list + */ + protected function claimWithSkipLocked(ConnectionInterface $connection, int $limit, int $now, int $expiresAt, string $token): array + { + return $connection->transaction(function () use ($connection, $limit, $now, $expiresAt, $token): array { + $rows = $this->query($connection) + ->where('available_at', '<=', $now) + ->orderBy('available_at')->orderBy('id') + ->limit($limit)->lock('FOR UPDATE SKIP LOCKED') + ->get()->all(); + + if ($rows === []) { + return []; + } + + $this->query($connection)->whereIn('id', array_column($rows, 'id'))->increment('attempts', 1, [ + 'lease_token' => $token, + 'available_at' => $expiresAt, + ]); + + foreach ($rows as $row) { + $row->eligible_at = $row->available_at; + $row->available_at = $expiresAt; + $row->lease_token = $token; + ++$row->attempts; + } + + return $rows; + }); + } + + /** + * Win eligible candidates through one atomic update on engines without skip-locked support. + * + * @return list + */ + protected function claimWithConditionalUpdate(ConnectionInterface $connection, int $limit, int $now, int $expiresAt, string $token): array + { + $candidates = $this->query($connection) + ->where('available_at', '<=', $now) + ->orderBy('available_at')->orderBy('id')->limit($limit) + ->pluck('available_at', 'id')->all(); + + if ($candidates === []) { + return []; + } + + $ids = array_keys($candidates); + $this->query($connection) + ->whereIn('id', $ids)->where('available_at', '<=', $now) + ->increment('attempts', 1, ['lease_token' => $token, 'available_at' => $expiresAt]); + + $rows = $this->query($connection)->whereIn('id', $ids)->where('lease_token', $token)->get()->all(); + + foreach ($rows as $row) { + $row->eligible_at = $candidates[$row->id]; + } + + return $rows; + } + + /** + * Settle a validated result without modifying rows claimed by a newer relay. + * + * @param non-empty-list $messages + * @return int the number of rows still owned and settled + */ + public function settle(string $database, array $messages, OutboxBatchResult $result): int + { + $connection = $this->getConnection($database); + $token = $messages[0]->leaseToken; + $failed = array_values(array_diff(array_column($messages, 'id'), $result->accepted, $result->deferred)); + + return $connection->transaction(function () use ($connection, $token, $result, $failed): int { + $settled = 0; + $maxBindings = $this->maxBindings($connection); + + foreach (array_chunk($result->accepted, $maxBindings - 1) as $ids) { + $settled += $this->query($connection)->where('lease_token', $token)->whereIn('id', $ids)->delete(); + } + + foreach (array_chunk($result->deferred, $maxBindings - 2) as $ids) { + $grammar = $connection->getQueryGrammar(); + $settled += $this->query($connection)->where('lease_token', $token)->whereIn('id', $ids)->update([ + 'lease_token' => null, + 'available_at' => new Expression($grammar->wrap('deliver_at')), + 'attempts' => new Expression($grammar->wrap('attempts') . ' - 1'), + ]); + } + + if ($failed !== []) { + $settled += $this->releaseFailed($connection, $token, $failed, $result->failure->getMessage()); + } + + return $settled; + }); + } + + /** + * Back off failed rows with seven retry deadlines rather than one case per row. + * + * @param non-empty-list $ids + */ + protected function releaseFailed(ConnectionInterface $connection, string $token, array $ids, string $error): int + { + $now = $this->currentTime(); + $deadlines = array_map(static fn (int $delay): int => $now + random_int(max(1, (int) ceil($delay / 2)), $delay), self::RETRY_DELAYS); + $error = mb_substr($error, 0, 4096); + $grammar = $connection->getQueryGrammar(); + $table = $grammar->wrapTable($this->table); + $attempts = $grammar->wrap('attempts'); + $cases = []; + + foreach (array_keys($deadlines) as $index) { + $cases[] = $index === count($deadlines) - 1 ? 'ELSE ?' : 'WHEN ' . $attempts . ' <= ' . ($index + 1) . ' THEN ?'; + } + + $case = 'CASE ' . implode(' ', $cases) . ' END'; + $affected = 0; + + foreach (array_chunk($ids, $this->maxBindings($connection) - count($deadlines) - 2) as $chunk) { + $parameters = implode(', ', array_fill(0, count($chunk), '?')); + $affected += $connection->update( + "UPDATE {$table} SET available_at = {$case}, lease_token = NULL, last_error = ? WHERE lease_token = ? AND id IN ({$parameters})", + [...$deadlines, $error, $token, ...$chunk], + ); + } + + return $affected; + } + + /** + * Get a bounded set of pending rows for operator inspection. + * + * @return list + */ + public function pending(string $database, int $limit): array + { + return $this->query($this->getConnection($database))->orderBy('id')->limit($limit)->get()->all(); + } + + /** + * Find a stored message by its job UUID on the writer. + */ + public function find(string $database, string $uuid): ?object + { + return $this->query($this->getConnection($database))->where('uuid', $uuid)->first(); + } + + /** + * Read backlog counts and the age of the first currently eligible row. + * + * @return array{total: int, eligible: int, delayed: int, leased: int, retrying: int, oldest_eligible_age: ?int} + */ + public function statistics(string $database): array + { + $connection = $this->getConnection($database); + $now = $this->currentTime(); + $counts = $this->query($connection)->selectRaw( + 'COUNT(*) AS total, + SUM(CASE WHEN available_at <= ? THEN 1 ELSE 0 END) AS eligible, + SUM(CASE WHEN lease_token IS NULL AND available_at > ? AND attempts = 0 THEN 1 ELSE 0 END) AS delayed, + SUM(CASE WHEN lease_token IS NOT NULL AND available_at > ? THEN 1 ELSE 0 END) AS leased, + SUM(CASE WHEN attempts > 0 AND (lease_token IS NULL OR available_at <= ?) THEN 1 ELSE 0 END) AS retrying, + MIN(CASE WHEN available_at <= ? THEN available_at END) AS oldest_eligible', + [$now, $now, $now, $now, $now], + )->first(); + + return [ + 'total' => (int) $counts->total, + 'eligible' => (int) $counts->eligible, + 'delayed' => (int) $counts->delayed, + 'leased' => (int) $counts->leased, + 'retrying' => (int) $counts->retrying, + 'oldest_eligible_age' => $counts->oldest_eligible === null ? null : $now - (int) $counts->oldest_eligible, + ]; + } + + /** + * Retry an attempted message unless a live relay owns it. + */ + public function retry(string $database, string $uuid): int + { + $now = $this->currentTime(); + + return $this->retryable($this->getConnection($database), $now)->where('uuid', $uuid)->update([ + 'available_at' => $now, + 'lease_token' => null, + 'last_error' => null, + ]); + } + + /** + * Retry the current backlog in bounded primary-key chunks, preserving attempt counts. + */ + public function retryAll(string $database): int + { + $connection = $this->getConnection($database); + $now = $this->currentTime(); + $ceiling = $this->retryable($connection, $now)->max('id'); + + if ($ceiling === null) { + return 0; + } + + $lastId = 0; + $affected = 0; + $limit = min(1000, $this->maxBindings($connection) - 5); + + do { + $ids = $this->retryable($connection, $now)->where('id', '>', $lastId)->where('id', '<=', $ceiling) + ->orderBy('id')->limit($limit)->pluck('id')->all(); + + if ($ids === []) { + break; + } + + $affected += $this->retryable($connection, $now)->whereIn('id', $ids)->update([ + 'available_at' => $now, + 'lease_token' => null, + 'last_error' => null, + ]); + $lastId = (int) end($ids); + } while (count($ids) === $limit); + + return $affected; + } + + /** + * Delete an inspected row only while its lease remains inactive and unchanged. + */ + public function forget(string $database, int $id, ?string $leaseToken): bool + { + return $this->withoutLiveLease($this->query($this->getConnection($database)), $this->currentTime()) + ->where('id', $id)->where('lease_token', $leaseToken)->delete() === 1; + } + + /** + * Select attempted rows that are safe for an operator to reschedule. + */ + protected function retryable(ConnectionInterface $connection, int $now): Builder + { + return $this->withoutLiveLease($this->query($connection), $now)->where('attempts', '>', 0); + } + + /** + * Exclude rows currently owned by a publishing relay. + */ + protected function withoutLiveLease(Builder $query, int $now): Builder + { + return $query->where(static function (Builder $query) use ($now): void { + $query->whereNull('lease_token')->orWhere('available_at', '<=', $now); + }); + } + + /** + * Query the configured table on the writer, including for operator mutations. + */ + protected function query(ConnectionInterface $connection): Builder + { + return $connection->table($this->table)->useWritePdo(); + } +} diff --git a/src/queue/src/Outbox/OutboxDispatcher.php b/src/queue/src/Outbox/OutboxDispatcher.php new file mode 100644 index 000000000..5f0ef6cc5 --- /dev/null +++ b/src/queue/src/Outbox/OutboxDispatcher.php @@ -0,0 +1,87 @@ +transactions->callbackApplicableTransactions() as $transaction) { + $active[$transaction->connection] = true; + } + + $explicit = $job->outboxDatabase ?? null; + + if ($explicit !== null) { + $connection = $this->resolver->connection($explicit); + + if ($active !== [] && ! isset($active[$connection->getNameWithReadWriteType() ?? ''])) { + throw new LogicException("Outbox database [{$explicit}] does not own an active transaction. Active connections: " . implode(', ', array_keys($active)) . '.'); + } + } elseif (count($active) > 1) { + throw new LogicException('Multiple database transactions are active [' . implode(', ', array_keys($active)) . ']. Select the outbox owner with onOutboxDatabase().'); + } else { + $connection = $this->resolver->connection(array_key_first($active) ?? $this->repository->getDatabases()[0]); + } + + $owner = $connection->getNameWithReadWriteType() ?? ''; + + if (ConnectionName::parse($owner)->isRead()) { + throw new LogicException("Outbox database [{$owner}] is a read connection. Select its base or ::write connection."); + } + + if (! in_array($connection->getName(), $this->repository->getDatabases(), true)) { + throw new LogicException("Outbox database [{$owner}] is not covered by queue.outbox.databases."); + } + + return $connection; + } + + /** + * Store one owner group and register rollback cleanup before dispatch acceptance. + * + * @param non-empty-list> $rows + * @param non-empty-list $jobs + */ + public function store(ConnectionInterface $connection, array $rows, array $jobs): void + { + $this->repository->insert($connection, $rows); + + if ($connection->transactionLevel() === 0) { + return; + } + + foreach ($jobs as $job) { + if (is_object($job) && ($snapshot = DispatchLockContext::snapshot($job)) !== null) { + $this->transactions->addCallbackForRollback( + static fn () => DispatchLockContext::releaseSnapshot($snapshot), + $connection->getNameWithReadWriteType() ?? '', + ); + } + } + } +} diff --git a/src/queue/src/Queue.php b/src/queue/src/Queue.php index 1ba59d3ad..54d940011 100644 --- a/src/queue/src/Queue.php +++ b/src/queue/src/Queue.php @@ -25,6 +25,7 @@ use Hypervel\Queue\Attributes\ReadsQueueAttributes; use Hypervel\Queue\Attributes\Timeout; use Hypervel\Queue\Attributes\Tries; +use Hypervel\Queue\Concerns\StagesOutboxJobs; use Hypervel\Queue\Events\JobPayloadFinalizing; use Hypervel\Queue\Events\JobQueued; use Hypervel\Queue\Events\JobQueueing; @@ -46,6 +47,7 @@ abstract class Queue { + use StagesOutboxJobs; use InteractsWithTime; use ReadsQueueAttributes; use ResolvesQueueRoutes; @@ -106,6 +108,7 @@ public function laterOn(UnitEnum|string|null $queue, DateInterval|DateTimeInterf public function bulk(array $jobs, mixed $data = '', UnitEnum|string|null $queue = null): mixed { $queue = $this->normalizeQueue($queue); + $jobs = $this->stageOutboxJobs($jobs, $data, $queue); foreach ((array) $jobs as $job) { $delay = $this->getJobDelay($job); @@ -120,6 +123,27 @@ public function bulk(array $jobs, mixed $data = '', UnitEnum|string|null $queue return null; } + /** + * Prepare payloads and delays for a set-based enqueue operation. + * + * @return array + */ + protected function prepareBatchJobs(array $jobs, mixed $data, ?string $queue): array + { + $prepared = []; + + foreach ($jobs as $job) { + $delay = $this->getJobDelay($job); + $prepared[] = [ + 'job' => $job, + 'delay' => $delay, + 'payload' => $this->createPayload($job, $queue, $data, $delay), + ]; + } + + return $prepared; + } + /** * Get the delay configured on the given job. */ @@ -413,6 +437,20 @@ public function setAfterCommitDispatcher(?Closure $dispatcher): static */ protected function enqueueUsing(object|string $job, string $payload, ?string $queue, DateInterval|DateTimeInterface|int|null $delay, callable $callback): mixed { + try { + $stageViaOutbox = $this->shouldStageViaOutbox($job); + } catch (CanceledException $exception) { + throw $exception; + } catch (Throwable $exception) { + $this->raiseJobQueueingFailedEvent($queue, $job, $payload, $delay, $exception); + + throw $exception; + } + + if ($stageViaOutbox) { + return $this->enqueueViaOutbox($job, $payload, $queue, $delay); + } + if ($this->shouldDispatchAfterCommit($job) && $this->container->has('db.transactions') ) { @@ -456,9 +494,17 @@ protected function deferBatchEnqueueAfterCommit( DatabaseTransactionsManager $transactions, array $jobs, Closure $enqueue, + ?string $connection = null, ): void { + // Inline work still belongs to its caller, including a failover's next attempt. + if (! $transactions->hasApplicableTransaction($connection)) { + $enqueue($this); + + return; + } + foreach ($jobs as $job) { - $this->addJobRollbackCallback($transactions, $job); + $this->addJobRollbackCallback($transactions, $job, $connection); } $callback = static function (Queue $owner) use ($jobs, $enqueue): mixed { @@ -483,11 +529,13 @@ protected function deferBatchEnqueueAfterCommit( $dispatcher = $this->afterCommitDispatcher; $transactions->addCallback( - static fn () => $dispatcher($callback) + static fn () => $dispatcher($callback), + $connection, ); } else { $transactions->addCallback( - fn () => $callback($this) + fn () => $callback($this), + $connection, ); } @@ -506,7 +554,8 @@ protected function enqueueNow(object|string $job, string $payload, ?string $queu try { $payload = $this->finalizePayloadForQueueing($queue, $job, $payload, $delay); $this->raiseJobQueueingEvent($queue, $job, $payload, $delay); - $jobId = $callback($this, $payload, $queue, $delay); + $storedPayload = $this->preparePayloadForPush($job, $payload, $queue); + $jobId = $callback($this, $storedPayload, $queue, $delay); } catch (CanceledException $exception) { throw $exception; } catch (Throwable $exception) { @@ -516,11 +565,72 @@ protected function enqueueNow(object|string $job, string $payload, ?string $queu } $this->acceptDispatchLocks($job); - $this->raiseJobQueuedEvent($queue, $jobId, $job, $payload, $delay); + $exception = null; + + try { + $this->handlePayloadPushed($storedPayload, $queue); + } catch (CanceledException $cancellation) { + throw $cancellation; + } catch (Throwable $throwable) { + $exception = $throwable; + } + + try { + $this->raiseJobQueuedEvent($queue, $jobId, $job, $payload, $delay); + } catch (CanceledException $cancellation) { + throw $cancellation; + } catch (Throwable $throwable) { + $exception ??= $throwable; + } + + if ($exception !== null) { + throw $exception; + } return $jobId; } + /** + * Prepare a finalized payload for storage without changing lifecycle event correlation. + */ + protected function preparePayloadForPush(object|string $job, string $payload, ?string $queue): string + { + return $payload; + } + + /** + * Handle a payload after the queue has accepted it. + */ + protected function handlePayloadPushed(string $payload, ?string $queue): void + { + } + + /** + * Notify every batch member and return the first ordinary notification failure. + * + * @template TMember + * @param array $members + * @param Closure(TMember): void ...$notifications + */ + protected function notifyBatchMembers(array $members, Closure ...$notifications): ?Throwable + { + $exception = null; + + foreach ($members as $member) { + foreach ($notifications as $notification) { + try { + $notification($member); + } catch (CanceledException $cancellation) { + throw $cancellation; + } catch (Throwable $throwable) { + $exception ??= $throwable; + } + } + } + + return $exception; + } + /** * Finalize an encoded payload immediately before queueing. * @@ -583,25 +693,31 @@ protected function partitionJobsByAfterCommit( protected function shouldDispatchAfterCommit(object|string $job): bool { if ($job instanceof ShouldQueueAfterCommit) { - return ! (isset($job->afterCommit) && $job->afterCommit === false); + $afterCommit = ! (isset($job->afterCommit) && $job->afterCommit === false); + } elseif (! $job instanceof Closure && is_object($job) && isset($job->afterCommit)) { + $afterCommit = $job->afterCommit; + } else { + $afterCommit = $this->dispatchAfterCommit; } - if (! $job instanceof Closure && is_object($job) && isset($job->afterCommit)) { - return $job->afterCommit; + if (! $afterCommit) { + return false; } - return $this->dispatchAfterCommit; + $forwarded = CoroutineContext::get(self::OUTBOX_FORWARDED_CONTEXT_KEY); + + return ($forwarded['job'] ?? null) !== $job || ! $forwarded['afterCommitHandled']; } /** * Register a transaction rollback callback that releases the job's locks. */ - protected function addJobRollbackCallback(DatabaseTransactionsManager $transactions, object|string $job): void + protected function addJobRollbackCallback(DatabaseTransactionsManager $transactions, object|string $job, ?string $connection = null): void { $callback = $this->createJobRollbackCallback($job); if ($callback !== null) { - $transactions->addCallbackForRollback($callback); + $transactions->addCallbackForRollback($callback, $connection); } } @@ -758,6 +874,7 @@ public function getConfig(): array public function setConfig(array $config): static { $this->config = $config; + $this->dispatchViaOutbox = $config['outbox'] ?? false; return $this; } diff --git a/src/queue/src/QueuePoolProxy.php b/src/queue/src/QueuePoolProxy.php index 8f0bd3e4e..ad94e2ca3 100644 --- a/src/queue/src/QueuePoolProxy.php +++ b/src/queue/src/QueuePoolProxy.php @@ -10,6 +10,7 @@ use Hypervel\Contracts\ObjectPool\Factory; use Hypervel\Contracts\Queue\IndexAwareQueue; use Hypervel\Contracts\Queue\Job; +use Hypervel\Contracts\Queue\PublishesOutboxMessages; use Hypervel\Contracts\Queue\Queue as QueueContract; use Hypervel\Database\ConnectionResolver; use Hypervel\ObjectPool\Lease; @@ -17,14 +18,19 @@ use Hypervel\ObjectPool\PoolErrorReporter; use Hypervel\ObjectPool\PoolProxy; use Hypervel\Queue\Jobs\Job as PoolLeaseAwareJob; +use Hypervel\Queue\Outbox\Concerns\PublishesOutboxMessagesSequentially; +use Hypervel\Queue\Outbox\OutboxBatchResult; +use Hypervel\Queue\Outbox\OutboxMessage; use Hypervel\Support\Collection; use RuntimeException; use Swoole\Coroutine\CanceledException; use Throwable; use UnitEnum; -class QueuePoolProxy extends PoolProxy implements QueueContract, IndexAwareQueue +class QueuePoolProxy extends PoolProxy implements QueueContract, IndexAwareQueue, PublishesOutboxMessages { + use PublishesOutboxMessagesSequentially; + /** * The logical connection name applied to each borrowed queue. */ @@ -223,6 +229,18 @@ public function bulk(array $jobs, mixed $data = '', UnitEnum|string|null $queue return $this->invoke(__FUNCTION__, func_get_args()); } + /** + * Publish a stored batch while borrowing its transport only once. + * + * @param list $messages + */ + public function publishOutboxMessages(array $messages): OutboxBatchResult + { + return $this->usingConnection(fn (QueueContract $queue): OutboxBatchResult => $queue instanceof PublishesOutboxMessages + ? $queue->publishOutboxMessages($messages) + : $this->publishOutboxMessagesSequentially($queue, $messages)); + } + /** * Run a callback with the underlying SQS queue. * @@ -342,14 +360,14 @@ protected function configureBorrowed(object $object): void /** * Run a callback with a borrowed queue. * - * @param Closure(Queue): mixed $callback + * @param Closure(Queue&QueueContract): mixed $callback */ protected function usingConnection(Closure $callback): mixed { $lease = $this->lease(); try { - /** @var Queue $queue */ + /** @var Queue&QueueContract $queue */ $queue = $lease->get(); $result = $callback($queue); } catch (Throwable $operationException) { diff --git a/src/queue/src/QueueServiceProvider.php b/src/queue/src/QueueServiceProvider.php index 3ec9787ec..296ed095f 100644 --- a/src/queue/src/QueueServiceProvider.php +++ b/src/queue/src/QueueServiceProvider.php @@ -19,6 +19,7 @@ use Hypervel\Queue\Console\ListenCommand; use Hypervel\Queue\Console\ListFailedCommand; use Hypervel\Queue\Console\MonitorCommand; +use Hypervel\Queue\Console\OutboxTableCommand; use Hypervel\Queue\Console\PauseCommand; use Hypervel\Queue\Console\PruneBatchesCommand; use Hypervel\Queue\Console\PruneFailedJobsCommand; @@ -32,6 +33,7 @@ use Hypervel\Queue\Failed\DatabaseUuidFailedJobProvider; use Hypervel\Queue\Failed\FileFailedJobProvider; use Hypervel\Queue\Failed\NullFailedJobProvider; +use Hypervel\Queue\Outbox\DatabaseOutboxRepository; use Hypervel\Support\ServiceProvider; use InvalidArgumentException; use Laravel\SerializableClosure\SerializableClosure; @@ -55,6 +57,7 @@ public function register(): void $this->registerListener(); $this->registerRoutes(); $this->registerFailedJobServices(); + $this->registerOutbox(); $this->commands([ BatchesTableCommand::class, @@ -65,6 +68,7 @@ public function register(): void ListenCommand::class, ListFailedCommand::class, MonitorCommand::class, + OutboxTableCommand::class, PauseCommand::class, PruneBatchesCommand::class, PruneFailedJobsCommand::class, @@ -219,6 +223,23 @@ protected function registerRoutes(): void $this->app->singleton('queue.routes', fn ($app) => $app->make(QueueRoutes::class)); } + /** + * Register transactional outbox services without resolving their storage. + */ + protected function registerOutbox(): void + { + $this->app->singleton(DatabaseOutboxRepository::class, function ($app): DatabaseOutboxRepository { + $config = $app->make('config'); + + return new DatabaseOutboxRepository( + $app->make('db'), + $config->array('queue.outbox.databases'), + $config->string('queue.outbox.table'), + $config->integer('queue.outbox.retry_after'), + ); + }); + } + /** * Register the failed job services. */ diff --git a/src/queue/src/RedisQueue.php b/src/queue/src/RedisQueue.php index a3ce84dfa..4e02a5416 100644 --- a/src/queue/src/RedisQueue.php +++ b/src/queue/src/RedisQueue.php @@ -9,11 +9,14 @@ use Hypervel\Contracts\Queue\ClearableQueue; use Hypervel\Contracts\Queue\IndexAwareQueue; use Hypervel\Contracts\Queue\Job as JobContract; +use Hypervel\Contracts\Queue\PublishesOutboxMessages; use Hypervel\Contracts\Queue\Queue as QueueContract; use Hypervel\Contracts\Redis\Factory as Redis; use Hypervel\Database\DatabaseTransactionsManager; use Hypervel\Queue\Jobs\InspectedJob; use Hypervel\Queue\Jobs\RedisJob; +use Hypervel\Queue\Outbox\OutboxBatchResult; +use Hypervel\Queue\Outbox\OutboxMessage; use Hypervel\Redis\RedisConnection; use Hypervel\Redis\RedisProxy; use Hypervel\Support\Collection; @@ -23,7 +26,7 @@ use Throwable; use UnitEnum; -class RedisQueue extends Queue implements QueueContract, ClearableQueue, IndexAwareQueue +class RedisQueue extends Queue implements QueueContract, ClearableQueue, IndexAwareQueue, PublishesOutboxMessages { public const int DEFAULT_RETRY_AFTER = 60; @@ -345,7 +348,7 @@ public function creationTimeOfOldestPendingJob(UnitEnum|string|null $queue = nul public function bulk(array $jobs, mixed $data = '', UnitEnum|string|null $queue = null): mixed { $queue = $this->normalizeQueue($queue); - $jobs = array_values($jobs); + $jobs = array_values($this->stageOutboxJobs($jobs, $data, $queue)); if ($jobs === []) { return null; @@ -392,17 +395,7 @@ static function (Queue $owner) use ($preparedJobs, $queue): void { */ protected function prepareBatchJobs(array $jobs, mixed $data, ?string $queue): array { - return (new Collection($jobs)) - ->map(function (object|string $job) use ($data, $queue): array { - $delay = $this->getJobDelay($job); - - return [ - 'job' => $job, - 'delay' => $delay, - 'payload' => $this->createPayload($job, $this->getQueue($queue), $data, $delay), - ]; - }) - ->all(); + return parent::prepareBatchJobs($jobs, $data, $this->getQueue($queue)); } /** @@ -422,7 +415,7 @@ protected function enqueueBatch(array $jobs, ?string $queue): void foreach ($jobs as $index => $job) { $this->raiseJobQueueingEvent($queue, $job['job'], $job['payload'], $job['delay']); - $jobs[$index]['payload'] = $this->preparePayloadForBulk( + $jobs[$index]['storedPayload'] = $this->preparePayloadForPush( $job['job'], $job['payload'], $queue, @@ -433,54 +426,116 @@ protected function enqueueBatch(array $jobs, ?string $queue): void foreach ($jobs as $job) { $arguments[] = $job['delay'] === null ? 'i' : $this->availableAt($job['delay']); - $arguments[] = $job['payload']; + $arguments[] = $job['storedPayload']; } $connection = $this->getConnection(); $this->isCluster ??= $connection->isCluster(); $queueKey = $this->getQueueRedisKey($queue); - $stored = $connection->evalWithShaCache( - LuaScripts::bulk(), - [$queueKey, $queueKey . ':notify', $queueKey . ':delayed'], - $arguments, - ); - - if (! is_int($stored) || $stored !== count($jobs)) { - throw new RuntimeException('Redis did not confirm every queued job in the batch.'); - } + $this->pushRawBatch($connection, $queueKey, $arguments); } catch (CanceledException $exception) { throw $exception; } catch (Throwable $exception) { - foreach ($jobs as $job) { - $this->raiseJobQueueingFailedEvent($queue, $job['job'], $job['payload'], $job['delay'], $exception); - } - - throw $exception; + throw $this->notifyBatchMembers( + $jobs, + fn (array $job) => $this->raiseJobQueueingFailedEvent($queue, $job['job'], $job['payload'], $job['delay'], $exception), + ) ?? $exception; } foreach ($jobs as $job) { $this->acceptDispatchLocks($job['job']); } - foreach ($jobs as $job) { - $this->handlePayloadPushedInBulk($job['payload'], $queue); - $this->raiseJobQueuedEvent($queue, null, $job['job'], $job['payload'], $job['delay']); + $exception = $this->notifyBatchMembers( + $jobs, + fn (array $job) => $this->handlePayloadPushed($job['storedPayload'], $queue), + fn (array $job) => $this->raiseJobQueuedEvent($queue, null, $job['job'], $job['payload'], $job['delay']), + ); + + if ($exception !== null) { + throw $exception; } } /** - * Prepare a payload for bulk storage. + * Publish stored outbox messages with one confirmed script per destination. + * + * @param list $messages */ - protected function preparePayloadForBulk(object|string $job, string $payload, ?string $queue): string + public function publishOutboxMessages(array $messages): OutboxBatchResult + { + $accepted = []; + + if ($messages === []) { + return new OutboxBatchResult; + } + + try { + $connection = $this->getConnection(); + $this->isCluster ??= $connection->isCluster(); + $groups = []; + + foreach ($messages as $message) { + $groups[$this->getQueueRedisKey($message->queue)][] = $message; + } + + foreach ($groups as $queueKey => $group) { + $prepared = []; + $arguments = []; + $now = $this->currentTime(); + + foreach ($group as $message) { + $payload = $this->preparePayloadForOutboxPublication($message->payload, $message->queue); + $prepared[] = ['payload' => $payload, 'queue' => $message->queue]; + $arguments[] = $message->deliverAt !== null && $message->deliverAt > $now ? $message->deliverAt : 'i'; + $arguments[] = $payload; + } + + $this->pushRawBatch($connection, $queueKey, $arguments); + + foreach ($group as $message) { + $accepted[] = $message->id; + } + + $failure = $this->notifyBatchMembers($prepared, fn (array $member) => $this->handlePayloadPushed($member['payload'], $member['queue'])); + + if ($failure !== null) { + throw $failure; + } + } + } catch (CanceledException $exception) { + throw $exception; + } catch (Throwable $exception) { + return new OutboxBatchResult($accepted, failure: $exception); + } + + return new OutboxBatchResult($accepted); + } + + /** + * Add publication-time metadata without recreating the stored job payload. + */ + protected function preparePayloadForOutboxPublication(string $payload, ?string $queue): string { return $payload; } /** - * Handle a payload that was stored as part of a batch. + * Store alternating availability and payload arguments with exact confirmation. + * + * @param list $arguments */ - protected function handlePayloadPushedInBulk(string $payload, ?string $queue): void + protected function pushRawBatch(RedisProxy $connection, string $queueKey, array $arguments): void { + $stored = $connection->evalWithShaCache( + LuaScripts::bulk(), + [$queueKey, $queueKey . ':notify', $queueKey . ':delayed'], + $arguments, + ); + + if (! is_int($stored) || $stored !== intdiv(count($arguments), 2)) { + throw new RuntimeException('Redis did not confirm every queued job in the batch.'); + } } /** @@ -723,6 +778,22 @@ public function getQueue(UnitEnum|string|null $queue): string return 'queues:' . $this->resolveQueue($queue === null || $queue === '' ? $this->default : $queue); } + /** + * Capture the logical destination before publication-time forwarding. + */ + protected function resolveOutboxQueue(?string $queue): ?string + { + return $queue === null || $queue === '' ? $this->default : $queue; + } + + /** + * Publish delayed Redis jobs immediately into their delayed set. + */ + protected function availableAtForOutbox(?string $queue, ?int $deliverAt): int + { + return $this->currentTime(); + } + /** * Get the cluster-safe Redis key for the given queue. * diff --git a/src/queue/src/SqsQueue.php b/src/queue/src/SqsQueue.php index 9573e3130..bddbb4639 100644 --- a/src/queue/src/SqsQueue.php +++ b/src/queue/src/SqsQueue.php @@ -5,6 +5,7 @@ namespace Hypervel\Queue; use Aws\Command; +use Aws\ResultInterface; use Aws\Sqs\Exception\SqsException; use Aws\Sqs\SqsClient; use DateInterval; @@ -13,10 +14,13 @@ use Hypervel\Contracts\Cache\Repository as CacheRepository; use Hypervel\Contracts\Queue\ClearableQueue; use Hypervel\Contracts\Queue\Job as JobContract; +use Hypervel\Contracts\Queue\PublishesOutboxMessages; use Hypervel\Contracts\Queue\Queue as QueueContract; use Hypervel\Database\DatabaseTransactionsManager; use Hypervel\ObjectPool\PoolErrorReporter; use Hypervel\Queue\Jobs\SqsJob; +use Hypervel\Queue\Outbox\OutboxBatchResult; +use Hypervel\Queue\Outbox\OutboxMessage; use Hypervel\Support\Arr; use Hypervel\Support\Collection; use Hypervel\Support\Str; @@ -26,7 +30,7 @@ use Throwable; use UnitEnum; -class SqsQueue extends Queue implements QueueContract, ClearableQueue +class SqsQueue extends Queue implements QueueContract, ClearableQueue, PublishesOutboxMessages { /** * The maximum SQS payload size in bytes (1 MB). @@ -282,7 +286,9 @@ public function later(DateInterval|DateTimeInterface|int $delay, object|string $ $queue = $this->normalizeQueue($queue); $queueName = $this->resolveQueueName($queue); - $this->ensureDelayIsSupported($delay, $queueName); + if (! $this->shouldStageViaOutbox($job)) { + $this->ensureDelayIsSupported($delay, $queueName); + } return $this->enqueueUsing( $job, @@ -315,7 +321,7 @@ static function ( public function bulk(array $jobs, mixed $data = '', UnitEnum|string|null $queue = null): mixed { $queue = $this->normalizeQueue($queue); - $jobs = array_values($jobs); + $jobs = array_values($this->stageOutboxJobs($jobs, $data, $queue)); if ($jobs === []) { return null; @@ -341,30 +347,14 @@ public function bulk(array $jobs, mixed $data = '', UnitEnum|string|null $queue } $messages = $this->prepareBatchMessages($afterCommit, $data, $queue); - - // A non-empty deferred group means partitionJobsByAfterCommit() resolved a transactions manager. - foreach ($afterCommit as $job) { - /** @var DatabaseTransactionsManager $transactions */ - $this->addJobRollbackCallback($transactions, $job); - } - - if ($this->afterCommitDispatcher !== null) { - $dispatcher = $this->afterCommitDispatcher; - - $transactions->addCallback( - static fn () => $dispatcher( - static function (Queue $owner) use ($messages, $queue): void { - /** @var SqsQueue $owner */ - $owner->sendBatchedMessages($messages, $queue); - } - ) - ); - - return null; - } - - $transactions->addCallback( - fn () => $this->sendBatchedMessages($messages, $queue) + /** @var DatabaseTransactionsManager $transactions */ + $this->deferBatchEnqueueAfterCommit( + $transactions, + $afterCommit, + static function (Queue $owner) use ($messages, $queue): void { + /** @var SqsQueue $owner */ + $owner->sendBatchedMessages($messages, $queue); + }, ); return null; @@ -386,6 +376,14 @@ protected function ensureBulkDelaysAreSupported(array $jobs, ?string $queue): vo } } + /** + * Reuse SQS payload preparation for set-based outbox staging. + */ + protected function prepareBatchJobs(array $jobs, mixed $data, ?string $queue): array + { + return $this->prepareBatchMessages($jobs, $data, $queue); + } + /** * Create the payload for each of the given jobs. * @@ -423,22 +421,27 @@ protected function prepareBatchMessages(array $jobs, mixed $data, ?string $queue */ protected function sendBatchedMessages(array $messages, ?string $queue): void { - foreach ($messages as $id => $message) { - $messages[$id]['payload'] = $this->finalizePayloadForQueueing( - $queue, - $message['job'], - $message['payload'], - $message['delay'], - ); - } - $outstandingMessageIds = array_fill_keys(array_keys($messages), true); try { + foreach ($messages as $id => $message) { + $messages[$id]['payload'] = $this->finalizePayloadForQueueing( + $queue, + $message['job'], + $message['payload'], + $message['delay'], + ); + } + $entries = []; $overflow = []; foreach ($messages as $id => $message) { + $message['payload'] = $messages[$id]['storedPayload'] = $this->preparePayloadForPush( + $message['job'], + $message['payload'], + $queue, + ); $entry = $this->prepareSendMessageBatchEntry($id, $message, $queue); if ($this->willOverflow($message['payload'])) { @@ -462,47 +465,19 @@ protected function sendBatchedMessages(array $messages, ?string $queue): void // Dispatch chunks serially so later messages cannot arrive ahead of an unsent failed chunk... foreach ($chunks as $chunk) { - foreach ($chunk as $entry) { - $message = $messages[$entry['Id']]; - - $this->raiseJobQueueingEvent($queue, $message['job'], $message['payload'], $message['delay']); - } - - $writtenPaths = []; - $attemptedPath = null; - try { foreach ($chunk as $entry) { - if (! isset($overflow[$entry['Id']])) { - continue; - } + $message = $messages[$entry['Id']]; - /** @var CacheRepository $store */ - $overflowPayload = $overflow[$entry['Id']]; - $attemptedPath = $overflowPayload['path']; - $this->storeOverflowPayload($store, $attemptedPath, $overflowPayload['payload']); - $writtenPaths[] = $attemptedPath; - $attemptedPath = null; - } - } catch (CanceledException $cancellation) { - if ($attemptedPath !== null) { - $writtenPaths[] = $attemptedPath; + $this->raiseJobQueueingEvent($queue, $message['job'], $message['payload'], $message['delay']); } - try { - /** @var CacheRepository $store */ - $this->cleanupOverflowPayloads($store, $writtenPaths); - } catch (CanceledException) { + if ($store !== null) { + $this->storeBatchOverflowPayloads($chunk, $overflow, $store); } - - throw $cancellation; + } catch (CanceledException $exception) { + throw $exception; } catch (Throwable $exception) { - if ($attemptedPath !== null) { - $writtenPaths[] = $attemptedPath; - } - - /** @var CacheRepository $store */ - $this->cleanupOverflowPayloads($store, $writtenPaths); $this->raiseBatchFailedEvents($messages, $outstandingMessageIds, $queue, $exception); throw $exception; @@ -523,6 +498,8 @@ protected function sendBatchedMessages(array $messages, ?string $queue): void throw $exception; } + $accepted = []; + foreach ($result['Successful'] ?? [] as $success) { if (! isset($outstandingMessageIds[$success['Id']])) { continue; @@ -530,53 +507,169 @@ protected function sendBatchedMessages(array $messages, ?string $queue): void $message = $messages[$success['Id']]; unset($outstandingMessageIds[$success['Id']]); + $this->acceptDispatchLocks($message['job']); + $accepted[] = $success; + } - $this->raiseJobQueuedEvent( - $queue, - $success['MessageId'], - $message['job'], - $message['payload'], - $message['delay'], - ); + $exception = null; + + if (! empty($result['Failed'])) { + $exception = $this->batchRejectionException($queueUrl, $chunk, $result); + + if ($store !== null) { + $rejectedPaths = []; + + foreach ($result['Failed'] as $rejected) { + if (isset($overflow[$rejected['Id']])) { + $rejectedPaths[] = $overflow[$rejected['Id']]['path']; + } + } + + $this->cleanupOverflowPayloads($store, $rejectedPaths); + } } - if (empty($result['Failed'])) { + $notificationFailure = $this->notifyBatchMembers( + $accepted, + fn (array $success) => $this->handlePayloadPushed($messages[$success['Id']]['storedPayload'], $queue), + function (array $success) use ($messages, $queue): void { + $message = $messages[$success['Id']]; + $this->raiseJobQueuedEvent($queue, $success['MessageId'], $message['job'], $message['payload'], $message['delay']); + }, + ); + $exception ??= $notificationFailure; + + if ($exception === null) { continue; } - $failure = $result['Failed'][0]; - $exception = new SqsException( - sprintf( - 'SQS SendMessageBatch rejected [%d] of [%d] messages. First failure [%s]: %s', - count($result['Failed']), - count($chunk), - $failure['Code'] ?? 'Unknown', - $failure['Message'] ?? '', - ), - new Command('SendMessageBatch', ['QueueUrl' => $queueUrl, 'Entries' => $chunk]), - [ - 'code' => $failure['Code'] ?? null, - 'message' => $failure['Message'] ?? null, - 'result' => $result, - ], - ); + try { + $this->raiseBatchFailedEvents($messages, $outstandingMessageIds, $queue, $exception); + } catch (CanceledException $cancellation) { + throw $cancellation; + } catch (Throwable $throwable) { + $notificationFailure ??= $throwable; + } + + throw $notificationFailure ?? $exception; + } + } + + /** + * Publish stored SQS messages with attempt-owned overflow payloads. + * + * @param list $messages + */ + public function publishOutboxMessages(array $messages): OutboxBatchResult + { + $accepted = []; - if ($store !== null) { - $rejectedPaths = []; + try { + $groups = []; + + foreach ($messages as $message) { + $groups[$this->getQueue($message->queue)][$message->id] = $message; + } - foreach ($result['Failed'] as $rejected) { - if (isset($overflow[$rejected['Id']])) { - $rejectedPaths[] = $overflow[$rejected['Id']]['path']; + foreach ($groups as $queueUrl => $group) { + $entries = []; + $overflow = []; + + foreach ($group as $message) { + $payload = $message->payload; + + if ($this->willOverflow($payload)) { + $path = static::EXTENDED_PAYLOAD_CACHE_PREFIX . $message->uuid . ':' . $message->leaseToken; + $overflow[$message->id] = ['path' => $path, 'payload' => $payload]; + $payload = json_encode(['@pointer' => $path], JSON_THROW_ON_ERROR); } + + $entries[] = [ + 'Id' => (string) $message->id, + 'MessageBody' => $payload, + ...($message->options['transport']['sqs'] ?? []), + ]; } - $this->cleanupOverflowPayloads($store, $rejectedPaths); - } + $store = $overflow === [] ? null : $this->overflowStore(); + $fifo = str_ends_with($queueUrl, '.fifo'); - $this->raiseBatchFailedEvents($messages, $outstandingMessageIds, $queue, $exception); + foreach ($this->chunkBatchEntries($entries) as $chunk) { + if ($store !== null) { + $this->storeBatchOverflowPayloads($chunk, $overflow, $store); + } + $now = $this->currentTime(); + + if (! $fifo) { + foreach ($chunk as &$entry) { + $deliverAt = $group[$entry['Id']]->deliverAt; + + if ($deliverAt !== null) { + $entry['DelaySeconds'] = min(900, max(0, $deliverAt - $now)); + } + } + unset($entry); + } + + // SDK failures are ambiguous; only explicit rejects own safe cleanup. + $result = $this->sqs->sendMessageBatch(['QueueUrl' => $queueUrl, 'Entries' => $chunk]); + $chunkIds = array_fill_keys(array_column($chunk, 'Id'), true); + + foreach ($result['Successful'] ?? [] as $success) { + if (isset($chunkIds[$success['Id']])) { + $accepted[] = (int) $success['Id']; + } + } + + if (! empty($result['Failed'])) { + if ($store !== null) { + $paths = []; + + foreach ($result['Failed'] as $rejected) { + if (isset($chunkIds[$rejected['Id']], $overflow[$rejected['Id']])) { + $paths[] = $overflow[$rejected['Id']]['path']; + } + } + + $this->cleanupOverflowPayloads($store, $paths); + } + + throw $this->batchRejectionException($queueUrl, $chunk, $result); + } + } + } + } catch (CanceledException $exception) { throw $exception; + } catch (Throwable $exception) { + return new OutboxBatchResult($accepted, failure: $exception); } + + return new OutboxBatchResult($accepted); + } + + /** + * Describe explicit SQS rejects while retaining the complete response. + */ + protected function batchRejectionException(string $queueUrl, array $chunk, ResultInterface $result): SqsException + { + $failure = $result['Failed'][0]; + + return new SqsException( + sprintf( + 'SQS SendMessageBatch rejected [%d] of [%d] messages. First failure [%s]: %s', + count($result['Failed']), + count($chunk), + $failure['Code'] ?? 'Unknown', + $failure['Message'] ?? '', + ), + new Command('SendMessageBatch', ['QueueUrl' => $queueUrl, 'Entries' => $chunk]), + [ + 'code' => $failure['Code'] ?? null, + 'message' => $failure['Message'] ?? null, + 'result' => $result, + ], + ); } /** @@ -588,16 +681,16 @@ protected function raiseBatchFailedEvents( ?string $queue, Throwable $exception, ): void { - foreach (array_keys($outstandingMessageIds) as $id) { - $message = $messages[$id]; + $notificationFailure = $this->notifyBatchMembers( + array_keys($outstandingMessageIds), + function (int $id) use ($messages, $queue, $exception): void { + $message = $messages[$id]; + $this->raiseJobQueueingFailedEvent($queue, $message['job'], $message['payload'], $message['delay'], $exception); + }, + ); - $this->raiseJobQueueingFailedEvent( - $queue, - $message['job'], - $message['payload'], - $message['delay'], - $exception, - ); + if ($notificationFailure !== null) { + throw $notificationFailure; } } @@ -691,6 +784,41 @@ protected function storeOverflowPayload(CacheRepository $store, string $path, st } } + /** + * Store one chunk's overflow payloads, cleaning every attempted write on failure. + */ + protected function storeBatchOverflowPayloads(array $chunk, array $overflow, CacheRepository $store): void + { + $paths = []; + + try { + foreach ($chunk as $entry) { + if (! isset($overflow[$entry['Id']])) { + continue; + } + + $payload = $overflow[$entry['Id']]; + $paths[] = $payload['path']; + $this->storeOverflowPayload($store, $payload['path'], $payload['payload']); + } + } catch (CanceledException $cancellation) { + if ($paths !== []) { + try { + $this->cleanupOverflowPayloads($store, $paths); + } catch (CanceledException) { + } + } + + throw $cancellation; + } catch (Throwable $exception) { + if ($paths !== []) { + $this->cleanupOverflowPayloads($store, $paths); + } + + throw $exception; + } + } + /** * Remove unpublished overflow payloads without hiding the primary failure. * @@ -869,6 +997,40 @@ public function getQueue(UnitEnum|string|null $queue): string : $queue; } + /** + * Capture the logical destination before publication-time forwarding. + */ + protected function resolveOutboxQueue(?string $queue): ?string + { + return $queue === null || $queue === '' ? $this->default : $queue; + } + + /** + * Capture FIFO options once while the original job and callbacks are available. + */ + protected function prepareOutboxOptions(object|string $job, ?string $queue, string $payload): array + { + $options = $this->getQueueableOptions($job, $queue, $payload); + + return $options === [] ? [] : ['transport' => ['sqs' => $options]]; + } + + /** + * Keep a delayed job staged until SQS can honor its remaining delay. + */ + protected function availableAtForOutbox(?string $queue, ?int $deliverAt): int + { + $now = $this->currentTime(); + + if ($deliverAt === null) { + return $now; + } + + return str_ends_with($this->resolveQueueName($queue), '.fifo') + ? $deliverAt + : max($now, $deliverAt - 900); + } + /** * Add the given suffix to the given queue name. */ diff --git a/src/queue/src/SyncQueue.php b/src/queue/src/SyncQueue.php index 21884e5ff..494ced555 100644 --- a/src/queue/src/SyncQueue.php +++ b/src/queue/src/SyncQueue.php @@ -4,8 +4,10 @@ namespace Hypervel\Queue; +use Closure; use DateInterval; use DateTimeInterface; +use Hypervel\Context\CoroutineContext; use Hypervel\Contracts\Events\Dispatcher as EventDispatcher; use Hypervel\Contracts\Queue\Job as JobContract; use Hypervel\Contracts\Queue\Queue as QueueContract; @@ -215,7 +217,14 @@ protected function executePayload(string $payload, ?string $queue = null): int { // JobProcessing hydrates the payload's context. Keep it through all job // events without replacing the context of the dispatching coroutine. - return $this->withLogContext(null, fn (): int => $this->processPayload($payload, $queue)); + return $this->withLogContext(null, function () use ($payload, $queue): int { + if (CoroutineContext::get(self::OUTBOX_FORWARDED_CONTEXT_KEY) === null) { + return $this->processPayload($payload, $queue); + } + + // A running handler may dispatch independent work through the same string handler. + return $this->withOutboxForwarding(null, fn (): int => $this->processPayload($payload, $queue)); + }); } /** @@ -357,6 +366,35 @@ public function later(DateInterval|DateTimeInterface|int $delay, object|string $ return $this->push($job, $data, $queue); } + /** + * Keep local queue execution out of durable outbox storage. + */ + protected function shouldStageViaOutbox(object|string $job): bool + { + return false; + } + + /** + * Degrade selected outbox delivery to local execution after commit. + */ + protected function shouldDispatchAfterCommit(object|string $job): bool + { + return $this->shouldDispatchViaOutbox($job) || parent::shouldDispatchAfterCommit($job); + } + + /** + * Honor the selected transaction without resolving outbox storage services. + */ + protected function deferEnqueueAfterCommit(DatabaseTransactionsManager $transactions, object|string $job, Closure $enqueue): void + { + $this->deferBatchEnqueueAfterCommit( + $transactions, + [$job], + $enqueue, + $this->resolveOutboxCommitConnection($transactions, $job), + ); + } + /** * Pop the next job off of the queue. */ diff --git a/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php b/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php new file mode 100644 index 000000000..446dc84f0 --- /dev/null +++ b/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php @@ -0,0 +1,40 @@ +id(); + $table->uuid('uuid')->unique(); + $table->string('connection'); + $table->string('queue')->nullable(); + $table->longText('payload'); + $table->text('options')->nullable(); + $table->unsignedBigInteger('deliver_at')->nullable(); + $table->unsignedInteger('attempts')->default(0); + $table->unsignedBigInteger('available_at'); + $table->uuid('lease_token')->nullable(); + $table->text('last_error')->nullable(); + $table->unsignedBigInteger('created_at'); + + $table->index(['available_at', 'id']); + }); + } + + /** + * Reverse the migrations. + */ + public function down(): void + { + Schema::dropIfExists('job_outbox'); + } +}; diff --git a/tests/Bus/DispatchLockContextTest.php b/tests/Bus/DispatchLockContextTest.php index 96ae1ceab..778ef531b 100644 --- a/tests/Bus/DispatchLockContextTest.php +++ b/tests/Bus/DispatchLockContextTest.php @@ -174,6 +174,7 @@ public function testMetadataSurvivesUntilAfterCommitPayloadCreation(string $queu Container::setInstance($container); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(fn (callable $callback) => $callback()); diff --git a/tests/Integration/Generators/QueueOutboxTableCommandTest.php b/tests/Integration/Generators/QueueOutboxTableCommandTest.php new file mode 100644 index 000000000..5d7d81082 --- /dev/null +++ b/tests/Integration/Generators/QueueOutboxTableCommandTest.php @@ -0,0 +1,29 @@ + 'outgoing_jobs']); + $this->artisan(OutboxTableCommand::class)->assertExitCode(0); + + $this->assertMigrationFileContains([ + 'use Hypervel\Database\Migrations\Migration;', + 'return new class extends Migration', + 'Schema::create(\'outgoing_jobs\', function (Blueprint $table) {', + '$table->uuid(\'uuid\')->unique();', + '$table->longText(\'payload\');', + '$table->unsignedBigInteger(\'deliver_at\')->nullable();', + '$table->index([\'available_at\', \'id\']);', + 'Schema::dropIfExists(\'outgoing_jobs\');', + ], 'create_outgoing_jobs_table.php'); + + $this->artisan('queue:outbox-table')->assertExitCode(1); + } +} diff --git a/tests/Integration/Queue/QueueConnectionTest.php b/tests/Integration/Queue/QueueConnectionTest.php index 3ddd5c7e8..d046bb0cd 100644 --- a/tests/Integration/Queue/QueueConnectionTest.php +++ b/tests/Integration/Queue/QueueConnectionTest.php @@ -59,6 +59,7 @@ public function testJobWontGetDispatchedInsideATransaction(): void { $this->app->singleton('db.transactions', function (): DatabaseTransactionsManager { $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('callbackApplicableTransactions') ->andReturn(new Collection([new DatabaseTransactionRecord('sqlite', 1)])); $transactionManager->expects('addCallback')->andReturn(null); @@ -93,6 +94,7 @@ public function testJobWontGetDispatchedInsideATransactionWhenExplicitlyIndicate $this->app->singleton('db.transactions', function (): DatabaseTransactionsManager { $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('callbackApplicableTransactions') ->andReturn(new Collection([new DatabaseTransactionRecord('sqlite', 1)])); $transactionManager->expects('addCallback')->andReturn(null); @@ -110,6 +112,7 @@ public function testUniqueJobWontGetDispatchedInsideATransaction(): void { $this->app->singleton('db.transactions', function (): DatabaseTransactionsManager { $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('callbackApplicableTransactions') ->andReturn(new Collection([new DatabaseTransactionRecord('sqlite', 1)])); $transactionManager->expects('addCallback')->andReturn(null); @@ -144,6 +147,7 @@ public function testUniqueJobWontGetDispatchedInsideATransactionWhenExplicitlyIn $this->app->singleton('db.transactions', function (): DatabaseTransactionsManager { $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('callbackApplicableTransactions') ->andReturn(new Collection([new DatabaseTransactionRecord('sqlite', 1)])); $transactionManager->expects('addCallback')->andReturn(null); diff --git a/tests/Queue/DatabaseQueueRollbackTest.php b/tests/Queue/DatabaseQueueRollbackTest.php new file mode 100644 index 000000000..89659a82f --- /dev/null +++ b/tests/Queue/DatabaseQueueRollbackTest.php @@ -0,0 +1,176 @@ +queue(); + $database = $queue->getDatabase(); + $cache = new Repository(new WorkerArrayStore); + $lock = $kind === 'unique' ? new UniqueLock($cache) : new DebounceLock($cache); + $job = (object) ['debounceFor' => 10]; + $lock->acquireForDispatch($job); + $database->beginTransaction(); + + match ($method) { + 'push' => $queue->push($job), + 'later' => $queue->later(30, $job), + 'bulk' => $queue->bulk([$job]), + }; + + $this->assertFalse(DispatchLockContext::has($job)); + $this->assertSame(1, $database->table('jobs')->count()); + $this->assertTrue($kind === 'unique' ? $cache->lock($lock::getKey($job))->isLocked() : $cache->has($lock::getKey($job))); + $database->rollBack(); + $this->assertSame(0, $database->table('jobs')->count()); + $this->assertFalse($kind === 'unique' ? $cache->lock($lock::getKey($job))->isLocked() : $cache->has($lock::getKey($job))); + } + + /** + * Provide the storage entry points for unique and debounce locks. + */ + public static function dispatchMethods(): array + { + return [ + 'push unique' => ['push', 'unique'], + 'later unique' => ['later', 'unique'], + 'bulk unique' => ['bulk', 'unique'], + 'push debounce' => ['push', 'debounce'], + 'later debounce' => ['later', 'debounce'], + 'bulk debounce' => ['bulk', 'debounce'], + ]; + } + + public function testChunkedBulkRegistersCleanupOnTheEnclosingSavepoint(): void + { + $queue = $this->queue(); + $database = $queue->getDatabase(); + $cache = new Repository(new WorkerArrayStore); + $jobs = array_map(static fn (int $id): object => (object) ['uniqueId' => $id], range(1, 3)); + $lock = new UniqueLock($cache); + + foreach ($jobs as $job) { + $this->assertTrue($lock->acquireForDispatch($job)); + } + + $database->beginTransaction(); + $database->beginTransaction(); + $queue->bulk($jobs); + $this->assertSame(2, $database->transactionLevel()); + $database->rollBack(); + + foreach ($jobs as $job) { + $this->assertFalse($cache->lock(UniqueLock::getKey($job))->isLocked()); + } + + $this->assertSame(0, $database->table('jobs')->count()); + $database->commit(); + } + + public function testDeadlockRetryCanReacquireTheRolledBackDispatchLock(): void + { + $queue = $this->queue(); + $cache = new Repository(new WorkerArrayStore); + $lock = new UniqueLock($cache); + $attempts = 0; + $job = new stdClass; + + $queue->getDatabase()->transaction(function () use ($queue, $lock, $job, &$attempts): void { + ++$attempts; + $this->assertTrue($lock->acquireForDispatch($job)); + $queue->push($job); + + if ($attempts === 1) { + throw new PDOException('deadlock detected', 40001); + } + }, 2); + + $this->assertSame(2, $attempts); + $this->assertSame(1, $queue->getDatabase()->table('jobs')->count()); + $this->assertTrue($cache->lock(UniqueLock::getKey($job))->isLocked()); + } + + public function testDurableInsertDoesNotRegisterCleanupOnAnotherTransaction(): void + { + $queue = $this->queue(); + $manager = $queue->getDatabase()->getTransactionManager(); + $other = new SQLiteConnection(new PDO('sqlite::memory:'), config: ['name' => 'other']); + $other->setTransactionManager($manager); + $other->beginTransaction(); + $cache = new Repository(new WorkerArrayStore); + $job = new stdClass; + $this->assertTrue((new UniqueLock($cache))->acquireForDispatch($job)); + + $queue->push($job); + $other->rollBack(); + + $this->assertSame(1, $queue->getDatabase()->table('jobs')->count()); + $this->assertTrue($cache->lock(UniqueLock::getKey($job))->isLocked()); + } + + /** + * Create a real queue with a small binding limit to exercise chunk savepoints. + */ + protected function queue(): DatabaseQueue + { + $container = new Container; + Container::setInstance($container); + $container->instance(EventDispatcher::class, new Dispatcher($container)); + $database = new RollbackQueueConnection(new PDO('sqlite::memory:'), config: ['name' => 'testing']); + $database->setReadWriteType('write'); + $database->setTransactionManager($manager = new DatabaseTransactionsManager); + $container->instance('db.transactions', $manager); + $database->getSchemaBuilder()->create('jobs', static function (Blueprint $table): void { + $table->id(); + $table->string('queue'); + $table->text('payload'); + $table->unsignedInteger('attempts'); + $table->unsignedBigInteger('reserved_at')->nullable(); + $table->unsignedBigInteger('available_at'); + $table->unsignedBigInteger('created_at'); + }); + $resolver = m::mock(ConnectionResolverInterface::class); + $resolver->shouldReceive('connection')->with('testing::write')->andReturn($database); + $queue = new DatabaseQueue($resolver, 'testing::write', 'jobs'); + $queue->setContainer($container); + $queue->setConnectionName('database'); + + return $queue; + } +} + +class RollbackQueueConnection extends SQLiteConnection +{ + /** + * Force a batch of three jobs through the binding-safe chunk path. + */ + protected function resolveMaxBindings(): int + { + return 12; + } +} diff --git a/tests/Queue/FailoverQueueTest.php b/tests/Queue/FailoverQueueTest.php index e81144a5c..0e187ce2b 100644 --- a/tests/Queue/FailoverQueueTest.php +++ b/tests/Queue/FailoverQueueTest.php @@ -23,6 +23,7 @@ use Hypervel\Contracts\Queue\ShouldQueueAfterCommit; use Hypervel\Database\DatabaseTransactionRecord; use Hypervel\Database\DatabaseTransactionsManager; +use Hypervel\Events\Dispatcher as EventDispatcher; use Hypervel\Events\QueuedClosure; use Hypervel\Queue\Attributes\Delay; use Hypervel\Queue\Events\QueueFailedOver; @@ -314,6 +315,58 @@ public function testPushReentersDeferralForATransactionOpenedByAnEarlierCommitCa $this->assertSame(['job'], $connection->pushedJobs); } + public function testInlineAfterCommitChildRetainsLockOwnershipForFallback(): void + { + $container = Container::getInstance(); + $container->instance('db.transactions', new DatabaseTransactionsManager); + $events = new EventDispatcher($container); + $container->instance(DispatcherContract::class, $events); + $container->instance('events', $events); + $failure = new RuntimeException('Scheduling failed before acceptance.'); + $first = new class($failure) extends SyncQueue { + /** + * Configure an after-commit child whose execution cannot start. + */ + public function __construct(private RuntimeException $failure) + { + parent::__construct(true); + } + + /** + * Fail before the queue can accept the dispatch. + */ + protected function executePayload(string $payload, ?string $queue = null): int + { + throw $this->failure; + } + }; + $first->setContainer($container)->setConnectionName('first'); + $second = m::mock(Queue::class); + $job = new FailoverQueueUniqueDebouncedJob; + $lock = new UniqueLock(new CacheRepository(new ArrayStore)); + $this->assertTrue($lock->acquireForDispatch($job)); + $second->expects('push')->with($job)->andReturnUsing(function ($job) use ($lock): string { + $this->assertTrue(DispatchLockContext::has($job)); + $this->assertFalse($lock->acquire($job)); + DispatchLockContext::accept($job); + + return 'accepted'; + }); + $events->listen(QueueFailedOver::class, function (QueueFailedOver $event) use ($failure): void { + $this->assertSame($failure, $event->exception); + }); + $failover = new FailoverQueue( + new FailoverQueueFakeManager(['first' => $first, 'second' => $second]), + $events, + ['first', 'second'], + ); + $failover->setContainer($container); + + $this->assertSame('accepted', $failover->push($job)); + $this->assertFalse(DispatchLockContext::has($job)); + $this->assertFalse($lock->acquire($job)); + } + public function testNestedCommitWaitsForTheRootTransaction(): void { $connection = new FailoverQueueSuccessfulConnection('sync'); diff --git a/tests/Queue/Outbox/DatabaseOutboxPublicationTest.php b/tests/Queue/Outbox/DatabaseOutboxPublicationTest.php new file mode 100644 index 000000000..ea569ab7b --- /dev/null +++ b/tests/Queue/Outbox/DatabaseOutboxPublicationTest.php @@ -0,0 +1,101 @@ +queue(); + $routes->forward(['orders' => 'processing', 'processing' => 'archive']); + $now = CarbonImmutable::now()->getTimestamp(); + $messages = [$this->message(1, 'orders', $now + 30), $this->message(2, 'reports', $now - 30)]; + $database->enableQueryLog(); + + $result = $queue->publishOutboxMessages($messages); + $inserts = array_filter($database->getQueryLog(), static fn (array $query): bool => str_starts_with($query['query'], 'insert into')); + $database->disableQueryLog(); + + $this->assertSame([1, 2], $result->accepted); + $this->assertNull($result->failure); + $this->assertCount(1, $inserts); + $rows = $database->table('jobs')->orderBy('id')->get()->all(); + $this->assertSame(['processing', 'reports'], array_column($rows, 'queue')); + $this->assertSame([$now + 30, $now], array_column($rows, 'available_at')); + $this->assertSame(array_column($messages, 'payload'), array_column($rows, 'payload')); + $this->assertSame(0, $database->transactionLevel()); + } + + public function testARejectedLaterChunkRollsBackEveryInsertedMessage(): void + { + [$queue, $database] = $this->queue(); + $database->unprepared("CREATE TRIGGER reject_job BEFORE INSERT ON jobs WHEN NEW.queue = 'rejected' BEGIN SELECT RAISE(ABORT, 'Queue rejected'); END"); + + $result = $queue->publishOutboxMessages([ + $this->message(1, 'orders'), $this->message(2, 'orders'), $this->message(3, 'rejected'), + ]); + + $this->assertSame([], $result->accepted); + $this->assertInstanceOf(QueryException::class, $result->failure); + $this->assertSame(0, $database->table('jobs')->count()); + $this->assertSame(0, $database->transactionLevel()); + } + + /** + * Create a real target jobs table with a two-row binding limit. + */ + protected function queue(): array + { + $container = new Container; + Container::setInstance($container); + $database = new OutboxPublicationConnection(new PDO('sqlite::memory:'), config: ['name' => 'target']); + $container->instance('db.schema', $database->getSchemaBuilder()); + $container->instance('queue.routes', $routes = new QueueRoutes); + Facade::setFacadeApplication($container); + (require dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000006_testbench_create_jobs_table.php')->up(); + $resolver = m::mock(ConnectionResolverInterface::class); + $resolver->shouldReceive('connection')->with('target')->andReturn($database); + $queue = new DatabaseQueue($resolver, 'target', 'jobs'); + $queue->setContainer($container)->setConnectionName('database'); + + return [$queue, $database, $routes]; + } + + /** + * Build a finalized payload without recreating its serialized command. + */ + protected function message(int $id, string $queue, ?int $deliverAt = null): OutboxMessage + { + $uuid = (string) Str::uuid(); + + return new OutboxMessage($id, $uuid, (string) Str::uuid(), 1, 'database', $queue, '{ "uuid": "' . $uuid . '" }', [], $deliverAt, 1, 1); + } +} + +class OutboxPublicationConnection extends SQLiteConnection +{ + /** + * Exercise binding-safe chunking with a small batch. + */ + protected function resolveMaxBindings(): int + { + return 12; + } +} diff --git a/tests/Queue/Outbox/DatabaseOutboxRepositoryTest.php b/tests/Queue/Outbox/DatabaseOutboxRepositoryTest.php new file mode 100644 index 000000000..43eb7b367 --- /dev/null +++ b/tests/Queue/Outbox/DatabaseOutboxRepositoryTest.php @@ -0,0 +1,227 @@ + true, + '--path' => dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php', + ]; + } + + public function testClaimPreservesPayloadAndOriginalEligibilityWithThreeStatements(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $now = Date::now()->getTimestamp(); + $repository = $this->repository(); + $connection = DB::connection(); + $first = $this->row($now - 10); + $second = $this->row($now - 30); + $future = $this->row($now + 60); + $repository->insert($connection, [$first, $second, $future]); + + $connection->maxBindings(); + $connection->lockForPopping(); + $transactionsStarted = 0; + $this->app->make('events')->listen(TransactionBeginning::class, static function () use (&$transactionsStarted): void { + ++$transactionsStarted; + }); + $connection->enableQueryLog(); + $connection->flushQueryLog(); + $messages = $repository->claim($connection->getName(), 2); + $queries = $connection->getQueryLog(); + $connection->disableQueryLog(); + + $this->assertCount(3, $queries); + $this->assertSame(0, $transactionsStarted); + $this->assertSame([$second['uuid'], $first['uuid']], array_column($messages, 'uuid')); + $this->assertSame([$now - 30, $now - 10], array_column($messages, 'eligibleAt')); + $this->assertSame($second['payload'], $messages[0]->payload); + $this->assertSame(['transport' => ['custom' => ['group' => 'orders']]], $messages[0]->options); + $this->assertSame(1, $messages[0]->attempts); + $this->assertSame($now, $messages[0]->createdAt); + $this->assertSame($messages[0]->leaseToken, $messages[1]->leaseToken); + $this->assertSame($now + 300, (int) DB::table('job_outbox')->where('uuid', $first['uuid'])->value('available_at')); + $this->assertSame([], $repository->claim($connection->getName(), 2)); + } + + public function testExpiredClaimCanBeRecoveredWithoutChangingDeliveryOrPayload(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $repository = $this->repository(); + $row = $this->row(Date::now()->getTimestamp()); + $row['deliver_at'] = Date::now()->getTimestamp() + 900; + $repository->insert(DB::connection(), [$row]); + $first = $repository->claim(DB::connection()->getName(), 10)[0]; + + Date::setTestNow(Date::now()->addSeconds(300)); + $second = $repository->claim(DB::connection()->getName(), 10)[0]; + + $this->assertSame($first->id, $second->id); + $this->assertNotSame($first->leaseToken, $second->leaseToken); + $this->assertSame(2, $second->attempts); + $this->assertSame($first->deliverAt, $second->deliverAt); + $this->assertSame($first->payload, $second->payload); + } + + public function testSettlementDeletesAcceptedDefersFutureAndBacksOffFailures(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $now = Date::now()->getTimestamp(); + $repository = $this->repository(); + $database = DB::connection()->getName(); + $accepted = $this->row($now); + $deferred = $this->row($now); + $deferred['deliver_at'] = $now + 2000; + $deferred['last_error'] = 'Previous failure'; + $failed = $this->row($now); + $failed['attempts'] = 6; + $repository->insert(DB::connection(), [$accepted, $deferred, $failed]); + $messages = $repository->claim($database, 10); + + $affected = $repository->settle($database, $messages, new OutboxBatchResult( + [$messages[0]->id], + [$messages[1]->id], + new RuntimeException(str_repeat('雪', 5000)), + )); + + $this->assertSame(3, $affected); + $this->assertNull($repository->find($database, $accepted['uuid'])); + $storedDeferred = $repository->find($database, $deferred['uuid']); + $this->assertNull($storedDeferred->lease_token); + $this->assertSame($now + 2000, (int) $storedDeferred->available_at); + $this->assertSame(0, (int) $storedDeferred->attempts); + $this->assertSame('Previous failure', $storedDeferred->last_error); + $storedFailed = $repository->find($database, $failed['uuid']); + $this->assertNull($storedFailed->lease_token); + $this->assertSame(7, (int) $storedFailed->attempts); + $this->assertGreaterThanOrEqual($now + 300, (int) $storedFailed->available_at); + $this->assertLessThanOrEqual($now + 600, (int) $storedFailed->available_at); + $this->assertSame(4096, mb_strlen($storedFailed->last_error)); + } + + public function testStaleSettlementCannotDeleteOrReleaseANewerClaim(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $repository = $this->repository(); + $database = DB::connection()->getName(); + $repository->insert(DB::connection(), [$this->row(Date::now()->getTimestamp()), $this->row(Date::now()->getTimestamp())]); + $old = $repository->claim($database, 10); + Date::setTestNow(Date::now()->addSeconds(300)); + $current = $repository->claim($database, 10); + + $this->assertSame(0, $repository->settle($database, $old, new OutboxBatchResult( + [$old[0]->id], + failure: new RuntimeException('Old attempt failed'), + ))); + $this->assertSame($current[0]->leaseToken, $repository->find($database, $current[0]->uuid)->lease_token); + $this->assertSame($current[1]->leaseToken, $repository->find($database, $current[1]->uuid)->lease_token); + $this->assertSame(2, $repository->settle($database, $current, new OutboxBatchResult(array_column($current, 'id')))); + } + + public function testOperatorQueriesDistinguishScheduledFailedAndLeasedRows(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $now = Date::now()->getTimestamp(); + $repository = $this->repository(); + $connection = DB::connection(); + $database = $connection->getName(); + $this->assertSame([ + 'total' => 0, 'eligible' => 0, 'delayed' => 0, 'leased' => 0, + 'retrying' => 0, 'oldest_eligible_age' => null, + ], $repository->statistics($database)); + $eligible = $this->row($now - 20); + $delayed = $this->row($now + 1000); + $leased = $this->row($now + 300); + $leased['lease_token'] = (string) Str::uuid(); + $leased['attempts'] = 1; + $failed = $this->row($now + 600); + $failed['attempts'] = 7; + $failed['last_error'] = 'Queue unavailable'; + $repository->insert(DB::connection(), [$eligible, $delayed, $leased, $failed]); + + $connection->enableQueryLog(); + $connection->flushQueryLog(); + $statistics = $repository->statistics($database); + $queries = $connection->getQueryLog(); + $connection->disableQueryLog(); + + $this->assertCount(1, $queries); + $this->assertSame([ + 'total' => 4, 'eligible' => 1, 'delayed' => 1, 'leased' => 1, + 'retrying' => 1, 'oldest_eligible_age' => 20, + ], $statistics); + $this->assertCount(2, $repository->pending($database, 2)); + $this->assertSame(0, $repository->retry($database, $delayed['uuid'])); + $this->assertSame(0, $repository->retry($database, $leased['uuid'])); + $this->assertSame(1, $repository->retryAll($database)); + $retried = $repository->find($database, $failed['uuid']); + $this->assertSame($now, (int) $retried->available_at); + $this->assertNull($retried->last_error); + $this->assertSame(7, (int) $retried->attempts); + $leasedRow = $repository->find($database, $leased['uuid']); + $this->assertFalse($repository->forget($database, $leasedRow->id, $leasedRow->lease_token)); + $this->assertFalse($repository->forget($database, $retried->id, (string) Str::uuid())); + $this->assertTrue($repository->forget($database, $retried->id, null)); + $this->assertNull($repository->find($database, $failed['uuid'])); + } + + public function testRepositoryRejectsUnconfiguredSourceBeforeQuerying(): void + { + $this->expectException(InvalidArgumentException::class); + $this->expectExceptionMessage('not configured in queue.outbox.databases'); + + $this->repository()->claim('not-configured', 100); + } + + /** + * Create a repository using the test's configured database. + */ + protected function repository(): DatabaseOutboxRepository + { + return new DatabaseOutboxRepository($this->app->make('db'), [DB::connection()->getName()], 'job_outbox', 300); + } + + /** + * Build a row with opaque payload bytes and transport metadata. + */ + protected function row(int $availableAt): array + { + return [ + 'uuid' => (string) Str::uuid(), + 'connection' => 'redis', + 'queue' => 'orders', + 'payload' => '{ "uuid": "opaque-payload", "data": {"text": "雪"} }', + 'options' => '{"transport":{"custom":{"group":"orders"}}}', + 'deliver_at' => null, + 'attempts' => 0, + 'available_at' => $availableAt, + 'lease_token' => null, + 'last_error' => null, + 'created_at' => Date::now()->getTimestamp(), + ]; + } +} diff --git a/tests/Queue/Outbox/OutboxDispatchTest.php b/tests/Queue/Outbox/OutboxDispatchTest.php new file mode 100644 index 000000000..2d8cad3db --- /dev/null +++ b/tests/Queue/Outbox/OutboxDispatchTest.php @@ -0,0 +1,342 @@ +make('config'); + $config->set('queue.outbox.databases', [$config->string('database.default')]); + } + + /** + * Load only the shipped outbox migration. + */ + protected function migrateFreshUsing(): array + { + return [ + '--realpath' => true, + '--path' => dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php', + ]; + } + + public function testSingleStagingPreservesFinalizedPayloadAndLifecycle(): void + { + $queue = $this->queue(); + $events = $this->app->make('events'); + $finalized = 0; + $observed = []; + $events->listen(JobPayloadFinalizing::class, function (JobPayloadFinalizing $event) use (&$finalized): void { + ++$finalized; + $event->payload = json_encode([...json_decode($event->payload, true), 'custom' => 'retained']); + }); + $events->listen(JobQueueing::class, function (JobQueueing $event) use (&$observed): void { + $observed[] = $event->payload; + }); + $events->listen(JobQueued::class, function (JobQueued $event) use (&$observed): void { + $observed[] = $event->payload; + }); + + $id = $queue->push((new OutboxDispatchJob)->viaOutbox()->beforeCommit()); + $row = DB::table('job_outbox')->sole(); + + $this->assertSame($row->uuid, $id); + $this->assertSame($id, json_decode($row->payload, true)['uuid']); + $this->assertSame([$row->payload, $row->payload], $observed); + $this->assertSame(1, $finalized); + $this->assertSame('redis', $row->connection); + $this->assertSame('orders', $row->queue); + $this->assertNull($row->options); + $this->assertSame([], $queue->published); + } + + public function testSelectionPrecedenceAndDirectDispatchAvoidOutboxResolution(): void + { + $queue = $this->queue(); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + $queue->push(new OutboxDispatchJob); + $queue->push((new MarkedOutboxDispatchJob)->withoutOutbox()); + $queue->setConfig(['outbox' => true]); + $queue->push((new OutboxDispatchJob)->withoutOutbox()); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + $this->assertCount(3, $queue->published); + + $queue->push(new OutboxDispatchJob); + $queue->setConfig([]); + $queue->push(new MarkedOutboxDispatchJob); + $queue->push((new OutboxDispatchJob)->viaOutbox()); + + $this->assertSame(3, DB::table('job_outbox')->count()); + $this->assertCount(3, $queue->published); + } + + public function testRawPublicationNeverStages(): void + { + $queue = $this->queue()->setConfig(['outbox' => true]); + $queue->pushRaw('{"uuid":"raw"}'); + + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + $this->assertSame(['{"uuid":"raw"}'], $queue->published); + } + + public function testBulkStagingUsesOneInsertAndFinalizesEachPayloadOnce(): void + { + $queue = $this->queue(); + $finalized = 0; + $this->app->make('events')->listen(JobPayloadFinalizing::class, function () use (&$finalized): void { + ++$finalized; + }); + $jobs = array_map(static fn (): OutboxDispatchJob => (new OutboxDispatchJob)->viaOutbox(), range(1, 100)); + $connection = DB::connection(); + $connection->enableQueryLog(); + $connection->flushQueryLog(); + + $this->assertNull($queue->bulk($jobs)); + $inserts = array_filter($connection->getQueryLog(), static fn (array $query): bool => str_starts_with($query['query'], 'insert into')); + $connection->disableQueryLog(); + + $this->assertCount(1, $inserts); + $this->assertSame(100, $finalized); + $this->assertSame(100, DB::table('job_outbox')->count()); + $this->assertSame([], $queue->published); + } + + public function testBulkPreparationFailureStoresNothingAndTerminatesEveryMember(): void + { + $queue = $this->queue(); + $failure = new RuntimeException('Finalization failed'); + $finalized = 0; + $failed = []; + $this->app->make('events')->listen(JobPayloadFinalizing::class, function () use (&$finalized, $failure): void { + if (++$finalized === 2) { + throw $failure; + } + }); + $this->app->make('events')->listen(JobQueueingFailed::class, function (JobQueueingFailed $event) use (&$failed): void { + $failed[] = $event->job; + }); + $jobs = [(new OutboxDispatchJob)->viaOutbox(), (new OutboxDispatchJob)->viaOutbox(), (new OutboxDispatchJob)->viaOutbox()]; + + try { + $queue->bulk($jobs); + $this->fail('Expected finalization to fail.'); + } catch (RuntimeException $exception) { + $this->assertSame($failure, $exception); + } + + $this->assertSame($jobs, $failed); + $this->assertSame(0, DB::table('job_outbox')->count()); + } + + public function testAcceptedBulkIsNotFailedWhenItsQueuedListenerThrows(): void + { + $queue = $this->queue(); + $failure = new RuntimeException('Observer failed'); + $notified = 0; + $failed = 0; + $this->app->make('events')->listen(JobQueued::class, function () use (&$notified, $failure): void { + ++$notified; + throw $failure; + }); + $this->app->make('events')->listen(JobQueueingFailed::class, function () use (&$failed): void { + ++$failed; + }); + + try { + $queue->bulk([(new OutboxDispatchJob)->viaOutbox(), (new OutboxDispatchJob)->viaOutbox()]); + $this->fail('Expected the observer error.'); + } catch (RuntimeException $exception) { + $this->assertSame($failure, $exception); + } + + $this->assertSame(2, $notified); + $this->assertSame(0, $failed); + $this->assertSame(2, DB::table('job_outbox')->count()); + } + + #[DataProvider('deliveryTimes')] + public function testStagingRetainsRequestedTimersWithoutPublishingBeforeTheyAreDue(string $kind, ?int $offset): void + { + Date::setTestNow('2040-01-01 12:00:00.500000'); + $queue = $this->queue(); + $delay = match ($kind) { + 'none' => null, + 'zero' => 0, + 'past' => Date::now()->subSecond(), + 'future' => Date::now()->addMilliseconds(250), + }; + $job = (new OutboxDispatchJob)->viaOutbox(); + $delay === null ? $queue->push($job) : $queue->later($delay, $job); + $row = DB::table('job_outbox')->sole(); + + $this->assertSame($offset === null ? null : Date::now()->getTimestamp() + $offset, $row->deliver_at === null ? null : (int) $row->deliver_at); + $this->assertSame(Date::now()->getTimestamp(), (int) $row->available_at); + } + + /** + * Provide timer selection and absolute deadline normalization cases. + */ + public static function deliveryTimes(): array + { + return [['none', null], ['zero', null], ['past', 0], ['future', 1]]; + } + + #[DataProvider('sqsDelays')] + public function testSqsStagingCapturesOptionsOnceAndWaitsForItsTransportWindow(string $method, bool $fifo, int $eligibleDelay): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $queue = new SqsQueue(m::mock(SqsClient::class), $fifo ? 'orders.fifo' : 'orders'); + $queue->setContainer($this->app)->setConnectionName('sqs'); + $job = (new OutboxFifoDispatchJob)->viaOutbox()->onGroup('tenant:17')->delay(1200); + + $method === 'bulk' ? $queue->bulk([$job]) : $queue->later(1200, $job); + $row = DB::table('job_outbox')->sole(); + $options = json_decode($row->options, true)['transport']['sqs']; + + $this->assertSame(Date::now()->getTimestamp() + 1200, (int) $row->deliver_at); + $this->assertSame(Date::now()->getTimestamp() + $eligibleDelay, (int) $row->available_at); + $this->assertSame('tenant:17', $options['MessageGroupId']); + $this->assertArrayNotHasKey('DelaySeconds', $options); + $this->assertSame($fifo ? 1 : 0, $job->deduplications); + + if ($fifo) { + $this->assertSame('captured-' . $row->uuid, $options['MessageDeduplicationId']); + } + } + + /** + * Exercise both SQS entry points and their distinct eligibility rules. + */ + public static function sqsDelays(): array + { + return [['later', false, 300], ['bulk', false, 300], ['later', true, 1200], ['bulk', true, 1200]]; + } + + public function testInvalidFinalUuidFailsBeforeStorage(): void + { + $this->app->make('events')->listen(JobPayloadFinalizing::class, function (JobPayloadFinalizing $event): void { + $event->payload = '{"uuid":"not-a-uuid"}'; + }); + + try { + $this->queue()->push((new OutboxDispatchJob)->viaOutbox()); + $this->fail('Expected an invalid payload.'); + } catch (InvalidPayloadException $exception) { + $this->assertSame('The outbox payload must contain a valid job UUID.', $exception->getMessage()); + } + + $this->assertSame(0, DB::table('job_outbox')->count()); + } + + public function testDatabaseFastPathResolutionFailureEmitsTheFailedLifecycle(): void + { + $queue = new DatabaseQueue($this->app->make(ConnectionResolverInterface::class), null, 'jobs'); + $queue->setContainer($this->app)->setConnectionName('database'); + $job = (new OutboxDispatchJob)->onOutboxDatabase('missing-owner'); + $failed = []; + $this->app->make('events')->listen(JobQueueingFailed::class, function (JobQueueingFailed $event) use (&$failed): void { + $failed[] = $event; + }); + + DB::beginTransaction(); + + try { + $queue->push($job); + $this->fail('Expected the invalid database configuration to fail.'); + } catch (InvalidArgumentException $exception) { + $this->assertCount(1, $failed); + $this->assertSame($job, $failed[0]->job); + $this->assertSame($exception, $failed[0]->exception); + } finally { + DB::rollBack(); + } + + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + $this->assertSame(0, DB::table('job_outbox')->count()); + } + + /** + * Create a Redis queue that records direct publication without network access. + */ + protected function queue(): RecordingOutboxRedisQueue + { + $queue = new RecordingOutboxRedisQueue(m::mock(Factory::class), 'orders'); + $queue->setContainer($this->app)->setConnectionName('redis'); + + return $queue; + } +} + +class OutboxDispatchJob +{ + use Queueable; +} + +class MarkedOutboxDispatchJob extends OutboxDispatchJob implements ShouldQueueViaOutbox +{ +} + +class OutboxFifoDispatchJob extends OutboxDispatchJob +{ + public int $deduplications = 0; + + /** + * Capture the finalized UUID while the original job is available. + */ + public function deduplicationId(string $payload, string $queue): string + { + ++$this->deduplications; + + return 'captured-' . json_decode($payload, true)['uuid']; + } +} + +class RecordingOutboxRedisQueue extends RedisQueue +{ + public array $published = []; + + /** + * Record a direct raw publication. + */ + public function pushRaw(string $payload, UnitEnum|string|null $queue = null, array $options = []): mixed + { + $this->published[] = $payload; + + return json_decode($payload, true)['uuid']; + } +} diff --git a/tests/Queue/Outbox/OutboxDispatcherTest.php b/tests/Queue/Outbox/OutboxDispatcherTest.php new file mode 100644 index 000000000..4ca4112bd --- /dev/null +++ b/tests/Queue/Outbox/OutboxDispatcherTest.php @@ -0,0 +1,95 @@ + $base]); + $connection->setTransactionManager($transactions); + + if ($role !== null) { + $connection->setReadWriteType($role); + } + + $connections[$name] = $connection; + } + + $resolver = m::mock(ConnectionResolverInterface::class); + $resolver->shouldReceive('connection')->andReturnUsing(fn (string $name): SQLiteConnection => $connections[$name]); + $repository = new DatabaseOutboxRepository($resolver, $configured, 'job_outbox', 300); + $dispatcher = new OutboxDispatcher($repository, $resolver, $transactions); + $job = new OutboxOwnershipJob; + + if ($explicit !== null) { + $job->onOutboxDatabase($explicit); + } + + try { + foreach ($active as $name) { + $connections[$name]->beginTransaction(); + } + + if ($error !== null) { + $this->expectException(LogicException::class); + $this->expectExceptionMessage($error); + } + + $owner = $dispatcher->databaseFor($job); + + $this->assertSame($connections[$expected], $owner); + } finally { + foreach ($connections as $connection) { + if ($connection->transactionLevel() > 0) { + $connection->rollBack(0); + } + } + } + } + + /** + * Provide distinct owner sessions and realistic selection conflicts. + */ + public static function ownershipCases(): array + { + return [ + 'configured default without transaction' => [[], null, ['business'], 'business', null], + 'explicit configured source without transaction' => [[], 'secondary', ['business', 'secondary'], 'secondary', null], + 'single active source overrides default' => [['secondary'], null, ['business', 'secondary'], 'secondary', null], + 'nested transactions retain one owner' => [['business', 'business'], null, ['business'], 'business', null], + 'writer is covered by base source' => [['business::write'], null, ['business'], 'business::write', null], + 'explicit writer selects one of two sessions' => [['business', 'business::write'], 'business::write', ['business'], 'business::write', null], + 'distinct sessions require explicit owner' => [['business', 'business::write'], null, ['business'], null, 'onOutboxDatabase()'], + 'explicit owner must participate' => [['business'], 'secondary', ['business', 'secondary'], null, 'does not own an active transaction'], + 'active source must be drainable' => [['secondary'], null, ['business'], null, 'not covered by queue.outbox.databases'], + 'active read session rejected' => [['business::read'], null, ['business'], null, 'is a read connection'], + 'explicit read session rejected' => [[], 'business::read', ['business'], null, 'is a read connection'], + ]; + } +} + +class OutboxOwnershipJob +{ + use Queueable; +} diff --git a/tests/Queue/Outbox/OutboxFastPathTest.php b/tests/Queue/Outbox/OutboxFastPathTest.php new file mode 100644 index 000000000..45f871100 --- /dev/null +++ b/tests/Queue/Outbox/OutboxFastPathTest.php @@ -0,0 +1,357 @@ +make('config'); + $config->set('database.default', 'source'); + $config->set('database.connections.source', ['driver' => 'sqlite', 'database' => ':memory:']); + $config->set('database.connections.secondary', ['driver' => 'sqlite', 'database' => ':memory:']); + $config->set('queue.outbox.databases', ['source', 'secondary']); + $config->set('queue.connections.database', [ + 'driver' => 'database', 'connection' => 'source', 'table' => 'jobs', + 'queue' => 'default', 'after_commit' => true, 'outbox' => true, + ]); + } + + #[DataProvider('transactionOutcomes')] + public function testLocalSelectionHonorsCommitAndRollbackWithoutSchema(bool $commit): void + { + $log = $this->app->make(OutboxExecutionLog::class); + $queue = $this->app->make('queue')->connection('sync'); + DB::beginTransaction(); + $queue->push((new OutboxLocalJob)->viaOutbox()->beforeCommit()); + $this->assertSame(0, $log->executions); + + $commit ? DB::commit() : DB::rollBack(); + + $this->assertSame($commit ? 1 : 0, $log->executions); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + } + + public function testLocalExplicitOwnerDoesNotWaitForAnUnrelatedTransaction(): void + { + $log = $this->app->make(OutboxExecutionLog::class); + DB::beginTransaction(); + DB::connection('secondary')->beginTransaction(); + $this->app->make('queue')->connection('sync')->push((new OutboxLocalJob)->onOutboxDatabase('secondary')); + + DB::connection('secondary')->commit(); + $this->assertSame(1, $log->executions); + DB::rollBack(); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + } + + public function testLocalExplicitOwnerRejectsAnUnrelatedActiveTransaction(): void + { + DB::beginTransaction(); + + try { + $this->expectException(LogicException::class); + $this->expectExceptionMessage('does not own an active transaction'); + $this->app->make('queue')->connection('sync')->push((new OutboxLocalJob)->onOutboxDatabase('secondary')); + } finally { + DB::rollBack(); + } + } + + public function testNullBulkAndQueueFakeDoNotResolveOutboxStorage(): void + { + $jobs = [new MarkedOutboxLocalJob, (new OutboxLocalJob)->viaOutbox()]; + $this->app->make('queue')->connection('null')->bulk($jobs); + $fake = new QueueFake($this->app); + $fake->bulk($jobs); + + $fake->assertPushed(OutboxLocalJob::class); + $fake->assertPushed(MarkedOutboxLocalJob::class); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + $this->assertSame(0, $this->app->make(OutboxExecutionLog::class)->executions); + } + + public function testDatabaseQueueUsesItsExistingAtomicStorageAndIgnoresAfterCommitDefault(): void + { + (require dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000006_testbench_create_jobs_table.php')->up(); + $queue = $this->app->make('queue')->connection('database'); + $id = $queue->push(new OutboxLocalJob); + $this->assertIsInt($id); + $this->assertSame(1, DB::table('jobs')->count()); + + DB::beginTransaction(); + $queue->bulk([new OutboxLocalJob, (new OutboxLocalJob)->onOutboxDatabase('source::write')]); + $this->assertSame(3, DB::table('jobs')->count()); + DB::rollBack(); + + $this->assertSame(1, DB::table('jobs')->count()); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + } + + public function testDatabaseQueueOnAnotherOwnerStagesInsideTheBusinessTransaction(): void + { + (require dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php')->up(); + config(['queue.connections.database.connection' => 'secondary']); + $queue = $this->app->make('queue')->connection('database'); + + DB::beginTransaction(); + $id = $queue->push(new OutboxLocalJob); + $this->assertSame($id, DB::table('job_outbox')->sole()->uuid); + DB::rollBack(); + $this->assertSame(0, DB::table('job_outbox')->count()); + } + + public function testFailoverSuppressesChildDefaultsAndRestoresThemAfterNestedFailure(): void + { + $primary = new FailingOutboxRedisQueue(m::mock(Factory::class), 'orders'); + $primary->setContainer($this->app)->setConnectionName('primary')->setConfig(['outbox' => true]); + $sync = $this->app->make('queue')->connection('sync'); + $manager = m::mock(QueueManager::class); + $manager->shouldReceive('connection')->with('primary')->andReturn($primary); + $manager->shouldReceive('connection')->with('sync')->andReturn($sync); + $inner = new FailoverQueue($manager, $this->app->make('events'), ['primary']); + $inner->setContainer($this->app)->setConnectionName('inner'); + $manager->shouldReceive('connection')->with('inner')->andReturn($inner); + $outer = new FailoverQueue($manager, $this->app->make('events'), ['inner', 'sync']); + $outer->setContainer($this->app)->setConnectionName('outer'); + + $outer->push(new OutboxLocalJob); + + $this->assertSame(1, $primary->attempts); + $this->assertSame(1, $this->app->make(OutboxExecutionLog::class)->executions); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + + (require dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php')->up(); + $primary->push(new OutboxLocalJob); + $this->assertSame(1, DB::table('job_outbox')->count()); + $this->assertSame(1, $primary->attempts); + } + + #[DataProvider('transactionOutcomes')] + public function testSelectedFailoverDefersTheWholeAttemptWithoutSchema(bool $commit): void + { + $manager = m::mock(QueueManager::class); + $manager->shouldReceive('connection')->with('sync')->andReturn($this->app->make('queue')->connection('sync')); + $queue = new FailoverQueue($manager, $this->app->make('events'), ['sync']); + $queue->setContainer($this->app)->setConnectionName('failover')->setConfig(['outbox' => true]); + $log = $this->app->make(OutboxExecutionLog::class); + DB::beginTransaction(); + $queue->push(new OutboxLocalJob); + $this->assertSame(0, $log->executions); + $commit ? DB::commit() : DB::rollBack(); + $this->assertSame($commit ? 1 : 0, $log->executions); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + } + + /** + * Provide the two terminal transaction outcomes. + */ + public static function transactionOutcomes(): array + { + return ['commit' => [true], 'rollback' => [false]]; + } + + public function testNestedFailoverPublishesAtItsExplicitOwnersCommitAndPreservesLocksAcrossFailure(): void + { + $failure = new RuntimeException('Execution could not start.'); + $first = new class($failure) extends SyncQueue { + /** + * Configure a failing after-commit child. + */ + public function __construct(private RuntimeException $failure) + { + parent::__construct(true); + } + + /** + * Fail before accepting the dispatch. + */ + protected function executePayload(string $payload, ?string $queue = null): int + { + throw $this->failure; + } + }; + $first->setContainer($this->app)->setConnectionName('first'); + $target = new FastPathRecordingRedisQueue(m::mock(Factory::class), dispatchAfterCommit: true); + $target->setContainer($this->app)->setConnectionName('target')->setConfig(['outbox' => true]); + $manager = m::mock(QueueManager::class); + $manager->shouldReceive('connection')->with('first')->andReturn($first); + $manager->shouldReceive('connection')->with('target')->andReturn($target); + $inner = new FailoverQueue($manager, $this->app->make('events'), ['first', 'target'], true); + $inner->setContainer($this->app)->setConnectionName('inner'); + $manager->shouldReceive('connection')->with('inner')->andReturn($inner); + $outer = new FailoverQueue($manager, $this->app->make('events'), ['inner']); + $outer->setContainer($this->app)->setConnectionName('outer'); + $job = (new UniqueOutboxLocalJob)->onOutboxDatabase('secondary')->afterCommit(); + $lock = new UniqueLock(new Repository(new ArrayStore)); + $this->assertTrue($lock->acquireForDispatch($job)); + $this->app->make('events')->listen(JobQueueing::class, function (JobQueueing $event) use ($job, $lock): void { + $this->assertSame($job, $event->job); + $this->assertTrue(DispatchLockContext::has($job)); + $this->assertFalse($lock->acquire($job)); + }); + + DB::beginTransaction(); + DB::connection('secondary')->beginTransaction(); + $this->assertNull($outer->push($job)); + $this->assertSame(0, $target->publications); + DB::connection('secondary')->commit(); + + $this->assertSame(1, $target->publications); + $this->assertFalse(DispatchLockContext::has($job)); + DB::rollBack(); + $this->assertFalse($lock->acquire($job)); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + } + + #[DataProvider('continuationKinds')] + public function testSyncFallbackDoesNotSuppressIndependentContinuation(string $kind): void + { + (require dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php')->up(); + $target = new FastPathRecordingRedisQueue(m::mock(Factory::class)); + $target->setContainer($this->app)->setConnectionName('target')->setConfig(['outbox' => true]); + $this->app->instance(RedisQueue::class, $target); + $manager = m::mock(QueueManager::class); + $manager->shouldReceive('connection')->with('sync')->andReturn($this->app->make('queue')->connection('sync')); + $failover = new FailoverQueue($manager, $this->app->make('events'), ['sync']); + $failover->setContainer($this->app)->setConnectionName('failover'); + + $failover->push( + $kind === 'object' ? new OutboxDispatchesContinuation : OutboxStringContinuation::class . '@fire', + ['chunk' => 1], + ); + + $row = DB::table('job_outbox')->sole(); + $this->assertSame('target', $row->connection); + $this->assertSame(0, $target->publications); + + if ($kind === 'string') { + $payload = json_decode($row->payload, true); + $this->assertSame(OutboxStringContinuation::class . '@fire', $payload['job']); + $this->assertSame(['chunk' => 2], $payload['data']); + } + } + + /** + * Provide supported object and string continuation dispatches. + */ + public static function continuationKinds(): array + { + return ['object' => ['object'], 'string' => ['string']]; + } +} + +class OutboxLocalJob +{ + use Queueable; + + /** + * Record execution through the real synchronous job handler. + */ + public function handle(OutboxExecutionLog $log): void + { + ++$log->executions; + } +} + +class MarkedOutboxLocalJob extends OutboxLocalJob implements ShouldQueueViaOutbox +{ +} + +class UniqueOutboxLocalJob extends OutboxLocalJob implements ShouldBeUnique +{ +} + +class OutboxDispatchesContinuation +{ + /** + * Dispatch an independently selected job from the running handler. + */ + public function handle(RedisQueue $queue): void + { + $queue->push((new OutboxLocalJob)->viaOutbox()); + } +} + +class OutboxStringContinuation +{ + /** + * Resolve the destination for subsequent chunks. + */ + public function __construct(private RedisQueue $queue) + { + } + + /** + * Queue the next chunk using the same string handler. + */ + public function fire(Job $job, array $data): void + { + $this->queue->push(self::class . '@fire', ['chunk' => $data['chunk'] + 1]); + } +} + +class FastPathRecordingRedisQueue extends RedisQueue +{ + public int $publications = 0; + + /** + * Record direct transport acceptance without an external Redis service. + */ + public function pushRaw(string $payload, UnitEnum|string|null $queue = null, array $options = []): mixed + { + ++$this->publications; + + return 'accepted'; + } +} + +class OutboxExecutionLog +{ + public int $executions = 0; +} + +class FailingOutboxRedisQueue extends RedisQueue +{ + public int $attempts = 0; + + /** + * Reject direct publication to exercise the ordinary failover path. + */ + public function pushRaw(string $payload, UnitEnum|string|null $queue = null, array $options = []): mixed + { + ++$this->attempts; + + throw new RuntimeException('Broker unavailable.'); + } +} diff --git a/tests/Queue/Outbox/OutboxTransactionTest.php b/tests/Queue/Outbox/OutboxTransactionTest.php new file mode 100644 index 000000000..9007304ee --- /dev/null +++ b/tests/Queue/Outbox/OutboxTransactionTest.php @@ -0,0 +1,180 @@ +queue(); + $firstUuid = null; + $finalized = 0; + $events->listen(JobPayloadFinalizing::class, function (JobPayloadFinalizing $event) use (&$firstUuid, &$finalized): void { + $payload = json_decode($event->payload, true); + $firstUuid ??= $payload['uuid']; + + if (++$finalized === 3) { + $payload['uuid'] = $firstUuid; + $event->payload = json_encode($payload); + } + }); + + for ($level = 0; $level < $depth; ++$level) { + $database->beginTransaction(); + } + + try { + $queue->bulk([ + (new OutboxTransactionJob)->viaOutbox(), + (new OutboxTransactionJob)->viaOutbox(), + (new OutboxTransactionJob)->viaOutbox(), + ]); + $this->fail('The duplicate UUID must reject the complete owner group.'); + } catch (QueryException $exception) { + $this->assertStringContainsString('UNIQUE constraint failed', $exception->getMessage()); + $this->assertSame($depth, $database->transactionLevel()); + $this->assertSame(0, $database->table('job_outbox')->count()); + } + + while ($database->transactionLevel() > 0) { + $database->commit(); + } + + $this->assertSame(0, $database->table('job_outbox')->count()); + } + + /** + * Exercise both a standalone insert transaction and nested application savepoints. + */ + public static function transactionDepths(): array + { + return [[0], [1], [2]]; + } + + public function testNestedRollbackReleasesAcceptedLocksButPreservesTheOuterGroup(): void + { + [$queue, $database] = $this->queue(); + $cache = new Repository(new WorkerArrayStore); + $lock = new UniqueLock($cache); + $outer = (new OutboxTransactionJob('outer'))->viaOutbox(); + $inner = (new OutboxTransactionJob('inner'))->viaOutbox(); + + $database->beginTransaction(); + $this->assertTrue($lock->acquireForDispatch($outer)); + $queue->push($outer); + $database->beginTransaction(); + $this->assertTrue($lock->acquireForDispatch($inner)); + $queue->bulk([$inner]); + + $this->assertFalse(DispatchLockContext::has($outer)); + $this->assertFalse(DispatchLockContext::has($inner)); + $database->rollBack(); + $this->assertSame(1, $database->table('job_outbox')->count()); + $this->assertFalse($cache->lock(UniqueLock::getKey($inner))->isLocked()); + $this->assertTrue($cache->lock(UniqueLock::getKey($outer))->isLocked()); + + $database->commit(); + $this->assertSame(1, $database->table('job_outbox')->count()); + $this->assertTrue($cache->lock(UniqueLock::getKey($outer))->isLocked()); + } + + public function testOuterRollbackReleasesEveryAcceptedChunkSnapshot(): void + { + [$queue, $database] = $this->queue(); + $cache = new Repository(new WorkerArrayStore); + $lock = new UniqueLock($cache); + $jobs = array_map(static fn (int $id): OutboxTransactionJob => (new OutboxTransactionJob((string) $id))->viaOutbox(), range(1, 3)); + + $database->beginTransaction(); + + foreach ($jobs as $job) { + $this->assertTrue($lock->acquireForDispatch($job)); + } + + $queue->bulk($jobs); + $this->assertSame(3, $database->table('job_outbox')->count()); + $database->rollBack(); + $this->assertSame(0, $database->table('job_outbox')->count()); + + foreach ($jobs as $job) { + $this->assertFalse($cache->lock(UniqueLock::getKey($job))->isLocked()); + } + } + + /** + * Create an independent SQLite owner using the shipped outbox schema. + * + * @return array{RedisQueue, OutboxTransactionConnection, Dispatcher} + */ + protected function queue(): array + { + $container = new Container; + Container::setInstance($container); + $events = new Dispatcher($container); + $container->instance('events', $events); + $container->instance(EventDispatcher::class, $events); + $database = new OutboxTransactionConnection(new PDO('sqlite::memory:'), config: ['name' => 'business']); + $database->setTransactionManager($transactions = new DatabaseTransactionsManager); + $container->instance('db.schema', $database->getSchemaBuilder()); + Facade::setFacadeApplication($container); + (require dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php')->up(); + + $resolver = m::mock(ConnectionResolverInterface::class); + $resolver->shouldReceive('connection')->with('business')->andReturn($database); + $repository = new DatabaseOutboxRepository($resolver, ['business'], 'job_outbox', 300); + $container->instance(OutboxDispatcher::class, new OutboxDispatcher($repository, $resolver, $transactions)); + $queue = new RedisQueue(m::mock(Factory::class), 'orders'); + $queue->setContainer($container)->setConnectionName('redis'); + + return [$queue, $database, $events]; + } +} + +class OutboxTransactionJob +{ + use Queueable; + + /** + * Identify a distinct dispatch-owned lock. + */ + public function __construct(public string $uniqueId = '') + { + } +} + +class OutboxTransactionConnection extends SQLiteConnection +{ + /** + * Force three outbox rows through two binding-safe insert chunks. + */ + protected function resolveMaxBindings(): int + { + return 26; + } +} diff --git a/tests/Queue/Outbox/PooledOutboxPublicationTest.php b/tests/Queue/Outbox/PooledOutboxPublicationTest.php new file mode 100644 index 000000000..fc6ee1909 --- /dev/null +++ b/tests/Queue/Outbox/PooledOutboxPublicationTest.php @@ -0,0 +1,128 @@ +makePartial(); + $messages = [$this->message(1), $this->message(2)]; + $expected = new OutboxBatchResult([1], failure: new RuntimeException('Partial rejection.')); + $queue->expects('publishOutboxMessages')->with($messages)->andReturn($expected); + $released = 0; + $pools = new PoolManager; + $proxy = $this->proxy($queue, $pools, $released); + + try { + $this->assertSame($expected, $proxy->publishOutboxMessages($messages)); + $this->assertSame(1, $released); + $this->assertSame(0, $pools->get($proxy->getPoolName())->getBorrowedCount()); + } finally { + $pools->purgeAll(); + } + } + + public function testSequentialFallbackRetainsConfirmedPrefixAndDefersFutureRows(): void + { + CarbonImmutable::setTestNow('2040-01-01 12:00:00'); + $queue = m::mock(SyncQueue::class)->makePartial(); + $messages = [$this->message(1), $this->message(2, 60), $this->message(3), $this->message(4), $this->message(5, 120)]; + $failure = new RuntimeException('Publication outcome unknown.'); + $queue->expects('pushRaw')->with($messages[0]->payload, 'orders', $messages[0]->options)->andReturn('accepted'); + $queue->expects('pushRaw')->with($messages[2]->payload, 'orders', $messages[2]->options)->andThrow($failure); + $released = 0; + $pools = new PoolManager; + $proxy = $this->proxy($queue, $pools, $released); + + try { + $result = $proxy->publishOutboxMessages($messages); + + $this->assertSame([1], $result->accepted); + $this->assertSame([2, 5], $result->deferred); + $this->assertSame($failure, $result->failure); + $this->assertSame(1, $released); + $this->assertSame(0, $pools->get($proxy->getPoolName())->getBorrowedCount()); + } finally { + $pools->purgeAll(); + } + } + + public function testCancellationReleasesTheBorrowedTransportAndRemainsCancellation(): void + { + $queue = m::mock(SyncQueue::class)->makePartial(); + $cancellation = new CanceledException; + $queue->expects('pushRaw')->andThrow($cancellation); + $released = 0; + $pools = new PoolManager; + $proxy = $this->proxy($queue, $pools, $released); + + try { + try { + $proxy->publishOutboxMessages([$this->message(1)]); + $this->fail('Expected publication to remain cancelled.'); + } catch (CanceledException $exception) { + $this->assertSame($cancellation, $exception); + } + + $this->assertSame(1, $released); + $this->assertSame(0, $pools->get($proxy->getPoolName())->getBorrowedCount()); + } finally { + $pools->purgeAll(); + } + } + + /** + * Create a single-slot proxy with observable release and no external transport. + */ + protected function proxy(SyncQueue $queue, PoolManager $pools, int &$released): QueuePoolProxy + { + return new QueuePoolProxy( + new PoolDefinition('outbox-publication', 'queue', 'outbox-test', PoolOptions::fromArray(['max_objects' => 1])), + static fn (): SyncQueue => $queue, + $pools, + static function () use (&$released): void { + ++$released; + }, + ); + } + + /** + * Build an opaque message passed through the generic raw contract. + */ + protected function message(int $id, ?int $delay = null): OutboxMessage + { + $uuid = (string) Str::uuid(); + + return new OutboxMessage( + $id, + $uuid, + (string) Str::uuid(), + 1, + 'custom', + 'orders', + json_encode(['uuid' => $uuid]), + ['transport' => ['custom' => ['key' => 'value']]], + $delay === null ? null : CarbonImmutable::now()->getTimestamp() + $delay, + 1, + 1, + ); + } +} diff --git a/tests/Queue/Outbox/RedisOutboxPublicationTest.php b/tests/Queue/Outbox/RedisOutboxPublicationTest.php new file mode 100644 index 000000000..0ad0b723e --- /dev/null +++ b/tests/Queue/Outbox/RedisOutboxPublicationTest.php @@ -0,0 +1,158 @@ +queue(cluster: $cluster); + $routes->forward(['orders' => 'processing', 'alias' => 'processing', 'processing' => 'archive']); + $events->listen(JobQueued::class, fn () => $this->fail('Relay must not repeat the producer lifecycle.')); + $messages = [$this->message(1, 'orders'), $this->message(2, 'alias', CarbonImmutable::now()->getTimestamp() + 30)]; + $key = $cluster ? 'queues:{processing}' : 'queues:processing'; + $connection->expects('evalWithShaCache')->with( + LuaScripts::bulk(), + [$key, $key . ':notify', $key . ':delayed'], + ['i', $messages[0]->payload, $messages[1]->deliverAt, $messages[1]->payload], + )->andReturn(2); + + $result = $queue->publishOutboxMessages($messages); + + $this->assertSame([1, 2], $result->accepted); + $this->assertNull($result->failure); + $this->assertSame([], $result->deferred); + } + + /** + * Provide standalone and Cluster key layouts. + */ + public static function clusterModes(): array + { + return [[false], [true]]; + } + + public function testLaterDestinationFailurePreservesEarlierConfirmation(): void + { + [$queue, $connection] = $this->queue(); + $failure = new RuntimeException('Redis connection lost.'); + $connection->expects('evalWithShaCache')->withArgs(fn (string $script, array $keys): bool => $keys[0] === 'queues:first')->andReturn(1); + $connection->expects('evalWithShaCache')->withArgs(fn (string $script, array $keys): bool => $keys[0] === 'queues:second')->andThrow($failure); + + $result = $queue->publishOutboxMessages([$this->message(1, 'first'), $this->message(2, 'second'), $this->message(3, 'third')]); + + $this->assertSame([1], $result->accepted); + $this->assertSame($failure, $result->failure); + } + + public function testIncompleteConfirmationRetainsTheEntireDestination(): void + { + [$queue, $connection] = $this->queue(); + $connection->expects('evalWithShaCache')->andReturn(1); + + $result = $queue->publishOutboxMessages([$this->message(1), $this->message(2)]); + + $this->assertSame([], $result->accepted); + $this->assertInstanceOf(RuntimeException::class, $result->failure); + } + + public function testHorizonStampsPublicationAndDrainsAcceptedNotificationsBeforeStopping(): void + { + CarbonImmutable::setTestNow('2040-01-01 12:00:00'); + [$queue, $connection, $events] = $this->queue(horizon: true); + $messages = [$this->message(1), $this->message(2), $this->message(3, 'other')]; + $pending = []; + $pushed = []; + $failure = new RuntimeException('Monitoring storage unavailable.'); + $events->listen(JobPending::class, function (JobPending $event) use (&$pending): void { + $pending[] = $event->payload->value; + }); + $events->listen(JobPushed::class, function (JobPushed $event) use (&$pushed, $failure): void { + $pushed[] = $event->payload->value; + throw $failure; + }); + $events->listen(JobQueued::class, fn () => $this->fail('Horizon relay must not emit JobQueued.')); + $stored = []; + $connection->expects('evalWithShaCache')->andReturnUsing(function (string $script, array $keys, array $arguments) use (&$stored): int { + $stored = [$arguments[1], $arguments[3]]; + + return 2; + }); + + $result = $queue->publishOutboxMessages($messages); + + $this->assertSame([1, 2], $result->accepted); + $this->assertSame($failure, $result->failure); + $this->assertSame($stored, $pending); + $this->assertSame($stored, $pushed); + + foreach ($stored as $index => $payload) { + $decoded = json_decode($payload, true); + $original = json_decode($messages[$index]->payload, true); + $this->assertArrayHasKey('pushedAt', $decoded); + unset($decoded['pushedAt']); + $this->assertSame($original, $decoded); + } + } + + /** + * Create a real driver over an observable transport boundary. + */ + protected function queue(bool $cluster = false, bool $horizon = false): array + { + $container = new Container; + Container::setInstance($container); + $events = new Dispatcher($container); + $container->instance('events', $events); + $container->instance(EventDispatcher::class, $events); + $container->instance('queue.routes', $routes = new QueueRoutes); + $connection = m::mock(RedisProxy::class); + $connection->shouldAllowMockingMethod('evalWithShaCache'); + $connection->expects('isCluster')->andReturn($cluster); + $factory = m::mock(Factory::class); + $factory->shouldReceive('connection')->andReturn($connection); + $queue = $horizon ? new HorizonRedisQueue($factory, 'orders') : new RedisQueue($factory, 'orders'); + $queue->setContainer($container)->setConnectionName('redis'); + + return [$queue, $connection, $events, $routes]; + } + + /** + * Build a finalized envelope with immutable staging classification. + */ + protected function message(int $id, string $queue = 'orders', ?int $deliverAt = null): OutboxMessage + { + $uuid = (string) Str::uuid(); + $payload = json_encode([ + 'uuid' => $uuid, 'id' => $uuid, 'attempts' => 0, + 'type' => 'job', 'tags' => ['tenant:17'], 'silenced' => false, + 'data' => ['command' => 'opaque serialized command'], + ]); + + return new OutboxMessage($id, $uuid, (string) Str::uuid(), 1, 'redis', $queue, $payload, [], $deliverAt, 1, 1); + } +} diff --git a/tests/Queue/Outbox/SqsOutboxPublicationTest.php b/tests/Queue/Outbox/SqsOutboxPublicationTest.php new file mode 100644 index 000000000..ecb048348 --- /dev/null +++ b/tests/Queue/Outbox/SqsOutboxPublicationTest.php @@ -0,0 +1,187 @@ +message(1, $fifo ? 'orders.fifo' : 'orders', $offset); + $client->expects('sendMessageBatch')->andReturnUsing(function (array $arguments) use ($fifo, $expected): Result { + $entry = $arguments['Entries'][0]; + $this->assertSame('https://sqs.test/account/orders' . ($fifo ? '.fifo' : ''), $arguments['QueueUrl']); + + if ($expected === null) { + $this->assertArrayNotHasKey('DelaySeconds', $entry); + } else { + $this->assertSame($expected, $entry['DelaySeconds']); + } + + if ($fifo) { + $this->assertSame('tenant:17', $entry['MessageGroupId']); + $this->assertSame('captured-deduplication', $entry['MessageDeduplicationId']); + } + + return new Result(['Successful' => [['Id' => '1', 'MessageId' => 'sqs-1']]]); + }); + + $result = $queue->publishOutboxMessages([$message]); + + $this->assertSame([1], $result->accepted); + $this->assertNull($result->failure); + } + + /** + * Provide absent, due, elapsed, capped and FIFO timers. + */ + public static function deliveryTimes(): array + { + return [[null, false, null], [0, false, 0], [-30, false, 0], [30, false, 30], [1000, false, 900], [0, true, null]]; + } + + public function testBatchSizeLimitsAndLaterSdkFailureRetainEarlierConfirmation(): void + { + $client = m::mock(SqsClient::class); + $queue = new SqsQueue($client, 'orders'); + $failure = new RuntimeException('SDK retries exhausted.'); + $calls = 0; + $client->expects('sendMessageBatch')->twice()->andReturnUsing(function (array $arguments) use (&$calls, $failure): Result { + if (++$calls === 2) { + $this->assertCount(1, $arguments['Entries']); + throw $failure; + } + + $this->assertCount(10, $arguments['Entries']); + + return new Result(['Successful' => array_map(static fn (array $entry): array => ['Id' => $entry['Id'], 'MessageId' => 'sqs-' . $entry['Id']], $arguments['Entries'])]); + }); + + $result = $queue->publishOutboxMessages(array_map(fn (int $id): OutboxMessage => $this->message($id), range(1, 11))); + + $this->assertSame(range(1, 10), $result->accepted); + $this->assertSame($failure, $result->failure); + } + + public function testByteLimitSplitsLargeMessagesBeforeTheCountLimit(): void + { + $client = m::mock(SqsClient::class); + $queue = new SqsQueue($client, 'orders'); + $client->expects('sendMessageBatch')->twice()->andReturnUsing(function (array $arguments): Result { + $this->assertCount(1, $arguments['Entries']); + + return new Result(['Successful' => [['Id' => $arguments['Entries'][0]['Id'], 'MessageId' => 'accepted']]]); + }); + + $result = $queue->publishOutboxMessages([ + $this->message(1, payload: json_encode(['data' => str_repeat('a', 600000)])), + $this->message(2, payload: json_encode(['data' => str_repeat('b', 600000)])), + ]); + + $this->assertSame([1, 2], $result->accepted); + $this->assertNull($result->failure); + } + + public function testExplicitRejectionCleansOnlyThatAttemptAndRetainsConfirmedEntries(): void + { + $container = new Container; + Container::setInstance($container); + $store = new Repository(new WorkerArrayStore); + $cache = m::mock(CacheFactory::class); + $cache->shouldReceive('store')->with(null)->andReturn($store); + $container->instance('cache', $cache); + $client = m::mock(SqsClient::class); + $queue = new SqsQueue($client, 'orders', overflowStorage: ['enabled' => true, 'always' => true]); + $queue->setContainer($container); + $first = $this->message(1); + $secondAttempt = new OutboxMessage( + $first->id, + $first->uuid, + (string) Str::uuid(), + 2, + $first->connection, + $first->queue, + $first->payload, + $first->options, + null, + 1, + 1, + ); + $accepted = $this->message(2); + $paths = []; + $calls = 0; + $client->expects('sendMessageBatch')->twice()->andReturnUsing(function (array $arguments) use (&$calls, &$paths, $store): Result { + foreach ($arguments['Entries'] as $entry) { + $path = json_decode($entry['MessageBody'], true)['@pointer']; + $paths[] = $path; + $this->assertTrue($store->has($path)); + } + + if (++$calls === 1) { + throw new RuntimeException('Acceptance is ambiguous.'); + } + + return new Result([ + 'Successful' => [['Id' => '2', 'MessageId' => 'confirmed']], + 'Failed' => [['Id' => '1', 'Code' => 'Rejected', 'Message' => 'Rejected entry', 'SenderFault' => true]], + ]); + }); + + $this->assertSame([], $queue->publishOutboxMessages([$first])->accepted); + $result = $queue->publishOutboxMessages([$secondAttempt, $accepted]); + + $this->assertSame([2], $result->accepted); + $this->assertInstanceOf(SqsException::class, $result->failure); + $this->assertStringContainsString('rejected [1] of [2]', $result->failure->getMessage()); + $this->assertNotSame($paths[0], $paths[1]); + $this->assertSame($first->payload, $store->get($paths[0])); + $this->assertFalse($store->has($paths[1])); + $this->assertSame($accepted->payload, $store->get($paths[2])); + } + + /** + * Build a claimed envelope with previously captured FIFO options. + */ + protected function message(int $id, string $queue = 'orders', ?int $offset = null, ?string $payload = null): OutboxMessage + { + $uuid = (string) Str::uuid(); + $options = str_ends_with($queue, '.fifo') + ? ['transport' => ['sqs' => ['MessageGroupId' => 'tenant:17', 'MessageDeduplicationId' => 'captured-deduplication']]] + : []; + + return new OutboxMessage( + $id, + $uuid, + (string) Str::uuid(), + 1, + 'sqs', + $queue, + $payload ?? json_encode(['uuid' => $uuid, 'data' => ['command' => 'opaque']]), + $options, + $offset === null ? null : CarbonImmutable::now()->getTimestamp() + $offset, + 1, + 1, + ); + } +} diff --git a/tests/Queue/QueueBackgroundQueueTest.php b/tests/Queue/QueueBackgroundQueueTest.php index e3652ff98..89ffd1909 100644 --- a/tests/Queue/QueueBackgroundQueueTest.php +++ b/tests/Queue/QueueBackgroundQueueTest.php @@ -204,6 +204,7 @@ public function testItAddsATransactionCallbackForAfterCommitJobs(): void $background->setConnectionName('background'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -217,6 +218,7 @@ public function testItAddsATransactionCallbackForInterfaceBasedAfterCommitJobs() $background->setConnectionName('background'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -230,6 +232,7 @@ public function testItAddsATransactionCallbackForAfterCommitUniqueJobs(): void $background->setConnectionName('background'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $transactionManager->shouldReceive('addCallbackForRollback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -247,6 +250,7 @@ public function testItAddsATransactionRollbackCallbackForAfterCommitDebouncedJob $background->setConnectionName('background'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $transactionManager->shouldReceive('addCallbackForRollback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -264,6 +268,7 @@ public function testItAddsATransactionCallbackForInterfaceBasedAfterCommitUnique $background->setConnectionName('background'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $transactionManager->shouldReceive('addCallbackForRollback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -361,6 +366,7 @@ public function testPushSnapshotsAfterCommitJobAtDispatch(): void $afterCommit = null; $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) use (&$afterCommit) { @@ -450,6 +456,7 @@ public function testLaterAddsTransactionCallbackForAfterCommitJobs(): void $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { @@ -472,6 +479,7 @@ public function testLaterAddsTransactionCallbackForInterfaceBasedAfterCommitJobs $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { @@ -494,6 +502,7 @@ public function testLaterAddsTransactionCallbackForAfterCommitUniqueJobs(): void $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { @@ -519,6 +528,7 @@ public function testLaterAddsTransactionRollbackCallbackForAfterCommitDebouncedJ $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { @@ -544,6 +554,7 @@ public function testLaterAddsTransactionCallbackForInterfaceBasedAfterCommitUniq $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { diff --git a/tests/Queue/QueueDatabaseQueueUnitTest.php b/tests/Queue/QueueDatabaseQueueUnitTest.php index d25f1ece1..9e4b0e5b6 100644 --- a/tests/Queue/QueueDatabaseQueueUnitTest.php +++ b/tests/Queue/QueueDatabaseQueueUnitTest.php @@ -531,6 +531,7 @@ public function testBulkRetainsDispatchOwnershipUntilTheTransactionCommits(): vo $resolver->shouldReceive('connection')->once()->with(null)->andReturn($connection = m::mock(PdoConnection::class)); $connection->shouldReceive('maxBindings')->once()->andReturn(6); + $connection->shouldReceive('transactionLevel')->twice()->andReturn(0); $connection->shouldReceive('transaction')->once()->andReturnUsing(function (Closure $callback) use ($first, $second): mixed { $this->assertTrue(DispatchLockContext::has($first)); $this->assertTrue(DispatchLockContext::has($second)); @@ -569,10 +570,15 @@ public function testBulkAcceptsEveryDispatchBeforeRaisingSuccessEvents(): void DispatchLockContext::registerDebounce($first, $cache, 'first', 'owner-1'); DispatchLockContext::registerDebounce($second, $cache, 'second', 'owner-2'); $exception = new RuntimeException('Listener failed.'); + $notified = []; $resolver->shouldReceive('connection')->once()->with(null)->andReturn($connection = m::mock(ConnectionInterface::class)); + $connection->shouldReceive('transactionLevel')->twice()->andReturn(0); $connection->shouldReceive('table')->once()->with('table')->andReturn($query = m::mock(Builder::class)); $query->shouldReceive('insert')->once()->andReturnTrue(); + $dispatcher->listen(JobQueued::class, static function (JobQueued $event) use (&$notified): void { + $notified[] = $event->job; + }); $dispatcher->listen(JobQueued::class, static function () use ($exception): never { throw $exception; }); @@ -586,6 +592,7 @@ public function testBulkAcceptsEveryDispatchBeforeRaisingSuccessEvents(): void $this->assertFalse(DispatchLockContext::has($first)); $this->assertFalse(DispatchLockContext::has($second)); + $this->assertSame([$first, $second], $notified); } public function testDelayAttributeIsRespectedWhenBulkPushing(): void diff --git a/tests/Queue/QueueDeferredQueueTest.php b/tests/Queue/QueueDeferredQueueTest.php index 5bf316199..4cd325652 100644 --- a/tests/Queue/QueueDeferredQueueTest.php +++ b/tests/Queue/QueueDeferredQueueTest.php @@ -185,6 +185,7 @@ public function testItAddsATransactionCallbackForAfterCommitJobs(): void $deferred->setConnectionName('deferred'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -198,6 +199,7 @@ public function testItAddsATransactionCallbackForInterfaceBasedAfterCommitJobs() $deferred->setConnectionName('deferred'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -211,6 +213,7 @@ public function testItAddsATransactionCallbackForAfterCommitUniqueJobs(): void $deferred->setConnectionName('deferred'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $transactionManager->shouldReceive('addCallbackForRollback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -228,6 +231,7 @@ public function testItAddsATransactionRollbackCallbackForAfterCommitDebouncedJob $deferred->setConnectionName('deferred'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $transactionManager->shouldReceive('addCallbackForRollback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -245,6 +249,7 @@ public function testItAddsATransactionCallbackForInterfaceBasedAfterCommitUnique $deferred->setConnectionName('deferred'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $transactionManager->shouldReceive('addCallbackForRollback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -343,6 +348,7 @@ public function testLaterAddsTransactionCallbackForAfterCommitJobs(): void $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { @@ -365,6 +371,7 @@ public function testLaterAddsTransactionCallbackForInterfaceBasedAfterCommitJobs $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { @@ -387,6 +394,7 @@ public function testLaterAddsTransactionCallbackForAfterCommitUniqueJobs(): void $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { @@ -412,6 +420,7 @@ public function testLaterAddsTransactionRollbackCallbackForAfterCommitDebouncedJ $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { @@ -437,6 +446,7 @@ public function testLaterAddsTransactionCallbackForInterfaceBasedAfterCommitUniq $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { diff --git a/tests/Queue/QueueRedisQueueTest.php b/tests/Queue/QueueRedisQueueTest.php index 5af48ddad..8b05940d1 100644 --- a/tests/Queue/QueueRedisQueueTest.php +++ b/tests/Queue/QueueRedisQueueTest.php @@ -207,7 +207,11 @@ public function testBulkAcceptsEveryDispatchBeforeRaisingSuccessEvents(): void DispatchLockContext::registerDebounce($first, $cache, 'first', 'owner-1'); DispatchLockContext::registerDebounce($second, $cache, 'second', 'owner-2'); $exception = new RuntimeException('Listener failed.'); + $notified = []; + $dispatcher->listen(JobQueued::class, static function (JobQueued $event) use (&$notified): void { + $notified[] = $event->job; + }); $dispatcher->listen(JobQueued::class, static function () use ($exception): never { throw $exception; }); @@ -221,6 +225,7 @@ public function testBulkAcceptsEveryDispatchBeforeRaisingSuccessEvents(): void $this->assertFalse(DispatchLockContext::has($first)); $this->assertFalse(DispatchLockContext::has($second)); + $this->assertSame([$first, $second], $notified); } public function testBulkRetainsDispatchOwnershipWhenRedisDoesNotConfirmEveryJob(): void diff --git a/tests/Queue/QueueSyncQueueTest.php b/tests/Queue/QueueSyncQueueTest.php index d09fb9a6e..12ef40d82 100644 --- a/tests/Queue/QueueSyncQueueTest.php +++ b/tests/Queue/QueueSyncQueueTest.php @@ -340,6 +340,7 @@ public function testItAddsATransactionCallbackForAfterCommitJobs(): void $sync->setConnectionName('sync'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('addCallback')->andReturn(null); $transactionManager->shouldNotReceive('addCallbackForRollback'); $container->instance('db.transactions', $transactionManager); @@ -354,6 +355,7 @@ public function testItAddsATransactionCallbackForInterfaceBasedAfterCommitJobs() $sync->setConnectionName('sync'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('addCallback')->andReturn(null); $transactionManager->shouldNotReceive('addCallbackForRollback'); $container->instance('db.transactions', $transactionManager); @@ -368,6 +370,7 @@ public function testItAddsATransactionCallbackForAfterCommitUniqueJobs(): void $sync->setConnectionName('sync'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('addCallback')->andReturn(null); $transactionManager->expects('addCallbackForRollback')->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -385,6 +388,7 @@ public function testItAddsATransactionRollbackCallbackForAfterCommitDebouncedJob $sync->setConnectionName('sync'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('addCallback')->andReturn(null); $transactionManager->expects('addCallbackForRollback')->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -402,6 +406,7 @@ public function testItAddsATransactionCallbackForInterfaceBasedAfterCommitUnique $sync->setConnectionName('sync'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('addCallback')->andReturn(null); $transactionManager->expects('addCallbackForRollback')->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -419,6 +424,7 @@ public function testAfterCommitUniqueJobWithoutDispatchOwnershipDoesNotAddRollba $sync->setConnectionName('sync'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('addCallback')->andReturnNull(); $transactionManager->shouldReceive('addCallbackForRollback')->never(); $container->instance('db.transactions', $transactionManager); diff --git a/tests/Queue/SqsBatchLifecycleTest.php b/tests/Queue/SqsBatchLifecycleTest.php new file mode 100644 index 000000000..8c4343b43 --- /dev/null +++ b/tests/Queue/SqsBatchLifecycleTest.php @@ -0,0 +1,192 @@ +queue(); + $cache = new Repository(new WorkerArrayStore); + $container->instance(Cache::class, $cache); + $container->instance(BusContract::class, new BusDispatcher($container, static fn () => $queue)); + $transactions = new DatabaseTransactionsManager; + $container->instance('db.transactions', $transactions); + $transactions->begin('testing', 1); + $sent = false; + $client->expects('sendMessageBatch')->andReturnUsing(static function () use (&$sent): Result { + $sent = true; + + return new Result(['Successful' => [['Id' => '0', 'MessageId' => 'accepted']]]); + }); + $job = new SqsUniqueBulkJob; + $job->afterCommit = $afterCommit; + + $pending = new PendingDispatch($job); + unset($pending); + + $this->assertSame(! $afterCommit, $sent); + $this->assertTrue($cache->lock(UniqueLock::getKey($job))->isLocked()); + $transactions->commit('testing', 1, 0); + $this->assertTrue($sent); + $this->assertFalse(DispatchLockContext::has($job)); + $this->assertTrue($cache->lock(UniqueLock::getKey($job))->isLocked()); + } + + #[TestWith([JobPayloadFinalizing::class])] + #[TestWith([JobQueueing::class])] + public function testPreparationFailureTerminatesEveryOutstandingMember(string $eventClass): void + { + [$queue, $client, $events] = $this->queue(); + $exception = new RuntimeException('Second member failed.'); + $count = 0; + $events->listen($eventClass, static function () use (&$count, $exception): void { + if (++$count === 2) { + throw $exception; + } + }); + $failed = []; + $events->listen(JobQueueingFailed::class, static function (JobQueueingFailed $event) use (&$failed): void { + $failed[] = $event; + }); + $client->shouldNotReceive('sendMessageBatch'); + + try { + $queue->bulk(['first', 'second', 'third']); + $this->fail('The second member should fail preparation.'); + } catch (RuntimeException $caught) { + $this->assertSame($exception, $caught); + } + + $this->assertSame(['first', 'second', 'third'], array_column($failed, 'job')); + $this->assertSame([$exception, $exception, $exception], array_column($failed, 'exception')); + } + + public function testFailureNotificationsDrainAndRetainThePrimaryAttemptError(): void + { + [$queue, $client, $events] = $this->queue(); + $primary = new RuntimeException('Transport failed.'); + $listener = new RuntimeException('Listener failed.'); + $client->expects('sendMessageBatch')->andThrow($primary); + $failed = []; + $events->listen(JobQueueingFailed::class, static function (JobQueueingFailed $event) use (&$failed): void { + $failed[] = $event; + }); + $events->listen(JobQueueingFailed::class, static fn () => throw $listener); + + try { + $queue->bulk(['first', 'second']); + $this->fail('The failure listener should throw.'); + } catch (RuntimeException $caught) { + $this->assertSame($listener, $caught); + } + + $this->assertSame(['first', 'second'], array_column($failed, 'job')); + $this->assertSame([$primary, $primary], array_column($failed, 'exception')); + } + + public function testSuccessListenerFailurePreservesAcceptedEntriesAndCleansExplicitRejects(): void + { + [$queue, $client, $events, $container] = $this->queue(['enabled' => true, 'always' => true, 'store' => 'overflow']); + $store = new Repository(new WorkerArrayStore); + $cache = m::mock(CacheFactory::class); + $cache->expects('store')->with('overflow')->andReturn($store); + $container->instance('cache', $cache); + $paths = []; + $client->expects('sendMessageBatch')->once()->andReturnUsing(static function (array $arguments) use (&$paths): Result { + $paths = array_map(static fn (array $entry): string => json_decode($entry['MessageBody'], true)['@pointer'], $arguments['Entries']); + + return new Result([ + 'Successful' => array_map(static fn (int $id): array => ['Id' => (string) $id, 'MessageId' => 'accepted-' . $id], range(0, 8)), + 'Failed' => [['Id' => '9', 'Code' => 'InvalidParameterValue', 'Message' => 'Rejected']], + ]); + }); + $queued = $failed = []; + $listener = new RuntimeException('Monitoring failed.'); + $events->listen(JobQueued::class, static function (JobQueued $event) use (&$queued): void { + $queued[] = $event->job; + }); + $events->listen(JobQueued::class, static fn () => throw $listener); + $events->listen(JobQueueingFailed::class, static function (JobQueueingFailed $event) use (&$failed): void { + $failed[] = $event->job; + }); + + try { + $queue->bulk(array_map('strval', range(1, 11))); + $this->fail('The monitoring failure should be rethrown.'); + } catch (RuntimeException $caught) { + $this->assertSame($listener, $caught); + } + + $this->assertSame(array_map('strval', range(1, 9)), $queued); + $this->assertSame(['10', '11'], $failed); + $this->assertFalse($store->has($paths[9])); + + foreach (array_slice($paths, 0, 9) as $path) { + $this->assertTrue($store->has($path)); + } + } + + /** + * Create a queue with real lifecycle dispatch and a controlled SDK boundary. + * + * @return array{SqsQueue, SqsClient, Dispatcher, Container} + */ + protected function queue(array $overflow = []): array + { + $container = new Container; + Container::setInstance($container); + $events = new Dispatcher($container); + $container->instance('events', $events); + $container->instance(EventsContract::class, $events); + $client = m::mock(SqsClient::class); + $queue = new SqsQueue($client, 'jobs', overflowStorage: $overflow); + $queue->setContainer($container); + $queue->setConnectionName('sqs'); + + return [$queue, $client, $events, $container]; + } +} + +class SqsUniqueBulkJob extends FakeSqsJob implements ShouldBeUnique +{ + /** + * Route a normal pending dispatch through the supported bulk API. + */ + public function queue(QueueContract $queue, object $command): mixed + { + return $queue->bulk([$command]); + } +} diff --git a/tests/Testbench/CommanderTest.php b/tests/Testbench/CommanderTest.php index f9ba39e1c..687f1cfaa 100644 --- a/tests/Testbench/CommanderTest.php +++ b/tests/Testbench/CommanderTest.php @@ -267,6 +267,7 @@ public function itCanCallCommanderUsingCliAndRunMigration(): void '0001_01_01_000006_testbench_create_jobs_table', '0001_01_01_000007_testbench_create_failed_jobs_table', '0001_01_01_000008_testbench_create_rate_limits_table', + '0001_01_01_000009_testbench_create_job_outbox_table', '2013_07_26_182750_create_testbench_users_table', ], DB::connection('sqlite')->table('migrations')->pluck('migration')->all()); }); From e083850b715ad2665e9dd311fba58a4d9f3568ac Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 20:48:16 +0000 Subject: [PATCH 08/33] feat(horizon): preserve queue lifecycle across outbox publication Classify staged jobs while the original object is available, then add publication timing only when the stored payload reaches Redis. Preserve Horizon identifiers, stored tags, silencing and retry metadata without redispatching the command. Use the shared preparation and post-acceptance hooks for normal pushes and batched outbox publication. Keep generic lifecycle payloads stable for tracing, accept confirmed jobs before observers, and preserve confirmed batches if a later notification fails. Encode combined classification and timing once, remove the obsolete last-pushed context handoff, and document the approved protected-hook routing for Horizon subclasses. Extend direct, delayed, raw-retry and batch lifecycle coverage. --- src/horizon/README.md | 1 + src/horizon/src/JobPayload.php | 21 ++- src/horizon/src/RedisQueue.php | 93 ++++------- tests/Horizon/Unit/RedisQueueTest.php | 152 ++++++++++++++++++ .../Horizon/Feature/QueueProcessingTest.php | 34 ---- .../Horizon/Feature/RedisPayloadTest.php | 53 ++++++ 6 files changed, 258 insertions(+), 96 deletions(-) diff --git a/src/horizon/README.md b/src/horizon/README.md index ff2fe6442..8a5fd43a7 100644 --- a/src/horizon/README.md +++ b/src/horizon/README.md @@ -11,5 +11,6 @@ Documentation: https://hypervel.org/docs/horizon - `horizon:listen` uses Hypervel's file watcher instead of a Node.js `chokidar` process, so Node is not required. A nonempty `horizon.watch` list replaces the `watcher` configuration's watch list, and `--poll` selects its scanning driver. See [automatically restarting Horizon](https://hypervel.org/docs/horizon#automatically-restarting-horizon). - Supervisors accept a `concurrency` option that lets each worker process run several jobs at once in coroutines. See [concurrency](https://hypervel.org/docs/horizon#concurrency). - Redis Cluster is configured on the named Redis connection selected by `horizon.use`. Laravel's top-level `database.redis.clusters` entries are not read. +- Horizon's `RedisQueue::push()` does not call `pushRaw()`. Custom queue subclasses should use `preparePayloadForPush()` and `handlePayloadPushed()` to customize normal publication; the latter runs after storage is accepted. Direct `pushRaw()` calls remain supported. Ported from: https://github.com/laravel/horizon diff --git a/src/horizon/src/JobPayload.php b/src/horizon/src/JobPayload.php index 4792e800f..995845858 100644 --- a/src/horizon/src/JobPayload.php +++ b/src/horizon/src/JobPayload.php @@ -95,25 +95,38 @@ public function isSilenced(): bool */ public function prepare(mixed $job): static { - return $this->prepareForQueueing($job)->markAsPushed(); + return $this->set([ + ...$this->queueingMetadata($job), + 'pushedAt' => str_replace(',', '.', (string) microtime(true)), + ]); } /** * Prepare the payload metadata while the original job is available. */ public function prepareForQueueing(mixed $job): static + { + $metadata = $this->queueingMetadata($job); + + return $metadata === [] ? $this : $this->set($metadata); + } + + /** + * Determine classification metadata without replacing it on raw retries. + */ + protected function queueingMetadata(mixed $job): array { if ($job === null && isset($this->decoded['type'])) { - return $this; + return []; } $tags = $this->determineTags($job); - return $this->set([ + return [ 'type' => $this->determineType($job), 'tags' => $tags, 'silenced' => $this->shouldBeSilenced($job, $tags), - ]); + ]; } /** diff --git a/src/horizon/src/RedisQueue.php b/src/horizon/src/RedisQueue.php index e0d27c008..d90f5400e 100644 --- a/src/horizon/src/RedisQueue.php +++ b/src/horizon/src/RedisQueue.php @@ -24,8 +24,6 @@ class RedisQueue extends BaseQueue { - public const string LAST_PUSHED_CONTEXT_KEY = '__horizon.queue.last_pushed'; - protected const string POPPING_QUEUE_CONTEXT_KEY = '__horizon.queue.popping'; /** @@ -49,11 +47,9 @@ public function push(object|string $job, mixed $data = '', UnitEnum|string|null $this->createPayload($job, $this->getQueue($queue), $data), $queue, null, - static function (BaseQueue $owner, $payload, $queue) use ($job) { + static function (BaseQueue $owner, string $payload, ?string $queue) { /** @var self $owner */ - $owner->setLastPushed($job); - - return $owner->pushRaw($payload, $queue); + return $owner->pushPreparedPayload($payload, $queue); } ); } @@ -64,25 +60,24 @@ static function (BaseQueue $owner, $payload, $queue) use ($job) { #[Override] public function pushRaw(string $payload, UnitEnum|string|null $queue = null, array $options = []): mixed { - $job = CoroutineContext::get(static::LAST_PUSHED_CONTEXT_KEY); + $payload = (new JobPayload($payload))->prepare(null); - try { - $payload = (new JobPayload($payload))->prepare($job); - - if ($this->hasEventListeners(JobPending::class)) { - $this->event($this->getQueue($queue), new JobPending($payload->value)); - } + if ($this->hasEventListeners(JobPending::class)) { + $this->event($this->getQueue($queue), new JobPending($payload->value)); + } - parent::pushRaw($payload->value, $queue, $options); + $this->pushPreparedPayload($payload->value, $queue, $options); + $this->handlePayloadPushed($payload->value, $this->normalizeQueue($queue)); - if ($this->hasEventListeners(JobPushed::class)) { - $this->event($this->getQueue($queue), new JobPushed($payload->value)); - } + return $payload->id(); + } - return $payload->id(); - } finally { - CoroutineContext::forget(static::LAST_PUSHED_CONTEXT_KEY); - } + /** + * Store a prepared payload without running Horizon's raw-push lifecycle again. + */ + protected function pushPreparedPayload(string $payload, UnitEnum|string|null $queue = null, array $options = []): mixed + { + return parent::pushRaw($payload, $queue, $options); } /** @@ -99,43 +94,33 @@ protected function createPayloadArray(array|object|string $job, ?string $queue, } /** - * Push a new job onto the queue after a delay. + * Classify a staged job without recording it as pending in Redis. */ - #[Override] - public function later(DateInterval|DateTimeInterface|int $delay, object|string $job, mixed $data = '', UnitEnum|string|null $queue = null): mixed + protected function preparePayloadForOutbox(object|string $job, string $payload): string { - $queue = $this->normalizeQueue($queue); - - return $this->enqueueUsing( - $job, - $this->createPayload($job, $this->getQueue($queue), $data, $delay), - $queue, - $delay, - static function (BaseQueue $owner, string $payload, ?string $queue, DateInterval|DateTimeInterface|int $delay) use ($job) { - // Horizon's Redis queue is not pooled, so after-commit dispatch returns this same queue type. - /** @var self $owner */ - $payload = (new JobPayload($payload))->prepare($job)->value; - - if ($owner->hasEventListeners(JobPending::class)) { - $owner->event($owner->getQueue($queue), new JobPending($payload)); - } + return (new JobPayload($payload))->prepareForQueueing($job)->value; + } - $id = $owner->laterRaw($delay, $payload, $queue); + /** + * Stamp a relayed payload and record its pending state before publication. + */ + #[Override] + protected function preparePayloadForOutboxPublication(string $payload, ?string $queue): string + { + $payload = (new JobPayload($payload))->markAsPushed()->value; - if ($owner->hasEventListeners(JobPushed::class)) { - $owner->event($owner->getQueue($queue), new JobPushed($payload)); - } + if ($this->hasEventListeners(JobPending::class)) { + $this->event($this->getQueue($queue), new JobPending($payload)); + } - return $id; - } - ); + return $payload; } /** - * Prepare a payload for bulk storage. + * Prepare a payload and record its pending state before publication. */ #[Override] - protected function preparePayloadForBulk(object|string $job, string $payload, ?string $queue): string + protected function preparePayloadForPush(object|string $job, string $payload, ?string $queue): string { $payload = (new JobPayload($payload))->prepare($job)->value; @@ -147,10 +132,10 @@ protected function preparePayloadForBulk(object|string $job, string $payload, ?s } /** - * Handle a payload that was stored as part of a batch. + * Notify Horizon after the queue has accepted the payload. */ #[Override] - protected function handlePayloadPushedInBulk(string $payload, ?string $queue): void + protected function handlePayloadPushed(string $payload, ?string $queue): void { if ($this->hasEventListeners(JobPushed::class)) { $this->event($this->getQueue($queue), new JobPushed($payload)); @@ -269,12 +254,4 @@ protected function hasEventListeners(string $event): bool return $this->container->bound(Dispatcher::class) && $this->container->make(Dispatcher::class)->hasListeners($event); } - - /** - * Set the job that last pushed to queue via the "push" method. - */ - protected function setLastPushed(object|string $job): void - { - CoroutineContext::set(static::LAST_PUSHED_CONTEXT_KEY, $job); - } } diff --git a/tests/Horizon/Unit/RedisQueueTest.php b/tests/Horizon/Unit/RedisQueueTest.php index ac47cfddf..1ec4d2d1a 100644 --- a/tests/Horizon/Unit/RedisQueueTest.php +++ b/tests/Horizon/Unit/RedisQueueTest.php @@ -4,13 +4,22 @@ namespace Hypervel\Tests\Horizon\Unit; +use Hypervel\Bus\DispatchLockContext; +use Hypervel\Bus\UniqueLock; +use Hypervel\Cache\Repository; +use Hypervel\Cache\WorkerArrayStore; +use Hypervel\Config\Repository as ConfigRepository; use Hypervel\Container\Container; use Hypervel\Contracts\Events\Dispatcher as DispatcherContract; use Hypervel\Contracts\Queue\Job; use Hypervel\Contracts\Redis\Factory; use Hypervel\Events\Dispatcher; +use Hypervel\Horizon\Events\JobPushed; use Hypervel\Horizon\Events\JobsMigrated; use Hypervel\Horizon\RedisQueue; +use Hypervel\Queue\Events\JobQueued; +use Hypervel\Queue\Events\JobQueueing; +use Hypervel\Queue\Events\JobQueueingFailed; use Hypervel\Queue\LuaScripts; use Hypervel\Redis\RedisProxy; use Hypervel\Tests\Horizon\UnitTestCase; @@ -18,11 +27,113 @@ use Mockery as m; use PHPUnit\Framework\Attributes\TestWith; use RedisException; +use RuntimeException; +use stdClass; +use Swoole\Coroutine\CanceledException; use function Hypervel\Coroutine\parallel; class RedisQueueTest extends UnitTestCase { + #[TestWith(['push'])] + #[TestWith(['later'])] + #[TestWith(['bulk'])] + #[TestWith(['failed-bulk'])] + public function testLifecyclePayloadCorrelationIsIndependentOfHorizonMetadata(string $method): void + { + $stored = []; + $failure = $method === 'failed-bulk' ? new RuntimeException('Redis unavailable.') : null; + [$queue, $events] = $this->publishingQueue($stored, $failure); + $queueing = $terminal = []; + $events->listen(JobQueueing::class, static function (JobQueueing $event) use (&$queueing): void { + $queueing[] = $event->payload; + }); + $events->listen([JobQueued::class, JobQueueingFailed::class], static function (JobQueued|JobQueueingFailed $event) use (&$terminal): void { + $terminal[] = $event->payload; + }); + + try { + match ($method) { + 'push' => $queue->push('first'), + 'later' => $queue->later(30, 'first'), + default => $queue->bulk(['first', 'second']), + }; + $this->assertNull($failure); + } catch (RuntimeException $caught) { + $this->assertSame($failure, $caught); + } + + $this->assertCount(str_contains($method, 'bulk') ? 2 : 1, $queueing); + $this->assertSame($queueing, $terminal); + + if ($failure === null) { + foreach ($stored as $index => $payload) { + $decoded = json_decode($payload, true); + $this->assertArrayHasKey('pushedAt', $decoded); + $this->assertSame('job', $decoded['type']); + $this->assertSame(json_decode($queueing[$index], true)['uuid'], $decoded['uuid']); + $this->assertArrayNotHasKey('pushedAt', json_decode($terminal[$index], true)); + } + } + } + + #[TestWith(['push'])] + #[TestWith(['later'])] + #[TestWith(['bulk'])] + public function testPostPushFailurePreservesAcceptanceAndNotifiesEveryMember(string $method): void + { + $stored = []; + [$queue, $events] = $this->publishingQueue($stored); + $exception = new RuntimeException('Horizon monitoring failed.'); + $events->listen(JobPushed::class, static fn () => throw $exception); + $queued = $failed = []; + $events->listen(JobQueued::class, static function (JobQueued $event) use (&$queued): void { + $queued[] = $event->job; + }); + $events->listen(JobQueueingFailed::class, static function (JobQueueingFailed $event) use (&$failed): void { + $failed[] = $event; + }); + $cache = new Repository(new WorkerArrayStore); + $job = new stdClass; + $this->assertTrue((new UniqueLock($cache))->acquireForDispatch($job)); + $jobs = $method === 'bulk' ? [$job, new stdClass] : [$job]; + + try { + match ($method) { + 'push' => $queue->push($job), + 'later' => $queue->later(30, $job), + 'bulk' => $queue->bulk($jobs), + }; + $this->fail('The monitoring failure should be rethrown.'); + } catch (RuntimeException $caught) { + $this->assertSame($exception, $caught); + } + + $this->assertCount(count($jobs), $stored); + $this->assertSame($jobs, $queued); + $this->assertSame([], $failed); + $this->assertFalse(DispatchLockContext::has($job)); + $this->assertTrue($cache->lock(UniqueLock::getKey($job))->isLocked()); + } + + public function testPostPushCancellationStopsNotificationsWithoutReplacingCancellation(): void + { + $stored = []; + [$queue, $events] = $this->publishingQueue($stored); + $cancellation = new CanceledException; + $events->listen(JobPushed::class, static fn () => throw $cancellation); + $events->listen([JobQueued::class, JobQueueingFailed::class], static fn () => throw new RuntimeException('Must not run after cancellation.')); + + try { + $queue->push('job'); + $this->fail('The post-push hook should cancel.'); + } catch (CanceledException $caught) { + $this->assertSame($cancellation, $caught); + } + + $this->assertCount(1, $stored); + } + public function testReadyNowReadsTheClusterSafeQueueKey(): void { $connection = m::mock(RedisProxy::class); @@ -140,4 +251,45 @@ private function queueRecordingMigrations(RedisProxy $connection, array &$queues return $queue; } + + /** + * Record transport payloads while exercising the actual Horizon enqueue lifecycle. + * + * @return array{RedisQueue, Dispatcher} + */ + private function publishingQueue(array &$stored, ?RuntimeException $failure = null): array + { + $connection = m::mock(RedisProxy::class); + $connection->shouldReceive('isCluster')->andReturnFalse(); + $connection->expects('evalWithShaCache')->andReturnUsing(static function (string $script, array $keys, array $arguments) use (&$stored, $failure): int { + if ($failure !== null) { + throw $failure; + } + + if ($script === LuaScripts::bulk()) { + for ($index = 1; $index < count($arguments); $index += 2) { + $stored[] = $arguments[$index]; + } + + return intdiv(count($arguments), 2); + } + + $stored[] = $arguments[$script === LuaScripts::later() ? 1 : 0]; + + return 1; + }); + $redis = m::mock(Factory::class); + $redis->shouldReceive('connection')->with('default')->andReturn($connection); + $container = new Container; + Container::setInstance($container); + $container->instance('config', new ConfigRepository); + $events = new Dispatcher($container); + $container->instance('events', $events); + $container->instance(DispatcherContract::class, $events); + $queue = new RedisQueue($redis, 'default', 'default'); + $queue->setContainer($container); + $queue->setConnectionName('redis'); + + return [$queue, $events]; + } } diff --git a/tests/Integration/Horizon/Feature/QueueProcessingTest.php b/tests/Integration/Horizon/Feature/QueueProcessingTest.php index 2d6bb4b2a..5adf56247 100644 --- a/tests/Integration/Horizon/Feature/QueueProcessingTest.php +++ b/tests/Integration/Horizon/Feature/QueueProcessingTest.php @@ -15,7 +15,6 @@ use Hypervel\Horizon\Events\JobsMigrated; use Hypervel\Horizon\Events\RedisEvent; use Hypervel\Horizon\RedisQueue; -use Hypervel\Queue\InvalidPayloadException; use Hypervel\Queue\Jobs\RedisJob; use Hypervel\Queue\Queue as BaseQueue; use Hypervel\Redis\Exceptions\LuaScriptException; @@ -229,28 +228,6 @@ public function testFailedBulkDoesNotRaisePushedEvents(): void $this->assertSame(0, $pushed); } - public function testPayloadPreparationFailureStillConsumesThePreviousJob(): void - { - $queue = new RedisQueueWithExposedLastPushed( - app('redis'), - 'default', - 'default', - ); - $queue->setContainer($this->app)->setConnectionName('redis'); - $queue->rememberLastPushed(new BasicJob); - - try { - $queue->pushRaw('{invalid'); - $this->fail('Expected the invalid payload to be rejected.'); - } catch (InvalidPayloadException) { - } - - $queue->pushRaw('{"id":"raw-after-failure","displayName":"Raw Job"}'); - - $payload = json_decode(Redis::connection('horizon')->hget('raw-after-failure', 'payload'), true); - $this->assertSame([], $payload['tags']); - } - public function testPendingJobsAreStoredWithTheirTags(): void { $id = Queue::push(new BasicJob); @@ -343,17 +320,6 @@ private function getQueueRedisKey(RedisQueue $queue, ?string $name = null): stri } } -class RedisQueueWithExposedLastPushed extends RedisQueue -{ - /** - * Set the job used to prepare the next payload. - */ - public function rememberLastPushed(object|string $job): void - { - $this->setLastPushed($job); - } -} - class AfterCommitHorizonJob extends BasicJob implements ShouldQueueAfterCommit { } diff --git a/tests/Integration/Horizon/Feature/RedisPayloadTest.php b/tests/Integration/Horizon/Feature/RedisPayloadTest.php index 03412879f..bf68140c9 100644 --- a/tests/Integration/Horizon/Feature/RedisPayloadTest.php +++ b/tests/Integration/Horizon/Feature/RedisPayloadTest.php @@ -65,6 +65,59 @@ public function testPreparationAndPublicationTimestampCanRunSeparately(): void $this->assertIsNumeric($payload->decoded['pushedAt']); } + public function testCombinedPreparationEncodesCustomizedMetadataOnce(): void + { + $payload = new class('{"id":"1"}') extends JobPayload { + public int $writes = 0; + + /** + * Count payload encoding operations. + */ + public function set(array $values): static + { + ++$this->writes; + + return parent::set($values); + } + + /** + * Customize the payload type. + */ + protected function determineType(mixed $job): string + { + return 'custom'; + } + + /** + * Customize the payload tags. + */ + protected function determineTags(mixed $job): array + { + return ['custom-tag']; + } + + /** + * Determine silencing from the customized tags. + */ + protected function shouldBeSilenced(mixed $job, array $tags): bool + { + return $tags === ['custom-tag']; + } + }; + + $payload->prepare(new StdClass); + + $this->assertSame(1, $payload->writes); + $this->assertSame('custom', $payload->decoded['type']); + $this->assertSame(['custom-tag'], $payload->decoded['tags']); + $this->assertTrue($payload->decoded['silenced']); + $this->assertIsNumeric($payload->decoded['pushedAt']); + $this->assertSame($payload->decoded, json_decode($payload->value, true)); + + $this->assertSame($payload, $payload->prepareForQueueing(null)); + $this->assertSame(1, $payload->writes); + } + public function testPreparingClassifiedPayloadWithoutOriginalJobPreservesMetadata(): void { $payload = new JobPayload(json_encode([ From 92b2c861ba15b618f44f7678c17173bd29c06d6d Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 20:48:16 +0000 Subject: [PATCH 09/33] perf(horizon): skip monitored-tag reads for untagged jobs Return immediately when the requested tag list is empty instead of loading the monitored tag set from Redis. Keep normal tag intersection behavior unchanged and cover both empty and populated requests. --- .../src/Repositories/RedisTagRepository.php | 4 ++++ tests/Horizon/Unit/RedisTagRepositoryTest.php | 21 +++++++++++++++++++ 2 files changed, 25 insertions(+) create mode 100644 tests/Horizon/Unit/RedisTagRepositoryTest.php diff --git a/src/horizon/src/Repositories/RedisTagRepository.php b/src/horizon/src/Repositories/RedisTagRepository.php index 4c47e802c..f2c49b177 100644 --- a/src/horizon/src/Repositories/RedisTagRepository.php +++ b/src/horizon/src/Repositories/RedisTagRepository.php @@ -35,6 +35,10 @@ public function monitoring(): array */ public function monitored(array $tags): array { + if ($tags === []) { + return []; + } + return array_intersect($tags, $this->monitoring()); } diff --git a/tests/Horizon/Unit/RedisTagRepositoryTest.php b/tests/Horizon/Unit/RedisTagRepositoryTest.php new file mode 100644 index 000000000..7ae401981 --- /dev/null +++ b/tests/Horizon/Unit/RedisTagRepositoryTest.php @@ -0,0 +1,21 @@ +shouldNotReceive('connection'); + + $this->assertSame([], (new RedisTagRepository($redis))->monitored([])); + } +} From c2e7eacd450671d9bd47f6fd2037d3885642d44e Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 20:48:30 +0000 Subject: [PATCH 10/33] docs(queue): record outbox checkpoint contracts and remaining work MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document exact resolved transaction identities for integrations and Horizon’s shared protected publication hooks. Keep additive outbox APIs out of the Laravel compatibility catalogue. Update the active implementation plan to the verified staging and transport design, including failover forwarding scope, inline lock ownership, finalized lifecycle payloads, atomic claim boundaries and single-query monitoring. Mark the implemented foundations clearly while retaining relay, operations, telemetry, live-engine coverage, canonical documentation and performance verification as unfinished work. --- ...pervel-transactional-queue-outbox-codex.md | 86 ++++++++++++------- src/docs/porting-from-laravel.md | 4 + 2 files changed, 57 insertions(+), 33 deletions(-) diff --git a/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md index 52d8f55eb..9a38dda18 100644 --- a/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md +++ b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md @@ -20,12 +20,18 @@ The result is a queue feature, not a generic event bus, inbox/deduplication syst ## Verified baseline +### Implementation status + +Implemented: framework prerequisites, public selection and wrapper propagation, schema/configuration, owner-aware staging and repository operations, and Redis/Horizon/database/SQS/pooled raw publication. It is not yet an end-to-end deployable outbox. + +Remaining work: relay and worker resource boundaries, operator commands, delivery events and telemetry; live-engine/concurrency and complete API-integration coverage; canonical documentation, approved performance checks and final feature verification. + ### Hypervel - `Queue::enqueueUsing()` is the common single-job seam. `enqueueNow()` guards payload finalization, `JobQueueing`, and the driver callback as one attempt; accepts dispatch locks only after storage/transport acceptance; and raises `JobQueued` afterwards. Non-cancellation attempt failures raise `JobQueueingFailed`, while `PendingDispatch` owns final cleanup before a queue accepts the job. - `DispatchLockContext` is the existing object-keyed `WeakMap` for dispatch-owned unique/debounce lock provenance. It carries the live cache repository needed for exact cleanup only while the job object remains live, exposes Laravel-compatible scalar unique metadata to payload hooks, and is cleared on acceptance/release. `Queueable::$uniqueLockOwner` is initialized to Laravel's empty-string default and owner-capable locks release through `UniqueLock::releaseOwned()`. - Database, Redis, and SQS have optimized `bulk()` implementations; the base implementation loops through `push()` / `later()`. `Bus::bulk()` groups jobs by queue connection and queue name. `Batch::add()` already calls queue `bulk()` inside the batch repository transaction. Database and Redis bulk paths already provide bounded, atomic storage/publication. -- `DatabaseTransactionsManager::callbackApplicableTransactions()` exposes coroutine-local transaction records. Its current base-name indexing conflates distinct role connections; complete the exact-owner prerequisite below before using these records to select outbox storage. Matching host/database configuration never proves atomicity. +- `DatabaseTransactionsManager::callbackApplicableTransactions()` exposes coroutine-local transaction records keyed by exact resolved connection identity, distinguishing base and role connections. Matching host/database configuration never proves atomicity. - `PdoConnection::lockForPopping()` and `maxBindings()` own live database capabilities and cache them only for the current physical session. MySQL/MariaDB/Vitess/PlanetScale, PostgreSQL, and SQLite supply their correct version-dependent values. `DatabaseQueue` already consumes both, and `Concerns\InsertsDatabaseRows` provides binding-safe, transactionally atomic chunked inserts. - A database queue insert already participates atomically when its queue database connection is the same active connection as the business transaction. Adding a second outbox table in that case would add latency without improving the guarantee. - `Queueable` already carries tri-state `afterCommit`, connection, queue, delay, chain, SQS message-group/deduplication, unique, and debounce state. `PendingDispatch`, `PendingBatch`, and `PendingChain` are the expected fluent surfaces. A valid batched job is only required to use `Batchable`, so batch-wide outbox state cannot assume that every job also uses `Queueable`. @@ -34,7 +40,7 @@ The result is a queue feature, not a generic event bus, inbox/deduplication syst - `ConnectionResolver::releaseIdleConnections()` can return idle sessions early, but preserves pinned and certain multi-server write sessions and does not cover all custom drivers or listener-owned resources. Keep separate short-lived relay phase coroutines for guaranteed cleanup; committing a transaction alone does not provide that guarantee. - SQS bulk delivery already chunks at 10 entries and the request byte cap, handles explicit per-entry success/failure, preserves FIFO options, and retains overflow pointers for ambiguous SDK failures. Reuse those rules rather than building a parallel SQS implementation. - `JobQueued::$id` already matches the Queue contract's `mixed` result, and raw Sync/Deferred/Background publication executes through each queue's intended timing behavior. -- Horizon already separates `JobPayload::prepareForQueueing()` from `markAsPushed()` while retaining `prepare()` as their composition. Raw retry preserves stored classification, and delayed publication stamps `pushedAt` at the actual push. Combined preparation currently encodes twice; remove that redundancy in the prerequisite below. +- Horizon separates `JobPayload::prepareForQueueing()` from `markAsPushed()` while retaining `prepare()` as their single-encoding composition. Raw retry preserves stored classification, and delayed publication stamps `pushedAt` at the actual push. - Generic Worker lifecycle callbacks and deferred signal callbacks already run through short-lived coroutine contexts, and worker delays use fakeable `Sleep`. The outbox worker must follow the same resource-ownership pattern. - Queue, Bus, and Horizon READMEs already follow the required order, and the application skeleton's queue configuration matches Foundation. Reuse the existing foundation instead of repeating completed cleanup. @@ -71,11 +77,14 @@ Primary upstream reading: Implement these at their owning boundary, before outbox integration. If already completed when implementation starts, verify their contracts and reuse them rather than repeating the work. -1. **Exact transaction identity.** Use the resolved connection object's `getNameWithReadWriteType()` consistently for transaction begin/commit/rollback/reset, applicable-transaction checks and connection-bound callbacks. Do not change `getName()` or derive identity from the caller's requested suffix: shared in-memory SQLite aliases return one owner object. Update `ManagesTransactions`, `Connection::resetTransactionState()`, Foundation testing's wrapping-transaction lookup and `RequestContextSynchronizer`, Pennant's transaction-record matching, and other consumers comparing record names. Distinct base/`::write` sessions must not consume each other's records or callbacks. **Owner approval before implementation:** role-qualified transaction-record names change values observable by extensions compared with Laravel's base-name records. Explain the correction and required exact-owner comparison in database compatibility documentation; preserve public signatures. Plan review does not replace approval for this or any other Laravel API departure. +1. **Exact transaction identity.** Use the resolved connection object's `getNameWithReadWriteType()` consistently for transaction begin/commit/rollback/reset, applicable-transaction checks and connection-bound callbacks. Do not change `getName()` or derive identity from the caller's requested suffix: shared in-memory SQLite aliases return one owner object. Update `ManagesTransactions`, `Connection::resetTransactionState()`, Foundation testing's wrapping-transaction lookup and `RequestContextSynchronizer`, Pennant's transaction-record matching, and other consumers comparing record names. Distinct base/`::write` sessions must not consume each other's records or callbacks. This owner-approved departure changes role-qualified transaction-record names observable by extensions compared with Laravel's base-name records. Explain the correction and required exact-owner comparison in database compatibility documentation; preserve public signatures. 2. **Context-safe dispatch-lock cleanup.** Add the supported boot registration method `DispatchLockContext::captureReleaseContextUsing(?callable $callback): void`, normalized to closures; registration appends, `null` clears. Its callbacks return a release wrapper or null, capturing only the context they need. Internal `captureReleaseContext(): ?Closure` composes wrappers in registration order, first outermost. Call it in `UniqueLock::acquireForDispatch()` and `DebounceLock::acquireForDispatch()` **before acquiring anything**, then pass its result into their internal registration methods. A throwing capture callback leaves no acquired lock/token. Unique and debounce acquisition are mutually exclusive, so capture once per acquisition. Records and snapshots retain the composed wrapper; release runs both existing owner-aware cleanup operations inside it, preserving existing error handling. Wrappers invoke the operation once and restore their prior context in `finally`. Without registrations return null immediately; without a captured wrapper run existing release code without allocating a release closure. Keep all record/capture operations internal, document the registration method's boot lifetime and callback contract in PHPDoc, and clear callbacks in the existing `flushState()`/test-subscriber path. Do not add a contextual-store interface, wrapper traversal, full coroutine-context snapshot, or cache implementation changes. 3. **Transactional storage rollback.** Immediately after a successful database-queue/outbox insert and before accepting dispatch locks, register an independent `DispatchLockContext::snapshot()` on the exact storage connection if its `transactionLevel() > 0`. Rollback calls `releaseSnapshot()` even after acceptance erases the live record. Apply to applicable single/bulk paths; after chunk-insertion savepoints complete, attach to the remaining enclosing transaction that can still undo the rows. Failed inserts and already-durable no-transaction staging register nothing. Existing deferred ownership remains intact; never release broker-accepted locks because a business transaction rolls back. Complete identity and capture-context fixes first. -4. **SQS batch lifecycle.** Move payload finalization and `JobQueueing` into the guarded unaccepted-attempt lifecycle in `sendBatchedMessages()`. Finish producer state for affected unaccepted messages on non-cancellation failure; preserve already confirmed chunks and never turn a later `JobQueued` listener error into rejection of accepted work. Follow the established Redis/database batch and cancellation behavior. -5. **Horizon combined preparation.** Make `JobPayload::prepare()` perform one encoding operation for classification plus `pushedAt`, preserving `prepareForQueueing()`, `markAsPushed()`, stored classification on raw retries, and supported public/protected customization. Do not alter ordinary publication timing. +4. **Batch acceptance and terminal notifications.** SQS deferred bulk must reuse `deferBatchEnqueueAfterCommit()`. Guard finalization, entry preparation, per-chunk `JobQueueing` and publication; accept every confirmed success before calling observers. Clean explicitly rejected overflow pointers before success notifications. Stop before the next chunk on a transport or notification failure, and fail only outstanding members. Across Redis/database/SQS, drain each bounded batch's terminal notifications despite ordinary listener failures, using a narrow protected `Queue::notifyBatchMembers(array $members, Closure ...$notifications): ?Throwable` helper. It returns the first ordinary notification error and immediately rethrows cancellation. Separate the Horizon post-push hook from `JobQueued` notifications so either ordinary failure cannot skip the other. Failure events carry the primary attempt error; rethrow the first listener error when one occurred, otherwise the primary error. Preserve confirmed acceptance and never emit failure for accepted jobs. Keep SQS's existing protected failure helper and per-attempted-chunk queueing events. +5. **Horizon preparation and acceptance.** Extract classification values into a protected `JobPayload` method shared by `prepare()` and `prepareForQueueing()`; `prepare()` sets them plus `pushedAt` in one encoding operation. Retain public split methods, stored classification on raw retries and Laravel's metadata customization methods. Move Redis's bulk preparation/post-push hooks to base Queue as `preparePayloadForPush()` / `handlePayloadPushed()`, used by single and optimized bulk paths. Preparation is guarded; the post-push hook runs only after dispatch-lock acceptance. On the single path, use local exception retention rather than batch arrays/closures: immediately rethrow cancellation, attempt `JobQueued` after an ordinary post-push error, then rethrow the first ordinary error. Horizon's stock monitoring listeners perform Redis I/O, so their failures must not turn successful storage into failed dispatch or release accepted locks. Horizon `push()` stores through a shared protected parent-Redis helper instead of invoking its public `pushRaw()` override; this subclass-routing departure is owner-approved. Base Redis routing stays unchanged. Horizon's inherited `later()` uses the shared hooks; external `pushRaw()` prepares with null and emits its own Pending/Pushed events. Remove the obsolete last-pushed context handoff and its state-leak test, not retain a compatibility no-op. Document the Horizon subclass routing change. + +6. **Broadcast public-property extraction.** Skip uninitialized public properties before reading them in `BroadcastEvent::getPayloadFromEvent()`. This supports untouched `Queueable` state and application-declared typed properties without changing initialized nulls, value formatting or `broadcastWith()` behavior. Do not add an outbox-specific property blacklist or reflection cache. +7. **Inline after-commit ownership.** When `deferBatchEnqueueAfterCommit()` has no applicable transaction for its selected connection, invoke the enqueue callback on the current queue and return without transferring dispatch-lock ownership, registering rollback callbacks or reacquiring a pooled queue. Inline failure belongs to the calling dispatch/failover attempt; only genuinely deferred callbacks own final cleanup. Keep the helper void and preserve local-driver return values. ### Outbox integration @@ -84,7 +93,7 @@ Keep the new mechanism under `Hypervel\Queue\Outbox`; use small integration poin - Bus/Foundation: add the marker, shared fluent state, pending-dispatch/batch methods, and wrapper-job propagation described below. - Queue: add selection and payload/option hooks, outbox services and commands, config, and the migration generator. Stage through the existing guarded enqueue lifecycle and reuse `DispatchLockContext`, `PdoConnection` capabilities, and `InsertsDatabaseRows`. - Queue drivers/proxies: implement the fast paths and raw-batch capability below; select/partition outbox work before SQS's direct FIFO-delay validation. Preserve direct validation, existing raw publication APIs, and direct overflow retention; isolate outbox overflow pointers by claim attempt. -- Horizon: classify staged payloads through the outbox-only hook and stamp them at relay publication through the existing split `JobPayload` lifecycle. The outbox path must not use Horizon's direct-push `CoroutineContext` handoff. +- Horizon: classify staged payloads through the outbox-only hook and stamp them at relay publication through the split `JobPayload` lifecycle. Direct publication uses the shared preparation/post-acceptance hooks, without a coroutine context handoff. - OpenTelemetry: consume guarded outbox delivery events without adding work to ordinary dispatch. - Foundation/application skeleton/docs: add matching outbox configuration and migrations, then add the canonical guide and concise links without revisiting completed baseline cleanup. - Doctor remains unchanged: do not copy runtime selection rules into a second setup detector or add relay-heartbeat machinery. Explicit setup, migrations, native storage failures, and outbox commands provide the required operating surface. @@ -118,7 +127,7 @@ public function withoutOutbox(): static; public function onOutboxDatabase(UnitEnum|string $connection): static; ``` -Framework code reads explicit state with `$job->viaOutbox ?? null` and `$job->outboxDatabase ?? null`. PHP serialization omits uninitialized typed properties, so ordinary jobs gain no payload fields or bytes. `onOutboxDatabase()` requires a non-empty resolved enum/string and also selects outbox delivery; `withoutOutbox()` unsets any selected database. Keep this state in `InteractsWithQueueOutbox`, used by `Queueable` and `Batchable`, so Laravel-compatible batch jobs that use only `Batchable` can participate. Do not add chain-specific outbox properties or `all*` methods: follow-up chain dispatch happens after the current handler completes, outside the business transaction, and the current queued job already supplies durable retry if that dispatch fails. Explicit Laravel-shaped properties are easier for jobs, wrappers, serialization, and agents to understand than a framework-only associative state bag. +Framework code reads explicit state with `$job->viaOutbox ?? null` and `$job->outboxDatabase ?? null`. PHP serialization omits uninitialized typed properties, so ordinary jobs gain no payload fields or bytes. `onOutboxDatabase()` requires a non-empty resolved enum/string and also selects outbox delivery; `withoutOutbox()` resets the selected database to null, preserving that explicit reset against batch defaults. Re-enabling with `viaOutbox()` then uses normal owner resolution. Keep this state in `InteractsWithQueueOutbox`, used by `Queueable` and `Batchable`, so Laravel-compatible batch jobs that use only `Batchable` can participate. Do not add chain-specific outbox properties or `all*` methods: follow-up chain dispatch happens after the current handler completes, outside the business transaction, and the current queued job already supplies durable retry if that dispatch fails. Explicit Laravel-shaped properties are easier for jobs, wrappers, serialization, and agents to understand than a framework-only associative state bag. Selection precedence is: @@ -142,13 +151,13 @@ Bus::batch($jobs) ->dispatch(); ``` -The dispatch return value for a staged single job is the finalized payload's job UUID, also indexed on the outbox row for operator lookup. Existing direct drivers retain their existing return values; bulk remains `null`. +The dispatch return value for a staged single job is the finalized payload's job UUID, also indexed on the outbox row for operator lookup. Staged-only bulk returns `null`; direct drivers retain their existing single and bulk return values. ### Laravel-compatible surface - Do not change signatures or ordinary semantics of `dispatch`, `push`, `later`, `bulk`, `afterCommit`, `beforeCommit`, batch, chain, Queue/Bus fakes, or the `Hypervel\Contracts\Queue\Queue` contract. `pushRaw()` remains a direct, non-staging escape hatch even when a connection default enables outbox; the relay uses the separate raw-batch SPI when available and `pushRaw()` as its generic fallback, avoiding recursion and preserving existing raw-payload behavior. - Add fluent methods to the same traits/builders that own their neighboring Laravel APIs; do not add an Outbox facade, transaction wrapper, alternate dispatcher, or environment-specific call site. -- Propagate outbox selection and the selected database from queued listeners, mailables, notifications, and broadcasts into their existing queue wrapper jobs with one assignment each beside the independent `afterCommit` propagation. Copy initialized tri-state state exactly; otherwise promote `ShouldQueueViaOutbox` to wrapper `viaOutbox = true`. Do not add producer-specific outbox APIs. +- Propagate outbox selection and the selected database from queued listeners, mailables, notifications, and broadcasts into their existing queue wrapper jobs beside the independent `afterCommit` propagation. Inline `isset($source->viaOutbox)` copies explicit true/false; otherwise promote `ShouldQueueViaOutbox` to wrapper `viaOutbox = true`. Copy a non-null owner with `isset($source->outboxDatabase)`. Null/absent selection falls through to the marker; untouched wrappers retain uninitialized fields. Keep batch-default initialization semantics separate. Do not add a shared public helper or producer-specific outbox APIs. - Keep application-visible names short and intention-revealing: `viaOutbox`, `withoutOutbox`, `onOutboxDatabase`, `ShouldQueueViaOutbox`, `queue:relay`, and the `queue:outbox` operator command family. - Add protected enqueue/destination/options hooks on the base queue and an optional raw-batch contract for optimized durable drivers. Database/local/failover queues override the smallest cohesive hook for their fast path; null already bypasses enqueueing. Do not build a central driver-name/type switch. Custom drivers retain existing behavior while base-lifecycle drivers gain staging and may opt into optimized publication. - Preserve Horizon's existing public `JobPayload::prepare($job)` method as the composition of its classification and actual-push operations. Use the narrower existing operations for outbox staging/publication without breaking the Laravel-compatible method. @@ -166,15 +175,15 @@ The dispatch return value for a staged single job is the finalized payload's job | Queue/Bus fake | Record the job in the fake only; never touch an outbox table | | Failover | Preserve normal failover, but defer the whole attempt until applicable transactions commit when outbox is selected; document that this degradation cannot provide crash durability | -Failover and outbox solve different policies, and failover may cross heterogeneous drivers whose wire payload/options are created only in the selected child. Do not invent a multi-destination outbox envelope. Instead, selected outbox behavior degrades to failover's existing all-applicable-transactions after-commit path, then performs ordinary child attempts. `FailoverQueue` always invokes children inside a coroutine-scoped outbox-suppression context, so a child's own connection default cannot stage successfully and prevent later fallbacks from being tried. The failover connection's explicit/marker/config selection is authoritative; child outbox defaults still apply when those children are used directly. The base selection path consults this context only after it would otherwise select outbox, so ordinary unselected queue dispatch pays no context lookup. +Failover and outbox solve different policies, and failover may cross heterogeneous drivers whose wire payload/options are created only in the selected child. Do not invent a multi-destination outbox envelope. Instead, selected outbox behavior degrades to failover's existing after-commit path, then performs ordinary child attempts. Scope a coroutine-local forwarding record to each job-bearing child call, suppressing outbox only for that matching job; exclude raw calls and restore before `QueueFailedOver` listeners. Clear forwarding state during synchronous payload execution so independent jobs, including same-handler string continuations, retain normal selection. The failover connection's explicit/marker/config selection is authoritative; child defaults still apply when used directly. -Driver switching never causes an exception merely because the selected driver is local, failover, or already durable. There are no environment checks; behavior follows runtime capability. Local/failover drivers register against the explicitly selected active transaction when supplied and reject only a false claim of atomic ownership (an explicit owner while transactions exist only on other connections); without an explicit owner they preserve their existing transaction-selection rules. +Driver switching never causes an exception merely because the selected driver is local, failover, or already durable. There are no environment checks; behavior follows runtime capability. Local/failover drivers register against the explicitly selected active transaction when supplied and reject only a false claim of atomic ownership (an explicit owner while transactions exist only on other connections); without an explicit owner they preserve their existing transaction-selection rules. Failover's explicit-owner callback enters the child-attempt loop directly with after-commit already satisfied; matching children must not revalidate the committed owner or defer onto an unrelated transaction. Ordinary all-transactions callbacks retain reentry for transactions opened by earlier callbacks. -A selected outbox keeps an SQS FIFO job staged until `deliver_at`, then publishes it without the forbidden per-message `DelaySeconds`. In `later()` and `bulk()`, select/partition outbox work before applying the existing FIFO checks to direct work. Capture `getQueueableOptions()` without a delay argument at staging; preserve its direct-call validation. No-transaction staging has the same delayed-publication behavior. +A selected outbox keeps an SQS FIFO job staged until `deliver_at`, then publishes it without the forbidden per-message `DelaySeconds`; any queue-level delay applies after publication. In `later()` and `bulk()`, select/partition outbox work before applying the existing FIFO checks to direct work. Capture `getQueueableOptions()` without a delay argument at staging; preserve its direct-call validation. No-transaction staging has the same delayed-publication behavior. A custom durable driver participates when it routes `push()` / `later()` / `bulk()` through the base enqueue hooks. The relay uses the optional raw-batch publication contract below when available and otherwise publishes due rows sequentially through the existing `pushRaw()` contract. Existing custom queues that override the public enqueue methods and bypass the base lifecycle retain their existing direct behavior; intercepting arbitrary overrides would require a permanent decorator and hot-path overhead for every queue operation. Document this extension contract instead of adding that machinery. -Evaluate fake/local/failover and already-atomic database fast paths before resolving outbox services or touching the outbox table. These paths need no outbox migration merely because a job carries the marker; local/failover drivers consult active transactions for their documented after-commit fallback. `NullQueue` already bypasses enqueueing and `PendingDispatch` releases its unaccepted locks; add no null-driver hook. +Evaluate fake/local/failover and already-atomic database fast paths before resolving outbox services or touching the outbox table. These paths need no outbox migration merely because a job carries the marker; local/failover drivers consult active transactions for their documented after-commit fallback. `NullQueue` bypasses single enqueueing and opts its inherited bulk path out of staging; `PendingDispatch` releases its unaccepted locks. ### Database ownership @@ -190,7 +199,7 @@ Configure each database source whose outbox table the relay drains; use configur ], ``` -Every shipped connection record includes `'outbox' => false`. Base `Queue::setConfig()` caches the optional, already-cast value in a typed flag, using `false` when an existing custom record omits it; `QueueManager` continues passing configuration once. On lazy service resolution, inject typed config values into their owning services through the provider, following the existing batch/failed-job providers; no `OutboxConfiguration` class. Validate a non-empty database-source list and positive `retry_after`. Native failures cover invalid table/connection configuration; do not add duplicate-name validation. Relay batch size is a worker option, not duplicate config. Retry delays are fixed at `OutboxRelay::RETRY_DELAYS = [1, 5, 15, 30, 60, 300, 600]`; staging chunks follow database binding limits, not another knob. +Every shipped connection record includes `'outbox' => false`. Base `Queue::setConfig()` caches the optional, already-cast value in a typed flag, using `false` when an existing custom record omits it; `QueueManager` continues passing configuration once. On lazy service resolution, inject typed config values into their owning services through the provider, following the existing batch/failed-job providers; no `OutboxConfiguration` class. Validate a non-empty database-source list and positive `retry_after`. Native failures cover invalid table/connection configuration; do not add duplicate-name validation. Relay batch size is a worker option, not duplicate config. Retry delays are fixed at `DatabaseOutboxRepository::RETRY_DELAYS = [1, 5, 15, 30, 60, 300, 600]`, beside the settlement code that applies them; staging chunks follow database binding limits, not another knob. Resolve the owning database as follows: @@ -229,8 +238,8 @@ Schema::create('{{table}}', function (Blueprint $table) { ``` - Bigint `id` keeps the hot index and range/order operations compact. `uuid` repeats the finalized payload's existing job UUID for indexed logs and operator commands; do not generate a second outbox identity. A fresh `lease_token` identifies only one claim attempt. -- All three time columns are integer Unix seconds supplied from the framework clock, following queue/cache operational deadline conventions. Bigint avoids PostgreSQL's signed 32-bit epoch limit. No database-clock default or timezone-dependent parsing: SQL session timezone and later `app.timezone` changes cannot move a staged instant. Keep message fields as `int`/`?int`, converting only for display. Preserve `InteractsWithTime::availableAt()`'s future ceil-to-second; already-due input becomes `deliver_at = null` with initial eligibility now. -- `deliver_at` is immutable intended delivery; `available_at` is next eligibility, including lease expiry and retry backoff. Optimized drivers compute non-negative remaining delay immediately before each transport chunk. Redis/database initially qualify now; standard SQS at `max(now, deliver_at - 900)`, with `DelaySeconds = min(900, max(0, remaining))`; FIFO and generic raw fallback qualify at `deliver_at ?? now`. Claim moves `available_at` to its lease deadline; retain the pre-claim value in the in-memory message as `eligibleAt` for ordering/latency. No separate reservation timestamp. +- All three time columns are integer Unix seconds supplied from the framework clock, following queue/cache operational deadline conventions. Bigint avoids PostgreSQL's signed 32-bit epoch limit. No database-clock default or timezone-dependent parsing: SQL session timezone and later `app.timezone` changes cannot move a staged instant. Keep message fields as `int`/`?int`, converting only for display. No timer (null or integer zero) stores `deliver_at = null`; a requested timer already due stores now. Preserve `InteractsWithTime::availableAt()`'s future ceil-to-second. A non-null timestamp preserves SQS's per-message timer override even after it expires. +- `deliver_at` is immutable intended delivery; `available_at` is next eligibility, including lease expiry and retry backoff. Optimized drivers compute non-negative remaining delay immediately before each transport chunk. Redis/database initially qualify now; standard SQS at `max(now, deliver_at - 900)`, with `DelaySeconds = min(900, max(0, remaining))` only when `deliver_at` is non-null. Otherwise omit it so the configured queue delay still applies. FIFO and generic raw fallback qualify at `deliver_at ?? now`; FIFO never sends `DelaySeconds`. Claim moves `available_at` to its lease deadline; retain the pre-claim value in the in-memory message as `eligibleAt` for ordering/latency. No separate reservation timestamp. - `queue` stores the unforwarded logical name with the connection's default expanded. Publication applies queue-name forwarding once through the normal storage path. Changing the default cannot reroute the row; changing a queue-name forward can. Connection forwarding is already resolved at staging and is not rerun. A queue-less custom driver may store null. Drain before changing captured connection routing or forwarding between incompatible FIFO/standard destinations. - `payload` is the finalized opaque envelope after payload hooks and `JobPayloadFinalizing`, not a job to redispatch through Bus. `longText` preserves its bytes; a database JSON type could normalize them. Adapters may add only delivery-time metadata such as Horizon's `pushedAt` or an SQS overflow pointer; SQL bytes, UUID and serialized command stay unchanged. Opaque `options` likewise uses text and is decoded only for publication. - `options` is null when empty, otherwise JSON containing only namespaced transport options needed after the object is gone, notably `{"transport":{"sqs":{}}}` for precomputed group/deduplication. Do not duplicate payload lock metadata or add an unused format version; migrate in-flight rows if a future format changes. Adapters use their own option members; native transport errors retain/back off rows. Do not add a driver-kind compatibility registry or business fields. @@ -253,6 +262,7 @@ src/bus/src/Batch.php src/bus/src/PendingBatch.php src/foundation/src/Bus/PendingDispatch.php src/queue/src/Concerns/InsertsDatabaseRows.php +src/queue/src/Concerns/StagesOutboxJobs.php src/queue/src/Outbox/ Concerns/PublishesOutboxMessagesSequentially.php DatabaseOutboxRepository.php @@ -302,13 +312,15 @@ Refactor the single and bulk enqueue paths so the cheap outbox-selection decisio 1. resolve/validate selection, database ownership, driver fast-path behavior, and logical destination before finalization; 2. retain dispatch ownership until storage succeeds; any preparatory snapshot is data only, not a registered rollback callback; 3. finalize the payload exactly once while the original job object exists, inside the guarded enqueue attempt; -4. run the narrow outbox-payload hook, decode and validate the finalized envelope and its UUID string consistently on every database engine, and compute absolute `deliver_at` plus target-specific opaque options in that same guarded attempt; +4. run the narrow outbox-payload hook into a separate stored-payload value, decode and validate that envelope and its UUID string consistently on every database engine, and compute absolute `deliver_at` plus target-specific opaque options in that same guarded attempt; 5. raise normal `JobQueueing`; 6. insert on the selected connection; immediately after success, register independent snapshot rollback cleanup only if that exact connection still has an active transaction; 7. let the existing enqueue lifecycle accept the dispatch locks and raise `JobQueued` with the job UUID after insertion, or raise `JobQueueingFailed` for every non-cancellation exception within the guarded attempt. The existing `DispatchLockContext` / `PendingDispatch` lifecycle owns failures before queue acceptance, including payload serialization and queue resolution. Reuse it rather than adding an outbox cleanup path. A failure before a valid payload exists rethrows without fabricating a queue lifecycle event; a failure inside the guarded enqueue attempt follows the existing `JobQueueingFailed` and cancellation rules. Do not report an accepted job as failed when a later `JobQueued` listener throws. +Generic lifecycle events always carry the same finalized payload string, before transport or outbox decoration. Keep the separately prepared stored payload for transport/storage and `handlePayloadPushed()`, without changing the lifecycle value. This preserves Sentry's exact-payload correlation and OTel's direct lookup. Single dispatch needs one extra local value; bulk members retain both values. SQS prepares stored bytes before byte-based chunking but keeps `JobQueueing` inside each attempted chunk. Outbox staging follows the same rule for `preparePayloadForOutbox()`; never substitute the stored Horizon envelope in its terminal events. + `JobQueued` means the configured logical queue accepted the job into its durable path; like a database-queue insert inside a caller transaction, that acceptance can still roll back with the transaction. It must not be delayed to another process. This preserves the original job object for listeners and completes OpenTelemetry producer state in the dispatching coroutine. Relay must not emit a second normal enqueue lifecycle. Provide additive protected queue hooks such as: @@ -333,10 +345,12 @@ protected function availableAtForOutbox( First-party drivers expand missing queues to their configured logical default without forwarding or deriving physical keys/URLs. The base options hook returns an empty array; initial availability is `deliver_at ?? now` because raw publication has no cross-driver delay contract. Redis/database become eligible immediately. SQS captures group/deduplication while the object exists, without relative `DelaySeconds`, and computes initial eligibility using the effective destination's standard/FIFO behavior. These inspections do not replace the stored logical name. At publication, normal driver routing resolves that name once. Register independent storage-rollback snapshots through the existing connection-aware transaction callbacks; do not reuse the live-record lookup callback after acceptance. -`Queue::setConfig()` caches the optional connection `outbox` boolean alongside the already retained immutable configuration; `QueueManager` already calls this once for every resolved concrete connection. Do not add another constructor argument to every driver/connector. Missing `outbox` remains `false` for existing custom configuration. Selection computes explicit state, marker, and the cached flag first; only a positive result checks the failover suppression context. `FailoverQueue` scopes that context with exception-safe restore around all child calls, including raw calls, so nested failover queues remain correct. +`Queue::setConfig()` caches the optional connection `outbox` boolean alongside the already retained immutable configuration; `QueueManager` already calls this once for every resolved concrete connection. Do not add another constructor argument to every driver/connector. Missing `outbox` remains `false` for existing custom configuration. Selection computes explicit state, marker, and the cached flag first; only a positive result checks the forwarding record. That record holds the job and an after-commit-handled boolean, set only by the explicit-owner callback. Base after-commit selection checks it only after its ordinary positive decision. Nested failover preserves a matching outer record. Keep the existing protected attempt method's signature; extract its child loop for direct and explicit-owner callback use. Base `Queue::bulk()` must preflight and partition outbox-selected jobs itself, stage its outbox partition with set-based owner-group inserts, then loop only the direct remainder. Beanstalkd and custom durable drivers that inherit the base method therefore get bulk staging without a driver override. Database, Redis, and SQS keep their optimized overrides but apply the same partition before their direct paths. +Keep selection and staging in `Concerns\StagesOutboxJobs`. Reuse Redis/database's protected `prepareBatchJobs()` shape as the base batch-preparation hook; SQS delegates to its existing `prepareBatchMessages()` and Beanstalkd supplies its normal payload-hook destination. This preserves each driver's payload hooks without adding a parallel serialization path. + Every implementation preserves order within each partition and groups staged rows by owning database. This prevents a later configuration error from partially staging an otherwise valid group. There is deliberately no ordering or atomicity guarantee between direct and outbox partitions, just as existing cross-connection bulk has none; callers requiring one atomic unit use one outbox owner. Finalize every owner group before its first insert, emit existing per-job queue lifecycle events, and use set-based storage. Each owner group is atomic across its insert chunks; distinct owner databases necessarily commit independently. A `PendingBatch` repository transaction provides one owner and rejects conflicting per-job choices. Preserve current `Bus::bulk()` semantics: it does not suddenly acquire unique/debounce locks or run `PreparesForDispatch`. ### Raw-batch publication SPI @@ -361,7 +375,7 @@ Implement the SPI where it provides a real batch/resource benefit: - database: one target-database transaction containing parameter-safe multi-row jobs-table inserts across logical queues, with each remaining delay converted to `available_at`; report success only after commit; - SQS: reuse the 10-entry and 1,048,576-byte request limits, FIFO fields, SDK retry assumptions, and per-entry response mapping; derive overflow pointers from job UUID plus lease token so separate publication attempts never share cleanup ownership; - pooled proxy: borrow once for the whole invocation, delegate when the borrowed queue implements the SPI, and otherwise invoke the shared sequential fallback on that same borrowed queue; never retain the lease; -- Horizon Redis: decorate the Redis batch path with `JobPending` before attempts and `JobPushed` only for accepted payloads. +- Horizon Redis: decorate the Redis batch path with `JobPending` before attempts and `JobPushed` only for accepted payloads. Record every confirmed destination's accepted IDs before draining post-push notifications. On an ordinary notification failure, stop before the next destination and return those accepted IDs plus the error; settlement deletes confirmed rows rather than republishing them. Do not add raw-delay methods to the general `Queue` contract. Custom durable drivers opt into staging through the documented base enqueue hooks; implementing the SPI is an optional publication optimization, and their connector remains otherwise unchanged. @@ -369,12 +383,12 @@ Do not add raw-delay methods to the general `Queue` contract. Custom durable dri ### Claim -- The concurrency-owning relay task calls claim through a fresh `Waiter` child with no copied request/transaction context. That child alone resolves the source database, commits the claim, returns immutable `OutboxMessage` values, and ends so `ConnectionResolver` releases the database lease before any broker work starts. +- The concurrency-owning relay task calls claim through a fresh `Waiter` child with no copied request/transaction context. That child alone resolves the source database, completes the claim, returns immutable `OutboxMessage` values, and ends so `ConnectionResolver` releases the database lease before any broker work starts. - Claim at most the validated worker batch limit eligible rows ordered by `available_at`, then `id`. -- Eligibility is `available_at <= now`. A claim sets one fresh UUID lease token, `available_at = now + retry_after`, and increments attempts in a short transaction. This moves live leases out of the hot eligibility range without a second timestamp/filter. Never hold SQL locks during broker calls. -- PostgreSQL: one CTE using `FOR UPDATE SKIP LOCKED` plus `UPDATE ... RETURNING`, returning the selected pre-update availability as `eligibleAt` with the updated token/attempt. Updating the indexed timestamp trades HOT-update eligibility for avoiding repeated live-lease scans; validate this tradeoff in the approved benchmark. +- Eligibility is `available_at <= now`. A claim atomically sets one fresh UUID lease token, `available_at = now + retry_after`, and increments attempts. This moves live leases out of the hot eligibility range without a second timestamp/filter. Use an explicit transaction only for the separate locking-select/update path; never hold SQL locks during broker calls. +- PostgreSQL: one autocommitted CTE using `FOR UPDATE SKIP LOCKED` plus `UPDATE ... RETURNING`, returning the selected pre-update availability as `eligibleAt` with the updated token/attempt. No extra BEGIN/COMMIT round trips. Updating the indexed timestamp trades HOT-update eligibility for avoiding repeated live-lease scans; validate this tradeoff in the approved benchmark. - Supported MySQL/MariaDB/Vitess: `SELECT ... FOR UPDATE SKIP LOCKED`, one guarded bulk update, and return the already selected rows with token/time/attempt fields reflected in memory rather than issuing another select. -- SQLite/engines without `SKIP LOCKED`: select an ordered, limited list of candidate IDs and their availability, perform one atomic eligibility-guarded token/expiry update restricted to those IDs, then load full rows restricted to those IDs bearing that token. Preserve original eligibility from the candidates. The second read distinguishes rows actually won; another claimant can cause an empty/partial result, not a double claim. SQLite serializes writers, so additional relays improve availability rather than write throughput. +- SQLite/engines without `SKIP LOCKED`: select an ordered, limited list of candidate IDs and their availability, perform one autocommitted atomic eligibility-guarded token/expiry update restricted to those IDs, then load full rows restricted to those IDs bearing that token. No encompassing transaction is needed: the token identifies this claim even if another claimant intervenes or reclaims an expired lease. Preserve original eligibility from the candidates. The second read distinguishes rows actually won; another claimant can cause an empty/partial result, not a double claim. SQLite serializes writers, so additional relays improve availability rather than write throughput. - Normalize every returned claim to its pre-claim `(eligibleAt, id)` order; `UPDATE ... RETURNING` and `WHERE IN` do not promise order, and the newly shared lease deadline is not the original eligibility. - Never return a connection, builder, cursor, lazy collection, or pool-backed object from the claim child. Materialize and validate the bounded result before it ends. - Reuse `PdoConnection::lockForPopping()` and `maxBindings()` for PDO-backed connections. Keep base connection types driver-neutral and follow DatabaseQueue's non-PDO lock/binding fallback; custom drivers are not required to expose PDO methods. Existing live-session capabilities cover engine variants; do not add another detector, contract or cache. @@ -407,7 +421,7 @@ Register these commands in `QueueServiceProvider`: - `queue:relay {database?}` — long-running relay plus `--once`, `--stop-when-empty`, `--batch=100`, `--sleep=1`, `--concurrency=1`, `--max-jobs=0`, `--max-time=0`, and `--memory=128`; validate batch 1–1,000 and positive sleep/concurrency. Sleep accepts fractions like `queue:work`. `--once` makes one synchronous round-robin pass, at most one bounded claim per configured/pinned database, with no empty/outage sleep. `--stop-when-empty` joins active tasks and exits after a complete source pass finds no eligible row; delayed/backoff rows do not keep it alive. `--max-jobs` counts claimed rows regardless of result; cap every final claim to the remaining allowance; - `queue:outbox {database?}` — pending-row diagnostics by configured database, ID, UUID, job display name read from the bounded payload result, target, attempts, lease state, age, and last error, ordered by ID and bounded by validated `--limit=100` (maximum 1,000), with `--json` matching `queue:failed`; otherwise visit all configured sources; never render payloads, transport options, context, or lock-owner tokens; -- `queue:outbox-monitor {database?}` — total, eligible, delayed, leased, and retrying counts plus the oldest currently eligible row's age, with explicit `--max` / `--age` alert thresholds and `--json` output for one/all configured databases; use one portable conditional-aggregate query plus one indexed `(available_at, id)` head lookup per database, and emit `OutboxBusy` like the existing `queue:monitor` command without another index or payload scan. Eligibility uses the claim predicate, and age starts at `available_at`, so intentionally delayed jobs do not create false alerts; +- `queue:outbox-monitor {database?}` — total, eligible, delayed, leased, and retrying counts plus the oldest currently eligible row's age, with explicit `--max` / `--age` alert thresholds and `--json` output for one/all configured databases; use one portable aggregate query with conditional counts and `MIN(CASE WHEN available_at <= ? THEN available_at END)` so counts and age share one read, and emit `OutboxBusy` like the existing `queue:monitor` command without another index or payload scan. Eligibility uses the claim predicate, and age starts at `available_at`, so intentionally delayed jobs do not create false alerts; - `queue:outbox-retry {uuid?*}` — targets only rows with `attempts > 0`, clears errors/leases, and makes selected rows (or explicit `all`) immediately available without erasing attempt history; accept `--database=` to pin a configured source, otherwise visit all sources. `all` snapshots each source's maximum ID among retryable rows and updates through that ceiling in bounded primary-key cursor chunks; an explicit UUID found on multiple unpinned sources fails rather than updating an ambiguous match. A never-attempted row waiting for its intended delivery time is not retryable; a previously failed row remains retryable regardless of its current backoff. Always reject live leases; - `queue:outbox-forget {uuid}` — confirmation, live-lease rejection, token-safe delete, and owner-matched unique-lock cleanup from the payload metadata; accept `--database=` to pin a configured source and fail rather than guess if an unpinned UUID exists on more than one source. There is no force path because an in-flight publication cannot be recalled; - `make:queue-outbox-table` / `queue:outbox-table` — migration generation matching existing queue table commands. @@ -429,9 +443,9 @@ Use the existing `JobPayload` lifecycle through its two semantic operations: - `prepareForQueueing($job)` while the object exists: type, tags, and silenced state; - `markAsPushed()` at actual Redis publication: `pushedAt` only. -Preserve `JobPayload::prepare($job)` as a single-pass combination of both operations. Direct immediate/delayed Horizon dispatch calls it once inside the actual publish callback, so after-commit timing is correct and the direct path gains no second JSON decode/encode. External `pushRaw()` uses the same single-pass method with `null`, preserving already-stored classification and refreshing only `pushedAt`. +Preserve `JobPayload::prepare($job)` as a single-pass combination of classification and timestamp values, not two encoding calls. Direct immediate/delayed Horizon dispatch calls it once through `preparePayloadForPush()` during the actual enqueue attempt, so after-commit timing stays correct. `handlePayloadPushed()` raises Horizon's post-storage event after acceptance. External `pushRaw()` remains stateless and uses the single-pass preparation method with `null`, preserving already-stored classification and refreshing only `pushedAt`. Base Redis's normal `pushRaw()` extension routing remains unchanged. -Add a narrow protected `preparePayloadForOutbox()` hook to base Queue that is called only after generic payload finalization on selected outbox paths and is otherwise a no-op. Horizon Redis overrides it to call `prepareForQueueing($job)` once before storage. Its raw-batch publication calls `markAsPushed()` once per payload immediately before Redis publication. Preserve job `uuid` / Horizon `id`; emit `JobPending` before each attempted publication and `JobPushed` only for explicit successes. The outbox path receives the original job through this hook; it must not reuse Horizon's direct-push coroutine handoff or classify with `null`. +Add a narrow protected `preparePayloadForOutbox()` hook to base Queue that is called only after generic payload finalization on selected outbox paths and is otherwise a no-op. Horizon Redis overrides it to call `prepareForQueueing($job)` once before storage, without `pushedAt` or `JobPending`. Its raw-batch publication calls `markAsPushed()` once per payload immediately before Redis publication. Preserve job `uuid` / Horizon `id`; emit `JobPending` before each attempted publication and `JobPushed` only for explicit successes. The staging hook receives the original job rather than classifying with `null`. ## Documentation cleanup @@ -440,28 +454,34 @@ Update in the same change so no stale behavior survives: - `src/docs/queue-outbox.md`: canonical setup, configuration, API precedence, transaction ownership, driver/custom-driver matrix, bulk/batches, `dispatchSync`/after-response limitations, delays, UUIDs, connection capture versus publication-time queue forwarding, deliberate driver changes, at-least-once/idempotency and best-effort dispatch ordering, SQS overflow attempt ownership/orphans, worker deployment/pool sizing, operations, Horizon and database fast paths; - `src/docs/documentation.md`: add the new guide to navigation; - `src/docs/queues.md`: add only a concise transactional-outbox introduction, when it is useful, one basic `viaOutbox()` example, and a link to `queue-outbox.md`; do not duplicate its setup, guarantees, driver matrix, or operations reference; -- `src/docs/events.md`, `mail.md`, `notifications.md`, and `broadcasting.md`: beside their existing after-commit guidance, add one concise example/cross-link showing that the producer's marker or fluent selection is propagated to its wrapper job; link to `queue-outbox.md` for guarantees rather than duplicating them; +- `src/docs/events.md`, `mail.md`, `notifications.md`, and `broadcasting.md`: beside their existing after-commit guidance, add one concise example/cross-link showing producer selection propagating to its wrapper job. Queued listeners are inspected without constructing them: use the marker or public property defaults, not constructor-fluent calls. When using `Queueable`, prefer the marker rather than incompatibly redeclaring its properties. Application-constructed producers can use the fluent methods. Link to `queue-outbox.md` for guarantees rather than duplicating them; - `src/docs/porting-from-laravel.md` and database README: explain exact resolved transaction identity for extensions observing role-qualified records. Do not add outbox catalogue entries: its opt-in APIs require no Laravel adaptation. Preserve Queue/Bus/Horizon README structure without adding additive-feature differences; - `/home/binaryfire/workspace/contrib/hypervel/hypervel`: match Foundation's outbox configuration and the Testbench schema in the next application migration; - configuration comments and migration command help: exact guarantees and required worker/table setup. +Clarify in listener/notification guidance that `viaOutbox()` is a fluent setter, not a getter hook like `viaConnection()` or `viaQueues()`. + Never imply exactly-once delivery, broker participation in the SQL transaction, or durable behavior for local drivers. Remove a comment only when the outbox implementation makes it stale; final comments explain only non-obvious invariants. ## Testing plan ### Framework prerequisites +- Broadcast extraction skips uninitialized Queueable/application properties while preserving initialized nulls and formatted values; retain existing custom-payload coverage. - Distinct base/`::write` sessions with simultaneous transactions and independent commit/rollback callbacks; nested savepoints, shared in-memory SQLite aliases, Foundation wrapping/context synchronization and Pennant rollback invalidation use resolved ownership. Do not use shared in-memory SQLite as proof of separate-session behavior. - Context-dependent cache fixture backed by `WorkerArrayStore`: acquire in context A, release record/snapshot in B, remove A's key and preserve B's same logical key. Cover unique and debounce separately, with a second registered wrapper proving composition order and restoration. A throwing capture callback acquires neither lock nor token. One real CoroutineQueue closing-timer regression covers release outside the acquiring coroutine. No repeated tests per caller or cache wrapper: the hook does not branch on wrapper types. - DatabaseQueue storage rollback releases its snapshot after acceptance, including nested rollback/deadlock retry; broker-accepted jobs are not released by unrelated rollback. Preserve existing deferred ownership coverage. -- SQS later-finalizer/queueing-listener failures finish unaccepted producer state without falsifying confirmed chunks; preserve cancellation. Reuse/extend affected existing tests. -- Horizon combined preparation encodes once and preserves classification/pushed timing/customization. Run the existing direct/retry/delayed tests; do not duplicate them under outbox names. +- With no applicable transaction, a local child failing before acceptance retains dispatch-lock ownership for failover's next child; local after-commit return values remain unchanged. +- SQS later-finalizer/queueing-listener failures finish unaccepted producer state without falsifying confirmed chunks; preserve cancellation and per-chunk queueing order. Custom job `queue()` dispatch through SQS bulk retains/delegates its acquired locks and releases only unaccepted work. Success-listener failure preserves all accepted entries, cleans explicit-reject pointers and prevents later sends. Reuse/extend affected tests. +- Every optimized batch driver drains successful terminal notifications despite an ordinary listener failure; cover failure-notification draining once at its shared boundary. Cancellation remains immediate. Do not duplicate these tests as a telemetry matrix. +- Horizon combined preparation encodes once and preserves classification/pushed timing/customization. A single/delayed post-push listener error retains the accepted lock, attempts `JobQueued` and emits no `JobQueueingFailed`; bulk also attempts that member's `JobQueued`. Compare exact generic lifecycle payload strings for direct/delayed/bulk and failed bulk to prevent correlation drift. Reuse existing direct/retry/delayed tests and remove only the obsolete last-pushed state test when its mechanism is removed. ### Contract and routing tests - `Queueable`/`Batchable` shared fluent state, pending dispatch/batch APIs, jobs that use only `Batchable`, jobs added to an existing batch, explicit per-job overrides, and the absence of chain-specific propagation/API state; -- precedence across explicit true/false, marker, and connection default; `beforeCommit`/`afterCommit` remain independent direct-publish controls; wrapper jobs preserve marker/explicit outbox state; and `dispatchSync`/`dispatchAfterResponse` retain their existing local/deferred semantics rather than pretending to provide pre-deferral durability; -- a custom durable transport without the raw-batch SPI stages normally and relays through the sequential raw fallback; failover selected by explicit state, marker, or its own connection default degrades to its existing after-commit attempt without schema access, child defaults are suppressed so fallbacks still run, nested suppression restores correctly, and a custom queue that bypasses the base lifecycle remains direct and is documented as not outbox-capable; preflight failure releases only locks acquired by the current dispatch; +- precedence across explicit true/false, marker, and connection default; `beforeCommit`/`afterCommit` remain independent direct-publish controls; wrapper jobs preserve explicit false over the marker and null falling through to the marker; and `dispatchSync`/`dispatchAfterResponse` retain their existing local/deferred semantics rather than pretending to provide pre-deferral durability; +- a custom durable transport without the raw-batch SPI stages normally and relays through the sequential raw fallback; a custom queue that bypasses the base lifecycle remains direct and is documented as not outbox-capable; preflight failure releases only locks acquired by the current dispatch; +- failover selection degrades to after-commit without schema access; child defaults are suppressed only for the forwarded job and restored afterwards. Nested failover honors explicit owner B's commit while A remains open, including lock ownership across a failed child; ordinary reentry remains intact. Independent jobs dispatched by a sync handler, including same-handler string continuations, retain outbox selection; - `pushRaw()` remains direct even on a connection whose default enables outbox; - local/null/fake/database fast paths never query the outbox; exact local transaction selection is honored; expanded defaults stay fixed and queue-name forwarding runs once, including `A → B`, `B → C` and a changed forward before publication; - staged single dispatch returns the same job UUID indexed on its row; direct driver return values remain unchanged. @@ -480,7 +500,7 @@ Never imply exactly-once delivery, broker participation in the SQL transaction, - queued jobs, closures, listeners, mailables, notifications, broadcasts, encrypted jobs, serialized models, custom payload hooks, OpenTelemetry propagation, unique jobs, debounced jobs, delayed jobs, batches, nested batched chains, and ordinary chains; wrapper producers propagate marker/explicit selection and database without changing their public APIs, while absent source state leaves the wrapper's typed fields uninitialized; - payload creation/finalization occurs once; malformed final envelopes or missing/invalid UUIDs fail uniformly before storage on every database engine; an outbox bulk finalizer failure leaves no earlier producer state open; opaque payload bytes remain unchanged in SQL and UUID/command bytes remain stable over relay retries apart from documented delivery-time transport metadata; -- SQS FIFO group/deduplication callback is evaluated once at staging; standard delays longer than 900 seconds wait until the broker window, FIFO delays wait until due, relay latency/backoff never rewrites immutable `deliver_at`, and publication after that instant uses zero additional delay; +- SQS FIFO group/deduplication callback is evaluated once at staging; standard delays longer than 900 seconds wait until the broker window, FIFO delays wait until due, and relay latency/backoff never rewrites immutable `deliver_at`. Parameterized staging/publication cases distinguish no timer from an already-due requested timer: standard publication omits `DelaySeconds` for null and uses zero after a requested timer expires; FIFO always omits it; - `Bus::bulk()` uses bounded multi-row inserts and preserves its existing lack of unique/debounce/preparation side effects; Beanstalkd and a custom durable driver inheriting base `Queue::bulk()` each stage one set-based insert per owner group rather than one insert per job, while their direct partitions retain the base loop; - fake assertions work without schema setup or outbox service resolution; an unselected job does not serialize the uninitialized outbox state. @@ -517,7 +537,7 @@ Place engine-specific cases under `tests/Integration/Queue/Database/{Postgres|My - Pin direct dispatch before/after query count, outbox service-resolution count, serialized fields, and payload transformations in automated tests: all remain unchanged when outbox is unselected. Use a repeatable local benchmark to measure the minimal cached job/config selection checks and investigate any material CPU or allocation regression; do not place unstable wall-time thresholds in CI. - Obtain the owner's confirmation that the machine is idle before any benchmark. Benchmark single and 1k/10k staging, lease claims, Redis Lua batches, database inserts and SQS request construction; compare query/network counts and allocation as well as throughput. -- Verify one insert per normal bulk chunk, one short claim transaction, publication using existing destination/protocol batches, and at most one accepted delete, deferred release and failed-row release per non-empty binding-safe settlement subset. Record peak borrowed source/transport connections and prove no lease spans unrelated work or sleep. +- Verify one insert per normal bulk chunk, claim transaction control only for the separate locking-select/update path, publication using existing destination/protocol batches, and at most one accepted delete, deferred release and failed-row release per non-empty binding-safe settlement subset. Record peak borrowed source/transport connections and prove no lease spans unrelated work or sleep. - Profile PostgreSQL/MySQL plans with representative pending/live-leased/retrying backlogs and concurrency. Compare the indexed availability/lease update against the prior separate-reservation predicate, including PostgreSQL index-write/HOT costs. Confirm the chosen design with evidence; resolve a substantive contrary result before changing the plan. Do not add indexes or tune engine settings speculatively. Run focused files immediately as they are added, then affected package suites, cross-package integration suites, analysis, and the repository checkpoint: diff --git a/src/docs/porting-from-laravel.md b/src/docs/porting-from-laravel.md index d4b30c7bd..c8958e163 100644 --- a/src/docs/porting-from-laravel.md +++ b/src/docs/porting-from-laravel.md @@ -803,6 +803,8 @@ Framework operations release idle database sessions at the boundaries listed in Do not manually commit or roll back the transaction owned by a `DB::transaction()` closure; Hypervel throws when that closure returns. Let `transaction()` manage completion, or use the [manual transaction API](/docs/{{version}}/database#manually-using-transactions) separately. Balanced nested transactions remain supported. +Extensions inspecting transaction records or registering callbacks directly with `DatabaseTransactionsManager` must use the resolved connection's `getNameWithReadWriteType()`, not its base `getName()`. Base, `::read` and `::write` sessions can own independent transactions. Resolve the connection first: shared in-memory SQLite aliases return one owner and therefore one transaction identity. Ordinary `afterCommit()` and `afterRollBack()` calls already select the correct owner. + When a package constructs `DatabaseStore`, `DatabaseSessionHandler`, `DatabaseQueue`, or `DatabaseBatchRepository` directly, pass the database connection resolver and configured connection name instead of retaining a resolved connection. Framework-configured drivers already use this form. Laravel's base `Connection` class exposes PDO methods. Hypervel's base `Connection` is driver-neutral, while its built-in SQL connections extend `PdoConnection`. Ported code that calls `getPdo`, `getReadPdo`, or another PDO-specific method should accept or narrow to `PdoConnection`. See [extending database connections](/docs/{{version}}/database#extending-database-connections) when porting a custom driver. @@ -859,6 +861,8 @@ Hypervel stores job batches in a relational database. Laravel's DynamoDB batch r With `Worker::$killOnTimeout = false`, a timeout interrupts job code with `Swoole\Coroutine\CanceledException`; catching `TimeoutExceededException` inside the job does not catch it. Use `finally` for cleanup and rethrow cancellation if caught. Failure callbacks and worker reporting still receive `TimeoutExceededException`. See [worker timeouts](/docs/{{version}}/queues#worker-timeouts). +Custom Horizon `RedisQueue` subclasses must not rely on `push()` calling their `pushRaw()` override. Use the protected `preparePayloadForPush()` and `handlePayloadPushed()` hooks for normal publication, with post-push work running after storage is accepted. Direct raw publication remains available through `pushRaw()`. + When a Laravel package offers optional support for an unsupported driver, remove that integration from the Hypervel port unless the package can safely provide it through a separate optional dependency. From 4af33e2b8b2a0a403c1450b1099165f249f40660 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 21:56:07 +0000 Subject: [PATCH 11/33] fix(cache): preserve named repository identity through custom and memo stores Retain the configured store name when a custom creator returns an unnamed repository, without overriding an explicit identity or naming ad hoc repositories. Carry the underlying name through memoization while preserving event suppression. This lets unique-job cleanup resolve the owning cache store when queued model restoration fails. Cover custom, memoized and memo-tagged repositories, explicit-name preservation, duplicate event suppression and the missing-model cleanup path. --- src/cache/src/CacheManager.php | 11 +++++- src/cache/src/Repository.php | 15 ++++++++ tests/Cache/CacheManagerTest.php | 43 +++++++++++++++++++++ tests/Integration/Queue/UniqueJobTest.php | 46 +++++++++++++++++++++-- 4 files changed, 110 insertions(+), 5 deletions(-) diff --git a/src/cache/src/CacheManager.php b/src/cache/src/CacheManager.php index 6c3e9aca4..1c40928dd 100644 --- a/src/cache/src/CacheManager.php +++ b/src/cache/src/CacheManager.php @@ -125,7 +125,7 @@ protected function createMemoizedRepository(string $driver): CacheRepository $repository = $this->repository( new MemoizedStore($driver, $store), - ['events' => false] + ['store' => $store->getName(), 'events' => false] ); return $isSpy ? Mockery::spy($repository) : $repository; @@ -146,7 +146,14 @@ public function resolve(string $name): CacheRepository $config = Arr::add($config, 'store', $name); - return $this->build($config); + $repository = $this->build($config); + + // Custom creators may wrap a store without passing its configuration. + if ($repository instanceof Repository && $repository->getName() === null) { + $repository->setName($config['store']); + } + + return $repository; } /** diff --git a/src/cache/src/Repository.php b/src/cache/src/Repository.php index 02857160a..304bc805a 100644 --- a/src/cache/src/Repository.php +++ b/src/cache/src/Repository.php @@ -1092,6 +1092,21 @@ public function getName(): ?string return $this->config['store'] ?? null; } + /** + * Set the cache store name before sharing the repository. + * + * Changing a shared repository's name affects cache events + * and queued unique-lock cleanup in every coroutine using that repository. + * + * @internal + */ + public function setName(string $name): static + { + $this->config['store'] = $name; + + return $this; + } + /** * Determine if a cached value exists. * diff --git a/tests/Cache/CacheManagerTest.php b/tests/Cache/CacheManagerTest.php index 819814af3..d256c8ca4 100644 --- a/tests/Cache/CacheManagerTest.php +++ b/tests/Cache/CacheManagerTest.php @@ -146,6 +146,24 @@ public function testItCanBuildRepositories(): void $this->assertInstanceOf(NullStore::class, $nullCache->getStore()); } + public function testCustomRepositoriesReceiveTheirConfiguredNameWithoutReplacingAnExplicitName(): void + { + $manager = new CacheManager($this->getApp([ + 'cache' => ['stores' => [ + 'custom' => ['driver' => 'custom'], + 'alias' => ['driver' => 'named'], + ]], + ])); + $manager->extend('custom', fn (): Repository => $manager->repository(new WorkerArrayStore)); + $named = new Repository(new WorkerArrayStore, ['store' => 'original']); + $manager->extend('named', static fn (): Repository => $named); + + $this->assertSame('custom', $manager->store('custom')->getName()); + $this->assertSame($named, $manager->store('alias')); + $this->assertSame('original', $named->getName()); + $this->assertNull($manager->repository(new WorkerArrayStore)->getName()); + } + public function testFailoverDriverPassesItsConfiguredNameToFailoverEvents(): void { $app = $this->getApp([ @@ -744,6 +762,31 @@ public function testForgetDriverForgetsTheCallingCoroutinesMemoizedRepository(): $this->assertNotSame($firstMemoizedStore->getInnerStore(), $secondMemoizedStore->getInnerStore()); } + public function testMemoizedRepositoriesPreserveStoreIdentityAndSuppressDuplicateEvents(): void + { + $app = $this->getApp(['cache' => ['stores' => ['named' => ['driver' => 'worker-array']]]]); + $events = new Event($app); + $app->instance(Dispatcher::class, $events); + $manager = new CacheManager($app); + $eventsSeen = []; + $events->listen('Hypervel\Cache\Events\*', static function (string $event, array $payload) use (&$eventsSeen): void { + $eventsSeen[] = $payload[0]->storeName; + }); + $memo = $manager->memo('named'); + + $this->assertSame('named', $memo->getName()); + $this->assertSame('named', $memo->tags('orders')->getName()); + $this->assertNull($memo->getEventDispatcher()); + $memo->put('key', 'value', 60); + $memo->get('key'); + $afterFirstRead = count($eventsSeen); + $memo->get('key'); + + $this->assertNotEmpty($eventsSeen); + $this->assertSame(['named'], array_values(array_unique($eventsSeen))); + $this->assertCount($afterFirstRead, $eventsSeen); + } + // REMOVED: CacheApcStoreTest, CacheDynamoDbStoreTest, CacheMemcachedConnectorTest, // CacheMemcachedStoreTest and their integration tests; these drivers are unsupported. diff --git a/tests/Integration/Queue/UniqueJobTest.php b/tests/Integration/Queue/UniqueJobTest.php index 966b71c95..7529f6a15 100644 --- a/tests/Integration/Queue/UniqueJobTest.php +++ b/tests/Integration/Queue/UniqueJobTest.php @@ -7,6 +7,8 @@ use Exception; use Hypervel\Bus\Queueable; use Hypervel\Bus\UniqueLock; +use Hypervel\Cache\Repository as CacheRepository; +use Hypervel\Cache\WorkerArrayStore; use Hypervel\Container\Container; use Hypervel\Contracts\Cache\Repository as Cache; use Hypervel\Contracts\Foundation\Application as ApplicationContract; @@ -26,6 +28,7 @@ use Hypervel\Testbench\Attributes\WithMigration; use Hypervel\Testbench\Factories\UserFactory; use Hypervel\Tests\Integration\Queue\QueueTestCase; +use PHPUnit\Framework\Attributes\DataProvider; #[WithMigration] #[WithMigration('cache')] @@ -41,6 +44,7 @@ protected function defineEnvironment(ApplicationContract $app): void $config = $app->make('config'); $config->set('cache.default', 'database'); + $config->set('cache.stores.custom_unique', ['driver' => 'custom_unique']); $config->set('queue.default', env('QUEUE_CONNECTION', 'database')); } @@ -212,13 +216,20 @@ public function testRetryOfOwnerlessUniqueUntilProcessingJobDoesNotReleaseSubseq $this->assertFalse($this->app->get(Cache::class)->lock($this->getLockKey($job), 10)->get()); } - public function testLockIsReleasedOnModelNotFoundException(): void + #[DataProvider('missingModelCacheRepositories')] + public function testLockIsReleasedOnModelNotFoundException(string $repository): void { UniqueTestSerializesModelsJob::$handled = false; + $manager = $this->app->make('cache'); + $manager->extend('custom_unique', static fn (): CacheRepository => $manager->repository(new WorkerArrayStore)); + /** @var User $user */ $user = UserFactory::new()->create(); - $job = new UniqueTestSerializesModelsJob($user); + $job = $repository === 'default' + ? new UniqueTestSerializesModelsJob($user) + : new UniqueViaSerializesModelsJob($user, $repository); + $cache = $repository === 'default' ? $this->app->get(Cache::class) : $job->uniqueVia(); $this->expectException(ModelNotFoundException::class); @@ -230,10 +241,18 @@ public function testLockIsReleasedOnModelNotFoundException(): void } finally { $this->assertFalse($job::$handled); $this->assertModelMissing($user); - $this->assertTrue($this->app->get(Cache::class)->lock($this->getLockKey($job), 10)->get()); + $this->assertTrue($cache->lock($this->getLockKey($job), 10)->get()); } } + /** + * Exercise named repositories whose identity must survive queued model restoration failure. + */ + public static function missingModelCacheRepositories(): array + { + return [['default'], ['memo'], ['custom_unique']]; + } + public function testModelNotFoundExceptionDoesNotReleaseSubsequentLock(): void { $this->markTestSkippedWhenUsingSyncQueueDriver(); @@ -499,6 +518,27 @@ public function __construct(public User $user) } } +class UniqueViaSerializesModelsJob extends UniqueTestSerializesModelsJob +{ + /** + * Create a unique job using the selected named cache wrapper. + */ + public function __construct(User $user, public string $repository) + { + parent::__construct($user); + } + + /** + * Resolve the cache repository that owns this job's unique lock. + */ + public function uniqueVia(): Cache + { + $manager = Container::getInstance()->make('cache'); + + return $this->repository === 'memo' ? $manager->memo('database') : $manager->store($this->repository); + } +} + class UniqueViaJob extends UniqueTestJob { public function uniqueVia(): Cache From d4413732f9d4f9623e87a6dc9c6f86ade3938ab8 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 21:56:17 +0000 Subject: [PATCH 12/33] feat(queue): relay outbox batches through bounded resource lifetimes Separate claims, connection-group publication, settlement and observers into fresh child coroutines. Join cleanup before advancing or propagating cancellation so blocked broker operations never retain a source database connection. Validate accepted and deferred row IDs without imposing array-key shape, settle only confirmed outcomes, and report publication and settlement failures independently. Observer failures cannot republish settled rows or throttle healthy delivery; reporter failures propagate. Add guarded batch delivery events and focused result, partial-failure, cancellation and observer tests. Exercise real single-slot database and queue pools to prove connection availability during blocked publication and cleanup after cancellation. --- .../src/Queue/PublishesOutboxMessages.php | 1 + src/queue/src/Events/OutboxBatchDelivered.php | 20 ++ .../src/Events/OutboxBatchDeliveryFailed.php | 24 ++ src/queue/src/Outbox/OutboxBatchResult.php | 4 +- src/queue/src/Outbox/OutboxRelay.php | 228 ++++++++++++++ .../Sqlite/WorkerResourceLifetimeTest.php | 117 +++++++ tests/Queue/Outbox/OutboxRelayTest.php | 290 ++++++++++++++++++ 7 files changed, 682 insertions(+), 2 deletions(-) create mode 100644 src/queue/src/Events/OutboxBatchDelivered.php create mode 100644 src/queue/src/Events/OutboxBatchDeliveryFailed.php create mode 100644 src/queue/src/Outbox/OutboxRelay.php create mode 100644 tests/Queue/Outbox/OutboxRelayTest.php diff --git a/src/contracts/src/Queue/PublishesOutboxMessages.php b/src/contracts/src/Queue/PublishesOutboxMessages.php index 5b4c63157..35d3c1b63 100644 --- a/src/contracts/src/Queue/PublishesOutboxMessages.php +++ b/src/contracts/src/Queue/PublishesOutboxMessages.php @@ -11,6 +11,7 @@ interface PublishesOutboxMessages { /** * Publish stored messages and identify confirmed acceptance or deferral. + * Only messages with a delivery time may be deferred; they become eligible again at that time. * * @param list $messages */ diff --git a/src/queue/src/Events/OutboxBatchDelivered.php b/src/queue/src/Events/OutboxBatchDelivered.php new file mode 100644 index 000000000..2d55933b5 --- /dev/null +++ b/src/queue/src/Events/OutboxBatchDelivered.php @@ -0,0 +1,20 @@ + $accepted - * @param list $deferred + * @param array $accepted + * @param array $deferred */ public function __construct( public array $accepted = [], diff --git a/src/queue/src/Outbox/OutboxRelay.php b/src/queue/src/Outbox/OutboxRelay.php new file mode 100644 index 000000000..9659b1493 --- /dev/null +++ b/src/queue/src/Outbox/OutboxRelay.php @@ -0,0 +1,228 @@ + + */ + public function claim(string $database, int $limit): array + { + return $this->inFreshCoroutine(fn (): array => $this->repository->claim($database, $limit)); + } + + /** + * Publish and settle a claimed batch, returning whether it completed without failure. + * + * @param list $messages + */ + public function deliver(string $database, array $messages): bool + { + $groups = []; + + foreach ($messages as $message) { + $groups[$message->connection][] = $message; + } + + $outcomes = []; + $successful = true; + + foreach ($groups as $group) { + $connection = $group[0]->connection; + $startedAt = hrtime(true); + $publishedAt = $this->currentTime(); + $result = $this->inFreshCoroutine(function () use ($connection, $group): OutboxBatchResult { + try { + $queue = $this->queues->connection($connection); + + return $queue instanceof PublishesOutboxMessages + ? $queue->publishOutboxMessages($group) + : $this->publishOutboxMessagesSequentially($queue, $group); + } catch (CanceledException|ChildCancellationException $exception) { + throw $exception; + } catch (Throwable $exception) { + return new OutboxBatchResult(failure: $exception); + } + }); + $duration = (hrtime(true) - $startedAt) / 1_000_000_000; + $result = $this->validateResult($group, $result); + $settlementFailure = $this->inFreshCoroutine(function () use ($database, $group, $result): ?Throwable { + try { + if ($this->repository->settle($database, $group, $result) !== count($group)) { + return new RuntimeException('Outbox lease ownership changed before the batch could be fully settled.'); + } + + return null; + } catch (CanceledException|ChildCancellationException $exception) { + throw $exception; + } catch (Throwable $exception) { + return $exception; + } + }); + + $successful = $successful && $result->failure === null && $settlementFailure === null; + $outcomes[] = [$connection, $group, $result, $settlementFailure, $duration, $publishedAt]; + } + + // Observers cannot prevent another connection's committed work from being delivered. + foreach ($outcomes as [$connection, $group, $result, $settlementFailure, $duration, $publishedAt]) { + $this->reportOutcome($database, $connection, $group, $result, $settlementFailure, $duration, $publishedAt); + } + + return $successful; + } + + /** + * Reject driver results that cannot safely identify the rows to settle. + * + * @param non-empty-list $messages + */ + protected function validateResult(array $messages, OutboxBatchResult $result): OutboxBatchResult + { + $input = []; + + foreach ($messages as $message) { + $input[$message->id] = $message; + } + + $seen = []; + $valid = true; + + foreach ([$result->accepted, $result->deferred] as $partition => $ids) { + foreach ($ids as $id) { + if (! is_int($id) || ! isset($input[$id]) || isset($seen[$id]) + || ($partition === 1 && $input[$id]->deliverAt === null)) { + $valid = false; + break 2; + } + + $seen[$id] = true; + } + } + + if (! $valid || ($result->failure === null && count($seen) !== count($messages))) { + return new OutboxBatchResult(failure: new LogicException( + 'Outbox publication must return disjoint input row IDs, defer only messages with a delivery time, and account for every row unless publication failed.', + previous: $result->failure, + )); + } + + return $result; + } + + /** + * Notify listeners and report failures after publication and settlement have released their resources. + * + * @param non-empty-list $messages + */ + protected function reportOutcome( + string $database, + string $connection, + array $messages, + OutboxBatchResult $result, + ?Throwable $settlementFailure, + float $duration, + int $publishedAt, + ): void { + $failure = $result->failure ?? $settlementFailure; + $attempted = count($messages) - count($result->deferred); + $eventClass = $failure === null ? OutboxBatchDelivered::class : OutboxBatchDeliveryFailed::class; + $notify = $attempted > 0 && $this->events->hasListeners($eventClass); + + if (! $notify && $failure === null) { + return; + } + + $this->inFreshCoroutine(function () use ($database, $connection, $messages, $result, $settlementFailure, $duration, $publishedAt, $failure, $attempted, $notify): void { + $listenerFailure = null; + + if ($notify) { + $eligibleAt = $publishedAt; + $deferred = array_fill_keys($result->deferred, true); + + foreach ($messages as $message) { + if (! isset($deferred[$message->id])) { + $eligibleAt = min($eligibleAt, $message->eligibleAt); + } + } + + $event = $failure === null + ? new OutboxBatchDelivered($database, $connection, count($result->accepted), $duration, $publishedAt - $eligibleAt) + : new OutboxBatchDeliveryFailed($database, $connection, count($result->accepted), $attempted - count($result->accepted), $duration, $publishedAt - $eligibleAt, $failure); + + try { + $this->events->dispatch($event); + } catch (CanceledException|ChildCancellationException $exception) { + throw $exception; + } catch (Throwable $exception) { + $listenerFailure = $exception; + } + } + + if ($failure !== null || $listenerFailure !== null) { + $context = [ + 'outbox_database' => $database, + 'queue_connection' => $connection, + 'outbox_ids' => array_column($messages, 'id'), + 'job_uuids' => array_column($messages, 'uuid'), + 'jobs' => array_map(static function (OutboxMessage $message): mixed { + $payload = json_decode($message->payload, true); + + return $payload['displayName'] ?? $payload['job'] ?? null; + }, $messages), + ]; + + foreach ([$result->failure, $settlementFailure, $listenerFailure] as $exception) { + if ($exception !== null) { + $this->exceptions->report($exception, $context); + } + } + } + }); + } + + /** + * Finish an operation's resource cleanup before returning or propagating cancellation. + * + * @template TReturn + * @param Closure(): TReturn $callback + * @return TReturn + */ + protected function inFreshCoroutine(Closure $callback): mixed + { + return (new Waiter(-1))->wait($callback, waitForChildTermination: true); + } +} diff --git a/tests/Integration/Queue/Database/Sqlite/WorkerResourceLifetimeTest.php b/tests/Integration/Queue/Database/Sqlite/WorkerResourceLifetimeTest.php index bb13dfdbb..7f68ec665 100644 --- a/tests/Integration/Queue/Database/Sqlite/WorkerResourceLifetimeTest.php +++ b/tests/Integration/Queue/Database/Sqlite/WorkerResourceLifetimeTest.php @@ -9,25 +9,41 @@ use Hypervel\Contracts\Foundation\Application as ApplicationContract; use Hypervel\Contracts\Queue\Factory as QueueManager; use Hypervel\Contracts\Queue\Job as JobContract; +use Hypervel\Contracts\Queue\PublishesOutboxMessages; use Hypervel\Contracts\Queue\Queue; use Hypervel\Coordinator\Timer; use Hypervel\Coroutine\Coroutine; use Hypervel\Coroutine\Waiter; +use Hypervel\Database\Events\QueryExecuted; use Hypervel\Database\Pool\PoolManager; use Hypervel\Engine\Channel; +use Hypervel\Engine\Coroutine as EngineCoroutine; use Hypervel\Filesystem\Filesystem; +use Hypervel\ObjectPool\PoolDefinition; +use Hypervel\ObjectPool\PoolManager as ObjectPoolManager; +use Hypervel\ObjectPool\PoolOptions; use Hypervel\Queue\Events\JobPopping; use Hypervel\Queue\Events\Looping; +use Hypervel\Queue\Events\OutboxBatchDelivered; use Hypervel\Queue\Events\WorkerIdle; use Hypervel\Queue\Events\WorkerPausing; use Hypervel\Queue\Events\WorkerStarting; use Hypervel\Queue\Events\WorkerStopping; +use Hypervel\Queue\Outbox\DatabaseOutboxRepository; +use Hypervel\Queue\Outbox\OutboxBatchResult; +use Hypervel\Queue\Outbox\OutboxRelay; +use Hypervel\Queue\QueuePoolProxy; +use Hypervel\Queue\SyncQueue; use Hypervel\Queue\Worker; use Hypervel\Queue\WorkerOptions; use Hypervel\Support\Facades\DB; +use Hypervel\Support\Str; use Hypervel\Testbench\TestCase; use Hypervel\Testing\ParallelTesting; use Mockery as m; +use PHPUnit\Framework\Attributes\DataProvider; +use Swoole\Coroutine\CanceledException; +use Throwable; class WorkerResourceLifetimeTest extends TestCase { @@ -68,6 +84,7 @@ protected function defineEnvironment(ApplicationContract $app): void $config = $app->make('config'); $config->set('database.default', 'worker_resource'); + $config->set('queue.outbox.databases', ['worker_resource']); $config->set('database.connections.worker_resource', [ 'driver' => 'sqlite', 'database' => $this->databaseDirectory . '/database.sqlite', @@ -80,6 +97,106 @@ protected function defineEnvironment(ApplicationContract $app): void ]); } + #[DataProvider('outboxPublicationOutcomes')] + public function testOutboxPublicationReleasesOneSlotPoolsAcrossBlockingAndCancellation(bool $cancel): void + { + $waiter = new Waiter(-1); + $repository = $this->app->make(DatabaseOutboxRepository::class); + $waiter->wait(function () use ($repository): void { + $migration = require dirname(__DIR__, 5) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php'; + $migration->up(); + $repository->insert(DB::connection(), [[ + 'uuid' => (string) Str::uuid(), 'connection' => 'transport', 'queue' => 'orders', + 'payload' => '{"displayName":"SendInvoice"}', 'options' => null, + 'deliver_at' => null, 'available_at' => 1, 'created_at' => 1, + ]]); + }); + $databasePool = $this->app->make(PoolManager::class)->pool('worker_resource'); + $objectPools = new ObjectPoolManager; + $started = new Channel(1); + $finish = new Channel(1); + $transport = m::mock(SyncQueue::class . ', ' . PublishesOutboxMessages::class)->makePartial(); + $transport->expects('publishOutboxMessages')->andReturnUsing(function (array $messages) use ($started, $finish, $databasePool): OutboxBatchResult { + $this->assertSame(1, $databasePool->getIdleCount()); + $started->push(true); + $this->assertTrue($finish->pop(5)); + + return new OutboxBatchResult(array_column($messages, 'id')); + }); + $proxy = new QueuePoolProxy( + new PoolDefinition('outbox-resource-test', 'queue', 'outbox-resource-test', PoolOptions::fromArray(['max_objects' => 1])), + static fn (): SyncQueue => $transport, + $objectPools, + ); + $queues = m::mock(QueueManager::class); + $queues->expects('connection')->with('transport')->andReturn($proxy); + $events = $this->app->make(EventDispatcher::class); + $settled = false; + $observed = false; + $events->listen(QueryExecuted::class, function (QueryExecuted $event) use ($objectPools, &$settled): void { + if (str_starts_with($event->sql, 'delete from "job_outbox"')) { + $this->assertSame(0, $objectPools->get('outbox-resource-test')->getBorrowedCount()); + $settled = true; + } + }); + $events->listen(OutboxBatchDelivered::class, function () use ($databasePool, &$observed): void { + $this->assertSame(1, $databasePool->getIdleCount()); + DB::selectOne('SELECT 1'); + $observed = true; + }); + $relay = new OutboxRelay($repository, $queues, $events, m::mock(ExceptionHandlerContract::class)); + $messages = $relay->claim('worker_resource', 100); + $failure = null; + $result = null; + $runner = Coroutine::create(static function () use ($relay, $messages, &$failure, &$result): void { + try { + $result = $relay->deliver('worker_resource', $messages); + } catch (Throwable $exception) { + $failure = $exception; + } + }); + + try { + $this->assertTrue($started->pop(2)); + $this->assertSame(1, $objectPools->get('outbox-resource-test')->getBorrowedCount()); + $this->assertSame(1, $databasePool->getIdleCount()); + $this->assertSame(1, $waiter->wait(static fn (): int => (int) DB::selectOne('SELECT 1 AS value')->value)); + + if ($cancel) { + EngineCoroutine::cancelById($runner, throwException: true); + } else { + $finish->push(true); + } + + Coroutine::join([$runner], 2); + $this->assertFalse(Coroutine::exists($runner)); + $this->assertSame(1, $databasePool->getIdleCount()); + $this->assertSame(0, $objectPools->get('outbox-resource-test')->getBorrowedCount()); + $this->assertSame(! $cancel, $settled); + $this->assertSame(! $cancel, $observed); + $this->assertSame($cancel ? 1 : 0, $waiter->wait(static fn (): int => DB::table('job_outbox')->count())); + + if ($cancel) { + $this->assertInstanceOf(CanceledException::class, $failure); + } else { + $this->assertNull($failure); + $this->assertTrue($result); + } + } finally { + $finish->close(); + Coroutine::join([$runner], 5); + $objectPools->purgeAll(); + } + } + + /** + * Cover normal settlement and cancellation while the transport is blocked. + */ + public static function outboxPublicationOutcomes(): array + { + return [[false], [true]]; + } + public function testLifecycleCallbacksReleasePooledConnectionsBeforeTheDaemonAdvances(): void { $connectionName = $this->app->make('config')->string('database.default'); diff --git a/tests/Queue/Outbox/OutboxRelayTest.php b/tests/Queue/Outbox/OutboxRelayTest.php new file mode 100644 index 000000000..dd935d07a --- /dev/null +++ b/tests/Queue/Outbox/OutboxRelayTest.php @@ -0,0 +1,290 @@ +message(1)]; + $phases = []; + $repository->expects('claim')->with('primary', 100)->andReturnUsing(function () use (&$phases, $messages): array { + $this->assertFalse(CoroutineContext::has('request-only')); + $phases[] = 'claim'; + Coroutine::defer(static function () use (&$phases): void { $phases[] = 'claim released'; }); + + return $messages; + }); + $queues->expects('connection')->with('redis')->andReturn($queue); + $queue->expects('publishOutboxMessages')->with($messages)->andReturnUsing(function () use (&$phases): OutboxBatchResult { + $this->assertSame(['claim', 'claim released'], $phases); + $this->assertFalse(CoroutineContext::has('request-only')); + $phases[] = 'publish'; + Coroutine::defer(static function () use (&$phases): void { $phases[] = 'publish released'; }); + + // Drivers may preserve keys when filtering a batch's row IDs. + return new OutboxBatchResult([7 => 1]); + }); + $repository->expects('settle')->andReturnUsing(function () use (&$phases): int { + $this->assertSame('publish released', end($phases)); + $phases[] = 'settle'; + Coroutine::defer(static function () use (&$phases): void { $phases[] = 'settle released'; }); + + return 1; + }); + $events->listen(OutboxBatchDelivered::class, function (OutboxBatchDelivered $event) use (&$phases): void { + $this->assertSame('settle released', end($phases)); + $this->assertFalse(CoroutineContext::has('request-only')); + $this->assertSame('primary', $event->database); + $this->assertSame(1, $event->accepted); + $this->assertSame(20, $event->oldestEligibleAge); + Coroutine::defer(static function () use (&$phases): void { $phases[] = 'observer released'; }); + }); + $relay = new OutboxRelay($repository, $queues, $events, $exceptions); + + $this->assertTrue($relay->deliver('primary', $relay->claim('primary', 100))); + $this->assertSame('observer released', end($phases)); + } + + public function testPartialResultsSettleBeforeAnotherConnectionAndReportingCannotBlockItsPublication(): void + { + $repository = m::mock(DatabaseOutboxRepository::class); + $queues = m::mock(Factory::class); + $first = m::mock(Queue::class . ', ' . PublishesOutboxMessages::class); + $second = m::mock(Queue::class . ', ' . PublishesOutboxMessages::class); + $events = new Dispatcher; + $exceptions = m::mock(ExceptionHandler::class); + $messages = [$this->message(1), $this->message(2), $this->message(3, 'other')]; + $failure = new RuntimeException('Second publication rejected.'); + $settled = []; + $queues->expects('connection')->with('redis')->andReturn($first); + $queues->expects('connection')->with('other')->andReturn($second); + $first->expects('publishOutboxMessages')->andReturn(new OutboxBatchResult([1], failure: $failure)); + $second->expects('publishOutboxMessages')->andReturnUsing(function () use (&$settled): OutboxBatchResult { + $this->assertSame([[1]], $settled); + + return new OutboxBatchResult([3]); + }); + $repository->shouldReceive('settle')->twice()->andReturnUsing(function (string $database, array $group, OutboxBatchResult $result) use (&$settled): int { + $settled[] = $result->accepted; + + return count($group); + }); + $events->listen(OutboxBatchDeliveryFailed::class, function (OutboxBatchDeliveryFailed $event) use ($failure): void { + $this->assertSame(1, $event->accepted); + $this->assertSame(1, $event->failed); + $this->assertSame($failure, $event->exception); + }); + $exceptions->expects('report')->with($failure, m::on(function (array $context) use (&$settled): bool { + $this->assertSame([[1], [3]], $settled); + $this->assertSame([1, 2], $context['outbox_ids']); + $this->assertSame(['SendInvoice', 'SendInvoice'], $context['jobs']); + + return true; + })); + + $this->assertFalse((new OutboxRelay($repository, $queues, $events, $exceptions))->deliver('primary', $messages)); + } + + #[DataProvider('invalidResults')] + public function testInvalidResultsReleaseEveryInputAsFailure(OutboxBatchResult $invalid): void + { + $repository = m::mock(DatabaseOutboxRepository::class); + $queues = m::mock(Factory::class); + $queue = m::mock(Queue::class . ', ' . PublishesOutboxMessages::class); + $exceptions = m::mock(ExceptionHandler::class); + $queues->expects('connection')->andReturn($queue); + $queue->expects('publishOutboxMessages')->andReturn($invalid); + $repository->expects('settle')->with('primary', m::type('array'), m::on(function (OutboxBatchResult $result): bool { + $this->assertSame([], $result->accepted); + $this->assertSame([], $result->deferred); + $this->assertInstanceOf(LogicException::class, $result->failure); + + return true; + }))->andReturn(2); + $exceptions->expects('report')->with(m::type(LogicException::class), m::type('array')); + + $this->assertFalse((new OutboxRelay($repository, $queues, new Dispatcher, $exceptions))->deliver('primary', [$this->message(1), $this->message(2)])); + } + + /** + * Provide invalid driver accounting without repeating transport tests. + */ + public static function invalidResults(): array + { + return [ + 'foreign ID' => [new OutboxBatchResult([1, 3])], + 'duplicate ID' => [new OutboxBatchResult([1, 1])], + 'overlap' => [new OutboxBatchResult([1, 2], [2])], + 'missing row without failure' => [new OutboxBatchResult([1])], + 'non-integer ID' => [new OutboxBatchResult(['1', 2])], + 'defer immediate job' => [new OutboxBatchResult([1], [2])], + ]; + } + + public function testGenericFallbackDefersFutureRowsWithoutEventsAndSettlesThemNormally(): void + { + $repository = m::mock(DatabaseOutboxRepository::class); + $queues = m::mock(Factory::class); + $queue = m::mock(Queue::class); + $exceptions = m::mock(ExceptionHandler::class); + $events = new Dispatcher; + $events->listen(OutboxBatchDelivered::class, fn () => $this->fail('Deferred rows were not delivered.')); + $events->listen(OutboxBatchDeliveryFailed::class, fn () => $this->fail('Deferral is not failure.')); + $queues->expects('connection')->andReturn($queue); + $repository->expects('settle')->with('primary', m::type('array'), m::on(fn (OutboxBatchResult $result): bool => $result->accepted === [] && $result->deferred === [1] && $result->failure === null))->andReturn(1); + + $this->assertTrue((new OutboxRelay($repository, $queues, $events, $exceptions))->deliver('primary', [$this->message(1, deliverAt: CarbonImmutable::now()->getTimestamp() + 60)])); + } + + public function testPublicationAndSettlementErrorsAreBothReportedWithPublicationOnTheEvent(): void + { + $repository = m::mock(DatabaseOutboxRepository::class); + $queues = m::mock(Factory::class); + $queue = m::mock(Queue::class); + $exceptions = m::mock(ExceptionHandler::class); + $events = new Dispatcher; + $publication = new RuntimeException('Broker unavailable.'); + $settlement = new RuntimeException('Database unavailable.'); + $queues->expects('connection')->andReturn($queue); + $queue->expects('pushRaw')->andThrow($publication); + $repository->expects('settle')->andThrow($settlement); + $exceptions->expects('report')->with($publication, m::type('array')); + $exceptions->expects('report')->with($settlement, m::type('array')); + $events->listen(OutboxBatchDeliveryFailed::class, function (OutboxBatchDeliveryFailed $event) use ($publication): void { + $this->assertSame($publication, $event->exception); + }); + + $this->assertFalse((new OutboxRelay($repository, $queues, $events, $exceptions))->deliver('primary', [$this->message(1)])); + } + + public function testLostSettlementOwnershipReportsConfirmedDeliveryWithoutRepublishing(): void + { + $repository = m::mock(DatabaseOutboxRepository::class); + $queues = m::mock(Factory::class); + $queue = m::mock(Queue::class); + $exceptions = m::mock(ExceptionHandler::class); + $events = new Dispatcher; + $queues->expects('connection')->andReturn($queue); + $queue->expects('pushRaw')->andReturn('accepted'); + $repository->expects('settle')->andReturn(0); + $events->listen(OutboxBatchDelivered::class, fn () => $this->fail('Lost ownership is a settlement failure.')); + $events->listen(OutboxBatchDeliveryFailed::class, function (OutboxBatchDeliveryFailed $event): void { + $this->assertSame(1, $event->accepted); + $this->assertSame(0, $event->failed); + $this->assertStringContainsString('lease ownership changed', $event->exception->getMessage()); + }); + $exceptions->expects('report')->with(m::type(RuntimeException::class), m::type('array')); + + $this->assertFalse((new OutboxRelay($repository, $queues, $events, $exceptions))->deliver('primary', [$this->message(1)])); + } + + #[DataProvider('reportingOutcomes')] + public function testListenerFailureDoesNotThrottleDeliveryAndReporterFailureEscapes(bool $reporterFails): void + { + $repository = m::mock(DatabaseOutboxRepository::class); + $queues = m::mock(Factory::class); + $queue = m::mock(Queue::class); + $exceptions = m::mock(ExceptionHandler::class); + $events = new Dispatcher; + $listenerFailure = new RuntimeException('Listener failed.'); + $reporterFailure = new RuntimeException('Reporter failed.'); + $released = false; + $queues->expects('connection')->andReturn($queue); + $queue->expects('pushRaw')->andReturn('accepted'); + $repository->expects('settle')->andReturn(1); + $events->listen(OutboxBatchDelivered::class, static function () use ($listenerFailure, &$released): void { + Coroutine::defer(static function () use (&$released): void { $released = true; }); + throw $listenerFailure; + }); + $events->listen(OutboxBatchDeliveryFailed::class, fn () => $this->fail('Listener failure must not change the delivery event.')); + $report = $exceptions->expects('report')->with($listenerFailure, m::type('array')); + + if ($reporterFails) { + $report->andThrow($reporterFailure); + } + + try { + $this->assertTrue((new OutboxRelay($repository, $queues, $events, $exceptions))->deliver('primary', [$this->message(1)])); + $this->assertFalse($reporterFails, 'Reporter failure must reach the worker.'); + } catch (RuntimeException $exception) { + $this->assertTrue($reporterFails); + $this->assertSame($reporterFailure, $exception); + } + + $this->assertTrue($released); + } + + /** + * Exercise observer reporting with a healthy and failed exception handler. + */ + public static function reportingOutcomes(): array + { + return [[false], [true]]; + } + + public function testIndependentPhaseCancellationReleasesResourcesAndStopsWithoutSettlement(): void + { + $repository = m::mock(DatabaseOutboxRepository::class); + $queues = m::mock(Factory::class); + $queue = m::mock(Queue::class); + $exceptions = m::mock(ExceptionHandler::class); + $cancellation = new CanceledException; + $released = false; + $queues->expects('connection')->andReturn($queue); + $queue->expects('pushRaw')->andReturnUsing(static function () use ($cancellation, &$released): never { + Coroutine::defer(static function () use (&$released): void { $released = true; }); + throw $cancellation; + }); + + try { + (new OutboxRelay($repository, $queues, new Dispatcher, $exceptions))->deliver('primary', [$this->message(1), $this->message(2, 'other')]); + $this->fail('Child cancellation must stop delivery.'); + } catch (ChildCancellationException $exception) { + $this->assertSame($cancellation, $exception->getPrevious()); + } + + $this->assertTrue($released); + } + + /** + * Build a message whose age and diagnostics are visible to observers. + */ + protected function message(int $id, string $connection = 'redis', ?int $deliverAt = null): OutboxMessage + { + $now = CarbonImmutable::now()->getTimestamp(); + + return new OutboxMessage($id, "job-{$id}", 'lease', 1, $connection, 'orders', '{"displayName":"SendInvoice"}', [], $deliverAt, $now - 20, $now - 30); + } +} From 957b82e3c88f79bf717c582d3a573c4b425269aa Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 21:56:26 +0000 Subject: [PATCH 13/33] feat(queue): coordinate concurrent outbox delivery with graceful shutdown Admit serial round-robin claims into bounded concurrent publication tasks, cap each claim against the remaining job allowance, and cool down failed sources without delaying healthy databases. Support once, stop-when-empty, source pinning, restart signals, pause and resume, memory and time limits. Keep invocation state local, isolate restart-cache reads and exception reporting, and join owned tasks before cancellation leaves the worker. Hold a failed delivery task through its retry sleep to protect unavailable transports from a fresh-row backlog. Test exact admission limits, concurrency, source fairness, lifecycle controls, resource cleanup and error propagation. --- src/queue/src/Outbox/OutboxWorker.php | 218 +++++++++++++++ src/queue/src/Outbox/OutboxWorkerOptions.php | 36 +++ tests/Queue/Outbox/OutboxWorkerTest.php | 277 +++++++++++++++++++ 3 files changed, 531 insertions(+) create mode 100644 src/queue/src/Outbox/OutboxWorker.php create mode 100644 src/queue/src/Outbox/OutboxWorkerOptions.php create mode 100644 tests/Queue/Outbox/OutboxWorkerTest.php diff --git a/src/queue/src/Outbox/OutboxWorker.php b/src/queue/src/Outbox/OutboxWorker.php new file mode 100644 index 000000000..57e873287 --- /dev/null +++ b/src/queue/src/Outbox/OutboxWorker.php @@ -0,0 +1,218 @@ +repository->getDatabases(); + + if ($database !== null) { + if (! in_array($database, $databases, true)) { + throw new InvalidArgumentException("Outbox database [{$database}] is not configured in queue.outbox.databases."); + } + + $databases = [$database]; + } + + $quit = false; + $paused = false; + $signalHandler = static function (int $signal) use (&$quit, &$paused): void { + match ($signal) { + SIGUSR2 => $paused = true, + SIGCONT => $paused = false, + default => $quit = true, + }; + }; + $this->signals->register($signalHandler, [SIGTERM, SIGINT, SIGQUIT, SIGUSR2, SIGCONT], $signalHandler); + $concurrent = new WaitConcurrent($options->concurrency); + $taskFailure = null; + $failure = null; + $status = Worker::EXIT_SUCCESS; + + try { + $startedAt = microtime(true); + $lastRestart = $options->once ? null : $this->restartTimestamp(); + $admitted = 0; + $visited = 0; + $emptyVisits = 0; + $retryAt = []; + $sourceCount = count($databases); + + while (! $quit && $taskFailure === null) { + if (memory_get_usage(true) / 1024 / 1024 >= $options->memory) { + $status = Worker::EXIT_MEMORY_LIMIT; + break; + } + + if (($options->maxJobs > 0 && $admitted >= $options->maxJobs) + || ($options->maxTime > 0 && microtime(true) - $startedAt >= $options->maxTime) + || ($options->once && $visited >= $sourceCount) + || (! $options->once && $lastRestart !== $this->restartTimestamp())) { + break; + } + + if ($paused) { + Sleep::usleep((int) ($options->sleep * 1_000_000)); + continue; + } + + if ($concurrent->isFull()) { + $concurrent->waitForAvailableSlot($options->sleep); + continue; + } + + $source = $databases[$visited++ % $sourceCount]; + $limit = $options->maxJobs > 0 ? min($options->batch, $options->maxJobs - $admitted) : $options->batch; + $messages = []; + + if (! isset($retryAt[$source]) || microtime(true) >= $retryAt[$source]) { + $messages = $this->claim($source, $limit); + + if ($messages === null) { + $retryAt[$source] = microtime(true) + $options->sleep; + } else { + unset($retryAt[$source]); + } + } + + if ($messages) { + $emptyVisits = 0; + $admitted += count($messages); + + if ($options->once) { + $this->relay->deliver($source, $messages); + } else { + $concurrent->create(function () use ($source, $messages, $options, &$taskFailure): void { + try { + if (! $this->relay->deliver($source, $messages)) { + Sleep::usleep((int) ($options->sleep * 1_000_000)); + } + } catch (CanceledException $exception) { + throw $exception; + } catch (Throwable $exception) { + $taskFailure ??= $exception; + } + }); + } + } elseif (++$emptyVisits >= $sourceCount) { + if ($options->stopWhenEmpty) { + break; + } + + if (! $options->once) { + Sleep::usleep((int) ($options->sleep * 1_000_000)); + } + + $emptyVisits = 0; + } + } + } catch (Throwable $exception) { + $failure = $exception; + + if ($exception instanceof CanceledException) { + $concurrent->cancel(); + } + } finally { + try { + $this->join($concurrent); + } catch (CanceledException $exception) { + if (! $failure instanceof CanceledException) { + $failure = $exception; + } + } finally { + $this->signals->unregister($signalHandler); + } + } + + if ($failure !== null || $taskFailure !== null) { + throw $failure ?? $taskFailure; + } + + return $status; + } + + /** + * Report a failed claim after its database-owning coroutine has ended. + * + * @return null|list null indicates a source failure + */ + protected function claim(string $database, int $limit): ?array + { + try { + return $this->relay->claim($database, $limit); + } catch (CanceledException|ChildCancellationException $exception) { + throw $exception; + } catch (Throwable $exception) { + (new Waiter(-1))->wait( + fn () => $this->exceptions->report($exception, ['outbox_database' => $database]), + waitForChildTermination: true, + ); + + return null; + } + } + + /** + * Read the restart signal without retaining a cache connection on the coordinator. + */ + protected function restartTimestamp(): ?int + { + return (new Waiter(-1))->wait(function (): ?int { + $timestamp = $this->cache->store()->get(Worker::RESTART_SIGNAL_CACHE_KEY); + + return $timestamp === null ? null : (int) $timestamp; + }, waitForChildTermination: true); + } + + /** + * Join every owned task before propagating cancellation, including repeated cancellation during cleanup. + */ + protected function join(WaitConcurrent $concurrent): void + { + $cancellation = null; + + do { + try { + $finished = $concurrent->wait(); + } catch (CanceledException $exception) { + $cancellation ??= $exception; + $finished = false; + } + } while (! $finished); + + if ($cancellation !== null) { + throw $cancellation; + } + } +} diff --git a/src/queue/src/Outbox/OutboxWorkerOptions.php b/src/queue/src/Outbox/OutboxWorkerOptions.php new file mode 100644 index 000000000..45692bbac --- /dev/null +++ b/src/queue/src/Outbox/OutboxWorkerOptions.php @@ -0,0 +1,36 @@ + 1000) { + throw new InvalidArgumentException('The outbox batch size must be between 1 and 1000.'); + } + + if ($sleep <= 0 || $concurrency < 1) { + throw new InvalidArgumentException('Outbox sleep and concurrency must be greater than zero.'); + } + + if ($maxJobs < 0 || $maxTime < 0 || $memory <= 0) { + throw new InvalidArgumentException('Outbox job and time limits must be non-negative, and memory must be greater than zero.'); + } + } +} diff --git a/tests/Queue/Outbox/OutboxWorkerTest.php b/tests/Queue/Outbox/OutboxWorkerTest.php new file mode 100644 index 000000000..40c483247 --- /dev/null +++ b/tests/Queue/Outbox/OutboxWorkerTest.php @@ -0,0 +1,277 @@ +worker(['primary', 'secondary']); + $releaseFirst = new Channel(1); + $running = 0; + $peak = 0; + $finished = 0; + $this->relay->expects('claim')->with('primary', 2)->andReturn([$this->message(1), $this->message(2)]); + $this->relay->expects('claim')->with('secondary', 1)->andReturn([$this->message(3)]); + $this->relay->shouldReceive('deliver')->twice()->andReturnUsing(function (string $database) use ($releaseFirst, &$running, &$peak, &$finished): bool { + $peak = max($peak, ++$running); + + try { + if ($database === 'primary') { + $this->assertTrue($releaseFirst->pop(2)); + } else { + $releaseFirst->push(true); + } + + ++$finished; + + return true; + } finally { + --$running; + } + }); + + $this->assertSame(Worker::EXIT_SUCCESS, $worker->run(null, new OutboxWorkerOptions(batch: 2, concurrency: 2, maxJobs: 3))); + $this->assertSame(2, $peak); + $this->assertSame(2, $finished); + $this->assertSame(0, $running); + } + + public function testFailedSourceCoolsDownWithoutDelayingHealthySources(): void + { + Sleep::fake(); + $worker = $this->worker(['broken', 'healthy']); + $failure = new RuntimeException('Connection refused.'); + $this->relay->expects('claim')->with('broken', 1)->andThrow($failure); + $this->relay->shouldReceive('claim')->with('healthy', 1)->times(3)->andReturn([$this->message(1)]); + $this->relay->shouldReceive('deliver')->with('healthy', m::type('array'))->times(3)->andReturn(true); + $this->exceptions->expects('report')->with($failure, ['outbox_database' => 'broken']); + + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(batch: 1, sleep: 60, maxJobs: 3))); + Sleep::assertNeverSlept(); + } + + public function testStopWhenEmptyTreatsFailedSourcesAsEmptyAndDoesNotRecheckAfterJoining(): void + { + Sleep::fake(); + $worker = $this->worker(['broken', 'empty']); + $failure = new RuntimeException('Missing table.'); + $this->relay->expects('claim')->with('broken', 100)->andThrow($failure); + $this->relay->expects('claim')->with('empty', 100)->andReturn([]); + $this->exceptions->expects('report')->with($failure, ['outbox_database' => 'broken']); + + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(stopWhenEmpty: true))); + Sleep::assertNeverSlept(); + } + + public function testOnceVisitsEverySourceWithoutSleepingEvenWhenDeliveryFails(): void + { + Sleep::fake(); + $worker = $this->worker(['first', 'second']); + $this->relay->expects('claim')->with('first', 100)->andReturn([$this->message(1)]); + $this->relay->expects('claim')->with('second', 100)->andReturn([]); + $this->relay->expects('deliver')->with('first', m::type('array'))->andReturn(false); + + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(once: true))); + Sleep::assertNeverSlept(); + } + + public function testPinnedSourceDoesNotClaimFromOtherConfiguredDatabases(): void + { + $worker = $this->worker(['first', 'second']); + $this->relay->expects('claim')->with('second', 100)->andReturn([]); + + $this->assertSame(0, $worker->run('second', new OutboxWorkerOptions(once: true))); + } + + public function testFailureSleepKeepsThePublicationSlotOccupied(): void + { + Sleep::fake(); + $worker = $this->worker(); + $claims = 0; + $this->relay->shouldReceive('claim')->twice()->andReturnUsing(function () use (&$claims): array { + return [$this->message(++$claims)]; + }); + $this->relay->expects('deliver')->andReturn(false); + $this->relay->expects('deliver')->andReturn(true); + Sleep::whenFakingSleep(function () use (&$claims): void { + $this->assertSame(1, $claims); + }); + + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(batch: 1, sleep: 0.25, maxJobs: 2))); + Sleep::assertSequence([Sleep::usleep(250_000)]); + } + + public function testEmptyPassSleepsOnceAndSignalStopsFurtherAdmission(): void + { + Sleep::fake(); + $worker = $this->worker(['first', 'second']); + $this->relay->expects('claim')->with('first', 100)->andReturn([]); + $this->relay->expects('claim')->with('second', 100)->andReturn([]); + Sleep::whenFakingSleep(function (): void { ($this->signalHandler)(SIGTERM); }); + + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(sleep: 0.25))); + Sleep::assertSequence([Sleep::usleep(250_000)]); + } + + public function testPauseStopsAdmissionUntilTheResumeSignal(): void + { + Sleep::fake(); + $worker = $this->worker(); + $claims = 0; + $sleeps = 0; + $this->relay->shouldReceive('claim')->twice()->andReturnUsing(function () use (&$claims): array { + return ++$claims === 1 ? [] : [$this->message(1)]; + }); + $this->relay->expects('deliver')->andReturn(true); + Sleep::whenFakingSleep(function () use (&$claims, &$sleeps): void { + $this->assertSame(1, $claims); + ($this->signalHandler)(++$sleeps === 1 ? SIGUSR2 : SIGCONT); + }); + + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(sleep: 0.25, maxJobs: 1))); + Sleep::assertSequence([Sleep::usleep(250_000), Sleep::usleep(250_000)]); + } + + public function testRestartReadsEndTheirCoroutineBeforeClaimingAndRestartStopsAdmission(): void + { + $worker = $this->worker(); + CoroutineContext::set('request-only', true); + $borrowed = false; + $reads = 0; + $this->cache->shouldReceive('get')->with(Worker::RESTART_SIGNAL_CACHE_KEY)->times(3)->andReturnUsing(function () use (&$borrowed, &$reads): int { + $this->assertFalse(CoroutineContext::has('request-only')); + $borrowed = true; + Coroutine::defer(static function () use (&$borrowed): void { $borrowed = false; }); + + return ++$reads === 3 ? 2 : 1; + }); + $this->relay->expects('claim')->andReturnUsing(function () use (&$borrowed): array { + $this->assertFalse($borrowed); + + return [$this->message(1)]; + }); + $this->relay->expects('deliver')->andReturn(true); + + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions)); + $this->assertFalse($borrowed); + } + + public function testEscapingTaskFailureStopsAdmissionAndPropagates(): void + { + $worker = $this->worker(); + $failure = new RuntimeException('Reporter unavailable.'); + $this->relay->expects('claim')->andReturn([$this->message(1)]); + $this->relay->expects('deliver')->andThrow($failure); + + try { + $worker->run(null, new OutboxWorkerOptions); + $this->fail('Task failure must reach the command.'); + } catch (RuntimeException $exception) { + $this->assertSame($failure, $exception); + } + } + + public function testCancellationJoinsTaskCleanupBeforeReturningToTheCaller(): void + { + $worker = $this->worker(); + $started = new Channel(1); + $blocker = new Channel(1); + $released = false; + $caught = null; + $this->relay->expects('claim')->andReturn([$this->message(1)]); + $this->relay->expects('deliver')->andReturnUsing(static function () use ($started, $blocker, &$released): bool { + Coroutine::defer(static function () use (&$released): void { $released = true; }); + $started->push(true); + $blocker->pop(5); + + return true; + }); + $coordinator = Coroutine::create(static function () use ($worker, &$caught): void { + try { + $worker->run(null, new OutboxWorkerOptions(maxJobs: 1)); + } catch (CanceledException $exception) { + $caught = $exception; + } + }); + + try { + $this->assertTrue($started->pop(2)); + EngineCoroutine::cancelById($coordinator, throwException: true); + Coroutine::join([$coordinator], 2); + $this->assertFalse(Coroutine::exists($coordinator)); + $this->assertInstanceOf(CanceledException::class, $caught); + $this->assertTrue($released); + } finally { + $blocker->close(); + Coroutine::join([$coordinator], 5); + } + } + + public function testMemoryLimitStopsBeforeClaiming(): void + { + $this->assertSame(Worker::EXIT_MEMORY_LIMIT, $this->worker()->run(null, new OutboxWorkerOptions(memory: 1))); + } + + /** + * Construct an isolated worker with observable dependencies and signal ownership. + */ + protected function worker(array $databases = ['primary']): OutboxWorker + { + $repository = m::mock(DatabaseOutboxRepository::class); + $repository->shouldReceive('getDatabases')->andReturn($databases); + $this->relay = m::mock(OutboxRelay::class); + $this->exceptions = m::mock(ExceptionHandler::class); + $this->cache = m::mock(Cache::class); + $this->cache->shouldReceive('get')->with(Worker::RESTART_SIGNAL_CACHE_KEY)->andReturn(null)->byDefault(); + $cacheFactory = m::mock(CacheFactory::class); + $cacheFactory->shouldReceive('store')->andReturn($this->cache); + $signals = m::mock(SignalRegistry::class); + $signals->expects('register')->andReturnUsing(function (object $owner, array $signals, Closure $callback): void { + $this->assertSame($owner, $callback); + $this->signalHandler = $callback; + }); + $signals->expects('unregister')->with(m::on(fn (object $owner): bool => $owner === $this->signalHandler)); + + return new OutboxWorker($repository, $this->relay, $cacheFactory, $this->exceptions, $signals); + } + + /** + * Create a claimed message for worker admission tests. + */ + protected function message(int $id): OutboxMessage + { + return new OutboxMessage($id, "job-{$id}", 'lease', 1, 'redis', 'default', '{}', [], null, 1, 1); + } +} From fb54177ebe70cde0b6d2da28701fff88784c93ed Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 21:56:37 +0000 Subject: [PATCH 14/33] feat(queue): add relay and transactional outbox operator commands Register queue:relay and bounded list, monitor, retry and forget commands. Expose readable UTC dates and compact errors in tables while preserving raw diagnostic values in JSON, and report mutually exclusive backlog states with optional threshold alerts. Resolve ambiguous job UUIDs explicitly, retain atomic live-lease guards and preserve retry history. Confirm destructive operations before querying; force skips only confirmation, never an active publication lease. Forget deletes the row before exact-owner unique-lock cleanup, reads only serialized scalar ownership metadata and reports post-delete cleanup failures without restoring discarded work. Cover command options, scripting, listing bounds, monitoring, source ambiguity and replacement-lock safety. --- .../Console/Concerns/FindsOutboxMessages.php | 34 +++ src/queue/src/Console/ForgetOutboxCommand.php | 117 ++++++++ src/queue/src/Console/ListOutboxCommand.php | 90 +++++++ .../src/Console/MonitorOutboxCommand.php | 68 +++++ src/queue/src/Console/RelayCommand.php | 61 +++++ src/queue/src/Console/RetryOutboxCommand.php | 74 +++++ src/queue/src/Events/OutboxBusy.php | 19 ++ .../src/Outbox/DatabaseOutboxRepository.php | 2 +- src/queue/src/QueueServiceProvider.php | 10 + tests/Queue/Outbox/OutboxCommandsTest.php | 254 ++++++++++++++++++ tests/Queue/RelayCommandTest.php | 80 ++++++ 11 files changed, 808 insertions(+), 1 deletion(-) create mode 100644 src/queue/src/Console/Concerns/FindsOutboxMessages.php create mode 100644 src/queue/src/Console/ForgetOutboxCommand.php create mode 100644 src/queue/src/Console/ListOutboxCommand.php create mode 100644 src/queue/src/Console/MonitorOutboxCommand.php create mode 100644 src/queue/src/Console/RelayCommand.php create mode 100644 src/queue/src/Console/RetryOutboxCommand.php create mode 100644 src/queue/src/Events/OutboxBusy.php create mode 100644 tests/Queue/Outbox/OutboxCommandsTest.php create mode 100644 tests/Queue/RelayCommandTest.php diff --git a/src/queue/src/Console/Concerns/FindsOutboxMessages.php b/src/queue/src/Console/Concerns/FindsOutboxMessages.php new file mode 100644 index 000000000..a30acb25b --- /dev/null +++ b/src/queue/src/Console/Concerns/FindsOutboxMessages.php @@ -0,0 +1,34 @@ +repository->getDatabases() : [$database] as $source) { + if (($message = $this->repository->find($source, $uuid)) === null) { + continue; + } + + if ($found !== null) { + throw new RuntimeException("Outbox message [{$uuid}] exists on multiple databases. Select one using --database."); + } + + $found = ['database' => $source, 'message' => $message]; + } + + return $found; + } +} diff --git a/src/queue/src/Console/ForgetOutboxCommand.php b/src/queue/src/Console/ForgetOutboxCommand.php new file mode 100644 index 000000000..f0a35fa59 --- /dev/null +++ b/src/queue/src/Console/ForgetOutboxCommand.php @@ -0,0 +1,117 @@ +argument('uuid'); + + // Ask before querying so an interactive prompt never holds a database connection. + if (! $this->confirmToProceed("Permanently discard outbox message [{$uuid}]", true)) { + return self::FAILURE; + } + + $found = $this->findOutboxMessage($uuid, $this->option('database')); + + if ($found === null) { + $this->components->error("No outbox message matches UUID [{$uuid}]."); + + return self::FAILURE; + } + + $message = $found['message']; + + if (! $this->repository->forget($found['database'], (int) $message->id, $message->lease_token)) { + $this->components->error("Outbox message [{$uuid}] is currently leased or changed during deletion."); + + return self::FAILURE; + } + + try { + $this->releaseUniqueLock($message->payload); + } catch (CanceledException $exception) { + throw $exception; + } catch (Throwable $exception) { + $this->components->error("Outbox message [{$uuid}] was deleted, but unique-lock cleanup failed: {$exception->getMessage()}"); + $this->exceptions->report($exception, ['outbox_database' => $found['database'], 'job_uuid' => $uuid]); + + return self::FAILURE; + } + + $this->components->info("Outbox message [{$uuid}] was deleted."); + + return self::SUCCESS; + } + + /** + * Release exact stored ownership without constructing the job or hydrating application context. + * + * Override this method to establish a custom cache namespace or resolve an ad hoc repository. + */ + protected function releaseUniqueLock(string $payload): void + { + $hidden = json_decode($payload, true, flags: JSON_THROW_ON_ERROR)['illuminate:log:context']['hidden'] ?? []; + + if (! array_key_exists('laravel_unique_job_key', $hidden)) { + return; + } + + $store = unserialize($hidden['laravel_unique_job_cache_store'] ?? 'N;', ['allowed_classes' => false]); + $key = unserialize($hidden['laravel_unique_job_key'], ['allowed_classes' => false]); + $owner = unserialize($hidden['laravel_unique_job_lock_owner'] ?? 'N;', ['allowed_classes' => false]); + + if (! is_string($store) || $store === '' || ! is_string($key) || $key === '' || ! is_string($owner) || $owner === '') { + throw new LogicException('Unique-lock cleanup requires a named cache store, key, and owner token.'); + } + + $cache = $this->cache->store($store); + + if (! $cache->getStore() instanceof LockProvider) { + throw new LogicException("Cache store [{$store}] does not support owner-matched lock cleanup."); + } + + UniqueLock::releaseOwned($cache, $key, $owner); + } +} diff --git a/src/queue/src/Console/ListOutboxCommand.php b/src/queue/src/Console/ListOutboxCommand.php new file mode 100644 index 000000000..835dc3965 --- /dev/null +++ b/src/queue/src/Console/ListOutboxCommand.php @@ -0,0 +1,90 @@ +option('limit'); + + if ($limit < 1 || $limit > 1000) { + throw new InvalidArgumentException('The outbox listing limit must be between 1 and 1000.'); + } + + $database = $this->argument('database'); + $rows = []; + $now = $this->currentTime(); + + foreach ($database === null ? $this->repository->getDatabases() : [$database] as $source) { + foreach ($this->repository->pending($source, $limit) as $message) { + $payload = json_decode($message->payload, true, flags: JSON_THROW_ON_ERROR); + $rows[] = [ + 'database' => $source, + 'id' => (int) $message->id, + 'uuid' => $message->uuid, + 'job' => $payload['displayName'] ?? $payload['job'] ?? '', + 'connection' => $message->connection, + 'queue' => $message->queue, + 'attempts' => (int) $message->attempts, + 'state' => match (true) { + $message->lease_token !== null && (int) $message->available_at > $now => 'leased', + (int) $message->available_at <= $now => 'eligible', + (int) $message->attempts > 0 => 'retrying', + default => 'delayed', + }, + 'age' => max(0, $now - (int) $message->created_at), + 'available_at' => (int) $message->available_at, + 'last_error' => $message->last_error, + ]; + } + } + + if ($this->option('json')) { + $this->output->writeln(json_encode($rows, JSON_THROW_ON_ERROR), OutputInterface::OUTPUT_RAW); + } elseif ($rows === []) { + $this->components->info('No pending outbox messages found.'); + } else { + $rows = array_map(static function (array $row): array { + $row['available_at'] = Date::createFromTimestampUTC($row['available_at'])->toDateTimeString(); + $row['last_error'] = Str::limit(Str::before($row['last_error'] ?? '', "\n"), 80); + + return $row; + }, $rows); + + $this->table(['Database', 'ID', 'UUID', 'Job', 'Connection', 'Queue', 'Attempts', 'State', 'Age (s)', 'Available At (UTC)', 'Last Error'], $rows); + } + } +} diff --git a/src/queue/src/Console/MonitorOutboxCommand.php b/src/queue/src/Console/MonitorOutboxCommand.php new file mode 100644 index 000000000..c5efe8baf --- /dev/null +++ b/src/queue/src/Console/MonitorOutboxCommand.php @@ -0,0 +1,68 @@ +option('max') === null ? null : (int) $this->option('max'); + $age = $this->option('age') === null ? null : (int) $this->option('age'); + + if (($maximum !== null && $maximum < 0) || ($age !== null && $age < 0)) { + throw new InvalidArgumentException('Outbox monitoring thresholds must be non-negative.'); + } + + $database = $this->argument('database'); + $rows = []; + + foreach ($database === null ? $this->repository->getDatabases() : [$database] as $source) { + $statistics = $this->repository->statistics($source); + $alert = ($maximum !== null && $statistics['total'] >= $maximum) + || ($age !== null && $statistics['oldest_eligible_age'] !== null && $statistics['oldest_eligible_age'] >= $age); + $rows[] = ['database' => $source, ...$statistics, 'status' => $alert ? 'ALERT' : 'OK']; + + if ($alert && $this->events->hasListeners(OutboxBusy::class)) { + $this->events->dispatch(new OutboxBusy($source, $statistics['total'], $statistics['eligible'], $statistics['oldest_eligible_age'])); + } + } + + if ($this->option('json')) { + $this->output->writeln(json_encode($rows, JSON_THROW_ON_ERROR), OutputInterface::OUTPUT_RAW); + } else { + $this->table(['Database', 'Total', 'Eligible', 'Delayed', 'Leased', 'Retrying', 'Oldest Eligible (s)', 'Status'], $rows); + } + } +} diff --git a/src/queue/src/Console/RelayCommand.php b/src/queue/src/Console/RelayCommand.php new file mode 100644 index 000000000..65ee006ed --- /dev/null +++ b/src/queue/src/Console/RelayCommand.php @@ -0,0 +1,61 @@ +argument('database'); + + // Avoid Symfony's stty subprocess probe inside this coroutine. + if (defined('STDIN') && stream_isatty(STDIN)) { + $this->components->info($database === null + ? 'Relaying outbox messages from all configured databases.' + : "Relaying outbox messages from [{$database}]."); + } + + return $this->worker->run($database, new OutboxWorkerOptions( + batch: (int) $this->option('batch'), + sleep: (float) $this->option('sleep'), + concurrency: (int) $this->option('concurrency'), + maxJobs: (int) $this->option('max-jobs'), + maxTime: (int) $this->option('max-time'), + memory: (float) $this->option('memory'), + stopWhenEmpty: (bool) $this->option('stop-when-empty'), + once: (bool) $this->option('once'), + )); + } +} diff --git a/src/queue/src/Console/RetryOutboxCommand.php b/src/queue/src/Console/RetryOutboxCommand.php new file mode 100644 index 000000000..f498cb472 --- /dev/null +++ b/src/queue/src/Console/RetryOutboxCommand.php @@ -0,0 +1,74 @@ +argument('uuid'))); + $database = $this->option('database'); + + if ($uuids === ['all']) { + $count = 0; + + foreach ($database === null ? $this->repository->getDatabases() : [$database] as $source) { + $count += $this->repository->retryAll($source); + } + + $this->components->info("Made {$count} outbox messages available for retry."); + + return self::SUCCESS; + } + + if ($uuids === [] || in_array('all', $uuids, true)) { + throw new InvalidArgumentException('Specify one or more job UUIDs, or "all" by itself.'); + } + + $status = self::SUCCESS; + + foreach ($uuids as $uuid) { + $found = $this->findOutboxMessage($uuid, $database); + + if ($found === null) { + $this->components->error("No outbox message matches UUID [{$uuid}]."); + $status = self::FAILURE; + } elseif ($this->repository->retry($found['database'], $uuid) === 0) { + $this->components->error("Outbox message [{$uuid}] has not been attempted or is currently leased."); + $status = self::FAILURE; + } else { + $this->components->info("Outbox message [{$uuid}] is available for retry."); + } + } + + return $status; + } +} diff --git a/src/queue/src/Events/OutboxBusy.php b/src/queue/src/Events/OutboxBusy.php new file mode 100644 index 000000000..4f6fea54e --- /dev/null +++ b/src/queue/src/Events/OutboxBusy.php @@ -0,0 +1,19 @@ + ? AND attempts = 0 THEN 1 ELSE 0 END) AS delayed, SUM(CASE WHEN lease_token IS NOT NULL AND available_at > ? THEN 1 ELSE 0 END) AS leased, - SUM(CASE WHEN attempts > 0 AND (lease_token IS NULL OR available_at <= ?) THEN 1 ELSE 0 END) AS retrying, + SUM(CASE WHEN lease_token IS NULL AND available_at > ? AND attempts > 0 THEN 1 ELSE 0 END) AS retrying, MIN(CASE WHEN available_at <= ? THEN available_at END) AS oldest_eligible', [$now, $now, $now, $now, $now], )->first(); diff --git a/src/queue/src/QueueServiceProvider.php b/src/queue/src/QueueServiceProvider.php index 296ed095f..5ddf5e9c4 100644 --- a/src/queue/src/QueueServiceProvider.php +++ b/src/queue/src/QueueServiceProvider.php @@ -16,17 +16,22 @@ use Hypervel\Queue\Console\FailedTableCommand; use Hypervel\Queue\Console\FlushFailedCommand; use Hypervel\Queue\Console\ForgetFailedCommand; +use Hypervel\Queue\Console\ForgetOutboxCommand; use Hypervel\Queue\Console\ListenCommand; use Hypervel\Queue\Console\ListFailedCommand; +use Hypervel\Queue\Console\ListOutboxCommand; use Hypervel\Queue\Console\MonitorCommand; +use Hypervel\Queue\Console\MonitorOutboxCommand; use Hypervel\Queue\Console\OutboxTableCommand; use Hypervel\Queue\Console\PauseCommand; use Hypervel\Queue\Console\PruneBatchesCommand; use Hypervel\Queue\Console\PruneFailedJobsCommand; +use Hypervel\Queue\Console\RelayCommand; use Hypervel\Queue\Console\RestartCommand; use Hypervel\Queue\Console\ResumeCommand; use Hypervel\Queue\Console\RetryBatchCommand; use Hypervel\Queue\Console\RetryCommand; +use Hypervel\Queue\Console\RetryOutboxCommand; use Hypervel\Queue\Console\TableCommand; use Hypervel\Queue\Console\WorkCommand; use Hypervel\Queue\Failed\DatabaseFailedJobProvider; @@ -65,17 +70,22 @@ public function register(): void FailedTableCommand::class, FlushFailedCommand::class, ForgetFailedCommand::class, + ForgetOutboxCommand::class, ListenCommand::class, ListFailedCommand::class, + ListOutboxCommand::class, MonitorCommand::class, + MonitorOutboxCommand::class, OutboxTableCommand::class, PauseCommand::class, PruneBatchesCommand::class, PruneFailedJobsCommand::class, + RelayCommand::class, RestartCommand::class, ResumeCommand::class, RetryBatchCommand::class, RetryCommand::class, + RetryOutboxCommand::class, TableCommand::class, WorkCommand::class, ]); diff --git a/tests/Queue/Outbox/OutboxCommandsTest.php b/tests/Queue/Outbox/OutboxCommandsTest.php new file mode 100644 index 000000000..a29876877 --- /dev/null +++ b/tests/Queue/Outbox/OutboxCommandsTest.php @@ -0,0 +1,254 @@ +make('config'); + $config->set('cache.default', 'worker-array'); + $config->set('queue.outbox.databases', [$config->string('database.default')]); + } + + /** + * Load only the shipped outbox schema. + */ + protected function migrateFreshUsing(): array + { + return [ + '--realpath' => true, + '--path' => dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php', + ]; + } + + public function testListingIsBoundedAndOmitsPayloadOptionsAndOwnershipTokens(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $error = str_repeat('Broker unavailable. ', 12) . "\nSecond error line."; + $first = $this->insert(['last_error' => $error]); + $this->insert(); + + $this->assertSame(0, Artisan::call('queue:outbox', ['--json' => true, '--limit' => 1])); + $output = Artisan::output(); + $rows = json_decode($output, true, flags: JSON_THROW_ON_ERROR); + $this->assertCount(1, $rows); + $this->assertSame($first, $rows[0]['uuid']); + $this->assertSame('SendInvoice', $rows[0]['job']); + $this->assertSame('eligible', $rows[0]['state']); + $this->assertSame(Date::now()->getTimestamp(), $rows[0]['available_at']); + $this->assertSame($error, $rows[0]['last_error']); + $this->assertArrayNotHasKey('payload', $rows[0]); + $this->assertArrayNotHasKey('options', $rows[0]); + $this->assertArrayNotHasKey('lease_token', $rows[0]); + $this->assertStringNotContainsString('private-content', $output); + + Artisan::call('queue:outbox', ['--limit' => 1]); + $output = Artisan::output(); + $this->assertStringContainsString('2040-01-01 12:00:00', $output); + $this->assertStringContainsString(Str::limit(Str::before($error, "\n"), 80), $output); + $this->assertStringNotContainsString('Second error line.', $output); + $this->assertStringNotContainsString($error, $output); + } + + public function testListingRejectsAnUnboundedLimit(): void + { + $this->expectException(InvalidArgumentException::class); + + Artisan::call('queue:outbox', ['--limit' => 1001]); + } + + public function testMonitorReportsDistinctStatesAndAlertsOnlyForEligibleAge(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $now = Date::now()->getTimestamp(); + $this->insert(['available_at' => $now + 60]); + $this->insert(['available_at' => $now + 60, 'attempts' => 1, 'lease_token' => (string) Str::uuid()]); + $this->insert(['available_at' => $now + 60, 'attempts' => 1]); + $seen = []; + $this->app->make('events')->listen(OutboxBusy::class, static function (OutboxBusy $event) use (&$seen): void { $seen[] = $event; }); + + Artisan::call('queue:outbox-monitor', ['--json' => true, '--age' => 10]); + $statistics = json_decode(Artisan::output(), true, flags: JSON_THROW_ON_ERROR)[0]; + $this->assertSame(3, $statistics['total']); + $this->assertSame(0, $statistics['eligible']); + $this->assertSame(1, $statistics['delayed']); + $this->assertSame(1, $statistics['leased']); + $this->assertSame(1, $statistics['retrying']); + $this->assertSame('OK', $statistics['status']); + $this->assertSame([], $seen); + + $this->insert(['available_at' => $now - 20]); + $this->insert(['available_at' => $now - 1, 'attempts' => 2]); + Artisan::call('queue:outbox-monitor', ['--json' => true, '--age' => 10]); + $statistics = json_decode(Artisan::output(), true, flags: JSON_THROW_ON_ERROR)[0]; + $this->assertSame(2, $statistics['eligible']); + $this->assertSame(1, $statistics['retrying']); + $this->assertSame($statistics['total'], $statistics['eligible'] + $statistics['delayed'] + $statistics['leased'] + $statistics['retrying']); + $this->assertCount(1, $seen); + $this->assertSame(5, $seen[0]->size); + $this->assertSame(20, $seen[0]->oldestEligibleAge); + } + + public function testRetryAllPreservesAttemptHistoryAndExcludesLiveAndNeverAttemptedRows(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $now = Date::now()->getTimestamp(); + $retry = $this->insert(['attempts' => 5, 'available_at' => $now + 500, 'last_error' => 'Broker offline']); + $live = $this->insert(['attempts' => 1, 'available_at' => $now + 500, 'lease_token' => (string) Str::uuid()]); + $new = $this->insert(['available_at' => $now + 500]); + + $this->assertSame(0, Artisan::call('queue:outbox-retry', ['uuid' => ['all']])); + $row = DB::table('job_outbox')->where('uuid', $retry)->first(); + $this->assertSame($now, (int) $row->available_at); + $this->assertSame(5, (int) $row->attempts); + $this->assertNull($row->last_error); + $this->assertSame($now + 500, (int) DB::table('job_outbox')->where('uuid', $live)->value('available_at')); + $this->assertSame($now + 500, (int) DB::table('job_outbox')->where('uuid', $new)->value('available_at')); + $this->assertSame(1, Artisan::call('queue:outbox-retry', ['uuid' => [$live, $new, 'missing']])); + $this->assertSame(0, Artisan::call('queue:outbox-retry', ['uuid' => [$retry], '--database' => DB::connection()->getName()])); + } + + public function testRetryRefusesCrossSourceUuidAmbiguityBeforeMutation(): void + { + $repository = m::mock(DatabaseOutboxRepository::class); + $repository->expects('getDatabases')->andReturn(['first', 'second']); + $repository->expects('find')->with('first', 'same')->andReturn((object) ['id' => 1]); + $repository->expects('find')->with('second', 'same')->andReturn((object) ['id' => 2]); + $this->app->instance(DatabaseOutboxRepository::class, $repository); + $this->expectException(RuntimeException::class); + $this->expectExceptionMessage('Select one using --database'); + + Artisan::call('queue:outbox-retry', ['uuid' => ['same']]); + } + + public function testForgetRequiresConfirmationAndRefusesLiveLeases(): void + { + $uuid = $this->insert(['attempts' => 1, 'lease_token' => (string) Str::uuid(), 'available_at' => Date::now()->getTimestamp() + 60]); + $prompt = 'Are you sure you want to run this command?'; + $this->artisan('queue:outbox-forget', ['uuid' => $uuid])->expectsConfirmation($prompt, 'no')->expectsOutputToContain('Command cancelled.')->assertExitCode(1); + $this->assertDatabaseHas('job_outbox', ['uuid' => $uuid]); + $this->artisan('queue:outbox-forget', ['uuid' => $uuid])->expectsConfirmation($prompt, 'yes')->assertExitCode(1); + $this->assertDatabaseHas('job_outbox', ['uuid' => $uuid]); + + $this->assertSame(1, Artisan::call('queue:outbox-forget', ['uuid' => $uuid, '--no-interaction' => true])); + $this->assertStringContainsString('Command cancelled.', Artisan::output()); + $this->assertSame(1, Artisan::call('queue:outbox-forget', ['uuid' => $uuid, '--no-interaction' => true, '--force' => true])); + $this->assertDatabaseHas('job_outbox', ['uuid' => $uuid]); + + DB::table('job_outbox')->where('uuid', $uuid)->update(['available_at' => Date::now()->getTimestamp() - 1]); + $this->assertSame(0, Artisan::call('queue:outbox-forget', ['uuid' => $uuid, '--no-interaction' => true, '--force' => true])); + $this->assertDatabaseMissing('job_outbox', ['uuid' => $uuid]); + } + + #[DataProvider('lockReplacement')] + public function testForgetReleasesOnlyItsExactUniqueOwnerWithoutHydratingOtherData(bool $replaced): void + { + $cache = Cache::store('worker-array'); + $lock = $cache->lock('unique-key'); + $this->assertTrue($lock->get()); + $uuid = $this->insert(['payload' => json_encode([ + 'data' => ['command' => 'Do not deserialize this job.'], + 'illuminate:log:context' => ['hidden' => [ + 'laravel_unique_job_cache_store' => serialize('worker-array'), + 'laravel_unique_job_key' => serialize('unique-key'), + 'laravel_unique_job_lock_owner' => serialize($lock->owner()), + 'unrelated' => 'Do not hydrate this context.', + ]], + ], JSON_THROW_ON_ERROR)]); + + if ($replaced) { + $lock->release(); + $replacement = $cache->lock('unique-key'); + $this->assertTrue($replacement->get()); + } + + $this->artisan('queue:outbox-forget', ['uuid' => $uuid]) + ->expectsConfirmation('Are you sure you want to run this command?', 'yes') + ->assertExitCode(0); + $this->assertDatabaseMissing('job_outbox', ['uuid' => $uuid]); + $this->assertSame(! $replaced, $cache->lock('unique-key')->get()); + } + + /** + * Cover current ownership and a later dispatch's replacement lock. + */ + public static function lockReplacement(): array + { + return [[false], [true]]; + } + + public function testForgetReportsUnresolvableOwnershipAfterDeletingWithoutGuessingTheDefaultStore(): void + { + $cache = Cache::store('worker-array'); + $lock = $cache->lock('unique-key'); + $this->assertTrue($lock->get()); + $uuid = $this->insert(['payload' => json_encode([ + 'illuminate:log:context' => ['hidden' => [ + 'laravel_unique_job_cache_store' => serialize(null), + 'laravel_unique_job_key' => serialize('unique-key'), + 'laravel_unique_job_lock_owner' => serialize($lock->owner()), + ]], + ], JSON_THROW_ON_ERROR)]); + $exceptions = m::mock(ExceptionHandler::class); + $exceptions->expects('report')->with(m::type(LogicException::class), m::on(fn (array $context): bool => $context['job_uuid'] === $uuid)); + $this->app->instance(ExceptionHandler::class, $exceptions); + + $this->artisan('queue:outbox-forget', ['uuid' => $uuid]) + ->expectsConfirmation('Are you sure you want to run this command?', 'yes') + ->expectsOutputToContain('was deleted, but unique-lock cleanup failed') + ->assertExitCode(1); + $this->assertDatabaseMissing('job_outbox', ['uuid' => $uuid]); + $this->assertFalse($cache->lock('unique-key')->get()); + } + + /** + * Insert an operational row using the shipped repository and schema. + */ + protected function insert(array $attributes = []): string + { + $uuid = (string) Str::uuid(); + $now = Date::now()->getTimestamp(); + $this->app->make(DatabaseOutboxRepository::class)->insert(DB::connection(), [array_replace([ + 'uuid' => $uuid, + 'connection' => 'redis', + 'queue' => 'invoices', + 'payload' => '{"displayName":"SendInvoice","data":{"command":"private-content"}}', + 'options' => '{"private-content":true}', + 'deliver_at' => null, + 'attempts' => 0, + 'available_at' => $now, + 'lease_token' => null, + 'last_error' => null, + 'created_at' => $now, + ], $attributes)]); + + return $uuid; + } +} diff --git a/tests/Queue/RelayCommandTest.php b/tests/Queue/RelayCommandTest.php new file mode 100644 index 000000000..bebad1f01 --- /dev/null +++ b/tests/Queue/RelayCommandTest.php @@ -0,0 +1,80 @@ +expects('run')->with('orders', m::on(function (OutboxWorkerOptions $options): bool { + $this->assertEquals(new OutboxWorkerOptions( + batch: 50, + sleep: 0.25, + concurrency: 4, + maxJobs: 500, + maxTime: 60, + memory: 256, + stopWhenEmpty: true, + once: true, + ), $options); + + return true; + }))->andReturn(12); + $this->app->instance(OutboxWorker::class, $worker); + + $this->assertSame(12, Artisan::call('queue:relay', [ + 'database' => 'orders', + '--batch' => '50', + '--sleep' => '0.25', + '--concurrency' => '4', + '--max-jobs' => '500', + '--max-time' => '60', + '--memory' => '256', + '--stop-when-empty' => true, + '--once' => true, + ])); + } + + public function testCommandDefaultsToAllConfiguredSources(): void + { + $worker = m::mock(OutboxWorker::class); + $worker->expects('run')->with(null, m::on(fn (OutboxWorkerOptions $options): bool => $options == new OutboxWorkerOptions))->andReturn(0); + $this->app->instance(OutboxWorker::class, $worker); + + $this->assertSame(0, Artisan::call('queue:relay')); + } + + #[DataProvider('invalidOptions')] + public function testInvalidOptionsFailBeforeTheWorkerStarts(array $options): void + { + $this->app->instance(OutboxWorker::class, m::mock(OutboxWorker::class)); + $this->expectException(InvalidArgumentException::class); + + Artisan::call('queue:relay', $options); + } + + /** + * Provide invalid runtime limits at the command boundary. + */ + public static function invalidOptions(): array + { + return [ + 'empty batch' => [['--batch' => 0]], + 'oversized batch' => [['--batch' => 1001]], + 'hot-loop sleep' => [['--sleep' => 0]], + 'no capacity' => [['--concurrency' => 0]], + 'negative maximum' => [['--max-jobs' => -1]], + ]; + } +} From 043e348bac2c4067ca0f37c852af1f4d783b8830 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 21:56:50 +0000 Subject: [PATCH 15/33] feat(opentelemetry): observe outbox delivery at batch boundaries Record confirmed delivered messages, batch publication duration and oldest eligible message age from guarded outbox events. Register listeners only when their metrics are enabled, without querying backlog, resolving queues, decoding payloads or starting another producer lifecycle. Use error-tagged duration observations for failed-batch counts instead of a redundant counter. Keep latency attributes independent of exception type and align its buckets with whole-second values to limit series and recording overhead. Expose matching published and normalized defaults, with tests for partial acceptance, settlement failure, disabled listeners and bounded metric series. --- src/opentelemetry/config/opentelemetry.php | 3 + .../Instrumentation/QueueInstrumentation.php | 77 +++++++++++++++++++ .../src/Support/ConfigurationNormalizer.php | 3 + .../QueueInstrumentationTest.php | 39 ++++++++++ .../PublishedConfigurationTest.php | 3 + 5 files changed, 125 insertions(+) diff --git a/src/opentelemetry/config/opentelemetry.php b/src/opentelemetry/config/opentelemetry.php index 1df81d4e9..a09dd81ce 100644 --- a/src/opentelemetry/config/opentelemetry.php +++ b/src/opentelemetry/config/opentelemetry.php @@ -218,6 +218,9 @@ 'messaging.client.consumed.messages' => true, 'messaging.process.duration' => true, 'hypervel.queue.jobs' => false, + 'hypervel.queue.outbox.delivered.messages' => true, + 'hypervel.queue.outbox.delivery.duration' => true, + 'hypervel.queue.outbox.delivery.latency' => true, ], ], SchedulerInstrumentation::class => [ diff --git a/src/opentelemetry/src/Instrumentation/QueueInstrumentation.php b/src/opentelemetry/src/Instrumentation/QueueInstrumentation.php index 65d46e49b..a31063427 100644 --- a/src/opentelemetry/src/Instrumentation/QueueInstrumentation.php +++ b/src/opentelemetry/src/Instrumentation/QueueInstrumentation.php @@ -22,6 +22,8 @@ use Hypervel\Queue\Events\JobQueued; use Hypervel\Queue\Events\JobQueueingFailed; use Hypervel\Queue\Events\JobTimedOut; +use Hypervel\Queue\Events\OutboxBatchDelivered; +use Hypervel\Queue\Events\OutboxBatchDeliveryFailed; use Hypervel\Queue\Queue; use Hypervel\Queue\QueueManager; use Hypervel\Queue\TimeoutExceededException; @@ -60,6 +62,12 @@ class QueueInstrumentation extends AbstractInstrumentation protected const string DEPTH_METRIC = 'hypervel.queue.jobs'; + protected const string OUTBOX_DELIVERED_METRIC = 'hypervel.queue.outbox.delivered.messages'; + + protected const string OUTBOX_DURATION_METRIC = 'hypervel.queue.outbox.delivery.duration'; + + protected const string OUTBOX_LATENCY_METRIC = 'hypervel.queue.outbox.delivery.latency'; + protected const string CONNECTION_ATTRIBUTE = 'hypervel.queue.connection'; protected const array DURATION_BOUNDARIES = [ @@ -89,6 +97,12 @@ class QueueInstrumentation extends AbstractInstrumentation protected ?HistogramInterface $processDuration = null; + protected ?CounterInterface $outboxDelivered = null; + + protected ?HistogramInterface $outboxDuration = null; + + protected ?HistogramInterface $outboxLatency = null; + protected bool $propagation = false; /** @var array */ @@ -131,6 +145,7 @@ protected function registerInstrumentation(): void } $this->registerMetrics(); + $this->registerOutboxMetrics(); $producerMetrics = $this->sentMessages !== null || $this->sendDuration !== null; $producerTelemetry = $this->tracer !== null || $producerMetrics; @@ -241,6 +256,68 @@ protected function registerMetrics(): void } } + /** + * Register outbox delivery metrics without observing or resolving ordinary queue operations. + */ + protected function registerOutboxMetrics(): void + { + $meter = null; + + if ($this->metricEnabled(self::OUTBOX_DELIVERED_METRIC)) { + $meter = $this->meterProvider->getMeter('hypervel.queue'); + $this->outboxDelivered = $meter->createCounter(self::OUTBOX_DELIVERED_METRIC, '{message}', 'Outbox messages confirmed by their destination queue.'); + } + + if ($this->metricEnabled(self::OUTBOX_DURATION_METRIC)) { + $meter ??= $this->meterProvider->getMeter('hypervel.queue'); + $this->outboxDuration = $meter->createHistogram( + self::OUTBOX_DURATION_METRIC, + 's', + 'Duration of outbox connection batch publication.', + ['ExplicitBucketBoundaries' => self::DURATION_BOUNDARIES], + ); + } + + if ($this->metricEnabled(self::OUTBOX_LATENCY_METRIC)) { + $meter ??= $this->meterProvider->getMeter('hypervel.queue'); + $this->outboxLatency = $meter->createHistogram( + self::OUTBOX_LATENCY_METRIC, + 's', + 'Age of the oldest eligible message at the start of each outbox connection publication batch.', + ['ExplicitBucketBoundaries' => [1, 5, 10, 30, 60, 300, 600, 1800, 3600]], + ); + } + + if ($meter !== null) { + $this->events->listen([OutboxBatchDelivered::class, OutboxBatchDeliveryFailed::class], function (OutboxBatchDelivered|OutboxBatchDeliveryFailed $event): void { + $this->recordOutboxDelivery($event); + }); + } + } + + /** + * Record confirmed acceptance separately from publication or settlement failure. + */ + protected function recordOutboxDelivery(OutboxBatchDelivered|OutboxBatchDeliveryFailed $event): void + { + $attributes = [ + self::CONNECTION_ATTRIBUTE => $event->connectionName, + 'hypervel.queue.outbox.database' => $event->database, + ]; + + if ($event->accepted > 0) { + $this->outboxDelivered?->add($event->accepted, $attributes); + } + + $this->outboxLatency?->record($event->oldestEligibleAge, $attributes); + + if ($event instanceof OutboxBatchDeliveryFailed) { + $attributes[ErrorAttributes::ERROR_TYPE] = $event->exception::class; + } + + $this->outboxDuration?->record($event->duration, $attributes); + } + /** * Inject propagation and start a persistent producer span. */ diff --git a/src/opentelemetry/src/Support/ConfigurationNormalizer.php b/src/opentelemetry/src/Support/ConfigurationNormalizer.php index 39b856ecf..8c471e27f 100644 --- a/src/opentelemetry/src/Support/ConfigurationNormalizer.php +++ b/src/opentelemetry/src/Support/ConfigurationNormalizer.php @@ -434,6 +434,9 @@ protected function instrumentationDefaults(): array 'messaging.client.consumed.messages' => true, 'messaging.process.duration' => true, 'hypervel.queue.jobs' => false, + 'hypervel.queue.outbox.delivered.messages' => true, + 'hypervel.queue.outbox.delivery.duration' => true, + 'hypervel.queue.outbox.delivery.latency' => true, ], ], SchedulerInstrumentation::class => [ diff --git a/tests/OpenTelemetry/Instrumentation/QueueInstrumentationTest.php b/tests/OpenTelemetry/Instrumentation/QueueInstrumentationTest.php index 0be3ffebc..b6fe135e8 100644 --- a/tests/OpenTelemetry/Instrumentation/QueueInstrumentationTest.php +++ b/tests/OpenTelemetry/Instrumentation/QueueInstrumentationTest.php @@ -26,6 +26,8 @@ use Hypervel\Queue\Events\JobQueueing; use Hypervel\Queue\Events\JobQueueingFailed; use Hypervel\Queue\Events\JobTimedOut; +use Hypervel\Queue\Events\OutboxBatchDelivered; +use Hypervel\Queue\Events\OutboxBatchDeliveryFailed; use Hypervel\Queue\NullQueue; use Hypervel\Queue\QueueManager; use Hypervel\Queue\SyncQueue; @@ -155,6 +157,8 @@ public function testRegistersNothingWhenEveryOutputIsDisabled(): void $this->assertFalse($this->events->hasListeners(JobQueueingFailed::class)); $this->assertFalse($this->events->hasListeners(JobProcessing::class)); $this->assertFalse($this->events->hasListeners(JobAttempted::class)); + $this->assertFalse($this->events->hasListeners(OutboxBatchDelivered::class)); + $this->assertFalse($this->events->hasListeners(OutboxBatchDeliveryFailed::class)); $this->assertSame(0, QueueInstrumentationPayloadQueue::payloadCallbackCount()); $this->assertSame(0, $this->clock->calls); } @@ -982,6 +986,41 @@ public function testPersistentOnlyConnectionsRegisterNoStaticPayloadHook(): void $this->assertSame(0, QueueInstrumentationPayloadQueue::payloadCallbackCount()); } + public function testOutboxMetricsCountAcceptanceAndSettlementFailuresWithoutASecondProducerLifecycle(): void + { + $this->instrumentation()->register($this->options([ + 'traces' => false, + 'propagation' => false, + 'metrics' => [ + 'hypervel.queue.outbox.delivered.messages' => true, + 'hypervel.queue.outbox.delivery.duration' => true, + 'hypervel.queue.outbox.delivery.latency' => true, + ], + ])); + $this->assertFalse($this->events->hasListeners(JobPayloadFinalizing::class)); + $this->assertFalse($this->events->hasListeners(JobQueued::class)); + $this->events->dispatch(new OutboxBatchDelivered('primary', 'redis', 3, 0.25, 10)); + $this->events->dispatch(new OutboxBatchDeliveryFailed('primary', 'redis', 2, 1, 0.5, 20, new RuntimeException('Partial acceptance.'))); + $this->events->dispatch(new OutboxBatchDeliveryFailed('primary', 'redis', 1, 0, 0.75, 30, new RuntimeException('Settlement failed.'))); + $this->metricReader->collect(); + + $delivered = $this->sumPoint('hypervel.queue.outbox.delivered.messages'); + $this->assertSame(6, $delivered->value); + $this->assertSame('primary', $delivered->attributes->get('hypervel.queue.outbox.database')); + $this->assertSame('redis', $delivered->attributes->get('hypervel.queue.connection')); + $duration = $this->metric('hypervel.queue.outbox.delivery.duration')->data->dataPoints; + $latency = $this->metric('hypervel.queue.outbox.delivery.latency')->data->dataPoints; + $failedBatches = array_filter($duration, static fn (HistogramDataPoint $point): bool => $point->attributes->get(ErrorAttributes::ERROR_TYPE) === RuntimeException::class); + $this->assertSame(2, array_sum(array_column($failedBatches, 'count'))); + $this->assertSame(3, array_sum(array_column($duration, 'count'))); + $this->assertSame(1.5, array_sum(array_column($duration, 'sum'))); + $this->assertCount(1, $latency); + $this->assertEquals(60, array_sum(array_column($latency, 'sum'))); + $this->assertCount(3, $this->metricExporter->collect()); + $this->assertSame([], $this->spanExporter->getSpans()); + $this->assertSame(0, $this->clock->calls); + } + public function testDepthCollectionRunsOnlyInEventWorkerZeroAndIsolatesBackendFailures(): void { $queues = m::mock(QueueManager::class); diff --git a/tests/OpenTelemetry/PublishedConfigurationTest.php b/tests/OpenTelemetry/PublishedConfigurationTest.php index 961a6d1cd..3edf92667 100644 --- a/tests/OpenTelemetry/PublishedConfigurationTest.php +++ b/tests/OpenTelemetry/PublishedConfigurationTest.php @@ -49,6 +49,9 @@ public function testPublishedDefaultsExposeTheCompleteMetricSurface(): void 'hypervel.object_pool.objects' => true, 'hypervel.object_pool.pending_requests' => true, 'hypervel.queue.jobs' => false, + 'hypervel.queue.outbox.delivered.messages' => true, + 'hypervel.queue.outbox.delivery.duration' => true, + 'hypervel.queue.outbox.delivery.latency' => true, 'hypervel.scheduler.task.duration' => true, 'hypervel.scheduler.task.executions' => true, 'hypervel.server.connections' => true, From f9da181710ada46e98aee2c450f5eb063c06fd47 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 21:56:50 +0000 Subject: [PATCH 16/33] docs(queue): align outbox plan with relay and operations contracts Specify coordinator-owned admission, resource-isolated delivery phases, cancellation cleanup, observer reporting and value-based result validation. Clarify cache identity and exact unique-lock cleanup, scriptable confirmation and disjoint monitoring states. Document the reduced batch telemetry surface and remaining OpenTelemetry reference updates. Record completed runtime and operator scope, with live-engine coverage, remaining API integration, documentation, approved performance checks and final verification still required. --- ...pervel-transactional-queue-outbox-codex.md | 34 ++++++++++++------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md index 9a38dda18..9a0e337df 100644 --- a/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md +++ b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md @@ -22,9 +22,9 @@ The result is a queue feature, not a generic event bus, inbox/deduplication syst ### Implementation status -Implemented: framework prerequisites, public selection and wrapper propagation, schema/configuration, owner-aware staging and repository operations, and Redis/Horizon/database/SQS/pooled raw publication. It is not yet an end-to-end deployable outbox. +Implemented: framework prerequisites, public selection and wrapper propagation, schema/configuration, owner-aware staging and repository operations, Redis/Horizon/database/SQS/pooled raw publication, relay/worker resource boundaries, operator commands, and guarded delivery events/telemetry. -Remaining work: relay and worker resource boundaries, operator commands, delivery events and telemetry; live-engine/concurrency and complete API-integration coverage; canonical documentation, approved performance checks and final feature verification. +Next: complete the remaining live-engine/concurrency and API-integration coverage. Canonical documentation, approved performance checks and final verification also remain. The full feature is not yet complete. ### Hypervel @@ -85,6 +85,7 @@ Implement these at their owning boundary, before outbox integration. If already 6. **Broadcast public-property extraction.** Skip uninitialized public properties before reading them in `BroadcastEvent::getPayloadFromEvent()`. This supports untouched `Queueable` state and application-declared typed properties without changing initialized nulls, value formatting or `broadcastWith()` behavior. Do not add an outbox-specific property blacklist or reflection cache. 7. **Inline after-commit ownership.** When `deferBatchEnqueueAfterCommit()` has no applicable transaction for its selected connection, invoke the enqueue callback on the current queue and return without transferring dispatch-lock ownership, registering rollback callbacks or reacquiring a pooled queue. Inline failure belongs to the calling dispatch/failover attempt; only genuinely deferred callbacks own final cleanup. Keep the helper void and preserve local-driver return values. +8. **Cache store identity.** In `CacheManager::resolve()`, fill a concrete `Repository`'s missing name from the normalized store configuration, preserving creator-supplied names and custom contract implementations. Add an internal construction-time `setName()` beside `getName()`. Memoized repositories retain the underlying repository's name while keeping events disabled. This fixes unique-job cleanup after missing-model errors for documented custom stores and memoized stores; do not name ad hoc repositories in `build()` or add per-lock detection. ### Outbox integration @@ -301,7 +302,7 @@ Do not create a listed type if it would only forward one call. Do not combine ro Container-cached outbox services remain immutable/stateless. Database capability detection stays on the live connection that owns the capability, without another service or cache. `OutboxMessage`, `OutboxBatchResult`, and `OutboxWorkerOptions` are immutable per-operation values created directly rather than container-resolved. Loop counters, current claims, stop reasons, and per-delivery data stay method-local. The outer concurrency task may retain only scalar/immutable values and result data: never a database connection, query builder, resolved queue instance, pool lease, or transport client. -Resource ownership is a hard invariant: **no relay database or transport connection remains borrowed while the relay performs work that does not require it**. Explicit idle release does not cover retained write sessions, every custom driver, or arbitrary callbacks. Use fresh child-coroutine boundaries for (1) source claim, (2) each target-connection publication group, and (3) that group's settlement. `Waiter(-1)` leaves phase duration to configured database/transport timeouts and daemon cancellation, not its unrelated ten-second default. Each child uses a fresh context, returns materialized values and ends before the next phase. Request-side staging instead uses the exact already-borrowed business connection for atomicity. +Resource ownership is a hard invariant: **no relay database or transport connection remains borrowed while the relay performs work that does not require it**. Explicit idle release does not cover retained write sessions, every custom driver, or arbitrary callbacks. Use fresh child-coroutine boundaries for (1) source claim, (2) each target-connection publication group, and (3) that group's settlement. `Waiter(-1)` with `waitForChildTermination: true` leaves phase duration to configured database/transport timeouts and ensures cancellation joins child cleanup. Each child uses a fresh context, returns materialized values and ends before the next phase. Request-side staging instead uses the exact already-borrowed business connection for atomicity. Reuse the prerequisite `DispatchLockContext` capture/snapshot lifecycle and existing hidden scalar unique metadata. Never persist live cache provenance or captured release closures, or create an outbox-owned lock registry. @@ -365,7 +366,7 @@ interface PublishesOutboxMessages } ``` -`OutboxMessage` carries row ID/job UUID/lease token/attempt, finalized payload, unforwarded logical queue, integer `deliverAt` (nullable), `eligibleAt` (pre-claim availability), `createdAt`, and opaque options. `OutboxBatchResult` contains unique lists of accepted and deferred row IDs plus an optional failure. Accepted and deferred IDs must be disjoint subsets of the input; a clean result accounts for every input through their union. Explicit transport rejects return confirmed IDs plus one aggregate exception containing the reject count, first reason and full response context; a later-chunk exception returns earlier confirmed IDs plus that exception. Cancellation is rethrown. Optimized implementations return no deferred IDs; sequential fallback uses them for future rows. Failed/unattempted rows are input minus accepted minus deferred and receive the same bounded failure text. Foreign/overlapping IDs or unaccounted-for input without failure are a broken contract: report a `LogicException` and settle nothing from that call. Transport response IDs stay inside the driver. +`OutboxMessage` carries row ID/job UUID/lease token/attempt, finalized payload, unforwarded logical queue, integer `deliverAt` (nullable), `eligibleAt` (pre-claim availability), `createdAt`, and opaque options. `OutboxBatchResult` contains arrays of unique accepted and deferred integer row IDs plus an optional failure; array keys do not affect settlement. Accepted and deferred IDs must be disjoint subsets of the input; a clean result accounts for every input through their union. Only messages with non-null `deliverAt` may be deferred; do not require that time to remain in the future during validation. Explicit transport rejects return confirmed IDs plus one aggregate exception containing the reject count, first reason and full response context; a later-chunk exception returns earlier confirmed IDs plus that exception. Cancellation is rethrown. Optimized implementations return no deferred IDs; sequential fallback uses them for future rows. Failed/unattempted rows are input minus accepted minus deferred and receive the same bounded failure text. Invalid results become an empty accepted/deferred result with a `LogicException`: retain every input row through ordinary token-guarded failure release/backoff and report the contract violation. Transport response IDs stay inside the driver. Use typed message/result boundaries and existing driver option handling; do not add a second transport validator. The generic fallback passes stored options through `pushRaw()`. Native publication errors and partial/ambiguous outcomes use the result contract above. @@ -383,7 +384,7 @@ Do not add raw-delay methods to the general `Queue` contract. Custom durable dri ### Claim -- The concurrency-owning relay task calls claim through a fresh `Waiter` child with no copied request/transaction context. That child alone resolves the source database, completes the claim, returns immutable `OutboxMessage` values, and ends so `ConnectionResolver` releases the database lease before any broker work starts. +- The coordinator waits for publication capacity, then claims through a fresh `Waiter` child with no copied request/transaction context. That child alone resolves the source database, completes the claim, returns immutable `OutboxMessage` values, and ends so `ConnectionResolver` releases the database lease before any broker work starts. Count actual claimed rows before starting their concurrent delivery task. Claims are serial per daemon, like `queue:work` pops; publication/settlement tasks and independent daemons remain concurrent. - Claim at most the validated worker batch limit eligible rows ordered by `available_at`, then `id`. - Eligibility is `available_at <= now`. A claim atomically sets one fresh UUID lease token, `available_at = now + retry_after`, and increments attempts. This moves live leases out of the hot eligibility range without a second timestamp/filter. Use an explicit transaction only for the separate locking-select/update path; never hold SQL locks during broker calls. - PostgreSQL: one autocommitted CTE using `FOR UPDATE SKIP LOCKED` plus `UPDATE ... RETURNING`, returning the selected pre-update availability as `eligibleAt` with the updated token/attempt. No extra BEGIN/COMMIT round trips. Updating the indexed timestamp trades HOT-update eligibility for avoiding repeated live-lease scans; validate this tradeoff in the approved benchmark. @@ -398,7 +399,7 @@ Do not add raw-delay methods to the general `Queue` contract. Custom durable dri - Group a claim by target connection, preserving order within each group. The driver partitions for existing Redis destination scripts, SQS URL/count/bytes, or database binding limits. This permits multi-row database inserts across queues and one pooled Beanstalk lease per connection batch without replacing existing driver mechanisms. - Publish each group in its own fresh child. Only that child resolves the queue, borrows its transport and returns a materialized result. Use the optional raw-batch contract or shared sequential fallback. Like `queue:retry`, stored rows honor the recorded connection's current configuration, including an operator deliberately changing it to sync/null; add no driver-kind rejection. Document draining before configuration changes that discard durability or invalidate stored transport options. Native send errors retain/back off affected rows. - The shared fallback moves rows whose `deliver_at` is still in the future into the result's deferred set without calling `pushRaw()`, then publishes the due rows in order and returns the accepted prefix plus any exception that stopped it. `QueuePoolProxy` invokes this same operation while its one borrowed queue is live when that queue lacks the SPI. Settlement releases deferred rows token-safely with `available_at = deliver_at`, restores the claim's attempt increment, and leaves `last_error` untouched. Deferral is not a publication failure: it does not occupy the task's failure-sleep slot or emit `OutboxBatchDeliveryFailed`. -- The driver's existing publish result is the acknowledgment—Redis/database replies, SQS success/failure entries, and Beanstalk IDs. Add no probe or extra broker round trip. Validate the returned accepted row IDs before settlement; an invalid result retains every row in that call and reports the driver-contract failure. +- The driver's existing publish result is the acknowledgment—Redis/database replies, SQS success/failure entries, and Beanstalk IDs. Add no probe or extra broker round trip. Validate results before settlement using the raw-batch contract above; invalid results release every input row into normal failure backoff without deleting any. - Settle each completed group immediately in a fresh child and short source transaction: delete accepted rows using `id IN (...)` plus `lease_token`; clear deferred tokens, restore availability to `deliver_at`, undo the claim's attempt increment and preserve `last_error`; release failed/unattempted rows with one token-guarded update per binding-safe chunk. Choose one jittered deadline per capped retry bucket using `RETRY_DELAYS`, random 50%-100% jitter and a minimum one-second delay. A fixed `CASE WHEN attempts <= 1 THEN ? ... ELSE ? END` binds seven deadlines, not per-row cases; clear the token and bind one bounded error text. Only ID lists require size-dependent chunking. If fewer rows match, retain successful settlement and leave newer owners untouched; report ownership loss once. End the child before the next group. - A failed connection group does not prevent publication/settlement of the other groups in the same claim. Collect and report their errors after all non-cancelled groups have been attempted, identifying affected row IDs, job UUIDs, and payload display names; decode display names only on failure so successful relay batches pay no diagnostic parsing cost. Cancellation stops further publication and leaves unsettled leases to expire. - After any result failure, keep that relay task's concurrency slot occupied for its positive `--sleep` interval. Row backoff alone cannot protect a misconfigured/unavailable target from a large backlog of first-attempt rows; bounded occupied slots throttle further claims without distributed circuit-breaker state. @@ -413,28 +414,33 @@ Do not add raw-delay methods to the general `Queue` contract. Custom durable dri Transactional rollback uses the prerequisite independent snapshot, including its captured release context. Existing deferred-dispatch callbacks retain their normal ownership lifecycle. Successful publication leaves locks for normal job processing. -`queue:outbox-forget` reads existing hidden cache-store/key/owner scalars with object deserialization disabled; never hydrate arbitrary context or unserialize the queued command. Delete the non-live row first, then release the owner-matched unique lock. Keep this cleanup in a protected command method receiving the stored envelope so integrations can establish a context-dependent cache namespace without copying SQL or changing the relay. Commands already resolve through the container; no new registry is needed. Cache and SQL cannot commit atomically: malformed metadata/post-delete cache failure is reported and may require manual cleanup, but never restores or republishes deliberately forgotten work. Debounce tokens have bounded TTL and need no explicit forget cleanup. No outbox lock abstraction or persisted release closure. +`queue:outbox-forget` reads only the existing hidden cache-store/key/owner scalars with object deserialization disabled; never hydrate arbitrary context or unserialize the queued command. Delete the non-live row first, then release the owner-matched unique lock. No unique key means no cleanup; otherwise require nonempty string store/key/owner and an owner-capable, resolvable store. A null store name can describe an ad hoc `uniqueVia()` repository, not necessarily the default store: never guess or force-release. Keep cleanup in a protected command method receiving the stored envelope so integrations can resolve custom repositories or establish a context-dependent cache namespace without copying SQL or changing the relay. Commands already resolve through the container; no new registry is needed. Cache and SQL cannot commit atomically: unresolved/malformed ownership or post-delete cache failure is reported with the job UUID and explicit notice that the row was deleted and lock cleanup failed; never restore or republish deliberately forgotten work. Debounce tokens have bounded TTL and need no explicit forget cleanup. No outbox lock abstraction or persisted release closure. ## Worker and operations Register these commands in `QueueServiceProvider`: - `queue:relay {database?}` — long-running relay plus `--once`, `--stop-when-empty`, `--batch=100`, `--sleep=1`, `--concurrency=1`, `--max-jobs=0`, `--max-time=0`, and `--memory=128`; validate batch 1–1,000 and positive sleep/concurrency. Sleep accepts fractions like `queue:work`. `--once` makes one synchronous round-robin pass, at most one bounded claim per configured/pinned database, with no empty/outage sleep. `--stop-when-empty` joins active tasks and exits after a complete source pass finds no eligible row; delayed/backoff rows do not keep it alive. `--max-jobs` counts claimed rows regardless of result; cap every final claim to the remaining allowance; -- `queue:outbox {database?}` — pending-row diagnostics by configured database, ID, UUID, job display name read from the bounded payload result, target, attempts, lease state, age, and last error, ordered by ID and bounded by validated `--limit=100` (maximum 1,000), with `--json` matching `queue:failed`; otherwise visit all configured sources; never render payloads, transport options, context, or lock-owner tokens; +- `queue:outbox {database?}` — pending-row diagnostics by configured database, ID, UUID, job display name read from the bounded payload result, target, attempts, lease state, age, and last error, ordered by ID and bounded by validated `--limit=100` (maximum 1,000), with `--json` matching `queue:failed`; otherwise visit all configured sources. Tables format availability as a date-time and abbreviate errors to their first line; JSON retains epoch values and complete stored errors. Never render payloads, transport options, context, or lock-owner tokens; - `queue:outbox-monitor {database?}` — total, eligible, delayed, leased, and retrying counts plus the oldest currently eligible row's age, with explicit `--max` / `--age` alert thresholds and `--json` output for one/all configured databases; use one portable aggregate query with conditional counts and `MIN(CASE WHEN available_at <= ? THEN available_at END)` so counts and age share one read, and emit `OutboxBusy` like the existing `queue:monitor` command without another index or payload scan. Eligibility uses the claim predicate, and age starts at `available_at`, so intentionally delayed jobs do not create false alerts; - `queue:outbox-retry {uuid?*}` — targets only rows with `attempts > 0`, clears errors/leases, and makes selected rows (or explicit `all`) immediately available without erasing attempt history; accept `--database=` to pin a configured source, otherwise visit all sources. `all` snapshots each source's maximum ID among retryable rows and updates through that ceiling in bounded primary-key cursor chunks; an explicit UUID found on multiple unpinned sources fails rather than updating an ambiguous match. A never-attempted row waiting for its intended delivery time is not retryable; a previously failed row remains retryable regardless of its current backoff. Always reject live leases; -- `queue:outbox-forget {uuid}` — confirmation, live-lease rejection, token-safe delete, and owner-matched unique-lock cleanup from the payload metadata; accept `--database=` to pin a configured source and fail rather than guess if an unpinned UUID exists on more than one source. There is no force path because an in-flight publication cannot be recalled; +- `queue:outbox-forget {uuid}` — confirmation, live-lease rejection, token-safe delete, and owner-matched unique-lock cleanup from the payload metadata; accept `--database=` to pin a configured source and fail rather than guess if an unpinned UUID exists on more than one source. Use `ConfirmableTrait` with confirmation required in every environment; `--force` skips only the prompt, never the live-lease guard, because an in-flight publication cannot be recalled; - `make:queue-outbox-table` / `queue:outbox-table` — migration generation matching existing queue table commands. The worker follows the relevant `queue:work` operational conventions: `queue:restart`, bounded coroutine concurrency, POSIX signals, memory/max-time exits, exception reporting, and no swallowed cancellation. It does not consult maintenance mode: relaying only moves committed jobs to their queue, while normal workers already control whether those jobs execute. One coordinator admits work in configured-database round-robin order only when `WaitConcurrent` has a free slot, so counters, source fairness, stop decisions, and the remaining `--max-jobs` allowance stay local and exact. -Retry/forget load mutation targets from the writer using existing query-builder support, not replica reads or a separately borrowed connection. Recheck token/live-lease predicates in the modifying statement so a concurrent claim cannot be reset/deleted. A live lease is `lease_token IS NOT NULL AND available_at > now`; expired tokens do not prevent recovery. Monitor counts distinguish those live leases from tokenless scheduled/backoff rows and use `available_at <= now` for eligibility. +- Cap each claim to `min(batch, maxJobs - admitted)` when limited. Wait for capacity with a bounded timeout so stop/restart checks remain responsive; do not reserve allowances for parallel claims. +- A failed claim is reported and counts as a no-work visit. Skip that source for `--sleep` using a coordinator-local retry-deadline map; healthy sources continue. After a complete consecutive no-work pass, sleep or, with `--stop-when-empty`, join admitted work and exit without another pass. Failed/cooling sources count as no-work; delayed/backoff rows need not drain. `--once` visits each source once without cooldown or sleep. +- Use the shared `SignalRegistry` with a per-run owner and local stop/pause flags; unregister in `finally`. Initial/subsequent restart-cache reads and lifecycle callbacks run in fresh children, since database-backed cache stores also borrow connections. Memory/time checks remain local. +- `OutboxRelay::deliver()` returns true on successful publication/settlement, including non-failure deferral; false triggers slot-held failure sleep. Tasks rethrow `CanceledException` and capture any other escaping `Throwable` into one first-failure reference. The coordinator stops admission, joins owned tasks and propagates the failure. Coordinator cancellation cancels and joins owned tasks before propagating its original exception; no separate cancelling flag or run-state class. -Each admitted task owns no pooled resource. It uses a fresh child for claim, each publication and settlement, passing only immutable messages/results. Guarded delivery events/reporting also run in a short-lived child because application callbacks may resolve clients. A listener failure is reported once; a reporter failure propagates. Neither alters settled rows, emits the opposite delivery event or republishes accepted work. Use fakeable `Sleep::usleep()` for fractional empty/failure delays and the existing capacity-wait primitive instead of polling busy slots. Graceful stop ceases admission and joins tasks so completed publication can settle. Supervise this dedicated process; scheduled `--once` is also supported. Size source/transport pools for relay concurrency and any other users, allowing a short-lived extra slot for restart-cache/event/reporting work on the same pool. Never make the relay a queued job dependent on the broker it must recover. +Retry/forget load mutation targets from the writer using existing query-builder support, not replica reads or a separately borrowed connection. Recheck token/live-lease predicates in the modifying statement so a concurrent claim cannot be reset/deleted. A live lease is `lease_token IS NOT NULL AND available_at > now`; expired tokens do not prevent recovery. Monitor states partition the total: eligible (`available_at <= now`), live leased, tokenless delayed (no attempts), or tokenless retrying (attempted and still in backoff), matching the listing. + +Each admitted task owns no pooled resource. It receives materialized claims and uses a fresh child for each publication and settlement, passing only immutable messages/results. Guarded delivery events/reporting also run in a short-lived child because application callbacks may resolve clients; skip this child when there is neither a listener nor an error. Use the exception handler's context array for row diagnostics. A listener failure is reported once without triggering delivery backoff; a reporter failure propagates. Neither alters settled rows, emits the opposite delivery event or republishes accepted work. When publication and settlement both fail, report both and retain the publication error on the event. Use fakeable `Sleep::usleep()` for fractional empty/failure delays and the existing capacity-wait primitive instead of polling busy slots. Graceful stop ceases admission and joins tasks so completed publication can settle. Supervise this dedicated process; scheduled `--once` is also supported. Size source/transport pools for relay concurrency and any other users, allowing a short-lived extra slot for restart-cache/event/reporting work on the same pool. Never make the relay a queued job dependent on the broker it must recover. Horizon continues supervising and visualizing Redis jobs only after relay publishes them. Do not add relational outbox polling, migrations, or supervisor machinery to Horizon. -Expose one guarded event per completed connection publication batch, with source database, target connection, accepted/failed counts, publish duration, oldest eligible age, and an optional failure, but no payload bodies: `OutboxBatchDelivered` only when every attempted row was published and settled, otherwise `OutboxBatchDeliveryFailed` for a result failure, partial acceptance, invalid result, lease-ownership loss, or settlement failure. Rows deferred before the generic fallback are excluded; a group containing only future rows emits no delivery event. The OpenTelemetry component consumes these events in the same change and records delivery count, failure count, eligible-to-publish latency, and publish duration. Rely on the monitor command for backlog/eligible-age gauges so ordinary dispatch does not issue count queries. +Expose one guarded event per completed connection publication batch, with source database, target connection, accepted/failed counts, publish duration, oldest eligible age, and an optional failure, but no payload bodies: `OutboxBatchDelivered` only when every attempted row was published and settled, otherwise `OutboxBatchDeliveryFailed` for a result failure, partial acceptance, invalid result, lease-ownership loss, or settlement failure. Rows deferred before the generic fallback are excluded; a group containing only future rows emits no delivery event. The OpenTelemetry component consumes these events and records delivery count, eligible-to-publish latency, and publish duration. Failed-batch counts come from duration observations tagged with `error.type`, not a duplicate counter. Latency uses whole-second buckets and no error-type attribute. Rely on the monitor command for backlog/eligible-age gauges so ordinary dispatch does not issue count queries. ## Horizon outbox integration @@ -453,6 +459,7 @@ Update in the same change so no stale behavior survives: - `src/docs/queue-outbox.md`: canonical setup, configuration, API precedence, transaction ownership, driver/custom-driver matrix, bulk/batches, `dispatchSync`/after-response limitations, delays, UUIDs, connection capture versus publication-time queue forwarding, deliberate driver changes, at-least-once/idempotency and best-effort dispatch ordering, SQS overflow attempt ownership/orphans, worker deployment/pool sizing, operations, Horizon and database fast paths; - `src/docs/documentation.md`: add the new guide to navigation; +- `src/docs/opentelemetry.md`: add the three outbox metrics and their defaults to the queue metrics reference; explain failed-batch counts through duration observations with `error.type`; - `src/docs/queues.md`: add only a concise transactional-outbox introduction, when it is useful, one basic `viaOutbox()` example, and a link to `queue-outbox.md`; do not duplicate its setup, guarantees, driver matrix, or operations reference; - `src/docs/events.md`, `mail.md`, `notifications.md`, and `broadcasting.md`: beside their existing after-commit guidance, add one concise example/cross-link showing producer selection propagating to its wrapper job. Queued listeners are inspected without constructing them: use the marker or public property defaults, not constructor-fluent calls. When using `Queueable`, prefer the marker rather than incompatibly redeclaring its properties. Application-constructed producers can use the fluent methods. Link to `queue-outbox.md` for guarantees rather than duplicating them; - `src/docs/porting-from-laravel.md` and database README: explain exact resolved transaction identity for extensions observing role-qualified records. Do not add outbox catalogue entries: its opt-in APIs require no Laravel adaptation. Preserve Queue/Bus/Horizon README structure without adding additive-feature differences; @@ -472,6 +479,7 @@ Never imply exactly-once delivery, broker participation in the SQL transaction, - Context-dependent cache fixture backed by `WorkerArrayStore`: acquire in context A, release record/snapshot in B, remove A's key and preserve B's same logical key. Cover unique and debounce separately, with a second registered wrapper proving composition order and restoration. A throwing capture callback acquires neither lock nor token. One real CoroutineQueue closing-timer regression covers release outside the acquiring coroutine. No repeated tests per caller or cache wrapper: the hook does not branch on wrapper types. - DatabaseQueue storage rollback releases its snapshot after acceptance, including nested rollback/deadlock retry; broker-accepted jobs are not released by unrelated rollback. Preserve existing deferred ownership coverage. - With no applicable transaction, a local child failing before acceptance retains dispatch-lock ownership for failover's next child; local after-commit return values remain unchanged. +- Manager-resolved custom and memoized repositories retain their cache names, including memo-tagged wrappers, while preserving creator-supplied names and memo event suppression. Missing-model unique-job cleanup releases the actual named memo/custom store's lock; ad hoc unnamed repositories retain their existing semantics. - SQS later-finalizer/queueing-listener failures finish unaccepted producer state without falsifying confirmed chunks; preserve cancellation and per-chunk queueing order. Custom job `queue()` dispatch through SQS bulk retains/delegates its acquired locks and releases only unaccepted work. Success-listener failure preserves all accepted entries, cleans explicit-reject pointers and prevents later sends. Reuse/extend affected tests. - Every optimized batch driver drains successful terminal notifications despite an ordinary listener failure; cover failure-notification draining once at its shared boundary. Cancellation remains immediate. Do not duplicate these tests as a telemetry matrix. - Horizon combined preparation encodes once and preserves classification/pushed timing/customization. A single/delayed post-push listener error retains the accepted lock, attempts `JobQueued` and emits no `JobQueueingFailed`; bulk also attempts that member's `JobQueued`. Compare exact generic lifecycle payload strings for direct/delayed/bulk and failed bulk to prevent correlation drift. Reuse existing direct/retry/delayed tests and remove only the obsolete last-pushed state test when its mechanism is removed. @@ -520,7 +528,7 @@ Place engine-specific cases under `tests/Integration/Queue/Database/{Postgres|My - Redis immediate/delayed Lua batching, exact confirmation and ambiguous failure after earlier confirmed destinations; - database queue multi-row insertion across logical queues and remaining-delay conversion; - SQS 10-entry/1,048,576-byte chunking, standard/FIFO outbox delay behavior, group/deduplication, aggregate explicit-reject failure, SDK exception after confirmed earlier chunks, and partial accepted-ID results; overflow storage reuses one pointer inside SDK retries but uses independent job-UUID/lease-token pointers across outbox attempts, cleanup for a later explicitly rejected attempt cannot remove an earlier ambiguous attempt's payload, confirmed unpublished pointers are cleaned up, and ambiguous attempts are retained; -- recorded connections honor current driver configuration and native transport failures retain rows; invalid/overlapping result IDs retain the whole call, sibling groups continue independently, and stale settlement leaves newer owners untouched; SPI-to-non-SPI replacement before `deliver_at` defers without publication, failure telemetry, attempt inflation or error mutation; +- recorded connections honor current driver configuration and native transport failures retain rows; invalid/overlapping result IDs and deferral without a delivery time release the whole call into failure backoff, sibling groups continue independently, and stale settlement leaves newer owners untouched; SPI-to-non-SPI replacement before `deliver_at` defers without publication, failure telemetry, attempt inflation or error mutation; - shared generic raw fallback accepted-prefix/deferred/ambiguous-failure results; Beanstalk uses it with one pooled lease per connection batch and switches tubes correctly; - only confirmed accepted IDs are deleted; every failed/unattempted row is released with the aggregate failure, while future fallback rows follow the non-failure deferral path; acceptance followed by delete failure produces a duplicate with the same job UUID and no lost row; - confirm settlement adds no broker probe beyond the normal publish call/result. From 343a6be7db92aff188dc498c2f50bf7d310fe21e Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 23:09:38 +0000 Subject: [PATCH 17/33] test(queue): pin set-based outbox staging and settlement Cover inherited Beanstalkd and custom-driver bulk staging while preserving direct partitions. Require multiple settlement rows per outcome to complete in three statements, and verify bounded retry-all updates exclude later arrivals. Exercise custom non-PDO connections through the driver-neutral repository contract, retaining real SQL behavior without requiring PDO capability methods. --- .../Outbox/DatabaseOutboxRepositoryTest.php | 78 ++++++++++++++++--- tests/Queue/Outbox/OutboxDispatchTest.php | 40 ++++++++++ 2 files changed, 109 insertions(+), 9 deletions(-) diff --git a/tests/Queue/Outbox/DatabaseOutboxRepositoryTest.php b/tests/Queue/Outbox/DatabaseOutboxRepositoryTest.php index 43eb7b367..b35caff3e 100644 --- a/tests/Queue/Outbox/DatabaseOutboxRepositoryTest.php +++ b/tests/Queue/Outbox/DatabaseOutboxRepositoryTest.php @@ -4,7 +4,12 @@ namespace Hypervel\Tests\Queue\Outbox; +use Closure; +use Hypervel\Database\ConnectionInterface; +use Hypervel\Database\ConnectionResolverInterface; +use Hypervel\Database\Events\QueryExecuted; use Hypervel\Database\Events\TransactionBeginning; +use Hypervel\Database\Query\Builder; use Hypervel\Foundation\Testing\RefreshDatabase; use Hypervel\Queue\Outbox\DatabaseOutboxRepository; use Hypervel\Queue\Outbox\OutboxBatchResult; @@ -13,6 +18,7 @@ use Hypervel\Support\Str; use Hypervel\Testbench\TestCase; use InvalidArgumentException; +use Mockery as m; use RuntimeException; class DatabaseOutboxRepositoryTest extends TestCase @@ -43,8 +49,6 @@ public function testClaimPreservesPayloadAndOriginalEligibilityWithThreeStatemen $future = $this->row($now + 60); $repository->insert($connection, [$first, $second, $future]); - $connection->maxBindings(); - $connection->lockForPopping(); $transactionsStarted = 0; $this->app->make('events')->listen(TransactionBeginning::class, static function () use (&$transactionsStarted): void { ++$transactionsStarted; @@ -87,6 +91,25 @@ public function testExpiredClaimCanBeRecoveredWithoutChangingDeliveryOrPayload() $this->assertSame($first->payload, $second->payload); } + public function testCustomNonPdoConnectionsUseTheConditionalClaimAndBindingFallback(): void + { + $database = DB::connection(); + $connection = m::mock(ConnectionInterface::class); + $connection->shouldReceive('table')->with('job_outbox')->andReturnUsing(fn (): Builder => $database->table('job_outbox')); + $connection->shouldReceive('transaction')->andReturnUsing(fn (Closure $callback): mixed => $database->transaction($callback)); + $resolver = m::mock(ConnectionResolverInterface::class); + $resolver->shouldReceive('connection')->with('custom')->andReturn($connection); + $repository = new DatabaseOutboxRepository($resolver, ['custom'], 'job_outbox', 300); + $row = $this->row(Date::now()->getTimestamp()); + $repository->insert($connection, [$row]); + + $messages = $repository->claim('custom', 1); + + $this->assertSame([$row['uuid']], array_column($messages, 'uuid')); + $this->assertSame(1, $repository->settle('custom', $messages, new OutboxBatchResult([$messages[0]->id]))); + $this->assertSame(0, $database->table('job_outbox')->count()); + } + public function testSettlementDeletesAcceptedDefersFutureAndBacksOffFailures(): void { Date::setTestNow('2040-01-01 12:00:00'); @@ -99,16 +122,29 @@ public function testSettlementDeletesAcceptedDefersFutureAndBacksOffFailures(): $deferred['last_error'] = 'Previous failure'; $failed = $this->row($now); $failed['attempts'] = 6; - $repository->insert(DB::connection(), [$accepted, $deferred, $failed]); + $repository->insert(DB::connection(), [ + $accepted, array_replace($accepted, ['uuid' => (string) Str::uuid()]), + $deferred, array_replace($deferred, ['uuid' => (string) Str::uuid()]), + $failed, array_replace($failed, ['uuid' => (string) Str::uuid()]), + ]); $messages = $repository->claim($database, 10); + $connection = DB::connection(); + $connection->enableQueryLog(); + $connection->flushQueryLog(); - $affected = $repository->settle($database, $messages, new OutboxBatchResult( - [$messages[0]->id], - [$messages[1]->id], - new RuntimeException(str_repeat('雪', 5000)), - )); + try { + $affected = $repository->settle($database, $messages, new OutboxBatchResult( + [$messages[0]->id, $messages[1]->id], + [$messages[2]->id, $messages[3]->id], + new RuntimeException(str_repeat('雪', 5000)), + )); + $queries = $connection->getQueryLog(); + } finally { + $connection->disableQueryLog(); + } - $this->assertSame(3, $affected); + $this->assertCount(3, $queries); + $this->assertSame(6, $affected); $this->assertNull($repository->find($database, $accepted['uuid'])); $storedDeferred = $repository->find($database, $deferred['uuid']); $this->assertNull($storedDeferred->lease_token); @@ -197,6 +233,30 @@ public function testRepositoryRejectsUnconfiguredSourceBeforeQuerying(): void $this->repository()->claim('not-configured', 100); } + public function testRetryAllUsesBoundedChunksAndExcludesMessagesAddedDuringTheOperation(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $now = Date::now()->getTimestamp(); + $repository = $this->repository(); + $database = DB::connection()->getName(); + $rows = array_map(fn (): array => array_replace($this->row($now + 600), ['attempts' => 1]), range(1, 1001)); + $repository->insert(DB::connection(), $rows); + $later = array_replace($this->row($now + 600), ['attempts' => 1]); + $updates = 0; + $this->app->make('events')->listen(QueryExecuted::class, static function (QueryExecuted $event) use ($repository, $later, &$updates): void { + if (str_starts_with($event->sql, 'update ') && str_contains($event->sql, 'job_outbox')) { + if (++$updates === 1) { + $repository->insert(DB::connection(), [$later]); + } + } + }); + + $this->assertSame(1001, $repository->retryAll($database)); + $this->assertSame(2, $updates); + $this->assertSame($now + 600, (int) $repository->find($database, $later['uuid'])->available_at); + $this->assertSame(1001, DB::table('job_outbox')->where('available_at', $now)->count()); + } + /** * Create a repository using the test's configured database. */ diff --git a/tests/Queue/Outbox/OutboxDispatchTest.php b/tests/Queue/Outbox/OutboxDispatchTest.php index 2d8cad3db..402eade6a 100644 --- a/tests/Queue/Outbox/OutboxDispatchTest.php +++ b/tests/Queue/Outbox/OutboxDispatchTest.php @@ -7,10 +7,12 @@ use Aws\Sqs\SqsClient; use Hypervel\Bus\Queueable; use Hypervel\Contracts\Foundation\Application; +use Hypervel\Contracts\Queue\Queue as QueueContract; use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; use Hypervel\Contracts\Redis\Factory; use Hypervel\Database\ConnectionResolverInterface; use Hypervel\Foundation\Testing\RefreshDatabase; +use Hypervel\Queue\BeanstalkdQueue; use Hypervel\Queue\DatabaseQueue; use Hypervel\Queue\Events\JobPayloadFinalizing; use Hypervel\Queue\Events\JobQueued; @@ -18,6 +20,7 @@ use Hypervel\Queue\Events\JobQueueingFailed; use Hypervel\Queue\InvalidPayloadException; use Hypervel\Queue\Outbox\OutboxDispatcher; +use Hypervel\Queue\Queue; use Hypervel\Queue\RedisQueue; use Hypervel\Queue\SqsQueue; use Hypervel\Support\Facades\Date; @@ -25,6 +28,9 @@ use Hypervel\Testbench\TestCase; use InvalidArgumentException; use Mockery as m; +use Pheanstalk\Contract\PheanstalkManagerInterface; +use Pheanstalk\Contract\PheanstalkPublisherInterface; +use Pheanstalk\Contract\PheanstalkSubscriberInterface; use PHPUnit\Framework\Attributes\DataProvider; use RuntimeException; use UnitEnum; @@ -136,6 +142,40 @@ public function testBulkStagingUsesOneInsertAndFinalizesEachPayloadOnce(): void $this->assertSame([], $queue->published); } + #[DataProvider('baseBulkDrivers')] + public function testBaseBulkStagesOneInsertAndKeepsTheDirectPartition(bool $beanstalk): void + { + $queue = $beanstalk + ? m::mock(BeanstalkdQueue::class, [m::mock(PheanstalkManagerInterface::class . ', ' . PheanstalkPublisherInterface::class . ', ' . PheanstalkSubscriberInterface::class), 'orders', 60])->makePartial() + : m::mock(Queue::class . ', ' . QueueContract::class)->makePartial(); + $queue->setContainer($this->app)->setConnectionName($beanstalk ? 'beanstalkd' : 'custom'); + $direct = (new OutboxDispatchJob)->withoutOutbox(); + $queue->expects('push')->with($direct, '', 'orders')->andReturn('direct'); + $jobs = array_map(static fn (): OutboxDispatchJob => (new OutboxDispatchJob)->viaOutbox(), range(1, 100)); + $connection = DB::connection(); + $connection->enableQueryLog(); + $connection->flushQueryLog(); + + try { + $queue->bulk([...$jobs, $direct], queue: 'orders'); + $inserts = array_filter($connection->getQueryLog(), static fn (array $query): bool => str_starts_with($query['query'], 'insert into')); + } finally { + $connection->disableQueryLog(); + } + + $this->assertCount(1, $inserts); + $this->assertSame(100, DB::table('job_outbox')->count()); + $this->assertSame(['orders'], DB::table('job_outbox')->distinct()->pluck('queue')->all()); + } + + /** + * Cover the first-party and custom users of the shared bulk implementation. + */ + public static function baseBulkDrivers(): array + { + return [[true], [false]]; + } + public function testBulkPreparationFailureStoresNothingAndTerminatesEveryMember(): void { $queue = $this->queue(); From 8996a5a6669d5a94855538263a0dd6d331708d59 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 23:09:56 +0000 Subject: [PATCH 18/33] test(queue): verify outbox claims on supported database engines Share live-engine coverage for bounded claim statements, original eligibility, independent write-role transaction ownership, competing claims and expired-lease settlement protection. Verify skip-locked progress past an uncommitted row and force competing fallback candidates on file-backed SQLite. Keep engine entry points in the existing CI discovery layout and migrate only the shipped outbox schema. --- .../Database/MariaDb/OutboxClaimTest.php | 13 + .../Queue/Database/MySql/OutboxClaimTest.php | 13 + .../Database/Postgres/OutboxClaimTest.php | 13 + .../Queue/Database/Sqlite/OutboxClaimTest.php | 56 ++++ .../Queue/Fixtures/OutboxClaimTestCase.php | 242 ++++++++++++++++++ 5 files changed, 337 insertions(+) create mode 100644 tests/Integration/Queue/Database/MariaDb/OutboxClaimTest.php create mode 100644 tests/Integration/Queue/Database/MySql/OutboxClaimTest.php create mode 100644 tests/Integration/Queue/Database/Postgres/OutboxClaimTest.php create mode 100644 tests/Integration/Queue/Database/Sqlite/OutboxClaimTest.php create mode 100644 tests/Integration/Queue/Fixtures/OutboxClaimTestCase.php diff --git a/tests/Integration/Queue/Database/MariaDb/OutboxClaimTest.php b/tests/Integration/Queue/Database/MariaDb/OutboxClaimTest.php new file mode 100644 index 000000000..cd631cf9d --- /dev/null +++ b/tests/Integration/Queue/Database/MariaDb/OutboxClaimTest.php @@ -0,0 +1,13 @@ +deleteDirectory(static::$databaseDirectory); + $filesystem->ensureDirectoryExists(static::$databaseDirectory); + touch(static::$databaseDirectory . '/database.sqlite'); + } + + /** + * Remove the file-backed database after its applications have shut down. + */ + public static function tearDownAfterClass(): void + { + (new Filesystem)->deleteDirectory(static::$databaseDirectory); + + parent::tearDownAfterClass(); + } + + /** + * Use independent sessions instead of SQLite's shared in-memory connection. + */ + protected function defineEnvironment(Application $app): void + { + parent::defineEnvironment($app); + + $config = $app->make('config'); + $database = $config->string('database.default'); + + if ($config->string("database.connections.{$database}.driver") === 'sqlite') { + $config->set("database.connections.{$database}.database", static::$databaseDirectory . '/database.sqlite'); + } + } +} diff --git a/tests/Integration/Queue/Fixtures/OutboxClaimTestCase.php b/tests/Integration/Queue/Fixtures/OutboxClaimTestCase.php new file mode 100644 index 000000000..7d5aea66e --- /dev/null +++ b/tests/Integration/Queue/Fixtures/OutboxClaimTestCase.php @@ -0,0 +1,242 @@ +make('config'); + $database = $config->string('database.default'); + $config->set('queue.outbox.databases', [$database]); + $config->set("database.connections.{$database}.pool.testing_enabled", true); + $config->set("database.connections.{$database}.pool.max_connections", 4); + $config->set("database.connections.{$database}.pool.heartbeat_interval", null); + } + + /** + * Migrate only the shipped operational outbox table. + */ + protected function migrateFreshUsing(): array + { + return [ + '--realpath' => true, + '--path' => dirname(__DIR__, 4) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php', + ]; + } + + public function testClaimBoundsAndStatementCountOnTheLiveEngine(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $now = Date::now()->getTimestamp(); + $repository = $this->app->make(DatabaseOutboxRepository::class); + $connection = DB::connection(); + $database = $connection->getName(); + $rows = [$this->row($now - 10), $this->row($now - 30), $this->row($now - 20), $this->row($now + 60)]; + $repository->insert($connection, $rows); + $skipLocked = $connection->lockForPopping() === 'FOR UPDATE SKIP LOCKED'; + $transactions = 0; + $this->app->make('events')->listen(TransactionBeginning::class, static function () use (&$transactions): void { ++$transactions; }); + $connection->enableQueryLog(); + $connection->flushQueryLog(); + + try { + $messages = $repository->claim($database, 2); + $queries = $connection->getQueryLog(); + } finally { + $connection->disableQueryLog(); + } + + $postgres = $connection->getDriverName() === 'pgsql'; + $this->assertCount($postgres ? 1 : ($skipLocked ? 2 : 3), $queries); + $this->assertSame($skipLocked && ! $postgres ? 1 : 0, $transactions); + $this->assertSame([$rows[1]['uuid'], $rows[2]['uuid']], array_column($messages, 'uuid')); + $this->assertSame([$now - 30, $now - 20], array_column($messages, 'eligibleAt')); + $this->assertSame($rows[1]['payload'], $messages[0]->payload); + $this->assertSame(['transport' => ['custom' => ['group' => 'orders']]], $messages[0]->options); + $this->assertSame(2, DB::table('job_outbox')->whereNotNull('lease_token')->count()); + $this->assertSame([$rows[0]['uuid']], array_column($repository->claim($database, 10), 'uuid')); + $this->assertSame(2, $repository->settle($database, $messages, new OutboxBatchResult(array_column($messages, 'id')))); + } + + public function testStagingUsesTheExactWriteRoleTransactionAndDrainsThroughItsBaseSource(): void + { + $base = DB::connection(); + $database = $base->getName(); + $writer = DB::connection($database . '::write'); + $queue = new RedisQueue(m::mock(Factory::class), 'orders'); + $queue->setContainer($this->app)->setConnectionName('redis'); + $base->beginTransaction(); + $writer->beginTransaction(); + + try { + $queue->push((new OutboxClaimJob)->onOutboxDatabase($database . '::write')); + $this->assertSame(1, $writer->table('job_outbox')->count()); + $this->assertSame(0, $base->table('job_outbox')->count()); + } finally { + $writer->rollBack(); + $base->rollBack(); + } + + $this->assertSame(0, $base->table('job_outbox')->count()); + $uuid = $writer->transaction(fn (): mixed => $queue->push((new OutboxClaimJob)->viaOutbox())); + $messages = $this->app->make(DatabaseOutboxRepository::class)->claim($database, 1); + + $this->assertSame([$uuid], array_column($messages, 'uuid')); + } + + public function testCompetingClaimsAndExpiredOwnershipOnIndependentSessions(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $now = Date::now()->getTimestamp(); + $repository = $this->app->make(DatabaseOutboxRepository::class); + $connection = DB::connection(); + $database = $connection->getName(); + $rows = array_map(fn (): array => $this->row($now), range(1, 4)); + $repository->insert($connection, $rows); + $fallback = $connection->lockForPopping() !== 'FOR UPDATE SKIP LOCKED'; + + // Force both fallback claimants to see the same candidates before either updates them. + if ($fallback) { + $selected = new Channel(1); + $selections = 0; + $this->app->make('events')->listen(QueryExecuted::class, function (QueryExecuted $event) use ($selected, &$selections): void { + if (str_starts_with($event->sql, 'select ') + && str_contains($event->sql, 'available_at') + && ! str_contains($event->sql, 'lease_token')) { + if (++$selections === 1) { + $this->assertTrue($selected->pop(5)); + } else { + $selected->push(true); + } + } + }); + } + + try { + [$first, $second] = parallel([ + fn (): array => $repository->claim($database, 2), + fn (): array => $repository->claim($database, 2), + ]); + } finally { + if ($fallback) { + $this->app->make('events')->forget(QueryExecuted::class); + $selected->close(); + } + } + + $firstIds = array_column($first, 'id'); + $secondIds = array_column($second, 'id'); + $this->assertSame([], array_values(array_intersect($firstIds, $secondIds))); + $this->assertCount($fallback ? 2 : 4, array_merge($firstIds, $secondIds)); + $old = array_merge($first, $second, $repository->claim($database, 10)); + $this->assertCount(4, $old); + $this->assertSame([], $repository->claim($database, 10)); + + Date::setTestNow(Date::now()->addSeconds(300)); + $current = $repository->claim($database, 10); + $this->assertCount(4, $current); + $this->assertSame([2, 2, 2, 2], array_column($current, 'attempts')); + $this->assertNotSame($old[0]->leaseToken, $current[0]->leaseToken); + + foreach ([$first, $second] as $stale) { + if ($stale !== []) { + $this->assertSame(0, $repository->settle($database, $stale, new OutboxBatchResult(array_column($stale, 'id')))); + } + } + + $this->assertSame(4, $repository->settle($database, $current, new OutboxBatchResult(array_column($current, 'id')))); + $this->assertSame(0, DB::table('job_outbox')->count()); + } + + public function testSkipLockedClaimsAdvancePastAnUncommittedOwner(): void + { + $connection = DB::connection(); + + if ($connection->lockForPopping() !== 'FOR UPDATE SKIP LOCKED') { + $this->markTestSkipped('This engine uses the conditional-update claim path.'); + } + + $repository = $this->app->make(DatabaseOutboxRepository::class); + $database = $connection->getName(); + $now = Date::now()->getTimestamp(); + $locked = $this->row($now - 20); + $available = $this->row($now - 10); + $repository->insert($connection, [$locked, $available]); + $ready = new Channel(1); + $release = new Channel(1); + + try { + [, $messages] = parallel([ + function () use ($locked, $ready, $release): void { + DB::transaction(function () use ($locked, $ready, $release): void { + DB::table('job_outbox')->where('uuid', $locked['uuid'])->lockForUpdate()->first(); + $ready->push(true); + $this->assertTrue($release->pop(5)); + }); + }, + function () use ($repository, $database, $ready, $release): array { + try { + $this->assertTrue($ready->pop(5)); + + return $repository->claim($database, 1); + } finally { + $release->push(true); + } + }, + ]); + } finally { + $ready->close(); + $release->close(); + } + + $this->assertSame([$available['uuid']], array_column($messages, 'uuid')); + $this->assertSame([$locked['uuid']], array_column($repository->claim($database, 1), 'uuid')); + } + + /** + * Create opaque stored bytes and exact epoch deadlines for claim tests. + */ + protected function row(int $availableAt): array + { + return [ + 'uuid' => (string) Str::uuid(), 'connection' => 'redis', 'queue' => 'orders', + 'payload' => '{ "displayName": "SendInvoice", "data": {"text": "雪"} }', + 'options' => '{"transport":{"custom":{"group":"orders"}}}', + 'deliver_at' => null, 'attempts' => 0, 'available_at' => $availableAt, + 'lease_token' => null, 'last_error' => null, 'created_at' => Date::now()->getTimestamp(), + ]; + } +} + +class OutboxClaimJob +{ + use Queueable; +} From a9f3aafe7a587cc6a1bebab144fb14bd558e3a9e Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 23:09:56 +0000 Subject: [PATCH 19/33] test(queue): exercise outbox producers and Redis worker delivery Exercise queued listeners, mailables, notifications and broadcasts through their existing public APIs and wrapper jobs, without broker access during staging. Run committed outbox payloads through the real Redis relay and worker, covering encrypted model restoration, stable payloads, unique-lock ownership, debounce, delays, closures, batches and nested chains. Reuse existing event-deferral coverage instead of duplicating it. --- .../Queue/Redis/OutboxDeliveryTest.php | 199 ++++++++++++++++++ tests/Queue/Outbox/OutboxProducerTest.php | 150 +++++++++++++ 2 files changed, 349 insertions(+) create mode 100644 tests/Integration/Queue/Redis/OutboxDeliveryTest.php create mode 100644 tests/Queue/Outbox/OutboxProducerTest.php diff --git a/tests/Integration/Queue/Redis/OutboxDeliveryTest.php b/tests/Integration/Queue/Redis/OutboxDeliveryTest.php new file mode 100644 index 000000000..7194b0c3b --- /dev/null +++ b/tests/Integration/Queue/Redis/OutboxDeliveryTest.php @@ -0,0 +1,199 @@ +make('config'); + $config->set('queue.default', 'redis'); + $config->set('queue.connections.redis.queue', 'outbox'); + $config->set('queue.outbox.databases', [$config->string('database.default')]); + $config->set('cache.default', 'worker-array'); + $config->set('app.key', str_repeat('a', 32)); + } + + /** + * Load only tables used by these dispatch and processing paths. + */ + protected function migrateFreshUsing(): array + { + $migrations = dirname(__DIR__, 4) . '/src/testbench/hypervel/migrations/'; + + return [ + '--realpath' => true, + '--path' => [ + $migrations . '0001_01_01_000000_testbench_create_users_table.php', + $migrations . '0001_01_01_000005_testbench_create_job_batches_table.php', + $migrations . '0001_01_01_000009_testbench_create_job_outbox_table.php', + ], + ]; + } + + public function testCommitRelayAndProcessingPreserveEncryptedModelsAndUniqueOwnership(): void + { + $user = UserFactory::new()->create(['name' => 'Before dispatch']); + $finalizations = 0; + $queued = 0; + $this->app->make('events')->listen(JobPayloadFinalizing::class, static function (JobPayloadFinalizing $event) use (&$finalizations): void { + ++$finalizations; + $payload = json_decode($event->payload, true, flags: JSON_THROW_ON_ERROR); + $event->payload = json_encode([...$payload, 'custom' => 'preserved'], JSON_THROW_ON_ERROR); + }); + $this->app->make('events')->listen(JobQueued::class, static function () use (&$queued): void { ++$queued; }); + DB::transaction(static function () use ($user): void { + OutboxEncryptedDeliveryJob::dispatch('private-result-key', $user)->viaOutbox(); + OutboxUniqueDeliveryJob::dispatch('unique')->viaOutbox(); + OutboxUniqueDeliveryJob::dispatch('unique')->viaOutbox(); + }); + $rows = DB::table('job_outbox')->orderBy('id')->get(); + $this->assertCount(2, $rows); + $this->assertSame(0, Queue::size()); + $this->assertStringNotContainsString('private-result-key', $rows[0]->payload); + $unique = new OutboxUniqueDeliveryJob('unique'); + $this->assertTrue(Cache::lock(UniqueLock::getKey($unique))->isLocked()); + $user->name = 'After dispatch'; + $user->save(); + + $this->assertSame(0, Artisan::call('queue:relay', ['--once' => true])); + $this->assertSame(0, DB::table('job_outbox')->count()); + $this->assertSame(2, Queue::size()); + $this->assertSame(2, $finalizations); + $this->assertSame(2, $queued); + $payloads = Queue::connection()->pendingJobs()->all(); + $this->assertSame($rows->pluck('uuid')->all(), array_map(static fn (InspectedJob $job): string => $job->uuid, $payloads)); + $this->assertSame( + $rows->map(static fn (object $row): array => json_decode($row->payload, true, flags: JSON_THROW_ON_ERROR))->all(), + array_map(static fn (InspectedJob $job): array => $job->payload, $payloads), + ); + $this->assertTrue(Cache::lock(UniqueLock::getKey($unique))->isLocked()); + + $this->work(); + $this->assertSame('After dispatch', Cache::get('private-result-key')); + $this->assertSame(1, Cache::get('unique')); + $this->assertFalse(Cache::lock(UniqueLock::getKey($unique))->isLocked()); + } + + public function testBatchesNestedChainsAndClosuresRunThroughTheExistingWorker(): void + { + $batch = Bus::batch([ + [new OutboxDeliveryJob('first'), new OutboxDeliveryJob('second')], + static function (): void { Cache::put('closure', true); }, + ])->viaOutbox()->dispatch(); + $batch->add([new OutboxDeliveryJob('added')]); + + $this->assertSame(3, DB::table('job_outbox')->count()); + $this->assertSame(4, $batch->fresh()->pendingJobs); + $this->assertSame(0, Queue::size()); + $this->assertSame(0, Artisan::call('queue:relay', ['--once' => true])); + $this->work(); + + $this->assertSame(1, Cache::get('first')); + $this->assertSame(1, Cache::get('second')); + $this->assertSame(1, Cache::get('added')); + $this->assertTrue(Cache::get('closure')); + $this->assertTrue($batch->fresh()->finished()); + $this->assertSame(0, DB::table('job_outbox')->count()); + $this->assertSame(0, Queue::size()); + } + + public function testDebouncedAndDelayedPayloadsRetainTheirOriginalDeliveryRules(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + OutboxDebouncedDeliveryJob::dispatch('debounced')->viaOutbox(); + OutboxDebouncedDeliveryJob::dispatch('debounced')->viaOutbox(); + OutboxDeliveryJob::dispatch('delayed')->viaOutbox()->delay(60); + $this->assertSame(0, Artisan::call('queue:relay', ['--once' => true])); + $this->assertSame(0, DB::table('job_outbox')->count()); + $this->assertNull(Queue::pop()); + + Date::setTestNow(Date::now()->addSeconds(61)); + $this->work(); + + $this->assertSame(1, Cache::get('debounced')); + $this->assertSame(1, Cache::get('delayed')); + $this->assertSame(0, Queue::size()); + } + + /** + * Process the ready queue using the public worker command. + */ + protected function work(): void + { + $this->assertSame(0, Artisan::call('queue:work', ['--stop-when-empty' => true, '--sleep' => 0, '--memory' => 1024])); + } +} + +class OutboxDeliveryJob implements ShouldQueue +{ + use Batchable; + use Dispatchable; + use InteractsWithQueue; + use Queueable; + use SerializesModels; + + /** + * Store a result key and optional model reference. + */ + public function __construct(public string $key, public ?User $user = null) + { + } + + /** + * Record execution in a store shared with the worker's child coroutine. + */ + public function handle(): void + { + Cache::put($this->key, $this->user?->name ?? ((int) Cache::get($this->key, 0) + 1)); + } +} + +class OutboxEncryptedDeliveryJob extends OutboxDeliveryJob implements ShouldBeEncrypted +{ +} + +class OutboxUniqueDeliveryJob extends OutboxDeliveryJob implements ShouldBeUnique +{ +} + +#[DebounceFor(30)] +class OutboxDebouncedDeliveryJob extends OutboxDeliveryJob +{ +} diff --git a/tests/Queue/Outbox/OutboxProducerTest.php b/tests/Queue/Outbox/OutboxProducerTest.php new file mode 100644 index 000000000..edd73f124 --- /dev/null +++ b/tests/Queue/Outbox/OutboxProducerTest.php @@ -0,0 +1,150 @@ +make('config'); + $config->set('queue.default', 'redis'); + $config->set('queue.outbox.databases', [$config->string('database.default')]); + } + + /** + * Load only the shipped outbox schema. + */ + protected function migrateFreshUsing(): array + { + return [ + '--realpath' => true, + '--path' => dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php', + ]; + } + + public function testPublicProducerApisStageTheirExistingWrapperJobs(): void + { + $this->app->instance('redis', m::mock(Factory::class)); + Event::listen('outbox.created', OutboxProducerListener::class); + + DB::transaction(static function (): void { + Event::dispatch('outbox.created', ['order-17']); + Mail::to('customer@example.com')->send(new OutboxProducerMailable); + NotificationFacade::route('mail', 'customer@example.com')->notify(new OutboxProducerNotification); + Event::dispatch(new OutboxProducerBroadcast('order-17')); + }); + + $rows = DB::table('job_outbox')->orderBy('id')->get(); + $jobs = $rows->map(static function (object $row): object { + $payload = json_decode($row->payload, true, flags: JSON_THROW_ON_ERROR); + + return unserialize($payload['data']['command']); + }); + + $this->assertSame([ + CallQueuedListener::class, + SendQueuedMailable::class, + SendQueuedNotifications::class, + BroadcastEvent::class, + ], $jobs->map(static fn (object $job): string => $job::class)->all()); + $this->assertSame(['order-17'], $jobs[0]->data); + $this->assertSame('customer@example.com', $jobs[1]->mailable->to[0]['address']); + $this->assertSame('customer@example.com', $jobs[2]->notifiables->first()->routes['mail']); + $this->assertSame('order-17', $jobs[3]->event->order); + $this->assertSame(['redis'], $rows->pluck('connection')->unique()->all()); + } +} + +class OutboxProducerListener implements ShouldQueueViaOutbox +{ + /** + * Handle the queued event. + */ + public function handle(string $order): void + { + } +} + +class OutboxProducerMailable extends Mailable implements ShouldQueueViaOutbox +{ + use Queueable; + + /** + * Build a mail message without an external template dependency. + */ + public function build(): static + { + return $this->subject('Order received')->html('Your order was received.'); + } +} + +class OutboxProducerNotification extends Notification implements ShouldQueueViaOutbox +{ + use Queueable; + + /** + * Select the queued mail notification channel. + */ + public function via(AnonymousNotifiable $notifiable): array + { + return ['mail']; + } + + /** + * Build the notification mail message. + */ + public function toMail(AnonymousNotifiable $notifiable): MailMessage + { + return (new MailMessage)->line('Your order was received.'); + } +} + +class OutboxProducerBroadcast implements ShouldBroadcast, ShouldQueueViaOutbox +{ + use Queueable; + + /** + * Identify the order in the broadcast payload. + */ + public function __construct(public string $order) + { + } + + /** + * Select the broadcast channel. + */ + public function broadcastOn(): array + { + return ['orders']; + } +} From bf01122fbdf91720adad3d38bdaf56935fd6fe6d Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 23:09:56 +0000 Subject: [PATCH 20/33] docs(queue): document transactional outbox setup and operations Add one canonical guide for selection, transactional ownership, driver behavior, delays, relay deployment, concurrency, recovery, operator commands and custom driver publication. Link the guide from queue and producer documentation. Explain whole-event after-commit timing, local-driver limitations, at-least-once processing and idempotency without duplicating the operating reference across pages. --- src/docs/broadcasting.md | 14 ++ src/docs/documentation.md | 1 + src/docs/events.md | 13 ++ src/docs/mail.md | 10 ++ src/docs/notifications.md | 8 + src/docs/queue-outbox.md | 319 ++++++++++++++++++++++++++++++++++++++ src/docs/queues.md | 18 +++ 7 files changed, 383 insertions(+) create mode 100644 src/docs/queue-outbox.md diff --git a/src/docs/broadcasting.md b/src/docs/broadcasting.md index b45529766..f23c7bf28 100644 --- a/src/docs/broadcasting.md +++ b/src/docs/broadcasting.md @@ -1001,6 +1001,20 @@ class ServerCreated implements ShouldBroadcast, ShouldDispatchAfterCommit > [!NOTE] > To learn more about working around these issues, please review the documentation regarding [queued jobs and database transactions](/docs/{{version}}/queues#jobs-and-database-transactions). +To store the broadcast job atomically with the business write, select the [transactional outbox](/docs/{{version}}/queue-outbox) on the broadcast event: + +```php +use Hypervel\Contracts\Broadcasting\ShouldBroadcast; +use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; + +class ServerCreated implements ShouldBroadcast, ShouldQueueViaOutbox +{ + // ... +} +``` + +Dispatch this event inside the transaction. Do not also defer the whole event with `ShouldDispatchAfterCommit`: that would postpone outbox staging until after commit. See the [outbox guide](/docs/{{version}}/queue-outbox#queued-producers) for setup and guarantees. + ## Authorizing Channels diff --git a/src/docs/documentation.md b/src/docs/documentation.md index 0f157e39d..212d87e83 100644 --- a/src/docs/documentation.md +++ b/src/docs/documentation.md @@ -61,6 +61,7 @@ - [Signals](/docs/{{version}}/signals) - [WebSockets](/docs/{{version}}/websockets) - [Queues](/docs/{{version}}/queues) + - [Transactional Outbox](/docs/{{version}}/queue-outbox) - [Rate Limiting](/docs/{{version}}/rate-limiting) - [Search](/docs/{{version}}/search) - [Strings](/docs/{{version}}/strings) diff --git a/src/docs/events.md b/src/docs/events.md index cc1b98ce7..0f5664543 100644 --- a/src/docs/events.md +++ b/src/docs/events.md @@ -521,6 +521,19 @@ class SendShipmentNotification implements ShouldQueueAfterCommit > [!NOTE] > To learn more about working around these issues, please review the documentation regarding [queued jobs and database transactions](/docs/{{version}}/queues#jobs-and-database-transactions). +To store a queued listener's job atomically with the business write, implement `ShouldQueueViaOutbox` on the listener: + +```php +use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; + +class SendShipmentNotification implements ShouldQueueViaOutbox +{ + // ... +} +``` + +Listener defaults are inspected without constructing the listener; use the marker rather than calling `viaOutbox()` in its constructor. Dispatch the event inside the transaction, without deferring the whole event through `ShouldDispatchAfterCommit`. See the [transactional outbox guide](/docs/{{version}}/queue-outbox#queued-producers) for setup and delivery guarantees. + ### Queued Listener Middleware diff --git a/src/docs/mail.md b/src/docs/mail.md index d802de402..ace6f928b 100644 --- a/src/docs/mail.md +++ b/src/docs/mail.md @@ -1240,6 +1240,16 @@ class OrderShipped extends Mailable implements ShouldQueue > [!NOTE] > To learn more about working around these issues, please review the documentation regarding [queued jobs and database transactions](/docs/{{version}}/queues#jobs-and-database-transactions). +For transactional delivery, a mailable using `Queueable` may select the [outbox](/docs/{{version}}/queue-outbox): + +```php +Mail::to($request->user())->queue( + (new OrderShipped($order))->viaOutbox() +); +``` + +The existing queued-mail wrapper carries this choice to the queue. See the [outbox guide](/docs/{{version}}/queue-outbox#queued-producers) for the required table, relay and delivery guarantees. + #### Queued Email Failures diff --git a/src/docs/notifications.md b/src/docs/notifications.md index 11810cb96..6e8b998d7 100644 --- a/src/docs/notifications.md +++ b/src/docs/notifications.md @@ -435,6 +435,14 @@ class InvoicePaid extends Notification implements ShouldQueue > [!NOTE] > To learn more about working around these issues, please review the documentation regarding [queued jobs and database transactions](/docs/{{version}}/queues#jobs-and-database-transactions). +For transactional delivery, a queued notification using `Queueable` may select the [outbox](/docs/{{version}}/queue-outbox): + +```php +$user->notify((new InvoicePaid($invoice))->viaOutbox()); +``` + +Each queued channel job carries the notification's choice. `viaOutbox()` is a fluent setter, not a hook returning a boolean or channel map. See the [outbox guide](/docs/{{version}}/queue-outbox#queued-producers) for setup and delivery guarantees. + #### Determining if a Queued Notification Should Be Sent diff --git a/src/docs/queue-outbox.md b/src/docs/queue-outbox.md new file mode 100644 index 000000000..57cb3d497 --- /dev/null +++ b/src/docs/queue-outbox.md @@ -0,0 +1,319 @@ +# Transactional Queue Outbox + +- [Introduction](#introduction) +- [Setup](#setup) +- [Dispatching Jobs](#dispatching-jobs) + - [Selecting the Outbox](#selecting-the-outbox) + - [Bulk Dispatch and Batches](#bulk-dispatch-and-batches) + - [Queued Listeners, Mail, Notifications and Broadcasts](#queued-producers) +- [Database Transactions](#database-transactions) +- [Queue Connections and Delays](#queue-connections-and-delays) +- [Running the Relay](#running-the-relay) + - [Concurrency and Shutdown](#concurrency-and-shutdown) + - [Failures and Recovery](#failures-and-recovery) +- [Inspecting and Managing Messages](#managing-messages) +- [Delivery Guarantees](#delivery-guarantees) +- [Custom Queue Drivers](#custom-queue-drivers) + + +## Introduction + +Dispatching a job [after a database transaction commits](/docs/{{version}}/queues#jobs-and-database-transactions) prevents a worker from reading uncommitted data. However, if the application stops between committing the transaction and publishing the job, the database change can remain without its intended job. + +The transactional outbox closes this gap. Hypervel stores the job in a database table as part of the same transaction, then a separate relay process publishes it to your queue. If the transaction rolls back, its outbox rows roll back too. If the relay stops or the queue is unavailable, committed rows remain available for recovery. + +```php +use App\Jobs\SendOrderReceipt; +use App\Models\Order; +use Hypervel\Support\Facades\DB; + +DB::transaction(function () use ($attributes) { + $order = Order::create($attributes); + + SendOrderReceipt::dispatch($order)->viaOutbox(); +}); +``` + +The outbox is an opt-in feature of the queue package. It uses your existing jobs, queue connections and workers, and does not require Horizon. Jobs which do not select it continue through their normal dispatch path without outbox database access. + + +## Setup + +The outbox settings live in your application's `config/queue.php` file: + +```php +'outbox' => [ + 'databases' => [ + env('QUEUE_OUTBOX_DATABASE', env('DB_CONNECTION', 'sqlite')), + ], + 'table' => env('QUEUE_OUTBOX_TABLE', 'job_outbox'), + 'retry_after' => (int) env('QUEUE_OUTBOX_RETRY_AFTER', 300), +], +``` + +The `databases` array lists the configured database connections whose outbox tables the relay will drain. Install the outbox table on each listed database. New applications include its migration; if your application does not, generate it using: + +```shell +php artisan make:queue-outbox-table +php artisan migrate +``` + +The generator uses the configured table name. `queue:outbox-table` is an alias for the same command. When using several databases, apply the migration on each appropriate connection using your normal migration deployment process. + +Finally, run the relay alongside your normal queue workers: + +```shell +php artisan queue:relay +php artisan queue:work redis +``` + +The relay publishes jobs; it does not replace the workers that process them. Horizon may continue managing your Redis workers as usual. + + +## Dispatching Jobs + + +### Selecting the Outbox + +Call `viaOutbox` when dispatching a job: + +```php +SendOrderReceipt::dispatch($order)->viaOutbox(); +``` + +To select the outbox for a job class by default, implement `ShouldQueueViaOutbox` instead of `ShouldQueue`. The marker extends `ShouldQueue`, so you do not need both: + +```php +use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; +use Hypervel\Foundation\Queue\Queueable; + +class SendOrderReceipt implements ShouldQueueViaOutbox +{ + use Queueable; + + // ... +} +``` + +Alternatively, set `outbox` to `true` on a named queue connection in `config/queue.php`. Existing connection configurations which omit this option default to `false`. + +You may override either default for a specific dispatch: + +```php +SendOrderReceipt::dispatch($order)->withoutOutbox(); +``` + +An explicit job choice takes precedence over the marker, followed by the queue connection's default. `afterCommit` and `beforeCommit` still control direct dispatch timing; neither disables the outbox. A selected outbox job is stored inside its owning transaction and cannot be relayed before that transaction commits. + +The fluent methods are available through the usual `Queueable` trait. A directly queued object may also set a public `viaOutbox` property to `true` or `false`. `viaOutbox()` is a setter, not a getter hook like `viaConnection()` or `viaQueues()`. + +A staged queue insertion returns the finalized job UUID instead of a broker identifier. The operator commands use this same UUID. Direct dispatch return values are unchanged. + + +### Bulk Dispatch and Batches + +Use the existing bulk API to stage several jobs efficiently: + +```php +use Hypervel\Support\Facades\Bus; + +Bus::bulk([ + (new SendOrderReceipt($firstOrder))->viaOutbox(), + (new SendOrderReceipt($secondOrder))->viaOutbox(), +]); +``` + +Within each queue bulk call, jobs sharing an outbox database are inserted together, using additional insert chunks only when the database's parameter limits require them. Each database group is atomic across those chunks. `Bus::bulk` keeps its existing grouping by queue connection and destination. A bulk containing direct jobs or several database owners is not one atomic unit, and publication order between its groups is not guaranteed. + +You may also select the outbox for a batch: + +```php +$batch = Bus::batch($jobs) + ->onConnection('redis') + ->viaOutbox() + ->dispatch(); + +$batch->add($moreJobs); +``` + +Batch defaults apply to newly added jobs too, without replacing an explicit choice on an individual job. Batch counters and outbox rows participate in the batch repository's transaction; conflicting outbox ownership is rejected. Jobs using only `Batchable`, rather than `Queueable`, can receive batch defaults. + +Chains retain their normal behavior. Selecting the outbox for the first job does not automatically select it for later jobs. Each later job is dispatched after its predecessor succeeds and uses its own selection or its queue connection's default. + + +### Queued Listeners, Mail, Notifications and Broadcasts + +Queued listeners, mailables, notifications and broadcast events propagate their outbox choice to their existing wrapper jobs. Implement `ShouldQueueViaOutbox` on the producer, or use its `Queueable` methods when constructing it: + +```php +use App\Mail\OrderReceipt; +use Hypervel\Support\Facades\Mail; + +Mail::to($customer)->queue( + (new OrderReceipt($order))->viaOutbox() +); +``` + +For queued listeners, use the marker or a public property default: the dispatcher reads these defaults without constructing the listener. Do not rely on a listener constructor calling `viaOutbox`. When a listener uses `Queueable`, prefer the marker instead of redeclaring that trait's properties. + +Dispatch the event while the business transaction is open. An event implementing `ShouldDispatchAfterCommit` postpones the entire event, including listener and broadcast queueing, until after commit. Adding outbox selection cannot make that later staging atomic with the earlier transaction. Select the outbox on the queued listener or broadcast event instead of deferring the whole event when you need transactional delivery. + + +## Database Transactions + +When one database connection has an active transaction, Hypervel stores the outbox row on that connection. Without an active transaction, it uses the first configured outbox database. You may select an owner explicitly: + +```php +SendOrderReceipt::dispatch($order) + ->onOutboxDatabase('billing'); +``` + +`onOutboxDatabase` also selects the outbox. The database must be included in `queue.outbox.databases`. If transactions are open on several connections, an explicit owner is required. Hypervel reports the active connections rather than guessing which business write belongs to the job. + +An explicitly selected owner must have an active transaction when other connections have active transactions. Only writes on that selected transaction are atomic with the job. Nested transactions on one connection remain one owner, and rolling back a savepoint also removes its outbox rows and releases their dispatch-owned locks. + +Ownership follows the resolved database session. For example, a `mysql::write` transaction is distinct from a `mysql` transaction when they use separate sessions. Select the exact owner when necessary, but configure only `mysql` as the relay source: do not list both names for the same outbox table. Read-role connections cannot own SQL outbox storage; use their base or write connection. + +Calling `withoutOutbox` clears an explicit database choice. Calling `viaOutbox` afterwards returns to normal owner selection. + +> [!WARNING] +> The outbox cannot make writes to different databases atomic with one another. It also cannot include an HTTP request, payment, email or filesystem write in the database transaction. Perform those effects in the queued job with suitable idempotency handling. + + +## Queue Connections and Delays + +Selecting the outbox does not require environment-specific job code when you switch queue drivers: + +| Queue connection | Behavior when the outbox is selected | +| --- | --- | +| Redis, SQS, Beanstalkd | Store the job in the outbox, then relay it to the queue. | +| Database, using the active owning connection | Insert directly into the jobs table inside that transaction; a second durable table adds no guarantee. | +| Database, with no active transaction | Insert directly into the durable jobs table. | +| Database, on another connection during a business transaction | Store the job on the business transaction's outbox, then relay it to the database queue. | +| Sync, background, deferred | Preserve local execution behavior after the selected transaction commits. These drivers do not gain crash durability. | +| Failover | Defer the complete failover attempt until the applicable transactions commit. Its normal child-driver attempts remain non-durable across a process crash. | +| Null | Discard the job normally without creating an outbox row. | +| Queue or Bus fake | Record the job only in the fake, without accessing an outbox table. | + +`dispatchSync` and after-response dispatch retain their existing local or deferred behavior. They are not substitutes for durable queued dispatch. Direct `pushRaw` calls also bypass outbox staging, even on a connection whose default enables it. + +Jobs keep their intended delivery time when the relay runs late or retries publication. Redis and database queues can receive delayed jobs immediately and hold them until due. Standard SQS jobs wait in the outbox until they fit its per-message delay window. SQS FIFO jobs wait until due before publication, without a per-message timer; the queue's own configured delay still applies. On standard SQS, omitting a job delay preserves the queue-level default, while an expired explicitly requested delay is sent as zero. + +The outbox captures the selected queue connection and logical queue name, including its default, at dispatch. Queue-name forwarding is applied once when publication occurs, using the current forwarding rules. Changing a connection's default queue does not reroute existing rows. + +The relay uses the recorded connection's current configuration. Drain pending rows before changing a connection to a non-durable driver, changing between incompatible SQS destination types, or otherwise invalidating its stored transport options. + + +## Running the Relay + +Run `queue:relay` under a process supervisor, just like `queue:work`. It is a dedicated CLI process which uses coroutines internally; it does not run as a job on the queue it is responsible for recovering. + +```shell +php artisan queue:relay --batch=100 --concurrency=4 +``` + +By default, it visits all configured outbox databases in turn. Pass a database name to dedicate a process to one source: + +```shell +php artisan queue:relay billing +``` + +The relay supports the following options: + +| Option | Default | Purpose | +| --- | --- | --- | +| `--batch` | `100` | Maximum rows per claim, from 1 to 1,000. | +| `--concurrency` | `1` | Maximum batches being delivered concurrently. | +| `--sleep` | `1` | Positive seconds to wait after an empty pass or delivery failure; fractions are supported. | +| `--max-jobs` | `0` | Stop after claiming this many rows; zero means unlimited. | +| `--max-time` | `0` | Stop admitting work after this many seconds; zero means unlimited. | +| `--memory` | `128` | Memory limit in megabytes. | +| `--stop-when-empty` | Off | Stop after a complete pass finds no eligible rows, then finish admitted work. | +| `--once` | Off | Process at most one batch from each selected database, without waiting between passes. | + +`--stop-when-empty` does not wait for delayed or backed-off rows to become eligible. Sources which fail or are cooling down count as empty for that pass. `--max-jobs` counts claimed rows, not successfully delivered jobs. `--once` runs synchronously and does not use concurrent delivery. + + +### Concurrency and Shutdown + +You may run several relay processes against the same outbox. Claims use short database operations and temporary ownership tokens, so one relay cannot settle rows reclaimed by another. Database connections and locks are released before broker publication. SQLite remains correct, but its serialized writes limit the benefit of adding relay processes. + +Size database and transport pools for your relay concurrency, allowing an additional short-lived connection when restart-cache reads or application listeners use the same pool. A relay process has its own pools; increasing its concurrency does not increase the capacity of another process's pools. + +The `queue.outbox.retry_after` setting is the claim lifetime. Set it above the longest expected lifetime of a claimed batch, including every destination group, network timeouts and SDK retries. A claim which outlives this value may be published again by another relay. There is no lease heartbeat. + +The relay observes `queue:restart` through the same configured cache store as ordinary queue workers. On restart or graceful shutdown, it stops claiming and finishes admitted work. `SIGTERM`, `SIGINT` and `SIGQUIT` request shutdown; `SIGUSR2` pauses admission and `SIGCONT` resumes it. Configure your supervisor's stop allowance to let active calls and settlement finish. + +Maintenance mode does not pause the relay. It only moves already committed jobs to their queues; your normal workers control whether those jobs execute. + + +### Failures and Recovery + +The relay removes rows after confirmed queue acceptance. Failed or uncertain publications remain retryable, with increasing, jittered backoff capped at ten minutes. There is no automatic maximum-attempt cutoff: a committed business action is not silently abandoned because its queue was unavailable overnight. + +Failed sources cool down without preventing healthy sources from being visited. A failed delivery batch also waits for `--sleep` before releasing its concurrency slot, limiting repeated attempts against an unavailable destination while new rows remain in the backlog. + +Exceptions include the source database, queue connection, row IDs, job UUIDs and display names. Delivery observer failures are reported without republishing accepted work or throttling healthy delivery. If exception reporting itself fails, the error reaches the command rather than being hidden. + +With SQS payload overflow enabled, separate relay attempts use separate payload pointers, so cleanup from one attempt cannot delete another attempt's payload. An uncertain SQS response may leave an offloaded payload behind, as it can during ordinary dispatch. Account for those objects in your cache/storage retention policy; the outbox does not change the existing overflow cleanup settings. + + +## Inspecting and Managing Messages + +List pending messages or select one database: + +```shell +php artisan queue:outbox --limit=100 +php artisan queue:outbox billing --json +``` + +The listing includes job UUIDs, destinations, attempts, availability and the last publication error. Tables show UTC dates and abbreviated errors; JSON retains epoch values and complete stored errors. Payload bodies and lock-owner tokens are not displayed. + +Monitor backlog size and the age of eligible work: + +```shell +php artisan queue:outbox-monitor --max=10000 --age=60 +``` + +The command reports eligible, delayed, leased and retrying rows. `--max` and `--age` are optional alert thresholds; reaching either emits `Hypervel\Queue\Events\OutboxBusy` when a listener is registered. Eligible age excludes time a job intentionally waits for its delivery date. Use `--json` for external monitoring. + +To retry attempted messages immediately without resetting their attempt history: + +```shell +php artisan queue:outbox-retry JOB_UUID +php artisan queue:outbox-retry all --database=billing +``` + +Retry refuses live leases and never-attempted delayed jobs. When the same UUID exists on several sources, specify `--database` rather than relying on a guess. + +To deliberately discard a message: + +```shell +php artisan queue:outbox-forget JOB_UUID --database=billing +``` + +This command asks for confirmation in every environment. For unattended use, `--force` skips the prompt but never bypasses live-lease protection: an already running publication cannot be recalled. + +After deleting the row, Hypervel releases its unique-job lock only when it can resolve the recorded cache store and exact owner. If cleanup fails, the command explicitly reports that the row was deleted but its lock could not be released. Custom or context-dependent cache integrations may override the command's protected `releaseUniqueLock` method to establish their namespace or resolve their repository. The command does not deserialize the queued job or hydrate arbitrary application context. + +Horizon displays Redis jobs after they reach Redis, not while they remain in the SQL outbox. The [OpenTelemetry integration](/docs/{{version}}/opentelemetry#queues) observes relay delivery counts, batch duration and eligible-message age without polling the outbox or creating a second producer span. + + +## Delivery Guarantees + +Delivery is **at least once**, not exactly once. If the queue accepts a message and the relay stops before deleting its outbox row, recovery can publish the same payload again. Its job UUID stays the same, but the outbox does not deduplicate processing or external effects. Make handlers idempotent where repeating an action would be harmful, using database constraints or a stable operation identifier supported by the external service. + +Publication order is best effort. Concurrent claims, delays and retries can change it. SQS FIFO orders messages by arrival within a message group, not by the order transactions originally staged them. + +Successful rows are deleted rather than archived. The outbox records pending delivery, not a business audit trail or a list of failed job executions. Once a worker receives a job, ordinary queue retry and failed-job behavior applies. + + +## Custom Queue Drivers + +Drivers which use the base queue's enqueue lifecycle can opt into staging without changing their public dispatch APIs. Overrides which bypass that lifecycle remain direct. The protected `resolveOutboxQueue`, `preparePayloadForOutbox`, `prepareOutboxOptions` and `availableAtForOutbox` hooks let a driver capture its logical destination, stored payload, transport options and initial eligibility while the original job is available. + +By default, the relay waits until a message is due and publishes it through the driver's existing `pushRaw` method. It does not call Bus or serialize the job again. A pooled driver is borrowed once for the publication batch. + +Drivers with a more efficient batch operation may implement `Hypervel\Contracts\Queue\PublishesOutboxMessages`. Its `publishOutboxMessages` method receives immutable `Hypervel\Queue\Outbox\OutboxMessage` values and returns an `OutboxBatchResult` containing confirmed accepted row IDs, deferred row IDs, and an optional failure. IDs refer to outbox rows, not broker identifiers; array keys are ignored. Accepted and deferred IDs must be unique, disjoint members of the input. A result without a failure must account for every input row. Only messages with an intended delivery time may be deferred. + +Report confirmed partial acceptance before returning a publication error. Keep uncertain outcomes retryable, and propagate coroutine cancellation. Do not emit the ordinary enqueue lifecycle again: it completed when the job was accepted into the outbox. diff --git a/src/docs/queues.md b/src/docs/queues.md index 450a19cec..635e4b9c2 100644 --- a/src/docs/queues.md +++ b/src/docs/queues.md @@ -25,6 +25,7 @@ - [Bulk Dispatching](#bulk-dispatching) - [Preparing Jobs Before Dispatch](#preparing-jobs-before-dispatch) - [Jobs & Database Transactions](#jobs-and-database-transactions) + - [Transactional Outbox](#transactional-outbox) - [Job Chaining](#job-chaining) - [Customizing The Queue and Connection](#customizing-the-queue-and-connection) - [Specifying Max Job Attempts / Timeout Values](#max-job-attempts-and-timeout) @@ -1504,6 +1505,23 @@ Likewise, if the `after_commit` configuration option is set to `true`, you may i ProcessPodcast::dispatch($podcast)->beforeCommit(); ``` + +### Transactional Outbox + +After-commit dispatch prevents a worker from seeing uncommitted data, but a process can still stop between committing the database transaction and publishing the job. When a job must survive that gap, use the [transactional outbox](/docs/{{version}}/queue-outbox): + +```php +use Hypervel\Support\Facades\DB; + +DB::transaction(function () use ($podcast) { + $podcast->save(); + + ProcessPodcast::dispatch($podcast)->viaOutbox(); +}); +``` + +Hypervel stores the job with the database write, then a separate relay publishes it to your existing queue. See the [outbox guide](/docs/{{version}}/queue-outbox) for setup, supported drivers, delivery guarantees and operations. + ### Job Chaining From 920c0b1040128d03503324dacf90d706029e12ad Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 23:10:12 +0000 Subject: [PATCH 21/33] docs(opentelemetry): explain transactional outbox delivery metrics Document queue-confirmed delivery counts, publication duration and eligible-age observations at connection-batch boundaries. Explain failure counting through duration attributes and the absence of backlog polling or a second producer lifecycle. Remove stale guidance that ordinary later batch-finalizer failures leave earlier producer states unfinished now that guarded queue lifecycles terminate outstanding work. --- src/docs/opentelemetry.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/src/docs/opentelemetry.md b/src/docs/opentelemetry.md index 0b13b1b18..081da00a9 100644 --- a/src/docs/opentelemetry.md +++ b/src/docs/opentelemetry.md @@ -189,6 +189,7 @@ The shipped defaults are: | Database and Redis pools | None | `db.client.connection.count`, `db.client.connection.max`, `db.client.connection.pending_requests` | | Cache | Events on the current span | `hypervel.cache.operations` | | Queue | Producer and consumer spans | `messaging.client.sent.messages`, `messaging.client.operation.duration`, `messaging.client.consumed.messages`, `messaging.process.duration`; `hypervel.queue.jobs` is off | +| Queue outbox | No additional spans | `hypervel.queue.outbox.delivered.messages`, `hypervel.queue.outbox.delivery.duration`, `hypervel.queue.outbox.delivery.latency` | | Scheduler | Task spans | `hypervel.scheduler.task.duration`, `hypervel.scheduler.task.executions` | | Console | Command spans | `hypervel.console.command.duration` | | Events | Exact allowlist, empty by default | None | @@ -318,7 +319,15 @@ The built-in `background` and `deferred` connections propagate the current conte Jobs dispatched after a database commit are finalized and timed only when the queue driver actually begins enqueueing them. Database and SQS batch operations retain one lifecycle per logical message. Later SQS chunks include the time spent sending earlier chunks because timing begins when the batch is finalized, and a later batch failure completes every still-outstanding message once. -Terminal queue correlation normally uses the exact final encoded payload without decoding it. If another finalizer rewrites that payload afterward, the package performs one terminal decode to recover Hypervel's job UUID. Payloads from custom drivers or `Queue::createPayloadUsing()` callbacks that do not contain a string UUID are accepted, omit `messaging.message.id`, and use exact-payload correlation only. If a later finalizer rewrites such a payload, its producer span remains started but is never ended or exported, and its state remains until coroutine or process exit. A later finalizer that removes or corrupts an existing UUID, or throws after earlier messages in the same batch were finalized, likewise leaves unmatched state. A long-running non-coroutine custom process that catches and continues after such failures may retain that state until it exits. +Terminal queue correlation normally uses the exact final encoded payload without decoding it. If another finalizer rewrites that payload afterward, the package performs one terminal decode to recover Hypervel's job UUID. Payloads from custom drivers or `Queue::createPayloadUsing()` callbacks that do not contain a string UUID are accepted, omit `messaging.message.id`, and use exact-payload correlation only. If a later finalizer rewrites such a payload, its producer span remains started but is never ended or exported, and its state remains until coroutine or process exit. A later finalizer that removes or corrupts an existing UUID likewise leaves unmatched state. A long-running non-coroutine custom process that catches and continues after such failures may retain that state until it exits. + +With the [transactional outbox](/docs/{{version}}/queue-outbox), producer telemetry finishes when staging accepts the job. The relay preserves its propagation fields and does not emit a second producer lifecycle. Three enabled-by-default metrics observe delivery at connection-batch boundaries: + +- `hypervel.queue.outbox.delivered.messages` counts queue-confirmed messages, including accepted messages whose SQL settlement subsequently fails. +- `hypervel.queue.outbox.delivery.duration` records publication duration in seconds, with `error.type` on publication or settlement failure. Its failed observation count supplies the number of failed batches without a separate counter. +- `hypervel.queue.outbox.delivery.latency` records the oldest attempted message's eligible age at publication start, in seconds. It is one observation per connection batch, not a per-message latency distribution, and excludes deferred rows. + +These metrics use the configured queue connection and outbox database as attributes. Latency does not add exception types as attributes. They perform no backlog queries or payload decoding and add no work to ordinary dispatch. Use `queue:outbox-monitor` for backlog counts and age alerts. Queue depth is off by default because it performs backend I/O during metric collection. Enable it with explicit targets: From 85f7979ccc4ae8589be501eaa826825e8cf45a4d Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 23:10:12 +0000 Subject: [PATCH 22/33] perf(queue): add repeatable transactional outbox workload harness Provide developer-only direct dispatch, transactional staging, real relay and concurrent claim-churn cases using isolated tables and queue names. Verify completed counts, released pool connections and resource cleanup on every run. Capture executing engine plans outside timing and compare PostgreSQL availability-index claims with a narrowly scoped reservation fixture. Make seeded lease counts explicit and label workload WAL, in-memory restart checks, CPU and heap measurements accurately. Keep baseline dispatch independent of feature-only fixtures, document fresh-process comparison methodology, and retain raw reports outside Git. Functional smoke checks are complete; performance measurements require an idle-machine window. --- .../Fixtures/OutboxBenchmarkJob.php | 27 ++ .../QueueOutbox/Fixtures/OutboxWorkload.php | 390 ++++++++++++++++++ .../Fixtures/ReservationRepository.php | 36 ++ tests/Benchmarks/QueueOutbox/README.md | 73 ++++ tests/Benchmarks/QueueOutbox/benchmark.php | 281 +++++++++++++ 5 files changed, 807 insertions(+) create mode 100644 tests/Benchmarks/QueueOutbox/Fixtures/OutboxBenchmarkJob.php create mode 100644 tests/Benchmarks/QueueOutbox/Fixtures/OutboxWorkload.php create mode 100644 tests/Benchmarks/QueueOutbox/Fixtures/ReservationRepository.php create mode 100644 tests/Benchmarks/QueueOutbox/README.md create mode 100644 tests/Benchmarks/QueueOutbox/benchmark.php diff --git a/tests/Benchmarks/QueueOutbox/Fixtures/OutboxBenchmarkJob.php b/tests/Benchmarks/QueueOutbox/Fixtures/OutboxBenchmarkJob.php new file mode 100644 index 000000000..c86748a82 --- /dev/null +++ b/tests/Benchmarks/QueueOutbox/Fixtures/OutboxBenchmarkJob.php @@ -0,0 +1,27 @@ +table = 'bench_outbox_' . $identifier; + $this->jobsTable = 'bench_jobs_' . $identifier; + } + + /** + * Build isolated tables from the shipped schema and capture one real payload. + */ + public function setUp(): void + { + $connection = $this->application->make('db')->connection($this->source); + $schema = $connection->getSchemaBuilder(); + $reservation = $this->settings['claim_design'] === 'reservation'; + + if ($reservation && $connection->getDriverName() !== 'pgsql') { + throw new InvalidArgumentException('The reservation comparison is PostgreSQL-only.'); + } + + $tables = [$this->table => 'outbox']; + if ($this->settings['target'] === 'database') { + $tables[$this->jobsTable] = 'jobs'; + } + + foreach ($tables as $table => $stub) { + if ($schema->hasTable($table)) { + throw new RuntimeException("Benchmark table [{$table}] already exists."); + } + + $path = tempnam(sys_get_temp_dir(), 'hypervel-outbox-migration-'); + if ($path === false) { + throw new RuntimeException('Could not create the temporary migration file.'); + } + $this->ownedTables[] = $table; + try { + $contents = file_get_contents(dirname(__DIR__, 4) . "/src/queue/src/Console/stubs/{$stub}.stub"); + file_put_contents($path, str_replace('{{table}}', $table, $contents)); + (require $path)->up(); + } finally { + unlink($path); + } + } + + if ($reservation) { + $schema->table($this->table, static fn (Blueprint $table) => $table->unsignedBigInteger('reserved_until')->nullable()); + $this->repository = new ReservationRepository($this->application->make('db'), [$this->source], $this->table, 3600); + } else { + $this->repository = $this->application->make(DatabaseOutboxRepository::class); + } + + // Redis staging never contacts Redis. It also avoids the database-queue atomic fast path. + $this->application->make('config')->set('queue.connections.benchmark_seed', [ + 'driver' => 'redis', 'connection' => 'default', 'queue' => $this->queue, 'retry_after' => 3600, + ]); + $this->application->make('queue')->connection('benchmark_seed') + ->push((new OutboxBenchmarkJob(str_repeat('x', $this->settings['payload_bytes'])))->viaOutbox()); + $row = (array) $connection->table($this->table)->useWritePdo()->first(); + unset($row['id']); + $row['connection'] = 'benchmark'; + if ($reservation) { + $row['reserved_until'] = null; + } + $this->template = $row; + $connection->table($this->table)->delete(); + } + + /** + * Seed outside timing; retain claim churn from warmup into the measured sample. + */ + public function prepare(int $jobs): void + { + if ($this->settings['case'] === 'relay') { + $this->insertRows($jobs); + } elseif ($this->settings['case'] === 'claims' && $this->distribution === []) { + $background = intdiv($this->settings['backlog'], 4); + $leased = $this->settings['leased']; + $eligible = max($this->settings['backlog'] - 2 * $background - $leased, $this->settings['batch'] * $this->settings['concurrency']); + $this->distribution = ['eligible' => $eligible, 'delayed' => $background, 'leased' => $leased, 'retrying' => $background]; + $now = time(); + $future = $now + 86400; + $this->insertRows($eligible); + $this->insertRows($background, ['available_at' => $future, 'deliver_at' => $future]); + $this->insertRows($background, ['available_at' => $future, 'attempts' => 5, 'last_error' => 'Benchmark retry backlog']); + $lease = ['available_at' => $future, 'attempts' => 1, 'lease_token' => (string) Str::uuid()]; + if ($this->settings['claim_design'] === 'reservation') { + $lease['available_at'] = $now - 60; + $lease['reserved_until'] = $future; + } + $this->insertRows($leased, $lease); + $this->analyze(); + } + } + + /** + * Exercise production staging, the real relay worker, or competing claims. + */ + public function execute(int $jobs): ?array + { + if ($this->settings['case'] === 'stage') { + $database = $this->application->make('db'); + $queues = $this->application->make('queue'); + for ($offset = 0; $offset < $jobs; $offset += $this->settings['batch']) { + $this->inFreshCoroutine(function () use ($database, $queues, $jobs, $offset): void { + $members = []; + for ($index = 0; $index < min($this->settings['batch'], $jobs - $offset); ++$index) { + $members[] = (new OutboxBenchmarkJob(str_repeat('x', $this->settings['payload_bytes'])))->viaOutbox(); + } + $database->connection($this->source)->transaction(function () use ($queues, $members): void { + if ($this->settings['batch'] === 1) { + $queues->connection('benchmark')->push($members[0]); + } else { + $queues->connection('benchmark')->bulk($members); + } + }); + }); + } + + return null; + } + + if ($this->settings['case'] === 'relay') { + $status = $this->application->make(OutboxWorker::class)->run($this->source, new OutboxWorkerOptions( + batch: $this->settings['batch'], + concurrency: $this->settings['concurrency'], + maxJobs: $jobs, + maxTime: 300, + memory: 1024, + stopWhenEmpty: true, + )); + if ($status !== 0) { + throw new RuntimeException("Relay exited with status {$status}."); + } + + return null; + } + + $callbacks = []; + $clients = min($jobs, $this->settings['concurrency']); + $deadline = hrtime(true) + 300_000_000_000; + for ($client = 0; $client < $clients; ++$client) { + $quota = intdiv($jobs, $clients) + ($client < $jobs % $clients ? 1 : 0); + $callbacks[] = function () use ($quota, $deadline): array { + $latencies = []; + $completed = 0; + $empty = 0; + while ($completed < $quota) { + if (hrtime(true) >= $deadline) { + throw new RuntimeException('The claims workload exceeded its five-minute limit.'); + } + $started = hrtime(true); + $messages = $this->inFreshCoroutine(fn (): array => $this->repository->claim($this->source, min($this->settings['batch'], $quota - $completed))); + $latencies[] = (hrtime(true) - $started) / 1e6; + if ($messages === []) { + ++$empty; + continue; + } + $this->inFreshCoroutine(function () use ($messages): void { + $settled = $this->repository->settle($this->source, $messages, new OutboxBatchResult(array_column($messages, 'id'))); + if ($settled !== count($messages)) { + throw new RuntimeException('A claimed batch lost ownership during the workload.'); + } + $this->insertRows(count($messages)); + }); + $completed += count($messages); + } + + return ['latencies' => $latencies, 'empty' => $empty]; + }; + } + + $results = parallel($callbacks, $clients); + $latencies = array_merge(...array_column($results, 'latencies')); + sort($latencies, SORT_NUMERIC); + + return [ + 'claim_operations' => count($latencies), + 'empty_claims' => array_sum(array_column($results, 'empty')), + 'claim_ms_p50' => $latencies[(int) ceil(count($latencies) * 0.5) - 1], + 'claim_ms_p95' => $latencies[(int) ceil(count($latencies) * 0.95) - 1], + 'backlog' => $this->distribution, + ]; + } + + /** + * Check completed work rather than treating an early or partial drain as faster. + */ + public function verify(int $jobs): void + { + $query = $this->application->make('db')->connection($this->source)->table($this->table)->useWritePdo(); + $expected = match ($this->settings['case']) { + 'stage' => $jobs, + 'claims' => array_sum($this->distribution), + default => 0, + }; + if ($query->count() !== $expected) { + throw new RuntimeException('The outbox remainder does not match the completed workload.'); + } + if ($this->settings['case'] === 'claims' && (clone $query)->whereNotNull('lease_token')->count() !== $this->distribution['leased']) { + throw new RuntimeException('The claims workload left unexpected leases.'); + } + if ($this->settings['case'] === 'relay' && $this->application->make('queue')->connection('benchmark')->size($this->queue) !== $jobs) { + throw new RuntimeException('The relay target does not contain every staged job.'); + } + } + + /** + * Clear warmup output without removing the mixed claims backlog. + */ + public function reset(): void + { + if ($this->settings['case'] !== 'claims') { + $this->application->make('db')->connection($this->source)->table($this->table)->delete(); + } + if ($this->settings['case'] === 'relay') { + $this->application->make('queue')->connection('benchmark')->clear($this->queue); + } + } + + /** + * Insert bounded batches of realistic opaque envelopes. + */ + private function insertRows(int $count, array $overrides = []): void + { + $connection = $this->application->make('db')->connection($this->source); + $payload = json_decode($this->template['payload'], true, flags: JSON_THROW_ON_ERROR); + for ($offset = 0; $offset < $count; $offset += 1000) { + $rows = []; + for ($index = 0; $index < min(1000, $count - $offset); ++$index) { + $uuid = (string) Str::uuid(); + $rows[] = array_replace($this->template, [ + 'uuid' => $uuid, 'payload' => json_encode(array_replace($payload, ['uuid' => $uuid]), JSON_THROW_ON_ERROR), + 'available_at' => time(), 'created_at' => time(), + ], $overrides); + } + $this->repository->insert($connection, $rows); + } + } + + /** + * End each resource-owning phase before starting unrelated work. + */ + protected function inFreshCoroutine(Closure $callback): mixed + { + return (new Waiter(-1))->wait($callback, waitForChildTermination: true); + } + + /** + * Refresh planner statistics without changing vacuum or engine settings. + */ + private function analyze(): void + { + $connection = $this->application->make('db')->connection($this->source); + $table = $connection->getQueryGrammar()->wrapTable($this->table); + $connection->statement(in_array($connection->getDriverName(), ['mysql', 'mariadb'], true) ? "ANALYZE TABLE {$table}" : "ANALYZE {$table}"); + } + + /** + * Capture actual claim SQL once and inspect executing plans after churn. + */ + public function explain(): array + { + $this->analyze(); + $connection = $this->application->make('db')->connection($this->source); + $connection->beginTransaction(); + $connection->enableQueryLog(); + $connection->flushQueryLog(); + try { + $this->repository->claim($this->source, $this->settings['batch']); + $queries = $connection->getQueryLog(); + } finally { + $connection->disableQueryLog(); + $connection->rollBack(); + } + + $plans = []; + foreach ($queries as $query) { + if (! str_starts_with(strtolower(ltrim($query['query'])), 'with ') + && ! str_starts_with(strtolower(ltrim($query['query'])), 'select ')) { + continue; + } + $prefix = match ($connection->getDriverName()) { + 'pgsql' => 'EXPLAIN (ANALYZE, BUFFERS, WAL, FORMAT JSON) ', + 'mysql' => 'EXPLAIN ANALYZE ', + 'mariadb' => 'ANALYZE FORMAT=JSON ', + 'sqlite' => 'EXPLAIN QUERY PLAN ', + default => throw new InvalidArgumentException('No plan command is defined for this database driver.'), + }; + $connection->beginTransaction(); + try { + $plans[] = ['sql' => $query['query'], 'plan' => $connection->selectFromWriteConnection($prefix . $query['query'], $query['bindings'])]; + } finally { + $connection->rollBack(); + } + } + + return $plans; + } + + /** + * Capture the cluster-wide logical WAL position outside the timed workload. + */ + public function walPosition(): ?string + { + $connection = $this->application->make('db')->connection($this->source); + + return $this->settings['case'] === 'claims' && $connection->getDriverName() === 'pgsql' + ? $connection->selectOne('SELECT pg_current_wal_insert_lsn() AS position', useReadPdo: false)->position + : null; + } + + /** + * Report total workload WAL, including refills, settlement and background activity. + */ + public function walBytes(?string $start): ?int + { + return $start === null ? null : (int) $this->application->make('db')->connection($this->source) + ->selectOne('SELECT pg_wal_lsn_diff(pg_current_wal_insert_lsn(), ?) AS bytes', [$start], false)->bytes; + } + + /** + * Describe the database and actual serialized payload without credentials. + */ + public function backend(): array + { + $connection = $this->application->make('db')->connection($this->source); + + return [ + 'driver' => $connection->getDriverName(), 'version' => $connection->getServerVersion(), + 'database' => $connection->getDatabaseName(), 'table' => $this->table, + 'payload_bytes' => strlen($this->template['payload']), + 'functional_only' => $connection->getDriverName() === 'sqlite', + ]; + } + + /** + * Remove only the tables this invocation created. + */ + public function tearDown(): void + { + $schema = $this->application->make('db')->connection($this->source)->getSchemaBuilder(); + foreach (array_reverse($this->ownedTables) as $table) { + $schema->dropIfExists($table); + if ($schema->hasTable($table)) { + throw new RuntimeException("Benchmark table [{$table}] was not removed."); + } + } + } +} diff --git a/tests/Benchmarks/QueueOutbox/Fixtures/ReservationRepository.php b/tests/Benchmarks/QueueOutbox/Fixtures/ReservationRepository.php new file mode 100644 index 000000000..a2bcf46c3 --- /dev/null +++ b/tests/Benchmarks/QueueOutbox/Fixtures/ReservationRepository.php @@ -0,0 +1,36 @@ +getQueryGrammar()->wrapTable($this->table); + + return $connection->selectFromWriteConnection( + "WITH candidates AS ( + SELECT id, available_at FROM {$table} + WHERE available_at <= ? AND (lease_token IS NULL OR reserved_until <= ?) + ORDER BY available_at, id LIMIT ? FOR UPDATE SKIP LOCKED + ) + UPDATE {$table} AS outgoing + SET lease_token = ?, reserved_until = ?, attempts = outgoing.attempts + 1 + FROM candidates WHERE outgoing.id = candidates.id + RETURNING outgoing.*, candidates.available_at AS eligible_at", + [$now, $now, $limit, $token, $expiresAt], + ); + } +} diff --git a/tests/Benchmarks/QueueOutbox/README.md b/tests/Benchmarks/QueueOutbox/README.md new file mode 100644 index 000000000..43d92a1b1 --- /dev/null +++ b/tests/Benchmarks/QueueOutbox/README.md @@ -0,0 +1,73 @@ +# Queue outbox benchmark + +This developer-only harness measures ordinary dispatch, transactional outbox staging, relay delivery, and concurrent SQL claims. It uses the real queue drivers, repository and relay worker. It does not execute application jobs, run in PHPUnit, or impose CI throughput thresholds. + +Run from the components repository root. Get fresh confirmation that the machine is idle before collecting performance measurements. Tiny invocations may check execution, counts and cleanup while the machine is busy; their timings are not performance evidence. + +```shell +timeout 600 php -d opcache.enable_cli=1 -d opcache.jit=disable \ + tests/Benchmarks/QueueOutbox/benchmark.php \ + --case=stage --database-connection=pgsql --jobs=10000 --batch=1000 \ + > /tmp/outbox-staging.json +``` + +The harness reads `.env` through `tests/bootstrap.php`. Use a disposable database with permission to create and drop tables. `--database-connection` selects an existing configured connection; its pool size is set by `--pool-size`. Omitting it creates a temporary file-backed SQLite database for functional checks only. SQLite timings do not represent a production networked database. + +Tables have random `bench_outbox_` / `bench_jobs_` names, and Redis uses a random queue name. Schema comes from the shipped queue migration stubs. The harness removes its own tables and queue keys in `finally`; it never flushes Redis or migrates existing application tables. Resource names are printed to stderr for manual cleanup after a forced kill. Do not run application workers against its queue. Normal exceptions also produce a nonzero exit status. + +## Cases + +| Case | Timed work | +| --- | --- | +| `direct` | Normal Redis single or bulk publication, without selecting the outbox. No SQL setup or outbox service resolution. | +| `stage` | Single or bulk dispatch through `viaOutbox()`, each call in its own SQL transaction. No Redis calls are needed. | +| `relay` | Actual `OutboxWorker` drain to Redis or a database queue. Seeded jobs are outside timing; claim, publication, settlement and worker lifecycle are inside. | +| `claims` | Concurrent claim → accepted settlement → refill cycles against a mixed backlog. Each claim and settlement/refill phase has its own resource lifetime. | + +`--batch=1` uses single dispatch for `direct` and `stage`; larger values use `bulk()`. Compare single, 1,000-job and 10,000-job staging calls using `--batch=1`, `1000`, and `10000`. Claims and relay batches follow the production worker limit of 1,000. + +Use `--target=database` for the database relay case; otherwise the target is Redis. `--redis-connection=default` selects the configured Redis connection. Publication uses its real driver batching and the normal database binding limits. Run each case separately: + +```shell +timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php --case=direct --jobs=10000 --batch=100 +timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php --case=stage --database-connection=mysql --jobs=10000 --batch=1000 +timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php --case=relay --database-connection=pgsql --jobs=10000 --batch=100 --concurrency=8 --pool-size=4 +timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php --case=relay --target=database --database-connection=pgsql --jobs=10000 --concurrency=8 --pool-size=4 +``` + +Concurrency applies to relay and claims. Direct and staging workloads are sequential producers. Vary pool size and concurrency together to distinguish pool pressure from transport latency. Existing tiny-pool integration tests prove the release boundaries; this harness checks that no connections remain borrowed when a sample finishes, without adding a sampling timer or per-operation pool instrumentation. The reported `cache_store` is `worker-array`: relay restart checks still run, but these timings exclude the external cache round trips a Redis or database cache would add. + +`--payload-bytes=256` controls the job's string field, not total serialized envelope size. `--warmup=100` runs outside timing; every invocation produces one measured sample. Warmup output is removed before direct, staging and relay measurements. Every run checks source remainder, target counts where applicable, remaining claims and cleanup. A partial drain is a failure, not a faster result. Relay and claim work have a five-minute bound; retain the external timeout for setup and transport failures too. + +## Claims and query plans + +`--backlog=10000` sizes the mixed backlog; delayed and retrying rows each take a quarter. `--leased` separately controls seeded live leases, defaulting to `batch × concurrency`, not a fraction of the backlog. The remaining rows are eligible, increased when necessary to provide at least one batch per concurrent claimant; the report gives the actual distribution. Seeded leases represent other in-flight work, in addition to the short-lived claims made by this process. Each settled batch is replaced with fresh eligible rows. Warmup churn is retained, so measurement is not limited to a freshly inserted table. Background deadlines are one day away, beyond the bounded run. + +Claim p50/p95 includes the repository operation, row materialization and child-coroutine cleanup, not just server execution time. Empty claims caused by competing fallback claimants count as attempts and are reported separately. Cycle throughput and WAL include settlement and refill, including UUID generation and payload encoding. Latency samples add common harness overhead and are retained only for this bounded invocation. + +```shell +timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php \ + --case=claims --database-connection=pgsql --jobs=10000 \ + --backlog=100000 --batch=100 --concurrency=8 --pool-size=4 --explain \ + > /tmp/outbox-claims.json +``` + +`--explain` refreshes table statistics and captures the actual claim SQL once after churn, outside timing. Executing plans run in transactions that are rolled back. PostgreSQL uses `EXPLAIN (ANALYZE, BUFFERS, WAL, FORMAT JSON)`; MySQL uses `EXPLAIN ANALYZE` for selects; MariaDB uses `ANALYZE FORMAT=JSON`; SQLite uses `EXPLAIN QUERY PLAN`. PostgreSQL's WAL option requires 13+, and MySQL's executing explain requires 8.0.18+. Engines lacking these commands fail with their native error; omit `--explain` there. No optimizer plan is asserted as a correctness requirement. + +### PostgreSQL claim-design comparison + +`--claim-design=availability` uses the production repository. `--claim-design=reservation` adds a benchmark-only, unindexed `reserved_until` column and replaces only the PostgreSQL claim CTE. It leaves indexed `available_at` unchanged and filters live reservations during selection. Both variants use the real accepted-delete settlement and equivalent logical backlogs. The reservation fixture is not a supported alternative production implementation. + +Run both variants in alternating fresh processes. Compare claim latency, buffers and rows filtered alongside write volume: avoiding index changes can reduce writes but require scanning live reservations. Compare the default seeded lease count with a larger `--leased` value representing several other relays; reservation scan cost grows with live leases, not simply with total backlog. No fillfactor, autovacuum or checkpoint tuning is performed. HOT updates still depend on available page space; an unindexed deadline does not guarantee them. + +The cluster-wide `pg_current_wal_insert_lsn()` delta includes the entire claim/settle/refill workload and unrelated database activity. It is not isolated index-write cost. The executing PostgreSQL plan supplies statement-local WAL and buffer evidence. Checkpoint state and full-page images can affect comparisons; use repeated samples rather than one ratio. See PostgreSQL's [WAL position functions](https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-ADMIN-BACKUP) and [EXPLAIN options](https://www.postgresql.org/docs/current/sql-explain.html). + +Once approved measurements settle the comparison, record the findings and remove the temporary reservation fixture and selector. Do not change the production schema based only on WAL bytes or a single small run. + +## Comparisons and interpretation + +For direct-path regression checks, copy this directory into an independently installed pre-feature worktree. The `direct` case uses only existing queue APIs and does not load feature-only fixtures. Use identical dependency versions, PHP configuration, opcache/JIT settings and services. Record revisions, uncommitted changes and commands with raw JSON outside Git. Alternate baseline/current order for at least ten fresh-process pairs; compare medians and spread, not the fastest sample. + +Direct publication and durable staging provide different guarantees; their throughput is not an apples-to-apples regression comparison. The report includes wall time, process CPU, heap/GC data and non-secret backend settings. CPU excludes database and Redis processes; heap growth is not an allocation count. Driver/network latency, payload size and database durability settings remain part of real throughput. + +For a quick functional check, use `--jobs=8 --batch=2 --warmup=2 --backlog=12`. Check all four cases, both relay targets and the PostgreSQL comparison when those services are available. Keep these checks separate from idle-machine measurements. There are no recorded performance results yet. diff --git a/tests/Benchmarks/QueueOutbox/benchmark.php b/tests/Benchmarks/QueueOutbox/benchmark.php new file mode 100644 index 000000000..60ab9edfc --- /dev/null +++ b/tests/Benchmarks/QueueOutbox/benchmark.php @@ -0,0 +1,281 @@ +#!/usr/bin/env php +wait($callback, waitForChildTermination: true); +} + +/** + * Run one sample in a new process, with setup and verification outside timing. + */ +function main(): void +{ + $options = getopt('', ['case:', 'target:', 'database-connection:', 'redis-connection:', 'jobs:', 'batch:', 'concurrency:', 'pool-size:', 'backlog:', 'leased:', 'payload-bytes:', 'warmup:', 'claim-design:', 'explain', 'help']); + + if (isset($options['help'])) { + echo "Usage: php tests/Benchmarks/QueueOutbox/benchmark.php [options]\n" + . " --case=direct|stage|relay|claims --target=redis|database (relay only)\n" + . " --database-connection=NAME (omit for temporary functional-only SQLite)\n" + . " --redis-connection=default --jobs=10000 --batch=100 --concurrency=1\n" + . " --pool-size=4 --backlog=10000 --leased=N (default: batch * concurrency)\n" + . " --payload-bytes=256 --warmup=100\n" + . " --claim-design=availability|reservation (PostgreSQL claims comparison)\n" + . " --explain (claims only; executing plans outside timing)\n" + . "Use a disposable database and an idle machine for measurements. See README.md.\n"; + + return; + } + + $case = $options['case'] ?? 'stage'; + $target = $options['target'] ?? 'redis'; + $design = $options['claim-design'] ?? 'availability'; + + if (! in_array($case, ['direct', 'stage', 'relay', 'claims'], true) + || ! in_array($target, ['redis', 'database'], true) + || ! in_array($design, ['availability', 'reservation'], true) + || ($target === 'database' && $case !== 'relay') + || ($design === 'reservation' && $case !== 'claims') + || (isset($options['explain']) && $case !== 'claims')) { + throw new InvalidArgumentException('Invalid case, target, claim design or explain combination; see --help.'); + } + + $settings = [ + 'case' => $case, + 'target' => $target, + 'claim_design' => $design, + 'jobs' => integerOption($options, 'jobs', 10_000), + 'batch' => integerOption($options, 'batch', 100), + 'concurrency' => integerOption($options, 'concurrency', 1), + 'pool_size' => integerOption($options, 'pool-size', 4), + 'backlog' => integerOption($options, 'backlog', 10_000, 4), + 'payload_bytes' => integerOption($options, 'payload-bytes', 256, 0), + 'warmup' => integerOption($options, 'warmup', 100, 0), + 'cache_store' => 'worker-array', + ]; + $settings['leased'] = integerOption($options, 'leased', $settings['batch'] * $settings['concurrency'], 0); + + if (in_array($case, ['relay', 'claims'], true) && $settings['batch'] > 1000) { + throw new InvalidArgumentException('Relay and claim batches must not exceed the worker limit of 1000.'); + } + + Bootstrapper::bootstrap(); + $application = Application::create(options: ['load_environment_variables' => false]); + $config = $application->make('config'); + $identifier = bin2hex(random_bytes(6)); + $queue = 'benchmark-outbox-' . $identifier; + $source = $options['database-connection'] ?? 'benchmark'; + $redis = $options['redis-connection'] ?? 'default'; + $sqlite = null; + $workload = null; + $failure = null; + $report = []; + + try { + if ($case !== 'direct' && ! isset($options['database-connection'])) { + $sqlite = tempnam(sys_get_temp_dir(), 'hypervel-outbox-benchmark-'); + + if ($sqlite === false) { + throw new RuntimeException('Could not create the temporary SQLite database.'); + } + + $config->set('database.connections.benchmark', ['driver' => 'sqlite', 'database' => $sqlite, 'prefix' => '', 'busy_timeout' => 5000]); + } + + if ($case !== 'direct') { + $connection = $config->array("database.connections.{$source}"); + $connection['pool'] = array_replace($connection['pool'] ?? [], [ + 'testing_enabled' => true, 'max_connections' => $settings['pool_size'], + 'heartbeat_interval' => null, 'idle_check_interval' => null, + ]); + $config->set("database.connections.{$source}", $connection); + $config->set('database.default', $source); + $config->set('queue.outbox', ['databases' => [$source], 'table' => 'bench_outbox_' . $identifier, 'retry_after' => 3600]); + } + + $redisConfig = $config->array("database.redis.{$redis}"); + $redisConfig['pool'] = array_replace($redisConfig['pool'] ?? [], ['max_connections' => $settings['pool_size'], 'heartbeat_interval' => null, 'idle_check_interval' => null]); + $config->set("database.redis.{$redis}", $redisConfig); + $config->set('cache.default', $settings['cache_store']); + $config->set('queue.connections.benchmark', $target === 'redis' + ? ['driver' => 'redis', 'connection' => $redis, 'queue' => $queue, 'retry_after' => 3600, 'block_for' => null, 'after_commit' => false] + : ['driver' => 'database', 'connection' => $source, 'table' => 'bench_jobs_' . $identifier, 'queue' => $queue, 'retry_after' => 3600, 'after_commit' => false]); + fwrite(STDERR, "Owned benchmark resources: queue={$queue}, tables=bench_outbox_{$identifier},bench_jobs_{$identifier}\n"); + + $completed = run(function () use ($application, $settings, $source, $queue, $redis, $identifier, $options, &$workload, &$report, &$failure): void { + $queues = $application->make('queue'); + $direct = static function (int $count) use ($queues, $settings, $queue): void { + for ($offset = 0; $offset < $count; $offset += $settings['batch']) { + $jobs = []; + + for ($index = 0; $index < min($settings['batch'], $count - $offset); ++$index) { + $jobs[] = new OutboxBenchmarkJob(str_repeat('x', $settings['payload_bytes'])); + } + + if ($settings['batch'] === 1) { + $queues->connection('benchmark')->push($jobs[0], queue: $queue); + } else { + $queues->connection('benchmark')->bulk($jobs, queue: $queue); + } + } + }; + + try { + if ($settings['case'] !== 'direct') { + $workload = new OutboxWorkload($application, $source, $queue, $identifier, $settings); + inFreshCoroutine(fn () => $workload->setUp()); + } + + $execute = $workload === null ? $direct : $workload->execute(...); + + if ($settings['warmup'] > 0) { + inFreshCoroutine(fn () => $workload?->prepare($settings['warmup'])); + inFreshCoroutine(fn () => $execute($settings['warmup'])); + inFreshCoroutine(function () use ($workload, $queues, $queue, $settings): void { + $workload?->verify($settings['warmup']); + if ($workload === null && $queues->connection('benchmark')->size($queue) !== $settings['warmup']) { + throw new RuntimeException('Direct warmup delivered an unexpected job count.'); + } + $workload?->reset(); + if ($workload === null) { + $queues->connection('benchmark')->clear($queue); + } + }); + } + + inFreshCoroutine(fn () => $workload?->prepare($settings['jobs'])); + $walStart = inFreshCoroutine(fn () => $workload?->walPosition()); + gc_collect_cycles(); + memory_reset_peak_usage(); + $heap = memory_get_usage(); + $gc = gc_status(); + $cpu = getrusage(); + $started = hrtime(true); + $details = inFreshCoroutine(fn () => $execute($settings['jobs'])); + $elapsed = (hrtime(true) - $started) / 1e9; + $cpuAfter = getrusage(); + $heapGrowth = memory_get_usage() - $heap; + $peakHeap = memory_get_peak_usage(); + $gcAfter = gc_status(); + $walBytes = inFreshCoroutine(fn () => $workload?->walBytes($walStart)); + inFreshCoroutine(function () use ($workload, $queues, $queue, $settings): void { + $workload?->verify($settings['jobs']); + if ($workload === null && $queues->connection('benchmark')->size($queue) !== $settings['jobs']) { + throw new RuntimeException('Direct dispatch delivered an unexpected job count.'); + } + }); + + $borrowed = []; + foreach ([DatabasePools::class, RedisPools::class] as $manager) { + foreach ($application->make($manager)->getPools() as $name => $pool) { + $borrowed[$manager . ':' . $name] = $pool->getBorrowedCount(); + } + } + if (array_sum($borrowed) !== 0) { + throw new RuntimeException('The workload retained a borrowed connection after completion.'); + } + + $report = [ + 'timestamp' => gmdate(DATE_ATOM), 'php' => PHP_VERSION, 'swoole' => phpversion('swoole'), + 'opcache_cli' => ini_get('opcache.enable_cli'), 'jit' => ini_get('opcache.jit'), + 'settings' => $settings, 'database_connection' => $settings['case'] === 'direct' ? null : $source, + 'redis_connection' => $redis, 'queue' => $queue, + 'backend' => inFreshCoroutine(fn () => $workload?->backend()), + 'redis' => $settings['target'] === 'redis' && in_array($settings['case'], ['direct', 'relay'], true) + ? inFreshCoroutine(function () use ($application, $redis): array { + $profile = $application->make('config')->array("database.redis.{$redis}"); + $information = $application->make('redis')->connection($redis)->info('server'); + + return [ + 'version' => $information['redis_version'] ?? $information['Server']['redis_version'] ?? null, + 'host' => $profile['host'] ?? null, 'port' => $profile['port'] ?? null, + 'database' => $profile['database'] ?? null, + ]; + }) : null, + 'seconds' => $elapsed, 'jobs_per_second' => $settings['jobs'] / $elapsed, + 'cpu_seconds' => ($cpuAfter['ru_utime.tv_sec'] + $cpuAfter['ru_stime.tv_sec'] - $cpu['ru_utime.tv_sec'] - $cpu['ru_stime.tv_sec']) + + ($cpuAfter['ru_utime.tv_usec'] + $cpuAfter['ru_stime.tv_usec'] - $cpu['ru_utime.tv_usec'] - $cpu['ru_stime.tv_usec']) / 1e6, + 'heap_growth_bytes' => $heapGrowth, 'peak_heap_bytes' => $peakHeap, + 'gc_runs' => $gcAfter['runs'] - $gc['runs'], 'gc_collected' => $gcAfter['collected'] - $gc['collected'], + 'gc_roots' => $gcAfter['roots'], 'borrowed_after_sample' => $borrowed, + 'cluster_workload_wal_bytes' => $walBytes, 'details' => $details, + 'plans' => isset($options['explain']) ? inFreshCoroutine(fn () => $workload->explain()) : null, + ]; + } catch (Throwable $exception) { + $failure = $exception; + } finally { + try { + inFreshCoroutine(function () use ($workload, $queues, $queue, $settings): void { + try { + $workload?->tearDown(); + } finally { + if ($settings['target'] === 'redis' && in_array($settings['case'], ['direct', 'relay'], true)) { + $queues->connection('benchmark')->clear($queue); + if ($queues->connection('benchmark')->size($queue) !== 0) { + throw new RuntimeException('Benchmark queue cleanup failed.'); + } + } + } + }); + } catch (Throwable $cleanupFailure) { + if ($failure !== null) { + fwrite(STDERR, 'Cleanup also failed: ' . $cleanupFailure . PHP_EOL); + } + $failure ??= $cleanupFailure; + } + } + }); + + if ($failure !== null) { + throw $failure; + } + if (! $completed) { + throw new RuntimeException('The benchmark coroutine did not complete.'); + } + + echo json_encode($report, JSON_PRETTY_PRINT | JSON_THROW_ON_ERROR) . PHP_EOL; + } finally { + $application->terminate(); + if (is_string($sqlite)) { + unlink($sqlite); + } + } +} + +try { + main(); +} catch (Throwable $exception) { + fwrite(STDERR, (string) $exception . PHP_EOL); + exit(1); +} From cb5f9604e84278218a6924bdf5653cf19666ce36 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Thu, 22 Oct 2026 23:10:12 +0000 Subject: [PATCH 23/33] docs(queue): record completed outbox implementation and measurement scope Mark implementation, coverage, documentation and the functionally verified benchmark harness complete. Retain the explicitly deferred idle-machine measurements without claiming performance results. Specify focused benchmark cases and realistic lease populations, preserve automated query and resource-lifetime guarantees, and align verification instructions with scoped reruns after localized corrections. --- ...pervel-transactional-queue-outbox-codex.md | 20 ++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md index 9a0e337df..138317a04 100644 --- a/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md +++ b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md @@ -24,7 +24,7 @@ The result is a queue feature, not a generic event bus, inbox/deduplication syst Implemented: framework prerequisites, public selection and wrapper propagation, schema/configuration, owner-aware staging and repository operations, Redis/Horizon/database/SQS/pooled raw publication, relay/worker resource boundaries, operator commands, and guarded delivery events/telemetry. -Next: complete the remaining live-engine/concurrency and API-integration coverage. Canonical documentation, approved performance checks and final verification also remain. The full feature is not yet complete. +Implementation, live-engine/concurrency and API-integration coverage, canonical documentation and the retained benchmark harness are complete and verified. Only performance measurements, including the PostgreSQL claim-design comparison, remain owner-deferred until an idle-machine window. The harness is functionally verified; no performance results are claimed. ### Hypervel @@ -543,19 +543,21 @@ Place engine-specific cases under `tests/Integration/Queue/Database/{Postgres|My ### Performance gates -- Pin direct dispatch before/after query count, outbox service-resolution count, serialized fields, and payload transformations in automated tests: all remain unchanged when outbox is unselected. Use a repeatable local benchmark to measure the minimal cached job/config selection checks and investigate any material CPU or allocation regression; do not place unstable wall-time thresholds in CI. -- Obtain the owner's confirmation that the machine is idle before any benchmark. Benchmark single and 1k/10k staging, lease claims, Redis Lua batches, database inserts and SQS request construction; compare query/network counts and allocation as well as throughput. -- Verify one insert per normal bulk chunk, claim transaction control only for the separate locking-select/update path, publication using existing destination/protocol batches, and at most one accepted delete, deferred release and failed-row release per non-empty binding-safe settlement subset. Record peak borrowed source/transport connections and prove no lease spans unrelated work or sleep. -- Profile PostgreSQL/MySQL plans with representative pending/live-leased/retrying backlogs and concurrency. Compare the indexed availability/lease update against the prior separate-reservation predicate, including PostgreSQL index-write/HOT costs. Confirm the chosen design with evidence; resolve a substantive contrary result before changing the plan. Do not add indexes or tune engine settings speculatively. +- Automated tests pin unselected dispatch's service resolution, serialized fields and payload transformations, set-based staging, bounded claim statements, transport batching and settlement statements. Use multiple rows per settlement outcome so query-count assertions detect per-row regressions. Tiny-pool tests own phase-release correctness; do not duplicate their instrumentation in benchmarks. +- Retain a developer-only harness under `tests/Benchmarks/QueueOutbox/`, following the existing Database and RateLimiter harnesses. Use `benchmark.php`, a README and cohesive fixtures only where needed. No production hooks, dependencies, Artisan command or CI timing thresholds. Raw reports and disposable data stay outside Git. +- Cases: direct Redis dispatch for baseline/current CPU and allocation comparison; transactional single and bulk staging (including 1k/10k jobs); real-worker relay to Redis/database queues; concurrent claim/accepted-settlement/refill cycles against mixed eligible, delayed, live-leased and retrying backlogs. Seeded live leases use an explicit `--leased` count, defaulting to batch × concurrency, rather than growing with backlog size. Include refill in cycle throughput and report claim latency separately. Seed deadlines beyond the bounded run. Report the in-memory restart-cache choice so relay timings do not imply external cache I/O. Use plain queued jobs so the direct case loads in a pre-feature worktree. +- Create isolated randomly named tables from shipped migration stubs and use a unique Redis queue; remove only owned resources in `finally`, never flush a shared Redis database. SQLite is functional-only evidence. Verify exact row/target counts and no remaining borrowed sessions on every run. Tiny invocations validate the harness while the machine is busy; draw no performance conclusions from them. +- Warm up outside timing; record wall time, process CPU, peak/ending heap and GC deltas, payload size and non-secret runtime/backend settings. Use fresh-process samples and alternate baseline/current worktrees with independent dependencies. Obtain fresh idle-machine confirmation before performance measurements. +- Keep executing query plans opt-in and outside timing: capture actual claim SQL once, then run PostgreSQL `EXPLAIN (ANALYZE, BUFFERS, WAL)`, MySQL `EXPLAIN ANALYZE` on its locking select, MariaDB `ANALYZE FORMAT=JSON`, or SQLite `EXPLAIN QUERY PLAN`. Roll back executing diagnostic statements. Document supported engine versions; no optimizer assertions or capability framework. +- PostgreSQL alone also supports a harness-only reservation comparison: add unindexed `reserved_until`, change only the claim CTE to retain indexed availability and filter live reservations, seed equivalent lease states, and reuse accepted settlement. Compare claim latency/scanning and statement WAL, not WAL alone. Whole-cycle `pg_current_wal_insert_lsn()` differences are cluster-wide workload WAL, including refill/settlement and background activity—not isolated index writes. Do not force checkpoints, change fillfactor/autovacuum or reset statistics. After approved measurements, record findings and remove the temporary comparison variant. Investigate substantive contrary evidence before changing production design. -Run focused files immediately as they are added, then affected package suites, cross-package integration suites, analysis, and the repository checkpoint: +Run focused files immediately as they are added. Verify coherent slices with formatter fixes, static analysis and affected package/integration suites. Use full verification for framework-wide changes; after localized corrections rerun only affected checks and finish any stages not yet completed. Do not repeat completed whole-framework verification merely to obtain one green invocation on the latest revision. ```bash cd /home/binaryfire/workspace/contrib/hypervel/components-queue-outbox composer test:parallel -- tests/Queue --filter Outbox composer test:parallel -- tests/Bus --filter Outbox composer test:parallel -- tests/Horizon --filter Outbox -composer fix cd /home/binaryfire/workspace/contrib/hypervel/hypervel composer cs-fix @@ -564,7 +566,7 @@ composer test ``` -Run each new or changed test file immediately. The filters are only a coherent-slice inner loop. `composer fix` includes `composer test:testbench`, satisfying the Testbench migration-change check without a duplicate run, plus formatting, analysis, parallel tests and dogfood suites. When verifying the Testbench change independently, run `composer test:testbench` directly. +The filters are examples, not substitutes for covering other affected behavior. `composer fix` includes formatting, analysis, parallel tests, `composer test:testbench` and dogfood suites; do not repeat those stages separately when already completed. When verifying a Testbench change independently, run `composer test:testbench` directly. Reuse unchanged skeleton verification and confirm Foundation configuration/schema parity. If implementation uses a components worktree, copy `.env` from the main components checkout and run `composer install` in the worktree; never symlink `vendor` or Composer state. @@ -575,7 +577,7 @@ If implementation uses a components worktree, copy `.env` from the main componen 3. Integrate single/bulk staging, existing fast paths, SQS delay selection, forwarding and the prerequisite snapshot lifecycle. 4. Add the raw-batch SPI and first-party driver implementations, including pooled and Horizon paths. 5. Add claim/publication/settlement resource scopes, relay/daemon and operator commands, guarded events, and OpenTelemetry listeners. -6. Complete canonical docs and the exact-owner compatibility note, approved benchmarks, focused suites, static analysis and `composer fix`. +6. Complete canonical docs, the exact-owner compatibility note, retained benchmark harness and scoped verification above. Finish review and commits before the owner-deferred performance measurements; investigate any later measured bottleneck through the same review workflow. Before completion, re-read every changed file and confirm: From 910e55a019778059cb18761f2646507eb100de55 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Fri, 23 Oct 2026 00:05:31 +0000 Subject: [PATCH 24/33] docs(queue): clarify outbox trait property defaults Warn that redeclaring outbox properties with defaults conflicts with the declarations supplied by Queueable and Batchable. Direct users to the marker for class defaults and fluent methods for individual objects. Limit public property-default guidance to classes without those trait-provided properties, including queued listeners whose defaults are inspected without construction. --- src/docs/queue-outbox.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/docs/queue-outbox.md b/src/docs/queue-outbox.md index 57cb3d497..db7bd3cb9 100644 --- a/src/docs/queue-outbox.md +++ b/src/docs/queue-outbox.md @@ -105,7 +105,7 @@ SendOrderReceipt::dispatch($order)->withoutOutbox(); An explicit job choice takes precedence over the marker, followed by the queue connection's default. `afterCommit` and `beforeCommit` still control direct dispatch timing; neither disables the outbox. A selected outbox job is stored inside its owning transaction and cannot be relayed before that transaction commits. -The fluent methods are available through the usual `Queueable` trait. A directly queued object may also set a public `viaOutbox` property to `true` or `false`. `viaOutbox()` is a setter, not a getter hook like `viaConnection()` or `viaQueues()`. +The fluent methods come from `Queueable` and `Batchable`. Do not redeclare their outbox properties with defaults; PHP rejects the conflicting trait definitions. Use the marker for a class default or fluent methods for individual objects. Classes without these trait-provided properties may instead declare a public `viaOutbox` default of `true` or `false`. `viaOutbox()` is a setter, not a getter hook like `viaConnection()` or `viaQueues()`. A staged queue insertion returns the finalized job UUID instead of a broker identifier. The operator commands use this same UUID. Direct dispatch return values are unchanged. @@ -154,7 +154,7 @@ Mail::to($customer)->queue( ); ``` -For queued listeners, use the marker or a public property default: the dispatcher reads these defaults without constructing the listener. Do not rely on a listener constructor calling `viaOutbox`. When a listener uses `Queueable`, prefer the marker instead of redeclaring that trait's properties. +For queued listeners, use the marker or, when the class has no trait-provided outbox properties, a public property default. The dispatcher reads these defaults without constructing the listener, so do not rely on its constructor calling `viaOutbox`. When a listener uses `Queueable` or `Batchable`, use the marker instead of redeclaring the trait properties. Dispatch the event while the business transaction is open. An event implementing `ShouldDispatchAfterCommit` postpones the entire event, including listener and broadcast queueing, until after commit. Adding outbox selection cannot make that later staging atomic with the earlier transaction. Select the outbox on the queued listener or broadcast event instead of deferring the whole event when you need transactional delivery. From 29a8f847484ea23c5d2141f5746e33b18e0febfe Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Fri, 23 Oct 2026 02:00:48 +0000 Subject: [PATCH 25/33] docs(queue): align plan guidance for outbox trait properties Require the marker for queued listeners using Queueable or Batchable. Both traits provide uninitialized outbox properties, so redeclaring those properties with defaults is a PHP conflict rather than an alternative convention. Align the active implementation plan with the canonical guide. No runtime behavior changes. --- ...2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md index 138317a04..60b8a6ea2 100644 --- a/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md +++ b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md @@ -461,7 +461,7 @@ Update in the same change so no stale behavior survives: - `src/docs/documentation.md`: add the new guide to navigation; - `src/docs/opentelemetry.md`: add the three outbox metrics and their defaults to the queue metrics reference; explain failed-batch counts through duration observations with `error.type`; - `src/docs/queues.md`: add only a concise transactional-outbox introduction, when it is useful, one basic `viaOutbox()` example, and a link to `queue-outbox.md`; do not duplicate its setup, guarantees, driver matrix, or operations reference; -- `src/docs/events.md`, `mail.md`, `notifications.md`, and `broadcasting.md`: beside their existing after-commit guidance, add one concise example/cross-link showing producer selection propagating to its wrapper job. Queued listeners are inspected without constructing them: use the marker or public property defaults, not constructor-fluent calls. When using `Queueable`, prefer the marker rather than incompatibly redeclaring its properties. Application-constructed producers can use the fluent methods. Link to `queue-outbox.md` for guarantees rather than duplicating them; +- `src/docs/events.md`, `mail.md`, `notifications.md`, and `broadcasting.md`: beside their existing after-commit guidance, add one concise example/cross-link showing producer selection propagating to its wrapper job. Queued listeners are inspected without constructing them: use the marker or public property defaults, not constructor-fluent calls. When a listener uses `Queueable` or `Batchable`, use the marker; PHP rejects redeclaring their outbox properties with defaults. Application-constructed producers can use the fluent methods. Link to `queue-outbox.md` for guarantees rather than duplicating them; - `src/docs/porting-from-laravel.md` and database README: explain exact resolved transaction identity for extensions observing role-qualified records. Do not add outbox catalogue entries: its opt-in APIs require no Laravel adaptation. Preserve Queue/Bus/Horizon README structure without adding additive-feature differences; - `/home/binaryfire/workspace/contrib/hypervel/hypervel`: match Foundation's outbox configuration and the Testbench schema in the next application migration; - configuration comments and migration command help: exact guarantees and required worker/table setup. From 4eed24fcb1f94d08a356be11d8b3b205e154d007 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Fri, 23 Oct 2026 02:55:16 +0000 Subject: [PATCH 26/33] perf(queue): retire the measured reservation comparison Remove the temporary PostgreSQL reservation-column fixture and its CLI selector after comparing it with the production claim path. Keep the retained harness focused on production dispatch, staging, relay and claim workloads. The comparison increased live-lease scan work without a consistent statement-WAL saving, so no production schema or claim changes are needed. Remove the unused schema import, option validation and alternate backlog setup. Verified targeted formatting and PHP syntax, all four tiny harness cases, both relay targets, PostgreSQL executing plans, exact counts and released resources. Rechecked SQLite claims after the final fixture cleanup. --- .../QueueOutbox/Fixtures/OutboxWorkload.php | 27 ++------------ .../Fixtures/ReservationRepository.php | 36 ------------------- tests/Benchmarks/QueueOutbox/benchmark.php | 9 ++--- 3 files changed, 5 insertions(+), 67 deletions(-) delete mode 100644 tests/Benchmarks/QueueOutbox/Fixtures/ReservationRepository.php diff --git a/tests/Benchmarks/QueueOutbox/Fixtures/OutboxWorkload.php b/tests/Benchmarks/QueueOutbox/Fixtures/OutboxWorkload.php index 918a44064..83cb340b1 100644 --- a/tests/Benchmarks/QueueOutbox/Fixtures/OutboxWorkload.php +++ b/tests/Benchmarks/QueueOutbox/Fixtures/OutboxWorkload.php @@ -7,7 +7,6 @@ use Closure; use Hypervel\Contracts\Foundation\Application; use Hypervel\Coroutine\Waiter; -use Hypervel\Database\Schema\Blueprint; use Hypervel\Queue\Outbox\DatabaseOutboxRepository; use Hypervel\Queue\Outbox\OutboxBatchResult; use Hypervel\Queue\Outbox\OutboxWorker; @@ -53,12 +52,6 @@ public function setUp(): void { $connection = $this->application->make('db')->connection($this->source); $schema = $connection->getSchemaBuilder(); - $reservation = $this->settings['claim_design'] === 'reservation'; - - if ($reservation && $connection->getDriverName() !== 'pgsql') { - throw new InvalidArgumentException('The reservation comparison is PostgreSQL-only.'); - } - $tables = [$this->table => 'outbox']; if ($this->settings['target'] === 'database') { $tables[$this->jobsTable] = 'jobs'; @@ -83,12 +76,7 @@ public function setUp(): void } } - if ($reservation) { - $schema->table($this->table, static fn (Blueprint $table) => $table->unsignedBigInteger('reserved_until')->nullable()); - $this->repository = new ReservationRepository($this->application->make('db'), [$this->source], $this->table, 3600); - } else { - $this->repository = $this->application->make(DatabaseOutboxRepository::class); - } + $this->repository = $this->application->make(DatabaseOutboxRepository::class); // Redis staging never contacts Redis. It also avoids the database-queue atomic fast path. $this->application->make('config')->set('queue.connections.benchmark_seed', [ @@ -99,9 +87,6 @@ public function setUp(): void $row = (array) $connection->table($this->table)->useWritePdo()->first(); unset($row['id']); $row['connection'] = 'benchmark'; - if ($reservation) { - $row['reserved_until'] = null; - } $this->template = $row; $connection->table($this->table)->delete(); } @@ -118,17 +103,11 @@ public function prepare(int $jobs): void $leased = $this->settings['leased']; $eligible = max($this->settings['backlog'] - 2 * $background - $leased, $this->settings['batch'] * $this->settings['concurrency']); $this->distribution = ['eligible' => $eligible, 'delayed' => $background, 'leased' => $leased, 'retrying' => $background]; - $now = time(); - $future = $now + 86400; + $future = time() + 86400; $this->insertRows($eligible); $this->insertRows($background, ['available_at' => $future, 'deliver_at' => $future]); $this->insertRows($background, ['available_at' => $future, 'attempts' => 5, 'last_error' => 'Benchmark retry backlog']); - $lease = ['available_at' => $future, 'attempts' => 1, 'lease_token' => (string) Str::uuid()]; - if ($this->settings['claim_design'] === 'reservation') { - $lease['available_at'] = $now - 60; - $lease['reserved_until'] = $future; - } - $this->insertRows($leased, $lease); + $this->insertRows($leased, ['available_at' => $future, 'attempts' => 1, 'lease_token' => (string) Str::uuid()]); $this->analyze(); } } diff --git a/tests/Benchmarks/QueueOutbox/Fixtures/ReservationRepository.php b/tests/Benchmarks/QueueOutbox/Fixtures/ReservationRepository.php deleted file mode 100644 index a2bcf46c3..000000000 --- a/tests/Benchmarks/QueueOutbox/Fixtures/ReservationRepository.php +++ /dev/null @@ -1,36 +0,0 @@ -getQueryGrammar()->wrapTable($this->table); - - return $connection->selectFromWriteConnection( - "WITH candidates AS ( - SELECT id, available_at FROM {$table} - WHERE available_at <= ? AND (lease_token IS NULL OR reserved_until <= ?) - ORDER BY available_at, id LIMIT ? FOR UPDATE SKIP LOCKED - ) - UPDATE {$table} AS outgoing - SET lease_token = ?, reserved_until = ?, attempts = outgoing.attempts + 1 - FROM candidates WHERE outgoing.id = candidates.id - RETURNING outgoing.*, candidates.available_at AS eligible_at", - [$now, $now, $limit, $token, $expiresAt], - ); - } -} diff --git a/tests/Benchmarks/QueueOutbox/benchmark.php b/tests/Benchmarks/QueueOutbox/benchmark.php index 60ab9edfc..ed749e404 100644 --- a/tests/Benchmarks/QueueOutbox/benchmark.php +++ b/tests/Benchmarks/QueueOutbox/benchmark.php @@ -42,7 +42,7 @@ function inFreshCoroutine(Closure $callback): mixed */ function main(): void { - $options = getopt('', ['case:', 'target:', 'database-connection:', 'redis-connection:', 'jobs:', 'batch:', 'concurrency:', 'pool-size:', 'backlog:', 'leased:', 'payload-bytes:', 'warmup:', 'claim-design:', 'explain', 'help']); + $options = getopt('', ['case:', 'target:', 'database-connection:', 'redis-connection:', 'jobs:', 'batch:', 'concurrency:', 'pool-size:', 'backlog:', 'leased:', 'payload-bytes:', 'warmup:', 'explain', 'help']); if (isset($options['help'])) { echo "Usage: php tests/Benchmarks/QueueOutbox/benchmark.php [options]\n" @@ -51,7 +51,6 @@ function main(): void . " --redis-connection=default --jobs=10000 --batch=100 --concurrency=1\n" . " --pool-size=4 --backlog=10000 --leased=N (default: batch * concurrency)\n" . " --payload-bytes=256 --warmup=100\n" - . " --claim-design=availability|reservation (PostgreSQL claims comparison)\n" . " --explain (claims only; executing plans outside timing)\n" . "Use a disposable database and an idle machine for measurements. See README.md.\n"; @@ -60,21 +59,17 @@ function main(): void $case = $options['case'] ?? 'stage'; $target = $options['target'] ?? 'redis'; - $design = $options['claim-design'] ?? 'availability'; if (! in_array($case, ['direct', 'stage', 'relay', 'claims'], true) || ! in_array($target, ['redis', 'database'], true) - || ! in_array($design, ['availability', 'reservation'], true) || ($target === 'database' && $case !== 'relay') - || ($design === 'reservation' && $case !== 'claims') || (isset($options['explain']) && $case !== 'claims')) { - throw new InvalidArgumentException('Invalid case, target, claim design or explain combination; see --help.'); + throw new InvalidArgumentException('Invalid case, target or explain combination; see --help.'); } $settings = [ 'case' => $case, 'target' => $target, - 'claim_design' => $design, 'jobs' => integerOption($options, 'jobs', 10_000), 'batch' => integerOption($options, 'batch', 100), 'concurrency' => integerOption($options, 'concurrency', 1), From 18e0de11771c05e4b6687bb0c69818aade328aac Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Fri, 23 Oct 2026 02:55:32 +0000 Subject: [PATCH 27/33] docs(queue): record outbox performance measurements Record paired ordinary-dispatch measurements, transactional staging, real relay throughput and concurrent claim churn across PostgreSQL, MySQL and MariaDB. Include runtime settings, sample sizes, medians and ranges, CPU and heap observations, and the PostgreSQL claim-plan and WAL comparison. Keep the failed oversubscribed sample and its investigation visible without claiming an unproven cause. Explain that claim latency includes checkout and that the relay concurrency-eight samples used four pool slots, rather than presenting them as matched-capacity scaling results. Update reproducible examples to match pool capacity to concurrency, link the measured report, and remove obsolete reservation-selector instructions. Preserve raw evidence outside Git; temporary benchmark resources have been removed. Production behavior remains unchanged. --- tests/Benchmarks/QueueOutbox/README.md | 20 ++--- tests/Benchmarks/QueueOutbox/results.md | 108 ++++++++++++++++++++++++ 2 files changed, 115 insertions(+), 13 deletions(-) create mode 100644 tests/Benchmarks/QueueOutbox/results.md diff --git a/tests/Benchmarks/QueueOutbox/README.md b/tests/Benchmarks/QueueOutbox/README.md index 43d92a1b1..ae58968a1 100644 --- a/tests/Benchmarks/QueueOutbox/README.md +++ b/tests/Benchmarks/QueueOutbox/README.md @@ -2,6 +2,8 @@ This developer-only harness measures ordinary dispatch, transactional outbox staging, relay delivery, and concurrent SQL claims. It uses the real queue drivers, repository and relay worker. It does not execute application jobs, run in PHPUnit, or impose CI throughput thresholds. +See [results.md](results.md) for recorded measurements and the PostgreSQL claim-design comparison. + Run from the components repository root. Get fresh confirmation that the machine is idle before collecting performance measurements. Tiny invocations may check execution, counts and cleanup while the machine is busy; their timings are not performance evidence. ```shell @@ -31,8 +33,8 @@ Use `--target=database` for the database relay case; otherwise the target is Red ```shell timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php --case=direct --jobs=10000 --batch=100 timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php --case=stage --database-connection=mysql --jobs=10000 --batch=1000 -timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php --case=relay --database-connection=pgsql --jobs=10000 --batch=100 --concurrency=8 --pool-size=4 -timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php --case=relay --target=database --database-connection=pgsql --jobs=10000 --concurrency=8 --pool-size=4 +timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php --case=relay --database-connection=pgsql --jobs=10000 --batch=100 --concurrency=8 --pool-size=8 +timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php --case=relay --target=database --database-connection=pgsql --jobs=10000 --concurrency=8 --pool-size=8 ``` Concurrency applies to relay and claims. Direct and staging workloads are sequential producers. Vary pool size and concurrency together to distinguish pool pressure from transport latency. Existing tiny-pool integration tests prove the release boundaries; this harness checks that no connections remain borrowed when a sample finishes, without adding a sampling timer or per-operation pool instrumentation. The reported `cache_store` is `worker-array`: relay restart checks still run, but these timings exclude the external cache round trips a Redis or database cache would add. @@ -43,31 +45,23 @@ Concurrency applies to relay and claims. Direct and staging workloads are sequen `--backlog=10000` sizes the mixed backlog; delayed and retrying rows each take a quarter. `--leased` separately controls seeded live leases, defaulting to `batch × concurrency`, not a fraction of the backlog. The remaining rows are eligible, increased when necessary to provide at least one batch per concurrent claimant; the report gives the actual distribution. Seeded leases represent other in-flight work, in addition to the short-lived claims made by this process. Each settled batch is replaced with fresh eligible rows. Warmup churn is retained, so measurement is not limited to a freshly inserted table. Background deadlines are one day away, beyond the bounded run. -Claim p50/p95 includes the repository operation, row materialization and child-coroutine cleanup, not just server execution time. Empty claims caused by competing fallback claimants count as attempts and are reported separately. Cycle throughput and WAL include settlement and refill, including UUID generation and payload encoding. Latency samples add common harness overhead and are retained only for this bounded invocation. +Claim p50/p95 includes connection checkout, the repository operation, row materialization and child-coroutine cleanup, not just server execution time. Use a pool at least as large as concurrency when comparing claim statements; smaller pools also measure waiting for other clients' settlement and refill. Empty claims caused by competing fallback claimants count as attempts and are reported separately. Cycle throughput and WAL include settlement and refill, including UUID generation and payload encoding. Latency samples add common harness overhead and are retained only for this bounded invocation. ```shell timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php \ --case=claims --database-connection=pgsql --jobs=10000 \ - --backlog=100000 --batch=100 --concurrency=8 --pool-size=4 --explain \ + --backlog=100000 --batch=100 --concurrency=8 --pool-size=8 --explain \ > /tmp/outbox-claims.json ``` `--explain` refreshes table statistics and captures the actual claim SQL once after churn, outside timing. Executing plans run in transactions that are rolled back. PostgreSQL uses `EXPLAIN (ANALYZE, BUFFERS, WAL, FORMAT JSON)`; MySQL uses `EXPLAIN ANALYZE` for selects; MariaDB uses `ANALYZE FORMAT=JSON`; SQLite uses `EXPLAIN QUERY PLAN`. PostgreSQL's WAL option requires 13+, and MySQL's executing explain requires 8.0.18+. Engines lacking these commands fail with their native error; omit `--explain` there. No optimizer plan is asserted as a correctness requirement. -### PostgreSQL claim-design comparison - -`--claim-design=availability` uses the production repository. `--claim-design=reservation` adds a benchmark-only, unindexed `reserved_until` column and replaces only the PostgreSQL claim CTE. It leaves indexed `available_at` unchanged and filters live reservations during selection. Both variants use the real accepted-delete settlement and equivalent logical backlogs. The reservation fixture is not a supported alternative production implementation. - -Run both variants in alternating fresh processes. Compare claim latency, buffers and rows filtered alongside write volume: avoiding index changes can reduce writes but require scanning live reservations. Compare the default seeded lease count with a larger `--leased` value representing several other relays; reservation scan cost grows with live leases, not simply with total backlog. No fillfactor, autovacuum or checkpoint tuning is performed. HOT updates still depend on available page space; an unindexed deadline does not guarantee them. - The cluster-wide `pg_current_wal_insert_lsn()` delta includes the entire claim/settle/refill workload and unrelated database activity. It is not isolated index-write cost. The executing PostgreSQL plan supplies statement-local WAL and buffer evidence. Checkpoint state and full-page images can affect comparisons; use repeated samples rather than one ratio. See PostgreSQL's [WAL position functions](https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-ADMIN-BACKUP) and [EXPLAIN options](https://www.postgresql.org/docs/current/sql-explain.html). -Once approved measurements settle the comparison, record the findings and remove the temporary reservation fixture and selector. Do not change the production schema based only on WAL bytes or a single small run. - ## Comparisons and interpretation For direct-path regression checks, copy this directory into an independently installed pre-feature worktree. The `direct` case uses only existing queue APIs and does not load feature-only fixtures. Use identical dependency versions, PHP configuration, opcache/JIT settings and services. Record revisions, uncommitted changes and commands with raw JSON outside Git. Alternate baseline/current order for at least ten fresh-process pairs; compare medians and spread, not the fastest sample. Direct publication and durable staging provide different guarantees; their throughput is not an apples-to-apples regression comparison. The report includes wall time, process CPU, heap/GC data and non-secret backend settings. CPU excludes database and Redis processes; heap growth is not an allocation count. Driver/network latency, payload size and database durability settings remain part of real throughput. -For a quick functional check, use `--jobs=8 --batch=2 --warmup=2 --backlog=12`. Check all four cases, both relay targets and the PostgreSQL comparison when those services are available. Keep these checks separate from idle-machine measurements. There are no recorded performance results yet. +For a quick functional check, use `--jobs=8 --batch=2 --warmup=2 --backlog=12`. Check all four cases, both relay targets and PostgreSQL claims with `--explain` when those services are available. Keep these checks separate from idle-machine measurements. diff --git a/tests/Benchmarks/QueueOutbox/results.md b/tests/Benchmarks/QueueOutbox/results.md new file mode 100644 index 000000000..6d1b89b69 --- /dev/null +++ b/tests/Benchmarks/QueueOutbox/results.md @@ -0,0 +1,108 @@ +# Queue outbox measurements + +Measured on October 23, 2026. The results support keeping the production outbox design unchanged. Ordinary single dispatch was effectively unchanged, bulk dispatch improved in the paired comparison, and the PostgreSQL reservation-column alternative added scan work without a consistent statement-WAL saving. See [README.md](README.md) for the retained harness. + +## Setup and interpretation + +- Baseline: `a67e2c9143078d4feca709869803be576604d9e4`. Measured feature branch: `29a8f847484ea23c5d2141f5746e33b18e0febfe`. Separate worktrees had independent Composer installs using the same lockfile. The comparison covers the complete feature branch, including its framework correctness changes, not only the outbox-selection check. +- Linux x86-64 VM exposing six CPUs, Intel Core i7-7700 at 3.60 GHz; PHP 8.4.26, Swoole 6.3.0RC1, CLI OPcache enabled, JIT disabled. +- Local PostgreSQL 17.11, MySQL 9.5.0, MariaDB 10.11.19 and Redis 8.2.1. PostgreSQL `fsync`, `synchronous_commit` and `full_page_writes` remained enabled. No forced checkpoints, statistics resets, fillfactor or autovacuum changes. +- Job contents: 256 bytes; stored outbox envelope: 1,246 bytes. Relay restart cache: `worker-array`, so these runs exclude external restart-cache round trips. Jobs were published, not executed. +- Each sample ran in a fresh process after warmup. Tables show median throughput with observed minimum–maximum, including outliers. CPU covers only the calling PHP process. Database/Redis server CPU is not included; heap growth is not an allocation count. + +These are local workload measurements, not production capacity promises. Database-backed wall times varied substantially while PHP CPU stayed steadier. Diagnostic PostgreSQL monitoring observed simultaneous `WalSync` and `WALWrite` waits, consistent with storage delays. It did not capture the exact cause of every outlier. Do not infer small throughput differences or choose a schema from wall time alone. + +Every successful sample checked exact completed counts, remaining lease counts where applicable, zero borrowed connections after completion, and removal of its own tables/queue. The failed oversubscribed run is recorded below rather than discarded. Raw reports, diagnostic logs and run scripts are retained outside Git. + +## Ordinary Redis dispatch + +Ten alternating baseline/current pairs per case. Single samples dispatched 20,000 jobs; bulk samples dispatched 100,000 in batches of 1,000. Warmup: 1,000 jobs. + +| Dispatch | Baseline jobs/s, median (range) | Feature jobs/s, median (range) | +| --- | ---: | ---: | +| Single | 5,074 (4,992–5,210) | 5,073 (4,981–5,180) | +| Bulk, 1,000 per call | 40,074 (39,107–41,351) | 43,491 (40,829–44,366) | + +Single throughput changed by −0.01% using the ratio of medians. Bulk's ratio of medians improved by 8.5%; all ten matched pairs improved, with a median paired change of 9.3% and a 1.1–12.8% range. The pooled bulk ranges overlap, so this is a measured paired improvement, not a claim that every future run will be faster. + +Median PHP CPU was 2.113 → 2.129 seconds for single samples and 1.990 → 1.809 seconds for bulk samples. Ending heap growth was 5,216 bytes on both revisions in both cases. The peak heap difference was about 24 KB in both cases. + +## Transactional staging + +Three samples per engine/batch size. Single samples staged 2,000 jobs; bulk samples staged 10,000. Warmup: 100 jobs. Each dispatch call had its own transaction, so bulk amortizes commit cost rather than providing the same workload as single dispatch. + +| Database | Jobs per call | Jobs/s, median (range) | +| --- | ---: | ---: | +| PostgreSQL | 1 | 266 (260–266) | +| PostgreSQL | 1,000 | 10,964 (9,425–11,241) | +| PostgreSQL | 10,000 | 11,312 (11,007–12,124) | +| MySQL | 1 | 86 (82–89) | +| MySQL | 1,000 | 1,046 (378–4,581) | +| MySQL | 10,000 | 7,504 (7,426–7,508) | +| MariaDB | 1 | 283 (276–284) | +| MariaDB | 1,000 | 11,281 (8,455–11,428) | +| MariaDB | 10,000 | 11,237 (9,021–11,657) | + +MySQL's 1,000-job batch samples took 2.18, 26.44 and 9.56 seconds while PHP CPU stayed around 0.4 seconds. The outliers are included; these runs do not establish an engine ranking. Bulk storage and binding-limit chunking completed correctly on all three engines. + +## Real relay worker + +PostgreSQL source, Redis or PostgreSQL database-queue target, four database/Redis pool slots, 30,000 jobs per sample and 1,000 warmup jobs. Three samples per configuration. Claims, publication, settlement and worker lifecycle were timed; initial seeding was not. + +| Target | Claim batch | Concurrency | Jobs/s, median (range) | +| --- | ---: | ---: | ---: | +| Redis | 100 | 1 | 8,328 (7,417–8,801) | +| Redis | 100 | 4 | 8,211 (7,770–8,421) | +| Redis | 100 | 8 | 7,301 (7,080–8,199) | +| Redis | 1,000 | 1 | 16,983 (7,971–17,519) | +| Redis | 1,000 | 4 | 22,387 (14,396–22,700) | +| Redis | 1,000 | 8 | 22,655 (21,515–23,118) | +| Database | 100 | 1 | 5,136 (723–6,395) | +| Database | 100 | 4 | 10,963 (9,450–11,153) | +| Database | 100 | 8 | 7,498 (7,101–11,011) | +| Database | 1,000 | 1 | 10,749 (8,376–10,952) | +| Database | 1,000 | 4 | 11,754 (6,867–14,457) | +| Database | 1,000 | 8 | 13,740 (10,850–14,373) | + +Larger batches helped on this setup. Pool capacity stayed fixed at four slots, so the concurrency-8 results include any checkout contention rather than isolating scaling with matched capacity. Size pools for relay concurrency as the guide describes; tasks release each phase's connection before starting the next. All relay samples drained completely and released their borrowed resources. The database-target, batch-100, concurrency-1 group includes a 41.48-second sample, versus 4.69 and 5.84 seconds; PHP CPU was 0.86–0.94 seconds. + +## Claims under churn + +Claims were followed by accepted settlement and replacement inserts, keeping the mixed backlog populated. Each sample completed 30,000 rows after 5,000 warmup rows, with batch 100, eight clients and eight pool slots. Claim latency includes checkout, materialization and child cleanup; total cycle throughput additionally includes settlement/refill, UUID generation and payload encoding. + +PostgreSQL used five alternating pairs per backlog/lease group. The temporary alternative added an unindexed `reserved_until` column and left indexed `available_at` unchanged, filtering live reservations during selection. Both used the production accepted-delete settlement. The comparison fixture and selector were removed after measurement; their measured source remains available at the revision above. + +| Actual backlog | Seeded live leases | Current cycle jobs/s, median (range) | Reservation cycle jobs/s, median (range) | +| --- | ---: | ---: | ---: | +| 10,000 | 800 | 16,789 (2,830–16,952) | 16,603 (11,843–17,079) | +| 13,800 | 8,000 | 16,582 (1,656–21,312) | 15,331 (14,535–18,607) | +| 100,000 | 800 | 10,732 (4,533–21,376) | 12,120 (5,775–19,676) | +| 100,000 | 8,000 | 10,437 (3,710–20,024) | 10,026 (5,695–18,311) | + +The 13,800-row group requested `--backlog=10000 --leased=8000`; the harness preserves 2,500 delayed and 2,500 retrying rows plus at least 800 eligible rows. Other groups match the requested backlog. The noisy cycle medians do not uniformly favor the current design. + +Executing PostgreSQL plans give the more direct scan comparison. Times include the claim statement's update, not just selection. Buffer hits below are on the eligibility-index scan; WAL is statement-local bytes, not the cluster-wide workload delta. + +| Backlog / live leases | Current statement ms, median (range) | Reservation statement ms, median (range) | Eligibility buffer hits, current / reservation | Statement WAL bytes, current / reservation | +| --- | ---: | ---: | ---: | ---: | +| 10,000 / 800 | 2.086 (1.898–2.197) | 2.505 (2.348–9.599) | 158 / 657 | 170,017 / 173,709 | +| 13,800 / 8,000 | 2.066 (1.805–2.153) | 4.948 (4.646–5.028) | 166 / 2,219 | 173,675 / 172,678 | +| 100,000 / 800 | 1.857 (1.653–2.392) | 2.309 (2.094–2.993) | 158 / 642 | 174,282 / 174,592 | +| 100,000 / 8,000 | 1.732 (1.631–1.873) | 4.581 (4.341–4.691) | 158 / 2,109 | 174,053 / 174,674 | + +The current query used bounded index scans without filtering live leases. The alternative filtered all 800 or 8,000 seeded live leases per statement, increased scan work and showed no consistent statement-WAL saving. An unindexed deadline alone does not guarantee HOT updates; free page space also matters. Keep the existing indexed eligibility/lease deadline and avoid another schema column. + +MySQL and MariaDB each had three samples at backlog 100,000 and 800 live leases: + +| Database | Cycle jobs/s, median (range) | +| --- | ---: | +| MySQL | 1,885 (1,083–2,050) | +| MariaDB | 3,635 (946–7,317) | + +Their executing SELECT plans used the eligibility index and returned 100 rows. Inspected examples took about 0.25 ms on MySQL and 0.17 ms on MariaDB; these SELECT times exclude the update and transaction commit. All 46 matched-capacity samples passed counts and resource cleanup. + +### Oversubscribed pool diagnostic + +Earlier PostgreSQL comparisons used eight clients and four slots. One 100,000-row, 8,000-live-lease sample failed with four checkout timeouts under the unchanged three-second `wait_timeout`. That failed sample has no throughput result. The completed earlier samples remain contention evidence, not the basis for the claim-design conclusion above. + +One trace-assisted repeat and five further monitored repeats passed with zero borrowed connections afterward. Server monitoring observed simultaneous `WalSync`/`WALWrite` waits up to about 1.44 seconds per statement. A settlement/refill phase includes both the settlement commit and the replacement insert's commit, so these waits are consistent with exceeding checkout time under pool contention. They do not prove the original failure's exact cause. No connection-lifetime defect was confirmed, and no timeout increase, retry workaround or production change was introduced. From 529cea1a72cda803adaa668d798d7c0b3ded23f8 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Fri, 23 Oct 2026 02:55:33 +0000 Subject: [PATCH 28/33] docs(queue): complete outbox measurement plan Mark implementation and performance measurements complete and link the retained benchmark report. Replace deferred-measurement and temporary-comparison instructions with the measured rationale for retaining indexed eligibility and the production claim path. Clarify heap versus allocation measurements and the checkout component of claim latency. Record the requirement to include performance tables with workload, concurrency and pool settings when opening the pull request. --- ...1522-hypervel-transactional-queue-outbox-codex.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md index 60b8a6ea2..6f5a3b1a3 100644 --- a/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md +++ b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md @@ -24,7 +24,9 @@ The result is a queue feature, not a generic event bus, inbox/deduplication syst Implemented: framework prerequisites, public selection and wrapper propagation, schema/configuration, owner-aware staging and repository operations, Redis/Horizon/database/SQS/pooled raw publication, relay/worker resource boundaries, operator commands, and guarded delivery events/telemetry. -Implementation, live-engine/concurrency and API-integration coverage, canonical documentation and the retained benchmark harness are complete and verified. Only performance measurements, including the PostgreSQL claim-design comparison, remain owner-deferred until an idle-machine window. The harness is functionally verified; no performance results are claimed. +Implementation, live-engine/concurrency and API-integration coverage, canonical documentation, the retained benchmark harness and performance measurements are complete. Measurements support the existing production design; see [benchmark results](../../tests/Benchmarks/QueueOutbox/results.md). + +When opening the PR, include a performance section with tables from the benchmark results and their workload, concurrency and pool settings. ### Hypervel @@ -387,7 +389,7 @@ Do not add raw-delay methods to the general `Queue` contract. Custom durable dri - The coordinator waits for publication capacity, then claims through a fresh `Waiter` child with no copied request/transaction context. That child alone resolves the source database, completes the claim, returns immutable `OutboxMessage` values, and ends so `ConnectionResolver` releases the database lease before any broker work starts. Count actual claimed rows before starting their concurrent delivery task. Claims are serial per daemon, like `queue:work` pops; publication/settlement tasks and independent daemons remain concurrent. - Claim at most the validated worker batch limit eligible rows ordered by `available_at`, then `id`. - Eligibility is `available_at <= now`. A claim atomically sets one fresh UUID lease token, `available_at = now + retry_after`, and increments attempts. This moves live leases out of the hot eligibility range without a second timestamp/filter. Use an explicit transaction only for the separate locking-select/update path; never hold SQL locks during broker calls. -- PostgreSQL: one autocommitted CTE using `FOR UPDATE SKIP LOCKED` plus `UPDATE ... RETURNING`, returning the selected pre-update availability as `eligibleAt` with the updated token/attempt. No extra BEGIN/COMMIT round trips. Updating the indexed timestamp trades HOT-update eligibility for avoiding repeated live-lease scans; validate this tradeoff in the approved benchmark. +- PostgreSQL: one autocommitted CTE using `FOR UPDATE SKIP LOCKED` plus `UPDATE ... RETURNING`, returning the selected pre-update availability as `eligibleAt` with the updated token/attempt. No extra BEGIN/COMMIT round trips. Updating the indexed timestamp trades HOT-update eligibility for avoiding repeated live-lease scans; the measured comparison supports retaining this design without an additional reservation column. - Supported MySQL/MariaDB/Vitess: `SELECT ... FOR UPDATE SKIP LOCKED`, one guarded bulk update, and return the already selected rows with token/time/attempt fields reflected in memory rather than issuing another select. - SQLite/engines without `SKIP LOCKED`: select an ordered, limited list of candidate IDs and their availability, perform one autocommitted atomic eligibility-guarded token/expiry update restricted to those IDs, then load full rows restricted to those IDs bearing that token. No encompassing transaction is needed: the token identifies this claim even if another claimant intervenes or reclaims an expired lease. Preserve original eligibility from the candidates. The second read distinguishes rows actually won; another claimant can cause an empty/partial result, not a double claim. SQLite serializes writers, so additional relays improve availability rather than write throughput. - Normalize every returned claim to its pre-claim `(eligibleAt, id)` order; `UPDATE ... RETURNING` and `WHERE IN` do not promise order, and the newly shared lease deadline is not the original eligibility. @@ -545,11 +547,11 @@ Place engine-specific cases under `tests/Integration/Queue/Database/{Postgres|My - Automated tests pin unselected dispatch's service resolution, serialized fields and payload transformations, set-based staging, bounded claim statements, transport batching and settlement statements. Use multiple rows per settlement outcome so query-count assertions detect per-row regressions. Tiny-pool tests own phase-release correctness; do not duplicate their instrumentation in benchmarks. - Retain a developer-only harness under `tests/Benchmarks/QueueOutbox/`, following the existing Database and RateLimiter harnesses. Use `benchmark.php`, a README and cohesive fixtures only where needed. No production hooks, dependencies, Artisan command or CI timing thresholds. Raw reports and disposable data stay outside Git. -- Cases: direct Redis dispatch for baseline/current CPU and allocation comparison; transactional single and bulk staging (including 1k/10k jobs); real-worker relay to Redis/database queues; concurrent claim/accepted-settlement/refill cycles against mixed eligible, delayed, live-leased and retrying backlogs. Seeded live leases use an explicit `--leased` count, defaulting to batch × concurrency, rather than growing with backlog size. Include refill in cycle throughput and report claim latency separately. Seed deadlines beyond the bounded run. Report the in-memory restart-cache choice so relay timings do not imply external cache I/O. Use plain queued jobs so the direct case loads in a pre-feature worktree. +- Cases: direct Redis dispatch for baseline/current CPU and heap comparison; transactional single and bulk staging (including 1k/10k jobs); real-worker relay to Redis/database queues; concurrent claim/accepted-settlement/refill cycles against mixed eligible, delayed, live-leased and retrying backlogs. Seeded live leases use an explicit `--leased` count, defaulting to batch × concurrency, rather than growing with backlog size. Include refill in cycle throughput and report claim latency separately, including checkout and child cleanup; use pool capacity at least equal to concurrency for claim-design comparisons. Seed deadlines beyond the bounded run. Report the in-memory restart-cache choice so relay timings do not imply external cache I/O. Use plain queued jobs so the direct case loads in a pre-feature worktree. - Create isolated randomly named tables from shipped migration stubs and use a unique Redis queue; remove only owned resources in `finally`, never flush a shared Redis database. SQLite is functional-only evidence. Verify exact row/target counts and no remaining borrowed sessions on every run. Tiny invocations validate the harness while the machine is busy; draw no performance conclusions from them. - Warm up outside timing; record wall time, process CPU, peak/ending heap and GC deltas, payload size and non-secret runtime/backend settings. Use fresh-process samples and alternate baseline/current worktrees with independent dependencies. Obtain fresh idle-machine confirmation before performance measurements. - Keep executing query plans opt-in and outside timing: capture actual claim SQL once, then run PostgreSQL `EXPLAIN (ANALYZE, BUFFERS, WAL)`, MySQL `EXPLAIN ANALYZE` on its locking select, MariaDB `ANALYZE FORMAT=JSON`, or SQLite `EXPLAIN QUERY PLAN`. Roll back executing diagnostic statements. Document supported engine versions; no optimizer assertions or capability framework. -- PostgreSQL alone also supports a harness-only reservation comparison: add unindexed `reserved_until`, change only the claim CTE to retain indexed availability and filter live reservations, seed equivalent lease states, and reuse accepted settlement. Compare claim latency/scanning and statement WAL, not WAL alone. Whole-cycle `pg_current_wal_insert_lsn()` differences are cluster-wide workload WAL, including refill/settlement and background activity—not isolated index writes. Do not force checkpoints, change fillfactor/autovacuum or reset statistics. After approved measurements, record findings and remove the temporary comparison variant. Investigate substantive contrary evidence before changing production design. +- Retain only the production claim path in the harness. The rejected reservation comparison and measured results belong in `tests/Benchmarks/QueueOutbox/results.md`: filtering live leases increased scan work without a consistent statement-WAL saving. Whole-cycle `pg_current_wal_insert_lsn()` differences are cluster-wide workload WAL, including refill/settlement and background activity—not isolated index writes. Do not force checkpoints, change fillfactor/autovacuum or reset statistics. Investigate substantive contrary evidence before changing production design. Run focused files immediately as they are added. Verify coherent slices with formatter fixes, static analysis and affected package/integration suites. Use full verification for framework-wide changes; after localized corrections rerun only affected checks and finish any stages not yet completed. Do not repeat completed whole-framework verification merely to obtain one green invocation on the latest revision. @@ -577,7 +579,7 @@ If implementation uses a components worktree, copy `.env` from the main componen 3. Integrate single/bulk staging, existing fast paths, SQS delay selection, forwarding and the prerequisite snapshot lifecycle. 4. Add the raw-batch SPI and first-party driver implementations, including pooled and Horizon paths. 5. Add claim/publication/settlement resource scopes, relay/daemon and operator commands, guarded events, and OpenTelemetry listeners. -6. Complete canonical docs, the exact-owner compatibility note, retained benchmark harness and scoped verification above. Finish review and commits before the owner-deferred performance measurements; investigate any later measured bottleneck through the same review workflow. +6. Complete canonical docs, the exact-owner compatibility note, retained benchmark harness, measurement results and scoped verification above. Investigate measured bottlenecks through the same review workflow. Before completion, re-read every changed file and confirm: From 294052aeb5bb880975b3037d7225dc93f3cd2963 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Fri, 23 Oct 2026 04:07:03 +0000 Subject: [PATCH 29/33] test(queue): account for runner memory in outbox relay tests Give ordinary relay tests an explicit memory budget so their behavior does not depend on how much of the framework PHPUnit has already loaded. Apply the same budget to Redis delivery tests that invoke the relay command. Keep the dedicated low-memory exit test and production worker defaults unchanged. This fixes premature memory-limit exits in the framework CI jobs without weakening relay lifecycle assertions. --- .../Queue/Redis/OutboxDeliveryTest.php | 6 ++--- tests/Queue/Outbox/OutboxWorkerTest.php | 22 +++++++++---------- 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/tests/Integration/Queue/Redis/OutboxDeliveryTest.php b/tests/Integration/Queue/Redis/OutboxDeliveryTest.php index 7194b0c3b..4fc31922a 100644 --- a/tests/Integration/Queue/Redis/OutboxDeliveryTest.php +++ b/tests/Integration/Queue/Redis/OutboxDeliveryTest.php @@ -92,7 +92,7 @@ public function testCommitRelayAndProcessingPreserveEncryptedModelsAndUniqueOwne $user->name = 'After dispatch'; $user->save(); - $this->assertSame(0, Artisan::call('queue:relay', ['--once' => true])); + $this->assertSame(0, Artisan::call('queue:relay', ['--once' => true, '--memory' => 1024])); $this->assertSame(0, DB::table('job_outbox')->count()); $this->assertSame(2, Queue::size()); $this->assertSame(2, $finalizations); @@ -122,7 +122,7 @@ static function (): void { Cache::put('closure', true); }, $this->assertSame(3, DB::table('job_outbox')->count()); $this->assertSame(4, $batch->fresh()->pendingJobs); $this->assertSame(0, Queue::size()); - $this->assertSame(0, Artisan::call('queue:relay', ['--once' => true])); + $this->assertSame(0, Artisan::call('queue:relay', ['--once' => true, '--memory' => 1024])); $this->work(); $this->assertSame(1, Cache::get('first')); @@ -140,7 +140,7 @@ public function testDebouncedAndDelayedPayloadsRetainTheirOriginalDeliveryRules( OutboxDebouncedDeliveryJob::dispatch('debounced')->viaOutbox(); OutboxDebouncedDeliveryJob::dispatch('debounced')->viaOutbox(); OutboxDeliveryJob::dispatch('delayed')->viaOutbox()->delay(60); - $this->assertSame(0, Artisan::call('queue:relay', ['--once' => true])); + $this->assertSame(0, Artisan::call('queue:relay', ['--once' => true, '--memory' => 1024])); $this->assertSame(0, DB::table('job_outbox')->count()); $this->assertNull(Queue::pop()); diff --git a/tests/Queue/Outbox/OutboxWorkerTest.php b/tests/Queue/Outbox/OutboxWorkerTest.php index 40c483247..5f7d6132f 100644 --- a/tests/Queue/Outbox/OutboxWorkerTest.php +++ b/tests/Queue/Outbox/OutboxWorkerTest.php @@ -62,7 +62,7 @@ public function testRoundRobinClaimsHaveAnExactJobLimitAndConcurrentPublication( } }); - $this->assertSame(Worker::EXIT_SUCCESS, $worker->run(null, new OutboxWorkerOptions(batch: 2, concurrency: 2, maxJobs: 3))); + $this->assertSame(Worker::EXIT_SUCCESS, $worker->run(null, new OutboxWorkerOptions(batch: 2, concurrency: 2, maxJobs: 3, memory: 1024))); $this->assertSame(2, $peak); $this->assertSame(2, $finished); $this->assertSame(0, $running); @@ -78,7 +78,7 @@ public function testFailedSourceCoolsDownWithoutDelayingHealthySources(): void $this->relay->shouldReceive('deliver')->with('healthy', m::type('array'))->times(3)->andReturn(true); $this->exceptions->expects('report')->with($failure, ['outbox_database' => 'broken']); - $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(batch: 1, sleep: 60, maxJobs: 3))); + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(batch: 1, sleep: 60, maxJobs: 3, memory: 1024))); Sleep::assertNeverSlept(); } @@ -91,7 +91,7 @@ public function testStopWhenEmptyTreatsFailedSourcesAsEmptyAndDoesNotRecheckAfte $this->relay->expects('claim')->with('empty', 100)->andReturn([]); $this->exceptions->expects('report')->with($failure, ['outbox_database' => 'broken']); - $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(stopWhenEmpty: true))); + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(memory: 1024, stopWhenEmpty: true))); Sleep::assertNeverSlept(); } @@ -103,7 +103,7 @@ public function testOnceVisitsEverySourceWithoutSleepingEvenWhenDeliveryFails(): $this->relay->expects('claim')->with('second', 100)->andReturn([]); $this->relay->expects('deliver')->with('first', m::type('array'))->andReturn(false); - $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(once: true))); + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(memory: 1024, once: true))); Sleep::assertNeverSlept(); } @@ -112,7 +112,7 @@ public function testPinnedSourceDoesNotClaimFromOtherConfiguredDatabases(): void $worker = $this->worker(['first', 'second']); $this->relay->expects('claim')->with('second', 100)->andReturn([]); - $this->assertSame(0, $worker->run('second', new OutboxWorkerOptions(once: true))); + $this->assertSame(0, $worker->run('second', new OutboxWorkerOptions(memory: 1024, once: true))); } public function testFailureSleepKeepsThePublicationSlotOccupied(): void @@ -129,7 +129,7 @@ public function testFailureSleepKeepsThePublicationSlotOccupied(): void $this->assertSame(1, $claims); }); - $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(batch: 1, sleep: 0.25, maxJobs: 2))); + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(batch: 1, sleep: 0.25, maxJobs: 2, memory: 1024))); Sleep::assertSequence([Sleep::usleep(250_000)]); } @@ -141,7 +141,7 @@ public function testEmptyPassSleepsOnceAndSignalStopsFurtherAdmission(): void $this->relay->expects('claim')->with('second', 100)->andReturn([]); Sleep::whenFakingSleep(function (): void { ($this->signalHandler)(SIGTERM); }); - $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(sleep: 0.25))); + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(sleep: 0.25, memory: 1024))); Sleep::assertSequence([Sleep::usleep(250_000)]); } @@ -160,7 +160,7 @@ public function testPauseStopsAdmissionUntilTheResumeSignal(): void ($this->signalHandler)(++$sleeps === 1 ? SIGUSR2 : SIGCONT); }); - $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(sleep: 0.25, maxJobs: 1))); + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(sleep: 0.25, maxJobs: 1, memory: 1024))); Sleep::assertSequence([Sleep::usleep(250_000), Sleep::usleep(250_000)]); } @@ -184,7 +184,7 @@ public function testRestartReadsEndTheirCoroutineBeforeClaimingAndRestartStopsAd }); $this->relay->expects('deliver')->andReturn(true); - $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions)); + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(memory: 1024))); $this->assertFalse($borrowed); } @@ -196,7 +196,7 @@ public function testEscapingTaskFailureStopsAdmissionAndPropagates(): void $this->relay->expects('deliver')->andThrow($failure); try { - $worker->run(null, new OutboxWorkerOptions); + $worker->run(null, new OutboxWorkerOptions(memory: 1024)); $this->fail('Task failure must reach the command.'); } catch (RuntimeException $exception) { $this->assertSame($failure, $exception); @@ -220,7 +220,7 @@ public function testCancellationJoinsTaskCleanupBeforeReturningToTheCaller(): vo }); $coordinator = Coroutine::create(static function () use ($worker, &$caught): void { try { - $worker->run(null, new OutboxWorkerOptions(maxJobs: 1)); + $worker->run(null, new OutboxWorkerOptions(maxJobs: 1, memory: 1024)); } catch (CanceledException $exception) { $caught = $exception; } From a5cd4171254dae67fa99eb61f08c659bc6be3a81 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Fri, 23 Oct 2026 04:07:03 +0000 Subject: [PATCH 30/33] fix(bus): validate batch outbox ownership before dispatch Reject declared outbox owners that resolve to a different session from the database batch repository before changing counters or publishing jobs. This prevents an independently rolled-back business transaction from removing jobs while leaving committed batch counters behind. Resolve ownership lazily, compare resolved session names rather than requested aliases, and preserve explicit outbox opt-outs and custom batch repositories. Keep validation at Batch::add rather than introducing queue-side batch context. Cover conflicting owners across staged, database and local delivery, same-owner rollback, shared SQLite aliases, and explicit opt-outs without resolving their unused owners. --- src/bus/src/Batch.php | 20 ++++- tests/Bus/BusBatchTest.php | 19 +++-- tests/Queue/Outbox/OutboxBatchTest.php | 106 +++++++++++++++++++++++++ 3 files changed, 137 insertions(+), 8 deletions(-) create mode 100644 tests/Queue/Outbox/OutboxBatchTest.php diff --git a/src/bus/src/Batch.php b/src/bus/src/Batch.php index 0096dd3ed..21f8be2a7 100644 --- a/src/bus/src/Batch.php +++ b/src/bus/src/Batch.php @@ -12,12 +12,14 @@ use Hypervel\Contracts\Events\Dispatcher; use Hypervel\Contracts\Queue\Factory as QueueFactory; use Hypervel\Contracts\Support\Arrayable; +use Hypervel\Database\ConnectionResolverInterface; use Hypervel\Queue\CallQueuedClosure; use Hypervel\Support\Arr; use Hypervel\Support\CarbonImmutable; use Hypervel\Support\Collection; use Hypervel\Support\Enumerable; use JsonSerializable; +use LogicException; use Throwable; class Batch implements Arrayable, JsonSerializable @@ -57,8 +59,10 @@ public function fresh(): ?Batch public function add(array|object $jobs): ?Batch { $count = 0; + $owner = null; + $resolver = null; - $jobs = Collection::wrap($jobs)->map(function ($job) use (&$count) { + $jobs = Collection::wrap($jobs)->map(function ($job) use (&$count, &$owner, &$resolver) { $job = $job instanceof Closure ? CallQueuedClosure::create($job) : $job; if (is_array($job)) { @@ -94,6 +98,20 @@ public function add(array|object $jobs): ?Batch } } + // The counter transaction supplies the implicit SQL owner; declared owners must not bypass it. + if (($database = $job->outboxDatabase ?? null) !== null + && ($job->viaOutbox ?? null) !== false + && $this->repository instanceof DatabaseBatchRepository) { + $owner ??= $this->repository->getConnection()->getNameWithReadWriteType(); + /** @var ConnectionResolverInterface $resolver */ + $resolver ??= Container::getInstance()->make('db'); + $selectedOwner = $resolver->connection($database)->getNameWithReadWriteType(); + + if ($selectedOwner !== $owner) { + throw new LogicException("Outbox database [{$selectedOwner}] does not match the batch database [{$owner}]. Use the batch database or omit the explicit outbox owner."); + } + } + return $job; }); diff --git a/tests/Bus/BusBatchTest.php b/tests/Bus/BusBatchTest.php index 9f2547b34..63a09b335 100644 --- a/tests/Bus/BusBatchTest.php +++ b/tests/Bus/BusBatchTest.php @@ -183,25 +183,30 @@ public function testJobsCanBeAddedToPendingBatch(): void public function testStoredOutboxDefaultsApplyToNewJobsWithoutReplacingExplicitChoices(): void { Queue::fake(); + $database = $this->app->make(DatabaseBatchRepository::class)->getConnection()->getName(); $defaultJob = new class { use Batchable; }; $directJob = (new TestBatchJob)->onOutboxDatabase('previous')->withoutOutbox(); - $otherOwnerJob = (new TestBatchJob)->onOutboxDatabase('other'); + $aliasOwnerJob = (new TestBatchJob)->onOutboxDatabase($database . '::write'); + $propertyOptOutJob = (new TestBatchJob)->onOutboxDatabase('unused'); + $propertyOptOutJob->viaOutbox = false; $nullJob = new TestBatchJob; $nullJob->viaOutbox = null; $nullJob->outboxDatabase = null; - $batch = Bus::batch([$defaultJob, $directJob, $otherOwnerJob, $nullJob]) - ->onOutboxDatabase('billing') + $batch = Bus::batch([$defaultJob, $directJob, $aliasOwnerJob, $propertyOptOutJob, $nullJob]) + ->onOutboxDatabase($database) ->dispatch(); $this->assertTrue($defaultJob->viaOutbox); - $this->assertSame('billing', $defaultJob->outboxDatabase); + $this->assertSame($database, $defaultJob->outboxDatabase); $this->assertFalse($directJob->viaOutbox); $this->assertNull($directJob->outboxDatabase); - $this->assertSame('other', $otherOwnerJob->outboxDatabase); + $this->assertSame($database . '::write', $aliasOwnerJob->outboxDatabase); + $this->assertFalse($propertyOptOutJob->viaOutbox); + $this->assertSame('unused', $propertyOptOutJob->outboxDatabase); $this->assertNull($nullJob->viaOutbox); $this->assertNull($nullJob->outboxDatabase); @@ -209,8 +214,8 @@ public function testStoredOutboxDefaultsApplyToNewJobsWithoutReplacingExplicitCh $batch->fresh()->add([$added]); $this->assertTrue($added->viaOutbox); - $this->assertSame('billing', $added->outboxDatabase); - $this->assertSame(5, $batch->fresh()->totalJobs); + $this->assertSame($database, $added->outboxDatabase); + $this->assertSame(6, $batch->fresh()->totalJobs); } public function testBatchOutboxDefaultsDoNotPropagateToLaterChainJobs(): void diff --git a/tests/Queue/Outbox/OutboxBatchTest.php b/tests/Queue/Outbox/OutboxBatchTest.php new file mode 100644 index 000000000..5ff8bcecd --- /dev/null +++ b/tests/Queue/Outbox/OutboxBatchTest.php @@ -0,0 +1,106 @@ +make('config'); + $config->set('database.default', 'batch'); + $config->set('database.connections.batch', ['driver' => 'sqlite', 'database' => ':memory:']); + $config->set('database.connections.business', ['driver' => 'sqlite', 'database' => ':memory:']); + $config->set('queue.batching.database', 'batch'); + $config->set('queue.outbox.databases', ['batch', 'business']); + $config->set('queue.connections.database', [ + 'driver' => 'database', 'connection' => 'business', 'table' => 'jobs', + 'queue' => 'default', 'after_commit' => false, + ]); + } + + /** + * Install the shipped operational schemas without contacting a broker. + */ + protected function setUpInCoroutine(): void + { + $migrations = dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/'; + Schema::swap(DB::connection('batch')->getSchemaBuilder()); + (require $migrations . '0001_01_01_000005_testbench_create_job_batches_table.php')->up(); + (require $migrations . '0001_01_01_000009_testbench_create_job_outbox_table.php')->up(); + Schema::swap(DB::connection('business')->getSchemaBuilder()); + (require $migrations . '0001_01_01_000006_testbench_create_jobs_table.php')->up(); + (require $migrations . '0001_01_01_000009_testbench_create_job_outbox_table.php')->up(); + $this->app->instance('redis', m::mock(Factory::class)); + } + + #[DataProvider('queueConnections')] + public function testConflictingOwnerCannotChangeAnExistingBatch(string $connection): void + { + $batch = Bus::batch([])->onConnection($connection)->onOutboxDatabase('business')->dispatch(); + DB::connection('business')->beginTransaction(); + + try { + $batch->add([new OutboxBatchJob]); + $this->fail('An outbox owner cannot differ from the batch counter transaction.'); + } catch (LogicException $exception) { + $this->assertStringContainsString('Outbox database [business] does not match the batch database [batch]', $exception->getMessage()); + $this->assertSame(0, $batch->fresh()->totalJobs); + $this->assertSame(0, $batch->fresh()->pendingJobs); + $this->assertSame(0, DB::connection('batch')->table('job_outbox')->count()); + $this->assertSame(0, DB::connection('business')->table('job_outbox')->count()); + $this->assertSame(0, DB::connection('business')->table('jobs')->count()); + } finally { + DB::connection('business')->rollBack(); + } + } + + /** + * Cover staged delivery, the database fast path and local after-commit delivery. + */ + public static function queueConnections(): array + { + return ['staged' => ['redis'], 'database' => ['database'], 'local' => ['sync']]; + } + + public function testSameOwnerRollsBackCountersAndStagedJobsTogether(): void + { + $batch = Bus::batch([])->onConnection('redis')->onOutboxDatabase('batch')->dispatch(); + DB::beginTransaction(); + + try { + $batch->add([new OutboxBatchJob]); + $this->assertSame(1, $batch->fresh()->totalJobs); + $this->assertSame(1, $batch->fresh()->pendingJobs); + $this->assertSame(1, DB::table('job_outbox')->count()); + } finally { + DB::rollBack(); + } + + $this->assertSame(0, $batch->fresh()->totalJobs); + $this->assertSame(0, $batch->fresh()->pendingJobs); + $this->assertSame(0, DB::table('job_outbox')->count()); + } +} + +class OutboxBatchJob +{ + use Batchable; + use Queueable; +} From 0d1cdb9739da6a19860b0420de918a37f9184ce2 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Fri, 23 Oct 2026 04:07:03 +0000 Subject: [PATCH 31/33] fix(queue): preserve immediate local outbox dispatch results Use the local after-commit fallback for outbox selection only when an applicable transaction exists. Without a transaction, synchronous dispatch retains its immediate result and delayed coroutine dispatch retains its timer identifier. Preserve ordinary explicit and configured after-commit behavior, including explicit-owner validation when transactions are active. Regression coverage checks both immediate results and the shared coroutine timer path without resolving outbox storage. --- src/queue/src/SyncQueue.php | 13 +++++++++++- tests/Queue/Outbox/OutboxFastPathTest.php | 26 +++++++++++++++++++++++ 2 files changed, 38 insertions(+), 1 deletion(-) diff --git a/src/queue/src/SyncQueue.php b/src/queue/src/SyncQueue.php index 494ced555..69d4be2d7 100644 --- a/src/queue/src/SyncQueue.php +++ b/src/queue/src/SyncQueue.php @@ -379,7 +379,18 @@ protected function shouldStageViaOutbox(object|string $job): bool */ protected function shouldDispatchAfterCommit(object|string $job): bool { - return $this->shouldDispatchViaOutbox($job) || parent::shouldDispatchAfterCommit($job); + if (parent::shouldDispatchAfterCommit($job)) { + return true; + } + + if (! $this->shouldDispatchViaOutbox($job) || ! $this->container->has('db.transactions')) { + return false; + } + + /** @var DatabaseTransactionsManager $transactions */ + $transactions = $this->container->make('db.transactions'); + + return $transactions->hasApplicableTransaction(); } /** diff --git a/tests/Queue/Outbox/OutboxFastPathTest.php b/tests/Queue/Outbox/OutboxFastPathTest.php index 45f871100..4b43964c9 100644 --- a/tests/Queue/Outbox/OutboxFastPathTest.php +++ b/tests/Queue/Outbox/OutboxFastPathTest.php @@ -14,6 +14,8 @@ use Hypervel\Contracts\Queue\ShouldBeUnique; use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; use Hypervel\Contracts\Redis\Factory; +use Hypervel\Coordinator\Timer; +use Hypervel\Queue\BackgroundQueue; use Hypervel\Queue\Events\JobQueueing; use Hypervel\Queue\FailoverQueue; use Hypervel\Queue\Outbox\OutboxDispatcher; @@ -47,6 +49,30 @@ protected function defineEnvironment(Application $app): void ]); } + public function testLocalSelectionWithoutATransactionPreservesImmediateResults(): void + { + $queue = $this->app->make('queue')->connection('sync'); + + $this->assertSame(0, $queue->push(new OutboxLocalJob)); + $this->assertSame(0, $queue->push((new OutboxLocalJob)->viaOutbox())); + $this->assertSame(0, $queue->later(0, new OutboxLocalJob)); + $this->assertSame(0, $queue->later(0, (new OutboxLocalJob)->viaOutbox())); + $this->assertSame(4, $this->app->make(OutboxExecutionLog::class)->executions); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + } + + public function testLocalSelectionWithoutATransactionPreservesTimerIds(): void + { + $timer = m::mock(Timer::class); + $timer->shouldReceive('after')->with(60.0, m::type('callable'))->twice()->andReturn(41, 42); + $queue = new BackgroundQueue(timer: $timer); + $queue->setContainer($this->app)->setConnectionName('local'); + + $this->assertSame(41, $queue->later(60, new OutboxLocalJob)); + $this->assertSame(42, $queue->later(60, (new OutboxLocalJob)->viaOutbox())); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + } + #[DataProvider('transactionOutcomes')] public function testLocalSelectionHonorsCommitAndRollbackWithoutSchema(bool $commit): void { From 81657adfa43197b81f89068c40829120ff91d0f4 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Fri, 23 Oct 2026 04:07:04 +0000 Subject: [PATCH 32/33] docs(queue): clarify batch outbox defaults and ownership Clarify that batch outbox defaults apply to each chain head, while successors retain their own selection and connection defaults. Align the active implementation plan with batch-owner preflight and local dispatch return-value preservation. Replace the completed PR-opening instruction with the remaining local-commit and owner-direction step. --- ...-10-22-1522-hypervel-transactional-queue-outbox-codex.md | 6 +++++- src/docs/queue-outbox.md | 2 +- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md index 6f5a3b1a3..8f2f14d51 100644 --- a/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md +++ b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md @@ -26,7 +26,7 @@ Implemented: framework prerequisites, public selection and wrapper propagation, Implementation, live-engine/concurrency and API-integration coverage, canonical documentation, the retained benchmark harness and performance measurements are complete. Measurements support the existing production design; see [benchmark results](../../tests/Benchmarks/QueueOutbox/results.md). -When opening the PR, include a performance section with tables from the benchmark results and their workload, concurrency and pool settings. +Complete local review-fix verification and commits, then wait for owner direction before pushing or replying to PR comments. ### Hypervel @@ -182,6 +182,8 @@ Failover and outbox solve different policies, and failover may cross heterogeneo Driver switching never causes an exception merely because the selected driver is local, failover, or already durable. There are no environment checks; behavior follows runtime capability. Local/failover drivers register against the explicitly selected active transaction when supplied and reject only a false claim of atomic ownership (an explicit owner while transactions exist only on other connections); without an explicit owner they preserve their existing transaction-selection rules. Failover's explicit-owner callback enters the child-attempt loop directly with after-commit already satisfied; matching children must not revalidate the committed owner or defer onto an unrelated transaction. Ordinary all-transactions callbacks retain reentry for transactions opened by earlier callbacks. +Outbox selection alone enters the local after-commit path only when an applicable transaction exists. Without one, preserve immediate sync return values and coroutine delay timer IDs. Existing explicit or configured `afterCommit` behavior remains unchanged. + A selected outbox keeps an SQS FIFO job staged until `deliver_at`, then publishes it without the forbidden per-message `DelaySeconds`; any queue-level delay applies after publication. In `later()` and `bulk()`, select/partition outbox work before applying the existing FIFO checks to direct work. Capture `getQueueableOptions()` without a delay argument at staging; preserve its direct-call validation. No-transaction staging has the same delayed-publication behavior. A custom durable driver participates when it routes `push()` / `later()` / `bulk()` through the base enqueue hooks. The relay uses the optional raw-batch publication contract below when available and otherwise publishes due rows sequentially through the existing `pushRaw()` contract. Existing custom queues that override the public enqueue methods and bypass the base lifecycle retain their existing direct behavior; intercepting arbitrary overrides would require a permanent decorator and hot-path overhead for every queue operation. Document this extension contract instead of adding that machinery. @@ -356,6 +358,8 @@ Keep selection and staging in `Concerns\StagesOutboxJobs`. Reuse Redis/database' Every implementation preserves order within each partition and groups staged rows by owning database. This prevents a later configuration error from partially staging an otherwise valid group. There is deliberately no ordering or atomicity guarantee between direct and outbox partitions, just as existing cross-connection bulk has none; callers requiring one atomic unit use one outbox owner. Finalize every owner group before its first insert, emit existing per-job queue lifecycle events, and use set-based storage. Each owner group is atomic across its insert chunks; distinct owner databases necessarily commit independently. A `PendingBatch` repository transaction provides one owner and rejects conflicting per-job choices. Preserve current `Bus::bulk()` semantics: it does not suddenly acquire unique/debounce locks or run `PreparesForDispatch`. +For a `DatabaseBatchRepository`, `Batch::add()` validates declared non-null outbox owners after applying defaults and before its transaction or queue call. Compare resolved session identities with the repository's connection, not requested aliases; skip explicitly false `viaOutbox` choices. This covers staged, database and local targets without queue-side state. Resolve the repository owner lazily once per add; leave custom batch repositories unchanged. + ### Raw-batch publication SPI `PublishesOutboxMessages` is a narrow optional capability in `Hypervel\Contracts\Queue`, beside `ClearableQueue` and `IndexAwareQueue`, not a new application API. Its signature may reference the queue-owned `OutboxMessage` and `OutboxBatchResult` value types under the repository's lazy contract-signature dependency convention. diff --git a/src/docs/queue-outbox.md b/src/docs/queue-outbox.md index db7bd3cb9..7d8d9e657 100644 --- a/src/docs/queue-outbox.md +++ b/src/docs/queue-outbox.md @@ -138,7 +138,7 @@ $batch->add($moreJobs); Batch defaults apply to newly added jobs too, without replacing an explicit choice on an individual job. Batch counters and outbox rows participate in the batch repository's transaction; conflicting outbox ownership is rejected. Jobs using only `Batchable`, rather than `Queueable`, can receive batch defaults. -Chains retain their normal behavior. Selecting the outbox for the first job does not automatically select it for later jobs. Each later job is dispatched after its predecessor succeeds and uses its own selection or its queue connection's default. +Chains retain their normal behavior. Selecting the outbox for the first job does not automatically select it for later jobs. Batch outbox defaults likewise apply only to the first job of each chain. Each later job is dispatched after its predecessor succeeds and uses its own selection or its queue connection's default. ### Queued Listeners, Mail, Notifications and Broadcasts From 2f11e2ec627ad9d17ea5e6b0b76ac64a590b9a38 Mon Sep 17 00:00:00 2001 From: Raj Siva-Rajah <5361908+binaryfire@users.noreply.github.com> Date: Fri, 23 Oct 2026 04:26:19 +0000 Subject: [PATCH 33/33] docs(queue): align outbox guidance with Laravel documentation style Explain outbox setup and delivery through application-facing language and examples rather than implementation terminology. Give delayed jobs, configuration changes, monitoring, retries and deletion their own sections while preserving the delivery and custom-driver contracts. Clarify required database registration, batch storage, local-driver transaction behavior, retry eligibility and telemetry interpretation. Match the surrounding prose in the queue, event, mail, notification, broadcast and porting guides, and keep their detailed instructions linked to the canonical outbox guide. Validate documentation anchors, cross-links and code fences. Update the active plan's remaining step to PR review and CI verification. No runtime behavior changes. --- ...pervel-transactional-queue-outbox-codex.md | 2 +- src/docs/broadcasting.md | 4 +- src/docs/events.md | 4 +- src/docs/mail.md | 4 +- src/docs/notifications.md | 4 +- src/docs/opentelemetry.md | 10 +- src/docs/porting-from-laravel.md | 4 +- src/docs/queue-outbox.md | 181 +++++++++++------- src/docs/queues.md | 4 +- 9 files changed, 130 insertions(+), 87 deletions(-) diff --git a/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md index 8f2f14d51..52534ea7e 100644 --- a/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md +++ b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md @@ -26,7 +26,7 @@ Implemented: framework prerequisites, public selection and wrapper propagation, Implementation, live-engine/concurrency and API-integration coverage, canonical documentation, the retained benchmark harness and performance measurements are complete. Measurements support the existing production design; see [benchmark results](../../tests/Benchmarks/QueueOutbox/results.md). -Complete local review-fix verification and commits, then wait for owner direction before pushing or replying to PR comments. +Complete the documentation style review, then resume PR review and CI verification on both remotes before merging. ### Hypervel diff --git a/src/docs/broadcasting.md b/src/docs/broadcasting.md index f23c7bf28..01a430d1e 100644 --- a/src/docs/broadcasting.md +++ b/src/docs/broadcasting.md @@ -1001,7 +1001,7 @@ class ServerCreated implements ShouldBroadcast, ShouldDispatchAfterCommit > [!NOTE] > To learn more about working around these issues, please review the documentation regarding [queued jobs and database transactions](/docs/{{version}}/queues#jobs-and-database-transactions). -To store the broadcast job atomically with the business write, select the [transactional outbox](/docs/{{version}}/queue-outbox) on the broadcast event: +If you would like to store a broadcast job as part of your database transaction, you may implement the `ShouldQueueViaOutbox` interface on the broadcast event. The [transactional outbox](/docs/{{version}}/queue-outbox) will then deliver the job to your queue after the transaction commits: ```php use Hypervel\Contracts\Broadcasting\ShouldBroadcast; @@ -1013,7 +1013,7 @@ class ServerCreated implements ShouldBroadcast, ShouldQueueViaOutbox } ``` -Dispatch this event inside the transaction. Do not also defer the whole event with `ShouldDispatchAfterCommit`: that would postpone outbox staging until after commit. See the [outbox guide](/docs/{{version}}/queue-outbox#queued-producers) for setup and guarantees. +Dispatch the event inside the transaction without implementing `ShouldDispatchAfterCommit`. That interface would postpone the entire event until after commit, preventing the broadcast job from being stored in the transaction. Before using this feature, follow the [outbox setup instructions](/docs/{{version}}/queue-outbox#setup). ## Authorizing Channels diff --git a/src/docs/events.md b/src/docs/events.md index 0f5664543..583273303 100644 --- a/src/docs/events.md +++ b/src/docs/events.md @@ -521,7 +521,7 @@ class SendShipmentNotification implements ShouldQueueAfterCommit > [!NOTE] > To learn more about working around these issues, please review the documentation regarding [queued jobs and database transactions](/docs/{{version}}/queues#jobs-and-database-transactions). -To store a queued listener's job atomically with the business write, implement `ShouldQueueViaOutbox` on the listener: +If you would like to store a queued listener's job as part of your database transaction, you may implement the `ShouldQueueViaOutbox` interface on the listener. The [transactional outbox](/docs/{{version}}/queue-outbox) will then deliver the job to your queue after the transaction commits: ```php use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; @@ -532,7 +532,7 @@ class SendShipmentNotification implements ShouldQueueViaOutbox } ``` -Listener defaults are inspected without constructing the listener; use the marker rather than calling `viaOutbox()` in its constructor. Dispatch the event inside the transaction, without deferring the whole event through `ShouldDispatchAfterCommit`. See the [transactional outbox guide](/docs/{{version}}/queue-outbox#queued-producers) for setup and delivery guarantees. +Dispatch the event inside the transaction, and do not implement `ShouldDispatchAfterCommit` on the event itself. That interface would postpone the entire event until after commit, preventing the listener's job from being stored in the transaction. Since Hypervel reads listener settings without constructing the listener, use `ShouldQueueViaOutbox` rather than calling `viaOutbox` in its constructor. Before using this feature, follow the [outbox setup instructions](/docs/{{version}}/queue-outbox#setup). ### Queued Listener Middleware diff --git a/src/docs/mail.md b/src/docs/mail.md index ace6f928b..85c81f665 100644 --- a/src/docs/mail.md +++ b/src/docs/mail.md @@ -1240,7 +1240,7 @@ class OrderShipped extends Mailable implements ShouldQueue > [!NOTE] > To learn more about working around these issues, please review the documentation regarding [queued jobs and database transactions](/docs/{{version}}/queues#jobs-and-database-transactions). -For transactional delivery, a mailable using `Queueable` may select the [outbox](/docs/{{version}}/queue-outbox): +If you would like to store a queued mailable as part of your database transaction, you may call the `viaOutbox` method provided by its `Queueable` trait: ```php Mail::to($request->user())->queue( @@ -1248,7 +1248,7 @@ Mail::to($request->user())->queue( ); ``` -The existing queued-mail wrapper carries this choice to the queue. See the [outbox guide](/docs/{{version}}/queue-outbox#queued-producers) for the required table, relay and delivery guarantees. +Queue the mailable inside the transaction. Hypervel will store its job in the [transactional outbox](/docs/{{version}}/queue-outbox) and deliver it to your queue after the transaction commits. Before using this feature, follow the [outbox setup instructions](/docs/{{version}}/queue-outbox#setup). #### Queued Email Failures diff --git a/src/docs/notifications.md b/src/docs/notifications.md index 6e8b998d7..b4de840af 100644 --- a/src/docs/notifications.md +++ b/src/docs/notifications.md @@ -435,13 +435,13 @@ class InvoicePaid extends Notification implements ShouldQueue > [!NOTE] > To learn more about working around these issues, please review the documentation regarding [queued jobs and database transactions](/docs/{{version}}/queues#jobs-and-database-transactions). -For transactional delivery, a queued notification using `Queueable` may select the [outbox](/docs/{{version}}/queue-outbox): +If you would like to store a queued notification's jobs as part of your database transaction, you may call the `viaOutbox` method provided by its `Queueable` trait: ```php $user->notify((new InvoicePaid($invoice))->viaOutbox()); ``` -Each queued channel job carries the notification's choice. `viaOutbox()` is a fluent setter, not a hook returning a boolean or channel map. See the [outbox guide](/docs/{{version}}/queue-outbox#queued-producers) for setup and delivery guarantees. +Send the notification inside the transaction. Hypervel will store each queued channel's job in the [transactional outbox](/docs/{{version}}/queue-outbox) and deliver it after commit. Unlike `viaQueues`, `viaOutbox` configures the notification when called; you do not implement it to return a setting. Before using this feature, follow the [outbox setup instructions](/docs/{{version}}/queue-outbox#setup). #### Determining if a Queued Notification Should Be Sent diff --git a/src/docs/opentelemetry.md b/src/docs/opentelemetry.md index 081da00a9..f416fd52d 100644 --- a/src/docs/opentelemetry.md +++ b/src/docs/opentelemetry.md @@ -321,13 +321,13 @@ Jobs dispatched after a database commit are finalized and timed only when the qu Terminal queue correlation normally uses the exact final encoded payload without decoding it. If another finalizer rewrites that payload afterward, the package performs one terminal decode to recover Hypervel's job UUID. Payloads from custom drivers or `Queue::createPayloadUsing()` callbacks that do not contain a string UUID are accepted, omit `messaging.message.id`, and use exact-payload correlation only. If a later finalizer rewrites such a payload, its producer span remains started but is never ended or exported, and its state remains until coroutine or process exit. A later finalizer that removes or corrupts an existing UUID likewise leaves unmatched state. A long-running non-coroutine custom process that catches and continues after such failures may retain that state until it exits. -With the [transactional outbox](/docs/{{version}}/queue-outbox), producer telemetry finishes when staging accepts the job. The relay preserves its propagation fields and does not emit a second producer lifecycle. Three enabled-by-default metrics observe delivery at connection-batch boundaries: +When using the [transactional outbox](/docs/{{version}}/queue-outbox), the producer span ends when the job is stored in the outbox. The relay preserves the job's trace context without creating another producer span. The following metrics are enabled by default and record delivery separately for each queue connection's batch: -- `hypervel.queue.outbox.delivered.messages` counts queue-confirmed messages, including accepted messages whose SQL settlement subsequently fails. -- `hypervel.queue.outbox.delivery.duration` records publication duration in seconds, with `error.type` on publication or settlement failure. Its failed observation count supplies the number of failed batches without a separate counter. -- `hypervel.queue.outbox.delivery.latency` records the oldest attempted message's eligible age at publication start, in seconds. It is one observation per connection batch, not a per-message latency distribution, and excludes deferred rows. +- `hypervel.queue.outbox.delivered.messages` counts messages accepted by the queue, including messages whose outbox records could not be deleted afterwards. +- `hypervel.queue.outbox.delivery.duration` measures how long sending the batch takes, in seconds. If sending or updating the outbox fails, the observation includes `error.type`. Count the duration observations that include `error.type` to determine the number of failed batches. +- `hypervel.queue.outbox.delivery.latency` records how long the oldest attempted message has been ready for delivery, in seconds, measured when sending begins. It records one value per connection batch, not one per message. Messages deferred until a future delivery time are excluded. -These metrics use the configured queue connection and outbox database as attributes. Latency does not add exception types as attributes. They perform no backlog queries or payload decoding and add no work to ordinary dispatch. Use `queue:outbox-monitor` for backlog counts and age alerts. +These metrics include the configured queue connection and outbox database as attributes. Only the duration metric includes exception types. Collecting them does not query pending outbox records or decode job payloads, and does not add work to ordinary dispatches. To monitor pending counts and waiting times, use the [`queue:outbox-monitor` command](/docs/{{version}}/queue-outbox#monitoring-the-outbox). Queue depth is off by default because it performs backend I/O during metric collection. Enable it with explicit targets: diff --git a/src/docs/porting-from-laravel.md b/src/docs/porting-from-laravel.md index c8958e163..18748f174 100644 --- a/src/docs/porting-from-laravel.md +++ b/src/docs/porting-from-laravel.md @@ -803,7 +803,7 @@ Framework operations release idle database sessions at the boundaries listed in Do not manually commit or roll back the transaction owned by a `DB::transaction()` closure; Hypervel throws when that closure returns. Let `transaction()` manage completion, or use the [manual transaction API](/docs/{{version}}/database#manually-using-transactions) separately. Balanced nested transactions remain supported. -Extensions inspecting transaction records or registering callbacks directly with `DatabaseTransactionsManager` must use the resolved connection's `getNameWithReadWriteType()`, not its base `getName()`. Base, `::read` and `::write` sessions can own independent transactions. Resolve the connection first: shared in-memory SQLite aliases return one owner and therefore one transaction identity. Ordinary `afterCommit()` and `afterRollBack()` calls already select the correct owner. +If your extension inspects transaction records or registers callbacks directly with `DatabaseTransactionsManager`, use the resolved connection's `getNameWithReadWriteType()` instead of `getName()`. Base, `::read`, and `::write` connections can have independent transactions. Always resolve the connection before reading its name; aliases for a shared in-memory SQLite connection resolve to the same session. Ordinary `afterCommit()` and `afterRollBack()` calls already use the correct connection. When a package constructs `DatabaseStore`, `DatabaseSessionHandler`, `DatabaseQueue`, or `DatabaseBatchRepository` directly, pass the database connection resolver and configured connection name instead of retaining a resolved connection. Framework-configured drivers already use this form. @@ -861,7 +861,7 @@ Hypervel stores job batches in a relational database. Laravel's DynamoDB batch r With `Worker::$killOnTimeout = false`, a timeout interrupts job code with `Swoole\Coroutine\CanceledException`; catching `TimeoutExceededException` inside the job does not catch it. Use `finally` for cleanup and rethrow cancellation if caught. Failure callbacks and worker reporting still receive `TimeoutExceededException`. See [worker timeouts](/docs/{{version}}/queues#worker-timeouts). -Custom Horizon `RedisQueue` subclasses must not rely on `push()` calling their `pushRaw()` override. Use the protected `preparePayloadForPush()` and `handlePayloadPushed()` hooks for normal publication, with post-push work running after storage is accepted. Direct raw publication remains available through `pushRaw()`. +If you extend Horizon's `RedisQueue`, do not rely on `push()` calling your `pushRaw()` override. Override the protected `preparePayloadForPush()` method to prepare normal pushes, or `handlePayloadPushed()` to run code after Redis accepts the job. Direct calls to `pushRaw()` remain supported. When a Laravel package offers optional support for an unsupported driver, remove that integration from the Hypervel port unless the package can safely provide it through a separate optional dependency. diff --git a/src/docs/queue-outbox.md b/src/docs/queue-outbox.md index 7d8d9e657..0ed15f4f1 100644 --- a/src/docs/queue-outbox.md +++ b/src/docs/queue-outbox.md @@ -5,22 +5,29 @@ - [Dispatching Jobs](#dispatching-jobs) - [Selecting the Outbox](#selecting-the-outbox) - [Bulk Dispatch and Batches](#bulk-dispatch-and-batches) - - [Queued Listeners, Mail, Notifications and Broadcasts](#queued-producers) + - [Queued Listeners, Mail, Notifications, and Broadcasts](#queued-producers) - [Database Transactions](#database-transactions) - [Queue Connections and Delays](#queue-connections-and-delays) + - [Delayed Jobs](#delayed-jobs) + - [Changing Queue Configuration](#changing-queue-configuration) - [Running the Relay](#running-the-relay) - [Concurrency and Shutdown](#concurrency-and-shutdown) - [Failures and Recovery](#failures-and-recovery) - [Inspecting and Managing Messages](#managing-messages) + - [Monitoring the Outbox](#monitoring-the-outbox) + - [Retrying Messages](#retrying-messages) + - [Deleting Messages](#deleting-messages) - [Delivery Guarantees](#delivery-guarantees) - [Custom Queue Drivers](#custom-queue-drivers) ## Introduction -Dispatching a job [after a database transaction commits](/docs/{{version}}/queues#jobs-and-database-transactions) prevents a worker from reading uncommitted data. However, if the application stops between committing the transaction and publishing the job, the database change can remain without its intended job. +When you dispatch a job [after a database transaction commits](/docs/{{version}}/queues#jobs-and-database-transactions), queue workers will not process it before the transaction completes. However, your application could stop after committing the transaction but before sending the job to the queue. In that case, your database changes would be saved, but the job would be lost. -The transactional outbox closes this gap. Hypervel stores the job in a database table as part of the same transaction, then a separate relay process publishes it to your queue. If the transaction rolls back, its outbox rows roll back too. If the relay stops or the queue is unavailable, committed rows remain available for recovery. +Hypervel's transactional outbox allows you to store a job as part of the same transaction as your database changes. A separate process, called the relay, sends the stored job to your queue after the transaction commits. If the transaction rolls back, the job is discarded as well. If the queue is unavailable, the job remains in the outbox until delivery succeeds. + +For example, you may call `viaOutbox` when dispatching a job inside a transaction: ```php use App\Jobs\SendOrderReceipt; @@ -34,12 +41,12 @@ DB::transaction(function () use ($attributes) { }); ``` -The outbox is an opt-in feature of the queue package. It uses your existing jobs, queue connections and workers, and does not require Horizon. Jobs which do not select it continue through their normal dispatch path without outbox database access. +The outbox works with your existing jobs, queue connections, and workers. It does not require Horizon. Jobs that do not use the outbox are dispatched normally without accessing the outbox table. ## Setup -The outbox settings live in your application's `config/queue.php` file: +You may configure the outbox in your application's `config/queue.php` configuration file: ```php 'outbox' => [ @@ -51,14 +58,14 @@ The outbox settings live in your application's `config/queue.php` file: ], ``` -The `databases` array lists the configured database connections whose outbox tables the relay will drain. Install the outbox table on each listed database. New applications include its migration; if your application does not, generate it using: +The `databases` option contains the names of the database connections the relay should read from. Each database needs an outbox table. New Hypervel applications include a migration for this table. If your application does not have this migration, you may create it using the `make:queue-outbox-table` Artisan command: ```shell php artisan make:queue-outbox-table php artisan migrate ``` -The generator uses the configured table name. `queue:outbox-table` is an alias for the same command. When using several databases, apply the migration on each appropriate connection using your normal migration deployment process. +The generated migration uses the table name from your outbox configuration. You may also use the `queue:outbox-table` alias to generate it. If you use several outbox databases, run the migration on each connection. Finally, run the relay alongside your normal queue workers: @@ -67,7 +74,7 @@ php artisan queue:relay php artisan queue:work redis ``` -The relay publishes jobs; it does not replace the workers that process them. Horizon may continue managing your Redis workers as usual. +The relay sends jobs to the queue; your queue workers still process them. If you use Horizon, it may continue managing your Redis workers as usual. ## Dispatching Jobs @@ -75,13 +82,13 @@ The relay publishes jobs; it does not replace the workers that process them. Hor ### Selecting the Outbox -Call `viaOutbox` when dispatching a job: +To dispatch a job through the outbox, you may chain the `viaOutbox` method onto your dispatch operation: ```php SendOrderReceipt::dispatch($order)->viaOutbox(); ``` -To select the outbox for a job class by default, implement `ShouldQueueViaOutbox` instead of `ShouldQueue`. The marker extends `ShouldQueue`, so you do not need both: +If a job should always use the outbox, you may implement the `ShouldQueueViaOutbox` interface on its class. This interface extends `ShouldQueue`, so you do not need to implement both: ```php use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; @@ -95,7 +102,7 @@ class SendOrderReceipt implements ShouldQueueViaOutbox } ``` -Alternatively, set `outbox` to `true` on a named queue connection in `config/queue.php`. Existing connection configurations which omit this option default to `false`. +Alternatively, you may set the `outbox` option to `true` on a queue connection in your `config/queue.php` file. This enables the outbox for jobs dispatched to that connection. When this option is omitted, it defaults to `false`. You may override either default for a specific dispatch: @@ -103,16 +110,18 @@ You may override either default for a specific dispatch: SendOrderReceipt::dispatch($order)->withoutOutbox(); ``` -An explicit job choice takes precedence over the marker, followed by the queue connection's default. `afterCommit` and `beforeCommit` still control direct dispatch timing; neither disables the outbox. A selected outbox job is stored inside its owning transaction and cannot be relayed before that transaction commits. +Calling `viaOutbox` or `withoutOutbox` overrides both the job's interface and the connection's configuration. Otherwise, a job uses the outbox if it implements `ShouldQueueViaOutbox` or its queue connection enables the `outbox` option. + +The `afterCommit` and `beforeCommit` methods do not disable the outbox. When a job uses the outbox, it is stored within the database transaction and cannot be delivered until that transaction commits. -The fluent methods come from `Queueable` and `Batchable`. Do not redeclare their outbox properties with defaults; PHP rejects the conflicting trait definitions. Use the marker for a class default or fluent methods for individual objects. Classes without these trait-provided properties may instead declare a public `viaOutbox` default of `true` or `false`. `viaOutbox()` is a setter, not a getter hook like `viaConnection()` or `viaQueues()`. +The `Queueable` and `Batchable` traits provide the outbox methods and properties. If your class uses either trait, use `ShouldQueueViaOutbox` for its default behavior rather than redeclaring the trait's properties with default values, which PHP does not allow. Classes without these properties may declare a public `viaOutbox` property with a default of `true` or `false`. Unlike `viaConnection` or `viaQueues`, `viaOutbox` configures the object when called; it is not a method that you implement to return a setting. -A staged queue insertion returns the finalized job UUID instead of a broker identifier. The operator commands use this same UUID. Direct dispatch return values are unchanged. +If you push a job using a queue method such as `Queue::push` and the job is stored in the outbox, the method returns the job's UUID instead of a queue-assigned ID. You may use this UUID with the [outbox commands](#managing-messages). Dispatches that bypass the outbox retain their usual return values. ### Bulk Dispatch and Batches -Use the existing bulk API to stage several jobs efficiently: +To store several jobs in the outbox, you may use the `bulk` method on the `Bus` facade: ```php use Hypervel\Support\Facades\Bus; @@ -123,7 +132,7 @@ Bus::bulk([ ]); ``` -Within each queue bulk call, jobs sharing an outbox database are inserted together, using additional insert chunks only when the database's parameter limits require them. Each database group is atomic across those chunks. `Bus::bulk` keeps its existing grouping by queue connection and destination. A bulk containing direct jobs or several database owners is not one atomic unit, and publication order between its groups is not guaranteed. +Hypervel groups bulk jobs by queue connection and queue name. Within each group, jobs using the same outbox database are inserted together. Large inserts are split to respect the database's parameter limit, but all inserts for that database group succeed or roll back together. Jobs sent directly to a queue or stored in different database groups do not share this guarantee, and may be delivered in a different order. You may also select the outbox for a batch: @@ -136,14 +145,14 @@ $batch = Bus::batch($jobs) $batch->add($moreJobs); ``` -Batch defaults apply to newly added jobs too, without replacing an explicit choice on an individual job. Batch counters and outbox rows participate in the batch repository's transaction; conflicting outbox ownership is rejected. Jobs using only `Batchable`, rather than `Queueable`, can receive batch defaults. +The batch's outbox settings also apply to jobs added later, unless a job has explicitly configured its own settings. With database batch storage, outbox jobs are stored in your batch database in the same transaction as the batch counters. Include this database in `queue.outbox.databases`. If a job selects a different outbox database, Hypervel throws an exception before updating the batch. Jobs only need the `Batchable` trait to receive these settings; `Queueable` is not required. Chains retain their normal behavior. Selecting the outbox for the first job does not automatically select it for later jobs. Batch outbox defaults likewise apply only to the first job of each chain. Each later job is dispatched after its predecessor succeeds and uses its own selection or its queue connection's default. -### Queued Listeners, Mail, Notifications and Broadcasts +### Queued Listeners, Mail, Notifications, and Broadcasts -Queued listeners, mailables, notifications and broadcast events propagate their outbox choice to their existing wrapper jobs. Implement `ShouldQueueViaOutbox` on the producer, or use its `Queueable` methods when constructing it: +You may also use the outbox with queued listeners, mailables, notifications, and broadcast events. Implement `ShouldQueueViaOutbox` on the class, or call `viaOutbox` if it uses the `Queueable` trait. For example, you may queue a mailable through the outbox: ```php use App\Mail\OrderReceipt; @@ -154,65 +163,82 @@ Mail::to($customer)->queue( ); ``` -For queued listeners, use the marker or, when the class has no trait-provided outbox properties, a public property default. The dispatcher reads these defaults without constructing the listener, so do not rely on its constructor calling `viaOutbox`. When a listener uses `Queueable` or `Batchable`, use the marker instead of redeclaring the trait properties. +For queued listeners, implement `ShouldQueueViaOutbox` instead of calling `viaOutbox` in the constructor. Hypervel reads the listener's settings without constructing it. A listener that does not use the outbox properties from `Queueable` or `Batchable` may also define a public property default, as described [above](#selecting-the-outbox). -Dispatch the event while the business transaction is open. An event implementing `ShouldDispatchAfterCommit` postpones the entire event, including listener and broadcast queueing, until after commit. Adding outbox selection cannot make that later staging atomic with the earlier transaction. Select the outbox on the queued listener or broadcast event instead of deferring the whole event when you need transactional delivery. +Dispatch events inside your database transaction so their queued listeners and broadcasts can be stored with your database changes. + +> [!WARNING] +> An event implementing `ShouldDispatchAfterCommit` is not dispatched until the transaction commits. Its listener and broadcast jobs therefore cannot be stored as part of that transaction. To use the outbox for these jobs, implement `ShouldQueueViaOutbox` on the listener or broadcast event instead of deferring the entire event. ## Database Transactions -When one database connection has an active transaction, Hypervel stores the outbox row on that connection. Without an active transaction, it uses the first configured outbox database. You may select an owner explicitly: +For queues that store jobs in the outbox, Hypervel uses the connection with the active transaction. Without an active transaction, it uses the first database in your outbox configuration. Whichever database stores the job must be listed in `queue.outbox.databases`. You may select a database using the `onOutboxDatabase` method: ```php SendOrderReceipt::dispatch($order) ->onOutboxDatabase('billing'); ``` -`onOutboxDatabase` also selects the outbox. The database must be included in `queue.outbox.databases`. If transactions are open on several connections, an explicit owner is required. Hypervel reports the active connections rather than guessing which business write belongs to the job. +Calling `onOutboxDatabase` also enables the outbox for the job. If transactions are open on several connections, you must select one explicitly. Otherwise, Hypervel throws an exception listing the active connections. -An explicitly selected owner must have an active transaction when other connections have active transactions. Only writes on that selected transaction are atomic with the job. Nested transactions on one connection remain one owner, and rolling back a savepoint also removes its outbox rows and releases their dispatch-owned locks. +If any database transactions are open, the selected connection must have an active transaction. Only changes made in that transaction are committed or rolled back with the job. Nested transactions on the same connection are supported; rolling back a savepoint also removes its outbox jobs and releases the unique and debounce locks acquired by those dispatches. -Ownership follows the resolved database session. For example, a `mysql::write` transaction is distinct from a `mysql` transaction when they use separate sessions. Select the exact owner when necessary, but configure only `mysql` as the relay source: do not list both names for the same outbox table. Read-role connections cannot own SQL outbox storage; use their base or write connection. +Use the connection that owns your transaction. For example, `mysql` and `mysql::write` can use separate database sessions with independent transactions. You may select `mysql::write` with `onOutboxDatabase`, but list only `mysql` in `queue.outbox.databases` so the relay reads the table once. Read connections such as `mysql::read` cannot store outbox jobs; use the base or write connection instead. -Calling `withoutOutbox` clears an explicit database choice. Calling `viaOutbox` afterwards returns to normal owner selection. +Calling `withoutOutbox` clears an explicit database choice. Calling `viaOutbox` afterwards lets Hypervel select the database using the rules above. > [!WARNING] -> The outbox cannot make writes to different databases atomic with one another. It also cannot include an HTTP request, payment, email or filesystem write in the database transaction. Perform those effects in the queued job with suitable idempotency handling. +> The outbox cannot combine changes to different databases into one transaction. It also cannot include an HTTP request, payment, email, or filesystem write in your database transaction. Perform these operations in the queued job, taking care to [handle duplicate delivery](#delivery-guarantees). ## Queue Connections and Delays -Selecting the outbox does not require environment-specific job code when you switch queue drivers: +You may switch queue drivers without removing `viaOutbox` calls from your jobs. Hypervel handles each driver as follows: | Queue connection | Behavior when the outbox is selected | | --- | --- | | Redis, SQS, Beanstalkd | Store the job in the outbox, then relay it to the queue. | -| Database, using the active owning connection | Insert directly into the jobs table inside that transaction; a second durable table adds no guarantee. | +| Database, using the active transaction's connection | Insert directly into the jobs table as part of that transaction. | | Database, with no active transaction | Insert directly into the durable jobs table. | | Database, on another connection during a business transaction | Store the job on the business transaction's outbox, then relay it to the database queue. | -| Sync, background, deferred | Preserve local execution behavior after the selected transaction commits. These drivers do not gain crash durability. | -| Failover | Defer the complete failover attempt until the applicable transactions commit. Its normal child-driver attempts remain non-durable across a process crash. | +| Sync, background, deferred | Wait for the current transaction to commit, or the transaction selected with `onOutboxDatabase`, then use the driver's normal execution behavior. Without a transaction, behave normally. Jobs are not stored for recovery after a process stops. | +| Failover | Wait for the applicable transactions to commit, then attempt normal failover delivery. The attempt is not stored for recovery after a process stops. | | Null | Discard the job normally without creating an outbox row. | | Queue or Bus fake | Record the job only in the fake, without accessing an outbox table. | -`dispatchSync` and after-response dispatch retain their existing local or deferred behavior. They are not substitutes for durable queued dispatch. Direct `pushRaw` calls also bypass outbox staging, even on a connection whose default enables it. +The `dispatchSync` and after-response dispatch methods retain their existing behavior; they do not store jobs in the outbox. Calling `pushRaw` also bypasses the outbox, even when the connection's configuration enables it. + + +### Delayed Jobs + +You may use the usual `delay` method when dispatching through the outbox: + +```php +SendOrderReceipt::dispatch($order)->viaOutbox()->delay(now()->plus(minutes: 10)); +``` + +Retrying delivery does not restart the job's delay. Redis and database queues can receive delayed jobs immediately and hold them until they are due. With standard SQS queues, the relay waits until the job is within SQS's per-message delay limit. With SQS FIFO queues, it waits until the job is due before sending it, since FIFO queues do not support per-message delays. -Jobs keep their intended delivery time when the relay runs late or retries publication. Redis and database queues can receive delayed jobs immediately and hold them until due. Standard SQS jobs wait in the outbox until they fit its per-message delay window. SQS FIFO jobs wait until due before publication, without a per-message timer; the queue's own configured delay still applies. On standard SQS, omitting a job delay preserves the queue-level default, while an expired explicitly requested delay is sent as zero. +Any delay configured on an SQS FIFO queue still applies after the relay sends the job. For standard SQS queues, omitting a job delay preserves the queue's default delay. An explicitly requested job delay overrides that default, including when the job is already due. -The outbox captures the selected queue connection and logical queue name, including its default, at dispatch. Queue-name forwarding is applied once when publication occurs, using the current forwarding rules. Changing a connection's default queue does not reroute existing rows. + +### Changing Queue Configuration -The relay uses the recorded connection's current configuration. Drain pending rows before changing a connection to a non-durable driver, changing between incompatible SQS destination types, or otherwise invalidating its stored transport options. +The outbox stores the job's queue connection and queue name when you dispatch it. If you did not specify a queue name, it stores the connection's default. Changing that default later does not move pending jobs to another queue. Queue-name forwarding, however, uses the rules in effect when the relay sends the job. + +The relay uses the connection's current configuration. Allow pending jobs to leave the outbox before switching to a driver that does not persist jobs, changing between standard and FIFO SQS queues, or making other changes that would invalidate the stored queue options. ## Running the Relay -Run `queue:relay` under a process supervisor, just like `queue:work`. It is a dedicated CLI process which uses coroutines internally; it does not run as a job on the queue it is responsible for recovering. +The `queue:relay` Artisan command sends committed outbox jobs to their queues. Like `queue:work`, it runs as a long-lived process and should be managed by a process supervisor. You may configure its batch size and concurrency when starting it: ```shell php artisan queue:relay --batch=100 --concurrency=4 ``` -By default, it visits all configured outbox databases in turn. Pass a database name to dedicate a process to one source: +By default, the relay reads from each configured outbox database in turn. To run a relay for a single database, pass its connection name to the command: ```shell php artisan queue:relay billing @@ -222,98 +248,115 @@ The relay supports the following options: | Option | Default | Purpose | | --- | --- | --- | -| `--batch` | `100` | Maximum rows per claim, from 1 to 1,000. | +| `--batch` | `100` | Maximum messages to take from the outbox at once, from 1 to 1,000. | | `--concurrency` | `1` | Maximum batches being delivered concurrently. | -| `--sleep` | `1` | Positive seconds to wait after an empty pass or delivery failure; fractions are supported. | -| `--max-jobs` | `0` | Stop after claiming this many rows; zero means unlimited. | -| `--max-time` | `0` | Stop admitting work after this many seconds; zero means unlimited. | +| `--sleep` | `1` | Seconds to wait after finding no work or after a delivery failure. Must be positive; fractions are supported. | +| `--max-jobs` | `0` | Stop after taking this many messages from the outbox; zero means unlimited. | +| `--max-time` | `0` | Stop taking new work after this many seconds; zero means unlimited. | | `--memory` | `128` | Memory limit in megabytes. | -| `--stop-when-empty` | Off | Stop after a complete pass finds no eligible rows, then finish admitted work. | -| `--once` | Off | Process at most one batch from each selected database, without waiting between passes. | +| `--stop-when-empty` | Off | Stop when no selected database has messages ready for delivery, after finishing work already started. | +| `--once` | Off | Process at most one batch from each selected database, without sleeping. | -`--stop-when-empty` does not wait for delayed or backed-off rows to become eligible. Sources which fail or are cooling down count as empty for that pass. `--max-jobs` counts claimed rows, not successfully delivered jobs. `--once` runs synchronously and does not use concurrent delivery. +The `--stop-when-empty` option does not wait for delayed messages or scheduled retries. A database that cannot be read, or is waiting to be retried, counts as empty for that pass. The `--max-jobs` option counts messages taken for delivery, including failed attempts. When using `--once`, batches are delivered one at a time regardless of the concurrency setting. ### Concurrency and Shutdown -You may run several relay processes against the same outbox. Claims use short database operations and temporary ownership tokens, so one relay cannot settle rows reclaimed by another. Database connections and locks are released before broker publication. SQLite remains correct, but its serialized writes limit the benefit of adding relay processes. +You may run several relay processes against the same outbox. Within each process, the `--concurrency` option controls how many batches may be delivered at once using coroutines. The relay reserves messages in a short database operation, then releases its database connections and locks before contacting the queue. If another relay takes over an expired reservation, the original relay can no longer delete those records. SQLite supports this behavior, but adding relays does not allow it to perform concurrent writes. + +Your database and queue connection pools should have enough connections for the configured concurrency. Allow an additional connection if restart checks or event listeners use the same pool. Each relay process has its own pools; increasing concurrency does not increase their size. -Size database and transport pools for your relay concurrency, allowing an additional short-lived connection when restart-cache reads or application listeners use the same pool. A relay process has its own pools; increasing its concurrency does not increase the capacity of another process's pools. +The `queue.outbox.retry_after` option controls how many seconds a batch remains reserved for one relay. Set it longer than the longest expected delivery attempt for an entire batch, including all queue connections, network timeouts, and SDK retries. Reservations are not extended while delivery is running. If a reservation expires, another relay may deliver the same messages. -The `queue.outbox.retry_after` setting is the claim lifetime. Set it above the longest expected lifetime of a claimed batch, including every destination group, network timeouts and SDK retries. A claim which outlives this value may be published again by another relay. There is no lease heartbeat. +You may gracefully restart relays using the `queue:restart` command. Relays read this signal from the same configured cache store as ordinary queue workers. On restart or shutdown, a relay stops taking new work and finishes the batches it has already started. -The relay observes `queue:restart` through the same configured cache store as ordinary queue workers. On restart or graceful shutdown, it stops claiming and finishes admitted work. `SIGTERM`, `SIGINT` and `SIGQUIT` request shutdown; `SIGUSR2` pauses admission and `SIGCONT` resumes it. Configure your supervisor's stop allowance to let active calls and settlement finish. +The `SIGTERM`, `SIGINT`, and `SIGQUIT` signals request shutdown. `SIGUSR2` pauses new work, and `SIGCONT` resumes it. Configure your process supervisor to allow enough time for active deliveries and their database updates to finish before forcibly stopping the relay. Maintenance mode does not pause the relay. It only moves already committed jobs to their queues; your normal workers control whether those jobs execute. ### Failures and Recovery -The relay removes rows after confirmed queue acceptance. Failed or uncertain publications remain retryable, with increasing, jittered backoff capped at ten minutes. There is no automatic maximum-attempt cutoff: a committed business action is not silently abandoned because its queue was unavailable overnight. +Once a queue confirms that it has accepted a job, the relay deletes the outbox record. If delivery fails or its outcome is unknown, the record remains available for another attempt. The delay between attempts increases up to ten minutes, with some random variation to avoid retrying all failed deliveries at once. Retries continue until delivery succeeds or you delete the message; there is no maximum attempt limit. -Failed sources cool down without preventing healthy sources from being visited. A failed delivery batch also waits for `--sleep` before releasing its concurrency slot, limiting repeated attempts against an unavailable destination while new rows remain in the backlog. +If the relay cannot read a database, it skips that database for the `--sleep` interval and continues reading the others. After a batch fails delivery, it occupies its concurrency slot for the `--sleep` interval before another batch can take its place. This limits repeated attempts against an unavailable queue even when many new messages are waiting. -Exceptions include the source database, queue connection, row IDs, job UUIDs and display names. Delivery observer failures are reported without republishing accepted work or throttling healthy delivery. If exception reporting itself fails, the error reaches the command rather than being hidden. +Delivery errors are reported through your application's exception handler with the database, queue connection, row IDs, job UUIDs, and job names. If a delivery event listener fails, its error is reported without sending accepted jobs again or delaying successful deliveries. If the exception handler itself fails, the error is allowed to stop the command. -With SQS payload overflow enabled, separate relay attempts use separate payload pointers, so cleanup from one attempt cannot delete another attempt's payload. An uncertain SQS response may leave an offloaded payload behind, as it can during ordinary dispatch. Account for those objects in your cache/storage retention policy; the outbox does not change the existing overflow cleanup settings. +When using SQS payload overflow, each relay attempt stores its payload separately so that cleanup from one attempt does not remove another attempt's data. If SQS does not return a clear result, the stored payload may remain after the attempt, just as with ordinary dispatch. Account for these payloads in your storage retention policy; the outbox does not change your existing overflow cleanup settings. ## Inspecting and Managing Messages -List pending messages or select one database: +You may inspect pending outbox messages using the `queue:outbox` Artisan command. By default, it lists messages from all configured outbox databases. You may pass a database name to inspect only that connection: ```shell php artisan queue:outbox --limit=100 php artisan queue:outbox billing --json ``` -The listing includes job UUIDs, destinations, attempts, availability and the last publication error. Tables show UTC dates and abbreviated errors; JSON retains epoch values and complete stored errors. Payload bodies and lock-owner tokens are not displayed. +The listing includes each job's UUID, destination, attempt count, next delivery time, and last delivery error. Table output displays UTC dates and abbreviated errors. The `--json` option returns Unix timestamps and complete stored errors instead. Neither format displays job payloads or lock-owner tokens. + + +### Monitoring the Outbox -Monitor backlog size and the age of eligible work: +To monitor the number of pending messages and how long they have been waiting for delivery, you may use the `queue:outbox-monitor` command: ```shell php artisan queue:outbox-monitor --max=10000 --age=60 ``` -The command reports eligible, delayed, leased and retrying rows. `--max` and `--age` are optional alert thresholds; reaching either emits `Hypervel\Queue\Events\OutboxBusy` when a listener is registered. Eligible age excludes time a job intentionally waits for its delivery date. Use `--json` for external monitoring. +The command reports messages ready for delivery, delayed messages, messages reserved by a relay, and messages waiting to retry. The optional `--max` and `--age` options set alert thresholds for the total pending count and oldest ready message's age in seconds. When either threshold is reached, Hypervel dispatches a `Hypervel\Queue\Events\OutboxBusy` event to registered listeners. Time spent waiting for an intentional delay is not included in the ready message's age. You may use `--json` to read these statistics from an external monitoring tool. + +Horizon displays Redis jobs after they reach Redis, not while they are waiting in the outbox. The [OpenTelemetry integration](/docs/{{version}}/opentelemetry#queues) provides delivery counts, batch durations, and waiting times without polling the outbox or creating another producer span. + + +### Retrying Messages -To retry attempted messages immediately without resetting their attempt history: +If you would like to retry messages before their next scheduled attempt, you may pass one or more UUIDs to the `queue:outbox-retry` command. This clears the last error but preserves the attempt count: ```shell php artisan queue:outbox-retry JOB_UUID php artisan queue:outbox-retry all --database=billing ``` -Retry refuses live leases and never-attempted delayed jobs. When the same UUID exists on several sources, specify `--database` rather than relying on a guess. +The `all` argument retries all previously attempted messages. You may limit it to one database using `--database`. If you are retrying a UUID that exists on more than one database, this option is required. -To deliberately discard a message: +Only messages that have already been attempted can be retried. Messages currently reserved by a relay cannot be retried manually. + + +### Deleting Messages + +To remove a message that should no longer be delivered, pass its UUID to the `queue:outbox-forget` command: ```shell php artisan queue:outbox-forget JOB_UUID --database=billing ``` -This command asks for confirmation in every environment. For unattended use, `--force` skips the prompt but never bypasses live-lease protection: an already running publication cannot be recalled. - -After deleting the row, Hypervel releases its unique-job lock only when it can resolve the recorded cache store and exact owner. If cleanup fails, the command explicitly reports that the row was deleted but its lock could not be released. Custom or context-dependent cache integrations may override the command's protected `releaseUniqueLock` method to establish their namespace or resolve their repository. The command does not deserialize the queued job or hydrate arbitrary application context. +This command asks for confirmation in every environment. You may pass `--force` to skip the prompt when running it unattended. Messages currently reserved by a relay cannot be deleted, even with `--force`, since an active delivery cannot be recalled. -Horizon displays Redis jobs after they reach Redis, not while they remain in the SQL outbox. The [OpenTelemetry integration](/docs/{{version}}/opentelemetry#queues) observes relay delivery counts, batch duration and eligible-message age without polling the outbox or creating a second producer span. +After deleting the record, Hypervel attempts to release the unique-job lock using its recorded cache store and lock owner. If the lock cannot be released, the command reports the failure, but the message remains deleted. For custom cache stores or context-dependent cache namespaces, you may extend `Hypervel\Queue\Console\ForgetOutboxCommand` and override its protected `releaseUniqueLock` method. The command reads only the stored lock metadata; it does not deserialize the job or restore arbitrary application context. ## Delivery Guarantees -Delivery is **at least once**, not exactly once. If the queue accepts a message and the relay stops before deleting its outbox row, recovery can publish the same payload again. Its job UUID stays the same, but the outbox does not deduplicate processing or external effects. Make handlers idempotent where repeating an action would be harmful, using database constraints or a stable operation identifier supported by the external service. +The outbox provides at-least-once delivery, which means a job may reach the queue more than once. For example, the relay could stop after the queue accepts a job but before its outbox record is deleted. When delivery resumes, the same job is sent again with the same UUID. -Publication order is best effort. Concurrent claims, delays and retries can change it. SQS FIFO orders messages by arrival within a message group, not by the order transactions originally staged them. +Your jobs should therefore be safe to process more than once. For example, you may use a unique database constraint to prevent duplicate records, or pass a stable operation identifier to a payment provider that supports idempotent requests. The outbox does not detect duplicate processing or undo external actions. + +Jobs are not guaranteed to reach the queue in dispatch order. Concurrent relays, delays, and retries can change that order. SQS FIFO preserves arrival order within a message group, not the order in which your transactions stored jobs in the outbox. Successful rows are deleted rather than archived. The outbox records pending delivery, not a business audit trail or a list of failed job executions. Once a worker receives a job, ordinary queue retry and failed-job behavior applies. ## Custom Queue Drivers -Drivers which use the base queue's enqueue lifecycle can opt into staging without changing their public dispatch APIs. Overrides which bypass that lifecycle remain direct. The protected `resolveOutboxQueue`, `preparePayloadForOutbox`, `prepareOutboxOptions` and `availableAtForOutbox` hooks let a driver capture its logical destination, stored payload, transport options and initial eligibility while the original job is available. +If you maintain a custom queue driver, its dispatch methods must use the base `Queue` class's enqueue methods to support the outbox. Methods that bypass this behavior continue sending jobs directly. You may customize outbox storage using the protected `resolveOutboxQueue`, `preparePayloadForOutbox`, `prepareOutboxOptions`, and `availableAtForOutbox` methods. These methods determine the queue name, stored payload, queue-specific options, and earliest delivery time while the original job is available. + +By default, the relay waits until a message is due and sends its stored payload through your driver's `pushRaw` method. It does not dispatch through the `Bus` facade or serialize the job again. If your driver uses a connection pool, the relay borrows one connection for the whole delivery batch. -By default, the relay waits until a message is due and publishes it through the driver's existing `pushRaw` method. It does not call Bus or serialize the job again. A pooled driver is borrowed once for the publication batch. +If your driver supports more efficient bulk delivery, you may implement `Hypervel\Contracts\Queue\PublishesOutboxMessages`. Its `publishOutboxMessages` method receives an array of immutable `Hypervel\Queue\Outbox\OutboxMessage` objects and returns a `Hypervel\Queue\Outbox\OutboxBatchResult`. The result contains the IDs of accepted messages, the IDs of messages deferred until their delivery time, and an optional failure. -Drivers with a more efficient batch operation may implement `Hypervel\Contracts\Queue\PublishesOutboxMessages`. Its `publishOutboxMessages` method receives immutable `Hypervel\Queue\Outbox\OutboxMessage` values and returns an `OutboxBatchResult` containing confirmed accepted row IDs, deferred row IDs, and an optional failure. IDs refer to outbox rows, not broker identifiers; array keys are ignored. Accepted and deferred IDs must be unique, disjoint members of the input. A result without a failure must account for every input row. Only messages with an intended delivery time may be deferred. +These IDs refer to outbox rows, not identifiers assigned by the queue. Array keys are ignored. Each ID must belong to an input message and appear only once across the accepted and deferred lists. A result without a failure must account for every message. Only messages with an intended delivery time may be deferred. -Report confirmed partial acceptance before returning a publication error. Keep uncertain outcomes retryable, and propagate coroutine cancellation. Do not emit the ordinary enqueue lifecycle again: it completed when the job was accepted into the outbox. +If only part of a batch succeeds, include the confirmed messages in the result alongside the error. Do not report a message as accepted if its outcome is unknown. Allow coroutine cancellation to propagate, and do not dispatch ordinary enqueue events again; those events were already dispatched when the job entered the outbox. diff --git a/src/docs/queues.md b/src/docs/queues.md index 635e4b9c2..cfdd22ed7 100644 --- a/src/docs/queues.md +++ b/src/docs/queues.md @@ -1508,7 +1508,7 @@ ProcessPodcast::dispatch($podcast)->beforeCommit(); ### Transactional Outbox -After-commit dispatch prevents a worker from seeing uncommitted data, but a process can still stop between committing the database transaction and publishing the job. When a job must survive that gap, use the [transactional outbox](/docs/{{version}}/queue-outbox): +Dispatching after commit prevents a worker from processing a job before your database changes are saved. However, your application could still stop after committing the transaction but before sending the job to the queue. To protect against losing the job in this case, you may use Hypervel's [transactional outbox](/docs/{{version}}/queue-outbox): ```php use Hypervel\Support\Facades\DB; @@ -1520,7 +1520,7 @@ DB::transaction(function () use ($podcast) { }); ``` -Hypervel stores the job with the database write, then a separate relay publishes it to your existing queue. See the [outbox guide](/docs/{{version}}/queue-outbox) for setup, supported drivers, delivery guarantees and operations. +Hypervel stores the job as part of the database transaction. A separate relay process then sends it to your existing queue after commit. Before using the outbox, follow its [setup instructions](/docs/{{version}}/queue-outbox#setup), including creating the table and running the relay. ### Job Chaining