diff --git a/AGENTS.md b/AGENTS.md index c839ea1dbe..5bab5ab185 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -119,7 +119,7 @@ During implementation, run new or changed test files immediately. After completi Use checks that match the change. For isolated changes, run `composer lint:fix`, `composer analyse`, and the affected tests. Run a single affected test file with PHPUnit. Use ParaTest when the affected tests span multiple files. Only run `composer fix` when changes could affect code beyond the affected tests; it already runs formatting, analysis, and all test suites, so do not run those checks separately first. -If a check fails, use targeted checks while correcting the issue, then run the failed check and each remaining check. Rerun an earlier check only if the correction could affect it. +After a failure, fix the cause and rerun only affected tests, formatting or static analysis. Then finish any verification not yet run. Repeat the full test suite only if the fix itself has framework-wide impact. ## Development Conventions diff --git a/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md new file mode 100644 index 0000000000..52534ea7ee --- /dev/null +++ b/docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md @@ -0,0 +1,599 @@ +# Hypervel Transactional Queue Outbox + +## Status and outcome + +Implement an optional transactional outbox in `hypervel/queue`. A job explicitly or conventionally routed through the outbox is serialized once, stored with the business write on the authoritative database transaction, and relayed to its existing queue connection after commit. This closes the database-commit / broker-publish crash window without replacing Queue, Bus, Horizon, queue drivers, or application job APIs. + +The completed feature must provide: + +- Laravel-style `viaOutbox()` / `withoutOutbox()` ergonomics plus a `ShouldQueueViaOutbox` marker and per-connection default; +- correct behavior when applications switch among Redis, SQS, Beanstalkd, database, sync, background, deferred, null, fake, and pooled connections; +- set-based outbox inserts for `Bus::bulk()` / batch work, using only the chunks required by documented database binding limits; +- horizontally scalable, lease-based relay workers with database-engine-specific claim paths; +- a staging/relay pipeline that preserves the original payload UUID, delay, SQS FIFO options, SQS overflow behavior, normal dispatch lifecycle, and Horizon metadata/events; +- an explicit at-least-once guarantee, including safe treatment of ambiguous transport outcomes; +- no outbox database work, service resolution, payload re-encoding, or additional outbox event allocation when the feature is not selected; +- production operations for monitoring, retrying, and deliberately forgetting pending rows; +- no dependency on Horizon and no third-party dependency. + +The result is a queue feature, not a generic event bus, inbox/deduplication system, workflow engine, CDC pipeline, or completed-message archive. Those concerns must not be smuggled into the implementation through aggregate metadata, status histories, a second dead-letter table, container swapping, or transport-specific application APIs. + +## Verified baseline + +### Implementation status + +Implemented: framework prerequisites, public selection and wrapper propagation, schema/configuration, owner-aware staging and repository operations, Redis/Horizon/database/SQS/pooled raw publication, relay/worker resource boundaries, operator commands, and guarded delivery events/telemetry. + +Implementation, live-engine/concurrency and API-integration coverage, canonical documentation, the retained benchmark harness and performance measurements are complete. Measurements support the existing production design; see [benchmark results](../../tests/Benchmarks/QueueOutbox/results.md). + +Complete the documentation style review, then resume PR review and CI verification on both remotes before merging. + +### Hypervel + +- `Queue::enqueueUsing()` is the common single-job seam. `enqueueNow()` guards payload finalization, `JobQueueing`, and the driver callback as one attempt; accepts dispatch locks only after storage/transport acceptance; and raises `JobQueued` afterwards. Non-cancellation attempt failures raise `JobQueueingFailed`, while `PendingDispatch` owns final cleanup before a queue accepts the job. +- `DispatchLockContext` is the existing object-keyed `WeakMap` for dispatch-owned unique/debounce lock provenance. It carries the live cache repository needed for exact cleanup only while the job object remains live, exposes Laravel-compatible scalar unique metadata to payload hooks, and is cleared on acceptance/release. `Queueable::$uniqueLockOwner` is initialized to Laravel's empty-string default and owner-capable locks release through `UniqueLock::releaseOwned()`. +- Database, Redis, and SQS have optimized `bulk()` implementations; the base implementation loops through `push()` / `later()`. `Bus::bulk()` groups jobs by queue connection and queue name. `Batch::add()` already calls queue `bulk()` inside the batch repository transaction. Database and Redis bulk paths already provide bounded, atomic storage/publication. +- `DatabaseTransactionsManager::callbackApplicableTransactions()` exposes coroutine-local transaction records keyed by exact resolved connection identity, distinguishing base and role connections. Matching host/database configuration never proves atomicity. +- `PdoConnection::lockForPopping()` and `maxBindings()` own live database capabilities and cache them only for the current physical session. MySQL/MariaDB/Vitess/PlanetScale, PostgreSQL, and SQLite supply their correct version-dependent values. `DatabaseQueue` already consumes both, and `Concerns\InsertsDatabaseRows` provides binding-safe, transactionally atomic chunked inserts. +- A database queue insert already participates atomically when its queue database connection is the same active connection as the business transaction. Adding a second outbox table in that case would add latency without improving the guarantee. +- `Queueable` already carries tri-state `afterCommit`, connection, queue, delay, chain, SQS message-group/deduplication, unique, and debounce state. `PendingDispatch`, `PendingBatch`, and `PendingChain` are the expected fluent surfaces. A valid batched job is only required to use `Batchable`, so batch-wide outbox state cannot assume that every job also uses `Queueable`. +- Final queue payloads already contain a stable job UUID. Hidden payload context carries the unique-job cache store, key, and owner; debounce ownership remains on the serialized job. Existing rollback cleanup depends on an unaccepted dispatch record; accepted transactional storage needs an independent release snapshot. +- `QueuePoolProxy` reacquires a pooled queue for after-commit dispatch rather than retaining a borrowed network client across a transaction. Raw outbox publication must likewise borrow once per batch and release immediately. +- `ConnectionResolver::releaseIdleConnections()` can return idle sessions early, but preserves pinned and certain multi-server write sessions and does not cover all custom drivers or listener-owned resources. Keep separate short-lived relay phase coroutines for guaranteed cleanup; committing a transaction alone does not provide that guarantee. +- SQS bulk delivery already chunks at 10 entries and the request byte cap, handles explicit per-entry success/failure, preserves FIFO options, and retains overflow pointers for ambiguous SDK failures. Reuse those rules rather than building a parallel SQS implementation. +- `JobQueued::$id` already matches the Queue contract's `mixed` result, and raw Sync/Deferred/Background publication executes through each queue's intended timing behavior. +- Horizon separates `JobPayload::prepareForQueueing()` from `markAsPushed()` while retaining `prepare()` as their single-encoding composition. Raw retry preserves stored classification, and delayed publication stamps `pushedAt` at the actual push. +- Generic Worker lifecycle callbacks and deferred signal callbacks already run through short-lived coroutine contexts, and worker delays use fakeable `Sleep`. The outbox worker must follow the same resource-ownership pattern. +- Queue, Bus, and Horizon READMEs already follow the required order, and the application skeleton's queue configuration matches Foundation. Reuse the existing foundation instead of repeating completed cleanup. + +### Reference findings + +- Reference checkouts live under `/home/binaryfire/workspace/references/`. `laravel-outbox` demonstrates bulk storage and `SKIP LOCKED`, but its request-wide mutable collector and temporary replacement of container `events` / `queue` bindings are unsafe under concurrent Swoole coroutines. Its aggregate IDs, payload HMAC, status/history fields, completed-row retention, second dead-letter table, and processor-job bootstrap are unnecessary for a native queue outbox. +- Laravel framework (`laravel/framework`) and Horizon (`laravel/horizon`) provide the public queue/batch/chain naming baseline; Hypervel's optimized bulk and transaction behavior is authoritative where it differs. +- Current Laravel's `Queueable`, `UniqueLock`, and hidden unique-job context carry the cache lock owner. Reusing that upstream behavior is safer and simpler than inventing outbox-only ownership state. +- MassTransit and NServiceBus persist the finalized outgoing transport operation and relay it later. Both confirm that the broker-send / outbox-delete boundary remains at-least-once and receivers must be idempotent. +- Watermill Forwarder / SQL confirms the small two-part shape: a transactional SQL publisher plus a daemon that forwards the stored envelope; its SQL publisher accepts multiple messages in one insert. +- Gruelbox Transaction Outbox validates bounded batches, retry scheduling, and skip-locked claiming. Eventuate validates CDC as an alternative deployment architecture, but CDC/Kafka infrastructure is unjustified for a framework queue primitive. +- TickerQ does support batch additions; it is not evidence against bulk APIs. Its scheduler concerns do not belong in this feature. +- PostgreSQL and MySQL both explicitly document `SKIP LOCKED` as suitable for multiple consumers of queue-like tables. SQLite remains a correct serialized fallback. + +Primary upstream reading: + +- +- +- +- +- +- +- +- +- +- +- +- +- + +## Framework changes + +### Independently useful prerequisites + +Implement these at their owning boundary, before outbox integration. If already completed when implementation starts, verify their contracts and reuse them rather than repeating the work. + +1. **Exact transaction identity.** Use the resolved connection object's `getNameWithReadWriteType()` consistently for transaction begin/commit/rollback/reset, applicable-transaction checks and connection-bound callbacks. Do not change `getName()` or derive identity from the caller's requested suffix: shared in-memory SQLite aliases return one owner object. Update `ManagesTransactions`, `Connection::resetTransactionState()`, Foundation testing's wrapping-transaction lookup and `RequestContextSynchronizer`, Pennant's transaction-record matching, and other consumers comparing record names. Distinct base/`::write` sessions must not consume each other's records or callbacks. This owner-approved departure changes role-qualified transaction-record names observable by extensions compared with Laravel's base-name records. Explain the correction and required exact-owner comparison in database compatibility documentation; preserve public signatures. +2. **Context-safe dispatch-lock cleanup.** Add the supported boot registration method `DispatchLockContext::captureReleaseContextUsing(?callable $callback): void`, normalized to closures; registration appends, `null` clears. Its callbacks return a release wrapper or null, capturing only the context they need. Internal `captureReleaseContext(): ?Closure` composes wrappers in registration order, first outermost. Call it in `UniqueLock::acquireForDispatch()` and `DebounceLock::acquireForDispatch()` **before acquiring anything**, then pass its result into their internal registration methods. A throwing capture callback leaves no acquired lock/token. Unique and debounce acquisition are mutually exclusive, so capture once per acquisition. Records and snapshots retain the composed wrapper; release runs both existing owner-aware cleanup operations inside it, preserving existing error handling. Wrappers invoke the operation once and restore their prior context in `finally`. Without registrations return null immediately; without a captured wrapper run existing release code without allocating a release closure. Keep all record/capture operations internal, document the registration method's boot lifetime and callback contract in PHPDoc, and clear callbacks in the existing `flushState()`/test-subscriber path. Do not add a contextual-store interface, wrapper traversal, full coroutine-context snapshot, or cache implementation changes. +3. **Transactional storage rollback.** Immediately after a successful database-queue/outbox insert and before accepting dispatch locks, register an independent `DispatchLockContext::snapshot()` on the exact storage connection if its `transactionLevel() > 0`. Rollback calls `releaseSnapshot()` even after acceptance erases the live record. Apply to applicable single/bulk paths; after chunk-insertion savepoints complete, attach to the remaining enclosing transaction that can still undo the rows. Failed inserts and already-durable no-transaction staging register nothing. Existing deferred ownership remains intact; never release broker-accepted locks because a business transaction rolls back. Complete identity and capture-context fixes first. +4. **Batch acceptance and terminal notifications.** SQS deferred bulk must reuse `deferBatchEnqueueAfterCommit()`. Guard finalization, entry preparation, per-chunk `JobQueueing` and publication; accept every confirmed success before calling observers. Clean explicitly rejected overflow pointers before success notifications. Stop before the next chunk on a transport or notification failure, and fail only outstanding members. Across Redis/database/SQS, drain each bounded batch's terminal notifications despite ordinary listener failures, using a narrow protected `Queue::notifyBatchMembers(array $members, Closure ...$notifications): ?Throwable` helper. It returns the first ordinary notification error and immediately rethrows cancellation. Separate the Horizon post-push hook from `JobQueued` notifications so either ordinary failure cannot skip the other. Failure events carry the primary attempt error; rethrow the first listener error when one occurred, otherwise the primary error. Preserve confirmed acceptance and never emit failure for accepted jobs. Keep SQS's existing protected failure helper and per-attempted-chunk queueing events. +5. **Horizon preparation and acceptance.** Extract classification values into a protected `JobPayload` method shared by `prepare()` and `prepareForQueueing()`; `prepare()` sets them plus `pushedAt` in one encoding operation. Retain public split methods, stored classification on raw retries and Laravel's metadata customization methods. Move Redis's bulk preparation/post-push hooks to base Queue as `preparePayloadForPush()` / `handlePayloadPushed()`, used by single and optimized bulk paths. Preparation is guarded; the post-push hook runs only after dispatch-lock acceptance. On the single path, use local exception retention rather than batch arrays/closures: immediately rethrow cancellation, attempt `JobQueued` after an ordinary post-push error, then rethrow the first ordinary error. Horizon's stock monitoring listeners perform Redis I/O, so their failures must not turn successful storage into failed dispatch or release accepted locks. Horizon `push()` stores through a shared protected parent-Redis helper instead of invoking its public `pushRaw()` override; this subclass-routing departure is owner-approved. Base Redis routing stays unchanged. Horizon's inherited `later()` uses the shared hooks; external `pushRaw()` prepares with null and emits its own Pending/Pushed events. Remove the obsolete last-pushed context handoff and its state-leak test, not retain a compatibility no-op. Document the Horizon subclass routing change. + +6. **Broadcast public-property extraction.** Skip uninitialized public properties before reading them in `BroadcastEvent::getPayloadFromEvent()`. This supports untouched `Queueable` state and application-declared typed properties without changing initialized nulls, value formatting or `broadcastWith()` behavior. Do not add an outbox-specific property blacklist or reflection cache. +7. **Inline after-commit ownership.** When `deferBatchEnqueueAfterCommit()` has no applicable transaction for its selected connection, invoke the enqueue callback on the current queue and return without transferring dispatch-lock ownership, registering rollback callbacks or reacquiring a pooled queue. Inline failure belongs to the calling dispatch/failover attempt; only genuinely deferred callbacks own final cleanup. Keep the helper void and preserve local-driver return values. +8. **Cache store identity.** In `CacheManager::resolve()`, fill a concrete `Repository`'s missing name from the normalized store configuration, preserving creator-supplied names and custom contract implementations. Add an internal construction-time `setName()` beside `getName()`. Memoized repositories retain the underlying repository's name while keeping events disabled. This fixes unique-job cleanup after missing-model errors for documented custom stores and memoized stores; do not name ad hoc repositories in `build()` or add per-lock detection. + +### Outbox integration + +Keep the new mechanism under `Hypervel\Queue\Outbox`; use small integration points in existing packages: + +- Bus/Foundation: add the marker, shared fluent state, pending-dispatch/batch methods, and wrapper-job propagation described below. +- Queue: add selection and payload/option hooks, outbox services and commands, config, and the migration generator. Stage through the existing guarded enqueue lifecycle and reuse `DispatchLockContext`, `PdoConnection` capabilities, and `InsertsDatabaseRows`. +- Queue drivers/proxies: implement the fast paths and raw-batch capability below; select/partition outbox work before SQS's direct FIFO-delay validation. Preserve direct validation, existing raw publication APIs, and direct overflow retention; isolate outbox overflow pointers by claim attempt. +- Horizon: classify staged payloads through the outbox-only hook and stamp them at relay publication through the split `JobPayload` lifecycle. Direct publication uses the shared preparation/post-acceptance hooks, without a coroutine context handoff. +- OpenTelemetry: consume guarded outbox delivery events without adding work to ordinary dispatch. +- Foundation/application skeleton/docs: add matching outbox configuration and migrations, then add the canonical guide and concise links without revisiting completed baseline cleanup. +- Doctor remains unchanged: do not copy runtime selection rules into a second setup detector or add relay-heartbeat machinery. Explicit setup, migrations, native storage failures, and outbox commands provide the required operating surface. + +## Decisions + +### Ownership and guarantee + +- Implement under `src/queue/src/Outbox`; the migration stub/command live with Queue, while the shipped application defaults live at their existing owner, `src/foundation/config/queue.php`. Horizon only adapts actual Redis publication. The feature works with Redis whether Horizon is installed or not. +- Outbox atomicity covers only writes on the selected database transaction. It cannot atomically cover another database, an email, filesystem/object-store write, HTTP call, or other external effect. +- Relay is at-least-once. If the target accepts a job and the relay crashes or cannot delete the row, the same job payload can be published again. Stable job UUIDs are retained; application handlers that cannot tolerate duplicates must be idempotent. +- Dispatch order is best-effort: concurrent claims, delays and retries can reorder publication. SQS FIFO preserves broker-arrival order within a message group, not the order application transactions staged their jobs. Do not add per-group sequencing machinery. +- An outbox is not an inbox. Do not claim exactly-once processing or add receiver deduplication to this work. +- Successful rows are deleted. Failed/ambiguous rows remain in the same table for retry. Do not retain completed rows, introduce statuses that duplicate nullable lease state, or add an automatic terminal dead-letter cutoff that can silently abandon a committed business action. + +### Selection and precedence + +Add the marker as an ordinary queued-job contract. Put the fluent state in a small `InteractsWithQueueOutbox` trait used by both `Queueable` and `Batchable`, because Laravel-compatible batches do not require `Queueable`: + +```php +interface ShouldQueueViaOutbox extends ShouldQueue +{ +} + +// InteractsWithQueueOutbox; intentionally uninitialized until explicitly set. +public ?bool $viaOutbox; +public ?string $outboxDatabase; + +public function viaOutbox(): static; +public function withoutOutbox(): static; +public function onOutboxDatabase(UnitEnum|string $connection): static; +``` + +Framework code reads explicit state with `$job->viaOutbox ?? null` and `$job->outboxDatabase ?? null`. PHP serialization omits uninitialized typed properties, so ordinary jobs gain no payload fields or bytes. `onOutboxDatabase()` requires a non-empty resolved enum/string and also selects outbox delivery; `withoutOutbox()` resets the selected database to null, preserving that explicit reset against batch defaults. Re-enabling with `viaOutbox()` then uses normal owner resolution. Keep this state in `InteractsWithQueueOutbox`, used by `Queueable` and `Batchable`, so Laravel-compatible batch jobs that use only `Batchable` can participate. Do not add chain-specific outbox properties or `all*` methods: follow-up chain dispatch happens after the current handler completes, outside the business transaction, and the current queued job already supplies durable retry if that dispatch fails. Explicit Laravel-shaped properties are easier for jobs, wrappers, serialization, and agents to understand than a framework-only associative state bag. + +Selection precedence is: + +1. explicit `withoutOutbox()` / `viaOutbox()` on the job; +2. `ShouldQueueViaOutbox`; +3. the target queue connection's `outbox` boolean; +4. direct existing queue behavior. + +`afterCommit()` / `beforeCommit()` and outbox selection remain independent. They control when a direct queue publish occurs; an outbox-selected dispatch inserts its durable row immediately inside the owning transaction, and delivery necessarily follows commit. `withoutOutbox()` is the explicit opt-out when a marker or connection default would select the outbox. No combination throws merely because an application changed queue drivers or commit defaults. + +Mirror the current-job fluent methods on `PendingDispatch`. Add `viaOutbox()`, `withoutOutbox()`, and `onOutboxDatabase()` to `PendingBatch`; these are aggregate defaults and never overwrite a job whose corresponding property is initialized. Persist only the exact `viaOutbox` / `outboxDatabase` option keys on a batch. `Batch::add()` applies them to every job it is about to pass to `bulk()`, including jobs added after initial dispatch. Do not add outbox methods to `PendingChain`, propagate outbox state to later chain members, or add a second `bulkViaOutbox()` API. Normal `Bus::bulk()` honors job/batch/connection selection and is the optimized path. + +Examples: + +```php +ChargeInvoice::dispatch($invoice)->viaOutbox(); + +Bus::batch($jobs) + ->onConnection('redis') + ->viaOutbox() + ->dispatch(); +``` + +The dispatch return value for a staged single job is the finalized payload's job UUID, also indexed on the outbox row for operator lookup. Staged-only bulk returns `null`; direct drivers retain their existing single and bulk return values. + +### Laravel-compatible surface + +- Do not change signatures or ordinary semantics of `dispatch`, `push`, `later`, `bulk`, `afterCommit`, `beforeCommit`, batch, chain, Queue/Bus fakes, or the `Hypervel\Contracts\Queue\Queue` contract. `pushRaw()` remains a direct, non-staging escape hatch even when a connection default enables outbox; the relay uses the separate raw-batch SPI when available and `pushRaw()` as its generic fallback, avoiding recursion and preserving existing raw-payload behavior. +- Add fluent methods to the same traits/builders that own their neighboring Laravel APIs; do not add an Outbox facade, transaction wrapper, alternate dispatcher, or environment-specific call site. +- Propagate outbox selection and the selected database from queued listeners, mailables, notifications, and broadcasts into their existing queue wrapper jobs beside the independent `afterCommit` propagation. Inline `isset($source->viaOutbox)` copies explicit true/false; otherwise promote `ShouldQueueViaOutbox` to wrapper `viaOutbox = true`. Copy a non-null owner with `isset($source->outboxDatabase)`. Null/absent selection falls through to the marker; untouched wrappers retain uninitialized fields. Keep batch-default initialization semantics separate. Do not add a shared public helper or producer-specific outbox APIs. +- Keep application-visible names short and intention-revealing: `viaOutbox`, `withoutOutbox`, `onOutboxDatabase`, `ShouldQueueViaOutbox`, `queue:relay`, and the `queue:outbox` operator command family. +- Add protected enqueue/destination/options hooks on the base queue and an optional raw-batch contract for optimized durable drivers. Database/local/failover queues override the smallest cohesive hook for their fast path; null already bypasses enqueueing. Do not build a central driver-name/type switch. Custom drivers retain existing behavior while base-lifecycle drivers gain staging and may opt into optimized publication. +- Preserve Horizon's existing public `JobPayload::prepare($job)` method as the composition of its classification and actual-push operations. Use the narrower existing operations for outbox staging/publication without breaking the Laravel-compatible method. + +### Driver behavior + +| Target at dispatch time | Selected outbox behavior | +|---|---| +| Redis, SQS, Beanstalkd, custom durable driver using the base enqueue lifecycle | Stage on the owning SQL database, then relay | +| Database queue, no active business transaction | Push directly; the jobs table is already durable | +| Database queue on the sole or explicitly selected active owning connection/session | Push directly inside that transaction; already atomic | +| Database queue on another connection while a business transaction is active | Stage on the business database, then relay to the database queue | +| Sync, background, deferred | Preserve local semantics; execute after successful commit when one exists, otherwise normally; an explicit database selects that exact active transaction; document that no crash durability is possible | +| Null | Preserve the no-op behavior and release only this dispatch's owner-token-matched unique/debounce locks | +| Queue/Bus fake | Record the job in the fake only; never touch an outbox table | +| Failover | Preserve normal failover, but defer the whole attempt until applicable transactions commit when outbox is selected; document that this degradation cannot provide crash durability | + +Failover and outbox solve different policies, and failover may cross heterogeneous drivers whose wire payload/options are created only in the selected child. Do not invent a multi-destination outbox envelope. Instead, selected outbox behavior degrades to failover's existing after-commit path, then performs ordinary child attempts. Scope a coroutine-local forwarding record to each job-bearing child call, suppressing outbox only for that matching job; exclude raw calls and restore before `QueueFailedOver` listeners. Clear forwarding state during synchronous payload execution so independent jobs, including same-handler string continuations, retain normal selection. The failover connection's explicit/marker/config selection is authoritative; child defaults still apply when used directly. + +Driver switching never causes an exception merely because the selected driver is local, failover, or already durable. There are no environment checks; behavior follows runtime capability. Local/failover drivers register against the explicitly selected active transaction when supplied and reject only a false claim of atomic ownership (an explicit owner while transactions exist only on other connections); without an explicit owner they preserve their existing transaction-selection rules. Failover's explicit-owner callback enters the child-attempt loop directly with after-commit already satisfied; matching children must not revalidate the committed owner or defer onto an unrelated transaction. Ordinary all-transactions callbacks retain reentry for transactions opened by earlier callbacks. + +Outbox selection alone enters the local after-commit path only when an applicable transaction exists. Without one, preserve immediate sync return values and coroutine delay timer IDs. Existing explicit or configured `afterCommit` behavior remains unchanged. + +A selected outbox keeps an SQS FIFO job staged until `deliver_at`, then publishes it without the forbidden per-message `DelaySeconds`; any queue-level delay applies after publication. In `later()` and `bulk()`, select/partition outbox work before applying the existing FIFO checks to direct work. Capture `getQueueableOptions()` without a delay argument at staging; preserve its direct-call validation. No-transaction staging has the same delayed-publication behavior. + +A custom durable driver participates when it routes `push()` / `later()` / `bulk()` through the base enqueue hooks. The relay uses the optional raw-batch publication contract below when available and otherwise publishes due rows sequentially through the existing `pushRaw()` contract. Existing custom queues that override the public enqueue methods and bypass the base lifecycle retain their existing direct behavior; intercepting arbitrary overrides would require a permanent decorator and hot-path overhead for every queue operation. Document this extension contract instead of adding that machinery. + +Evaluate fake/local/failover and already-atomic database fast paths before resolving outbox services or touching the outbox table. These paths need no outbox migration merely because a job carries the marker; local/failover drivers consult active transactions for their documented after-commit fallback. `NullQueue` bypasses single enqueueing and opts its inherited bulk path out of staging; `PendingDispatch` releases its unaccepted locks. + +### Database ownership + +Configure each database source whose outbox table the relay drains; use configured connection names, not both a base name and its `::write` alias: + +```php +'outbox' => [ + 'databases' => [ + env('QUEUE_OUTBOX_DATABASE', env('DB_CONNECTION', 'sqlite')), + ], + 'table' => env('QUEUE_OUTBOX_TABLE', 'job_outbox'), + 'retry_after' => (int) env('QUEUE_OUTBOX_RETRY_AFTER', 300), +], +``` + +Every shipped connection record includes `'outbox' => false`. Base `Queue::setConfig()` caches the optional, already-cast value in a typed flag, using `false` when an existing custom record omits it; `QueueManager` continues passing configuration once. On lazy service resolution, inject typed config values into their owning services through the provider, following the existing batch/failed-job providers; no `OutboxConfiguration` class. Validate a non-empty database-source list and positive `retry_after`. Native failures cover invalid table/connection configuration; do not add duplicate-name validation. Relay batch size is a worker option, not duplicate config. Retry delays are fixed at `DatabaseOutboxRepository::RETRY_DELAYS = [1, 5, 15, 30, 60, 300, 600]`, beside the settlement code that applies them; staging chunks follow database binding limits, not another knob. + +Resolve the owning database as follows: + +- one applicable active transaction: use that connection; an explicit different database is an error; +- multiple active transaction connections: require `onOutboxDatabase()` naming one of them; otherwise throw an exception that lists the active connections and names the required method. Document that only the explicitly selected transaction is atomic with the job; +- no active transaction: use the explicit database or the first configured database; +- whenever SQL staging is required, its source must be covered by `databases`, so the relay cannot silently ignore the rows. For base/`::write` transactions, compare the resolved owner's `getName()` with the configured source names, while inserting and registering rollback on the exact `getNameWithReadWriteType()` session. + +Nested records on one resolved connection count as one owner. Explicit selection must identify an applicable owner when transactions exist; otherwise it selects configured storage. Resolve transaction identities from objects, not requested suffixes or host/database comparisons. `onOutboxDatabase('mysql::write')` selects that exact active session while `databases => ['mysql']` covers draining its table. Do not list both as sources: that would duplicate polls/counts and make UUID lookup falsely ambiguous. Shared in-memory SQLite aliases remain one owner. Do not borrow an unrelated base connection merely to inspect an alias. Install the migration on each configured source; avoid listing different names for the same physical outbox table rather than adding topology detection. Workers visit sources round-robin or pin one source through their database argument. + +SQL outbox staging is writer-owned. During preflight, before finalization or insertion, reject a resolved owner for which `ConnectionName::parse($owner->getNameWithReadWriteType())->isRead()` is true, including explicit no-transaction selection. Name that connection in the error and direct the caller to its base/`::write` connection; existing pre-acceptance cleanup releases dispatch locks. Do not probe endpoints or exempt read aliases without separate read configuration. Shared in-memory SQLite resolves to its base owner and remains supported. This rule governs SQL staging, not local after-commit fallback or ordinary unselected queue APIs. + +Do not add a dynamic shard/source registry. Configured database connections are the complete, explicit set of sources the relay drains. + +## Storage model + +Generate with `make:queue-outbox-table` (alias `queue:outbox-table`) using the configured table name. `OutboxTableCommand` extends `MigrationGeneratorCommand` with normal duplicate-table detection; no combined migration needs an override. Commit the same schema in `src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php` and `/home/binaryfire/workspace/contrib/hypervel/hypervel/database/migrations/0001_01_01_000009_create_job_outbox_table.php`; do not maintain a third test-only migration copy. + +```php +Schema::create('{{table}}', function (Blueprint $table) { + $table->id(); + $table->uuid('uuid')->unique(); + $table->string('connection'); + $table->string('queue')->nullable(); + $table->longText('payload'); + $table->text('options')->nullable(); + $table->unsignedBigInteger('deliver_at')->nullable(); + $table->unsignedInteger('attempts')->default(0); + $table->unsignedBigInteger('available_at'); + $table->uuid('lease_token')->nullable(); + $table->text('last_error')->nullable(); + $table->unsignedBigInteger('created_at'); + + $table->index(['available_at', 'id']); +}); +``` + +- Bigint `id` keeps the hot index and range/order operations compact. `uuid` repeats the finalized payload's existing job UUID for indexed logs and operator commands; do not generate a second outbox identity. A fresh `lease_token` identifies only one claim attempt. +- All three time columns are integer Unix seconds supplied from the framework clock, following queue/cache operational deadline conventions. Bigint avoids PostgreSQL's signed 32-bit epoch limit. No database-clock default or timezone-dependent parsing: SQL session timezone and later `app.timezone` changes cannot move a staged instant. Keep message fields as `int`/`?int`, converting only for display. No timer (null or integer zero) stores `deliver_at = null`; a requested timer already due stores now. Preserve `InteractsWithTime::availableAt()`'s future ceil-to-second. A non-null timestamp preserves SQS's per-message timer override even after it expires. +- `deliver_at` is immutable intended delivery; `available_at` is next eligibility, including lease expiry and retry backoff. Optimized drivers compute non-negative remaining delay immediately before each transport chunk. Redis/database initially qualify now; standard SQS at `max(now, deliver_at - 900)`, with `DelaySeconds = min(900, max(0, remaining))` only when `deliver_at` is non-null. Otherwise omit it so the configured queue delay still applies. FIFO and generic raw fallback qualify at `deliver_at ?? now`; FIFO never sends `DelaySeconds`. Claim moves `available_at` to its lease deadline; retain the pre-claim value in the in-memory message as `eligibleAt` for ordering/latency. No separate reservation timestamp. +- `queue` stores the unforwarded logical name with the connection's default expanded. Publication applies queue-name forwarding once through the normal storage path. Changing the default cannot reroute the row; changing a queue-name forward can. Connection forwarding is already resolved at staging and is not rerun. A queue-less custom driver may store null. Drain before changing captured connection routing or forwarding between incompatible FIFO/standard destinations. +- `payload` is the finalized opaque envelope after payload hooks and `JobPayloadFinalizing`, not a job to redispatch through Bus. `longText` preserves its bytes; a database JSON type could normalize them. Adapters may add only delivery-time metadata such as Horizon's `pushedAt` or an SQS overflow pointer; SQL bytes, UUID and serialized command stay unchanged. Opaque `options` likewise uses text and is decoded only for publication. +- `options` is null when empty, otherwise JSON containing only namespaced transport options needed after the object is gone, notably `{"transport":{"sqs":{}}}` for precomputed group/deduplication. Do not duplicate payload lock metadata or add an unused format version; migrate in-flight rows if a future format changes. Adapters use their own option members; native transport errors retain/back off rows. Do not add a driver-kind compatibility registry or business fields. +- No tenant or shard column: this is an operational table like `jobs`, `failed_jobs`, and `job_batches`; application context already travels inside the finalized payload. +- No `updated_at`, `status`, `processed_at`, history, payload hash, or stack trace. Eligibility is exactly `available_at <= now`; a live lease has a non-null token and `available_at > now`. Full exceptions are reported once; `last_error` is bounded to 4,096 characters. + +Use Query Builder/raw SQL, never Eloquent, for this hot operational table. Raw claim SQL wraps configured identifiers through the active grammar and binds every value. A one-statement bulk relies on that statement's atomicity; a bulk requiring multiple binding-safe insert chunks runs inside `Connection::transaction()` (a savepoint when already nested), so even a caller that catches the exception cannot commit a partial owner group. Reuse `InsertsDatabaseRows`; do not add a second chunk calculation or expose another configuration knob. Claim and settlement sizes may be lower than the requested worker batch when an engine's binding/expression limit requires it. + +## Internal architecture + +Keep public surface small and group the cohesive implementation under `Hypervel\Queue\Outbox`: + +```text +src/contracts/src/Queue/PublishesOutboxMessages.php +src/contracts/src/Queue/ShouldQueueViaOutbox.php +src/bus/src/InteractsWithQueueOutbox.php +src/bus/src/Queueable.php +src/bus/src/Batchable.php +src/bus/src/Batch.php +src/bus/src/PendingBatch.php +src/foundation/src/Bus/PendingDispatch.php +src/queue/src/Concerns/InsertsDatabaseRows.php +src/queue/src/Concerns/StagesOutboxJobs.php +src/queue/src/Outbox/ + Concerns/PublishesOutboxMessagesSequentially.php + DatabaseOutboxRepository.php + OutboxBatchResult.php + OutboxDispatcher.php + OutboxMessage.php + OutboxRelay.php + OutboxWorker.php + OutboxWorkerOptions.php +src/queue/src/Console/ + ForgetOutboxCommand.php + ListOutboxCommand.php + MonitorOutboxCommand.php + OutboxTableCommand.php + RelayCommand.php + RetryOutboxCommand.php + stubs/outbox.stub +src/queue/src/Events/ + OutboxBatchDelivered.php + OutboxBatchDeliveryFailed.php + OutboxBusy.php +``` + +The tree lists the new mechanism and primary API surfaces, not every modified file. Integration also touches the existing queue drivers, `FailoverQueue`, pool proxies, Events `Dispatcher`/listener wrappers, `SendQueuedMailable`, `SendQueuedNotifications`, `BroadcastEvent`, Horizon `RedisQueue`/`JobPayload`, OpenTelemetry `QueueInstrumentation`, and the prerequisite files identified above. + +Do not create a listed type if it would only forward one call. Do not combine routing, SQL claims, network publication and daemon lifecycle into one god class. Responsibilities are: + +- base `Queue` plus small protected hooks: selection precedence, local/database fast paths, normal lifecycle events, default-expanded logical destination and driver option capture; +- `InsertsDatabaseRows`: reuse the existing binding-safe, transactionally atomic chunk operation and its existing queued-job insertion error; +- `OutboxDispatcher`: transaction/database ownership, preflight validation, staged single/bulk coordination, and rollback registration without containing SQL claim or broker publication logic; +- `DatabaseOutboxRepository`: multi-row insert, engine-specific claim, token-guarded delete/release, retry/forget/list/monitor queries; +- `PublishesOutboxMessagesSequentially`: the one shared generic fallback used by the relay and pooled proxy, partitioning future rows and returning accepted-prefix/deferred/failure outcomes without duplicating the algorithm; +- `OutboxRelay`: group claimed rows by target connection, choose optimized raw-batch publication or the shared generic raw fallback, and settle only explicit outcomes; +- `OutboxWorker`: stateless daemon loop, bounded task admission, round-robin database fairness, restart/signal/memory/time/empty rules, and ownership of the short-lived child-coroutine boundaries described below; +- command classes: input/output only. + +Container-cached outbox services remain immutable/stateless. Database capability detection stays on the live connection that owns the capability, without another service or cache. `OutboxMessage`, `OutboxBatchResult`, and `OutboxWorkerOptions` are immutable per-operation values created directly rather than container-resolved. Loop counters, current claims, stop reasons, and per-delivery data stay method-local. The outer concurrency task may retain only scalar/immutable values and result data: never a database connection, query builder, resolved queue instance, pool lease, or transport client. + +Resource ownership is a hard invariant: **no relay database or transport connection remains borrowed while the relay performs work that does not require it**. Explicit idle release does not cover retained write sessions, every custom driver, or arbitrary callbacks. Use fresh child-coroutine boundaries for (1) source claim, (2) each target-connection publication group, and (3) that group's settlement. `Waiter(-1)` with `waitForChildTermination: true` leaves phase duration to configured database/transport timeouts and ensures cancellation joins child cleanup. Each child uses a fresh context, returns materialized values and ends before the next phase. Request-side staging instead uses the exact already-borrowed business connection for atomicity. + +Reuse the prerequisite `DispatchLockContext` capture/snapshot lifecycle and existing hidden scalar unique metadata. Never persist live cache provenance or captured release closures, or create an outbox-owned lock registry. + +### Queue integration seam + +Refactor the single and bulk enqueue paths so the cheap outbox-selection decision happens before ordinary after-commit deferral. Read the connection default from the typed boolean cached when the queue receives its immutable config; when neither job nor connection selects outbox, do not resolve an outbox service. Once selected, complete these steps inside the existing guarded enqueue attempt: + +1. resolve/validate selection, database ownership, driver fast-path behavior, and logical destination before finalization; +2. retain dispatch ownership until storage succeeds; any preparatory snapshot is data only, not a registered rollback callback; +3. finalize the payload exactly once while the original job object exists, inside the guarded enqueue attempt; +4. run the narrow outbox-payload hook into a separate stored-payload value, decode and validate that envelope and its UUID string consistently on every database engine, and compute absolute `deliver_at` plus target-specific opaque options in that same guarded attempt; +5. raise normal `JobQueueing`; +6. insert on the selected connection; immediately after success, register independent snapshot rollback cleanup only if that exact connection still has an active transaction; +7. let the existing enqueue lifecycle accept the dispatch locks and raise `JobQueued` with the job UUID after insertion, or raise `JobQueueingFailed` for every non-cancellation exception within the guarded attempt. + +The existing `DispatchLockContext` / `PendingDispatch` lifecycle owns failures before queue acceptance, including payload serialization and queue resolution. Reuse it rather than adding an outbox cleanup path. A failure before a valid payload exists rethrows without fabricating a queue lifecycle event; a failure inside the guarded enqueue attempt follows the existing `JobQueueingFailed` and cancellation rules. Do not report an accepted job as failed when a later `JobQueued` listener throws. + +Generic lifecycle events always carry the same finalized payload string, before transport or outbox decoration. Keep the separately prepared stored payload for transport/storage and `handlePayloadPushed()`, without changing the lifecycle value. This preserves Sentry's exact-payload correlation and OTel's direct lookup. Single dispatch needs one extra local value; bulk members retain both values. SQS prepares stored bytes before byte-based chunking but keeps `JobQueueing` inside each attempted chunk. Outbox staging follows the same rule for `preparePayloadForOutbox()`; never substitute the stored Horizon envelope in its terminal events. + +`JobQueued` means the configured logical queue accepted the job into its durable path; like a database-queue insert inside a caller transaction, that acceptance can still roll back with the transaction. It must not be delayed to another process. This preserves the original job object for listeners and completes OpenTelemetry producer state in the dispatching coroutine. Relay must not emit a second normal enqueue lifecycle. + +Provide additive protected queue hooks such as: + +```php +protected function resolveOutboxQueue(?string $queue): ?string; +protected function preparePayloadForOutbox( + object|string $job, + string $payload, +): string; +protected function prepareOutboxOptions( + object|string $job, + ?string $queue, + string $payload, +): array; + +protected function availableAtForOutbox( + ?string $queue, + ?int $deliverAt, +): int; +``` + +First-party drivers expand missing queues to their configured logical default without forwarding or deriving physical keys/URLs. The base options hook returns an empty array; initial availability is `deliver_at ?? now` because raw publication has no cross-driver delay contract. Redis/database become eligible immediately. SQS captures group/deduplication while the object exists, without relative `DelaySeconds`, and computes initial eligibility using the effective destination's standard/FIFO behavior. These inspections do not replace the stored logical name. At publication, normal driver routing resolves that name once. Register independent storage-rollback snapshots through the existing connection-aware transaction callbacks; do not reuse the live-record lookup callback after acceptance. + +`Queue::setConfig()` caches the optional connection `outbox` boolean alongside the already retained immutable configuration; `QueueManager` already calls this once for every resolved concrete connection. Do not add another constructor argument to every driver/connector. Missing `outbox` remains `false` for existing custom configuration. Selection computes explicit state, marker, and the cached flag first; only a positive result checks the forwarding record. That record holds the job and an after-commit-handled boolean, set only by the explicit-owner callback. Base after-commit selection checks it only after its ordinary positive decision. Nested failover preserves a matching outer record. Keep the existing protected attempt method's signature; extract its child loop for direct and explicit-owner callback use. + +Base `Queue::bulk()` must preflight and partition outbox-selected jobs itself, stage its outbox partition with set-based owner-group inserts, then loop only the direct remainder. Beanstalkd and custom durable drivers that inherit the base method therefore get bulk staging without a driver override. Database, Redis, and SQS keep their optimized overrides but apply the same partition before their direct paths. + +Keep selection and staging in `Concerns\StagesOutboxJobs`. Reuse Redis/database's protected `prepareBatchJobs()` shape as the base batch-preparation hook; SQS delegates to its existing `prepareBatchMessages()` and Beanstalkd supplies its normal payload-hook destination. This preserves each driver's payload hooks without adding a parallel serialization path. + +Every implementation preserves order within each partition and groups staged rows by owning database. This prevents a later configuration error from partially staging an otherwise valid group. There is deliberately no ordering or atomicity guarantee between direct and outbox partitions, just as existing cross-connection bulk has none; callers requiring one atomic unit use one outbox owner. Finalize every owner group before its first insert, emit existing per-job queue lifecycle events, and use set-based storage. Each owner group is atomic across its insert chunks; distinct owner databases necessarily commit independently. A `PendingBatch` repository transaction provides one owner and rejects conflicting per-job choices. Preserve current `Bus::bulk()` semantics: it does not suddenly acquire unique/debounce locks or run `PreparesForDispatch`. + +For a `DatabaseBatchRepository`, `Batch::add()` validates declared non-null outbox owners after applying defaults and before its transaction or queue call. Compare resolved session identities with the repository's connection, not requested aliases; skip explicitly false `viaOutbox` choices. This covers staged, database and local targets without queue-side state. Resolve the repository owner lazily once per add; leave custom batch repositories unchanged. + +### Raw-batch publication SPI + +`PublishesOutboxMessages` is a narrow optional capability in `Hypervel\Contracts\Queue`, beside `ClearableQueue` and `IndexAwareQueue`, not a new application API. Its signature may reference the queue-owned `OutboxMessage` and `OutboxBatchResult` value types under the repository's lazy contract-signature dependency convention. + +```php +interface PublishesOutboxMessages +{ + /** @param list $messages */ + public function publishOutboxMessages(array $messages): OutboxBatchResult; +} +``` + +`OutboxMessage` carries row ID/job UUID/lease token/attempt, finalized payload, unforwarded logical queue, integer `deliverAt` (nullable), `eligibleAt` (pre-claim availability), `createdAt`, and opaque options. `OutboxBatchResult` contains arrays of unique accepted and deferred integer row IDs plus an optional failure; array keys do not affect settlement. Accepted and deferred IDs must be disjoint subsets of the input; a clean result accounts for every input through their union. Only messages with non-null `deliverAt` may be deferred; do not require that time to remain in the future during validation. Explicit transport rejects return confirmed IDs plus one aggregate exception containing the reject count, first reason and full response context; a later-chunk exception returns earlier confirmed IDs plus that exception. Cancellation is rethrown. Optimized implementations return no deferred IDs; sequential fallback uses them for future rows. Failed/unattempted rows are input minus accepted minus deferred and receive the same bounded failure text. Invalid results become an empty accepted/deferred result with a `LogicException`: retain every input row through ordinary token-guarded failure release/backoff and report the contract violation. Transport response IDs stay inside the driver. + +Use typed message/result boundaries and existing driver option handling; do not add a second transport validator. The generic fallback passes stored options through `pushRaw()`. Native publication errors and partial/ambiguous outcomes use the result contract above. + +Implement the SPI where it provides a real batch/resource benefit: + +- Redis: reuse `LuaScripts::bulk()` and its exact stored-count confirmation once per resolved destination, combining immediate/delayed entries and retaining existing Cluster-safe keys/script handling; preserve confirmed earlier destinations if a later call fails; +- database: one target-database transaction containing parameter-safe multi-row jobs-table inserts across logical queues, with each remaining delay converted to `available_at`; report success only after commit; +- SQS: reuse the 10-entry and 1,048,576-byte request limits, FIFO fields, SDK retry assumptions, and per-entry response mapping; derive overflow pointers from job UUID plus lease token so separate publication attempts never share cleanup ownership; +- pooled proxy: borrow once for the whole invocation, delegate when the borrowed queue implements the SPI, and otherwise invoke the shared sequential fallback on that same borrowed queue; never retain the lease; +- Horizon Redis: decorate the Redis batch path with `JobPending` before attempts and `JobPushed` only for accepted payloads. Record every confirmed destination's accepted IDs before draining post-push notifications. On an ordinary notification failure, stop before the next destination and return those accepted IDs plus the error; settlement deletes confirmed rows rather than republishing them. + +Do not add raw-delay methods to the general `Queue` contract. Custom durable drivers opt into staging through the documented base enqueue hooks; implementing the SPI is an optional publication optimization, and their connector remains otherwise unchanged. + +## Relay algorithm and failure handling + +### Claim + +- The coordinator waits for publication capacity, then claims through a fresh `Waiter` child with no copied request/transaction context. That child alone resolves the source database, completes the claim, returns immutable `OutboxMessage` values, and ends so `ConnectionResolver` releases the database lease before any broker work starts. Count actual claimed rows before starting their concurrent delivery task. Claims are serial per daemon, like `queue:work` pops; publication/settlement tasks and independent daemons remain concurrent. +- Claim at most the validated worker batch limit eligible rows ordered by `available_at`, then `id`. +- Eligibility is `available_at <= now`. A claim atomically sets one fresh UUID lease token, `available_at = now + retry_after`, and increments attempts. This moves live leases out of the hot eligibility range without a second timestamp/filter. Use an explicit transaction only for the separate locking-select/update path; never hold SQL locks during broker calls. +- PostgreSQL: one autocommitted CTE using `FOR UPDATE SKIP LOCKED` plus `UPDATE ... RETURNING`, returning the selected pre-update availability as `eligibleAt` with the updated token/attempt. No extra BEGIN/COMMIT round trips. Updating the indexed timestamp trades HOT-update eligibility for avoiding repeated live-lease scans; the measured comparison supports retaining this design without an additional reservation column. +- Supported MySQL/MariaDB/Vitess: `SELECT ... FOR UPDATE SKIP LOCKED`, one guarded bulk update, and return the already selected rows with token/time/attempt fields reflected in memory rather than issuing another select. +- SQLite/engines without `SKIP LOCKED`: select an ordered, limited list of candidate IDs and their availability, perform one autocommitted atomic eligibility-guarded token/expiry update restricted to those IDs, then load full rows restricted to those IDs bearing that token. No encompassing transaction is needed: the token identifies this claim even if another claimant intervenes or reclaims an expired lease. Preserve original eligibility from the candidates. The second read distinguishes rows actually won; another claimant can cause an empty/partial result, not a double claim. SQLite serializes writers, so additional relays improve availability rather than write throughput. +- Normalize every returned claim to its pre-claim `(eligibleAt, id)` order; `UPDATE ... RETURNING` and `WHERE IN` do not promise order, and the newly shared lease deadline is not the original eligibility. +- Never return a connection, builder, cursor, lazy collection, or pool-backed object from the claim child. Materialize and validate the bounded result before it ends. +- Reuse `PdoConnection::lockForPopping()` and `maxBindings()` for PDO-backed connections. Keep base connection types driver-neutral and follow DatabaseQueue's non-PDO lock/binding fallback; custom drivers are not required to expose PDO methods. Existing live-session capabilities cover engine variants; do not add another detector, contract or cache. + +### Publish and settle + +- Group a claim by target connection, preserving order within each group. The driver partitions for existing Redis destination scripts, SQS URL/count/bytes, or database binding limits. This permits multi-row database inserts across queues and one pooled Beanstalk lease per connection batch without replacing existing driver mechanisms. +- Publish each group in its own fresh child. Only that child resolves the queue, borrows its transport and returns a materialized result. Use the optional raw-batch contract or shared sequential fallback. Like `queue:retry`, stored rows honor the recorded connection's current configuration, including an operator deliberately changing it to sync/null; add no driver-kind rejection. Document draining before configuration changes that discard durability or invalidate stored transport options. Native send errors retain/back off affected rows. +- The shared fallback moves rows whose `deliver_at` is still in the future into the result's deferred set without calling `pushRaw()`, then publishes the due rows in order and returns the accepted prefix plus any exception that stopped it. `QueuePoolProxy` invokes this same operation while its one borrowed queue is live when that queue lacks the SPI. Settlement releases deferred rows token-safely with `available_at = deliver_at`, restores the claim's attempt increment, and leaves `last_error` untouched. Deferral is not a publication failure: it does not occupy the task's failure-sleep slot or emit `OutboxBatchDeliveryFailed`. +- The driver's existing publish result is the acknowledgment—Redis/database replies, SQS success/failure entries, and Beanstalk IDs. Add no probe or extra broker round trip. Validate results before settlement using the raw-batch contract above; invalid results release every input row into normal failure backoff without deleting any. +- Settle each completed group immediately in a fresh child and short source transaction: delete accepted rows using `id IN (...)` plus `lease_token`; clear deferred tokens, restore availability to `deliver_at`, undo the claim's attempt increment and preserve `last_error`; release failed/unattempted rows with one token-guarded update per binding-safe chunk. Choose one jittered deadline per capped retry bucket using `RETRY_DELAYS`, random 50%-100% jitter and a minimum one-second delay. A fixed `CASE WHEN attempts <= 1 THEN ? ... ELSE ? END` binds seven deadlines, not per-row cases; clear the token and bind one bounded error text. Only ID lists require size-dependent chunking. If fewer rows match, retain successful settlement and leave newer owners untouched; report ownership loss once. End the child before the next group. +- A failed connection group does not prevent publication/settlement of the other groups in the same claim. Collect and report their errors after all non-cancelled groups have been attempted, identifying affected row IDs, job UUIDs, and payload display names; decode display names only on failure so successful relay batches pay no diagnostic parsing cost. Cancellation stops further publication and leaves unsettled leases to expire. +- After any result failure, keep that relay task's concurrency slot occupied for its positive `--sleep` interval. Row backoff alone cannot protect a misconfigured/unavailable target from a large backlog of first-attempt rows; bounded occupied slots throttle further claims without distributed circuit-breaker state. +- If broker acceptance succeeds but delete fails or the process crashes, lease expiry makes the row retryable; a duplicate is correct. If the network outcome is ambiguous, retain and retry. +- Guard all settlement by lease token so a stale worker cannot delete/release rows reclaimed by another worker. +- Require `retry_after` to exceed the maximum expected lifetime of one claimed batch, including every connection-group publication and SDK retry. Do not add a heartbeat coroutine or callback machinery; an exceptionally slow cycle may produce a duplicate after lease expiry but cannot lose the row. +- On graceful stop, stop claiming new work and let active completed calls settle before joining them. Cancellation is never converted to an ordinary failure; an unsettled lease expires naturally. +- Parent cancellation while waiting for a phase propagates as `CanceledException`; a child that independently cancels while its parent remains active follows `Waiter`'s existing `ChildCancellationException` contract and is reported as a phase failure. Do not catch either merely to log and continue claiming. +- Continue retries indefinitely. Backoff prevents poison rows or outages from hot-looping, ordering plus `SKIP LOCKED` prevents one row from blocking later work, and monitor alerts surface permanent failures. + +### Lock cleanup + +Transactional rollback uses the prerequisite independent snapshot, including its captured release context. Existing deferred-dispatch callbacks retain their normal ownership lifecycle. Successful publication leaves locks for normal job processing. + +`queue:outbox-forget` reads only the existing hidden cache-store/key/owner scalars with object deserialization disabled; never hydrate arbitrary context or unserialize the queued command. Delete the non-live row first, then release the owner-matched unique lock. No unique key means no cleanup; otherwise require nonempty string store/key/owner and an owner-capable, resolvable store. A null store name can describe an ad hoc `uniqueVia()` repository, not necessarily the default store: never guess or force-release. Keep cleanup in a protected command method receiving the stored envelope so integrations can resolve custom repositories or establish a context-dependent cache namespace without copying SQL or changing the relay. Commands already resolve through the container; no new registry is needed. Cache and SQL cannot commit atomically: unresolved/malformed ownership or post-delete cache failure is reported with the job UUID and explicit notice that the row was deleted and lock cleanup failed; never restore or republish deliberately forgotten work. Debounce tokens have bounded TTL and need no explicit forget cleanup. No outbox lock abstraction or persisted release closure. + +## Worker and operations + +Register these commands in `QueueServiceProvider`: + +- `queue:relay {database?}` — long-running relay plus `--once`, `--stop-when-empty`, `--batch=100`, `--sleep=1`, `--concurrency=1`, `--max-jobs=0`, `--max-time=0`, and `--memory=128`; validate batch 1–1,000 and positive sleep/concurrency. Sleep accepts fractions like `queue:work`. `--once` makes one synchronous round-robin pass, at most one bounded claim per configured/pinned database, with no empty/outage sleep. `--stop-when-empty` joins active tasks and exits after a complete source pass finds no eligible row; delayed/backoff rows do not keep it alive. `--max-jobs` counts claimed rows regardless of result; cap every final claim to the remaining allowance; +- `queue:outbox {database?}` — pending-row diagnostics by configured database, ID, UUID, job display name read from the bounded payload result, target, attempts, lease state, age, and last error, ordered by ID and bounded by validated `--limit=100` (maximum 1,000), with `--json` matching `queue:failed`; otherwise visit all configured sources. Tables format availability as a date-time and abbreviate errors to their first line; JSON retains epoch values and complete stored errors. Never render payloads, transport options, context, or lock-owner tokens; +- `queue:outbox-monitor {database?}` — total, eligible, delayed, leased, and retrying counts plus the oldest currently eligible row's age, with explicit `--max` / `--age` alert thresholds and `--json` output for one/all configured databases; use one portable aggregate query with conditional counts and `MIN(CASE WHEN available_at <= ? THEN available_at END)` so counts and age share one read, and emit `OutboxBusy` like the existing `queue:monitor` command without another index or payload scan. Eligibility uses the claim predicate, and age starts at `available_at`, so intentionally delayed jobs do not create false alerts; +- `queue:outbox-retry {uuid?*}` — targets only rows with `attempts > 0`, clears errors/leases, and makes selected rows (or explicit `all`) immediately available without erasing attempt history; accept `--database=` to pin a configured source, otherwise visit all sources. `all` snapshots each source's maximum ID among retryable rows and updates through that ceiling in bounded primary-key cursor chunks; an explicit UUID found on multiple unpinned sources fails rather than updating an ambiguous match. A never-attempted row waiting for its intended delivery time is not retryable; a previously failed row remains retryable regardless of its current backoff. Always reject live leases; +- `queue:outbox-forget {uuid}` — confirmation, live-lease rejection, token-safe delete, and owner-matched unique-lock cleanup from the payload metadata; accept `--database=` to pin a configured source and fail rather than guess if an unpinned UUID exists on more than one source. Use `ConfirmableTrait` with confirmation required in every environment; `--force` skips only the prompt, never the live-lease guard, because an in-flight publication cannot be recalled; +- `make:queue-outbox-table` / `queue:outbox-table` — migration generation matching existing queue table commands. + +The worker follows the relevant `queue:work` operational conventions: `queue:restart`, bounded coroutine concurrency, POSIX signals, memory/max-time exits, exception reporting, and no swallowed cancellation. It does not consult maintenance mode: relaying only moves committed jobs to their queue, while normal workers already control whether those jobs execute. One coordinator admits work in configured-database round-robin order only when `WaitConcurrent` has a free slot, so counters, source fairness, stop decisions, and the remaining `--max-jobs` allowance stay local and exact. + +- Cap each claim to `min(batch, maxJobs - admitted)` when limited. Wait for capacity with a bounded timeout so stop/restart checks remain responsive; do not reserve allowances for parallel claims. +- A failed claim is reported and counts as a no-work visit. Skip that source for `--sleep` using a coordinator-local retry-deadline map; healthy sources continue. After a complete consecutive no-work pass, sleep or, with `--stop-when-empty`, join admitted work and exit without another pass. Failed/cooling sources count as no-work; delayed/backoff rows need not drain. `--once` visits each source once without cooldown or sleep. +- Use the shared `SignalRegistry` with a per-run owner and local stop/pause flags; unregister in `finally`. Initial/subsequent restart-cache reads and lifecycle callbacks run in fresh children, since database-backed cache stores also borrow connections. Memory/time checks remain local. +- `OutboxRelay::deliver()` returns true on successful publication/settlement, including non-failure deferral; false triggers slot-held failure sleep. Tasks rethrow `CanceledException` and capture any other escaping `Throwable` into one first-failure reference. The coordinator stops admission, joins owned tasks and propagates the failure. Coordinator cancellation cancels and joins owned tasks before propagating its original exception; no separate cancelling flag or run-state class. + +Retry/forget load mutation targets from the writer using existing query-builder support, not replica reads or a separately borrowed connection. Recheck token/live-lease predicates in the modifying statement so a concurrent claim cannot be reset/deleted. A live lease is `lease_token IS NOT NULL AND available_at > now`; expired tokens do not prevent recovery. Monitor states partition the total: eligible (`available_at <= now`), live leased, tokenless delayed (no attempts), or tokenless retrying (attempted and still in backoff), matching the listing. + +Each admitted task owns no pooled resource. It receives materialized claims and uses a fresh child for each publication and settlement, passing only immutable messages/results. Guarded delivery events/reporting also run in a short-lived child because application callbacks may resolve clients; skip this child when there is neither a listener nor an error. Use the exception handler's context array for row diagnostics. A listener failure is reported once without triggering delivery backoff; a reporter failure propagates. Neither alters settled rows, emits the opposite delivery event or republishes accepted work. When publication and settlement both fail, report both and retain the publication error on the event. Use fakeable `Sleep::usleep()` for fractional empty/failure delays and the existing capacity-wait primitive instead of polling busy slots. Graceful stop ceases admission and joins tasks so completed publication can settle. Supervise this dedicated process; scheduled `--once` is also supported. Size source/transport pools for relay concurrency and any other users, allowing a short-lived extra slot for restart-cache/event/reporting work on the same pool. Never make the relay a queued job dependent on the broker it must recover. + +Horizon continues supervising and visualizing Redis jobs only after relay publishes them. Do not add relational outbox polling, migrations, or supervisor machinery to Horizon. + +Expose one guarded event per completed connection publication batch, with source database, target connection, accepted/failed counts, publish duration, oldest eligible age, and an optional failure, but no payload bodies: `OutboxBatchDelivered` only when every attempted row was published and settled, otherwise `OutboxBatchDeliveryFailed` for a result failure, partial acceptance, invalid result, lease-ownership loss, or settlement failure. Rows deferred before the generic fallback are excluded; a group containing only future rows emits no delivery event. The OpenTelemetry component consumes these events and records delivery count, eligible-to-publish latency, and publish duration. Failed-batch counts come from duration observations tagged with `error.type`, not a duplicate counter. Latency uses whole-second buckets and no error-type attribute. Rely on the monitor command for backlog/eligible-age gauges so ordinary dispatch does not issue count queries. + +## Horizon outbox integration + +Use the existing `JobPayload` lifecycle through its two semantic operations: + +- `prepareForQueueing($job)` while the object exists: type, tags, and silenced state; +- `markAsPushed()` at actual Redis publication: `pushedAt` only. + +Preserve `JobPayload::prepare($job)` as a single-pass combination of classification and timestamp values, not two encoding calls. Direct immediate/delayed Horizon dispatch calls it once through `preparePayloadForPush()` during the actual enqueue attempt, so after-commit timing stays correct. `handlePayloadPushed()` raises Horizon's post-storage event after acceptance. External `pushRaw()` remains stateless and uses the single-pass preparation method with `null`, preserving already-stored classification and refreshing only `pushedAt`. Base Redis's normal `pushRaw()` extension routing remains unchanged. + +Add a narrow protected `preparePayloadForOutbox()` hook to base Queue that is called only after generic payload finalization on selected outbox paths and is otherwise a no-op. Horizon Redis overrides it to call `prepareForQueueing($job)` once before storage, without `pushedAt` or `JobPending`. Its raw-batch publication calls `markAsPushed()` once per payload immediately before Redis publication. Preserve job `uuid` / Horizon `id`; emit `JobPending` before each attempted publication and `JobPushed` only for explicit successes. The staging hook receives the original job rather than classifying with `null`. + +## Documentation cleanup + +Update in the same change so no stale behavior survives: + +- `src/docs/queue-outbox.md`: canonical setup, configuration, API precedence, transaction ownership, driver/custom-driver matrix, bulk/batches, `dispatchSync`/after-response limitations, delays, UUIDs, connection capture versus publication-time queue forwarding, deliberate driver changes, at-least-once/idempotency and best-effort dispatch ordering, SQS overflow attempt ownership/orphans, worker deployment/pool sizing, operations, Horizon and database fast paths; +- `src/docs/documentation.md`: add the new guide to navigation; +- `src/docs/opentelemetry.md`: add the three outbox metrics and their defaults to the queue metrics reference; explain failed-batch counts through duration observations with `error.type`; +- `src/docs/queues.md`: add only a concise transactional-outbox introduction, when it is useful, one basic `viaOutbox()` example, and a link to `queue-outbox.md`; do not duplicate its setup, guarantees, driver matrix, or operations reference; +- `src/docs/events.md`, `mail.md`, `notifications.md`, and `broadcasting.md`: beside their existing after-commit guidance, add one concise example/cross-link showing producer selection propagating to its wrapper job. Queued listeners are inspected without constructing them: use the marker or public property defaults, not constructor-fluent calls. When a listener uses `Queueable` or `Batchable`, use the marker; PHP rejects redeclaring their outbox properties with defaults. Application-constructed producers can use the fluent methods. Link to `queue-outbox.md` for guarantees rather than duplicating them; +- `src/docs/porting-from-laravel.md` and database README: explain exact resolved transaction identity for extensions observing role-qualified records. Do not add outbox catalogue entries: its opt-in APIs require no Laravel adaptation. Preserve Queue/Bus/Horizon README structure without adding additive-feature differences; +- `/home/binaryfire/workspace/contrib/hypervel/hypervel`: match Foundation's outbox configuration and the Testbench schema in the next application migration; +- configuration comments and migration command help: exact guarantees and required worker/table setup. + +Clarify in listener/notification guidance that `viaOutbox()` is a fluent setter, not a getter hook like `viaConnection()` or `viaQueues()`. + +Never imply exactly-once delivery, broker participation in the SQL transaction, or durable behavior for local drivers. Remove a comment only when the outbox implementation makes it stale; final comments explain only non-obvious invariants. + +## Testing plan + +### Framework prerequisites + +- Broadcast extraction skips uninitialized Queueable/application properties while preserving initialized nulls and formatted values; retain existing custom-payload coverage. +- Distinct base/`::write` sessions with simultaneous transactions and independent commit/rollback callbacks; nested savepoints, shared in-memory SQLite aliases, Foundation wrapping/context synchronization and Pennant rollback invalidation use resolved ownership. Do not use shared in-memory SQLite as proof of separate-session behavior. +- Context-dependent cache fixture backed by `WorkerArrayStore`: acquire in context A, release record/snapshot in B, remove A's key and preserve B's same logical key. Cover unique and debounce separately, with a second registered wrapper proving composition order and restoration. A throwing capture callback acquires neither lock nor token. One real CoroutineQueue closing-timer regression covers release outside the acquiring coroutine. No repeated tests per caller or cache wrapper: the hook does not branch on wrapper types. +- DatabaseQueue storage rollback releases its snapshot after acceptance, including nested rollback/deadlock retry; broker-accepted jobs are not released by unrelated rollback. Preserve existing deferred ownership coverage. +- With no applicable transaction, a local child failing before acceptance retains dispatch-lock ownership for failover's next child; local after-commit return values remain unchanged. +- Manager-resolved custom and memoized repositories retain their cache names, including memo-tagged wrappers, while preserving creator-supplied names and memo event suppression. Missing-model unique-job cleanup releases the actual named memo/custom store's lock; ad hoc unnamed repositories retain their existing semantics. +- SQS later-finalizer/queueing-listener failures finish unaccepted producer state without falsifying confirmed chunks; preserve cancellation and per-chunk queueing order. Custom job `queue()` dispatch through SQS bulk retains/delegates its acquired locks and releases only unaccepted work. Success-listener failure preserves all accepted entries, cleans explicit-reject pointers and prevents later sends. Reuse/extend affected tests. +- Every optimized batch driver drains successful terminal notifications despite an ordinary listener failure; cover failure-notification draining once at its shared boundary. Cancellation remains immediate. Do not duplicate these tests as a telemetry matrix. +- Horizon combined preparation encodes once and preserves classification/pushed timing/customization. A single/delayed post-push listener error retains the accepted lock, attempts `JobQueued` and emits no `JobQueueingFailed`; bulk also attempts that member's `JobQueued`. Compare exact generic lifecycle payload strings for direct/delayed/bulk and failed bulk to prevent correlation drift. Reuse existing direct/retry/delayed tests and remove only the obsolete last-pushed state test when its mechanism is removed. + +### Contract and routing tests + +- `Queueable`/`Batchable` shared fluent state, pending dispatch/batch APIs, jobs that use only `Batchable`, jobs added to an existing batch, explicit per-job overrides, and the absence of chain-specific propagation/API state; +- precedence across explicit true/false, marker, and connection default; `beforeCommit`/`afterCommit` remain independent direct-publish controls; wrapper jobs preserve explicit false over the marker and null falling through to the marker; and `dispatchSync`/`dispatchAfterResponse` retain their existing local/deferred semantics rather than pretending to provide pre-deferral durability; +- a custom durable transport without the raw-batch SPI stages normally and relays through the sequential raw fallback; a custom queue that bypasses the base lifecycle remains direct and is documented as not outbox-capable; preflight failure releases only locks acquired by the current dispatch; +- failover selection degrades to after-commit without schema access; child defaults are suppressed only for the forwarded job and restored afterwards. Nested failover honors explicit owner B's commit while A remains open, including lock ownership across a failed child; ordinary reentry remains intact. Independent jobs dispatched by a sync handler, including same-handler string continuations, retain outbox selection; +- `pushRaw()` remains direct even on a connection whose default enables outbox; +- local/null/fake/database fast paths never query the outbox; exact local transaction selection is honored; expanded defaults stay fixed and queue-name forwarding runs once, including `A → B`, `B → C` and a changed forward before publication; +- staged single dispatch returns the same job UUID indexed on its row; direct driver return values remain unchanged. + +### Transaction tests + +- no transaction, commit, rollback, nested savepoint commit/rollback, and rollback lock release; +- exactly one active connection, nested records on it, multiple active owners with/without explicit selection, and unconfigured-source rejection; a `::write` owner covered by its configured base stages on the exact alias session and is drained/countable once through the base source; +- a read-role owner is rejected before insertion with no staged row and existing dispatch-lock cleanup; cover active ownership and explicit no-transaction selection within the owner tests, not a separate suite; +- same database queue connection inserts directly and rolls back atomically; a differently named/session connection stages and relays; +- bulk is all-or-nothing across insert chunks within each owner group both with no outer transaction and when an outer caller catches the exception after nested-savepoint rollback; separate owners and direct/outbox partitions are independently atomic and preserve only internal order; batch counters and outbox rows commit/roll back together; +- outbox preflight, payload-finalizer, option, and storage failures follow the existing lifecycle: every unaccepted single/bulk member raises `JobQueueingFailed`, dispatch-owned locks remain exact-owner safe, cancellation remains cancellation, and an accepted job is never reported as failed because a later `JobQueued` listener throws; +- rollback snapshots register on the selected owner and release only that dispatch's locks even after acceptance removes its live record; successful publication preserves the normal queued-job lock lifetime. + +### Payload and API integration tests + +- queued jobs, closures, listeners, mailables, notifications, broadcasts, encrypted jobs, serialized models, custom payload hooks, OpenTelemetry propagation, unique jobs, debounced jobs, delayed jobs, batches, nested batched chains, and ordinary chains; wrapper producers propagate marker/explicit selection and database without changing their public APIs, while absent source state leaves the wrapper's typed fields uninitialized; +- payload creation/finalization occurs once; malformed final envelopes or missing/invalid UUIDs fail uniformly before storage on every database engine; an outbox bulk finalizer failure leaves no earlier producer state open; opaque payload bytes remain unchanged in SQL and UUID/command bytes remain stable over relay retries apart from documented delivery-time transport metadata; +- SQS FIFO group/deduplication callback is evaluated once at staging; standard delays longer than 900 seconds wait until the broker window, FIFO delays wait until due, and relay latency/backoff never rewrites immutable `deliver_at`. Parameterized staging/publication cases distinguish no timer from an already-due requested timer: standard publication omits `DelaySeconds` for null and uses zero after a requested timer expires; FIFO always omits it; +- `Bus::bulk()` uses bounded multi-row inserts and preserves its existing lack of unique/debounce/preparation side effects; Beanstalkd and a custom durable driver inheriting base `Queue::bulk()` each stage one set-based insert per owner group rather than one insert per job, while their direct partitions retain the base loop; +- fake assertions work without schema setup or outbox service resolution; an unselected job does not serialize the uninitialized outbox state. + +### Persistence and concurrency tests + +- schema/migration command/config validation, byte-preserving payloads, opaque nullable text options, public UUID lookup and the hot composite index; staged time fields equal the frozen framework clock's epoch values, with one post-2038 fixture rather than a timezone/type matrix; normalize past dates and never truncate a future fractional due time early; +- indexed eligibility, pre-claim order/latency, attempt increment, lease expiry, token-guarded stale settlement, capped-bucket jitter bounds and last-error truncation; +- two/many concurrent relays do not claim the same live row; a killed claimant is recoverable after `retry_after`; +- instrumented tiny database/queue pools prove claim connection release before transport blocking, transport release before settlement, settlement release before another target or failure sleep, and bounded forward progress under concurrency; phase boundaries return no pool-backed objects and the long-lived coordinator/concurrency task retains none; +- PostgreSQL CTE/returning, MySQL/MariaDB skip-locked claim and competing fallback claims/full-row reload, consuming existing connection capabilities rather than retesting their engine-version detection; custom non-PDO paths do not assume PDO methods; +- query-count and bounded-fetch assertions cover one-statement PostgreSQL, two-statement MySQL/MariaDB and three-statement fallback claims, excluding ordinary transaction control. Verify limits in the SQL path, not just a truncated returned array. Do not assert a specific optimizer plan or pretend a mock proves live Vitess behavior; inspect representative query plans during approved benchmarks. + +Place engine-specific cases under `tests/Integration/Queue/Database/{Postgres|MySql|MariaDb|Sqlite}` so the existing database workflow discovers them. Live Redis cases stay under `tests/Integration/Queue/Redis` and use `InteractsWithRedis`; do not pretend SQLite or a mock validates PostgreSQL/MySQL locking or Redis behavior. + +### Transport tests + +- Redis immediate/delayed Lua batching, exact confirmation and ambiguous failure after earlier confirmed destinations; +- database queue multi-row insertion across logical queues and remaining-delay conversion; +- SQS 10-entry/1,048,576-byte chunking, standard/FIFO outbox delay behavior, group/deduplication, aggregate explicit-reject failure, SDK exception after confirmed earlier chunks, and partial accepted-ID results; overflow storage reuses one pointer inside SDK retries but uses independent job-UUID/lease-token pointers across outbox attempts, cleanup for a later explicitly rejected attempt cannot remove an earlier ambiguous attempt's payload, confirmed unpublished pointers are cleaned up, and ambiguous attempts are retained; +- recorded connections honor current driver configuration and native transport failures retain rows; invalid/overlapping result IDs and deferral without a delivery time release the whole call into failure backoff, sibling groups continue independently, and stale settlement leaves newer owners untouched; SPI-to-non-SPI replacement before `deliver_at` defers without publication, failure telemetry, attempt inflation or error mutation; +- shared generic raw fallback accepted-prefix/deferred/ambiguous-failure results; Beanstalk uses it with one pooled lease per connection batch and switches tubes correctly; +- only confirmed accepted IDs are deleted; every failed/unattempted row is released with the aggregate failure, while future fallback rows follow the non-failure deferral path; acceptance followed by delete failure produces a duplicate with the same job UUID and no lost row; +- confirm settlement adds no broker probe beyond the normal publish call/result. + +### Worker, Horizon, and telemetry tests + +- round-robin sources, positive fractional empty/outage sleeps, backlog drain, once/stop/max-jobs/memory/restart/signal/cancellation behavior, maintenance-mode independence and exception reporting; +- list bounds/display names/source selection; monitor counts distinguish live leases from future tokenless rows and alert on eligible age; retry/forget use writer reads and atomic live-lease guards, bounded `all`, confirmation/source pinning and cross-source UUID disambiguation; forget releases exact scalar unique ownership, exposes its narrow cleanup override and reports malformed metadata. Retain bounded debounce expiry without adding cleanup machinery; +- Horizon outbox paths preserve type/tags/silenced, refresh only `pushedAt` and emit Pending/Pushed correctly on partial results; prerequisite coverage owns shared direct/retry/delay regressions; +- OpenTelemetry producer state terminates during staging; relay batch instruments do not leak state or duplicate normal enqueue metrics; +- delivery-listener and exception-reporter failures release their child-coroutine resources and cannot alter an already-settled row, emit the opposite delivery event, or cause a republish. + +### Performance gates + +- Automated tests pin unselected dispatch's service resolution, serialized fields and payload transformations, set-based staging, bounded claim statements, transport batching and settlement statements. Use multiple rows per settlement outcome so query-count assertions detect per-row regressions. Tiny-pool tests own phase-release correctness; do not duplicate their instrumentation in benchmarks. +- Retain a developer-only harness under `tests/Benchmarks/QueueOutbox/`, following the existing Database and RateLimiter harnesses. Use `benchmark.php`, a README and cohesive fixtures only where needed. No production hooks, dependencies, Artisan command or CI timing thresholds. Raw reports and disposable data stay outside Git. +- Cases: direct Redis dispatch for baseline/current CPU and heap comparison; transactional single and bulk staging (including 1k/10k jobs); real-worker relay to Redis/database queues; concurrent claim/accepted-settlement/refill cycles against mixed eligible, delayed, live-leased and retrying backlogs. Seeded live leases use an explicit `--leased` count, defaulting to batch × concurrency, rather than growing with backlog size. Include refill in cycle throughput and report claim latency separately, including checkout and child cleanup; use pool capacity at least equal to concurrency for claim-design comparisons. Seed deadlines beyond the bounded run. Report the in-memory restart-cache choice so relay timings do not imply external cache I/O. Use plain queued jobs so the direct case loads in a pre-feature worktree. +- Create isolated randomly named tables from shipped migration stubs and use a unique Redis queue; remove only owned resources in `finally`, never flush a shared Redis database. SQLite is functional-only evidence. Verify exact row/target counts and no remaining borrowed sessions on every run. Tiny invocations validate the harness while the machine is busy; draw no performance conclusions from them. +- Warm up outside timing; record wall time, process CPU, peak/ending heap and GC deltas, payload size and non-secret runtime/backend settings. Use fresh-process samples and alternate baseline/current worktrees with independent dependencies. Obtain fresh idle-machine confirmation before performance measurements. +- Keep executing query plans opt-in and outside timing: capture actual claim SQL once, then run PostgreSQL `EXPLAIN (ANALYZE, BUFFERS, WAL)`, MySQL `EXPLAIN ANALYZE` on its locking select, MariaDB `ANALYZE FORMAT=JSON`, or SQLite `EXPLAIN QUERY PLAN`. Roll back executing diagnostic statements. Document supported engine versions; no optimizer assertions or capability framework. +- Retain only the production claim path in the harness. The rejected reservation comparison and measured results belong in `tests/Benchmarks/QueueOutbox/results.md`: filtering live leases increased scan work without a consistent statement-WAL saving. Whole-cycle `pg_current_wal_insert_lsn()` differences are cluster-wide workload WAL, including refill/settlement and background activity—not isolated index writes. Do not force checkpoints, change fillfactor/autovacuum or reset statistics. Investigate substantive contrary evidence before changing production design. + +Run focused files immediately as they are added. Verify coherent slices with formatter fixes, static analysis and affected package/integration suites. Use full verification for framework-wide changes; after localized corrections rerun only affected checks and finish any stages not yet completed. Do not repeat completed whole-framework verification merely to obtain one green invocation on the latest revision. + +```bash +cd /home/binaryfire/workspace/contrib/hypervel/components-queue-outbox +composer test:parallel -- tests/Queue --filter Outbox +composer test:parallel -- tests/Bus --filter Outbox +composer test:parallel -- tests/Horizon --filter Outbox + +cd /home/binaryfire/workspace/contrib/hypervel/hypervel +composer cs-fix +composer analyse +composer test + +``` + +The filters are examples, not substitutes for covering other affected behavior. `composer fix` includes formatting, analysis, parallel tests, `composer test:testbench` and dogfood suites; do not repeat those stages separately when already completed. When verifying a Testbench change independently, run `composer test:testbench` directly. Reuse unchanged skeleton verification and confirm Foundation configuration/schema parity. + +If implementation uses a components worktree, copy `.env` from the main components checkout and run `composer install` in the worktree; never symlink `vendor` or Composer state. + +## Implementation order and acceptance + +1. Complete and verify the independently useful framework prerequisites in dependency order, then add marker/shared fluent state and selection tests. +2. Add config, migration command/schema, immutable value objects, `DatabaseOutboxRepository`, and transaction/multi-database tests. +3. Integrate single/bulk staging, existing fast paths, SQS delay selection, forwarding and the prerequisite snapshot lifecycle. +4. Add the raw-batch SPI and first-party driver implementations, including pooled and Horizon paths. +5. Add claim/publication/settlement resource scopes, relay/daemon and operator commands, guarded events, and OpenTelemetry listeners. +6. Complete canonical docs, the exact-owner compatibility note, retained benchmark harness, measurement results and scoped verification above. Investigate measured bottlenecks through the same review workflow. + +Before completion, re-read every changed file and confirm: + +- no mutable cross-coroutine state, pool lease retained beyond its owning phase or established queued-job terminal lifecycle, container swap, hidden environment branch, or network call under a SQL lock; +- source claim, each target publication, settlement, and arbitrary listener/reporter execution end their borrowing coroutine before unrelated work, another target, or throttling sleep begins; +- every crash boundary retains a retryable row or has a confirmed target acceptance, with duplicates preferred over loss; +- ordinary queue, batch, chain, Horizon, and OpenTelemetry behavior remains coherent; +- disabled/local/database-fast-path dispatch performs no outbox I/O; +- Foundation and application-skeleton queue defaults and outbox migrations agree; +- `queue-outbox.md` is the single detailed outbox reference and shorter queue/producer mentions link to it without duplicating guarantees; +- all configured owners are drainable, exact transaction identity is preserved, queue-name forwarding occurs once, current connection configuration is honored, and all SQL settlement is lease-token guarded; +- public names read like first-party Laravel queue APIs while internals remain Hypervel-native; +- no dead status/history/dead-letter/CDC/inbox abstraction, stale comment, duplicate README line, or superseded documentation remains. diff --git a/src/broadcasting/src/BroadcastEvent.php b/src/broadcasting/src/BroadcastEvent.php index 8a0552c80e..e434a3d0b4 100644 --- a/src/broadcasting/src/BroadcastEvent.php +++ b/src/broadcasting/src/BroadcastEvent.php @@ -7,6 +7,7 @@ use Hypervel\Bus\Queueable; use Hypervel\Contracts\Broadcasting\Factory as BroadcastingFactory; use Hypervel\Contracts\Queue\ShouldQueue; +use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; use Hypervel\Contracts\Support\Arrayable; use Hypervel\Queue\Attributes\Backoff; use Hypervel\Queue\Attributes\DeleteWhenMissingModels; @@ -67,6 +68,17 @@ public function __construct(mixed $event) $this->timeout = $this->getAttributeValue($event, Timeout::class, 'timeout'); $this->backoff = $this->getAttributeValue($event, Backoff::class, 'backoff'); $this->afterCommit = property_exists($event, 'afterCommit') ? $event->afterCommit : null; + + if (isset($event->viaOutbox)) { + $this->viaOutbox = $event->viaOutbox; + } elseif ($event instanceof ShouldQueueViaOutbox) { + $this->viaOutbox = true; + } + + if (isset($event->outboxDatabase)) { + $this->outboxDatabase = $event->outboxDatabase; + } + $this->maxExceptions = $this->getAttributeValue($event, MaxExceptions::class, 'maxExceptions'); $this->deleteWhenMissingModels = $this->getAttributeValue($event, DeleteWhenMissingModels::class, 'deleteWhenMissingModels', $this->deleteWhenMissingModels); } @@ -115,6 +127,10 @@ protected function getPayloadFromEvent(mixed $event): array $payload = []; foreach ((new ReflectionClass($event))->getProperties(ReflectionProperty::IS_PUBLIC) as $property) { + if (! $property->isInitialized($event)) { + continue; + } + $payload[$property->getName()] = $this->formatProperty($property->getValue($event)); } diff --git a/src/bus/src/Batch.php b/src/bus/src/Batch.php index 79735c2b06..21f8be2a7e 100644 --- a/src/bus/src/Batch.php +++ b/src/bus/src/Batch.php @@ -12,12 +12,14 @@ use Hypervel\Contracts\Events\Dispatcher; use Hypervel\Contracts\Queue\Factory as QueueFactory; use Hypervel\Contracts\Support\Arrayable; +use Hypervel\Database\ConnectionResolverInterface; use Hypervel\Queue\CallQueuedClosure; use Hypervel\Support\Arr; use Hypervel\Support\CarbonImmutable; use Hypervel\Support\Collection; use Hypervel\Support\Enumerable; use JsonSerializable; +use LogicException; use Throwable; class Batch implements Arrayable, JsonSerializable @@ -57,8 +59,10 @@ public function fresh(): ?Batch public function add(array|object $jobs): ?Batch { $count = 0; + $owner = null; + $resolver = null; - $jobs = Collection::wrap($jobs)->map(function ($job) use (&$count) { + $jobs = Collection::wrap($jobs)->map(function ($job) use (&$count, &$owner, &$resolver) { $job = $job instanceof Closure ? CallQueuedClosure::create($job) : $job; if (is_array($job)) { @@ -76,11 +80,37 @@ public function add(array|object $jobs): ?Batch $first->allOnConnection($this->options['connection']); } - return $first->chain($chain->slice(1)->values()->all()); + $job = $first->chain($chain->slice(1)->values()->all()); + } else { + $job->withBatchId($this->id); + + ++$count; } - $job->withBatchId($this->id); - ++$count; + if (array_key_exists('viaOutbox', $this->options) || array_key_exists('outboxDatabase', $this->options)) { + // Preserve initialized nulls as well as explicit values on each job. + $properties = get_object_vars($job); + + foreach (['viaOutbox', 'outboxDatabase'] as $property) { + if (array_key_exists($property, $this->options) && ! array_key_exists($property, $properties)) { + $job->{$property} = $this->options[$property]; + } + } + } + + // The counter transaction supplies the implicit SQL owner; declared owners must not bypass it. + if (($database = $job->outboxDatabase ?? null) !== null + && ($job->viaOutbox ?? null) !== false + && $this->repository instanceof DatabaseBatchRepository) { + $owner ??= $this->repository->getConnection()->getNameWithReadWriteType(); + /** @var ConnectionResolverInterface $resolver */ + $resolver ??= Container::getInstance()->make('db'); + $selectedOwner = $resolver->connection($database)->getNameWithReadWriteType(); + + if ($selectedOwner !== $owner) { + throw new LogicException("Outbox database [{$selectedOwner}] does not match the batch database [{$owner}]. Use the batch database or omit the explicit outbox owner."); + } + } return $job; }); diff --git a/src/bus/src/Batchable.php b/src/bus/src/Batchable.php index bb90b12f00..4c5ba4e10c 100644 --- a/src/bus/src/Batchable.php +++ b/src/bus/src/Batchable.php @@ -11,6 +11,8 @@ trait Batchable { + use InteractsWithQueueOutbox; + /** * The batch ID (if applicable). */ diff --git a/src/bus/src/DebounceLock.php b/src/bus/src/DebounceLock.php index 71c078a6cd..5d4822e3f9 100644 --- a/src/bus/src/DebounceLock.php +++ b/src/bus/src/DebounceLock.php @@ -45,13 +45,14 @@ public function acquire(mixed $job, ?int $debounceFor = null, ?int $maxWait = nu public function acquireForDispatch(object $job, ?int $debounceFor = null, ?int $maxWait = null): array { [$cache, $key, $ttl, $maxWait] = $this->resolveLockValues($job, $debounceFor, $maxWait); + $releaseContext = DispatchLockContext::captureReleaseContext(); $result = $this->acquireResolvedLock($cache, $key, $ttl, $maxWait); if (isset(class_uses_recursive($job)[Queueable::class])) { $job->debounceOwner = $result['owner']; } - DispatchLockContext::registerDebounce($job, $cache, $key, $result['owner']); + DispatchLockContext::registerDebounce($job, $cache, $key, $result['owner'], $releaseContext); return $result; } diff --git a/src/bus/src/DispatchLockContext.php b/src/bus/src/DispatchLockContext.php index ec1b101d4d..d8e28d5aaa 100644 --- a/src/bus/src/DispatchLockContext.php +++ b/src/bus/src/DispatchLockContext.php @@ -4,6 +4,7 @@ namespace Hypervel\Bus; +use Closure; use Hypervel\Contracts\Cache\Repository as Cache; use Throwable; use WeakMap; @@ -11,8 +12,6 @@ /** * Track dispatch lock ownership until a queue accepts the job. * - * @internal - * * @phpstan-type UniqueMetadata array{ * laravel_unique_job_cache_store: null|string, * laravel_unique_job_key: string, @@ -23,11 +22,13 @@ * metadata: null|UniqueMetadata, * unique: null|LockProvenance, * debounce: null|LockProvenance, + * releaseContext: null|Closure(Closure(): void): void, * delegated: bool * } * @phpstan-type LockSnapshot array{ * unique: null|LockProvenance, - * debounce: null|LockProvenance + * debounce: null|LockProvenance, + * releaseContext: null|Closure(Closure(): void): void * } */ class DispatchLockContext @@ -39,8 +40,73 @@ class DispatchLockContext */ protected static ?WeakMap $records = null; + /** + * The callbacks that capture context before dispatch lock acquisition. + * + * @var list + */ + protected static array $releaseContextCallbacks = []; + + /** + * Register a callback that captures the context needed for deferred lock release. + * + * Boot-only. Callbacks persist for the worker lifetime and affect subsequent dispatches. + * Return a wrapper that invokes the release operation once and restores the previous + * context in finally, or null when no context is needed. The first wrapper is outermost. + * Passing null clears registrations; capture runs before acquiring a lock. + * + * @param null|callable(): (null|Closure(Closure(): void): void) $callback + */ + public static function captureReleaseContextUsing(?callable $callback): void + { + if ($callback === null) { + static::$releaseContextCallbacks = []; + + return; + } + + static::$releaseContextCallbacks[] = $callback(...); + } + + /** + * Capture and compose the current release context. + * + * @internal + * + * @return null|Closure(Closure(): void): void + */ + public static function captureReleaseContext(): ?Closure + { + $context = null; + + foreach (static::$releaseContextCallbacks as $callback) { + if (($wrapper = $callback()) === null) { + continue; + } + + if ($context === null) { + $context = $wrapper; + + continue; + } + + $outer = $context; + $context = static function (Closure $release) use ($outer, $wrapper): void { + $outer(static function () use ($wrapper, $release): void { + $wrapper($release); + }); + }; + } + + return $context; + } + /** * Register an acquired unique lock. + * + * @internal + * + * @param null|Closure(Closure(): void): void $releaseContext */ public static function registerUnique( object $job, @@ -48,6 +114,7 @@ public static function registerUnique( ?string $cacheStore, string $key, string $owner, + ?Closure $releaseContext = null, ): void { /** @var WeakMap $records */ // @phpstan-ignore assign.propertyType (PHPStan rejects an empty WeakMap for this invariant closed-shape value.) @@ -61,29 +128,38 @@ public static function registerUnique( 'laravel_unique_job_lock_owner' => $owner, ]; $record['unique'] = compact('cache', 'key', 'owner'); + $record['releaseContext'] = $releaseContext; $records[$job] = $record; } /** * Register an acquired debounce lock. + * + * @internal + * + * @param null|Closure(Closure(): void): void $releaseContext */ public static function registerDebounce( object $job, Cache $cache, string $key, string $owner, + ?Closure $releaseContext = null, ): void { /** @var WeakMap $records */ // @phpstan-ignore assign.propertyType (PHPStan rejects an empty WeakMap for this invariant closed-shape value.) $records = static::$records ??= new WeakMap; $record = $records[$job] ?? static::newRecord(); $record['debounce'] = compact('cache', 'key', 'owner'); + $record['releaseContext'] = $releaseContext; $records[$job] = $record; } /** * Peek at hidden unique-job payload metadata. * + * @internal + * * @return null|UniqueMetadata */ public static function peekPayloadMetadata(object $job): ?array @@ -93,6 +169,8 @@ public static function peekPayloadMetadata(object $job): ?array /** * Determine whether a dispatch owns any locks. + * + * @internal */ public static function has(object $job): bool { @@ -101,6 +179,8 @@ public static function has(object $job): bool /** * Delegate a dispatch's lock ownership to a registered callback. + * + * @internal */ public static function delegate(object $job): bool { @@ -109,6 +189,8 @@ public static function delegate(object $job): bool /** * Claim lock ownership in a delegated callback. + * + * @internal */ public static function claim(object $job): bool { @@ -117,6 +199,8 @@ public static function claim(object $job): bool /** * Mark a dispatch as accepted by a queue. + * + * @internal */ public static function accept(object $job): void { @@ -129,6 +213,8 @@ public static function accept(object $job): void /** * Release locks still owned by a dispatch. + * + * @internal */ public static function release(object $job): void { @@ -149,11 +235,14 @@ public static function release(object $job): void static::releaseSnapshot([ 'unique' => $record['unique'], 'debounce' => $record['debounce'], + 'releaseContext' => $record['releaseContext'], ]); } /** - * Take a release snapshot for work accepted by an in-memory timer. + * Take an independent release snapshot before accepting a dispatch. + * + * @internal * * @return null|LockSnapshot */ @@ -168,15 +257,36 @@ public static function snapshot(object $job): ?array return [ 'unique' => $record['unique'], 'debounce' => $record['debounce'], + 'releaseContext' => $record['releaseContext'], ]; } /** - * Release locks from an in-memory timer snapshot. + * Release locks from an independent dispatch snapshot. + * + * @internal * * @param LockSnapshot $snapshot */ public static function releaseSnapshot(array $snapshot): void + { + if ($snapshot['releaseContext'] !== null) { + $snapshot['releaseContext'](static function () use ($snapshot): void { + static::releaseLocks($snapshot); + }); + + return; + } + + static::releaseLocks($snapshot); + } + + /** + * Release both owned locks even if one cleanup fails. + * + * @param LockSnapshot $snapshot + */ + protected static function releaseLocks(array $snapshot): void { $exception = null; @@ -252,6 +362,7 @@ protected static function newRecord(): array 'metadata' => null, 'unique' => null, 'debounce' => null, + 'releaseContext' => null, 'delegated' => false, ]; } @@ -262,5 +373,6 @@ protected static function newRecord(): array public static function flushState(): void { static::$records = null; + static::$releaseContextCallbacks = []; } } diff --git a/src/bus/src/InteractsWithQueueOutbox.php b/src/bus/src/InteractsWithQueueOutbox.php new file mode 100644 index 0000000000..234426ed1c --- /dev/null +++ b/src/bus/src/InteractsWithQueueOutbox.php @@ -0,0 +1,63 @@ +viaOutbox = true; + + return $this; + } + + /** + * Dispatch the job without the transactional outbox. + */ + public function withoutOutbox(): static + { + $this->viaOutbox = false; + $this->outboxDatabase = null; + + return $this; + } + + /** + * Stage the job on the given database connection. + */ + public function onOutboxDatabase(UnitEnum|string $connection): static + { + $connection = (string) enum_value($connection); + + if ($connection === '') { + throw new InvalidArgumentException('The outbox database connection must not be empty.'); + } + + $this->outboxDatabase = $connection; + $this->viaOutbox = true; + + return $this; + } +} diff --git a/src/bus/src/PendingBatch.php b/src/bus/src/PendingBatch.php index ab42b52773..dd00e2d17c 100644 --- a/src/bus/src/PendingBatch.php +++ b/src/bus/src/PendingBatch.php @@ -12,6 +12,7 @@ use Hypervel\Support\Arr; use Hypervel\Support\Collection; use Hypervel\Support\Traits\Conditionable; +use InvalidArgumentException; use Laravel\SerializableClosure\SerializableClosure; use RuntimeException; use Throwable; @@ -283,6 +284,45 @@ public function queue(): ?string return $this->options['queue'] ?? null; } + /** + * Dispatch batched jobs through the transactional outbox by default. + */ + public function viaOutbox(): static + { + $this->options['viaOutbox'] = true; + + return $this; + } + + /** + * Dispatch batched jobs without the transactional outbox by default. + */ + public function withoutOutbox(): static + { + $this->options['viaOutbox'] = false; + + unset($this->options['outboxDatabase']); + + return $this; + } + + /** + * Stage batched jobs on the given database connection by default. + */ + public function onOutboxDatabase(UnitEnum|string $connection): static + { + $connection = (string) enum_value($connection); + + if ($connection === '') { + throw new InvalidArgumentException('The outbox database connection must not be empty.'); + } + + $this->options['outboxDatabase'] = $connection; + $this->options['viaOutbox'] = true; + + return $this; + } + /** * Add additional data into the batch's options array. */ diff --git a/src/bus/src/Queueable.php b/src/bus/src/Queueable.php index 9a8d5f5b81..15678c77fe 100644 --- a/src/bus/src/Queueable.php +++ b/src/bus/src/Queueable.php @@ -20,6 +20,8 @@ trait Queueable { + use InteractsWithQueueOutbox; + /** * The name of the connection the job should be sent to. */ diff --git a/src/bus/src/UniqueLock.php b/src/bus/src/UniqueLock.php index c3c235c100..03fcbd5c31 100644 --- a/src/bus/src/UniqueLock.php +++ b/src/bus/src/UniqueLock.php @@ -45,6 +45,7 @@ public function acquireForDispatch(object $job): bool { [$uniqueFor, $cache, $key, $supportsOwnership] = $this->resolveLockValues($job); $cacheStore = $cache instanceof Repository ? $cache->getName() : null; + $releaseContext = DispatchLockContext::captureReleaseContext(); $owner = $this->acquireResolvedLock( $job, @@ -59,7 +60,7 @@ public function acquireForDispatch(object $job): bool return false; } - DispatchLockContext::registerUnique($job, $cache, $cacheStore, $key, $owner); + DispatchLockContext::registerUnique($job, $cache, $cacheStore, $key, $owner, $releaseContext); return true; } diff --git a/src/cache/src/CacheManager.php b/src/cache/src/CacheManager.php index 6c3e9aca47..1c40928dd9 100644 --- a/src/cache/src/CacheManager.php +++ b/src/cache/src/CacheManager.php @@ -125,7 +125,7 @@ protected function createMemoizedRepository(string $driver): CacheRepository $repository = $this->repository( new MemoizedStore($driver, $store), - ['events' => false] + ['store' => $store->getName(), 'events' => false] ); return $isSpy ? Mockery::spy($repository) : $repository; @@ -146,7 +146,14 @@ public function resolve(string $name): CacheRepository $config = Arr::add($config, 'store', $name); - return $this->build($config); + $repository = $this->build($config); + + // Custom creators may wrap a store without passing its configuration. + if ($repository instanceof Repository && $repository->getName() === null) { + $repository->setName($config['store']); + } + + return $repository; } /** diff --git a/src/cache/src/Repository.php b/src/cache/src/Repository.php index 02857160a7..304bc805aa 100644 --- a/src/cache/src/Repository.php +++ b/src/cache/src/Repository.php @@ -1092,6 +1092,21 @@ public function getName(): ?string return $this->config['store'] ?? null; } + /** + * Set the cache store name before sharing the repository. + * + * Changing a shared repository's name affects cache events + * and queued unique-lock cleanup in every coroutine using that repository. + * + * @internal + */ + public function setName(string $name): static + { + $this->config['store'] = $name; + + return $this; + } + /** * Determine if a cached value exists. * diff --git a/src/contracts/src/Queue/PublishesOutboxMessages.php b/src/contracts/src/Queue/PublishesOutboxMessages.php new file mode 100644 index 0000000000..35d3c1b63a --- /dev/null +++ b/src/contracts/src/Queue/PublishesOutboxMessages.php @@ -0,0 +1,19 @@ + $messages + */ + public function publishOutboxMessages(array $messages): OutboxBatchResult; +} diff --git a/src/contracts/src/Queue/ShouldQueueViaOutbox.php b/src/contracts/src/Queue/ShouldQueueViaOutbox.php new file mode 100644 index 0000000000..881bd3b408 --- /dev/null +++ b/src/contracts/src/Queue/ShouldQueueViaOutbox.php @@ -0,0 +1,9 @@ +transactionsManager?->commit( - $this->getName() ?? '', + $this->getNameWithReadWriteType() ?? '', $levelBeingCommitted, $this->transactions ); @@ -182,7 +182,7 @@ public function beginTransaction(): void try { $this->transactionsManager?->begin( - $this->getName() ?? '', + $this->getNameWithReadWriteType() ?? '', $this->transactions ); @@ -259,7 +259,7 @@ public function commit(): void try { $this->transactionsManager?->commit( - $this->getName() ?? '', + $this->getNameWithReadWriteType() ?? '', $levelBeingCommitted, $this->transactions ); @@ -369,7 +369,7 @@ public function rollBack(?int $toLevel = null): void try { $this->transactionsManager?->rollback( - $this->getName() ?? '', + $this->getNameWithReadWriteType() ?? '', $this->transactions ); } catch (CanceledException $cancellation) { @@ -447,7 +447,7 @@ public function hasApplicableTransaction(): bool return false; } - return $this->transactionsManager?->hasApplicableTransaction($this->getName() ?? '') ?? true; + return $this->transactionsManager?->hasApplicableTransaction($this->getNameWithReadWriteType() ?? '') ?? true; } /** @@ -460,7 +460,7 @@ public function hasApplicableTransaction(): bool public function afterCommit(callable $callback): void { if ($this->transactionsManager) { - $this->transactionsManager->addCallback($callback, $this->getName() ?? ''); + $this->transactionsManager->addCallback($callback, $this->getNameWithReadWriteType() ?? ''); return; } @@ -492,7 +492,7 @@ public function afterCommitOrNow(callable $callback): void public function afterRollBack(callable $callback): void { if ($this->transactionsManager) { - $this->transactionsManager->addCallbackForRollback($callback, $this->getName() ?? ''); + $this->transactionsManager->addCallbackForRollback($callback, $this->getNameWithReadWriteType() ?? ''); return; } diff --git a/src/database/src/Connection.php b/src/database/src/Connection.php index 83d295716b..9553d63a9a 100755 --- a/src/database/src/Connection.php +++ b/src/database/src/Connection.php @@ -1003,7 +1003,7 @@ protected function resetTransactionState(): void { $this->transactions = 0; - $this->transactionsManager?->rollback($this->getName() ?? '', 0); + $this->transactionsManager?->rollback($this->getNameWithReadWriteType() ?? '', 0); } /** diff --git a/src/database/src/DatabaseTransactionsManager.php b/src/database/src/DatabaseTransactionsManager.php index a52c9e76ee..3e96b46030 100755 --- a/src/database/src/DatabaseTransactionsManager.php +++ b/src/database/src/DatabaseTransactionsManager.php @@ -288,7 +288,7 @@ protected function executeRollbackCallbacks(Collection $transactions): void * connection. The callback runs when that transaction and its enclosing stack on the * same connection commit. * - * @param null|string $connection base name from Connection::getName() + * @param null|string $connection resolved owner name from Connection::getNameWithReadWriteType() */ public function addCallback(callable $callback, ?string $connection = null): void { @@ -307,7 +307,7 @@ public function addCallback(callable $callback, ?string $connection = null): voi * connection. The callback runs when that transaction and its enclosing stack on the * same connection roll back. * - * @param null|string $connection base name from Connection::getName() + * @param null|string $connection resolved owner name from Connection::getNameWithReadWriteType() */ public function addCallbackForRollback(callable $callback, ?string $connection = null): void { diff --git a/src/docs/broadcasting.md b/src/docs/broadcasting.md index b455297666..01a430d1e5 100644 --- a/src/docs/broadcasting.md +++ b/src/docs/broadcasting.md @@ -1001,6 +1001,20 @@ class ServerCreated implements ShouldBroadcast, ShouldDispatchAfterCommit > [!NOTE] > To learn more about working around these issues, please review the documentation regarding [queued jobs and database transactions](/docs/{{version}}/queues#jobs-and-database-transactions). +If you would like to store a broadcast job as part of your database transaction, you may implement the `ShouldQueueViaOutbox` interface on the broadcast event. The [transactional outbox](/docs/{{version}}/queue-outbox) will then deliver the job to your queue after the transaction commits: + +```php +use Hypervel\Contracts\Broadcasting\ShouldBroadcast; +use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; + +class ServerCreated implements ShouldBroadcast, ShouldQueueViaOutbox +{ + // ... +} +``` + +Dispatch the event inside the transaction without implementing `ShouldDispatchAfterCommit`. That interface would postpone the entire event until after commit, preventing the broadcast job from being stored in the transaction. Before using this feature, follow the [outbox setup instructions](/docs/{{version}}/queue-outbox#setup). + ## Authorizing Channels diff --git a/src/docs/documentation.md b/src/docs/documentation.md index 0f157e39d5..212d87e83f 100644 --- a/src/docs/documentation.md +++ b/src/docs/documentation.md @@ -61,6 +61,7 @@ - [Signals](/docs/{{version}}/signals) - [WebSockets](/docs/{{version}}/websockets) - [Queues](/docs/{{version}}/queues) + - [Transactional Outbox](/docs/{{version}}/queue-outbox) - [Rate Limiting](/docs/{{version}}/rate-limiting) - [Search](/docs/{{version}}/search) - [Strings](/docs/{{version}}/strings) diff --git a/src/docs/events.md b/src/docs/events.md index cc1b98ce72..5832733035 100644 --- a/src/docs/events.md +++ b/src/docs/events.md @@ -521,6 +521,19 @@ class SendShipmentNotification implements ShouldQueueAfterCommit > [!NOTE] > To learn more about working around these issues, please review the documentation regarding [queued jobs and database transactions](/docs/{{version}}/queues#jobs-and-database-transactions). +If you would like to store a queued listener's job as part of your database transaction, you may implement the `ShouldQueueViaOutbox` interface on the listener. The [transactional outbox](/docs/{{version}}/queue-outbox) will then deliver the job to your queue after the transaction commits: + +```php +use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; + +class SendShipmentNotification implements ShouldQueueViaOutbox +{ + // ... +} +``` + +Dispatch the event inside the transaction, and do not implement `ShouldDispatchAfterCommit` on the event itself. That interface would postpone the entire event until after commit, preventing the listener's job from being stored in the transaction. Since Hypervel reads listener settings without constructing the listener, use `ShouldQueueViaOutbox` rather than calling `viaOutbox` in its constructor. Before using this feature, follow the [outbox setup instructions](/docs/{{version}}/queue-outbox#setup). + ### Queued Listener Middleware diff --git a/src/docs/mail.md b/src/docs/mail.md index d802de4020..85c81f665d 100644 --- a/src/docs/mail.md +++ b/src/docs/mail.md @@ -1240,6 +1240,16 @@ class OrderShipped extends Mailable implements ShouldQueue > [!NOTE] > To learn more about working around these issues, please review the documentation regarding [queued jobs and database transactions](/docs/{{version}}/queues#jobs-and-database-transactions). +If you would like to store a queued mailable as part of your database transaction, you may call the `viaOutbox` method provided by its `Queueable` trait: + +```php +Mail::to($request->user())->queue( + (new OrderShipped($order))->viaOutbox() +); +``` + +Queue the mailable inside the transaction. Hypervel will store its job in the [transactional outbox](/docs/{{version}}/queue-outbox) and deliver it to your queue after the transaction commits. Before using this feature, follow the [outbox setup instructions](/docs/{{version}}/queue-outbox#setup). + #### Queued Email Failures diff --git a/src/docs/notifications.md b/src/docs/notifications.md index 11810cb963..b4de840af2 100644 --- a/src/docs/notifications.md +++ b/src/docs/notifications.md @@ -435,6 +435,14 @@ class InvoicePaid extends Notification implements ShouldQueue > [!NOTE] > To learn more about working around these issues, please review the documentation regarding [queued jobs and database transactions](/docs/{{version}}/queues#jobs-and-database-transactions). +If you would like to store a queued notification's jobs as part of your database transaction, you may call the `viaOutbox` method provided by its `Queueable` trait: + +```php +$user->notify((new InvoicePaid($invoice))->viaOutbox()); +``` + +Send the notification inside the transaction. Hypervel will store each queued channel's job in the [transactional outbox](/docs/{{version}}/queue-outbox) and deliver it after commit. Unlike `viaQueues`, `viaOutbox` configures the notification when called; you do not implement it to return a setting. Before using this feature, follow the [outbox setup instructions](/docs/{{version}}/queue-outbox#setup). + #### Determining if a Queued Notification Should Be Sent diff --git a/src/docs/opentelemetry.md b/src/docs/opentelemetry.md index 0b13b1b183..f416fd52d3 100644 --- a/src/docs/opentelemetry.md +++ b/src/docs/opentelemetry.md @@ -189,6 +189,7 @@ The shipped defaults are: | Database and Redis pools | None | `db.client.connection.count`, `db.client.connection.max`, `db.client.connection.pending_requests` | | Cache | Events on the current span | `hypervel.cache.operations` | | Queue | Producer and consumer spans | `messaging.client.sent.messages`, `messaging.client.operation.duration`, `messaging.client.consumed.messages`, `messaging.process.duration`; `hypervel.queue.jobs` is off | +| Queue outbox | No additional spans | `hypervel.queue.outbox.delivered.messages`, `hypervel.queue.outbox.delivery.duration`, `hypervel.queue.outbox.delivery.latency` | | Scheduler | Task spans | `hypervel.scheduler.task.duration`, `hypervel.scheduler.task.executions` | | Console | Command spans | `hypervel.console.command.duration` | | Events | Exact allowlist, empty by default | None | @@ -318,7 +319,15 @@ The built-in `background` and `deferred` connections propagate the current conte Jobs dispatched after a database commit are finalized and timed only when the queue driver actually begins enqueueing them. Database and SQS batch operations retain one lifecycle per logical message. Later SQS chunks include the time spent sending earlier chunks because timing begins when the batch is finalized, and a later batch failure completes every still-outstanding message once. -Terminal queue correlation normally uses the exact final encoded payload without decoding it. If another finalizer rewrites that payload afterward, the package performs one terminal decode to recover Hypervel's job UUID. Payloads from custom drivers or `Queue::createPayloadUsing()` callbacks that do not contain a string UUID are accepted, omit `messaging.message.id`, and use exact-payload correlation only. If a later finalizer rewrites such a payload, its producer span remains started but is never ended or exported, and its state remains until coroutine or process exit. A later finalizer that removes or corrupts an existing UUID, or throws after earlier messages in the same batch were finalized, likewise leaves unmatched state. A long-running non-coroutine custom process that catches and continues after such failures may retain that state until it exits. +Terminal queue correlation normally uses the exact final encoded payload without decoding it. If another finalizer rewrites that payload afterward, the package performs one terminal decode to recover Hypervel's job UUID. Payloads from custom drivers or `Queue::createPayloadUsing()` callbacks that do not contain a string UUID are accepted, omit `messaging.message.id`, and use exact-payload correlation only. If a later finalizer rewrites such a payload, its producer span remains started but is never ended or exported, and its state remains until coroutine or process exit. A later finalizer that removes or corrupts an existing UUID likewise leaves unmatched state. A long-running non-coroutine custom process that catches and continues after such failures may retain that state until it exits. + +When using the [transactional outbox](/docs/{{version}}/queue-outbox), the producer span ends when the job is stored in the outbox. The relay preserves the job's trace context without creating another producer span. The following metrics are enabled by default and record delivery separately for each queue connection's batch: + +- `hypervel.queue.outbox.delivered.messages` counts messages accepted by the queue, including messages whose outbox records could not be deleted afterwards. +- `hypervel.queue.outbox.delivery.duration` measures how long sending the batch takes, in seconds. If sending or updating the outbox fails, the observation includes `error.type`. Count the duration observations that include `error.type` to determine the number of failed batches. +- `hypervel.queue.outbox.delivery.latency` records how long the oldest attempted message has been ready for delivery, in seconds, measured when sending begins. It records one value per connection batch, not one per message. Messages deferred until a future delivery time are excluded. + +These metrics include the configured queue connection and outbox database as attributes. Only the duration metric includes exception types. Collecting them does not query pending outbox records or decode job payloads, and does not add work to ordinary dispatches. To monitor pending counts and waiting times, use the [`queue:outbox-monitor` command](/docs/{{version}}/queue-outbox#monitoring-the-outbox). Queue depth is off by default because it performs backend I/O during metric collection. Enable it with explicit targets: diff --git a/src/docs/porting-from-laravel.md b/src/docs/porting-from-laravel.md index d4b30c7bd7..18748f174a 100644 --- a/src/docs/porting-from-laravel.md +++ b/src/docs/porting-from-laravel.md @@ -803,6 +803,8 @@ Framework operations release idle database sessions at the boundaries listed in Do not manually commit or roll back the transaction owned by a `DB::transaction()` closure; Hypervel throws when that closure returns. Let `transaction()` manage completion, or use the [manual transaction API](/docs/{{version}}/database#manually-using-transactions) separately. Balanced nested transactions remain supported. +If your extension inspects transaction records or registers callbacks directly with `DatabaseTransactionsManager`, use the resolved connection's `getNameWithReadWriteType()` instead of `getName()`. Base, `::read`, and `::write` connections can have independent transactions. Always resolve the connection before reading its name; aliases for a shared in-memory SQLite connection resolve to the same session. Ordinary `afterCommit()` and `afterRollBack()` calls already use the correct connection. + When a package constructs `DatabaseStore`, `DatabaseSessionHandler`, `DatabaseQueue`, or `DatabaseBatchRepository` directly, pass the database connection resolver and configured connection name instead of retaining a resolved connection. Framework-configured drivers already use this form. Laravel's base `Connection` class exposes PDO methods. Hypervel's base `Connection` is driver-neutral, while its built-in SQL connections extend `PdoConnection`. Ported code that calls `getPdo`, `getReadPdo`, or another PDO-specific method should accept or narrow to `PdoConnection`. See [extending database connections](/docs/{{version}}/database#extending-database-connections) when porting a custom driver. @@ -859,6 +861,8 @@ Hypervel stores job batches in a relational database. Laravel's DynamoDB batch r With `Worker::$killOnTimeout = false`, a timeout interrupts job code with `Swoole\Coroutine\CanceledException`; catching `TimeoutExceededException` inside the job does not catch it. Use `finally` for cleanup and rethrow cancellation if caught. Failure callbacks and worker reporting still receive `TimeoutExceededException`. See [worker timeouts](/docs/{{version}}/queues#worker-timeouts). +If you extend Horizon's `RedisQueue`, do not rely on `push()` calling your `pushRaw()` override. Override the protected `preparePayloadForPush()` method to prepare normal pushes, or `handlePayloadPushed()` to run code after Redis accepts the job. Direct calls to `pushRaw()` remain supported. + When a Laravel package offers optional support for an unsupported driver, remove that integration from the Hypervel port unless the package can safely provide it through a separate optional dependency. diff --git a/src/docs/queue-outbox.md b/src/docs/queue-outbox.md new file mode 100644 index 0000000000..0ed15f4f1b --- /dev/null +++ b/src/docs/queue-outbox.md @@ -0,0 +1,362 @@ +# Transactional Queue Outbox + +- [Introduction](#introduction) +- [Setup](#setup) +- [Dispatching Jobs](#dispatching-jobs) + - [Selecting the Outbox](#selecting-the-outbox) + - [Bulk Dispatch and Batches](#bulk-dispatch-and-batches) + - [Queued Listeners, Mail, Notifications, and Broadcasts](#queued-producers) +- [Database Transactions](#database-transactions) +- [Queue Connections and Delays](#queue-connections-and-delays) + - [Delayed Jobs](#delayed-jobs) + - [Changing Queue Configuration](#changing-queue-configuration) +- [Running the Relay](#running-the-relay) + - [Concurrency and Shutdown](#concurrency-and-shutdown) + - [Failures and Recovery](#failures-and-recovery) +- [Inspecting and Managing Messages](#managing-messages) + - [Monitoring the Outbox](#monitoring-the-outbox) + - [Retrying Messages](#retrying-messages) + - [Deleting Messages](#deleting-messages) +- [Delivery Guarantees](#delivery-guarantees) +- [Custom Queue Drivers](#custom-queue-drivers) + + +## Introduction + +When you dispatch a job [after a database transaction commits](/docs/{{version}}/queues#jobs-and-database-transactions), queue workers will not process it before the transaction completes. However, your application could stop after committing the transaction but before sending the job to the queue. In that case, your database changes would be saved, but the job would be lost. + +Hypervel's transactional outbox allows you to store a job as part of the same transaction as your database changes. A separate process, called the relay, sends the stored job to your queue after the transaction commits. If the transaction rolls back, the job is discarded as well. If the queue is unavailable, the job remains in the outbox until delivery succeeds. + +For example, you may call `viaOutbox` when dispatching a job inside a transaction: + +```php +use App\Jobs\SendOrderReceipt; +use App\Models\Order; +use Hypervel\Support\Facades\DB; + +DB::transaction(function () use ($attributes) { + $order = Order::create($attributes); + + SendOrderReceipt::dispatch($order)->viaOutbox(); +}); +``` + +The outbox works with your existing jobs, queue connections, and workers. It does not require Horizon. Jobs that do not use the outbox are dispatched normally without accessing the outbox table. + + +## Setup + +You may configure the outbox in your application's `config/queue.php` configuration file: + +```php +'outbox' => [ + 'databases' => [ + env('QUEUE_OUTBOX_DATABASE', env('DB_CONNECTION', 'sqlite')), + ], + 'table' => env('QUEUE_OUTBOX_TABLE', 'job_outbox'), + 'retry_after' => (int) env('QUEUE_OUTBOX_RETRY_AFTER', 300), +], +``` + +The `databases` option contains the names of the database connections the relay should read from. Each database needs an outbox table. New Hypervel applications include a migration for this table. If your application does not have this migration, you may create it using the `make:queue-outbox-table` Artisan command: + +```shell +php artisan make:queue-outbox-table +php artisan migrate +``` + +The generated migration uses the table name from your outbox configuration. You may also use the `queue:outbox-table` alias to generate it. If you use several outbox databases, run the migration on each connection. + +Finally, run the relay alongside your normal queue workers: + +```shell +php artisan queue:relay +php artisan queue:work redis +``` + +The relay sends jobs to the queue; your queue workers still process them. If you use Horizon, it may continue managing your Redis workers as usual. + + +## Dispatching Jobs + + +### Selecting the Outbox + +To dispatch a job through the outbox, you may chain the `viaOutbox` method onto your dispatch operation: + +```php +SendOrderReceipt::dispatch($order)->viaOutbox(); +``` + +If a job should always use the outbox, you may implement the `ShouldQueueViaOutbox` interface on its class. This interface extends `ShouldQueue`, so you do not need to implement both: + +```php +use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; +use Hypervel\Foundation\Queue\Queueable; + +class SendOrderReceipt implements ShouldQueueViaOutbox +{ + use Queueable; + + // ... +} +``` + +Alternatively, you may set the `outbox` option to `true` on a queue connection in your `config/queue.php` file. This enables the outbox for jobs dispatched to that connection. When this option is omitted, it defaults to `false`. + +You may override either default for a specific dispatch: + +```php +SendOrderReceipt::dispatch($order)->withoutOutbox(); +``` + +Calling `viaOutbox` or `withoutOutbox` overrides both the job's interface and the connection's configuration. Otherwise, a job uses the outbox if it implements `ShouldQueueViaOutbox` or its queue connection enables the `outbox` option. + +The `afterCommit` and `beforeCommit` methods do not disable the outbox. When a job uses the outbox, it is stored within the database transaction and cannot be delivered until that transaction commits. + +The `Queueable` and `Batchable` traits provide the outbox methods and properties. If your class uses either trait, use `ShouldQueueViaOutbox` for its default behavior rather than redeclaring the trait's properties with default values, which PHP does not allow. Classes without these properties may declare a public `viaOutbox` property with a default of `true` or `false`. Unlike `viaConnection` or `viaQueues`, `viaOutbox` configures the object when called; it is not a method that you implement to return a setting. + +If you push a job using a queue method such as `Queue::push` and the job is stored in the outbox, the method returns the job's UUID instead of a queue-assigned ID. You may use this UUID with the [outbox commands](#managing-messages). Dispatches that bypass the outbox retain their usual return values. + + +### Bulk Dispatch and Batches + +To store several jobs in the outbox, you may use the `bulk` method on the `Bus` facade: + +```php +use Hypervel\Support\Facades\Bus; + +Bus::bulk([ + (new SendOrderReceipt($firstOrder))->viaOutbox(), + (new SendOrderReceipt($secondOrder))->viaOutbox(), +]); +``` + +Hypervel groups bulk jobs by queue connection and queue name. Within each group, jobs using the same outbox database are inserted together. Large inserts are split to respect the database's parameter limit, but all inserts for that database group succeed or roll back together. Jobs sent directly to a queue or stored in different database groups do not share this guarantee, and may be delivered in a different order. + +You may also select the outbox for a batch: + +```php +$batch = Bus::batch($jobs) + ->onConnection('redis') + ->viaOutbox() + ->dispatch(); + +$batch->add($moreJobs); +``` + +The batch's outbox settings also apply to jobs added later, unless a job has explicitly configured its own settings. With database batch storage, outbox jobs are stored in your batch database in the same transaction as the batch counters. Include this database in `queue.outbox.databases`. If a job selects a different outbox database, Hypervel throws an exception before updating the batch. Jobs only need the `Batchable` trait to receive these settings; `Queueable` is not required. + +Chains retain their normal behavior. Selecting the outbox for the first job does not automatically select it for later jobs. Batch outbox defaults likewise apply only to the first job of each chain. Each later job is dispatched after its predecessor succeeds and uses its own selection or its queue connection's default. + + +### Queued Listeners, Mail, Notifications, and Broadcasts + +You may also use the outbox with queued listeners, mailables, notifications, and broadcast events. Implement `ShouldQueueViaOutbox` on the class, or call `viaOutbox` if it uses the `Queueable` trait. For example, you may queue a mailable through the outbox: + +```php +use App\Mail\OrderReceipt; +use Hypervel\Support\Facades\Mail; + +Mail::to($customer)->queue( + (new OrderReceipt($order))->viaOutbox() +); +``` + +For queued listeners, implement `ShouldQueueViaOutbox` instead of calling `viaOutbox` in the constructor. Hypervel reads the listener's settings without constructing it. A listener that does not use the outbox properties from `Queueable` or `Batchable` may also define a public property default, as described [above](#selecting-the-outbox). + +Dispatch events inside your database transaction so their queued listeners and broadcasts can be stored with your database changes. + +> [!WARNING] +> An event implementing `ShouldDispatchAfterCommit` is not dispatched until the transaction commits. Its listener and broadcast jobs therefore cannot be stored as part of that transaction. To use the outbox for these jobs, implement `ShouldQueueViaOutbox` on the listener or broadcast event instead of deferring the entire event. + + +## Database Transactions + +For queues that store jobs in the outbox, Hypervel uses the connection with the active transaction. Without an active transaction, it uses the first database in your outbox configuration. Whichever database stores the job must be listed in `queue.outbox.databases`. You may select a database using the `onOutboxDatabase` method: + +```php +SendOrderReceipt::dispatch($order) + ->onOutboxDatabase('billing'); +``` + +Calling `onOutboxDatabase` also enables the outbox for the job. If transactions are open on several connections, you must select one explicitly. Otherwise, Hypervel throws an exception listing the active connections. + +If any database transactions are open, the selected connection must have an active transaction. Only changes made in that transaction are committed or rolled back with the job. Nested transactions on the same connection are supported; rolling back a savepoint also removes its outbox jobs and releases the unique and debounce locks acquired by those dispatches. + +Use the connection that owns your transaction. For example, `mysql` and `mysql::write` can use separate database sessions with independent transactions. You may select `mysql::write` with `onOutboxDatabase`, but list only `mysql` in `queue.outbox.databases` so the relay reads the table once. Read connections such as `mysql::read` cannot store outbox jobs; use the base or write connection instead. + +Calling `withoutOutbox` clears an explicit database choice. Calling `viaOutbox` afterwards lets Hypervel select the database using the rules above. + +> [!WARNING] +> The outbox cannot combine changes to different databases into one transaction. It also cannot include an HTTP request, payment, email, or filesystem write in your database transaction. Perform these operations in the queued job, taking care to [handle duplicate delivery](#delivery-guarantees). + + +## Queue Connections and Delays + +You may switch queue drivers without removing `viaOutbox` calls from your jobs. Hypervel handles each driver as follows: + +| Queue connection | Behavior when the outbox is selected | +| --- | --- | +| Redis, SQS, Beanstalkd | Store the job in the outbox, then relay it to the queue. | +| Database, using the active transaction's connection | Insert directly into the jobs table as part of that transaction. | +| Database, with no active transaction | Insert directly into the durable jobs table. | +| Database, on another connection during a business transaction | Store the job on the business transaction's outbox, then relay it to the database queue. | +| Sync, background, deferred | Wait for the current transaction to commit, or the transaction selected with `onOutboxDatabase`, then use the driver's normal execution behavior. Without a transaction, behave normally. Jobs are not stored for recovery after a process stops. | +| Failover | Wait for the applicable transactions to commit, then attempt normal failover delivery. The attempt is not stored for recovery after a process stops. | +| Null | Discard the job normally without creating an outbox row. | +| Queue or Bus fake | Record the job only in the fake, without accessing an outbox table. | + +The `dispatchSync` and after-response dispatch methods retain their existing behavior; they do not store jobs in the outbox. Calling `pushRaw` also bypasses the outbox, even when the connection's configuration enables it. + + +### Delayed Jobs + +You may use the usual `delay` method when dispatching through the outbox: + +```php +SendOrderReceipt::dispatch($order)->viaOutbox()->delay(now()->plus(minutes: 10)); +``` + +Retrying delivery does not restart the job's delay. Redis and database queues can receive delayed jobs immediately and hold them until they are due. With standard SQS queues, the relay waits until the job is within SQS's per-message delay limit. With SQS FIFO queues, it waits until the job is due before sending it, since FIFO queues do not support per-message delays. + +Any delay configured on an SQS FIFO queue still applies after the relay sends the job. For standard SQS queues, omitting a job delay preserves the queue's default delay. An explicitly requested job delay overrides that default, including when the job is already due. + + +### Changing Queue Configuration + +The outbox stores the job's queue connection and queue name when you dispatch it. If you did not specify a queue name, it stores the connection's default. Changing that default later does not move pending jobs to another queue. Queue-name forwarding, however, uses the rules in effect when the relay sends the job. + +The relay uses the connection's current configuration. Allow pending jobs to leave the outbox before switching to a driver that does not persist jobs, changing between standard and FIFO SQS queues, or making other changes that would invalidate the stored queue options. + + +## Running the Relay + +The `queue:relay` Artisan command sends committed outbox jobs to their queues. Like `queue:work`, it runs as a long-lived process and should be managed by a process supervisor. You may configure its batch size and concurrency when starting it: + +```shell +php artisan queue:relay --batch=100 --concurrency=4 +``` + +By default, the relay reads from each configured outbox database in turn. To run a relay for a single database, pass its connection name to the command: + +```shell +php artisan queue:relay billing +``` + +The relay supports the following options: + +| Option | Default | Purpose | +| --- | --- | --- | +| `--batch` | `100` | Maximum messages to take from the outbox at once, from 1 to 1,000. | +| `--concurrency` | `1` | Maximum batches being delivered concurrently. | +| `--sleep` | `1` | Seconds to wait after finding no work or after a delivery failure. Must be positive; fractions are supported. | +| `--max-jobs` | `0` | Stop after taking this many messages from the outbox; zero means unlimited. | +| `--max-time` | `0` | Stop taking new work after this many seconds; zero means unlimited. | +| `--memory` | `128` | Memory limit in megabytes. | +| `--stop-when-empty` | Off | Stop when no selected database has messages ready for delivery, after finishing work already started. | +| `--once` | Off | Process at most one batch from each selected database, without sleeping. | + +The `--stop-when-empty` option does not wait for delayed messages or scheduled retries. A database that cannot be read, or is waiting to be retried, counts as empty for that pass. The `--max-jobs` option counts messages taken for delivery, including failed attempts. When using `--once`, batches are delivered one at a time regardless of the concurrency setting. + + +### Concurrency and Shutdown + +You may run several relay processes against the same outbox. Within each process, the `--concurrency` option controls how many batches may be delivered at once using coroutines. The relay reserves messages in a short database operation, then releases its database connections and locks before contacting the queue. If another relay takes over an expired reservation, the original relay can no longer delete those records. SQLite supports this behavior, but adding relays does not allow it to perform concurrent writes. + +Your database and queue connection pools should have enough connections for the configured concurrency. Allow an additional connection if restart checks or event listeners use the same pool. Each relay process has its own pools; increasing concurrency does not increase their size. + +The `queue.outbox.retry_after` option controls how many seconds a batch remains reserved for one relay. Set it longer than the longest expected delivery attempt for an entire batch, including all queue connections, network timeouts, and SDK retries. Reservations are not extended while delivery is running. If a reservation expires, another relay may deliver the same messages. + +You may gracefully restart relays using the `queue:restart` command. Relays read this signal from the same configured cache store as ordinary queue workers. On restart or shutdown, a relay stops taking new work and finishes the batches it has already started. + +The `SIGTERM`, `SIGINT`, and `SIGQUIT` signals request shutdown. `SIGUSR2` pauses new work, and `SIGCONT` resumes it. Configure your process supervisor to allow enough time for active deliveries and their database updates to finish before forcibly stopping the relay. + +Maintenance mode does not pause the relay. It only moves already committed jobs to their queues; your normal workers control whether those jobs execute. + + +### Failures and Recovery + +Once a queue confirms that it has accepted a job, the relay deletes the outbox record. If delivery fails or its outcome is unknown, the record remains available for another attempt. The delay between attempts increases up to ten minutes, with some random variation to avoid retrying all failed deliveries at once. Retries continue until delivery succeeds or you delete the message; there is no maximum attempt limit. + +If the relay cannot read a database, it skips that database for the `--sleep` interval and continues reading the others. After a batch fails delivery, it occupies its concurrency slot for the `--sleep` interval before another batch can take its place. This limits repeated attempts against an unavailable queue even when many new messages are waiting. + +Delivery errors are reported through your application's exception handler with the database, queue connection, row IDs, job UUIDs, and job names. If a delivery event listener fails, its error is reported without sending accepted jobs again or delaying successful deliveries. If the exception handler itself fails, the error is allowed to stop the command. + +When using SQS payload overflow, each relay attempt stores its payload separately so that cleanup from one attempt does not remove another attempt's data. If SQS does not return a clear result, the stored payload may remain after the attempt, just as with ordinary dispatch. Account for these payloads in your storage retention policy; the outbox does not change your existing overflow cleanup settings. + + +## Inspecting and Managing Messages + +You may inspect pending outbox messages using the `queue:outbox` Artisan command. By default, it lists messages from all configured outbox databases. You may pass a database name to inspect only that connection: + +```shell +php artisan queue:outbox --limit=100 +php artisan queue:outbox billing --json +``` + +The listing includes each job's UUID, destination, attempt count, next delivery time, and last delivery error. Table output displays UTC dates and abbreviated errors. The `--json` option returns Unix timestamps and complete stored errors instead. Neither format displays job payloads or lock-owner tokens. + + +### Monitoring the Outbox + +To monitor the number of pending messages and how long they have been waiting for delivery, you may use the `queue:outbox-monitor` command: + +```shell +php artisan queue:outbox-monitor --max=10000 --age=60 +``` + +The command reports messages ready for delivery, delayed messages, messages reserved by a relay, and messages waiting to retry. The optional `--max` and `--age` options set alert thresholds for the total pending count and oldest ready message's age in seconds. When either threshold is reached, Hypervel dispatches a `Hypervel\Queue\Events\OutboxBusy` event to registered listeners. Time spent waiting for an intentional delay is not included in the ready message's age. You may use `--json` to read these statistics from an external monitoring tool. + +Horizon displays Redis jobs after they reach Redis, not while they are waiting in the outbox. The [OpenTelemetry integration](/docs/{{version}}/opentelemetry#queues) provides delivery counts, batch durations, and waiting times without polling the outbox or creating another producer span. + + +### Retrying Messages + +If you would like to retry messages before their next scheduled attempt, you may pass one or more UUIDs to the `queue:outbox-retry` command. This clears the last error but preserves the attempt count: + +```shell +php artisan queue:outbox-retry JOB_UUID +php artisan queue:outbox-retry all --database=billing +``` + +The `all` argument retries all previously attempted messages. You may limit it to one database using `--database`. If you are retrying a UUID that exists on more than one database, this option is required. + +Only messages that have already been attempted can be retried. Messages currently reserved by a relay cannot be retried manually. + + +### Deleting Messages + +To remove a message that should no longer be delivered, pass its UUID to the `queue:outbox-forget` command: + +```shell +php artisan queue:outbox-forget JOB_UUID --database=billing +``` + +This command asks for confirmation in every environment. You may pass `--force` to skip the prompt when running it unattended. Messages currently reserved by a relay cannot be deleted, even with `--force`, since an active delivery cannot be recalled. + +After deleting the record, Hypervel attempts to release the unique-job lock using its recorded cache store and lock owner. If the lock cannot be released, the command reports the failure, but the message remains deleted. For custom cache stores or context-dependent cache namespaces, you may extend `Hypervel\Queue\Console\ForgetOutboxCommand` and override its protected `releaseUniqueLock` method. The command reads only the stored lock metadata; it does not deserialize the job or restore arbitrary application context. + + +## Delivery Guarantees + +The outbox provides at-least-once delivery, which means a job may reach the queue more than once. For example, the relay could stop after the queue accepts a job but before its outbox record is deleted. When delivery resumes, the same job is sent again with the same UUID. + +Your jobs should therefore be safe to process more than once. For example, you may use a unique database constraint to prevent duplicate records, or pass a stable operation identifier to a payment provider that supports idempotent requests. The outbox does not detect duplicate processing or undo external actions. + +Jobs are not guaranteed to reach the queue in dispatch order. Concurrent relays, delays, and retries can change that order. SQS FIFO preserves arrival order within a message group, not the order in which your transactions stored jobs in the outbox. + +Successful rows are deleted rather than archived. The outbox records pending delivery, not a business audit trail or a list of failed job executions. Once a worker receives a job, ordinary queue retry and failed-job behavior applies. + + +## Custom Queue Drivers + +If you maintain a custom queue driver, its dispatch methods must use the base `Queue` class's enqueue methods to support the outbox. Methods that bypass this behavior continue sending jobs directly. You may customize outbox storage using the protected `resolveOutboxQueue`, `preparePayloadForOutbox`, `prepareOutboxOptions`, and `availableAtForOutbox` methods. These methods determine the queue name, stored payload, queue-specific options, and earliest delivery time while the original job is available. + +By default, the relay waits until a message is due and sends its stored payload through your driver's `pushRaw` method. It does not dispatch through the `Bus` facade or serialize the job again. If your driver uses a connection pool, the relay borrows one connection for the whole delivery batch. + +If your driver supports more efficient bulk delivery, you may implement `Hypervel\Contracts\Queue\PublishesOutboxMessages`. Its `publishOutboxMessages` method receives an array of immutable `Hypervel\Queue\Outbox\OutboxMessage` objects and returns a `Hypervel\Queue\Outbox\OutboxBatchResult`. The result contains the IDs of accepted messages, the IDs of messages deferred until their delivery time, and an optional failure. + +These IDs refer to outbox rows, not identifiers assigned by the queue. Array keys are ignored. Each ID must belong to an input message and appear only once across the accepted and deferred lists. A result without a failure must account for every message. Only messages with an intended delivery time may be deferred. + +If only part of a batch succeeds, include the confirmed messages in the result alongside the error. Do not report a message as accepted if its outcome is unknown. Allow coroutine cancellation to propagate, and do not dispatch ordinary enqueue events again; those events were already dispatched when the job entered the outbox. diff --git a/src/docs/queues.md b/src/docs/queues.md index 450a19cec2..cfdd22ed70 100644 --- a/src/docs/queues.md +++ b/src/docs/queues.md @@ -25,6 +25,7 @@ - [Bulk Dispatching](#bulk-dispatching) - [Preparing Jobs Before Dispatch](#preparing-jobs-before-dispatch) - [Jobs & Database Transactions](#jobs-and-database-transactions) + - [Transactional Outbox](#transactional-outbox) - [Job Chaining](#job-chaining) - [Customizing The Queue and Connection](#customizing-the-queue-and-connection) - [Specifying Max Job Attempts / Timeout Values](#max-job-attempts-and-timeout) @@ -1504,6 +1505,23 @@ Likewise, if the `after_commit` configuration option is set to `true`, you may i ProcessPodcast::dispatch($podcast)->beforeCommit(); ``` + +### Transactional Outbox + +Dispatching after commit prevents a worker from processing a job before your database changes are saved. However, your application could still stop after committing the transaction but before sending the job to the queue. To protect against losing the job in this case, you may use Hypervel's [transactional outbox](/docs/{{version}}/queue-outbox): + +```php +use Hypervel\Support\Facades\DB; + +DB::transaction(function () use ($podcast) { + $podcast->save(); + + ProcessPodcast::dispatch($podcast)->viaOutbox(); +}); +``` + +Hypervel stores the job as part of the database transaction. A separate relay process then sends it to your existing queue after commit. Before using the outbox, follow its [setup instructions](/docs/{{version}}/queue-outbox#setup), including creating the table and running the relay. + ### Job Chaining diff --git a/src/events/src/Dispatcher.php b/src/events/src/Dispatcher.php index 4226b2b693..66b84c5bee 100755 --- a/src/events/src/Dispatcher.php +++ b/src/events/src/Dispatcher.php @@ -25,6 +25,7 @@ use Hypervel\Contracts\Queue\ShouldBeUniqueUntilProcessing; use Hypervel\Contracts\Queue\ShouldQueue; use Hypervel\Contracts\Queue\ShouldQueueAfterCommit; +use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; use Hypervel\Database\DatabaseTransactionsManager; use Hypervel\Queue\Attributes\Backoff; use Hypervel\Queue\Attributes\Connection; @@ -1041,6 +1042,16 @@ protected function propagateListenerOptions(mixed $listener, CallQueuedListener $job->afterCommit = property_exists($listener, 'afterCommit') ? $listener->afterCommit : null; } + if (isset($listener->viaOutbox)) { + $job->viaOutbox = $listener->viaOutbox; + } elseif ($listener instanceof ShouldQueueViaOutbox) { + $job->viaOutbox = true; + } + + if (isset($listener->outboxDatabase)) { + $job->outboxDatabase = $listener->outboxDatabase; + } + $job->backoff = method_exists($listener, 'backoff') ? $listener->backoff(...$data) : $this->getAttributeValue($listener, Backoff::class, 'backoff'); $job->maxExceptions = $this->getAttributeValue($listener, MaxExceptions::class, 'maxExceptions'); $job->retryUntil = method_exists($listener, 'retryUntil') ? $listener->retryUntil(...$data) : null; diff --git a/src/foundation/config/queue.php b/src/foundation/config/queue.php index 962ff713ea..fd49879487 100644 --- a/src/foundation/config/queue.php +++ b/src/foundation/config/queue.php @@ -59,16 +59,19 @@ 'sync' => [ 'driver' => 'sync', 'after_commit' => false, + 'outbox' => false, ], 'background' => [ 'driver' => 'background', 'after_commit' => true, + 'outbox' => false, ], 'deferred' => [ 'driver' => 'deferred', 'after_commit' => true, + 'outbox' => false, ], 'database' => [ @@ -78,6 +81,7 @@ 'queue' => env('DB_QUEUE', 'default'), 'retry_after' => (int) env('DB_QUEUE_RETRY_AFTER', 90), 'after_commit' => false, + 'outbox' => false, ], 'beanstalkd' => [ @@ -88,6 +92,7 @@ 'retry_after' => (int) env('BEANSTALKD_QUEUE_RETRY_AFTER', 90), 'block_for' => 0, 'after_commit' => true, + 'outbox' => false, 'pool' => [ 'min_retained_objects' => 1, 'max_objects' => 10, @@ -108,6 +113,7 @@ 'suffix' => env('SQS_SUFFIX'), 'region' => env('AWS_DEFAULT_REGION', 'us-east-1'), 'after_commit' => true, + 'outbox' => false, 'overflow' => [ 'enabled' => (bool) env('SQS_OVERFLOW_ENABLED', false), 'store' => env('SQS_OVERFLOW_STORE'), @@ -133,6 +139,7 @@ // A value of 0 waits indefinitely and may delay worker shutdown. 'block_for' => null, 'after_commit' => true, + 'outbox' => false, ], 'failover' => [ @@ -142,9 +149,34 @@ 'deferred', ], 'after_commit' => true, + 'outbox' => false, ], ], + /* + |-------------------------------------------------------------------------- + | Transactional Outbox + |-------------------------------------------------------------------------- + | + | Outbox delivery stores jobs with your business transaction before sending + | them to a durable queue. Create the table on every listed database and + | run queue:relay. The retry timeout must exceed a full delivery batch, + | including transport timeouts and SDK retries, to avoid early redelivery. + | + | Jobs opt in with viaOutbox(), ShouldQueueViaOutbox, or their connection's + | outbox option. Local and failover queues retain after-commit behavior + | without crash durability; an already-atomic database queue inserts directly. + | + */ + + 'outbox' => [ + 'databases' => [ + env('QUEUE_OUTBOX_DATABASE', env('DB_CONNECTION', 'sqlite')), + ], + 'table' => env('QUEUE_OUTBOX_TABLE', 'job_outbox'), + 'retry_after' => (int) env('QUEUE_OUTBOX_RETRY_AFTER', 300), + ], + /* |-------------------------------------------------------------------------- | Job Batching diff --git a/src/foundation/src/Bus/PendingDispatch.php b/src/foundation/src/Bus/PendingDispatch.php index ace4fe525e..1f0bf3d271 100644 --- a/src/foundation/src/Bus/PendingDispatch.php +++ b/src/foundation/src/Bus/PendingDispatch.php @@ -146,6 +146,36 @@ public function beforeCommit(): static return $this; } + /** + * Dispatch the job through the transactional outbox. + */ + public function viaOutbox(): static + { + $this->job->viaOutbox(); + + return $this; + } + + /** + * Dispatch the job without the transactional outbox. + */ + public function withoutOutbox(): static + { + $this->job->withoutOutbox(); + + return $this; + } + + /** + * Stage the job on the given database connection. + */ + public function onOutboxDatabase(UnitEnum|string $connection): static + { + $this->job->onOutboxDatabase($connection); + + return $this; + } + /** * Set the jobs that should run if this job is successful. */ diff --git a/src/foundation/src/Testing/DatabaseTransactionsManager.php b/src/foundation/src/Testing/DatabaseTransactionsManager.php index 246e10736b..c9c7112db8 100644 --- a/src/foundation/src/Testing/DatabaseTransactionsManager.php +++ b/src/foundation/src/Testing/DatabaseTransactionsManager.php @@ -57,7 +57,7 @@ public function beginTransactions(DatabaseManager $database): void try { $connection->beginTransaction(); - $this->wrappingTransactions[] = $this->getCurrentTransactionForConnection($connection->getName() ?? ''); + $this->wrappingTransactions[] = $this->getCurrentTransactionForConnection($connection->getNameWithReadWriteType() ?? ''); } finally { if ($dispatcher !== null) { $connection->setEventDispatcher($dispatcher); diff --git a/src/foundation/src/Testing/RequestContextSynchronizer.php b/src/foundation/src/Testing/RequestContextSynchronizer.php index 1e79f64b2d..4e163f66e6 100644 --- a/src/foundation/src/Testing/RequestContextSynchronizer.php +++ b/src/foundation/src/Testing/RequestContextSynchronizer.php @@ -26,7 +26,7 @@ public function syncTransactionState( $names = []; foreach ($connections as $connection) { - $names[$connection->getName() ?? ''] = true; + $names[$connection->getNameWithReadWriteType() ?? ''] = true; } foreach (['pending', 'committed'] as $property) { diff --git a/src/horizon/README.md b/src/horizon/README.md index ff2fe64429..8a5fd43a71 100644 --- a/src/horizon/README.md +++ b/src/horizon/README.md @@ -11,5 +11,6 @@ Documentation: https://hypervel.org/docs/horizon - `horizon:listen` uses Hypervel's file watcher instead of a Node.js `chokidar` process, so Node is not required. A nonempty `horizon.watch` list replaces the `watcher` configuration's watch list, and `--poll` selects its scanning driver. See [automatically restarting Horizon](https://hypervel.org/docs/horizon#automatically-restarting-horizon). - Supervisors accept a `concurrency` option that lets each worker process run several jobs at once in coroutines. See [concurrency](https://hypervel.org/docs/horizon#concurrency). - Redis Cluster is configured on the named Redis connection selected by `horizon.use`. Laravel's top-level `database.redis.clusters` entries are not read. +- Horizon's `RedisQueue::push()` does not call `pushRaw()`. Custom queue subclasses should use `preparePayloadForPush()` and `handlePayloadPushed()` to customize normal publication; the latter runs after storage is accepted. Direct `pushRaw()` calls remain supported. Ported from: https://github.com/laravel/horizon diff --git a/src/horizon/src/JobPayload.php b/src/horizon/src/JobPayload.php index 4792e800fe..995845858e 100644 --- a/src/horizon/src/JobPayload.php +++ b/src/horizon/src/JobPayload.php @@ -95,25 +95,38 @@ public function isSilenced(): bool */ public function prepare(mixed $job): static { - return $this->prepareForQueueing($job)->markAsPushed(); + return $this->set([ + ...$this->queueingMetadata($job), + 'pushedAt' => str_replace(',', '.', (string) microtime(true)), + ]); } /** * Prepare the payload metadata while the original job is available. */ public function prepareForQueueing(mixed $job): static + { + $metadata = $this->queueingMetadata($job); + + return $metadata === [] ? $this : $this->set($metadata); + } + + /** + * Determine classification metadata without replacing it on raw retries. + */ + protected function queueingMetadata(mixed $job): array { if ($job === null && isset($this->decoded['type'])) { - return $this; + return []; } $tags = $this->determineTags($job); - return $this->set([ + return [ 'type' => $this->determineType($job), 'tags' => $tags, 'silenced' => $this->shouldBeSilenced($job, $tags), - ]); + ]; } /** diff --git a/src/horizon/src/RedisQueue.php b/src/horizon/src/RedisQueue.php index e0d27c008c..d90f5400e3 100644 --- a/src/horizon/src/RedisQueue.php +++ b/src/horizon/src/RedisQueue.php @@ -24,8 +24,6 @@ class RedisQueue extends BaseQueue { - public const string LAST_PUSHED_CONTEXT_KEY = '__horizon.queue.last_pushed'; - protected const string POPPING_QUEUE_CONTEXT_KEY = '__horizon.queue.popping'; /** @@ -49,11 +47,9 @@ public function push(object|string $job, mixed $data = '', UnitEnum|string|null $this->createPayload($job, $this->getQueue($queue), $data), $queue, null, - static function (BaseQueue $owner, $payload, $queue) use ($job) { + static function (BaseQueue $owner, string $payload, ?string $queue) { /** @var self $owner */ - $owner->setLastPushed($job); - - return $owner->pushRaw($payload, $queue); + return $owner->pushPreparedPayload($payload, $queue); } ); } @@ -64,25 +60,24 @@ static function (BaseQueue $owner, $payload, $queue) use ($job) { #[Override] public function pushRaw(string $payload, UnitEnum|string|null $queue = null, array $options = []): mixed { - $job = CoroutineContext::get(static::LAST_PUSHED_CONTEXT_KEY); + $payload = (new JobPayload($payload))->prepare(null); - try { - $payload = (new JobPayload($payload))->prepare($job); - - if ($this->hasEventListeners(JobPending::class)) { - $this->event($this->getQueue($queue), new JobPending($payload->value)); - } + if ($this->hasEventListeners(JobPending::class)) { + $this->event($this->getQueue($queue), new JobPending($payload->value)); + } - parent::pushRaw($payload->value, $queue, $options); + $this->pushPreparedPayload($payload->value, $queue, $options); + $this->handlePayloadPushed($payload->value, $this->normalizeQueue($queue)); - if ($this->hasEventListeners(JobPushed::class)) { - $this->event($this->getQueue($queue), new JobPushed($payload->value)); - } + return $payload->id(); + } - return $payload->id(); - } finally { - CoroutineContext::forget(static::LAST_PUSHED_CONTEXT_KEY); - } + /** + * Store a prepared payload without running Horizon's raw-push lifecycle again. + */ + protected function pushPreparedPayload(string $payload, UnitEnum|string|null $queue = null, array $options = []): mixed + { + return parent::pushRaw($payload, $queue, $options); } /** @@ -99,43 +94,33 @@ protected function createPayloadArray(array|object|string $job, ?string $queue, } /** - * Push a new job onto the queue after a delay. + * Classify a staged job without recording it as pending in Redis. */ - #[Override] - public function later(DateInterval|DateTimeInterface|int $delay, object|string $job, mixed $data = '', UnitEnum|string|null $queue = null): mixed + protected function preparePayloadForOutbox(object|string $job, string $payload): string { - $queue = $this->normalizeQueue($queue); - - return $this->enqueueUsing( - $job, - $this->createPayload($job, $this->getQueue($queue), $data, $delay), - $queue, - $delay, - static function (BaseQueue $owner, string $payload, ?string $queue, DateInterval|DateTimeInterface|int $delay) use ($job) { - // Horizon's Redis queue is not pooled, so after-commit dispatch returns this same queue type. - /** @var self $owner */ - $payload = (new JobPayload($payload))->prepare($job)->value; - - if ($owner->hasEventListeners(JobPending::class)) { - $owner->event($owner->getQueue($queue), new JobPending($payload)); - } + return (new JobPayload($payload))->prepareForQueueing($job)->value; + } - $id = $owner->laterRaw($delay, $payload, $queue); + /** + * Stamp a relayed payload and record its pending state before publication. + */ + #[Override] + protected function preparePayloadForOutboxPublication(string $payload, ?string $queue): string + { + $payload = (new JobPayload($payload))->markAsPushed()->value; - if ($owner->hasEventListeners(JobPushed::class)) { - $owner->event($owner->getQueue($queue), new JobPushed($payload)); - } + if ($this->hasEventListeners(JobPending::class)) { + $this->event($this->getQueue($queue), new JobPending($payload)); + } - return $id; - } - ); + return $payload; } /** - * Prepare a payload for bulk storage. + * Prepare a payload and record its pending state before publication. */ #[Override] - protected function preparePayloadForBulk(object|string $job, string $payload, ?string $queue): string + protected function preparePayloadForPush(object|string $job, string $payload, ?string $queue): string { $payload = (new JobPayload($payload))->prepare($job)->value; @@ -147,10 +132,10 @@ protected function preparePayloadForBulk(object|string $job, string $payload, ?s } /** - * Handle a payload that was stored as part of a batch. + * Notify Horizon after the queue has accepted the payload. */ #[Override] - protected function handlePayloadPushedInBulk(string $payload, ?string $queue): void + protected function handlePayloadPushed(string $payload, ?string $queue): void { if ($this->hasEventListeners(JobPushed::class)) { $this->event($this->getQueue($queue), new JobPushed($payload)); @@ -269,12 +254,4 @@ protected function hasEventListeners(string $event): bool return $this->container->bound(Dispatcher::class) && $this->container->make(Dispatcher::class)->hasListeners($event); } - - /** - * Set the job that last pushed to queue via the "push" method. - */ - protected function setLastPushed(object|string $job): void - { - CoroutineContext::set(static::LAST_PUSHED_CONTEXT_KEY, $job); - } } diff --git a/src/horizon/src/Repositories/RedisTagRepository.php b/src/horizon/src/Repositories/RedisTagRepository.php index 4c47e802c6..f2c49b1774 100644 --- a/src/horizon/src/Repositories/RedisTagRepository.php +++ b/src/horizon/src/Repositories/RedisTagRepository.php @@ -35,6 +35,10 @@ public function monitoring(): array */ public function monitored(array $tags): array { + if ($tags === []) { + return []; + } + return array_intersect($tags, $this->monitoring()); } diff --git a/src/mail/src/SendQueuedMailable.php b/src/mail/src/SendQueuedMailable.php index eef662a460..29ec7d74cf 100644 --- a/src/mail/src/SendQueuedMailable.php +++ b/src/mail/src/SendQueuedMailable.php @@ -10,6 +10,7 @@ use Hypervel\Contracts\Mail\Mailable as MailableContract; use Hypervel\Contracts\Queue\ShouldBeEncrypted; use Hypervel\Contracts\Queue\ShouldQueueAfterCommit; +use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; use Hypervel\Queue\Attributes\Backoff; use Hypervel\Queue\Attributes\Connection; use Hypervel\Queue\Attributes\MaxExceptions; @@ -60,6 +61,16 @@ public function __construct( $this->afterCommit = property_exists($mailable, 'afterCommit') ? $mailable->afterCommit : null; } + if (isset($mailable->viaOutbox)) { + $this->viaOutbox = $mailable->viaOutbox; + } elseif ($mailable instanceof ShouldQueueViaOutbox) { + $this->viaOutbox = true; + } + + if (isset($mailable->outboxDatabase)) { + $this->outboxDatabase = $mailable->outboxDatabase; + } + $this->connection = $this->getAttributeValue($mailable, Connection::class, 'connection'); $this->maxExceptions = $this->getAttributeValue($mailable, MaxExceptions::class, 'maxExceptions'); $this->queue = $this->getAttributeValue($mailable, QueueAttribute::class, 'queue'); diff --git a/src/notifications/src/SendQueuedNotifications.php b/src/notifications/src/SendQueuedNotifications.php index 566ce6d2ed..e3555df6cf 100644 --- a/src/notifications/src/SendQueuedNotifications.php +++ b/src/notifications/src/SendQueuedNotifications.php @@ -9,6 +9,7 @@ use Hypervel\Contracts\Queue\ShouldBeEncrypted; use Hypervel\Contracts\Queue\ShouldQueue; use Hypervel\Contracts\Queue\ShouldQueueAfterCommit; +use Hypervel\Contracts\Queue\ShouldQueueViaOutbox; use Hypervel\Database\Eloquent\Collection as EloquentCollection; use Hypervel\Database\Eloquent\Model; use Hypervel\Queue\Attributes\Backoff; @@ -95,6 +96,16 @@ public function __construct(mixed $notifiables, mixed $notification, ?array $cha $this->afterCommit = property_exists($notification, 'afterCommit') ? $notification->afterCommit : null; } + if (isset($notification->viaOutbox)) { + $this->viaOutbox = $notification->viaOutbox; + } elseif ($notification instanceof ShouldQueueViaOutbox) { + $this->viaOutbox = true; + } + + if (isset($notification->outboxDatabase)) { + $this->outboxDatabase = $notification->outboxDatabase; + } + $this->shouldBeEncrypted = $notification instanceof ShouldBeEncrypted; } diff --git a/src/opentelemetry/config/opentelemetry.php b/src/opentelemetry/config/opentelemetry.php index 1df81d4e95..a09dd81ce0 100644 --- a/src/opentelemetry/config/opentelemetry.php +++ b/src/opentelemetry/config/opentelemetry.php @@ -218,6 +218,9 @@ 'messaging.client.consumed.messages' => true, 'messaging.process.duration' => true, 'hypervel.queue.jobs' => false, + 'hypervel.queue.outbox.delivered.messages' => true, + 'hypervel.queue.outbox.delivery.duration' => true, + 'hypervel.queue.outbox.delivery.latency' => true, ], ], SchedulerInstrumentation::class => [ diff --git a/src/opentelemetry/src/Instrumentation/QueueInstrumentation.php b/src/opentelemetry/src/Instrumentation/QueueInstrumentation.php index 65d46e49bb..a310634277 100644 --- a/src/opentelemetry/src/Instrumentation/QueueInstrumentation.php +++ b/src/opentelemetry/src/Instrumentation/QueueInstrumentation.php @@ -22,6 +22,8 @@ use Hypervel\Queue\Events\JobQueued; use Hypervel\Queue\Events\JobQueueingFailed; use Hypervel\Queue\Events\JobTimedOut; +use Hypervel\Queue\Events\OutboxBatchDelivered; +use Hypervel\Queue\Events\OutboxBatchDeliveryFailed; use Hypervel\Queue\Queue; use Hypervel\Queue\QueueManager; use Hypervel\Queue\TimeoutExceededException; @@ -60,6 +62,12 @@ class QueueInstrumentation extends AbstractInstrumentation protected const string DEPTH_METRIC = 'hypervel.queue.jobs'; + protected const string OUTBOX_DELIVERED_METRIC = 'hypervel.queue.outbox.delivered.messages'; + + protected const string OUTBOX_DURATION_METRIC = 'hypervel.queue.outbox.delivery.duration'; + + protected const string OUTBOX_LATENCY_METRIC = 'hypervel.queue.outbox.delivery.latency'; + protected const string CONNECTION_ATTRIBUTE = 'hypervel.queue.connection'; protected const array DURATION_BOUNDARIES = [ @@ -89,6 +97,12 @@ class QueueInstrumentation extends AbstractInstrumentation protected ?HistogramInterface $processDuration = null; + protected ?CounterInterface $outboxDelivered = null; + + protected ?HistogramInterface $outboxDuration = null; + + protected ?HistogramInterface $outboxLatency = null; + protected bool $propagation = false; /** @var array */ @@ -131,6 +145,7 @@ protected function registerInstrumentation(): void } $this->registerMetrics(); + $this->registerOutboxMetrics(); $producerMetrics = $this->sentMessages !== null || $this->sendDuration !== null; $producerTelemetry = $this->tracer !== null || $producerMetrics; @@ -241,6 +256,68 @@ protected function registerMetrics(): void } } + /** + * Register outbox delivery metrics without observing or resolving ordinary queue operations. + */ + protected function registerOutboxMetrics(): void + { + $meter = null; + + if ($this->metricEnabled(self::OUTBOX_DELIVERED_METRIC)) { + $meter = $this->meterProvider->getMeter('hypervel.queue'); + $this->outboxDelivered = $meter->createCounter(self::OUTBOX_DELIVERED_METRIC, '{message}', 'Outbox messages confirmed by their destination queue.'); + } + + if ($this->metricEnabled(self::OUTBOX_DURATION_METRIC)) { + $meter ??= $this->meterProvider->getMeter('hypervel.queue'); + $this->outboxDuration = $meter->createHistogram( + self::OUTBOX_DURATION_METRIC, + 's', + 'Duration of outbox connection batch publication.', + ['ExplicitBucketBoundaries' => self::DURATION_BOUNDARIES], + ); + } + + if ($this->metricEnabled(self::OUTBOX_LATENCY_METRIC)) { + $meter ??= $this->meterProvider->getMeter('hypervel.queue'); + $this->outboxLatency = $meter->createHistogram( + self::OUTBOX_LATENCY_METRIC, + 's', + 'Age of the oldest eligible message at the start of each outbox connection publication batch.', + ['ExplicitBucketBoundaries' => [1, 5, 10, 30, 60, 300, 600, 1800, 3600]], + ); + } + + if ($meter !== null) { + $this->events->listen([OutboxBatchDelivered::class, OutboxBatchDeliveryFailed::class], function (OutboxBatchDelivered|OutboxBatchDeliveryFailed $event): void { + $this->recordOutboxDelivery($event); + }); + } + } + + /** + * Record confirmed acceptance separately from publication or settlement failure. + */ + protected function recordOutboxDelivery(OutboxBatchDelivered|OutboxBatchDeliveryFailed $event): void + { + $attributes = [ + self::CONNECTION_ATTRIBUTE => $event->connectionName, + 'hypervel.queue.outbox.database' => $event->database, + ]; + + if ($event->accepted > 0) { + $this->outboxDelivered?->add($event->accepted, $attributes); + } + + $this->outboxLatency?->record($event->oldestEligibleAge, $attributes); + + if ($event instanceof OutboxBatchDeliveryFailed) { + $attributes[ErrorAttributes::ERROR_TYPE] = $event->exception::class; + } + + $this->outboxDuration?->record($event->duration, $attributes); + } + /** * Inject propagation and start a persistent producer span. */ diff --git a/src/opentelemetry/src/Support/ConfigurationNormalizer.php b/src/opentelemetry/src/Support/ConfigurationNormalizer.php index 39b856ecfc..8c471e27fa 100644 --- a/src/opentelemetry/src/Support/ConfigurationNormalizer.php +++ b/src/opentelemetry/src/Support/ConfigurationNormalizer.php @@ -434,6 +434,9 @@ protected function instrumentationDefaults(): array 'messaging.client.consumed.messages' => true, 'messaging.process.duration' => true, 'hypervel.queue.jobs' => false, + 'hypervel.queue.outbox.delivered.messages' => true, + 'hypervel.queue.outbox.delivery.duration' => true, + 'hypervel.queue.outbox.delivery.latency' => true, ], ], SchedulerInstrumentation::class => [ diff --git a/src/pennant/src/Drivers/Decorator.php b/src/pennant/src/Drivers/Decorator.php index 7de16f1eb0..2e85772dcb 100644 --- a/src/pennant/src/Drivers/Decorator.php +++ b/src/pennant/src/Drivers/Decorator.php @@ -6,7 +6,6 @@ use Closure; use Hypervel\Contracts\Container\Container; -use Hypervel\Database\ConnectionName; use Hypervel\Database\DatabaseManager; use Hypervel\Database\DatabaseTransactionRecord; use Hypervel\Database\DatabaseTransactionsManager; @@ -975,13 +974,13 @@ protected function trackTransaction(PennantState $state, string $connection): vo return; } - $base = ConnectionName::parse($connection)->base; + $owner = $db->connection($connection)->getNameWithReadWriteType(); /** @var DatabaseTransactionsManager $transactions */ $transactions = $this->container->make('db.transactions'); $state->forgetOnRollback( - $transactions->callbackApplicableTransactions()->last(fn (DatabaseTransactionRecord $record) => $record->connection === $base), + $transactions->callbackApplicableTransactions()->last(fn (DatabaseTransactionRecord $record) => $record->connection === $owner), $connection, ); } diff --git a/src/queue/src/BeanstalkdQueue.php b/src/queue/src/BeanstalkdQueue.php index d84d90ffbd..27773944ce 100644 --- a/src/queue/src/BeanstalkdQueue.php +++ b/src/queue/src/BeanstalkdQueue.php @@ -300,6 +300,22 @@ public function getQueue(UnitEnum|string|null $queue): string return $this->resolveQueue($queue === null || $queue === '' ? $this->default : $queue); } + /** + * Capture the logical destination before publication-time forwarding. + */ + protected function resolveOutboxQueue(?string $queue): ?string + { + return $queue === null || $queue === '' ? $this->default : $queue; + } + + /** + * Prepare bulk payloads with the same hook destination as individual pushes. + */ + protected function prepareBatchJobs(array $jobs, mixed $data, ?string $queue): array + { + return parent::prepareBatchJobs($jobs, $data, $this->getQueue($queue)); + } + /** * Get the underlying Pheanstalk instance. * diff --git a/src/queue/src/Concerns/InsertsDatabaseRows.php b/src/queue/src/Concerns/InsertsDatabaseRows.php index 9ab34c0318..3ba8967e46 100644 --- a/src/queue/src/Concerns/InsertsDatabaseRows.php +++ b/src/queue/src/Concerns/InsertsDatabaseRows.php @@ -5,6 +5,7 @@ namespace Hypervel\Queue\Concerns; use Hypervel\Database\ConnectionInterface; +use Hypervel\Database\PdoConnection; use RuntimeException; trait InsertsDatabaseRows @@ -18,9 +19,8 @@ protected function insertDatabaseRows( ConnectionInterface $connection, string $table, array $rows, - int $maxBindings, ): void { - $rowsPerInsert = intdiv($maxBindings, count(reset($rows))); + $rowsPerInsert = intdiv($this->maxBindings($connection), count(reset($rows))); if (count($rows) <= $rowsPerInsert) { $this->insertDatabaseRowChunk($connection, $table, $rows); @@ -46,4 +46,14 @@ protected function insertDatabaseRowChunk(ConnectionInterface $connection, strin throw new RuntimeException('Unable to insert queued jobs into the database.'); } } + + /** + * Get the statement binding limit without requiring a PDO-backed custom driver. + */ + protected function maxBindings(ConnectionInterface $connection): int + { + return $connection instanceof PdoConnection + ? $connection->maxBindings() + : PdoConnection::DEFAULT_MAX_BINDINGS; + } } diff --git a/src/queue/src/Concerns/StagesOutboxJobs.php b/src/queue/src/Concerns/StagesOutboxJobs.php new file mode 100644 index 0000000000..35c2406fbb --- /dev/null +++ b/src/queue/src/Concerns/StagesOutboxJobs.php @@ -0,0 +1,271 @@ +viaOutbox ?? ($job instanceof ShouldQueueViaOutbox || $this->dispatchViaOutbox); + + return $selected && (CoroutineContext::get(self::OUTBOX_FORWARDED_CONTEXT_KEY)['job'] ?? null) !== $job; + } + + /** + * Run a child attempt or handler with its own forwarding context. + * + * @param null|array{job: object|string, afterCommitHandled: bool} $forwarded + */ + protected function withOutboxForwarding(?array $forwarded, callable $callback): mixed + { + $previous = CoroutineContext::get(self::OUTBOX_FORWARDED_CONTEXT_KEY); + + if ($previous === $forwarded) { + return $callback(); + } + + $forwarded === null + ? CoroutineContext::forget(self::OUTBOX_FORWARDED_CONTEXT_KEY) + : CoroutineContext::set(self::OUTBOX_FORWARDED_CONTEXT_KEY, $forwarded); + + try { + return $callback(); + } finally { + $previous === null + ? CoroutineContext::forget(self::OUTBOX_FORWARDED_CONTEXT_KEY) + : CoroutineContext::set(self::OUTBOX_FORWARDED_CONTEXT_KEY, $previous); + } + } + + /** + * Determine whether this driver needs SQL staging for the selected job. + */ + protected function shouldStageViaOutbox(object|string $job): bool + { + return $this->shouldDispatchViaOutbox($job); + } + + /** + * Select an explicit transaction for a local or failover after-commit fallback. + */ + protected function resolveOutboxCommitConnection(DatabaseTransactionsManager $transactions, object|string $job): ?string + { + if (! $this->shouldDispatchViaOutbox($job) || ($database = $job->outboxDatabase ?? null) === null) { + return null; + } + + /** @var ConnectionResolverInterface $resolver */ + $resolver = $this->container->make('db'); + $owner = $resolver->connection($database)->getNameWithReadWriteType() ?? ''; + $active = $transactions->callbackApplicableTransactions(); + + if ($active->isNotEmpty() && ! $transactions->hasApplicableTransaction($owner)) { + throw new LogicException("Outbox database [{$database}] does not own an active transaction."); + } + + return $owner; + } + + /** + * Stage a single job through its normal guarded enqueue lifecycle. + */ + protected function enqueueViaOutbox(object|string $job, string $payload, ?string $queue, DateInterval|DateTimeInterface|int|null $delay): string + { + try { + $dispatcher = $this->container->make(OutboxDispatcher::class); + $connection = $dispatcher->databaseFor($job); + $payload = $this->finalizePayloadForQueueing($queue, $job, $payload, $delay); + $row = $this->prepareOutboxRow($job, $payload, $queue, $delay); + $this->raiseJobQueueingEvent($queue, $job, $payload, $delay); + $dispatcher->store($connection, [$row], [$job]); + } catch (CanceledException $exception) { + throw $exception; + } catch (Throwable $exception) { + $this->raiseJobQueueingFailedEvent($queue, $job, $payload, $delay, $exception); + + throw $exception; + } + + $this->acceptDispatchLocks($job); + $this->raiseJobQueuedEvent($queue, $row['uuid'], $job, $payload, $delay); + + return $row['uuid']; + } + + /** + * Stage selected jobs in owner groups and return the direct remainder. + */ + protected function stageOutboxJobs(array $jobs, mixed $data, ?string $queue): array + { + $direct = $jobs; + $selected = []; + + foreach ($jobs as $index => $job) { + if ($this->shouldStageViaOutbox($job)) { + $selected[] = $job; + unset($direct[$index]); + } + } + + if ($selected === []) { + return $jobs; + } + + $dispatcher = $this->container->make(OutboxDispatcher::class); + $prepared = $this->prepareBatchJobs($selected, $data, $queue); + $groups = []; + + try { + // Resolve every owner before finalization or insertion of any group. + foreach ($prepared as $index => $member) { + $connection = $dispatcher->databaseFor($member['job']); + $name = $connection->getNameWithReadWriteType() ?? ''; + $groups[$name]['connection'] = $connection; + $groups[$name]['members'][] = $index; + } + + foreach ($groups as $group) { + $rows = []; + $groupJobs = []; + + foreach ($group['members'] as $index) { + $member = &$prepared[$index]; + $member['payload'] = $this->finalizePayloadForQueueing($queue, $member['job'], $member['payload'], $member['delay']); + $rows[] = $this->prepareOutboxRow($member['job'], $member['payload'], $queue, $member['delay']); + $groupJobs[] = $member['job']; + unset($member); + } + + foreach ($group['members'] as $index) { + $member = $prepared[$index]; + $this->raiseJobQueueingEvent($queue, $member['job'], $member['payload'], $member['delay']); + } + + $dispatcher->store($group['connection'], $rows, $groupJobs); + $accepted = []; + + foreach ($group['members'] as $position => $index) { + $member = $prepared[$index]; + $this->acceptDispatchLocks($member['job']); + $member['uuid'] = $rows[$position]['uuid']; + $accepted[] = $member; + unset($prepared[$index]); + } + + $failure = $this->notifyBatchMembers($accepted, function (array $member) use ($queue): void { + $this->raiseJobQueuedEvent($queue, $member['uuid'], $member['job'], $member['payload'], $member['delay']); + }); + + if ($failure !== null) { + throw $failure; + } + } + } catch (CanceledException $exception) { + throw $exception; + } catch (Throwable $exception) { + $failure = $this->notifyBatchMembers($prepared, function (array $member) use ($queue, $exception): void { + $this->raiseJobQueueingFailedEvent($queue, $member['job'], $member['payload'], $member['delay'], $exception); + }); + + throw $failure ?? $exception; + } + + return array_values($direct); + } + + /** + * Build a byte-preserving row from a finalized payload. + */ + protected function prepareOutboxRow(object|string $job, string $payload, ?string $queue, DateInterval|DateTimeInterface|int|null $delay): array + { + $storedPayload = $this->preparePayloadForOutbox($job, $payload); + + try { + $decoded = json_decode($storedPayload, true, flags: JSON_THROW_ON_ERROR); + } catch (JsonException $exception) { + throw new InvalidPayloadException('Unable to decode the outbox payload: ' . $exception->getMessage(), $storedPayload); + } + + if (! is_array($decoded) || ! is_string($decoded['uuid'] ?? null) || ! Str::isUuid($decoded['uuid'])) { + throw new InvalidPayloadException('The outbox payload must contain a valid job UUID.', $storedPayload); + } + + $queue = $this->resolveOutboxQueue($queue); + $now = $this->currentTime(); + // Even an already-due timer overrides SQS's queue-level delay. + $deliverAt = empty($delay) ? null : max($this->availableAt($delay), $now); + $options = $this->prepareOutboxOptions($job, $queue, $storedPayload); + + return [ + 'uuid' => $decoded['uuid'], + 'connection' => $this->connectionName, + 'queue' => $queue, + 'payload' => $storedPayload, + 'options' => $options === [] ? null : json_encode($options, JSON_THROW_ON_ERROR), + 'deliver_at' => $deliverAt, + 'attempts' => 0, + 'available_at' => $this->availableAtForOutbox($queue, $deliverAt), + 'lease_token' => null, + 'last_error' => null, + 'created_at' => $now, + ]; + } + + /** + * Expand the logical destination without applying publication-time forwarding. + */ + protected function resolveOutboxQueue(?string $queue): ?string + { + return $queue; + } + + /** + * Add metadata required while the original job is still available. + */ + protected function preparePayloadForOutbox(object|string $job, string $payload): string + { + return $payload; + } + + /** + * Capture transport options that cannot be recovered from the stored envelope. + */ + protected function prepareOutboxOptions(object|string $job, ?string $queue, string $payload): array + { + return []; + } + + /** + * Determine when a stored message may first be published. + */ + protected function availableAtForOutbox(?string $queue, ?int $deliverAt): int + { + return $deliverAt ?? $this->currentTime(); + } +} diff --git a/src/queue/src/Connectors/SqsConnector.php b/src/queue/src/Connectors/SqsConnector.php index 2603d73d69..ceb872a6fb 100644 --- a/src/queue/src/Connectors/SqsConnector.php +++ b/src/queue/src/Connectors/SqsConnector.php @@ -90,6 +90,7 @@ protected function clientConfiguration(array $config): array 'prefix', 'suffix', 'after_commit', + 'outbox', 'key', 'secret', 'token', diff --git a/src/queue/src/Console/Concerns/FindsOutboxMessages.php b/src/queue/src/Console/Concerns/FindsOutboxMessages.php new file mode 100644 index 0000000000..a30acb25b8 --- /dev/null +++ b/src/queue/src/Console/Concerns/FindsOutboxMessages.php @@ -0,0 +1,34 @@ +repository->getDatabases() : [$database] as $source) { + if (($message = $this->repository->find($source, $uuid)) === null) { + continue; + } + + if ($found !== null) { + throw new RuntimeException("Outbox message [{$uuid}] exists on multiple databases. Select one using --database."); + } + + $found = ['database' => $source, 'message' => $message]; + } + + return $found; + } +} diff --git a/src/queue/src/Console/ForgetOutboxCommand.php b/src/queue/src/Console/ForgetOutboxCommand.php new file mode 100644 index 0000000000..f0a35fa599 --- /dev/null +++ b/src/queue/src/Console/ForgetOutboxCommand.php @@ -0,0 +1,117 @@ +argument('uuid'); + + // Ask before querying so an interactive prompt never holds a database connection. + if (! $this->confirmToProceed("Permanently discard outbox message [{$uuid}]", true)) { + return self::FAILURE; + } + + $found = $this->findOutboxMessage($uuid, $this->option('database')); + + if ($found === null) { + $this->components->error("No outbox message matches UUID [{$uuid}]."); + + return self::FAILURE; + } + + $message = $found['message']; + + if (! $this->repository->forget($found['database'], (int) $message->id, $message->lease_token)) { + $this->components->error("Outbox message [{$uuid}] is currently leased or changed during deletion."); + + return self::FAILURE; + } + + try { + $this->releaseUniqueLock($message->payload); + } catch (CanceledException $exception) { + throw $exception; + } catch (Throwable $exception) { + $this->components->error("Outbox message [{$uuid}] was deleted, but unique-lock cleanup failed: {$exception->getMessage()}"); + $this->exceptions->report($exception, ['outbox_database' => $found['database'], 'job_uuid' => $uuid]); + + return self::FAILURE; + } + + $this->components->info("Outbox message [{$uuid}] was deleted."); + + return self::SUCCESS; + } + + /** + * Release exact stored ownership without constructing the job or hydrating application context. + * + * Override this method to establish a custom cache namespace or resolve an ad hoc repository. + */ + protected function releaseUniqueLock(string $payload): void + { + $hidden = json_decode($payload, true, flags: JSON_THROW_ON_ERROR)['illuminate:log:context']['hidden'] ?? []; + + if (! array_key_exists('laravel_unique_job_key', $hidden)) { + return; + } + + $store = unserialize($hidden['laravel_unique_job_cache_store'] ?? 'N;', ['allowed_classes' => false]); + $key = unserialize($hidden['laravel_unique_job_key'], ['allowed_classes' => false]); + $owner = unserialize($hidden['laravel_unique_job_lock_owner'] ?? 'N;', ['allowed_classes' => false]); + + if (! is_string($store) || $store === '' || ! is_string($key) || $key === '' || ! is_string($owner) || $owner === '') { + throw new LogicException('Unique-lock cleanup requires a named cache store, key, and owner token.'); + } + + $cache = $this->cache->store($store); + + if (! $cache->getStore() instanceof LockProvider) { + throw new LogicException("Cache store [{$store}] does not support owner-matched lock cleanup."); + } + + UniqueLock::releaseOwned($cache, $key, $owner); + } +} diff --git a/src/queue/src/Console/ListOutboxCommand.php b/src/queue/src/Console/ListOutboxCommand.php new file mode 100644 index 0000000000..835dc39659 --- /dev/null +++ b/src/queue/src/Console/ListOutboxCommand.php @@ -0,0 +1,90 @@ +option('limit'); + + if ($limit < 1 || $limit > 1000) { + throw new InvalidArgumentException('The outbox listing limit must be between 1 and 1000.'); + } + + $database = $this->argument('database'); + $rows = []; + $now = $this->currentTime(); + + foreach ($database === null ? $this->repository->getDatabases() : [$database] as $source) { + foreach ($this->repository->pending($source, $limit) as $message) { + $payload = json_decode($message->payload, true, flags: JSON_THROW_ON_ERROR); + $rows[] = [ + 'database' => $source, + 'id' => (int) $message->id, + 'uuid' => $message->uuid, + 'job' => $payload['displayName'] ?? $payload['job'] ?? '', + 'connection' => $message->connection, + 'queue' => $message->queue, + 'attempts' => (int) $message->attempts, + 'state' => match (true) { + $message->lease_token !== null && (int) $message->available_at > $now => 'leased', + (int) $message->available_at <= $now => 'eligible', + (int) $message->attempts > 0 => 'retrying', + default => 'delayed', + }, + 'age' => max(0, $now - (int) $message->created_at), + 'available_at' => (int) $message->available_at, + 'last_error' => $message->last_error, + ]; + } + } + + if ($this->option('json')) { + $this->output->writeln(json_encode($rows, JSON_THROW_ON_ERROR), OutputInterface::OUTPUT_RAW); + } elseif ($rows === []) { + $this->components->info('No pending outbox messages found.'); + } else { + $rows = array_map(static function (array $row): array { + $row['available_at'] = Date::createFromTimestampUTC($row['available_at'])->toDateTimeString(); + $row['last_error'] = Str::limit(Str::before($row['last_error'] ?? '', "\n"), 80); + + return $row; + }, $rows); + + $this->table(['Database', 'ID', 'UUID', 'Job', 'Connection', 'Queue', 'Attempts', 'State', 'Age (s)', 'Available At (UTC)', 'Last Error'], $rows); + } + } +} diff --git a/src/queue/src/Console/MonitorOutboxCommand.php b/src/queue/src/Console/MonitorOutboxCommand.php new file mode 100644 index 0000000000..c5efe8bafa --- /dev/null +++ b/src/queue/src/Console/MonitorOutboxCommand.php @@ -0,0 +1,68 @@ +option('max') === null ? null : (int) $this->option('max'); + $age = $this->option('age') === null ? null : (int) $this->option('age'); + + if (($maximum !== null && $maximum < 0) || ($age !== null && $age < 0)) { + throw new InvalidArgumentException('Outbox monitoring thresholds must be non-negative.'); + } + + $database = $this->argument('database'); + $rows = []; + + foreach ($database === null ? $this->repository->getDatabases() : [$database] as $source) { + $statistics = $this->repository->statistics($source); + $alert = ($maximum !== null && $statistics['total'] >= $maximum) + || ($age !== null && $statistics['oldest_eligible_age'] !== null && $statistics['oldest_eligible_age'] >= $age); + $rows[] = ['database' => $source, ...$statistics, 'status' => $alert ? 'ALERT' : 'OK']; + + if ($alert && $this->events->hasListeners(OutboxBusy::class)) { + $this->events->dispatch(new OutboxBusy($source, $statistics['total'], $statistics['eligible'], $statistics['oldest_eligible_age'])); + } + } + + if ($this->option('json')) { + $this->output->writeln(json_encode($rows, JSON_THROW_ON_ERROR), OutputInterface::OUTPUT_RAW); + } else { + $this->table(['Database', 'Total', 'Eligible', 'Delayed', 'Leased', 'Retrying', 'Oldest Eligible (s)', 'Status'], $rows); + } + } +} diff --git a/src/queue/src/Console/OutboxTableCommand.php b/src/queue/src/Console/OutboxTableCommand.php new file mode 100644 index 0000000000..06b4c32bd3 --- /dev/null +++ b/src/queue/src/Console/OutboxTableCommand.php @@ -0,0 +1,45 @@ +hypervel->make('config')->string('queue.outbox.table'); + } + + /** + * Get the path to the migration stub file. + */ + protected function migrationStubFile(): string + { + return __DIR__ . '/stubs/outbox.stub'; + } +} diff --git a/src/queue/src/Console/RelayCommand.php b/src/queue/src/Console/RelayCommand.php new file mode 100644 index 0000000000..65ee006ed7 --- /dev/null +++ b/src/queue/src/Console/RelayCommand.php @@ -0,0 +1,61 @@ +argument('database'); + + // Avoid Symfony's stty subprocess probe inside this coroutine. + if (defined('STDIN') && stream_isatty(STDIN)) { + $this->components->info($database === null + ? 'Relaying outbox messages from all configured databases.' + : "Relaying outbox messages from [{$database}]."); + } + + return $this->worker->run($database, new OutboxWorkerOptions( + batch: (int) $this->option('batch'), + sleep: (float) $this->option('sleep'), + concurrency: (int) $this->option('concurrency'), + maxJobs: (int) $this->option('max-jobs'), + maxTime: (int) $this->option('max-time'), + memory: (float) $this->option('memory'), + stopWhenEmpty: (bool) $this->option('stop-when-empty'), + once: (bool) $this->option('once'), + )); + } +} diff --git a/src/queue/src/Console/RetryOutboxCommand.php b/src/queue/src/Console/RetryOutboxCommand.php new file mode 100644 index 0000000000..f498cb472c --- /dev/null +++ b/src/queue/src/Console/RetryOutboxCommand.php @@ -0,0 +1,74 @@ +argument('uuid'))); + $database = $this->option('database'); + + if ($uuids === ['all']) { + $count = 0; + + foreach ($database === null ? $this->repository->getDatabases() : [$database] as $source) { + $count += $this->repository->retryAll($source); + } + + $this->components->info("Made {$count} outbox messages available for retry."); + + return self::SUCCESS; + } + + if ($uuids === [] || in_array('all', $uuids, true)) { + throw new InvalidArgumentException('Specify one or more job UUIDs, or "all" by itself.'); + } + + $status = self::SUCCESS; + + foreach ($uuids as $uuid) { + $found = $this->findOutboxMessage($uuid, $database); + + if ($found === null) { + $this->components->error("No outbox message matches UUID [{$uuid}]."); + $status = self::FAILURE; + } elseif ($this->repository->retry($found['database'], $uuid) === 0) { + $this->components->error("Outbox message [{$uuid}] has not been attempted or is currently leased."); + $status = self::FAILURE; + } else { + $this->components->info("Outbox message [{$uuid}] is available for retry."); + } + } + + return $status; + } +} diff --git a/src/queue/src/Console/stubs/outbox.stub b/src/queue/src/Console/stubs/outbox.stub new file mode 100644 index 0000000000..bc2782d04a --- /dev/null +++ b/src/queue/src/Console/stubs/outbox.stub @@ -0,0 +1,41 @@ +id(); + $table->uuid('uuid')->unique(); + $table->string('connection'); + $table->string('queue')->nullable(); + $table->longText('payload'); + $table->text('options')->nullable(); + $table->unsignedBigInteger('deliver_at')->nullable(); + $table->unsignedInteger('attempts')->default(0); + $table->unsignedBigInteger('available_at'); + $table->uuid('lease_token')->nullable(); + $table->text('last_error')->nullable(); + $table->unsignedBigInteger('created_at'); + + $table->index(['available_at', 'id']); + }); + } + + /** + * Reverse the migrations. + */ + public function down(): void + { + Schema::dropIfExists('{{table}}'); + } +}; diff --git a/src/queue/src/DatabaseQueue.php b/src/queue/src/DatabaseQueue.php index ae7b78121c..b533b63346 100644 --- a/src/queue/src/DatabaseQueue.php +++ b/src/queue/src/DatabaseQueue.php @@ -6,8 +6,10 @@ use DateInterval; use DateTimeInterface; +use Hypervel\Bus\DispatchLockContext; use Hypervel\Contracts\Queue\ClearableQueue; use Hypervel\Contracts\Queue\Job; +use Hypervel\Contracts\Queue\PublishesOutboxMessages; use Hypervel\Contracts\Queue\Queue as QueueContract; use Hypervel\Database\ConnectionInterface; use Hypervel\Database\ConnectionResolverInterface; @@ -21,13 +23,15 @@ use Hypervel\Queue\Jobs\DatabaseJob; use Hypervel\Queue\Jobs\DatabaseJobRecord; use Hypervel\Queue\Jobs\InspectedJob; +use Hypervel\Queue\Outbox\OutboxBatchResult; +use Hypervel\Queue\Outbox\OutboxMessage; use Hypervel\Support\CarbonImmutable; use Hypervel\Support\Collection; use Swoole\Coroutine\CanceledException; use Throwable; use UnitEnum; -class DatabaseQueue extends Queue implements QueueContract, ClearableQueue +class DatabaseQueue extends Queue implements QueueContract, ClearableQueue, PublishesOutboxMessages { use DetectsConcurrencyErrors; use DetectsLostConnections; @@ -279,8 +283,11 @@ public function push(object|string $job, mixed $data = '', UnitEnum|string|null $this->createPayload($job, $this->getQueue($queue), $data), $queue, null, - static function (DatabaseQueue $owner, string $payload, ?string $queue) { - return $owner->pushToDatabase($queue, $payload); + static function (DatabaseQueue $owner, string $payload, ?string $queue) use ($job) { + $id = $owner->pushToDatabase($queue, $payload); + $owner->addStoredJobRollbackCallback($job); + + return $id; } ); } @@ -310,8 +317,11 @@ static function ( string $payload, ?string $queue, DateInterval|DateTimeInterface|int $delay - ) { - return $owner->pushToDatabase($queue, $payload, $delay); + ) use ($job) { + $id = $owner->pushToDatabase($queue, $payload, $delay); + $owner->addStoredJobRollbackCallback($job); + + return $id; } ); } @@ -326,7 +336,7 @@ static function ( public function bulk(array $jobs, mixed $data = '', UnitEnum|string|null $queue = null): mixed { $queue = $this->normalizeQueue($queue); - $jobs = array_values($jobs); + $jobs = array_values($this->stageOutboxJobs($jobs, $data, $queue)); if ($jobs === []) { return null; @@ -372,17 +382,7 @@ static function (Queue $owner) use ($preparedJobs, $queue): mixed { */ protected function prepareBatchJobs(array $jobs, mixed $data, ?string $queue): array { - return (new Collection($jobs)) - ->map(function (object|string $job) use ($data, $queue): array { - $delay = $this->getJobDelay($job); - - return [ - 'job' => $job, - 'delay' => $delay, - 'payload' => $this->createPayload($job, $this->getQueue($queue), $data, $delay), - ]; - }) - ->all(); + return parent::prepareBatchJobs($jobs, $data, $this->getQueue($queue)); } /** @@ -402,15 +402,13 @@ protected function enqueueBatch(array $jobs, ?string $queue): mixed // Every payload must be final before any batch member begins its // existing JobQueueing lifecycle or the database write. - foreach ($jobs as $job) { + foreach ($jobs as $index => $job) { $this->raiseJobQueueingEvent($queue, $job['job'], $job['payload'], $job['delay']); + $jobs[$index]['storedPayload'] = $this->preparePayloadForPush($job['job'], $job['payload'], $queue); } $now = $this->availableAt(); $connection = $this->getDatabase(); - $maxBindings = $connection instanceof PdoConnection - ? $connection->maxBindings() - : PdoConnection::DEFAULT_MAX_BINDINGS; $this->insertDatabaseRows( $connection, @@ -418,33 +416,99 @@ protected function enqueueBatch(array $jobs, ?string $queue): mixed (new Collection($jobs)) ->map(fn (array $job): array => $this->buildDatabaseRecord( $this->getQueue($queue), - $job['payload'], + $job['storedPayload'], $job['delay'] !== null ? $this->availableAt($job['delay']) : $now, ))->all(), - $maxBindings, ); - $result = true; - } catch (CanceledException $exception) { - throw $exception; - } catch (Throwable $exception) { + foreach ($jobs as $job) { - $this->raiseJobQueueingFailedEvent($queue, $job['job'], $job['payload'], $job['delay'], $exception); + $this->addStoredJobRollbackCallback($job['job'], $connection); } + $result = true; + } catch (CanceledException $exception) { throw $exception; + } catch (Throwable $exception) { + throw $this->notifyBatchMembers( + $jobs, + fn (array $job) => $this->raiseJobQueueingFailedEvent($queue, $job['job'], $job['payload'], $job['delay'], $exception), + ) ?? $exception; } foreach ($jobs as $job) { $this->acceptDispatchLocks($job['job']); } - foreach ($jobs as $job) { - $this->raiseJobQueuedEvent($queue, null, $job['job'], $job['payload'], $job['delay']); + $exception = $this->notifyBatchMembers( + $jobs, + fn (array $job) => $this->handlePayloadPushed($job['storedPayload'], $queue), + fn (array $job) => $this->raiseJobQueuedEvent($queue, null, $job['job'], $job['payload'], $job['delay']), + ); + + if ($exception !== null) { + throw $exception; } return $result; } + /** + * Release accepted dispatch locks if their stored job is rolled back. + */ + protected function addStoredJobRollbackCallback(object|string $job, ?ConnectionInterface $connection = null): void + { + $snapshot = is_object($job) ? DispatchLockContext::snapshot($job) : null; + + if ($snapshot === null) { + return; + } + + $connection ??= $this->getDatabase(); + + if ($connection->transactionLevel() > 0) { + /** @var DatabaseTransactionsManager $transactions */ + $transactions = $this->container->make('db.transactions'); + $transactions->addCallbackForRollback( + static fn () => DispatchLockContext::releaseSnapshot($snapshot), + $connection->getNameWithReadWriteType() ?? '', + ); + } + } + + /** + * Publish a raw outbox batch atomically across its logical queues. + * + * @param list $messages + */ + public function publishOutboxMessages(array $messages): OutboxBatchResult + { + if ($messages === []) { + return new OutboxBatchResult; + } + + try { + $connection = $this->getDatabase(); + $now = $this->currentTime(); + $rows = []; + + foreach ($messages as $message) { + $rows[] = $this->buildDatabaseRecord( + $this->getQueue($message->queue), + $message->payload, + max($now, $message->deliverAt ?? $now), + ); + } + + $this->insertDatabaseRows($connection, $this->table, $rows); + } catch (CanceledException $exception) { + throw $exception; + } catch (Throwable $exception) { + return new OutboxBatchResult(failure: $exception); + } + + return new OutboxBatchResult(array_column($messages, 'id')); + } + /** * Release a reserved job back onto the queue after (n) seconds. */ @@ -691,6 +755,59 @@ public function getQueue(UnitEnum|string|null $queue): string return $this->resolveQueue($queue === null || $queue === '' ? $this->default : $queue); } + /** + * Avoid a second durable table when the jobs insert already belongs to its owner. + */ + protected function shouldStageViaOutbox(object|string $job): bool + { + if (! $this->shouldDispatchViaOutbox($job) || ! $this->container->bound('db.transactions')) { + return false; + } + + /** @var DatabaseTransactionsManager $transactions */ + $transactions = $this->container->make('db.transactions'); + $active = []; + + foreach ($transactions->callbackApplicableTransactions() as $transaction) { + $active[$transaction->connection] = true; + } + + if ($active === []) { + return false; + } + + $target = $this->getDatabase()->getNameWithReadWriteType() ?? ''; + $explicit = $job->outboxDatabase ?? null; + + return ! (isset($active[$target]) && ($explicit === null + ? count($active) === 1 + : $this->resolver->connection($explicit)->getNameWithReadWriteType() === $target)); + } + + /** + * Keep an already-atomic outbox-selected insert inside its transaction. + */ + protected function shouldDispatchAfterCommit(object|string $job): bool + { + return ! $this->shouldDispatchViaOutbox($job) && parent::shouldDispatchAfterCommit($job); + } + + /** + * Capture the logical destination before publication-time forwarding. + */ + protected function resolveOutboxQueue(?string $queue): ?string + { + return $queue === null || $queue === '' ? $this->default : $queue; + } + + /** + * Publish delayed database jobs immediately with their future availability. + */ + protected function availableAtForOutbox(?string $queue, ?int $deliverAt): int + { + return $this->currentTime(); + } + /** * Get the underlying database connection. */ diff --git a/src/queue/src/Events/OutboxBatchDelivered.php b/src/queue/src/Events/OutboxBatchDelivered.php new file mode 100644 index 0000000000..2d55933b58 --- /dev/null +++ b/src/queue/src/Events/OutboxBatchDelivered.php @@ -0,0 +1,20 @@ +queueRoutes()->forwardedQueueForConnection($queue, $this->connectionName ?? null); } + /** + * Preserve failover attempts instead of capturing an undecided destination. + */ + protected function shouldStageViaOutbox(object|string $job): bool + { + return false; + } + + /** + * Degrade selected outbox delivery to an after-commit failover attempt. + */ + protected function shouldDispatchAfterCommit(object|string $job): bool + { + return $this->shouldDispatchViaOutbox($job) || parent::shouldDispatchAfterCommit($job); + } + /** * Attempt the given method on all connections. * @@ -248,16 +264,38 @@ protected function attemptOnAllConnections(string $method, array $arguments, obj } } + return $this->attemptOnConnections($method, $arguments, $job); + } + + /** + * Forward an operation to the first available child connection. + */ + protected function attemptOnConnections( + string $method, + array $arguments, + object|string|null $job, + bool $afterCommitHandled = false, + ): mixed { $contextKey = self::FAILING_QUEUES_CONTEXT_PREFIX . spl_object_id($this); $state = CoroutineContext::get($contextKey); $failingQueues = $state !== null && $state[0]->get() === $this ? $state[1] : []; [$lastException, $failedQueues] = [null, []]; + $forwarded = CoroutineContext::get(self::OUTBOX_FORWARDED_CONTEXT_KEY); + + if ($job !== null && ($forwarded['job'] ?? null) !== $job) { + $forwarded = ['job' => $job, 'afterCommitHandled' => $afterCommitHandled]; + } try { foreach ($this->connections as $connection) { try { - return $this->manager->connection($connection)->{$method}(...$arguments); + return $job === null + ? $this->manager->connection($connection)->{$method}(...$arguments) + : $this->withOutboxForwarding( + $forwarded, + fn () => $this->manager->connection($connection)->{$method}(...$arguments), + ); } catch (CanceledException $exception) { throw $exception; } catch (Throwable $e) { @@ -303,6 +341,10 @@ protected function deferUntilAllTransactionsCommit( return false; } + if (($selectedConnection = $this->resolveOutboxCommitConnection($transactions, $job)) !== null) { + $connections = [$selectedConnection]; + } + $context = ContextRepository::hasInstance() ? ContextRepository::getInstance()->replicate() : null; $pendingConnections = array_fill_keys($connections, true); $settled = false; @@ -325,7 +367,7 @@ static function () use (&$settled, $releaseLocks): void { foreach ($connections as $connection) { $transactions->addCallback( - function () use (&$pendingConnections, &$settled, $connection, $method, $arguments, $job, $context): void { + function () use (&$pendingConnections, &$settled, $connection, $method, $arguments, $job, $context, $selectedConnection): void { if ($settled) { return; } @@ -344,7 +386,9 @@ function () use (&$pendingConnections, &$settled, $connection, $method, $argumen // Child payload hooks run after commit but must see the dispatch-time log context. try { - $this->withLogContext($context, fn () => $this->attemptOnAllConnections($method, $arguments, $job)); + $this->withLogContext($context, fn () => $selectedConnection !== null + ? $this->attemptOnConnections($method, $arguments, $job, true) + : $this->attemptOnAllConnections($method, $arguments, $job)); } finally { if (is_object($job)) { DispatchLockContext::release($job); diff --git a/src/queue/src/NullQueue.php b/src/queue/src/NullQueue.php index b015c1eeba..b86173b561 100644 --- a/src/queue/src/NullQueue.php +++ b/src/queue/src/NullQueue.php @@ -157,6 +157,14 @@ public function later(DateInterval|DateTimeInterface|int $delay, object|string $ return null; } + /** + * Keep inherited bulk dispatch a no-op even when jobs select the outbox. + */ + protected function shouldStageViaOutbox(object|string $job): bool + { + return false; + } + /** * Pop the next job off of the queue. */ diff --git a/src/queue/src/Outbox/Concerns/PublishesOutboxMessagesSequentially.php b/src/queue/src/Outbox/Concerns/PublishesOutboxMessagesSequentially.php new file mode 100644 index 0000000000..be8fa02f75 --- /dev/null +++ b/src/queue/src/Outbox/Concerns/PublishesOutboxMessagesSequentially.php @@ -0,0 +1,51 @@ + $messages + */ + protected function publishOutboxMessagesSequentially(Queue $queue, array $messages): OutboxBatchResult + { + $due = []; + $deferred = []; + $accepted = []; + $now = $this->currentTime(); + + foreach ($messages as $message) { + if ($message->deliverAt !== null && $message->deliverAt > $now) { + $deferred[] = $message->id; + } else { + $due[] = $message; + } + } + + try { + foreach ($due as $message) { + $queue->pushRaw($message->payload, $message->queue, $message->options); + $accepted[] = $message->id; + } + } catch (CanceledException $exception) { + throw $exception; + } catch (Throwable $exception) { + return new OutboxBatchResult($accepted, $deferred, $exception); + } + + return new OutboxBatchResult($accepted, $deferred); + } +} diff --git a/src/queue/src/Outbox/DatabaseOutboxRepository.php b/src/queue/src/Outbox/DatabaseOutboxRepository.php new file mode 100644 index 0000000000..a9e2c375ba --- /dev/null +++ b/src/queue/src/Outbox/DatabaseOutboxRepository.php @@ -0,0 +1,403 @@ + $databases + */ + public function __construct( + protected ConnectionResolverInterface $resolver, + protected array $databases, + protected string $table, + protected int $retryAfter, + ) { + if ($databases === []) { + throw new InvalidArgumentException('At least one outbox database must be configured.'); + } + + if ($retryAfter <= 0) { + throw new InvalidArgumentException('The outbox retry_after must be greater than zero.'); + } + } + + /** + * Get the configured outbox sources. + * + * @return list + */ + public function getDatabases(): array + { + return $this->databases; + } + + /** + * Resolve a configured outbox source in the current coroutine. + */ + public function getConnection(string $database): ConnectionInterface + { + if (! in_array($database, $this->databases, true)) { + throw new InvalidArgumentException("Outbox database [{$database}] is not configured in queue.outbox.databases."); + } + + return $this->resolver->connection($database); + } + + /** + * Insert finalized rows on the connection that owns the business transaction. + * + * @param non-empty-list> $rows + */ + public function insert(ConnectionInterface $connection, array $rows): void + { + $this->insertDatabaseRows($connection, $this->table, $rows); + } + + /** + * Claim eligible messages without retaining a database resource in the result. + * + * @return list + */ + public function claim(string $database, int $limit): array + { + $connection = $this->getConnection($database); + $now = $this->currentTime(); + $token = (string) Str::uuid(); + $expiresAt = $now + $this->retryAfter; + $lock = $connection instanceof PdoConnection ? $connection->lockForPopping() : true; + $limit = min($limit, $this->maxBindings($connection) - 3); + + if ($lock === 'FOR UPDATE SKIP LOCKED' && $connection->getDriverName() === 'pgsql') { + $rows = $this->claimOnPostgres($connection, $limit, $now, $expiresAt, $token); + } elseif ($lock === 'FOR UPDATE SKIP LOCKED') { + $rows = $this->claimWithSkipLocked($connection, $limit, $now, $expiresAt, $token); + } else { + $rows = $this->claimWithConditionalUpdate($connection, $limit, $now, $expiresAt, $token); + } + + usort($rows, static fn (object $first, object $second): int => [(int) $first->eligible_at, (int) $first->id] <=> [(int) $second->eligible_at, (int) $second->id]); + + return array_map(static fn (object $row): OutboxMessage => new OutboxMessage( + (int) $row->id, + $row->uuid, + $row->lease_token, + (int) $row->attempts, + $row->connection, + $row->queue, + $row->payload, + $row->options === null ? [] : json_decode($row->options, true, flags: JSON_THROW_ON_ERROR), + $row->deliver_at === null ? null : (int) $row->deliver_at, + (int) $row->eligible_at, + (int) $row->created_at, + ), $rows); + } + + /** + * Claim a PostgreSQL batch in one statement, retaining its original eligibility. + * + * @return list + */ + protected function claimOnPostgres(ConnectionInterface $connection, int $limit, int $now, int $expiresAt, string $token): array + { + $table = $connection->getQueryGrammar()->wrapTable($this->table); + + return $connection->selectFromWriteConnection( + "WITH candidates AS ( + SELECT id, available_at FROM {$table} + WHERE available_at <= ? ORDER BY available_at, id + LIMIT ? FOR UPDATE SKIP LOCKED + ) + UPDATE {$table} AS outgoing + SET lease_token = ?, available_at = ?, attempts = outgoing.attempts + 1 + FROM candidates WHERE outgoing.id = candidates.id + RETURNING outgoing.*, candidates.available_at AS eligible_at", + [$now, $limit, $token, $expiresAt], + ); + } + + /** + * Claim rows on engines that support skipping another relay's locked rows. + * + * @return list + */ + protected function claimWithSkipLocked(ConnectionInterface $connection, int $limit, int $now, int $expiresAt, string $token): array + { + return $connection->transaction(function () use ($connection, $limit, $now, $expiresAt, $token): array { + $rows = $this->query($connection) + ->where('available_at', '<=', $now) + ->orderBy('available_at')->orderBy('id') + ->limit($limit)->lock('FOR UPDATE SKIP LOCKED') + ->get()->all(); + + if ($rows === []) { + return []; + } + + $this->query($connection)->whereIn('id', array_column($rows, 'id'))->increment('attempts', 1, [ + 'lease_token' => $token, + 'available_at' => $expiresAt, + ]); + + foreach ($rows as $row) { + $row->eligible_at = $row->available_at; + $row->available_at = $expiresAt; + $row->lease_token = $token; + ++$row->attempts; + } + + return $rows; + }); + } + + /** + * Win eligible candidates through one atomic update on engines without skip-locked support. + * + * @return list + */ + protected function claimWithConditionalUpdate(ConnectionInterface $connection, int $limit, int $now, int $expiresAt, string $token): array + { + $candidates = $this->query($connection) + ->where('available_at', '<=', $now) + ->orderBy('available_at')->orderBy('id')->limit($limit) + ->pluck('available_at', 'id')->all(); + + if ($candidates === []) { + return []; + } + + $ids = array_keys($candidates); + $this->query($connection) + ->whereIn('id', $ids)->where('available_at', '<=', $now) + ->increment('attempts', 1, ['lease_token' => $token, 'available_at' => $expiresAt]); + + $rows = $this->query($connection)->whereIn('id', $ids)->where('lease_token', $token)->get()->all(); + + foreach ($rows as $row) { + $row->eligible_at = $candidates[$row->id]; + } + + return $rows; + } + + /** + * Settle a validated result without modifying rows claimed by a newer relay. + * + * @param non-empty-list $messages + * @return int the number of rows still owned and settled + */ + public function settle(string $database, array $messages, OutboxBatchResult $result): int + { + $connection = $this->getConnection($database); + $token = $messages[0]->leaseToken; + $failed = array_values(array_diff(array_column($messages, 'id'), $result->accepted, $result->deferred)); + + return $connection->transaction(function () use ($connection, $token, $result, $failed): int { + $settled = 0; + $maxBindings = $this->maxBindings($connection); + + foreach (array_chunk($result->accepted, $maxBindings - 1) as $ids) { + $settled += $this->query($connection)->where('lease_token', $token)->whereIn('id', $ids)->delete(); + } + + foreach (array_chunk($result->deferred, $maxBindings - 2) as $ids) { + $grammar = $connection->getQueryGrammar(); + $settled += $this->query($connection)->where('lease_token', $token)->whereIn('id', $ids)->update([ + 'lease_token' => null, + 'available_at' => new Expression($grammar->wrap('deliver_at')), + 'attempts' => new Expression($grammar->wrap('attempts') . ' - 1'), + ]); + } + + if ($failed !== []) { + $settled += $this->releaseFailed($connection, $token, $failed, $result->failure->getMessage()); + } + + return $settled; + }); + } + + /** + * Back off failed rows with seven retry deadlines rather than one case per row. + * + * @param non-empty-list $ids + */ + protected function releaseFailed(ConnectionInterface $connection, string $token, array $ids, string $error): int + { + $now = $this->currentTime(); + $deadlines = array_map(static fn (int $delay): int => $now + random_int(max(1, (int) ceil($delay / 2)), $delay), self::RETRY_DELAYS); + $error = mb_substr($error, 0, 4096); + $grammar = $connection->getQueryGrammar(); + $table = $grammar->wrapTable($this->table); + $attempts = $grammar->wrap('attempts'); + $cases = []; + + foreach (array_keys($deadlines) as $index) { + $cases[] = $index === count($deadlines) - 1 ? 'ELSE ?' : 'WHEN ' . $attempts . ' <= ' . ($index + 1) . ' THEN ?'; + } + + $case = 'CASE ' . implode(' ', $cases) . ' END'; + $affected = 0; + + foreach (array_chunk($ids, $this->maxBindings($connection) - count($deadlines) - 2) as $chunk) { + $parameters = implode(', ', array_fill(0, count($chunk), '?')); + $affected += $connection->update( + "UPDATE {$table} SET available_at = {$case}, lease_token = NULL, last_error = ? WHERE lease_token = ? AND id IN ({$parameters})", + [...$deadlines, $error, $token, ...$chunk], + ); + } + + return $affected; + } + + /** + * Get a bounded set of pending rows for operator inspection. + * + * @return list + */ + public function pending(string $database, int $limit): array + { + return $this->query($this->getConnection($database))->orderBy('id')->limit($limit)->get()->all(); + } + + /** + * Find a stored message by its job UUID on the writer. + */ + public function find(string $database, string $uuid): ?object + { + return $this->query($this->getConnection($database))->where('uuid', $uuid)->first(); + } + + /** + * Read backlog counts and the age of the first currently eligible row. + * + * @return array{total: int, eligible: int, delayed: int, leased: int, retrying: int, oldest_eligible_age: ?int} + */ + public function statistics(string $database): array + { + $connection = $this->getConnection($database); + $now = $this->currentTime(); + $counts = $this->query($connection)->selectRaw( + 'COUNT(*) AS total, + SUM(CASE WHEN available_at <= ? THEN 1 ELSE 0 END) AS eligible, + SUM(CASE WHEN lease_token IS NULL AND available_at > ? AND attempts = 0 THEN 1 ELSE 0 END) AS delayed, + SUM(CASE WHEN lease_token IS NOT NULL AND available_at > ? THEN 1 ELSE 0 END) AS leased, + SUM(CASE WHEN lease_token IS NULL AND available_at > ? AND attempts > 0 THEN 1 ELSE 0 END) AS retrying, + MIN(CASE WHEN available_at <= ? THEN available_at END) AS oldest_eligible', + [$now, $now, $now, $now, $now], + )->first(); + + return [ + 'total' => (int) $counts->total, + 'eligible' => (int) $counts->eligible, + 'delayed' => (int) $counts->delayed, + 'leased' => (int) $counts->leased, + 'retrying' => (int) $counts->retrying, + 'oldest_eligible_age' => $counts->oldest_eligible === null ? null : $now - (int) $counts->oldest_eligible, + ]; + } + + /** + * Retry an attempted message unless a live relay owns it. + */ + public function retry(string $database, string $uuid): int + { + $now = $this->currentTime(); + + return $this->retryable($this->getConnection($database), $now)->where('uuid', $uuid)->update([ + 'available_at' => $now, + 'lease_token' => null, + 'last_error' => null, + ]); + } + + /** + * Retry the current backlog in bounded primary-key chunks, preserving attempt counts. + */ + public function retryAll(string $database): int + { + $connection = $this->getConnection($database); + $now = $this->currentTime(); + $ceiling = $this->retryable($connection, $now)->max('id'); + + if ($ceiling === null) { + return 0; + } + + $lastId = 0; + $affected = 0; + $limit = min(1000, $this->maxBindings($connection) - 5); + + do { + $ids = $this->retryable($connection, $now)->where('id', '>', $lastId)->where('id', '<=', $ceiling) + ->orderBy('id')->limit($limit)->pluck('id')->all(); + + if ($ids === []) { + break; + } + + $affected += $this->retryable($connection, $now)->whereIn('id', $ids)->update([ + 'available_at' => $now, + 'lease_token' => null, + 'last_error' => null, + ]); + $lastId = (int) end($ids); + } while (count($ids) === $limit); + + return $affected; + } + + /** + * Delete an inspected row only while its lease remains inactive and unchanged. + */ + public function forget(string $database, int $id, ?string $leaseToken): bool + { + return $this->withoutLiveLease($this->query($this->getConnection($database)), $this->currentTime()) + ->where('id', $id)->where('lease_token', $leaseToken)->delete() === 1; + } + + /** + * Select attempted rows that are safe for an operator to reschedule. + */ + protected function retryable(ConnectionInterface $connection, int $now): Builder + { + return $this->withoutLiveLease($this->query($connection), $now)->where('attempts', '>', 0); + } + + /** + * Exclude rows currently owned by a publishing relay. + */ + protected function withoutLiveLease(Builder $query, int $now): Builder + { + return $query->where(static function (Builder $query) use ($now): void { + $query->whereNull('lease_token')->orWhere('available_at', '<=', $now); + }); + } + + /** + * Query the configured table on the writer, including for operator mutations. + */ + protected function query(ConnectionInterface $connection): Builder + { + return $connection->table($this->table)->useWritePdo(); + } +} diff --git a/src/queue/src/Outbox/OutboxBatchResult.php b/src/queue/src/Outbox/OutboxBatchResult.php new file mode 100644 index 0000000000..50ec31dc44 --- /dev/null +++ b/src/queue/src/Outbox/OutboxBatchResult.php @@ -0,0 +1,23 @@ + $accepted + * @param array $deferred + */ + public function __construct( + public array $accepted = [], + public array $deferred = [], + public ?Throwable $failure = null, + ) { + } +} diff --git a/src/queue/src/Outbox/OutboxDispatcher.php b/src/queue/src/Outbox/OutboxDispatcher.php new file mode 100644 index 0000000000..5f0ef6cc5c --- /dev/null +++ b/src/queue/src/Outbox/OutboxDispatcher.php @@ -0,0 +1,87 @@ +transactions->callbackApplicableTransactions() as $transaction) { + $active[$transaction->connection] = true; + } + + $explicit = $job->outboxDatabase ?? null; + + if ($explicit !== null) { + $connection = $this->resolver->connection($explicit); + + if ($active !== [] && ! isset($active[$connection->getNameWithReadWriteType() ?? ''])) { + throw new LogicException("Outbox database [{$explicit}] does not own an active transaction. Active connections: " . implode(', ', array_keys($active)) . '.'); + } + } elseif (count($active) > 1) { + throw new LogicException('Multiple database transactions are active [' . implode(', ', array_keys($active)) . ']. Select the outbox owner with onOutboxDatabase().'); + } else { + $connection = $this->resolver->connection(array_key_first($active) ?? $this->repository->getDatabases()[0]); + } + + $owner = $connection->getNameWithReadWriteType() ?? ''; + + if (ConnectionName::parse($owner)->isRead()) { + throw new LogicException("Outbox database [{$owner}] is a read connection. Select its base or ::write connection."); + } + + if (! in_array($connection->getName(), $this->repository->getDatabases(), true)) { + throw new LogicException("Outbox database [{$owner}] is not covered by queue.outbox.databases."); + } + + return $connection; + } + + /** + * Store one owner group and register rollback cleanup before dispatch acceptance. + * + * @param non-empty-list> $rows + * @param non-empty-list $jobs + */ + public function store(ConnectionInterface $connection, array $rows, array $jobs): void + { + $this->repository->insert($connection, $rows); + + if ($connection->transactionLevel() === 0) { + return; + } + + foreach ($jobs as $job) { + if (is_object($job) && ($snapshot = DispatchLockContext::snapshot($job)) !== null) { + $this->transactions->addCallbackForRollback( + static fn () => DispatchLockContext::releaseSnapshot($snapshot), + $connection->getNameWithReadWriteType() ?? '', + ); + } + } + } +} diff --git a/src/queue/src/Outbox/OutboxMessage.php b/src/queue/src/Outbox/OutboxMessage.php new file mode 100644 index 0000000000..e760ac87dc --- /dev/null +++ b/src/queue/src/Outbox/OutboxMessage.php @@ -0,0 +1,26 @@ + + */ + public function claim(string $database, int $limit): array + { + return $this->inFreshCoroutine(fn (): array => $this->repository->claim($database, $limit)); + } + + /** + * Publish and settle a claimed batch, returning whether it completed without failure. + * + * @param list $messages + */ + public function deliver(string $database, array $messages): bool + { + $groups = []; + + foreach ($messages as $message) { + $groups[$message->connection][] = $message; + } + + $outcomes = []; + $successful = true; + + foreach ($groups as $group) { + $connection = $group[0]->connection; + $startedAt = hrtime(true); + $publishedAt = $this->currentTime(); + $result = $this->inFreshCoroutine(function () use ($connection, $group): OutboxBatchResult { + try { + $queue = $this->queues->connection($connection); + + return $queue instanceof PublishesOutboxMessages + ? $queue->publishOutboxMessages($group) + : $this->publishOutboxMessagesSequentially($queue, $group); + } catch (CanceledException|ChildCancellationException $exception) { + throw $exception; + } catch (Throwable $exception) { + return new OutboxBatchResult(failure: $exception); + } + }); + $duration = (hrtime(true) - $startedAt) / 1_000_000_000; + $result = $this->validateResult($group, $result); + $settlementFailure = $this->inFreshCoroutine(function () use ($database, $group, $result): ?Throwable { + try { + if ($this->repository->settle($database, $group, $result) !== count($group)) { + return new RuntimeException('Outbox lease ownership changed before the batch could be fully settled.'); + } + + return null; + } catch (CanceledException|ChildCancellationException $exception) { + throw $exception; + } catch (Throwable $exception) { + return $exception; + } + }); + + $successful = $successful && $result->failure === null && $settlementFailure === null; + $outcomes[] = [$connection, $group, $result, $settlementFailure, $duration, $publishedAt]; + } + + // Observers cannot prevent another connection's committed work from being delivered. + foreach ($outcomes as [$connection, $group, $result, $settlementFailure, $duration, $publishedAt]) { + $this->reportOutcome($database, $connection, $group, $result, $settlementFailure, $duration, $publishedAt); + } + + return $successful; + } + + /** + * Reject driver results that cannot safely identify the rows to settle. + * + * @param non-empty-list $messages + */ + protected function validateResult(array $messages, OutboxBatchResult $result): OutboxBatchResult + { + $input = []; + + foreach ($messages as $message) { + $input[$message->id] = $message; + } + + $seen = []; + $valid = true; + + foreach ([$result->accepted, $result->deferred] as $partition => $ids) { + foreach ($ids as $id) { + if (! is_int($id) || ! isset($input[$id]) || isset($seen[$id]) + || ($partition === 1 && $input[$id]->deliverAt === null)) { + $valid = false; + break 2; + } + + $seen[$id] = true; + } + } + + if (! $valid || ($result->failure === null && count($seen) !== count($messages))) { + return new OutboxBatchResult(failure: new LogicException( + 'Outbox publication must return disjoint input row IDs, defer only messages with a delivery time, and account for every row unless publication failed.', + previous: $result->failure, + )); + } + + return $result; + } + + /** + * Notify listeners and report failures after publication and settlement have released their resources. + * + * @param non-empty-list $messages + */ + protected function reportOutcome( + string $database, + string $connection, + array $messages, + OutboxBatchResult $result, + ?Throwable $settlementFailure, + float $duration, + int $publishedAt, + ): void { + $failure = $result->failure ?? $settlementFailure; + $attempted = count($messages) - count($result->deferred); + $eventClass = $failure === null ? OutboxBatchDelivered::class : OutboxBatchDeliveryFailed::class; + $notify = $attempted > 0 && $this->events->hasListeners($eventClass); + + if (! $notify && $failure === null) { + return; + } + + $this->inFreshCoroutine(function () use ($database, $connection, $messages, $result, $settlementFailure, $duration, $publishedAt, $failure, $attempted, $notify): void { + $listenerFailure = null; + + if ($notify) { + $eligibleAt = $publishedAt; + $deferred = array_fill_keys($result->deferred, true); + + foreach ($messages as $message) { + if (! isset($deferred[$message->id])) { + $eligibleAt = min($eligibleAt, $message->eligibleAt); + } + } + + $event = $failure === null + ? new OutboxBatchDelivered($database, $connection, count($result->accepted), $duration, $publishedAt - $eligibleAt) + : new OutboxBatchDeliveryFailed($database, $connection, count($result->accepted), $attempted - count($result->accepted), $duration, $publishedAt - $eligibleAt, $failure); + + try { + $this->events->dispatch($event); + } catch (CanceledException|ChildCancellationException $exception) { + throw $exception; + } catch (Throwable $exception) { + $listenerFailure = $exception; + } + } + + if ($failure !== null || $listenerFailure !== null) { + $context = [ + 'outbox_database' => $database, + 'queue_connection' => $connection, + 'outbox_ids' => array_column($messages, 'id'), + 'job_uuids' => array_column($messages, 'uuid'), + 'jobs' => array_map(static function (OutboxMessage $message): mixed { + $payload = json_decode($message->payload, true); + + return $payload['displayName'] ?? $payload['job'] ?? null; + }, $messages), + ]; + + foreach ([$result->failure, $settlementFailure, $listenerFailure] as $exception) { + if ($exception !== null) { + $this->exceptions->report($exception, $context); + } + } + } + }); + } + + /** + * Finish an operation's resource cleanup before returning or propagating cancellation. + * + * @template TReturn + * @param Closure(): TReturn $callback + * @return TReturn + */ + protected function inFreshCoroutine(Closure $callback): mixed + { + return (new Waiter(-1))->wait($callback, waitForChildTermination: true); + } +} diff --git a/src/queue/src/Outbox/OutboxWorker.php b/src/queue/src/Outbox/OutboxWorker.php new file mode 100644 index 0000000000..57e8732875 --- /dev/null +++ b/src/queue/src/Outbox/OutboxWorker.php @@ -0,0 +1,218 @@ +repository->getDatabases(); + + if ($database !== null) { + if (! in_array($database, $databases, true)) { + throw new InvalidArgumentException("Outbox database [{$database}] is not configured in queue.outbox.databases."); + } + + $databases = [$database]; + } + + $quit = false; + $paused = false; + $signalHandler = static function (int $signal) use (&$quit, &$paused): void { + match ($signal) { + SIGUSR2 => $paused = true, + SIGCONT => $paused = false, + default => $quit = true, + }; + }; + $this->signals->register($signalHandler, [SIGTERM, SIGINT, SIGQUIT, SIGUSR2, SIGCONT], $signalHandler); + $concurrent = new WaitConcurrent($options->concurrency); + $taskFailure = null; + $failure = null; + $status = Worker::EXIT_SUCCESS; + + try { + $startedAt = microtime(true); + $lastRestart = $options->once ? null : $this->restartTimestamp(); + $admitted = 0; + $visited = 0; + $emptyVisits = 0; + $retryAt = []; + $sourceCount = count($databases); + + while (! $quit && $taskFailure === null) { + if (memory_get_usage(true) / 1024 / 1024 >= $options->memory) { + $status = Worker::EXIT_MEMORY_LIMIT; + break; + } + + if (($options->maxJobs > 0 && $admitted >= $options->maxJobs) + || ($options->maxTime > 0 && microtime(true) - $startedAt >= $options->maxTime) + || ($options->once && $visited >= $sourceCount) + || (! $options->once && $lastRestart !== $this->restartTimestamp())) { + break; + } + + if ($paused) { + Sleep::usleep((int) ($options->sleep * 1_000_000)); + continue; + } + + if ($concurrent->isFull()) { + $concurrent->waitForAvailableSlot($options->sleep); + continue; + } + + $source = $databases[$visited++ % $sourceCount]; + $limit = $options->maxJobs > 0 ? min($options->batch, $options->maxJobs - $admitted) : $options->batch; + $messages = []; + + if (! isset($retryAt[$source]) || microtime(true) >= $retryAt[$source]) { + $messages = $this->claim($source, $limit); + + if ($messages === null) { + $retryAt[$source] = microtime(true) + $options->sleep; + } else { + unset($retryAt[$source]); + } + } + + if ($messages) { + $emptyVisits = 0; + $admitted += count($messages); + + if ($options->once) { + $this->relay->deliver($source, $messages); + } else { + $concurrent->create(function () use ($source, $messages, $options, &$taskFailure): void { + try { + if (! $this->relay->deliver($source, $messages)) { + Sleep::usleep((int) ($options->sleep * 1_000_000)); + } + } catch (CanceledException $exception) { + throw $exception; + } catch (Throwable $exception) { + $taskFailure ??= $exception; + } + }); + } + } elseif (++$emptyVisits >= $sourceCount) { + if ($options->stopWhenEmpty) { + break; + } + + if (! $options->once) { + Sleep::usleep((int) ($options->sleep * 1_000_000)); + } + + $emptyVisits = 0; + } + } + } catch (Throwable $exception) { + $failure = $exception; + + if ($exception instanceof CanceledException) { + $concurrent->cancel(); + } + } finally { + try { + $this->join($concurrent); + } catch (CanceledException $exception) { + if (! $failure instanceof CanceledException) { + $failure = $exception; + } + } finally { + $this->signals->unregister($signalHandler); + } + } + + if ($failure !== null || $taskFailure !== null) { + throw $failure ?? $taskFailure; + } + + return $status; + } + + /** + * Report a failed claim after its database-owning coroutine has ended. + * + * @return null|list null indicates a source failure + */ + protected function claim(string $database, int $limit): ?array + { + try { + return $this->relay->claim($database, $limit); + } catch (CanceledException|ChildCancellationException $exception) { + throw $exception; + } catch (Throwable $exception) { + (new Waiter(-1))->wait( + fn () => $this->exceptions->report($exception, ['outbox_database' => $database]), + waitForChildTermination: true, + ); + + return null; + } + } + + /** + * Read the restart signal without retaining a cache connection on the coordinator. + */ + protected function restartTimestamp(): ?int + { + return (new Waiter(-1))->wait(function (): ?int { + $timestamp = $this->cache->store()->get(Worker::RESTART_SIGNAL_CACHE_KEY); + + return $timestamp === null ? null : (int) $timestamp; + }, waitForChildTermination: true); + } + + /** + * Join every owned task before propagating cancellation, including repeated cancellation during cleanup. + */ + protected function join(WaitConcurrent $concurrent): void + { + $cancellation = null; + + do { + try { + $finished = $concurrent->wait(); + } catch (CanceledException $exception) { + $cancellation ??= $exception; + $finished = false; + } + } while (! $finished); + + if ($cancellation !== null) { + throw $cancellation; + } + } +} diff --git a/src/queue/src/Outbox/OutboxWorkerOptions.php b/src/queue/src/Outbox/OutboxWorkerOptions.php new file mode 100644 index 0000000000..45692bbac5 --- /dev/null +++ b/src/queue/src/Outbox/OutboxWorkerOptions.php @@ -0,0 +1,36 @@ + 1000) { + throw new InvalidArgumentException('The outbox batch size must be between 1 and 1000.'); + } + + if ($sleep <= 0 || $concurrency < 1) { + throw new InvalidArgumentException('Outbox sleep and concurrency must be greater than zero.'); + } + + if ($maxJobs < 0 || $maxTime < 0 || $memory <= 0) { + throw new InvalidArgumentException('Outbox job and time limits must be non-negative, and memory must be greater than zero.'); + } + } +} diff --git a/src/queue/src/Queue.php b/src/queue/src/Queue.php index 1ba59d3adb..54d940011a 100644 --- a/src/queue/src/Queue.php +++ b/src/queue/src/Queue.php @@ -25,6 +25,7 @@ use Hypervel\Queue\Attributes\ReadsQueueAttributes; use Hypervel\Queue\Attributes\Timeout; use Hypervel\Queue\Attributes\Tries; +use Hypervel\Queue\Concerns\StagesOutboxJobs; use Hypervel\Queue\Events\JobPayloadFinalizing; use Hypervel\Queue\Events\JobQueued; use Hypervel\Queue\Events\JobQueueing; @@ -46,6 +47,7 @@ abstract class Queue { + use StagesOutboxJobs; use InteractsWithTime; use ReadsQueueAttributes; use ResolvesQueueRoutes; @@ -106,6 +108,7 @@ public function laterOn(UnitEnum|string|null $queue, DateInterval|DateTimeInterf public function bulk(array $jobs, mixed $data = '', UnitEnum|string|null $queue = null): mixed { $queue = $this->normalizeQueue($queue); + $jobs = $this->stageOutboxJobs($jobs, $data, $queue); foreach ((array) $jobs as $job) { $delay = $this->getJobDelay($job); @@ -120,6 +123,27 @@ public function bulk(array $jobs, mixed $data = '', UnitEnum|string|null $queue return null; } + /** + * Prepare payloads and delays for a set-based enqueue operation. + * + * @return array + */ + protected function prepareBatchJobs(array $jobs, mixed $data, ?string $queue): array + { + $prepared = []; + + foreach ($jobs as $job) { + $delay = $this->getJobDelay($job); + $prepared[] = [ + 'job' => $job, + 'delay' => $delay, + 'payload' => $this->createPayload($job, $queue, $data, $delay), + ]; + } + + return $prepared; + } + /** * Get the delay configured on the given job. */ @@ -413,6 +437,20 @@ public function setAfterCommitDispatcher(?Closure $dispatcher): static */ protected function enqueueUsing(object|string $job, string $payload, ?string $queue, DateInterval|DateTimeInterface|int|null $delay, callable $callback): mixed { + try { + $stageViaOutbox = $this->shouldStageViaOutbox($job); + } catch (CanceledException $exception) { + throw $exception; + } catch (Throwable $exception) { + $this->raiseJobQueueingFailedEvent($queue, $job, $payload, $delay, $exception); + + throw $exception; + } + + if ($stageViaOutbox) { + return $this->enqueueViaOutbox($job, $payload, $queue, $delay); + } + if ($this->shouldDispatchAfterCommit($job) && $this->container->has('db.transactions') ) { @@ -456,9 +494,17 @@ protected function deferBatchEnqueueAfterCommit( DatabaseTransactionsManager $transactions, array $jobs, Closure $enqueue, + ?string $connection = null, ): void { + // Inline work still belongs to its caller, including a failover's next attempt. + if (! $transactions->hasApplicableTransaction($connection)) { + $enqueue($this); + + return; + } + foreach ($jobs as $job) { - $this->addJobRollbackCallback($transactions, $job); + $this->addJobRollbackCallback($transactions, $job, $connection); } $callback = static function (Queue $owner) use ($jobs, $enqueue): mixed { @@ -483,11 +529,13 @@ protected function deferBatchEnqueueAfterCommit( $dispatcher = $this->afterCommitDispatcher; $transactions->addCallback( - static fn () => $dispatcher($callback) + static fn () => $dispatcher($callback), + $connection, ); } else { $transactions->addCallback( - fn () => $callback($this) + fn () => $callback($this), + $connection, ); } @@ -506,7 +554,8 @@ protected function enqueueNow(object|string $job, string $payload, ?string $queu try { $payload = $this->finalizePayloadForQueueing($queue, $job, $payload, $delay); $this->raiseJobQueueingEvent($queue, $job, $payload, $delay); - $jobId = $callback($this, $payload, $queue, $delay); + $storedPayload = $this->preparePayloadForPush($job, $payload, $queue); + $jobId = $callback($this, $storedPayload, $queue, $delay); } catch (CanceledException $exception) { throw $exception; } catch (Throwable $exception) { @@ -516,11 +565,72 @@ protected function enqueueNow(object|string $job, string $payload, ?string $queu } $this->acceptDispatchLocks($job); - $this->raiseJobQueuedEvent($queue, $jobId, $job, $payload, $delay); + $exception = null; + + try { + $this->handlePayloadPushed($storedPayload, $queue); + } catch (CanceledException $cancellation) { + throw $cancellation; + } catch (Throwable $throwable) { + $exception = $throwable; + } + + try { + $this->raiseJobQueuedEvent($queue, $jobId, $job, $payload, $delay); + } catch (CanceledException $cancellation) { + throw $cancellation; + } catch (Throwable $throwable) { + $exception ??= $throwable; + } + + if ($exception !== null) { + throw $exception; + } return $jobId; } + /** + * Prepare a finalized payload for storage without changing lifecycle event correlation. + */ + protected function preparePayloadForPush(object|string $job, string $payload, ?string $queue): string + { + return $payload; + } + + /** + * Handle a payload after the queue has accepted it. + */ + protected function handlePayloadPushed(string $payload, ?string $queue): void + { + } + + /** + * Notify every batch member and return the first ordinary notification failure. + * + * @template TMember + * @param array $members + * @param Closure(TMember): void ...$notifications + */ + protected function notifyBatchMembers(array $members, Closure ...$notifications): ?Throwable + { + $exception = null; + + foreach ($members as $member) { + foreach ($notifications as $notification) { + try { + $notification($member); + } catch (CanceledException $cancellation) { + throw $cancellation; + } catch (Throwable $throwable) { + $exception ??= $throwable; + } + } + } + + return $exception; + } + /** * Finalize an encoded payload immediately before queueing. * @@ -583,25 +693,31 @@ protected function partitionJobsByAfterCommit( protected function shouldDispatchAfterCommit(object|string $job): bool { if ($job instanceof ShouldQueueAfterCommit) { - return ! (isset($job->afterCommit) && $job->afterCommit === false); + $afterCommit = ! (isset($job->afterCommit) && $job->afterCommit === false); + } elseif (! $job instanceof Closure && is_object($job) && isset($job->afterCommit)) { + $afterCommit = $job->afterCommit; + } else { + $afterCommit = $this->dispatchAfterCommit; } - if (! $job instanceof Closure && is_object($job) && isset($job->afterCommit)) { - return $job->afterCommit; + if (! $afterCommit) { + return false; } - return $this->dispatchAfterCommit; + $forwarded = CoroutineContext::get(self::OUTBOX_FORWARDED_CONTEXT_KEY); + + return ($forwarded['job'] ?? null) !== $job || ! $forwarded['afterCommitHandled']; } /** * Register a transaction rollback callback that releases the job's locks. */ - protected function addJobRollbackCallback(DatabaseTransactionsManager $transactions, object|string $job): void + protected function addJobRollbackCallback(DatabaseTransactionsManager $transactions, object|string $job, ?string $connection = null): void { $callback = $this->createJobRollbackCallback($job); if ($callback !== null) { - $transactions->addCallbackForRollback($callback); + $transactions->addCallbackForRollback($callback, $connection); } } @@ -758,6 +874,7 @@ public function getConfig(): array public function setConfig(array $config): static { $this->config = $config; + $this->dispatchViaOutbox = $config['outbox'] ?? false; return $this; } diff --git a/src/queue/src/QueuePoolProxy.php b/src/queue/src/QueuePoolProxy.php index 8f0bd3e4ef..ad94e2ca3d 100644 --- a/src/queue/src/QueuePoolProxy.php +++ b/src/queue/src/QueuePoolProxy.php @@ -10,6 +10,7 @@ use Hypervel\Contracts\ObjectPool\Factory; use Hypervel\Contracts\Queue\IndexAwareQueue; use Hypervel\Contracts\Queue\Job; +use Hypervel\Contracts\Queue\PublishesOutboxMessages; use Hypervel\Contracts\Queue\Queue as QueueContract; use Hypervel\Database\ConnectionResolver; use Hypervel\ObjectPool\Lease; @@ -17,14 +18,19 @@ use Hypervel\ObjectPool\PoolErrorReporter; use Hypervel\ObjectPool\PoolProxy; use Hypervel\Queue\Jobs\Job as PoolLeaseAwareJob; +use Hypervel\Queue\Outbox\Concerns\PublishesOutboxMessagesSequentially; +use Hypervel\Queue\Outbox\OutboxBatchResult; +use Hypervel\Queue\Outbox\OutboxMessage; use Hypervel\Support\Collection; use RuntimeException; use Swoole\Coroutine\CanceledException; use Throwable; use UnitEnum; -class QueuePoolProxy extends PoolProxy implements QueueContract, IndexAwareQueue +class QueuePoolProxy extends PoolProxy implements QueueContract, IndexAwareQueue, PublishesOutboxMessages { + use PublishesOutboxMessagesSequentially; + /** * The logical connection name applied to each borrowed queue. */ @@ -223,6 +229,18 @@ public function bulk(array $jobs, mixed $data = '', UnitEnum|string|null $queue return $this->invoke(__FUNCTION__, func_get_args()); } + /** + * Publish a stored batch while borrowing its transport only once. + * + * @param list $messages + */ + public function publishOutboxMessages(array $messages): OutboxBatchResult + { + return $this->usingConnection(fn (QueueContract $queue): OutboxBatchResult => $queue instanceof PublishesOutboxMessages + ? $queue->publishOutboxMessages($messages) + : $this->publishOutboxMessagesSequentially($queue, $messages)); + } + /** * Run a callback with the underlying SQS queue. * @@ -342,14 +360,14 @@ protected function configureBorrowed(object $object): void /** * Run a callback with a borrowed queue. * - * @param Closure(Queue): mixed $callback + * @param Closure(Queue&QueueContract): mixed $callback */ protected function usingConnection(Closure $callback): mixed { $lease = $this->lease(); try { - /** @var Queue $queue */ + /** @var Queue&QueueContract $queue */ $queue = $lease->get(); $result = $callback($queue); } catch (Throwable $operationException) { diff --git a/src/queue/src/QueueServiceProvider.php b/src/queue/src/QueueServiceProvider.php index 3ec9787ec8..5ddf5e9c46 100644 --- a/src/queue/src/QueueServiceProvider.php +++ b/src/queue/src/QueueServiceProvider.php @@ -16,22 +16,29 @@ use Hypervel\Queue\Console\FailedTableCommand; use Hypervel\Queue\Console\FlushFailedCommand; use Hypervel\Queue\Console\ForgetFailedCommand; +use Hypervel\Queue\Console\ForgetOutboxCommand; use Hypervel\Queue\Console\ListenCommand; use Hypervel\Queue\Console\ListFailedCommand; +use Hypervel\Queue\Console\ListOutboxCommand; use Hypervel\Queue\Console\MonitorCommand; +use Hypervel\Queue\Console\MonitorOutboxCommand; +use Hypervel\Queue\Console\OutboxTableCommand; use Hypervel\Queue\Console\PauseCommand; use Hypervel\Queue\Console\PruneBatchesCommand; use Hypervel\Queue\Console\PruneFailedJobsCommand; +use Hypervel\Queue\Console\RelayCommand; use Hypervel\Queue\Console\RestartCommand; use Hypervel\Queue\Console\ResumeCommand; use Hypervel\Queue\Console\RetryBatchCommand; use Hypervel\Queue\Console\RetryCommand; +use Hypervel\Queue\Console\RetryOutboxCommand; use Hypervel\Queue\Console\TableCommand; use Hypervel\Queue\Console\WorkCommand; use Hypervel\Queue\Failed\DatabaseFailedJobProvider; use Hypervel\Queue\Failed\DatabaseUuidFailedJobProvider; use Hypervel\Queue\Failed\FileFailedJobProvider; use Hypervel\Queue\Failed\NullFailedJobProvider; +use Hypervel\Queue\Outbox\DatabaseOutboxRepository; use Hypervel\Support\ServiceProvider; use InvalidArgumentException; use Laravel\SerializableClosure\SerializableClosure; @@ -55,6 +62,7 @@ public function register(): void $this->registerListener(); $this->registerRoutes(); $this->registerFailedJobServices(); + $this->registerOutbox(); $this->commands([ BatchesTableCommand::class, @@ -62,16 +70,22 @@ public function register(): void FailedTableCommand::class, FlushFailedCommand::class, ForgetFailedCommand::class, + ForgetOutboxCommand::class, ListenCommand::class, ListFailedCommand::class, + ListOutboxCommand::class, MonitorCommand::class, + MonitorOutboxCommand::class, + OutboxTableCommand::class, PauseCommand::class, PruneBatchesCommand::class, PruneFailedJobsCommand::class, + RelayCommand::class, RestartCommand::class, ResumeCommand::class, RetryBatchCommand::class, RetryCommand::class, + RetryOutboxCommand::class, TableCommand::class, WorkCommand::class, ]); @@ -219,6 +233,23 @@ protected function registerRoutes(): void $this->app->singleton('queue.routes', fn ($app) => $app->make(QueueRoutes::class)); } + /** + * Register transactional outbox services without resolving their storage. + */ + protected function registerOutbox(): void + { + $this->app->singleton(DatabaseOutboxRepository::class, function ($app): DatabaseOutboxRepository { + $config = $app->make('config'); + + return new DatabaseOutboxRepository( + $app->make('db'), + $config->array('queue.outbox.databases'), + $config->string('queue.outbox.table'), + $config->integer('queue.outbox.retry_after'), + ); + }); + } + /** * Register the failed job services. */ diff --git a/src/queue/src/RedisQueue.php b/src/queue/src/RedisQueue.php index a3ce84dfab..4e02a5416d 100644 --- a/src/queue/src/RedisQueue.php +++ b/src/queue/src/RedisQueue.php @@ -9,11 +9,14 @@ use Hypervel\Contracts\Queue\ClearableQueue; use Hypervel\Contracts\Queue\IndexAwareQueue; use Hypervel\Contracts\Queue\Job as JobContract; +use Hypervel\Contracts\Queue\PublishesOutboxMessages; use Hypervel\Contracts\Queue\Queue as QueueContract; use Hypervel\Contracts\Redis\Factory as Redis; use Hypervel\Database\DatabaseTransactionsManager; use Hypervel\Queue\Jobs\InspectedJob; use Hypervel\Queue\Jobs\RedisJob; +use Hypervel\Queue\Outbox\OutboxBatchResult; +use Hypervel\Queue\Outbox\OutboxMessage; use Hypervel\Redis\RedisConnection; use Hypervel\Redis\RedisProxy; use Hypervel\Support\Collection; @@ -23,7 +26,7 @@ use Throwable; use UnitEnum; -class RedisQueue extends Queue implements QueueContract, ClearableQueue, IndexAwareQueue +class RedisQueue extends Queue implements QueueContract, ClearableQueue, IndexAwareQueue, PublishesOutboxMessages { public const int DEFAULT_RETRY_AFTER = 60; @@ -345,7 +348,7 @@ public function creationTimeOfOldestPendingJob(UnitEnum|string|null $queue = nul public function bulk(array $jobs, mixed $data = '', UnitEnum|string|null $queue = null): mixed { $queue = $this->normalizeQueue($queue); - $jobs = array_values($jobs); + $jobs = array_values($this->stageOutboxJobs($jobs, $data, $queue)); if ($jobs === []) { return null; @@ -392,17 +395,7 @@ static function (Queue $owner) use ($preparedJobs, $queue): void { */ protected function prepareBatchJobs(array $jobs, mixed $data, ?string $queue): array { - return (new Collection($jobs)) - ->map(function (object|string $job) use ($data, $queue): array { - $delay = $this->getJobDelay($job); - - return [ - 'job' => $job, - 'delay' => $delay, - 'payload' => $this->createPayload($job, $this->getQueue($queue), $data, $delay), - ]; - }) - ->all(); + return parent::prepareBatchJobs($jobs, $data, $this->getQueue($queue)); } /** @@ -422,7 +415,7 @@ protected function enqueueBatch(array $jobs, ?string $queue): void foreach ($jobs as $index => $job) { $this->raiseJobQueueingEvent($queue, $job['job'], $job['payload'], $job['delay']); - $jobs[$index]['payload'] = $this->preparePayloadForBulk( + $jobs[$index]['storedPayload'] = $this->preparePayloadForPush( $job['job'], $job['payload'], $queue, @@ -433,54 +426,116 @@ protected function enqueueBatch(array $jobs, ?string $queue): void foreach ($jobs as $job) { $arguments[] = $job['delay'] === null ? 'i' : $this->availableAt($job['delay']); - $arguments[] = $job['payload']; + $arguments[] = $job['storedPayload']; } $connection = $this->getConnection(); $this->isCluster ??= $connection->isCluster(); $queueKey = $this->getQueueRedisKey($queue); - $stored = $connection->evalWithShaCache( - LuaScripts::bulk(), - [$queueKey, $queueKey . ':notify', $queueKey . ':delayed'], - $arguments, - ); - - if (! is_int($stored) || $stored !== count($jobs)) { - throw new RuntimeException('Redis did not confirm every queued job in the batch.'); - } + $this->pushRawBatch($connection, $queueKey, $arguments); } catch (CanceledException $exception) { throw $exception; } catch (Throwable $exception) { - foreach ($jobs as $job) { - $this->raiseJobQueueingFailedEvent($queue, $job['job'], $job['payload'], $job['delay'], $exception); - } - - throw $exception; + throw $this->notifyBatchMembers( + $jobs, + fn (array $job) => $this->raiseJobQueueingFailedEvent($queue, $job['job'], $job['payload'], $job['delay'], $exception), + ) ?? $exception; } foreach ($jobs as $job) { $this->acceptDispatchLocks($job['job']); } - foreach ($jobs as $job) { - $this->handlePayloadPushedInBulk($job['payload'], $queue); - $this->raiseJobQueuedEvent($queue, null, $job['job'], $job['payload'], $job['delay']); + $exception = $this->notifyBatchMembers( + $jobs, + fn (array $job) => $this->handlePayloadPushed($job['storedPayload'], $queue), + fn (array $job) => $this->raiseJobQueuedEvent($queue, null, $job['job'], $job['payload'], $job['delay']), + ); + + if ($exception !== null) { + throw $exception; } } /** - * Prepare a payload for bulk storage. + * Publish stored outbox messages with one confirmed script per destination. + * + * @param list $messages */ - protected function preparePayloadForBulk(object|string $job, string $payload, ?string $queue): string + public function publishOutboxMessages(array $messages): OutboxBatchResult + { + $accepted = []; + + if ($messages === []) { + return new OutboxBatchResult; + } + + try { + $connection = $this->getConnection(); + $this->isCluster ??= $connection->isCluster(); + $groups = []; + + foreach ($messages as $message) { + $groups[$this->getQueueRedisKey($message->queue)][] = $message; + } + + foreach ($groups as $queueKey => $group) { + $prepared = []; + $arguments = []; + $now = $this->currentTime(); + + foreach ($group as $message) { + $payload = $this->preparePayloadForOutboxPublication($message->payload, $message->queue); + $prepared[] = ['payload' => $payload, 'queue' => $message->queue]; + $arguments[] = $message->deliverAt !== null && $message->deliverAt > $now ? $message->deliverAt : 'i'; + $arguments[] = $payload; + } + + $this->pushRawBatch($connection, $queueKey, $arguments); + + foreach ($group as $message) { + $accepted[] = $message->id; + } + + $failure = $this->notifyBatchMembers($prepared, fn (array $member) => $this->handlePayloadPushed($member['payload'], $member['queue'])); + + if ($failure !== null) { + throw $failure; + } + } + } catch (CanceledException $exception) { + throw $exception; + } catch (Throwable $exception) { + return new OutboxBatchResult($accepted, failure: $exception); + } + + return new OutboxBatchResult($accepted); + } + + /** + * Add publication-time metadata without recreating the stored job payload. + */ + protected function preparePayloadForOutboxPublication(string $payload, ?string $queue): string { return $payload; } /** - * Handle a payload that was stored as part of a batch. + * Store alternating availability and payload arguments with exact confirmation. + * + * @param list $arguments */ - protected function handlePayloadPushedInBulk(string $payload, ?string $queue): void + protected function pushRawBatch(RedisProxy $connection, string $queueKey, array $arguments): void { + $stored = $connection->evalWithShaCache( + LuaScripts::bulk(), + [$queueKey, $queueKey . ':notify', $queueKey . ':delayed'], + $arguments, + ); + + if (! is_int($stored) || $stored !== intdiv(count($arguments), 2)) { + throw new RuntimeException('Redis did not confirm every queued job in the batch.'); + } } /** @@ -723,6 +778,22 @@ public function getQueue(UnitEnum|string|null $queue): string return 'queues:' . $this->resolveQueue($queue === null || $queue === '' ? $this->default : $queue); } + /** + * Capture the logical destination before publication-time forwarding. + */ + protected function resolveOutboxQueue(?string $queue): ?string + { + return $queue === null || $queue === '' ? $this->default : $queue; + } + + /** + * Publish delayed Redis jobs immediately into their delayed set. + */ + protected function availableAtForOutbox(?string $queue, ?int $deliverAt): int + { + return $this->currentTime(); + } + /** * Get the cluster-safe Redis key for the given queue. * diff --git a/src/queue/src/SqsQueue.php b/src/queue/src/SqsQueue.php index 9573e31309..bddbb46391 100644 --- a/src/queue/src/SqsQueue.php +++ b/src/queue/src/SqsQueue.php @@ -5,6 +5,7 @@ namespace Hypervel\Queue; use Aws\Command; +use Aws\ResultInterface; use Aws\Sqs\Exception\SqsException; use Aws\Sqs\SqsClient; use DateInterval; @@ -13,10 +14,13 @@ use Hypervel\Contracts\Cache\Repository as CacheRepository; use Hypervel\Contracts\Queue\ClearableQueue; use Hypervel\Contracts\Queue\Job as JobContract; +use Hypervel\Contracts\Queue\PublishesOutboxMessages; use Hypervel\Contracts\Queue\Queue as QueueContract; use Hypervel\Database\DatabaseTransactionsManager; use Hypervel\ObjectPool\PoolErrorReporter; use Hypervel\Queue\Jobs\SqsJob; +use Hypervel\Queue\Outbox\OutboxBatchResult; +use Hypervel\Queue\Outbox\OutboxMessage; use Hypervel\Support\Arr; use Hypervel\Support\Collection; use Hypervel\Support\Str; @@ -26,7 +30,7 @@ use Throwable; use UnitEnum; -class SqsQueue extends Queue implements QueueContract, ClearableQueue +class SqsQueue extends Queue implements QueueContract, ClearableQueue, PublishesOutboxMessages { /** * The maximum SQS payload size in bytes (1 MB). @@ -282,7 +286,9 @@ public function later(DateInterval|DateTimeInterface|int $delay, object|string $ $queue = $this->normalizeQueue($queue); $queueName = $this->resolveQueueName($queue); - $this->ensureDelayIsSupported($delay, $queueName); + if (! $this->shouldStageViaOutbox($job)) { + $this->ensureDelayIsSupported($delay, $queueName); + } return $this->enqueueUsing( $job, @@ -315,7 +321,7 @@ static function ( public function bulk(array $jobs, mixed $data = '', UnitEnum|string|null $queue = null): mixed { $queue = $this->normalizeQueue($queue); - $jobs = array_values($jobs); + $jobs = array_values($this->stageOutboxJobs($jobs, $data, $queue)); if ($jobs === []) { return null; @@ -341,30 +347,14 @@ public function bulk(array $jobs, mixed $data = '', UnitEnum|string|null $queue } $messages = $this->prepareBatchMessages($afterCommit, $data, $queue); - - // A non-empty deferred group means partitionJobsByAfterCommit() resolved a transactions manager. - foreach ($afterCommit as $job) { - /** @var DatabaseTransactionsManager $transactions */ - $this->addJobRollbackCallback($transactions, $job); - } - - if ($this->afterCommitDispatcher !== null) { - $dispatcher = $this->afterCommitDispatcher; - - $transactions->addCallback( - static fn () => $dispatcher( - static function (Queue $owner) use ($messages, $queue): void { - /** @var SqsQueue $owner */ - $owner->sendBatchedMessages($messages, $queue); - } - ) - ); - - return null; - } - - $transactions->addCallback( - fn () => $this->sendBatchedMessages($messages, $queue) + /** @var DatabaseTransactionsManager $transactions */ + $this->deferBatchEnqueueAfterCommit( + $transactions, + $afterCommit, + static function (Queue $owner) use ($messages, $queue): void { + /** @var SqsQueue $owner */ + $owner->sendBatchedMessages($messages, $queue); + }, ); return null; @@ -386,6 +376,14 @@ protected function ensureBulkDelaysAreSupported(array $jobs, ?string $queue): vo } } + /** + * Reuse SQS payload preparation for set-based outbox staging. + */ + protected function prepareBatchJobs(array $jobs, mixed $data, ?string $queue): array + { + return $this->prepareBatchMessages($jobs, $data, $queue); + } + /** * Create the payload for each of the given jobs. * @@ -423,22 +421,27 @@ protected function prepareBatchMessages(array $jobs, mixed $data, ?string $queue */ protected function sendBatchedMessages(array $messages, ?string $queue): void { - foreach ($messages as $id => $message) { - $messages[$id]['payload'] = $this->finalizePayloadForQueueing( - $queue, - $message['job'], - $message['payload'], - $message['delay'], - ); - } - $outstandingMessageIds = array_fill_keys(array_keys($messages), true); try { + foreach ($messages as $id => $message) { + $messages[$id]['payload'] = $this->finalizePayloadForQueueing( + $queue, + $message['job'], + $message['payload'], + $message['delay'], + ); + } + $entries = []; $overflow = []; foreach ($messages as $id => $message) { + $message['payload'] = $messages[$id]['storedPayload'] = $this->preparePayloadForPush( + $message['job'], + $message['payload'], + $queue, + ); $entry = $this->prepareSendMessageBatchEntry($id, $message, $queue); if ($this->willOverflow($message['payload'])) { @@ -462,47 +465,19 @@ protected function sendBatchedMessages(array $messages, ?string $queue): void // Dispatch chunks serially so later messages cannot arrive ahead of an unsent failed chunk... foreach ($chunks as $chunk) { - foreach ($chunk as $entry) { - $message = $messages[$entry['Id']]; - - $this->raiseJobQueueingEvent($queue, $message['job'], $message['payload'], $message['delay']); - } - - $writtenPaths = []; - $attemptedPath = null; - try { foreach ($chunk as $entry) { - if (! isset($overflow[$entry['Id']])) { - continue; - } + $message = $messages[$entry['Id']]; - /** @var CacheRepository $store */ - $overflowPayload = $overflow[$entry['Id']]; - $attemptedPath = $overflowPayload['path']; - $this->storeOverflowPayload($store, $attemptedPath, $overflowPayload['payload']); - $writtenPaths[] = $attemptedPath; - $attemptedPath = null; - } - } catch (CanceledException $cancellation) { - if ($attemptedPath !== null) { - $writtenPaths[] = $attemptedPath; + $this->raiseJobQueueingEvent($queue, $message['job'], $message['payload'], $message['delay']); } - try { - /** @var CacheRepository $store */ - $this->cleanupOverflowPayloads($store, $writtenPaths); - } catch (CanceledException) { + if ($store !== null) { + $this->storeBatchOverflowPayloads($chunk, $overflow, $store); } - - throw $cancellation; + } catch (CanceledException $exception) { + throw $exception; } catch (Throwable $exception) { - if ($attemptedPath !== null) { - $writtenPaths[] = $attemptedPath; - } - - /** @var CacheRepository $store */ - $this->cleanupOverflowPayloads($store, $writtenPaths); $this->raiseBatchFailedEvents($messages, $outstandingMessageIds, $queue, $exception); throw $exception; @@ -523,6 +498,8 @@ protected function sendBatchedMessages(array $messages, ?string $queue): void throw $exception; } + $accepted = []; + foreach ($result['Successful'] ?? [] as $success) { if (! isset($outstandingMessageIds[$success['Id']])) { continue; @@ -530,53 +507,169 @@ protected function sendBatchedMessages(array $messages, ?string $queue): void $message = $messages[$success['Id']]; unset($outstandingMessageIds[$success['Id']]); + $this->acceptDispatchLocks($message['job']); + $accepted[] = $success; + } - $this->raiseJobQueuedEvent( - $queue, - $success['MessageId'], - $message['job'], - $message['payload'], - $message['delay'], - ); + $exception = null; + + if (! empty($result['Failed'])) { + $exception = $this->batchRejectionException($queueUrl, $chunk, $result); + + if ($store !== null) { + $rejectedPaths = []; + + foreach ($result['Failed'] as $rejected) { + if (isset($overflow[$rejected['Id']])) { + $rejectedPaths[] = $overflow[$rejected['Id']]['path']; + } + } + + $this->cleanupOverflowPayloads($store, $rejectedPaths); + } } - if (empty($result['Failed'])) { + $notificationFailure = $this->notifyBatchMembers( + $accepted, + fn (array $success) => $this->handlePayloadPushed($messages[$success['Id']]['storedPayload'], $queue), + function (array $success) use ($messages, $queue): void { + $message = $messages[$success['Id']]; + $this->raiseJobQueuedEvent($queue, $success['MessageId'], $message['job'], $message['payload'], $message['delay']); + }, + ); + $exception ??= $notificationFailure; + + if ($exception === null) { continue; } - $failure = $result['Failed'][0]; - $exception = new SqsException( - sprintf( - 'SQS SendMessageBatch rejected [%d] of [%d] messages. First failure [%s]: %s', - count($result['Failed']), - count($chunk), - $failure['Code'] ?? 'Unknown', - $failure['Message'] ?? '', - ), - new Command('SendMessageBatch', ['QueueUrl' => $queueUrl, 'Entries' => $chunk]), - [ - 'code' => $failure['Code'] ?? null, - 'message' => $failure['Message'] ?? null, - 'result' => $result, - ], - ); + try { + $this->raiseBatchFailedEvents($messages, $outstandingMessageIds, $queue, $exception); + } catch (CanceledException $cancellation) { + throw $cancellation; + } catch (Throwable $throwable) { + $notificationFailure ??= $throwable; + } + + throw $notificationFailure ?? $exception; + } + } + + /** + * Publish stored SQS messages with attempt-owned overflow payloads. + * + * @param list $messages + */ + public function publishOutboxMessages(array $messages): OutboxBatchResult + { + $accepted = []; - if ($store !== null) { - $rejectedPaths = []; + try { + $groups = []; + + foreach ($messages as $message) { + $groups[$this->getQueue($message->queue)][$message->id] = $message; + } - foreach ($result['Failed'] as $rejected) { - if (isset($overflow[$rejected['Id']])) { - $rejectedPaths[] = $overflow[$rejected['Id']]['path']; + foreach ($groups as $queueUrl => $group) { + $entries = []; + $overflow = []; + + foreach ($group as $message) { + $payload = $message->payload; + + if ($this->willOverflow($payload)) { + $path = static::EXTENDED_PAYLOAD_CACHE_PREFIX . $message->uuid . ':' . $message->leaseToken; + $overflow[$message->id] = ['path' => $path, 'payload' => $payload]; + $payload = json_encode(['@pointer' => $path], JSON_THROW_ON_ERROR); } + + $entries[] = [ + 'Id' => (string) $message->id, + 'MessageBody' => $payload, + ...($message->options['transport']['sqs'] ?? []), + ]; } - $this->cleanupOverflowPayloads($store, $rejectedPaths); - } + $store = $overflow === [] ? null : $this->overflowStore(); + $fifo = str_ends_with($queueUrl, '.fifo'); - $this->raiseBatchFailedEvents($messages, $outstandingMessageIds, $queue, $exception); + foreach ($this->chunkBatchEntries($entries) as $chunk) { + if ($store !== null) { + $this->storeBatchOverflowPayloads($chunk, $overflow, $store); + } + $now = $this->currentTime(); + + if (! $fifo) { + foreach ($chunk as &$entry) { + $deliverAt = $group[$entry['Id']]->deliverAt; + + if ($deliverAt !== null) { + $entry['DelaySeconds'] = min(900, max(0, $deliverAt - $now)); + } + } + unset($entry); + } + + // SDK failures are ambiguous; only explicit rejects own safe cleanup. + $result = $this->sqs->sendMessageBatch(['QueueUrl' => $queueUrl, 'Entries' => $chunk]); + $chunkIds = array_fill_keys(array_column($chunk, 'Id'), true); + + foreach ($result['Successful'] ?? [] as $success) { + if (isset($chunkIds[$success['Id']])) { + $accepted[] = (int) $success['Id']; + } + } + + if (! empty($result['Failed'])) { + if ($store !== null) { + $paths = []; + + foreach ($result['Failed'] as $rejected) { + if (isset($chunkIds[$rejected['Id']], $overflow[$rejected['Id']])) { + $paths[] = $overflow[$rejected['Id']]['path']; + } + } + + $this->cleanupOverflowPayloads($store, $paths); + } + + throw $this->batchRejectionException($queueUrl, $chunk, $result); + } + } + } + } catch (CanceledException $exception) { throw $exception; + } catch (Throwable $exception) { + return new OutboxBatchResult($accepted, failure: $exception); } + + return new OutboxBatchResult($accepted); + } + + /** + * Describe explicit SQS rejects while retaining the complete response. + */ + protected function batchRejectionException(string $queueUrl, array $chunk, ResultInterface $result): SqsException + { + $failure = $result['Failed'][0]; + + return new SqsException( + sprintf( + 'SQS SendMessageBatch rejected [%d] of [%d] messages. First failure [%s]: %s', + count($result['Failed']), + count($chunk), + $failure['Code'] ?? 'Unknown', + $failure['Message'] ?? '', + ), + new Command('SendMessageBatch', ['QueueUrl' => $queueUrl, 'Entries' => $chunk]), + [ + 'code' => $failure['Code'] ?? null, + 'message' => $failure['Message'] ?? null, + 'result' => $result, + ], + ); } /** @@ -588,16 +681,16 @@ protected function raiseBatchFailedEvents( ?string $queue, Throwable $exception, ): void { - foreach (array_keys($outstandingMessageIds) as $id) { - $message = $messages[$id]; + $notificationFailure = $this->notifyBatchMembers( + array_keys($outstandingMessageIds), + function (int $id) use ($messages, $queue, $exception): void { + $message = $messages[$id]; + $this->raiseJobQueueingFailedEvent($queue, $message['job'], $message['payload'], $message['delay'], $exception); + }, + ); - $this->raiseJobQueueingFailedEvent( - $queue, - $message['job'], - $message['payload'], - $message['delay'], - $exception, - ); + if ($notificationFailure !== null) { + throw $notificationFailure; } } @@ -691,6 +784,41 @@ protected function storeOverflowPayload(CacheRepository $store, string $path, st } } + /** + * Store one chunk's overflow payloads, cleaning every attempted write on failure. + */ + protected function storeBatchOverflowPayloads(array $chunk, array $overflow, CacheRepository $store): void + { + $paths = []; + + try { + foreach ($chunk as $entry) { + if (! isset($overflow[$entry['Id']])) { + continue; + } + + $payload = $overflow[$entry['Id']]; + $paths[] = $payload['path']; + $this->storeOverflowPayload($store, $payload['path'], $payload['payload']); + } + } catch (CanceledException $cancellation) { + if ($paths !== []) { + try { + $this->cleanupOverflowPayloads($store, $paths); + } catch (CanceledException) { + } + } + + throw $cancellation; + } catch (Throwable $exception) { + if ($paths !== []) { + $this->cleanupOverflowPayloads($store, $paths); + } + + throw $exception; + } + } + /** * Remove unpublished overflow payloads without hiding the primary failure. * @@ -869,6 +997,40 @@ public function getQueue(UnitEnum|string|null $queue): string : $queue; } + /** + * Capture the logical destination before publication-time forwarding. + */ + protected function resolveOutboxQueue(?string $queue): ?string + { + return $queue === null || $queue === '' ? $this->default : $queue; + } + + /** + * Capture FIFO options once while the original job and callbacks are available. + */ + protected function prepareOutboxOptions(object|string $job, ?string $queue, string $payload): array + { + $options = $this->getQueueableOptions($job, $queue, $payload); + + return $options === [] ? [] : ['transport' => ['sqs' => $options]]; + } + + /** + * Keep a delayed job staged until SQS can honor its remaining delay. + */ + protected function availableAtForOutbox(?string $queue, ?int $deliverAt): int + { + $now = $this->currentTime(); + + if ($deliverAt === null) { + return $now; + } + + return str_ends_with($this->resolveQueueName($queue), '.fifo') + ? $deliverAt + : max($now, $deliverAt - 900); + } + /** * Add the given suffix to the given queue name. */ diff --git a/src/queue/src/SyncQueue.php b/src/queue/src/SyncQueue.php index 21884e5ff3..69d4be2d7a 100644 --- a/src/queue/src/SyncQueue.php +++ b/src/queue/src/SyncQueue.php @@ -4,8 +4,10 @@ namespace Hypervel\Queue; +use Closure; use DateInterval; use DateTimeInterface; +use Hypervel\Context\CoroutineContext; use Hypervel\Contracts\Events\Dispatcher as EventDispatcher; use Hypervel\Contracts\Queue\Job as JobContract; use Hypervel\Contracts\Queue\Queue as QueueContract; @@ -215,7 +217,14 @@ protected function executePayload(string $payload, ?string $queue = null): int { // JobProcessing hydrates the payload's context. Keep it through all job // events without replacing the context of the dispatching coroutine. - return $this->withLogContext(null, fn (): int => $this->processPayload($payload, $queue)); + return $this->withLogContext(null, function () use ($payload, $queue): int { + if (CoroutineContext::get(self::OUTBOX_FORWARDED_CONTEXT_KEY) === null) { + return $this->processPayload($payload, $queue); + } + + // A running handler may dispatch independent work through the same string handler. + return $this->withOutboxForwarding(null, fn (): int => $this->processPayload($payload, $queue)); + }); } /** @@ -357,6 +366,46 @@ public function later(DateInterval|DateTimeInterface|int $delay, object|string $ return $this->push($job, $data, $queue); } + /** + * Keep local queue execution out of durable outbox storage. + */ + protected function shouldStageViaOutbox(object|string $job): bool + { + return false; + } + + /** + * Degrade selected outbox delivery to local execution after commit. + */ + protected function shouldDispatchAfterCommit(object|string $job): bool + { + if (parent::shouldDispatchAfterCommit($job)) { + return true; + } + + if (! $this->shouldDispatchViaOutbox($job) || ! $this->container->has('db.transactions')) { + return false; + } + + /** @var DatabaseTransactionsManager $transactions */ + $transactions = $this->container->make('db.transactions'); + + return $transactions->hasApplicableTransaction(); + } + + /** + * Honor the selected transaction without resolving outbox storage services. + */ + protected function deferEnqueueAfterCommit(DatabaseTransactionsManager $transactions, object|string $job, Closure $enqueue): void + { + $this->deferBatchEnqueueAfterCommit( + $transactions, + [$job], + $enqueue, + $this->resolveOutboxCommitConnection($transactions, $job), + ); + } + /** * Pop the next job off of the queue. */ diff --git a/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php b/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php new file mode 100644 index 0000000000..446dc84f04 --- /dev/null +++ b/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php @@ -0,0 +1,40 @@ +id(); + $table->uuid('uuid')->unique(); + $table->string('connection'); + $table->string('queue')->nullable(); + $table->longText('payload'); + $table->text('options')->nullable(); + $table->unsignedBigInteger('deliver_at')->nullable(); + $table->unsignedInteger('attempts')->default(0); + $table->unsignedBigInteger('available_at'); + $table->uuid('lease_token')->nullable(); + $table->text('last_error')->nullable(); + $table->unsignedBigInteger('created_at'); + + $table->index(['available_at', 'id']); + }); + } + + /** + * Reverse the migrations. + */ + public function down(): void + { + Schema::dropIfExists('job_outbox'); + } +}; diff --git a/tests/Benchmarks/QueueOutbox/Fixtures/OutboxBenchmarkJob.php b/tests/Benchmarks/QueueOutbox/Fixtures/OutboxBenchmarkJob.php new file mode 100644 index 0000000000..c86748a82f --- /dev/null +++ b/tests/Benchmarks/QueueOutbox/Fixtures/OutboxBenchmarkJob.php @@ -0,0 +1,27 @@ +table = 'bench_outbox_' . $identifier; + $this->jobsTable = 'bench_jobs_' . $identifier; + } + + /** + * Build isolated tables from the shipped schema and capture one real payload. + */ + public function setUp(): void + { + $connection = $this->application->make('db')->connection($this->source); + $schema = $connection->getSchemaBuilder(); + $tables = [$this->table => 'outbox']; + if ($this->settings['target'] === 'database') { + $tables[$this->jobsTable] = 'jobs'; + } + + foreach ($tables as $table => $stub) { + if ($schema->hasTable($table)) { + throw new RuntimeException("Benchmark table [{$table}] already exists."); + } + + $path = tempnam(sys_get_temp_dir(), 'hypervel-outbox-migration-'); + if ($path === false) { + throw new RuntimeException('Could not create the temporary migration file.'); + } + $this->ownedTables[] = $table; + try { + $contents = file_get_contents(dirname(__DIR__, 4) . "/src/queue/src/Console/stubs/{$stub}.stub"); + file_put_contents($path, str_replace('{{table}}', $table, $contents)); + (require $path)->up(); + } finally { + unlink($path); + } + } + + $this->repository = $this->application->make(DatabaseOutboxRepository::class); + + // Redis staging never contacts Redis. It also avoids the database-queue atomic fast path. + $this->application->make('config')->set('queue.connections.benchmark_seed', [ + 'driver' => 'redis', 'connection' => 'default', 'queue' => $this->queue, 'retry_after' => 3600, + ]); + $this->application->make('queue')->connection('benchmark_seed') + ->push((new OutboxBenchmarkJob(str_repeat('x', $this->settings['payload_bytes'])))->viaOutbox()); + $row = (array) $connection->table($this->table)->useWritePdo()->first(); + unset($row['id']); + $row['connection'] = 'benchmark'; + $this->template = $row; + $connection->table($this->table)->delete(); + } + + /** + * Seed outside timing; retain claim churn from warmup into the measured sample. + */ + public function prepare(int $jobs): void + { + if ($this->settings['case'] === 'relay') { + $this->insertRows($jobs); + } elseif ($this->settings['case'] === 'claims' && $this->distribution === []) { + $background = intdiv($this->settings['backlog'], 4); + $leased = $this->settings['leased']; + $eligible = max($this->settings['backlog'] - 2 * $background - $leased, $this->settings['batch'] * $this->settings['concurrency']); + $this->distribution = ['eligible' => $eligible, 'delayed' => $background, 'leased' => $leased, 'retrying' => $background]; + $future = time() + 86400; + $this->insertRows($eligible); + $this->insertRows($background, ['available_at' => $future, 'deliver_at' => $future]); + $this->insertRows($background, ['available_at' => $future, 'attempts' => 5, 'last_error' => 'Benchmark retry backlog']); + $this->insertRows($leased, ['available_at' => $future, 'attempts' => 1, 'lease_token' => (string) Str::uuid()]); + $this->analyze(); + } + } + + /** + * Exercise production staging, the real relay worker, or competing claims. + */ + public function execute(int $jobs): ?array + { + if ($this->settings['case'] === 'stage') { + $database = $this->application->make('db'); + $queues = $this->application->make('queue'); + for ($offset = 0; $offset < $jobs; $offset += $this->settings['batch']) { + $this->inFreshCoroutine(function () use ($database, $queues, $jobs, $offset): void { + $members = []; + for ($index = 0; $index < min($this->settings['batch'], $jobs - $offset); ++$index) { + $members[] = (new OutboxBenchmarkJob(str_repeat('x', $this->settings['payload_bytes'])))->viaOutbox(); + } + $database->connection($this->source)->transaction(function () use ($queues, $members): void { + if ($this->settings['batch'] === 1) { + $queues->connection('benchmark')->push($members[0]); + } else { + $queues->connection('benchmark')->bulk($members); + } + }); + }); + } + + return null; + } + + if ($this->settings['case'] === 'relay') { + $status = $this->application->make(OutboxWorker::class)->run($this->source, new OutboxWorkerOptions( + batch: $this->settings['batch'], + concurrency: $this->settings['concurrency'], + maxJobs: $jobs, + maxTime: 300, + memory: 1024, + stopWhenEmpty: true, + )); + if ($status !== 0) { + throw new RuntimeException("Relay exited with status {$status}."); + } + + return null; + } + + $callbacks = []; + $clients = min($jobs, $this->settings['concurrency']); + $deadline = hrtime(true) + 300_000_000_000; + for ($client = 0; $client < $clients; ++$client) { + $quota = intdiv($jobs, $clients) + ($client < $jobs % $clients ? 1 : 0); + $callbacks[] = function () use ($quota, $deadline): array { + $latencies = []; + $completed = 0; + $empty = 0; + while ($completed < $quota) { + if (hrtime(true) >= $deadline) { + throw new RuntimeException('The claims workload exceeded its five-minute limit.'); + } + $started = hrtime(true); + $messages = $this->inFreshCoroutine(fn (): array => $this->repository->claim($this->source, min($this->settings['batch'], $quota - $completed))); + $latencies[] = (hrtime(true) - $started) / 1e6; + if ($messages === []) { + ++$empty; + continue; + } + $this->inFreshCoroutine(function () use ($messages): void { + $settled = $this->repository->settle($this->source, $messages, new OutboxBatchResult(array_column($messages, 'id'))); + if ($settled !== count($messages)) { + throw new RuntimeException('A claimed batch lost ownership during the workload.'); + } + $this->insertRows(count($messages)); + }); + $completed += count($messages); + } + + return ['latencies' => $latencies, 'empty' => $empty]; + }; + } + + $results = parallel($callbacks, $clients); + $latencies = array_merge(...array_column($results, 'latencies')); + sort($latencies, SORT_NUMERIC); + + return [ + 'claim_operations' => count($latencies), + 'empty_claims' => array_sum(array_column($results, 'empty')), + 'claim_ms_p50' => $latencies[(int) ceil(count($latencies) * 0.5) - 1], + 'claim_ms_p95' => $latencies[(int) ceil(count($latencies) * 0.95) - 1], + 'backlog' => $this->distribution, + ]; + } + + /** + * Check completed work rather than treating an early or partial drain as faster. + */ + public function verify(int $jobs): void + { + $query = $this->application->make('db')->connection($this->source)->table($this->table)->useWritePdo(); + $expected = match ($this->settings['case']) { + 'stage' => $jobs, + 'claims' => array_sum($this->distribution), + default => 0, + }; + if ($query->count() !== $expected) { + throw new RuntimeException('The outbox remainder does not match the completed workload.'); + } + if ($this->settings['case'] === 'claims' && (clone $query)->whereNotNull('lease_token')->count() !== $this->distribution['leased']) { + throw new RuntimeException('The claims workload left unexpected leases.'); + } + if ($this->settings['case'] === 'relay' && $this->application->make('queue')->connection('benchmark')->size($this->queue) !== $jobs) { + throw new RuntimeException('The relay target does not contain every staged job.'); + } + } + + /** + * Clear warmup output without removing the mixed claims backlog. + */ + public function reset(): void + { + if ($this->settings['case'] !== 'claims') { + $this->application->make('db')->connection($this->source)->table($this->table)->delete(); + } + if ($this->settings['case'] === 'relay') { + $this->application->make('queue')->connection('benchmark')->clear($this->queue); + } + } + + /** + * Insert bounded batches of realistic opaque envelopes. + */ + private function insertRows(int $count, array $overrides = []): void + { + $connection = $this->application->make('db')->connection($this->source); + $payload = json_decode($this->template['payload'], true, flags: JSON_THROW_ON_ERROR); + for ($offset = 0; $offset < $count; $offset += 1000) { + $rows = []; + for ($index = 0; $index < min(1000, $count - $offset); ++$index) { + $uuid = (string) Str::uuid(); + $rows[] = array_replace($this->template, [ + 'uuid' => $uuid, 'payload' => json_encode(array_replace($payload, ['uuid' => $uuid]), JSON_THROW_ON_ERROR), + 'available_at' => time(), 'created_at' => time(), + ], $overrides); + } + $this->repository->insert($connection, $rows); + } + } + + /** + * End each resource-owning phase before starting unrelated work. + */ + protected function inFreshCoroutine(Closure $callback): mixed + { + return (new Waiter(-1))->wait($callback, waitForChildTermination: true); + } + + /** + * Refresh planner statistics without changing vacuum or engine settings. + */ + private function analyze(): void + { + $connection = $this->application->make('db')->connection($this->source); + $table = $connection->getQueryGrammar()->wrapTable($this->table); + $connection->statement(in_array($connection->getDriverName(), ['mysql', 'mariadb'], true) ? "ANALYZE TABLE {$table}" : "ANALYZE {$table}"); + } + + /** + * Capture actual claim SQL once and inspect executing plans after churn. + */ + public function explain(): array + { + $this->analyze(); + $connection = $this->application->make('db')->connection($this->source); + $connection->beginTransaction(); + $connection->enableQueryLog(); + $connection->flushQueryLog(); + try { + $this->repository->claim($this->source, $this->settings['batch']); + $queries = $connection->getQueryLog(); + } finally { + $connection->disableQueryLog(); + $connection->rollBack(); + } + + $plans = []; + foreach ($queries as $query) { + if (! str_starts_with(strtolower(ltrim($query['query'])), 'with ') + && ! str_starts_with(strtolower(ltrim($query['query'])), 'select ')) { + continue; + } + $prefix = match ($connection->getDriverName()) { + 'pgsql' => 'EXPLAIN (ANALYZE, BUFFERS, WAL, FORMAT JSON) ', + 'mysql' => 'EXPLAIN ANALYZE ', + 'mariadb' => 'ANALYZE FORMAT=JSON ', + 'sqlite' => 'EXPLAIN QUERY PLAN ', + default => throw new InvalidArgumentException('No plan command is defined for this database driver.'), + }; + $connection->beginTransaction(); + try { + $plans[] = ['sql' => $query['query'], 'plan' => $connection->selectFromWriteConnection($prefix . $query['query'], $query['bindings'])]; + } finally { + $connection->rollBack(); + } + } + + return $plans; + } + + /** + * Capture the cluster-wide logical WAL position outside the timed workload. + */ + public function walPosition(): ?string + { + $connection = $this->application->make('db')->connection($this->source); + + return $this->settings['case'] === 'claims' && $connection->getDriverName() === 'pgsql' + ? $connection->selectOne('SELECT pg_current_wal_insert_lsn() AS position', useReadPdo: false)->position + : null; + } + + /** + * Report total workload WAL, including refills, settlement and background activity. + */ + public function walBytes(?string $start): ?int + { + return $start === null ? null : (int) $this->application->make('db')->connection($this->source) + ->selectOne('SELECT pg_wal_lsn_diff(pg_current_wal_insert_lsn(), ?) AS bytes', [$start], false)->bytes; + } + + /** + * Describe the database and actual serialized payload without credentials. + */ + public function backend(): array + { + $connection = $this->application->make('db')->connection($this->source); + + return [ + 'driver' => $connection->getDriverName(), 'version' => $connection->getServerVersion(), + 'database' => $connection->getDatabaseName(), 'table' => $this->table, + 'payload_bytes' => strlen($this->template['payload']), + 'functional_only' => $connection->getDriverName() === 'sqlite', + ]; + } + + /** + * Remove only the tables this invocation created. + */ + public function tearDown(): void + { + $schema = $this->application->make('db')->connection($this->source)->getSchemaBuilder(); + foreach (array_reverse($this->ownedTables) as $table) { + $schema->dropIfExists($table); + if ($schema->hasTable($table)) { + throw new RuntimeException("Benchmark table [{$table}] was not removed."); + } + } + } +} diff --git a/tests/Benchmarks/QueueOutbox/README.md b/tests/Benchmarks/QueueOutbox/README.md new file mode 100644 index 0000000000..ae58968a13 --- /dev/null +++ b/tests/Benchmarks/QueueOutbox/README.md @@ -0,0 +1,67 @@ +# Queue outbox benchmark + +This developer-only harness measures ordinary dispatch, transactional outbox staging, relay delivery, and concurrent SQL claims. It uses the real queue drivers, repository and relay worker. It does not execute application jobs, run in PHPUnit, or impose CI throughput thresholds. + +See [results.md](results.md) for recorded measurements and the PostgreSQL claim-design comparison. + +Run from the components repository root. Get fresh confirmation that the machine is idle before collecting performance measurements. Tiny invocations may check execution, counts and cleanup while the machine is busy; their timings are not performance evidence. + +```shell +timeout 600 php -d opcache.enable_cli=1 -d opcache.jit=disable \ + tests/Benchmarks/QueueOutbox/benchmark.php \ + --case=stage --database-connection=pgsql --jobs=10000 --batch=1000 \ + > /tmp/outbox-staging.json +``` + +The harness reads `.env` through `tests/bootstrap.php`. Use a disposable database with permission to create and drop tables. `--database-connection` selects an existing configured connection; its pool size is set by `--pool-size`. Omitting it creates a temporary file-backed SQLite database for functional checks only. SQLite timings do not represent a production networked database. + +Tables have random `bench_outbox_` / `bench_jobs_` names, and Redis uses a random queue name. Schema comes from the shipped queue migration stubs. The harness removes its own tables and queue keys in `finally`; it never flushes Redis or migrates existing application tables. Resource names are printed to stderr for manual cleanup after a forced kill. Do not run application workers against its queue. Normal exceptions also produce a nonzero exit status. + +## Cases + +| Case | Timed work | +| --- | --- | +| `direct` | Normal Redis single or bulk publication, without selecting the outbox. No SQL setup or outbox service resolution. | +| `stage` | Single or bulk dispatch through `viaOutbox()`, each call in its own SQL transaction. No Redis calls are needed. | +| `relay` | Actual `OutboxWorker` drain to Redis or a database queue. Seeded jobs are outside timing; claim, publication, settlement and worker lifecycle are inside. | +| `claims` | Concurrent claim → accepted settlement → refill cycles against a mixed backlog. Each claim and settlement/refill phase has its own resource lifetime. | + +`--batch=1` uses single dispatch for `direct` and `stage`; larger values use `bulk()`. Compare single, 1,000-job and 10,000-job staging calls using `--batch=1`, `1000`, and `10000`. Claims and relay batches follow the production worker limit of 1,000. + +Use `--target=database` for the database relay case; otherwise the target is Redis. `--redis-connection=default` selects the configured Redis connection. Publication uses its real driver batching and the normal database binding limits. Run each case separately: + +```shell +timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php --case=direct --jobs=10000 --batch=100 +timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php --case=stage --database-connection=mysql --jobs=10000 --batch=1000 +timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php --case=relay --database-connection=pgsql --jobs=10000 --batch=100 --concurrency=8 --pool-size=8 +timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php --case=relay --target=database --database-connection=pgsql --jobs=10000 --concurrency=8 --pool-size=8 +``` + +Concurrency applies to relay and claims. Direct and staging workloads are sequential producers. Vary pool size and concurrency together to distinguish pool pressure from transport latency. Existing tiny-pool integration tests prove the release boundaries; this harness checks that no connections remain borrowed when a sample finishes, without adding a sampling timer or per-operation pool instrumentation. The reported `cache_store` is `worker-array`: relay restart checks still run, but these timings exclude the external cache round trips a Redis or database cache would add. + +`--payload-bytes=256` controls the job's string field, not total serialized envelope size. `--warmup=100` runs outside timing; every invocation produces one measured sample. Warmup output is removed before direct, staging and relay measurements. Every run checks source remainder, target counts where applicable, remaining claims and cleanup. A partial drain is a failure, not a faster result. Relay and claim work have a five-minute bound; retain the external timeout for setup and transport failures too. + +## Claims and query plans + +`--backlog=10000` sizes the mixed backlog; delayed and retrying rows each take a quarter. `--leased` separately controls seeded live leases, defaulting to `batch × concurrency`, not a fraction of the backlog. The remaining rows are eligible, increased when necessary to provide at least one batch per concurrent claimant; the report gives the actual distribution. Seeded leases represent other in-flight work, in addition to the short-lived claims made by this process. Each settled batch is replaced with fresh eligible rows. Warmup churn is retained, so measurement is not limited to a freshly inserted table. Background deadlines are one day away, beyond the bounded run. + +Claim p50/p95 includes connection checkout, the repository operation, row materialization and child-coroutine cleanup, not just server execution time. Use a pool at least as large as concurrency when comparing claim statements; smaller pools also measure waiting for other clients' settlement and refill. Empty claims caused by competing fallback claimants count as attempts and are reported separately. Cycle throughput and WAL include settlement and refill, including UUID generation and payload encoding. Latency samples add common harness overhead and are retained only for this bounded invocation. + +```shell +timeout 600 php tests/Benchmarks/QueueOutbox/benchmark.php \ + --case=claims --database-connection=pgsql --jobs=10000 \ + --backlog=100000 --batch=100 --concurrency=8 --pool-size=8 --explain \ + > /tmp/outbox-claims.json +``` + +`--explain` refreshes table statistics and captures the actual claim SQL once after churn, outside timing. Executing plans run in transactions that are rolled back. PostgreSQL uses `EXPLAIN (ANALYZE, BUFFERS, WAL, FORMAT JSON)`; MySQL uses `EXPLAIN ANALYZE` for selects; MariaDB uses `ANALYZE FORMAT=JSON`; SQLite uses `EXPLAIN QUERY PLAN`. PostgreSQL's WAL option requires 13+, and MySQL's executing explain requires 8.0.18+. Engines lacking these commands fail with their native error; omit `--explain` there. No optimizer plan is asserted as a correctness requirement. + +The cluster-wide `pg_current_wal_insert_lsn()` delta includes the entire claim/settle/refill workload and unrelated database activity. It is not isolated index-write cost. The executing PostgreSQL plan supplies statement-local WAL and buffer evidence. Checkpoint state and full-page images can affect comparisons; use repeated samples rather than one ratio. See PostgreSQL's [WAL position functions](https://www.postgresql.org/docs/current/functions-admin.html#FUNCTIONS-ADMIN-BACKUP) and [EXPLAIN options](https://www.postgresql.org/docs/current/sql-explain.html). + +## Comparisons and interpretation + +For direct-path regression checks, copy this directory into an independently installed pre-feature worktree. The `direct` case uses only existing queue APIs and does not load feature-only fixtures. Use identical dependency versions, PHP configuration, opcache/JIT settings and services. Record revisions, uncommitted changes and commands with raw JSON outside Git. Alternate baseline/current order for at least ten fresh-process pairs; compare medians and spread, not the fastest sample. + +Direct publication and durable staging provide different guarantees; their throughput is not an apples-to-apples regression comparison. The report includes wall time, process CPU, heap/GC data and non-secret backend settings. CPU excludes database and Redis processes; heap growth is not an allocation count. Driver/network latency, payload size and database durability settings remain part of real throughput. + +For a quick functional check, use `--jobs=8 --batch=2 --warmup=2 --backlog=12`. Check all four cases, both relay targets and PostgreSQL claims with `--explain` when those services are available. Keep these checks separate from idle-machine measurements. diff --git a/tests/Benchmarks/QueueOutbox/benchmark.php b/tests/Benchmarks/QueueOutbox/benchmark.php new file mode 100644 index 0000000000..ed749e404c --- /dev/null +++ b/tests/Benchmarks/QueueOutbox/benchmark.php @@ -0,0 +1,276 @@ +#!/usr/bin/env php +wait($callback, waitForChildTermination: true); +} + +/** + * Run one sample in a new process, with setup and verification outside timing. + */ +function main(): void +{ + $options = getopt('', ['case:', 'target:', 'database-connection:', 'redis-connection:', 'jobs:', 'batch:', 'concurrency:', 'pool-size:', 'backlog:', 'leased:', 'payload-bytes:', 'warmup:', 'explain', 'help']); + + if (isset($options['help'])) { + echo "Usage: php tests/Benchmarks/QueueOutbox/benchmark.php [options]\n" + . " --case=direct|stage|relay|claims --target=redis|database (relay only)\n" + . " --database-connection=NAME (omit for temporary functional-only SQLite)\n" + . " --redis-connection=default --jobs=10000 --batch=100 --concurrency=1\n" + . " --pool-size=4 --backlog=10000 --leased=N (default: batch * concurrency)\n" + . " --payload-bytes=256 --warmup=100\n" + . " --explain (claims only; executing plans outside timing)\n" + . "Use a disposable database and an idle machine for measurements. See README.md.\n"; + + return; + } + + $case = $options['case'] ?? 'stage'; + $target = $options['target'] ?? 'redis'; + + if (! in_array($case, ['direct', 'stage', 'relay', 'claims'], true) + || ! in_array($target, ['redis', 'database'], true) + || ($target === 'database' && $case !== 'relay') + || (isset($options['explain']) && $case !== 'claims')) { + throw new InvalidArgumentException('Invalid case, target or explain combination; see --help.'); + } + + $settings = [ + 'case' => $case, + 'target' => $target, + 'jobs' => integerOption($options, 'jobs', 10_000), + 'batch' => integerOption($options, 'batch', 100), + 'concurrency' => integerOption($options, 'concurrency', 1), + 'pool_size' => integerOption($options, 'pool-size', 4), + 'backlog' => integerOption($options, 'backlog', 10_000, 4), + 'payload_bytes' => integerOption($options, 'payload-bytes', 256, 0), + 'warmup' => integerOption($options, 'warmup', 100, 0), + 'cache_store' => 'worker-array', + ]; + $settings['leased'] = integerOption($options, 'leased', $settings['batch'] * $settings['concurrency'], 0); + + if (in_array($case, ['relay', 'claims'], true) && $settings['batch'] > 1000) { + throw new InvalidArgumentException('Relay and claim batches must not exceed the worker limit of 1000.'); + } + + Bootstrapper::bootstrap(); + $application = Application::create(options: ['load_environment_variables' => false]); + $config = $application->make('config'); + $identifier = bin2hex(random_bytes(6)); + $queue = 'benchmark-outbox-' . $identifier; + $source = $options['database-connection'] ?? 'benchmark'; + $redis = $options['redis-connection'] ?? 'default'; + $sqlite = null; + $workload = null; + $failure = null; + $report = []; + + try { + if ($case !== 'direct' && ! isset($options['database-connection'])) { + $sqlite = tempnam(sys_get_temp_dir(), 'hypervel-outbox-benchmark-'); + + if ($sqlite === false) { + throw new RuntimeException('Could not create the temporary SQLite database.'); + } + + $config->set('database.connections.benchmark', ['driver' => 'sqlite', 'database' => $sqlite, 'prefix' => '', 'busy_timeout' => 5000]); + } + + if ($case !== 'direct') { + $connection = $config->array("database.connections.{$source}"); + $connection['pool'] = array_replace($connection['pool'] ?? [], [ + 'testing_enabled' => true, 'max_connections' => $settings['pool_size'], + 'heartbeat_interval' => null, 'idle_check_interval' => null, + ]); + $config->set("database.connections.{$source}", $connection); + $config->set('database.default', $source); + $config->set('queue.outbox', ['databases' => [$source], 'table' => 'bench_outbox_' . $identifier, 'retry_after' => 3600]); + } + + $redisConfig = $config->array("database.redis.{$redis}"); + $redisConfig['pool'] = array_replace($redisConfig['pool'] ?? [], ['max_connections' => $settings['pool_size'], 'heartbeat_interval' => null, 'idle_check_interval' => null]); + $config->set("database.redis.{$redis}", $redisConfig); + $config->set('cache.default', $settings['cache_store']); + $config->set('queue.connections.benchmark', $target === 'redis' + ? ['driver' => 'redis', 'connection' => $redis, 'queue' => $queue, 'retry_after' => 3600, 'block_for' => null, 'after_commit' => false] + : ['driver' => 'database', 'connection' => $source, 'table' => 'bench_jobs_' . $identifier, 'queue' => $queue, 'retry_after' => 3600, 'after_commit' => false]); + fwrite(STDERR, "Owned benchmark resources: queue={$queue}, tables=bench_outbox_{$identifier},bench_jobs_{$identifier}\n"); + + $completed = run(function () use ($application, $settings, $source, $queue, $redis, $identifier, $options, &$workload, &$report, &$failure): void { + $queues = $application->make('queue'); + $direct = static function (int $count) use ($queues, $settings, $queue): void { + for ($offset = 0; $offset < $count; $offset += $settings['batch']) { + $jobs = []; + + for ($index = 0; $index < min($settings['batch'], $count - $offset); ++$index) { + $jobs[] = new OutboxBenchmarkJob(str_repeat('x', $settings['payload_bytes'])); + } + + if ($settings['batch'] === 1) { + $queues->connection('benchmark')->push($jobs[0], queue: $queue); + } else { + $queues->connection('benchmark')->bulk($jobs, queue: $queue); + } + } + }; + + try { + if ($settings['case'] !== 'direct') { + $workload = new OutboxWorkload($application, $source, $queue, $identifier, $settings); + inFreshCoroutine(fn () => $workload->setUp()); + } + + $execute = $workload === null ? $direct : $workload->execute(...); + + if ($settings['warmup'] > 0) { + inFreshCoroutine(fn () => $workload?->prepare($settings['warmup'])); + inFreshCoroutine(fn () => $execute($settings['warmup'])); + inFreshCoroutine(function () use ($workload, $queues, $queue, $settings): void { + $workload?->verify($settings['warmup']); + if ($workload === null && $queues->connection('benchmark')->size($queue) !== $settings['warmup']) { + throw new RuntimeException('Direct warmup delivered an unexpected job count.'); + } + $workload?->reset(); + if ($workload === null) { + $queues->connection('benchmark')->clear($queue); + } + }); + } + + inFreshCoroutine(fn () => $workload?->prepare($settings['jobs'])); + $walStart = inFreshCoroutine(fn () => $workload?->walPosition()); + gc_collect_cycles(); + memory_reset_peak_usage(); + $heap = memory_get_usage(); + $gc = gc_status(); + $cpu = getrusage(); + $started = hrtime(true); + $details = inFreshCoroutine(fn () => $execute($settings['jobs'])); + $elapsed = (hrtime(true) - $started) / 1e9; + $cpuAfter = getrusage(); + $heapGrowth = memory_get_usage() - $heap; + $peakHeap = memory_get_peak_usage(); + $gcAfter = gc_status(); + $walBytes = inFreshCoroutine(fn () => $workload?->walBytes($walStart)); + inFreshCoroutine(function () use ($workload, $queues, $queue, $settings): void { + $workload?->verify($settings['jobs']); + if ($workload === null && $queues->connection('benchmark')->size($queue) !== $settings['jobs']) { + throw new RuntimeException('Direct dispatch delivered an unexpected job count.'); + } + }); + + $borrowed = []; + foreach ([DatabasePools::class, RedisPools::class] as $manager) { + foreach ($application->make($manager)->getPools() as $name => $pool) { + $borrowed[$manager . ':' . $name] = $pool->getBorrowedCount(); + } + } + if (array_sum($borrowed) !== 0) { + throw new RuntimeException('The workload retained a borrowed connection after completion.'); + } + + $report = [ + 'timestamp' => gmdate(DATE_ATOM), 'php' => PHP_VERSION, 'swoole' => phpversion('swoole'), + 'opcache_cli' => ini_get('opcache.enable_cli'), 'jit' => ini_get('opcache.jit'), + 'settings' => $settings, 'database_connection' => $settings['case'] === 'direct' ? null : $source, + 'redis_connection' => $redis, 'queue' => $queue, + 'backend' => inFreshCoroutine(fn () => $workload?->backend()), + 'redis' => $settings['target'] === 'redis' && in_array($settings['case'], ['direct', 'relay'], true) + ? inFreshCoroutine(function () use ($application, $redis): array { + $profile = $application->make('config')->array("database.redis.{$redis}"); + $information = $application->make('redis')->connection($redis)->info('server'); + + return [ + 'version' => $information['redis_version'] ?? $information['Server']['redis_version'] ?? null, + 'host' => $profile['host'] ?? null, 'port' => $profile['port'] ?? null, + 'database' => $profile['database'] ?? null, + ]; + }) : null, + 'seconds' => $elapsed, 'jobs_per_second' => $settings['jobs'] / $elapsed, + 'cpu_seconds' => ($cpuAfter['ru_utime.tv_sec'] + $cpuAfter['ru_stime.tv_sec'] - $cpu['ru_utime.tv_sec'] - $cpu['ru_stime.tv_sec']) + + ($cpuAfter['ru_utime.tv_usec'] + $cpuAfter['ru_stime.tv_usec'] - $cpu['ru_utime.tv_usec'] - $cpu['ru_stime.tv_usec']) / 1e6, + 'heap_growth_bytes' => $heapGrowth, 'peak_heap_bytes' => $peakHeap, + 'gc_runs' => $gcAfter['runs'] - $gc['runs'], 'gc_collected' => $gcAfter['collected'] - $gc['collected'], + 'gc_roots' => $gcAfter['roots'], 'borrowed_after_sample' => $borrowed, + 'cluster_workload_wal_bytes' => $walBytes, 'details' => $details, + 'plans' => isset($options['explain']) ? inFreshCoroutine(fn () => $workload->explain()) : null, + ]; + } catch (Throwable $exception) { + $failure = $exception; + } finally { + try { + inFreshCoroutine(function () use ($workload, $queues, $queue, $settings): void { + try { + $workload?->tearDown(); + } finally { + if ($settings['target'] === 'redis' && in_array($settings['case'], ['direct', 'relay'], true)) { + $queues->connection('benchmark')->clear($queue); + if ($queues->connection('benchmark')->size($queue) !== 0) { + throw new RuntimeException('Benchmark queue cleanup failed.'); + } + } + } + }); + } catch (Throwable $cleanupFailure) { + if ($failure !== null) { + fwrite(STDERR, 'Cleanup also failed: ' . $cleanupFailure . PHP_EOL); + } + $failure ??= $cleanupFailure; + } + } + }); + + if ($failure !== null) { + throw $failure; + } + if (! $completed) { + throw new RuntimeException('The benchmark coroutine did not complete.'); + } + + echo json_encode($report, JSON_PRETTY_PRINT | JSON_THROW_ON_ERROR) . PHP_EOL; + } finally { + $application->terminate(); + if (is_string($sqlite)) { + unlink($sqlite); + } + } +} + +try { + main(); +} catch (Throwable $exception) { + fwrite(STDERR, (string) $exception . PHP_EOL); + exit(1); +} diff --git a/tests/Benchmarks/QueueOutbox/results.md b/tests/Benchmarks/QueueOutbox/results.md new file mode 100644 index 0000000000..6d1b89b699 --- /dev/null +++ b/tests/Benchmarks/QueueOutbox/results.md @@ -0,0 +1,108 @@ +# Queue outbox measurements + +Measured on October 23, 2026. The results support keeping the production outbox design unchanged. Ordinary single dispatch was effectively unchanged, bulk dispatch improved in the paired comparison, and the PostgreSQL reservation-column alternative added scan work without a consistent statement-WAL saving. See [README.md](README.md) for the retained harness. + +## Setup and interpretation + +- Baseline: `a67e2c9143078d4feca709869803be576604d9e4`. Measured feature branch: `29a8f847484ea23c5d2141f5746e33b18e0febfe`. Separate worktrees had independent Composer installs using the same lockfile. The comparison covers the complete feature branch, including its framework correctness changes, not only the outbox-selection check. +- Linux x86-64 VM exposing six CPUs, Intel Core i7-7700 at 3.60 GHz; PHP 8.4.26, Swoole 6.3.0RC1, CLI OPcache enabled, JIT disabled. +- Local PostgreSQL 17.11, MySQL 9.5.0, MariaDB 10.11.19 and Redis 8.2.1. PostgreSQL `fsync`, `synchronous_commit` and `full_page_writes` remained enabled. No forced checkpoints, statistics resets, fillfactor or autovacuum changes. +- Job contents: 256 bytes; stored outbox envelope: 1,246 bytes. Relay restart cache: `worker-array`, so these runs exclude external restart-cache round trips. Jobs were published, not executed. +- Each sample ran in a fresh process after warmup. Tables show median throughput with observed minimum–maximum, including outliers. CPU covers only the calling PHP process. Database/Redis server CPU is not included; heap growth is not an allocation count. + +These are local workload measurements, not production capacity promises. Database-backed wall times varied substantially while PHP CPU stayed steadier. Diagnostic PostgreSQL monitoring observed simultaneous `WalSync` and `WALWrite` waits, consistent with storage delays. It did not capture the exact cause of every outlier. Do not infer small throughput differences or choose a schema from wall time alone. + +Every successful sample checked exact completed counts, remaining lease counts where applicable, zero borrowed connections after completion, and removal of its own tables/queue. The failed oversubscribed run is recorded below rather than discarded. Raw reports, diagnostic logs and run scripts are retained outside Git. + +## Ordinary Redis dispatch + +Ten alternating baseline/current pairs per case. Single samples dispatched 20,000 jobs; bulk samples dispatched 100,000 in batches of 1,000. Warmup: 1,000 jobs. + +| Dispatch | Baseline jobs/s, median (range) | Feature jobs/s, median (range) | +| --- | ---: | ---: | +| Single | 5,074 (4,992–5,210) | 5,073 (4,981–5,180) | +| Bulk, 1,000 per call | 40,074 (39,107–41,351) | 43,491 (40,829–44,366) | + +Single throughput changed by −0.01% using the ratio of medians. Bulk's ratio of medians improved by 8.5%; all ten matched pairs improved, with a median paired change of 9.3% and a 1.1–12.8% range. The pooled bulk ranges overlap, so this is a measured paired improvement, not a claim that every future run will be faster. + +Median PHP CPU was 2.113 → 2.129 seconds for single samples and 1.990 → 1.809 seconds for bulk samples. Ending heap growth was 5,216 bytes on both revisions in both cases. The peak heap difference was about 24 KB in both cases. + +## Transactional staging + +Three samples per engine/batch size. Single samples staged 2,000 jobs; bulk samples staged 10,000. Warmup: 100 jobs. Each dispatch call had its own transaction, so bulk amortizes commit cost rather than providing the same workload as single dispatch. + +| Database | Jobs per call | Jobs/s, median (range) | +| --- | ---: | ---: | +| PostgreSQL | 1 | 266 (260–266) | +| PostgreSQL | 1,000 | 10,964 (9,425–11,241) | +| PostgreSQL | 10,000 | 11,312 (11,007–12,124) | +| MySQL | 1 | 86 (82–89) | +| MySQL | 1,000 | 1,046 (378–4,581) | +| MySQL | 10,000 | 7,504 (7,426–7,508) | +| MariaDB | 1 | 283 (276–284) | +| MariaDB | 1,000 | 11,281 (8,455–11,428) | +| MariaDB | 10,000 | 11,237 (9,021–11,657) | + +MySQL's 1,000-job batch samples took 2.18, 26.44 and 9.56 seconds while PHP CPU stayed around 0.4 seconds. The outliers are included; these runs do not establish an engine ranking. Bulk storage and binding-limit chunking completed correctly on all three engines. + +## Real relay worker + +PostgreSQL source, Redis or PostgreSQL database-queue target, four database/Redis pool slots, 30,000 jobs per sample and 1,000 warmup jobs. Three samples per configuration. Claims, publication, settlement and worker lifecycle were timed; initial seeding was not. + +| Target | Claim batch | Concurrency | Jobs/s, median (range) | +| --- | ---: | ---: | ---: | +| Redis | 100 | 1 | 8,328 (7,417–8,801) | +| Redis | 100 | 4 | 8,211 (7,770–8,421) | +| Redis | 100 | 8 | 7,301 (7,080–8,199) | +| Redis | 1,000 | 1 | 16,983 (7,971–17,519) | +| Redis | 1,000 | 4 | 22,387 (14,396–22,700) | +| Redis | 1,000 | 8 | 22,655 (21,515–23,118) | +| Database | 100 | 1 | 5,136 (723–6,395) | +| Database | 100 | 4 | 10,963 (9,450–11,153) | +| Database | 100 | 8 | 7,498 (7,101–11,011) | +| Database | 1,000 | 1 | 10,749 (8,376–10,952) | +| Database | 1,000 | 4 | 11,754 (6,867–14,457) | +| Database | 1,000 | 8 | 13,740 (10,850–14,373) | + +Larger batches helped on this setup. Pool capacity stayed fixed at four slots, so the concurrency-8 results include any checkout contention rather than isolating scaling with matched capacity. Size pools for relay concurrency as the guide describes; tasks release each phase's connection before starting the next. All relay samples drained completely and released their borrowed resources. The database-target, batch-100, concurrency-1 group includes a 41.48-second sample, versus 4.69 and 5.84 seconds; PHP CPU was 0.86–0.94 seconds. + +## Claims under churn + +Claims were followed by accepted settlement and replacement inserts, keeping the mixed backlog populated. Each sample completed 30,000 rows after 5,000 warmup rows, with batch 100, eight clients and eight pool slots. Claim latency includes checkout, materialization and child cleanup; total cycle throughput additionally includes settlement/refill, UUID generation and payload encoding. + +PostgreSQL used five alternating pairs per backlog/lease group. The temporary alternative added an unindexed `reserved_until` column and left indexed `available_at` unchanged, filtering live reservations during selection. Both used the production accepted-delete settlement. The comparison fixture and selector were removed after measurement; their measured source remains available at the revision above. + +| Actual backlog | Seeded live leases | Current cycle jobs/s, median (range) | Reservation cycle jobs/s, median (range) | +| --- | ---: | ---: | ---: | +| 10,000 | 800 | 16,789 (2,830–16,952) | 16,603 (11,843–17,079) | +| 13,800 | 8,000 | 16,582 (1,656–21,312) | 15,331 (14,535–18,607) | +| 100,000 | 800 | 10,732 (4,533–21,376) | 12,120 (5,775–19,676) | +| 100,000 | 8,000 | 10,437 (3,710–20,024) | 10,026 (5,695–18,311) | + +The 13,800-row group requested `--backlog=10000 --leased=8000`; the harness preserves 2,500 delayed and 2,500 retrying rows plus at least 800 eligible rows. Other groups match the requested backlog. The noisy cycle medians do not uniformly favor the current design. + +Executing PostgreSQL plans give the more direct scan comparison. Times include the claim statement's update, not just selection. Buffer hits below are on the eligibility-index scan; WAL is statement-local bytes, not the cluster-wide workload delta. + +| Backlog / live leases | Current statement ms, median (range) | Reservation statement ms, median (range) | Eligibility buffer hits, current / reservation | Statement WAL bytes, current / reservation | +| --- | ---: | ---: | ---: | ---: | +| 10,000 / 800 | 2.086 (1.898–2.197) | 2.505 (2.348–9.599) | 158 / 657 | 170,017 / 173,709 | +| 13,800 / 8,000 | 2.066 (1.805–2.153) | 4.948 (4.646–5.028) | 166 / 2,219 | 173,675 / 172,678 | +| 100,000 / 800 | 1.857 (1.653–2.392) | 2.309 (2.094–2.993) | 158 / 642 | 174,282 / 174,592 | +| 100,000 / 8,000 | 1.732 (1.631–1.873) | 4.581 (4.341–4.691) | 158 / 2,109 | 174,053 / 174,674 | + +The current query used bounded index scans without filtering live leases. The alternative filtered all 800 or 8,000 seeded live leases per statement, increased scan work and showed no consistent statement-WAL saving. An unindexed deadline alone does not guarantee HOT updates; free page space also matters. Keep the existing indexed eligibility/lease deadline and avoid another schema column. + +MySQL and MariaDB each had three samples at backlog 100,000 and 800 live leases: + +| Database | Cycle jobs/s, median (range) | +| --- | ---: | +| MySQL | 1,885 (1,083–2,050) | +| MariaDB | 3,635 (946–7,317) | + +Their executing SELECT plans used the eligibility index and returned 100 rows. Inspected examples took about 0.25 ms on MySQL and 0.17 ms on MariaDB; these SELECT times exclude the update and transaction commit. All 46 matched-capacity samples passed counts and resource cleanup. + +### Oversubscribed pool diagnostic + +Earlier PostgreSQL comparisons used eight clients and four slots. One 100,000-row, 8,000-live-lease sample failed with four checkout timeouts under the unchanged three-second `wait_timeout`. That failed sample has no throughput result. The completed earlier samples remain contention evidence, not the basis for the claim-design conclusion above. + +One trace-assisted repeat and five further monitored repeats passed with zero borrowed connections afterward. Server monitoring observed simultaneous `WalSync`/`WALWrite` waits up to about 1.44 seconds per statement. A settlement/refill phase includes both the settlement commit and the replacement insert's commit, so these waits are consistent with exceeding checkout time under pool contention. They do not prove the original failure's exact cause. No connection-lifetime defect was confirmed, and no timeout increase, retry workaround or production change was introduced. diff --git a/tests/Broadcasting/BroadcastEventTest.php b/tests/Broadcasting/BroadcastEventTest.php index f5617e3d9e..1546ff9598 100644 --- a/tests/Broadcasting/BroadcastEventTest.php +++ b/tests/Broadcasting/BroadcastEventTest.php @@ -7,6 +7,7 @@ use Exception; use Hypervel\Broadcasting\BroadcastEvent; use Hypervel\Broadcasting\InteractsWithBroadcasting; +use Hypervel\Bus\Queueable; use Hypervel\Contracts\Broadcasting\Broadcaster; use Hypervel\Contracts\Broadcasting\Factory as BroadcastingFactory; use Hypervel\Contracts\Broadcasting\ShouldBroadcast; @@ -56,6 +57,27 @@ public function testManualParameterSpecification(): void (new BroadcastEvent($event))->handle($manager); } + public function testUninitializedPublicPropertiesAreOmittedFromThePayload(): void + { + $event = new class extends TestBroadcastEvent { + use Queueable; + + public string $uninitialized; + + public ?string $optional = null; + }; + + $payload = get_object_vars($event); + $payload['collection'] = ['foo' => 'bar']; + + $broadcaster = m::mock(Broadcaster::class); + $broadcaster->expects('broadcast')->with(['test-channel'], $event::class, $payload); + $manager = m::mock(BroadcastingFactory::class); + $manager->expects('connection')->with(null)->andReturn($broadcaster); + + (new BroadcastEvent($event))->handle($manager); + } + public function testSpecificBroadcasterGiven(): void { $broadcaster = m::mock(Broadcaster::class); diff --git a/tests/Bus/BusBatchTest.php b/tests/Bus/BusBatchTest.php index a1d0d27d57..63a09b3356 100644 --- a/tests/Bus/BusBatchTest.php +++ b/tests/Bus/BusBatchTest.php @@ -180,6 +180,57 @@ public function testJobsCanBeAddedToPendingBatch(): void $this->assertCount(2, $batch->jobs); } + public function testStoredOutboxDefaultsApplyToNewJobsWithoutReplacingExplicitChoices(): void + { + Queue::fake(); + $database = $this->app->make(DatabaseBatchRepository::class)->getConnection()->getName(); + + $defaultJob = new class { + use Batchable; + }; + $directJob = (new TestBatchJob)->onOutboxDatabase('previous')->withoutOutbox(); + $aliasOwnerJob = (new TestBatchJob)->onOutboxDatabase($database . '::write'); + $propertyOptOutJob = (new TestBatchJob)->onOutboxDatabase('unused'); + $propertyOptOutJob->viaOutbox = false; + $nullJob = new TestBatchJob; + $nullJob->viaOutbox = null; + $nullJob->outboxDatabase = null; + + $batch = Bus::batch([$defaultJob, $directJob, $aliasOwnerJob, $propertyOptOutJob, $nullJob]) + ->onOutboxDatabase($database) + ->dispatch(); + + $this->assertTrue($defaultJob->viaOutbox); + $this->assertSame($database, $defaultJob->outboxDatabase); + $this->assertFalse($directJob->viaOutbox); + $this->assertNull($directJob->outboxDatabase); + $this->assertSame($database . '::write', $aliasOwnerJob->outboxDatabase); + $this->assertFalse($propertyOptOutJob->viaOutbox); + $this->assertSame('unused', $propertyOptOutJob->outboxDatabase); + $this->assertNull($nullJob->viaOutbox); + $this->assertNull($nullJob->outboxDatabase); + + $added = new TestBatchJob; + $batch->fresh()->add([$added]); + + $this->assertTrue($added->viaOutbox); + $this->assertSame($database, $added->outboxDatabase); + $this->assertSame(6, $batch->fresh()->totalJobs); + } + + public function testBatchOutboxDefaultsDoNotPropagateToLaterChainJobs(): void + { + Queue::fake(); + + $first = new TestBatchJob; + $second = new TestBatchJob; + + Bus::batch([[$first, $second]])->viaOutbox()->dispatch(); + + $this->assertTrue($first->viaOutbox); + $this->assertArrayNotHasKey('viaOutbox', get_object_vars(unserialize($first->chained[0]))); + } + public function testJobsCanBeAddedToThePendingBatchFromIterable(): void { $batch = new PendingBatch($this->app, collect()); diff --git a/tests/Bus/DispatchLockContextTest.php b/tests/Bus/DispatchLockContextTest.php index 96ae1ceab0..778ef531bf 100644 --- a/tests/Bus/DispatchLockContextTest.php +++ b/tests/Bus/DispatchLockContextTest.php @@ -174,6 +174,7 @@ public function testMetadataSurvivesUntilAfterCommitPayloadCreation(string $queu Container::setInstance($container); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(fn (callable $callback) => $callback()); diff --git a/tests/Bus/DispatchLockReleaseContextTest.php b/tests/Bus/DispatchLockReleaseContextTest.php new file mode 100644 index 0000000000..c69a275a8e --- /dev/null +++ b/tests/Bus/DispatchLockReleaseContextTest.php @@ -0,0 +1,248 @@ +registerContextCapture(); + $order = []; + DispatchLockContext::captureReleaseContextUsing(static function () use (&$order): Closure { + return static function (Closure $release) use (&$order): void { + $order[] = CoroutineContext::get(ReleaseContextCacheStore::CONTEXT_KEY); + $release(); + }; + }); + $cache = new Repository(new ReleaseContextCacheStore); + $lock = $kind === 'unique' ? new UniqueLock($cache) : new DebounceLock($cache); + $job = new ReleaseContextJob; + + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, 'first:'); + $lock->acquireForDispatch($job); + $key = $lock::getKey($job); + $captured = DispatchLockContext::snapshot($job); + + if ($snapshot) { + DispatchLockContext::accept($job); + } + + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, 'second:'); + $other = new ReleaseContextJob; + $lock->acquireForDispatch($other); + + if ($snapshot) { + DispatchLockContext::releaseSnapshot($captured); + } else { + DispatchLockContext::release($job); + } + + $this->assertSame(['first:'], $order); + $this->assertSame('second:', CoroutineContext::get(ReleaseContextCacheStore::CONTEXT_KEY)); + $this->assertTrue($kind === 'unique' ? $cache->lock($key)->isLocked() : $cache->has($key)); + + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, 'first:'); + $this->assertFalse($kind === 'unique' ? $cache->lock($key)->isLocked() : $cache->has($key)); + DispatchLockContext::release($other); + $this->assertSame('first:', CoroutineContext::get(ReleaseContextCacheStore::CONTEXT_KEY)); + } + + /** + * Provide each lock's live-record and accepted-snapshot release paths. + */ + public static function releasePaths(): array + { + return [ + 'unique live record' => ['unique', false], + 'unique accepted snapshot' => ['unique', true], + 'debounce live record' => ['debounce', false], + 'debounce accepted snapshot' => ['debounce', true], + ]; + } + + public function testCaptureFailureLeavesNeitherLockNorTokenAcquired(): void + { + $exception = new RuntimeException('Cannot capture context.'); + DispatchLockContext::captureReleaseContextUsing(static fn () => throw $exception); + $cache = new Repository(new WorkerArrayStore); + $job = new ReleaseContextJob; + + foreach ([new UniqueLock($cache), new DebounceLock($cache)] as $lock) { + try { + $lock->acquireForDispatch($job); + $this->fail('Context capture should have failed before acquisition.'); + } catch (RuntimeException $caught) { + $this->assertSame($exception, $caught); + } + } + + $this->assertFalse($cache->lock(UniqueLock::getKey($job))->isLocked()); + $this->assertFalse($cache->has(DebounceLock::getKey($job))); + $this->assertFalse(DispatchLockContext::has($job)); + } + + public function testReleaseFailureRestoresTheCallingContext(): void + { + $this->registerContextCapture(); + $exception = new RuntimeException('Cleanup failed.'); + DispatchLockContext::captureReleaseContextUsing(static fn () => static fn () => throw $exception); + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, 'first:'); + $job = new ReleaseContextJob; + (new UniqueLock(new Repository(new ReleaseContextCacheStore)))->acquireForDispatch($job); + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, 'second:'); + + try { + DispatchLockContext::release($job); + $this->fail('The release wrapper should have failed.'); + } catch (RuntimeException $caught) { + $this->assertSame($exception, $caught); + } + + $this->assertSame('second:', CoroutineContext::get(ReleaseContextCacheStore::CONTEXT_KEY)); + $this->assertFalse(DispatchLockContext::has($job)); + } + + public function testClosingTimerReleasesTheLockOutsideItsAcquiringCoroutine(): void + { + $this->registerContextCapture(); + $released = new Channel(1); + DispatchLockContext::captureReleaseContextUsing(static function () use ($released): Closure { + return static function (Closure $release) use ($released): void { + $release(); + $released->push(true); + }; + }); + + $container = new Container; + Container::setInstance($container); + $container->instance(Dispatcher::class, new EventsDispatcher($container)); + $timer = new Timer; + $queue = new BackgroundQueue(timer: $timer); + $queue->setContainer($container); + $queue->setConnectionName('background'); + $cache = new Repository(new ReleaseContextCacheStore); + $job = new ReleaseContextJob; + + try { + (new Waiter)->wait(function () use ($queue, $cache, $job): void { + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, 'first:'); + $this->assertTrue((new UniqueLock($cache))->acquireForDispatch($job)); + $queue->later(60, $job); + $this->assertFalse(DispatchLockContext::has($job)); + }); + + CoordinatorManager::until(Constants::WORKER_EXIT)->resume(); + $this->assertTrue($released->pop(2)); + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, 'first:'); + $this->assertFalse($cache->lock(UniqueLock::getKey($job))->isLocked()); + } finally { + $timer->clearAll(); + $released->close(); + } + } + + public function testNullCaptureAndClearingRegistrationsDoNotWrapRelease(): void + { + DispatchLockContext::captureReleaseContextUsing(static fn () => null); + $this->assertNull(DispatchLockContext::captureReleaseContext()); + DispatchLockContext::captureReleaseContextUsing(static fn () => throw new RuntimeException); + DispatchLockContext::captureReleaseContextUsing(null); + $this->assertNull(DispatchLockContext::captureReleaseContext()); + } + + /** + * Capture the cache prefix without retaining an entire coroutine context. + */ + protected function registerContextCapture(): void + { + DispatchLockContext::captureReleaseContextUsing(static function (): Closure { + $prefix = CoroutineContext::get(ReleaseContextCacheStore::CONTEXT_KEY); + + return static function (Closure $release) use ($prefix): void { + $previous = CoroutineContext::get(ReleaseContextCacheStore::CONTEXT_KEY); + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, $prefix); + + try { + $release(); + } finally { + CoroutineContext::set(ReleaseContextCacheStore::CONTEXT_KEY, $previous); + } + }; + }); + } +} + +class ReleaseContextJob implements ShouldBeUnique +{ + public int $debounceFor = 10; +} + +class ReleaseContextCacheStore extends WorkerArrayStore +{ + public const string CONTEXT_KEY = '__tests.dispatch_lock_prefix'; + + /** + * Resolve the namespace from the current execution. + */ + public function getPrefix(): string + { + return CoroutineContext::get(self::CONTEXT_KEY, ''); + } + + /** + * Create a lock in the current namespace. + */ + public function lock(string $name, int $seconds = 0, ?string $owner = null): ArrayLock + { + return parent::lock($this->getPrefix() . $name, $seconds, $owner); + } + + /** + * Read a value from the current namespace. + */ + public function get(string $key): mixed + { + return parent::get($this->getPrefix() . $key); + } + + /** + * Store a value in the current namespace. + */ + public function put(string $key, mixed $value, int $seconds): bool + { + return parent::put($this->getPrefix() . $key, $value, $seconds); + } + + /** + * Remove a value from the current namespace. + */ + public function forget(string $key): bool + { + return parent::forget($this->getPrefix() . $key); + } +} diff --git a/tests/Bus/OutboxOptionsTest.php b/tests/Bus/OutboxOptionsTest.php new file mode 100644 index 0000000000..637308076b --- /dev/null +++ b/tests/Bus/OutboxOptionsTest.php @@ -0,0 +1,107 @@ +assertStringNotContainsString('viaOutbox', serialize($job)); + $this->assertStringNotContainsString('outboxDatabase', serialize($job)); + + $this->assertSame($job, $job->onOutboxDatabase(OutboxDatabase::Billing)); + $this->assertTrue($job->viaOutbox); + $this->assertSame('billing', $job->outboxDatabase); + + $restored = unserialize(serialize($job)); + $this->assertTrue($restored->viaOutbox); + $this->assertSame('billing', $restored->outboxDatabase); + + $this->assertSame($job, $job->withoutOutbox()); + $this->assertFalse($job->viaOutbox); + $this->assertNull($job->outboxDatabase); + $this->assertSame($job, $job->viaOutbox()); + $this->assertTrue($job->viaOutbox); + $this->assertNull($job->outboxDatabase); + } + } + + public function testPendingDispatchForwardsOutboxOptions(): void + { + $job = new OutboxQueueableJob; + $dispatcher = m::mock(Dispatcher::class); + Container::getInstance()->instance(Dispatcher::class, $dispatcher); + $dispatcher->expects('dispatch')->with($job)->andReturnNull(); + + $pending = new PendingDispatch($job); + $this->assertSame($pending, $pending->onOutboxDatabase('billing')); + $this->assertSame('billing', $job->outboxDatabase); + $this->assertSame($pending, $pending->withoutOutbox()); + $this->assertFalse($job->viaOutbox); + $this->assertNull($job->outboxDatabase); + $this->assertSame($pending, $pending->viaOutbox()); + $this->assertTrue($job->viaOutbox); + + unset($pending); + } + + public function testPendingBatchStoresOnlyOutboxDefaults(): void + { + $batch = new PendingBatch(Container::getInstance(), new Collection); + $this->assertSame($batch, $batch->onOutboxDatabase(OutboxDatabase::Billing)); + $this->assertSame(['outboxDatabase' => 'billing', 'viaOutbox' => true], $batch->options); + $this->assertSame($batch, $batch->withoutOutbox()); + $this->assertSame(['viaOutbox' => false], $batch->options); + $this->assertSame($batch, $batch->viaOutbox()); + $this->assertSame(['viaOutbox' => true], $batch->options); + } + + public function testEmptyDatabaseIsRejectedWithoutChangingSelection(): void + { + foreach ([new OutboxQueueableJob, new PendingBatch(Container::getInstance(), new Collection)] as $target) { + try { + $target->onOutboxDatabase(''); + $this->fail('An empty owner must not select outbox delivery.'); + } catch (InvalidArgumentException $exception) { + $this->assertSame('The outbox database connection must not be empty.', $exception->getMessage()); + } + + $this->assertFalse(isset($target->viaOutbox)); + } + } +} + +class OutboxQueueableJob +{ + use Queueable; +} + +class OutboxBatchableJob +{ + use Batchable; +} + +class OutboxCombinedJob +{ + use Batchable; + use Queueable; +} + +enum OutboxDatabase: string +{ + case Billing = 'billing'; +} diff --git a/tests/Cache/CacheManagerTest.php b/tests/Cache/CacheManagerTest.php index 819814af36..d256c8ca4c 100644 --- a/tests/Cache/CacheManagerTest.php +++ b/tests/Cache/CacheManagerTest.php @@ -146,6 +146,24 @@ public function testItCanBuildRepositories(): void $this->assertInstanceOf(NullStore::class, $nullCache->getStore()); } + public function testCustomRepositoriesReceiveTheirConfiguredNameWithoutReplacingAnExplicitName(): void + { + $manager = new CacheManager($this->getApp([ + 'cache' => ['stores' => [ + 'custom' => ['driver' => 'custom'], + 'alias' => ['driver' => 'named'], + ]], + ])); + $manager->extend('custom', fn (): Repository => $manager->repository(new WorkerArrayStore)); + $named = new Repository(new WorkerArrayStore, ['store' => 'original']); + $manager->extend('named', static fn (): Repository => $named); + + $this->assertSame('custom', $manager->store('custom')->getName()); + $this->assertSame($named, $manager->store('alias')); + $this->assertSame('original', $named->getName()); + $this->assertNull($manager->repository(new WorkerArrayStore)->getName()); + } + public function testFailoverDriverPassesItsConfiguredNameToFailoverEvents(): void { $app = $this->getApp([ @@ -744,6 +762,31 @@ public function testForgetDriverForgetsTheCallingCoroutinesMemoizedRepository(): $this->assertNotSame($firstMemoizedStore->getInnerStore(), $secondMemoizedStore->getInnerStore()); } + public function testMemoizedRepositoriesPreserveStoreIdentityAndSuppressDuplicateEvents(): void + { + $app = $this->getApp(['cache' => ['stores' => ['named' => ['driver' => 'worker-array']]]]); + $events = new Event($app); + $app->instance(Dispatcher::class, $events); + $manager = new CacheManager($app); + $eventsSeen = []; + $events->listen('Hypervel\Cache\Events\*', static function (string $event, array $payload) use (&$eventsSeen): void { + $eventsSeen[] = $payload[0]->storeName; + }); + $memo = $manager->memo('named'); + + $this->assertSame('named', $memo->getName()); + $this->assertSame('named', $memo->tags('orders')->getName()); + $this->assertNull($memo->getEventDispatcher()); + $memo->put('key', 'value', 60); + $memo->get('key'); + $afterFirstRead = count($eventsSeen); + $memo->get('key'); + + $this->assertNotEmpty($eventsSeen); + $this->assertSame(['named'], array_values(array_unique($eventsSeen))); + $this->assertCount($afterFirstRead, $eventsSeen); + } + // REMOVED: CacheApcStoreTest, CacheDynamoDbStoreTest, CacheMemcachedConnectorTest, // CacheMemcachedStoreTest and their integration tests; these drivers are unsupported. diff --git a/tests/Database/DatabaseTransactionIdentityTest.php b/tests/Database/DatabaseTransactionIdentityTest.php new file mode 100644 index 0000000000..938ad14b44 --- /dev/null +++ b/tests/Database/DatabaseTransactionIdentityTest.php @@ -0,0 +1,113 @@ +connections(); + $callbacks = []; + + $base->beginTransaction(); + $writer->beginTransaction(); + $base->afterCommit(function () use (&$callbacks): void { + $callbacks[] = 'base committed'; + }); + $writer->afterRollBack(function () use (&$callbacks): void { + $callbacks[] = 'writer rolled back'; + }); + + $writer->transaction(function () use ($writer, &$callbacks): void { + $writer->afterCommit(function () use (&$callbacks): void { + $callbacks[] = 'writer committed'; + }); + }); + + $base->commit(); + + $this->assertSame(['base committed'], $callbacks); + $this->assertTrue($writer->hasApplicableTransaction()); + $this->assertFalse($base->hasApplicableTransaction()); + + $writer->rollBack(); + + $this->assertSame(['base committed', 'writer rolled back'], $callbacks); + $this->assertTrue($writer->getTransactionManager()->getPendingTransactions()->isEmpty()); + $this->assertTrue($writer->getTransactionManager()->getCommittedTransactions()->isEmpty()); + } + + public function testRollingBackASavepointOnlyReleasesItsOwnersCallbacks(): void + { + [$base, $writer] = $this->connections(); + $callbacks = []; + + $base->beginTransaction(); + $writer->beginTransaction(); + $writer->beginTransaction(); + $writer->afterRollBack(function () use (&$callbacks): void { + $callbacks[] = 'savepoint rolled back'; + }); + $base->afterCommit(function () use (&$callbacks): void { + $callbacks[] = 'base committed'; + }); + $writer->rollBack(); + + $this->assertSame(['savepoint rolled back'], $callbacks); + + $writer->afterCommit(function () use (&$callbacks): void { + $callbacks[] = 'writer committed'; + }); + $writer->commit(); + $this->assertSame(['savepoint rolled back', 'writer committed'], $callbacks); + + $base->commit(); + $this->assertSame(['savepoint rolled back', 'writer committed', 'base committed'], $callbacks); + } + + public function testDisconnectOnlyResetsItsOwnersTransactionRecords(): void + { + [$base, $writer] = $this->connections(); + $callbacks = []; + + $base->beginTransaction(); + $writer->beginTransaction(); + $base->afterRollBack(function () use (&$callbacks): void { + $callbacks[] = 'base rolled back'; + }); + $writer->afterRollBack(function () use (&$callbacks): void { + $callbacks[] = 'writer rolled back'; + }); + + $writer->disconnect(); + $this->assertSame(['writer rolled back'], $callbacks); + $this->assertTrue($base->hasApplicableTransaction()); + + $base->rollBack(); + $this->assertSame(['writer rolled back', 'base rolled back'], $callbacks); + } + + /** + * Create independent sessions for the same configured connection name. + * + * @return array{SQLiteConnection, SQLiteConnection} + */ + protected function connections(): array + { + $manager = new DatabaseTransactionsManager; + $base = new SQLiteConnection(new PDO('sqlite::memory:'), config: ['name' => 'testing']); + $writer = new SQLiteConnection(new PDO('sqlite::memory:'), config: ['name' => 'testing']); + $writer->setReadWriteType('write'); + $base->setTransactionManager($manager); + $writer->setTransactionManager($manager); + + return [$base, $writer]; + } +} diff --git a/tests/Foundation/Testing/DatabaseTransactionsManagerTest.php b/tests/Foundation/Testing/DatabaseTransactionsManagerTest.php index 5fbfdd004b..e008e17ed2 100644 --- a/tests/Foundation/Testing/DatabaseTransactionsManagerTest.php +++ b/tests/Foundation/Testing/DatabaseTransactionsManagerTest.php @@ -138,6 +138,38 @@ public function testPooledChildTransactionsDoNotCountAsTheirParentsWrapper(): vo $this->assertFalse($manager->hasApplicableTransaction('foo')); } + public function testRoleWrappersExcludeOnlyTheirResolvedOwnersTransaction(): void + { + $manager = new DatabaseTransactionsManager(['foo::write']); + $database = m::mock(DatabaseManager::class); + $writer = new SQLiteConnection(new PDO('sqlite::memory:'), config: ['name' => 'foo']); + $writer->setReadWriteType('write'); + $database->shouldReceive('connection')->once()->with('foo::write')->andReturn($writer); + $manager->beginTransactions($database); + + $this->assertFalse($writer->hasApplicableTransaction()); + + $base = new SQLiteConnection(new PDO('sqlite::memory:'), config: ['name' => 'foo']); + $base->setTransactionManager($manager); + $base->beginTransaction(); + $this->assertTrue($base->hasApplicableTransaction()); + + $called = false; + $writer->transaction(function () use ($writer, &$called): void { + $writer->afterCommit(function () use (&$called): void { + $called = true; + }); + }); + + $this->assertTrue($called); + $this->assertTrue($base->hasApplicableTransaction()); + $this->assertFalse($writer->hasApplicableTransaction()); + + $base->rollBack(); + $writer->rollBack(); + $this->assertTrue($manager->getPendingTransactions()->isEmpty()); + } + /** * Begin the real wrappers while keeping these manager tests independent of an application. */ diff --git a/tests/Foundation/Testing/RefreshDatabaseTest.php b/tests/Foundation/Testing/RefreshDatabaseTest.php index d091677b87..571cb06bed 100644 --- a/tests/Foundation/Testing/RefreshDatabaseTest.php +++ b/tests/Foundation/Testing/RefreshDatabaseTest.php @@ -251,7 +251,7 @@ public function testRefreshDatabaseDoesNotPublishMigratedWhenTheAfterHookFails() $eventDispatcher = m::mock(Dispatcher::class); $connection = m::mock(PdoConnection::class); $connection->expects('getPdo')->andReturn($pdo); - $connection->expects('getName')->andReturn('default'); + $connection->expects('getNameWithReadWriteType')->andReturn('default'); $connection->expects('setTransactionManager'); $connection->expects('getEventDispatcher')->twice()->andReturn($eventDispatcher); $connection->expects('unsetEventDispatcher')->twice(); @@ -661,7 +661,7 @@ public function testRollbackRestoresTheDispatcherAndKeepsMigrationBookkeepingWhe protected function getMockedDatabase(): DatabaseManager { $connection = m::mock(ConnectionInterface::class); - $connection->expects('getName')->andReturn('default'); + $connection->expects('getNameWithReadWriteType')->andReturn('default'); $eventDispatcher = m::mock(Dispatcher::class); $connection->expects('getEventDispatcher') ->twice() diff --git a/tests/Foundation/Testing/RequestContextSynchronizerTest.php b/tests/Foundation/Testing/RequestContextSynchronizerTest.php index df54dd6aa6..6717bdf7b3 100644 --- a/tests/Foundation/Testing/RequestContextSynchronizerTest.php +++ b/tests/Foundation/Testing/RequestContextSynchronizerTest.php @@ -7,11 +7,45 @@ use ArrayObject; use Hypervel\Context\CoroutineContext; use Hypervel\Coroutine\Waiter; +use Hypervel\Database\DatabaseTransactionRecord; +use Hypervel\Database\DatabaseTransactionState; +use Hypervel\Database\SQLiteConnection; use Hypervel\Foundation\Testing\RequestContextSynchronizer; use Hypervel\Tests\TestCase; +use PDO; class RequestContextSynchronizerTest extends TestCase { + public function testTransactionSyncOnlyReplacesTheRetainedRoleOwnersRecords(): void + { + $base = new DatabaseTransactionRecord('testing', 1); + $writer = new DatabaseTransactionRecord('testing::write', 1); + $savepoint = new DatabaseTransactionRecord('testing::write', 2, $writer); + $source = new DatabaseTransactionState; + $source->pending->push($writer); + $source->committed->push($savepoint); + $source->current['testing::write'] = $writer; + + $target = new DatabaseTransactionState; + $target->pending->push($base, $writer, $savepoint); + $target->current = ['testing' => $base, 'testing::write' => $savepoint]; + + $connection = new SQLiteConnection(new PDO('sqlite::memory:'), config: ['name' => 'testing']); + $connection->setReadWriteType('write'); + $synchronizer = new RequestContextSynchronizer; + $synchronizer->syncTransactionState($source, $target, ['testing::write' => $connection]); + + $this->assertSame([$base, $writer], $target->pending->all()); + $this->assertSame([$savepoint], $target->committed->all()); + $this->assertSame(['testing' => $base, 'testing::write' => $writer], $target->current); + + $synchronizer->syncTransactionState(new DatabaseTransactionState, $target, ['testing::write' => $connection]); + + $this->assertSame([$base], $target->pending->all()); + $this->assertTrue($target->committed->isEmpty()); + $this->assertSame(['testing' => $base], $target->current); + } + public function testSyncContextKeysToParentCopiesPresentKeysAndForgetsMissingKeys(): void { CoroutineContext::set('foundation.testing.present', 'old'); diff --git a/tests/Horizon/Unit/RedisQueueTest.php b/tests/Horizon/Unit/RedisQueueTest.php index ac47cfddf2..1ec4d2d1a8 100644 --- a/tests/Horizon/Unit/RedisQueueTest.php +++ b/tests/Horizon/Unit/RedisQueueTest.php @@ -4,13 +4,22 @@ namespace Hypervel\Tests\Horizon\Unit; +use Hypervel\Bus\DispatchLockContext; +use Hypervel\Bus\UniqueLock; +use Hypervel\Cache\Repository; +use Hypervel\Cache\WorkerArrayStore; +use Hypervel\Config\Repository as ConfigRepository; use Hypervel\Container\Container; use Hypervel\Contracts\Events\Dispatcher as DispatcherContract; use Hypervel\Contracts\Queue\Job; use Hypervel\Contracts\Redis\Factory; use Hypervel\Events\Dispatcher; +use Hypervel\Horizon\Events\JobPushed; use Hypervel\Horizon\Events\JobsMigrated; use Hypervel\Horizon\RedisQueue; +use Hypervel\Queue\Events\JobQueued; +use Hypervel\Queue\Events\JobQueueing; +use Hypervel\Queue\Events\JobQueueingFailed; use Hypervel\Queue\LuaScripts; use Hypervel\Redis\RedisProxy; use Hypervel\Tests\Horizon\UnitTestCase; @@ -18,11 +27,113 @@ use Mockery as m; use PHPUnit\Framework\Attributes\TestWith; use RedisException; +use RuntimeException; +use stdClass; +use Swoole\Coroutine\CanceledException; use function Hypervel\Coroutine\parallel; class RedisQueueTest extends UnitTestCase { + #[TestWith(['push'])] + #[TestWith(['later'])] + #[TestWith(['bulk'])] + #[TestWith(['failed-bulk'])] + public function testLifecyclePayloadCorrelationIsIndependentOfHorizonMetadata(string $method): void + { + $stored = []; + $failure = $method === 'failed-bulk' ? new RuntimeException('Redis unavailable.') : null; + [$queue, $events] = $this->publishingQueue($stored, $failure); + $queueing = $terminal = []; + $events->listen(JobQueueing::class, static function (JobQueueing $event) use (&$queueing): void { + $queueing[] = $event->payload; + }); + $events->listen([JobQueued::class, JobQueueingFailed::class], static function (JobQueued|JobQueueingFailed $event) use (&$terminal): void { + $terminal[] = $event->payload; + }); + + try { + match ($method) { + 'push' => $queue->push('first'), + 'later' => $queue->later(30, 'first'), + default => $queue->bulk(['first', 'second']), + }; + $this->assertNull($failure); + } catch (RuntimeException $caught) { + $this->assertSame($failure, $caught); + } + + $this->assertCount(str_contains($method, 'bulk') ? 2 : 1, $queueing); + $this->assertSame($queueing, $terminal); + + if ($failure === null) { + foreach ($stored as $index => $payload) { + $decoded = json_decode($payload, true); + $this->assertArrayHasKey('pushedAt', $decoded); + $this->assertSame('job', $decoded['type']); + $this->assertSame(json_decode($queueing[$index], true)['uuid'], $decoded['uuid']); + $this->assertArrayNotHasKey('pushedAt', json_decode($terminal[$index], true)); + } + } + } + + #[TestWith(['push'])] + #[TestWith(['later'])] + #[TestWith(['bulk'])] + public function testPostPushFailurePreservesAcceptanceAndNotifiesEveryMember(string $method): void + { + $stored = []; + [$queue, $events] = $this->publishingQueue($stored); + $exception = new RuntimeException('Horizon monitoring failed.'); + $events->listen(JobPushed::class, static fn () => throw $exception); + $queued = $failed = []; + $events->listen(JobQueued::class, static function (JobQueued $event) use (&$queued): void { + $queued[] = $event->job; + }); + $events->listen(JobQueueingFailed::class, static function (JobQueueingFailed $event) use (&$failed): void { + $failed[] = $event; + }); + $cache = new Repository(new WorkerArrayStore); + $job = new stdClass; + $this->assertTrue((new UniqueLock($cache))->acquireForDispatch($job)); + $jobs = $method === 'bulk' ? [$job, new stdClass] : [$job]; + + try { + match ($method) { + 'push' => $queue->push($job), + 'later' => $queue->later(30, $job), + 'bulk' => $queue->bulk($jobs), + }; + $this->fail('The monitoring failure should be rethrown.'); + } catch (RuntimeException $caught) { + $this->assertSame($exception, $caught); + } + + $this->assertCount(count($jobs), $stored); + $this->assertSame($jobs, $queued); + $this->assertSame([], $failed); + $this->assertFalse(DispatchLockContext::has($job)); + $this->assertTrue($cache->lock(UniqueLock::getKey($job))->isLocked()); + } + + public function testPostPushCancellationStopsNotificationsWithoutReplacingCancellation(): void + { + $stored = []; + [$queue, $events] = $this->publishingQueue($stored); + $cancellation = new CanceledException; + $events->listen(JobPushed::class, static fn () => throw $cancellation); + $events->listen([JobQueued::class, JobQueueingFailed::class], static fn () => throw new RuntimeException('Must not run after cancellation.')); + + try { + $queue->push('job'); + $this->fail('The post-push hook should cancel.'); + } catch (CanceledException $caught) { + $this->assertSame($cancellation, $caught); + } + + $this->assertCount(1, $stored); + } + public function testReadyNowReadsTheClusterSafeQueueKey(): void { $connection = m::mock(RedisProxy::class); @@ -140,4 +251,45 @@ private function queueRecordingMigrations(RedisProxy $connection, array &$queues return $queue; } + + /** + * Record transport payloads while exercising the actual Horizon enqueue lifecycle. + * + * @return array{RedisQueue, Dispatcher} + */ + private function publishingQueue(array &$stored, ?RuntimeException $failure = null): array + { + $connection = m::mock(RedisProxy::class); + $connection->shouldReceive('isCluster')->andReturnFalse(); + $connection->expects('evalWithShaCache')->andReturnUsing(static function (string $script, array $keys, array $arguments) use (&$stored, $failure): int { + if ($failure !== null) { + throw $failure; + } + + if ($script === LuaScripts::bulk()) { + for ($index = 1; $index < count($arguments); $index += 2) { + $stored[] = $arguments[$index]; + } + + return intdiv(count($arguments), 2); + } + + $stored[] = $arguments[$script === LuaScripts::later() ? 1 : 0]; + + return 1; + }); + $redis = m::mock(Factory::class); + $redis->shouldReceive('connection')->with('default')->andReturn($connection); + $container = new Container; + Container::setInstance($container); + $container->instance('config', new ConfigRepository); + $events = new Dispatcher($container); + $container->instance('events', $events); + $container->instance(DispatcherContract::class, $events); + $queue = new RedisQueue($redis, 'default', 'default'); + $queue->setContainer($container); + $queue->setConnectionName('redis'); + + return [$queue, $events]; + } } diff --git a/tests/Horizon/Unit/RedisTagRepositoryTest.php b/tests/Horizon/Unit/RedisTagRepositoryTest.php new file mode 100644 index 0000000000..7ae4019814 --- /dev/null +++ b/tests/Horizon/Unit/RedisTagRepositoryTest.php @@ -0,0 +1,21 @@ +shouldNotReceive('connection'); + + $this->assertSame([], (new RedisTagRepository($redis))->monitored([])); + } +} diff --git a/tests/Integration/Generators/QueueOutboxTableCommandTest.php b/tests/Integration/Generators/QueueOutboxTableCommandTest.php new file mode 100644 index 0000000000..5d7d810826 --- /dev/null +++ b/tests/Integration/Generators/QueueOutboxTableCommandTest.php @@ -0,0 +1,29 @@ + 'outgoing_jobs']); + $this->artisan(OutboxTableCommand::class)->assertExitCode(0); + + $this->assertMigrationFileContains([ + 'use Hypervel\Database\Migrations\Migration;', + 'return new class extends Migration', + 'Schema::create(\'outgoing_jobs\', function (Blueprint $table) {', + '$table->uuid(\'uuid\')->unique();', + '$table->longText(\'payload\');', + '$table->unsignedBigInteger(\'deliver_at\')->nullable();', + '$table->index([\'available_at\', \'id\']);', + 'Schema::dropIfExists(\'outgoing_jobs\');', + ], 'create_outgoing_jobs_table.php'); + + $this->artisan('queue:outbox-table')->assertExitCode(1); + } +} diff --git a/tests/Integration/Horizon/Feature/QueueProcessingTest.php b/tests/Integration/Horizon/Feature/QueueProcessingTest.php index 2d6bb4b2af..5adf562477 100644 --- a/tests/Integration/Horizon/Feature/QueueProcessingTest.php +++ b/tests/Integration/Horizon/Feature/QueueProcessingTest.php @@ -15,7 +15,6 @@ use Hypervel\Horizon\Events\JobsMigrated; use Hypervel\Horizon\Events\RedisEvent; use Hypervel\Horizon\RedisQueue; -use Hypervel\Queue\InvalidPayloadException; use Hypervel\Queue\Jobs\RedisJob; use Hypervel\Queue\Queue as BaseQueue; use Hypervel\Redis\Exceptions\LuaScriptException; @@ -229,28 +228,6 @@ public function testFailedBulkDoesNotRaisePushedEvents(): void $this->assertSame(0, $pushed); } - public function testPayloadPreparationFailureStillConsumesThePreviousJob(): void - { - $queue = new RedisQueueWithExposedLastPushed( - app('redis'), - 'default', - 'default', - ); - $queue->setContainer($this->app)->setConnectionName('redis'); - $queue->rememberLastPushed(new BasicJob); - - try { - $queue->pushRaw('{invalid'); - $this->fail('Expected the invalid payload to be rejected.'); - } catch (InvalidPayloadException) { - } - - $queue->pushRaw('{"id":"raw-after-failure","displayName":"Raw Job"}'); - - $payload = json_decode(Redis::connection('horizon')->hget('raw-after-failure', 'payload'), true); - $this->assertSame([], $payload['tags']); - } - public function testPendingJobsAreStoredWithTheirTags(): void { $id = Queue::push(new BasicJob); @@ -343,17 +320,6 @@ private function getQueueRedisKey(RedisQueue $queue, ?string $name = null): stri } } -class RedisQueueWithExposedLastPushed extends RedisQueue -{ - /** - * Set the job used to prepare the next payload. - */ - public function rememberLastPushed(object|string $job): void - { - $this->setLastPushed($job); - } -} - class AfterCommitHorizonJob extends BasicJob implements ShouldQueueAfterCommit { } diff --git a/tests/Integration/Horizon/Feature/RedisPayloadTest.php b/tests/Integration/Horizon/Feature/RedisPayloadTest.php index 03412879f1..bf68140c92 100644 --- a/tests/Integration/Horizon/Feature/RedisPayloadTest.php +++ b/tests/Integration/Horizon/Feature/RedisPayloadTest.php @@ -65,6 +65,59 @@ public function testPreparationAndPublicationTimestampCanRunSeparately(): void $this->assertIsNumeric($payload->decoded['pushedAt']); } + public function testCombinedPreparationEncodesCustomizedMetadataOnce(): void + { + $payload = new class('{"id":"1"}') extends JobPayload { + public int $writes = 0; + + /** + * Count payload encoding operations. + */ + public function set(array $values): static + { + ++$this->writes; + + return parent::set($values); + } + + /** + * Customize the payload type. + */ + protected function determineType(mixed $job): string + { + return 'custom'; + } + + /** + * Customize the payload tags. + */ + protected function determineTags(mixed $job): array + { + return ['custom-tag']; + } + + /** + * Determine silencing from the customized tags. + */ + protected function shouldBeSilenced(mixed $job, array $tags): bool + { + return $tags === ['custom-tag']; + } + }; + + $payload->prepare(new StdClass); + + $this->assertSame(1, $payload->writes); + $this->assertSame('custom', $payload->decoded['type']); + $this->assertSame(['custom-tag'], $payload->decoded['tags']); + $this->assertTrue($payload->decoded['silenced']); + $this->assertIsNumeric($payload->decoded['pushedAt']); + $this->assertSame($payload->decoded, json_decode($payload->value, true)); + + $this->assertSame($payload, $payload->prepareForQueueing(null)); + $this->assertSame(1, $payload->writes); + } + public function testPreparingClassifiedPayloadWithoutOriginalJobPreservesMetadata(): void { $payload = new JobPayload(json_encode([ diff --git a/tests/Integration/Queue/Database/MariaDb/OutboxClaimTest.php b/tests/Integration/Queue/Database/MariaDb/OutboxClaimTest.php new file mode 100644 index 0000000000..cd631cf9d1 --- /dev/null +++ b/tests/Integration/Queue/Database/MariaDb/OutboxClaimTest.php @@ -0,0 +1,13 @@ +deleteDirectory(static::$databaseDirectory); + $filesystem->ensureDirectoryExists(static::$databaseDirectory); + touch(static::$databaseDirectory . '/database.sqlite'); + } + + /** + * Remove the file-backed database after its applications have shut down. + */ + public static function tearDownAfterClass(): void + { + (new Filesystem)->deleteDirectory(static::$databaseDirectory); + + parent::tearDownAfterClass(); + } + + /** + * Use independent sessions instead of SQLite's shared in-memory connection. + */ + protected function defineEnvironment(Application $app): void + { + parent::defineEnvironment($app); + + $config = $app->make('config'); + $database = $config->string('database.default'); + + if ($config->string("database.connections.{$database}.driver") === 'sqlite') { + $config->set("database.connections.{$database}.database", static::$databaseDirectory . '/database.sqlite'); + } + } +} diff --git a/tests/Integration/Queue/Database/Sqlite/WorkerResourceLifetimeTest.php b/tests/Integration/Queue/Database/Sqlite/WorkerResourceLifetimeTest.php index bb13dfdbb2..7f68ec665e 100644 --- a/tests/Integration/Queue/Database/Sqlite/WorkerResourceLifetimeTest.php +++ b/tests/Integration/Queue/Database/Sqlite/WorkerResourceLifetimeTest.php @@ -9,25 +9,41 @@ use Hypervel\Contracts\Foundation\Application as ApplicationContract; use Hypervel\Contracts\Queue\Factory as QueueManager; use Hypervel\Contracts\Queue\Job as JobContract; +use Hypervel\Contracts\Queue\PublishesOutboxMessages; use Hypervel\Contracts\Queue\Queue; use Hypervel\Coordinator\Timer; use Hypervel\Coroutine\Coroutine; use Hypervel\Coroutine\Waiter; +use Hypervel\Database\Events\QueryExecuted; use Hypervel\Database\Pool\PoolManager; use Hypervel\Engine\Channel; +use Hypervel\Engine\Coroutine as EngineCoroutine; use Hypervel\Filesystem\Filesystem; +use Hypervel\ObjectPool\PoolDefinition; +use Hypervel\ObjectPool\PoolManager as ObjectPoolManager; +use Hypervel\ObjectPool\PoolOptions; use Hypervel\Queue\Events\JobPopping; use Hypervel\Queue\Events\Looping; +use Hypervel\Queue\Events\OutboxBatchDelivered; use Hypervel\Queue\Events\WorkerIdle; use Hypervel\Queue\Events\WorkerPausing; use Hypervel\Queue\Events\WorkerStarting; use Hypervel\Queue\Events\WorkerStopping; +use Hypervel\Queue\Outbox\DatabaseOutboxRepository; +use Hypervel\Queue\Outbox\OutboxBatchResult; +use Hypervel\Queue\Outbox\OutboxRelay; +use Hypervel\Queue\QueuePoolProxy; +use Hypervel\Queue\SyncQueue; use Hypervel\Queue\Worker; use Hypervel\Queue\WorkerOptions; use Hypervel\Support\Facades\DB; +use Hypervel\Support\Str; use Hypervel\Testbench\TestCase; use Hypervel\Testing\ParallelTesting; use Mockery as m; +use PHPUnit\Framework\Attributes\DataProvider; +use Swoole\Coroutine\CanceledException; +use Throwable; class WorkerResourceLifetimeTest extends TestCase { @@ -68,6 +84,7 @@ protected function defineEnvironment(ApplicationContract $app): void $config = $app->make('config'); $config->set('database.default', 'worker_resource'); + $config->set('queue.outbox.databases', ['worker_resource']); $config->set('database.connections.worker_resource', [ 'driver' => 'sqlite', 'database' => $this->databaseDirectory . '/database.sqlite', @@ -80,6 +97,106 @@ protected function defineEnvironment(ApplicationContract $app): void ]); } + #[DataProvider('outboxPublicationOutcomes')] + public function testOutboxPublicationReleasesOneSlotPoolsAcrossBlockingAndCancellation(bool $cancel): void + { + $waiter = new Waiter(-1); + $repository = $this->app->make(DatabaseOutboxRepository::class); + $waiter->wait(function () use ($repository): void { + $migration = require dirname(__DIR__, 5) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php'; + $migration->up(); + $repository->insert(DB::connection(), [[ + 'uuid' => (string) Str::uuid(), 'connection' => 'transport', 'queue' => 'orders', + 'payload' => '{"displayName":"SendInvoice"}', 'options' => null, + 'deliver_at' => null, 'available_at' => 1, 'created_at' => 1, + ]]); + }); + $databasePool = $this->app->make(PoolManager::class)->pool('worker_resource'); + $objectPools = new ObjectPoolManager; + $started = new Channel(1); + $finish = new Channel(1); + $transport = m::mock(SyncQueue::class . ', ' . PublishesOutboxMessages::class)->makePartial(); + $transport->expects('publishOutboxMessages')->andReturnUsing(function (array $messages) use ($started, $finish, $databasePool): OutboxBatchResult { + $this->assertSame(1, $databasePool->getIdleCount()); + $started->push(true); + $this->assertTrue($finish->pop(5)); + + return new OutboxBatchResult(array_column($messages, 'id')); + }); + $proxy = new QueuePoolProxy( + new PoolDefinition('outbox-resource-test', 'queue', 'outbox-resource-test', PoolOptions::fromArray(['max_objects' => 1])), + static fn (): SyncQueue => $transport, + $objectPools, + ); + $queues = m::mock(QueueManager::class); + $queues->expects('connection')->with('transport')->andReturn($proxy); + $events = $this->app->make(EventDispatcher::class); + $settled = false; + $observed = false; + $events->listen(QueryExecuted::class, function (QueryExecuted $event) use ($objectPools, &$settled): void { + if (str_starts_with($event->sql, 'delete from "job_outbox"')) { + $this->assertSame(0, $objectPools->get('outbox-resource-test')->getBorrowedCount()); + $settled = true; + } + }); + $events->listen(OutboxBatchDelivered::class, function () use ($databasePool, &$observed): void { + $this->assertSame(1, $databasePool->getIdleCount()); + DB::selectOne('SELECT 1'); + $observed = true; + }); + $relay = new OutboxRelay($repository, $queues, $events, m::mock(ExceptionHandlerContract::class)); + $messages = $relay->claim('worker_resource', 100); + $failure = null; + $result = null; + $runner = Coroutine::create(static function () use ($relay, $messages, &$failure, &$result): void { + try { + $result = $relay->deliver('worker_resource', $messages); + } catch (Throwable $exception) { + $failure = $exception; + } + }); + + try { + $this->assertTrue($started->pop(2)); + $this->assertSame(1, $objectPools->get('outbox-resource-test')->getBorrowedCount()); + $this->assertSame(1, $databasePool->getIdleCount()); + $this->assertSame(1, $waiter->wait(static fn (): int => (int) DB::selectOne('SELECT 1 AS value')->value)); + + if ($cancel) { + EngineCoroutine::cancelById($runner, throwException: true); + } else { + $finish->push(true); + } + + Coroutine::join([$runner], 2); + $this->assertFalse(Coroutine::exists($runner)); + $this->assertSame(1, $databasePool->getIdleCount()); + $this->assertSame(0, $objectPools->get('outbox-resource-test')->getBorrowedCount()); + $this->assertSame(! $cancel, $settled); + $this->assertSame(! $cancel, $observed); + $this->assertSame($cancel ? 1 : 0, $waiter->wait(static fn (): int => DB::table('job_outbox')->count())); + + if ($cancel) { + $this->assertInstanceOf(CanceledException::class, $failure); + } else { + $this->assertNull($failure); + $this->assertTrue($result); + } + } finally { + $finish->close(); + Coroutine::join([$runner], 5); + $objectPools->purgeAll(); + } + } + + /** + * Cover normal settlement and cancellation while the transport is blocked. + */ + public static function outboxPublicationOutcomes(): array + { + return [[false], [true]]; + } + public function testLifecycleCallbacksReleasePooledConnectionsBeforeTheDaemonAdvances(): void { $connectionName = $this->app->make('config')->string('database.default'); diff --git a/tests/Integration/Queue/Fixtures/OutboxClaimTestCase.php b/tests/Integration/Queue/Fixtures/OutboxClaimTestCase.php new file mode 100644 index 0000000000..7d5aea66e6 --- /dev/null +++ b/tests/Integration/Queue/Fixtures/OutboxClaimTestCase.php @@ -0,0 +1,242 @@ +make('config'); + $database = $config->string('database.default'); + $config->set('queue.outbox.databases', [$database]); + $config->set("database.connections.{$database}.pool.testing_enabled", true); + $config->set("database.connections.{$database}.pool.max_connections", 4); + $config->set("database.connections.{$database}.pool.heartbeat_interval", null); + } + + /** + * Migrate only the shipped operational outbox table. + */ + protected function migrateFreshUsing(): array + { + return [ + '--realpath' => true, + '--path' => dirname(__DIR__, 4) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php', + ]; + } + + public function testClaimBoundsAndStatementCountOnTheLiveEngine(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $now = Date::now()->getTimestamp(); + $repository = $this->app->make(DatabaseOutboxRepository::class); + $connection = DB::connection(); + $database = $connection->getName(); + $rows = [$this->row($now - 10), $this->row($now - 30), $this->row($now - 20), $this->row($now + 60)]; + $repository->insert($connection, $rows); + $skipLocked = $connection->lockForPopping() === 'FOR UPDATE SKIP LOCKED'; + $transactions = 0; + $this->app->make('events')->listen(TransactionBeginning::class, static function () use (&$transactions): void { ++$transactions; }); + $connection->enableQueryLog(); + $connection->flushQueryLog(); + + try { + $messages = $repository->claim($database, 2); + $queries = $connection->getQueryLog(); + } finally { + $connection->disableQueryLog(); + } + + $postgres = $connection->getDriverName() === 'pgsql'; + $this->assertCount($postgres ? 1 : ($skipLocked ? 2 : 3), $queries); + $this->assertSame($skipLocked && ! $postgres ? 1 : 0, $transactions); + $this->assertSame([$rows[1]['uuid'], $rows[2]['uuid']], array_column($messages, 'uuid')); + $this->assertSame([$now - 30, $now - 20], array_column($messages, 'eligibleAt')); + $this->assertSame($rows[1]['payload'], $messages[0]->payload); + $this->assertSame(['transport' => ['custom' => ['group' => 'orders']]], $messages[0]->options); + $this->assertSame(2, DB::table('job_outbox')->whereNotNull('lease_token')->count()); + $this->assertSame([$rows[0]['uuid']], array_column($repository->claim($database, 10), 'uuid')); + $this->assertSame(2, $repository->settle($database, $messages, new OutboxBatchResult(array_column($messages, 'id')))); + } + + public function testStagingUsesTheExactWriteRoleTransactionAndDrainsThroughItsBaseSource(): void + { + $base = DB::connection(); + $database = $base->getName(); + $writer = DB::connection($database . '::write'); + $queue = new RedisQueue(m::mock(Factory::class), 'orders'); + $queue->setContainer($this->app)->setConnectionName('redis'); + $base->beginTransaction(); + $writer->beginTransaction(); + + try { + $queue->push((new OutboxClaimJob)->onOutboxDatabase($database . '::write')); + $this->assertSame(1, $writer->table('job_outbox')->count()); + $this->assertSame(0, $base->table('job_outbox')->count()); + } finally { + $writer->rollBack(); + $base->rollBack(); + } + + $this->assertSame(0, $base->table('job_outbox')->count()); + $uuid = $writer->transaction(fn (): mixed => $queue->push((new OutboxClaimJob)->viaOutbox())); + $messages = $this->app->make(DatabaseOutboxRepository::class)->claim($database, 1); + + $this->assertSame([$uuid], array_column($messages, 'uuid')); + } + + public function testCompetingClaimsAndExpiredOwnershipOnIndependentSessions(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $now = Date::now()->getTimestamp(); + $repository = $this->app->make(DatabaseOutboxRepository::class); + $connection = DB::connection(); + $database = $connection->getName(); + $rows = array_map(fn (): array => $this->row($now), range(1, 4)); + $repository->insert($connection, $rows); + $fallback = $connection->lockForPopping() !== 'FOR UPDATE SKIP LOCKED'; + + // Force both fallback claimants to see the same candidates before either updates them. + if ($fallback) { + $selected = new Channel(1); + $selections = 0; + $this->app->make('events')->listen(QueryExecuted::class, function (QueryExecuted $event) use ($selected, &$selections): void { + if (str_starts_with($event->sql, 'select ') + && str_contains($event->sql, 'available_at') + && ! str_contains($event->sql, 'lease_token')) { + if (++$selections === 1) { + $this->assertTrue($selected->pop(5)); + } else { + $selected->push(true); + } + } + }); + } + + try { + [$first, $second] = parallel([ + fn (): array => $repository->claim($database, 2), + fn (): array => $repository->claim($database, 2), + ]); + } finally { + if ($fallback) { + $this->app->make('events')->forget(QueryExecuted::class); + $selected->close(); + } + } + + $firstIds = array_column($first, 'id'); + $secondIds = array_column($second, 'id'); + $this->assertSame([], array_values(array_intersect($firstIds, $secondIds))); + $this->assertCount($fallback ? 2 : 4, array_merge($firstIds, $secondIds)); + $old = array_merge($first, $second, $repository->claim($database, 10)); + $this->assertCount(4, $old); + $this->assertSame([], $repository->claim($database, 10)); + + Date::setTestNow(Date::now()->addSeconds(300)); + $current = $repository->claim($database, 10); + $this->assertCount(4, $current); + $this->assertSame([2, 2, 2, 2], array_column($current, 'attempts')); + $this->assertNotSame($old[0]->leaseToken, $current[0]->leaseToken); + + foreach ([$first, $second] as $stale) { + if ($stale !== []) { + $this->assertSame(0, $repository->settle($database, $stale, new OutboxBatchResult(array_column($stale, 'id')))); + } + } + + $this->assertSame(4, $repository->settle($database, $current, new OutboxBatchResult(array_column($current, 'id')))); + $this->assertSame(0, DB::table('job_outbox')->count()); + } + + public function testSkipLockedClaimsAdvancePastAnUncommittedOwner(): void + { + $connection = DB::connection(); + + if ($connection->lockForPopping() !== 'FOR UPDATE SKIP LOCKED') { + $this->markTestSkipped('This engine uses the conditional-update claim path.'); + } + + $repository = $this->app->make(DatabaseOutboxRepository::class); + $database = $connection->getName(); + $now = Date::now()->getTimestamp(); + $locked = $this->row($now - 20); + $available = $this->row($now - 10); + $repository->insert($connection, [$locked, $available]); + $ready = new Channel(1); + $release = new Channel(1); + + try { + [, $messages] = parallel([ + function () use ($locked, $ready, $release): void { + DB::transaction(function () use ($locked, $ready, $release): void { + DB::table('job_outbox')->where('uuid', $locked['uuid'])->lockForUpdate()->first(); + $ready->push(true); + $this->assertTrue($release->pop(5)); + }); + }, + function () use ($repository, $database, $ready, $release): array { + try { + $this->assertTrue($ready->pop(5)); + + return $repository->claim($database, 1); + } finally { + $release->push(true); + } + }, + ]); + } finally { + $ready->close(); + $release->close(); + } + + $this->assertSame([$available['uuid']], array_column($messages, 'uuid')); + $this->assertSame([$locked['uuid']], array_column($repository->claim($database, 1), 'uuid')); + } + + /** + * Create opaque stored bytes and exact epoch deadlines for claim tests. + */ + protected function row(int $availableAt): array + { + return [ + 'uuid' => (string) Str::uuid(), 'connection' => 'redis', 'queue' => 'orders', + 'payload' => '{ "displayName": "SendInvoice", "data": {"text": "雪"} }', + 'options' => '{"transport":{"custom":{"group":"orders"}}}', + 'deliver_at' => null, 'attempts' => 0, 'available_at' => $availableAt, + 'lease_token' => null, 'last_error' => null, 'created_at' => Date::now()->getTimestamp(), + ]; + } +} + +class OutboxClaimJob +{ + use Queueable; +} diff --git a/tests/Integration/Queue/QueueConnectionTest.php b/tests/Integration/Queue/QueueConnectionTest.php index 3ddd5c7e89..d046bb0cdf 100644 --- a/tests/Integration/Queue/QueueConnectionTest.php +++ b/tests/Integration/Queue/QueueConnectionTest.php @@ -59,6 +59,7 @@ public function testJobWontGetDispatchedInsideATransaction(): void { $this->app->singleton('db.transactions', function (): DatabaseTransactionsManager { $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('callbackApplicableTransactions') ->andReturn(new Collection([new DatabaseTransactionRecord('sqlite', 1)])); $transactionManager->expects('addCallback')->andReturn(null); @@ -93,6 +94,7 @@ public function testJobWontGetDispatchedInsideATransactionWhenExplicitlyIndicate $this->app->singleton('db.transactions', function (): DatabaseTransactionsManager { $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('callbackApplicableTransactions') ->andReturn(new Collection([new DatabaseTransactionRecord('sqlite', 1)])); $transactionManager->expects('addCallback')->andReturn(null); @@ -110,6 +112,7 @@ public function testUniqueJobWontGetDispatchedInsideATransaction(): void { $this->app->singleton('db.transactions', function (): DatabaseTransactionsManager { $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('callbackApplicableTransactions') ->andReturn(new Collection([new DatabaseTransactionRecord('sqlite', 1)])); $transactionManager->expects('addCallback')->andReturn(null); @@ -144,6 +147,7 @@ public function testUniqueJobWontGetDispatchedInsideATransactionWhenExplicitlyIn $this->app->singleton('db.transactions', function (): DatabaseTransactionsManager { $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('callbackApplicableTransactions') ->andReturn(new Collection([new DatabaseTransactionRecord('sqlite', 1)])); $transactionManager->expects('addCallback')->andReturn(null); diff --git a/tests/Integration/Queue/Redis/OutboxDeliveryTest.php b/tests/Integration/Queue/Redis/OutboxDeliveryTest.php new file mode 100644 index 0000000000..4fc31922af --- /dev/null +++ b/tests/Integration/Queue/Redis/OutboxDeliveryTest.php @@ -0,0 +1,199 @@ +make('config'); + $config->set('queue.default', 'redis'); + $config->set('queue.connections.redis.queue', 'outbox'); + $config->set('queue.outbox.databases', [$config->string('database.default')]); + $config->set('cache.default', 'worker-array'); + $config->set('app.key', str_repeat('a', 32)); + } + + /** + * Load only tables used by these dispatch and processing paths. + */ + protected function migrateFreshUsing(): array + { + $migrations = dirname(__DIR__, 4) . '/src/testbench/hypervel/migrations/'; + + return [ + '--realpath' => true, + '--path' => [ + $migrations . '0001_01_01_000000_testbench_create_users_table.php', + $migrations . '0001_01_01_000005_testbench_create_job_batches_table.php', + $migrations . '0001_01_01_000009_testbench_create_job_outbox_table.php', + ], + ]; + } + + public function testCommitRelayAndProcessingPreserveEncryptedModelsAndUniqueOwnership(): void + { + $user = UserFactory::new()->create(['name' => 'Before dispatch']); + $finalizations = 0; + $queued = 0; + $this->app->make('events')->listen(JobPayloadFinalizing::class, static function (JobPayloadFinalizing $event) use (&$finalizations): void { + ++$finalizations; + $payload = json_decode($event->payload, true, flags: JSON_THROW_ON_ERROR); + $event->payload = json_encode([...$payload, 'custom' => 'preserved'], JSON_THROW_ON_ERROR); + }); + $this->app->make('events')->listen(JobQueued::class, static function () use (&$queued): void { ++$queued; }); + DB::transaction(static function () use ($user): void { + OutboxEncryptedDeliveryJob::dispatch('private-result-key', $user)->viaOutbox(); + OutboxUniqueDeliveryJob::dispatch('unique')->viaOutbox(); + OutboxUniqueDeliveryJob::dispatch('unique')->viaOutbox(); + }); + $rows = DB::table('job_outbox')->orderBy('id')->get(); + $this->assertCount(2, $rows); + $this->assertSame(0, Queue::size()); + $this->assertStringNotContainsString('private-result-key', $rows[0]->payload); + $unique = new OutboxUniqueDeliveryJob('unique'); + $this->assertTrue(Cache::lock(UniqueLock::getKey($unique))->isLocked()); + $user->name = 'After dispatch'; + $user->save(); + + $this->assertSame(0, Artisan::call('queue:relay', ['--once' => true, '--memory' => 1024])); + $this->assertSame(0, DB::table('job_outbox')->count()); + $this->assertSame(2, Queue::size()); + $this->assertSame(2, $finalizations); + $this->assertSame(2, $queued); + $payloads = Queue::connection()->pendingJobs()->all(); + $this->assertSame($rows->pluck('uuid')->all(), array_map(static fn (InspectedJob $job): string => $job->uuid, $payloads)); + $this->assertSame( + $rows->map(static fn (object $row): array => json_decode($row->payload, true, flags: JSON_THROW_ON_ERROR))->all(), + array_map(static fn (InspectedJob $job): array => $job->payload, $payloads), + ); + $this->assertTrue(Cache::lock(UniqueLock::getKey($unique))->isLocked()); + + $this->work(); + $this->assertSame('After dispatch', Cache::get('private-result-key')); + $this->assertSame(1, Cache::get('unique')); + $this->assertFalse(Cache::lock(UniqueLock::getKey($unique))->isLocked()); + } + + public function testBatchesNestedChainsAndClosuresRunThroughTheExistingWorker(): void + { + $batch = Bus::batch([ + [new OutboxDeliveryJob('first'), new OutboxDeliveryJob('second')], + static function (): void { Cache::put('closure', true); }, + ])->viaOutbox()->dispatch(); + $batch->add([new OutboxDeliveryJob('added')]); + + $this->assertSame(3, DB::table('job_outbox')->count()); + $this->assertSame(4, $batch->fresh()->pendingJobs); + $this->assertSame(0, Queue::size()); + $this->assertSame(0, Artisan::call('queue:relay', ['--once' => true, '--memory' => 1024])); + $this->work(); + + $this->assertSame(1, Cache::get('first')); + $this->assertSame(1, Cache::get('second')); + $this->assertSame(1, Cache::get('added')); + $this->assertTrue(Cache::get('closure')); + $this->assertTrue($batch->fresh()->finished()); + $this->assertSame(0, DB::table('job_outbox')->count()); + $this->assertSame(0, Queue::size()); + } + + public function testDebouncedAndDelayedPayloadsRetainTheirOriginalDeliveryRules(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + OutboxDebouncedDeliveryJob::dispatch('debounced')->viaOutbox(); + OutboxDebouncedDeliveryJob::dispatch('debounced')->viaOutbox(); + OutboxDeliveryJob::dispatch('delayed')->viaOutbox()->delay(60); + $this->assertSame(0, Artisan::call('queue:relay', ['--once' => true, '--memory' => 1024])); + $this->assertSame(0, DB::table('job_outbox')->count()); + $this->assertNull(Queue::pop()); + + Date::setTestNow(Date::now()->addSeconds(61)); + $this->work(); + + $this->assertSame(1, Cache::get('debounced')); + $this->assertSame(1, Cache::get('delayed')); + $this->assertSame(0, Queue::size()); + } + + /** + * Process the ready queue using the public worker command. + */ + protected function work(): void + { + $this->assertSame(0, Artisan::call('queue:work', ['--stop-when-empty' => true, '--sleep' => 0, '--memory' => 1024])); + } +} + +class OutboxDeliveryJob implements ShouldQueue +{ + use Batchable; + use Dispatchable; + use InteractsWithQueue; + use Queueable; + use SerializesModels; + + /** + * Store a result key and optional model reference. + */ + public function __construct(public string $key, public ?User $user = null) + { + } + + /** + * Record execution in a store shared with the worker's child coroutine. + */ + public function handle(): void + { + Cache::put($this->key, $this->user?->name ?? ((int) Cache::get($this->key, 0) + 1)); + } +} + +class OutboxEncryptedDeliveryJob extends OutboxDeliveryJob implements ShouldBeEncrypted +{ +} + +class OutboxUniqueDeliveryJob extends OutboxDeliveryJob implements ShouldBeUnique +{ +} + +#[DebounceFor(30)] +class OutboxDebouncedDeliveryJob extends OutboxDeliveryJob +{ +} diff --git a/tests/Integration/Queue/UniqueJobTest.php b/tests/Integration/Queue/UniqueJobTest.php index 966b71c953..7529f6a152 100644 --- a/tests/Integration/Queue/UniqueJobTest.php +++ b/tests/Integration/Queue/UniqueJobTest.php @@ -7,6 +7,8 @@ use Exception; use Hypervel\Bus\Queueable; use Hypervel\Bus\UniqueLock; +use Hypervel\Cache\Repository as CacheRepository; +use Hypervel\Cache\WorkerArrayStore; use Hypervel\Container\Container; use Hypervel\Contracts\Cache\Repository as Cache; use Hypervel\Contracts\Foundation\Application as ApplicationContract; @@ -26,6 +28,7 @@ use Hypervel\Testbench\Attributes\WithMigration; use Hypervel\Testbench\Factories\UserFactory; use Hypervel\Tests\Integration\Queue\QueueTestCase; +use PHPUnit\Framework\Attributes\DataProvider; #[WithMigration] #[WithMigration('cache')] @@ -41,6 +44,7 @@ protected function defineEnvironment(ApplicationContract $app): void $config = $app->make('config'); $config->set('cache.default', 'database'); + $config->set('cache.stores.custom_unique', ['driver' => 'custom_unique']); $config->set('queue.default', env('QUEUE_CONNECTION', 'database')); } @@ -212,13 +216,20 @@ public function testRetryOfOwnerlessUniqueUntilProcessingJobDoesNotReleaseSubseq $this->assertFalse($this->app->get(Cache::class)->lock($this->getLockKey($job), 10)->get()); } - public function testLockIsReleasedOnModelNotFoundException(): void + #[DataProvider('missingModelCacheRepositories')] + public function testLockIsReleasedOnModelNotFoundException(string $repository): void { UniqueTestSerializesModelsJob::$handled = false; + $manager = $this->app->make('cache'); + $manager->extend('custom_unique', static fn (): CacheRepository => $manager->repository(new WorkerArrayStore)); + /** @var User $user */ $user = UserFactory::new()->create(); - $job = new UniqueTestSerializesModelsJob($user); + $job = $repository === 'default' + ? new UniqueTestSerializesModelsJob($user) + : new UniqueViaSerializesModelsJob($user, $repository); + $cache = $repository === 'default' ? $this->app->get(Cache::class) : $job->uniqueVia(); $this->expectException(ModelNotFoundException::class); @@ -230,10 +241,18 @@ public function testLockIsReleasedOnModelNotFoundException(): void } finally { $this->assertFalse($job::$handled); $this->assertModelMissing($user); - $this->assertTrue($this->app->get(Cache::class)->lock($this->getLockKey($job), 10)->get()); + $this->assertTrue($cache->lock($this->getLockKey($job), 10)->get()); } } + /** + * Exercise named repositories whose identity must survive queued model restoration failure. + */ + public static function missingModelCacheRepositories(): array + { + return [['default'], ['memo'], ['custom_unique']]; + } + public function testModelNotFoundExceptionDoesNotReleaseSubsequentLock(): void { $this->markTestSkippedWhenUsingSyncQueueDriver(); @@ -499,6 +518,27 @@ public function __construct(public User $user) } } +class UniqueViaSerializesModelsJob extends UniqueTestSerializesModelsJob +{ + /** + * Create a unique job using the selected named cache wrapper. + */ + public function __construct(User $user, public string $repository) + { + parent::__construct($user); + } + + /** + * Resolve the cache repository that owns this job's unique lock. + */ + public function uniqueVia(): Cache + { + $manager = Container::getInstance()->make('cache'); + + return $this->repository === 'memo' ? $manager->memo('database') : $manager->store($this->repository); + } +} + class UniqueViaJob extends UniqueTestJob { public function uniqueVia(): Cache diff --git a/tests/OpenTelemetry/Instrumentation/QueueInstrumentationTest.php b/tests/OpenTelemetry/Instrumentation/QueueInstrumentationTest.php index 0be3ffebc3..b6fe135e84 100644 --- a/tests/OpenTelemetry/Instrumentation/QueueInstrumentationTest.php +++ b/tests/OpenTelemetry/Instrumentation/QueueInstrumentationTest.php @@ -26,6 +26,8 @@ use Hypervel\Queue\Events\JobQueueing; use Hypervel\Queue\Events\JobQueueingFailed; use Hypervel\Queue\Events\JobTimedOut; +use Hypervel\Queue\Events\OutboxBatchDelivered; +use Hypervel\Queue\Events\OutboxBatchDeliveryFailed; use Hypervel\Queue\NullQueue; use Hypervel\Queue\QueueManager; use Hypervel\Queue\SyncQueue; @@ -155,6 +157,8 @@ public function testRegistersNothingWhenEveryOutputIsDisabled(): void $this->assertFalse($this->events->hasListeners(JobQueueingFailed::class)); $this->assertFalse($this->events->hasListeners(JobProcessing::class)); $this->assertFalse($this->events->hasListeners(JobAttempted::class)); + $this->assertFalse($this->events->hasListeners(OutboxBatchDelivered::class)); + $this->assertFalse($this->events->hasListeners(OutboxBatchDeliveryFailed::class)); $this->assertSame(0, QueueInstrumentationPayloadQueue::payloadCallbackCount()); $this->assertSame(0, $this->clock->calls); } @@ -982,6 +986,41 @@ public function testPersistentOnlyConnectionsRegisterNoStaticPayloadHook(): void $this->assertSame(0, QueueInstrumentationPayloadQueue::payloadCallbackCount()); } + public function testOutboxMetricsCountAcceptanceAndSettlementFailuresWithoutASecondProducerLifecycle(): void + { + $this->instrumentation()->register($this->options([ + 'traces' => false, + 'propagation' => false, + 'metrics' => [ + 'hypervel.queue.outbox.delivered.messages' => true, + 'hypervel.queue.outbox.delivery.duration' => true, + 'hypervel.queue.outbox.delivery.latency' => true, + ], + ])); + $this->assertFalse($this->events->hasListeners(JobPayloadFinalizing::class)); + $this->assertFalse($this->events->hasListeners(JobQueued::class)); + $this->events->dispatch(new OutboxBatchDelivered('primary', 'redis', 3, 0.25, 10)); + $this->events->dispatch(new OutboxBatchDeliveryFailed('primary', 'redis', 2, 1, 0.5, 20, new RuntimeException('Partial acceptance.'))); + $this->events->dispatch(new OutboxBatchDeliveryFailed('primary', 'redis', 1, 0, 0.75, 30, new RuntimeException('Settlement failed.'))); + $this->metricReader->collect(); + + $delivered = $this->sumPoint('hypervel.queue.outbox.delivered.messages'); + $this->assertSame(6, $delivered->value); + $this->assertSame('primary', $delivered->attributes->get('hypervel.queue.outbox.database')); + $this->assertSame('redis', $delivered->attributes->get('hypervel.queue.connection')); + $duration = $this->metric('hypervel.queue.outbox.delivery.duration')->data->dataPoints; + $latency = $this->metric('hypervel.queue.outbox.delivery.latency')->data->dataPoints; + $failedBatches = array_filter($duration, static fn (HistogramDataPoint $point): bool => $point->attributes->get(ErrorAttributes::ERROR_TYPE) === RuntimeException::class); + $this->assertSame(2, array_sum(array_column($failedBatches, 'count'))); + $this->assertSame(3, array_sum(array_column($duration, 'count'))); + $this->assertSame(1.5, array_sum(array_column($duration, 'sum'))); + $this->assertCount(1, $latency); + $this->assertEquals(60, array_sum(array_column($latency, 'sum'))); + $this->assertCount(3, $this->metricExporter->collect()); + $this->assertSame([], $this->spanExporter->getSpans()); + $this->assertSame(0, $this->clock->calls); + } + public function testDepthCollectionRunsOnlyInEventWorkerZeroAndIsolatesBackendFailures(): void { $queues = m::mock(QueueManager::class); diff --git a/tests/OpenTelemetry/PublishedConfigurationTest.php b/tests/OpenTelemetry/PublishedConfigurationTest.php index 961a6d1cd0..3edf92667d 100644 --- a/tests/OpenTelemetry/PublishedConfigurationTest.php +++ b/tests/OpenTelemetry/PublishedConfigurationTest.php @@ -49,6 +49,9 @@ public function testPublishedDefaultsExposeTheCompleteMetricSurface(): void 'hypervel.object_pool.objects' => true, 'hypervel.object_pool.pending_requests' => true, 'hypervel.queue.jobs' => false, + 'hypervel.queue.outbox.delivered.messages' => true, + 'hypervel.queue.outbox.delivery.duration' => true, + 'hypervel.queue.outbox.delivery.latency' => true, 'hypervel.scheduler.task.duration' => true, 'hypervel.scheduler.task.executions' => true, 'hypervel.server.connections' => true, diff --git a/tests/Pennant/TransactionTest.php b/tests/Pennant/TransactionTest.php index 651f1f63d4..362222fde2 100644 --- a/tests/Pennant/TransactionTest.php +++ b/tests/Pennant/TransactionTest.php @@ -6,6 +6,7 @@ use Hypervel\Contracts\Foundation\Application; use Hypervel\Database\Events\QueryExecuted; +use Hypervel\Database\SimpleConnectionResolver; use Hypervel\Foundation\Testing\RefreshDatabase; use Hypervel\Pennant\Drivers\DatabaseDriver; use Hypervel\Pennant\Feature; @@ -32,6 +33,9 @@ protected function defineEnvironment(Application $app): void 'driver' => 'database', 'cache' => ['store' => 'array'], ]); + $config->set('database.connections.independent', ['driver' => 'sqlite', 'database' => ':memory:']); + $config->set('pennant.stores.independent', ['driver' => 'database', 'connection' => 'independent']); + $config->set('pennant.stores.independent-writer', ['driver' => 'database', 'connection' => 'independent::write']); } protected function setUpInCoroutine(): void @@ -106,6 +110,41 @@ public function testCommittingKeepsValuesMemoizedInsideTheTransaction(): void $this->assertSame(0, $queries->count); } + public function testRoleRollbackForgetsOnlyValuesTrackedByThatSession(): void + { + $resolver = $this->app->make('db.resolver'); + $this->app->instance('db.resolver', new SimpleConnectionResolver($this->app->make('db'))); + $base = DB::connection('independent'); + $writer = DB::connection('independent::write'); + $this->assertNotSame($base, $writer); + + $this->app->instance(TransactionFeatureWithBeforeHook::class, $feature = new TransactionFeatureWithBeforeHook); + $feature->value = 'inside'; + + try { + $base->beginTransaction(); + $writer->beginTransaction(); + + foreach (['independent', 'independent-writer'] as $store) { + Feature::store($store)->define(TransactionFeatureWithBeforeHook::class); + $this->assertSame('inside', Feature::store($store)->for('taylor')->value(TransactionFeatureWithBeforeHook::class)); + } + + $writer->rollBack(); + $feature->value = 'after'; + + $this->assertSame('after', Feature::store('independent-writer')->for('taylor')->value(TransactionFeatureWithBeforeHook::class)); + $this->assertSame('inside', Feature::store('independent')->for('taylor')->value(TransactionFeatureWithBeforeHook::class)); + + $base->rollBack(); + $this->assertSame('after', Feature::store('independent')->for('taylor')->value(TransactionFeatureWithBeforeHook::class)); + } finally { + $base->rollBack(0); + $writer->rollBack(0); + $this->app->instance('db.resolver', $resolver); + } + } + public function testFlushingAStoreKeepsOtherStoresValuesTrackedForRollback(): void { Feature::store('secondary')->define('foo', fn () => 'resolved'); diff --git a/tests/Queue/DatabaseQueueRollbackTest.php b/tests/Queue/DatabaseQueueRollbackTest.php new file mode 100644 index 0000000000..89659a82fb --- /dev/null +++ b/tests/Queue/DatabaseQueueRollbackTest.php @@ -0,0 +1,176 @@ +queue(); + $database = $queue->getDatabase(); + $cache = new Repository(new WorkerArrayStore); + $lock = $kind === 'unique' ? new UniqueLock($cache) : new DebounceLock($cache); + $job = (object) ['debounceFor' => 10]; + $lock->acquireForDispatch($job); + $database->beginTransaction(); + + match ($method) { + 'push' => $queue->push($job), + 'later' => $queue->later(30, $job), + 'bulk' => $queue->bulk([$job]), + }; + + $this->assertFalse(DispatchLockContext::has($job)); + $this->assertSame(1, $database->table('jobs')->count()); + $this->assertTrue($kind === 'unique' ? $cache->lock($lock::getKey($job))->isLocked() : $cache->has($lock::getKey($job))); + $database->rollBack(); + $this->assertSame(0, $database->table('jobs')->count()); + $this->assertFalse($kind === 'unique' ? $cache->lock($lock::getKey($job))->isLocked() : $cache->has($lock::getKey($job))); + } + + /** + * Provide the storage entry points for unique and debounce locks. + */ + public static function dispatchMethods(): array + { + return [ + 'push unique' => ['push', 'unique'], + 'later unique' => ['later', 'unique'], + 'bulk unique' => ['bulk', 'unique'], + 'push debounce' => ['push', 'debounce'], + 'later debounce' => ['later', 'debounce'], + 'bulk debounce' => ['bulk', 'debounce'], + ]; + } + + public function testChunkedBulkRegistersCleanupOnTheEnclosingSavepoint(): void + { + $queue = $this->queue(); + $database = $queue->getDatabase(); + $cache = new Repository(new WorkerArrayStore); + $jobs = array_map(static fn (int $id): object => (object) ['uniqueId' => $id], range(1, 3)); + $lock = new UniqueLock($cache); + + foreach ($jobs as $job) { + $this->assertTrue($lock->acquireForDispatch($job)); + } + + $database->beginTransaction(); + $database->beginTransaction(); + $queue->bulk($jobs); + $this->assertSame(2, $database->transactionLevel()); + $database->rollBack(); + + foreach ($jobs as $job) { + $this->assertFalse($cache->lock(UniqueLock::getKey($job))->isLocked()); + } + + $this->assertSame(0, $database->table('jobs')->count()); + $database->commit(); + } + + public function testDeadlockRetryCanReacquireTheRolledBackDispatchLock(): void + { + $queue = $this->queue(); + $cache = new Repository(new WorkerArrayStore); + $lock = new UniqueLock($cache); + $attempts = 0; + $job = new stdClass; + + $queue->getDatabase()->transaction(function () use ($queue, $lock, $job, &$attempts): void { + ++$attempts; + $this->assertTrue($lock->acquireForDispatch($job)); + $queue->push($job); + + if ($attempts === 1) { + throw new PDOException('deadlock detected', 40001); + } + }, 2); + + $this->assertSame(2, $attempts); + $this->assertSame(1, $queue->getDatabase()->table('jobs')->count()); + $this->assertTrue($cache->lock(UniqueLock::getKey($job))->isLocked()); + } + + public function testDurableInsertDoesNotRegisterCleanupOnAnotherTransaction(): void + { + $queue = $this->queue(); + $manager = $queue->getDatabase()->getTransactionManager(); + $other = new SQLiteConnection(new PDO('sqlite::memory:'), config: ['name' => 'other']); + $other->setTransactionManager($manager); + $other->beginTransaction(); + $cache = new Repository(new WorkerArrayStore); + $job = new stdClass; + $this->assertTrue((new UniqueLock($cache))->acquireForDispatch($job)); + + $queue->push($job); + $other->rollBack(); + + $this->assertSame(1, $queue->getDatabase()->table('jobs')->count()); + $this->assertTrue($cache->lock(UniqueLock::getKey($job))->isLocked()); + } + + /** + * Create a real queue with a small binding limit to exercise chunk savepoints. + */ + protected function queue(): DatabaseQueue + { + $container = new Container; + Container::setInstance($container); + $container->instance(EventDispatcher::class, new Dispatcher($container)); + $database = new RollbackQueueConnection(new PDO('sqlite::memory:'), config: ['name' => 'testing']); + $database->setReadWriteType('write'); + $database->setTransactionManager($manager = new DatabaseTransactionsManager); + $container->instance('db.transactions', $manager); + $database->getSchemaBuilder()->create('jobs', static function (Blueprint $table): void { + $table->id(); + $table->string('queue'); + $table->text('payload'); + $table->unsignedInteger('attempts'); + $table->unsignedBigInteger('reserved_at')->nullable(); + $table->unsignedBigInteger('available_at'); + $table->unsignedBigInteger('created_at'); + }); + $resolver = m::mock(ConnectionResolverInterface::class); + $resolver->shouldReceive('connection')->with('testing::write')->andReturn($database); + $queue = new DatabaseQueue($resolver, 'testing::write', 'jobs'); + $queue->setContainer($container); + $queue->setConnectionName('database'); + + return $queue; + } +} + +class RollbackQueueConnection extends SQLiteConnection +{ + /** + * Force a batch of three jobs through the binding-safe chunk path. + */ + protected function resolveMaxBindings(): int + { + return 12; + } +} diff --git a/tests/Queue/FailoverQueueTest.php b/tests/Queue/FailoverQueueTest.php index e81144a5c4..0e187ce2b1 100644 --- a/tests/Queue/FailoverQueueTest.php +++ b/tests/Queue/FailoverQueueTest.php @@ -23,6 +23,7 @@ use Hypervel\Contracts\Queue\ShouldQueueAfterCommit; use Hypervel\Database\DatabaseTransactionRecord; use Hypervel\Database\DatabaseTransactionsManager; +use Hypervel\Events\Dispatcher as EventDispatcher; use Hypervel\Events\QueuedClosure; use Hypervel\Queue\Attributes\Delay; use Hypervel\Queue\Events\QueueFailedOver; @@ -314,6 +315,58 @@ public function testPushReentersDeferralForATransactionOpenedByAnEarlierCommitCa $this->assertSame(['job'], $connection->pushedJobs); } + public function testInlineAfterCommitChildRetainsLockOwnershipForFallback(): void + { + $container = Container::getInstance(); + $container->instance('db.transactions', new DatabaseTransactionsManager); + $events = new EventDispatcher($container); + $container->instance(DispatcherContract::class, $events); + $container->instance('events', $events); + $failure = new RuntimeException('Scheduling failed before acceptance.'); + $first = new class($failure) extends SyncQueue { + /** + * Configure an after-commit child whose execution cannot start. + */ + public function __construct(private RuntimeException $failure) + { + parent::__construct(true); + } + + /** + * Fail before the queue can accept the dispatch. + */ + protected function executePayload(string $payload, ?string $queue = null): int + { + throw $this->failure; + } + }; + $first->setContainer($container)->setConnectionName('first'); + $second = m::mock(Queue::class); + $job = new FailoverQueueUniqueDebouncedJob; + $lock = new UniqueLock(new CacheRepository(new ArrayStore)); + $this->assertTrue($lock->acquireForDispatch($job)); + $second->expects('push')->with($job)->andReturnUsing(function ($job) use ($lock): string { + $this->assertTrue(DispatchLockContext::has($job)); + $this->assertFalse($lock->acquire($job)); + DispatchLockContext::accept($job); + + return 'accepted'; + }); + $events->listen(QueueFailedOver::class, function (QueueFailedOver $event) use ($failure): void { + $this->assertSame($failure, $event->exception); + }); + $failover = new FailoverQueue( + new FailoverQueueFakeManager(['first' => $first, 'second' => $second]), + $events, + ['first', 'second'], + ); + $failover->setContainer($container); + + $this->assertSame('accepted', $failover->push($job)); + $this->assertFalse(DispatchLockContext::has($job)); + $this->assertFalse($lock->acquire($job)); + } + public function testNestedCommitWaitsForTheRootTransaction(): void { $connection = new FailoverQueueSuccessfulConnection('sync'); diff --git a/tests/Queue/Outbox/DatabaseOutboxPublicationTest.php b/tests/Queue/Outbox/DatabaseOutboxPublicationTest.php new file mode 100644 index 0000000000..ea569ab7bc --- /dev/null +++ b/tests/Queue/Outbox/DatabaseOutboxPublicationTest.php @@ -0,0 +1,101 @@ +queue(); + $routes->forward(['orders' => 'processing', 'processing' => 'archive']); + $now = CarbonImmutable::now()->getTimestamp(); + $messages = [$this->message(1, 'orders', $now + 30), $this->message(2, 'reports', $now - 30)]; + $database->enableQueryLog(); + + $result = $queue->publishOutboxMessages($messages); + $inserts = array_filter($database->getQueryLog(), static fn (array $query): bool => str_starts_with($query['query'], 'insert into')); + $database->disableQueryLog(); + + $this->assertSame([1, 2], $result->accepted); + $this->assertNull($result->failure); + $this->assertCount(1, $inserts); + $rows = $database->table('jobs')->orderBy('id')->get()->all(); + $this->assertSame(['processing', 'reports'], array_column($rows, 'queue')); + $this->assertSame([$now + 30, $now], array_column($rows, 'available_at')); + $this->assertSame(array_column($messages, 'payload'), array_column($rows, 'payload')); + $this->assertSame(0, $database->transactionLevel()); + } + + public function testARejectedLaterChunkRollsBackEveryInsertedMessage(): void + { + [$queue, $database] = $this->queue(); + $database->unprepared("CREATE TRIGGER reject_job BEFORE INSERT ON jobs WHEN NEW.queue = 'rejected' BEGIN SELECT RAISE(ABORT, 'Queue rejected'); END"); + + $result = $queue->publishOutboxMessages([ + $this->message(1, 'orders'), $this->message(2, 'orders'), $this->message(3, 'rejected'), + ]); + + $this->assertSame([], $result->accepted); + $this->assertInstanceOf(QueryException::class, $result->failure); + $this->assertSame(0, $database->table('jobs')->count()); + $this->assertSame(0, $database->transactionLevel()); + } + + /** + * Create a real target jobs table with a two-row binding limit. + */ + protected function queue(): array + { + $container = new Container; + Container::setInstance($container); + $database = new OutboxPublicationConnection(new PDO('sqlite::memory:'), config: ['name' => 'target']); + $container->instance('db.schema', $database->getSchemaBuilder()); + $container->instance('queue.routes', $routes = new QueueRoutes); + Facade::setFacadeApplication($container); + (require dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000006_testbench_create_jobs_table.php')->up(); + $resolver = m::mock(ConnectionResolverInterface::class); + $resolver->shouldReceive('connection')->with('target')->andReturn($database); + $queue = new DatabaseQueue($resolver, 'target', 'jobs'); + $queue->setContainer($container)->setConnectionName('database'); + + return [$queue, $database, $routes]; + } + + /** + * Build a finalized payload without recreating its serialized command. + */ + protected function message(int $id, string $queue, ?int $deliverAt = null): OutboxMessage + { + $uuid = (string) Str::uuid(); + + return new OutboxMessage($id, $uuid, (string) Str::uuid(), 1, 'database', $queue, '{ "uuid": "' . $uuid . '" }', [], $deliverAt, 1, 1); + } +} + +class OutboxPublicationConnection extends SQLiteConnection +{ + /** + * Exercise binding-safe chunking with a small batch. + */ + protected function resolveMaxBindings(): int + { + return 12; + } +} diff --git a/tests/Queue/Outbox/DatabaseOutboxRepositoryTest.php b/tests/Queue/Outbox/DatabaseOutboxRepositoryTest.php new file mode 100644 index 0000000000..b35caff3e5 --- /dev/null +++ b/tests/Queue/Outbox/DatabaseOutboxRepositoryTest.php @@ -0,0 +1,287 @@ + true, + '--path' => dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php', + ]; + } + + public function testClaimPreservesPayloadAndOriginalEligibilityWithThreeStatements(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $now = Date::now()->getTimestamp(); + $repository = $this->repository(); + $connection = DB::connection(); + $first = $this->row($now - 10); + $second = $this->row($now - 30); + $future = $this->row($now + 60); + $repository->insert($connection, [$first, $second, $future]); + + $transactionsStarted = 0; + $this->app->make('events')->listen(TransactionBeginning::class, static function () use (&$transactionsStarted): void { + ++$transactionsStarted; + }); + $connection->enableQueryLog(); + $connection->flushQueryLog(); + $messages = $repository->claim($connection->getName(), 2); + $queries = $connection->getQueryLog(); + $connection->disableQueryLog(); + + $this->assertCount(3, $queries); + $this->assertSame(0, $transactionsStarted); + $this->assertSame([$second['uuid'], $first['uuid']], array_column($messages, 'uuid')); + $this->assertSame([$now - 30, $now - 10], array_column($messages, 'eligibleAt')); + $this->assertSame($second['payload'], $messages[0]->payload); + $this->assertSame(['transport' => ['custom' => ['group' => 'orders']]], $messages[0]->options); + $this->assertSame(1, $messages[0]->attempts); + $this->assertSame($now, $messages[0]->createdAt); + $this->assertSame($messages[0]->leaseToken, $messages[1]->leaseToken); + $this->assertSame($now + 300, (int) DB::table('job_outbox')->where('uuid', $first['uuid'])->value('available_at')); + $this->assertSame([], $repository->claim($connection->getName(), 2)); + } + + public function testExpiredClaimCanBeRecoveredWithoutChangingDeliveryOrPayload(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $repository = $this->repository(); + $row = $this->row(Date::now()->getTimestamp()); + $row['deliver_at'] = Date::now()->getTimestamp() + 900; + $repository->insert(DB::connection(), [$row]); + $first = $repository->claim(DB::connection()->getName(), 10)[0]; + + Date::setTestNow(Date::now()->addSeconds(300)); + $second = $repository->claim(DB::connection()->getName(), 10)[0]; + + $this->assertSame($first->id, $second->id); + $this->assertNotSame($first->leaseToken, $second->leaseToken); + $this->assertSame(2, $second->attempts); + $this->assertSame($first->deliverAt, $second->deliverAt); + $this->assertSame($first->payload, $second->payload); + } + + public function testCustomNonPdoConnectionsUseTheConditionalClaimAndBindingFallback(): void + { + $database = DB::connection(); + $connection = m::mock(ConnectionInterface::class); + $connection->shouldReceive('table')->with('job_outbox')->andReturnUsing(fn (): Builder => $database->table('job_outbox')); + $connection->shouldReceive('transaction')->andReturnUsing(fn (Closure $callback): mixed => $database->transaction($callback)); + $resolver = m::mock(ConnectionResolverInterface::class); + $resolver->shouldReceive('connection')->with('custom')->andReturn($connection); + $repository = new DatabaseOutboxRepository($resolver, ['custom'], 'job_outbox', 300); + $row = $this->row(Date::now()->getTimestamp()); + $repository->insert($connection, [$row]); + + $messages = $repository->claim('custom', 1); + + $this->assertSame([$row['uuid']], array_column($messages, 'uuid')); + $this->assertSame(1, $repository->settle('custom', $messages, new OutboxBatchResult([$messages[0]->id]))); + $this->assertSame(0, $database->table('job_outbox')->count()); + } + + public function testSettlementDeletesAcceptedDefersFutureAndBacksOffFailures(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $now = Date::now()->getTimestamp(); + $repository = $this->repository(); + $database = DB::connection()->getName(); + $accepted = $this->row($now); + $deferred = $this->row($now); + $deferred['deliver_at'] = $now + 2000; + $deferred['last_error'] = 'Previous failure'; + $failed = $this->row($now); + $failed['attempts'] = 6; + $repository->insert(DB::connection(), [ + $accepted, array_replace($accepted, ['uuid' => (string) Str::uuid()]), + $deferred, array_replace($deferred, ['uuid' => (string) Str::uuid()]), + $failed, array_replace($failed, ['uuid' => (string) Str::uuid()]), + ]); + $messages = $repository->claim($database, 10); + $connection = DB::connection(); + $connection->enableQueryLog(); + $connection->flushQueryLog(); + + try { + $affected = $repository->settle($database, $messages, new OutboxBatchResult( + [$messages[0]->id, $messages[1]->id], + [$messages[2]->id, $messages[3]->id], + new RuntimeException(str_repeat('雪', 5000)), + )); + $queries = $connection->getQueryLog(); + } finally { + $connection->disableQueryLog(); + } + + $this->assertCount(3, $queries); + $this->assertSame(6, $affected); + $this->assertNull($repository->find($database, $accepted['uuid'])); + $storedDeferred = $repository->find($database, $deferred['uuid']); + $this->assertNull($storedDeferred->lease_token); + $this->assertSame($now + 2000, (int) $storedDeferred->available_at); + $this->assertSame(0, (int) $storedDeferred->attempts); + $this->assertSame('Previous failure', $storedDeferred->last_error); + $storedFailed = $repository->find($database, $failed['uuid']); + $this->assertNull($storedFailed->lease_token); + $this->assertSame(7, (int) $storedFailed->attempts); + $this->assertGreaterThanOrEqual($now + 300, (int) $storedFailed->available_at); + $this->assertLessThanOrEqual($now + 600, (int) $storedFailed->available_at); + $this->assertSame(4096, mb_strlen($storedFailed->last_error)); + } + + public function testStaleSettlementCannotDeleteOrReleaseANewerClaim(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $repository = $this->repository(); + $database = DB::connection()->getName(); + $repository->insert(DB::connection(), [$this->row(Date::now()->getTimestamp()), $this->row(Date::now()->getTimestamp())]); + $old = $repository->claim($database, 10); + Date::setTestNow(Date::now()->addSeconds(300)); + $current = $repository->claim($database, 10); + + $this->assertSame(0, $repository->settle($database, $old, new OutboxBatchResult( + [$old[0]->id], + failure: new RuntimeException('Old attempt failed'), + ))); + $this->assertSame($current[0]->leaseToken, $repository->find($database, $current[0]->uuid)->lease_token); + $this->assertSame($current[1]->leaseToken, $repository->find($database, $current[1]->uuid)->lease_token); + $this->assertSame(2, $repository->settle($database, $current, new OutboxBatchResult(array_column($current, 'id')))); + } + + public function testOperatorQueriesDistinguishScheduledFailedAndLeasedRows(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $now = Date::now()->getTimestamp(); + $repository = $this->repository(); + $connection = DB::connection(); + $database = $connection->getName(); + $this->assertSame([ + 'total' => 0, 'eligible' => 0, 'delayed' => 0, 'leased' => 0, + 'retrying' => 0, 'oldest_eligible_age' => null, + ], $repository->statistics($database)); + $eligible = $this->row($now - 20); + $delayed = $this->row($now + 1000); + $leased = $this->row($now + 300); + $leased['lease_token'] = (string) Str::uuid(); + $leased['attempts'] = 1; + $failed = $this->row($now + 600); + $failed['attempts'] = 7; + $failed['last_error'] = 'Queue unavailable'; + $repository->insert(DB::connection(), [$eligible, $delayed, $leased, $failed]); + + $connection->enableQueryLog(); + $connection->flushQueryLog(); + $statistics = $repository->statistics($database); + $queries = $connection->getQueryLog(); + $connection->disableQueryLog(); + + $this->assertCount(1, $queries); + $this->assertSame([ + 'total' => 4, 'eligible' => 1, 'delayed' => 1, 'leased' => 1, + 'retrying' => 1, 'oldest_eligible_age' => 20, + ], $statistics); + $this->assertCount(2, $repository->pending($database, 2)); + $this->assertSame(0, $repository->retry($database, $delayed['uuid'])); + $this->assertSame(0, $repository->retry($database, $leased['uuid'])); + $this->assertSame(1, $repository->retryAll($database)); + $retried = $repository->find($database, $failed['uuid']); + $this->assertSame($now, (int) $retried->available_at); + $this->assertNull($retried->last_error); + $this->assertSame(7, (int) $retried->attempts); + $leasedRow = $repository->find($database, $leased['uuid']); + $this->assertFalse($repository->forget($database, $leasedRow->id, $leasedRow->lease_token)); + $this->assertFalse($repository->forget($database, $retried->id, (string) Str::uuid())); + $this->assertTrue($repository->forget($database, $retried->id, null)); + $this->assertNull($repository->find($database, $failed['uuid'])); + } + + public function testRepositoryRejectsUnconfiguredSourceBeforeQuerying(): void + { + $this->expectException(InvalidArgumentException::class); + $this->expectExceptionMessage('not configured in queue.outbox.databases'); + + $this->repository()->claim('not-configured', 100); + } + + public function testRetryAllUsesBoundedChunksAndExcludesMessagesAddedDuringTheOperation(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $now = Date::now()->getTimestamp(); + $repository = $this->repository(); + $database = DB::connection()->getName(); + $rows = array_map(fn (): array => array_replace($this->row($now + 600), ['attempts' => 1]), range(1, 1001)); + $repository->insert(DB::connection(), $rows); + $later = array_replace($this->row($now + 600), ['attempts' => 1]); + $updates = 0; + $this->app->make('events')->listen(QueryExecuted::class, static function (QueryExecuted $event) use ($repository, $later, &$updates): void { + if (str_starts_with($event->sql, 'update ') && str_contains($event->sql, 'job_outbox')) { + if (++$updates === 1) { + $repository->insert(DB::connection(), [$later]); + } + } + }); + + $this->assertSame(1001, $repository->retryAll($database)); + $this->assertSame(2, $updates); + $this->assertSame($now + 600, (int) $repository->find($database, $later['uuid'])->available_at); + $this->assertSame(1001, DB::table('job_outbox')->where('available_at', $now)->count()); + } + + /** + * Create a repository using the test's configured database. + */ + protected function repository(): DatabaseOutboxRepository + { + return new DatabaseOutboxRepository($this->app->make('db'), [DB::connection()->getName()], 'job_outbox', 300); + } + + /** + * Build a row with opaque payload bytes and transport metadata. + */ + protected function row(int $availableAt): array + { + return [ + 'uuid' => (string) Str::uuid(), + 'connection' => 'redis', + 'queue' => 'orders', + 'payload' => '{ "uuid": "opaque-payload", "data": {"text": "雪"} }', + 'options' => '{"transport":{"custom":{"group":"orders"}}}', + 'deliver_at' => null, + 'attempts' => 0, + 'available_at' => $availableAt, + 'lease_token' => null, + 'last_error' => null, + 'created_at' => Date::now()->getTimestamp(), + ]; + } +} diff --git a/tests/Queue/Outbox/OutboxBatchTest.php b/tests/Queue/Outbox/OutboxBatchTest.php new file mode 100644 index 0000000000..5ff8bcecdd --- /dev/null +++ b/tests/Queue/Outbox/OutboxBatchTest.php @@ -0,0 +1,106 @@ +make('config'); + $config->set('database.default', 'batch'); + $config->set('database.connections.batch', ['driver' => 'sqlite', 'database' => ':memory:']); + $config->set('database.connections.business', ['driver' => 'sqlite', 'database' => ':memory:']); + $config->set('queue.batching.database', 'batch'); + $config->set('queue.outbox.databases', ['batch', 'business']); + $config->set('queue.connections.database', [ + 'driver' => 'database', 'connection' => 'business', 'table' => 'jobs', + 'queue' => 'default', 'after_commit' => false, + ]); + } + + /** + * Install the shipped operational schemas without contacting a broker. + */ + protected function setUpInCoroutine(): void + { + $migrations = dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/'; + Schema::swap(DB::connection('batch')->getSchemaBuilder()); + (require $migrations . '0001_01_01_000005_testbench_create_job_batches_table.php')->up(); + (require $migrations . '0001_01_01_000009_testbench_create_job_outbox_table.php')->up(); + Schema::swap(DB::connection('business')->getSchemaBuilder()); + (require $migrations . '0001_01_01_000006_testbench_create_jobs_table.php')->up(); + (require $migrations . '0001_01_01_000009_testbench_create_job_outbox_table.php')->up(); + $this->app->instance('redis', m::mock(Factory::class)); + } + + #[DataProvider('queueConnections')] + public function testConflictingOwnerCannotChangeAnExistingBatch(string $connection): void + { + $batch = Bus::batch([])->onConnection($connection)->onOutboxDatabase('business')->dispatch(); + DB::connection('business')->beginTransaction(); + + try { + $batch->add([new OutboxBatchJob]); + $this->fail('An outbox owner cannot differ from the batch counter transaction.'); + } catch (LogicException $exception) { + $this->assertStringContainsString('Outbox database [business] does not match the batch database [batch]', $exception->getMessage()); + $this->assertSame(0, $batch->fresh()->totalJobs); + $this->assertSame(0, $batch->fresh()->pendingJobs); + $this->assertSame(0, DB::connection('batch')->table('job_outbox')->count()); + $this->assertSame(0, DB::connection('business')->table('job_outbox')->count()); + $this->assertSame(0, DB::connection('business')->table('jobs')->count()); + } finally { + DB::connection('business')->rollBack(); + } + } + + /** + * Cover staged delivery, the database fast path and local after-commit delivery. + */ + public static function queueConnections(): array + { + return ['staged' => ['redis'], 'database' => ['database'], 'local' => ['sync']]; + } + + public function testSameOwnerRollsBackCountersAndStagedJobsTogether(): void + { + $batch = Bus::batch([])->onConnection('redis')->onOutboxDatabase('batch')->dispatch(); + DB::beginTransaction(); + + try { + $batch->add([new OutboxBatchJob]); + $this->assertSame(1, $batch->fresh()->totalJobs); + $this->assertSame(1, $batch->fresh()->pendingJobs); + $this->assertSame(1, DB::table('job_outbox')->count()); + } finally { + DB::rollBack(); + } + + $this->assertSame(0, $batch->fresh()->totalJobs); + $this->assertSame(0, $batch->fresh()->pendingJobs); + $this->assertSame(0, DB::table('job_outbox')->count()); + } +} + +class OutboxBatchJob +{ + use Batchable; + use Queueable; +} diff --git a/tests/Queue/Outbox/OutboxCommandsTest.php b/tests/Queue/Outbox/OutboxCommandsTest.php new file mode 100644 index 0000000000..a29876877f --- /dev/null +++ b/tests/Queue/Outbox/OutboxCommandsTest.php @@ -0,0 +1,254 @@ +make('config'); + $config->set('cache.default', 'worker-array'); + $config->set('queue.outbox.databases', [$config->string('database.default')]); + } + + /** + * Load only the shipped outbox schema. + */ + protected function migrateFreshUsing(): array + { + return [ + '--realpath' => true, + '--path' => dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php', + ]; + } + + public function testListingIsBoundedAndOmitsPayloadOptionsAndOwnershipTokens(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $error = str_repeat('Broker unavailable. ', 12) . "\nSecond error line."; + $first = $this->insert(['last_error' => $error]); + $this->insert(); + + $this->assertSame(0, Artisan::call('queue:outbox', ['--json' => true, '--limit' => 1])); + $output = Artisan::output(); + $rows = json_decode($output, true, flags: JSON_THROW_ON_ERROR); + $this->assertCount(1, $rows); + $this->assertSame($first, $rows[0]['uuid']); + $this->assertSame('SendInvoice', $rows[0]['job']); + $this->assertSame('eligible', $rows[0]['state']); + $this->assertSame(Date::now()->getTimestamp(), $rows[0]['available_at']); + $this->assertSame($error, $rows[0]['last_error']); + $this->assertArrayNotHasKey('payload', $rows[0]); + $this->assertArrayNotHasKey('options', $rows[0]); + $this->assertArrayNotHasKey('lease_token', $rows[0]); + $this->assertStringNotContainsString('private-content', $output); + + Artisan::call('queue:outbox', ['--limit' => 1]); + $output = Artisan::output(); + $this->assertStringContainsString('2040-01-01 12:00:00', $output); + $this->assertStringContainsString(Str::limit(Str::before($error, "\n"), 80), $output); + $this->assertStringNotContainsString('Second error line.', $output); + $this->assertStringNotContainsString($error, $output); + } + + public function testListingRejectsAnUnboundedLimit(): void + { + $this->expectException(InvalidArgumentException::class); + + Artisan::call('queue:outbox', ['--limit' => 1001]); + } + + public function testMonitorReportsDistinctStatesAndAlertsOnlyForEligibleAge(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $now = Date::now()->getTimestamp(); + $this->insert(['available_at' => $now + 60]); + $this->insert(['available_at' => $now + 60, 'attempts' => 1, 'lease_token' => (string) Str::uuid()]); + $this->insert(['available_at' => $now + 60, 'attempts' => 1]); + $seen = []; + $this->app->make('events')->listen(OutboxBusy::class, static function (OutboxBusy $event) use (&$seen): void { $seen[] = $event; }); + + Artisan::call('queue:outbox-monitor', ['--json' => true, '--age' => 10]); + $statistics = json_decode(Artisan::output(), true, flags: JSON_THROW_ON_ERROR)[0]; + $this->assertSame(3, $statistics['total']); + $this->assertSame(0, $statistics['eligible']); + $this->assertSame(1, $statistics['delayed']); + $this->assertSame(1, $statistics['leased']); + $this->assertSame(1, $statistics['retrying']); + $this->assertSame('OK', $statistics['status']); + $this->assertSame([], $seen); + + $this->insert(['available_at' => $now - 20]); + $this->insert(['available_at' => $now - 1, 'attempts' => 2]); + Artisan::call('queue:outbox-monitor', ['--json' => true, '--age' => 10]); + $statistics = json_decode(Artisan::output(), true, flags: JSON_THROW_ON_ERROR)[0]; + $this->assertSame(2, $statistics['eligible']); + $this->assertSame(1, $statistics['retrying']); + $this->assertSame($statistics['total'], $statistics['eligible'] + $statistics['delayed'] + $statistics['leased'] + $statistics['retrying']); + $this->assertCount(1, $seen); + $this->assertSame(5, $seen[0]->size); + $this->assertSame(20, $seen[0]->oldestEligibleAge); + } + + public function testRetryAllPreservesAttemptHistoryAndExcludesLiveAndNeverAttemptedRows(): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $now = Date::now()->getTimestamp(); + $retry = $this->insert(['attempts' => 5, 'available_at' => $now + 500, 'last_error' => 'Broker offline']); + $live = $this->insert(['attempts' => 1, 'available_at' => $now + 500, 'lease_token' => (string) Str::uuid()]); + $new = $this->insert(['available_at' => $now + 500]); + + $this->assertSame(0, Artisan::call('queue:outbox-retry', ['uuid' => ['all']])); + $row = DB::table('job_outbox')->where('uuid', $retry)->first(); + $this->assertSame($now, (int) $row->available_at); + $this->assertSame(5, (int) $row->attempts); + $this->assertNull($row->last_error); + $this->assertSame($now + 500, (int) DB::table('job_outbox')->where('uuid', $live)->value('available_at')); + $this->assertSame($now + 500, (int) DB::table('job_outbox')->where('uuid', $new)->value('available_at')); + $this->assertSame(1, Artisan::call('queue:outbox-retry', ['uuid' => [$live, $new, 'missing']])); + $this->assertSame(0, Artisan::call('queue:outbox-retry', ['uuid' => [$retry], '--database' => DB::connection()->getName()])); + } + + public function testRetryRefusesCrossSourceUuidAmbiguityBeforeMutation(): void + { + $repository = m::mock(DatabaseOutboxRepository::class); + $repository->expects('getDatabases')->andReturn(['first', 'second']); + $repository->expects('find')->with('first', 'same')->andReturn((object) ['id' => 1]); + $repository->expects('find')->with('second', 'same')->andReturn((object) ['id' => 2]); + $this->app->instance(DatabaseOutboxRepository::class, $repository); + $this->expectException(RuntimeException::class); + $this->expectExceptionMessage('Select one using --database'); + + Artisan::call('queue:outbox-retry', ['uuid' => ['same']]); + } + + public function testForgetRequiresConfirmationAndRefusesLiveLeases(): void + { + $uuid = $this->insert(['attempts' => 1, 'lease_token' => (string) Str::uuid(), 'available_at' => Date::now()->getTimestamp() + 60]); + $prompt = 'Are you sure you want to run this command?'; + $this->artisan('queue:outbox-forget', ['uuid' => $uuid])->expectsConfirmation($prompt, 'no')->expectsOutputToContain('Command cancelled.')->assertExitCode(1); + $this->assertDatabaseHas('job_outbox', ['uuid' => $uuid]); + $this->artisan('queue:outbox-forget', ['uuid' => $uuid])->expectsConfirmation($prompt, 'yes')->assertExitCode(1); + $this->assertDatabaseHas('job_outbox', ['uuid' => $uuid]); + + $this->assertSame(1, Artisan::call('queue:outbox-forget', ['uuid' => $uuid, '--no-interaction' => true])); + $this->assertStringContainsString('Command cancelled.', Artisan::output()); + $this->assertSame(1, Artisan::call('queue:outbox-forget', ['uuid' => $uuid, '--no-interaction' => true, '--force' => true])); + $this->assertDatabaseHas('job_outbox', ['uuid' => $uuid]); + + DB::table('job_outbox')->where('uuid', $uuid)->update(['available_at' => Date::now()->getTimestamp() - 1]); + $this->assertSame(0, Artisan::call('queue:outbox-forget', ['uuid' => $uuid, '--no-interaction' => true, '--force' => true])); + $this->assertDatabaseMissing('job_outbox', ['uuid' => $uuid]); + } + + #[DataProvider('lockReplacement')] + public function testForgetReleasesOnlyItsExactUniqueOwnerWithoutHydratingOtherData(bool $replaced): void + { + $cache = Cache::store('worker-array'); + $lock = $cache->lock('unique-key'); + $this->assertTrue($lock->get()); + $uuid = $this->insert(['payload' => json_encode([ + 'data' => ['command' => 'Do not deserialize this job.'], + 'illuminate:log:context' => ['hidden' => [ + 'laravel_unique_job_cache_store' => serialize('worker-array'), + 'laravel_unique_job_key' => serialize('unique-key'), + 'laravel_unique_job_lock_owner' => serialize($lock->owner()), + 'unrelated' => 'Do not hydrate this context.', + ]], + ], JSON_THROW_ON_ERROR)]); + + if ($replaced) { + $lock->release(); + $replacement = $cache->lock('unique-key'); + $this->assertTrue($replacement->get()); + } + + $this->artisan('queue:outbox-forget', ['uuid' => $uuid]) + ->expectsConfirmation('Are you sure you want to run this command?', 'yes') + ->assertExitCode(0); + $this->assertDatabaseMissing('job_outbox', ['uuid' => $uuid]); + $this->assertSame(! $replaced, $cache->lock('unique-key')->get()); + } + + /** + * Cover current ownership and a later dispatch's replacement lock. + */ + public static function lockReplacement(): array + { + return [[false], [true]]; + } + + public function testForgetReportsUnresolvableOwnershipAfterDeletingWithoutGuessingTheDefaultStore(): void + { + $cache = Cache::store('worker-array'); + $lock = $cache->lock('unique-key'); + $this->assertTrue($lock->get()); + $uuid = $this->insert(['payload' => json_encode([ + 'illuminate:log:context' => ['hidden' => [ + 'laravel_unique_job_cache_store' => serialize(null), + 'laravel_unique_job_key' => serialize('unique-key'), + 'laravel_unique_job_lock_owner' => serialize($lock->owner()), + ]], + ], JSON_THROW_ON_ERROR)]); + $exceptions = m::mock(ExceptionHandler::class); + $exceptions->expects('report')->with(m::type(LogicException::class), m::on(fn (array $context): bool => $context['job_uuid'] === $uuid)); + $this->app->instance(ExceptionHandler::class, $exceptions); + + $this->artisan('queue:outbox-forget', ['uuid' => $uuid]) + ->expectsConfirmation('Are you sure you want to run this command?', 'yes') + ->expectsOutputToContain('was deleted, but unique-lock cleanup failed') + ->assertExitCode(1); + $this->assertDatabaseMissing('job_outbox', ['uuid' => $uuid]); + $this->assertFalse($cache->lock('unique-key')->get()); + } + + /** + * Insert an operational row using the shipped repository and schema. + */ + protected function insert(array $attributes = []): string + { + $uuid = (string) Str::uuid(); + $now = Date::now()->getTimestamp(); + $this->app->make(DatabaseOutboxRepository::class)->insert(DB::connection(), [array_replace([ + 'uuid' => $uuid, + 'connection' => 'redis', + 'queue' => 'invoices', + 'payload' => '{"displayName":"SendInvoice","data":{"command":"private-content"}}', + 'options' => '{"private-content":true}', + 'deliver_at' => null, + 'attempts' => 0, + 'available_at' => $now, + 'lease_token' => null, + 'last_error' => null, + 'created_at' => $now, + ], $attributes)]); + + return $uuid; + } +} diff --git a/tests/Queue/Outbox/OutboxDispatchTest.php b/tests/Queue/Outbox/OutboxDispatchTest.php new file mode 100644 index 0000000000..402eade6a0 --- /dev/null +++ b/tests/Queue/Outbox/OutboxDispatchTest.php @@ -0,0 +1,382 @@ +make('config'); + $config->set('queue.outbox.databases', [$config->string('database.default')]); + } + + /** + * Load only the shipped outbox migration. + */ + protected function migrateFreshUsing(): array + { + return [ + '--realpath' => true, + '--path' => dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php', + ]; + } + + public function testSingleStagingPreservesFinalizedPayloadAndLifecycle(): void + { + $queue = $this->queue(); + $events = $this->app->make('events'); + $finalized = 0; + $observed = []; + $events->listen(JobPayloadFinalizing::class, function (JobPayloadFinalizing $event) use (&$finalized): void { + ++$finalized; + $event->payload = json_encode([...json_decode($event->payload, true), 'custom' => 'retained']); + }); + $events->listen(JobQueueing::class, function (JobQueueing $event) use (&$observed): void { + $observed[] = $event->payload; + }); + $events->listen(JobQueued::class, function (JobQueued $event) use (&$observed): void { + $observed[] = $event->payload; + }); + + $id = $queue->push((new OutboxDispatchJob)->viaOutbox()->beforeCommit()); + $row = DB::table('job_outbox')->sole(); + + $this->assertSame($row->uuid, $id); + $this->assertSame($id, json_decode($row->payload, true)['uuid']); + $this->assertSame([$row->payload, $row->payload], $observed); + $this->assertSame(1, $finalized); + $this->assertSame('redis', $row->connection); + $this->assertSame('orders', $row->queue); + $this->assertNull($row->options); + $this->assertSame([], $queue->published); + } + + public function testSelectionPrecedenceAndDirectDispatchAvoidOutboxResolution(): void + { + $queue = $this->queue(); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + $queue->push(new OutboxDispatchJob); + $queue->push((new MarkedOutboxDispatchJob)->withoutOutbox()); + $queue->setConfig(['outbox' => true]); + $queue->push((new OutboxDispatchJob)->withoutOutbox()); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + $this->assertCount(3, $queue->published); + + $queue->push(new OutboxDispatchJob); + $queue->setConfig([]); + $queue->push(new MarkedOutboxDispatchJob); + $queue->push((new OutboxDispatchJob)->viaOutbox()); + + $this->assertSame(3, DB::table('job_outbox')->count()); + $this->assertCount(3, $queue->published); + } + + public function testRawPublicationNeverStages(): void + { + $queue = $this->queue()->setConfig(['outbox' => true]); + $queue->pushRaw('{"uuid":"raw"}'); + + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + $this->assertSame(['{"uuid":"raw"}'], $queue->published); + } + + public function testBulkStagingUsesOneInsertAndFinalizesEachPayloadOnce(): void + { + $queue = $this->queue(); + $finalized = 0; + $this->app->make('events')->listen(JobPayloadFinalizing::class, function () use (&$finalized): void { + ++$finalized; + }); + $jobs = array_map(static fn (): OutboxDispatchJob => (new OutboxDispatchJob)->viaOutbox(), range(1, 100)); + $connection = DB::connection(); + $connection->enableQueryLog(); + $connection->flushQueryLog(); + + $this->assertNull($queue->bulk($jobs)); + $inserts = array_filter($connection->getQueryLog(), static fn (array $query): bool => str_starts_with($query['query'], 'insert into')); + $connection->disableQueryLog(); + + $this->assertCount(1, $inserts); + $this->assertSame(100, $finalized); + $this->assertSame(100, DB::table('job_outbox')->count()); + $this->assertSame([], $queue->published); + } + + #[DataProvider('baseBulkDrivers')] + public function testBaseBulkStagesOneInsertAndKeepsTheDirectPartition(bool $beanstalk): void + { + $queue = $beanstalk + ? m::mock(BeanstalkdQueue::class, [m::mock(PheanstalkManagerInterface::class . ', ' . PheanstalkPublisherInterface::class . ', ' . PheanstalkSubscriberInterface::class), 'orders', 60])->makePartial() + : m::mock(Queue::class . ', ' . QueueContract::class)->makePartial(); + $queue->setContainer($this->app)->setConnectionName($beanstalk ? 'beanstalkd' : 'custom'); + $direct = (new OutboxDispatchJob)->withoutOutbox(); + $queue->expects('push')->with($direct, '', 'orders')->andReturn('direct'); + $jobs = array_map(static fn (): OutboxDispatchJob => (new OutboxDispatchJob)->viaOutbox(), range(1, 100)); + $connection = DB::connection(); + $connection->enableQueryLog(); + $connection->flushQueryLog(); + + try { + $queue->bulk([...$jobs, $direct], queue: 'orders'); + $inserts = array_filter($connection->getQueryLog(), static fn (array $query): bool => str_starts_with($query['query'], 'insert into')); + } finally { + $connection->disableQueryLog(); + } + + $this->assertCount(1, $inserts); + $this->assertSame(100, DB::table('job_outbox')->count()); + $this->assertSame(['orders'], DB::table('job_outbox')->distinct()->pluck('queue')->all()); + } + + /** + * Cover the first-party and custom users of the shared bulk implementation. + */ + public static function baseBulkDrivers(): array + { + return [[true], [false]]; + } + + public function testBulkPreparationFailureStoresNothingAndTerminatesEveryMember(): void + { + $queue = $this->queue(); + $failure = new RuntimeException('Finalization failed'); + $finalized = 0; + $failed = []; + $this->app->make('events')->listen(JobPayloadFinalizing::class, function () use (&$finalized, $failure): void { + if (++$finalized === 2) { + throw $failure; + } + }); + $this->app->make('events')->listen(JobQueueingFailed::class, function (JobQueueingFailed $event) use (&$failed): void { + $failed[] = $event->job; + }); + $jobs = [(new OutboxDispatchJob)->viaOutbox(), (new OutboxDispatchJob)->viaOutbox(), (new OutboxDispatchJob)->viaOutbox()]; + + try { + $queue->bulk($jobs); + $this->fail('Expected finalization to fail.'); + } catch (RuntimeException $exception) { + $this->assertSame($failure, $exception); + } + + $this->assertSame($jobs, $failed); + $this->assertSame(0, DB::table('job_outbox')->count()); + } + + public function testAcceptedBulkIsNotFailedWhenItsQueuedListenerThrows(): void + { + $queue = $this->queue(); + $failure = new RuntimeException('Observer failed'); + $notified = 0; + $failed = 0; + $this->app->make('events')->listen(JobQueued::class, function () use (&$notified, $failure): void { + ++$notified; + throw $failure; + }); + $this->app->make('events')->listen(JobQueueingFailed::class, function () use (&$failed): void { + ++$failed; + }); + + try { + $queue->bulk([(new OutboxDispatchJob)->viaOutbox(), (new OutboxDispatchJob)->viaOutbox()]); + $this->fail('Expected the observer error.'); + } catch (RuntimeException $exception) { + $this->assertSame($failure, $exception); + } + + $this->assertSame(2, $notified); + $this->assertSame(0, $failed); + $this->assertSame(2, DB::table('job_outbox')->count()); + } + + #[DataProvider('deliveryTimes')] + public function testStagingRetainsRequestedTimersWithoutPublishingBeforeTheyAreDue(string $kind, ?int $offset): void + { + Date::setTestNow('2040-01-01 12:00:00.500000'); + $queue = $this->queue(); + $delay = match ($kind) { + 'none' => null, + 'zero' => 0, + 'past' => Date::now()->subSecond(), + 'future' => Date::now()->addMilliseconds(250), + }; + $job = (new OutboxDispatchJob)->viaOutbox(); + $delay === null ? $queue->push($job) : $queue->later($delay, $job); + $row = DB::table('job_outbox')->sole(); + + $this->assertSame($offset === null ? null : Date::now()->getTimestamp() + $offset, $row->deliver_at === null ? null : (int) $row->deliver_at); + $this->assertSame(Date::now()->getTimestamp(), (int) $row->available_at); + } + + /** + * Provide timer selection and absolute deadline normalization cases. + */ + public static function deliveryTimes(): array + { + return [['none', null], ['zero', null], ['past', 0], ['future', 1]]; + } + + #[DataProvider('sqsDelays')] + public function testSqsStagingCapturesOptionsOnceAndWaitsForItsTransportWindow(string $method, bool $fifo, int $eligibleDelay): void + { + Date::setTestNow('2040-01-01 12:00:00'); + $queue = new SqsQueue(m::mock(SqsClient::class), $fifo ? 'orders.fifo' : 'orders'); + $queue->setContainer($this->app)->setConnectionName('sqs'); + $job = (new OutboxFifoDispatchJob)->viaOutbox()->onGroup('tenant:17')->delay(1200); + + $method === 'bulk' ? $queue->bulk([$job]) : $queue->later(1200, $job); + $row = DB::table('job_outbox')->sole(); + $options = json_decode($row->options, true)['transport']['sqs']; + + $this->assertSame(Date::now()->getTimestamp() + 1200, (int) $row->deliver_at); + $this->assertSame(Date::now()->getTimestamp() + $eligibleDelay, (int) $row->available_at); + $this->assertSame('tenant:17', $options['MessageGroupId']); + $this->assertArrayNotHasKey('DelaySeconds', $options); + $this->assertSame($fifo ? 1 : 0, $job->deduplications); + + if ($fifo) { + $this->assertSame('captured-' . $row->uuid, $options['MessageDeduplicationId']); + } + } + + /** + * Exercise both SQS entry points and their distinct eligibility rules. + */ + public static function sqsDelays(): array + { + return [['later', false, 300], ['bulk', false, 300], ['later', true, 1200], ['bulk', true, 1200]]; + } + + public function testInvalidFinalUuidFailsBeforeStorage(): void + { + $this->app->make('events')->listen(JobPayloadFinalizing::class, function (JobPayloadFinalizing $event): void { + $event->payload = '{"uuid":"not-a-uuid"}'; + }); + + try { + $this->queue()->push((new OutboxDispatchJob)->viaOutbox()); + $this->fail('Expected an invalid payload.'); + } catch (InvalidPayloadException $exception) { + $this->assertSame('The outbox payload must contain a valid job UUID.', $exception->getMessage()); + } + + $this->assertSame(0, DB::table('job_outbox')->count()); + } + + public function testDatabaseFastPathResolutionFailureEmitsTheFailedLifecycle(): void + { + $queue = new DatabaseQueue($this->app->make(ConnectionResolverInterface::class), null, 'jobs'); + $queue->setContainer($this->app)->setConnectionName('database'); + $job = (new OutboxDispatchJob)->onOutboxDatabase('missing-owner'); + $failed = []; + $this->app->make('events')->listen(JobQueueingFailed::class, function (JobQueueingFailed $event) use (&$failed): void { + $failed[] = $event; + }); + + DB::beginTransaction(); + + try { + $queue->push($job); + $this->fail('Expected the invalid database configuration to fail.'); + } catch (InvalidArgumentException $exception) { + $this->assertCount(1, $failed); + $this->assertSame($job, $failed[0]->job); + $this->assertSame($exception, $failed[0]->exception); + } finally { + DB::rollBack(); + } + + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + $this->assertSame(0, DB::table('job_outbox')->count()); + } + + /** + * Create a Redis queue that records direct publication without network access. + */ + protected function queue(): RecordingOutboxRedisQueue + { + $queue = new RecordingOutboxRedisQueue(m::mock(Factory::class), 'orders'); + $queue->setContainer($this->app)->setConnectionName('redis'); + + return $queue; + } +} + +class OutboxDispatchJob +{ + use Queueable; +} + +class MarkedOutboxDispatchJob extends OutboxDispatchJob implements ShouldQueueViaOutbox +{ +} + +class OutboxFifoDispatchJob extends OutboxDispatchJob +{ + public int $deduplications = 0; + + /** + * Capture the finalized UUID while the original job is available. + */ + public function deduplicationId(string $payload, string $queue): string + { + ++$this->deduplications; + + return 'captured-' . json_decode($payload, true)['uuid']; + } +} + +class RecordingOutboxRedisQueue extends RedisQueue +{ + public array $published = []; + + /** + * Record a direct raw publication. + */ + public function pushRaw(string $payload, UnitEnum|string|null $queue = null, array $options = []): mixed + { + $this->published[] = $payload; + + return json_decode($payload, true)['uuid']; + } +} diff --git a/tests/Queue/Outbox/OutboxDispatcherTest.php b/tests/Queue/Outbox/OutboxDispatcherTest.php new file mode 100644 index 0000000000..4ca4112bd7 --- /dev/null +++ b/tests/Queue/Outbox/OutboxDispatcherTest.php @@ -0,0 +1,95 @@ + $base]); + $connection->setTransactionManager($transactions); + + if ($role !== null) { + $connection->setReadWriteType($role); + } + + $connections[$name] = $connection; + } + + $resolver = m::mock(ConnectionResolverInterface::class); + $resolver->shouldReceive('connection')->andReturnUsing(fn (string $name): SQLiteConnection => $connections[$name]); + $repository = new DatabaseOutboxRepository($resolver, $configured, 'job_outbox', 300); + $dispatcher = new OutboxDispatcher($repository, $resolver, $transactions); + $job = new OutboxOwnershipJob; + + if ($explicit !== null) { + $job->onOutboxDatabase($explicit); + } + + try { + foreach ($active as $name) { + $connections[$name]->beginTransaction(); + } + + if ($error !== null) { + $this->expectException(LogicException::class); + $this->expectExceptionMessage($error); + } + + $owner = $dispatcher->databaseFor($job); + + $this->assertSame($connections[$expected], $owner); + } finally { + foreach ($connections as $connection) { + if ($connection->transactionLevel() > 0) { + $connection->rollBack(0); + } + } + } + } + + /** + * Provide distinct owner sessions and realistic selection conflicts. + */ + public static function ownershipCases(): array + { + return [ + 'configured default without transaction' => [[], null, ['business'], 'business', null], + 'explicit configured source without transaction' => [[], 'secondary', ['business', 'secondary'], 'secondary', null], + 'single active source overrides default' => [['secondary'], null, ['business', 'secondary'], 'secondary', null], + 'nested transactions retain one owner' => [['business', 'business'], null, ['business'], 'business', null], + 'writer is covered by base source' => [['business::write'], null, ['business'], 'business::write', null], + 'explicit writer selects one of two sessions' => [['business', 'business::write'], 'business::write', ['business'], 'business::write', null], + 'distinct sessions require explicit owner' => [['business', 'business::write'], null, ['business'], null, 'onOutboxDatabase()'], + 'explicit owner must participate' => [['business'], 'secondary', ['business', 'secondary'], null, 'does not own an active transaction'], + 'active source must be drainable' => [['secondary'], null, ['business'], null, 'not covered by queue.outbox.databases'], + 'active read session rejected' => [['business::read'], null, ['business'], null, 'is a read connection'], + 'explicit read session rejected' => [[], 'business::read', ['business'], null, 'is a read connection'], + ]; + } +} + +class OutboxOwnershipJob +{ + use Queueable; +} diff --git a/tests/Queue/Outbox/OutboxFastPathTest.php b/tests/Queue/Outbox/OutboxFastPathTest.php new file mode 100644 index 0000000000..4b43964c97 --- /dev/null +++ b/tests/Queue/Outbox/OutboxFastPathTest.php @@ -0,0 +1,383 @@ +make('config'); + $config->set('database.default', 'source'); + $config->set('database.connections.source', ['driver' => 'sqlite', 'database' => ':memory:']); + $config->set('database.connections.secondary', ['driver' => 'sqlite', 'database' => ':memory:']); + $config->set('queue.outbox.databases', ['source', 'secondary']); + $config->set('queue.connections.database', [ + 'driver' => 'database', 'connection' => 'source', 'table' => 'jobs', + 'queue' => 'default', 'after_commit' => true, 'outbox' => true, + ]); + } + + public function testLocalSelectionWithoutATransactionPreservesImmediateResults(): void + { + $queue = $this->app->make('queue')->connection('sync'); + + $this->assertSame(0, $queue->push(new OutboxLocalJob)); + $this->assertSame(0, $queue->push((new OutboxLocalJob)->viaOutbox())); + $this->assertSame(0, $queue->later(0, new OutboxLocalJob)); + $this->assertSame(0, $queue->later(0, (new OutboxLocalJob)->viaOutbox())); + $this->assertSame(4, $this->app->make(OutboxExecutionLog::class)->executions); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + } + + public function testLocalSelectionWithoutATransactionPreservesTimerIds(): void + { + $timer = m::mock(Timer::class); + $timer->shouldReceive('after')->with(60.0, m::type('callable'))->twice()->andReturn(41, 42); + $queue = new BackgroundQueue(timer: $timer); + $queue->setContainer($this->app)->setConnectionName('local'); + + $this->assertSame(41, $queue->later(60, new OutboxLocalJob)); + $this->assertSame(42, $queue->later(60, (new OutboxLocalJob)->viaOutbox())); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + } + + #[DataProvider('transactionOutcomes')] + public function testLocalSelectionHonorsCommitAndRollbackWithoutSchema(bool $commit): void + { + $log = $this->app->make(OutboxExecutionLog::class); + $queue = $this->app->make('queue')->connection('sync'); + DB::beginTransaction(); + $queue->push((new OutboxLocalJob)->viaOutbox()->beforeCommit()); + $this->assertSame(0, $log->executions); + + $commit ? DB::commit() : DB::rollBack(); + + $this->assertSame($commit ? 1 : 0, $log->executions); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + } + + public function testLocalExplicitOwnerDoesNotWaitForAnUnrelatedTransaction(): void + { + $log = $this->app->make(OutboxExecutionLog::class); + DB::beginTransaction(); + DB::connection('secondary')->beginTransaction(); + $this->app->make('queue')->connection('sync')->push((new OutboxLocalJob)->onOutboxDatabase('secondary')); + + DB::connection('secondary')->commit(); + $this->assertSame(1, $log->executions); + DB::rollBack(); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + } + + public function testLocalExplicitOwnerRejectsAnUnrelatedActiveTransaction(): void + { + DB::beginTransaction(); + + try { + $this->expectException(LogicException::class); + $this->expectExceptionMessage('does not own an active transaction'); + $this->app->make('queue')->connection('sync')->push((new OutboxLocalJob)->onOutboxDatabase('secondary')); + } finally { + DB::rollBack(); + } + } + + public function testNullBulkAndQueueFakeDoNotResolveOutboxStorage(): void + { + $jobs = [new MarkedOutboxLocalJob, (new OutboxLocalJob)->viaOutbox()]; + $this->app->make('queue')->connection('null')->bulk($jobs); + $fake = new QueueFake($this->app); + $fake->bulk($jobs); + + $fake->assertPushed(OutboxLocalJob::class); + $fake->assertPushed(MarkedOutboxLocalJob::class); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + $this->assertSame(0, $this->app->make(OutboxExecutionLog::class)->executions); + } + + public function testDatabaseQueueUsesItsExistingAtomicStorageAndIgnoresAfterCommitDefault(): void + { + (require dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000006_testbench_create_jobs_table.php')->up(); + $queue = $this->app->make('queue')->connection('database'); + $id = $queue->push(new OutboxLocalJob); + $this->assertIsInt($id); + $this->assertSame(1, DB::table('jobs')->count()); + + DB::beginTransaction(); + $queue->bulk([new OutboxLocalJob, (new OutboxLocalJob)->onOutboxDatabase('source::write')]); + $this->assertSame(3, DB::table('jobs')->count()); + DB::rollBack(); + + $this->assertSame(1, DB::table('jobs')->count()); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + } + + public function testDatabaseQueueOnAnotherOwnerStagesInsideTheBusinessTransaction(): void + { + (require dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php')->up(); + config(['queue.connections.database.connection' => 'secondary']); + $queue = $this->app->make('queue')->connection('database'); + + DB::beginTransaction(); + $id = $queue->push(new OutboxLocalJob); + $this->assertSame($id, DB::table('job_outbox')->sole()->uuid); + DB::rollBack(); + $this->assertSame(0, DB::table('job_outbox')->count()); + } + + public function testFailoverSuppressesChildDefaultsAndRestoresThemAfterNestedFailure(): void + { + $primary = new FailingOutboxRedisQueue(m::mock(Factory::class), 'orders'); + $primary->setContainer($this->app)->setConnectionName('primary')->setConfig(['outbox' => true]); + $sync = $this->app->make('queue')->connection('sync'); + $manager = m::mock(QueueManager::class); + $manager->shouldReceive('connection')->with('primary')->andReturn($primary); + $manager->shouldReceive('connection')->with('sync')->andReturn($sync); + $inner = new FailoverQueue($manager, $this->app->make('events'), ['primary']); + $inner->setContainer($this->app)->setConnectionName('inner'); + $manager->shouldReceive('connection')->with('inner')->andReturn($inner); + $outer = new FailoverQueue($manager, $this->app->make('events'), ['inner', 'sync']); + $outer->setContainer($this->app)->setConnectionName('outer'); + + $outer->push(new OutboxLocalJob); + + $this->assertSame(1, $primary->attempts); + $this->assertSame(1, $this->app->make(OutboxExecutionLog::class)->executions); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + + (require dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php')->up(); + $primary->push(new OutboxLocalJob); + $this->assertSame(1, DB::table('job_outbox')->count()); + $this->assertSame(1, $primary->attempts); + } + + #[DataProvider('transactionOutcomes')] + public function testSelectedFailoverDefersTheWholeAttemptWithoutSchema(bool $commit): void + { + $manager = m::mock(QueueManager::class); + $manager->shouldReceive('connection')->with('sync')->andReturn($this->app->make('queue')->connection('sync')); + $queue = new FailoverQueue($manager, $this->app->make('events'), ['sync']); + $queue->setContainer($this->app)->setConnectionName('failover')->setConfig(['outbox' => true]); + $log = $this->app->make(OutboxExecutionLog::class); + DB::beginTransaction(); + $queue->push(new OutboxLocalJob); + $this->assertSame(0, $log->executions); + $commit ? DB::commit() : DB::rollBack(); + $this->assertSame($commit ? 1 : 0, $log->executions); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + } + + /** + * Provide the two terminal transaction outcomes. + */ + public static function transactionOutcomes(): array + { + return ['commit' => [true], 'rollback' => [false]]; + } + + public function testNestedFailoverPublishesAtItsExplicitOwnersCommitAndPreservesLocksAcrossFailure(): void + { + $failure = new RuntimeException('Execution could not start.'); + $first = new class($failure) extends SyncQueue { + /** + * Configure a failing after-commit child. + */ + public function __construct(private RuntimeException $failure) + { + parent::__construct(true); + } + + /** + * Fail before accepting the dispatch. + */ + protected function executePayload(string $payload, ?string $queue = null): int + { + throw $this->failure; + } + }; + $first->setContainer($this->app)->setConnectionName('first'); + $target = new FastPathRecordingRedisQueue(m::mock(Factory::class), dispatchAfterCommit: true); + $target->setContainer($this->app)->setConnectionName('target')->setConfig(['outbox' => true]); + $manager = m::mock(QueueManager::class); + $manager->shouldReceive('connection')->with('first')->andReturn($first); + $manager->shouldReceive('connection')->with('target')->andReturn($target); + $inner = new FailoverQueue($manager, $this->app->make('events'), ['first', 'target'], true); + $inner->setContainer($this->app)->setConnectionName('inner'); + $manager->shouldReceive('connection')->with('inner')->andReturn($inner); + $outer = new FailoverQueue($manager, $this->app->make('events'), ['inner']); + $outer->setContainer($this->app)->setConnectionName('outer'); + $job = (new UniqueOutboxLocalJob)->onOutboxDatabase('secondary')->afterCommit(); + $lock = new UniqueLock(new Repository(new ArrayStore)); + $this->assertTrue($lock->acquireForDispatch($job)); + $this->app->make('events')->listen(JobQueueing::class, function (JobQueueing $event) use ($job, $lock): void { + $this->assertSame($job, $event->job); + $this->assertTrue(DispatchLockContext::has($job)); + $this->assertFalse($lock->acquire($job)); + }); + + DB::beginTransaction(); + DB::connection('secondary')->beginTransaction(); + $this->assertNull($outer->push($job)); + $this->assertSame(0, $target->publications); + DB::connection('secondary')->commit(); + + $this->assertSame(1, $target->publications); + $this->assertFalse(DispatchLockContext::has($job)); + DB::rollBack(); + $this->assertFalse($lock->acquire($job)); + $this->assertFalse($this->app->resolved(OutboxDispatcher::class)); + } + + #[DataProvider('continuationKinds')] + public function testSyncFallbackDoesNotSuppressIndependentContinuation(string $kind): void + { + (require dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php')->up(); + $target = new FastPathRecordingRedisQueue(m::mock(Factory::class)); + $target->setContainer($this->app)->setConnectionName('target')->setConfig(['outbox' => true]); + $this->app->instance(RedisQueue::class, $target); + $manager = m::mock(QueueManager::class); + $manager->shouldReceive('connection')->with('sync')->andReturn($this->app->make('queue')->connection('sync')); + $failover = new FailoverQueue($manager, $this->app->make('events'), ['sync']); + $failover->setContainer($this->app)->setConnectionName('failover'); + + $failover->push( + $kind === 'object' ? new OutboxDispatchesContinuation : OutboxStringContinuation::class . '@fire', + ['chunk' => 1], + ); + + $row = DB::table('job_outbox')->sole(); + $this->assertSame('target', $row->connection); + $this->assertSame(0, $target->publications); + + if ($kind === 'string') { + $payload = json_decode($row->payload, true); + $this->assertSame(OutboxStringContinuation::class . '@fire', $payload['job']); + $this->assertSame(['chunk' => 2], $payload['data']); + } + } + + /** + * Provide supported object and string continuation dispatches. + */ + public static function continuationKinds(): array + { + return ['object' => ['object'], 'string' => ['string']]; + } +} + +class OutboxLocalJob +{ + use Queueable; + + /** + * Record execution through the real synchronous job handler. + */ + public function handle(OutboxExecutionLog $log): void + { + ++$log->executions; + } +} + +class MarkedOutboxLocalJob extends OutboxLocalJob implements ShouldQueueViaOutbox +{ +} + +class UniqueOutboxLocalJob extends OutboxLocalJob implements ShouldBeUnique +{ +} + +class OutboxDispatchesContinuation +{ + /** + * Dispatch an independently selected job from the running handler. + */ + public function handle(RedisQueue $queue): void + { + $queue->push((new OutboxLocalJob)->viaOutbox()); + } +} + +class OutboxStringContinuation +{ + /** + * Resolve the destination for subsequent chunks. + */ + public function __construct(private RedisQueue $queue) + { + } + + /** + * Queue the next chunk using the same string handler. + */ + public function fire(Job $job, array $data): void + { + $this->queue->push(self::class . '@fire', ['chunk' => $data['chunk'] + 1]); + } +} + +class FastPathRecordingRedisQueue extends RedisQueue +{ + public int $publications = 0; + + /** + * Record direct transport acceptance without an external Redis service. + */ + public function pushRaw(string $payload, UnitEnum|string|null $queue = null, array $options = []): mixed + { + ++$this->publications; + + return 'accepted'; + } +} + +class OutboxExecutionLog +{ + public int $executions = 0; +} + +class FailingOutboxRedisQueue extends RedisQueue +{ + public int $attempts = 0; + + /** + * Reject direct publication to exercise the ordinary failover path. + */ + public function pushRaw(string $payload, UnitEnum|string|null $queue = null, array $options = []): mixed + { + ++$this->attempts; + + throw new RuntimeException('Broker unavailable.'); + } +} diff --git a/tests/Queue/Outbox/OutboxProducerTest.php b/tests/Queue/Outbox/OutboxProducerTest.php new file mode 100644 index 0000000000..edd73f124c --- /dev/null +++ b/tests/Queue/Outbox/OutboxProducerTest.php @@ -0,0 +1,150 @@ +make('config'); + $config->set('queue.default', 'redis'); + $config->set('queue.outbox.databases', [$config->string('database.default')]); + } + + /** + * Load only the shipped outbox schema. + */ + protected function migrateFreshUsing(): array + { + return [ + '--realpath' => true, + '--path' => dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php', + ]; + } + + public function testPublicProducerApisStageTheirExistingWrapperJobs(): void + { + $this->app->instance('redis', m::mock(Factory::class)); + Event::listen('outbox.created', OutboxProducerListener::class); + + DB::transaction(static function (): void { + Event::dispatch('outbox.created', ['order-17']); + Mail::to('customer@example.com')->send(new OutboxProducerMailable); + NotificationFacade::route('mail', 'customer@example.com')->notify(new OutboxProducerNotification); + Event::dispatch(new OutboxProducerBroadcast('order-17')); + }); + + $rows = DB::table('job_outbox')->orderBy('id')->get(); + $jobs = $rows->map(static function (object $row): object { + $payload = json_decode($row->payload, true, flags: JSON_THROW_ON_ERROR); + + return unserialize($payload['data']['command']); + }); + + $this->assertSame([ + CallQueuedListener::class, + SendQueuedMailable::class, + SendQueuedNotifications::class, + BroadcastEvent::class, + ], $jobs->map(static fn (object $job): string => $job::class)->all()); + $this->assertSame(['order-17'], $jobs[0]->data); + $this->assertSame('customer@example.com', $jobs[1]->mailable->to[0]['address']); + $this->assertSame('customer@example.com', $jobs[2]->notifiables->first()->routes['mail']); + $this->assertSame('order-17', $jobs[3]->event->order); + $this->assertSame(['redis'], $rows->pluck('connection')->unique()->all()); + } +} + +class OutboxProducerListener implements ShouldQueueViaOutbox +{ + /** + * Handle the queued event. + */ + public function handle(string $order): void + { + } +} + +class OutboxProducerMailable extends Mailable implements ShouldQueueViaOutbox +{ + use Queueable; + + /** + * Build a mail message without an external template dependency. + */ + public function build(): static + { + return $this->subject('Order received')->html('Your order was received.'); + } +} + +class OutboxProducerNotification extends Notification implements ShouldQueueViaOutbox +{ + use Queueable; + + /** + * Select the queued mail notification channel. + */ + public function via(AnonymousNotifiable $notifiable): array + { + return ['mail']; + } + + /** + * Build the notification mail message. + */ + public function toMail(AnonymousNotifiable $notifiable): MailMessage + { + return (new MailMessage)->line('Your order was received.'); + } +} + +class OutboxProducerBroadcast implements ShouldBroadcast, ShouldQueueViaOutbox +{ + use Queueable; + + /** + * Identify the order in the broadcast payload. + */ + public function __construct(public string $order) + { + } + + /** + * Select the broadcast channel. + */ + public function broadcastOn(): array + { + return ['orders']; + } +} diff --git a/tests/Queue/Outbox/OutboxRelayTest.php b/tests/Queue/Outbox/OutboxRelayTest.php new file mode 100644 index 0000000000..dd935d07a1 --- /dev/null +++ b/tests/Queue/Outbox/OutboxRelayTest.php @@ -0,0 +1,290 @@ +message(1)]; + $phases = []; + $repository->expects('claim')->with('primary', 100)->andReturnUsing(function () use (&$phases, $messages): array { + $this->assertFalse(CoroutineContext::has('request-only')); + $phases[] = 'claim'; + Coroutine::defer(static function () use (&$phases): void { $phases[] = 'claim released'; }); + + return $messages; + }); + $queues->expects('connection')->with('redis')->andReturn($queue); + $queue->expects('publishOutboxMessages')->with($messages)->andReturnUsing(function () use (&$phases): OutboxBatchResult { + $this->assertSame(['claim', 'claim released'], $phases); + $this->assertFalse(CoroutineContext::has('request-only')); + $phases[] = 'publish'; + Coroutine::defer(static function () use (&$phases): void { $phases[] = 'publish released'; }); + + // Drivers may preserve keys when filtering a batch's row IDs. + return new OutboxBatchResult([7 => 1]); + }); + $repository->expects('settle')->andReturnUsing(function () use (&$phases): int { + $this->assertSame('publish released', end($phases)); + $phases[] = 'settle'; + Coroutine::defer(static function () use (&$phases): void { $phases[] = 'settle released'; }); + + return 1; + }); + $events->listen(OutboxBatchDelivered::class, function (OutboxBatchDelivered $event) use (&$phases): void { + $this->assertSame('settle released', end($phases)); + $this->assertFalse(CoroutineContext::has('request-only')); + $this->assertSame('primary', $event->database); + $this->assertSame(1, $event->accepted); + $this->assertSame(20, $event->oldestEligibleAge); + Coroutine::defer(static function () use (&$phases): void { $phases[] = 'observer released'; }); + }); + $relay = new OutboxRelay($repository, $queues, $events, $exceptions); + + $this->assertTrue($relay->deliver('primary', $relay->claim('primary', 100))); + $this->assertSame('observer released', end($phases)); + } + + public function testPartialResultsSettleBeforeAnotherConnectionAndReportingCannotBlockItsPublication(): void + { + $repository = m::mock(DatabaseOutboxRepository::class); + $queues = m::mock(Factory::class); + $first = m::mock(Queue::class . ', ' . PublishesOutboxMessages::class); + $second = m::mock(Queue::class . ', ' . PublishesOutboxMessages::class); + $events = new Dispatcher; + $exceptions = m::mock(ExceptionHandler::class); + $messages = [$this->message(1), $this->message(2), $this->message(3, 'other')]; + $failure = new RuntimeException('Second publication rejected.'); + $settled = []; + $queues->expects('connection')->with('redis')->andReturn($first); + $queues->expects('connection')->with('other')->andReturn($second); + $first->expects('publishOutboxMessages')->andReturn(new OutboxBatchResult([1], failure: $failure)); + $second->expects('publishOutboxMessages')->andReturnUsing(function () use (&$settled): OutboxBatchResult { + $this->assertSame([[1]], $settled); + + return new OutboxBatchResult([3]); + }); + $repository->shouldReceive('settle')->twice()->andReturnUsing(function (string $database, array $group, OutboxBatchResult $result) use (&$settled): int { + $settled[] = $result->accepted; + + return count($group); + }); + $events->listen(OutboxBatchDeliveryFailed::class, function (OutboxBatchDeliveryFailed $event) use ($failure): void { + $this->assertSame(1, $event->accepted); + $this->assertSame(1, $event->failed); + $this->assertSame($failure, $event->exception); + }); + $exceptions->expects('report')->with($failure, m::on(function (array $context) use (&$settled): bool { + $this->assertSame([[1], [3]], $settled); + $this->assertSame([1, 2], $context['outbox_ids']); + $this->assertSame(['SendInvoice', 'SendInvoice'], $context['jobs']); + + return true; + })); + + $this->assertFalse((new OutboxRelay($repository, $queues, $events, $exceptions))->deliver('primary', $messages)); + } + + #[DataProvider('invalidResults')] + public function testInvalidResultsReleaseEveryInputAsFailure(OutboxBatchResult $invalid): void + { + $repository = m::mock(DatabaseOutboxRepository::class); + $queues = m::mock(Factory::class); + $queue = m::mock(Queue::class . ', ' . PublishesOutboxMessages::class); + $exceptions = m::mock(ExceptionHandler::class); + $queues->expects('connection')->andReturn($queue); + $queue->expects('publishOutboxMessages')->andReturn($invalid); + $repository->expects('settle')->with('primary', m::type('array'), m::on(function (OutboxBatchResult $result): bool { + $this->assertSame([], $result->accepted); + $this->assertSame([], $result->deferred); + $this->assertInstanceOf(LogicException::class, $result->failure); + + return true; + }))->andReturn(2); + $exceptions->expects('report')->with(m::type(LogicException::class), m::type('array')); + + $this->assertFalse((new OutboxRelay($repository, $queues, new Dispatcher, $exceptions))->deliver('primary', [$this->message(1), $this->message(2)])); + } + + /** + * Provide invalid driver accounting without repeating transport tests. + */ + public static function invalidResults(): array + { + return [ + 'foreign ID' => [new OutboxBatchResult([1, 3])], + 'duplicate ID' => [new OutboxBatchResult([1, 1])], + 'overlap' => [new OutboxBatchResult([1, 2], [2])], + 'missing row without failure' => [new OutboxBatchResult([1])], + 'non-integer ID' => [new OutboxBatchResult(['1', 2])], + 'defer immediate job' => [new OutboxBatchResult([1], [2])], + ]; + } + + public function testGenericFallbackDefersFutureRowsWithoutEventsAndSettlesThemNormally(): void + { + $repository = m::mock(DatabaseOutboxRepository::class); + $queues = m::mock(Factory::class); + $queue = m::mock(Queue::class); + $exceptions = m::mock(ExceptionHandler::class); + $events = new Dispatcher; + $events->listen(OutboxBatchDelivered::class, fn () => $this->fail('Deferred rows were not delivered.')); + $events->listen(OutboxBatchDeliveryFailed::class, fn () => $this->fail('Deferral is not failure.')); + $queues->expects('connection')->andReturn($queue); + $repository->expects('settle')->with('primary', m::type('array'), m::on(fn (OutboxBatchResult $result): bool => $result->accepted === [] && $result->deferred === [1] && $result->failure === null))->andReturn(1); + + $this->assertTrue((new OutboxRelay($repository, $queues, $events, $exceptions))->deliver('primary', [$this->message(1, deliverAt: CarbonImmutable::now()->getTimestamp() + 60)])); + } + + public function testPublicationAndSettlementErrorsAreBothReportedWithPublicationOnTheEvent(): void + { + $repository = m::mock(DatabaseOutboxRepository::class); + $queues = m::mock(Factory::class); + $queue = m::mock(Queue::class); + $exceptions = m::mock(ExceptionHandler::class); + $events = new Dispatcher; + $publication = new RuntimeException('Broker unavailable.'); + $settlement = new RuntimeException('Database unavailable.'); + $queues->expects('connection')->andReturn($queue); + $queue->expects('pushRaw')->andThrow($publication); + $repository->expects('settle')->andThrow($settlement); + $exceptions->expects('report')->with($publication, m::type('array')); + $exceptions->expects('report')->with($settlement, m::type('array')); + $events->listen(OutboxBatchDeliveryFailed::class, function (OutboxBatchDeliveryFailed $event) use ($publication): void { + $this->assertSame($publication, $event->exception); + }); + + $this->assertFalse((new OutboxRelay($repository, $queues, $events, $exceptions))->deliver('primary', [$this->message(1)])); + } + + public function testLostSettlementOwnershipReportsConfirmedDeliveryWithoutRepublishing(): void + { + $repository = m::mock(DatabaseOutboxRepository::class); + $queues = m::mock(Factory::class); + $queue = m::mock(Queue::class); + $exceptions = m::mock(ExceptionHandler::class); + $events = new Dispatcher; + $queues->expects('connection')->andReturn($queue); + $queue->expects('pushRaw')->andReturn('accepted'); + $repository->expects('settle')->andReturn(0); + $events->listen(OutboxBatchDelivered::class, fn () => $this->fail('Lost ownership is a settlement failure.')); + $events->listen(OutboxBatchDeliveryFailed::class, function (OutboxBatchDeliveryFailed $event): void { + $this->assertSame(1, $event->accepted); + $this->assertSame(0, $event->failed); + $this->assertStringContainsString('lease ownership changed', $event->exception->getMessage()); + }); + $exceptions->expects('report')->with(m::type(RuntimeException::class), m::type('array')); + + $this->assertFalse((new OutboxRelay($repository, $queues, $events, $exceptions))->deliver('primary', [$this->message(1)])); + } + + #[DataProvider('reportingOutcomes')] + public function testListenerFailureDoesNotThrottleDeliveryAndReporterFailureEscapes(bool $reporterFails): void + { + $repository = m::mock(DatabaseOutboxRepository::class); + $queues = m::mock(Factory::class); + $queue = m::mock(Queue::class); + $exceptions = m::mock(ExceptionHandler::class); + $events = new Dispatcher; + $listenerFailure = new RuntimeException('Listener failed.'); + $reporterFailure = new RuntimeException('Reporter failed.'); + $released = false; + $queues->expects('connection')->andReturn($queue); + $queue->expects('pushRaw')->andReturn('accepted'); + $repository->expects('settle')->andReturn(1); + $events->listen(OutboxBatchDelivered::class, static function () use ($listenerFailure, &$released): void { + Coroutine::defer(static function () use (&$released): void { $released = true; }); + throw $listenerFailure; + }); + $events->listen(OutboxBatchDeliveryFailed::class, fn () => $this->fail('Listener failure must not change the delivery event.')); + $report = $exceptions->expects('report')->with($listenerFailure, m::type('array')); + + if ($reporterFails) { + $report->andThrow($reporterFailure); + } + + try { + $this->assertTrue((new OutboxRelay($repository, $queues, $events, $exceptions))->deliver('primary', [$this->message(1)])); + $this->assertFalse($reporterFails, 'Reporter failure must reach the worker.'); + } catch (RuntimeException $exception) { + $this->assertTrue($reporterFails); + $this->assertSame($reporterFailure, $exception); + } + + $this->assertTrue($released); + } + + /** + * Exercise observer reporting with a healthy and failed exception handler. + */ + public static function reportingOutcomes(): array + { + return [[false], [true]]; + } + + public function testIndependentPhaseCancellationReleasesResourcesAndStopsWithoutSettlement(): void + { + $repository = m::mock(DatabaseOutboxRepository::class); + $queues = m::mock(Factory::class); + $queue = m::mock(Queue::class); + $exceptions = m::mock(ExceptionHandler::class); + $cancellation = new CanceledException; + $released = false; + $queues->expects('connection')->andReturn($queue); + $queue->expects('pushRaw')->andReturnUsing(static function () use ($cancellation, &$released): never { + Coroutine::defer(static function () use (&$released): void { $released = true; }); + throw $cancellation; + }); + + try { + (new OutboxRelay($repository, $queues, new Dispatcher, $exceptions))->deliver('primary', [$this->message(1), $this->message(2, 'other')]); + $this->fail('Child cancellation must stop delivery.'); + } catch (ChildCancellationException $exception) { + $this->assertSame($cancellation, $exception->getPrevious()); + } + + $this->assertTrue($released); + } + + /** + * Build a message whose age and diagnostics are visible to observers. + */ + protected function message(int $id, string $connection = 'redis', ?int $deliverAt = null): OutboxMessage + { + $now = CarbonImmutable::now()->getTimestamp(); + + return new OutboxMessage($id, "job-{$id}", 'lease', 1, $connection, 'orders', '{"displayName":"SendInvoice"}', [], $deliverAt, $now - 20, $now - 30); + } +} diff --git a/tests/Queue/Outbox/OutboxTransactionTest.php b/tests/Queue/Outbox/OutboxTransactionTest.php new file mode 100644 index 0000000000..9007304eee --- /dev/null +++ b/tests/Queue/Outbox/OutboxTransactionTest.php @@ -0,0 +1,180 @@ +queue(); + $firstUuid = null; + $finalized = 0; + $events->listen(JobPayloadFinalizing::class, function (JobPayloadFinalizing $event) use (&$firstUuid, &$finalized): void { + $payload = json_decode($event->payload, true); + $firstUuid ??= $payload['uuid']; + + if (++$finalized === 3) { + $payload['uuid'] = $firstUuid; + $event->payload = json_encode($payload); + } + }); + + for ($level = 0; $level < $depth; ++$level) { + $database->beginTransaction(); + } + + try { + $queue->bulk([ + (new OutboxTransactionJob)->viaOutbox(), + (new OutboxTransactionJob)->viaOutbox(), + (new OutboxTransactionJob)->viaOutbox(), + ]); + $this->fail('The duplicate UUID must reject the complete owner group.'); + } catch (QueryException $exception) { + $this->assertStringContainsString('UNIQUE constraint failed', $exception->getMessage()); + $this->assertSame($depth, $database->transactionLevel()); + $this->assertSame(0, $database->table('job_outbox')->count()); + } + + while ($database->transactionLevel() > 0) { + $database->commit(); + } + + $this->assertSame(0, $database->table('job_outbox')->count()); + } + + /** + * Exercise both a standalone insert transaction and nested application savepoints. + */ + public static function transactionDepths(): array + { + return [[0], [1], [2]]; + } + + public function testNestedRollbackReleasesAcceptedLocksButPreservesTheOuterGroup(): void + { + [$queue, $database] = $this->queue(); + $cache = new Repository(new WorkerArrayStore); + $lock = new UniqueLock($cache); + $outer = (new OutboxTransactionJob('outer'))->viaOutbox(); + $inner = (new OutboxTransactionJob('inner'))->viaOutbox(); + + $database->beginTransaction(); + $this->assertTrue($lock->acquireForDispatch($outer)); + $queue->push($outer); + $database->beginTransaction(); + $this->assertTrue($lock->acquireForDispatch($inner)); + $queue->bulk([$inner]); + + $this->assertFalse(DispatchLockContext::has($outer)); + $this->assertFalse(DispatchLockContext::has($inner)); + $database->rollBack(); + $this->assertSame(1, $database->table('job_outbox')->count()); + $this->assertFalse($cache->lock(UniqueLock::getKey($inner))->isLocked()); + $this->assertTrue($cache->lock(UniqueLock::getKey($outer))->isLocked()); + + $database->commit(); + $this->assertSame(1, $database->table('job_outbox')->count()); + $this->assertTrue($cache->lock(UniqueLock::getKey($outer))->isLocked()); + } + + public function testOuterRollbackReleasesEveryAcceptedChunkSnapshot(): void + { + [$queue, $database] = $this->queue(); + $cache = new Repository(new WorkerArrayStore); + $lock = new UniqueLock($cache); + $jobs = array_map(static fn (int $id): OutboxTransactionJob => (new OutboxTransactionJob((string) $id))->viaOutbox(), range(1, 3)); + + $database->beginTransaction(); + + foreach ($jobs as $job) { + $this->assertTrue($lock->acquireForDispatch($job)); + } + + $queue->bulk($jobs); + $this->assertSame(3, $database->table('job_outbox')->count()); + $database->rollBack(); + $this->assertSame(0, $database->table('job_outbox')->count()); + + foreach ($jobs as $job) { + $this->assertFalse($cache->lock(UniqueLock::getKey($job))->isLocked()); + } + } + + /** + * Create an independent SQLite owner using the shipped outbox schema. + * + * @return array{RedisQueue, OutboxTransactionConnection, Dispatcher} + */ + protected function queue(): array + { + $container = new Container; + Container::setInstance($container); + $events = new Dispatcher($container); + $container->instance('events', $events); + $container->instance(EventDispatcher::class, $events); + $database = new OutboxTransactionConnection(new PDO('sqlite::memory:'), config: ['name' => 'business']); + $database->setTransactionManager($transactions = new DatabaseTransactionsManager); + $container->instance('db.schema', $database->getSchemaBuilder()); + Facade::setFacadeApplication($container); + (require dirname(__DIR__, 3) . '/src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php')->up(); + + $resolver = m::mock(ConnectionResolverInterface::class); + $resolver->shouldReceive('connection')->with('business')->andReturn($database); + $repository = new DatabaseOutboxRepository($resolver, ['business'], 'job_outbox', 300); + $container->instance(OutboxDispatcher::class, new OutboxDispatcher($repository, $resolver, $transactions)); + $queue = new RedisQueue(m::mock(Factory::class), 'orders'); + $queue->setContainer($container)->setConnectionName('redis'); + + return [$queue, $database, $events]; + } +} + +class OutboxTransactionJob +{ + use Queueable; + + /** + * Identify a distinct dispatch-owned lock. + */ + public function __construct(public string $uniqueId = '') + { + } +} + +class OutboxTransactionConnection extends SQLiteConnection +{ + /** + * Force three outbox rows through two binding-safe insert chunks. + */ + protected function resolveMaxBindings(): int + { + return 26; + } +} diff --git a/tests/Queue/Outbox/OutboxWorkerTest.php b/tests/Queue/Outbox/OutboxWorkerTest.php new file mode 100644 index 0000000000..5f7d6132f2 --- /dev/null +++ b/tests/Queue/Outbox/OutboxWorkerTest.php @@ -0,0 +1,277 @@ +worker(['primary', 'secondary']); + $releaseFirst = new Channel(1); + $running = 0; + $peak = 0; + $finished = 0; + $this->relay->expects('claim')->with('primary', 2)->andReturn([$this->message(1), $this->message(2)]); + $this->relay->expects('claim')->with('secondary', 1)->andReturn([$this->message(3)]); + $this->relay->shouldReceive('deliver')->twice()->andReturnUsing(function (string $database) use ($releaseFirst, &$running, &$peak, &$finished): bool { + $peak = max($peak, ++$running); + + try { + if ($database === 'primary') { + $this->assertTrue($releaseFirst->pop(2)); + } else { + $releaseFirst->push(true); + } + + ++$finished; + + return true; + } finally { + --$running; + } + }); + + $this->assertSame(Worker::EXIT_SUCCESS, $worker->run(null, new OutboxWorkerOptions(batch: 2, concurrency: 2, maxJobs: 3, memory: 1024))); + $this->assertSame(2, $peak); + $this->assertSame(2, $finished); + $this->assertSame(0, $running); + } + + public function testFailedSourceCoolsDownWithoutDelayingHealthySources(): void + { + Sleep::fake(); + $worker = $this->worker(['broken', 'healthy']); + $failure = new RuntimeException('Connection refused.'); + $this->relay->expects('claim')->with('broken', 1)->andThrow($failure); + $this->relay->shouldReceive('claim')->with('healthy', 1)->times(3)->andReturn([$this->message(1)]); + $this->relay->shouldReceive('deliver')->with('healthy', m::type('array'))->times(3)->andReturn(true); + $this->exceptions->expects('report')->with($failure, ['outbox_database' => 'broken']); + + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(batch: 1, sleep: 60, maxJobs: 3, memory: 1024))); + Sleep::assertNeverSlept(); + } + + public function testStopWhenEmptyTreatsFailedSourcesAsEmptyAndDoesNotRecheckAfterJoining(): void + { + Sleep::fake(); + $worker = $this->worker(['broken', 'empty']); + $failure = new RuntimeException('Missing table.'); + $this->relay->expects('claim')->with('broken', 100)->andThrow($failure); + $this->relay->expects('claim')->with('empty', 100)->andReturn([]); + $this->exceptions->expects('report')->with($failure, ['outbox_database' => 'broken']); + + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(memory: 1024, stopWhenEmpty: true))); + Sleep::assertNeverSlept(); + } + + public function testOnceVisitsEverySourceWithoutSleepingEvenWhenDeliveryFails(): void + { + Sleep::fake(); + $worker = $this->worker(['first', 'second']); + $this->relay->expects('claim')->with('first', 100)->andReturn([$this->message(1)]); + $this->relay->expects('claim')->with('second', 100)->andReturn([]); + $this->relay->expects('deliver')->with('first', m::type('array'))->andReturn(false); + + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(memory: 1024, once: true))); + Sleep::assertNeverSlept(); + } + + public function testPinnedSourceDoesNotClaimFromOtherConfiguredDatabases(): void + { + $worker = $this->worker(['first', 'second']); + $this->relay->expects('claim')->with('second', 100)->andReturn([]); + + $this->assertSame(0, $worker->run('second', new OutboxWorkerOptions(memory: 1024, once: true))); + } + + public function testFailureSleepKeepsThePublicationSlotOccupied(): void + { + Sleep::fake(); + $worker = $this->worker(); + $claims = 0; + $this->relay->shouldReceive('claim')->twice()->andReturnUsing(function () use (&$claims): array { + return [$this->message(++$claims)]; + }); + $this->relay->expects('deliver')->andReturn(false); + $this->relay->expects('deliver')->andReturn(true); + Sleep::whenFakingSleep(function () use (&$claims): void { + $this->assertSame(1, $claims); + }); + + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(batch: 1, sleep: 0.25, maxJobs: 2, memory: 1024))); + Sleep::assertSequence([Sleep::usleep(250_000)]); + } + + public function testEmptyPassSleepsOnceAndSignalStopsFurtherAdmission(): void + { + Sleep::fake(); + $worker = $this->worker(['first', 'second']); + $this->relay->expects('claim')->with('first', 100)->andReturn([]); + $this->relay->expects('claim')->with('second', 100)->andReturn([]); + Sleep::whenFakingSleep(function (): void { ($this->signalHandler)(SIGTERM); }); + + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(sleep: 0.25, memory: 1024))); + Sleep::assertSequence([Sleep::usleep(250_000)]); + } + + public function testPauseStopsAdmissionUntilTheResumeSignal(): void + { + Sleep::fake(); + $worker = $this->worker(); + $claims = 0; + $sleeps = 0; + $this->relay->shouldReceive('claim')->twice()->andReturnUsing(function () use (&$claims): array { + return ++$claims === 1 ? [] : [$this->message(1)]; + }); + $this->relay->expects('deliver')->andReturn(true); + Sleep::whenFakingSleep(function () use (&$claims, &$sleeps): void { + $this->assertSame(1, $claims); + ($this->signalHandler)(++$sleeps === 1 ? SIGUSR2 : SIGCONT); + }); + + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(sleep: 0.25, maxJobs: 1, memory: 1024))); + Sleep::assertSequence([Sleep::usleep(250_000), Sleep::usleep(250_000)]); + } + + public function testRestartReadsEndTheirCoroutineBeforeClaimingAndRestartStopsAdmission(): void + { + $worker = $this->worker(); + CoroutineContext::set('request-only', true); + $borrowed = false; + $reads = 0; + $this->cache->shouldReceive('get')->with(Worker::RESTART_SIGNAL_CACHE_KEY)->times(3)->andReturnUsing(function () use (&$borrowed, &$reads): int { + $this->assertFalse(CoroutineContext::has('request-only')); + $borrowed = true; + Coroutine::defer(static function () use (&$borrowed): void { $borrowed = false; }); + + return ++$reads === 3 ? 2 : 1; + }); + $this->relay->expects('claim')->andReturnUsing(function () use (&$borrowed): array { + $this->assertFalse($borrowed); + + return [$this->message(1)]; + }); + $this->relay->expects('deliver')->andReturn(true); + + $this->assertSame(0, $worker->run(null, new OutboxWorkerOptions(memory: 1024))); + $this->assertFalse($borrowed); + } + + public function testEscapingTaskFailureStopsAdmissionAndPropagates(): void + { + $worker = $this->worker(); + $failure = new RuntimeException('Reporter unavailable.'); + $this->relay->expects('claim')->andReturn([$this->message(1)]); + $this->relay->expects('deliver')->andThrow($failure); + + try { + $worker->run(null, new OutboxWorkerOptions(memory: 1024)); + $this->fail('Task failure must reach the command.'); + } catch (RuntimeException $exception) { + $this->assertSame($failure, $exception); + } + } + + public function testCancellationJoinsTaskCleanupBeforeReturningToTheCaller(): void + { + $worker = $this->worker(); + $started = new Channel(1); + $blocker = new Channel(1); + $released = false; + $caught = null; + $this->relay->expects('claim')->andReturn([$this->message(1)]); + $this->relay->expects('deliver')->andReturnUsing(static function () use ($started, $blocker, &$released): bool { + Coroutine::defer(static function () use (&$released): void { $released = true; }); + $started->push(true); + $blocker->pop(5); + + return true; + }); + $coordinator = Coroutine::create(static function () use ($worker, &$caught): void { + try { + $worker->run(null, new OutboxWorkerOptions(maxJobs: 1, memory: 1024)); + } catch (CanceledException $exception) { + $caught = $exception; + } + }); + + try { + $this->assertTrue($started->pop(2)); + EngineCoroutine::cancelById($coordinator, throwException: true); + Coroutine::join([$coordinator], 2); + $this->assertFalse(Coroutine::exists($coordinator)); + $this->assertInstanceOf(CanceledException::class, $caught); + $this->assertTrue($released); + } finally { + $blocker->close(); + Coroutine::join([$coordinator], 5); + } + } + + public function testMemoryLimitStopsBeforeClaiming(): void + { + $this->assertSame(Worker::EXIT_MEMORY_LIMIT, $this->worker()->run(null, new OutboxWorkerOptions(memory: 1))); + } + + /** + * Construct an isolated worker with observable dependencies and signal ownership. + */ + protected function worker(array $databases = ['primary']): OutboxWorker + { + $repository = m::mock(DatabaseOutboxRepository::class); + $repository->shouldReceive('getDatabases')->andReturn($databases); + $this->relay = m::mock(OutboxRelay::class); + $this->exceptions = m::mock(ExceptionHandler::class); + $this->cache = m::mock(Cache::class); + $this->cache->shouldReceive('get')->with(Worker::RESTART_SIGNAL_CACHE_KEY)->andReturn(null)->byDefault(); + $cacheFactory = m::mock(CacheFactory::class); + $cacheFactory->shouldReceive('store')->andReturn($this->cache); + $signals = m::mock(SignalRegistry::class); + $signals->expects('register')->andReturnUsing(function (object $owner, array $signals, Closure $callback): void { + $this->assertSame($owner, $callback); + $this->signalHandler = $callback; + }); + $signals->expects('unregister')->with(m::on(fn (object $owner): bool => $owner === $this->signalHandler)); + + return new OutboxWorker($repository, $this->relay, $cacheFactory, $this->exceptions, $signals); + } + + /** + * Create a claimed message for worker admission tests. + */ + protected function message(int $id): OutboxMessage + { + return new OutboxMessage($id, "job-{$id}", 'lease', 1, 'redis', 'default', '{}', [], null, 1, 1); + } +} diff --git a/tests/Queue/Outbox/OutboxWrapperTest.php b/tests/Queue/Outbox/OutboxWrapperTest.php new file mode 100644 index 0000000000..9945a8777f --- /dev/null +++ b/tests/Queue/Outbox/OutboxWrapperTest.php @@ -0,0 +1,167 @@ +wrap($source, $wrapper); + $this->assertArrayNotHasKey('viaOutbox', get_object_vars($job)); + $this->assertArrayNotHasKey('outboxDatabase', get_object_vars($job)); + + $source->onOutboxDatabase('billing'); + $job = $this->wrap($source, $wrapper); + $this->assertTrue($job->viaOutbox); + $this->assertSame('billing', $job->outboxDatabase); + + $source = new $marked; + $this->assertTrue($this->wrap($source, $wrapper)->viaOutbox); + + $source->viaOutbox = null; + $this->assertTrue($this->wrap($source, $wrapper)->viaOutbox); + + $source->withoutOutbox(); + $job = $this->wrap($source, $wrapper); + $this->assertFalse($job->viaOutbox); + $this->assertArrayNotHasKey('outboxDatabase', get_object_vars($job)); + } + + /** + * Provide the independent producer wrapper boundaries. + */ + public static function producerTypes(): array + { + return [ + 'mail' => [OutboxMailable::class, MarkedOutboxMailable::class, SendQueuedMailable::class], + 'notification' => [OutboxNotification::class, MarkedOutboxNotification::class, SendQueuedNotifications::class], + 'broadcast' => [OutboxBroadcast::class, MarkedOutboxBroadcast::class, BroadcastEvent::class], + ]; + } + + #[DataProvider('listenerTypes')] + public function testListenerDefaultsPropagateToQueuedJobs(string $listener, ?bool $selected, ?string $database): void + { + $queue = new QueueFake(new Container); + $dispatcher = new Dispatcher; + $dispatcher->setQueueResolver(fn () => $queue); + $dispatcher->listen('order.created', $listener); + + $dispatcher->dispatch('order.created'); + + $queue->assertPushed(CallQueuedListener::class, function (CallQueuedListener $job) use ($selected, $database): bool { + $this->assertSame($selected, $job->viaOutbox ?? null); + $this->assertSame($database, $job->outboxDatabase ?? null); + + if ($selected === null) { + $this->assertArrayNotHasKey('viaOutbox', get_object_vars($job)); + } + + return true; + }); + } + + /** + * Provide listener defaults read without invoking constructors. + */ + public static function listenerTypes(): array + { + return [ + 'absent' => [OutboxListener::class, null, null], + 'marker with uninitialized state' => [MarkedOutboxListener::class, true, null], + 'null with marker' => [NullMarkedOutboxListener::class, true, 'billing'], + 'explicit false overrides marker' => [DirectMarkedOutboxListener::class, false, null], + 'explicit true' => [SelectedOutboxListener::class, true, 'billing'], + ]; + } + + /** + * Construct the wrapper through its normal producer boundary. + */ + protected function wrap(object $source, string $wrapper): object + { + return $wrapper === SendQueuedNotifications::class + ? new SendQueuedNotifications([], $source) + : new $wrapper($source); + } +} + +class OutboxMailable extends Mailable +{ + use Queueable; +} + +class MarkedOutboxMailable extends OutboxMailable implements ShouldQueueViaOutbox +{ +} + +class OutboxNotification extends Notification +{ + use Queueable; +} + +class MarkedOutboxNotification extends OutboxNotification implements ShouldQueueViaOutbox +{ +} + +class OutboxBroadcast +{ + use Queueable; +} + +class MarkedOutboxBroadcast extends OutboxBroadcast implements ShouldQueueViaOutbox +{ +} + +class OutboxListener implements ShouldQueue +{ + /** + * Handle the queued event. + */ + public function handle(): void + { + } +} + +class MarkedOutboxListener extends OutboxListener implements ShouldQueueViaOutbox +{ + use Queueable; +} + +class NullMarkedOutboxListener extends OutboxListener implements ShouldQueueViaOutbox +{ + public ?bool $viaOutbox = null; + + public string $outboxDatabase = 'billing'; +} + +class DirectMarkedOutboxListener extends OutboxListener implements ShouldQueueViaOutbox +{ + public bool $viaOutbox = false; +} + +class SelectedOutboxListener extends OutboxListener +{ + public bool $viaOutbox = true; + + public string $outboxDatabase = 'billing'; +} diff --git a/tests/Queue/Outbox/PooledOutboxPublicationTest.php b/tests/Queue/Outbox/PooledOutboxPublicationTest.php new file mode 100644 index 0000000000..fc6ee1909f --- /dev/null +++ b/tests/Queue/Outbox/PooledOutboxPublicationTest.php @@ -0,0 +1,128 @@ +makePartial(); + $messages = [$this->message(1), $this->message(2)]; + $expected = new OutboxBatchResult([1], failure: new RuntimeException('Partial rejection.')); + $queue->expects('publishOutboxMessages')->with($messages)->andReturn($expected); + $released = 0; + $pools = new PoolManager; + $proxy = $this->proxy($queue, $pools, $released); + + try { + $this->assertSame($expected, $proxy->publishOutboxMessages($messages)); + $this->assertSame(1, $released); + $this->assertSame(0, $pools->get($proxy->getPoolName())->getBorrowedCount()); + } finally { + $pools->purgeAll(); + } + } + + public function testSequentialFallbackRetainsConfirmedPrefixAndDefersFutureRows(): void + { + CarbonImmutable::setTestNow('2040-01-01 12:00:00'); + $queue = m::mock(SyncQueue::class)->makePartial(); + $messages = [$this->message(1), $this->message(2, 60), $this->message(3), $this->message(4), $this->message(5, 120)]; + $failure = new RuntimeException('Publication outcome unknown.'); + $queue->expects('pushRaw')->with($messages[0]->payload, 'orders', $messages[0]->options)->andReturn('accepted'); + $queue->expects('pushRaw')->with($messages[2]->payload, 'orders', $messages[2]->options)->andThrow($failure); + $released = 0; + $pools = new PoolManager; + $proxy = $this->proxy($queue, $pools, $released); + + try { + $result = $proxy->publishOutboxMessages($messages); + + $this->assertSame([1], $result->accepted); + $this->assertSame([2, 5], $result->deferred); + $this->assertSame($failure, $result->failure); + $this->assertSame(1, $released); + $this->assertSame(0, $pools->get($proxy->getPoolName())->getBorrowedCount()); + } finally { + $pools->purgeAll(); + } + } + + public function testCancellationReleasesTheBorrowedTransportAndRemainsCancellation(): void + { + $queue = m::mock(SyncQueue::class)->makePartial(); + $cancellation = new CanceledException; + $queue->expects('pushRaw')->andThrow($cancellation); + $released = 0; + $pools = new PoolManager; + $proxy = $this->proxy($queue, $pools, $released); + + try { + try { + $proxy->publishOutboxMessages([$this->message(1)]); + $this->fail('Expected publication to remain cancelled.'); + } catch (CanceledException $exception) { + $this->assertSame($cancellation, $exception); + } + + $this->assertSame(1, $released); + $this->assertSame(0, $pools->get($proxy->getPoolName())->getBorrowedCount()); + } finally { + $pools->purgeAll(); + } + } + + /** + * Create a single-slot proxy with observable release and no external transport. + */ + protected function proxy(SyncQueue $queue, PoolManager $pools, int &$released): QueuePoolProxy + { + return new QueuePoolProxy( + new PoolDefinition('outbox-publication', 'queue', 'outbox-test', PoolOptions::fromArray(['max_objects' => 1])), + static fn (): SyncQueue => $queue, + $pools, + static function () use (&$released): void { + ++$released; + }, + ); + } + + /** + * Build an opaque message passed through the generic raw contract. + */ + protected function message(int $id, ?int $delay = null): OutboxMessage + { + $uuid = (string) Str::uuid(); + + return new OutboxMessage( + $id, + $uuid, + (string) Str::uuid(), + 1, + 'custom', + 'orders', + json_encode(['uuid' => $uuid]), + ['transport' => ['custom' => ['key' => 'value']]], + $delay === null ? null : CarbonImmutable::now()->getTimestamp() + $delay, + 1, + 1, + ); + } +} diff --git a/tests/Queue/Outbox/RedisOutboxPublicationTest.php b/tests/Queue/Outbox/RedisOutboxPublicationTest.php new file mode 100644 index 0000000000..0ad0b723ed --- /dev/null +++ b/tests/Queue/Outbox/RedisOutboxPublicationTest.php @@ -0,0 +1,158 @@ +queue(cluster: $cluster); + $routes->forward(['orders' => 'processing', 'alias' => 'processing', 'processing' => 'archive']); + $events->listen(JobQueued::class, fn () => $this->fail('Relay must not repeat the producer lifecycle.')); + $messages = [$this->message(1, 'orders'), $this->message(2, 'alias', CarbonImmutable::now()->getTimestamp() + 30)]; + $key = $cluster ? 'queues:{processing}' : 'queues:processing'; + $connection->expects('evalWithShaCache')->with( + LuaScripts::bulk(), + [$key, $key . ':notify', $key . ':delayed'], + ['i', $messages[0]->payload, $messages[1]->deliverAt, $messages[1]->payload], + )->andReturn(2); + + $result = $queue->publishOutboxMessages($messages); + + $this->assertSame([1, 2], $result->accepted); + $this->assertNull($result->failure); + $this->assertSame([], $result->deferred); + } + + /** + * Provide standalone and Cluster key layouts. + */ + public static function clusterModes(): array + { + return [[false], [true]]; + } + + public function testLaterDestinationFailurePreservesEarlierConfirmation(): void + { + [$queue, $connection] = $this->queue(); + $failure = new RuntimeException('Redis connection lost.'); + $connection->expects('evalWithShaCache')->withArgs(fn (string $script, array $keys): bool => $keys[0] === 'queues:first')->andReturn(1); + $connection->expects('evalWithShaCache')->withArgs(fn (string $script, array $keys): bool => $keys[0] === 'queues:second')->andThrow($failure); + + $result = $queue->publishOutboxMessages([$this->message(1, 'first'), $this->message(2, 'second'), $this->message(3, 'third')]); + + $this->assertSame([1], $result->accepted); + $this->assertSame($failure, $result->failure); + } + + public function testIncompleteConfirmationRetainsTheEntireDestination(): void + { + [$queue, $connection] = $this->queue(); + $connection->expects('evalWithShaCache')->andReturn(1); + + $result = $queue->publishOutboxMessages([$this->message(1), $this->message(2)]); + + $this->assertSame([], $result->accepted); + $this->assertInstanceOf(RuntimeException::class, $result->failure); + } + + public function testHorizonStampsPublicationAndDrainsAcceptedNotificationsBeforeStopping(): void + { + CarbonImmutable::setTestNow('2040-01-01 12:00:00'); + [$queue, $connection, $events] = $this->queue(horizon: true); + $messages = [$this->message(1), $this->message(2), $this->message(3, 'other')]; + $pending = []; + $pushed = []; + $failure = new RuntimeException('Monitoring storage unavailable.'); + $events->listen(JobPending::class, function (JobPending $event) use (&$pending): void { + $pending[] = $event->payload->value; + }); + $events->listen(JobPushed::class, function (JobPushed $event) use (&$pushed, $failure): void { + $pushed[] = $event->payload->value; + throw $failure; + }); + $events->listen(JobQueued::class, fn () => $this->fail('Horizon relay must not emit JobQueued.')); + $stored = []; + $connection->expects('evalWithShaCache')->andReturnUsing(function (string $script, array $keys, array $arguments) use (&$stored): int { + $stored = [$arguments[1], $arguments[3]]; + + return 2; + }); + + $result = $queue->publishOutboxMessages($messages); + + $this->assertSame([1, 2], $result->accepted); + $this->assertSame($failure, $result->failure); + $this->assertSame($stored, $pending); + $this->assertSame($stored, $pushed); + + foreach ($stored as $index => $payload) { + $decoded = json_decode($payload, true); + $original = json_decode($messages[$index]->payload, true); + $this->assertArrayHasKey('pushedAt', $decoded); + unset($decoded['pushedAt']); + $this->assertSame($original, $decoded); + } + } + + /** + * Create a real driver over an observable transport boundary. + */ + protected function queue(bool $cluster = false, bool $horizon = false): array + { + $container = new Container; + Container::setInstance($container); + $events = new Dispatcher($container); + $container->instance('events', $events); + $container->instance(EventDispatcher::class, $events); + $container->instance('queue.routes', $routes = new QueueRoutes); + $connection = m::mock(RedisProxy::class); + $connection->shouldAllowMockingMethod('evalWithShaCache'); + $connection->expects('isCluster')->andReturn($cluster); + $factory = m::mock(Factory::class); + $factory->shouldReceive('connection')->andReturn($connection); + $queue = $horizon ? new HorizonRedisQueue($factory, 'orders') : new RedisQueue($factory, 'orders'); + $queue->setContainer($container)->setConnectionName('redis'); + + return [$queue, $connection, $events, $routes]; + } + + /** + * Build a finalized envelope with immutable staging classification. + */ + protected function message(int $id, string $queue = 'orders', ?int $deliverAt = null): OutboxMessage + { + $uuid = (string) Str::uuid(); + $payload = json_encode([ + 'uuid' => $uuid, 'id' => $uuid, 'attempts' => 0, + 'type' => 'job', 'tags' => ['tenant:17'], 'silenced' => false, + 'data' => ['command' => 'opaque serialized command'], + ]); + + return new OutboxMessage($id, $uuid, (string) Str::uuid(), 1, 'redis', $queue, $payload, [], $deliverAt, 1, 1); + } +} diff --git a/tests/Queue/Outbox/SqsOutboxPublicationTest.php b/tests/Queue/Outbox/SqsOutboxPublicationTest.php new file mode 100644 index 0000000000..ecb0483483 --- /dev/null +++ b/tests/Queue/Outbox/SqsOutboxPublicationTest.php @@ -0,0 +1,187 @@ +message(1, $fifo ? 'orders.fifo' : 'orders', $offset); + $client->expects('sendMessageBatch')->andReturnUsing(function (array $arguments) use ($fifo, $expected): Result { + $entry = $arguments['Entries'][0]; + $this->assertSame('https://sqs.test/account/orders' . ($fifo ? '.fifo' : ''), $arguments['QueueUrl']); + + if ($expected === null) { + $this->assertArrayNotHasKey('DelaySeconds', $entry); + } else { + $this->assertSame($expected, $entry['DelaySeconds']); + } + + if ($fifo) { + $this->assertSame('tenant:17', $entry['MessageGroupId']); + $this->assertSame('captured-deduplication', $entry['MessageDeduplicationId']); + } + + return new Result(['Successful' => [['Id' => '1', 'MessageId' => 'sqs-1']]]); + }); + + $result = $queue->publishOutboxMessages([$message]); + + $this->assertSame([1], $result->accepted); + $this->assertNull($result->failure); + } + + /** + * Provide absent, due, elapsed, capped and FIFO timers. + */ + public static function deliveryTimes(): array + { + return [[null, false, null], [0, false, 0], [-30, false, 0], [30, false, 30], [1000, false, 900], [0, true, null]]; + } + + public function testBatchSizeLimitsAndLaterSdkFailureRetainEarlierConfirmation(): void + { + $client = m::mock(SqsClient::class); + $queue = new SqsQueue($client, 'orders'); + $failure = new RuntimeException('SDK retries exhausted.'); + $calls = 0; + $client->expects('sendMessageBatch')->twice()->andReturnUsing(function (array $arguments) use (&$calls, $failure): Result { + if (++$calls === 2) { + $this->assertCount(1, $arguments['Entries']); + throw $failure; + } + + $this->assertCount(10, $arguments['Entries']); + + return new Result(['Successful' => array_map(static fn (array $entry): array => ['Id' => $entry['Id'], 'MessageId' => 'sqs-' . $entry['Id']], $arguments['Entries'])]); + }); + + $result = $queue->publishOutboxMessages(array_map(fn (int $id): OutboxMessage => $this->message($id), range(1, 11))); + + $this->assertSame(range(1, 10), $result->accepted); + $this->assertSame($failure, $result->failure); + } + + public function testByteLimitSplitsLargeMessagesBeforeTheCountLimit(): void + { + $client = m::mock(SqsClient::class); + $queue = new SqsQueue($client, 'orders'); + $client->expects('sendMessageBatch')->twice()->andReturnUsing(function (array $arguments): Result { + $this->assertCount(1, $arguments['Entries']); + + return new Result(['Successful' => [['Id' => $arguments['Entries'][0]['Id'], 'MessageId' => 'accepted']]]); + }); + + $result = $queue->publishOutboxMessages([ + $this->message(1, payload: json_encode(['data' => str_repeat('a', 600000)])), + $this->message(2, payload: json_encode(['data' => str_repeat('b', 600000)])), + ]); + + $this->assertSame([1, 2], $result->accepted); + $this->assertNull($result->failure); + } + + public function testExplicitRejectionCleansOnlyThatAttemptAndRetainsConfirmedEntries(): void + { + $container = new Container; + Container::setInstance($container); + $store = new Repository(new WorkerArrayStore); + $cache = m::mock(CacheFactory::class); + $cache->shouldReceive('store')->with(null)->andReturn($store); + $container->instance('cache', $cache); + $client = m::mock(SqsClient::class); + $queue = new SqsQueue($client, 'orders', overflowStorage: ['enabled' => true, 'always' => true]); + $queue->setContainer($container); + $first = $this->message(1); + $secondAttempt = new OutboxMessage( + $first->id, + $first->uuid, + (string) Str::uuid(), + 2, + $first->connection, + $first->queue, + $first->payload, + $first->options, + null, + 1, + 1, + ); + $accepted = $this->message(2); + $paths = []; + $calls = 0; + $client->expects('sendMessageBatch')->twice()->andReturnUsing(function (array $arguments) use (&$calls, &$paths, $store): Result { + foreach ($arguments['Entries'] as $entry) { + $path = json_decode($entry['MessageBody'], true)['@pointer']; + $paths[] = $path; + $this->assertTrue($store->has($path)); + } + + if (++$calls === 1) { + throw new RuntimeException('Acceptance is ambiguous.'); + } + + return new Result([ + 'Successful' => [['Id' => '2', 'MessageId' => 'confirmed']], + 'Failed' => [['Id' => '1', 'Code' => 'Rejected', 'Message' => 'Rejected entry', 'SenderFault' => true]], + ]); + }); + + $this->assertSame([], $queue->publishOutboxMessages([$first])->accepted); + $result = $queue->publishOutboxMessages([$secondAttempt, $accepted]); + + $this->assertSame([2], $result->accepted); + $this->assertInstanceOf(SqsException::class, $result->failure); + $this->assertStringContainsString('rejected [1] of [2]', $result->failure->getMessage()); + $this->assertNotSame($paths[0], $paths[1]); + $this->assertSame($first->payload, $store->get($paths[0])); + $this->assertFalse($store->has($paths[1])); + $this->assertSame($accepted->payload, $store->get($paths[2])); + } + + /** + * Build a claimed envelope with previously captured FIFO options. + */ + protected function message(int $id, string $queue = 'orders', ?int $offset = null, ?string $payload = null): OutboxMessage + { + $uuid = (string) Str::uuid(); + $options = str_ends_with($queue, '.fifo') + ? ['transport' => ['sqs' => ['MessageGroupId' => 'tenant:17', 'MessageDeduplicationId' => 'captured-deduplication']]] + : []; + + return new OutboxMessage( + $id, + $uuid, + (string) Str::uuid(), + 1, + 'sqs', + $queue, + $payload ?? json_encode(['uuid' => $uuid, 'data' => ['command' => 'opaque']]), + $options, + $offset === null ? null : CarbonImmutable::now()->getTimestamp() + $offset, + 1, + 1, + ); + } +} diff --git a/tests/Queue/QueueBackgroundQueueTest.php b/tests/Queue/QueueBackgroundQueueTest.php index e3652ff980..89ffd19098 100644 --- a/tests/Queue/QueueBackgroundQueueTest.php +++ b/tests/Queue/QueueBackgroundQueueTest.php @@ -204,6 +204,7 @@ public function testItAddsATransactionCallbackForAfterCommitJobs(): void $background->setConnectionName('background'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -217,6 +218,7 @@ public function testItAddsATransactionCallbackForInterfaceBasedAfterCommitJobs() $background->setConnectionName('background'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -230,6 +232,7 @@ public function testItAddsATransactionCallbackForAfterCommitUniqueJobs(): void $background->setConnectionName('background'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $transactionManager->shouldReceive('addCallbackForRollback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -247,6 +250,7 @@ public function testItAddsATransactionRollbackCallbackForAfterCommitDebouncedJob $background->setConnectionName('background'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $transactionManager->shouldReceive('addCallbackForRollback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -264,6 +268,7 @@ public function testItAddsATransactionCallbackForInterfaceBasedAfterCommitUnique $background->setConnectionName('background'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $transactionManager->shouldReceive('addCallbackForRollback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -361,6 +366,7 @@ public function testPushSnapshotsAfterCommitJobAtDispatch(): void $afterCommit = null; $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) use (&$afterCommit) { @@ -450,6 +456,7 @@ public function testLaterAddsTransactionCallbackForAfterCommitJobs(): void $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { @@ -472,6 +479,7 @@ public function testLaterAddsTransactionCallbackForInterfaceBasedAfterCommitJobs $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { @@ -494,6 +502,7 @@ public function testLaterAddsTransactionCallbackForAfterCommitUniqueJobs(): void $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { @@ -519,6 +528,7 @@ public function testLaterAddsTransactionRollbackCallbackForAfterCommitDebouncedJ $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { @@ -544,6 +554,7 @@ public function testLaterAddsTransactionCallbackForInterfaceBasedAfterCommitUniq $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { diff --git a/tests/Queue/QueueDatabaseQueueUnitTest.php b/tests/Queue/QueueDatabaseQueueUnitTest.php index d25f1ece16..9e4b0e5b6f 100644 --- a/tests/Queue/QueueDatabaseQueueUnitTest.php +++ b/tests/Queue/QueueDatabaseQueueUnitTest.php @@ -531,6 +531,7 @@ public function testBulkRetainsDispatchOwnershipUntilTheTransactionCommits(): vo $resolver->shouldReceive('connection')->once()->with(null)->andReturn($connection = m::mock(PdoConnection::class)); $connection->shouldReceive('maxBindings')->once()->andReturn(6); + $connection->shouldReceive('transactionLevel')->twice()->andReturn(0); $connection->shouldReceive('transaction')->once()->andReturnUsing(function (Closure $callback) use ($first, $second): mixed { $this->assertTrue(DispatchLockContext::has($first)); $this->assertTrue(DispatchLockContext::has($second)); @@ -569,10 +570,15 @@ public function testBulkAcceptsEveryDispatchBeforeRaisingSuccessEvents(): void DispatchLockContext::registerDebounce($first, $cache, 'first', 'owner-1'); DispatchLockContext::registerDebounce($second, $cache, 'second', 'owner-2'); $exception = new RuntimeException('Listener failed.'); + $notified = []; $resolver->shouldReceive('connection')->once()->with(null)->andReturn($connection = m::mock(ConnectionInterface::class)); + $connection->shouldReceive('transactionLevel')->twice()->andReturn(0); $connection->shouldReceive('table')->once()->with('table')->andReturn($query = m::mock(Builder::class)); $query->shouldReceive('insert')->once()->andReturnTrue(); + $dispatcher->listen(JobQueued::class, static function (JobQueued $event) use (&$notified): void { + $notified[] = $event->job; + }); $dispatcher->listen(JobQueued::class, static function () use ($exception): never { throw $exception; }); @@ -586,6 +592,7 @@ public function testBulkAcceptsEveryDispatchBeforeRaisingSuccessEvents(): void $this->assertFalse(DispatchLockContext::has($first)); $this->assertFalse(DispatchLockContext::has($second)); + $this->assertSame([$first, $second], $notified); } public function testDelayAttributeIsRespectedWhenBulkPushing(): void diff --git a/tests/Queue/QueueDeferredQueueTest.php b/tests/Queue/QueueDeferredQueueTest.php index 5bf3161997..4cd325652e 100644 --- a/tests/Queue/QueueDeferredQueueTest.php +++ b/tests/Queue/QueueDeferredQueueTest.php @@ -185,6 +185,7 @@ public function testItAddsATransactionCallbackForAfterCommitJobs(): void $deferred->setConnectionName('deferred'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -198,6 +199,7 @@ public function testItAddsATransactionCallbackForInterfaceBasedAfterCommitJobs() $deferred->setConnectionName('deferred'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -211,6 +213,7 @@ public function testItAddsATransactionCallbackForAfterCommitUniqueJobs(): void $deferred->setConnectionName('deferred'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $transactionManager->shouldReceive('addCallbackForRollback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -228,6 +231,7 @@ public function testItAddsATransactionRollbackCallbackForAfterCommitDebouncedJob $deferred->setConnectionName('deferred'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $transactionManager->shouldReceive('addCallbackForRollback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -245,6 +249,7 @@ public function testItAddsATransactionCallbackForInterfaceBasedAfterCommitUnique $deferred->setConnectionName('deferred'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback')->once()->andReturn(null); $transactionManager->shouldReceive('addCallbackForRollback')->once()->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -343,6 +348,7 @@ public function testLaterAddsTransactionCallbackForAfterCommitJobs(): void $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { @@ -365,6 +371,7 @@ public function testLaterAddsTransactionCallbackForInterfaceBasedAfterCommitJobs $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { @@ -387,6 +394,7 @@ public function testLaterAddsTransactionCallbackForAfterCommitUniqueJobs(): void $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { @@ -412,6 +420,7 @@ public function testLaterAddsTransactionRollbackCallbackForAfterCommitDebouncedJ $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { @@ -437,6 +446,7 @@ public function testLaterAddsTransactionCallbackForInterfaceBasedAfterCommitUniq $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->shouldReceive('addCallback') ->once() ->andReturnUsing(function ($callback) { diff --git a/tests/Queue/QueueRedisQueueTest.php b/tests/Queue/QueueRedisQueueTest.php index 5af48ddadc..8b05940d16 100644 --- a/tests/Queue/QueueRedisQueueTest.php +++ b/tests/Queue/QueueRedisQueueTest.php @@ -207,7 +207,11 @@ public function testBulkAcceptsEveryDispatchBeforeRaisingSuccessEvents(): void DispatchLockContext::registerDebounce($first, $cache, 'first', 'owner-1'); DispatchLockContext::registerDebounce($second, $cache, 'second', 'owner-2'); $exception = new RuntimeException('Listener failed.'); + $notified = []; + $dispatcher->listen(JobQueued::class, static function (JobQueued $event) use (&$notified): void { + $notified[] = $event->job; + }); $dispatcher->listen(JobQueued::class, static function () use ($exception): never { throw $exception; }); @@ -221,6 +225,7 @@ public function testBulkAcceptsEveryDispatchBeforeRaisingSuccessEvents(): void $this->assertFalse(DispatchLockContext::has($first)); $this->assertFalse(DispatchLockContext::has($second)); + $this->assertSame([$first, $second], $notified); } public function testBulkRetainsDispatchOwnershipWhenRedisDoesNotConfirmEveryJob(): void diff --git a/tests/Queue/QueueSyncQueueTest.php b/tests/Queue/QueueSyncQueueTest.php index d09fb9a6e1..12ef40d826 100644 --- a/tests/Queue/QueueSyncQueueTest.php +++ b/tests/Queue/QueueSyncQueueTest.php @@ -340,6 +340,7 @@ public function testItAddsATransactionCallbackForAfterCommitJobs(): void $sync->setConnectionName('sync'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('addCallback')->andReturn(null); $transactionManager->shouldNotReceive('addCallbackForRollback'); $container->instance('db.transactions', $transactionManager); @@ -354,6 +355,7 @@ public function testItAddsATransactionCallbackForInterfaceBasedAfterCommitJobs() $sync->setConnectionName('sync'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('addCallback')->andReturn(null); $transactionManager->shouldNotReceive('addCallbackForRollback'); $container->instance('db.transactions', $transactionManager); @@ -368,6 +370,7 @@ public function testItAddsATransactionCallbackForAfterCommitUniqueJobs(): void $sync->setConnectionName('sync'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('addCallback')->andReturn(null); $transactionManager->expects('addCallbackForRollback')->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -385,6 +388,7 @@ public function testItAddsATransactionRollbackCallbackForAfterCommitDebouncedJob $sync->setConnectionName('sync'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('addCallback')->andReturn(null); $transactionManager->expects('addCallbackForRollback')->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -402,6 +406,7 @@ public function testItAddsATransactionCallbackForInterfaceBasedAfterCommitUnique $sync->setConnectionName('sync'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('addCallback')->andReturn(null); $transactionManager->expects('addCallbackForRollback')->andReturn(null); $container->instance('db.transactions', $transactionManager); @@ -419,6 +424,7 @@ public function testAfterCommitUniqueJobWithoutDispatchOwnershipDoesNotAddRollba $sync->setConnectionName('sync'); $container = $this->getContainer(); $transactionManager = m::mock(DatabaseTransactionsManager::class); + $transactionManager->expects('hasApplicableTransaction')->with(null)->andReturnTrue(); $transactionManager->expects('addCallback')->andReturnNull(); $transactionManager->shouldReceive('addCallbackForRollback')->never(); $container->instance('db.transactions', $transactionManager); diff --git a/tests/Queue/RelayCommandTest.php b/tests/Queue/RelayCommandTest.php new file mode 100644 index 0000000000..bebad1f015 --- /dev/null +++ b/tests/Queue/RelayCommandTest.php @@ -0,0 +1,80 @@ +expects('run')->with('orders', m::on(function (OutboxWorkerOptions $options): bool { + $this->assertEquals(new OutboxWorkerOptions( + batch: 50, + sleep: 0.25, + concurrency: 4, + maxJobs: 500, + maxTime: 60, + memory: 256, + stopWhenEmpty: true, + once: true, + ), $options); + + return true; + }))->andReturn(12); + $this->app->instance(OutboxWorker::class, $worker); + + $this->assertSame(12, Artisan::call('queue:relay', [ + 'database' => 'orders', + '--batch' => '50', + '--sleep' => '0.25', + '--concurrency' => '4', + '--max-jobs' => '500', + '--max-time' => '60', + '--memory' => '256', + '--stop-when-empty' => true, + '--once' => true, + ])); + } + + public function testCommandDefaultsToAllConfiguredSources(): void + { + $worker = m::mock(OutboxWorker::class); + $worker->expects('run')->with(null, m::on(fn (OutboxWorkerOptions $options): bool => $options == new OutboxWorkerOptions))->andReturn(0); + $this->app->instance(OutboxWorker::class, $worker); + + $this->assertSame(0, Artisan::call('queue:relay')); + } + + #[DataProvider('invalidOptions')] + public function testInvalidOptionsFailBeforeTheWorkerStarts(array $options): void + { + $this->app->instance(OutboxWorker::class, m::mock(OutboxWorker::class)); + $this->expectException(InvalidArgumentException::class); + + Artisan::call('queue:relay', $options); + } + + /** + * Provide invalid runtime limits at the command boundary. + */ + public static function invalidOptions(): array + { + return [ + 'empty batch' => [['--batch' => 0]], + 'oversized batch' => [['--batch' => 1001]], + 'hot-loop sleep' => [['--sleep' => 0]], + 'no capacity' => [['--concurrency' => 0]], + 'negative maximum' => [['--max-jobs' => -1]], + ]; + } +} diff --git a/tests/Queue/SqsBatchLifecycleTest.php b/tests/Queue/SqsBatchLifecycleTest.php new file mode 100644 index 0000000000..8c4343b430 --- /dev/null +++ b/tests/Queue/SqsBatchLifecycleTest.php @@ -0,0 +1,192 @@ +queue(); + $cache = new Repository(new WorkerArrayStore); + $container->instance(Cache::class, $cache); + $container->instance(BusContract::class, new BusDispatcher($container, static fn () => $queue)); + $transactions = new DatabaseTransactionsManager; + $container->instance('db.transactions', $transactions); + $transactions->begin('testing', 1); + $sent = false; + $client->expects('sendMessageBatch')->andReturnUsing(static function () use (&$sent): Result { + $sent = true; + + return new Result(['Successful' => [['Id' => '0', 'MessageId' => 'accepted']]]); + }); + $job = new SqsUniqueBulkJob; + $job->afterCommit = $afterCommit; + + $pending = new PendingDispatch($job); + unset($pending); + + $this->assertSame(! $afterCommit, $sent); + $this->assertTrue($cache->lock(UniqueLock::getKey($job))->isLocked()); + $transactions->commit('testing', 1, 0); + $this->assertTrue($sent); + $this->assertFalse(DispatchLockContext::has($job)); + $this->assertTrue($cache->lock(UniqueLock::getKey($job))->isLocked()); + } + + #[TestWith([JobPayloadFinalizing::class])] + #[TestWith([JobQueueing::class])] + public function testPreparationFailureTerminatesEveryOutstandingMember(string $eventClass): void + { + [$queue, $client, $events] = $this->queue(); + $exception = new RuntimeException('Second member failed.'); + $count = 0; + $events->listen($eventClass, static function () use (&$count, $exception): void { + if (++$count === 2) { + throw $exception; + } + }); + $failed = []; + $events->listen(JobQueueingFailed::class, static function (JobQueueingFailed $event) use (&$failed): void { + $failed[] = $event; + }); + $client->shouldNotReceive('sendMessageBatch'); + + try { + $queue->bulk(['first', 'second', 'third']); + $this->fail('The second member should fail preparation.'); + } catch (RuntimeException $caught) { + $this->assertSame($exception, $caught); + } + + $this->assertSame(['first', 'second', 'third'], array_column($failed, 'job')); + $this->assertSame([$exception, $exception, $exception], array_column($failed, 'exception')); + } + + public function testFailureNotificationsDrainAndRetainThePrimaryAttemptError(): void + { + [$queue, $client, $events] = $this->queue(); + $primary = new RuntimeException('Transport failed.'); + $listener = new RuntimeException('Listener failed.'); + $client->expects('sendMessageBatch')->andThrow($primary); + $failed = []; + $events->listen(JobQueueingFailed::class, static function (JobQueueingFailed $event) use (&$failed): void { + $failed[] = $event; + }); + $events->listen(JobQueueingFailed::class, static fn () => throw $listener); + + try { + $queue->bulk(['first', 'second']); + $this->fail('The failure listener should throw.'); + } catch (RuntimeException $caught) { + $this->assertSame($listener, $caught); + } + + $this->assertSame(['first', 'second'], array_column($failed, 'job')); + $this->assertSame([$primary, $primary], array_column($failed, 'exception')); + } + + public function testSuccessListenerFailurePreservesAcceptedEntriesAndCleansExplicitRejects(): void + { + [$queue, $client, $events, $container] = $this->queue(['enabled' => true, 'always' => true, 'store' => 'overflow']); + $store = new Repository(new WorkerArrayStore); + $cache = m::mock(CacheFactory::class); + $cache->expects('store')->with('overflow')->andReturn($store); + $container->instance('cache', $cache); + $paths = []; + $client->expects('sendMessageBatch')->once()->andReturnUsing(static function (array $arguments) use (&$paths): Result { + $paths = array_map(static fn (array $entry): string => json_decode($entry['MessageBody'], true)['@pointer'], $arguments['Entries']); + + return new Result([ + 'Successful' => array_map(static fn (int $id): array => ['Id' => (string) $id, 'MessageId' => 'accepted-' . $id], range(0, 8)), + 'Failed' => [['Id' => '9', 'Code' => 'InvalidParameterValue', 'Message' => 'Rejected']], + ]); + }); + $queued = $failed = []; + $listener = new RuntimeException('Monitoring failed.'); + $events->listen(JobQueued::class, static function (JobQueued $event) use (&$queued): void { + $queued[] = $event->job; + }); + $events->listen(JobQueued::class, static fn () => throw $listener); + $events->listen(JobQueueingFailed::class, static function (JobQueueingFailed $event) use (&$failed): void { + $failed[] = $event->job; + }); + + try { + $queue->bulk(array_map('strval', range(1, 11))); + $this->fail('The monitoring failure should be rethrown.'); + } catch (RuntimeException $caught) { + $this->assertSame($listener, $caught); + } + + $this->assertSame(array_map('strval', range(1, 9)), $queued); + $this->assertSame(['10', '11'], $failed); + $this->assertFalse($store->has($paths[9])); + + foreach (array_slice($paths, 0, 9) as $path) { + $this->assertTrue($store->has($path)); + } + } + + /** + * Create a queue with real lifecycle dispatch and a controlled SDK boundary. + * + * @return array{SqsQueue, SqsClient, Dispatcher, Container} + */ + protected function queue(array $overflow = []): array + { + $container = new Container; + Container::setInstance($container); + $events = new Dispatcher($container); + $container->instance('events', $events); + $container->instance(EventsContract::class, $events); + $client = m::mock(SqsClient::class); + $queue = new SqsQueue($client, 'jobs', overflowStorage: $overflow); + $queue->setContainer($container); + $queue->setConnectionName('sqs'); + + return [$queue, $client, $events, $container]; + } +} + +class SqsUniqueBulkJob extends FakeSqsJob implements ShouldBeUnique +{ + /** + * Route a normal pending dispatch through the supported bulk API. + */ + public function queue(QueueContract $queue, object $command): mixed + { + return $queue->bulk([$command]); + } +} diff --git a/tests/Testbench/CommanderTest.php b/tests/Testbench/CommanderTest.php index f9ba39e1c2..687f1cfaa6 100644 --- a/tests/Testbench/CommanderTest.php +++ b/tests/Testbench/CommanderTest.php @@ -267,6 +267,7 @@ public function itCanCallCommanderUsingCliAndRunMigration(): void '0001_01_01_000006_testbench_create_jobs_table', '0001_01_01_000007_testbench_create_failed_jobs_table', '0001_01_01_000008_testbench_create_rate_limits_table', + '0001_01_01_000009_testbench_create_job_outbox_table', '2013_07_26_182750_create_testbench_users_table', ], DB::connection('sqlite')->table('migrations')->pluck('migration')->all()); });