Add an opt-in transactional queue outbox by binaryfire · Pull Request #70 · hypervel/components-backup

binaryfire · 2026-10-23T03:14:23Z

Summary

This adds an opt-in transactional outbox to the queue package. Jobs are stored alongside the business write in the same database transaction, then published to their existing queue by a separate relay process.

afterCommit() prevents a worker from seeing uncommitted data, but it cannot close the gap between committing the database transaction and publishing the job. If the process stops in that gap, the business write survives without its job. The outbox closes that gap without replacing the existing queue drivers, dispatch APIs, or workers.

DB::transaction(function () use ($attributes) {
    $order = Order::create($attributes);

    SendOrderReceipt::dispatch($order)->viaOutbox();
});

Usage

Applications can select the outbox per dispatch with viaOutbox(), per job class with ShouldQueueViaOutbox, or per queue connection with outbox => true. withoutOutbox() overrides either default. onOutboxDatabase() selects the owning connection when more than one transaction is active.

The existing bulk and batch APIs use set-based inserts. Batch defaults also apply to jobs added later. Queued listeners, mailables, notifications, and broadcasts pass their selection to their existing wrapper jobs. Jobs that do not select the feature keep their normal dispatch path, without outbox database access or outbox service resolution.

Setup consists of an outbox table and a supervised relay alongside the existing workers:

php artisan make:queue-outbox-table
php artisan migrate
php artisan queue:relay --batch=100 --concurrency=4
php artisan queue:work redis

Horizon remains optional and continues managing Redis jobs after publication. The detailed guide is src/docs/queue-outbox.md; the queue and producer guides link to it.

Delivery and recovery

Staging uses the exact database session that owns the transaction. Rollback removes the rows and releases dispatch-owned locks. With several active connections, the caller must choose an owner rather than relying on a guess.
The relay claims bounded batches, publishes through the existing drivers, and deletes only confirmed deliveries. PostgreSQL uses a single claim statement; MySQL and MariaDB use locking reads with SKIP LOCKED; SQLite has an atomic-update fallback.
Database connections and locks are released before broker calls. Publication and settlement have separate, short resource lifetimes. Multiple relay processes and bounded concurrent batches are supported.
Redis and database publication are batched. SQS uses its request limits and partial-success responses. Other drivers use their existing raw-payload API, with one pooled connection per publication batch.
Failed and uncertain deliveries remain retryable with jittered backoff. Failed sources and destinations are throttled without blocking healthy sources. Ownership tokens prevent a stale relay from deleting another relay's claim.
Payload UUIDs, intended delays, SQS FIFO options, and Horizon classification survive staging. Separate SQS overflow publication attempts have separate cleanup ownership; existing overflow retention settings are unchanged.

The guarantee is at least once, not exactly once. A crash after broker acceptance but before settlement can publish the same job again. Handlers must be idempotent where repeated effects would be harmful. Publication order is best effort, and the outbox cannot make writes to separate databases or external services atomic.

Driver compatibility and operations

Selecting the feature does not require changing job code when switching drivers. A database queue already participating in the owning transaction inserts directly. Sync, deferred, background, and failover connections preserve their execution behavior through an after-commit fallback, without claiming crash durability. Null and fake queues do not create outbox rows. Direct pushRaw() remains a bypass.

The relay supports restart signals, graceful shutdown, bounded runtime and job limits, and one-shot execution. Operator commands provide pending-row inspection, monitoring, immediate retry, and deliberate removal with owner-matched unique-lock cleanup:

php artisan queue:outbox
php artisan queue:outbox-monitor --max=10000 --age=60
php artisan queue:outbox-retry JOB_UUID
php artisan queue:outbox-forget JOB_UUID

Delivery events and OpenTelemetry metrics are recorded at batch boundaries. There is no backlog polling on the application dispatch path and no duplicate producer span at relay time.

Related correctness changes

The integration also fixes behavior shared with ordinary dispatch:

Transaction records and callbacks identify the resolved base/read/write session, preventing separate sessions from consuming each other's callbacks.
Dispatch-lock cleanup captures the context in which the lock was acquired. Transactional storage keeps an independent rollback snapshot after acceptance.
Batch notifications preserve confirmed acceptance even when an observer fails. Horizon's post-publication work cannot turn an accepted job into a failed dispatch or release its lock.
Generic lifecycle events retain the finalized payload used by existing tracing integrations, separate from transport decoration.
Uninitialized public broadcast properties are skipped, and custom or memoized cache repositories retain their store identity for unique-lock cleanup.

Public dispatch method signatures remain unchanged. Two extension behaviors are documented: role-qualified transaction records now use the exact session name, and Horizon's normal push() uses the shared Redis storage helper instead of routing through its public pushRaw() override. Direct pushRaw() remains supported.

Performance

Measurements compare the pre-feature baseline with this branch, using independent installs with identical dependencies. The environment was a Linux VM exposing six CPUs, PHP 8.4.26, Swoole 6.3.0RC1, CLI OPcache enabled, JIT disabled, and local PostgreSQL 17.11, MySQL 9.5.0, MariaDB 10.11.19, and Redis 8.2.1. PostgreSQL durability settings stayed enabled. Job contents were 256 bytes; stored outbox envelopes were 1,246 bytes.

These measurements include the branch's related correctness changes. They measure publication, not execution of application jobs. Tables show median jobs/second with the observed range.

Ordinary dispatch, without the outbox

Ten alternating baseline/current pairs per case. Single runs dispatched 20,000 jobs; bulk runs dispatched 100,000 jobs in batches of 1,000. Warmup was 1,000 jobs.

Dispatch	Baseline jobs/s	This branch jobs/s
Single	5,074 (4,992–5,210)	5,073 (4,981–5,180)
Bulk, 1,000 per call	40,074 (39,107–41,351)	43,491 (40,829–44,366)

Single dispatch was effectively unchanged. Bulk improved by 8.5% using the ratio of medians; every matched pair improved. Median PHP CPU for bulk fell from 1.990 to 1.809 seconds. Ending heap growth was identical between revisions in both cases; the peak heap difference was about 24 KB.

Transactional staging

Three samples per engine and batch size. Single runs staged 2,000 jobs; bulk runs staged 10,000. Each call had its own transaction, so bulk amortizes commit cost.

Database	Single jobs/s	1,000 per call jobs/s	10,000 per call jobs/s
PostgreSQL	266 (260–266)	10,964 (9,425–11,241)	11,312 (11,007–12,124)
MySQL	86 (82–89)	1,046 (378–4,581)	7,504 (7,426–7,508)
MariaDB	283 (276–284)	11,281 (8,455–11,428)	11,237 (9,021–11,657)

Database wall times varied substantially while PHP CPU stayed steadier. The ranges retain those outliers; these results are not an engine ranking.

Relay

PostgreSQL source, 30,000 jobs per sample, 1,000 warmup jobs, and three samples per configuration. The table below uses four concurrent tasks and four pool slots. Claim, publication, settlement, and worker lifecycle are timed; seeding is not. Restart checks use an in-memory cache.

Target	Claim batch	Jobs/s
Redis	100	8,211 (7,770–8,421)
Redis	1,000	22,387 (14,396–22,700)
Database	100	10,963 (9,450–11,153)
Database	1,000	11,754 (6,867–14,457)

The full report also includes concurrency-one and concurrency-eight runs. Those retained four pool slots, so the concurrency-eight results include pool contention rather than isolating scaling with matched capacity.

Claim design

A PostgreSQL comparison tested the current indexed eligibility deadline against a separate, unindexed reservation deadline. Five alternating pairs per workload used eight clients, eight pool slots, batches of 100, and a mixed backlog under continuous claim/settle/refill activity.

Backlog / live leases	Current claim statement, median ms	Reservation alternative, median ms	Eligibility-index buffer hits, current / alternative
10,000 / 800	2.086	2.505	158 / 657
13,800 / 8,000	2.066	4.948	166 / 2,219
100,000 / 800	1.857	2.309	158 / 642
100,000 / 8,000	1.732	4.581	158 / 2,109

The alternative scanned live leases and showed no consistent statement-WAL saving. The production design is unchanged, and the temporary comparison code is removed. An earlier oversubscribed run hit checkout timeouts; the report retains that failure and the follow-up investigation rather than hiding it or changing timeout settings. No connection-lifetime defect was confirmed.

The retained harness and full results, including commands, ranges, runtime settings, resource checks, and diagnostic limitations, are in tests/Benchmarks/QueueOutbox/.

Verification

Local verification covered formatting, source and type-fixture static analysis, the framework tests, Testbench package contracts, and focused checks after corrections. Live database tests cover PostgreSQL, MySQL, MariaDB, and SQLite; Redis integration tests exercise actual publication and worker delivery.

Coverage includes transaction ownership and rollback, batch atomicity, producer propagation, partial and uncertain publication, lease expiry and stale settlement, cancellation, observer failures, operator commands, and tiny-pool resource lifetimes. Benchmark runs check completed counts and released connections, and clean up their own tables and queue keys. No timing thresholds were added to CI.

Summary by cubic

Adds an opt-in transactional outbox so queued jobs can be stored in the same database transaction as the business write and published by a separate relay. This closes the crash window between commit and publish without replacing existing queue drivers, dispatch APIs, or workers.

New Features

Select the outbox with viaOutbox(), the ShouldQueueViaOutbox marker, or per-connection outbox => true; withoutOutbox() overrides any default, and onOutboxDatabase() picks the owning connection.
Queued listeners, mailables, notifications, and broadcasts propagate the selection; unselected jobs keep their ordinary dispatch path without outbox access.
Setup is one migration plus a supervised queue:relay alongside existing workers; Horizon remains optional and continues managing jobs after publication.
The relay claims bounded batches, publishes through the existing drivers, and deletes only confirmed deliveries; failed and uncertain deliveries retry with jittered backoff and throttled sources.
Delivery is at-least-once, so handlers must be idempotent where repeated effects are harmful. Sync, background, deferred, and failover connections degrade to an after-commit fallback without crash durability; without a transaction, local dispatch keeps its immediate result.
Adds operator commands for listing, monitoring, retrying, and discarding pending messages, plus batch-boundary delivery events and OpenTelemetry metrics.
Includes related correctness fixes: resolved base/read/write transaction identity for records and callbacks, context-captured dispatch-lock cleanup, preserved confirmed acceptance on observer failures, finalized payload retention for tracing, and batch outbox owners validated against the batch repository before dispatch so a rolled-back business transaction cannot leave committed batch counters.
Ordinary single dispatch is effectively unchanged; bulk dispatch improved in paired measurements.

Migration

Run php artisan make:queue-outbox-table and php artisan migrate, then start queue:relay alongside queue:work.
Extensions registering callbacks directly with DatabaseTransactionsManager must use the resolved connection's getNameWithReadWriteType() instead of base getName().

^{Written for commit 2f11e2e. Summary will update on new commits.}

Note

Add opt-in transactional queue outbox with relay worker and console commands

Adds a database-backed transactional outbox. Jobs marked via viaOutbox(), onOutboxDatabase(), the ShouldQueueViaOutbox marker, or a queue connection default are staged in a job outbox table inside the dispatching transaction, then published by a relay worker after commit. See StagesOutboxJobs.php, OutboxDispatcher.php, and OutboxWorker.php.
Adds outbox publishers for Redis, SQS, database, and queue-pool transports, honoring delayed delivery, FIFO fields, and batch limits. SQS delayed jobs are staged no more than 900 seconds before delivery.
Adds console commands: queue:relay, queue:outbox (listing), queue:outbox-monitor, queue:outbox-retry, queue:outbox-forget, and make:queue-outbox-table, plus new OutboxBusy, OutboxBatchDelivered, and OutboxBatchDeliveryFailed events and OpenTelemetry delivery metrics.
Extends opt-in APIs to batches (PendingBatch::viaOutbox and friends), pending dispatches, and queued mailables, notifications, broadcasts, and listeners; unused outbox properties are not serialized.
Refactors transaction-owner identity so transaction records, callbacks, and applicability checks use the connection name with its read/write role (ManagesTransactions, DatabaseTransactionsManager, RequestContextSynchronizer). Also reworks Horizon RedisQueue payload preparation around prepareForQueueing / push-time hooks and removes the last-pushed coroutine context.
Behavioral Change: queue enqueue now raises queued events and accepts dispatch locks after transport acceptance; a post-acceptance listener failure is rethrown after all member notifications. Transaction callbacks keyed on base connection names no longer match — extensions comparing against getName() must switch to getNameWithReadWriteType(). RedisTagRepository returns early for empty tag lists. SyncQueue jobs selected for outbox are deferred until commit instead of stored.

^{Macroscope summarized 2f11e2e.}

Reconcile the outbox design with the current queue, database, coroutine, Horizon and telemetry implementations while preserving opt-in queue APIs and the unchanged ordinary publication path. Specify exact transaction-session ownership separately from relay source coverage, writer-owned staging, and post-insert rollback snapshots. Add framework prerequisites for context-safe dispatch-lock release, SQS bulk lifecycle cleanup and single-pass Horizon preparation. Require owner approval before changing role-qualified transaction-record compatibility. Use bigint epoch deadlines and available_at lease expiry, bounded retry-bucket updates, existing Redis bulk Lua publication and normal queue forwarding. Keep concurrent relay resource scopes and operational commands while removing the configuration holder, redundant validators and speculative diagnostics. Refresh schema, integration boundaries, documentation, test coverage and benchmark requirements. Keep private integrations outside the public plan and avoid duplicate tests or fixed optimizer-plan assertions. Validation: source-backed design review and full plan self-review completed; staged whitespace checks passed. Documentation only; no implementation tests or benchmarks run.

Choose rerun scope from the correction's impact rather than repeating the original broad command. Finish outstanding verification and reserve another full test-suite run for fixes with framework-wide impact.

Use the resolved base/read/write connection identity for transaction lifecycle records and connection-bound callbacks. Independent role sessions must not consume each other’s commit or rollback work, while shared in-memory SQLite aliases retain their single owner. Align Foundation wrapping transactions, request context synchronization and Pennant invalidation with exact ownership. Add role-session and rollback regressions and document the extension-facing identity convention.

Capture release context before unique or debounce lock acquisition and retain the composed wrapper with live ownership records and independent rollback snapshots. Release through the acquisition context while restoring the caller’s context afterwards. Keep registration boot-scoped, avoid closure allocation when no wrapper is installed, and clear registrations with existing framework test cleanup. Cover exact-owner release, composition, capture failures and delayed coroutine cleanup without introducing a separate lock registry.

Add the ShouldQueueViaOutbox marker and fluent viaOutbox, withoutOutbox and onOutboxDatabase options to the existing job and pending-dispatch surfaces. Keep nullable state uninitialized until selected so ordinary serialized jobs gain no fields. Apply batch defaults without overwriting initialized job choices, including jobs added later and first jobs in batched chains. Propagate effective selection through queued listeners, mail, notifications and broadcasts without introducing producer-specific APIs. Preserve explicit false over markers and null fallthrough to markers. Skip uninitialized broadcast properties while retaining initialized nulls and existing payload formatting. Cover public APIs, batch precedence and wrapper propagation.

Introduce immutable stored-message and batch-result values plus the optional PublishesOutboxMessages capability. Keep accepted, deferred and failed outcomes separate so confirmed publications are not retried when later work fails. Provide one sequential fallback for non-batching transports and pooled adapters. Defer future messages without publishing, retain accepted prefixes on ordinary failure, and propagate coroutine cancellation unchanged. Preserve the existing Queue contract and pushRaw API.

Integrate opt-in staging into the existing single and bulk queue lifecycle. Resolve exact transaction owners before writing, reject ambiguous or read-role ownership, preserve finalized payload bytes and UUIDs, and store owner groups with binding-safe atomic inserts. Reuse direct database and local-driver fast paths without outbox storage access. Add the outbox schema, migration generator, lazy configuration and repository operations. Use atomic PostgreSQL claims, short locking-select transactions where required, conditional claims elsewhere, lease-token settlement and bounded retry operations. Read monitoring counts and oldest eligibility in one aggregate query. Publish Redis batches with confirmed Lua counts, database batches through shared binding-aware insertion, and SQS batches with protocol size limits and explicit partial results. Preserve queue-default versus requested delays, FIFO options and claim-specific overflow ownership. Pooled publication borrows once per batch; generic lifecycle events retain their finalized payload correlation. Accept confirmed dispatch locks before terminal observers and retain independent rollback snapshots for transactional storage. Keep failover forwarding job-scoped, satisfy explicit-owner callbacks without reentering unrelated transactions, and preserve independent dispatches from sync handlers. Inline after-commit work retains caller ownership and avoids a second pooled borrow. Cover staging precedence, owner selection, atomic bulk rollback, driver fast paths, partial acceptance, cancellation, payload identity, failover regressions and schema generation. This checkpoint supplies staging and transport foundations; relay orchestration, operator commands and end-to-end runtime coverage remain in the active implementation plan.

Classify staged jobs while the original object is available, then add publication timing only when the stored payload reaches Redis. Preserve Horizon identifiers, stored tags, silencing and retry metadata without redispatching the command. Use the shared preparation and post-acceptance hooks for normal pushes and batched outbox publication. Keep generic lifecycle payloads stable for tracing, accept confirmed jobs before observers, and preserve confirmed batches if a later notification fails. Encode combined classification and timing once, remove the obsolete last-pushed context handoff, and document the approved protected-hook routing for Horizon subclasses. Extend direct, delayed, raw-retry and batch lifecycle coverage.

Return immediately when the requested tag list is empty instead of loading the monitored tag set from Redis. Keep normal tag intersection behavior unchanged and cover both empty and populated requests.

Document exact resolved transaction identities for integrations and Horizon’s shared protected publication hooks. Keep additive outbox APIs out of the Laravel compatibility catalogue. Update the active implementation plan to the verified staging and transport design, including failover forwarding scope, inline lock ownership, finalized lifecycle payloads, atomic claim boundaries and single-query monitoring. Mark the implemented foundations clearly while retaining relay, operations, telemetry, live-engine coverage, canonical documentation and performance verification as unfinished work.

…o stores Retain the configured store name when a custom creator returns an unnamed repository, without overriding an explicit identity or naming ad hoc repositories. Carry the underlying name through memoization while preserving event suppression. This lets unique-job cleanup resolve the owning cache store when queued model restoration fails. Cover custom, memoized and memo-tagged repositories, explicit-name preservation, duplicate event suppression and the missing-model cleanup path.

Separate claims, connection-group publication, settlement and observers into fresh child coroutines. Join cleanup before advancing or propagating cancellation so blocked broker operations never retain a source database connection. Validate accepted and deferred row IDs without imposing array-key shape, settle only confirmed outcomes, and report publication and settlement failures independently. Observer failures cannot republish settled rows or throttle healthy delivery; reporter failures propagate. Add guarded batch delivery events and focused result, partial-failure, cancellation and observer tests. Exercise real single-slot database and queue pools to prove connection availability during blocked publication and cleanup after cancellation.

…down Admit serial round-robin claims into bounded concurrent publication tasks, cap each claim against the remaining job allowance, and cool down failed sources without delaying healthy databases. Support once, stop-when-empty, source pinning, restart signals, pause and resume, memory and time limits. Keep invocation state local, isolate restart-cache reads and exception reporting, and join owned tasks before cancellation leaves the worker. Hold a failed delivery task through its retry sleep to protect unavailable transports from a fresh-row backlog. Test exact admission limits, concurrency, source fairness, lifecycle controls, resource cleanup and error propagation.

Register queue:relay and bounded list, monitor, retry and forget commands. Expose readable UTC dates and compact errors in tables while preserving raw diagnostic values in JSON, and report mutually exclusive backlog states with optional threshold alerts. Resolve ambiguous job UUIDs explicitly, retain atomic live-lease guards and preserve retry history. Confirm destructive operations before querying; force skips only confirmation, never an active publication lease. Forget deletes the row before exact-owner unique-lock cleanup, reads only serialized scalar ownership metadata and reports post-delete cleanup failures without restoring discarded work. Cover command options, scripting, listing bounds, monitoring, source ambiguity and replacement-lock safety.

Record confirmed delivered messages, batch publication duration and oldest eligible message age from guarded outbox events. Register listeners only when their metrics are enabled, without querying backlog, resolving queues, decoding payloads or starting another producer lifecycle. Use error-tagged duration observations for failed-batch counts instead of a redundant counter. Keep latency attributes independent of exception type and align its buckets with whole-second values to limit series and recording overhead. Expose matching published and normalized defaults, with tests for partial acceptance, settlement failure, disabled listeners and bounded metric series.

Specify coordinator-owned admission, resource-isolated delivery phases, cancellation cleanup, observer reporting and value-based result validation. Clarify cache identity and exact unique-lock cleanup, scriptable confirmation and disjoint monitoring states. Document the reduced batch telemetry surface and remaining OpenTelemetry reference updates. Record completed runtime and operator scope, with live-engine coverage, remaining API integration, documentation, approved performance checks and final verification still required.

Cover inherited Beanstalkd and custom-driver bulk staging while preserving direct partitions. Require multiple settlement rows per outcome to complete in three statements, and verify bounded retry-all updates exclude later arrivals. Exercise custom non-PDO connections through the driver-neutral repository contract, retaining real SQL behavior without requiring PDO capability methods.

Share live-engine coverage for bounded claim statements, original eligibility, independent write-role transaction ownership, competing claims and expired-lease settlement protection. Verify skip-locked progress past an uncommitted row and force competing fallback candidates on file-backed SQLite. Keep engine entry points in the existing CI discovery layout and migrate only the shipped outbox schema.

Exercise queued listeners, mailables, notifications and broadcasts through their existing public APIs and wrapper jobs, without broker access during staging. Run committed outbox payloads through the real Redis relay and worker, covering encrypted model restoration, stable payloads, unique-lock ownership, debounce, delays, closures, batches and nested chains. Reuse existing event-deferral coverage instead of duplicating it.

Add one canonical guide for selection, transactional ownership, driver behavior, delays, relay deployment, concurrency, recovery, operator commands and custom driver publication. Link the guide from queue and producer documentation. Explain whole-event after-commit timing, local-driver limitations, at-least-once processing and idempotency without duplicating the operating reference across pages.

Document queue-confirmed delivery counts, publication duration and eligible-age observations at connection-batch boundaries. Explain failure counting through duration attributes and the absence of backlog polling or a second producer lifecycle. Remove stale guidance that ordinary later batch-finalizer failures leave earlier producer states unfinished now that guarded queue lifecycles terminate outstanding work.

Provide developer-only direct dispatch, transactional staging, real relay and concurrent claim-churn cases using isolated tables and queue names. Verify completed counts, released pool connections and resource cleanup on every run. Capture executing engine plans outside timing and compare PostgreSQL availability-index claims with a narrowly scoped reservation fixture. Make seeded lease counts explicit and label workload WAL, in-memory restart checks, CPU and heap measurements accurately. Keep baseline dispatch independent of feature-only fixtures, document fresh-process comparison methodology, and retain raw reports outside Git. Functional smoke checks are complete; performance measurements require an idle-machine window.

…cope Mark implementation, coverage, documentation and the functionally verified benchmark harness complete. Retain the explicitly deferred idle-machine measurements without claiming performance results. Specify focused benchmark cases and realistic lease populations, preserve automated query and resource-lifetime guarantees, and align verification instructions with scoped reruns after localized corrections.

Warn that redeclaring outbox properties with defaults conflicts with the declarations supplied by Queueable and Batchable. Direct users to the marker for class defaults and fluent methods for individual objects. Limit public property-default guidance to classes without those trait-provided properties, including queued listeners whose defaults are inspected without construction.

Require the marker for queued listeners using Queueable or Batchable. Both traits provide uninitialized outbox properties, so redeclaring those properties with defaults is a PHP conflict rather than an alternative convention. Align the active implementation plan with the canonical guide. No runtime behavior changes.

Remove the temporary PostgreSQL reservation-column fixture and its CLI selector after comparing it with the production claim path. Keep the retained harness focused on production dispatch, staging, relay and claim workloads. The comparison increased live-lease scan work without a consistent statement-WAL saving, so no production schema or claim changes are needed. Remove the unused schema import, option validation and alternate backlog setup. Verified targeted formatting and PHP syntax, all four tiny harness cases, both relay targets, PostgreSQL executing plans, exact counts and released resources. Rechecked SQLite claims after the final fixture cleanup.

Record paired ordinary-dispatch measurements, transactional staging, real relay throughput and concurrent claim churn across PostgreSQL, MySQL and MariaDB. Include runtime settings, sample sizes, medians and ranges, CPU and heap observations, and the PostgreSQL claim-plan and WAL comparison. Keep the failed oversubscribed sample and its investigation visible without claiming an unproven cause. Explain that claim latency includes checkout and that the relay concurrency-eight samples used four pool slots, rather than presenting them as matched-capacity scaling results. Update reproducible examples to match pool capacity to concurrency, link the measured report, and remove obsolete reservation-selector instructions. Preserve raw evidence outside Git; temporary benchmark resources have been removed. Production behavior remains unchanged.

Mark implementation and performance measurements complete and link the retained benchmark report. Replace deferred-measurement and temporary-comparison instructions with the measured rationale for retaining indexed eligibility and the production claim path. Clarify heap versus allocation measurements and the checkout component of claim latency. Record the requirement to include performance tables with workload, concurrency and pool settings when opening the pull request.

coderabbitai · 2026-10-23T03:14:32Z

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Source mode: Auto
Repository: hypervel/components-backup
Branch: feature/queue-outbox
File: .coderabbit.yaml
Commit: 2f11e2e
Layers: Repository YAML
Review profile: CHILL
Plan: Advanced
Run ID: eea4c6fb-e83d-4911-8b07-04685b003ece

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

qodo-free-for-open-source-projects · 2026-10-23T03:16:58Z

PR Summary by Qodo

Add an opt-in transactional queue outbox and relay

✨ Enhancement 🐞 Bug fix 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Store selected jobs with business writes so committed transactions retain their jobs.
• Relay committed jobs through existing queue drivers with retries and operator controls.
• Fix transaction ownership, lock cleanup, and publication lifecycle behavior; add integration
 coverage.

Diagram

graph TD
  A["Job dispatch"] --> B["Outbox selection"] --> C["Transactional staging"] --> D[("Outbox table")] --> E["Relay worker"] --> F["Queue drivers"] --> G["Existing workers"]
  B -->|not selected| F

High-Level Assessment

The following are alternative approaches to this PR:

1. Use after-commit publication only

➕ No outbox table or relay to operate.
➖ A crash after commit can permanently lose the job.

2. Publish through database change-data capture

➕ Can reuse an existing CDC pipeline where one is already operated.
➖ Adds infrastructure and database-specific integration; complicates reuse of existing queue payloads and drivers.

Recommendation: Keep the opt-in SQL outbox and separate relay: it closes the commit-to-publication loss window while retaining current dispatch APIs and workers. Reviewers should verify lease ownership, ambiguous delivery, and the documented at-least-once limit particularly closely.

Files changed (132) +10445 / -327

Enhancement (44) +3057 / -256

BroadcastEvent.phpPropagate broadcast outbox selection +16/-0

Propagate broadcast outbox selection
• Copies outbox options to the queued broadcast wrapper and skips uninitialized public event properties.
src/broadcasting/src/BroadcastEvent.php

Batch.phpApply outbox defaults to added batch jobs +15/-3

Apply outbox defaults to added batch jobs
• Propagates stored batch outbox options to jobs and chains added after batch creation without overriding explicit job settings.
src/bus/src/Batch.php

Batchable.phpExpose outbox options on batchable jobs +2/-0

Expose outbox options on batchable jobs
• Makes the shared outbox-selection methods available to batchable jobs.
src/bus/src/Batchable.php

InteractsWithQueueOutbox.phpAdd per-job outbox selection methods +63/-0

Add per-job outbox selection methods
• Introduces viaOutbox(), withoutOutbox(), and onOutboxDatabase() without adding initialized state to ordinary jobs.
src/bus/src/InteractsWithQueueOutbox.php

PendingBatch.phpConfigure outbox defaults on pending batches +40/-0

Configure outbox defaults on pending batches
• Adds batch-level opt-in, opt-out, and owning-database selection.
src/bus/src/PendingBatch.php

Queueable.phpMake queueable jobs outbox-selectable +2/-0

Make queueable jobs outbox-selectable
• Includes the shared outbox-selection trait in queueable jobs.
src/bus/src/Queueable.php

PublishesOutboxMessages.phpDefine batched outbox publication contract +19/-0

Define batched outbox publication contract
• Provides a transport interface for publishing stored messages and reporting confirmed outcomes.
src/contracts/src/Queue/PublishesOutboxMessages.php

ShouldQueueViaOutbox.phpDefine job-level outbox marker +9/-0

Define job-level outbox marker
• Adds a contract for jobs that select outbox delivery by class.
src/contracts/src/Queue/ShouldQueueViaOutbox.php

Dispatcher.phpPropagate listener outbox options +11/-0

Propagate listener outbox options
• Passes listener-level selection and owning-database settings to queued listener jobs.
src/events/src/Dispatcher.php

PendingDispatch.phpExpose outbox choices during pending dispatch +30/-0

Expose outbox choices during pending dispatch
• Adds fluent outbox selection and owner methods before dispatch completes.
src/foundation/src/Bus/PendingDispatch.php

RedisQueue.phpIntegrate Horizon with shared publication lifecycle +35/-58

Integrate Horizon with shared publication lifecycle
• Prepares staged classifications, stamps relayed jobs at publication, and avoids routing normal push through the public raw-push override.
src/horizon/src/RedisQueue.php

RedisTagRepository.phpHandle relayed Horizon tag records +4/-0

Handle relayed Horizon tag records
• Adjusts tag recording for the staged-and-published job lifecycle.
src/horizon/src/Repositories/RedisTagRepository.php

SendQueuedMailable.phpForward mailable outbox options +11/-0

Forward mailable outbox options
• Transfers the original mailable's outbox choice and database to its queued wrapper.
src/mail/src/SendQueuedMailable.php

SendQueuedNotifications.phpForward notification outbox options +11/-0

Forward notification outbox options
• Transfers notification outbox selection and database to queued notification jobs.
src/notifications/src/SendQueuedNotifications.php

QueueInstrumentation.phpRecord relay delivery metrics +77/-0

Record relay delivery metrics
• Observes batch delivery events for confirmed counts, publication duration, and eligible-message age.
src/opentelemetry/src/Instrumentation/QueueInstrumentation.php

BeanstalkdQueue.phpResolve Beanstalkd outbox destination +16/-0

Resolve Beanstalkd outbox destination
• Preserves the logical queue destination needed when staged messages are later published.
src/queue/src/BeanstalkdQueue.php

StagesOutboxJobs.phpStage selected jobs on their transaction owner +271/-0

Stage selected jobs on their transaction owner
• Selects jobs, prepares finalized payloads, groups bulk inserts by owner, and retains dispatch lifecycle and rollback behavior.
src/queue/src/Concerns/StagesOutboxJobs.php

FindsOutboxMessages.phpShare operator message lookup +34/-0

Share operator message lookup
• Locates a UUID across configured outbox databases for retry and removal commands.
src/queue/src/Console/Concerns/FindsOutboxMessages.php

ForgetOutboxCommand.phpAdd deliberate outbox removal command +117/-0

Add deliberate outbox removal command
• Deletes an unleased message after confirmation and releases its uniquely owned cache lock.
src/queue/src/Console/ForgetOutboxCommand.php

ListOutboxCommand.phpAdd pending outbox inspection +90/-0

Add pending outbox inspection
• Lists bounded pending messages for operator diagnosis.
src/queue/src/Console/ListOutboxCommand.php

MonitorOutboxCommand.phpAdd outbox backlog monitoring +68/-0

Add outbox backlog monitoring
• Reports backlog and age against operator-selected thresholds.
src/queue/src/Console/MonitorOutboxCommand.php

OutboxTableCommand.phpGenerate outbox table migration +45/-0

Generate outbox table migration
• Adds the make:queue-outbox-table migration generator.
src/queue/src/Console/OutboxTableCommand.php

RelayCommand.phpExpose the supervised relay command +61/-0

Expose the supervised relay command
• Adds source selection and limits for batch size, concurrency, runtime, memory, and one-shot operation.
src/queue/src/Console/RelayCommand.php

RetryOutboxCommand.phpAdd pending-message retry controls +74/-0

Add pending-message retry controls
• Allows operators to reschedule retryable messages without taking over live leases.
src/queue/src/Console/RetryOutboxCommand.php

DatabaseQueue.phpReuse atomic database inserts and publish batches +147/-30

Reuse atomic database inserts and publish batches
• Bypasses redundant staging when a database job already shares the owner transaction, adds rollback-lock cleanup, and supports set-based relay publication.
src/queue/src/DatabaseQueue.php

OutboxBatchDelivered.phpEmit confirmed batch delivery +20/-0

Emit confirmed batch delivery
• Defines a batch-boundary event carrying accepted counts and delivery timing.
src/queue/src/Events/OutboxBatchDelivered.php

OutboxBatchDeliveryFailed.phpEmit failed batch delivery +24/-0

Emit failed batch delivery
• Defines failure details and confirmed acceptance counts for relay observers.
src/queue/src/Events/OutboxBatchDeliveryFailed.php

OutboxBusy.phpDescribe busy outbox status +19/-0

Describe busy outbox status
• Adds an event for operator backlog monitoring.
src/queue/src/Events/OutboxBusy.php

FailoverQueue.phpPreserve failover semantics for selected jobs +47/-3

Preserve failover semantics for selected jobs
• Uses an after-commit fallback instead of staging an undecided destination and prevents child attempts from restaging.
src/queue/src/FailoverQueue.php

NullQueue.phpKeep null-queue dispatch out of storage +8/-0

Keep null-queue dispatch out of storage
• Prevents selected jobs on a null transport from creating outbox rows.
src/queue/src/NullQueue.php

PublishesOutboxMessagesSequentially.phpProvide raw-payload publication fallback +51/-0

Provide raw-payload publication fallback
• Publishes stored payloads through drivers without a specialized batch publisher while tracking confirmed results.
src/queue/src/Outbox/Concerns/PublishesOutboxMessagesSequentially.php

DatabaseOutboxRepository.phpImplement durable claims and settlement +403/-0

Implement durable claims and settlement
• Inserts rows, claims with engine-appropriate concurrency control, settles by lease token, backs off failures, and supports operator queries.
src/queue/src/Outbox/DatabaseOutboxRepository.php

OutboxBatchResult.phpRepresent publication outcomes +23/-0

Represent publication outcomes
• Carries accepted and deferred row IDs plus publication failure information.
src/queue/src/Outbox/OutboxBatchResult.php

OutboxDispatcher.phpResolve transaction-owning outbox storage +87/-0

Resolve transaction-owning outbox storage
• Rejects ambiguous or uncovered owners, inserts through the owning session, and registers post-insert rollback-lock snapshots.
src/queue/src/Outbox/OutboxDispatcher.php

OutboxMessage.phpRepresent a claimed stored message +26/-0

Represent a claimed stored message
• Provides a connection-free value object for relay publication and settlement.
src/queue/src/Outbox/OutboxMessage.php

OutboxRelay.phpPublish and settle isolated relay batches +228/-0

Publish and settle isolated relay batches
• Groups claims by destination, validates confirmed outcomes, separates resource scopes, and emits delivery events.
src/queue/src/Outbox/OutboxRelay.php

OutboxWorker.phpCoordinate concurrent relay execution +218/-0

Coordinate concurrent relay execution
• Polls configured sources with bounded concurrency, throttles failures, and handles restart signals and graceful shutdown.
src/queue/src/Outbox/OutboxWorker.php

OutboxWorkerOptions.phpValidate relay runtime options +36/-0

Validate relay runtime options
• Defines bounded batch, concurrency, sleep, memory, and stopping settings.
src/queue/src/Outbox/OutboxWorkerOptions.php

Queue.phpIntegrate opt-in staging with queue lifecycle +128/-11

Integrate opt-in staging with queue lifecycle
• Routes selected single and bulk jobs to staging while keeping ordinary publication lazy and preserving finalized lifecycle payloads.
src/queue/src/Queue.php

QueuePoolProxy.phpPublish batches on one pooled queue lease +21/-3

Publish batches on one pooled queue lease
• Delegates specialized publication or the raw fallback while borrowing one transport per batch.
src/queue/src/QueuePoolProxy.php

QueueServiceProvider.phpRegister outbox services and commands +31/-0

Register outbox services and commands
• Registers a lazily resolved repository and the relay, migration, inspection, monitoring, retry, and forget commands.
src/queue/src/QueueServiceProvider.php

RedisQueue.phpBatch Redis outbox publication +107/-36

Batch Redis outbox publication
• Reuses the bulk Lua script for stored payloads and keeps acceptance distinct from post-publication observer failures.
src/queue/src/RedisQueue.php

SqsQueue.phpPublish SQS outbox batches with partial outcomes +273/-111

Publish SQS outbox batches with partial outcomes
• Preserves transport options and delay limits, tracks individual confirmations, and isolates overflow-payload cleanup by attempt.
src/queue/src/SqsQueue.php

SyncQueue.phpFallback to after-commit local execution +39/-1

Fallback to after-commit local execution
• Keeps selected sync and inherited local queues out of durable storage while honoring the selected transaction.
src/queue/src/SyncQueue.php

Bug fix (12) +176 / -28

DebounceLock.phpCapture debounce-lock release context +2/-1

Capture debounce-lock release context
• Records the dispatch-time context needed to release acquired debounce locks later.
src/bus/src/DebounceLock.php

DispatchLockContext.phpSnapshot context-aware dispatch locks +117/-5

Snapshot context-aware dispatch locks
• Adds composable release-context capture and independent ownership snapshots for deferred or rollback cleanup.
src/bus/src/DispatchLockContext.php

UniqueLock.phpCapture unique-lock release context +2/-1

Capture unique-lock release context
• Associates acquired unique locks with their dispatch-time release context.
src/bus/src/UniqueLock.php

CacheManager.phpRetain named cache-store identity +9/-2

Retain named cache-store identity
• Preserves repository store names, including for custom or memoized repositories used in unique-lock cleanup.
src/cache/src/CacheManager.php

Repository.phpAllow repository store names to be recorded +15/-0

Allow repository store names to be recorded
• Adds an internal setter for the name associated with a shared cache repository.
src/cache/src/Repository.php

ManagesTransactions.phpKey transaction callbacks by exact session +7/-7

Key transaction callbacks by exact session
• Uses role-qualified connection identities for transaction records and commit or rollback callbacks.
src/database/src/Concerns/ManagesTransactions.php

Connection.phpPreserve transaction session identity +1/-1

Preserve transaction session identity
• Aligns connection transaction handling with resolved base, read, or write session names.
src/database/src/Connection.php

DatabaseTransactionsManager.phpResolve role-qualified transaction records +2/-2

Resolve role-qualified transaction records
• Adjusts transaction-manager behavior for exact connection-session ownership.
src/database/src/DatabaseTransactionsManager.php

DatabaseTransactionsManager.phpKeep testing callbacks session-specific +1/-1

Keep testing callbacks session-specific
• Aligns test transaction tracking with role-qualified connection identities.
src/foundation/src/Testing/DatabaseTransactionsManager.php

RequestContextSynchronizer.phpSynchronize exact transaction identities +1/-1

Synchronize exact transaction identities
• Keeps request test context consistent with resolved transaction sessions.
src/foundation/src/Testing/RequestContextSynchronizer.php

JobPayload.phpSeparate Horizon classification from push time +17/-4

Separate Horizon classification from push time
• Allows classification when staging and publication timestamps when jobs actually enter Redis.
src/horizon/src/JobPayload.php

Decorator.phpRespect resolved transaction sessions +2/-3

Respect resolved transaction sessions
• Adjusts feature-flag transaction integration to use the correct connection identity.
src/pennant/src/Drivers/Decorator.php

Refactor (1) +12 / -2

InsertsDatabaseRows.phpShare bounded database inserts +12/-2

Share bounded database inserts
• Lets queue and outbox storage use the same binding-aware, set-based insertion helper.
src/queue/src/Concerns/InsertsDatabaseRows.php

Tests (55) +5950 / -39

0001_01_01_000009_testbench_create_job_outbox_table.phpProvision Testbench outbox storage +40/-0

Provision Testbench outbox storage
• Adds the outbox schema to package test migrations.
src/testbench/hypervel/migrations/0001_01_01_000009_testbench_create_job_outbox_table.php

OutboxBenchmarkJob.phpDefine benchmark job fixture +27/-0

Define benchmark job fixture
• Provides a stable payload for dispatch and staging measurements.
tests/Benchmarks/QueueOutbox/Fixtures/OutboxBenchmarkJob.php

OutboxWorkload.phpBuild repeatable outbox workloads +369/-0

Build repeatable outbox workloads
• Seeds, dispatches, relays, verifies counts, and cleans up benchmark resources.
tests/Benchmarks/QueueOutbox/Fixtures/OutboxWorkload.php

benchmark.phpAdd executable outbox benchmark harness +276/-0

Add executable outbox benchmark harness
• Measures ordinary dispatch, transactional staging, and relay configurations without CI timing assertions.
tests/Benchmarks/QueueOutbox/benchmark.php

BroadcastEventTest.phpTest queued broadcast option propagation +22/-0

Test queued broadcast option propagation
• Checks outbox settings and uninitialized broadcast properties.
tests/Broadcasting/BroadcastEventTest.php

BusBatchTest.phpTest batch outbox defaults +46/-0

Test batch outbox defaults
• Verifies selection on batch members and later additions.
tests/Bus/BusBatchTest.php

DispatchLockContextTest.phpExtend dispatch-lock context coverage +1/-0

Extend dispatch-lock context coverage
• Covers updated lock-context behavior.
tests/Bus/DispatchLockContextTest.php

DispatchLockReleaseContextTest.phpTest deferred lock release contexts +248/-0

Test deferred lock release contexts
• Exercises captured contexts, snapshots, and release across dispatch and rollback boundaries.
tests/Bus/DispatchLockReleaseContextTest.php

OutboxOptionsTest.phpTest job and batch selection options +107/-0

Test job and batch selection options
• Verifies fluent opt-in, opt-out, and explicit owning-database behavior.
tests/Bus/OutboxOptionsTest.php

CacheManagerTest.phpTest cache repository identity +43/-0

Test cache repository identity
• Checks store-name preservation for custom and reused repositories.
tests/Cache/CacheManagerTest.php

DatabaseTransactionIdentityTest.phpTest exact transaction-session ownership +113/-0

Test exact transaction-session ownership
• Verifies that base, read, and write sessions do not consume one another's callbacks.
tests/Database/DatabaseTransactionIdentityTest.php

DatabaseTransactionsManagerTest.phpTest session-specific test transactions +32/-0

Test session-specific test transactions
• Covers transaction-manager behavior under role-qualified connection names.
tests/Foundation/Testing/DatabaseTransactionsManagerTest.php

RefreshDatabaseTest.phpAlign refresh-database expectations +2/-2

Align refresh-database expectations
• Updates assertions for resolved transaction-session identities.
tests/Foundation/Testing/RefreshDatabaseTest.php

RequestContextSynchronizerTest.phpTest synchronized transaction context +34/-0

Test synchronized transaction context
• Checks request-context handling for distinct database sessions.
tests/Foundation/Testing/RequestContextSynchronizerTest.php

RedisQueueTest.phpTest Horizon publication lifecycle +152/-0

Test Horizon publication lifecycle
• Checks staged classification, push-time preparation, raw dispatch, and observer behavior.
tests/Horizon/Unit/RedisQueueTest.php

RedisTagRepositoryTest.phpTest Horizon tag persistence +21/-0

Test Horizon tag persistence
• Covers tag repository behavior for the revised publication lifecycle.
tests/Horizon/Unit/RedisTagRepositoryTest.php

QueueOutboxTableCommandTest.phpTest outbox migration generation +29/-0

Test outbox migration generation
• Verifies the generator produces the expected migration.
tests/Integration/Generators/QueueOutboxTableCommandTest.php

QueueProcessingTest.phpRemove obsolete Horizon context test +0/-34

Remove obsolete Horizon context test
• Removes a test and helper tied to the deleted last-pushed context mechanism.
tests/Integration/Horizon/Feature/QueueProcessingTest.php

RedisPayloadTest.phpTest relayed Horizon payloads +53/-0

Test relayed Horizon payloads
• Checks classification and publication metadata in Redis integration.
tests/Integration/Horizon/Feature/RedisPayloadTest.php

OutboxClaimTest.phpExercise MariaDB outbox claims +13/-0

Exercise MariaDB outbox claims
• Runs shared claim and lease scenarios against MariaDB.
tests/Integration/Queue/Database/MariaDb/OutboxClaimTest.php

OutboxClaimTest.phpExercise MySQL outbox claims +13/-0

Exercise MySQL outbox claims
• Runs shared claim and lease scenarios against MySQL.
tests/Integration/Queue/Database/MySql/OutboxClaimTest.php

OutboxClaimTest.phpExercise PostgreSQL outbox claims +13/-0

Exercise PostgreSQL outbox claims
• Runs shared claim and lease scenarios against PostgreSQL.
tests/Integration/Queue/Database/Postgres/OutboxClaimTest.php

OutboxClaimTest.phpExercise SQLite outbox claims +56/-0

Exercise SQLite outbox claims
• Tests conditional-update claims and settlement on SQLite.
tests/Integration/Queue/Database/Sqlite/OutboxClaimTest.php

WorkerResourceLifetimeTest.phpTest relay connection lifetimes +117/-0

Test relay connection lifetimes
• Uses constrained SQLite resources to check claim, publication, and settlement separation.
tests/Integration/Queue/Database/Sqlite/WorkerResourceLifetimeTest.php

OutboxClaimTestCase.phpShare cross-engine claim scenarios +242/-0

Share cross-engine claim scenarios
• Defines concurrency, lease-expiry, retry, and stale-settlement assertions for database integrations.
tests/Integration/Queue/Fixtures/OutboxClaimTestCase.php

QueueConnectionTest.phpCheck queue outbox configuration wiring +4/-0

Check queue outbox configuration wiring
• Verifies queue connections receive their outbox defaults.
tests/Integration/Queue/QueueConnectionTest.php

OutboxDeliveryTest.phpTest end-to-end Redis relay delivery +199/-0

Test end-to-end Redis relay delivery
• Stages committed jobs and verifies publication and worker delivery against Redis.
tests/Integration/Queue/Redis/OutboxDeliveryTest.php

UniqueJobTest.phpTest unique-job outbox cleanup +43/-3

Test unique-job outbox cleanup
• Covers unique-lock ownership across acceptance, rollback, and cleanup.
tests/Integration/Queue/UniqueJobTest.php

QueueInstrumentationTest.phpTest outbox metric recording +39/-0

Test outbox metric recording
• Checks batch-event instruments without duplicating producer observations.
tests/OpenTelemetry/Instrumentation/QueueInstrumentationTest.php

PublishedConfigurationTest.phpTest published metric configuration +3/-0

Test published metric configuration
• Asserts outbox instruments appear in published telemetry settings.
tests/OpenTelemetry/PublishedConfigurationTest.php

TransactionTest.phpTest Pennant transaction isolation +39/-0

Test Pennant transaction isolation
• Checks feature-flag behavior across role-qualified transaction sessions.
tests/Pennant/TransactionTest.php

DatabaseQueueRollbackTest.phpTest atomic database-job rollback cleanup +176/-0

Test atomic database-job rollback cleanup
• Verifies direct database inserts and accepted locks follow owning-transaction rollback.
tests/Queue/DatabaseQueueRollbackTest.php

FailoverQueueTest.phpTest selected failover fallback +53/-0

Test selected failover fallback
• Checks after-commit attempts and preservation of child failover behavior.
tests/Queue/FailoverQueueTest.php

DatabaseOutboxPublicationTest.phpTest batched database publication +101/-0

Test batched database publication
• Verifies stored messages are inserted into database queues with intended availability.
tests/Queue/Outbox/DatabaseOutboxPublicationTest.php

DatabaseOutboxRepositoryTest.phpTest repository claim and operations +287/-0

Test repository claim and operations
• Covers insertion, ownership-checked settlement, retry, statistics, and safe removal.
tests/Queue/Outbox/DatabaseOutboxRepositoryTest.php

OutboxCommandsTest.phpTest outbox operator commands +254/-0

Test outbox operator commands
• Exercises inspection, monitoring, retry, and deliberate forget behavior.
tests/Queue/Outbox/OutboxCommandsTest.php

OutboxDispatchTest.phpTest outbox dispatch lifecycles +382/-0

Test outbox dispatch lifecycles
• Checks selected dispatch, bulk staging, payload metadata, and acceptance events.
tests/Queue/Outbox/OutboxDispatchTest.php

OutboxDispatcherTest.phpTest owner selection and staging +95/-0

Test owner selection and staging
• Checks explicit ownership, ambiguity rejection, and transaction-aware storage.
tests/Queue/Outbox/OutboxDispatcherTest.php

OutboxFastPathTest.phpProtect ordinary dispatch paths +357/-0

Protect ordinary dispatch paths
• Checks unselected jobs avoid outbox storage or service resolution.
tests/Queue/Outbox/OutboxFastPathTest.php

OutboxProducerTest.phpTest producer wrapper propagation +150/-0

Test producer wrapper propagation
• Covers queued listener, mail, notification, and broadcast selections.
tests/Queue/Outbox/OutboxProducerTest.php

OutboxRelayTest.phpTest relay result accounting +290/-0

Test relay result accounting
• Exercises partial acceptance, deferred work, failure reporting, and settlement validation.
tests/Queue/Outbox/OutboxRelayTest.php

OutboxTransactionTest.phpTest atomic staging and rollback +180/-0

Test atomic staging and rollback
• Checks owner-session inserts, rollback removal, and multi-connection selection.
tests/Queue/Outbox/OutboxTransactionTest.php

OutboxWorkerTest.phpTest relay coordination and shutdown +277/-0

Test relay coordination and shutdown
• Checks source polling, throttling, limits, signals, and concurrent task completion.
tests/Queue/Outbox/OutboxWorkerTest.php

OutboxWrapperTest.phpTest wrapper queue compatibility +167/-0

Test wrapper queue compatibility
• Checks selected and unselected behavior for non-durable and forwarding queue types.
tests/Queue/Outbox/OutboxWrapperTest.php

PooledOutboxPublicationTest.phpTest pooled publisher ownership +128/-0

Test pooled publisher ownership
• Verifies one borrowed connection per publication batch and timely release.
tests/Queue/Outbox/PooledOutboxPublicationTest.php

RedisOutboxPublicationTest.phpTest Redis publication batches +158/-0

Test Redis publication batches
• Checks bulk-script confirmation, destinations, delays, and failure accounting.
tests/Queue/Outbox/RedisOutboxPublicationTest.php

SqsOutboxPublicationTest.phpTest SQS relay publication +187/-0

Test SQS relay publication
• Covers request chunking, FIFO options, delays, partial responses, and overflow ownership.
tests/Queue/Outbox/SqsOutboxPublicationTest.php

QueueBackgroundQueueTest.phpTest background fallback +11/-0

Test background fallback
• Checks selected jobs preserve background execution behavior.
tests/Queue/QueueBackgroundQueueTest.php

QueueDatabaseQueueUnitTest.phpTest database queue selection +7/-0

Test database queue selection
• Covers direct transactional database publication and outbox selection.
tests/Queue/QueueDatabaseQueueUnitTest.php

QueueDeferredQueueTest.phpTest deferred fallback +10/-0

Test deferred fallback
• Checks selected jobs retain deferred execution semantics.
tests/Queue/QueueDeferredQueueTest.php

QueueRedisQueueTest.phpTest Redis queue integration +5/-0

Test Redis queue integration
• Extends queue tests for outbox-related publication behavior.
tests/Queue/QueueRedisQueueTest.php

QueueSyncQueueTest.phpTest sync fallback +6/-0

Test sync fallback
• Checks selected jobs execute after commit without durable staging.
tests/Queue/QueueSyncQueueTest.php

RelayCommandTest.phpTest relay command options +80/-0

Test relay command options
• Verifies command arguments and worker-option forwarding.
tests/Queue/RelayCommandTest.php

SqsBatchLifecycleTest.phpTest SQS batch acceptance lifecycle +192/-0

Test SQS batch acceptance lifecycle
• Checks partial acceptance, notification failures, lock handling, and overflow cleanup.
tests/Queue/SqsBatchLifecycleTest.php

CommanderTest.phpRegister Testbench outbox commands +1/-0

Register Testbench outbox commands
• Checks the new queue operator commands are available in Testbench.
tests/Testbench/CommanderTest.php

Documentation (15) +1170 / -2

AGENTS.mdClarify repository guidance +1/-1

Clarify repository guidance
• Adjusts contributor guidance for the expanded queue work.
AGENTS.md

2026-10-22-1522-hypervel-transactional-queue-outbox-codex.mdRecord the outbox implementation plan +595/-0

Record the outbox implementation plan
• Documents transaction ownership, relay design, driver integration, operations, verification, and performance requirements.
docs/plans/2026-10-22-1522-hypervel-transactional-queue-outbox-codex.md

README.mdLink database outbox guidance +1/-0

Link database outbox guidance
• Points database readers to the transactional queue outbox documentation.
src/database/README.md

broadcasting.mdDocument outbox-backed broadcasts +14/-0

Document outbox-backed broadcasts
• Explains outbox selection and limitations for queued broadcasts.
src/docs/broadcasting.md

documentation.mdIndex the outbox guide +1/-0

Index the outbox guide
• Adds the transactional outbox guide to the documentation index.
src/docs/documentation.md

events.mdDocument queued-listener outbox usage +13/-0

Document queued-listener outbox usage
• Explains how queued event listeners select outbox delivery.
src/docs/events.md

mail.mdDocument queued-mail outbox usage +10/-0

Document queued-mail outbox usage
• Describes outbox selection for queued mailables.
src/docs/mail.md

notifications.mdDocument queued-notification outbox usage +8/-0

Document queued-notification outbox usage
• Describes outbox selection for queued notifications.
src/docs/notifications.md

opentelemetry.mdDocument outbox delivery metrics +10/-1

Document outbox delivery metrics
• Describes relay metrics and how publication telemetry relates to producer tracing.
src/docs/opentelemetry.md

porting-from-laravel.mdNote outbox portability behavior +4/-0

Note outbox portability behavior
• Adds outbox-related guidance for applications porting from Laravel.
src/docs/porting-from-laravel.md

queue-outbox.mdAdd transactional outbox operations guide +319/-0

Add transactional outbox operations guide
• Covers setup, selection, delivery guarantees, driver behavior, recovery, commands, and monitoring.
src/docs/queue-outbox.md

queues.mdLink queue users to the outbox guide +18/-0

Link queue users to the outbox guide
• Introduces opt-in outbox usage from the main queue documentation.
src/docs/queues.md

README.mdLink Horizon outbox documentation +1/-0

Link Horizon outbox documentation
• Points Horizon users to the outbox integration guide.
src/horizon/README.md

README.mdDocument benchmark reproduction +67/-0

Document benchmark reproduction
• Records setup, commands, measurement scope, and interpretation constraints.
tests/Benchmarks/QueueOutbox/README.md

results.mdRecord measured outbox performance +108/-0

Record measured outbox performance
• Reports baseline comparisons, database staging, relay throughput, and claim-design investigation.
tests/Benchmarks/QueueOutbox/results.md

Other (5) +80 / -0

queue.phpPublish opt-in outbox configuration +32/-0

Publish opt-in outbox configuration
• Defaults queue connections to ordinary dispatch and configures outbox source databases, table, and retry timeout.
src/foundation/config/queue.php

opentelemetry.phpConfigure outbox telemetry +3/-0

Configure outbox telemetry
• Adds outbox delivery instruments to published telemetry configuration.
src/opentelemetry/config/opentelemetry.php

ConfigurationNormalizer.phpNormalize outbox metric settings +3/-0

Normalize outbox metric settings
• Includes outbox instruments in telemetry configuration normalization.
src/opentelemetry/src/Support/ConfigurationNormalizer.php

SqsConnector.phpPass SQS outbox configuration +1/-0

Pass SQS outbox configuration
• Carries the connection's outbox setting into the constructed SQS queue.
src/queue/src/Connectors/SqsConnector.php

outbox.stubDefine outbox migration schema +41/-0

Define outbox migration schema
• Creates the durable payload, availability, attempt, lease, and indexing columns required by the relay.
src/queue/src/Console/stubs/outbox.stub

qodo-free-for-open-source-projects · 2026-10-23T03:18:30Z

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Later jobs in batched chains skip the outbox 🐞 Bug ≡ Correctness

Description

Batch::add() applies the batch’s viaOutbox and outboxDatabase defaults only to the head of an
array-based chain, after $first->chain() has serialized the other members into $first->chained.
With Bus::batch([[$a, $b]])->viaOutbox(), $a uses the outbox, but $b and later members are
dispatched without those defaults unless they independently select the outbox; an
onOutboxDatabase() default is lost the same way.

Code

src/bus/src/Batch.php[R86-94]

+            if (array_key_exists('viaOutbox', $this->options) || array_key_exists('outboxDatabase', $this->options)) {
+                // Preserve initialized nulls as well as explicit values on each job.
+                $properties = get_object_vars($job);
+
+                foreach (['viaOutbox', 'outboxDatabase'] as $property) {
+                    if (array_key_exists($property, $this->options) && ! array_key_exists($property, $properties)) {
+                        $job->{$property} = $this->options[$property];
+                    }
+                }

Evidence
prepareBatchedChain() sets the batch ID on every member, but $first->chain() serializes the
followers before Batch::add() applies outbox defaults to the returned head. On continuation,
Queueable::dispatchNextJobInChain() unserializes a follower and copies connection, queue, and
catch callbacks, but not viaOutbox or outboxDatabase; outbox selection therefore reads the
follower’s unchanged values.
src/bus/src/Batch.php[64-95]
src/bus/src/Batch.php[124-131]
src/bus/src/Queueable.php[295-312]
src/bus/src/Batch.php[56-97]
src/bus/src/Batch.php[119-126]
src/bus/src/Queueable.php[295-313]
src/queue/src/Concerns/StagesOutboxJobs.php[31-38]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Batch-level `viaOutbox()` and `onOutboxDatabase()` defaults reach only the head of a batched chain. Followers are serialized before those defaults are applied and can later be dispatched without the intended outbox selection.
## Fix Focus Areas
- src/bus/src/Batch.php[64-95]
- src/bus/src/Batch.php[124-131]
- src/bus/src/Queueable.php[295-313]
## Recommended Fix
Move default application into a helper, such as `applyOutboxDefaults(object $job): object`, that sets `viaOutbox` and `outboxDatabase` only when each property is uninitialized, preserving explicit per-job choices. Call it for every member inside `prepareBatchedChain()` before `$first->chain(...)` serializes followers, and retain its use for non-array entries. Add a test that runs a batched chain with `viaOutbox()` and verifies that the second member retains `viaOutbox === true` after unserialization and when dispatched.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Tip of the day

💡 Did you know, you can reply 'qodo' on any finding to push back, ask questions, or dig deeper

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

greptile-apps · 2026-10-23T03:22:23Z

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; both previous findings are addressed and no new actionable issues remain.

Summary

Adds an opt-in transactional queue outbox that stores jobs with database writes and relays them through existing queue drivers.

Selected jobs are saved with business data before an existing queue receives them.
A separate relay publishes staged jobs and leaves uncertain deliveries ready to retry.
Transaction callbacks and dispatch locks keep the connection and context that own them.
Horizon keeps its job tracking while relayed jobs enter Redis.
Guides and benchmark tools explain how to set up and measure the outbox.

Both previous findings are addressed. No new actionable issues were found.

_{Reviews (2) · Last reviewed commit: "docs(queue): align outbox guidance with ..." · Reviewed by Greptile}

Give ordinary relay tests an explicit memory budget so their behavior does not depend on how much of the framework PHPUnit has already loaded. Apply the same budget to Redis delivery tests that invoke the relay command. Keep the dedicated low-memory exit test and production worker defaults unchanged. This fixes premature memory-limit exits in the framework CI jobs without weakening relay lifecycle assertions.

Reject declared outbox owners that resolve to a different session from the database batch repository before changing counters or publishing jobs. This prevents an independently rolled-back business transaction from removing jobs while leaving committed batch counters behind. Resolve ownership lazily, compare resolved session names rather than requested aliases, and preserve explicit outbox opt-outs and custom batch repositories. Keep validation at Batch::add rather than introducing queue-side batch context. Cover conflicting owners across staged, database and local delivery, same-owner rollback, shared SQLite aliases, and explicit opt-outs without resolving their unused owners.

Use the local after-commit fallback for outbox selection only when an applicable transaction exists. Without a transaction, synchronous dispatch retains its immediate result and delayed coroutine dispatch retains its timer identifier. Preserve ordinary explicit and configured after-commit behavior, including explicit-owner validation when transactions are active. Regression coverage checks both immediate results and the shared coroutine timer path without resolving outbox storage.

Clarify that batch outbox defaults apply to each chain head, while successors retain their own selection and connection defaults. Align the active implementation plan with batch-owner preflight and local dispatch return-value preservation. Replace the completed PR-opening instruction with the remaining local-commit and owner-direction step.

Explain outbox setup and delivery through application-facing language and examples rather than implementation terminology. Give delayed jobs, configuration changes, monitoring, retries and deletion their own sections while preserving the delivery and custom-driver contracts. Clarify required database registration, batch storage, local-driver transaction behavior, retry eligibility and telemetry interpretation. Match the surrounding prose in the queue, event, mail, notification, broadcast and porting guides, and keep their detailed instructions linked to the canonical outbox guide. Validate documentation anchors, cross-links and code fences. Update the active plan's remaining step to PR review and CI verification. No runtime behavior changes.

binaryfire added 28 commits October 22, 2026 17:10

Clarify targeted verification after failures

0e726ce

Choose rerun scope from the correction's impact rather than repeating the original broad command. Finish outstanding verification and reserve another full test-suite run for fixes with framework-wide impact.

perf(horizon): skip monitored-tag reads for untagged jobs

92b2c86

Return immediately when the requested tag list is empty instead of loading the monitored tag set from Redis. Keep normal tag intersection behavior unchanged and cover both empty and populated requests.

qodo-free-for-open-source-projects Bot reviewed Oct 23, 2026

View reviewed changes

Comment thread src/bus/src/Batch.php

greptile-apps Bot reviewed Oct 23, 2026

View reviewed changes

Comment thread src/queue/src/Outbox/OutboxDispatcher.php

Comment thread src/queue/src/SyncQueue.php Outdated

binaryfire added 5 commits October 23, 2026 04:07

binaryfire merged commit 76da349 into 0.4 Oct 23, 2026
51 checks passed

Navigation Menu

Repository navigation

Add an opt-in transactional queue outbox - #70

Add an opt-in transactional queue outbox#70
binaryfire merged 33 commits into
0.4from
feature/queue-outbox

binaryfire commented Oct 23, 2026 •

edited by macroscopeapp Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 23, 2026 •

edited

Loading

Review skipped

Uh oh!

qodo-free-for-open-source-projects Bot commented Oct 23, 2026

Uh oh!

qodo-free-for-open-source-projects Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

binaryfire commented Oct 23, 2026 • edited by macroscopeapp Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Usage

Delivery and recovery

Driver compatibility and operations

Related correctness changes

Performance

Ordinary dispatch, without the outbox

Transactional staging

Relay

Claim design

Verification

Summary by cubic

Add opt-in transactional queue outbox with relay worker and console commands

Uh oh!

coderabbitai Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

qodo-free-for-open-source-projects Bot commented Oct 23, 2026

PR Summary by Qodo

Uh oh!

qodo-free-for-open-source-projects Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

binaryfire commented Oct 23, 2026 •

edited by macroscopeapp Bot

Loading

coderabbitai Bot commented Oct 23, 2026 •

edited

Loading

qodo-free-for-open-source-projects Bot commented Oct 23, 2026 •

edited

Loading

greptile-apps Bot commented Oct 23, 2026 •

edited

Loading