Repository navigation
Conversation
AdamGS
force-pushed
the
adamg/remove-map-fn-limit
branch
from
July 17, 2026 09:45
112e14e to
a684568
Compare
AdamGS
force-pushed
the
adamg/remove-map-fn-limit
branch
from
July 17, 2026 12:39
882dc6e to
eaff866
Compare
AdamGS
force-pushed
the
adamg/remove-map-fn-limit
branch
from
July 17, 2026 14:50
b0b6794 to
da0d99c
Compare
AdamGS
force-pushed
the
adamg/remove-map-fn-limit
branch
from
July 17, 2026 15:47
3b330f4 to
bf8ba8a
Compare
AdamGS
force-pushed
the
adamg/remove-map-fn-limit
branch
2 times, most recently
from
July 28, 2026 10:35
7015650 to
26a82c5
Compare
AdamGS
added
action/bench-sql
AdamGS
force-pushed
the
adamg/remove-map-fn-limit
branch
2 times, most recently
from
July 29, 2026 16:04
f1ba9b0 to
01e195e
Compare
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.032x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (1.098x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.041x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.104x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (0.997x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.046x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Vortex queries 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.766x ✅, 2↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.005x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.017x ➖, 1↑ 1↓)
datafusion / vortex-compact / ns (0.993x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.107x ➖, 1↑ 3↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.030x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.065x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.010x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 3↑ 4↓)
duckdb / vortex-compact / ns (0.928x ➖, 2↑ 1↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.997x ➖, 1↑ 1↓)
duckdb / vortex-compact / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.992x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.985x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.008x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.012x ➖, 1↑ 2↓)
duckdb / vortex-compact / ns (0.935x ➖, 3↑ 0↓)
duckdb / parquet / ns (0.984x ➖, 0↑ 0↓)
File Size Changes (200 files changed, +0.0% overall, 103↑ 97↓)
Totals:
|
Contributor
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.089x ➖, 2↑ 3↓)
datafusion / vortex-compact / ns (1.017x ➖, 1↑ 1↓)
datafusion / parquet / ns (0.957x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (0.985x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.010x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.990x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.992x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.003x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 1↑ 3↓)
duckdb / vortex-compact / ns (0.997x ➖, 2↑ 1↓)
duckdb / parquet / ns (1.010x ➖, 2↑ 3↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (1.000x ➖, 1↑ 2↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 1↑ 4↓)
duckdb / vortex-compact / ns (1.004x ➖, 1↑ 6↓)
duckdb / parquet / ns (0.991x ➖, 4↑ 1↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.013x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.002x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.986x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.990x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.016x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: String Encoding 📖Commits: PR vortex / vortex-file-compressed / ms (1.003x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / % (1.000x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.099x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (0.994x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.048x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.027x ➖, 1↑ 0↓)
duckdb / vortex-compact / ns (0.971x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.052x ➖, 0↑ 1↓)
|
Contributor
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.983x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / ratio (0.995x ➖, 0↑ 0↓)
vortex / parquet / ns (0.991x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (1.029x ➖, 0↑ 5↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.001x ➖, 0↑ 2↓)
datafusion / parquet / ns (0.968x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.037x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.026x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
vortex / arrow-ipc / ns (1.024x ➖, 0↑ 1↓)
random-access / vortex-file-compressed / ns (1.022x ➖, 0↑ 1↓)
random-access / parquet / ns (0.981x ➖, 1↑ 0↓)
random-access / lance / ns (0.989x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Random Access (S3) 📖Commits: PR How to read Verdict and Engines
random-access / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
random-access / parquet / ns (0.995x ➖, 0↑ 0↓)
random-access / lance / ns (0.980x ➖, 0↑ 0↓)
|
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
- Split tasks return plain `VortexResult<Option<ArrayRef>>` futures. The scan stream ends after its first error, replacing the recoverable/terminal `TaskResult` split and the `TaskFuture` wrapper. - Filtered limited scans run in two phases: split filters run ahead, each filtered mask reserves rows against the limit, and only granted rows are projected. Ordered scans reserve in split order, so they now return the earliest matching rows without projecting rows past the limit, removing the output-trimming path. - `ScanBuilder` and `RepeatedScan` hold a single `ScanLimit` (per-execution or shared with sibling partitions) instead of separate `limit` and `row_limit` fields. - Document why the DataFusion opener may apply a limit after its filter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QR9qZkPoFCuJ7rHyryscxT Signed-off-by: Claude <noreply@anthropic.com>
Filtered scans with a limit go back to filtering and projecting splits ahead of the consumer, with the projection's I/O prefetched alongside the filter. Emitted arrays take their rows from the limit and the excess is discarded, for ordered and unordered scans alike. Split tasks are still built lazily so no new splits start once the budget is spent. Filterless limited scans keep reserving rows before projection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QR9qZkPoFCuJ7rHyryscxT Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QR9qZkPoFCuJ7rHyryscxT Signed-off-by: Claude <noreply@anthropic.com>
Scan consumers convert each split to Arrow (or canonical form) inside the spawned split tasks again, restoring the parallel decode they had before `map` was removed. `ScanBuilder`, `RepeatedScan` and the scan consumers go back to their generic `develop` signatures, as do `ScanBuilder::build` and the public `RepeatedScan::execute_stream`. Filtered scans with a limit must trim their output before mapping, so they apply the map function in a second stage of CPU tasks after the rows are taken from the limit. All other scans apply it within the split task. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QR9qZkPoFCuJ7rHyryscxT Signed-off-by: Claude <noreply@anthropic.com>
Scans now support a filter together with a limit, so: - `test_arrow_indices_use_scan_filter_limit_validation`, which expected the combination to be rejected, becomes a test of the filtered, limited result. - The Hugging Face dataset reader passes its limit to the scan even when a filter is set, instead of scanning every matching row and stopping on the Python side. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QR9qZkPoFCuJ7rHyryscxT Signed-off-by: Claude <noreply@anthropic.com>
DataFusion removes its own limit operator once `DataSource::with_fetch` accepts the limit, so the source has to return exactly `fetch` rows. Unordered scans already share one budget across partitions, but ordered partitions each apply the limit locally, so the v2 source now trims the flattened batch stream to the limit. Also drop the stale claim in the `ScanRequest::limit` docs that filtered partitions trim their selection mask before projection. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QR9qZkPoFCuJ7rHyryscxT Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Reviving #7565. Scans previously rejected combining a filter with a limit. This change lets callers push the limit into Vortex and stop after enough matching rows have been produced.
What changes are included in this PR?
ScanBuilder::mapand the generic result types, applying the mapper to trimmed arrays.What APIs are changed? Are there any user-facing changes?
ScanBuilder::with_limitnow supports scans with filters.ScanBuilder::mapandScanBuilder::buildremain available.buildandRepeatedScan::executereject limits because independently scheduled split futures cannot coordinate them.Validation
Regression coverage includes filtered limits, ordering, shared budgets, sparse selections, error termination, and task construction. Tests and formatting were not run during this review revision, following repository guidance.