perf: speed up DictionaryArray hashing by Rich-T-kid · Pull Request #26159 · apache/datafusion

Rich-T-kid · 2026-10-09T14:15:39Z

Which issue does this PR close?

Closes perf: speed up hash_dictionary_scatter hot loop #26168.

Rationale for this change

see issue, the cost for hashing dictionary arrays comes down to a writing corresponding values to their key indices, we can speed this up with branchless code.

What changes are included in this PR?

Rewrites the scatter as four const-generic specializations:

8-way unrolled gather for the no-nulls case
single-check loop for nullable values
64-key bitmask-driven path for nullable keys (branchless gather+csel for
valid values, trailing_zeros bit-walk otherwise)
scalar tail for non-multiple-of-64 lengths

What is the testing strategy for this PR?

existing test cover this path

Are there any user-facing changes?

no

Rich-T-kid · 2026-10-09T14:17:32Z

run benchmark with_hashes

adriangbot · 2026-10-09T14:20:19Z

Benchmark for this request failed before finishing (Kubernetes reason: BackoffLimitExceeded).

Benchmarks requested: with_hashes

Runner log (last 40 lines)

2026-10-09T14:19:45.128146Z  INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/26159, benchmarks=with_hashes
2026-10-09T14:19:45.187849Z  INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
2026-10-09T14:19:45.187869Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
2026-10-09T14:19:50.197440Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/26159/head:rich-T-kid/faster-dict-hash", "main"], cwd="/workspace/datafusion-branch"
2026-10-09T14:20:05.201836Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "rich-T-kid/faster-dict-hash"], cwd="/workspace/datafusion-branch"
2026-10-09T14:20:10.203962Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
2026-10-09T14:20:15.205905Z ERROR runner benchmark failed error=git merge-base: git merge-base HEAD origin/main exited with code 1
stdout:

stderr:

Kubernetes message

Job has reached the specified backoff limit

File an issue against this benchmark runner

codecov-commenter · 2026-10-09T16:03:57Z

Codecov Report

❌ Patch coverage is 55.84416% with 68 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.76%. Comparing base (5134a1a) to head (2b50497).
⚠️ Report is 674 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/common/src/hash_utils.rs	55.84%	67 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #26159      +/-   ##
==========================================
+ Coverage   81.38%   82.76%   +1.38%     
==========================================
  Files        1116     1148      +32     
  Lines      397960   451186   +53226     
  Branches   397960   451186   +53226     
==========================================
+ Hits       323880   373435   +49555     
+ Misses      55120    55008     -112     
- Partials    18960    22743    +3783

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Rich-T-kid · 2026-10-09T16:49:09Z

show benchmark queue

adriangbot · 2026-10-09T16:49:11Z

Hi @Rich-T-kid, you asked to view the benchmark queue (#26159 (comment)).

No pending jobs.

File an issue against this benchmark runner

Rich-T-kid · 2026-10-09T16:49:21Z

run benchmark with_hashes

adriangbot · 2026-10-09T16:53:27Z

Benchmark for this request failed before finishing (Kubernetes reason: BackoffLimitExceeded).

Benchmarks requested: with_hashes

Runner log (last 40 lines)

2026-10-09T16:52:52.209214Z  INFO runner starting benchmark runner bench_type=Datafusion, pr_url=https://github.com/apache/datafusion/pull/26159, benchmarks=with_hashes
2026-10-09T16:52:52.292335Z  INFO benchmark_controller::runner::bench_datafusion === Cloning PR branch ===
2026-10-09T16:52:52.292357Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["clone", "--depth=200", "https://github.com/apache/datafusion.git", "/workspace/datafusion-branch"], cwd="/"
2026-10-09T16:52:57.323576Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["fetch", "origin", "refs/pull/26159/head:rich-T-kid/faster-dict-hash", "main"], cwd="/workspace/datafusion-branch"
2026-10-09T16:53:12.327073Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["checkout", "rich-T-kid/faster-dict-hash"], cwd="/workspace/datafusion-branch"
2026-10-09T16:53:17.329088Z  INFO benchmark_controller::runner::shell running command cmd=git, args=["merge-base", "HEAD", "origin/main"], cwd="/workspace/datafusion-branch"
2026-10-09T16:53:22.330831Z ERROR runner benchmark failed error=git merge-base: git merge-base HEAD origin/main exited with code 1
stdout:

stderr:

Kubernetes message

Job has reached the specified backoff limit

File an issue against this benchmark runner

Rich-T-kid · 2026-10-09T19:04:24Z

run benchmark with_hashes

Rich-T-kid · 2026-10-09T19:04:34Z

run benchmark with_hashes
env:
BENCH_FILTER: dict

adriangbot · 2026-10-09T19:07:38Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6087428589-3254-vjlgx 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/faster-dict-hash (05bcf90) to 1c49b7f (merge-base) diff

Run configuration

run benchmark with_hashes

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-10-09T19:10:12Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6087431329-3255-7ltrx 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/faster-dict-hash (05bcf90) to 1c49b7f (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dict"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-10-09T19:13:33Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/faster-dict-hash (05bcf90) to 1c49b7f (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dict"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        HEAD                                   rich-T-kid_faster-dict-hash
-----                                        ----                                   ---------------------------
dictionary_utf8_int32: multiple, no nulls    1.42     14.7±0.01µs        ? ?/sec    1.00     10.3±0.01µs        ? ?/sec
dictionary_utf8_int32: multiple, nulls       2.10     28.9±0.08µs        ? ?/sec    1.00     13.8±0.01µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.46      4.6±0.01µs        ? ?/sec    1.00      3.2±0.00µs        ? ?/sec
dictionary_utf8_int32: single, nulls         2.37      9.4±0.01µs        ? ?/sec    1.00      4.0±0.00µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric	Value
Wall time	90.0s
Peak memory	33.1 MiB
Avg memory	11.2 MiB
CPU user	47.8s
CPU sys	0.1s
Peak spill	0 B

with_hashes — branch

Metric	Value
Wall time	100.0s
Peak memory	35.6 MiB
Avg memory	10.4 MiB
CPU user	44.7s
CPU sys	0.1s
Peak spill	0 B

File an issue against this benchmark runner

Rich-T-kid · 2026-10-09T19:27:32Z

run benchmark with_hashes
env:
BENCH_FILTER: dict

Rich-T-kid · 2026-10-09T19:30:24Z

cc @mightsleep @alamb if your in the mood to review more bit manipulation code 😄. I think this is less intense than whats going on in arrow-rs 😆

adriangbot · 2026-10-09T19:31:04Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6087790133-3256-9fbgp 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/faster-dict-hash (05bcf90) to 1c49b7f (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dict"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-10-09T19:32:56Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/faster-dict-hash (05bcf90) to 1c49b7f (merge-base) diff

Run configuration

run benchmark with_hashes

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                         HEAD                                   rich-T-kid_faster-dict-hash
-----                                         ----                                   ---------------------------
dense_union: multiple, no nulls               1.00     44.0±0.09µs        ? ?/sec    1.00     44.2±0.06µs        ? ?/sec
dense_union: single, no nulls                 1.00     14.9±0.01µs        ? ?/sec    1.00     15.0±0.02µs        ? ?/sec
dense_union_sliced: 1/10 of 81920 rows        1.00     22.4±0.02µs        ? ?/sec    1.01     22.6±0.02µs        ? ?/sec
dense_union_sliced: 1/2 of 16384 rows         1.01     22.6±0.03µs        ? ?/sec    1.00     22.5±0.02µs        ? ?/sec
dense_union_sliced: 1/5 of 40960 rows         1.00     22.4±0.02µs        ? ?/sec    1.00     22.5±0.02µs        ? ?/sec
dictionary_utf8_int32: multiple, no nulls     1.43     14.8±0.02µs        ? ?/sec    1.00     10.4±0.01µs        ? ?/sec
dictionary_utf8_int32: multiple, nulls        2.13     29.4±0.14µs        ? ?/sec    1.00     13.8±0.02µs        ? ?/sec
dictionary_utf8_int32: single, no nulls       1.45      4.6±0.01µs        ? ?/sec    1.00      3.2±0.00µs        ? ?/sec
dictionary_utf8_int32: single, nulls          2.39      9.4±0.01µs        ? ?/sec    1.00      4.0±0.01µs        ? ?/sec
fixed_size_list_array: multiple, no nulls     1.03    111.2±0.55µs        ? ?/sec    1.00    108.3±0.79µs        ? ?/sec
fixed_size_list_array: multiple, nulls        1.00    119.1±0.12µs        ? ?/sec    1.00    119.2±0.07µs        ? ?/sec
fixed_size_list_array: single, no nulls       1.00     36.2±0.04µs        ? ?/sec    1.00     36.2±0.02µs        ? ?/sec
fixed_size_list_array: single, nulls          1.00     40.0±0.12µs        ? ?/sec    1.01     40.5±0.32µs        ? ?/sec
int64: multiple, no nulls                     1.00     16.9±0.02µs        ? ?/sec    1.00     16.9±0.01µs        ? ?/sec
int64: multiple, nulls                        1.00     26.7±0.01µs        ? ?/sec    1.00     26.7±0.05µs        ? ?/sec
int64: single, no nulls                       1.00      4.5±0.00µs        ? ?/sec    1.01      4.6±0.01µs        ? ?/sec
int64: single, nulls                          1.00      8.7±0.00µs        ? ?/sec    1.00      8.7±0.01µs        ? ?/sec
large_utf8: multiple, no nulls                1.00    120.1±1.34µs        ? ?/sec    1.00    120.5±0.53µs        ? ?/sec
large_utf8: multiple, nulls                   1.00    144.5±0.64µs        ? ?/sec    1.00    144.6±0.73µs        ? ?/sec
large_utf8: single, no nulls                  1.01     38.9±0.19µs        ? ?/sec    1.00     38.7±0.16µs        ? ?/sec
large_utf8: single, nulls                     1.00     34.6±0.35µs        ? ?/sec    1.00     34.6±0.35µs        ? ?/sec
list_array: multiple, no nulls                1.00    139.9±0.13µs        ? ?/sec    1.01    142.0±0.30µs        ? ?/sec
list_array: multiple, nulls                   1.00    168.1±0.14µs        ? ?/sec    1.00    168.5±0.26µs        ? ?/sec
list_array: single, no nulls                  1.00     46.9±0.03µs        ? ?/sec    1.01     47.6±0.12µs        ? ?/sec
list_array: single, nulls                     1.00     56.5±0.53µs        ? ?/sec    1.00     56.5±0.04µs        ? ?/sec
list_array_sliced: 1/10 of 81920 rows         1.00     46.9±0.03µs        ? ?/sec    1.01     47.3±0.04µs        ? ?/sec
list_array_sliced: 1/2 of 16384 rows          1.04     49.7±0.25µs        ? ?/sec    1.00     47.7±0.04µs        ? ?/sec
list_array_sliced: 1/5 of 40960 rows          1.00     46.9±0.06µs        ? ?/sec    1.03     48.5±1.17µs        ? ?/sec
list_view_array: multiple, no nulls           1.00    135.0±0.19µs        ? ?/sec    1.01    136.3±0.15µs        ? ?/sec
list_view_array: multiple, nulls              1.01    145.4±0.41µs        ? ?/sec    1.00    143.7±0.15µs        ? ?/sec
list_view_array: single, no nulls             1.00     45.3±0.05µs        ? ?/sec    1.01     45.7±0.04µs        ? ?/sec
list_view_array: single, nulls                1.00     48.0±0.10µs        ? ?/sec    1.00     48.1±0.05µs        ? ?/sec
map_array: multiple, no nulls                 1.00    240.3±0.18µs        ? ?/sec    1.01    242.1±0.14µs        ? ?/sec
map_array: multiple, nulls                    1.00    248.1±0.25µs        ? ?/sec    1.00    249.3±0.31µs        ? ?/sec
map_array: single, no nulls                   1.00     80.4±0.06µs        ? ?/sec    1.01     80.8±0.12µs        ? ?/sec
map_array: single, nulls                      1.00     83.8±0.21µs        ? ?/sec    1.00     83.9±0.40µs        ? ?/sec
map_array_sliced: 1/10 of 81920 rows          1.00     80.3±0.06µs        ? ?/sec    1.01     81.3±0.23µs        ? ?/sec
map_array_sliced: 1/2 of 16384 rows           1.00     80.6±0.17µs        ? ?/sec    1.00     80.5±0.04µs        ? ?/sec
map_array_sliced: 1/5 of 40960 rows           1.00     80.4±0.04µs        ? ?/sec    1.00     80.6±0.14µs        ? ?/sec
mixed: 3 columns (int64, utf8, utf8_view)     1.00     80.8±1.73µs        ? ?/sec    1.02     82.2±0.11µs        ? ?/sec
null: multiple, no nulls                      1.00      2.5±0.00µs        ? ?/sec    1.00      2.5±0.00µs        ? ?/sec
null: single, no nulls                        1.00   1111.3±3.48ns        ? ?/sec    1.00   1110.4±4.53ns        ? ?/sec
run_array_int32: multiple, no nulls           1.01      5.1±0.01µs        ? ?/sec    1.00      5.0±0.01µs        ? ?/sec
run_array_int32: multiple, nulls              1.00      6.7±0.01µs        ? ?/sec    1.01      6.8±0.01µs        ? ?/sec
run_array_int32: single, no nulls             1.03   1892.6±4.46ns        ? ?/sec    1.00   1829.5±4.48ns        ? ?/sec
run_array_int32: single, nulls                1.00      2.4±0.01µs        ? ?/sec    1.04      2.5±0.00µs        ? ?/sec
sparse_union (2 types): multiple, no nulls    1.00    102.5±0.38µs        ? ?/sec    1.00    102.5±0.47µs        ? ?/sec
sparse_union (2 types): single, no nulls      1.15     39.4±0.35µs        ? ?/sec    1.00     34.4±0.13µs        ? ?/sec
sparse_union (utf8): multiple, no nulls       1.00    234.2±1.57µs        ? ?/sec    1.03    240.6±1.06µs        ? ?/sec
sparse_union (utf8): single, no nulls         1.00     78.8±0.43µs        ? ?/sec    1.02     80.7±0.45µs        ? ?/sec
sparse_union: multiple, no nulls              1.03    102.0±5.68µs        ? ?/sec    1.00     98.6±0.49µs        ? ?/sec
sparse_union: single, no nulls                1.00     32.7±0.36µs        ? ?/sec    1.01     33.1±0.17µs        ? ?/sec
sparse_union_sliced: 1/10 of 81920 rows       1.00     33.5±0.18µs        ? ?/sec    1.02     34.2±1.64µs        ? ?/sec
sparse_union_sliced: 1/2 of 16384 rows        1.00     33.4±0.25µs        ? ?/sec    1.00     33.3±0.11µs        ? ?/sec
sparse_union_sliced: 1/5 of 40960 rows        1.01     33.3±0.38µs        ? ?/sec    1.00     33.1±0.26µs        ? ?/sec
struct_array: multiple, no nulls              1.00    161.6±0.72µs        ? ?/sec    1.01    163.4±0.42µs        ? ?/sec
struct_array: multiple, nulls                 1.01    180.3±2.57µs        ? ?/sec    1.00    178.8±0.62µs        ? ?/sec
struct_array: single, no nulls                1.01     54.8±0.16µs        ? ?/sec    1.00     54.2±0.17µs        ? ?/sec
struct_array: single, nulls                   1.00     59.8±0.33µs        ? ?/sec    1.00     60.0±0.31µs        ? ?/sec
utf8: multiple, no nulls                      1.00    115.0±0.45µs        ? ?/sec    1.00    114.7±0.38µs        ? ?/sec
utf8: multiple, nulls                         1.00    161.1±0.90µs        ? ?/sec    1.00    161.1±0.73µs        ? ?/sec
utf8: single, no nulls                        1.00     29.6±0.16µs        ? ?/sec    1.00     29.5±0.14µs        ? ?/sec
utf8: single, nulls                           1.00     44.7±0.55µs        ? ?/sec    1.00     44.6±0.65µs        ? ?/sec
utf8_view (small): multiple, no nulls         1.00     19.0±0.07µs        ? ?/sec    1.00     19.0±0.01µs        ? ?/sec
utf8_view (small): multiple, nulls            1.00     29.2±0.04µs        ? ?/sec    1.00     29.3±0.05µs        ? ?/sec
utf8_view (small): single, no nulls           1.00      5.0±0.01µs        ? ?/sec    1.00      5.0±0.00µs        ? ?/sec
utf8_view (small): single, nulls              1.00      9.6±0.03µs        ? ?/sec    1.01      9.7±0.01µs        ? ?/sec
utf8_view: multiple, no nulls                 1.05    101.9±0.17µs        ? ?/sec    1.00     97.2±0.20µs        ? ?/sec
utf8_view: multiple, nulls                    1.15    123.9±0.37µs        ? ?/sec    1.00    107.5±0.16µs        ? ?/sec
utf8_view: single, no nulls                   1.15     35.1±0.10µs        ? ?/sec    1.00     30.4±0.06µs        ? ?/sec
utf8_view: single, nulls                      1.12     36.6±0.06µs        ? ?/sec    1.00     32.7±0.06µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric	Value
Wall time	765.2s
Peak memory	40.1 MiB
Avg memory	33.5 MiB
CPU user	885.3s
CPU sys	0.8s
Peak spill	0 B

with_hashes — branch

Metric	Value
Wall time	740.2s
Peak memory	42.1 MiB
Avg memory	35.9 MiB
CPU user	881.3s
CPU sys	0.8s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-10-09T19:33:55Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/faster-dict-hash (05bcf90) to 1c49b7f (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dict"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        HEAD                                   rich-T-kid_faster-dict-hash
-----                                        ----                                   ---------------------------
dictionary_utf8_int32: multiple, no nulls    1.43     14.8±0.19µs        ? ?/sec    1.00     10.4±0.08µs        ? ?/sec
dictionary_utf8_int32: multiple, nulls       2.09     28.8±0.05µs        ? ?/sec    1.00     13.8±0.02µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.46      4.6±0.01µs        ? ?/sec    1.00      3.2±0.00µs        ? ?/sec
dictionary_utf8_int32: single, nulls         2.38      9.5±0.02µs        ? ?/sec    1.00      4.0±0.01µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric	Value
Wall time	85.0s
Peak memory	35.8 MiB
Avg memory	12.8 MiB
CPU user	45.9s
CPU sys	0.1s
Peak spill	0 B

with_hashes — branch

Metric	Value
Wall time	75.0s
Peak memory	32.5 MiB
Avg memory	13.1 MiB
CPU user	44.7s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

alamb

Thanks @Rich-T-kid -- this is still pretty intense LOL. I left some thoughts let me know whatyou think

Rework hash_dictionary_scatter so the key-null handling runs in 64-bit BitChunks instead of per-row Option unwrap, and route all-valid chunks plus the no-null-keys fast path through an 8-way unrolled dense scatter that keeps the dict gather pipeline full. Mixed chunks use a branchless csel blend (safe-clamped indices) when dict values are all valid; the rarer HAS_NULL_VALUES mixed path falls back to the per-bit scan for correctness. Benchmarks (datafusion/common/benches/with_hashes.rs, Apple M4 Max): dictionary_utf8_int32 single, no nulls : 2.91 µs -> 1.83 µs (-37%) dictionary_utf8_int32 multiple, no nulls: 9.10 µs -> 5.67 µs (-38%) dictionary_utf8_int32 single, nulls : 7.11 µs -> 2.81 µs (-60%) dictionary_utf8_int32 multiple, nulls : 20.90 µs -> 8.21 µs (-61%) Non-dict benchmarks (int64, utf8_view small) stay within noise.

Rich-T-kid · 2026-10-09T20:52:20Z

run benchmark with_hashes
env:
BENCH_FILTER: dict

Rich-T-kid · 2026-10-09T20:52:24Z

run benchmark with_hashes
env:
BENCH_FILTER: dict

mightsleep · 2026-10-09T20:56:37Z

yes hello,
the slim version keeps the no-nulls gain, but it also drops the null-key one. On x86 (Zen 5) I measured main + get_unchecked at 1.01x / 0.99x on "single, nulls" / "multiple, nulls". With your 64-bit mask walk kept only for mixed chunks (full chunks and the no-nulls case use the same plain get_unchecked loop), it was 3.0x / 2.4x, without the manual unroll alamb wanted gone. Might be worth bringing that part back, with the chunk test.

adriangbot · 2026-10-09T21:05:38Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6089059634-3260-cx85d 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/faster-dict-hash (88d50e0) to d137137 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dict"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-10-09T21:09:00Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/faster-dict-hash (88d50e0) to d137137 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dict"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        HEAD                                   rich-T-kid_faster-dict-hash
-----                                        ----                                   ---------------------------
dictionary_utf8_int32: multiple, no nulls    1.39     15.0±0.09µs        ? ?/sec    1.00     10.8±0.01µs        ? ?/sec
dictionary_utf8_int32: multiple, nulls       1.13     29.0±0.05µs        ? ?/sec    1.00     25.7±0.04µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.43      4.6±0.01µs        ? ?/sec    1.00      3.2±0.00µs        ? ?/sec
dictionary_utf8_int32: single, nulls         1.16      9.5±0.02µs        ? ?/sec    1.00      8.2±0.01µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric	Value
Wall time	85.0s
Peak memory	35.8 MiB
Avg memory	12.8 MiB
CPU user	45.7s
CPU sys	0.0s
Peak spill	0 B

with_hashes — branch

Metric	Value
Wall time	105.0s
Peak memory	32.8 MiB
Avg memory	9.6 MiB
CPU user	44.7s
CPU sys	0.1s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-10-09T21:10:14Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c6089060732-3261-swn8x 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing rich-T-kid/faster-dict-hash (88d50e0) to d137137 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dict"

Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-10-09T21:13:10Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing rich-T-kid/faster-dict-hash (88d50e0) to d137137 (merge-base) diff

Run configuration

run benchmark with_hashes
env:
  BENCH_FILTER: "dict"

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                        HEAD                                   rich-T-kid_faster-dict-hash
-----                                        ----                                   ---------------------------
dictionary_utf8_int32: multiple, no nulls    1.36     14.7±0.01µs        ? ?/sec    1.00     10.8±0.01µs        ? ?/sec
dictionary_utf8_int32: multiple, nulls       1.14     29.1±0.03µs        ? ?/sec    1.00     25.6±0.02µs        ? ?/sec
dictionary_utf8_int32: single, no nulls      1.43      4.6±0.02µs        ? ?/sec    1.00      3.2±0.00µs        ? ?/sec
dictionary_utf8_int32: single, nulls         1.16      9.5±0.02µs        ? ?/sec    1.00      8.2±0.01µs        ? ?/sec

Resource Usage

with_hashes — base (merge-base)

Metric	Value
Wall time	90.0s
Peak memory	32.5 MiB
Avg memory	11.3 MiB
CPU user	45.7s
CPU sys	0.0s
Peak spill	0 B

with_hashes — branch

Metric	Value
Wall time	75.0s
Peak memory	33.8 MiB
Avg memory	13.5 MiB
CPU user	48.2s
CPU sys	0.0s
Peak spill	0 B

File an issue against this benchmark runner

mightsleep · 2026-10-09T21:14:00Z

Bot numbers match mine, nulls went from 2.1-2.4x to 1.13-1.16x without the mask walk.

Rich-T-kid · 2026-10-10T01:11:06Z

hmm I think its okay to break up the changes then. I think a 15-43% is good for a single PR and then I can build on top of it afterwards. I think @alamb is also in favor of this style

Rich-T-kid · 2026-10-10T16:53:15Z

-                    } else {
-                        *hash = dict_hashes[idx];
-                    }
+                    let dict_hash = unsafe { *dict_hashes.get_unchecked(idx) };


should add a saftey comment

Rich-T-kid force-pushed the rich-T-kid/faster-dict-hash branch from d00c2a0 to 63799c2 Compare October 9, 2026 14:16

github-actions Bot added the common Related to common crate label Oct 9, 2026

Rich-T-kid force-pushed the rich-T-kid/faster-dict-hash branch from 05bcf90 to 2b50497 Compare October 9, 2026 19:30

Rich-T-kid marked this pull request as ready for review October 9, 2026 19:30

alamb reviewed Oct 9, 2026

View reviewed changes

Comment thread datafusion/common/src/hash_utils.rs

Comment thread datafusion/common/src/hash_utils.rs Outdated

Rich-T-kid added 5 commits October 9, 2026 16:46

test: cover dict hash chunks

d5b570e

test: move buffer imports

2e011c9

perf: remove dict hash bounds checks

732211b

test: remove dict chunk regression

88d50e0

Rich-T-kid force-pushed the rich-T-kid/faster-dict-hash branch from 6117012 to 88d50e0 Compare October 9, 2026 20:52

Rich-T-kid commented Oct 10, 2026

View reviewed changes

Conversation

Rich-T-kid commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

codecov-commenter commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

Rich-T-kid commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

Rich-T-kid commented Oct 9, 2026

Uh oh!

mightsleep commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

adriangbot commented Oct 9, 2026

Uh oh!

mightsleep commented Oct 9, 2026

Uh oh!

Rich-T-kid commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Rich-T-kid Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

Rich-T-kid commented Oct 9, 2026 •

edited

Loading

codecov-commenter commented Oct 9, 2026 •

edited

Loading

Rich-T-kid commented Oct 9, 2026 •

edited

Loading

Rich-T-kid commented Oct 10, 2026 •

edited

Loading