Benchmarking¶

Vortex has two categories of benchmarks: microbenchmarks for individual operations, and SQL benchmarks for end-to-end query performance.

Microbenchmarks¶

Microbenchmarks use the Divan framework and live in benches/ directories within individual crates.

Run microbenchmarks for a specific crate with:

cargo bench -p <crate-name>

Best Practices¶

Separate setup from profiled code¶

Always use bencher.with_inputs(|| ...) so fixture construction is excluded from timing:

bencher
    .with_inputs(|| bench_fixture()))
    .bench_refs(|(array, indices)| {
        array.take(indices.to_array()).unwrap()
    });

Exclude `Drop` from measurements¶

Divan measures only the closure body, not the Drop of its return value. Structure your benchmark so that expensive drops happen via the return value or via bench_refs inputs.

Return the value from the closure — Divan will drop it after timing stops:

bencher
    .with_inputs(|| make_big_vec())
    .bench_values(|v| transform(v))  // drop of the result is NOT timed

Use bench_refs — the input is dropped after the entire sample loop, not per-iteration:

bencher
    .with_inputs(|| make_big_vec())
    .bench_refs(|v| v.sort())  // v is dropped outside the timed region

Structure your benchmark so that expensive drops happen via the return value or via bench_refs inputs.

Black-box inputs to prevent compiler optimization¶

The compiler can constant-fold or eliminate work if it can prove that inputs are known at compile time.

Values provided through with_inputs are automatically black-boxed by Divan — no action needed:

// ✓ `array` and `indices` are automatically black-boxed by Divan
bencher
    .with_inputs(|| (&prebuilt_array, &prebuilt_indices))
    .bench_refs(|(array, indices)| array.take(indices.to_array()).unwrap());

Captured variables¶

Variables captured from the surrounding scope are not black-boxed. Wrap them with divan::black_box() or pass them through with_inputs instead:

let array = make_array();

// ✗ `array` is captured — the compiler may optimize based on its known contents
bencher.bench(|| process(&array));

// ✓ Option A: pass through with_inputs
bencher
    .with_inputs(|| &array)
    .bench_refs(|array| process(array));

// ✓ Option B: explicit black_box on the capture
bencher.bench(|| process(divan::black_box(&array)));

Return values and manual loops¶

Return values are automatically black-boxed. You only need explicit black_box for side-effect-free results inside manual loops:

bencher.with_inputs(|| &array).bench_refs(|array| {
    for idx in 0..len {
        divan::black_box(array.scalar_at(idx).unwrap());
    }
});

Use deterministic, seeded RNG¶

Always use StdRng::seed_from_u64(N) for reproducible data generation:

let mut rng = StdRng::seed_from_u64(0);

Parameterize with `args`, `consts`, and `types`¶

Use Divan’s parameterization features and define parameter arrays as named constants:

const NUM_INDICES: &[usize] = &[1_000, 10_000, 100_000];
const VECTOR_SIZE: &[usize] = &[16, 256, 2048, 8192];

#[divan::bench(args = NUM_INDICES, consts = VECTOR_SIZE)]
fn my_bench<const N: usize>(bencher: Bencher, num_indices: usize) { ... }

Keep per-iteration execution time under 1 ms¶

1 ms is the maximum, not a soft target. Each individual iteration of the benchmarked closure must complete in less than 1 ms. This is to keep benchmarks snappy, locally and on CI.

A benchmark that needs longer than that is measuring too much work at once. Shrink the input size until a single iteration fits, split it into smaller parameterized cases, or gate it with #[cfg(not(codspeed))] if it genuinely cannot be made to fit.

The number to check against the budget is the per-iteration time, not the time the whole benchmark binary takes. CodSpeed reports exactly that: its performance report on a pull request lists the per-iteration time under HEAD for every benchmark the pull request adds or changes, so check any new benchmark there before merging.

Keep per-iteration work above the harness floor¶

The budget has a floor as well as a ceiling. CodSpeed runs each benchmark once and adds a fixed cost of roughly half a microsecond of reported time around the closure. A closure that does tens of nanoseconds of real work, such as one small allocation or a fast path that finds nothing to do, reports mostly that floor, and the floor moves by more than 10% between runs of identical code. Such benchmarks flag regressions on pull requests that do not touch Rust at all.

Aim for at least a few microseconds of real work per iteration:

When the operation itself is tiny, repeat it a fixed number of times inside the closure and black-box each result, as vortex-buffer/benches/allocation.rs does.
Drop degenerate inputs, such as a zero-byte allocation or a compaction with nothing to move.
Size inputs by bytes rather than element count so narrow and wide types land in the same range, as vortex-array/benches/filter_fixed_width.rs does.

Benchmarks tagged #[cpu_features] run on the walltime legs instead, where the floor is timer resolution and per-iteration jitter. Give those at least tens of microseconds per iteration, and keep the working set inside the L2 cache of the leg machines (1 MiB on the Graviton leg) when the benchmark is about kernel code rather than memory bandwidth.

Gate CodSpeed-incompatible benchmarks¶

Use #[cfg(not(codspeed))] for benchmarks that are incompatible with CodSpeed.

Keep third-party and frozen baselines out of CodSpeed¶

A benchmark of code that Vortex does not own, such as an arrow-rs kernel over the same data, or of a frozen copy of an old Vortex implementation, cannot regress because of a pull request, so a change in its number is never actionable. On the walltime legs these baselines were among the noisiest series in the suite. Keep them for local cargo bench comparisons, but gate them with #[cfg(not(codspeed))] and leave them untagged, as vortex-compute/benches/lane_kernels.rs and vortex-buffer/benches/collect_bool.rs do.

CodSpeed’s single-run model¶

CI benchmarks run under CodSpeed’s CPU simulation, which executes each benchmark exactly once and estimates CPU cycles from the instruction trace — including cache and memory access costs. This has several implications:

sample_count and sample_size have no effect — CodSpeed always runs one iteration.
Results are deterministic — the simulated cycle count is derived from the instruction trace, not wall-clock time, so there is no noise from system load or scheduling.
System calls are excluded — CodSpeed only measures user-space code. Benchmarks that rely on I/O or kernel interactions will not reflect those costs, so they should use the walltime instrument or be gated with #[cfg(not(codspeed))].

Use `mimalloc` as the global allocator¶

Every benchmark binary uses mimalloc as its global allocator:

use mimalloc::MiMalloc;
#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;

The system allocator’s cost depends on its state, which differs between runner images and between runs, so allocation inside a timed region made several simulation benchmarks flip between two values on pull requests that could not have affected them. mimalloc does the same work every time, and one allocator for every binary means no benchmark measures a different allocator from its neighbours. Add the two lines to every new benchmark file; each crate with benchmarks already has the mimalloc dev-dependency.

SQL Benchmarks¶

SQL benchmarks measure end-to-end query performance across different engines and file formats. The vortex-bench crate provides a common Benchmark trait that each benchmark suite implements, defining its queries, data generation, and expected results.

Available suites include TPC-H, TPC-DS, ClickBench, FineWeb, and others. Each suite can be run against multiple engines (DataFusion, DuckDB) and formats (Parquet, Vortex, Vortex Compact, Lance, DuckDB native).

Data Generation¶

Before running SQL benchmarks, test data must be generated:

uv run --project bench-orchestrator vx-bench prepare-data <benchmark> --format parquet,vortex

The data generator creates base Parquet data and converts it to each requested format. Scale factors are configurable per suite (e.g. --opt scale-factor=10.0 for TPC-H SF=10).

Running SQL Benchmarks¶

SQL benchmarks can be run directly via their per-engine binaries:

cargo run --release --bin datafusion-bench -- <benchmark>
cargo run --release --bin duckdb-bench -- <benchmark>

Orchestrator¶

The bench-orchestrator is a Python CLI tool (vx-bench) that coordinates running SQL benchmarks across multiple engines, stores results, and provides comparison tooling.

See bench-orchestrator/README.md for installation, commands, and example workflows.

For CI, the reusable SQL workflow now drives vx-bench directly:

uv run --project bench-orchestrator vx-bench prepare-data tpch \
  --formats-json '["parquet","vortex","vortex-compact"]' \
  --opt scale-factor=1.0

uv run --project bench-orchestrator vx-bench run tpch \
  --targets-json '[{"engine":"datafusion","format":"parquet"},{"engine":"duckdb","format":"vortex"}]' \
  --output results.json \
  --no-build

CI Benchmarks¶

Benchmarks run automatically on all commits to develop and can be run on-demand for PRs:

Post-commit – compression, string encoding, random access, and SQL benchmarks run on every commit to develop, with results uploaded for historical tracking.
Random access – action/bench-random-access runs only the random-access benchmark.
Compression – action/bench-compress runs only the compression benchmark.
String encoding – action/bench-string runs only the string encoding benchmark.
GPU compression – action/bench-gpu-compress runs the allow-listed Vortex decompression cases on a GPU runner.
SQL – action/bench-sql runs the pr preset, which excludes vortex-compact.
SQL Compact – action/bench-sql-compact runs the pr-compact preset, which benchmarks vortex-compact plus Parquet control rows used to distinguish code changes from runner drift.
All CPU benchmarks – action/bench-all runs random access, compression, string encoding, and the pr-all SQL preset, which combines the pr and pr-compact coverage without repeating shared jobs. Do not combine it with other benchmark labels; GPU compression is the only exception.

All CI benchmarks run on dedicated instances with the release_debug profile and -C target-cpu=native to produce representative numbers.

Results can be viewed at bench.vortex.dev.