Concepts

Vortex is a modular ecosystem for working with compressed columnar data: in-memory, on-disk, over-the-wire, and integrated with query engines.

• DTypes
  • Logical Types
  • Vs. Arrow
• Arrays
  • Encodings
  • Statistics
  • Execution
  • Buffer Handles
• Expressions
  • Scalar Functions
  • Scalar Function Arrays
  • Type Checking
• Layouts
  • Built-in Layouts
  • Layout Strategies
• File Format
  • Layout Strategy
  • Compression Strategies
• Scan API
  • Motivation
  • Source
  • Filter Pushdown
  • Projection Pushdown
  • Query Engine Integration

Core Concepts

DTypes are Vortex’s logical type system. Types like UTF8 describe what data means without dictating physical layout, allowing the same logical data to use different encodings.

Arrays are the in-memory representation. Unlike Arrow, Vortex arrays can be compressed, so an integer array can be bit-packed rather than stored as a flat buffer. Serialized formats can reuse array buffers for zero-copy I/O, while plugins adapt historical formats to current array implementations.

Compute functions operate directly on compressed arrays where possible, dispatching to encoding-specific kernels or falling back to canonical implementations.

Storage & I/O

Layouts organize arrays into larger-than-memory datasets (e.g., chunked row groups) and can read from any block storage: local disk, object stores, caches, etc.

File Format (.vortex files) serialize layouts to disk with efficient segment retrieval, FlatBuffer metadata for O(1) schema access, and support for memory mapping.

IPC Format provides streaming transfer of compressed arrays.

Versioning explains how library implementations, serialized formats, and editions evolve while preserving read compatibility.

Integrations

Language bindings: Rust, Python, Java, C

Query engines: DataFusion, DuckDB, Spark, Polars, Ray