File Format

Important

The Vortex file format’s stability guarantee starts with version 0.36.0 of the Vortex Rust crates and edition core2025.05.0. Later versions of those crates retain read support for the components in frozen editions. Versioning explains the guarantee and the requirements for draft and custom components.

See also

The majority of the complexity of the Vortex file format is encapsulated in Vortex Layouts. Unless you are interested in the specific byte layout of the file, you are probably looking for that documentation!

Recall that Vortex Layouts provide a mechanism to efficiently query large serialized Vortex arrays. The Vortex File Format is designed to provide a container for these serialized arrays, as well as footer definition that allows efficiently querying the layout.

Other considerations for the Vortex file format include:

• File compatibility. Editions constrain which serialized components a writer can use. A writer built with newer Vortex crates can target an older edition that its intended readers support.

• Fine-grained encryption.

• Efficient access for both local disk and cloud storage.

• Minimal overhead reading few columns or rows from wide or long arrays.

File Specification

The Vortex file format has a very small definition, with much of the complexity encapsulated in Vortex Layouts.

<4 bytes>  magic number 'VTXF'
...        segments of binary data, optionally with inter-segment padding
...        postscript data
<2 bytes>  u16 version tag
<2 bytes>  u16 postscript length
<4 bytes>  magic number 'VTXF'

The file format begins and ends with the 4-byte magic number VTXF. Immediately prior to the trailing magic number are two 16-bit integers: the version tag and the length of the postscript.

Notably, this minimal notion of a Vortex file effectively includes only the byte ranges, alignment, encryption, and compression configurations for other pieces of metadata.

Postscript

The postscript contains the locations of:

1. a dtype segment representing the top-level logical data type (i.e., schema)

2. a layout segment containing the root Layout

3. a statistics segment containing file-level per-field statistics (e.g., minima and maxima of each field/column, for whole-file pruning)

4. a footer segment containing a dictionary-encoded segment map, and other shared configuration such as compression and encryption schemes

5. up to 16 user-defined metadata segments, each identified by a unique, non-empty UTF-8 key of at most 64 bytes

The postscript carries a locator (offset, length, and alignment) for each metadata segment that is present; a file written without user metadata (and any file predating this feature) carries none. Readers do not load the opaque metadata values by default. Opt-in metadata reads resolve each locator separately, allowing values outside the initial file-tail read to be fetched without reading the intervening file contents.

/// The `Postscript` is guaranteed by the file format to never exceed
/// 65528 bytes (i.e., u16::MAX - 8 bytes) in length, and is immediately
/// followed by an 8-byte `EndOfFile` struct.
///
/// An initial read of a Vortex file defaults to at least 64KB (u16::MAX bytes) and therefore
/// is guaranteed to cover at least the Postscript.
///
/// The reason for a postscript at all is to ensure minimal but all necessary footer information
/// can be read in two round trips. Since the DType is optional and possibly large, it lives in
/// its own segment. If the footer were arbitrary size, with a pointer to the DType segment, then
/// in the worst case we would need one round trip to read the footer length, one to read the full
/// footer and parse the DType offset, and a third to fetch the DType segment.
///
/// The segments pointed to by the postscript have inline compression and encryption specs to avoid
/// the need to fetch encryption schemes up-front.
///
/// New fields must be appended to preserve FlatBuffers field-order compatibility with old readers.
table Postscript {
    /// Segment containing the root `DType` flatbuffer.
    dtype: PostscriptSegment;
    /// Segment containing the root `Layout` flatbuffer (required).
    layout: PostscriptSegment;
    /// Segment containing the file-level `Statistics` flatbuffer.
    statistics: PostscriptSegment;
    /// Segment containing the 'Footer' flatbuffer (required)
    footer: PostscriptSegment;
    /// User-defined metadata segments keyed by string. Keys must be unique, non-empty, and at most
    /// 64 UTF-8 bytes; readers reject postscripts that violate these limits.
    metadata: [PostscriptMetadata];
}

/// A keyed user-defined metadata segment.
table PostscriptMetadata {
    key: string (required);
    segment: PostscriptSegment (required);
}

/// A `PostscriptSegment` describes the location of a segment in the file without referencing any
/// specification objects. That is, encryption and compression are defined inline.
table PostscriptSegment {
    offset: uint64;
    length: uint32;
    alignment_exponent: uint8;
    _compression: CompressionSpec;
    _encryption: EncryptionSpec;
}

Data Type

Both viewed arrays and viewed layouts require an external DType to instantiate them. This helps us to avoid redundancy in the serialized format since it is very common for a child array or layout to inherit or infer its data type from the parent type.

The root DType segment is a flat buffer serialized DType object. See DType Format for more information.

Note

Unlike many columnar formats, the DType of a Vortex file is not required to be a StructDType. It is perfectly valid to store a Float64 array, a Boolean array, or any other root data type.

Footer

The footer is a flat buffer serialized Footer object. This object contains all the information required to load the root Layout object into a usable LayoutReader). For example, it contains the locations, compression schemes, encryption schemes, and required alignment of all segments in the file.

/// The `FileStatistics` object contains file-level statistics for the Vortex file.
table FileStatistics {
    /// Statistics for each field in the root schema. If the root schema is not a struct, there will
    /// be a single entry in this array.
    field_stats: [ArrayStats];
}

/// The `Registry` object stores dictionary-encoded configuration for segments,
/// compression schemes, encryption schemes, etc.
table Footer {
    // Dictionary-encoded array specs, up to u16::MAX.
    array_specs: [ArraySpec];
    // Dictionary-encoded layout specs, up to u16::MAX.
    layout_specs: [LayoutSpec];
    // Dictionary-encoded segment specs, up to u32::MAX.
    segment_specs: [SegmentSpec];
    // Dictionary-encoded compress specs, up to u3::MAX (8).
    compression_specs: [CompressionSpec];
    // Dictionary-encoded encryption specs, up to u16::MAX.
    encryption_specs: [EncryptionSpec];
}

/// An `ArraySpec` describes the type of a particular array.
///
/// These are identified by a globally unique string identifier, and looked up in the Vortex registry
/// at read-time.
table ArraySpec {
    id: string (required);
}

/// A `LayoutSpec` describes the type of a particular layout.
///
/// These are identified by a globally unique string identifier, and looked up in the Vortex registry
/// at read-time.
table LayoutSpec {
    id: string (required);
}

/// A `SegmentSpec` acts as the locator for a buffer within the file.
struct SegmentSpec {
    /// Offset relative to the start of the file.
    offset: uint64;
    /// Length in bytes of the segment.
    length: uint32;
    /// Base-2 exponent of the alignment of the segment.
    alignment_exponent: uint8;
    // These two fields are reserved for future use and act as pointers
    // into `FileLayout::compression_schemes` and `FileLayout::encryption_schemes`
    // respectively. They are not used in the current version of the file format.
    _compression: uint8;
    _encryption: uint16;
}

enum CompressionScheme: uint8 {
    None = 0,
    LZ4 = 1,
    ZLib = 2,
    ZStd = 3,
}

/// Definition of a compression scheme.
table CompressionSpec {
    scheme: CompressionScheme;
}

table EncryptionSpec {
}

The footer is separated from the Data Type such that large schemas can be omitted from the file if they can be shared or fetched from an external source.

Reified File Example

Since Vortex files are largely self-describing, many mainstays of other columnar file formats (e.g., whether or not to have row groups) are decided by the writer, rather than being a rigid part of the specification. To build intuition, consider an example Vortex file with two non-nullable columns, “A” of type i32, and “B” of type UTF-8. Using the defaults as of June 2025, it might look as follows.

Backward Compatibility

Later Vortex library versions retain read support for frozen formats, beginning with the formats in core2025.05.0, supported from version 0.36.0. The reader must retain the required component implementations, including optional plugins. See Versioning for the guarantee and its boundaries.

Forward Compatibility

Newer writers can already produce files for older readers by selecting editions whose formats those readers support. That allows applications to upgrade independently while continuing to exchange files in supported formats.

Reading a newly introduced format with an older reader is a separate capability. A proposed approach embeds WebAssembly decoding logic for new encodings and layouts in the file. An older reader with the required execution support could then interpret them without a native implementation. This approach is not implemented and is not part of the edition compatibility guarantee.