Skip to main content

Storage

FLUX storage is a segmented append-only log engine with recovery and retention controls.

Log Structure​

Each topic-partition writes to segment files:

data/<topic>-<partition>-segment-<segment_id>.log

Record format:

<offset>|<timestamp_ms>|<crc32_hex>|<value>

Example:

0|1715840400000|8c736521|created

Indexes​

Side indexes per topic-partition:

data/orders-2-offset.idx
data/orders-2-time.idx

These support future optimized seek/fetch behavior.

Partitioning​

  • key hashing: FNV-1a
  • same key -> same partition (stable for fixed partition count)
  • per-partition ordering preserved

Recovery​

On startup:

  1. discover/sort segment files
  2. replay records in order
  3. verify checksum per record
  4. skip corrupt records
  5. rebuild in-memory state

Retention​

Retention enforcement runs on write and removes old segments by:

  • max age (FLUX_RETENTION_MAX_AGE_SECONDS)
  • max bytes (FLUX_RETENTION_MAX_BYTES)

Durability Knobs​

  • FLUX_FLUSH_INTERVAL_MS
  • FLUX_FLUSH_BYTES
  • FLUX_FSYNC_MODE (always, interval, never)

Persistence Files​

  • segment logs and indexes
  • offsets.json for committed offsets
  • groups.json for durable consumer-group state

Local Replica Mirror Files​

Current replication visibility also writes same-node replica mirror files:

data/orders-2-replica-1-segment-000000.log
data/orders-2-replica-2-segment-000000.log