Skip to main content

Flux

FLUX is a Go-based event streaming system focused on distributed messaging fundamentals with a clean TypeScript integration path.

This documentation is meant to be a complete practical guide: architecture, protocol, durability, replication semantics, consumer-group behavior, deployment, and SDK usage.

Current Product Capabilities​

  • Versioned Text TCP Protocol (V1) with Correlation IDs: every request and response uses the same fixed format, so clients can parse reliably and match each response to the correct request.
  • Deterministic Key-Based Partition Routing: if partition count stays the same, the same key always goes to the same partition, so events for that key stay in order.
  • Segmented Append-Only Storage per Topic-Partition: messages are written in sequence to log segments, which makes writes fast and makes replay straightforward.
  • Checksum-Based Startup Recovery: when the broker restarts, it checks saved records for corruption and skips bad records instead of failing the whole startup.
  • Retention by Age and Size: old data is deleted automatically based on configured time limits and total storage limits.
  • Replication Manager with ISR + High Watermark Tracking: FLUX keeps track of which replicas are up to date, and only lets consumers read messages that are safely replicated.
  • Produce Ack Modes (acks=0|1|all): acks=0 or acks=1 responds faster but gives weaker safety; acks=all is safer but usually slower because it waits for stronger replication conditions.
  • Leader/Follower Partition Role Transitions: a partition can be set to leader (accept writes) or follower (reject writes), which helps enforce role-based write control.
  • Consumer Groups with Generation-Based Membership: every rebalance creates a new generation number (a version id for current group membership), so old consumers cannot keep committing with stale ownership.
  • Join/Sync/Heartbeat/Leave Lifecycle: consumers join the group, sync assignments, send heartbeats to stay active, and leave cleanly during shutdown.
  • Generation-Safe Offset Commits: a commit is accepted only if member id, generation, and partition ownership all match current group state.
  • Durable Offsets (offsets.json) and Group Metadata (groups.json): committed offsets and group state are saved to disk, so they survive broker restarts.
  • TypeScript SDK for Application Usage: application teams can produce, consume, and manage group flow from TypeScript without manually building protocol strings.
  • High-Level TypeScript Runtime API for Producer/Consumer Workflows: FLUX also provides a higher-level runtime API for common producer and consumer loops, reducing repeated lifecycle code.
  • Python SDK for Application Usage: Python applications can use low-level protocol APIs and high-level runtime producer/consumer workflows with parity to TypeScript.
  • Control Plane Metadata Scaffold: FLUX now has a dedicated controller + metadata store boundary for cluster metadata operations (topic lifecycle, broker state, partition leadership).
  • Admin Protocol Commands: FLUX now exposes admin operations directly over protocol (ADMIN_CREATE_TOPIC, ADMIN_REGISTER_BROKER, ADMIN_BROKER_HEARTBEAT, ADMIN_SET_PARTITION_LEADER, ADMIN_GET_METADATA).
  • Metadata Migration on Runtime Path: produce/consume paths now ensure topic metadata is represented in control-plane state and partition routing decisions read from metadata snapshot context.
  • Multi-Broker Docker Runtime Bootstrap: FLUX can now run as three independent broker processes (broker-0, broker-1, broker-2) with explicit broker IDs, advertised listeners, and cluster peer config.
  • Broker-to-Broker Replication RPC Scaffold: BROKER_FETCH and BROKER_REPLICA_ACK contracts plus follower worker loop scaffolding are implemented for cross-node replication flow development and testing.

Current Scope​

FLUX now supports both:

  • single-broker local runtime for fast iteration
  • three-broker Docker runtime bootstrap for distributed systems development

This means:

  • strong educational and integration-test value
  • deterministic protocol behavior for key distributed systems concepts
  • realistic multi-node process identity/topology in Docker
  • no full durable multi-broker metadata quorum yet

Not Yet Available​

  • full production-grade multi-broker network replication with automated orchestration and failover
  • full distributed metadata quorum with networked consensus and durable controller log replication
  • automatic partition leader election across multiple brokers
  • advanced feature parity for transactions, exactly-once, and multi-broker routing

Core Concepts​

ConceptMeaning
Topicnamed event stream
Partitionordered shard of a topic
Offsetposition of a message in a partition
Segmentbounded log file chunk
ISRin-sync replica set used for committed visibility
High watermarkhighest committed offset visible to consumers
Generationconsumer-group membership epoch

Documentation Map​

Foundations​

  1. Getting Started
  2. Architecture
  3. Protocol

Internals​

  1. Storage
  2. Replication
  3. Broker Runtime
  4. Group Coordinator
  5. Delivery Semantics

SDK​

  1. TypeScript SDK
  2. Python SDK (covered in Getting Started + repository README until dedicated page lands)

Operations​

  1. Deployment
  2. Operations Runbook
  3. Performance and Capacity
  4. Security Model

Strategy​

  1. Roadmap
  2. Multi-Cluster Strategy
  3. Contributing

Source of Truth​