Skip to main content

Architecture

This chapter explains FLUX architecture in depth: runtime boundaries, module responsibilities, request paths, data ownership, concurrency model, and durability semantics.

FLUX currently supports single-broker local runtime and a 3-broker Docker runtime bootstrap. Its internal decomposition is intentionally designed to evolve toward durable distributed control/data plane separation.

Implemented Topology Diagram (Current)​

flowchart LR
    C[Clients: TypeScript/Python SDK] --> B0[broker-0 id=0 :9092]
    C --> B1[broker-1 id=1 :9093]
    C --> B2[broker-2 id=2 :9094]

    B0 --- B1
    B1 --- B2
    B0 --- B2

    subgraph BrokerNode["Each broker process contains"]
      N[Network Handler + Protocol V1]
      S[Storage Engine]
      G[Group Coordinator]
      R[Replication Manager]
      CP[Controller boundary + metadata APIs]
    end

Important current-state note:

  • We now run 3 independent broker processes in Docker with distinct broker IDs and advertised listeners.
  • Controller metadata storage is still in-memory per process (not a durable distributed quorum yet).
  • Cross-node replication RPC contract and follower worker loop are implemented as scaffolding and test harnesses, with full production-grade failover still planned.

1) Architectural Overview​

At runtime, each FLUX broker process opens a TCP port and accepts plain text requests one line at a time. In Docker cluster mode, this runs as three independent broker processes with explicit IDs and advertised listeners.

Internally it is split into these domains:

  • Protocol domain: parse/format wire envelopes (V1|<correlation_id>|<command>|<args>)
  • Network domain: per-connection request loop and command dispatch
  • Storage domain: segmented append-only logs + indexes + retention + recovery
  • Replication domain: ISR/high watermark state and ack semantics
  • Coordination domain: consumer-group lifecycle and offset commit validation
  • Composition domain: dependency wiring (broker.Broker)

Even in single-node mode, this separation makes behavior explicit and testable.

2) Process and Runtime Model​

Startup sequence​

  1. load and validate config (internal/config)
  2. initialize storage (including on-disk replay)
  3. initialize coordinator (offset + group metadata load)
  4. initialize replication manager
  5. bind TCP listener
  6. accept connections and spawn per-connection goroutines

Connection model​

  • each accepted socket is handled by network.HandleConnection
  • requests are line-delimited
  • each line is parsed independently
  • each response is one line in protocol envelope format

This is a simple request/response model rather than a streaming multiplex protocol.

3) Module-by-Module Deep Dive​

cmd/broker​

Responsibility:

  • runtime entrypoint
  • config loading
  • dependency construction
  • listener lifecycle

Important property:

  • keeps business logic out of main; orchestration only

internal/broker​

Responsibility:

  • composition root
  • holds pointers to Storage, Coordinator, Replication manager

Important property:

  • command handlers depend on a single runtime aggregate instead of global state

internal/network​

Responsibility:

  • parse request lines
  • route command names to command handlers
  • translate domain errors into protocol errors

Important property:

  • protocol-level validation in handlers is centralized and deterministic

internal/protocol​

Responsibility:

  • request parsing (V1|correlation|command|args)
  • response formatting (OK / ERR envelopes)
  • argument conversion helpers

Important property:

  • wire contract is isolated from domain modules

internal/storage​

Responsibility:

  • append records to partition segments
  • segment rollover
  • index append (offset/time)
  • startup recovery and checksum validation
  • retention enforcement (age/size)

Important property:

  • storage is authoritative for record existence and partition append order

internal/replication​

Responsibility:

  • track leader offset per partition
  • track follower progress and freshness
  • compute ISR and high watermark
  • enforce acks=all wait semantics
  • manage partition role (leader / follower)

Important property:

  • commit visibility boundary is represented by high watermark

internal/coordinator​

Responsibility:

  • group membership state machine (JOIN, SYNC, HEARTBEAT, LEAVE)
  • generation advancement and rebalance
  • assignment strategies (range/round_robin)
  • commit fencing (generation + ownership)
  • persistence of group metadata and offsets

Important property:

  • stale members are fenced via generation checks, reducing ownership races

4) Data Ownership and State Boundaries​

Understanding FLUX requires knowing who owns which state.

Storage-owned state​

  • log segments
  • in-memory partition message cache
  • segment/index metadata

Replication-owned state​

  • leader offsets
  • follower offsets / last seen
  • ISR membership flags
  • high watermark
  • partition role

Coordinator-owned state​

  • group generation
  • member liveness
  • assignment map
  • committed offsets (with offset manager)

Cross-domain interaction rule​

  • Storage writes are not automatically “committed-visible”
  • visibility to consumers is gated by replication high watermark
  • offset commit validity is gated by coordinator generation ownership

This explicit split mirrors distributed systems design patterns where durability, commit, and ownership are distinct concepts.

5) End-to-End Request Paths​

Request/Data Flow Diagram (Current)​

sequenceDiagram
    participant App as App Client
    participant Leader as Leader Broker
    participant Follower as Follower Broker

    App->>Leader: PRODUCE(topic,key:value,acks)
    Leader->>Leader: Append to partition log
    Leader->>Leader: Advance leader offset
    opt Replication worker active
      Follower->>Leader: BROKER_FETCH(topic,partition,offset)
      Leader-->>Follower: records + leader HW
      Follower->>Follower: AppendReplicated(records)
      Follower->>Leader: BROKER_REPLICA_ACK(offset)
      Leader->>Leader: Update follower progress / ISR / HW
    end
    Leader-->>App: OK partition,offset,hw (or timeout/error)

Produce path (PRODUCE)​

  1. network handler validates args (topic, key:value, acks)
  2. partition chosen via key hash
  3. replication role checked (leader required)
  4. storage append returns (partition, offset)
  5. replication leader offset advanced
  6. if acks=all, wait until high watermark reaches target and ISR minimum satisfied
  7. response returns partition, offset, hw

Key architectural point:

  • append success and commit success are separate phases

Consume path (CONSUME)​

  1. parse topic/partition/offset
  2. read current high watermark from replication manager
  3. read candidate messages from storage
  4. filter output to offset <= high watermark
  5. return committed-visible set only

Key architectural point:

  • consumers do not observe uncommitted records

Group lifecycle path (JOIN / SYNC / HEARTBEAT / LEAVE)​

  • JOIN: creates/updates member and may bump generation and rebalance
  • SYNC: validates generation and returns assignment snapshot
  • HEARTBEAT: refreshes liveness for current generation
  • LEAVE: explicit removal and rebalance

Key architectural point:

  • generation is an epoch fence for ownership correctness

Commit path (COMMIT)​

  1. parse group/topic/member/generation/partition/offset
  2. coordinator validates member is current + owns partition
  3. offset manager persists commit

Key architectural point:

  • commit is a coordination decision, not just a write

6) Durability Model and On-Disk Layout​

FLUX writes several classes of files under FLUX_DATA_DIR.

Record data and indexes​

  • <topic>-<partition>-segment-<id>.log
  • <topic>-<partition>-offset.idx
  • <topic>-<partition>-time.idx

Replication visibility artifacts (current local simulation)​

  • <topic>-<partition>-replica-<id>-segment-<id>.log

Coordination metadata​

  • offsets.json (committed offsets)
  • groups.json (group generation, members, assignments)

Recovery behavior​

On restart:

  • storage replays segments with checksum verification
  • offsets/group metadata reload from JSON snapshots
  • runtime resumes from persisted control state

7) Concurrency and Synchronization​

FLUX uses explicit mutex boundaries per subsystem.

  • storage mutex: append/read/retention state consistency
  • replication mutex: ISR/high-watermark transitions consistency
  • coordinator mutex: membership/generation/assignment consistency

Why this matters:

  • avoids cross-subsystem races in current single-process model
  • makes tests deterministic
  • provides clear migration points for future distributed state separation

8) Error Model and Failure Semantics​

Protocol-level errors are explicit and stable.

Examples:

  • BAD_REQUEST: malformed args
  • UNKNOWN_COMMAND: unsupported command
  • NOT_LEADER: produce attempted on follower role partition
  • REPLICATION_TIMEOUT: acks=all deadline exceeded
  • GENERATION_MISMATCH: stale or invalid member epoch/ownership

Design property:

  • errors map directly to architectural invariants (leadership, ISR commit, epoch fencing)

9) Current Limits vs Future Architecture​

Current state (intentionally constrained):

  • single-broker local mode plus 3-broker Docker bootstrap mode
  • replication manager semantics are stable; full replication orchestration/failover is still in progress
  • no external durable metadata quorum yet

Planned evolution path:

  1. metadata quorum and controller plane
  2. broker registration/fencing
  3. true broker-to-broker replication
  4. leader-aware client routing

This keeps the codebase pedagogically useful while progressively introducing production distributed complexity.

10) How to Read the Code with This Model​

Recommended code-reading order:

  1. internal/network/handler.go
  2. internal/protocol/protocol.go
  3. internal/storage/storage.go
  4. internal/replication/manager.go
  5. internal/coordinator/group.go and offset.go
  6. tests for each module

If you follow this order, the architectural invariants above map directly to implementation details and test cases.