Skip to main content

Internals: Broker Runtime

This chapter walks through broker internals as if you are debugging production behavior.

Startup Path​

  1. load config and validate runtime knobs
  2. initialize storage (recovery replay + retention state)
  3. initialize coordinator (offsets + group metadata)
  4. initialize replication manager
  5. start TCP listener and spawn per-connection handlers

Connection Handling Model​

  • each accepted TCP connection is handled by one goroutine
  • each request line is parsed into V1 envelope fields
  • handler routes by command and writes one line response

Request Processing Pipeline​

wire line -> ParseRequest -> command handler -> state mutation -> response envelope

Core In-Memory State​

  • storage in-memory message cache per topic-partition
  • replication partition state (leader offset, follower offsets, ISR flags)
  • coordinator group state (generation, members, assignments)

Disk State​

  • segments + indexes per topic-partition
  • offsets.json for committed offsets
  • groups.json for group metadata

Fail/Recovery Behavior​

  • malformed requests are rejected with protocol error
  • on restart, storage rebuilds from segments
  • group and offset state reloads from JSON snapshots

Current Concurrency Boundaries​

  • storage manager uses mutex for append/consume state
  • coordinator manager uses mutex for group lifecycle
  • replication manager uses mutex for ISR/HW transitions

These boundaries are simple and safe for current single-node architecture.