Roadmap

This roadmap tracks FLUX progression from prototype runtime to production-grade distributed streaming.

Phase Status

Phase	Status
Phase 0: Prototype hardening	Completed
Phase 1: Storage engine v1	Completed
Phase 2: Replication and partition leadership	Completed
Phase 3: Consumer coordination v2	Completed
Phase 4: Metadata quorum and control plane	Completed (scaffold level)
Phase 5: Real distributed cluster runtime	In progress
Phase 6: Durable controller quorum	Planned
Phase 7: Security and multi-tenancy	Planned
Phase 8: Observability and operations	Planned
Phase 9: Performance and scale validation	Planned

Completed Highlights

Phase 0

• protocol hardening and response envelopes
• structured config and validation
• structured logging and module test coverage

Phase 1

• segmented append-only storage
• offset/time indexes
• startup recovery + checksum validation
• retention and durability knobs

Phase 2

• follower progress protocol (REPLICA_FETCH)
• ISR and high watermark tracking
• role transitions (SET_PARTITION_ROLE)
• produce ack modes and committed-read behavior
• under-replicated partition signaling

Phase 3 (Completed)

• heartbeat/session-timeout group lifecycle
• join/sync/rebalance generation behavior
• range + round-robin assignors
• durable group metadata (groups.json)
• generation-safe commit validation
• TypeScript SDK high-level producer/consumer runtime
• Python SDK high-level producer/consumer runtime parity
• SDK retry/backoff + lifecycle hooks (onAssign, onRevoke, onCrash)

Phase 4 (Started)

Completed today and this sprint:

• control-plane metadata model scaffold
• controller service abstraction over metadata store
• raft-shaped in-memory metadata store with command apply + snapshot seams
• broker wiring for controller usage
• admin protocol commands:
  • ADMIN_CREATE_TOPIC
  • ADMIN_REGISTER_BROKER
  • ADMIN_BROKER_HEARTBEAT
  • ADMIN_SET_PARTITION_LEADER
  • ADMIN_GET_METADATA
• runtime metadata migration:
  • produce/consume paths now consult controller-backed topic/partition metadata
  • topic metadata bootstrap path added for compatibility during transition

Phase 5 (In Progress)

Completed now:

• broker-to-broker replication RPC contracts:
  • BROKER_FETCH
  • BROKER_REPLICA_ACK
• follower replication loop scaffold (fetch -> apply -> ack)
• unit + integration harness coverage for replication flow
• 3-broker Docker runtime bootstrap with broker identity + advertised listener config:
  • FLUX_BROKER_ID
  • FLUX_ADVERTISED_HOST
  • FLUX_ADVERTISED_PORT
  • FLUX_CLUSTER_PEERS

Next Focus

1. Implement full broker-to-broker replication orchestration across separate nodes.
2. Enforce committed-read visibility from replicated high watermark across nodes.
3. Validate failover flow in integration tests (leader crash -> election/promotion -> continued traffic).
4. Begin durable controller quorum implementation after Phase 5 runtime hardening.

For detailed execution tracking, see the Production Plan.