Roadmap

This roadmap tracks FLUX progression from single-node prototype to production-grade distributed streaming.

Phase Status

Phase	Status
Phase 0: Prototype hardening	Completed
Phase 1: Storage engine v1	Completed
Phase 2: Replication and partition leadership	Completed
Phase 3: Consumer coordination v2	Completed
Phase 4: Metadata quorum and control plane	In progress
Phase 5: Security and multi-tenancy	Planned
Phase 6: Observability and operations	Planned
Phase 7: Performance and scale validation	Planned

Completed Highlights

Phase 0

• protocol hardening and response envelopes
• structured config and validation
• structured logging and module test coverage

Phase 1

• segmented append-only storage
• offset/time indexes
• startup recovery + checksum validation
• retention and durability knobs

Phase 2

• follower progress protocol (REPLICA_FETCH)
• ISR and high watermark tracking
• role transitions (SET_PARTITION_ROLE)
• produce ack modes and committed-read behavior
• under-replicated partition signaling

Phase 3 (Completed)

• heartbeat/session-timeout group lifecycle
• join/sync/rebalance generation behavior
• range + round-robin assignors
• durable group metadata (groups.json)
• generation-safe commit validation
• TypeScript SDK high-level producer/consumer runtime
• Python SDK high-level producer/consumer runtime parity
• SDK retry/backoff + lifecycle hooks (onAssign, onRevoke, onCrash)

Phase 4 (Started)

Completed today and this sprint:

• control-plane metadata model scaffold
• controller service abstraction over metadata store
• raft-shaped in-memory metadata store with command apply + snapshot seams
• broker wiring for controller usage
• admin protocol commands:
  • ADMIN_CREATE_TOPIC
  • ADMIN_REGISTER_BROKER
  • ADMIN_BROKER_HEARTBEAT
  • ADMIN_SET_PARTITION_LEADER
  • ADMIN_GET_METADATA
• runtime metadata migration:
  • produce/consume paths now consult controller-backed topic/partition metadata
  • topic metadata bootstrap path added for compatibility during transition

Next Focus

1. Implement real distributed metadata quorum (networked consensus and durable log).
2. Add automatic partition leader election beyond manual/admin assignment.
3. Add broker fencing and stronger health-state transitions.
4. Advance client-plane metadata discovery + multi-broker routing.

For detailed execution tracking, see the Production Plan.