Performance and Capacity

This chapter provides practical tuning guidance for current FLUX architecture.

Throughput Drivers

• number of partitions
• message size distribution
• fsync mode
• flush interval/bytes thresholds
• consumer poll interval and commit frequency

Producer-Side Patterns

• batch messages in SDK producer where possible
• use acks=1 for higher throughput when full commit guarantees are not required
• use acks=all for stronger durability at higher latency

Storage Tuning

Key knobs:

• FLUX_SEGMENT_MAX_BYTES
• FLUX_FLUSH_INTERVAL_MS
• FLUX_FLUSH_BYTES
• FLUX_FSYNC_MODE

Tradeoff summary:

• stronger durability settings increase write latency
• larger segments reduce rotation overhead but can slow some recovery paths

Consumer Tuning

• lower poll interval reduces end-to-end latency
• larger processing batches improve throughput
• heartbeat interval should stay well below session timeout

Capacity Planning (Single Node)

Estimate:

• write bytes/sec per topic
• retention window bytes
• disk growth: write rate * retention window

Plan headroom for:

• active segments
• index files
• replica mirror files
• offsets.json and groups.json

Benchmark Strategy

Measure separately:

• produce p50/p95 latency
• consume lag at steady state
• acks=all timeout rates under follower lag simulation
• restart recovery time versus dataset size