Skip to main content

Architecture

Distributed Object Storage separates request handling, metadata transactions, file storage, and replication work so product behavior remains explicit.

Components​

ComponentResponsibility
HTTP APIvalidates requests and serves upload, presign, and download routes
Service layercoordinates object operations
Repositoryperforms PostgreSQL transactions
Storage adapterreads and writes object bytes
Replication workerprocesses durable replication jobs
Migratorauto-applies embedded SQL migrations on startup

Upload Path​

request -> path validation -> primary write -> metadata transaction -> replication job -> response

The metadata row and replication job are committed together so secondary-copy work is not lost after a successful upload.

Within the same upload transaction:

  1. previous latest row for (bucket, object_key) is unset
  2. new object row is inserted with is_latest=true
  3. replication job row is inserted

Replication Path​

pending -> running -> completed
pending -> running -> pending
pending -> running -> failed

The worker retries transient failures and records terminal failures after retry exhaustion.

Current implementation details:

  • job claiming uses FOR UPDATE SKIP LOCKED for safe concurrent workers
  • retry backoff is quadratic (attempt_count^2 seconds)
  • replica records are idempotent per (object_id, node_name)

Deployment Model​

The current release uses local filesystem nodes and PostgreSQL with startup schema migration application. Future releases are expected to add stronger operational controls, authentication, and broader object-storage APIs.