System Architecture & Design

1. System Overview

What it is

• An embedded, single-process storage engine with sharded single-writer runtimes and lock-free snapshot readers.

Component Diagram

flowchart TB
  API["Public API: pomai::DB"] --> MM[MembraneManager]
  MM --> ENG["Engine (per membrane)"]
  ENG --> SHARD["Shard (per shard)"]
  SHARD --> RT[ShardRuntime]
  RT --> WAL["WAL per shard"]
  RT --> MEM[Active MemTable]
  RT --> FMEM[Frozen MemTables]
  RT --> SEG[Segments]
  SEG --> MAN[Shard Manifest]
  ENG --> POOL[Search Thread Pool]
            

Data Paths

• Write path: API → shard mailbox → WAL append → Active MemTable → optional soft freeze → snapshot publish.
• Read path: API → snapshot load → frozen memtables → segments → merge.
• Persistence: Freeze flushes frozen memtables to segments, updates manifest, resets WAL.

Request Lifecycle (Upsert)

sequenceDiagram
  participant Client
  participant DB as pomai::DB
  participant Shard as ShardRuntime
  participant WAL
  participant Mem as Active MemTable
  participant Snap as ShardSnapshot

  Client->>DB: Put(id, vec)
  DB->>Shard: Enqueue PutCmd
  Shard->>WAL: AppendPut
  Shard->>Mem: MemTable::Put
  alt count >= 5000
    Shard->>Snap: RotateMemTable + PublishSnapshot
  end
  Shard-->>Client: Status::Ok
            

2. Consistency Model

Core Guarantees

• Snapshot Isolation per shard: Reads operate on a single immutable ShardSnapshot.
• Bounded Staleness: Writes are visible only after active MemTable rotation (soft freeze).

State Machine

stateDiagram-v2
  [*] --> Active
  Active --> Frozen: "soft freeze (>=5000)"
  Active --> Frozen: "explicit Freeze (rotates)"
  Frozen --> Segment: Freeze flush
            

Timeline Example

T0: Put(id=1) -> appended to WAL + Active MemTable
T1: Get(id=1) -> NotFound (active not in snapshot)
T2: Active count reaches 5000 -> RotateMemTable + PublishSnapshot
T3: Get(id=1) -> Found (visible in snapshot)

3. CBR-S Implementation

CBR-S stands for Cluster-Based Routing System. It handles Global-Centroid Shard Assignment + Routed Query + Dual-Epoch Probe.

Routing Table Format

• File: ROUTING (binary payload + CRC32C)
• Contains: centroids, owner_shard mapping.

Dual-Epoch Probe Strategy

1. Compute nearest centroids to query.
2. Choose probe count (default 2).
3. Map centroid IDs to shard IDs.
4. Dual-Epoch Logic: Union with previous table's shard set if available to mitigate misses during transition.