Operations

Demo day is easy. Day 400 is the job — upserts, sizing, tenants, and the restore you pray you'll never need.

▶ Watch this reel

What you'll learn

Indexing & updates
Sizing & cost
Multi-tenancy & security
Backup & DR

Remember this

Deterministic IDs + upserts make re-ingestion idempotent; CDC beats rebuilds; deletes need budget
Size: N × dims × bytes × index overhead × 2 headroom; shard before the node fills; tenancy enforced in one code path
Source docs are the crown jewels; index is derived — snapshot for RTO, rehearse the restore

Indexing & updates

Upsert by deterministic ID (sha1(doc_id:chunk_idx)) → idempotent re-ingestion.
Deletes: ANN graph repair is costly → tombstone + scheduled compact.
CDC (source events) over periodic rebuilds; full rebuild = fallback.

Sizing & cost

Memory = N × dims × bytes × (1 + HNSW ~0.3-0.5) × 2 (headroom).
Quantization (float16 / PQ) when memory bites — gate on golden-set recall.
Plan sharding before the node fills; resharding full stores is surgery.

Multi-tenancy & security

Filter-based: cheap, one forgotten WHERE from a breach → enforce centrally + test.
Per-tenant collections: structural isolation, moderate overhead — most SaaS tiers.
Per-tenant DB/cluster: regulated tiers, cost follows assurance.

Backup & DR

Source docs = crown jewels (PIT recovery). Embeddings/index = derived → snapshots for RTO.
Rehearse restores quarterly, time them against the promised RTO.
Untested backup = rumor.

Code: The operations layer — writes, tenancy, restore drill

from hashlib import sha1

# --- 1. Deterministic upserts --------------------------------------
def chunk_id(doc_id: str, idx: int) -> str:
    return sha1(f"{doc_id}:{idx}".encode()).hexdigest()[:16]

for doc in changed_docs():              # from CDC events, not a rebuild
    for i, chunk in enumerate(chunk(doc.text)):
        store.upsert(id=chunk_id(doc.id, i),
                     vec=embed(chunk),
                     payload={"doc_id": doc.id, "v": doc.version,
                              "tenant": doc.tenant, "text": chunk})

# --- 2. Tenant enforcement (one path, tested) ----------------------
def search(q, tenant: str, k=5):
    assert tenant, "tenant is mandatory"
    return store.search(embed(q), k=k, where={"tenant": tenant})

# test:  search(q, tenant=None) must raise — CI enforces it

# --- 3. Restore drill (quarterly, timed) ---------------------------
def restore_drill():
    t0 = time.time()
    for doc in source_store.list_all():          # the crown jewels
        reindex(doc)                              # embed + upsert
    elapsed = time.time() - t0
    assert elapsed < RTO_SECONDS, f"rebuild {elapsed:.0f}s > RTO"
    # Snapshot alternative: store.snapshot() → restore in minutes

# --- 4. Delete = tombstone + compact -------------------------------
# store.delete(ids) marks; a scheduled job runs store.compact()
# off-peak — latency stays smooth, graph gets repaired.