Performance & Cost

The same answer can cost 10× more or arrive 5× slower depending on engineering. Caching, routing, streaming, fallbacks — the optimization stack, honestly prioritized.

What you'll learn

1. Caching layers
2. Model routing
3. Latency & resilience
4. Quotas & throughput

Remember this

• Three caches compound: static-first prompts for provider cache reuse, exact response cache, high-threshold semantic cache — with versioned keys and TTLs
• Model routing sends easy traffic to cheap lanes with EXTERNAL verification and policy exceptions; PTU covers the steady baseline, on-demand the peaks
• Latency = streaming + parallelism + small contexts; resilience = multi-provider fallback with constant response shapes and idempotent retries; capacity pressure sheds load by design

Code: The request pipeline with every lever in place

async def handle(q, user):
    key = cache_key(q, user.tenant, prompt_version())
    if hit := response_cache.get(key):            # exact
        return hit
    if near := semantic_cache.search(embed(q), 0.95):
        return near                               # paraphrase

    route = triage(q)                             # cheap | strong
    try:
        async with token_bucket(user):            # smoothing
            stream = route.model.stream(assemble_static_first(q))
            return await emit_sse(stream,         # first token <1s
                                  on_complete=lambda r: (
                                      response_cache.set(key, r),
                                      semantic_cache.add(q, r)))
    except ProviderUnavailable:
        return await fallback_provider.handle(q)  # same shape, degraded note