Designing an Async Semantic Caching Layer for Enterprise LLM Gateways

When engineering enterprise LLM gateways, most teams start with standard HTTP payload caching. They hash the incoming request—combining prompt text, system instructions, and generation parameters—with SHA-256 and look up the string key in Redis. This works for identical automated tasks, such as scheduled hourly reporting scripts. But in human-driven workflows, exact-string matching yields a depressing hit rate: typically under 5%. A user typing "What were our Q3 sales in North India?" misses the cache entirely if another user previously asked "Show Q3 revenue for North India region."

To bridge this gap, teams attempt to build a semantic cache. The premise is straightforward: convert incoming prompt queries into vector embeddings, measure the distance between the new prompt and previous queries stored in a vector index, and return the cached completion if the distance falls below a set similarity threshold. However, naive implementations frequently backfire. If your semantic lookup pipeline takes 120ms to fetch embeddings from an external API and evaluate vector distance, only to result in a cache miss 70% of the time, you have introduced an unacceptably high latency tax across your entire gateway.

The Core Architectural Constraint: The Latency Budget

To make a semantic cache viable for real-time application workloads, the entire cache evaluation path must complete in under 15ms. If evaluation takes longer, the tail latency added to cache misses degrades user experience far more than sporadic cache hits save it.

When we audit enterprise client architectures during a Systems Audit & Blueprint, we regularly encounter designs that call OpenAI's embedding endpoints synchronously inside the HTTP gateway thread just to check for cached responses. Generating an embedding over a remote HTTPS connection introduces 80ms to 180ms of network overhead. That completely invalidates the performance rationale for caching in the first place.

Building a Two-Tier Cache Pipeline

To keep cache lookup overhead well under 15ms, we decouple query evaluation into a two-tier pipeline combining deterministic exact-string lookup with local vector inference:

Tier 1: L1 Deterministic Hash Cache

Before executing any vector math, the incoming request string is normalized (stripped of whitespace, lowercased, and punctuation-cleaned) and hashed alongside the tenant identifier. The gateway queries a standard Redis key-value store. This lookup costs under 1ms. If an exact match exists within the active TTL window, the gateway returns the cached response immediately, bypassing embedding generation entirely.

Tier 2: L2 On-Device Vector Semantic Index

If Tier 1 misses, the gateway routes the normalized prompt to an in-process embedding engine using ONNX Runtime. By running a compact, quantized transformer model like all-MiniLM-L6-v2 (384 dimensions) directly on CPU or local acceleration co-located with the gateway process, vector generation completes in 3ms to 6ms without any network calls.

Once the 384-dimensional vector is generated, the gateway executes an Approximate Nearest Neighbor (ANN) search against an HNSW (Hierarchical Navigable Small World) vector index. Using optimized engines such as Redis Search and Query Documentation, vector distance calculations across hundreds of thousands of prompt vectors complete in under 4ms.

Navigating Distance Thresholds and Collisions

The primary hazard in semantic caching is returning a semantically close but contextually incorrect answer. In enterprise analytics, "What was our Q3 EBITDA?" and "What was our Q3 Revenue?" reside extremely close to each other in vector embedding space. Returning a cached EBITDA figure for a Revenue query is a severe data accuracy failure.

To eliminate false positive collisions, a production semantic cache must enforce three structural rules:

• Strict Cosine Similarity Thresholds: While standard vector retrieval applications accept similarity scores around 0.78, semantic caching for analytical queries demands a strict minimum threshold between 0.92 and 0.95. Anything below 0.92 is treated as a definitive cache miss.
• Metadata Pre-Filtering: Vector distance calculations must never run across raw database indexes without scope constraints. Tenant ID, role-based access control (RBAC) groups, and target data schema version must serve as hard pre-filters in the HNSW index before vector distance is evaluated.
• Dynamic TTL Based on Distance Decay: Matches evaluated at 0.98 similarity receive a standard 24-hour TTL, whereas responses matched at 0.92 are assigned a short 1-hour TTL, ensuring marginal matches decay rapidly from memory.

Asynchronous Write Backs and Invalidation

Writing new prompt-response pairs to the semantic cache must never block the client response stream. When a cache miss occurs and the primary LLM streams its completion, the gateway streams tokens directly to the client while enqueuing an asynchronous write task into a background worker queue using BullMQ or Celery.

The background worker receives the complete prompt and response pair, generates the vector entry using the local ONNX embedding pipeline, attaches required tenant metadata tags, and writes the record to the vector store atomically.

Cache Invalidation Strategies

Unlike conventional SQL-backed web applications where cache invalidation happens via key deletion on mutation, invalidating a semantic cache requires tag-based scope clearing. When underlying database tables or RAG documentation stores undergo ETL refreshes, an orchestration hook broadcasts a clear event that removes all cache vectors associated with those specific dataset identifiers.

Production Results and Tradeoffs

Implementing this two-tiered semantic caching model across multi-tenant gateways delivers measurable infrastructure improvements:

• Latency Reduction: Average p95 latency drops from 2,100ms down to 14ms on cache hits.
• Cost Efficiency: Billable token volume on primary LLM providers decreases by 60% to 70% in high-frequency analytical portals.
• Compact Memory Footprint: A 384-dimensional float32 vector in an HNSW index occupies less than 2 KB per entry including metadata, allowing millions of historical queries to fit into lightweight Redis instances.

By keeping embedding generation local and setting rigorous vector similarity limits, enterprise gateways reduce compute costs and response latency without compromising semantic precision.

If your semantic cache introduces 40ms of embedding computation just to decide that a query is a cache miss, you have built a latency tax, not an optimization.