Architecting High-Throughput Webhook Gateways: Fastify, Redis Streams, and Backpressure

The Webhook Ingestion Death Spiral

Last year, an enterprise platform we worked with ran into a catastrophic failure mode during a flash sale campaign. Their web infrastructure was processing incoming webhook callbacks from payment gateways, warehouse inventory managers, and third-party logistics platforms. Every time a payment status changed or an item was scanned, an HTTP POST payload hit their primary application backend.

Under quiet baseline conditions, the endpoint worked fine. It received the payload, verified the signature, opened a relational database transaction, updated five different tables, and returned a HTTP 200 OK response within 180 milliseconds. But during peak load, when webhook traffic spiked from 20 requests per second to over 3,000 requests per second, the architecture fell apart. The relational database ran out of available connection pools. Request queues backed up inside the web worker nodes, latency spiked to 12 seconds per request, and HTTP timeouts began cascading across the network. Because the payment provider received 504 Gateway Timeouts, it automatically initiated aggressive exponential retries, multiplying the incoming traffic load by three. Within ten minutes, the primary portal went completely offline.

This failure pattern is shockingly common in modern enterprise application development. Teams frequently mistake an asynchronous event ingestion problem for a standard CRUD application route. If your HTTP endpoint executes a database write before sending a 200 OK header back to a third-party gateway, you are running an expensive downtime incident waiting to happen.

Why Serverless Lambdas Are Not a Silver Bullet

The standard architectural recommendation from public cloud providers is to toss serverless functions in front of an API gateway. The logic sounds clean on paper: serverless functions auto-scale instantly to absorb thousands of concurrent incoming HTTP POST calls, isolating your primary web app server from the initial ingress spike.

In practice, offloading raw webhook ingestion directly to serverless functions creates two expensive problems. First, serverless functions running on AWS Lambda or Google Cloud Functions still need to write that payload somewhere. If three thousand concurrent lambdas spin up in five seconds, they will instantly exhaust your database connection proxy or exhaust your connection limit on Managed PostgreSQL or MySQL. Second, serverless pricing models punish high-frequency, short-lived executions due to minimum billing durations and API gateway overhead charges. Billing $0.0000002 per invocation sounds negligible until you are processing 80 million webhook callbacks per month, at which point the infrastructure bill becomes eye-watering.

To solve this permanently for client systems, we moved away from serverless ingress handlers and built a dedicated, stateless ingestion tier using Fastify and Redis Streams. You can learn how we structure these production environments by reading about our custom web development services.

Building the Intake Gateway: Fastify and Schema Validation

When engineering an ingress service, your primary objective is speed and memory efficiency. The HTTP handler must do only three things before closing the connection: validate that the payload structure is valid, verify the cryptographic signature (HMAC), and append the event payload to an append-only log in memory.

We selected Fastify over Express or NestJS for this layer due to its near-zero overhead routing and compiled JSON schema parsing. Fastify utilizes compile-time JSON schema validation powered by Ajv. Instead of parsing arbitrary JSON blobs into dynamic JavaScript objects at runtime, Fastify pre-compiles the expected webhook schema into optimized functions during application boot.

When a third-party server hits the ingestion gateway, Fastify verifies the headers and HMAC signature using node crypto constructs, executes the compiled schema validation synchronously, and immediately streams the raw string payload into Redis. By adhering closely to Fastify's official performance guidelines, the entire HTTP lifecycle completes in under 8 milliseconds, maintaining a tiny memory footprint on lightweight runtime nodes.

Decoupling Ingestion from Execution with Redis Streams

The core engine behind this resilient architecture is Redis Streams using the XADD primitive. Unlike traditional Redis Pub/Sub, which is ephemeral and loses messages if a subscriber disconnects, Redis Streams provides a durable, ordered, append-only log that resides in memory with optional persistence to disk.

When an incoming HTTP POST payload clears signature validation, the Fastify route issues a non-blocking XADD command to append the raw payload into a stream named webhooks:incoming. Because Redis handles string appends in single-digit microseconds, the web worker completes the command and returns an immediate 202 Accepted or 200 OK response to the external sender. The upstream sender considers the delivery successful and terminates its retry timer.

Behind the stream sits an independent pool of background worker processes configured into Redis Consumer Groups using XREADGROUP. Consumer groups allow us to distribute the processing workload across dozens of worker containers without race conditions or duplicate processing. Each worker pulls a batch of 50 events from the stream, parses the body, opens necessary database connections, and executes business logic asynchronously.

Managing Backpressure and Downstream Outages

The greatest advantage of decoupling ingress from processing via Redis Streams is backpressure control. Downstream services—such as PostgreSQL databases, third-party analytics APIs, or search indexes—have finite throughput limits. If an upstream service sends 5,000 webhooks per second, but your database cluster can only safely process 800 complex writes per second, a synchronous architecture crashes.

With Redis Streams acting as a shock absorber, the intake gateway continues accepting 5,000 webhooks per second into memory without breaking a sweat. Meanwhile, the worker pool continues consuming events at its maximum safe capacity of 800 writes per second. The stream depth temporarily expands, holding the unconsumed messages safely in queue. Once the external traffic burst subsides, the background consumers catch up and drain the stream back to zero.

If a worker process crashes mid-execution while updating the database, Redis Streams tracks unacknowledged messages in the Pending Entries List (PEL). A supervisor background process periodically queries the PEL using XPENDING and claims stale messages using XCLAIM, re-assigning them to healthy worker nodes. Zero events are dropped, zero database connections are exhausted, and HTTP client connections never time out.

Accelerating Workflow Strategy with AI Tooling

When building complex queue management systems, edge cases in concurrent state management are difficult to simulate manually. During our development workflow, we utilized Claude Code and Cursor to accelerate test suite creation. We prompted the AI assistant to write custom k6 load testing scripts that deliberately injected network jitter, dropped TCP packets, and simulated random database locks during consumer execution.

Rather than relying on generic code generation, we used AI models as automated red-team tools. We instructed the LLM to inspect our Redis stream consumer handling logic specifically looking for unhandled promise rejections during connection dropouts. This workflow cut our integration testing phase down from weeks to days, surfacing edge-case consumer lockups before any code touched production deployment environments.

Real-World Operational Results

Replacing synchronous HTTP webhook handlers with a dedicated Fastify and Redis Stream ingestion pipeline completely eliminates spike-induced outages. In production deployments handling millions of daily events, this pattern maintains a consistent p99 HTTP latency of under 8 milliseconds under heavy load.

Infrastructure costs decrease substantially because you no longer need to scale out expensive web application containers just to hold open long-lived HTTP client connections. Lightweight Fastify gateway nodes handle massive concurrency on minimal RAM and CPU allocations, leaving your heavy business logic and database queries to run at a controlled, predictable pace in background worker pools.

If your HTTP endpoint executes a database write before sending a 200 OK header back to a third-party gateway, you are running an expensive downtime incident waiting to happen.