Building Asynchronous AI Task Queues for B2B SaaS Products

The Synchronous LLM Anti-Pattern in B2B SaaS

When engineering early-stage B2B SaaS MVPs, the fastest way to implement an AI feature is usually the worst way for production stability: making a synchronous HTTP call directly inside your API route handler or Next.js Server Action. You accept a POST request, call OpenAI or Anthropic, wait 15 to 45 seconds while the model generates text or executes tool calls, and then respond to the client.

This synchronous pattern fails immediately when scaled to real enterprise users. Standard edge networks and load balancers impose strict request timeouts—typically 60 seconds on Cloudflare or 15 seconds on standard Vercel hobby tiers. If an upstream LLM experiences latency spikes or if an agent executes multiple sequential tool calls, your server returns a 544 or 502 gateway timeout. Even when the request succeeds, the client thread remains locked, creating an unhelpful, laggy user interface.

When reviewing architecture on our client projects, converting synchronous AI pipelines into asynchronous background queues is usually the single highest-leverage performance fix we deploy.

The Solution: Producer-Worker Queue with Server-Sent Events

To deliver enterprise-grade stability, AI workflows must be decoupled into three distinct layers: request intake, background processing, and real-time state streaming. This decoupled design ensures that user-facing API endpoints return response codes within milliseconds, regardless of how long the backend model takes to execute.

1. Intake and Producer Handlers

When a user triggers an AI operation—such as parsing a 50-page PDF or running a financial reconciliation workflow—the web server immediately writes a task record to PostgreSQL with a pending state. It then enqueues a job payload onto a distributed Redis queue and returns an immediate 202 Accepted status containing a unique job_id.

2. Background Processing via BullMQ

Background worker processes run isolated from the web server. Using a queue library like BullMQ documentation on top of Redis, workers pull jobs off the queue concurrently. Workers handle rate-limiting, backoff strategies, and transient upstream API failures gracefully without risking the web server's memory or execution context.

3. Unidirectional Updates via Server-Sent Events (SSE)

Instead of forcing the frontend to spam the backend with continuous HTTP polling every second, the client opens an HTTP connection using Server-Sent Events (SSE). As the background worker makes progress—such as chunking text, querying a vector store, or generating structured JSON—it broadcasts lightweight JSON events over the SSE stream directly to the user client.

Architecting the Task Queue with BullMQ and Redis

BullMQ provides robust primitive abstractions for task scheduling, retries, and concurrency limits out of the box. Below is a structured blueprint for setting up worker concurrency and backoff retry logic for LLM tasks.

Worker Definition and Rate-Limit Handling

When calling external LLM providers, rate limits (Tokens Per Minute and Requests Per Minute) are your primary constraint. BullMQ allows you to cap worker concurrency and automatically pause execution when hitting HTTP 429 status codes from upstream APIs.

// worker.ts import { Worker } from 'bullmq'; import { redisConnection } from './redisConfig'; export const aiWorker = new Worker('ai-processing-queue', async job => { const { documentId, prompt } = job.data; // Step 1: Update status in primary DB await updateJobStatus(job.id, 'PROCESSING'); // Step 2: Execute long-running AI workflow const result = await executeAgentChain(documentId, prompt); // Step 3: Save results await saveResult(documentId, result); return result; }, { connection: redisConnection, concurrency: 5, limiter: { max: 20, duration: 60000 // Limit to 20 jobs per minute per worker } });

Handling Failure Modes and Enterprise Idempotency

In B2B SaaS, double-submitting a task or partially processing a document costs real infrastructure dollars and corrupts state data. Your background queue system must handle three specific edge cases:

• Network Interruptions and Timeouts: Implement exponential backoff. If an LLM call fails due to a rate limit, the queue should retry after 2s, 4s, 8s, up to a defined maximum threshold before marking the job as failed.
• Worker Crashes: Use BullMQ lock duration settings to ensure that if a worker node crashes mid-execution, Redis reassigns the stalled job to an active worker automatically after the lock expires.
• Idempotent Job Keys: Generate deterministic job IDs based on a SHA-256 hash of the user ID and the target payload. If a user double-clicks a action button, BullMQ deduplicates the payload and prevents duplicate LLM invocations.

Server-Sent Events vs WebSockets for SaaS Dashboards

Engineers often default to WebSockets for real-time features. However, for 95% of AI SaaS applications, WebSockets add unnecessary state complexity. WebSockets are full-duplex, requiring complex load balancer sticky sessions and reconnection logic.

Server-Sent Events (SSE), by contrast, operate over standard HTTP, natively support automatic reconnection via standard browser APIs, and work seamlessly through standard HTTP/2 multiplexing. You push text chunks and workflow progress indicators downstream from server to client over a persistent read stream, closing the stream once the queue emits a completed event.

Key Architecture Guidelines for Modern SaaS Builders

Building a successful AI-enabled B2B SaaS product requires isolating high-latency dependencies away from user request lifecycles. By combining lightweight web handlers, persistent BullMQ workers, and unidirectional SSE streams, you achieve a system that handles scaling spikes without dropped connections or degraded frontend user experiences.

If your B2B SaaS user is sitting in front of a spinning loader waiting for an LLM to generate a complex document over an HTTP connection, your architecture is broken.