Queues

Queues provide reliable job processing for background work. Unlike messaging where every subscriber gets every message, queues ensure only one consumer processes each job.

How Queues Differ from Messaging

Aspect	Messaging	Queues
Delivery	Every subscriber gets every message	One consumer per job
Acknowledgment	None (fire-and-forget)	Explicit job.ack() required
Failure handling	Message lost if subscriber fails	Job redelivered to another consumer
Use case	Broadcasting, notifications	Background jobs, task processing

┌──────────┐       ┌─────────┐       ┌────────────┐
│ Producer │ ───▶  │  Queue  │ ───▶  │ Consumer A │  ✓ Gets job 1
└──────────┘       │         │       └────────────┘
                   │  Job 1  │       ┌────────────┐
                   │  Job 2  │ ───▶  │ Consumer B │  ✓ Gets job 2
                   │  Job 3  │       └────────────┘
                   └─────────┘       ┌────────────┐
                                     │ Consumer C │  (idle, waiting)
                                     └────────────┘

Publishing Jobs

const queue = await client.initQueue('email-queue');

await queue.publish('welcome-email', {
  to: 'user@example.com',
  subject: 'Welcome!',
  template: 'welcome'
});

Jobs are persisted until a consumer acknowledges them.

Consuming Jobs

const queue = await client.initQueue('email-queue');

await queue.consume(
{ topic: 'welcome-email', name: 'email-worker', group: 'workers' },
async (job) => {
  // Process the job
  await sendEmail(job.message);
  
  // Acknowledge completion - REQUIRED
  await job.ack();
}
);

Acknowledge Your Jobs

If you don't call job.ack(), the job will be redelivered after the visibility timeout. Always acknowledge after successful processing.

Consumer Groups

Multiple consumers can process jobs from the same queue in parallel:

// Worker 1 (process A)
await queue.consume({ topic: 'images', name: 'worker-1', group: 'processors' }, processImage);

// Worker 2 (separate process/machine)
await queue.consume({ topic: 'images', name: 'worker-2', group: 'processors' }, processImage);

// Worker 3
await queue.consume({ topic: 'images', name: 'worker-3', group: 'processors' }, processImage);

// Jobs are distributed across all active consumers

Add more workers to process jobs faster — RelayX handles the distribution.

The Acknowledgment Contract

Acknowledgment tells RelayX "I've successfully handled this job, don't send it again."

Action	Result
job.ack() called	Job removed permanently
Consumer crashes before ack()	Job redelivered after timeout
Consumer disconnects before ack()	Job redelivered after timeout
Timeout expires without ack()	Job redelivered to any available consumer

Handling Failures

Transient Failures (Retry Later)

Don't acknowledge — let the timeout trigger redelivery:

await queue.consume({ topic: 'webhooks', name: 'webhook-worker', group: 'workers' }, async (job) => {
try {
  await callWebhook(job.message);
  await job.ack();
} catch (error) {
  if (isTransientError(error)) {
    // Don't ack - will retry after timeout
    console.log('Transient error, will retry:', error.message);
    return;
  }
  // Permanent failure - ack to prevent infinite retries
  await job.ack();
  await logFailure(job.message, error);
}
});

Negative Acknowledgment (Retry with Delay)

Use job.nack() to explicitly request redelivery with a delay:

await queue.consume({ topic: 'tasks', name: 'task-worker', group: 'workers' }, async (job) => {
try {
  await processTask(job.message);
  await job.ack();
} catch (error) {
  // Retry after 5 seconds
  await job.nack(5000);
}
});

Use Cases

Use Case	Why Queues?
Email sending	Each email should send exactly once
Image/video processing	Long-running, distribute across workers
Payment processing	Must not double-charge; needs acknowledgment
Webhook delivery	Retry on failure, guaranteed delivery
Data pipeline stages	Reliable handoff between processing steps

Common Mistakes

Assuming exactly-once execution

Jobs may be delivered more than once if workers crash or timeout. Design your handlers to be idempotent.

Acknowledging too early

Acknowledging before work is complete can cause silent data loss. Always acknowledge after successful processing.

Using queues for real-time messaging

Queues add latency for reliability. For real-time fan-out, use Messaging.

When Queues Are the Wrong Tool

Queues are not ideal for:

• Scheduled or delayed execution (no built-in scheduler)
• Multi-step workflows (use a workflow engine)
• Real-time broadcasting (use Messaging)

The Right Mental Model

A queue job may run more than once, and it's only finished when acknowledged.

Design your job handlers with that assumption, and queues will behave reliably under failure.

Need Help?

Join our Discord server, post your concern & someone from our team will help you out ✌️