What is a job queue and why does your application need one?

Published on 2026-10-20 by frfrand

Qué es una cola de tareas, cómo trabajan los workers, reintentos, idempotencia y qué hacer con trabajos fallidos.

A job queue allows storing pending tasks so that one or more processes can execute them without forcing the user to wait while they are done. That is why it appears behind emails, reports, videos, imports, and automations.

Imagine someone clicks “Generate monthly report” and the application takes three minutes to finish. Keeping the browser waiting all that time would be uncomfortable and fragile. A queue offers a better way to organize that work.

The comparison with a kitchen

In a restaurant, orders arrive, are written down, and then the cooks prepare them. The waiter does not stay by the oven waiting for each dish: they can continue attending to other tables.

In an application, a web server can register a job in a queue and respond immediately: “We received your request.” Then a worker takes the task, processes it, and records whether it finished, failed, or needs a new attempt.

The queue separates the moment of requesting something from the moment it is actually completed.

How is a queue-based system organized?

Usually there are three parts:

Producer: the application that creates the task. For example, an API that receives the request to generate a PDF.

Queue or broker: the mechanism that keeps jobs available for its consumers, according to its delivery and persistence rules.

Consumer or worker: the process that takes jobs and executes the operations.

A result can also be saved in a database so that the user can check progress. Not all queues include result storage or tracking interfaces.

Real example: sending a thousand emails

Suppose a store needs to notify a thousand customers about a change in hours. It would not make sense for whoever clicked "Send" to be stuck with the page blocked until the last email.

The application can create sending jobs and process them with workers. This also makes it possible to limit how many messages go out simultaneously and how to react when the email provider returns a temporary error.

But there is a detail: the fact that a task is queued does not guarantee that the email reaches the recipient. You have to distinguish between accepted work, attempted sending, and confirmed delivery.

What happens when a worker fails?

A task can fail because a connection dropped, a file does not exist, or an external service is offline.

Some systems allow retrying the job, waiting an interval, or sending it to a failed queue. It is not advisable to retry infinitely without control: you could repeat an operation that always fails and consume all resources.

In addition, a worker can complete an action but fail before recording that it finished. If the system runs that task again, you could repeat a charge or send a notification twice.

The importance of idempotency

An idempotent operation is designed so that repeating it does not cause additional unwanted effects relative to its final result. For tasks such as "confirm this payment" or "register this reservation," it is essential to avoid duplicates.

You can use unique identifiers and records of processed operations. It is also advisable to design the database to prevent inconsistent states, rather than relying solely on queue behavior.

A system promising "at least once" delivery precisely means that repeated executions may exist. That guarantee is not the same as exactly-once execution of the entire business logic.

Is Redis useful as a queue?

Redis can be part of messaging and queue architectures; tools like Celery and other systems can use suitable brokers according to their configuration.

But a cache and a queue are not the same, even if they sometimes use the same software. A cache entry can be lost with no further consequences than repeating a query; a lost billing task can be much more serious.

When tasks are important, review durability, acknowledgments, retries, monitoring, and recovery.

When do you not need a queue?

If an operation takes a few milliseconds, completes within the request, and has no special requirements, incorporating multiple workers and a broker can be unnecessary complexity.

Before adding another piece of infrastructure, measure how long the job takes, what happens when it fails, and how many tasks arrive at the same time.

What does this have to do with ArduMaker?

In ArduMaker, applications and services such as Redis and n8n can run within VPS. A backend deployed there can organize jobs with a queue and workers if its design requires it. But the platform does not automatically define the retry policy nor prevent duplications: that logic belongs to the project.

In one sentence: a queue allows deferring and distributing jobs. The hard part is not putting tasks on hold, but knowing what to do when they repeat, fail, or disappear.

Sources: Celery Guide · Redis Streams Documentation · What is a worker in ArduMaker.