Picking a queue driver: database, Redis or a managed service

Written By: BrainSoft In Backend

Almost every project eventually needs background work: sending email, resizing images, syncing with an external API, generating a report. The first version usually runs inline and nobody notices. Then a request takes four seconds because it is waiting on a third-party endpoint, and someone says we should put this on a queue. The question that follows is which queue.

Start with the database table you already have. Move to Redis when throughput or polling load becomes a real problem, and only reach for a managed service when you need durability guarantees, fan-out or operational help you cannot provide yourself. This post walks through how those three options behave in practice.

The database driver is the right default

If you are on Laravel, Rails, Django or anything with a built-in database queue backend, you can have a working queue in about ten minutes. The jobs table lives next to your other tables, so migrations, backups and local development all behave the same way. There is no second piece of infrastructure to run, no connection string to rotate, no extra service to explain to whoever is on call.

The costs are real but predictable. Every worker polls the table for new rows, so an idle worker still runs a query every few seconds. Row locking under contention can get ugly if you have dozens of workers pulling from the same table. And on MySQL or Postgres, a large backlog of failed jobs will bloat the table and slow down the queries that scan it.

For a few thousand jobs a day and a handful of workers, none of that matters. It is the option with the fewest moving parts, and fewer moving parts is worth a lot.

Redis when polling hurts

Redis-backed queues push and pop from a list or a sorted set instead of querying a table. Workers block on a pop rather than polling, so a job is picked up in milliseconds instead of up to a poll interval later. That latency difference is the main reason teams switch, and the second reason is that the polling load disappears from your primary database.

What you give up is durability. Depending on how Redis is configured, a crash can lose recent writes. If your jobs are emails and cache warmups, that is fine. If they are payment captures, it is not. You also now have a second stateful service to monitor, back up and size correctly.

QUEUE_CONNECTION=redis
REDIS_HOST=127.0.0.1
REDIS_PORT=6379

php artisan queue:work redis --tries=3 --timeout=90

One practical note: keep the queue on its own Redis database or instance, separate from your cache. A cache flush command that wipes pending jobs is a bad afternoon. If you need scheduling, delayed jobs and retries with backoff, a library like Laravel Horizon or Sidekiq gives you a dashboard and a supervisor that restarts dead workers.

Managed queues and when they earn their keep

SQS, RabbitMQ on a hosted plan, Google Pub/Sub and similar services exist because running a reliable broker is genuinely hard. They give you at-least-once delivery, dead-letter queues, visibility timeouts and horizontal scale without you tuning anything. If you are already on AWS, SQS is a few clicks and costs almost nothing at low volume.

The trade-offs are latency, cost at scale and a harder local development story. Round trips to a managed broker are slower than a local Redis pop, so very high-throughput pipelines can get expensive. Testing locally usually means either pointing at a real queue in a sandbox account or running a compatible emulator, which is one more thing that can drift from production.

Managed queues pay off when you need fan-out to multiple consumers, when jobs must survive a full region failure, or when your team simply does not want to be paged about a broker at 3am. If that describes you, the operational savings usually beat the raw cost. If it does not, you are paying for guarantees you will not use.

A decision you can make in an afternoon

Under a few thousand jobs a day, one app server, no strict latency target: use the database driver.
Job latency matters, or polling is loading your primary database: move to Redis, on its own instance.
You need dead-letter queues, cross-region durability or multi-consumer fan-out: use a managed service.
Whatever you pick, make jobs idempotent. At-least-once delivery means a job will occasionally run twice, and retries are the norm, not the exception.

Most teams I have worked with start on the database, outgrow it somewhere around the point where the jobs table crosses a few million rows, and then move to Redis. Very few need a managed broker before that. If you would rather have someone look at your current setup before you migrate, get in touch and we can talk through the specifics.

Frequently asked questions

Can I switch queue drivers later without rewriting my jobs?

Usually yes. Frameworks like Laravel and Rails abstract the driver behind a common interface, so changing a config value is often enough. The exceptions are jobs that rely on driver-specific features such as SQS message attributes or Redis sorted-set scheduling, which need small adjustments.

How do I stop the same job from running twice?

You cannot prevent it entirely with any at-least-once system. Instead, make the job idempotent: check whether the work was already done before doing it, use unique keys on writes, and wrap multi-step operations in a transaction or a lock keyed on the job payload.

Is a Redis queue safe if the server restarts?

It depends on persistence settings. With append-only file enabled and fsync set to every second, you lose at most about a second of writes. With persistence off, you lose whatever was in memory. For anything financially or legally significant, use a broker with stronger durability guarantees.

#Backend