Why Run Multiple Resques?

You might think it's weird to run more than one Resque and so why do you even need an app like this? This article will explain why.

Resque Failure Modes

There are three general ways that your Resque instance can become unavailable:

• Redis or the machine it's on dies/goes down
• Redis fills up
• All connections in use, no new connections can be established

The questions to ask when managing your technical infrastructure would then be:

• How likely are each of these failure modes?
• What is the impact of these failures?

The answers to these depend on many things. If you are using Redis-as-a-Service, the chance of all three of these happening is good. Free or cheap Redis instances limit the number of connections and storage available. Hosting over "the cloud" carries inherent flakiness that can make your Redis appear unavaiable.

Given this, what is the impact of these failures? Two things happen:

• You cannot process existing jobs
• You cannot queue new jobs

While the inability to process jobs could be tolerated for a time (we are backgrounding things for a reason), the inability to queue new jobs could be catastrophic.

With all this in mind, let's talk about how this impacts various technical architectures

Technical architectures

If all of your code is in one monolithic Rails app, you have bigger problems and using multiple Resques will not solve them. You can move along now.

If, on the other hand, you deploy multiple single purpose apps and services, you want to avoid single points of failure.

Consider a two-application setup, where your public website (www) serves your customers and an admin console (admin) is used by your Customer Service and Support team. Suppose that both systems require background jobs.

Now, let's suppose they both share the same Resque instance. You now have a situation where your admin server, if it were to create a large influx of jobs, can take down your public website.

This is what's called Not Good™.

Suppose, instead, that you ran two separate Resque instances. The influx of jobs from your admin service would still take the admin service down, but your public website would be unaffected.

That's Why You Always Run Multiple Resques

If you are running more than one application that uses Resque, give each app its own Resque (and Redis). This insulates your applications from Resque failures introduced by others.

The problem, however, is that the included resque-web can only show you one resque. Typically, you mount it inside your Rails app, or deploy it for each Resque. Resque-Brain shows them all. That's its purpose.