Very large amount of requests made to Redis

[migbash]

I have a single queue that I have been testing with Bull and upon deploying the script I checked my Redis usage after the job was completed and it just kept going up and up.

• What is the reason for so many Redis - Queue requests ?
• How can I reduce the number of such requests and what parameters are needed ?

[manast]

Do you have some code (simple) that exhibits this issue?

[migbash]

@manast yes! Here you go:

const Queue = require('bull');

// ~~~~~~~~~~~~~~~~~~~~~~~~
// [❗] BULL CRITICAL
// ~~~~~~~~~~~~~~~~~~~~~~~~

const settings = {
  stalledInterval: 600000,   // NOTE: 10 min. : (milliseconds) How often check for stalled jobs (use 0 for never checking)
  guardInterval: 5000,       // NOTE: (milliseconds) Poll interval for delayed jobs and added jobs.
  drainDelay: 300            // NOTE: (milliseconds) A timeout for when the queue is in drained state (empty waiting for jobs).
}
const cacheQueueFeaturedMatch = new Queue (
  'cacheQueueFeaturedMatch', 
  { 
    redis: { 
      port: "", // <UPSTASH-PORT>, 
      host: "", // "<UPSTASH-HOIST>", 
      password: "", // "<UPSTASH-PASSWORD", 
      tls: {} // enabled with TLS </UPSTASH-HOIST>
    },
    settings: settings
  }
);
const job_settings = {
  timeout: 300000,            // NOTE: 5 min. : The number of milliseconds after which the job should be fail with a timeout error [optional]
  removeOnComplete: 50        // NOTE: If true, removes the job when it successfully
                              // completes. A number specified the amount of jobs to keep. Default behavior is to keep the job in the completed set.
}
const cacheQueueProcessName = "cacheQueueFeaturedMatch"
const cacheTarget = "REDIS CACHE | featured match"
let logs = []

// ~~~~~~~~~~~~~~~~~~~~~~~~
//  [MAIN] ENDPOINT METHOD
// ~~~~~~~~~~~~~~~~~~~~~~~~

async function main() {

  // [ℹ] producers [JOBS]
  const job = await cacheQueueFeaturedMatch.add({}, job_settings);
  console.log(`${cacheQueueProcessName} -> job_id: ${job.id}`)

}

// ~~~~~~~~~~~~~~~~~~~~~~~~
//  [MAIN] BULL WORKERS 
// ~~~~~~~~~~~~~~~~~~~~~~~~

cacheQueueFeaturedMatch.process (async function (job, done) {
  // console.log(job.data.argumentList);
  // console.log(job.data)

  logs = []
  logs.push(`${job.id}`);

  /* 
  do stuff
  */

  const t0 = performance.now();
  const t1 = performance.now();

  done(null, { logs: logs });

}).catch(err => {
  console.log(err)
});

main()

---

Just doing node script.js after installing npm install bull the REDIS requests just rise and rise in insane pace and levels.

Doing about 3000 request in 2 hours approx. with 3 connections active:

All while sitting idle and after running the node script.js

There must be a way or configuration that I am missing to get the absurd request-size under-control.

[manast]

I run the script using a local Redis instance and nothing special happens. However I see 2 major issues in your code, 1. it seems like you are using stash. Stash is not compatible with Bull. 2. You are using a processor function with async AND done, you must use either async or done, not both.

[migbash]

I see... I believe I was misguided in terms of the use of Upstash as a bull-queue as a Redis use, didn't know. Are there any other Redis Services like Upstash that also do not work ?

I will try with another Redis service and see what the usage is like.

The reason I had async inside the:

(async function (job, done) {

is because I had some methods I call inside the .process section do stuff that I used await for before calling done().

If I use async should I use the Promise based completion ?

[manast]

If you use async do not use "done", it is not needed.
Upstash is the only service that is not compatible with Bull. Upstash is not really Redis, it is a custom implementation.

[manast]

Btw, if you are new to Bull, better to jump to BullMQ and use typescript for much better ergonomics...

[migbash]

Hi @manast!

I have switched over to BullMQ and have connected and setup a Docker-Redis local image and I still see the same exact issue with the number of requests being done literally once every second almost:

When investigating with MONITOR command for REDIS

---

Is there a way to reduce the number of requests made from the Worker / Queue to Redis ?

It's too much.

My idea and use case is to just not have (or very little amount) of requests made to Redis once a Job is picked up by the Worker and is completed. But it appears the request continue after job completion and are a lot.

Also I see there are no AdvancedSettings for BullMQ - Queue ?! Like there is in Bull V3 ?

[manast]

I am not sure what you mean by "too many requests". The requests performed are necessary. The stalled check script is run once every 30 seconds by default.
Here you can find all the worker options: https://api.docs.bullmq.io/interfaces/WorkerOptions.html

[migbash]

Ok, I understand.

But where is the timeout setting from the Bull V3 ?
Cannot find it on BullMQ to terminate a job after X milliseconds ?

[nicolas-besnard]

Hello!

I just started using BullMQ and got the exact same issue. I have 1 worker with concurrency: 3, and between 00am and 9am, Redis received 5K requests. I seems quite a lot and will make it really expensive on Upstash for example for a very low usage.

I'm also afraid of the cost once you start scaling to more Workers and using Upstash

[manast]

Currently using BullMQ with Upstash is not very cost effective, but I know this is something they are working on to improve.

[matt-koevort]

Hey @manast - do you know if Upstash are tracking that somewhere?

[nara-golfdistrict-com]

@manast We are also running into this issue and unfortunately for us, it is running in several hundred thousands of reads and writes with just 2 jobs.

[manast]

After this PR it will be possible to use a larger drainDelay setting: taskforcesh/bullmq#2514
This will allow us to reduce the amount of calls to Upstash when the queue is idle. But keep in mind that BullMQ performs a lot of calls when processing jobs too.

[Bartek532]

Did you find out any answer for this?

[manast]

@Bartek532 did you increase the drainDelay setting as suggested above?

[stanxing]

We have same issues, bullmq makes a lot of calls to redis resulting in 100% cpu usage.

[manast]

@stanxing that may be normal depending on your workload, number of workers, and so on, you need to provide more information so that we can give you a better suggestion. In any case, have you tried to increase the drainDelay option as suggested above?