Add Worker/Queues to process notification based on status

[SerpentBytes]

Description

This PR resolves #113, #115.

What's working?

• Worker/Queue pair to run checks every 5 mins for DNS record and certs expiring in less than a month
• Send Notification once.

Steps to Test / Re-Test

• Delete redis data and restart the Redis container (clear any previous jobs)
• npm run setup (to update to DB)
• Run the project (make sure Redis is up and running)
• npm run db:studio (to see the lastNotified field getting updated)
• Navigate to the MailHog interface at http://localhost:8025 to see notifications after 5 mins (see Note)

Note
To test if the jobs are repeating without waiting for 5 mins, modify the value of repeat here to every 5 seconds.

[humphd]

I read through this, and I'm going to suggest that we don't need a dedicated worker/queue for this. When we build the flow for creating/updating/deleting DNS, we're going to need something very similar to what @dadolhay has done for certs (i.e., a series of jobs that flow in order). Part of those jobs will be responding to the state changes, and sending a notification can be part of that.

If we run this query as a repeating job, we'll end-up spamming people with notifications every day. I don't think this will be a good experience, and it would be better to do it in a controlled way, based on a user action (e.g., user clicks Update or Delete etc and we start the flow).

Having said that, you have the basis here (and knowledge) for another feature we need, namely, running a query every day to expire domains, delete expired certs, send notifications of upcoming expirations, etc.

Would you be willing to evolve this code to focus on that? Maybe start with sending a notification if a domain or cert will expire in 1 month?

[humphd]

Filed #270 for the DNS flow I'm describing above.

[ghost]

I agree with what @humphd proposed, with the exception, that this should not only run once a day, it should run every hour or even every 5 minutes.

If you allow tasks to pile up throughout the day, it will have to send a lot of mails all at once, and it is a lot easier to hit a rate limit + 3rd party providers do not really like large spike-like loads. I'd suggest to run it often, and only handle the tasks that are due, that represents a lot more even load on the system.

[humphd]

I agree with what @humphd proposed, with the exception, that this should not only run once a day, it should run every hour or even every 5 minutes.

If you allow tasks to pile up throughout the day, it will have to send a lot of mails all at once, and it is a lot easier to hit a rate limit + 3rd party providers do not really like large spike-like loads. I'd suggest to run it often, and only handle the tasks that are due, that represents a lot more even load on the system.

Sounds good, and something we can do load testing on later, then measure and tune.

[humphd]

One thing to call out with repeat jobs: we're going to need to not spam users. For example, if you run this every 5 mins (or every day), we need some way to keep track of the fact that we've already notified a user and don't need to do it again. There are various ways to approach this, and we'll need one of them.

[SerpentBytes]

I have modified the code to check for expiring records and certificates every 5 mins and persist if the user is notified. At the moment the type is boolean, but we can replace it with a number value so we can send two reminders.

[humphd]

Let's move this to 0.5. I'm not going to be able to review it fully in time for shipping today.

[humphd]

Looks like there's a conflict to sort out still.

[SerpentBytes]

It Looks like there's a conflict to sort out still.

Yes. I am just waiting for a few more reviews.

[ghost]

Nice work!

[humphd]

Looks good to me.