feat: Dns reconciler queue

[ghost]

• Moves the complete reconciler to under queues
• Creates a queue for it with automatic repeat. It makes certain that only 1 repeat job can exist
• Adds a new table, Uses a unique enum so only ever 1 row can exist in the table
• Alters the reconciler main code, so that it only runs if there is no concurrency problem (because of multiple instances in the swarm), and it is requested to run
• Introduces 'limp mode'. If bulk update fails, it will try to run it one by one, and it points the offending update out in the logs

Closes #396

[humphd]

This is great. I love normal vs. limp mode! Lots of great problem solving and interesting patterns in here.

I've left a bunch of questions and comments.

[ghost]

@humphd Thank you! Updates pushed, and I'll ask for some help with the e2e tests.
My only feeling is that it might be failing because of the updated database schema, but the video shows it standing on the main screen (before login) not doing anything.

No rush, we can talk about it tomorrow! Again, thanks for the very quick reviews!

[Genne23v]

The code is well organized and comments are easy to understand. I added only one comment. Feel free to update or keep it as is.

[humphd]

Debugging this locally, the following change gets the e2e tests to pass for me locally:

diff --git a/app/root.tsx b/app/root.tsx
index a4a8f08..89550eb 100644
--- a/app/root.tsx
+++ b/app/root.tsx
@@ -12,7 +12,6 @@ import {

 import { getUser } from './session.server';
 import theme from './theme';
-import { addReconcilerJob } from './queues/reconciler/reconciler-queue.server';

 import type { MetaFunction, LoaderArgs, LinksFunction } from '@remix-run/node';

@@ -37,8 +36,6 @@ export async function loader({ request, context }: LoaderArgs) {
   });
 }

-addReconcilerJob();
-
 function Document({ children }: { children: React.ReactNode }) {
   // We need the nonce generated by the server via helmetjs to put on scripts
   const { nonce } = useLoaderData<typeof loader>();
diff --git a/server.ts b/server.ts
index a604867..7765fd0 100644
--- a/server.ts
+++ b/server.ts
@@ -8,6 +8,7 @@ import helmet from 'helmet';
 import cors from 'cors';

 import logger from '~/lib/logger.server';
+import { addReconcilerJob } from '~/queues/reconciler/reconciler-queue.server';

 import type { Request, Response } from 'express';

@@ -81,6 +82,8 @@ const port = process.env.PORT || 8080;
 const server = app.listen(port, () => {
   // require the built app so we're ready when the first request comes in
   require(BUILD_DIR);
+  // start the DNS reconciler
+  addReconcilerJob();
   logger.info(`✅ app ready: http://localhost:${port}`);
 });

[ghost]

@humphd Thanks for the suggestion! It seems to have solved the issue!

[ghost]

@humphd When you are around, can I get an approve if all is well?

[humphd]

🚢 ship it!