DNS reconciler

[ghost]

Our own internal DB should be the "primary source of truth".

We should

• Load all records from DB
• Load all records from route53
• Pair each db record with route53 record on exact match only (a record has to be identified by type, subdomain and content, as there may be multiple TXT records with the same subdomain but with different content, just to bring an example)
• loop through all the pairs:
• If route53 part is missing => create it
• if db part missing => delete record in route53
• if both parts of the pair is present => nothing to do
• Collect all reauired changes take the first 1000 elements from the list
• send the changes to route53

---

We would also need to implement some kind of throttling for this that mimics lodash (but qith queues)
https://lodash.com/docs/#throttle
where options = {leading: true, trailing: true}

[ghost]

This would also modify our state handling for records
We sould only have "pending" and "synced" statuses
When a record is created or altered => set it to pending
When the reconciler has finished with a specific record => set it to "synced"
When a record is deleted => delete it from the db directly, reconciler will take care of route53 the next time

This would also completely take care of the "cannot edit while being created/updated/deleted" problem

[humphd]

According to the AWS docs for ListResourceRecordSet, we can get up to 300 resource record sets at a time. The docs go on:

Specifying where to start listing records

You can use the name and type elements to specify the resource record set that the list begins with:

If you do not specify Name or Type
The results begin with the first resource record set that the hosted zone contains.

If you specify Name but not Type
The results begin with the first resource record set in the list whose name is greater than or equal to Name.

If you specify Type but not Name
Amazon Route 53 returns the InvalidInput error.

If you specify both Name and Type
The results begin with the first resource record set in the list whose name is greater than or equal to Name, and whose type is greater than or equal to Type.

So we "page" our way through > 300 records. We might want to build some test code that creates 300+ in moto.

Also, the docs talk about PENDING records being returned too, so we'll have to be aware of this.

Next, we have to be careful not to issue ChangeResourceRecordSets commands while we are paging the results:

To ensure that you get an accurate listing of the resource record sets for a hosted zone at a point in time, do not submit a ChangeResourceRecordSets request while you're paging through the results of a ListResourceRecordSets request. If you do, some pages may display results without the latest changes while other pages display results with the latest changes.

And in terms of how to do this:

If a ListResourceRecordSets command returns more than one page of results, the value of IsTruncated is true. To display the next page of results, get the values of NextRecordName, NextRecordType, and NextRecordIdentifier (if any) from the response. Then submit another ListResourceRecordSets request, and specify those values for StartRecordName, StartRecordType, and StartRecordIdentifier.

It sounds like this won't be trivial to do properly. But I agree that it will give us confidence that our ground-truth isn't drifting apart from our DNS setup (something that is already happening on staging).

We should come up with a plan for doing this in parallel to our current implementation so we don't lose any of our current velocity. Later we can refactor over to using this on the whole.