Attempt to Restore from Linode Backup

[nelsonic]

Zipping the data as part of dwyl/hits#81 appears to have deleted some data ... dwyl/hits#83 😞
Thankfully I'm paying for backups on Linode so now it's time to test the process!

[nelsonic]

https://manager.linode.com/linodes/backups/linode3429883

https://manager.linode.com/linodes/backup_details/linode3429883?id=119009843

Add a new linode for the backup: https://manager.linode.com/linodes/add

Wait ...

Brand new:

Try the restore again:

Modal ...

Wait ...

Keep waiting ...

Go for a walk outside ...

[nelsonic]

Failed:

[nelsonic]

Trying again from an earlier backup.

[nelsonic]

Failed:

[nelsonic]

Opened support ticket:

Ticket: https://manager.linode.com/support/ticket/11990994

[nelsonic]

Failed again: 😞

[nelsonic]

Attempting to restore from the oldest available backup:

Failed:

Restore failed, 283556 troubled files DEBUG: {"symlnk":1056,"zero_length":230,"create":282267,"sparse":3} 

[nelsonic]

Unable to boot the instance:

Network Helper did not run: could not determine distribution or distribution version  

Which means we cannot actually run the instance.
This is a horrible UX because it says "Success" but then we cannot actually access the instance!

[nelsonic]

22h later still no acknowledgement on the Linode Support Ticket ... ⏳ 😞
https://manager.linode.com/support/ticket/11990994

[nelsonic]

Got reply from Chris @ Linode Support:

Attempting to restore again so that they can help me.
(I deleted the failed instances because they were useless to me ...)

[nelsonic]

This is further than it's got in previous attempts:

Still failed:

Replied to Chris:

[nelsonic]

Got a reply from Tim at Linode:

Sadly, the restore Tim attempted did also failed with the same error:

Waiting for another reply ...

[nelsonic]

Latest reply from Linode Support:

Going to try and boot the image now ... 🤞

[nelsonic]

https://manager.linode.com/linodes/dashboard/linode14305623

Running:

ssh [email protected]

Looks like there's a lot more data on the recovered instance.
going to attempt to run the app.

OK. that appears to be all working. Let's get back to zipping up the data ... dwyl/hits#81

[nelsonic]

[nelsonic]

Lowering priority as the data was restored from backup. ✅

[nelsonic]

Next:

• log back in to restored server: ssh [email protected] and download old logs
• log in to production server ssh [email protected] and download latest logs
• attempt to insert text (ALF) log into PostgreSQL using script on localhost
• investigate simplified Elixir (Phoenix) deployment: Effortless Elixir releases and deployment learn-elixir#129
• decide if we are going to continue using Linode or switch to AWS for PSQL-as-a-Service?
• Deploy to chosen environment
• Update LetsEncrypt to autorenew if possible
• Update DNS to point to Phoenix server.
• Monitor the service!

[nelsonic]

Considering using Docker to run Phoenix and PostgreSQL ... dwyl/learn-docker#15 (comment)
Reading: https://pspdfkit.com/blog/2018/how-to-run-your-phoenix-application-with-docker/

[nelsonic]

Continue: https://cloud.linode.com/dashboard

[nelsonic]

https://cloud.linode.com/linodes/3429883/summary

[nelsonic]

[nelsonic]

du -S . | sort -nr | head -25

what's up with the ridiculous 8GB forever log? 🤷‍♂️

Let's clear them: https://stackoverflow.com/questions/31721860/how-to-clean-foreverjs-logs-for-running-processes

forever cleanlogs

That gives the instance a bit more breathing room. But still going to delete it!

[nelsonic]

https://devconnected.com/how-to-count-files-in-directory-on-linux

ls | wc -l