Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate data from Filesystem to Postgres #81

Open
5 of 11 tasks
nelsonic opened this issue Jun 1, 2019 · 7 comments
Open
5 of 11 tasks

Migrate data from Filesystem to Postgres #81

nelsonic opened this issue Jun 1, 2019 · 7 comments

Comments

@nelsonic
Copy link
Member

nelsonic commented Jun 1, 2019

At present the Node.js MVP saves data to the instance filesystem because that was the simplest way of storing data without having to manage any database.
see: hits-nodejs/lib/db_filesystem.js#L26-L68

This worked well for MVP as it streams the contents of the file each time a request is made and counts the lines in the file as the count. Node.js shines at this because fs.createReadStream is non-blocking.
Anyway, with the migration to phoenix, we need to:

  • write an Elixir Script that parses the data in the filesystem and inserts it into PostgreSQL

Tasks

I estimate that this will take me T4h because I have to:

  • Log-in to the Linode instance and inspect how many directories have been created
    (this is the number of people - GitHub usernames - using the hits badge)
  • zip the data on Linode instance to get it off the instance - because I will be running the Elixir script on my localhost.
  • download the data to localhost
  • unzip data on localhost
  • Download and install Postgres on Mac: https://postgresapp.com/downloads.html
  • write the script
    • insert data into hits_dev PostgreSQL on localhost
  • run the script
  • export the data from hits_dev PostgreSQL on localhost
  • load the data on the remote server
@nelsonic
Copy link
Member Author

nelsonic commented Jun 2, 2019

While attempting to run the tar command:

tar -zcvf logs.tar.gz logs/

as per
https://unix.stackexchange.com/questions/93139/can-i-zip-an-entire-folder-using-gzip
we received the following error:

tar: logs: file changed as we read it

Now deleting the .gz files recursively:

find . -name "*.gz" -type f -delete

https://askubuntu.com/questions/377438/how-can-i-recursively-delete-all-files-of-a-specific-extension-in-the-current-di

Temporarily stopping the node.js server in order to perform the archive.

forever stop server.js

Now running:

tar -zcvf logs.tar.gz logs/

And it's taking forever ... ⏳

@nelsonic
Copy link
Member Author

nelsonic commented Jun 2, 2019

https://stackoverflow.com/questions/9427553/how-to-download-a-file-from-server-using-ssh
Format:

scp [email protected]:foobar.txt /local/dir

Actual:

scp [email protected]:hits/logs.tar.gz ./logs.tar.gz

Worked:
image

@nelsonic
Copy link
Member Author

nelsonic commented Jun 2, 2019

@nelsonic
Copy link
Member Author

nelsonic commented Jun 2, 2019

@nelsonic
Copy link
Member Author

nelsonic commented Jun 5, 2019

Roadblock: dwyl/learn-devops#55

@nelsonic
Copy link
Member Author

https://serverfault.com/questions/264595/can-scp-copy-directories-recursively

scp -rp [email protected]:hits/logs ./
scp -rp [email protected]:hits/logs ./

in-progress:
Screen Shot 2021-05-23 at 12 02 15 AM

heading to bed let's see if it works overnight.

@nelsonic
Copy link
Member Author

This is why we can't have nice things:
Screen Shot 2021-05-23 at 6 18 55 AM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant