-
Notifications
You must be signed in to change notification settings - Fork 206
Process
Matt Gaunt edited this page Jan 7, 2022
·
1 revision
- Crawler.rb queries the GitHub API and writes the events to a file locally on the VM
- These files are per hour
- At 5mins past every hour, a cron job compresses the last file and uploads it to cloud storage
- At 8mins past every hour,
ruby upload.rb
is run to upload the compressed file to bigquery
The yearly tables are created by a cron job at 5am on January 1st.
?How are these created?