v0.2.1

Improvements and bugfixes for all weather tools. weather-dl is much faster & more robust. weather-mv now uses a pluggable infrastructure, which makes iterations faster. weather-sp is mid transition to arbitrary splits.

Thanks to our new OSS contributors, @pranay101 and @pbattaglia!

Current Status

weather-dl: Major fixes

• This release introduces a fix to #98, which makes the downloader faster and more robust. With this change, there is no need to override the autoscaling algorithm – so, it now has less moving parts.
• Uploads have better retry logic. Users should experience less crashes from network errors in the pipeline.
• The downloader structure has been refactored to be more testable.
• Examples of JSON configs were added.
• Address a critical NameError bug that occurred during a refactor.

weather-mv: Refactor

• The mover has been refactored to use a pluggable infrastructure. This makes it easier to develop local runs, and to write weather data to other sources besides BQ.

weather-sp: Skipping logic

• A non-API changing feature has been added to the splitter: now, already splitted data will be skipped. Users can override this feature with -f,--force.
• The documentation for the splitter has been improved.

General

The release process now produces smaller binaries (we're now ignoring test data).

What's Changed

• Added example JSON config by @pranay101 in #52
• Refactored weather-mv to work with pluggable Data Sinks. by @alxmrs in #101
• Update weather-sp's templating system to allow users to specify level and shortname. by @alxmrs in #105
• Upload to cloud is robust to socket timeout errors. by @alxmrs in #110
• Fix wrong output file example in weather-sp readme. by @uhager in #111
• Added skipping logic to weather-sp by @alxmrs in #108
• Lower default num-requests for MARS to make it more robust. by @alxmrs in #113
• New data-oriented task distribution strategy. by @alxmrs in #116
• Downloader refactor: extracted out partitioning; tested pipeline args. by @alxmrs in #117
• Fix minor bug: main session needs to be saved. by @alxmrs in #120
• (#123) Fixed beam not being able to access global namespace + minor related bug. by @pbattaglia in #124
• Shrinking the size of the package release artifacts by @alxmrs in #122

New Contributors

• @pranay101 made their first contribution in #52
• @pbattaglia made their first contribution in #124

Full Changelog: v0.2.0...v0.2.1