-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
weather-mv
: Add columns to Schema for faster geolocation queries
#49
Comments
Hi @alxmrs, I plan on working on this but I am still in the process of trying to understand the code. |
This sounds right to me. I recommend looking at the logic here (https:/google/weather-tools/blob/main/weather_mv/loader_pipeline/pipeline.py#L370). I think the column should be added around this area, and to the associated functions. Check out this BQ documentation, too: https://cloud.google.com/bigquery/docs/geospatial-data
Yes, I think this is a great starting point. If you'd like additional help, maybe we can set up a 1:1 meeting.
Sure thing. The general flow will look like:
How does all that sound? |
Heads up: The files changed around a lot in #101. |
A suggestion from @lakshmanok:
ticket updated: 2022-03-24
Acceptance Criteria
weather-mv
can make use of the Geography capabilities of BigQuery.GEOGRAPHY
, which uses "point geography" from the raw data's lat/lng data.Implementation Notes
GEOGRAPHY
type. Namely,ST_GEOPOINT
is a function that can take lat/lng FLOAT64 columns and create aGEOGRAPHY
Point value from it.weather-tools/weather_mv/loader_pipeline/bq.py
Line 70 in fd0c5e4
weather-tools/weather_mv/loader_pipeline/bq.py
Line 173 in fd0c5e4
Extended feature to delight users
Implementing extra columns to include the polygon of the (grid) area where the values are relevant is a great bonus feature. However, it will not be possible for all types of input data. This experience is only possible if the XArray DataFrame includes extra information, like a coordinate or variable attribute.
For now, while we implement the first part of this ticket, let's also investigate if such metadata exists on our happy-path data sources. If it does, we'll create a follow up ticket to implement this.
Acceptance Criteria
The text was updated successfully, but these errors were encountered: