Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove 0 / no-data value for a Google Drive export for areas outside of of the export geometry's bounding box #56

Closed
2 tasks
davidoesch opened this issue Feb 22, 2024 · 4 comments
Assignees
Labels
enhancement New feature or request good first issue Good for newcomers
Milestone

Comments

@davidoesch
Copy link
Collaborator

davidoesch commented Feb 22, 2024

Is your feature request related to a problem? Please describe.
When exporting the image, I use the bounding box of the geometry.

The no-data values within the bounding box are correct 9999) but I cannot figure out how to set the no-data value of the pixels outside of the bounding box; these seem to default to 0.

image

Describe the solution you'd like

@davidoesch davidoesch added the enhancement New feature or request label Feb 22, 2024
@davidoesch davidoesch added the good first issue Good for newcomers label Apr 11, 2024
@davidoesch davidoesch added this to the v1.0.0 milestone Apr 11, 2024
@Tschoun Tschoun closed this as completed May 6, 2024
@davidoesch
Copy link
Collaborator Author

it is not fixed yet

@davidoesch davidoesch reopened this May 15, 2024
@Tschoun Tschoun self-assigned this Jun 24, 2024
Tschoun added a commit that referenced this issue Jun 25, 2024
#56
- outside of the swiss perimeter is nan, inside, but in the swiss perimeter but outside the flightpath is zero.
- line 605 in step0_processor_s2_sr.py is causing issues.
@davidoesch
Copy link
Collaborator Author

davidoesch commented Jul 11, 2024

needs to be fixed in the satromo_publisher step at

as seprate util function?, since in the GEE assets it is already masked correctly
The approach is: to get from metafile the asset ID the get the masked /non masked area

``
asset_id = 'projects/satromo-int/assets/COL_S2_SR_HARMONIZED_SWISS/S2-L2A_mosaic_2024-07-05T104021_bands-10m'

Load the asset.

asset = ee.Image(asset_id)

Select the first band of the image and convert it to an integral image.

first_band = asset.select(0).toInt()

Create a binary mask.

binary_mask = first_band.mask()

Invert the mask to get the non-masked areas.

non_masked = binary_mask.Not()

Convert the non-masked areas to vectors.

vectors = non_masked.reduceToVectors(geometryType='polygon', maxPixels=1e9, bestEffort=True)

Convert the vectors to GeoJSON.

geojson = vectors.getInfo()

Extract the features from GeoJSON

features = geojson['features']

Extract geometries and properties

geometries = [shape(feature['geometry']) for feature in features]
properties = [feature['properties'] for feature in features]

Create a GeoDataFrame

gdf = gpd.GeoDataFrame(properties, geometry=geometries)

Define output shapefile path

output_shapefile = 'output.shp'

Define the CRS

crs = {'init': 'epsg:2056'}

Export to Shapefile with specified CRS

gdf.to_file(output_shapefile, driver='ESRI Shapefile', crs=crs)

print(f"Shapefile exported successfully to {output_shapefile}")

``

  • then cast this as config.BUFFER. / use it a config.BUFFER
  • Make sur that you check if output.shp exists so it does not need to be claulcated everytime

@davidoesch davidoesch assigned davidoesch and unassigned Tschoun Jul 11, 2024
@davidoesch
Copy link
Collaborator Author

We solve thsi differently: We precompute for each orbit a buffer https://code.earthengine.google.com/279232b086bcfb5d456fe26eda6b09c6
the apply this buffer when creating the mosaic in satromo_publisher: Advantage: we save EECU hours, and we alawys have the orginal data masked in GEE assets. Disadvantage: with orbit shifts we will miss some tiles

davidoesch added a commit that referenced this issue Jul 15, 2024
@davidoesch
Copy link
Collaborator Author

fixed on dev

davidoesch added a commit that referenced this issue Jul 17, 2024
* catch gdrive errors, remove backup on S3 INT

-fix for #86
-removed the backup on S3 INT option with rclone

* Functional util_addproperties.py

Script to add properties to existing GEE assets.
#79

* added MSG LST step0

feat: Add processing pipeline for daily LandSurface mosaics over Switzerland with step0_processor_msg_lst.py

- Introduced a new script to access and upload LandSurface (LST) data over Switzerland to Google Earth Engine (GEE).
- Utilized CMS SAF data provided by MeteoSwiss via s3 on data.geo.admin.ch to store as SATROMO assets and to calculate VCI, TCI, and VHI.
- Added functionality to retrieve and process NetCDF files, extracting relevant attributes and dimensions.
- Implemented the export of NetCDF bands to GeoTIFF format and scaling operations using Rasterio.
- Added functions to generate LST mosaics for specific dates and handle the upload process to Google Cloud Storage and GEE.
- Ensured proper error handling and cleanup of temporary files and datasets.
- added in dev_config a new PRODUCT_MSG which configs the LST
- moved util_addproperties to main_functions

* updated requirements.txt for MSG LST

-Update dependencies in requirements.txt for LST METEOSWISS processor
- removed unnecessary mount point to google cloud

* added MSG LST CLIMA

MSG LST  and MSG LST CLIMA can be calculated with the step0 function as the otehr products. They won't have a step1. This eases reprocessing
-satromo processor: added PRODUCT_MSG_CLIMA and PRODUCT_MSG exit points
-dev_config: added RODUCT_MSG_CLIMA and PRODUCT_MSG

 new
-step0_processor_msg_lst.py
-step0_processor_msg_lst_clima.py

* added command line option for date

we can now reprocess a specific date from command line 'python satromo_processor.py prod_config.py 2024-06-12', the config file is a must when us with date option

* first steps to get rid of the black part in the visualisation

#56
- outside of the swiss perimeter is nan, inside, but in the swiss perimeter but outside the flightpath is zero.
- line 605 in step0_processor_s2_sr.py is causing issues.

* Added satromo_processor.drawio.svg

* Update satromo_processor.drawio

* Update README.md

Added Description for Single Scene Processing via Command Line

* Removes black border

Fix for  #56

* Moved ch_buffer_5000m to assets

* moved ch_buffer_5000m.shp to assets path

* Thumbnail

Thumbnail:
        Fixed aspect ratio for VHI and S2 SR.
        Updated background to accommodate new S2 SR tiles cut to the effective path.
        Changed no data values accordingly.

util_listassets.py:
        Added functionality to list all assets of a collection.

util_moveassets_terrainshadow.py:
        Script to initialize Google Earth Engine (GEE) and Google Drive authentication.
        Converts Day-Of-Year (DOY) to datetime strings.
        Manages the copying and updating of assets within Google Earth Engine.

satromo_publish.py:
        Added detection for VHI cases to use the Swiss buffer.

step0_processor_s2_sr.py:
        Replaced swisswalti with swisssurface.
        Set terrainShadowDetectionPrecalculated to True to use pre-calculated shadows.

step1_processor_vhi.py:
        Updated to use T235959 time format instead of 240000 as per definitions with BAFU and MeteoSwiss.

* Update requirements.txt

set certifi to latest

* Update step0_processor_s2_sr.py

fix missing slash at the end of the shadow terrain collection

* Prepared for deploy to prod

---------

Co-authored-by: Joan Sturm <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

2 participants