Add spatial resolution to the properties for easier filtering possibilities #79
Comments
|
Add this to the existing assets in projects/satromo-prod/assets/col/S2_SR_HARMONIZED_SWISS/ |
Tschoun
added a commit
that referenced
this issue
May 15, 2024
* Set up the step1_processor_vhi - changes in the dev_config file regarding additional variables for the V1 (VHI) product - renamed it from step1_processor_pv1 to step1_processor_vhi -> adjusted in the satromo_processor - deleted step1_processor_pv1 (not intuitive which product it is refering to) - set up the whole step1_processor_vhi. * Create util_castassetbands.py used to fix #73 * change color table for VHI thumbnail. * from_origin fix * update for VHI -added white bg for no data in thumbnail * step1_processor_vhi : renamed -10m export drive: renamed _10m to -10m * Update VHI timestamp to 240000 format addressed changes in #76 * added 'pixel_size_meter' to S2_SR and VHI #79 * Stop processing if data not yet complete - don't process, if necessary number of tiles for specific overpass is not yet reached (#61) - don't process is the collection for cloud probability (either CloudScore+ or S2 cloud probability) is not yet complete, i.e. doesn't have the necessary number of tiles for the specific overpass (#80) * Fixed abortion in case of incomplete tile coverage #80 - wrong number of tiles for orbit 108 corrected. - changed from if/elif to only if conditions - return after every condition that is met, not only at the end of the if/elif conditional. * Fixed datetime error in publish problems with the iso standard of YYY-MM-DDT240000, now only checking for the date, disregarding the time. * fixed iso8601 issue for publish - removed breakpoint in satromo_publish.py - main_publish_stac_fsdi.py Check if raw_item ends with "240000", since python does not recognize the newest version of ISO8601 of October 2022: "An amendment was published in October 2022 featuring minor technical clarifications and attempts to remove ambiguities in definitions. The most significant change, however, was the reintroduction of the "24:00:00" format to refer to the instant at the end of a calendar day." * Fix if raw_item ends with "240000" * update preview link to map.geo Fix for #77 * VHI thumbnail with swiss border rivers lakes * extract warnregions fix for #72 * Update requirements.txt used for the main_extract_warnregions funtion. pandas and pyarrow * removed double rasterio entry * added warnregions SHP and config * added the first version for the extraction and push * reverted the ISO8601 end of day to 23:59:59 * Delete ch.swisstopo.swisseo_vhi_v100_mosaic_2023-07-13T235959_forest-10m.tif * Delete validate_geoparquet.py * change to PRODUCT_VHI naming and add no_data propertiy to config. for more intuitive naming * Update main_extract_warnregions.py handling empty values * Added Warnregions Version 2.5.2024 Adding the official version of the warning regions as per email from Andreas Helbling (BAFU) dated 02.05.2024 and the corresponding link on SharePoint." * Delete assets/warnregionen_vhi_2056.qmd * Delete assets/warnregionen_vhi_2056.cpg * added date to Parquet export * Added warnregions BAFU_Grossregionen_v3 Adding the official version of the warning regionsbased on BAFU_Grossregionen_v3 as per email from Andreas Helbling (BAFU) dated 02.05.2024 and the corresponding link on SharePoint. * Delete assets/warnregionen_vhi_2056.xml * Set regioname to BAFU_Grossregionen_v3 * added current for warnregions * Delete ch.swisstopo.swisseo_vhi_v100_mosaic_2023-07-13T235959_forest-10m.tif * delete orphans in Processing * prepare processor and prod_config for branch merge * Prepare merge - update prod_config * Update README.md --------- Co-authored-by: David Oesch <[email protected]>
Tschoun
added a commit
that referenced
this issue
May 31, 2024
Script to add properties to existing GEE assets. #79
|
Added 'pixel_size_meter' to all swissEO S2-SR assets in the SATROMO/prod environment. |
davidoesch
added a commit
that referenced
this issue
Jul 17, 2024
* catch gdrive errors, remove backup on S3 INT -fix for #86 -removed the backup on S3 INT option with rclone * Functional util_addproperties.py Script to add properties to existing GEE assets. #79 * added MSG LST step0 feat: Add processing pipeline for daily LandSurface mosaics over Switzerland with step0_processor_msg_lst.py - Introduced a new script to access and upload LandSurface (LST) data over Switzerland to Google Earth Engine (GEE). - Utilized CMS SAF data provided by MeteoSwiss via s3 on data.geo.admin.ch to store as SATROMO assets and to calculate VCI, TCI, and VHI. - Added functionality to retrieve and process NetCDF files, extracting relevant attributes and dimensions. - Implemented the export of NetCDF bands to GeoTIFF format and scaling operations using Rasterio. - Added functions to generate LST mosaics for specific dates and handle the upload process to Google Cloud Storage and GEE. - Ensured proper error handling and cleanup of temporary files and datasets. - added in dev_config a new PRODUCT_MSG which configs the LST - moved util_addproperties to main_functions * updated requirements.txt for MSG LST -Update dependencies in requirements.txt for LST METEOSWISS processor - removed unnecessary mount point to google cloud * added MSG LST CLIMA MSG LST and MSG LST CLIMA can be calculated with the step0 function as the otehr products. They won't have a step1. This eases reprocessing -satromo processor: added PRODUCT_MSG_CLIMA and PRODUCT_MSG exit points -dev_config: added RODUCT_MSG_CLIMA and PRODUCT_MSG new -step0_processor_msg_lst.py -step0_processor_msg_lst_clima.py * added command line option for date we can now reprocess a specific date from command line 'python satromo_processor.py prod_config.py 2024-06-12', the config file is a must when us with date option * first steps to get rid of the black part in the visualisation #56 - outside of the swiss perimeter is nan, inside, but in the swiss perimeter but outside the flightpath is zero. - line 605 in step0_processor_s2_sr.py is causing issues. * Added satromo_processor.drawio.svg * Update satromo_processor.drawio * Update README.md Added Description for Single Scene Processing via Command Line * Removes black border Fix for #56 * Moved ch_buffer_5000m to assets * moved ch_buffer_5000m.shp to assets path * Thumbnail Thumbnail: Fixed aspect ratio for VHI and S2 SR. Updated background to accommodate new S2 SR tiles cut to the effective path. Changed no data values accordingly. util_listassets.py: Added functionality to list all assets of a collection. util_moveassets_terrainshadow.py: Script to initialize Google Earth Engine (GEE) and Google Drive authentication. Converts Day-Of-Year (DOY) to datetime strings. Manages the copying and updating of assets within Google Earth Engine. satromo_publish.py: Added detection for VHI cases to use the Swiss buffer. step0_processor_s2_sr.py: Replaced swisswalti with swisssurface. Set terrainShadowDetectionPrecalculated to True to use pre-calculated shadows. step1_processor_vhi.py: Updated to use T235959 time format instead of 240000 as per definitions with BAFU and MeteoSwiss. * Update requirements.txt set certifi to latest * Update step0_processor_s2_sr.py fix missing slash at the end of the shadow terrain collection * Prepared for deploy to prod --------- Co-authored-by: Joan Sturm <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
There is currently no metadata in the collection that allows the user to filter for only the 10m images vs the 20m images in the collection. The spatial resolution is only part of the file/asset name.
Current filtering process
Expected behavior
The text was updated successfully, but these errors were encountered: