Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add arctic weather satelliter l1b reader #2565

Draft
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

adybbroe
Copy link
Contributor

@adybbroe adybbroe commented Sep 5, 2023

This PR adds a reader for the ESA Arctic Weather Satellite (AWS) l1b format.

The AWS file format is netCDF and supposed to follow the EPS-SG MWS level-1b format. See https://www.eumetsat.int/media/44139

However, please be aware that there are some issues with the level-1b format of the test data we have acquired from ESA. It is yet unsure to what extent ESA will adapt to our feedback.

  • Closes #xxxx
  • Tests added
  • Fully documented
  • Add your name to AUTHORS.md if not there already

@adybbroe
Copy link
Contributor Author

adybbroe commented Sep 5, 2023

Here is a view of channel 1 remapped to a near-ided perspective projection over northern Europe:

aws_testdata_50 3GHz_20230707_1200_nsper_swe

@codecov
Copy link

codecov bot commented Sep 5, 2023

Codecov Report

Attention: Patch coverage is 98.58491% with 3 lines in your changes missing coverage. Please review.

Project coverage is 96.06%. Comparing base (9e3b342) to head (a84d874).
Report is 79 commits behind head on main.

Files with missing lines Patch % Lines
satpy/readers/aws_l1b.py 97.70% 2 Missing ⚠️
satpy/tests/reader_tests/test_aws_l1b.py 99.20% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main    #2565    +/-   ##
========================================
  Coverage   96.06%   96.06%            
========================================
  Files         373      375     +2     
  Lines       54465    54691   +226     
========================================
+ Hits        52321    52539   +218     
- Misses       2144     2152     +8     
Flag Coverage Δ
behaviourtests 3.97% <0.00%> (-0.02%) ⬇️
unittests 96.16% <98.58%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@mraspaud mraspaud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just two comments for now.

name: aws_l1b_nc
short_name: AWS L1B RAD NetCDF4
long_name: AWS L1B Radiance (NetCDF4)
description: Reader for the ESA AWS (Arctic Weather Satellite) Sounder level-1b files in netCDF4.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a link to online documentation of the format we could use?

satpy/etc/readers/aws_l1b_nc.yaml Outdated Show resolved Hide resolved
@mraspaud mraspaud marked this pull request as ready for review November 7, 2023 09:01
@coveralls
Copy link

coveralls commented Nov 7, 2023

Pull Request Test Coverage Report for Build 11277592514

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 210 of 213 (98.59%) changed or added relevant lines in 2 files are covered.
  • 33 unchanged lines in 6 files lost coverage.
  • Overall coverage decreased (-0.003%) to 96.161%

Changes Missing Coverage Covered Lines Changed/Added Lines %
satpy/tests/reader_tests/test_aws_l1b.py 124 125 99.2%
satpy/readers/aws_l1b.py 86 88 97.73%
Files with Coverage Reduction New Missed Lines %
satpy/composites/config_loader.py 2 94.12%
satpy/tests/utils.py 2 93.16%
satpy/tests/reader_tests/gms/test_gms5_vissr_l1b.py 3 98.67%
satpy/tests/reader_tests/gms/test_gms5_vissr_navigation.py 3 97.18%
satpy/readers/msi_safe.py 4 98.41%
satpy/utils.py 19 92.15%
Totals Coverage Status
Change from base Build 10957325874: -0.003%
Covered Lines: 52774
Relevant Lines: 54881

💛 - Coveralls

@adybbroe adybbroe self-assigned this Nov 7, 2023
@adybbroe adybbroe added enhancement code enhancements, features, improvements component:readers labels Nov 7, 2023
Copy link
Contributor Author

@adybbroe adybbroe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks very nice, many thanks!

@adybbroe
Copy link
Contributor Author

adybbroe commented Nov 17, 2023

So, just a comment/observation. The AWS testdata file delivered August 2023 has the datasets structured a bit odd and different from most other satellite datasets (e.g. the EPS-SG MWS testdata which format it is supposed to follow).

Here a phrase communicated to ESA abouth this layout issue:

"The ordering of the dimensions is unexpected, e.g. for “aws_toa_brightness_temperature” we have (19, 145, 4991) which is (n_channels, n_fovs, n_scans). But in netCDF the first dimension is always the one that varies the most slowly. For MWS the order is reversed, i.e. (n_scans, n_fovs, n_channels)."

This ordering currently has the unfortunate consequence that the optimal radius of influence used for remapping cannot be determined. The method geocentric_resolution in the CoordinateDefinition class in the geometry.py module of Pyresample assumes the first dimension is the scanlines, but here it is the pixel number across scan instead, so the points found are the points close to nadir of the first scanline (instead of the outer most edge of the scan in the middle of the swath). In the testdata set delivered the first 10-20 scanlines or so are missing-data in the geolocation (for some strange reason).

I am preparing a PR for Pyresample on this.

Until now one has to provide a reasonable value (say 40000) for radius_of_influence parameter explicitly in the resample call on the scene object.

When doing this one may be able create an image like the below using code similar to this (I have here skipped the decoration/coast overlay part for simplification):

scn = Scene(filenames=FILENAMES, reader='aws_l1b_nc')
composite_name = 'mw183_humidity_surface'
scn.load([composite_name])

local = scn.resample(AREAID, radius_of_influence=40000)
local.show(composite_name)

aws_testdata_radsim_mw183_humidity_surface_20240115_1111_nsper_swe

Copy link
Member

@pnuu pnuu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments inline. I'd like some clarifiactions, but in general LGTM!

Comment on lines +21 to +22
- QH
- QV
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, what are these polarizations? It's been ages when I've done anything with microwave data, and from there I just remember H (horizontal linear polarization), V (vertical), VH (send vertical, receive horizontal, for radars), LHC (left-handed circular), RHC but nothing starting with Q comes to mind.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pnuu The Q here stands for "Quasi"
"QV" means Vertical polarization in nadir, but as the mirror rotates towards the scan edges, the radiation measured will be effectively containing more and more contribution from H and less from V.
By design it is not possible to measure only one component.

Comment on lines +282 to +284
central: 325.15
side: 1.2
bandwidth: 0.8
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify, there's one 800 MHz wide side-band 1.2 GHz below and above the central frequenzy?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines 30 to 36
AWS_CHANNEL_NAMES_TO_NUMBER = {"1": 1, "2": 2, "3": 3, "4": 4,
"5": 5, "6": 6, "7": 7, "8": 8,
"9": 9, "10": 10, "11": 11, "12": 12,
"13": 13, "14": 14, "15": 15, "16": 16,
"17": 17, "18": 18, "19": 19}

AWS_CHANNEL_NAMES = list(AWS_CHANNEL_NAMES_TO_NUMBER.keys())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name -> number conversion is not used, so this could be simplified to

Suggested change
AWS_CHANNEL_NAMES_TO_NUMBER = {"1": 1, "2": 2, "3": 3, "4": 4,
"5": 5, "6": 6, "7": 7, "8": 8,
"9": 9, "10": 10, "11": 11, "12": 12,
"13": 13, "14": 14, "15": 15, "16": 16,
"17": 17, "18": 18, "19": 19}
AWS_CHANNEL_NAMES = list(AWS_CHANNEL_NAMES_TO_NUMBER.keys())
AWS_CHANNEL_NAMES = [str(i) for i in range(1, 20)]

And if there's a need for the conversion you could have a utility function to convert from name to the number. Or just use int(name) .

def _name2num(name):
    return int(name)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose I was not sure if we should name the channels 1 to 19 or as indicated on the ESA web site by using two digits where the first one indicate the group...
https://www.esa.int/Applications/Observing_the_Earth/Meteorological_missions/Arctic_Weather_Satellite/The_instrument

satpy/readers/aws_l1b.py Outdated Show resolved Hide resolved
satpy/tests/reader_tests/test_aws_l1b.py Show resolved Hide resolved
satpy/tests/reader_tests/test_aws_l1b.py Outdated Show resolved Hide resolved
@adybbroe
Copy link
Contributor Author

I somehow thought we had merged this long ago?
But, it makes sense not to merge it now, as we will have new test data from ESA in about a month, where the issues with the format that we raised with the, should be solved. Then we can update the reader for the new format.
So, we can wait another month or two perhaps, or what do you say @mraspaud and @pnuu ?

Sorry for not coming back to your comments/questions earlier!

@mraspaud
Copy link
Member

Any news on the new test dataset?

@adybbroe
Copy link
Contributor Author

Test data is supposed to come in ~2 weeks according to latest news from ESA (August 16). Let's hope this latest forecast is more accurate than the previous.

@adybbroe adybbroe marked this pull request as draft October 1, 2024 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:readers enhancement code enhancements, features, improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants