Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

handle groups in teca_cf_writer #801

Open
taobrienlbl opened this issue Oct 10, 2024 · 4 comments
Open

handle groups in teca_cf_writer #801

taobrienlbl opened this issue Oct 10, 2024 · 4 comments
Assignees
Labels
1_high_priority an issue that should be fixed prior to the next release I/O high priority I/O issues

Comments

@taobrienlbl
Copy link
Collaborator

@ghweber - great job on #799! I can now use teca_metadata_probe on output from HENS, and I can almost use teca_bayesian_ar_detect. It looks like the coordinate name global/lat and global/lon is passed on in the TECA metadata in a way that causes the teca_cf_writer stage to fail:

taobrien@nid001105:/pscratch/sd/t/taobrien/tmp$ module use /global/common/software/m1517/teca/perlmutter_gpu/develop-9190a290/modulefiles/
taobrien@nid001105:/pscratch/sd/t/taobrien/tmp$ module load teca
taobrien@nid001105:/pscratch/sd/t/taobrien/tmp$ srun -N 1 -G 4 -n 1 \
    teca_bayesian_ar_detect \
    --input_regex ${data_root}/ensemble_out_03708_2023-06-23-00-00-00\.nc \
    --x_axis_variable global/lon  \
    --y_axis_variable global/lat \
    --ivt=global/ivt \
    --output_file TECA_BARD_%t%_EM_03708.nc
ERROR: [0:140098274140160] [/pscratch/sd/t/taobrien/TECA_superbuild/site/TECA_superbuild/build-gpu-develop-9190a290/TECA-prefix/src/TECA/io/teca_cf_layout_manager.cxx:309 5.0.0-324-g9190a29]
ERROR: failed to define dimensions for coordinate axis 1 "global/lat" NetCDF: Name contains illegal characters

Two potential solutions come to mind, one simple, the other more complex:

  1. modify the TECA metadata to remove the global/ part from variable names and instead add a metadata key that stores the netCDF group of the variable; or
  2. modify teca_cf_writer to handle groups: e.g., create lat in the global group if the variable name is global/lat

What do you think? And would you be able to take this on?

@taobrienlbl taobrienlbl added I/O high priority I/O issues 1_high_priority an issue that should be fixed prior to the next release labels Oct 10, 2024
@ghweber
Copy link
Collaborator

ghweber commented Oct 10, 2024

I'll take a look. I lean toward solution 2, but will see what makes the most sense.

@taobrienlbl
Copy link
Collaborator Author

that sounds good to me - thank you! If we do go with solution 2, it might be good to add an option to the CF writer to override the group for all variables (e.g., set the group to / to avoid groups in the output file). If this is implemented as a BOOST command line option, then it could be specified for any TECA app through the advanced command line options.

@ghweber
Copy link
Collaborator

ghweber commented Oct 12, 2024

What would you do in the case where this leads to multiple variables with the same name?

@taobrienlbl
Copy link
Collaborator Author

Throw an error, I think. If there is a variable name conflict resulting from the user trying to put variables in the root group, that would suggest that the file should in fact have groups and that putting variables in root is probably a bad idea.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
1_high_priority an issue that should be fixed prior to the next release I/O high priority I/O issues
Projects
None yet
Development

No branches or pull requests

2 participants