-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Flexible splits #125
Flexible splits #125
Conversation
Let the user specify along which dimensions to split. To avoid mismatches between the split dimensions and the output file pattern, the split dimensions are extracted directly from the output formatting.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Part 1 of this CL review. Will work on the next review now, but this should give you stuff to work on sooner.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will look at the core of NetCDF splitting tomorrow.
…FileInfo, removed redundant method in FileSplitter.
…ools into flexible_splits
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Allow more flexible splitting.
Dimensions to split along are specified using output formatting to avoid mismatch between formatting and splitting.
Possible splits for GRIB files are all metadata fields, like shortName, typeOfLevel, date, time, step, forecastTime, etc.
Possible splits for NetCDF files are 'level', 'time', 'variable'. If the specified dimension (time, level) is not in the file, and exception will the thrown.
Output formatting can be given using an output template that contains formatting marks, or an output directory with an additional formatting string. If an output directory is specified without formatting, file type-specific defaults are used.