A first pass at regenerating atlases that fail our validation scripts

[alessandrofelder]

Is your feature request related to a problem? Please describe.
Thanks to brainglobe/bg-atlasgen#116, we now have a mature-enough first version of a validation framework for BrainGlobe atlases, and have run it on all existing atlases. This has helped us identify a number of validation failures. Many of these consist of a mismatch between the annotation and the root mesh extents - a possible source of this may already be fixed in the code by brainglobe/bg-atlasgen#62 but the atlases have not been regenerated yet.

Describe the solution you'd like

• regenerate atlases that fail locally
• run the validation again
• push new versions of the atlases that have been fixed by regeneration to GIN, hopefully at least [BUG] Spinal cord mesh rotated at 90 degrees in meshio/napari #231 will be fixed by this.
• document the atlases that still fail the validation in a new issue.

Describe alternatives you've considered
/

Additional context
/

[adamltyson]

Would this be a good time to pick up the original idea for packaging atlases following updates, namely a script that can run unattended on HPC/workstation to:

• Repackage all atlases
• Validate
• Upload to PyPI

The current process is a bit of a pain

[alessandrofelder]

This might be a good idea!
Minor thing that is unclear in my head: would we bump versions of atlases when the code has changed, but the resulting atlas has not?

[alessandrofelder]

An alternative might be to embed the commit hash that an atlas was created with in its metadata 🤔

[adamltyson]

Minor thing that is unclear in my head: would we bump versions of atlases when the code has changed, but the resulting atlas has not?

Originally we had an idea that the atlas versions would have the same major version as the atlas API. So maybe only bump the atlases when the atlas API has a major bump?

[adamltyson]

I.e. atlas version 2.x.y would be x.y. version compatible with version 2 of the API.

[alessandrofelder]

Would this be a good time to pick up the original idea for packaging atlases following updates, namely a script that can run unattended on HPC/workstation to:

• Repackage all atlases
• Validate
• Upload to PyPI

In the last point, do you mean releasing the code to PyPI and/or uploading the atlases to GIN, @adamltyson ?

[adamltyson]

Sorry, yes uploading to GIN (not PyPI).

[alessandrofelder]

namely a script that can run unattended on HPC/workstation to

Yes, I think this is a good way to address this issue. This script would either...

• need to check for differences after regeneration to be able to decide whether to push a new version to GIN, or...
• always bump the (minor) version of all atlases if they were generated with a new tagged version of brainglobe-atlasapi

What do you think @viktorpm @adamltyson ?

2.x.y would be x.y. version compatible with version 2 of the API.

I am happy with this.

[adamltyson]

need to check for differences after regeneration to be able to decide whether to push a new version to GIN, or...

I think this one. If we want to keep old versions on GIN, we'd be uploading huge amounts of data unnecessarily.

[viktorpm]

I.e. atlas version 2.x.y would be x.y. version compatible with version 2 of the API.

Yes, I agree. This sounds reasonable