Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Many existing service permissions on group anonymous lost after upgrading from 1.7.3 to 3.5.1 #403

Closed
tlvu opened this issue Mar 25, 2021 · 0 comments · Fixed by #407
Assignees
Labels
bug Problem, error, or invalid behaviour

Comments

@tlvu
Copy link
Contributor

tlvu commented Mar 25, 2021

Describe the bug
Test upgrade of production instance for PR bird-house/birdhouse-deploy#107. This is done by importing the Magpie Postgres DB from prod pavics.ouranos.ca into staging medus.ouranos.ca.

In this case, the upgrade failed on imported DB from prod pavics.ouranos.ca.

However, medus.ouranos.ca itself was successfully upgraded via the autodeploy and no errors found.

My dev VM was also upgraded and no Jenkins errors, except the bug found in #401 where the Anonymous group, Thredds config page completely crash.

Basically, it seems like the upgrade is a random hit-or-miss depending on the existing Magpie data. 3 different environments upgraded with 3 completely different results.

Thredds perm before

crop-Screenshot_2021-03-25 Magpie Administration-boreas-be4-gr-anon-thredds

Thredds perm after (lost the perm on the top level "thredds")

crop-Screenshot_2021-03-25 Magpie-boreas-after-upgrade-group-anon-thredds

WPS perm before

crop-Screenshot_2021-03-25 Magpie Administration-boreas-be4-anon-group-wps

WPS perm after

crop-Screenshot_2021-03-25 Magpie-boreas-after-upgrade-group-anon-wps png

All the other services used to have all permissions turned on before and all lost after upgrade

crop-Screenshot_2021-03-25 Magpie-boreas-after-upgrade-group-anon-geoserverwms png

crop-Screenshot_2021-03-25 Magpie-boreas-after-upgrade-group-anon-ncwms png

crop-Screenshot_2021-03-25 Magpie-boreas-after-upgrade-group-anon-wfs png

Jenkins test suite effectively failed corresponding to those permissions lost.

Thredds perms lost:

OSError: [Errno -78] NetCDF: Authorization failure: b'https://medus.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/cccma/CanESM2/historical/day/atmos/r1i1p1/tasmin/tasmin_day_CanESM2_historical_r1i1p1_18500101-20051231.nc'

WPS perms lost:

16:08:21  ServiceException: <?xml version="1.0" encoding="utf-8"?>
16:08:21  <ExceptionReport version="1.0.0"
16:08:21      xmlns="http://www.opengis.net/ows/1.1"
16:08:21      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
16:08:21      xsi:schemaLocation="http://www.opengis.net/ows/1.1 http://schemas.opengis.net/ows/1.1.0/owsExceptionReport.xsd">
16:08:21      <Exception exceptionCode="NoApplicableCode" locator="AccessForbidden">
16:08:21          <ExceptionText>Not authorized to access this resource. User does not meet required permissions.</ExceptionText>
16:08:21      </Exception>
16:08:21  </ExceptionReport>

To Reproduce
Steps to reproduce the behavior:

  1. Save /data/magpie_persist folder from prod pavics.ouranos.ca: cd /data; tar czf magpie_persist.prod.tgz magpie_persist
  2. scp magpie_persist.prod.tgz to medus
  3. login to medus
  4. cd /path/to/birdhouse-deploy/birdhouse
  5. ./pavics-compose.sh down
  6. git checkout 1.11.24 # before PR Magpie thredds bird-house/birdhouse-deploy#107
  7. cd /data
  8. rm -rf magpie_persist
  9. tar xzf magpie_persist.prod.tgz # restore Magpie DB with prod version
  10. cd /path/to/birdhouse-deploy/birdhouse
  11. ./pavics-compose.sh up -d
  12. Update env.local MAGPIE_ADMIN_PASSWORD with prod passwd for Twitcher to be able to access Magpie since unable to change admin passwd back to the old value on medus, issue [BUG] User admin unable to change his own password after upgrade from 1.7.3 to 3.5.1  #402
  13. ./pavics-compose.sh restart twitcher # for Twitcher to get new Magpie admin passwd
  14. Baseline working state: trigger Jenkins test suite, ensure all pass except pavics_thredds.ipynb that requires new Magpie
  15. Baseline working state: take screen shot of existing services permissions on group Anonymous
  16. git checkout 7be3eeaf0464511abc89e93d1f553de2d7413b59 # Pr merge commit of new Magpie
  17. ./pavics-compose.sh up -d # upgrade to new Magpie
  18. docker logs magpie: check no DB migration error
  19. Trigger Jenkins test suite again
  20. Take screen shot of new services permissions on group Anonymous

Expected behavior
Should not lose existing permissions, Jenkins test suite should have same result as before upgrade.

@tlvu tlvu added the bug Problem, error, or invalid behaviour label Mar 25, 2021
tlvu added a commit to bird-house/birdhouse-deploy that referenced this issue Mar 26, 2021
Revert "Merge pull request #107 from bird-house/magpie-thredds"

This reverts commit 7be3eea, reversing changes made to c0ffb41.

PR #107 upgrade path is unstable, see issue Ouranosinc/Magpie#403 and issue Ouranosinc/Magpie#401.

Revert so we can remove the autodeploy block on production instances and give us time to properly test any upcoming fixes.  Also unblock Finch to go to production #138.

Branch https:/bird-house/birdhouse-deploy/tree/restore-previous-broken-magpie-upgrade-so-we-can-work-on-a-fix created so we have a place to work on a fix.

Matching notebooks revert: Ouranosinc/pavics-sdi#211
fmigneault added a commit that referenced this issue Mar 26, 2021
resolve bad migration operation causing drop of permissions (fixes #403)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Problem, error, or invalid behaviour
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants