Remove pavics-datacatalog from stack #287
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
documentation
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1190/Result : failure BIRDHOUSE_DEPLOY_BRANCH : fix-206 DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https:// PAVICS-e2e-workflow-tests Pipeline ResultsTests URL :NOTEBOOK TEST RESULTS
|
|
run tests |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1211/Result : failure BIRDHOUSE_DEPLOY_BRANCH : fix-206 DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https:// PAVICS-e2e-workflow-tests Pipeline ResultsTests URL :NOTEBOOK TEST RESULTS
|
|
Self note, need to align with #284 to remove extra configs not needed anymore. |
E2E Test ResultsDACCS-iac Pipeline ResultsBuild URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1213/Result : failure BIRDHOUSE_DEPLOY_BRANCH : fix-206 DACCS_CONFIGS_BRANCH : master PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master PAVICS_SDI_BRANCH : master DESTROY_INFRA_ON_EXIT : true PAVICS_HOST : https://host-140-154.rdext.crim.ca PAVICS-e2e-workflow-tests Pipeline ResultsTests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/944/NOTEBOOK TEST RESULTS |
|
All the notebooks seem to pass on our prod version, so not clear to me where the nb failures are coming from. |
This was referenced Feb 10, 2023
fmigneault
added a commit
that referenced
this pull request
Apr 20, 2023
## Overview
Please include a summary of the changes and which issues are fixed.
Please also include relevant motivation and context.
List any dependencies that are required for this change.
## Breaking changes
- CanarieAPI: update to `0.7.1`.
- The Docker running `CanarieAPI` is now using Python 3 (since `0.4.x` tags).
Configurations need to be updated if any specific Python 2 definitions were used.
See [2to3](https://docs.python.org/3/library/2to3.html) to help migrate configurations automatically if necessary.
- Update the [CanarieAPI configuration](birdhouse/config/canarie-api/docker_configuration.py.template) to use
Python 3.x executable code.
## Changes
- CanarieAPI: update to `0.7.1`.
- The server node now provides a generic ``server`` configuration for the current ``platform`` definition.
- Added multiple missing docuementation references for all the services included within `CanarieAPI` configurations.
- With new `CanarieAPI` version, a slightly improved UI with more service details are provided for the active server:
- Add optional variables witht defaults to define reference Docker image version tags.
Following optional variables are defined by default. These are used as reference in the respective Docker compose
service definition of these components, as well as in their `CanarieAPI` configuration to retrieve the release time
of the tag, and refer to relevant URL references as needed.
- `CATALOG_VERSION`
- `FINCH_VERSION`
- `FLYINGPIGEON_VERSION`
- `GEOSERVER_VERSION`
- `HUMMINGBIRD_VERSION`
- `MALLEEFOWL_VERSION`
- `RAVEN_VERSION`
## Fixes:
- CanarieAPI: update to `0.7.1`.
- Fixes an `AttributeError` raised due to misconfiguration of the Web Application with Flask 2.x definitions
(relates to [Ouranosinc/CanarieAPI#10](Ouranosinc/CanarieAPI#10)).
- Skip over `0.4.x`, `0.5.x`, `0.6.x` versions to avoid issue related to `cron` job monitoring and log parser
command failures in order to collect configured service statistics and statuses
(see also [Ouranosinc/CanarieAPI#14](Ouranosinc/CanarieAPI#14)).
- Weaver: update CanarieAPI monitoring definitions
- Move monitoring of public endpoint under [optional-components/canarie-api-full-monitoring][canarie-monitor].
- Add monitoring of private endpoint by default when using Weaver component.
- Cowbird: update CanarieAPI monitoring definitions
- Add monitoring of public endpoint under [optional-components/canarie-api-full-monitoring][canarie-monitor].
- Add public Magpie permission on Cowbird entrypoint only to allow its monitoring.
## Additional Information
Resolves the following log error.
```
proxy | [2023-01-31 19:37:01 +0000] [37] [DEBUG] GET /canarie/
proxy | [2023-01-31 19:37:01,708] [37] [INFO] app_object : Disconnecting from database
proxy | [2023-01-31 19:37:01,709] [37] [DEBUG] app_object : Using db filename : /opt/local/src/CanarieAPI/stats.db
proxy | [2023-01-31 19:37:01 +0000] [37] [DEBUG] Closing connection.
proxy | [2023-01-31 19:37:02 +0000] [37] [DEBUG] GET /canarie/background.jpg
proxy | [2023-01-31 19:37:02,176] [37] [INFO] app_object : Disconnecting from database
proxy | [2023-01-31 19:37:02,176] [37] [DEBUG] app_object : Using db filename : /opt/local/src/CanarieAPI/stats.db
proxy | [2023-01-31 19:37:02 +0000] [37] [ERROR] Error handling request /canarie/background.jpg
proxy | Traceback (most recent call last):
proxy | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/async.py", line 56, in handle
proxy | self.handle_request(listener_name, req, client, addr)
proxy | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 152, in handle_request
proxy | super(GeventWorker, self).handle_request(*args)
proxy | File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/async.py", line 107, in handle_request
proxy | respiter = self.wsgi(environ, resp.start_response)
proxy | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1997, in __call__
proxy | return self.wsgi_app(environ, start_response)
proxy | File "/opt/local/src/CanarieAPI/canarieapi/reverse_proxied.py", line 33, in __call__
proxy | return self.app(environ, start_response)
proxy | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1985, in wsgi_app
proxy | response = self.handle_exception(e)
proxy | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1532, in handle_exception
proxy | handler = self._find_error_handler(InternalServerError())
proxy | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1449, in _find_error_handler
proxy | .get(code))
proxy | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1440, in find_handler
proxy | handler = handler_map.get(cls)
proxy | AttributeError: 'function' object has no attribute 'get'
```
The above problem lead to unresponsive CanarieAPI, which in turn caused the platform to fail responding with successful monitoring statuses for requests toward the configured services, which in turn, caused the stack to never completely boot. When the stack failed to boot, the *End2End Test Results* (example: #283 (comment)) could not run due to the unresponsive instance, which is the cause of the incomplete output:
`````
Tests URL :
NOTEBOOK TEST RESULTS
````
</code>
``
`````
# To Do (in follow-up PRs)
- remove deprecated configs deleted in #287
- remove deprecated configs deleted in #291
- remove deprecated configs deleted in #292
|
Can this be closed ? |
|
@huard I wonder if we should wait until STAC is included (#297), before we transition everything? |
|
I think @mishaschwartz has a plan to handle all the components that will be removed or disabled. |
|
Ok, I'll close the PRs. Once the big reorg is done, I can go through the docs and remove the obsolete material. |
mishaschwartz
added a commit
that referenced
this pull request
Jul 6, 2023
## Overview Move unused and unmaintained components to a separate `deprecated-components/` subdirectory and remove them from the DEFAULT_CONF_DIRS list if required. ## Changes **Non-breaking changes** **Breaking changes** - The following components are no longer present by default: - catalog - frontend - malleefowl - ncops - ncwms2 - project-api - solr - phoenix ## Related Issue / Discussion - Resolves #214 - Closes #287 - Resolves #206 - Closes #291 - Resolves #9 - Closes #292 - Resolves #290 ## Additional Information
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
The Pavics-datacatalog is not used, unmaintained, and not part of future development plans. Catalogs are currently provided by intake static files, and will eventually be accompanied by a STAC catalog.
This PR has not been tested, and frankly, I don't know what I'm doing.
Changes
Non-breaking changes
Breaking changes
Related Issue / Discussion
Additional Information
Links to other issues or sources.