Feature/dac 300 weaver compose #114

fmigneault · 2021-01-12T23:19:15Z

Live instance (at CRIM with daccs-iac):
https://host-140-36.rdext.crim.ca/twitcher/ows/proxy/weaver
(OpenAPI doc endpoint for info: https://host-140-36.rdext.crim.ca/weaver/api)

Example of flyingpigeon's WPS-1 subset_bbox process converted to WPS-REST via Weaver:
https://host-140-36.rdext.crim.ca/weaver/providers/flyingpigeon/processes/subset_bbox

Dependencies

employ docker-compose v3.4 and shared logging config #152 (docker-compose v3.4 & logging)
Utility docker-compose optional components #153 (optional components - db ports & wps-healthchecks)
customization env vars #154 (env var DATA_PERSIST_ROOT and server-related docs)

note: until they are merged, changes of this PR will continue to contain all of them
preview of simplified modifications: changes
all above PRs are combined in branch tmp-everything-except-weaver to generate the this diff changes

Changes

Adds Weaver to the stack using optional-component.
Adds worker image & docker-proxy to allow Weaver booting a docker Application Package as sibling container.
Adds Magpie permissions for Weaver.
Adds all WPS birds of the stack as known providers of Weaver.
Bump docker-compose to 3.4 to resolve network issue.
Add x-logging config to reuse definition across compose. (fixes Free disk space problem due to docker images logging #10)
Add generic metadata variables that can be shared and displayed in responses of multiple services.
Add DATA_PERSIST_ROOT to allow changing default /data where persistence dirs are generated.

Extra

Add optional-component wps-healthchecks (developed while testing Weaver integration, but not directly needed by it any more).
Add optional-component database-external-ports, to allow exposing port for dev instances, but that should definitely not be open by default.

Resolves https://www.crim.ca/jira/browse/DAC-25
Resolves https://www.crim.ca/jira/browse/DAC-300
Resolves crim-ca/weaver#13

daccs-iac config : https://www.crim.ca/stash/projects/DACCS/repos/daccs-configs/pull-requests/1

dbyrns

I got some questions, but overall this is a nice PR.

birdhouse/optional-components/testthredds/docker-compose-extra.yml

birdhouse/optional-components/weaver/conf.extra-service.d/weaver.conf.template

birdhouse/optional-components/weaver/config/weaver/data_sources.json.template

birdhouse/optional-components/weaver/config/weaver/weaver.ini.template

birdhouse/optional-components/weaver/config/weaver/wps_processes.yml.template

birdhouse/optional-components/weaver/default.env

fmigneault · 2021-01-13T18:34:34Z

Jenkins-bot failing due to Ouranosinc/PAVICS-e2e-workflow-tests#58

birdhouse/optional-components/testthredds/docker-compose-extra.yml

birdhouse/optional-components/weaver/conf.extra-service.d/weaver.conf.template

dbyrns

I think the volume mapping needs to be reviewed.

birdhouse/optional-components/weaver/default.env

birdhouse/config/postgres/credentials.env.template

tlvu

Just starting review. Not done yet. 1 change request: move Weaver to birdhouse/components/ instead.

birdhouse/config/magpie/permissions.cfg.template

birdhouse/docker-compose.yml

birdhouse/optional-components/README.md

fmigneault · 2021-01-14T16:01:10Z

Just starting review. Not done yet. 1 change request: move Weaver to birdhouse/components/ instead.

I thought the whole point was to make it optional?
@dbyrns can you confirm?

I refactored everything as optional for this...

birdhouse/components/README.rst

birdhouse/config/magpie/permissions.cfg.template

tlvu

Finished first pass of my review. Still need to deploy it and play with it. Will probably have more feedbacks once I actually try to use it.

Big component, great work plugging into the stack. I think you got the hang of how the pluggable architecture work. As one of my comment, does weaver allow other birds to plug themselves into weaver config?

birdhouse/components/weaver/config/weaver/weaver.ini.template

birdhouse/components/weaver/config/weaver/data_sources.json.template

birdhouse/components/weaver/config/weaver/wps_processes.yml.template

tlvu · 2021-01-15T23:25:12Z

birdhouse/components/weaver/default.env

+'
+# extend the original 'VARS' from 'birdhouse/pavics-compose.sh' to employ them for template substitution
+# adding them to 'VARS', they will also be validated in case of override of 'default.env' using 'env.local'
+VARS="$VARS $EXTRA_VARS"


You sure you want to add to VARS and not OPTIONAL_VARS? I have not checked but if all these have default values, OPTIONAL_VARS is much more appropriate. See https:/bird-house/birdhouse-deploy/blob/e10a2ef9bd77498f1af22267cad5d181d0da65ef/birdhouse/components/monitoring/default.env for example

All variables are mandatory in the sense that if they are overridden by an empty string, compose won't work at all.
So although they are optional in terms of "no need any override", they must be validated.
My understanding of OPTIONAL_VARS was that empty value will still pass.
Was I wrong? If so, I don't understand why there are 2 variables.

birdhouse/components/weaver/docker-compose-extra.yml

birdhouse/default.env

birdhouse/optional-components/wps-healthchecks/docker-compose-extra.yml

birdhouse/pavics-compose.sh

…otes the requirement

…eeded, default to original hardcoded values

…ners + remove default open db ports in favor of optional-component to be explicitly defined when needed

fmigneault · 2021-10-27T22:28:28Z

New test triggered with latest master merged & added changes. Results should appear eventually...

tlvu

LGTM with latest changes for canarie-api.

tlvu · 2021-10-27T22:48:45Z

This is why I think there is not much to test in terms of upgrade deployment, since nothing will be added if those components are not added manually in your/others config.

Exact. As long as Weaver is not activated, nothing should break. That's why I can accept this PR as-is.

But once it is activated, we know it can start the first time but we have no idea how it integrate itself into the upgrade process (autodeploy) because that scenario is not tested by the daccs-iac.

crim-jenkins-bot · 2021-10-27T23:35:10Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/720/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : feature/DAC-300-weaver-compose
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-8.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/637/

NOTEBOOK TEST RESULTS

    
[2021-10-27T23:19:53.607Z] ============================= test session starts ==============================
[2021-10-27T23:19:53.607Z] platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
[2021-10-27T23:19:53.607Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2021-10-27T23:19:53.607Z] plugins: anyio-3.3.0, dash-1.21.0, nbval-0.9.6, tornasync-0.6.0.post2
[2021-10-27T23:19:53.607Z] collected 210 items
[2021-10-27T23:19:53.607Z] 
[2021-10-27T23:20:03.308Z] notebooks-auth/test_thredds.ipynb ...........                            [  5%]
[2021-10-27T23:20:24.950Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  8%]
[2021-10-27T23:20:35.547Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 11%]
[2021-10-27T23:20:44.295Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 15%]
[2021-10-27T23:20:47.421Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 20%]
[2021-10-27T23:22:38.996Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb F                 [ 20%]
[2021-10-27T23:22:38.997Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 20%]
[2021-10-27T23:22:45.951Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 24%]
[2021-10-27T23:22:53.820Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 26%]
[2021-10-27T23:26:54.320Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 33%]
[2021-10-27T23:27:46.343Z] ..............                                                           [ 40%]
[2021-10-27T23:27:55.054Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 42%]
[2021-10-27T23:27:58.123Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 46%]
[2021-10-27T23:28:27.987Z] .................                                                        [ 54%]
[2021-10-27T23:28:35.339Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 56%]
[2021-10-27T23:28:46.655Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 61%]
[2021-10-27T23:28:56.541Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 64%]
[2021-10-27T23:29:44.365Z] finch-master/docs/source/notebooks/subset.ipynb ....................     [ 73%]
[2021-10-27T23:29:45.768Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 74%]
[2021-10-27T23:29:51.045Z] ......                                                                   [ 77%]
[2021-10-27T23:30:13.049Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 77%]
[2021-10-27T23:30:30.169Z] .............                                                            [ 83%]
[2021-10-27T23:30:40.196Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
[2021-10-27T23:31:25.532Z] ....s.                                                                   [ 87%]
[2021-10-27T23:31:35.530Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]
[2021-10-27T23:31:43.345Z] ...                                                                      [ 89%]
[2021-10-27T23:32:01.499Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 89%]
[2021-10-27T23:32:28.913Z] ......                                                                   [ 92%]
[2021-10-27T23:32:39.524Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2021-10-27T23:34:17.999Z] notebooks/stress-tests.ipynb ....                                        [100%]
[2021-10-27T23:34:18.000Z] 
[2021-10-27T23:34:18.000Z] =================================== FAILURES ===================================

crim-jenkins-bot · 2021-10-28T00:18:18Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/721/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : feature/DAC-300-weaver-compose
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-8.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/638/

NOTEBOOK TEST RESULTS

    
[2021-10-28T00:04:24.286Z] ============================= test session starts ==============================
[2021-10-28T00:04:24.286Z] platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
[2021-10-28T00:04:24.286Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2021-10-28T00:04:24.286Z] plugins: anyio-3.3.0, dash-1.21.0, nbval-0.9.6, tornasync-0.6.0.post2
[2021-10-28T00:04:24.286Z] collected 210 items
[2021-10-28T00:04:24.286Z] 
[2021-10-28T00:04:32.759Z] notebooks-auth/test_thredds.ipynb ...........                            [  5%]
[2021-10-28T00:04:55.596Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  8%]
[2021-10-28T00:05:06.606Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 11%]
[2021-10-28T00:05:15.623Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 15%]
[2021-10-28T00:05:18.074Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 20%]
[2021-10-28T00:07:09.585Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb F                 [ 20%]
[2021-10-28T00:07:09.585Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 20%]
[2021-10-28T00:07:13.498Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 24%]
[2021-10-28T00:07:20.696Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 26%]
[2021-10-28T00:10:20.186Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 33%]
[2021-10-28T00:11:18.164Z] ..............                                                           [ 40%]
[2021-10-28T00:11:26.011Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 42%]
[2021-10-28T00:11:28.303Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 46%]
[2021-10-28T00:11:51.438Z] .................                                                        [ 54%]
[2021-10-28T00:11:58.126Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 56%]
[2021-10-28T00:12:08.565Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 61%]
[2021-10-28T00:12:16.610Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 64%]
[2021-10-28T00:13:02.692Z] finch-master/docs/source/notebooks/subset.ipynb ....................     [ 73%]
[2021-10-28T00:13:04.079Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 74%]
[2021-10-28T00:13:07.435Z] ......                                                                   [ 77%]
[2021-10-28T00:13:25.573Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 77%]
[2021-10-28T00:13:41.580Z] .............                                                            [ 83%]
[2021-10-28T00:13:51.604Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 84%]
[2021-10-28T00:14:31.758Z] ....s.                                                                   [ 87%]
[2021-10-28T00:14:41.782Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 87%]
[2021-10-28T00:14:48.558Z] ...                                                                      [ 89%]
[2021-10-28T00:15:06.710Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 89%]
[2021-10-28T00:15:38.040Z] ......                                                                   [ 92%]
[2021-10-28T00:15:48.016Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2021-10-28T00:17:25.944Z] notebooks/stress-tests.ipynb ....                                        [100%]
[2021-10-28T00:17:25.945Z] 
[2021-10-28T00:17:25.945Z] =================================== FAILURES ===================================

crim-jenkins-bot · 2021-10-28T01:02:48Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/722/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : feature/DAC-300-weaver-compose
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-8.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/639/

NOTEBOOK TEST RESULTS

    
[2021-10-28T00:48:12.013Z] ============================= test session starts ==============================
[2021-10-28T00:48:12.013Z] platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
[2021-10-28T00:48:12.013Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2021-10-28T00:48:12.013Z] plugins: anyio-3.3.0, dash-1.21.0, nbval-0.9.6, tornasync-0.6.0.post2
[2021-10-28T00:48:12.013Z] collected 221 items
[2021-10-28T00:48:12.013Z] 
[2021-10-28T00:48:21.218Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2021-10-28T00:49:07.495Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .FFFFF.        [  8%]
[2021-10-28T00:49:17.300Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2021-10-28T00:49:26.375Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 14%]
[2021-10-28T00:49:28.792Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 19%]
[2021-10-28T00:51:20.389Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb F                 [ 19%]
[2021-10-28T00:51:20.389Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 19%]
[2021-10-28T00:51:25.033Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 23%]
[2021-10-28T00:51:32.449Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 25%]
[2021-10-28T00:55:05.777Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 32%]
[2021-10-28T00:55:51.752Z] ..............                                                           [ 38%]
[2021-10-28T00:55:58.501Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 40%]
[2021-10-28T00:56:00.740Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 43%]
[2021-10-28T00:56:22.970Z] .................                                                        [ 51%]
[2021-10-28T00:56:29.055Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 53%]
[2021-10-28T00:56:30.462Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 54%]
[2021-10-28T00:56:32.910Z] .FFFFFFFF.                                                               [ 58%]
[2021-10-28T00:56:44.553Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 63%]
[2021-10-28T00:56:52.303Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 66%]
[2021-10-28T00:57:35.875Z] finch-master/docs/source/notebooks/subset.ipynb ....................     [ 75%]
[2021-10-28T00:57:37.786Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 75%]
[2021-10-28T00:57:40.601Z] ......                                                                   [ 78%]
[2021-10-28T00:58:02.569Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 78%]
[2021-10-28T00:58:14.980Z] .............                                                            [ 84%]
[2021-10-28T00:58:24.983Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2021-10-28T00:59:08.237Z] ....s.                                                                   [ 87%]
[2021-10-28T00:59:16.408Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2021-10-28T00:59:24.220Z] ...                                                                      [ 89%]
[2021-10-28T00:59:42.350Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2021-10-28T01:00:11.444Z] ......                                                                   [ 92%]
[2021-10-28T01:00:20.893Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2021-10-28T01:01:59.387Z] notebooks/stress-tests.ipynb ....                                        [100%]
[2021-10-28T01:01:59.387Z] 
[2021-10-28T01:01:59.387Z] =================================== FAILURES ===================================

crim-jenkins-bot · 2021-10-28T13:42:43Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/724/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : feature/DAC-300-weaver-compose
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-8.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/641/

NOTEBOOK TEST RESULTS

    
[2021-10-28T13:27:18.715Z] ============================= test session starts ==============================
[2021-10-28T13:27:18.715Z] platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
[2021-10-28T13:27:18.715Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2021-10-28T13:27:18.715Z] plugins: anyio-3.3.0, dash-1.21.0, nbval-0.9.6, tornasync-0.6.0.post2
[2021-10-28T13:27:18.715Z] collected 221 items
[2021-10-28T13:27:18.715Z] 
[2021-10-28T13:27:28.156Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2021-10-28T13:27:54.486Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  8%]
[2021-10-28T13:28:07.176Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2021-10-28T13:28:16.310Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 14%]
[2021-10-28T13:28:18.887Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 19%]
[2021-10-28T13:30:10.422Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb F                 [ 19%]
[2021-10-28T13:30:11.364Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 19%]
[2021-10-28T13:30:17.014Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 23%]
[2021-10-28T13:30:23.759Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 25%]
[2021-10-28T13:33:56.439Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 32%]
[2021-10-28T13:34:51.025Z] ..............                                                           [ 38%]
[2021-10-28T13:35:00.457Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 40%]
[2021-10-28T13:35:02.709Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 43%]
[2021-10-28T13:35:27.308Z] .................                                                        [ 51%]
[2021-10-28T13:35:35.414Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 53%]
[2021-10-28T13:35:36.806Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 54%]
[2021-10-28T13:35:47.941Z] .......FF.                                                               [ 58%]
[2021-10-28T13:35:58.301Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 63%]
[2021-10-28T13:36:07.516Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 66%]
[2021-10-28T13:36:51.527Z] finch-master/docs/source/notebooks/subset.ipynb ....................     [ 75%]
[2021-10-28T13:36:52.471Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 75%]
[2021-10-28T13:36:56.031Z] ......                                                                   [ 78%]
[2021-10-28T13:37:14.169Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 78%]
[2021-10-28T13:37:30.340Z] .............                                                            [ 84%]
[2021-10-28T13:37:40.395Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2021-10-28T13:38:24.876Z] ....s.                                                                   [ 87%]
[2021-10-28T13:38:37.130Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2021-10-28T13:38:45.631Z] ...                                                                      [ 89%]
[2021-10-28T13:39:12.266Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2021-10-28T13:39:45.941Z] ......                                                                   [ 92%]
[2021-10-28T13:39:57.205Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2021-10-28T13:41:35.359Z] notebooks/stress-tests.ipynb ....                                        [100%]
[2021-10-28T13:41:35.360Z] 
[2021-10-28T13:41:35.360Z] =================================== FAILURES ===================================

crim-jenkins-bot · 2021-10-28T15:47:05Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/725/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : feature/DAC-300-weaver-compose
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : false
PAVICS_HOST : https://host-140-8.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/642/

NOTEBOOK TEST RESULTS

    
[2021-10-28T15:32:05.341Z] ============================= test session starts ==============================
[2021-10-28T15:32:05.341Z] platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
[2021-10-28T15:32:05.341Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2021-10-28T15:32:05.341Z] plugins: anyio-3.3.0, dash-1.21.0, nbval-0.9.6, tornasync-0.6.0.post2
[2021-10-28T15:32:05.341Z] collected 221 items
[2021-10-28T15:32:05.341Z] 
[2021-10-28T15:32:14.729Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2021-10-28T15:32:38.513Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  8%]
[2021-10-28T15:32:49.000Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2021-10-28T15:33:00.848Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 14%]
[2021-10-28T15:33:03.694Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 19%]
[2021-10-28T15:34:55.223Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb F                 [ 19%]
[2021-10-28T15:34:55.223Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 19%]
[2021-10-28T15:34:59.978Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 23%]
[2021-10-28T15:35:07.189Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 25%]
[2021-10-28T15:38:52.009Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 32%]
[2021-10-28T15:39:47.822Z] ..............                                                           [ 38%]
[2021-10-28T15:39:57.626Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 40%]
[2021-10-28T15:39:59.582Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 43%]
[2021-10-28T15:40:22.676Z] .................                                                        [ 51%]
[2021-10-28T15:40:30.505Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 53%]
[2021-10-28T15:40:31.887Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 54%]
[2021-10-28T15:40:34.584Z] .FFFFFFFF.                                                               [ 58%]
[2021-10-28T15:40:45.731Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 63%]
[2021-10-28T15:40:54.116Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 66%]
[2021-10-28T15:41:39.809Z] finch-master/docs/source/notebooks/subset.ipynb ....................     [ 75%]
[2021-10-28T15:41:41.726Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 75%]
[2021-10-28T15:41:44.788Z] ......                                                                   [ 78%]
[2021-10-28T15:42:02.946Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 78%]
[2021-10-28T15:42:19.559Z] .............                                                            [ 84%]
[2021-10-28T15:42:29.555Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2021-10-28T15:43:12.707Z] ....s.                                                                   [ 87%]
[2021-10-28T15:43:24.949Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2021-10-28T15:43:31.622Z] ...                                                                      [ 89%]
[2021-10-28T15:43:49.741Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2021-10-28T15:44:18.890Z] ......                                                                   [ 92%]
[2021-10-28T15:44:29.457Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2021-10-28T15:46:07.459Z] notebooks/stress-tests.ipynb ....                                        [100%]
[2021-10-28T15:46:07.460Z] 
[2021-10-28T15:46:07.460Z] =================================== FAILURES ===================================

…s just on the edge of completion in some cases

crim-jenkins-bot · 2021-10-28T17:55:43Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/727/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : feature/DAC-300-weaver-compose
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-4.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/643/

NOTEBOOK TEST RESULTS

    
[2021-10-28T17:39:55.451Z] ============================= test session starts ==============================
[2021-10-28T17:39:55.451Z] platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
[2021-10-28T17:39:55.451Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2021-10-28T17:39:55.451Z] plugins: anyio-3.3.0, dash-1.21.0, nbval-0.9.6, tornasync-0.6.0.post2
[2021-10-28T17:39:55.451Z] collected 221 items
[2021-10-28T17:39:55.451Z] 
[2021-10-28T17:40:03.883Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2021-10-28T17:40:24.273Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  8%]
[2021-10-28T17:40:34.350Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2021-10-28T17:40:46.578Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 14%]
[2021-10-28T17:40:49.143Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 19%]
[2021-10-28T17:42:40.661Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb F                 [ 19%]
[2021-10-28T17:42:40.661Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 19%]
[2021-10-28T17:42:45.301Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 23%]
[2021-10-28T17:42:52.159Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 25%]
[2021-10-28T17:47:20.641Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 32%]
[2021-10-28T17:48:10.970Z] ..............                                                           [ 38%]
[2021-10-28T17:48:18.723Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 40%]
[2021-10-28T17:48:20.939Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 43%]
[2021-10-28T17:48:43.957Z] .................                                                        [ 51%]
[2021-10-28T17:48:50.323Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 53%]
[2021-10-28T17:48:52.232Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 54%]
[2021-10-28T17:49:23.044Z] ......FFF.                                                               [ 58%]
[2021-10-28T17:49:32.364Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 63%]
[2021-10-28T17:49:41.211Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 66%]
[2021-10-28T17:50:27.045Z] finch-master/docs/source/notebooks/subset.ipynb ....................     [ 75%]
[2021-10-28T17:50:27.045Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 75%]
[2021-10-28T17:50:30.104Z] ......                                                                   [ 78%]
[2021-10-28T17:50:48.241Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 78%]
[2021-10-28T17:51:04.408Z] .............                                                            [ 84%]
[2021-10-28T17:51:14.448Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2021-10-28T17:52:00.995Z] ....s.                                                                   [ 87%]
[2021-10-28T17:52:10.994Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2021-10-28T17:52:18.174Z] ...                                                                      [ 89%]
[2021-10-28T17:52:36.294Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2021-10-28T17:53:03.164Z] ......                                                                   [ 92%]
[2021-10-28T17:53:15.677Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2021-10-28T17:54:54.125Z] notebooks/stress-tests.ipynb ....                                        [100%]
[2021-10-28T17:54:54.125Z] 
[2021-10-28T17:54:54.125Z] =================================== FAILURES ===================================

…rce restart weaver/worker

crim-jenkins-bot · 2021-10-29T19:06:03Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/733/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : feature/DAC-300-weaver-compose
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-91.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/645/

NOTEBOOK TEST RESULTS

    
[2021-10-29T18:38:58.074Z] ============================= test session starts ==============================
[2021-10-29T18:38:58.074Z] platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
[2021-10-29T18:38:58.074Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2021-10-29T18:38:58.074Z] plugins: anyio-3.3.0, dash-1.21.0, nbval-0.9.6, tornasync-0.6.0.post2
[2021-10-29T18:38:58.074Z] collected 221 items
[2021-10-29T18:38:58.074Z] 
[2021-10-29T18:39:07.087Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2021-10-29T18:39:36.166Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  8%]
[2021-10-29T18:40:10.837Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb .FFFF.         [ 10%]
[2021-10-29T18:40:18.585Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 14%]
[2021-10-29T18:40:21.539Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 19%]
[2021-10-29T18:40:36.456Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb .                 [ 19%]
[2021-10-29T18:40:37.032Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 19%]
[2021-10-29T18:40:42.429Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 23%]
[2021-10-29T18:40:48.589Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 25%]
[2021-10-29T18:45:03.451Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ...FF.FF.FFF.FF [ 32%]
[2021-10-29T18:45:35.603Z] FFFFFF.FFFFFFF                                                           [ 38%]
[2021-10-29T18:45:38.715Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 40%]
[2021-10-29T18:45:40.628Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 43%]
[2021-10-29T18:48:05.306Z] ....FFFFFF.FFFFFF                                                        [ 51%]
[2021-10-29T18:48:05.307Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 53%]
[2021-10-29T18:48:05.307Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 54%]
[2021-10-29T18:48:31.894Z] ......FFF.                                                               [ 58%]
[2021-10-29T18:48:41.386Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 63%]
[2021-10-29T18:48:49.429Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 66%]
[2021-10-29T18:49:43.284Z] finch-master/docs/source/notebooks/subset.ipynb .....FFFF.....FFF...     [ 75%]
[2021-10-29T18:52:04.838Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb F [ 75%]
[2021-10-29T18:54:11.390Z] FFF.F.                                                                   [ 78%]
[2021-10-29T18:56:17.914Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb F [ 78%]
[2021-10-29T18:57:04.639Z] FFF.FF.FFFFF.                                                            [ 84%]
[2021-10-29T18:59:26.262Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb F [ 85%]
[2021-10-29T19:02:19.538Z] FFFFs.                                                                   [ 87%]
[2021-10-29T19:02:26.118Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2021-10-29T19:02:33.987Z] ...                                                                      [ 89%]
[2021-10-29T19:02:48.899Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2021-10-29T19:03:21.297Z] ......                                                                   [ 92%]
[2021-10-29T19:03:33.517Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2021-10-29T19:04:56.005Z] notebooks/stress-tests.ipynb ....                                        [100%]
[2021-10-29T19:04:56.005Z] 
[2021-10-29T19:04:56.005Z] =================================== FAILURES ===================================

crim-jenkins-bot · 2021-10-29T20:33:01Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/734/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : feature/DAC-300-weaver-compose
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : false
PAVICS_HOST : https://host-140-91.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/646/

NOTEBOOK TEST RESULTS

    
[2021-10-29T20:23:42.877Z] ============================= test session starts ==============================
[2021-10-29T20:23:42.877Z] platform linux -- Python 3.7.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
[2021-10-29T20:23:42.877Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2021-10-29T20:23:42.877Z] plugins: anyio-3.3.0, dash-1.21.0, nbval-0.9.6, tornasync-0.6.0.post2
[2021-10-29T20:23:42.877Z] collected 221 items
[2021-10-29T20:23:42.877Z] 
[2021-10-29T20:23:50.932Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2021-10-29T20:24:36.837Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  8%]
[2021-10-29T20:24:40.386Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb .FFFF.         [ 10%]
[2021-10-29T20:24:50.050Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 14%]
[2021-10-29T20:24:52.675Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 19%]
[2021-10-29T20:25:07.578Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb .                 [ 19%]
[2021-10-29T20:25:08.524Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb E       [ 19%]
[2021-10-29T20:25:14.155Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 23%]
[2021-10-29T20:25:20.663Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 25%]
[2021-10-29T20:27:54.192Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ...FF.FF.FFF.FF [ 32%]
[2021-10-29T20:27:54.192Z] FFFFFF.FFFFFFF                                                           [ 38%]
[2021-10-29T20:27:57.352Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 40%]
[2021-10-29T20:27:59.532Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 43%]
[2021-10-29T20:28:04.072Z] ....FFFFFF.FFFFFF                                                        [ 51%]
[2021-10-29T20:28:10.827Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 53%]
[2021-10-29T20:28:12.227Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 54%]
[2021-10-29T20:28:28.518Z] ..........                                                               [ 58%]
[2021-10-29T20:28:38.904Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 63%]
[2021-10-29T20:28:46.688Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 66%]
[2021-10-29T20:29:11.157Z] finch-master/docs/source/notebooks/subset.ipynb .....FFFF.....FFF...     [ 75%]
[2021-10-29T20:29:12.541Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb F [ 75%]
[2021-10-29T20:29:13.930Z] FFF.F.                                                                   [ 78%]
[2021-10-29T20:29:16.487Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb F [ 78%]
[2021-10-29T20:29:19.555Z] FFF.FF.FFFFF.                                                            [ 84%]
[2021-10-29T20:29:23.738Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb F [ 85%]
[2021-10-29T20:29:23.738Z] FFFFs.                                                                   [ 87%]
[2021-10-29T20:29:33.726Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2021-10-29T20:29:40.294Z] ...                                                                      [ 89%]
[2021-10-29T20:29:55.196Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2021-10-29T20:30:23.938Z] ......                                                                   [ 92%]
[2021-10-29T20:30:36.193Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2021-10-29T20:32:14.550Z] notebooks/stress-tests.ipynb ....                                        [100%]
[2021-10-29T20:32:14.551Z] 
[2021-10-29T20:32:14.551Z] ==================================== ERRORS ====================================

# Overview Add any variation of sub-dir notebooks for artifact collection in Jenkins. For instance, following were omitted: https:/Ouranosinc/pavics-sdi/tree/master/docs/source/notebook-components At the same time, ensure following are not matched: https:/Ouranosinc/pavics-sdi/tree/master/docs/source/deprecated ## Related Issue / Discussion Useful for collecting results from bird-house/birdhouse-deploy#114

dbyrns · 2021-11-01T14:51:24Z

But once it is activated, we know it can start the first time but we have no idea how it integrate itself into the upgrade process (autodeploy) because that scenario is not tested by the daccs-iac.

@tlvu just so you know, indeed this process is not tested by the IAC, but it is tested by our staging environment that will be upgraded as soon as the PR is merged. So if Hirondelle goes well after the merge we will know everything is OK. I thought that we all agreed on that procedure. A PR is accepted based on the IAC tests. An organization deploys on prod using their fork after they check their own staging with master. An organization is not required to validate the upgrade on other organizations staging but can support them in case of documented issues.

tlvu · 2021-11-01T16:13:38Z

But once it is activated, we know it can start the first time but we have no idea how it integrate itself into the upgrade process (autodeploy) because that scenario is not tested by the daccs-iac.

@tlvu just so you know, indeed this process is not tested by the IAC, but it is tested by our staging environment that will be upgraded as soon as the PR is merged. So if Hirondelle goes well after the merge we will know everything is OK. I thought that we all agreed on that procedure. A PR is accepted based on the IAC tests. An organization deploys on prod using their fork after they check their own staging with master. An organization is not required to validate the upgrade on other organizations staging but can support them in case of documented issues.

@dbyrns Just to be sure, there are 2 aspects to an upgrade during the autodeploy: 1) the code, 2) the exiting data that might need migration.

So I understand for number 2, each org with each different production data will be in a better position to test their own data.

However for number 1, that code can be tested by the feature owner, weather it can run successfully inside the autodeploy process, at least without any errors for the most basic case where no data migration is required.

Both the autodeploy and weaver are optional components so it's not a blocking issue so I accepted this PR. But it could still introduce a regression for autodeploy and I think we all agree regression is bad.

So that's why I mentioned in comment #114 (comment) "Just indicate in the changelog clearly that autodeploy is not tested, in case another org (PCIC) wants to activate this." This is to prevent another org that already has autodeploy currently activated to be careful with activating weaver at the same time. And my comment has been ignored.

dbyrns · 2021-11-01T17:45:55Z

For the changelog, you are right, a message could have been added, surely an honest mistake.
To test the autodeploy though, I still think that the staging is the right way to validate that, even for the code part. Let's talk about that in our next meeting.

## Overview Move `mongodb` network to `mongodb` image directly and update `phoenix`/`weaver` images accordingly. ## Changes **Non-breaking changes** - Apply ``mongodb`` network to ``mongodb`` image in order to allow ``phoenix`` to properly reference it. - Remove ``mongodb`` definition from ``./components/weaver`` since the extended ``mongodb`` network is already provided. **Breaking changes** - n/a ## Related Issue / Discussion - Resolves #114 (comment)

tlvu · 2021-11-03T00:43:53Z

birdhouse/components/weaver/config/canarie-api/weaver_config.py.template

@@ -0,0 +1,9 @@
+SERVICES['node']['monitoring'].update({


@fmigneault another problem detected after go-live in prod yesterday:

This weaver_config.py should be added to .gitignore, else autodeploy is broken because it would detect an unclean checkout.

Do you want a separate PR for this fix only (with version bump and all?).

Absolutely. A hotfix also deserve a proper release to have an identifier to communicate to your users. You would do the same for Magpie I figure.

fmigneault requested review from huard, matprov, dbyrns and tlvu January 12, 2021 23:19

dbyrns reviewed Jan 13, 2021

View reviewed changes

matprov reviewed Jan 13, 2021

View reviewed changes

birdhouse/optional-components/testthredds/docker-compose-extra.yml Outdated Show resolved Hide resolved

matprov reviewed Jan 13, 2021

View reviewed changes

birdhouse/optional-components/weaver/conf.extra-service.d/weaver.conf.template Outdated Show resolved Hide resolved

matprov approved these changes Jan 13, 2021

View reviewed changes

dbyrns requested changes Jan 13, 2021

View reviewed changes

birdhouse/optional-components/weaver/default.env Outdated Show resolved Hide resolved

tlvu reviewed Jan 13, 2021

View reviewed changes

birdhouse/config/postgres/credentials.env.template Outdated Show resolved Hide resolved

tlvu requested changes Jan 13, 2021

View reviewed changes

birdhouse/config/magpie/permissions.cfg.template Show resolved Hide resolved

birdhouse/docker-compose.yml Outdated Show resolved Hide resolved

birdhouse/docker-compose.yml Show resolved Hide resolved

birdhouse/optional-components/README.md Outdated Show resolved Hide resolved

tlvu reviewed Jan 15, 2021

View reviewed changes

birdhouse/components/README.rst Outdated Show resolved Hide resolved

birdhouse/config/magpie/permissions.cfg.template Show resolved Hide resolved

tlvu requested changes Jan 15, 2021

View reviewed changes

tlvu mentioned this pull request Feb 10, 2021

secured THREDDS tests Ouranosinc/PAVICS-e2e-workflow-tests#55

Merged

Zvax and others added 14 commits February 22, 2021 15:41

basic working /processes route

7ba12e8

pushes nginx files inside proxy container

c3cb446

explicitely adds environment variables to the templating engine and n…

a7d5e5b

…otes the requirement

fixes postgres-pavics database not found psql error

47e59f0

unhardcodes various twitcher and weaver configs

4a87f1a

adds path config from environment for weaver wps

381c0d1

first pass aligning params with deploy variables

721ed5c

tag mongodb 3.6.21 + remap volumes for docker-compose 3.4

4a107d6

employ log rotation shared logging config

278714f

add DATA_PERSIST_ROOT value to override root data directory path as n…

b376620

…eeded, default to original hardcoded values

move all weaver configs under their optional-component directory

bac7b32

add server/contact/support variables that can be shared across contai…

35aece5

…ners + remove default open db ports in favor of optional-component to be explicitly defined when needed

revert mongodb 3.4

f3c6687

fix weaver twitcher nginx variable

784f00d

fmigneault requested a review from tlvu October 27, 2021 22:28

tlvu approved these changes Oct 27, 2021

View reviewed changes

double the time for weaver's wps provider timeout (120s) since 60s wa…

fba7632

…s just on the edge of completion in some cases

bird-house deleted a comment from crim-jenkins-bot Oct 28, 2021

detect error celery setup although docker-compose command is ok to fo…

d3aabec

…rce restart weaver/worker

bird-house deleted a comment from crim-jenkins-bot Oct 29, 2021

fmigneault mentioned this pull request Oct 29, 2021

collect missing notebooks artifacts in Jenkins Ouranosinc/PAVICS-e2e-workflow-tests#91

Merged

Bump version: 1.16.2 → 1.17.0

c770f9c

github-actions bot added the documentation Improvements or additions to documentation label Nov 1, 2021

fmigneault merged commit 23b5dab into master Nov 1, 2021

fmigneault deleted the feature/DAC-300-weaver-compose branch November 1, 2021 14:44

bird-house deleted a comment from crim-jenkins-bot Nov 1, 2021

fmigneault mentioned this pull request Nov 2, 2021

fix mongodb network #205

Merged

tlvu reviewed Nov 3, 2021

View reviewed changes

fmigneault mentioned this pull request Nov 3, 2021

ignore generated weaver_config under canarie-api #220

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/dac 300 weaver compose #114

Feature/dac 300 weaver compose #114

fmigneault commented Jan 12, 2021 •

edited

Loading

dbyrns left a comment

fmigneault commented Jan 13, 2021

dbyrns left a comment

tlvu left a comment

fmigneault commented Jan 14, 2021

tlvu left a comment

tlvu Jan 15, 2021

fmigneault Jan 16, 2021 •

edited

Loading

fmigneault commented Oct 27, 2021

tlvu left a comment

tlvu commented Oct 27, 2021

crim-jenkins-bot commented Oct 27, 2021

crim-jenkins-bot commented Oct 28, 2021

crim-jenkins-bot commented Oct 28, 2021

crim-jenkins-bot commented Oct 28, 2021

crim-jenkins-bot commented Oct 28, 2021

crim-jenkins-bot commented Oct 28, 2021

crim-jenkins-bot commented Oct 29, 2021

crim-jenkins-bot commented Oct 29, 2021

dbyrns commented Nov 1, 2021

tlvu commented Nov 1, 2021

dbyrns commented Nov 1, 2021

tlvu Nov 3, 2021

fmigneault Nov 3, 2021

tlvu Nov 3, 2021

Feature/dac 300 weaver compose #114

Feature/dac 300 weaver compose #114

Conversation

fmigneault commented Jan 12, 2021 • edited Loading

Dependencies

Changes

Extra

dbyrns left a comment

Choose a reason for hiding this comment

fmigneault commented Jan 13, 2021

dbyrns left a comment

Choose a reason for hiding this comment

tlvu left a comment

Choose a reason for hiding this comment

fmigneault commented Jan 14, 2021

tlvu left a comment

Choose a reason for hiding this comment

tlvu Jan 15, 2021

Choose a reason for hiding this comment

fmigneault Jan 16, 2021 • edited Loading

Choose a reason for hiding this comment

fmigneault commented Oct 27, 2021

tlvu left a comment

Choose a reason for hiding this comment

tlvu commented Oct 27, 2021

crim-jenkins-bot commented Oct 27, 2021

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Oct 28, 2021

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Oct 28, 2021

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Oct 28, 2021

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Oct 28, 2021

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Oct 28, 2021

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Oct 29, 2021

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Oct 29, 2021

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

dbyrns commented Nov 1, 2021

tlvu commented Nov 1, 2021

dbyrns commented Nov 1, 2021

tlvu Nov 3, 2021

Choose a reason for hiding this comment

fmigneault Nov 3, 2021

Choose a reason for hiding this comment

tlvu Nov 3, 2021

Choose a reason for hiding this comment

fmigneault commented Jan 12, 2021 •

edited

Loading

fmigneault Jan 16, 2021 •

edited

Loading