add components representation and JSON response endpoint #281

fmigneault · 2023-01-25T23:31:35Z

Overview

New endpoint for reporting platform components.

Changes

Non-breaking changes

Proxy: add /components endpoint that provides a JSON list of loaded components by the platform.

Prior to this functionality, it was impossible to know which potential capabilities, services or behaviors were to be
expected by a given DACCS/PAVICS/birdhouse instance. Using this endpoint, nodes can obtain minimal machine-readable
details about their supported capabilities, allowing better interoperability.

Furthermore, developers maintaining distinct stacks and integrating different features can have a better
understanding of behaviors by the various web services when performing requests against a given node.

Breaking changes

n/a (if the operation fails to load/transform components into JSON, the list will simply be empty - silent failure)

Related Issue / Discussion

n/a

Additional Information

The sample JSON output in the script is the actual response received when requesting /components on a test instance.

crim-jenkins-bot · 2023-01-26T00:15:28Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac/40/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : add-json-components-endpoint
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/930/

NOTEBOOK TEST RESULTS

    
[2023-01-25T23:39:28.988Z] ============================= test session starts ==============================
[2023-01-25T23:39:28.988Z] platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
[2023-01-25T23:39:28.988Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-01-25T23:39:28.989Z] plugins: anyio-3.6.2, dash-2.7.0, nbval-0.9.6, tornasync-0.6.0.post2
[2023-01-25T23:39:28.989Z] collected 263 items
[2023-01-25T23:39:28.989Z] 
[2023-01-25T23:39:37.472Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-01-25T23:39:47.134Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  6%]
[2023-01-25T23:39:55.266Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  9%]
[2023-01-25T23:40:04.969Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 12%]
[2023-01-25T23:40:07.756Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 15%]
[2023-01-25T23:40:29.461Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb FFFFFFFFFFF.        [ 20%]
[2023-01-25T23:40:31.388Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 20%]
[2023-01-25T23:40:38.431Z] ...............                                                          [ 26%]
[2023-01-25T23:40:47.966Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 28%]
[2023-01-25T23:41:00.304Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb F                 [ 28%]
[2023-01-25T23:41:16.134Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 31%]
[2023-01-25T23:41:17.522Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 31%]
[2023-01-25T23:41:23.898Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 34%]
[2023-01-25T23:41:29.057Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb ....F       [ 36%]
[2023-01-25T23:45:14.384Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 41%]
[2023-01-25T23:46:23.875Z] .............                                                            [ 46%]
[2023-01-25T23:46:35.573Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 48%]
[2023-01-25T23:46:38.877Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 51%]
[2023-01-25T23:47:01.308Z] .................                                                        [ 57%]
[2023-01-25T23:47:08.802Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 59%]
[2023-01-25T23:47:10.728Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2023-01-25T23:47:32.766Z] .........                                                                [ 63%]
[2023-01-25T23:47:43.936Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 67%]
[2023-01-25T23:47:54.106Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 69%]
[2023-01-25T23:48:24.711Z] finch-master/docs/source/notebooks/subset.ipynb ....................ss.  [ 78%]
[2023-01-25T23:48:26.634Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2023-01-25T23:48:30.209Z] ......                                                                   [ 81%]
[2023-01-25T23:48:56.879Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 81%]
[2023-01-25T23:49:14.479Z] .............                                                            [ 86%]
[2023-01-25T23:49:24.501Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2023-01-25T23:50:02.617Z] ....s.                                                                   [ 89%]
[2023-01-25T23:50:15.029Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-01-25T23:50:19.725Z] ...                                                                      [ 90%]
[2023-01-25T23:50:37.871Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2023-01-25T23:51:06.718Z] ......                                                                   [ 93%]
[2023-01-25T23:51:17.568Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2023-01-25T23:54:52.873Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-01-25T23:54:52.873Z] 
[2023-01-25T23:54:52.873Z] =================================== FAILURES ===================================

crim-jenkins-bot · 2023-01-26T00:15:28Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1145/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : add-json-components-endpoint
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/930/

NOTEBOOK TEST RESULTS

    
[2023-01-25T23:39:28.988Z] ============================= test session starts ==============================
[2023-01-25T23:39:28.988Z] platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
[2023-01-25T23:39:28.988Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-01-25T23:39:28.989Z] plugins: anyio-3.6.2, dash-2.7.0, nbval-0.9.6, tornasync-0.6.0.post2
[2023-01-25T23:39:28.989Z] collected 263 items
[2023-01-25T23:39:28.989Z] 
[2023-01-25T23:39:37.472Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-01-25T23:39:47.134Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  6%]
[2023-01-25T23:39:55.266Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  9%]
[2023-01-25T23:40:04.969Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 12%]
[2023-01-25T23:40:07.756Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 15%]
[2023-01-25T23:40:29.461Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb FFFFFFFFFFF.        [ 20%]
[2023-01-25T23:40:31.388Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 20%]
[2023-01-25T23:40:38.431Z] ...............                                                          [ 26%]
[2023-01-25T23:40:47.966Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 28%]
[2023-01-25T23:41:00.304Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb F                 [ 28%]
[2023-01-25T23:41:16.134Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 31%]
[2023-01-25T23:41:17.522Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 31%]
[2023-01-25T23:41:23.898Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 34%]
[2023-01-25T23:41:29.057Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb ....F       [ 36%]
[2023-01-25T23:45:14.384Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 41%]
[2023-01-25T23:46:23.875Z] .............                                                            [ 46%]
[2023-01-25T23:46:35.573Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 48%]
[2023-01-25T23:46:38.877Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 51%]
[2023-01-25T23:47:01.308Z] .................                                                        [ 57%]
[2023-01-25T23:47:08.802Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 59%]
[2023-01-25T23:47:10.728Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2023-01-25T23:47:32.766Z] .........                                                                [ 63%]
[2023-01-25T23:47:43.936Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 67%]
[2023-01-25T23:47:54.106Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 69%]
[2023-01-25T23:48:24.711Z] finch-master/docs/source/notebooks/subset.ipynb ....................ss.  [ 78%]
[2023-01-25T23:48:26.634Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2023-01-25T23:48:30.209Z] ......                                                                   [ 81%]
[2023-01-25T23:48:56.879Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 81%]
[2023-01-25T23:49:14.479Z] .............                                                            [ 86%]
[2023-01-25T23:49:24.501Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2023-01-25T23:50:02.617Z] ....s.                                                                   [ 89%]
[2023-01-25T23:50:15.029Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-01-25T23:50:19.725Z] ...                                                                      [ 90%]
[2023-01-25T23:50:37.871Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2023-01-25T23:51:06.718Z] ......                                                                   [ 93%]
[2023-01-25T23:51:17.568Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2023-01-25T23:54:52.873Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-01-25T23:54:52.873Z] 
[2023-01-25T23:54:52.873Z] =================================== FAILURES ===================================

tlvu

The idea to show the various capabilities to help interoperability is very useful.

If we keep the current form, rename the included script.

But in the current form, maybe some private config might leak out.

Maybe each component can declare how it wants to be shown instead so then it can decide to not show anything as well, if it is a private component.

birdhouse/pavics-compose.sh

birdhouse/scripts/get-components-json.sh

huard

I like the idea, can't really judge how it's done.

crim-jenkins-bot · 2023-01-31T17:12:26Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac/49/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : add-json-components-endpoint
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-36.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/939/

NOTEBOOK TEST RESULTS

    
[2023-01-31T16:15:55.868Z] ============================= test session starts ==============================
[2023-01-31T16:15:55.868Z] platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
[2023-01-31T16:15:55.868Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-01-31T16:15:55.868Z] plugins: anyio-3.6.2, dash-2.7.0, nbval-0.9.6, tornasync-0.6.0.post2
[2023-01-31T16:15:55.868Z] collected 263 items
[2023-01-31T16:15:55.868Z] 
[2023-01-31T16:16:06.469Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-01-31T16:16:40.980Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  6%]
[2023-01-31T16:16:52.370Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  9%]
[2023-01-31T16:17:00.438Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 12%]
[2023-01-31T16:17:03.342Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 15%]
[2023-01-31T16:27:59.921Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ......F.....        [ 20%]
[2023-01-31T16:28:05.897Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 20%]
[2023-01-31T16:28:21.888Z] ...............                                                          [ 26%]
[2023-01-31T16:28:58.517Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb Cannot contact linux1: java.lang.InterruptedException
[2023-01-31T16:29:06.148Z] .....    [ 28%]
[2023-01-31T16:29:06.148Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb F                 [ 28%]
[2023-01-31T16:29:17.893Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 31%]
[2023-01-31T16:29:19.652Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 31%]
[2023-01-31T16:29:25.770Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 34%]
[2023-01-31T16:29:32.058Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb ....F       [ 36%]
[2023-01-31T16:33:08.222Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 41%]
[2023-01-31T16:34:20.892Z] .............                                                            [ 46%]
[2023-01-31T16:34:23.759Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 48%]
[2023-01-31T16:34:27.116Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 51%]
[2023-01-31T16:34:53.909Z] .................                                                        [ 57%]
[2023-01-31T16:35:00.341Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 59%]
[2023-01-31T16:35:02.253Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2023-01-31T16:35:18.385Z] .........                                                                [ 63%]
[2023-01-31T16:35:30.677Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 67%]
[2023-01-31T16:35:41.297Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 69%]
[2023-01-31T16:36:13.501Z] finch-master/docs/source/notebooks/subset.ipynb ....................ss.  [ 78%]
[2023-01-31T16:36:14.902Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2023-01-31T16:36:18.472Z] ......                                                                   [ 81%]
[2023-01-31T16:36:40.495Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb F [ 81%]
[2023-01-31T16:36:56.821Z] F.F..........                                                            [ 86%]
[2023-01-31T16:37:06.835Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb F [ 87%]
[2023-01-31T16:37:39.586Z] ....s.                                                                   [ 89%]
[2023-01-31T16:37:51.821Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-01-31T16:37:56.998Z] ...                                                                      [ 90%]
[2023-01-31T16:38:19.774Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2023-01-31T16:38:45.948Z] ......                                                                   [ 93%]
[2023-01-31T16:38:49.804Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2023-01-31T16:42:24.828Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-01-31T16:42:24.830Z] 
[2023-01-31T16:42:24.830Z] =================================== FAILURES ===================================

crim-jenkins-bot · 2023-01-31T17:12:26Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1154/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : add-json-components-endpoint
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-36.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/939/

NOTEBOOK TEST RESULTS

    
[2023-01-31T16:15:55.868Z] ============================= test session starts ==============================
[2023-01-31T16:15:55.868Z] platform linux -- Python 3.8.15, pytest-7.2.0, pluggy-1.0.0
[2023-01-31T16:15:55.868Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2023-01-31T16:15:55.868Z] plugins: anyio-3.6.2, dash-2.7.0, nbval-0.9.6, tornasync-0.6.0.post2
[2023-01-31T16:15:55.868Z] collected 263 items
[2023-01-31T16:15:55.868Z] 
[2023-01-31T16:16:06.469Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-01-31T16:16:40.980Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  6%]
[2023-01-31T16:16:52.370Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  9%]
[2023-01-31T16:17:00.438Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 12%]
[2023-01-31T16:17:03.342Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 15%]
[2023-01-31T16:27:59.921Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ......F.....        [ 20%]
[2023-01-31T16:28:05.897Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 20%]
[2023-01-31T16:28:21.888Z] ...............                                                          [ 26%]
[2023-01-31T16:28:58.517Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb Cannot contact linux1: java.lang.InterruptedException
[2023-01-31T16:29:06.148Z] .....    [ 28%]
[2023-01-31T16:29:06.148Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb F                 [ 28%]
[2023-01-31T16:29:17.893Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 31%]
[2023-01-31T16:29:19.652Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 31%]
[2023-01-31T16:29:25.770Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 34%]
[2023-01-31T16:29:32.058Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb ....F       [ 36%]
[2023-01-31T16:33:08.222Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 41%]
[2023-01-31T16:34:20.892Z] .............                                                            [ 46%]
[2023-01-31T16:34:23.759Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 48%]
[2023-01-31T16:34:27.116Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 51%]
[2023-01-31T16:34:53.909Z] .................                                                        [ 57%]
[2023-01-31T16:35:00.341Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 59%]
[2023-01-31T16:35:02.253Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2023-01-31T16:35:18.385Z] .........                                                                [ 63%]
[2023-01-31T16:35:30.677Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 67%]
[2023-01-31T16:35:41.297Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 69%]
[2023-01-31T16:36:13.501Z] finch-master/docs/source/notebooks/subset.ipynb ....................ss.  [ 78%]
[2023-01-31T16:36:14.902Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2023-01-31T16:36:18.472Z] ......                                                                   [ 81%]
[2023-01-31T16:36:40.495Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb F [ 81%]
[2023-01-31T16:36:56.821Z] F.F..........                                                            [ 86%]
[2023-01-31T16:37:06.835Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb F [ 87%]
[2023-01-31T16:37:39.586Z] ....s.                                                                   [ 89%]
[2023-01-31T16:37:51.821Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-01-31T16:37:56.998Z] ...                                                                      [ 90%]
[2023-01-31T16:38:19.774Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2023-01-31T16:38:45.948Z] ......                                                                   [ 93%]
[2023-01-31T16:38:49.804Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2023-01-31T16:42:24.828Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-01-31T16:42:24.830Z] 
[2023-01-31T16:42:24.830Z] =================================== FAILURES ===================================

crim-jenkins-bot · 2023-01-31T18:18:13Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac/52/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : add-json-components-endpoint
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-36.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS

    
</code>

crim-jenkins-bot · 2023-01-31T18:18:19Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1157/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : add-json-components-endpoint
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-36.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS

    
</code>

tlvu · 2023-02-01T19:49:55Z

Maybe each component can declare how it wants to be shown instead so then it can decide to not show anything as well, if it is a private component.

I think I have a cheap solution to this: add a new var EXTRA_CONF_DIRS_PRIVATE. So any components that want to stay private goes there so anything in the original EXTRA_CONF_DIRS can freely be public.

fmigneault · 2023-02-02T00:23:58Z

@tlvu

I think I have a cheap solution to this: add a new var EXTRA_CONF_DIRS_PRIVATE. So any components that want to stay private goes there so anything in the original EXTRA_CONF_DIRS can freely be public.

We can either filter out EXTRA_CONF_DIRS with items in EXTRA_CONF_DIRS_PRIVATE, or work the other way around, include EXTRA_CONF_DIRS_PRIVATE in EXTRA_CONF_DIRS after get-components-json.include.sh is called. The first doesn't modify EXTRA_CONF_DIRS, but the second method avoids possible mismatching issues. Any approach you prefer?

crim-jenkins-bot · 2023-02-02T04:55:29Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1166/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : add-json-components-endpoint
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-88.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS

    
</code>

crim-jenkins-bot · 2023-02-02T05:00:40Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1167/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : add-json-components-endpoint
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-88.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS

    
</code>

tlvu

After 122f196 I still see my private components in external repos listed with their full path.

$ curl --silent https://medus.ouranos.ca/components/
{"components": ["bird-house/birdhouse-deploy:optional-components/canarie-api-full-monitoring",
"bird-house/birdhouse-deploy:optional-components/wps-healthchecks",
"bird-house/birdhouse-deploy:optional-components/testthredds",
"bird-house/birdhouse-deploy:optional-components/generic_bird",
"bird-house/birdhouse-deploy:components/weaver",
"bird-house/birdhouse-deploy:components/scheduler",
"bird-house/birdhouse-deploy:components/monitoring",
"/path/to/repo1/config-dir",
"/path/to/repo2/config-dir"]}

Maybe you want to add some absolute path in your env.local for testing.

crim-jenkins-bot · 2023-02-02T19:50:40Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1169/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : add-json-components-endpoint
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-46.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS

    
</code>

tlvu

This works on my staging machine.

However, chdir is a typo and please do not exit directly in this script to avoid leaving the entire platform down during unattended autodeploy.

birdhouse/scripts/get-components-json.include.sh

crim-jenkins-bot · 2023-02-03T04:51:42Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1171/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : add-json-components-endpoint
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-46.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS

    
</code>

crim-jenkins-bot · 2023-02-03T04:55:19Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1172/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : add-json-components-endpoint
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-46.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS

    
</code>

tlvu

Please do not use exit in this script and no need for any cd.

birdhouse/scripts/get-components-json.include.sh

tlvu · 2023-02-03T04:59:06Z

birdhouse/scripts/get-components-json.include.sh

+fi
+
+# resolve path considering if sourced or executed, and whether from current dir, pavics-compose include or another dir
+BIRDHOUSE_DEPLOY_COMPONENTS_ROOT=$(dirname -- "$(realpath "$0")")


I am not sure why you need to discover BIRDHOUSE_DEPLOY_COMPONENTS_ROOT. Since this script is being source by pavics-compose.sh its PWD will always be the same location of pavics-compose.sh since pavics-compose.sh needs to enforce that so it can find the docker-compose.yml.

So no need to perform any cd in the first place.

Just to avoid getting this issue if we ever end up calling it instead of sourcing it. This is the actual reason why it didn't filter your absolute paths the first time, while it was working on my side. I was sourcing it directly from another dir to test.

Ah ok. Just curious, in what other scenario would you want to just source this one outside of pavics-compose.sh?

In fact, testing it outside of pavics-compose.sh is dangerous since you will forget to test it with pavics-compose.sh !

I tested both to be sure afterward, but I prefer to test it directly than having to go through pavics-compose and quickly breaking out of it. Anyway, they should both work for now and simply remain empty if anything fails.

crim-jenkins-bot · 2023-02-03T05:02:02Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1173/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : add-json-components-endpoint
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-46.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS

    
</code>

…SON resolution on error

crim-jenkins-bot · 2023-02-03T05:18:57Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1174/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : add-json-components-endpoint
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-46.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS

    
</code>

birdhouse/scripts/get-components-json.include.sh

tlvu · 2023-02-07T18:13:30Z

@fmigneault I found out on pour production curl https://pavics.ouranos.ca/components/ is empty, it was deployed by the autodeploy. Have you tested this feature under the autodeploy environment?

fmigneault · 2023-02-08T18:14:16Z

@tlvu

I found out on pour production curl pavics.ouranos.ca/components is empty, it was deployed by the autodeploy. Have you tested this feature under the autodeploy environment?

No. We are not using autodeploy anywhere on our side.
Is it possibly related to a similar problem as identified in #288?
The components script only uses EXTRA_CONF_DIRS, so it must not be properly defined only in that situation, as I expect your prod to have many components enabled.

fmigneault added 2 commits January 25, 2023 18:18

add components representation and JSON response endpoint

0c667a4

update changelog

c21fe2d

fmigneault requested review from huard, matprov and tlvu January 25, 2023 23:31

fmigneault self-assigned this Jan 25, 2023

tlvu requested changes Jan 26, 2023

View reviewed changes

birdhouse/pavics-compose.sh Outdated Show resolved Hide resolved

birdhouse/scripts/get-components-json.sh Outdated Show resolved Hide resolved

birdhouse/scripts/get-components-json.sh Outdated Show resolved Hide resolved

huard approved these changes Jan 30, 2023

View reviewed changes

rename script with include indication + change permissions

b24a7e5

fmigneault requested a review from tlvu January 31, 2023 16:07

Merge branch 'master' into add-json-components-endpoint

8f934b8

filter out enabled EXTRA_CONF_DIRS components to birdhouse only

122f196

fmigneault force-pushed the add-json-components-endpoint branch from 1efb04b to 122f196 Compare February 2, 2023 04:59

tlvu requested changes Feb 2, 2023

View reviewed changes

fix path resolution for sourced component JSON representation

edd94b1

fmigneault requested a review from tlvu February 2, 2023 19:49

tlvu approved these changes Feb 2, 2023

View reviewed changes

birdhouse/scripts/get-components-json.include.sh Outdated Show resolved Hide resolved

birdhouse/scripts/get-components-json.include.sh Outdated Show resolved Hide resolved

replace cd + add check for error handling

be664b9

update message and exit 0 on failed known component resolution

462dabd

tlvu requested changes Feb 3, 2023

View reviewed changes

Bump version: 1.22.10 → 1.22.11

d0d5c98

github-actions bot added ci/operations Continuous Integration components documentation Improvements or additions to documentation labels Feb 3, 2023

replace exit by return to resume pavics-compose sourcing components J…

180ac46

…SON resolution on error

fmigneault requested a review from tlvu February 3, 2023 05:17

tlvu requested changes Feb 3, 2023

View reviewed changes

birdhouse/scripts/get-components-json.include.sh Show resolved Hide resolved

birdhouse/scripts/get-components-json.include.sh Show resolved Hide resolved

tlvu approved these changes Feb 3, 2023

View reviewed changes

fmigneault merged commit fe9a633 into master Feb 3, 2023

fmigneault deleted the add-json-components-endpoint branch February 3, 2023 23:50

fmigneault added a commit that referenced this pull request Feb 10, 2023

remove obsolete script file included in #281

e45c0b6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add components representation and JSON response endpoint #281

add components representation and JSON response endpoint #281

fmigneault commented Jan 25, 2023 •

edited

Loading

crim-jenkins-bot commented Jan 26, 2023

crim-jenkins-bot commented Jan 26, 2023

tlvu left a comment

huard left a comment

crim-jenkins-bot commented Jan 31, 2023

crim-jenkins-bot commented Jan 31, 2023

crim-jenkins-bot commented Jan 31, 2023

crim-jenkins-bot commented Jan 31, 2023

tlvu commented Feb 1, 2023

fmigneault commented Feb 2, 2023

crim-jenkins-bot commented Feb 2, 2023

crim-jenkins-bot commented Feb 2, 2023

tlvu left a comment

crim-jenkins-bot commented Feb 2, 2023

tlvu left a comment

crim-jenkins-bot commented Feb 3, 2023

crim-jenkins-bot commented Feb 3, 2023

tlvu left a comment

tlvu Feb 3, 2023

fmigneault Feb 3, 2023

tlvu Feb 3, 2023

fmigneault Feb 3, 2023 •

edited

Loading

crim-jenkins-bot commented Feb 3, 2023

crim-jenkins-bot commented Feb 3, 2023

tlvu commented Feb 7, 2023

fmigneault commented Feb 8, 2023

add components representation and JSON response endpoint #281

add components representation and JSON response endpoint #281

Conversation

fmigneault commented Jan 25, 2023 • edited Loading

Overview

Changes

Related Issue / Discussion

Additional Information

crim-jenkins-bot commented Jan 26, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Jan 26, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

tlvu left a comment

Choose a reason for hiding this comment

huard left a comment

Choose a reason for hiding this comment

crim-jenkins-bot commented Jan 31, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Jan 31, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Jan 31, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Jan 31, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

tlvu commented Feb 1, 2023

fmigneault commented Feb 2, 2023

crim-jenkins-bot commented Feb 2, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Feb 2, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

tlvu left a comment

Choose a reason for hiding this comment

crim-jenkins-bot commented Feb 2, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

tlvu left a comment

Choose a reason for hiding this comment

crim-jenkins-bot commented Feb 3, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Feb 3, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

tlvu left a comment

Choose a reason for hiding this comment

tlvu Feb 3, 2023

Choose a reason for hiding this comment

fmigneault Feb 3, 2023

Choose a reason for hiding this comment

tlvu Feb 3, 2023

Choose a reason for hiding this comment

fmigneault Feb 3, 2023 • edited Loading

Choose a reason for hiding this comment

crim-jenkins-bot commented Feb 3, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

crim-jenkins-bot commented Feb 3, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

fmigneault commented Jan 25, 2023 •

edited

Loading

fmigneault Feb 3, 2023 •

edited

Loading