-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add s3 benchmark and deploy pipeline (#66)
* rename remove_blobfs_files role - remove both blobfs and kvfs files * Rename remove_blobfs role and task name * rename format_blobfs role * rename format_blobfs subtasks * add log dir var * Refactor for KVFS - put the format script in conf dir, not bin dir - put the log file in log dir, not bin dir - use abs paths for everything * KVFS enhancements - stat for dss_formatter and set var - rename substasks * rename template and refactor for kvfs * handle both blobfs and kvfs * handle legacy and new conf file names * Only parse log for errors if blobfs * save dss_target.out to log dir, not bin dir * update path of dss_target.py.out * update dss_target.py.out path * remove reference to unused gcc var for datamover * don't deploy GCC when deploying client * remove gcc from datamover * Don't deploy GCC * remove gcc defaults * Stat dss_formatter and install gcc if not present * Optionally call gcc setenv only if needed * revise start target - use abs path of target script - put target script in conf dir, not bin dir * fix gcc condition with dss_formatter - incorrectly used inverse logic * put minio scripts in conf dir, not bin dir * add format disks err file * don't run compaction if dss_formatter is present * Revise format logic - blobfs logfile now created from template script - kvfs format logfile created after format task - if kvfs format fails - no logfile - missing logfile used for criteria for future format decision - this accommodates dss_formatter returns non-0 on failed format * Cleanup start_dss_host - put script in /etc/dss not bin dir - output logs to /var/log/dss - don't needlessly loop through dss_host.py tasks * Output logs to /var/log/dss * execute spdk setup script by abs path - don't output log if empty * remove correct paths during format * execute spdk script directly, not dss_Target.py reset * don't run compaction if dss_formatter * remove chdir args * allow dss_target_config.sh script to be called anywhere - no chdir needed * set compaction default to 'no' - revert compaction string on false condition * remove target conf dir on uninstall * Only check blobfs format log if the format file output exists * Don't check if file exists, we must assume it exists if we are runnign this check * Add deploy stage to dss-ansible
- Loading branch information
Showing
69 changed files
with
1,080 additions
and
355 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,8 @@ | ||
| # Internal testing inventory files | ||
| inv_* | ||
| .vscode/** | ||
|
|
||
| # VS Code config | ||
| .vscode | ||
|
|
||
| # JUNIT XML Test Results | ||
| *.xml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,21 +1,9 @@ | ||
| variables: | ||
| BRANCH_NAME: $CI_COMMIT_BRANCH | ||
| SONAR_BRANCH: -Dsonar.branch.name=$CI_COMMIT_BRANCH | ||
|
|
||
| image: | ||
| name: dss-build_$BRANCH_NAME | ||
|
|
||
| workflow: | ||
| rules: | ||
| - if: $CI_PIPELINE_SOURCE == "merge_request_event" | ||
| variables: | ||
| BRANCH_NAME: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME | ||
| - if: $CI_COMMIT_BRANCH == "master" && $CI_PIPELINE_SOURCE == "push" | ||
| - if: $CI_COMMIT_BRANCH =~ /^(stable|feature)\/.*/ && $CI_PIPELINE_SOURCE == "push" | ||
| include: | ||
| - .gitlab/defaults.yml | ||
|
|
||
| stages: | ||
| - build | ||
| - lint | ||
|
|
||
| ansible-lint: | ||
| stage: lint | ||
| script: ansible-lint * | ||
| - deploy | ||
| - test | ||
| - sync |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| deploy DSS: | ||
| stage: deploy | ||
| image: | ||
| name: $CI_REGISTRY/$ANSIBLE_PROJECT_PATH/$BRANCH_NAME:$DOCKERFILE_NAME | ||
| pull_policy: always | ||
| environment: | ||
| name: $ANSIBLE_INVENTORY | ||
| url: $CI_SERVER_URL/dfs/dss/dss-ansible/-/blob/inventory/$ANSIBLE_INVENTORY | ||
| before_script: | ||
| # Clone ansible repo | ||
| - git config --global http.sslVerify false | ||
| - git config --global user.name "$CI_USERNAME" | ||
| - git config --global user.email "$CI_EMAIL" | ||
| - git clone https://$CI_USERNAME:$CI_TOKEN@$CI_SERVER_HOST/$ANSIBLE_PROJECT_PATH.git --branch $ANSIBLE_BRANCH ../dss-ansible | ||
| - cd ../dss-ansible | ||
| # Get inventory file | ||
| - git fetch origin inventory | ||
| - git restore --source origin/inventory -- $ANSIBLE_INVENTORY | ||
| # Hack to disregard task output from JUNIT callback module | ||
| - sed -i -E "s/dump =.+/dump = ''/g" /usr/local/lib/python3.11/site-packages/ansible/plugins/callback/junit.py | ||
| script: | ||
| - | | ||
| ansible-playbook -i $ANSIBLE_INVENTORY playbooks/download_artifacts.yml \ | ||
| -e "download_artifacts=true" \ | ||
| -e "artifacts_url=$MINIO_HOST_URL/dss-artifacts" \ | ||
| -e "artifacts_branch=$BRANCH_NAME" | ||
| - ansible-playbook -i $ANSIBLE_INVENTORY playbooks/remove_dss_software.yml | ||
| - ansible-playbook -i $ANSIBLE_INVENTORY playbooks/deploy_dss_software.yml | ||
| artifacts: | ||
| when: always | ||
| reports: | ||
| junit: "*.xml" | ||
| variables: | ||
| ANSIBLE_PROJECT_PATH: dfs/dss/dss-ansible | ||
| ANSIBLE_BRANCH: MIN-2148-add-deploy-stage | ||
| GIT_STRATEGY: none | ||
| DOCKERFILE_NAME: rocky8 | ||
| ANSIBLE_CONFIG: ../dss-ansible/ansible.cfg | ||
| ANSIBLE_INVENTORY: inv_$CI_PROJECT_NAME.ini | ||
| ANSIBLE_FORCE_COLOR: "true" | ||
| JUNIT_OUTPUT_DIR: $CI_PROJECT_DIR | ||
| JUNIT_TASK_CLASS: "yes" | ||
| JUNIT_INCLUDE_SETUP_TASKS_IN_REPORT: "no" | ||
| ANSIBLE_CALLBACK_WHITELIST: junit | ||
| rules: | ||
| - !reference [.default_rules, merge_and_push] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| build docker: | ||
| stage: build | ||
| image: docker:25.0.3-git | ||
| variables: | ||
| ANSIBLE_PROJECT_PATH: dfs/dss/dss-ansible | ||
| ANSIBLE_BRANCH: MIN-2148-add-deploy-stage | ||
| GIT_STRATEGY: none | ||
| DOCKERFILE_NAME: rocky8 | ||
| DOCKERFILE_PATH: scripts/docker/$DOCKERFILE_NAME.DOCKERFILE | ||
| # IMAGE_TAG: $CI_REGISTRY_IMAGE/$BRANCH_NAME:$DOCKERFILE_NAME | ||
| IMAGE_TAG: $CI_REGISTRY/$ANSIBLE_PROJECT_PATH/$BRANCH_NAME:$DOCKERFILE_NAME | ||
| CACHE_TAG: ${IMAGE_TAG}-cache | ||
| before_script: | ||
| # Clone dss-ansible repo | ||
| - git config --global http.sslVerify false | ||
| - git config --global user.name "$CI_USERNAME" | ||
| - git config --global user.email "$CI_EMAIL" | ||
| - git clone https://$CI_USERNAME:$CI_TOKEN@$CI_SERVER_HOST/$ANSIBLE_PROJECT_PATH.git --branch $ANSIBLE_BRANCH . | ||
| # Install certs so buildkit can access Gitlab container registry | ||
| - echo "$SSI_ROOTCA_CERT" > /usr/local/share/ca-certificates/SSI-RootCA.crt | ||
| - echo "$SSI_ISSUINGCA_CERT" > /usr/local/share/ca-certificates/SSI-ISSUINGCA.crt | ||
| - echo "$MSL_ETX_CERT" > /usr/local/share/ca-certificates/msl-etx.samsung.com.crt | ||
| - update-ca-certificates --fresh > /dev/null | ||
| # Configure buildkitd.toml to use newly-installed certs | ||
| - | | ||
| cat <<EOF > /buildkitd.toml | ||
| [registry."$CI_REGISTRY"] | ||
| ca=["/etc/ssl/certs/ca-certificates.crt"] | ||
| EOF | ||
| # Initialize buildkit with custom config | ||
| - docker buildx create --driver=docker-container --name=buildkit-builder --use --config /buildkitd.toml | ||
| # Login to Gitlab container registry | ||
| - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY | ||
| script: | ||
| - | | ||
| docker buildx build \ | ||
| --cache-from type=registry,ref=$CACHE_TAG \ | ||
| --cache-to type=registry,ref=$CACHE_TAG \ | ||
| --push \ | ||
| --tag $IMAGE_TAG \ | ||
| --file $DOCKERFILE_PATH . \ | ||
| --provenance false | ||
| rules: | ||
| - !reference [.default_rules, merge_and_push] | ||
| - if: '$CI_PIPELINE_SOURCE == "parent_pipeline"' | ||
| when: always | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| include: | ||
| - project: dfs/dss/dss | ||
| ref: master | ||
| file: .gitlab/defaults.yml | ||
| - project: dfs/dss/dss | ||
| ref: master | ||
| file: .gitlab/sync-github.yml | ||
| - .gitlab/lint.yml | ||
| - .gitlab/build.yml | ||
| - .gitlab/deploy.yml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| include: .gitlab/ansible.yml | ||
|
|
||
| deploy DSS with upstream dss-sdk artifacts: | ||
| extends: deploy DSS | ||
| stage: deploy | ||
| script: | ||
| - | | ||
| ansible-playbook -i $ANSIBLE_INVENTORY playbooks/download_artifacts.yml \ | ||
| -e "download_artifacts=true" \ | ||
| -e "artifacts_url=$MINIO_HOST_URL/dss-artifacts" \ | ||
| -e "artifacts_branch=$BRANCH_NAME" | ||
| - rm -f artifacts/nkv-target* | ||
| - rm -f artifacts/nkv-sdk-bin* | ||
| - cp $CI_PROJECT_DIR/df_out/nkv-target-*.tgz artifacts/ | ||
| - cp $CI_PROJECT_DIR/host_out/nkv-sdk-bin-*.tgz artifacts/ | ||
| - ansible-playbook -i $ANSIBLE_INVENTORY playbooks/remove_dss_software.yml | ||
| - ansible-playbook -i $ANSIBLE_INVENTORY playbooks/deploy_dss_software.yml | ||
| - ansible-playbook -i $ANSIBLE_INVENTORY playbooks/test_nkv_test_cli.yml -e nkv_test_cli_test=suite -e nkv_test_cli_suite=suite003 | ||
| - ansible-playbook -i $ANSIBLE_INVENTORY playbooks/test_s3_benchmark.yml | ||
| needs: | ||
| - build docker | ||
| - project: dfs/dss/dss-sdk | ||
| job: build dss-sdk | ||
| ref: $UPSTREAM_REF | ||
| artifacts: true | ||
| rules: | ||
| - if: $CI_PIPELINE_SOURCE == "parent_pipeline" && $CI_MERGE_REQUEST_SOURCE_PROJECT_PATH == "dfs/dss/dss-sdk" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| ansible-lint: | ||
| stage: lint | ||
| script: ansible-lint * | ||
| needs: [] | ||
| rules: | ||
| - !reference [.default_rules, merge_and_push] | ||
| - if: '$CI_PIPELINE_SOURCE == "parent_pipeline"' | ||
| when: never |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.