Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release pipelines should pin their image versions #2197

Closed
19 tasks done
AlanGreene opened this issue Sep 19, 2024 · 7 comments
Closed
19 tasks done

Release pipelines should pin their image versions #2197

AlanGreene opened this issue Sep 19, 2024 · 7 comments
Labels
kind/misc Categorizes issue or PR as a miscellaneuous one.

Comments

@AlanGreene
Copy link
Member

AlanGreene commented Sep 19, 2024

All nightly releases started failing on September 19th due to an issue with the koparse image, see fix in #2196

The nightly releases picked up the bad image as they were using the latest tag.

All images should be pinned using an appropriate image digest to ensure repeatable builds and to avoid picking up unexpected changes. From a quick review of the first few nightly releases many projects are using the latest tag of at least the ko, ko-gcloud, and koparse images.

List of releases in the tekton-nightly namespace:

Taking the most recent one as an example, these are the unpinned images used, most are using latest:

wait-task

task step image
git-clone clone gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init:v0.18.1 (should specify digest too)
unit-tests unit-test docker.io/library/golang:latest
build-tests build docker.io/library/golang:latest
publish-images create-ko-yaml busybox
publish-images container-registry-auth gcr.io/go-containerregistry/crane:debug
publish-images run-ko gcr.io/tekton-releases/dogfooding/ko:latest
publish-images koparse gcr.io/tekton-releases/dogfooding/koparse:latest
publish-images tag-images gcr.io/go-containerregistry/crane:debug
report-bucket create-results alpine
@AlanGreene AlanGreene added the kind/misc Categorizes issue or PR as a miscellaneuous one. label Sep 19, 2024
@AlanGreene AlanGreene changed the title Nightly releases should pin their image versions Nightly release pipelines should pin their image versions Sep 22, 2024
@AlanGreene AlanGreene changed the title Nightly release pipelines should pin their image versions Release pipelines should pin their image versions Sep 26, 2024
@AlanGreene
Copy link
Member Author

AlanGreene commented Sep 26, 2024

Dashboard and Pipelines nightly releases are passing again, updated to use the latest koparse image, and now publishing their images to ghcr.io.

Andrea also has a PR open against Triggers to update it to the latest koparse and move to ghcr.io.

PRs open for other projects, custom tasks, interceptors, etc. to pin images to known working state.

@AlanGreene
Copy link
Member Author

AlanGreene commented Sep 27, 2024

Triggers nightly release is passing again, also updated to the latest koparse image and publishing images to ghcr.io

The following nightly releases are also passing again:

  • Operator
  • cloudevents
  • concurrency
  • pipeline-in-pod
  • pipeline-to-taskrun
  • pipelines-in-pipelines
  • task-loops
  • wait-task
  • workflows

Chains nightly is failing due to a known bad test: tektoncd/chains#1178

cel nightly is failing due to wrong go version:

2024-09-27T02:08:39.542964111Z Error: error processing import paths in "/workspace/go/src/github.com/tektoncd/experimental/cel/config/500-controller.yaml": error resolving image references: build: go build: exit status 1: # [golang.org/x/sys/unix](http://golang.org/x/sys/unix)
2024-09-27T02:08:39.543058620Z vendor/golang.org/x/sys/unix/syscall.go:83:16: unsafe.Slice requires go1.17 or later (-lang was set to go1.16; check go.mod)

add-team-members, add-pr-body, and add-pr-body-ci nightly are failing due to missing task:

Pipeline tekton-nightly/add-team-members-release can't be Run; it contains Tasks that don't exist: Couldn't retrieve Task "": error requesting remote resource: error getting "bundleresolver" "tekton-nightly/bundles-c741b1a1ad5c6d7d07337a952fbd7d6a": could not find object in image with kind: task and name: git-batch-merge

pr-commenter and pr-status-updater are failing due to:

2024-09-27T00:01:06.270600691Z go: updates to go.mod needed; to update it:
2024-09-27T00:01:06.270635326Z 	go mod tidy

@afrittoli
Copy link
Member

afrittoli commented Oct 2, 2024

add-team-members-release

This is now working fine, I fixed the references to the wrong tasks.

add-pr-body-ci

This now failing for the same go mod tidy issue as the pr-* ones

add-pr-body

This now fails with a koparse issue

PR to fix: #2225

afrittoli added a commit to afrittoli/plumbing that referenced this issue Oct 2, 2024
Related to: tektoncd#2197

Signed-off-by: Andrea Frittoli <[email protected]>
@afrittoli
Copy link
Member

Fix for the CEL component: tektoncd/experimental#979

tekton-robot pushed a commit that referenced this issue Oct 3, 2024
Related to: #2197

Signed-off-by: Andrea Frittoli <[email protected]>
@afrittoli
Copy link
Member

afrittoli commented Oct 3, 2024

Remaining issues:

  • pr-commenter:
+ IMAGES=' http://gcr.io/tekton-nightly/github.com/tektoncd/plumbing/tekton/ci/custom-tasks/pr-commenter/cmd/pr-commenter:v20241003-d627097ada'
/tekton/scripts/script-3-bmfdd: line 19: syntax error: unterminated quoted string
  • add-pr-body:
+ IMAGES_PATH=http://gcr.io/tekton-nightly/github.com/tektoncd/plumbing
+ '[' tekton/ci/interceptors/add-pr-body '!='  ]
+ IMAGES_PATH=http://gcr.io/tekton-nightly/github.com/tektoncd/plumbing/tekton/ci/interceptors/add-pr-body
+ IMAGES='http://gcr.io/tekton-nightly/github.com/tektoncd/plumbing/tekton/ci/interceptors/add-pr-body/cmd/add-pr-body:v20241003-d627097ada'
+ koparse --path /workspace/output/v20241003-d627097ada/release.yaml --base http://gcr.io/tekton-nightly/github.com/tektoncd/plumbing/tekton/ci/interceptors/add-pr-body)--images http://gcr.io/tekton-nightly/github.com/tektoncd/plumbing/tekton/ci/interceptors/add-pr-body/cmd/add-pr-body:v20241003-d627097ada
Expected images did not match: Images ['http://gcr.io/tekton-nightly/github.com/tektoncd/plumbing/tekton/ci/interceptors/add-pr-body/cmd/add-pr-body:v20241003-d627097ada'] were expected but missing.

@afrittoli
Copy link
Member

All remaining issues have been fixed.

@AlanGreene
Copy link
Member Author

Thanks for seeing this over the line @afrittoli 🙇 I'll be back next week

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/misc Categorizes issue or PR as a miscellaneuous one.
Projects
None yet
Development

No branches or pull requests

2 participants