Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve test stability #614

Merged

Conversation

SaschaSchwarze0
Copy link
Member

@SaschaSchwarze0 SaschaSchwarze0 commented Feb 22, 2021

Changes

PR Use GitHub action for e2e test #611 should not have been merged by the robot. It had an approval, but the tests were not successful.

Basically, after having added the fourth Kubernetes version to the test matrix, the actions became unstable. Statements in the documentation like this leave room for interpretation, but I assume there is some resource limit and CPU is becoming the bottleneck.

I am therefore trying to limit the parallelism.

In addition, I am making changes to remove race conditions:

  • in e2e, I am waiting for the Build to be registered before creating a BuildRun
  • in integration, I am waiting for the Build to have a registration before creating a BuildRun
  • in integration, after creating a namespace, I am waiting for the default service account to exist and contain the token secret
  • in integration, I added a polling to wait for the owner updates to happen
  • in integration, I commented our a part where a test case assumed to see a Pending TaskRun and BuildRun

Finally, I have to disable the runs with more than one Kubernetes version. The reason is that I see intermittent API calls (performed by the test code) failing with etcdserver: request timed out. We have some retry behavior for them for get calls in e2e, but not for creations and updates. And we have no retry as of now in integration. I am mainly seeing it for creations (namespace, clusterbuildstrategy) in integration.

Submitter Checklist

  • Includes tests if functionality changed/was added
  • Includes docs if changes are user-facing
  • Release notes block has been filled in, or marked NONE

Release Notes

NONE

@openshift-ci-robot openshift-ci-robot added release-note-none Label for when a PR does not need a release note do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Feb 22, 2021
@SaschaSchwarze0 SaschaSchwarze0 force-pushed the sascha-e2e branch 12 times, most recently from 02a4567 to 141c1c5 Compare February 23, 2021 13:25
@SaschaSchwarze0 SaschaSchwarze0 changed the title WIP Improve test stability Improve test stability Feb 23, 2021
@openshift-ci-robot openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 23, 2021
Copy link
Contributor

@qu1queee qu1queee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SaschaSchwarze0 thanks, nice!
/lgtm
/approve

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 24, 2021
@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: qu1queee

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 24, 2021
@openshift-merge-robot openshift-merge-robot merged commit f370fe3 into shipwright-io:master Feb 24, 2021
@SaschaSchwarze0 SaschaSchwarze0 deleted the sascha-e2e branch February 24, 2021 10:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. release-note-none Label for when a PR does not need a release note
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants