-
Notifications
You must be signed in to change notification settings - Fork 738
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Automate conformance testing #686
Comments
I don't mind sonobuoy, but if we can achieve the same thing using the e2e.test binary directly I would prefer that mostly because it's one less potential source of bugs to worry about. For example, something like this should be possible and we'll get the test output in real-time straight to stdout rather than having to wait for
|
Fair point, Matthew. I've changed the issue title and description accordingly. |
Adds a .circleci/config.yml file that will be used to house the complex workflows we want to use for automated functional e2e testing. Issue aws#686
The names of environment variables accepted by aws-k8s-tester changed when the managed node group functionality was introduced. This commit updates the integration testing scripts to call aws-k8s-tester (v.0.5.4 which is the release needed with the fix for aws/aws-k8s-tester#70) with these updated environment variables. We decrease the number of parallel builds of the echo job from 100 to 3 and the number of completions for that job from 1000 to 30. This decreases the setup time of the cluster by about 10 minutes. Finally, I added in a short-circuit to prevent double-deprovisioning of the cluster if, say, a stacktrace occurred when running the aws-k8s-tester tool. Issue aws#686 Issue aws#784 Issue aws#786
The names of environment variables accepted by aws-k8s-tester changed when the managed node group functionality was introduced. This commit updates the integration testing scripts to call aws-k8s-tester (v.0.5.4 which is the release needed with the fix for aws/aws-k8s-tester#70) with these updated environment variables. We decrease the number of parallel builds of the echo job from 100 to 3 and the number of completions for that job from 1000 to 30. This decreases the setup time of the cluster by about 10 minutes. Finally, I added in a short-circuit to prevent double-deprovisioning of the cluster if, say, a stacktrace occurred when running the aws-k8s-tester tool. Issue aws#686 Issue aws#784 Issue aws#786
The names of environment variables accepted by aws-k8s-tester changed when the managed node group functionality was introduced. This commit updates the integration testing scripts to call aws-k8s-tester (v.0.5.4 which is the release needed with the fix for aws/aws-k8s-tester#70) with these updated environment variables. We decrease the number of parallel builds of the echo job from 100 to 3 and the number of completions for that job from 1000 to 30. This decreases the setup time of the cluster by about 10 minutes. Finally, I added in a short-circuit to prevent double-deprovisioning of the cluster if, say, a stacktrace occurred when running the aws-k8s-tester tool. Issue aws#686 Issue aws#784 Issue aws#786
The names of environment variables accepted by aws-k8s-tester changed when the managed node group functionality was introduced. This commit updates the integration testing scripts to call aws-k8s-tester (v.0.5.4 which is the release needed with the fix for aws/aws-k8s-tester#70) with these updated environment variables. We decrease the number of parallel builds of the echo job from 100 to 3 and the number of completions for that job from 1000 to 30. This decreases the setup time of the cluster by about 10 minutes. Finally, I added in a short-circuit to prevent double-deprovisioning of the cluster if, say, a stacktrace occurred when running the aws-k8s-tester tool. Issue #686 Issue #784 Issue #786
The names of environment variables accepted by aws-k8s-tester changed when the managed node group functionality was introduced. This commit updates the integration testing scripts to call aws-k8s-tester (v.0.5.4 which is the release needed with the fix for aws/aws-k8s-tester#70) with these updated environment variables. We decrease the number of parallel builds of the echo job from 100 to 3 and the number of completions for that job from 1000 to 30. This decreases the setup time of the cluster by about 10 minutes. Finally, I added in a short-circuit to prevent double-deprovisioning of the cluster if, say, a stacktrace occurred when running the aws-k8s-tester tool. Issue aws#686 Issue aws#784 Issue aws#786 (cherry picked from commit 8291df3)
The names of environment variables accepted by aws-k8s-tester changed when the managed node group functionality was introduced. This commit updates the integration testing scripts to call aws-k8s-tester (v.0.5.4 which is the release needed with the fix for aws/aws-k8s-tester#70) with these updated environment variables. We decrease the number of parallel builds of the echo job from 100 to 3 and the number of completions for that job from 1000 to 30. This decreases the setup time of the cluster by about 10 minutes. Finally, I added in a short-circuit to prevent double-deprovisioning of the cluster if, say, a stacktrace occurred when running the aws-k8s-tester tool. Issue #686 Issue #784 Issue #786 (cherry picked from commit 8291df3)
Ran
Had to skip the following two test because of kubernetes/kubernetes#88064
Updated "flakeAttempts" to 2 so that we don't see failure on flaky tests. single test failures will still be reported as failure but test suite doesn't fail. Added "failFast" parameter so that we will know early if something fails even with retry. Usually conformance tests takes upto 2 hours to complete. SaranBalaji90@2264ffd has the changes required. Still testing CI integration in my repo. |
The EKS team currently runs the Sonobouy conformance test suite manually before releasing the CNI plugin. We want to both automate this process and place all test definitions into the public repo.
Alternately, to avoid waiting for Sonobuoy logs, we could do this instead (suggested from @wongma7):
The text was updated successfully, but these errors were encountered: