-
Notifications
You must be signed in to change notification settings - Fork 114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Retry only failed test when --num-flaky-test-attempts is provided #2429
Comments
The linked issue points to Flank delegating the retry logic to the Firebase backend, but the documentation doesn't concretely describe exactly what will be re-run, e.g. entire execution, shard, individual test case. I opened a conversation internally with folks to investigate. |
Following up, the behavior is working as intended. The documentation specifies that the execution will be retried. In certain cases that might mean retrying a shard instead of the entire set of tests, but explicitly does not mean retrying individual test cases. I'll work to get the documentation clarified. |
GIVEN --num-flaky-test-attempts is N,
WHEN flank is run ,
THEN
I have tried running flank v23.07.0, and provided the argument --num-flaky-test-attempts=4, for 2 test named as :
I expected the "alwaysPass" test to be executed only once, without further retry. But, it was executed for 4 times as shown in the attached screenshot.
Based on my understanding of the previous issue : #636
I believe that the retry feature should only be executed on failing test ?
Can someone help to provide advice ?
The text was updated successfully, but these errors were encountered: