Chromium Code Reviews Chromium Code Reviews
Issue
1289163002:
Improve flakiness logic for print-flaky-tests. (Closed)
DescriptionImprove flakiness logic for print-flaky-tests.
We only compare the 2nd attempt of a run to the tests expected results to
see if it was very flaky. Instead we should see if any of the runs had
an unexpected result. That way, any test that gets marked flaky will
stop getting printed out by this script as soon as the bots have cycled once.
Also, use TestExpectations.result_was_expected to determine if a result was
actually expected. The previous "in" check didn't work for cases like Text,
which are a subset of Failure instead of verbatim include in TestExpectations
as Text.
Finally, just use the length of the results entry to decide if something
is very flaky vs regular flaky. If there are <=2 entries, then it's not
very flaky. That catches the cases of a test that reliably fails as well
as the case of a test that fails the first attempt and runs as expected
on the second run (not necessarily passing on the second attempt, just
matching the listing in TestExpectations for that run).
R=joelo@chromium.org
Committed: https://src.chromium.org/viewvc/blink?view=rev&revision=200768
Patch Set 1 #
Total comments: 3
Messages
Total messages: 5 (1 generated)
|
||||||||||||||||||||||||||||||||||||||||||||||
