Disable all the tests that are flaking more than 5% on ChromeOS builders.

Disable all the tests that are flaking more than 5% on ChromeOS builders.

Data from http://chromium-build-logs.appspot.com/flakiness_dashboard?master=chromium.chromiumos.

BUG=386891, 396494, 396496, 396497, 396499, 396501, 396502, 396504, 396505, 393208, 223888, 396506, 387045, 396507, 396509, 396510
R=viettrungluu@chromium.org
TBR=sky, stevenjb

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=284938

[viettrungluu, 2014-07-23 15:29:28]

lgtm w/nit

https://codereview.chromium.org/407073010/diff/1/chrome/browser/media/chrome_...
File chrome/browser/media/chrome_webrtc_browsertest.cc (right):

https://codereview.chromium.org/407073010/diff/1/chrome/browser/media/chrome_...
chrome/browser/media/chrome_webrtc_browsertest.cc:45: #define
MAYBE_RunsAudioVideoWebRTCCallInTwoTabs
DISABLED_RunsAudioVideoWebRTCCallInTwoTabs
nit: avoid the long lines

[jam, 2014-07-23 16:03:11]

https://codereview.chromium.org/407073010/diff/1/chrome/browser/media/chrome_...
File chrome/browser/media/chrome_webrtc_browsertest.cc (right):

https://codereview.chromium.org/407073010/diff/1/chrome/browser/media/chrome_...
chrome/browser/media/chrome_webrtc_browsertest.cc:45: #define
MAYBE_RunsAudioVideoWebRTCCallInTwoTabs
DISABLED_RunsAudioVideoWebRTCCallInTwoTabs
On 2014/07/23 15:29:28, viettrungluu wrote:
> nit: avoid the long lines

Done.

[jam, 2014-07-23 16:05:12]

tbr'ing for owners

[eseidel, 2014-07-23 16:19:49]

We clearly need a separate file to hold expectations.  It's very difficult as-is
to see anything resembling trends when the expectations are right next to the
individual tests.

Also interesting that these become disabled in this mode, which means that they
can't ever easily be turned back on when they stop being flaky.

In any case, thank you very much for disabling these.

[ojan, 2014-07-23 18:14:12]

I don't think we should be making changes like this for the code yellow. Doing
this manually temporarily plugs a hole. Other tests will become flaky over the
course of the coming weeks and some of these tests will stop being flaky.

I'd rather we focus on shipping a systematic, long-term solution (ignorer-bot).
This change actually hurts the code yellow because it makes it hard to tell if
the system we put in place to deal with these things automatically actually
works well.

[Sergey Berezin, 2014-07-23 18:22:09]

On 2014/07/23 18:14:12, ojan-only-code-yellow-reviews wrote:
> I don't think we should be making changes like this for the code yellow. Doing
> this manually temporarily plugs a hole. Other tests will become flaky over the
> course of the coming weeks and some of these tests will stop being flaky.
> 
> I'd rather we focus on shipping a systematic, long-term solution
(ignorer-bot).
> This change actually hurts the code yellow because it makes it hard to tell if
> the system we put in place to deal with these things automatically actually
> works well.

We still have plenty of tests with flakiness over 1%, which is the current
target for the ignorer bot. I personally think this will warm people up to the
idea that we mean business by disabling their tests, even before the ignorer bot
is fully ready.

[jam, 2014-07-24 20:18:02]

On 2014/07/23 16:19:49, eseidel wrote:
> We clearly need a separate file to hold expectations.

Let's discuss this with a wider audience, i.e. chromium-dev.

>  It's very difficult as-is
> to see anything resembling trends when the expectations are right next to the
> individual tests.

it's still possible to have trends since gtest harness can tell how many tests
(and which) tests are disabled. so one can see what % of tests are disabled.
obviously tests can be disabled for other reasons, but generally in chrome
they're disabled because they're flaky.

> 
> Also interesting that these become disabled in this mode, which means that
they
> can't ever easily be turned back on when they stop being flaky.
> 
> In any case, thank you very much for disabling these.

[jam, 2014-07-24 20:20:31]

On 2014/07/23 18:14:12, ojan-only-code-yellow-reviews wrote:
> I don't think we should be making changes like this for the code yellow. Doing
> this manually temporarily plugs a hole. Other tests will become flaky over the
> course of the coming weeks and some of these tests will stop being flaky.

I'm curious what makes you say that stops will stop being flaky on their own?

there are still many other flaky tests as Sergey mentioned. I picked 5% as an
arbitrary threshold for now. Some of these tests have been flaky for a long
time, so I wanted to put the dashboard that we got recently to good use.

> 
> I'd rather we focus on shipping a systematic, long-term solution
(ignorer-bot).
> This change actually hurts the code yellow because it makes it hard to tell if
> the system we put in place to deal with these things automatically actually
> works well.

i don't think that's true, there are still hundreds of tests that flake between
1% and 5% of the time. there's no good reason to keep real flaky tests running
while the ignorer bot is being worked on.

[ojan, 2014-07-24 21:25:49]

On 2014/07/24 at 20:20:31, jam wrote:
> On 2014/07/23 18:14:12, ojan-only-code-yellow-reviews wrote:
> > I don't think we should be making changes like this for the code yellow.
Doing
> > this manually temporarily plugs a hole. Other tests will become flaky over
the
> > course of the coming weeks and some of these tests will stop being flaky.
> 
> I'm curious what makes you say that stops will stop being flaky on their own?

In my experience, it's very common for tests to become flaky for a short-period
of time and then stop being flaky later. Flakiness-inducing bugs get introduced
and fixed all the time. But if the tests have been flaky for a long time, then
it's less likely that they'll get fixed on their own.

[chromium-reviews, 2014-07-24 21:37:36]

On Thu, Jul 24, 2014 at 2:25 PM, <ojan@chromium.org> wrote:

> In my experience, it's very common for tests to become flaky for a
> short-period
> of time and then stop being flaky later. Flakiness-inducing bugs get
> introduced
> and fixed all the time. But if the tests have been flaky for a long time,
> then
> it's less likely that they'll get fixed on their own.
>

Yes! Please don't disable flaky tests too quickly if they've been passing
consistently for a while before that, it's often a sign that the test is
actually exposing a new bug somewhere else in the system that should be
fixed.

To unsubscribe from this group and stop receiving emails from it, send an email
to chromium-reviews+unsubscribe@chromium.org.

[eseidel, 2014-07-24 21:40:31]

A flaky test is currently treated like a compile failure and closes the tree and
stops work for 100s of people.

Except a flaky test is in fact worse than a compile failure, because it comes
back.

It's still worse, in that it not only closes the tree, it also slows down the
CQ.

We've tried not disabling flaky tests for years.  It didn't work.  We're now in
a code yellow and we're trying something different.

Once we have a list of disable/flaky tests we can burn that list down.  This
process has worked fine for Blink for years.  Somehow we haven't destroyed the
product yet, despite years of flaky test disabling. :)

[jam, 2014-07-25 19:45:29]

On 2014/07/24 21:25:49, ojan-only-code-yellow-reviews wrote:
> On 2014/07/24 at 20:20:31, jam wrote:
> > On 2014/07/23 18:14:12, ojan-only-code-yellow-reviews wrote:
> > > I don't think we should be making changes like this for the code yellow.
> Doing
> > > this manually temporarily plugs a hole. Other tests will become flaky over
> the
> > > course of the coming weeks and some of these tests will stop being flaky.
> > 
> > I'm curious what makes you say that stops will stop being flaky on their
own?
> 
> In my experience, it's very common for tests to become flaky for a
short-period
> of time and then stop being flaky later. Flakiness-inducing bugs get
introduced
> and fixed all the time. But if the tests have been flaky for a long time, then
> it's less likely that they'll get fixed on their own.

I wonder if layout tests' failure modes are different from (gtest) tests in
chromium repo? That hasn't been my experience with unit and browser tests.

[jam, 2014-07-25 19:52:00]

I've sent an email about this with more folks, since most people will miss it in
a code review thread. I think a meeting will be helpful.

On 2014/07/24 21:40:31, eseidel wrote:
> A flaky test is currently treated like a compile failure and closes the tree
and
> stops work for 100s of people.

Not quite. The unit and browser test harnesses retry tests 3 times to work
around flakiness. So a test that flakes once, and passes on retry doesn't affect
developers.

If a test suddenly starts failing frequently, then that might be good hint that
some regression happened recently. That's what happened with the gpu maps tests
earlier this week. Disabling that test would have unintended consequences.

We had discussed earlier that the system would detect flakiness and figure out
what caused it using the bisect bot. That is very useful and I'm not sure if
people are working on that?

Also note that some of the flakiness is induced by running tests in parallel, so
the flakiness rates are exaggerated because they count those tests that fail
when sharded (say because they're slow and get starved).

> 
> Except a flaky test is in fact worse than a compile failure, because it comes
> back.
> 
> It's still worse, in that it not only closes the tree, it also slows down the
> CQ.

per above, this doesn't slow down the CQ unless the test flakes 4 times. if a
test is flaking that often, it should be at the top of a flakiness dashboard and
we should have a bug filed. This should be treated high priority, since it could
be exposing actual problems. A bot that tracks when the test started failing
should occur, to find the root cause. If the test has always been failing, then
disabling it is fine. But that is a one time thing, once we have this system
running.

> 
> We've tried not disabling flaky tests for years.  It didn't work.  We're now
in
> a code yellow and we're trying something different.
> 
> Once we have a list of disable/flaky tests we can burn that list down.  This
> process has worked fine for Blink for years.  Somehow we haven't destroyed the
> product yet, despite years of flaky test disabling. :)

[eseidel, 2014-07-25 20:10:27]

I guess it's just all a matter of degrees.  I was not aware that gtest retried 3
times.  LayoutTests only retry once last I knew.  n^4 gets smaller much faster
than n^2. :)

Retry systems can work around flakiness yes (and we do have them, and I'm very
glad for it).  A retry system is just throwing time at the problem though. 
Sufficiently surgically placed retry systems trade less time than a full rebuild
or a full 'click the CQ again' of course.

A flaky test still closes trees and slows down the CQ (by hitting higher and
higher retry levels) just at lower percentages.

If gtest retries 3 times, then a 1% flake should only succeed in closing the
tree only 1e-8 builds (effectively never).  A 10% flake will close the tree 1 in
10,000 (about once a year for chromium bots) and a 50% flake 7 out of 100 (a
couple times a day), etc.

Maybe LayoutTests should also retry 3 times.  I think we didn't do that because
it just slows down the common case of a test being deterministic.