Add --sanitizer-mode to run-webkit-tests

When we run sanitizers like ASAN and LSAN, we typically don't care about whether the tests run exactly like they do normally; we are just looking for errors that the sanitizers have detected.

This flag changes run-webkit-tests to ignore text and image failures and just look for crashes and timeouts (most crashes will be sanitizer-triggered).

In theory most of the time the tests *should* run the same way, but in practice we see a lot of noise that no one has the time to deal with, so this change should reduce a lot of the redness we see on the bots.

BUG=336971, 374043
Committed: https://src.chromium.org/viewvc/blink?view=rev&revision=175488

[earthdok, 2014-05-22 12:51:42]

Hey!

1) How does this connect with the out-of-band logging approach we discussed for
ASan/LSan? (I was about to start working on that.) It appears that you're still
looking at stderr. Should I wait until you commit this and then change this code
to use out-of-band logs?

2) What is the rationale behind ignoring non-ASan errors in ASan runs? If an
ASan run behaves differently than a regular run, we certainly want to know.
Especially since there are plans to replace libstdc++ with libc++ in ASan
builds, which could be enough to cause test failures (there's code which
incorrectly relies on libstdc++-specific behavior).

3) I suggest to name this --enable-sanitizer-mode (and replace ASan with
sanitizer accordingly) since ThreadSanitizer, MemorySanitizer and LeakSanitizer
would also use this.

[Dirk Pranke, 2014-05-22 19:37:39]

On 2014/05/22 12:51:42, earthdok wrote:
> Hey!
> 
> 1) How does this connect with the out-of-band logging approach we discussed
for
> ASan/LSan? (I was about to start working on that.) It appears that you're
still
> looking at stderr. Should I wait until you commit this and then change this
code
> to use out-of-band logs?
> 

My changes are not intended to replace the out-of-band-logging approach 
of bug 374043.

Rather, they are designed to introduce a core difference in run-webkit-tests
to treat sanitized runs differently from non-sanitized run, as below ...

> 2) What is the rationale behind ignoring non-ASan errors in ASan runs? If an
> ASan run behaves differently than a regular run, we certainly want to know.
> Especially since there are plans to replace libstdc++ with libc++ in ASan
> builds, which could be enough to cause test failures (there's code which
> incorrectly relies on libstdc++-specific behavior).
>

Well, in theory I'd agree with you that an ASAN build should produce the same
as a non-ASAN build. In practice, we've found that it doesn't; there are often
a few tests that fail in odd ways. However, the memory sheriffs don't have the
cycles to investigate those failures, and the normal blink gardeners don't 
tend to care about ASAN-induced failures that aren't actually memory errors
(since they don't happen in a non-ASAN build, they're really more just
flakiness).

So the non-memory failures just end up being viewed as noise, and this change is
intended
to eliminate the noise. 

Someday, it might be nice to not view this as noise ...

> 3) I suggest to name this --enable-sanitizer-mode (and replace ASan with
> sanitizer accordingly) since ThreadSanitizer, MemorySanitizer and
LeakSanitizer
> would also use this.

Yeah, that name is better. Will change it.

So, I'd expect to land this change and then have you would on your
post-processing
changes, as this should (hopefully) make it easier to integrate the two and keep
things clearer.

[earthdok, 2014-05-22 20:31:57]

On 2014/05/22 19:37:39, Dirk Pranke wrote:
> Well, in theory I'd agree with you that an ASAN build should produce the same
> as a non-ASAN build. In practice, we've found that it doesn't; there are often
> a few tests that fail in odd ways. However, the memory sheriffs don't have the
> cycles to investigate those failures, and the normal blink gardeners don't 
> tend to care about ASAN-induced failures that aren't actually memory errors
> (since they don't happen in a non-ASAN build, they're really more just
> flakiness).

Regarding memory sheriffs, I'm curious what you base that on. We're talking
about Chromium sheriffs, correct?

Judging by the sheriff handoff emails, most shifts lately have been pretty
quiet. Also, I don't think memory
sheriffs watch the ASan layout bot at all, since it's not on
chromium.memory.fyi.

[Dirk Pranke, 2014-05-23 16:09:28]

On Thu, May 22, 2014 at 1:31 PM, <earthdok@chromium.org> wrote:

> On 2014/05/22 19:37:39, Dirk Pranke wrote:
>
>> Well, in theory I'd agree with you that an ASAN build should produce the
>> same
>> as a non-ASAN build. In practice, we've found that it doesn't; there are
>> often
>> a few tests that fail in odd ways. However, the memory sheriffs don't
>> have the
>> cycles to investigate those failures, and the normal blink gardeners don't
>> tend to care about ASAN-induced failures that aren't actually memory
>> errors
>> (since they don't happen in a non-ASAN build, they're really more just
>> flakiness).
>>
>
> Regarding memory sheriffs, I'm curious what you base that on. We're talking
> about Chromium sheriffs, correct?
>

I'm referring to whomever watches the chromium.memory.* waterfalls. I agree
that they don't currently watch the chromium.webkit bot; the blink
gardeners are expected to be doing so.

I seem to recall some comment from inferno@ somewhere that if we made the
WebKit Linux ASAN bot only fail on memory failures, it might be more
interesting to the memory sheriffs. I don't think this particular point
matters that much at the moment; I just want to get the bot reliably
detecting all memory errors and not turning red for things we don't care
about.

-- Dirk

To unsubscribe from this group and stop receiving emails from it, send an email
to blink-reviews+unsubscribe@chromium.org.

[blink-reviews, 2014-05-25 13:45:48]

On Fri, May 23, 2014 at 9:09 AM, Dirk Pranke <dpranke@chromium.org> wrote:

>
>
>
> On Thu, May 22, 2014 at 1:31 PM, <earthdok@chromium.org> wrote:
>
>> On 2014/05/22 19:37:39, Dirk Pranke wrote:
>>
>>> Well, in theory I'd agree with you that an ASAN build should produce the
>>> same
>>> as a non-ASAN build. In practice, we've found that it doesn't; there are
>>> often
>>> a few tests that fail in odd ways. However, the memory sheriffs don't
>>> have the
>>> cycles to investigate those failures, and the normal blink gardeners
>>> don't
>>> tend to care about ASAN-induced failures that aren't actually memory
>>> errors
>>> (since they don't happen in a non-ASAN build, they're really more just
>>> flakiness).
>>>
>>
>> Regarding memory sheriffs, I'm curious what you base that on. We're
>> talking
>> about Chromium sheriffs, correct?
>>
>
> I'm referring to whomever watches the chromium.memory.* waterfalls. I
> agree that they don't currently watch the chromium.webkit bot; the blink
> gardeners are expected to be doing so.
>
> I seem to recall some comment from inferno@ somewhere that if we made the
> WebKit Linux ASAN bot only fail on memory failures, it might be more
> interesting to the memory sheriffs. I don't think this particular point
> matters that much at the moment; I just want to get the bot reliably
> detecting all memory errors and not turning red for things we don't care
> about.
>

+1, we should just ignore the functional test failures, timeouts, etc for
this bot. That way memory sheriff will just be looking at security failures.


>
> -- Dirk
>
>

To unsubscribe from this group and stop receiving emails from it, send an email
to blink-reviews+unsubscribe@chromium.org.

[Alexander Potapenko, 2014-05-26 11:46:59]

Again, I'd like to emphasize that it's regular (Chrome, Webkit)
sheriffs' job to watch ASan bots. Memory sheriffs watch the
non-tree-closing MFYI waterfall.
Sorry for the off-top.

On Sun, May 25, 2014 at 5:45 PM, Abhishek Arya <aarya@google.com> wrote:
> On Fri, May 23, 2014 at 9:09 AM, Dirk Pranke <dpranke@chromium.org> wrote:
>>
>>
>>
>>
>> On Thu, May 22, 2014 at 1:31 PM, <earthdok@chromium.org> wrote:
>>>
>>> On 2014/05/22 19:37:39, Dirk Pranke wrote:
>>>>
>>>> Well, in theory I'd agree with you that an ASAN build should produce the
>>>> same
>>>> as a non-ASAN build. In practice, we've found that it doesn't; there are
>>>> often
>>>> a few tests that fail in odd ways. However, the memory sheriffs don't
>>>> have the
>>>> cycles to investigate those failures, and the normal blink gardeners
>>>> don't
>>>> tend to care about ASAN-induced failures that aren't actually memory
>>>> errors
>>>> (since they don't happen in a non-ASAN build, they're really more just
>>>> flakiness).
>>>
>>>
>>> Regarding memory sheriffs, I'm curious what you base that on. We're
>>> talking
>>> about Chromium sheriffs, correct?
>>
>>
>> I'm referring to whomever watches the chromium.memory.* waterfalls. I
>> agree that they don't currently watch the chromium.webkit bot; the blink
>> gardeners are expected to be doing so.
>>
>> I seem to recall some comment from inferno@ somewhere that if we made the
>> WebKit Linux ASAN bot only fail on memory failures, it might be more
>> interesting to the memory sheriffs. I don't think this particular point
>> matters that much at the moment; I just want to get the bot reliably
>> detecting all memory errors and not turning red for things we don't care
>> about.
>
>
> +1, we should just ignore the functional test failures, timeouts, etc for
> this bot. That way memory sheriff will just be looking at security failures.
>
>>
>>
>> -- Dirk
>>
>



-- 
Alexander Potapenko
Software Engineer
Google Moscow

To unsubscribe from this group and stop receiving emails from it, send an email
to blink-reviews+unsubscribe@chromium.org.

[blink-reviews, 2014-05-26 11:52:51]

I wish we used an easier-to-understand naming scheme...

2014-05-26 15:46 GMT+04:00 Alexander Potapenko <glider@chromium.org>:
> Again, I'd like to emphasize that it's regular (Chrome, Webkit)
> sheriffs' job to watch ASan bots. Memory sheriffs watch the
> non-tree-closing MFYI waterfall.
> Sorry for the off-top.
>
> On Sun, May 25, 2014 at 5:45 PM, Abhishek Arya <aarya@google.com> wrote:
>> On Fri, May 23, 2014 at 9:09 AM, Dirk Pranke <dpranke@chromium.org> wrote:
>>>
>>>
>>>
>>>
>>> On Thu, May 22, 2014 at 1:31 PM, <earthdok@chromium.org> wrote:
>>>>
>>>> On 2014/05/22 19:37:39, Dirk Pranke wrote:
>>>>>
>>>>> Well, in theory I'd agree with you that an ASAN build should produce the
>>>>> same
>>>>> as a non-ASAN build. In practice, we've found that it doesn't; there are
>>>>> often
>>>>> a few tests that fail in odd ways. However, the memory sheriffs don't
>>>>> have the
>>>>> cycles to investigate those failures, and the normal blink gardeners
>>>>> don't
>>>>> tend to care about ASAN-induced failures that aren't actually memory
>>>>> errors
>>>>> (since they don't happen in a non-ASAN build, they're really more just
>>>>> flakiness).
>>>>
>>>>
>>>> Regarding memory sheriffs, I'm curious what you base that on. We're
>>>> talking
>>>> about Chromium sheriffs, correct?
>>>
>>>
>>> I'm referring to whomever watches the chromium.memory.* waterfalls. I
>>> agree that they don't currently watch the chromium.webkit bot; the blink
>>> gardeners are expected to be doing so.
>>>
>>> I seem to recall some comment from inferno@ somewhere that if we made the
>>> WebKit Linux ASAN bot only fail on memory failures, it might be more
>>> interesting to the memory sheriffs. I don't think this particular point
>>> matters that much at the moment; I just want to get the bot reliably
>>> detecting all memory errors and not turning red for things we don't care
>>> about.
>>
>>
>> +1, we should just ignore the functional test failures, timeouts, etc for
>> this bot. That way memory sheriff will just be looking at security failures.
>>
>>>
>>>
>>> -- Dirk
>>>
>>
>
>
>
> --
> Alexander Potapenko
> Software Engineer
> Google Moscow

To unsubscribe from this group and stop receiving emails from it, send an email
to blink-reviews+unsubscribe@chromium.org.

[earthdok, 2014-05-26 12:34:08]

On 2014/05/25 13:45:48, blink-reviews_chromium.org wrote:
> On Fri, May 23, 2014 at 9:09 AM, Dirk Pranke <mailto:dpranke@chromium.org>
wrote:
> 
> >
> >
> >
> > On Thu, May 22, 2014 at 1:31 PM, <mailto:earthdok@chromium.org> wrote:
> >
> >> On 2014/05/22 19:37:39, Dirk Pranke wrote:
> >>
> >>> Well, in theory I'd agree with you that an ASAN build should produce the
> >>> same
> >>> as a non-ASAN build. In practice, we've found that it doesn't; there are
> >>> often
> >>> a few tests that fail in odd ways. However, the memory sheriffs don't
> >>> have the
> >>> cycles to investigate those failures, and the normal blink gardeners
> >>> don't
> >>> tend to care about ASAN-induced failures that aren't actually memory
> >>> errors
> >>> (since they don't happen in a non-ASAN build, they're really more just
> >>> flakiness).
> >>>
> >>
> >> Regarding memory sheriffs, I'm curious what you base that on. We're
> >> talking
> >> about Chromium sheriffs, correct?
> >>
> >
> > I'm referring to whomever watches the chromium.memory.* waterfalls. I
> > agree that they don't currently watch the chromium.webkit bot; the blink
> > gardeners are expected to be doing so.
> >
> > I seem to recall some comment from inferno@ somewhere that if we made the
> > WebKit Linux ASAN bot only fail on memory failures, it might be more
> > interesting to the memory sheriffs. I don't think this particular point
> > matters that much at the moment; I just want to get the bot reliably
> > detecting all memory errors and not turning red for things we don't care
> > about.
> >
> 
> +1, we should just ignore the functional test failures, timeouts, etc for
> this bot. That way memory sheriff will just be looking at security failures.

If we ignore non-ASan failures, how do we know the tests are actually running
the code they are meant to test? They could start crashing before doing anything
useful, and the bot would stay green forever.

[Dirk Pranke, 2014-05-28 18:57:24]

On 2014/05/26 12:34:08, earthdok wrote:
> On 2014/05/25 13:45:48, http://blink-reviews_chromium.org wrote:
> > On Fri, May 23, 2014 at 9:09 AM, Dirk Pranke <mailto:dpranke@chromium.org>
> wrote:
> > 
> > >
> > >
> > >
> > > On Thu, May 22, 2014 at 1:31 PM, <mailto:earthdok@chromium.org> wrote:
> > >
> > >> On 2014/05/22 19:37:39, Dirk Pranke wrote:
> > >>
> > >>> Well, in theory I'd agree with you that an ASAN build should produce the
> > >>> same
> > >>> as a non-ASAN build. In practice, we've found that it doesn't; there are
> > >>> often
> > >>> a few tests that fail in odd ways. However, the memory sheriffs don't
> > >>> have the
> > >>> cycles to investigate those failures, and the normal blink gardeners
> > >>> don't
> > >>> tend to care about ASAN-induced failures that aren't actually memory
> > >>> errors
> > >>> (since they don't happen in a non-ASAN build, they're really more just
> > >>> flakiness).
> > >>>
> > >>
> > >> Regarding memory sheriffs, I'm curious what you base that on. We're
> > >> talking
> > >> about Chromium sheriffs, correct?
> > >>
> > >
> > > I'm referring to whomever watches the chromium.memory.* waterfalls. I
> > > agree that they don't currently watch the chromium.webkit bot; the blink
> > > gardeners are expected to be doing so.
> > >
> > > I seem to recall some comment from inferno@ somewhere that if we made the
> > > WebKit Linux ASAN bot only fail on memory failures, it might be more
> > > interesting to the memory sheriffs. I don't think this particular point
> > > matters that much at the moment; I just want to get the bot reliably
> > > detecting all memory errors and not turning red for things we don't care
> > > about.
> > >
> > 
> > +1, we should just ignore the functional test failures, timeouts, etc for
> > this bot. That way memory sheriff will just be looking at security failures.
> 
> If we ignore non-ASan failures, how do we know the tests are actually running
> the code they are meant to test? They could start crashing before doing
anything
> useful, and the bot would stay green forever.

That is a risk. I'm not sure how significant a risk this is. It might be a
reasonable compromise to make sure we were failing tests that were crashing for
*any* reason, not just sanitizer-related ones.

[Dirk Pranke, 2014-06-02 23:27:02]

Please take a look? This should allow our ASAN bots to stay a lot more green and
provide useful info when we do get failures.

[Dirk Pranke, 2014-06-03 20:04:05]

updated description :).

[ojan, 2014-06-04 02:17:17]

lgtm

I would be fine with us being even more conservative and ignoring timeouts. Lets
get something we can actually keep up with and then expand. Right now these bots
are pretty much ignored.

[ojan, 2014-06-04 02:17:37]

The CQ bit was checked by ojan@chromium.org

[commit-bot: I haz the power, 2014-06-04 02:18:10]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/dpranke@chromium.org/294973008/40001

[commit-bot: I haz the power, 2014-06-04 04:36:09]

FYI, CQ is re-trying this CL (attempt #1).
The failing builders are:
  win_blink_rel on tryserver.blink
(http://build.chromium.org/p/tryserver.blink/builders/win_blink_rel/builds/10524)
  win_gpu_triggered_tests on tryserver.chromium.gpu
(http://build.chromium.org/p/tryserver.chromium.gpu/builders/win_gpu_triggered...)

[commit-bot: I haz the power, 2014-06-04 04:36:33]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-06-04 04:36:34]

Try jobs failed on following builders:
  win_gpu_triggered_tests on tryserver.chromium.gpu
(http://build.chromium.org/p/tryserver.chromium.gpu/builders/win_gpu_triggered...)

[earthdok, 2014-06-04 13:42:20]

The CQ bit was checked by earthdok@chromium.org

[commit-bot: I haz the power, 2014-06-04 13:42:58]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/dpranke@chromium.org/294973008/40001

[commit-bot: I haz the power, 2014-06-04 14:35:24]

Change committed as 175488