Add ClangToTLinuxAsanLibfuzzer

Add ClangToTLinuxAsanLibfuzzer

This bot builds a fuzzer with the pinned libFuzzer revision
and ToT clang.

BUG=635952
Committed: https://chromium.googlesource.com/chromium/tools/build/+/e0d6ce3e5ab8c9c27ee2147334802a41502f7eb6

[Oliver Chang, 2016-08-31 16:09:04]

Description was changed from

==========
Add ClangToTLinuxAsanLibfuzzer

BUG=635952
==========

to

==========
Add ClangToTLinuxAsanLibfuzzer

BUG=635952
==========

[Oliver Chang, 2016-08-31 16:09:04]

ochang@chromium.org changed reviewers:
+ aizatsky@chromium.org, dpranke@chromium.org, thakis@chromium.org

[Oliver Chang, 2016-08-31 16:09:18]

ptal, https://codereview.chromium.org/2294963003/ for the mb change.

[Nico, 2016-08-31 16:13:04]

Thanks! This builds with trunk clang but pinned libfuzzer, yes?

https://codereview.chromium.org/2293913004/diff/1/scripts/slave/recipes/chrom...
File scripts/slave/recipes/chromium_libfuzzer_clang_tot.py (right):

https://codereview.chromium.org/2293913004/diff/1/scripts/slave/recipes/chrom...
scripts/slave/recipes/chromium_libfuzzer_clang_tot.py:50:
api.chromium.run_mb(mastername, buildername, use_goma=False)
This looks like a pretty standard config – why can't this just be in
chromium_fyi.py?

[Oliver Chang, 2016-08-31 16:21:15]

On 2016/08/31 16:13:04, Nico wrote:
> Thanks! This builds with trunk clang but pinned libfuzzer, yes?

Yep! 

> 
>
https://codereview.chromium.org/2293913004/diff/1/scripts/slave/recipes/chrom...
> File scripts/slave/recipes/chromium_libfuzzer_clang_tot.py (right):
> 
>
https://codereview.chromium.org/2293913004/diff/1/scripts/slave/recipes/chrom...
> scripts/slave/recipes/chromium_libfuzzer_clang_tot.py:50:
> api.chromium.run_mb(mastername, buildername, use_goma=False)
> This looks like a pretty standard config – why can't this just be in
> chromium_fyi.py?

Mostly because I wasn't sure if it's possible to run custom steps e.g.
"out/Release/empty_fuzzer -runs=1" in chromium_fyi.py, and we will probably add
a couple more steps like this in the future.

[Dirk Pranke, 2016-08-31 16:24:29]

lgtm.

I don't think we need to reuse the existing recipes if the builder is trying to
do something fundamentally different (and simpler).

However, if you do plan to start running some of the regular gtests, we should
switch to the regular chromium recipes.

[Oliver Chang, 2016-08-31 16:33:55]

Description was changed from

==========
Add ClangToTLinuxAsanLibfuzzer

BUG=635952
==========

to

==========
Add ClangToTLinuxAsanLibfuzzer

This bot builds a fuzzer with the pinned libFuzzer revision
and ToT clang.

BUG=635952
==========

[Nico, 2016-08-31 16:56:02]

On 2016/08/31 16:21:15, Oliver Chang wrote:
> On 2016/08/31 16:13:04, Nico wrote:
> > Thanks! This builds with trunk clang but pinned libfuzzer, yes?
> 
> Yep! 
> 
> > 
> >
>
https://codereview.chromium.org/2293913004/diff/1/scripts/slave/recipes/chrom...
> > File scripts/slave/recipes/chromium_libfuzzer_clang_tot.py (right):
> > 
> >
>
https://codereview.chromium.org/2293913004/diff/1/scripts/slave/recipes/chrom...
> > scripts/slave/recipes/chromium_libfuzzer_clang_tot.py:50:
> > api.chromium.run_mb(mastername, buildername, use_goma=False)
> > This looks like a pretty standard config – why can't this just be in
> > chromium_fyi.py?
> 
> Mostly because I wasn't sure if it's possible to run custom steps e.g.
> "out/Release/empty_fuzzer -runs=1" in chromium_fyi.py, and we will probably
add
> a couple more steps like this in the future.

Just add

        "args": [
          "-runs=1"
        ],

in testing/buildbot/chromium.fyi.json I think?

[Oliver Chang, 2016-08-31 17:21:28]

On 2016/08/31 16:56:02, Nico wrote:
> On 2016/08/31 16:21:15, Oliver Chang wrote:
> > On 2016/08/31 16:13:04, Nico wrote:
> > > Thanks! This builds with trunk clang but pinned libfuzzer, yes?
> > 
> > Yep! 
> > 
> > > 
> > >
> >
>
https://codereview.chromium.org/2293913004/diff/1/scripts/slave/recipes/chrom...
> > > File scripts/slave/recipes/chromium_libfuzzer_clang_tot.py (right):
> > > 
> > >
> >
>
https://codereview.chromium.org/2293913004/diff/1/scripts/slave/recipes/chrom...
> > > scripts/slave/recipes/chromium_libfuzzer_clang_tot.py:50:
> > > api.chromium.run_mb(mastername, buildername, use_goma=False)
> > > This looks like a pretty standard config – why can't this just be in
> > > chromium_fyi.py?
> > 
> > Mostly because I wasn't sure if it's possible to run custom steps e.g.
> > "out/Release/empty_fuzzer -runs=1" in chromium_fyi.py, and we will probably
> add
> > a couple more steps like this in the future.
> 
> Just add
> 
>         "args": [
>           "-runs=1"
>         ],
> 
> in testing/buildbot/chromium.fyi.json I think?

Hmm, I still think it might be better for us to have a separate recipe, as it
gives more flexibility when we want to do more complex things in the future
like: 

cmake -GNinja  -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
-DLLVM_USE_SANITIZER=Address -DLLVM_USE_SANITIZE_COVERAGE=YES
-DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON /path/to/llvm
ninja check-fuzzer

[Nico, 2016-08-31 17:25:58]

On 2016/08/31 17:21:28, Oliver Chang wrote:
> On 2016/08/31 16:56:02, Nico wrote:
> > On 2016/08/31 16:21:15, Oliver Chang wrote:
> > > On 2016/08/31 16:13:04, Nico wrote:
> > > > Thanks! This builds with trunk clang but pinned libfuzzer, yes?
> > > 
> > > Yep! 
> > > 
> > > > 
> > > >
> > >
> >
>
https://codereview.chromium.org/2293913004/diff/1/scripts/slave/recipes/chrom...
> > > > File scripts/slave/recipes/chromium_libfuzzer_clang_tot.py (right):
> > > > 
> > > >
> > >
> >
>
https://codereview.chromium.org/2293913004/diff/1/scripts/slave/recipes/chrom...
> > > > scripts/slave/recipes/chromium_libfuzzer_clang_tot.py:50:
> > > > api.chromium.run_mb(mastername, buildername, use_goma=False)
> > > > This looks like a pretty standard config – why can't this just be
in
> > > > chromium_fyi.py?
> > > 
> > > Mostly because I wasn't sure if it's possible to run custom steps e.g.
> > > "out/Release/empty_fuzzer -runs=1" in chromium_fyi.py, and we will
probably
> > add
> > > a couple more steps like this in the future.
> > 
> > Just add
> > 
> >         "args": [
> >           "-runs=1"
> >         ],
> > 
> > in testing/buildbot/chromium.fyi.json I think?
> 
> Hmm, I still think it might be better for us to have a separate recipe, as it
> gives more flexibility when we want to do more complex things in the future
> like: 
> 
> cmake -GNinja  -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
> -DLLVM_USE_SANITIZER=Address -DLLVM_USE_SANITIZE_COVERAGE=YES
> -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON /path/to/llvm
> ninja check-fuzzer

1. YAGNI :-)
2. What'd be the motivation for doing that?

[Oliver Chang, 2016-08-31 19:19:55]

On 2016/08/31 17:25:58, Nico wrote:
> On 2016/08/31 17:21:28, Oliver Chang wrote:
> > On 2016/08/31 16:56:02, Nico wrote:
> > > On 2016/08/31 16:21:15, Oliver Chang wrote:
> > > > On 2016/08/31 16:13:04, Nico wrote:
> > > > > Thanks! This builds with trunk clang but pinned libfuzzer, yes?
> > > > 
> > > > Yep! 
> > > > 
> > > > > 
> > > > >
> > > >
> > >
> >
>
https://codereview.chromium.org/2293913004/diff/1/scripts/slave/recipes/chrom...
> > > > > File scripts/slave/recipes/chromium_libfuzzer_clang_tot.py (right):
> > > > > 
> > > > >
> > > >
> > >
> >
>
https://codereview.chromium.org/2293913004/diff/1/scripts/slave/recipes/chrom...
> > > > > scripts/slave/recipes/chromium_libfuzzer_clang_tot.py:50:
> > > > > api.chromium.run_mb(mastername, buildername, use_goma=False)
> > > > > This looks like a pretty standard config – why can't this just be
> in
> > > > > chromium_fyi.py?
> > > > 
> > > > Mostly because I wasn't sure if it's possible to run custom steps e.g.
> > > > "out/Release/empty_fuzzer -runs=1" in chromium_fyi.py, and we will
> probably
> > > add
> > > > a couple more steps like this in the future.
> > > 
> > > Just add
> > > 
> > >         "args": [
> > >           "-runs=1"
> > >         ],
> > > 
> > > in testing/buildbot/chromium.fyi.json I think?
> > 
> > Hmm, I still think it might be better for us to have a separate recipe, as
it
> > gives more flexibility when we want to do more complex things in the future
> > like: 
> > 
> > cmake -GNinja  -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
> > -DLLVM_USE_SANITIZER=Address -DLLVM_USE_SANITIZE_COVERAGE=YES
> > -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON /path/to/llvm
> > ninja check-fuzzer
> 
> 1. YAGNI :-)
> 2. What'd be the motivation for doing that?

That should run a bunch of more comprehensive tests for libFuzzer, but might be
pretty annoying to set up an not necessary for now. Just an option in case
running empty_fuzzer -runs=1 doesn't turn out to be enough and something breaks
again that we don't notice.

Also, I notice that tests are specified under a "gtest_tests" key in
chromium.fyi.json. Can I expect this to work with non-gtest binaries? It seems
to me that there's a lot of magic happening here that isn't necessary for what
we want to do :)

[Dirk Pranke, 2016-08-31 19:35:05]

On 2016/08/31 19:19:55, Oliver Chang wrote:
> On 2016/08/31 17:25:58, Nico wrote:
> > On 2016/08/31 17:21:28, Oliver Chang wrote:
> > > On 2016/08/31 16:56:02, Nico wrote:
> > > > On 2016/08/31 16:21:15, Oliver Chang wrote:
> > > > > On 2016/08/31 16:13:04, Nico wrote:
> > > > > > Thanks! This builds with trunk clang but pinned libfuzzer, yes?
> > > > > 
> > > > > Yep! 
> > > > > 
> > > > > > 
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://codereview.chromium.org/2293913004/diff/1/scripts/slave/recipes/chrom...
> > > > > > File scripts/slave/recipes/chromium_libfuzzer_clang_tot.py (right):
> > > > > > 
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://codereview.chromium.org/2293913004/diff/1/scripts/slave/recipes/chrom...
> > > > > > scripts/slave/recipes/chromium_libfuzzer_clang_tot.py:50:
> > > > > > api.chromium.run_mb(mastername, buildername, use_goma=False)
> > > > > > This looks like a pretty standard config – why can't this just
be
> > in
> > > > > > chromium_fyi.py?
> > > > > 
> > > > > Mostly because I wasn't sure if it's possible to run custom steps e.g.
> > > > > "out/Release/empty_fuzzer -runs=1" in chromium_fyi.py, and we will
> > probably
> > > > add
> > > > > a couple more steps like this in the future.
> > > > 
> > > > Just add
> > > > 
> > > >         "args": [
> > > >           "-runs=1"
> > > >         ],
> > > > 
> > > > in testing/buildbot/chromium.fyi.json I think?
> > > 
> > > Hmm, I still think it might be better for us to have a separate recipe, as
> it
> > > gives more flexibility when we want to do more complex things in the
future
> > > like: 
> > > 
> > > cmake -GNinja  -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
> > > -DLLVM_USE_SANITIZER=Address -DLLVM_USE_SANITIZE_COVERAGE=YES
> > > -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON /path/to/llvm
> > > ninja check-fuzzer
> > 
> > 1. YAGNI :-)
> > 2. What'd be the motivation for doing that?
> 
> That should run a bunch of more comprehensive tests for libFuzzer, but might
be
> pretty annoying to set up an not necessary for now. Just an option in case
> running empty_fuzzer -runs=1 doesn't turn out to be enough and something
breaks
> again that we don't notice.
> 
> Also, I notice that tests are specified under a "gtest_tests" key in
> chromium.fyi.json. Can I expect this to work with non-gtest binaries? It seems
> to me that there's a lot of magic happening here that isn't necessary for what
> we want to do :)

There is, indeed. 

Entries in the 'gtest_tests' keys are expected to conform to the gtest +
base/test/launcher
command line API.

Really, we want everything to be configured src-side, and as part of that it
might be ideal
to have a simple per-builder 'steps' entry for running arbitrary commands, but
we don't have
that now.

So, I really think this CL is best done as-is, rather than trying to use the
chromium recipes.
We can change things later as our needs evolve.

[Oliver Chang, 2016-09-01 16:06:06]

The CQ bit was checked by ochang@chromium.org

[commit-bot: I haz the power, 2016-09-01 16:06:14]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-09-01 16:14:24]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-09-01 16:14:25]

Try jobs failed on following builders:
  Build Presubmit on luci.infra.try (JOB_FAILED,
https://luci-milo.appspot.com/swarming/task/30ff51e283607110)

[Oliver Chang, 2016-09-01 16:19:25]

The CQ bit was checked by ochang@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-09-01 16:19:34]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-09-01 16:24:04]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-09-01 16:24:05]

Dry run: This issue passed the CQ dry run.

[Oliver Chang, 2016-09-01 16:26:45]

The CQ bit was checked by ochang@chromium.org

[Oliver Chang, 2016-09-01 16:26:46]

The patchset sent to the CQ was uploaded after l-g-t-m from dpranke@chromium.org
Link to the patchset: https://codereview.chromium.org/2293913004/#ps20001
(title: "Fix typo")

[commit-bot: I haz the power, 2016-09-01 16:26:55]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-09-01 16:31:11]

Description was changed from

==========
Add ClangToTLinuxAsanLibfuzzer

This bot builds a fuzzer with the pinned libFuzzer revision
and ToT clang.

BUG=635952
==========

to

==========
Add ClangToTLinuxAsanLibfuzzer

This bot builds a fuzzer with the pinned libFuzzer revision
and ToT clang.

BUG=635952

Committed:
https://chromium.googlesource.com/chromium/tools/build/+/e0d6ce3e5ab8c9c27ee2...
==========

[commit-bot: I haz the power, 2016-09-01 16:31:12]

Committed patchset #2 (id:20001) as
https://chromium.googlesource.com/chromium/tools/build/+/e0d6ce3e5ab8c9c27ee2...