Adding Google benchmarking library.

Adding Google benchmarking library.

In Chromium we do not have general tools for writing microbenchmarks, even though
we need it.

Therefor we constantly reinvent the wheel and resulting benchmarks take very long
time to write, much less precise than they should be, run longer than they can etc.

Some examples:
https://cs.chromium.org/chromium/src/base/json/json_perftest.cc
https://cs.chromium.org/chromium/src/base/message_loop/message_pump_perftest.cc
https://cs.chromium.org/chromium/src/base/threading/thread_perftest.cc
https://codereview.chromium.org/2745733005
https://codereview.chromium.org/2854353002

Google has an open source library: https://github.com/google/benchmark for doing
microbenchmarking. It has decent documentation and there is also a talk at CppCon
by Chandler Carruth https://youtu.be/nXaxk27zwlk

This cl adds this library to third party and adds a small example using it.

BUG=682215

[dyaroshev, 2017-05-05 22:26:27]

Patchset #1 (id:1) has been deleted

[dyaroshev, 2017-05-05 22:53:31]

Description was changed from

==========
Adding microbenchmarking library.

BUG=682215
==========

to

==========
Adding Google benchmarking library.

In Chromium we do not have general tools for writing microbenchmarks, even
though
we need it.

Therefor we constantly reinvent the wheel and resulting benchmarks take very
long
time to write, much less precise than they should be, run longer than they can
etc.

Some examples:
https://cs.chromium.org/chromium/src/base/json/json_perftest.cc
https://cs.chromium.org/chromium/src/base/message_loop/message_pump_perftest.cc
https://cs.chromium.org/chromium/src/base/threading/thread_perftest.cc
https://codereview.chromium.org/2745733005
https://codereview.chromium.org/2854353002

Google has an open source library: https://github.com/google/benchmark for doing
microbenchmarking. It has decent documentation and there is also a talk at
CppCon
by Chandler Carruth  https://youtu.be/nXaxk27zwlk

This cl adds this library to third party and adds a small example using it.

BUG=682215
==========

[dyaroshev, 2017-05-05 22:53:31]

dyaroshev@yandex-team.ru changed reviewers:
+ brettw@chromium.org, danakj@chromium.org, jdoerrie@chromium.org

[dyaroshev, 2017-05-05 22:53:56]

Description was changed from

==========
Adding Google benchmarking library.

In Chromium we do not have general tools for writing microbenchmarks, even
though
we need it.

Therefor we constantly reinvent the wheel and resulting benchmarks take very
long
time to write, much less precise than they should be, run longer than they can
etc.

Some examples:
https://cs.chromium.org/chromium/src/base/json/json_perftest.cc
https://cs.chromium.org/chromium/src/base/message_loop/message_pump_perftest.cc
https://cs.chromium.org/chromium/src/base/threading/thread_perftest.cc
https://codereview.chromium.org/2745733005
https://codereview.chromium.org/2854353002

Google has an open source library: https://github.com/google/benchmark for doing
microbenchmarking. It has decent documentation and there is also a talk at
CppCon
by Chandler Carruth  https://youtu.be/nXaxk27zwlk

This cl adds this library to third party and adds a small example using it.

BUG=682215
==========

to

==========
Adding Google benchmarking library.

In Chromium we do not have general tools for writing microbenchmarks, even
though
we need it.

Therefor we constantly reinvent the wheel and resulting benchmarks take very
long
time to write, much less precise than they should be, run longer than they can
etc.

Some examples:
https://cs.chromium.org/chromium/src/base/json/json_perftest.cc
https://cs.chromium.org/chromium/src/base/message_loop/message_pump_perftest.cc
https://cs.chromium.org/chromium/src/base/threading/thread_perftest.cc
https://codereview.chromium.org/2745733005
https://codereview.chromium.org/2854353002

Google has an open source library: https://github.com/google/benchmark for doing
microbenchmarking. It has decent documentation and there is also a talk at
CppCon
by Chandler Carruth https://youtu.be/nXaxk27zwlk

This cl adds this library to third party and adds a small example using it.

BUG=682215
==========

[dyaroshev, 2017-05-05 22:58:00]

dyaroshev@yandex-team.ru changed reviewers:
+ nduca@chromium.org

[dyaroshev, 2017-05-05 23:19:11]

Hi!

Are you generally OK with adding this library? Even without regular runs, this
is very handy tool to have at our disposal, at the very least benchmarks are
precise, fast and easy to write (without it I personally wasn't able to
benchmark adding 10 elements to an empty set).

Help wanted:
 - I had to disable a few checks, because the checkers kept running on a
third_party library - what do I do?
 - Who do I list as owners for the third party library?
 - I've just cloned github repo and then removed git related information. Some
third_party libraries are added like this, others
   are added as a nested repo and then downloaded via DEPS. I have very little
understanding on how/why/what to do here.

PS: here are results on my machine, if you are curious:
https://drive.google.com/file/d/0B5pZ43DhqaQDeXVlVHpoYUozOEE/view?usp=sharing -
please keep in mind that my machine is very likely to be much faster that the
user's one and results can differ.

[dyaroshev, 2017-05-05 23:21:21]

In third_party I wrote BUILD.gn and README.chromium. Other files are taken as is
from library.

[dyaroshev, 2017-05-09 21:38:22]

On 2017/05/05 23:21:21, dyaroshev wrote:
> In third_party I wrote BUILD.gn and README.chromium. Other files are taken as
is
> from library.

Ping. I think you would love this library.

[jdoerrie, 2017-05-10 20:40:18]

On 2017/05/09 21:38:22, dyaroshev wrote:
> On 2017/05/05 23:21:21, dyaroshev wrote:
> > In third_party I wrote BUILD.gn and README.chromium. Other files are taken
as
> is
> > from library.
> 
> Ping. I think you would love this library.

I am very interested in this library, but I have no experience with the process
of adding third_party libraries. Therefore I am not qualified to review this CL.

[danakj, 2017-05-10 20:45:53]

On 2017/05/10 20:40:18, jdoerrie (slow this week) wrote:
> On 2017/05/09 21:38:22, dyaroshev wrote:
> > On 2017/05/05 23:21:21, dyaroshev wrote:
> > > In third_party I wrote BUILD.gn and README.chromium. Other files are taken
> as
> > is
> > > from library.
> > 
> > Ping. I think you would love this library.
> 
> I am very interested in this library, but I have no experience with the
process
> of adding third_party libraries. Therefore I am not qualified to review this
CL.

brettw's a third_party OWNER but you could try another if he's not available
https://cs.chromium.org/chromium/src/third_party/OWNERS?q=third_party/owners&...

[dyaroshev, 2017-05-10 22:06:36]

On 2017/05/10 20:45:53, danakj wrote:
> On 2017/05/10 20:40:18, jdoerrie (slow this week) wrote:
> > On 2017/05/09 21:38:22, dyaroshev wrote:
> > > On 2017/05/05 23:21:21, dyaroshev wrote:
> > > > In third_party I wrote BUILD.gn and README.chromium. Other files are
taken
> > as
> > > is
> > > > from library.
> > > 
> > > Ping. I think you would love this library.
> > 
> > I am very interested in this library, but I have no experience with the
> process
> > of adding third_party libraries. Therefore I am not qualified to review this
> CL.
> 
> brettw's a third_party OWNER but you could try another if he's not available
>
https://cs.chromium.org/chromium/src/third_party/OWNERS?q=third_party/owners&...

To be completely honest, I was hoping he'd volunteer)

[brettw, 2017-05-10 22:36:45]

See these instructions for adding third_party libraries:
https://chromium.googlesource.com/chromium/src.git/+/master/docs/adding_to_th...

This is probably not possible for non-Googlers since some of the signoffs are
internal. We need these even for Google-written code (though legal and
open-source reviews are generally rubberstamps in this case).

I'll look at it and see if I have an opinion on adding this...

[brettw, 2017-05-10 23:11:54]

It looks like we don't want to add this.

Please see:
https://groups.google.com/a/chromium.org/d/msg/chromium-dev/PFqPgWLPUQ0/Dy6QE...
for the infra team's direction on this.

Separately, I'm pretty wary of adding more stuff. In principle this is pretty
easy and adding yet another way to do it just complicates things. We have
infrastructure for doing this and for outputting the logs in a specific format
that feed into the perf testing waterfalls. We're already having enough problems
with things like gmock adding a large number of magic macros and a complicated
system that people need to learn. And now gmock has decided to change things
around and we're stuck on an old version.

The base_perftests thing is kind of an orphan. It's not hooked up on any bots
and is entirely ad-hoc. It's not even hooked into the base perf testing
framework. We could certainly improve things without introducing new libraries.
I don't think we would want to hook up base_perftests to the waterfall,
especially in the context of wanting to reduce the amount of perf tests are run.
The current tests are primarily useful for local testing when you make specific
changes.

[dyaroshev, 2017-05-11 11:29:51]

On 2017/05/10 23:11:54, brettw (behind--catching up) wrote:
> It looks like we don't want to add this.
> 
> Please see:
>
https://groups.google.com/a/chromium.org/d/msg/chromium-dev/PFqPgWLPUQ0/Dy6QE...
> for the infra team's direction on this.
> 
> Separately, I'm pretty wary of adding more stuff. In principle this is pretty
> easy and adding yet another way to do it just complicates things. We have
> infrastructure for doing this and for outputting the logs in a specific format
> that feed into the perf testing waterfalls. We're already having enough
problems
> with things like gmock adding a large number of magic macros and a complicated
> system that people need to learn. And now gmock has decided to change things
> around and we're stuck on an old version.
> 
> The base_perftests thing is kind of an orphan. It's not hooked up on any bots
> and is entirely ad-hoc. It's not even hooked into the base perf testing
> framework. We could certainly improve things without introducing new
libraries.
> I don't think we would want to hook up base_perftests to the waterfall,
> especially in the context of wanting to reduce the amount of perf tests are
run.
> The current tests are primarily useful for local testing when you make
specific
> changes.

I read the post about adding third-party library and it doesn't answer my
questions, unfortunately:
1 - How to disable checks for a directory
2 - How to build third party library - I just created a BUILD.gn file, maybe
there is some way of incorporating CMAKE.
3 - When to add third party directly in repo and when to duplicate it on
specific servers.

I hear your concerns with respect to adding this library, but let me try to
convince you again.

1 - Writing good benchmarks is very hard and just calling things a bunch of
times in a loop doesn't cut it - precision is awful, inconsistent results. I've
spent a few days trying to benchmark sets - I ended up with a benchmark that
runs for 4 minutes and still doesn't give me consistent results.

2 - The main purpose of adding this library is to make base_perftests (and like
those) reliable and easy to create and run. This library allows to measure small
inputs, at least I can not do it well without it. Tricks with volatile, noinline
and staff like that are hard and you have to know what you do. People who wrote
this library do know.

3 - Even though macrosses have certain level of complexity to them, it's nothing
compared trying measuring small timings by hand.

4 - I don't actually suggest adding base_perftests to waterfall (I don't even
see your waterfall result)). I'd just say this is useful for us, as the way to
test your local changes.

So, I would ask, do you think that having microbenchmarks to test your own
changes in the repo is valuable? If no, I understand, but I don't actually
believe that at least I can do microbenchmarks without at least functionality
from this library.

[brettw, 2017-05-12 18:08:25]

On 2017/05/11 11:29:51, dyaroshev wrote:
> On 2017/05/10 23:11:54, brettw (behind--catching up) wrote:
> > It looks like we don't want to add this.
> > 
> > Please see:
> >
>
https://groups.google.com/a/chromium.org/d/msg/chromium-dev/PFqPgWLPUQ0/Dy6QE...
> > for the infra team's direction on this.
> > 
> > Separately, I'm pretty wary of adding more stuff. In principle this is
pretty
> > easy and adding yet another way to do it just complicates things. We have
> > infrastructure for doing this and for outputting the logs in a specific
format
> > that feed into the perf testing waterfalls. We're already having enough
> problems
> > with things like gmock adding a large number of magic macros and a
complicated
> > system that people need to learn. And now gmock has decided to change things
> > around and we're stuck on an old version.
> > 
> > The base_perftests thing is kind of an orphan. It's not hooked up on any
bots
> > and is entirely ad-hoc. It's not even hooked into the base perf testing
> > framework. We could certainly improve things without introducing new
> libraries.
> > I don't think we would want to hook up base_perftests to the waterfall,
> > especially in the context of wanting to reduce the amount of perf tests are
> run.
> > The current tests are primarily useful for local testing when you make
> specific
> > changes.
> 
> I read the post about adding third-party library and it doesn't answer my
> questions, unfortunately:
> 1 - How to disable checks for a directory
> 2 - How to build third party library - I just created a BUILD.gn file, maybe
> there is some way of incorporating CMAKE.
> 3 - When to add third party directly in repo and when to duplicate it on
> specific servers.
> 
> I hear your concerns with respect to adding this library, but let me try to
> convince you again.

This isn't just me, I talked to some others too who have similar opinions. We do
a lot of perf testing, I would check out some of the other infrastructure. If
there are additions we can make to these things that support your use-case
better, we should discuss those.

This doesn't integrate with our perf testing infrastructure, and the main
problem we have lately is too much complexity rather than specific tasks being
hard. I have found some microbenchmarks useful, but only rarely so. We're
unlikely to start running these tests on the bots, for example, so the benefit
of this doesn't justify adding more complexity to the project.

[dyaroshev, 2017-05-12 18:13:30]

On 2017/05/12 at 18:08:25, brettw wrote:
> This isn't just me, I talked to some others too who have similar opinions. We
do a lot of perf testing, I would check out some of the other infrastructure. If
there are additions we can make to these things that support your use-case
better, we should discuss those.
> 
> This doesn't integrate with our perf testing infrastructure, and the main
problem we have lately is too much complexity rather than specific tasks being
hard. I have found some microbenchmarks useful, but only rarely so. We're
unlikely to start running these tests on the bots, for example, so the benefit
of this doesn't justify adding more complexity to the project.

Ok, I see your point, thanks for considering.

Of course I would make the additions to the infrastructure, if there were easy
enough - unfortunately fighting optimiser and time measuring is not and it
requires too much specific knowledge that I don't have.

I can write benchmarks on a case by case basis, when I need to. It has a few
downsides but doable.