telemetry: Add ct_run_benchmark to run benchmarks in CT's repository

telemetry: Add ct_run_benchmark to run benchmarks in CT's repository.

Motivation:
Cluster Telemetry has its own set of page sets. This change is to get around copying
the CT page sets to the perf/page_sets/ at every run.


BUG=skia:3168

[rmistry, 2014-11-17 22:24:54]

rmistry@google.com changed reviewers:
+ ernstm@chromium.org

[rmistry, 2014-11-19 22:07:24]

Friendly Ping

[rmistry, 2014-11-21 19:14:05]

Friendly ping.

[ernstm, 2014-11-22 00:22:19]

Could you instead create benchmarks for all the page sets you need in the
cluster telemetry repository and add a customized run_benchmarks command that
looks for them in the right location?

[rmistry, 2014-11-22 00:30:30]

On 2014/11/22 00:22:19, ernstm wrote:
> Could you instead create benchmarks for all the page sets you need in the
> cluster telemetry repository and add a customized run_benchmarks command that
> looks for them in the right location?

I tried to create a generic solution because I was not sure how many page sets
will be needed in the future. This is because for Cluster Telemetry, there are
constantly new page set types being added: IndexSample10k, Mobile10k etc.

[rmistry, 2014-11-22 00:32:51]

On 2014/11/22 00:30:30, rmistry wrote:
> On 2014/11/22 00:22:19, ernstm wrote:
> > Could you instead create benchmarks for all the page sets you need in the
> > cluster telemetry repository and add a customized run_benchmarks command
that
> > looks for them in the right location?
> 
> I tried to create a generic solution because I was not sure how many page sets
> will be needed in the future. This is because for Cluster Telemetry, there are
> constantly new page set types being added: IndexSample10k, Mobile10k etc.

Oh I may have misunderstood what you meant. Did you mean the new
ct_run_benchmarks command will know to look in the right place for the pagesets?
if so that SGTM.

[ernstm, 2014-11-22 00:56:09]

On 2014/11/22 00:32:51, rmistry wrote:
> On 2014/11/22 00:30:30, rmistry wrote:
> > On 2014/11/22 00:22:19, ernstm wrote:
> > > Could you instead create benchmarks for all the page sets you need in the
> > > cluster telemetry repository and add a customized run_benchmarks command
> that
> > > looks for them in the right location?
> > 
> > I tried to create a generic solution because I was not sure how many page
sets
> > will be needed in the future. This is because for Cluster Telemetry, there
are
> > constantly new page set types being added: IndexSample10k, Mobile10k etc.
> 
> Oh I may have misunderstood what you meant. Did you mean the new
> ct_run_benchmarks command will know to look in the right place for the
pagesets?
> if so that SGTM.

you would need a ct_run_benchmarks, that knows where to look for you
*benchmarks*. Then you would need to create one benchmark for each page set.
Look for example at tools/perf/benchmarks/smoothness.py. There is one benchmark
in this file for every page sets. You would have to do the same for your page
sets, just add them locally somewhere in the cluster telemetry repository and
point ct_run_benchmark to them.

[rmistry, 2014-11-24 13:51:50]

On 2014/11/22 00:56:09, ernstm wrote:
> On 2014/11/22 00:32:51, rmistry wrote:
> > On 2014/11/22 00:30:30, rmistry wrote:
> > > On 2014/11/22 00:22:19, ernstm wrote:
> > > > Could you instead create benchmarks for all the page sets you need in
the
> > > > cluster telemetry repository and add a customized run_benchmarks command
> > that
> > > > looks for them in the right location?
> > > 
> > > I tried to create a generic solution because I was not sure how many page
> sets
> > > will be needed in the future. This is because for Cluster Telemetry, there
> are
> > > constantly new page set types being added: IndexSample10k, Mobile10k etc.
> > 
> > Oh I may have misunderstood what you meant. Did you mean the new
> > ct_run_benchmarks command will know to look in the right place for the
> pagesets?
> > if so that SGTM.
> 
> you would need a ct_run_benchmarks, that knows where to look for you
> *benchmarks*. Then you would need to create one benchmark for each page set.
> Look for example at tools/perf/benchmarks/smoothness.py. There is one
benchmark
> in this file for every page sets. You would have to do the same for your page
> sets, just add them locally somewhere in the cluster telemetry repository and
> point ct_run_benchmark to them.

K to make sure I understand:
I will have my own benchmarks similar to tools/perf/benchmarks/smoothness.py in
CT's repository which will make use of the measurements in
tools/perf/measurements.

How do I set up ct_run_benchmark to use my own benchmarks in a different
repository? Looking at the existing run_benchmark in
https://chromium.googlesource.com/chromium/src/+/master/tools/perf/run_benchmark
I am not sure how to proceed.

Thanks!

[ernstm, 2014-11-24 17:54:12]

> K to make sure I understand:
> I will have my own benchmarks similar to tools/perf/benchmarks/smoothness.py
in
> CT's repository which will make use of the measurements in
> tools/perf/measurements.

Correct
 
> How do I set up ct_run_benchmark to use my own benchmarks in a different
> repository? Looking at the existing run_benchmark in
>
https://chromium.googlesource.com/chromium/src/+/master/tools/perf/run_benchmark
> I am not sure how to proceed.

ct_run_benchmark would have to specify an additional base directory (that is
where we recursively search for benchmarks), e.g. in line 17 of
run_benchmark.py:
benchmark_runner.config = environment.Environment([base_dir, ct_base_dir])

[rmistry, 2014-11-24 19:14:41]

Patchset #2 (id:20001) has been deleted

[rmistry, 2014-11-24 19:14:47]

Patchset #2 (id:40001) has been deleted

[rmistry, 2014-11-24 19:18:36]

On 2014/11/24 17:54:12, ernstm wrote:
> > K to make sure I understand:
> > I will have my own benchmarks similar to tools/perf/benchmarks/smoothness.py
> in
> > CT's repository which will make use of the measurements in
> > tools/perf/measurements.
> 
> Correct
>  
> > How do I set up ct_run_benchmark to use my own benchmarks in a different
> > repository? Looking at the existing run_benchmark in
> >
>
https://chromium.googlesource.com/chromium/src/+/master/tools/perf/run_benchmark
> > I am not sure how to proceed.
> 
> ct_run_benchmark would have to specify an additional base directory (that is
> where we recursively search for benchmarks), e.g. in line 17 of
> run_benchmark.py:
> benchmark_runner.config = environment.Environment([base_dir, ct_base_dir])

Done. PTAL

[ernstm, 2014-11-24 20:15:38]

I though the ct_run_benchmark would live in the CT repository. Can you move it
there?

https://codereview.chromium.org/730033005/diff/80001/tools/perf/benchmarks/sk...
File tools/perf/benchmarks/skpicture_printer.py (right):

https://codereview.chromium.org/730033005/diff/80001/tools/perf/benchmarks/sk...
tools/perf/benchmarks/skpicture_printer.py:11: def
MatchPageSetName(page_set_name, page_set_base_dir):
Why does this need to become a public method of the module?

[rmistry, 2014-11-24 21:36:21]

On 2014/11/24 20:15:38, ernstm wrote:
> I though the ct_run_benchmark would live in the CT repository. Can you move it
> there?

Yes, did so and it is working.

Looking at what we have now (ct_run_benchmark and individual benchmarks in the
CT repository) and comparing it with PatchSet1 in this CL.
I prefer what I had in PatchSet1, because if there are any changes to benchmarks
or the run_benchmark script then it will be fixed in telemetry's repo.
Multiple times I have had to update my pagesets because of telemetry's
refactorings and other changes (it is a constantly moving codebase). With the
current change I will have to update the pagesets as well as the benchmarks and
ct_run_benchmark when there are changes. I would rather move as much as I can
into telemetry's codebase vs moving things out. I have not yet figured out how
to move CTs pagesets in here but that is my eventual goal.

WDYT?


> 
>
https://codereview.chromium.org/730033005/diff/80001/tools/perf/benchmarks/sk...
> File tools/perf/benchmarks/skpicture_printer.py (right):
> 
>
https://codereview.chromium.org/730033005/diff/80001/tools/perf/benchmarks/sk...
> tools/perf/benchmarks/skpicture_printer.py:11: def
> MatchPageSetName(page_set_name, page_set_base_dir):
> Why does this need to become a public method of the module?

[ernstm, 2014-11-25 00:28:35]

> Looking at what we have now (ct_run_benchmark and individual benchmarks in the
> CT repository) and comparing it with PatchSet1 in this CL.
> I prefer what I had in PatchSet1, because if there are any changes to
benchmarks
> or the run_benchmark script then it will be fixed in telemetry's repo.
> Multiple times I have had to update my pagesets because of telemetry's
> refactorings and other changes (it is a constantly moving codebase). With the
> current change I will have to update the pagesets as well as the benchmarks
and
> ct_run_benchmark when there are changes. I would rather move as much as I can
> into telemetry's codebase vs moving things out. I have not yet figured out how
> to move CTs pagesets in here but that is my eventual goal.
> 
> WDYT?

Why can't you move your page sets into telemetry?

[rmistry, 2014-11-25 01:52:03]

On 2014/11/25 00:28:35, ernstm wrote:
> > Looking at what we have now (ct_run_benchmark and individual benchmarks in
the
> > CT repository) and comparing it with PatchSet1 in this CL.
> > I prefer what I had in PatchSet1, because if there are any changes to
> benchmarks
> > or the run_benchmark script then it will be fixed in telemetry's repo.
> > Multiple times I have had to update my pagesets because of telemetry's
> > refactorings and other changes (it is a constantly moving codebase). With
the
> > current change I will have to update the pagesets as well as the benchmarks
> and
> > ct_run_benchmark when there are changes. I would rather move as much as I
can
> > into telemetry's codebase vs moving things out. I have not yet figured out
how
> > to move CTs pagesets in here but that is my eventual goal.
> > 
> > WDYT?
> 
> Why can't you move your page sets into telemetry?

I have 10k page sets, I figured it was too many. I could move them into a ct
directory under page_sets if that works for you guys?

[ernstm, 2014-11-25 18:33:01]

ernstm@chromium.org changed reviewers:
+ dtu@chromium.org, tonyg@chromium.org

[ernstm, 2014-11-25 18:33:02]

+dtu +tonyg

[tonyg, 2014-12-09 00:32:01]

Sorry for the delay. PageSets on the order of 10k URLs are fine here. The data
is all stored in cloud storage. The smoke tests should make sure they're not
broken by a refactor.

This change lgtm if we get rid of the ct_ abbreviation so that people understand
what it is. Maybe run_cluster_benchmark?

[rmistry, 2014-12-09 12:58:00]

On 2014/12/09 00:32:01, tonyg wrote:
> Sorry for the delay. PageSets on the order of 10k URLs are fine here. The data
> is all stored in cloud storage. The smoke tests should make sure they're not
> broken by a refactor.

By "data is all stored in cloud storage" I am assuming you mean that the
archives that correspond to the pagesets are stored in cloud storage but not the
pagesets themselves.
Also to clarify, here I mean 10k pagesets not a few pagesets with 10k URLs. This
means there will be 10k (or more) new files checked into the repository. Is
there any way to store these pagesets in cloud storage instead?
Apologies if I misunderstood something, I would love to figure out a way to
store the many pagesets of cluster telemetry and the few pagesets of Skia with
the rest of telemetry's pagesets to make sure we do not break with the frequent
changes to telemetry.

> 
> This change lgtm if we get rid of the ct_ abbreviation so that people
understand
> what it is. Maybe run_cluster_benchmark?

[tonyg, 2014-12-10 04:10:44]

On 2014/12/09 12:58:00, rmistry wrote:
> On 2014/12/09 00:32:01, tonyg wrote:
> > Sorry for the delay. PageSets on the order of 10k URLs are fine here. The
data
> > is all stored in cloud storage. The smoke tests should make sure they're not
> > broken by a refactor.
> 
> By "data is all stored in cloud storage" I am assuming you mean that the
> archives that correspond to the pagesets are stored in cloud storage but not
the
> pagesets themselves.
> Also to clarify, here I mean 10k pagesets not a few pagesets with 10k URLs.
This
> means there will be 10k (or more) new files checked into the repository

Oops, I misunderstood. Checking in 10k files probably won't be great for the git
repo. It also wouldn't be fair to ask devs to update them. Could you point me to
an example page set please?

> . Is
> there any way to store these pagesets in cloud storage instead?
> Apologies if I misunderstood something, I would love to figure out a way to
> store the many pagesets of cluster telemetry and the few pagesets of Skia with
> the rest of telemetry's pagesets to make sure we do not break with the
frequent
> changes to telemetry.
> 
> > 
> > This change lgtm if we get rid of the ct_ abbreviation so that people
> understand
> > what it is. Maybe run_cluster_benchmark?

[rmistry, 2014-12-10 14:04:01]

On 2014/12/10 04:10:44, tonyg wrote:
> On 2014/12/09 12:58:00, rmistry wrote:
> > On 2014/12/09 00:32:01, tonyg wrote:
> > > Sorry for the delay. PageSets on the order of 10k URLs are fine here. The
> data
> > > is all stored in cloud storage. The smoke tests should make sure they're
not
> > > broken by a refactor.
> > 
> > By "data is all stored in cloud storage" I am assuming you mean that the
> > archives that correspond to the pagesets are stored in cloud storage but not
> the
> > pagesets themselves.
> > Also to clarify, here I mean 10k pagesets not a few pagesets with 10k URLs.
> This
> > means there will be 10k (or more) new files checked into the repository
> 
> Oops, I misunderstood. Checking in 10k files probably won't be great for the
git
> repo. It also wouldn't be fair to ask devs to update them. Could you point me
to
> an example page set please?


Here is an example page set:


# Use of this source code is governed by a BSD-style license that can be        
# found in the LICENSE file.                                                    
# pylint: disable=W0401,W0614                                                   
                                                                                
from telemetry.page import page as page_module                                  
from telemetry.page import page_set as page_set_module                          
                                                                                
                                                                                
class TypicalAlexaPage(page_module.Page):                                       
                                                                                
  def __init__(self, url, page_set):                                            
    super(TypicalAlexaPage, self).__init__(url=url, page_set=page_set)          
    self.user_agent_type = 'mobile'                                             
    self.archive_data_file =
'/b/storage/webpage_archives/Dummy100/alexa1-1.json'
                                                                                
  def RunSmoothness(self, action_runner):                                       
    action_runner.ScrollElement()                                               
                                                                                
  def RunRepaint(self, action_runner):                                          
    action_runner.RepaintContinuously(seconds=5)                                
                                                                                
                                                                                
class Alexa1_1PageSet(page_set_module.PageSet):                                 
                                                                                
  def __init__(self):                                                           
    super(Alexa1_1PageSet, self).__init__(                                      
      user_agent_type='mobile',                                                 
      archive_data_file='/b/storage/webpage_archives/Dummy100/alexa1-1.json')   
                                                                                
    urls_list = ['http://www.google.com']                                       
                                                                                
    for url in urls_list:                                                       
      self.AddPage(TypicalAlexaPage(url, self))

[rmistry, 2014-12-10 14:07:00]

I would also like to revisit my changes in
https://codereview.chromium.org/730033005/#ps1
I prefer that change than to create a separate run_cluster_benchmark, because if
there are any changes to the benchmarks then it will be fixed in telemetry's
repo.

[tonyg, 2015-01-14 18:47:09]

tonyg@chromium.org changed reviewers:
- tonyg@chromium.org