Add sharding support to browser_test_runner.

Add sharding support to browser_test_runner.

The flags --total-shards and --shard-index control which tests are run
on which shard.

BUG=chromium:352807
Committed: https://chromium.googlesource.com/external/github.com/catapult-project/catapult/+/2984402997902f914c920b8134dc1da5dbb42dd6

[Ken Russell (switch to Gerrit), 2016-06-13 23:20:59]

kbr@chromium.org changed reviewers:
+ nednguyen@google.com

[Ken Russell (switch to Gerrit), 2016-06-13 23:20:59]

Ned: please review. Thanks.

Mo: FYI.

A follow-on Chromium CL will hook this up to Swarming.

https://codereview.chromium.org/2065853002/diff/1/telemetry/telemetry/testing...
File telemetry/telemetry/testing/browser_test_runner.py (right):

https://codereview.chromium.org/2065853002/diff/1/telemetry/telemetry/testing...
telemetry/telemetry/testing/browser_test_runner.py:157: break
This isn't strictly required, but seems like a reasonable fix.

[Zhenyao Mo, 2016-06-14 01:42:52]

This is great!

[nednguyen, 2016-06-14 01:48:33]

lgtm

https://codereview.chromium.org/2065853002/diff/1/telemetry/telemetry/testing...
File telemetry/telemetry/testing/browser_test_runner.py (right):

https://codereview.chromium.org/2065853002/diff/1/telemetry/telemetry/testing...
telemetry/telemetry/testing/browser_test_runner.py:157: break
On 2016/06/13 23:20:59, Ken Russell wrote:
> This isn't strictly required, but seems like a reasonable fix.

ok. I actually think it should throw exceptions if there are duplicates.

[Ken Russell (switch to Gerrit), 2016-06-14 17:41:31]

https://codereview.chromium.org/2065853002/diff/1/telemetry/telemetry/testing...
File telemetry/telemetry/testing/browser_test_runner.py (right):

https://codereview.chromium.org/2065853002/diff/1/telemetry/telemetry/testing...
telemetry/telemetry/testing/browser_test_runner.py:157: break
On 2016/06/14 01:48:33, nednguyen wrote:
> On 2016/06/13 23:20:59, Ken Russell wrote:
> > This isn't strictly required, but seems like a reasonable fix.
> 
> ok. I actually think it should throw exceptions if there are duplicates.

Right, ValidateDistinctNames, above, does that.

[Ken Russell (switch to Gerrit), 2016-06-14 17:41:51]

The CQ bit was checked by kbr@chromium.org

[Ken Russell (switch to Gerrit), 2016-06-14 17:41:52]

The patchset sent to the CQ was uploaded after l-g-t-m from nednguyen@google.com
Link to the patchset: https://codereview.chromium.org/2065853002/#ps20001
(title: "Rebased.")

[commit-bot: I haz the power, 2016-06-14 17:42:02]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/2065853002/20001

[nednguyen, 2016-06-14 17:49:45]

https://codereview.chromium.org/2065853002/diff/1/telemetry/telemetry/testing...
File telemetry/telemetry/testing/browser_test_runner.py (right):

https://codereview.chromium.org/2065853002/diff/1/telemetry/telemetry/testing...
telemetry/telemetry/testing/browser_test_runner.py:157: break
On 2016/06/14 17:41:31, Ken Russell wrote:
> On 2016/06/14 01:48:33, nednguyen wrote:
> > On 2016/06/13 23:20:59, Ken Russell wrote:
> > > This isn't strictly required, but seems like a reasonable fix.
> > 
> > ok. I actually think it should throw exceptions if there are duplicates.
> 
> Right, ValidateDistinctNames, above, does that.

Ah, I don't remember what we did :P

[commit-bot: I haz the power, 2016-06-14 18:05:27]

Description was changed from

==========
Add sharding support to browser_test_runner.

The flags --total-shards and --shard-index control which tests are run
on which shard.

BUG=chromium:352807
==========

to

==========
Add sharding support to browser_test_runner.

The flags --total-shards and --shard-index control which tests are run
on which shard.

BUG=chromium:352807

Committed:
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...
==========

[commit-bot: I haz the power, 2016-06-14 18:05:28]

Committed patchset #2 (id:20001) as
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...

[nednguyen, 2016-06-17 04:38:00]

https://codereview.chromium.org/2065853002/diff/20001/telemetry/telemetry/tes...
File telemetry/telemetry/testing/browser_test_runner.py (right):

https://codereview.chromium.org/2065853002/diff/20001/telemetry/telemetry/tes...
telemetry/telemetry/testing/browser_test_runner.py:52: |shard_index| is
currently being run.
Now that I think about this, why do we need such a complicated algorithm for
this? Can we just do s.t like:

for i in xrange(num_tests):
  if i % total_shards == shard_index:
      add num_tests[i] to list_of_tests_to_run

?

[Ken Russell (switch to Gerrit), 2016-06-22 02:02:51]

On 2016/06/17 04:38:00, nednguyen wrote:
>
https://codereview.chromium.org/2065853002/diff/20001/telemetry/telemetry/tes...
> File telemetry/telemetry/testing/browser_test_runner.py (right):
> 
>
https://codereview.chromium.org/2065853002/diff/20001/telemetry/telemetry/tes...
> telemetry/telemetry/testing/browser_test_runner.py:52: |shard_index| is
> currently being run.
> Now that I think about this, why do we need such a complicated algorithm for
> this? Can we just do s.t like:
> 
> for i in xrange(num_tests):
>   if i % total_shards == shard_index:
>       add num_tests[i] to list_of_tests_to_run
> 
> ?

I was trying to split things up alphabetically.

We are probably going to have to change this algorithm in the future because the
shards aren't well balanced right now and it's impractical to manually balance
them. I am looking into producing
https://www.chromium.org/developers/the-json-test-results-format and reading it
back in to get per-test execution times and use that to distribute tests among
shards.

[nednguyen, 2016-06-22 02:12:59]

On 2016/06/22 02:02:51, Ken Russell wrote:
> On 2016/06/17 04:38:00, nednguyen wrote:
> >
>
https://codereview.chromium.org/2065853002/diff/20001/telemetry/telemetry/tes...
> > File telemetry/telemetry/testing/browser_test_runner.py (right):
> > 
> >
>
https://codereview.chromium.org/2065853002/diff/20001/telemetry/telemetry/tes...
> > telemetry/telemetry/testing/browser_test_runner.py:52: |shard_index| is
> > currently being run.
> > Now that I think about this, why do we need such a complicated algorithm for
> > this? Can we just do s.t like:
> > 
> > for i in xrange(num_tests):
> >   if i % total_shards == shard_index:
> >       add num_tests[i] to list_of_tests_to_run
> > 
> > ?
> 
> I was trying to split things up alphabetically.
> 
> We are probably going to have to change this algorithm in the future because
the
> shards aren't well balanced right now and it's impractical to manually balance
> them. I am looking into producing
> https://www.chromium.org/developers/the-json-test-results-format and reading
it
> back in to get per-test execution times and use that to distribute tests among
> shards.

That sounds awesome.