[Android] Add support for external sharding to the test runner.

[Android] Add support for external sharding to the test runner.

This adds support for sharding via GTEST_SHARD_INDEX and
GTEST_TOTAL_SHARDS (along with their equivalent --test-launcher-*
command-line options), matching the behavior of //base/test/launcher
for both gtests and instrumentation tests.

BUG=692200
Review-Url: https://codereview.chromium.org/2743873003
Cr-Commit-Position: refs/heads/master@{#457001}
Committed: https://chromium.googlesource.com/chromium/src/+/84188b5c73a3bd70944aac7028dbdd449e9e5b73

[jbudorick, 2017-03-10 23:32:38]

The CQ bit was checked by jbudorick@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-03-10 23:33:39]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-03-11 00:11:59]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-03-11 00:12:00]

Dry run: Try jobs failed on following builders:
  linux_android_rel_ng on master.tryserver.chromium.android (JOB_FAILED,
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_androi...)

[jbudorick, 2017-03-11 01:06:37]

jbudorick@chromium.org changed reviewers:
+ bpastene@chromium.org

[jbudorick, 2017-03-11 01:06:37]

https://codereview.chromium.org/2743873003/diff/1/testing/buildbot/chromium.l...
File testing/buildbot/chromium.linux.json (right):

https://codereview.chromium.org/2743873003/diff/1/testing/buildbot/chromium.l...
testing/buildbot/chromium.linux.json:279: "shards": 2
This is for demonstration purposes only.

[mikecase (-- gone --), 2017-03-13 18:44:50]

mikecase@chromium.org changed reviewers:
+ mikecase@chromium.org

[mikecase (-- gone --), 2017-03-13 18:44:50]

https://codereview.chromium.org/2743873003/diff/20001/build/android/pylib/loc...
File build/android/pylib/local/device/local_device_test_run.py (right):

https://codereview.chromium.org/2743873003/diff/20001/build/android/pylib/loc...
build/android/pylib/local/device/local_device_test_run.py:183: if
hash(self._GetUniqueTestName(t)) % total_shards == shard_index]
I was curious how uniform the distribution of "hash(<typical test names>) %
SHARDS" would be. Here are the results I got for running against a set of Chrome
test names.

2 shards: [830, 797]
3 shards: [528, 534, 565]
4 shards: [403, 391, 427, 406]
5 shards: [333, 312, 328, 313, 341]
6 shards: [275, 257, 278, 253, 277, 287]
7 shards: [237, 226, 217, 232, 230, 249, 236]

The shard sizes are pretty even. But you could get them perfectly even by just
alphabetizing the test names, then just modding the array indices. Your approach
I guess is more robust if some shards for some reason have a different set of
tests?

[jbudorick, 2017-03-13 19:02:20]

https://codereview.chromium.org/2743873003/diff/20001/build/android/pylib/loc...
File build/android/pylib/local/device/local_device_test_run.py (right):

https://codereview.chromium.org/2743873003/diff/20001/build/android/pylib/loc...
build/android/pylib/local/device/local_device_test_run.py:183: if
hash(self._GetUniqueTestName(t)) % total_shards == shard_index]
On 2017/03/13 18:44:50, mikecase wrote:
> I was curious how uniform the distribution of "hash(<typical test names>) %
> SHARDS" would be. Here are the results I got for running against a set of
Chrome
> test names.
> 
> 2 shards: [830, 797]
> 3 shards: [528, 534, 565]
> 4 shards: [403, 391, 427, 406]
> 5 shards: [333, 312, 328, 313, 341]
> 6 shards: [275, 257, 278, 253, 277, 287]
> 7 shards: [237, 226, 217, 232, 230, 249, 236]
> 
> The shard sizes are pretty even. But you could get them perfectly even by just
> alphabetizing the test names, then just modding the array indices. Your
approach
> I guess is more robust if some shards for some reason have a different set of
> tests?

What I have here is also:
 - consistent with how base_test_launcher handles it (% hash algorithm, which I
didn't bother investigating):
https://codesearch.chromium.org/chromium/src/base/test/launcher/test_launcher...
 - stable -- holding hash method and total number of shards constant, a given
test will always get mapped to the same shard

[bpastene, 2017-03-14 01:03:47]

https://codereview.chromium.org/2743873003/diff/20001/build/android/pylib/loc...
File build/android/pylib/local/device/local_device_test_run.py (right):

https://codereview.chromium.org/2743873003/diff/20001/build/android/pylib/loc...
build/android/pylib/local/device/local_device_test_run.py:180: def
_ApplyExternalSharding(self, tests, shard_index, total_shards):
It might be helpful to log the shard args here. I know the base test launcher
prints something like:
"This is shard 1/5"

https://codereview.chromium.org/2743873003/diff/20001/build/android/pylib/loc...
build/android/pylib/local/device/local_device_test_run.py:181: return [
Any interest in complaining and/or exploding if shard_index > total_shards?

https://codereview.chromium.org/2743873003/diff/20001/build/android/test_runn...
File build/android/test_runner.py (left):

https://codereview.chromium.org/2743873003/diff/20001/build/android/test_runn...
build/android/test_runner.py:168: os.environ['PATH'] = adb_dir + os.pathsep +
os.environ['PATH']
Removing adb from PATH? That sounds scary. (and also unrelated to this cl?)

https://codereview.chromium.org/2743873003/diff/20001/testing/buildbot/chromi...
File testing/buildbot/chromium.linux.json (right):

https://codereview.chromium.org/2743873003/diff/20001/testing/buildbot/chromi...
testing/buildbot/chromium.linux.json:279: "shards": 2
Doooooope.

[jbudorick, 2017-03-14 15:36:01]

https://codereview.chromium.org/2743873003/diff/20001/build/android/pylib/loc...
File build/android/pylib/local/device/local_device_test_run.py (right):

https://codereview.chromium.org/2743873003/diff/20001/build/android/pylib/loc...
build/android/pylib/local/device/local_device_test_run.py:180: def
_ApplyExternalSharding(self, tests, shard_index, total_shards):
On 2017/03/14 01:03:46, bpastene wrote:
> It might be helpful to log the shard args here. I know the base test launcher
> prints something like:
> "This is shard 1/5"

Done.

https://codereview.chromium.org/2743873003/diff/20001/build/android/pylib/loc...
build/android/pylib/local/device/local_device_test_run.py:181: return [
On 2017/03/14 01:03:46, bpastene wrote:
> Any interest in complaining and/or exploding if shard_index > total_shards?

Done.

https://codereview.chromium.org/2743873003/diff/20001/build/android/test_runn...
File build/android/test_runner.py (left):

https://codereview.chromium.org/2743873003/diff/20001/build/android/test_runn...
build/android/test_runner.py:168: os.environ['PATH'] = adb_dir + os.pathsep +
os.environ['PATH']
On 2017/03/14 01:03:47, bpastene wrote:
> Removing adb from PATH? That sounds scary. (and also unrelated to this cl?)

oops, this should be in the prior CL in the chain.

[bpastene, 2017-03-14 17:36:26]

lgtm

[jbudorick, 2017-03-14 20:54:12]

ptal at shards

[bpastene, 2017-03-14 21:02:13]

lgtm + comment

https://codereview.chromium.org/2743873003/diff/60001/testing/buildbot/chromi...
File testing/buildbot/chromium.swarm.json (right):

https://codereview.chromium.org/2743873003/diff/60001/testing/buildbot/chromi...
testing/buildbot/chromium.swarm.json:9: "shards": 7,
I'm pretty sure "shards" should go in the "swarming" dict. At least, that's how
your single larng tryjob did it

[jbudorick, 2017-03-14 22:15:15]

jbudorick@chromium.org changed reviewers:
+ dpranke@chromium.org

[jbudorick, 2017-03-14 22:15:16]

+dpranke for //testing/buildbot stamp

https://codereview.chromium.org/2743873003/diff/60001/testing/buildbot/chromi...
File testing/buildbot/chromium.swarm.json (right):

https://codereview.chromium.org/2743873003/diff/60001/testing/buildbot/chromi...
testing/buildbot/chromium.swarm.json:9: "shards": 7,
On 2017/03/14 21:02:13, bpastene wrote:
> I'm pretty sure "shards" should go in the "swarming" dict. At least, that's
how
> your single larng tryjob did it

yep, fixed.

[Dirk Pranke, 2017-03-14 23:48:08]

lgtm

[jbudorick, 2017-03-14 23:51:54]

The CQ bit was checked by jbudorick@chromium.org

[jbudorick, 2017-03-14 23:51:54]

The patchset sent to the CQ was uploaded after l-g-t-m from
bpastene@chromium.org
Link to the patchset: https://codereview.chromium.org/2743873003/#ps80001
(title: "move shards into swarming spec")

[jbudorick, 2017-03-14 23:52:13]

The CQ bit was checked by jbudorick@chromium.org

[commit-bot: I haz the power, 2017-03-14 23:52:43]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-03-15 00:54:56]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-03-15 00:54:58]

Try jobs failed on following builders:
  android_n5x_swarming_rel on master.tryserver.chromium.android (JOB_FAILED,
https://build.chromium.org/p/tryserver.chromium.android/builders/android_n5x_...)

[jbudorick, 2017-03-15 04:30:23]

The CQ bit was checked by jbudorick@chromium.org

[jbudorick, 2017-03-15 04:30:23]

The patchset sent to the CQ was uploaded after l-g-t-m from
bpastene@chromium.org, dpranke@chromium.org
Link to the patchset: https://codereview.chromium.org/2743873003/#ps70008
(title: "borked rebase")

[commit-bot: I haz the power, 2017-03-15 04:30:41]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-03-15 05:29:28]

CQ is committing da patch.

Bot data: {"patchset_id": 70008, "attempt_start_ts": 1489552223566720,
"parent_rev": "1f8b68a795ee37c7e22de5deeb0eadfb28ec7b08", "commit_rev":
"84188b5c73a3bd70944aac7028dbdd449e9e5b73"}

[commit-bot: I haz the power, 2017-03-15 05:30:25]

Description was changed from

==========
[Android] Add support for external sharding to the test runner.

This adds support for sharding via GTEST_SHARD_INDEX and
GTEST_TOTAL_SHARDS (along with their equivalent --test-launcher-*
command-line options), matching the behavior of //base/test/launcher
for both gtests and instrumentation tests.

BUG=692200
==========

to

==========
[Android] Add support for external sharding to the test runner.

This adds support for sharding via GTEST_SHARD_INDEX and
GTEST_TOTAL_SHARDS (along with their equivalent --test-launcher-*
command-line options), matching the behavior of //base/test/launcher
for both gtests and instrumentation tests.

BUG=692200

Review-Url: https://codereview.chromium.org/2743873003
Cr-Commit-Position: refs/heads/master@{#457001}
Committed:
https://chromium.googlesource.com/chromium/src/+/84188b5c73a3bd70944aac7028db...
==========

[commit-bot: I haz the power, 2017-03-15 05:30:26]

Committed patchset #6 (id:70008) as
https://chromium.googlesource.com/chromium/src/+/84188b5c73a3bd70944aac7028db...