Mod a hash of test names to determine a shard, rather than the test index.

Mod a hash of test names to determine a shard, rather than the test index.

This keeps tests on a particular shard, until their name changes.

Currently tests can move to a different waterfall bot between runs;
whenever a test is added or removed "before" it in the test suite. This
makes it very hard to track failure history of a particular test, since
it can "disappear" from prior builds on that bot.

Also, since the shards do not have synced cycles, a revision that breaks
test "A" might never be in a blamelist. E.g. Bad revision X breaks test
"A", good revision X+1 adds unrelated test "B". Blamelists
bot0:[X],[X+1] and bot1:[X, X+1]. Only bot0 will run test "A", and only
on its second run, but it will not blame revision "X". Note "X" could
even be the revision that adds test "A".

Hashing the name does not guarantee an even distribution of tests.
However, the duration of a test is not predictable, so the change to
load variance should balance out except in pathological cases or very
small test suites.

BUG=372461
TBR=phajdan.jr@chromium.org

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=272920

[tapted, 2014-05-22 11:45:30]

Hi Paweł - WDYT - is this a good idea?

[Paweł Hajdan Jr., 2014-05-22 13:37:52]

I see the reasons for this, and it's probably worth trying.

I'm worried about the possibly uneven distribution of tests between shards.

Adding Scott and John for advice - what do you think?

[sky, 2014-05-22 15:06:01]

On 2014/05/22 13:37:52, Paweł Hajdan Jr. wrote:
> I see the reasons for this, and it's probably worth trying.
> 
> I'm worried about the possibly uneven distribution of tests between shards.

In practice how does the distribution look for our tests?

> Adding Scott and John for advice - what do you think?

[tapted, 2014-05-23 00:11:51]

On 2014/05/22 15:06:01, sky wrote:
> On 2014/05/22 13:37:52, Paweł Hajdan Jr. wrote:
> > I see the reasons for this, and it's probably worth trying.
> > 
> > I'm worried about the possibly uneven distribution of tests between shards.
> 
> In practice how does the distribution look for our tests?

looking at browser_tests on ChromeOS

2 shards (target 1845): 1835, 1856
3 shards (target 1230): 1275, 1228, 1188
4 shards (target 0922): 912, 922, 944, 913

worst case: 3.7% more tests than the target.

On windows, it's creeps up to 5.7% worst case with the current set of tests:

2 shards (target 1578): 1550 1607
3 shards (target 1052): 1112 1030 1015
4 shards (target 0789): 771 810 779 797

2- and 4-shard cases are much more balanced. Maybe superfasthash.c isn't such a
fan of mod-3 :/

If this sounds bad, I can run the checks again using base/md5.h which should
have a better distribution, just taking 4 bytes from the digest and modding
that.

[Paweł Hajdan Jr., 2014-05-23 09:03:58]

I'm fine with that distribution. If there are no objections from Scott or John,
I'd happily approve.

[jam, 2014-05-23 15:28:06]

On 2014/05/23 09:03:58, Paweł Hajdan Jr. wrote:
> I'm fine with that distribution. If there are no objections from Scott or
John,
> I'd happily approve.

sounds like an improvement to me, thanks. just keep an eye out after landing it
to make sure none of the testers become very imbalanced

[sky, 2014-05-23 15:53:02]

LGTM too

On Fri, May 23, 2014 at 8:28 AM,  <jam@chromium.org> wrote:
> On 2014/05/23 09:03:58, Paweł Hajdan Jr. wrote:
>>
>> I'm fine with that distribution. If there are no objections from Scott or
>
> John,
>>
>> I'd happily approve.
>
>
> sounds like an improvement to me, thanks. just keep an eye out after landing
> it
> to make sure none of the testers become very imbalanced
>
> https://codereview.chromium.org/292973006/

To unsubscribe from this group and stop receiving emails from it, send an email
to chromium-reviews+unsubscribe@chromium.org.

[sky, 2014-05-23 15:53:58]

LGTM

[tapted, 2014-05-27 02:32:14]

The CQ bit was checked by tapted@chromium.org

[commit-bot: I haz the power, 2014-05-27 02:32:22]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/tapted@chromium.org/292973006/20001

[commit-bot: I haz the power, 2014-05-27 03:17:11]

FYI, CQ is re-trying this CL (attempt #1).
Please consider checking whether the failures are real,
and report flakes to chrome-troopers@google.com.
The failing builders are:
  chromium_presubmit on tryserver.chromium
(http://build.chromium.org/p/tryserver.chromium/builders/chromium_presubmit/bu...)

[commit-bot: I haz the power, 2014-05-27 03:21:41]

Change committed as 272920