Iteratively increase sample size for good/bad classification.

Iteratively increase sample size for good/bad classification.

This change makes the bisect recipe use scipy's implementation of
Mann-Whitney's U test to determine if a significant different exists
between two given revisions.

It also changes the bisection step by requiring more tests to be run
when the sample is insufficient to establish a difference between the
revision being evaluated and either of the FKBR/LKGR.

R=dtu,qyearsley,prasadv
BUG=565606
Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=298810

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=298830

[RobertoCN, 2016-01-23 03:07:48]

Please take a look, this still needs more thorough testing and to make the
initial regression check (good significantly different from bad) also iterative,
but other than that please critique this approach.

[qyearsley, 2016-01-26 19:26:00]

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
File scripts/slave/recipe_modules/auto_bisect/bisector.py (right):

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/bisector.py:130: 'See json.output for
details']
How will this show up on the build page? Will there be a log link (along with
stdio and json.output) named either "True" or "False"?

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
File scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py (right):

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:42: self.values
= self.values or [] + results['values']
`or` is less tightly binding than `+`, so this is equivalent to:

  self.values = self.values or ([] + results['values'])

And if results['values'] is a list, then ([] + results['values']) should be the
same as results['values'] (I think).

Should this be changed to the following?

  self.values = (self.values or []) + results['values']

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:128: if
self.custom_reps: # pragma: no cover
Two spaces before #.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:132: # shapiro's
test for normality [even though we don't really use it])
The parenthetical part of this comment could probably be removed.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:137: # more.
I'm not sure whether this comment is helpful; maybe it could be removed?

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:197:
self.bisector.wait_for_any([self])
This triggers the bisect to run another |self.custom_reps| times, right?

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:215: sizes =
map(lambda x: len(x.values),  [self.bisector.lkgr, self,
Extra space after the comma.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:220:
min_increase = math.ceil(smallest_sample_size/3.0)
Optional: could add spaces around the / operator.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:223: def
_check_revision_good(self):
Parts of the previous docstring could be kept here -- overall, it still returns
True if it's going to be considered "good", and False otherwise.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:236: if
diff_from_good and diff_from_bad:
A blank line above this if block might slightly help with readability.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:239: # handle
the secondary regression by spinning a second job.
No second job is started as of this CL, right?

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
File
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different.py
(right):

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different.py:46:
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
Why is it necessary to do this through anaconda? Otherwise scipy isn't
available?

(Could add a comment about this somewhere.)

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
File
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different_test.py
(right):

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different_test.py:18:
# pylint: enable=unused-variable
I think it also works to just put the pylint: disable=unused-variable on the
same line as `import scipy`, and then it should only affect that one line.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different_test.py:26:
['', '[1, 2, 3, 3, 2, 1]', '[1, 2, 2, 2, 5, 0]', 0.05], self.conda_path)
Does this actually run anaconda and run the statistical tests?

Since the main() function doesn't return values, and it just prints them, it's a
bit hard to test in a unit test; maybe it would be better to put the main logic
in another function which returns the results dict, and then main() just calls
that function and prints the results as JSON. Then, in the test, you could call
the function with the logic and assert that you get certain results.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
File scripts/slave/recipe_modules/auto_bisect/revision_state.py (right):

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/revision_state.py:34: NEED_MORE_DATA,
You could add a comment for this status -- still not enough evidence to classify
this revision as good or bad, waiting for more test runs (although it's fairly
clear from the name, the other statuses are also somewhat clear from the name).

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/revision_state.py:155:
RevisionState.NEED_MORE_DATA]:
Below when checking whether status is one of several things, you use a tuple
rather than list; could also change this to use a tuple.

[RobertoCN, 2016-02-01 17:29:51]

Addressed all feedback.

Also hardcoding 5 as the minimum required sample size.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
File scripts/slave/recipe_modules/auto_bisect/bisector.py (right):

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/bisector.py:130: 'See json.output for
details']
On 2016/01/26 19:26:00, qyearsley wrote:
> How will this show up on the build page? Will there be a log link (along with
> stdio and json.output) named either "True" or "False"?

https://goo.gl/photos/E9AuQkm9C8Mpjyvw7

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
File scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py (right):

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:42: self.values
= self.values or [] + results['values']
On 2016/01/26 19:26:00, qyearsley wrote:
> `or` is less tightly binding than `+`, so this is equivalent to:
> 
>   self.values = self.values or ([] + results['values'])
> 
> And if results['values'] is a list, then ([] + results['values']) should be
the
> same as results['values'] (I think).
> 
> Should this be changed to the following?
> 
>   self.values = (self.values or []) + results['values']

Done.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:128: if
self.custom_reps: # pragma: no cover
On 2016/01/26 19:26:00, qyearsley wrote:
> Two spaces before #.

Done.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:132: # shapiro's
test for normality [even though we don't really use it])
On 2016/01/26 19:26:00, qyearsley wrote:
> The parenthetical part of this comment could probably be removed.

Done.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:137: # more.
On 2016/01/26 19:26:00, qyearsley wrote:
> I'm not sure whether this comment is helpful; maybe it could be removed?

Yep, I was supposed to remove it and forgot.

Done.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:197:
self.bisector.wait_for_any([self])
On 2016/01/26 19:26:00, qyearsley wrote:
> This triggers the bisect to run another |self.custom_reps| times, right?

Yeah.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:215: sizes =
map(lambda x: len(x.values),  [self.bisector.lkgr, self,
On 2016/01/26 19:26:00, qyearsley wrote:
> Extra space after the comma.

Done.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:220:
min_increase = math.ceil(smallest_sample_size/3.0)
On 2016/01/26 19:26:00, qyearsley wrote:
> Optional: could add spaces around the / operator.

Done.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:223: def
_check_revision_good(self):
On 2016/01/26 19:26:00, qyearsley wrote:
> Parts of the previous docstring could be kept here -- overall, it still
returns
> True if it's going to be considered "good", and False otherwise.

Done.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:236: if
diff_from_good and diff_from_bad:
On 2016/01/26 19:26:00, qyearsley wrote:
> A blank line above this if block might slightly help with readability.

Done.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:239: # handle
the secondary regression by spinning a second job.
On 2016/01/26 19:26:00, qyearsley wrote:
> No second job is started as of this CL, right?

Correct, for now we'll go with the larger regression (calculated by distance of
means)

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
File
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different.py
(right):

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different.py:46:
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
On 2016/01/26 19:26:00, qyearsley wrote:
> Why is it necessary to do this through anaconda? Otherwise scipy isn't
> available?
> 
> (Could add a comment about this somewhere.)

Added the explanation to the docstring of bisector.significantly_different().

Done.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
File
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different_test.py
(right):

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different_test.py:18:
# pylint: enable=unused-variable
On 2016/01/26 19:26:00, qyearsley wrote:
> I think it also works to just put the pylint: disable=unused-variable on the
> same line as `import scipy`, and then it should only affect that one line.

Done.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different_test.py:26:
['', '[1, 2, 3, 3, 2, 1]', '[1, 2, 2, 2, 5, 0]', 0.05], self.conda_path)
On 2016/01/26 19:26:00, qyearsley wrote:
> Does this actually run anaconda and run the statistical tests?
> 
> Since the main() function doesn't return values, and it just prints them, it's
a
> bit hard to test in a unit test; maybe it would be better to put the main
logic
> in another function which returns the results dict, and then main() just calls
> that function and prints the results as JSON. Then, in the test, you could
call
> the function with the logic and assert that you get certain results.

Done.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
File scripts/slave/recipe_modules/auto_bisect/revision_state.py (right):

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/revision_state.py:34: NEED_MORE_DATA,
On 2016/01/26 19:26:00, qyearsley wrote:
> You could add a comment for this status -- still not enough evidence to
classify
> this revision as good or bad, waiting for more test runs (although it's fairly
> clear from the name, the other statuses are also somewhat clear from the
name).

Done.

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/revision_state.py:155:
RevisionState.NEED_MORE_DATA]:
On 2016/01/26 19:26:00, qyearsley wrote:
> Below when checking whether status is one of several things, you use a tuple
> rather than list; could also change this to use a tuple.

Done.

[qyearsley, 2016-02-02 00:40:41]

LGTM now, probably should wait for feedback from Dave or Prasad before
committing.

[RobertoCN, 2016-02-03 23:11:45]

I will test this on my local machine to make sure it works as intended, but the
code is pretty much ready. PTAL.

[dtu, 2016-02-05 00:18:02]

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
File scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py (right):

https://codereview.chromium.org/1610203003/diff/20001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:219: delta =
target_sample_size - smallest_sample_size
delta is unused.

https://codereview.chromium.org/1610203003/diff/80001/scripts/slave/recipe_mo...
File scripts/slave/recipe_modules/auto_bisect/bisector.py (right):

https://codereview.chromium.org/1610203003/diff/80001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/bisector.py:48:
REGRESSION_CHECK_TIMEOUT = 6 * 60 * 60
Is that six hours? Do we really have that much time?

https://codereview.chromium.org/1610203003/diff/80001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/bisector.py:89:
'required_initial_confidence', 80))
Should we rethink what this value is? Also make it a constant?

https://codereview.chromium.org/1610203003/diff/80001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/bisector.py:486: return dummy_result
Seems like there's a bunch of technical debt related to this dummy test data.
It's written in a cross-cutting way.

https://codereview.chromium.org/1610203003/diff/80001/scripts/slave/recipe_mo...
File scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py (right):

https://codereview.chromium.org/1610203003/diff/80001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:44: self.values
= (self.values or []) + results['values']
Why do you need this? Isn't self.values already initialized to []?

https://codereview.chromium.org/1610203003/diff/80001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:200: # We need
at least 5 samples for the hypothesis testing to work properly.
Sorry, not strictly true as written. Only true with these assumptions:
- Mann-Whitney U
- 2-sided
- Significance level of 0.01

https://codereview.chromium.org/1610203003/diff/80001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/perf_revision_state.py:240:
diff_from_good = self.bisector.significantly_different(
If you reorganize this function, you can avoid the code duplication.

while True:
  diff_from_good = abc
  diff_from_bad = xyz

  if diff_from_good and diff_from_bad:
    # Calculate stuff and return.
  elif diff_from_good or diff_from_bad:
    # Return.
  else:
    # Retest.

https://codereview.chromium.org/1610203003/diff/80001/scripts/slave/recipe_mo...
File
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different.py
(right):

https://codereview.chromium.org/1610203003/diff/80001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different.py:8:
CONDA_SCRIPT = """
Why not put it in its own Python file?

https://codereview.chromium.org/1610203003/diff/80001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different.py:49:
if (results['shapiro_p_value'][0] < significance and
It's strange to mix logic between here and the Anaconda script. Best to put it
all in one place. You can make this script just a thin wrapper for the Anaconda
script.

https://codereview.chromium.org/1610203003/diff/80001/scripts/slave/recipe_mo...
File
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different_test.py
(right):

https://codereview.chromium.org/1610203003/diff/80001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different_test.py:1:
#!/usr/bin/python
#!/usr/bin/env python

https://codereview.chromium.org/1610203003/diff/80001/scripts/slave/recipe_mo...
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different_test.py:17:
self.conda_path = 'python'
sys.executable

[RobertoCN, 2016-02-08 20:48:11]

PTAL

[RobertoCN, 2016-02-11 01:38:08]

I think this is it, PTAL.

[RobertoCN, 2016-02-11 21:49:37]

The CQ bit was checked by robertocn@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-02-11 21:49:49]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1610203003/160001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1610203003/160001

[commit-bot: I haz the power, 2016-02-11 21:59:29]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-02-11 21:59:30]

Dry run: Try jobs failed on following builders:
  Build Presubmit on tryserver.infra (JOB_FAILED,
https://build.chromium.org/p/tryserver.infra/builders/Build%20Presubmit/build...)

[RobertoCN, 2016-02-11 22:30:49]

The CQ bit was checked by robertocn@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-02-11 22:30:53]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1610203003/180001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1610203003/180001

[commit-bot: I haz the power, 2016-02-11 22:33:46]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-02-11 22:33:47]

Dry run: This issue passed the CQ dry run.

[Vadim Sh., 2016-02-12 00:17:11]

vadimsh@chromium.org changed reviewers:
+ vadimsh@chromium.org

[Vadim Sh., 2016-02-12 00:17:12]

https://codereview.chromium.org/1610203003/diff/180001/scripts/slave/recipe_m...
File
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different.py
(right):

https://codereview.chromium.org/1610203003/diff/180001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different.py:14:
anaconda_path = r'c:\conda-py-scientific\bin\python'
It is actually C:\conda-py-scientific\python.exe (no 'bin' on windows).

[Vadim Sh., 2016-02-12 00:36:14]

https://codereview.chromium.org/1610203003/diff/180001/scripts/slave/recipe_m...
File
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different.py
(right):

https://codereview.chromium.org/1610203003/diff/180001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different.py:14:
anaconda_path = r'c:\conda-py-scientific\bin\python'
On 2016/02/12 00:17:12, Vadim Sh. wrote:
> It is actually C:\conda-py-scientific\python.exe (no 'bin' on windows).

Oh, and also clear PYTHONPATH environ variable when calling another python, just
to avoid weird stuff when one environment tries to use modules from another
environment.

[RobertoCN, 2016-02-12 01:58:15]

On 2016/02/12 at 00:36:14, vadimsh wrote:
>
https://codereview.chromium.org/1610203003/diff/180001/scripts/slave/recipe_m...
> File
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different.py
(right):
> 
>
https://codereview.chromium.org/1610203003/diff/180001/scripts/slave/recipe_m...
>
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different.py:14:
anaconda_path = r'c:\conda-py-scientific\bin\python'
> On 2016/02/12 00:17:12, Vadim Sh. wrote:
> > It is actually C:\conda-py-scientific\python.exe (no 'bin' on windows).
> 
> Oh, and also clear PYTHONPATH environ variable when calling another python,
just to avoid weird stuff when one environment tries to use modules from another
environment.

Thanks for catching that!

Done.

[RobertoCN, 2016-02-12 01:59:00]

The CQ bit was checked by robertocn@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-02-12 01:59:07]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1610203003/220001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1610203003/220001

[Vadim Sh., 2016-02-12 02:01:31]

https://codereview.chromium.org/1610203003/diff/220001/scripts/slave/recipe_m...
File
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different.py
(right):

https://codereview.chromium.org/1610203003/diff/220001/scripts/slave/recipe_m...
scripts/slave/recipe_modules/auto_bisect/resources/significantly_different.py:26:
del conda_environ["PYTHONPATH"]
nit: conda_environ.pop("PYTHONPATH", None)

Otherwise you'll get KeyError if PYTHONPATH is not set.

[commit-bot: I haz the power, 2016-02-12 02:01:37]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-02-12 02:01:38]

Dry run: This issue passed the CQ dry run.

[RobertoCN, 2016-02-16 18:48:51]

The CQ bit was checked by robertocn@chromium.org

[RobertoCN, 2016-02-16 18:48:51]

The patchset sent to the CQ was uploaded after l-g-t-m from
qyearsley@chromium.org
Link to the patchset: https://codereview.chromium.org/1610203003/#ps220001
(title: "Fixing bad rebase.")

[commit-bot: I haz the power, 2016-02-16 18:49:01]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1610203003/220001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1610203003/220001

[commit-bot: I haz the power, 2016-02-16 18:51:53]

Description was changed from

==========
Iteratively increase sample size for good/bad classification.

This change makes the bisect recipe use scipy's implementation of
Mann-Whitney's U test to determine if a significant different exists
between two given revisions.

It also changes the bisection step by requiring more tests to be run
when the sample is insufficient to establish a difference between the
revision being evaluated and either of the FKBR/LKGR.

R=dtu,qyearsley,prasadv
BUG=565606
==========

to

==========
Iteratively increase sample size for good/bad classification.

This change makes the bisect recipe use scipy's implementation of
Mann-Whitney's U test to determine if a significant different exists
between two given revisions.

It also changes the bisection step by requiring more tests to be run
when the sample is insufficient to establish a difference between the
revision being evaluated and either of the FKBR/LKGR.

R=dtu,qyearsley,prasadv
BUG=565606

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=298810
==========

[commit-bot: I haz the power, 2016-02-16 18:51:54]

Committed patchset #12 (id:220001) as
http://src.chromium.org/viewvc/chrome?view=rev&revision=298810

[RobertoCN, 2016-02-16 19:48:15]

A revert of this CL (patchset #12 id:220001) has been created in
https://codereview.chromium.org/1702013004/ by robertocn@chromium.org.

The reason for reverting is: return_code bisects fail with this approach.

[RobertoCN, 2016-02-17 22:24:02]

The CQ bit was checked by robertocn@chromium.org

[RobertoCN, 2016-02-17 22:24:02]

The patchset sent to the CQ was uploaded after l-g-t-m from
qyearsley@chromium.org
Link to the patchset: https://codereview.chromium.org/1610203003/#ps260001
(title: "Rebasing")

[commit-bot: I haz the power, 2016-02-17 22:24:06]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1610203003/260001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1610203003/260001

[commit-bot: I haz the power, 2016-02-17 22:27:06]

Description was changed from

==========
Iteratively increase sample size for good/bad classification.

This change makes the bisect recipe use scipy's implementation of
Mann-Whitney's U test to determine if a significant different exists
between two given revisions.

It also changes the bisection step by requiring more tests to be run
when the sample is insufficient to establish a difference between the
revision being evaluated and either of the FKBR/LKGR.

R=dtu,qyearsley,prasadv
BUG=565606

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=298810
==========

to

==========
Iteratively increase sample size for good/bad classification.

This change makes the bisect recipe use scipy's implementation of
Mann-Whitney's U test to determine if a significant different exists
between two given revisions.

It also changes the bisection step by requiring more tests to be run
when the sample is insufficient to establish a difference between the
revision being evaluated and either of the FKBR/LKGR.

R=dtu,qyearsley,prasadv
BUG=565606

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=298810

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=298830
==========

[commit-bot: I haz the power, 2016-02-17 22:27:07]

Committed patchset #14 (id:260001) as
http://src.chromium.org/viewvc/chrome?view=rev&revision=298830