Requiring confidence in initial regression range before bisecting.

tools/auto_bisect/bisect_perf_regression.py

 #!/usr/bin/env python
 # Copyright (c) 2013 The Chromium Authors. All rights reserved.
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
 """Performance Test Bisect Tool
 
 This script bisects a series of changelists using binary search. It starts at
 a bad revision where a performance metric has regressed, and asks for a last
 known-good revision. It will then binary search across this revision range by
@@ skipping 32 matching lines @@
 import shutil
 import StringIO
 import sys
 import time
 import zipfile
 
 sys.path.append(os.path.join(
     os.path.dirname(__file__), os.path.pardir, 'telemetry'))
 
 from bisect_results import BisectResults
+from bisect_results import ConfidenceScore
 import bisect_utils
 import builder
 import math_utils
 import request_build
 import source_control as source_control_module
 from telemetry.util import cloud_storage
 
 # Below is the map of "depot" names to information about each depot. Each depot
 # is a repository, and in the process of bisecting, revision ranges in these
 # repositories may also be bisected.
@@ skipping 2375 matching lines @@
             'a working \'good\' revision to start from.\n\nError: %s' %
             good_results[0])
         return results
 
 
       # We need these reference values to determine if later runs should be
       # classified as pass or fail.
       known_bad_value = bad_results[0]
       known_good_value = good_results[0]
 
+      regression_confidence = ConfidenceScore(known_bad_value['values'],
+                                              known_good_value['values'])
+      if regression_confidence < HIGH_CONFIDENCE:

[qyearsley, 2014/10/12 04:07:21]

I'm not 100% sure it's good practice to re-use the HIGH_CONFIDENCE here for this
slightly different purpose.

[RobertoCN, 2014/10/17 22:44:37]

Done.

+        results.error = ('The results of the performance test on the \'good\' '
+                         'and \'bad\' revisions are so close together that we '
+                         'can\'t be confident this is a regression. Try '
+                         'bisecting a different revision range.')

[qyearsley, 2014/10/12 04:07:22]

I think that this message can be improved. We want to communicate that:
   1. We couldn't reproduce it in this run with this test and metric and
platform combination.
   2. This gives evidence that there is no regression between "good" and "bad",
but it might still potentially be able to be reproduced on a different platform.

[RobertoCN, 2014/10/17 22:44:37]

Done.

[qyearsley, 2014/10/17 23:53:52]

Sometimes when the bisect can't reproduce the regression, the next step to take
is to try on a different platform with the same revision range.

So, I suggest changing the last sentence here; either expand it to "Try
bisecting a different revision range or using a different platform." or just
remove it.

+        return results
       # Can just mark the good and bad revisions explicitly here since we
       # already know the results.
       bad_revision_data = revision_data[revision_list[0]]
       bad_revision_data['external'] = bad_results[2]
       bad_revision_data['perf_time'] = bad_results[3]
       bad_revision_data['build_time'] = bad_results[4]
       bad_revision_data['passed'] = False
       bad_revision_data['value'] = known_bad_value
 
       good_revision_data = revision_data[revision_list[max_revision]]
@@ skipping 799 matching lines @@
       # bugs. If you change this, please update the perf dashboard as well.
       bisect_utils.OutputAnnotationStepStart('Results')
     print 'Error: %s' % e.message
     if opts.output_buildbot_annotations:
       bisect_utils.OutputAnnotationStepClosed()
   return 1
 
 
 if __name__ == '__main__':
   sys.exit(main())