Making check_initial_confidence verify return_code bisects.

scripts/slave/recipe_modules/auto_bisect/bisector_test.py

 #!/usr/bin/env python
 # Copyright 2015 The Chromium Authors. All rights reserved.
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
 import copy
 import os
 import sys
 import unittest
 
@@ skipping 131 matching lines @@
     bisector.good_rev.values = [3, 3, 3, 3, 3, 3]
     bisector.bad_rev.values = [3, 3, 3, 3]
     self.assertFalse(bisector.check_initial_confidence())
     self.assertTrue(len(bisector.bad_rev.values) >=
                     auto_bisect.bisector.MAX_REQUIRED_SAMPLES or
                     len(bisector.good_rev.values) >=
                     auto_bisect.bisector.MAX_REQUIRED_SAMPLES)
 
   @mock.patch.object(auto_bisect.bisector.Bisector, 'significantly_different',
                      mock.MagicMock())
-  def test_check_initial_confidence_not_required(self):
+  def test_check_initial_confidence_return_code(self):
     return_code_config = self.bisect_config
     return_code_config['test_type'] = 'return_code'
     # When confidence is not required, confidence_score should not be called.

[qyearsley, 2016/03/23 20:54:40]

This comment is now confusing since this test method doesn't reference use or
check confidence_score (but it checks that significantly_different is not called
-- which could again be a separate little test method).

[RobertoCN, 2016/03/23 21:52:09]

Removed the check for not calling significantly different for two reasons:
- Bypass repro check is no longer a use case (the parameter is ignored)
- Significantly_different has been modified to support return_code bisects.

Split the test for return code confidence check into pass and fail.

     bisector = auto_bisect.bisector.Bisector(self.dummy_api, return_code_config,
                                              MockRevisionClass)
+    bisector.good_rev.mean_value = 0
+    bisector.bad_rev.mean_value = 0
+    self.assertFalse(bisector.check_initial_confidence())
+    bisector.bad_rev.mean_value = 1
     self.assertTrue(bisector.check_initial_confidence())
     self.assertFalse(bisector.significantly_different.called)
 

[qyearsley, 2016/03/23 20:54:40]

This test method might be a bit clearer if it's split into separate methods that
each test one simple behavior/case: there could be one case where
check_initial_confidence() returns True and another where it returns False.

[RobertoCN, 2016/03/23 21:52:09]

Done.

 
 if __name__ == '__main__':
   unittest.main()  # pragma: no cover