Fetch retry summary and use it to determine what tests to rebaseline.

third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py

Index: third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py
diff --git a/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py b/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py
index 04c49a98a83f535bf00da3397bc88e6f4be41d64..c7ffe4e336050e3c43e7b6c9b461f025c5f75be6 100644
--- a/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py
+++ b/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py
@@ -59,6 +59,16 @@ class RebaselineCLTest(BaseTestCase, LoggingTestCase):
         })
         self.command.rietveld = Rietveld(web)
 
+        self.tool.buildbot.set_retry_sumary_json(Build('MOCK Try Win', 5000), json.dumps({
+            'failures': [
+                'fast/dom/prototype-newtest.html',
+                'fast/dom/prototype-taco.html',
+                'fast/dom/prototype-inheritance.html',
+                'svg/dynamic-updates/SVGFEDropShadowElement-dom-stdDeviation-attr.html',
+            ],
+            'ignored': [],
+        }))
+
         # Write to the mock filesystem so that these tests are considered to exist.
         port = self.mac_port
         tests = [
@@ -138,6 +148,27 @@ class RebaselineCLTest(BaseTestCase, LoggingTestCase):
             'INFO: Rebaselining fast/dom/prototype-taco.html\n',
         ])
 
+    def test_execute_with_flaky_test_that_fails_on_retry(self):

[wkorman, 2016/10/24 21:47:38]

Do we have test coverage to make sure with code added in this change, with
--only-changed-tests passed:
- there is no avail retry summary (would want to fetch then)
- there is a retry summary, the test(s) changed in the current change *are not*
failing in the retry (would not want to fetch then)
- there is a retry summary, the test(s) changed in the current change *are*
failing in the retry (would want to fetch then)

My expectation in parens but open to thought.

[qyearsley, 2016/10/25 21:53:52]

In the first patchset:

 1. scenario 1 isn't covered yet
 2. scenario 2 should be covered by the other test methods above, since a canned
response is added in setUp with buildbot.set_retry_sumary_json, with no tests
"ignored"; in schenario 2, where the tests fail with the patch but not in the
retry without the patch, then that means they *should* be rebaselined and we
would want to fetch then.
 3. scenario 3 should be covered by this method,
test_execute_with_flaky_test_that_fails_on_retry, where a test is in the
"ignored" list and no baseline is downloaded for that test.

Now added a test for the first case.

[wkorman, 2016/10/25 23:15:34]

LOL, I had it backwards, but thanks to the tests now we can have confidence!

+        # In this example, the --only-changed-tests flag is not given, but
+        # svg/dynamic-updates/SVGFEDropShadowElement-dom-stdDeviation-attr.html
+        # failed both with and without the patch in the try job, so it is not
+        # rebaselined.
+        self.tool.buildbot.set_retry_sumary_json(Build('MOCK Try Win', 5000), json.dumps({
+            'failures': [
+                'fast/dom/prototype-taco.html',
+                'fast/dom/prototype-inheritance.html',
+            ],
+            'ignored': ['svg/dynamic-updates/SVGFEDropShadowElement-dom-stdDeviation-attr.html'],
+        }))
+        self.command.execute(self.command_options(issue=11112222, only_changed_tests=True), [], self.tool)
+        self.assertLog([
+            'INFO: Tests to rebaseline:\n',
+            'INFO:   fast/dom/prototype-inheritance.html: MOCK Try Win (5000)\n',
+            'INFO:   fast/dom/prototype-taco.html: MOCK Try Win (5000)\n',
+            'INFO: Rebaselining fast/dom/prototype-inheritance.html\n',
+            'INFO: Rebaselining fast/dom/prototype-taco.html\n',
+        ])
+
     def test_execute_with_nonexistent_test(self):
         self.command.execute(self.command_options(issue=11112222), ['some/non/existent/test.html'], self.tool)
         self.assertLog([