Support filling in baselines from other platforms in rebaseline-cl.

third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py

 # Copyright 2016 The Chromium Authors. All rights reserved.
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
 """A command to fetch new baselines from try jobs for the current CL."""
 
 import json
 import logging
 import optparse
 
@@ skipping 18 matching lines @@
         super(RebaselineCL, self).__init__(options=[
             optparse.make_option(
                 '--dry-run', action='store_true', default=False,
                 help='Dry run mode; list actions that would be performed but do not do anything.'),
             optparse.make_option(
                 '--only-changed-tests', action='store_true', default=False,
                 help='Only download new baselines for tests that are changed in the CL.'),
             optparse.make_option(
                 '--no-trigger-jobs', dest='trigger_jobs', action='store_false', default=True,
                 help='Do not trigger any try jobs.'),
+            optparse.make_option(
+                '--fill-missing', dest='fill_missing', action='store_true', default=False,
+                help='If some builds have no results available, use results from other builds.'),

[wkorman, 2017/04/10 20:12:27]

This reads as if the port will be the same and the only thing we are potentially
doing is falling back to an earlier build if it had suitable baseline. But I
think we could in fact switch ports? Perhaps worth clarifying documentation.

[qyearsley, 2017/04/10 23:52:09]

Ah, yes, not just "use results from other builds", but "use results from other
platforms' try jobs".

Done.

             self.no_optimize_option,
             self.results_directory_option,
         ])
 
     def execute(self, options, args, tool):
         self._tool = tool
 
         # TODO(qyearsley): Move this call to somewhere else.
         WPTManifest.ensure_manifest(tool)
 
@@ skipping 17 matching lines @@
             _log.info('There are existing pending builds for:')
             for builder in sorted(builders_with_pending_builds):
                 _log.info('  %s', builder)
         builders_with_no_results = self.builders_with_no_results(builds)
 
         if options.trigger_jobs and builders_with_no_results:
             self.trigger_builds(builders_with_no_results)
             _log.info('Please re-run webkit-patch rebaseline-cl once all pending try jobs have finished.')
             return 1
 
-        if builders_with_no_results:
+        if builders_with_no_results and not options.fill_missing:
             # TODO(qyearsley): Support trying to continue as long as there are
             # some results from some builder; see http://crbug.com/673966.
             _log.error('The following builders have no results:')
             for builder in builders_with_no_results:
                 _log.error('  %s', builder)
             return 1
 
         _log.debug('Getting results for issue %d.', issue_number)
         builds_to_results = self._fetch_results(builds)
         if builds_to_results is None:
             return 1
 
         test_baseline_set = TestBaselineSet(tool)
         if args:
             for test in args:
                 for build in builds:
                     test_baseline_set.add(test, build)
         else:
             test_baseline_set = self._make_test_baseline_set(
                 builds_to_results,
                 only_changed_tests=options.only_changed_tests)
 
+        if options.fill_missing:
+            self.fill_in_missing_results(test_baseline_set)
+
         _log.debug('Rebaselining: %s', test_baseline_set)
 
         if not options.dry_run:
             self.rebaseline(options, test_baseline_set)
         return 0
 
     def _get_issue_number(self):
         """Returns the current CL issue number, or None."""
         issue = self.git_cl().get_issue_number()
         if not issue.isdigit():
@@ skipping 104 matching lines @@
         buildbot = self._tool.buildbot
         content = buildbot.fetch_retry_summary_json(build)
         if content is None:
             return None
         try:
             retry_summary = json.loads(content)
             return retry_summary['failures']
         except (ValueError, KeyError):
             _log.warning('Unexpected retry summary content:\n%s', content)
             return None
+
+    def fill_in_missing_results(self, test_baseline_set):
+        """Modifies test_baseline_set to guarantee that there are entries for
+        all ports for all tests to rebaseline.
+
+        For a each test prefix, if there is entry for some port, then an entry
+        should be added
+        """
+        all_ports = {self._tool.builders.port_name_for_builder_name(b) for b in self._try_bots()}
+        for test_prefix in test_baseline_set.test_prefixes():
+            build_port_pairs = test_baseline_set.build_port_pairs(test_prefix)
+            missing_ports = all_ports - {p for _, p in build_port_pairs}
+            for port in missing_ports:
+                # TODO(qyearsley): Decide what build to use for a given port

[wkorman, 2017/04/10 20:12:27]

Just looking to make sure I understand, is this noted work basically revising to
incorporate the same/similar platform (port) fallback logic that we use for
optimizing baselines?

[qyearsley, 2017/04/10 23:52:10]

Something similar, but I believe we can't entirely rely on the fallback graph.

For example, suppose we just have results for win7. The baseline fallback path
for linux-trusty is ["linux", "win"], and "win" corresponds to win10. Similarly,
the baseline fallback path for mac10.10 is ["mac-mac10.10", "mac-mac10.11",
"mac"], which doesn't include any windows platforms.

Fallback graph:
https://drive.google.com/a/google.com/file/d/0Bzxz6fWz5JOIWlVMQkJXc1Bjd2M/vie...

So I guess one approach would be to start with the fallback path, but then if
there are no available builds for any ports there, then just use any build?

One approach would be to construct a dict of port to list of other ports
starting with similar ports just by listing all the supported versions, e.g.:


def similar_ports_dict():
  android = ['android-' + v for v in AndroidPort.SUPPORTED_VERSIONS]
  linux = ['linux-' + v for v in LinuxPort.SUPPORTED_VERSIONS]
  mac = ['mac-' + v for v in MacPort.SUPPORTED_VERSIONS]
  win = ['win-' + v for v in WinPort.SUPPORTED_VERSIONS]
  similar_ports = {}
  for v in android:
    similar_ports[v] = android + linux + mac + win
  for v in linux:
    similar_ports[v] = linux + android + mac + win
  for v in mac:
    similar_ports[v] = mac + linux + android + win
  for v in win:
    similar_ports[v] = win + linux + android + mac
  return similar_ports

My main thought here is just that, for example, if we just need to fill in one
mac port, then we should default to using another mac result first; but if there
are no other mac builds available, then probably trying to use any result is
OK...


Does that sound roughly correct? I'm hoping I can use something a little simpler
than the above.

[wkorman, 2017/04/11 20:36:49]

Yes, something like this sounds correct to me. Perhaps checkpoint with drott@ as
well for another opinion since he spends a lot of time with cross platform font
baselines. Doing this as a separate change makes sense to me.

[qyearsley, 2017/04/11 23:23:21]

Sounds good, will do :-)

+                # in a more sophisticated way, such that a build with a
+                # "similar" port will be used when available.
+                build = build_port_pairs[0][0]
+                test_baseline_set.add(test_prefix, build, port)
+        return test_baseline_set