rebaseline-cl: Add logic for selecting a related builder when filling in results.

rebaseline-cl: Add logic for selecting a related builder when filling in results.

This is a follow-up to https://codereview.chromium.org/2803143002 which adds
some logic for deciding what builder to use when we're trying to rebaseline
from try jobs and platforms have no results available.

Initially, the logic was: just choose any build (the first build).
This would change it to: prefer choosing some build with the same basic OS.

BUG=673966
Review-Url: https://codereview.chromium.org/2808043005
Cr-Commit-Position: refs/heads/master@{#464675}
Committed: https://chromium.googlesource.com/chromium/src/+/8f030f1d89d3a3ccc4bcf8cc687a7ad6f413256c

[qyearsley, 2017-04-12 19:29:19]

qyearsley@chromium.org changed reviewers:
+ wkorman@chromium.org

[wkorman, 2017-04-13 16:41:35]

lgtm

https://codereview.chromium.org/2808043005/diff/1/third_party/WebKit/Tools/Sc...
File third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py
(right):

https://codereview.chromium.org/2808043005/diff/1/third_party/WebKit/Tools/Sc...
third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py:260: if
'-' not in port:
Suggest adding comment to document what format we're expecting to parse here.
Could also be worth validating the format matches via a regexp and if we use a
regexp then we could just use the matched group. Thought is that, in future,
someone might unwittingly decide to add a new port with another hyphen, or
otherwise deviate accidentally.

https://codereview.chromium.org/2808043005/diff/1/third_party/WebKit/Tools/Sc...
third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py:264: #
If any Build exists with the same OS, use the first one.
What is the order of the builds at this point -- unsorted, or sorted by build
id, or something else? Probably best would be to use the most recent one?

Also if I am not mistaken not all builds have all baselines, sometimes they
don't bother running layout tests if the code change didn't trigger them, will
we fall back and reconsider other builds in that case? Or maybe for the builders
we have configured for use by rebaseline-cl they do always run all layout tests?

[qyearsley, 2017-04-14 00:12:59]

The CQ bit was checked by qyearsley@chromium.org to run a CQ dry run

[qyearsley, 2017-04-14 00:13:03]

The CQ bit was unchecked by qyearsley@chromium.org

[qyearsley, 2017-04-14 00:13:05]

The CQ bit was checked by qyearsley@chromium.org

[qyearsley, 2017-04-14 00:13:06]

The patchset sent to the CQ was uploaded after l-g-t-m from wkorman@chromium.org
Link to the patchset: https://codereview.chromium.org/2808043005/#ps20001
(title: "Added a comment")

[qyearsley, 2017-04-14 00:13:08]

https://codereview.chromium.org/2808043005/diff/1/third_party/WebKit/Tools/Sc...
File third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py
(right):

https://codereview.chromium.org/2808043005/diff/1/third_party/WebKit/Tools/Sc...
third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py:260: if
'-' not in port:
On 2017/04/13 at 16:41:35, wkorman wrote:
> Suggest adding comment to document what format we're expecting to parse here.
Could also be worth validating the format matches via a regexp and if we use a
regexp then we could just use the matched group. Thought is that, in future,
someone might unwittingly decide to add a new port with another hyphen, or
otherwise deviate accidentally.

Right, good idea to be cautious.

The actual format I'm expecting here is "<port>-<version>", or
"test-<port>-<version>"; now added a comment to explain this.

The full port names now are created by the determine_full_port_name in the Port
subclasses, which generally always returns <port>-<version>, except in a couple
cases, for example AndroidPort always just returns "android". And TestPort
returns "test-<os>-<version>".

With regards to validating this, there is a related bug about asserting a
baseline directory name convention (crbug.com/660234); also added a note to that
bug about this.

https://codereview.chromium.org/2808043005/diff/1/third_party/WebKit/Tools/Sc...
third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py:264: #
If any Build exists with the same OS, use the first one.
On 2017/04/13 at 16:41:35, wkorman wrote:
> What is the order of the builds at this point -- unsorted, or sorted by build
id, or something else? Probably best would be to use the most recent one?

At this point, they should be in the order returned by GitCL.latest_try_jobs,
which is sorted (lexically) by builder name then build number, but including
only the latest build for reach builder. So, for example, it might be:
[Build("android_blink_rel", 3456), Build("linux_trusty_blink_rel", 6544),
Build("mac10.10_blink_rel, 32433"), ...]

https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitp...

> Also if I am not mistaken not all builds have all baselines, sometimes they
don't bother running layout tests if the code change didn't trigger them, will
we fall back and reconsider other builds in that case? Or maybe for the builders
we have configured for use by rebaseline-cl they do always run all layout tests?

Right, the builders we have configured for use by rebaseline-cl (the
tryserver.blink bots linux_trusty_blink_rel, etc.) always run all the layout
tests.

At this point in the code, I think I'm assuming that there should definitely be
baselines available because earlier on, we should try to download the layout
test results from Google Storage, and only include a build if results were
successfully fetched already.

[commit-bot: I haz the power, 2017-04-14 00:13:54]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-14 01:01:31]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-14 01:01:32]

Try jobs failed on following builders:
  linux_chromium_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[qyearsley, 2017-04-14 03:19:40]

The CQ bit was checked by qyearsley@chromium.org

[qyearsley, 2017-04-14 03:19:40]

The patchset sent to the CQ was uploaded after l-g-t-m from wkorman@chromium.org
Link to the patchset: https://codereview.chromium.org/2808043005/#ps40001
(title: "Update test")

[commit-bot: I haz the power, 2017-04-14 03:19:58]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-14 04:20:14]

CQ is committing da patch.

Bot data: {"patchset_id": 40001, "attempt_start_ts": 1492139980452920,
"parent_rev": "1210f0f9a5941ce09b2ece0c4d2c593ef64986e5", "commit_rev":
"8f030f1d89d3a3ccc4bcf8cc687a7ad6f413256c"}

[commit-bot: I haz the power, 2017-04-14 04:21:22]

Description was changed from

==========
rebaseline-cl: Add logic for selecting a related builder when filling in
results.

This is a follow-up to https://codereview.chromium.org/2803143002 which adds
some logic for deciding what builder to use when we're trying to rebaseline
from try jobs and platforms have no results available.

Initially, the logic was: just choose any build (the first build).
This would change it to: prefer choosing some build with the same basic OS.

BUG=673966
==========

to

==========
rebaseline-cl: Add logic for selecting a related builder when filling in
results.

This is a follow-up to https://codereview.chromium.org/2803143002 which adds
some logic for deciding what builder to use when we're trying to rebaseline
from try jobs and platforms have no results available.

Initially, the logic was: just choose any build (the first build).
This would change it to: prefer choosing some build with the same basic OS.

BUG=673966

Review-Url: https://codereview.chromium.org/2808043005
Cr-Commit-Position: refs/heads/master@{#464675}
Committed:
https://chromium.googlesource.com/chromium/src/+/8f030f1d89d3a3ccc4bcf8cc687a...
==========

[commit-bot: I haz the power, 2017-04-14 04:21:23]

Committed patchset #3 (id:40001) as
https://chromium.googlesource.com/chromium/src/+/8f030f1d89d3a3ccc4bcf8cc687a...