Reland of In rebaseline, include build number information with "test_prefix_list" dicts.

Reland of In rebaseline, include build number information with "test_prefix_list" dicts. (patchset #1 id:1 of https://codereview.chromium.org/2151253004/ )

Reason for revert:
Making a reland CL to investigate and try to reproduce the unit test failures (e.g. https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.10/builds/20653/steps/webkit_python_tests/logs/stdio)

Original issue's description:
> Revert of In rebaseline, include build number information with "test_prefix_list" dicts. (patchset #4 id:60001 of https://codereview.chromium.org/2144873004/ )
>
> Reason for revert:
> Seems to be causing failures in webkit_python_tests on multiple bots (e.g. https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.10/builds/20653).
>
> Original issue's description:
> > In rebaseline, include build number information with "test_prefix_list" dicts.
> >
> > When rebaselining, the main argument passed to the rebaseline method is a "test_prefix_list", which is currently a dict mapping test prefixes to builder names to baseline file suffixes.
> >
> > If we want to change _suffixes_for_actual_failures to fetch actual failures for try jobs, then that method needs to know the build number as well.
> >
> > This CL would replace the builder in all "test_prefix_list" dicts with a Build object, which includes builder name and build number, but build number is optional.
> >
> > Planned follow-up: changing build_data() to use a method to fetch and cache results for particular builds, and change _suffixes_for_actual_failures fetch results for particular builds using that method as well. This would make the "skip_checking_actual_results" workaround unnecessary. I could also include that change in this CL.
> >
> > Committed: https://crrev.com/e5830ddcf7cc0fc87decede44a0a0baa4d5e8c61
> > Cr-Commit-Position: refs/heads/master@{#405783}
>
> TBR=dpranke@chromium.org,wkorman@chromium.org,qyearsley@chromium.org
> # Skipping CQ checks because original CL landed less than 1 days ago.
> NOPRESUBMIT=true
> NOTREECHECKS=true
> NOTRY=true
>
> Committed: https://crrev.com/110c6f3a6dfa490acca83b2daa49615ee821ac8a
> Cr-Commit-Position: refs/heads/master@{#405804}

TBR=dpranke@chromium.org,wkorman@chromium.org,wjmaclean@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true

Committed: https://crrev.com/074ad1aeeecfa153389899fd76dcb4dcfc8c3872
Cr-Commit-Position: refs/heads/master@{#406085}

[qyearsley, 2016-07-15 20:34:45]

Created Reland of In rebaseline, include build number information with
"test_prefix_list" dicts.

[qyearsley, 2016-07-15 20:36:06]

On 2016/07/15 at 20:34:45, qyearsley wrote:
> Created Reland of In rebaseline, include build number information with
"test_prefix_list" dicts.

(I haven't found the reason for the unit test failures yet; will upload a patch
here if I found the cause.)

[qyearsley, 2016-07-15 20:49:10]

https://codereview.chromium.org/2153973002/diff/1/third_party/WebKit/Tools/Sc...
File third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py
(right):

https://codereview.chromium.org/2153973002/diff/1/third_party/WebKit/Tools/Sc...
third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py:394: for
build in test_prefix_list[test_prefix]:
Hypothesis: The failure was non-deterministic, and happened because here (and on
line 394) I'm depending on a particular order of iterating through keys in a
dictionary.

[qyearsley, 2016-07-15 20:57:55]

On 2016/07/15 at 20:49:10, qyearsley wrote:
>
https://codereview.chromium.org/2153973002/diff/1/third_party/WebKit/Tools/Sc...
> File third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py
(right):
> 
>
https://codereview.chromium.org/2153973002/diff/1/third_party/WebKit/Tools/Sc...
> third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py:394: for
build in test_prefix_list[test_prefix]:
> Hypothesis: The failure was non-deterministic, and happened because here (and
on line 394) I'm depending on a particular order of iterating through keys in a
dictionary.

Hypothesis confirmed, because I could get the same failure to reproduce by
sorting the keys in different orders.

The fix in the latest patch changes it so that Build objects are sorted in
_rebaseline_commands and _optimize_commands.

In a follow-up CL, I'd like to change all "test_prefix_list" dicts to be
instances of a class; maybe that class should be responsible for returning an
ordered list of Build objects for a particular test prefix.

[wkorman, 2016-07-18 20:02:07]

On 2016/07/15 at 20:57:55, qyearsley wrote:
> On 2016/07/15 at 20:49:10, qyearsley wrote:
> >
https://codereview.chromium.org/2153973002/diff/1/third_party/WebKit/Tools/Sc...
> > File third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py
(right):
> > 
> >
https://codereview.chromium.org/2153973002/diff/1/third_party/WebKit/Tools/Sc...
> > third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py:394:
for build in test_prefix_list[test_prefix]:
> > Hypothesis: The failure was non-deterministic, and happened because here
(and on line 394) I'm depending on a particular order of iterating through keys
in a dictionary.
> 
> Hypothesis confirmed, because I could get the same failure to reproduce by
sorting the keys in different orders.
> 
> The fix in the latest patch changes it so that Build objects are sorted in
_rebaseline_commands and _optimize_commands.
> 
> In a follow-up CL, I'd like to change all "test_prefix_list" dicts to be
instances of a class; maybe that class should be responsible for returning an
ordered list of Build objects for a particular test prefix.

Yes, if order matters it would be good to centralize it in some manner.

[wkorman, 2016-07-18 20:02:14]

lgtm

https://codereview.chromium.org/2153973002/diff/120001/third_party/WebKit/Too...
File third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py
(right):

https://codereview.chromium.org/2153973002/diff/120001/third_party/WebKit/Too...
third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py:555: for
new baseilnes. For example:
baselines

[qyearsley, 2016-07-18 20:40:31]

https://codereview.chromium.org/2153973002/diff/120001/third_party/WebKit/Too...
File third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py
(right):

https://codereview.chromium.org/2153973002/diff/120001/third_party/WebKit/Too...
third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py:555: for
new baseilnes. For example:
On 2016/07/18 at 20:02:14, wkorman wrote:
> baselines

fixed

[qyearsley, 2016-07-18 20:40:34]

The CQ bit was checked by qyearsley@chromium.org

[qyearsley, 2016-07-18 20:40:34]

The patchset sent to the CQ was uploaded after l-g-t-m from wkorman@chromium.org
Link to the patchset: https://codereview.chromium.org/2153973002/#ps140001
(title: "Fix spelling baseilnes -> baselines")

[commit-bot: I haz the power, 2016-07-18 20:41:20]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-07-18 20:42:06]

Committed patchset #3 (id:140001)

[commit-bot: I haz the power, 2016-07-18 20:44:03]

Description was changed from

==========
Reland of In rebaseline, include build number information with
"test_prefix_list" dicts. (patchset #1 id:1 of
https://codereview.chromium.org/2151253004/ )

Reason for revert:
Making a reland CL to investigate and try to reproduce the unit test failures
(e.g.
https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.10/build...)

Original issue's description:
> Revert of In rebaseline, include build number information with
"test_prefix_list" dicts. (patchset #4 id:60001 of
https://codereview.chromium.org/2144873004/ )
> 
> Reason for revert:
> Seems to be causing failures in webkit_python_tests on multiple bots (e.g.
https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.10/build...).
> 
> Original issue's description:
> > In rebaseline, include build number information with "test_prefix_list"
dicts.
> >
> > When rebaselining, the main argument passed to the rebaseline method is a
"test_prefix_list", which is currently a dict mapping test prefixes to builder
names to baseline file suffixes.
> >
> > If we want to change _suffixes_for_actual_failures to fetch actual failures
for try jobs, then that method needs to know the build number as well.
> >
> > This CL would replace the builder in all "test_prefix_list" dicts with a
Build object, which includes builder name and build number, but build number is
optional.
> >
> > Planned follow-up: changing build_data() to use a method to fetch and cache
results for particular builds, and change _suffixes_for_actual_failures fetch
results for particular builds using that method as well. This would make the
"skip_checking_actual_results" workaround unnecessary. I could also include that
change in this CL.
> >
> > Committed: https://crrev.com/e5830ddcf7cc0fc87decede44a0a0baa4d5e8c61
> > Cr-Commit-Position: refs/heads/master@{#405783}
> 
> TBR=dpranke@chromium.org,wkorman@chromium.org,qyearsley@chromium.org
> # Skipping CQ checks because original CL landed less than 1 days ago.
> NOPRESUBMIT=true
> NOTREECHECKS=true
> NOTRY=true
> 
> Committed: https://crrev.com/110c6f3a6dfa490acca83b2daa49615ee821ac8a
> Cr-Commit-Position: refs/heads/master@{#405804}

TBR=dpranke@chromium.org,wkorman@chromium.org,wjmaclean@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true
==========

to

==========
Reland of In rebaseline, include build number information with
"test_prefix_list" dicts. (patchset #1 id:1 of
https://codereview.chromium.org/2151253004/ )

Reason for revert:
Making a reland CL to investigate and try to reproduce the unit test failures
(e.g.
https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.10/build...)

Original issue's description:
> Revert of In rebaseline, include build number information with
"test_prefix_list" dicts. (patchset #4 id:60001 of
https://codereview.chromium.org/2144873004/ )
>
> Reason for revert:
> Seems to be causing failures in webkit_python_tests on multiple bots (e.g.
https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Mac10.10/build...).
>
> Original issue's description:
> > In rebaseline, include build number information with "test_prefix_list"
dicts.
> >
> > When rebaselining, the main argument passed to the rebaseline method is a
"test_prefix_list", which is currently a dict mapping test prefixes to builder
names to baseline file suffixes.
> >
> > If we want to change _suffixes_for_actual_failures to fetch actual failures
for try jobs, then that method needs to know the build number as well.
> >
> > This CL would replace the builder in all "test_prefix_list" dicts with a
Build object, which includes builder name and build number, but build number is
optional.
> >
> > Planned follow-up: changing build_data() to use a method to fetch and cache
results for particular builds, and change _suffixes_for_actual_failures fetch
results for particular builds using that method as well. This would make the
"skip_checking_actual_results" workaround unnecessary. I could also include that
change in this CL.
> >
> > Committed: https://crrev.com/e5830ddcf7cc0fc87decede44a0a0baa4d5e8c61
> > Cr-Commit-Position: refs/heads/master@{#405783}
>
> TBR=dpranke@chromium.org,wkorman@chromium.org,qyearsley@chromium.org
> # Skipping CQ checks because original CL landed less than 1 days ago.
> NOPRESUBMIT=true
> NOTREECHECKS=true
> NOTRY=true
>
> Committed: https://crrev.com/110c6f3a6dfa490acca83b2daa49615ee821ac8a
> Cr-Commit-Position: refs/heads/master@{#405804}

TBR=dpranke@chromium.org,wkorman@chromium.org,wjmaclean@chromium.org
# Skipping CQ checks because original CL landed less than 1 days ago.
NOPRESUBMIT=true
NOTREECHECKS=true
NOTRY=true

Committed: https://crrev.com/074ad1aeeecfa153389899fd76dcb4dcfc8c3872
Cr-Commit-Position: refs/heads/master@{#406085}
==========

[commit-bot: I haz the power, 2016-07-18 20:44:04]

Patchset 3 (id:??) landed as
https://crrev.com/074ad1aeeecfa153389899fd76dcb4dcfc8c3872
Cr-Commit-Position: refs/heads/master@{#406085}