In rebaseline-cl: add option to only rebaseline tests changed in the CL.

In rebaseline-cl: add option to only rebaseline tests changed in the CL.

In this CL:
 - Add a flag --only-changed-tests to rebaseline-cl, to specify that only baselines for tests changed in the CL should be downloaded, rather than all baselines for any failed tests.
 - Add a helper function changed_files to rietveld to get the list of changed files for a CL.
 - Add a helper method layout_test_name to WebKitFinder to return the test name given the path from the chromium repo base.

Purpose: There are sometimes flaky test failures when running the try jobs, and sometimes users will only want baselines for tests modified in the CL. In some cases they'll change source files and want baselines for tests not changed in the current CL though.

So, I believe we may want an option to choose between downloading all new baselines, and downloading just those for changed tests.

BUG=474273, 634971
Committed: https://crrev.com/625bd62a29b931b2a172f9cb7c1e5ad36101c98b
Cr-Commit-Position: refs/heads/master@{#411794}

[qyearsley, 2016-08-11 17:19:46]

Description was changed from

==========
In rebaseline-cl: by default, only rebaseline tests changed in the CL.

BUG=474273
==========

to

==========
In rebaseline-cl: by default, only rebaseline tests changed in the CL.

In this CL:
 - Add a flag --all-tests to rebaseline-cl, to specify that baselines for all
tests with new baselines should be downloaded, rather than just baselines for
tests changed in the CL. The new default behavior is to consider only tests
changed in the CL.
 - Add a helper function changed_files to rietveld to get the list of changed
files for a CL.
 - Add a helper method layout_test_name to WebKitFinder to return the test name
given the path from the chromium repo base.

Purpose: There are sometimes flaky test failures when running the try jobs, and
most of the time I expect users will usually only want baselines for tests
modified in the CL; in some cases they'll change source files and want baselines
for tests not changed in the current CL though, so I think we may want an option
to choose between downloading all new baselines, and downloading just those for
changed tests.

BUG=474273
==========

[qyearsley, 2016-08-11 17:19:46]

qyearsley@chromium.org changed reviewers:
+ dpranke@chromium.org, wkorman@chromium.org

[wkorman, 2016-08-11 22:28:09]

lgtm

The use case is interesting. I feel like it is often the case when working on
Blink that I end up making a change that impacts tens++ of layout tests that I'm
not actually modifying in my change.

Maybe 60/40, weighted toward case where I need to rebaseline things I am not
modifying in my change.

We can see what people think. Do the try servers retry with/without patch? Do
they re-run failed layout tests a few times to make sure they were really
failing and not just flaky, as is done when running locally?

[qyearsley, 2016-08-11 22:43:50]

On 2016/08/11 at 22:28:09, wkorman wrote:
> lgtm
> 
> The use case is interesting. I feel like it is often the case when working on
Blink that I end up making a change that impacts tens++ of layout tests that I'm
not actually modifying in my change.
> 
> Maybe 60/40, weighted toward case where I need to rebaseline things I am not
modifying in my change.
> 
> We can see what people think. Do the try servers retry with/without patch? Do
they re-run failed layout tests a few times to make sure they were really
failing and not just flaky, as is done when running locally?

Looks like the try bots do run with and without patch, and they do re-run failed
layout tests, as is done locally.

Considering this, maybe it's best to make the "download baselines for all
failing tests" mode default, and make "only download baselines for tests changed
in the CL" as an option provided for convenience? (This change done in latest
patchset.)

[wkorman, 2016-08-11 22:44:55]

On 2016/08/11 at 22:43:50, qyearsley wrote:
> On 2016/08/11 at 22:28:09, wkorman wrote:
> > lgtm
> > 
> > The use case is interesting. I feel like it is often the case when working
on Blink that I end up making a change that impacts tens++ of layout tests that
I'm not actually modifying in my change.
> > 
> > Maybe 60/40, weighted toward case where I need to rebaseline things I am not
modifying in my change.
> > 
> > We can see what people think. Do the try servers retry with/without patch?
Do they re-run failed layout tests a few times to make sure they were really
failing and not just flaky, as is done when running locally?
> 
> Looks like the try bots do run with and without patch, and they do re-run
failed layout tests, as is done locally.
> 
> Considering this, maybe it's best to make the "download baselines for all
failing tests" mode default, and make "only download baselines for tests changed
in the CL" as an option provided for convenience? (This change done in latest
patchset.)

OK, yes, this default sounds better given the new info.

[qyearsley, 2016-08-11 22:45:34]

Description was changed from

==========
In rebaseline-cl: by default, only rebaseline tests changed in the CL.

In this CL:
 - Add a flag --all-tests to rebaseline-cl, to specify that baselines for all
tests with new baselines should be downloaded, rather than just baselines for
tests changed in the CL. The new default behavior is to consider only tests
changed in the CL.
 - Add a helper function changed_files to rietveld to get the list of changed
files for a CL.
 - Add a helper method layout_test_name to WebKitFinder to return the test name
given the path from the chromium repo base.

Purpose: There are sometimes flaky test failures when running the try jobs, and
most of the time I expect users will usually only want baselines for tests
modified in the CL; in some cases they'll change source files and want baselines
for tests not changed in the current CL though, so I think we may want an option
to choose between downloading all new baselines, and downloading just those for
changed tests.

BUG=474273
==========

to

==========
In rebaseline-cl: add option to only rebaseline tests changed in the CL.

In this CL:
 - Add a flag --only-changed-tests to rebaseline-cl, to specify that only
baselines for tests changed in the CL should be downloaded, rather than all
baselines for any failed tests.
 - Add a helper function changed_files to rietveld to get the list of changed
files for a CL.
 - Add a helper method layout_test_name to WebKitFinder to return the test name
given the path from the chromium repo base.

Purpose: There are sometimes flaky test failures when running the try jobs, and
sometimes users will only want baselines for tests modified in the CL. In some
cases they'll change source files and want baselines for tests not changed in
the current CL though.

So, I believe we may want an option to choose between downloading all new
baselines, and downloading just those for changed tests.

BUG=474273
==========

[qyearsley, 2016-08-12 17:33:12]

The CQ bit was checked by qyearsley@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-08-12 17:33:32]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-08-12 18:46:16]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-08-12 18:46:17]

Dry run: This issue passed the CQ dry run.

[qyearsley, 2016-08-12 21:12:06]

The CQ bit was checked by qyearsley@chromium.org

[qyearsley, 2016-08-12 21:12:07]

The patchset sent to the CQ was uploaded after l-g-t-m from wkorman@chromium.org
Link to the patchset: https://codereview.chromium.org/2237083002/#ps40001
(title: "Rebased")

[commit-bot: I haz the power, 2016-08-12 21:12:36]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-08-12 22:18:08]

Committed patchset #3 (id:40001)

[commit-bot: I haz the power, 2016-08-12 22:20:04]

Description was changed from

==========
In rebaseline-cl: add option to only rebaseline tests changed in the CL.

In this CL:
 - Add a flag --only-changed-tests to rebaseline-cl, to specify that only
baselines for tests changed in the CL should be downloaded, rather than all
baselines for any failed tests.
 - Add a helper function changed_files to rietveld to get the list of changed
files for a CL.
 - Add a helper method layout_test_name to WebKitFinder to return the test name
given the path from the chromium repo base.

Purpose: There are sometimes flaky test failures when running the try jobs, and
sometimes users will only want baselines for tests modified in the CL. In some
cases they'll change source files and want baselines for tests not changed in
the current CL though.

So, I believe we may want an option to choose between downloading all new
baselines, and downloading just those for changed tests.

BUG=474273
==========

to

==========
In rebaseline-cl: add option to only rebaseline tests changed in the CL.

In this CL:
 - Add a flag --only-changed-tests to rebaseline-cl, to specify that only
baselines for tests changed in the CL should be downloaded, rather than all
baselines for any failed tests.
 - Add a helper function changed_files to rietveld to get the list of changed
files for a CL.
 - Add a helper method layout_test_name to WebKitFinder to return the test name
given the path from the chromium repo base.

Purpose: There are sometimes flaky test failures when running the try jobs, and
sometimes users will only want baselines for tests modified in the CL. In some
cases they'll change source files and want baselines for tests not changed in
the current CL though.

So, I believe we may want an option to choose between downloading all new
baselines, and downloading just those for changed tests.

BUG=474273

Committed: https://crrev.com/625bd62a29b931b2a172f9cb7c1e5ad36101c98b
Cr-Commit-Position: refs/heads/master@{#411794}
==========

[commit-bot: I haz the power, 2016-08-12 22:20:05]

Patchset 3 (id:??) landed as
https://crrev.com/625bd62a29b931b2a172f9cb7c1e5ad36101c98b
Cr-Commit-Position: refs/heads/master@{#411794}

[qyearsley, 2016-08-13 18:26:22]

Description was changed from

==========
In rebaseline-cl: add option to only rebaseline tests changed in the CL.

In this CL:
 - Add a flag --only-changed-tests to rebaseline-cl, to specify that only
baselines for tests changed in the CL should be downloaded, rather than all
baselines for any failed tests.
 - Add a helper function changed_files to rietveld to get the list of changed
files for a CL.
 - Add a helper method layout_test_name to WebKitFinder to return the test name
given the path from the chromium repo base.

Purpose: There are sometimes flaky test failures when running the try jobs, and
sometimes users will only want baselines for tests modified in the CL. In some
cases they'll change source files and want baselines for tests not changed in
the current CL though.

So, I believe we may want an option to choose between downloading all new
baselines, and downloading just those for changed tests.

BUG=474273

Committed: https://crrev.com/625bd62a29b931b2a172f9cb7c1e5ad36101c98b
Cr-Commit-Position: refs/heads/master@{#411794}
==========

to

==========
In rebaseline-cl: add option to only rebaseline tests changed in the CL.

In this CL:
 - Add a flag --only-changed-tests to rebaseline-cl, to specify that only
baselines for tests changed in the CL should be downloaded, rather than all
baselines for any failed tests.
 - Add a helper function changed_files to rietveld to get the list of changed
files for a CL.
 - Add a helper method layout_test_name to WebKitFinder to return the test name
given the path from the chromium repo base.

Purpose: There are sometimes flaky test failures when running the try jobs, and
sometimes users will only want baselines for tests modified in the CL. In some
cases they'll change source files and want baselines for tests not changed in
the current CL though.

So, I believe we may want an option to choose between downloading all new
baselines, and downloading just those for changed tests.

BUG=474273,634971

Committed: https://crrev.com/625bd62a29b931b2a172f9cb7c1e5ad36101c98b
Cr-Commit-Position: refs/heads/master@{#411794}
==========