Make layout test harness extensible with custom layout tests directory

Make layout test harness extensible with custom layout tests directory

- Moves apache config for layout tests to within Tools/Scripts
- This will make it easier to re-use for devtools tests.

BUG=660477
Committed: https://crrev.com/98e38c15f03f244abf103caa9abd6da265cda794
Cr-Commit-Position: refs/heads/master@{#429973}

[chenwilliam, 2016-10-28 21:25:52]

Description was changed from

==========
DevTools: Proof-of-concept of moving layout tests to /Source/devtools

This is a minimal demonstration of how we can move the inspector* layout
tests to within /Source/devtools.

Most of this CL is copying test & config files from LayoutTests. The changes
to the layout test harness are small (e.g. port/base.py, run_webkit_tests.py)

BUG=660477
==========

to

==========
DevTools: Proof-of-concept of moving layout tests to /Source/devtools

This is a proof-of-concept of how we can move the inspector* layout
tests to within /Source/devtools. This only includes 8 tests so the diff
is manageable but I've ran this across all inspector tests (~1100) and all 
but ~12 tests are passing.

Most of this CL is copying test & config files from LayoutTests. The changes
to the actual layout test harness are small (e.g. port/base.py,
run_webkit_tests.py)

- Includes several types of inspector tests, but not inspector-protocol
which are backend tests and will not be moved
- Includes TestExpectations, SlowTests, and VirtualTestSuites for inspector
tests

The plan would be to follow-up with a CL that moves all of the inspector
tests and ensures the bots run the inspector test step.

BUG=660477
==========

[chenwilliam, 2016-10-28 21:25:53]

chenwilliam@chromium.org changed reviewers:
+ dpranke@chromium.org, pfeldman@chromium.org

[chenwilliam, 2016-10-28 21:31:34]

Dirk - please take a look.

I talked with Pavel about how to share this plan more broadly and he plans on
talking with other blink leads to get their feedback and then we'll follow-up
with an announcement on blink-dev that explains the change.

[Dirk Pranke, 2016-10-28 21:48:47]

Yup, this looks about like what I'd expect, and the strategy sounds right.

I'd like to not clone the http/conf directory, though, because we have to change
that from time to time and keeping two copies up-to-date would be annoying.

I suggest we try to move things from LayoutTests/http/conf to
Tools/Scripts/apache_config or some name like that.

[chenwilliam, 2016-10-31 22:44:18]

Description was changed from

==========
DevTools: Proof-of-concept of moving layout tests to /Source/devtools

This is a proof-of-concept of how we can move the inspector* layout
tests to within /Source/devtools. This only includes 8 tests so the diff
is manageable but I've ran this across all inspector tests (~1100) and all 
but ~12 tests are passing.

Most of this CL is copying test & config files from LayoutTests. The changes
to the actual layout test harness are small (e.g. port/base.py,
run_webkit_tests.py)

- Includes several types of inspector tests, but not inspector-protocol
which are backend tests and will not be moved
- Includes TestExpectations, SlowTests, and VirtualTestSuites for inspector
tests

The plan would be to follow-up with a CL that moves all of the inspector
tests and ensures the bots run the inspector test step.

BUG=660477
==========

to

==========
DevTools: Setup test harness to move devtools layout tests

This lays the foundation for moving the inspector* layout
tests to within /Source/devtools. This only includes 8 tests so the diff
is manageable but I've ran this across all inspector tests (~1100) and all 
but ~12 tests are passing.

- Includes several types of inspector tests, but not inspector-protocol
which are backend tests and will not be moved
- Includes TestExpectations, SlowTests, and VirtualTestSuites for inspector
tests
- Moves the apache config to within Tools/Scripts (easier to re-use for devtools
tests)

The plan is to follow-up with a CL that moves all of the inspector
tests and ensures the bots run the inspector test step.

BUG=660477
==========

[chenwilliam, 2016-10-31 22:56:00]

PTAL. I applied your suggestion Dirk to move the apache config for better reuse.
I'm hoping to merge this patch and then follow with two CLs:

1. CL to extend BlinkTest[0] and run this small batch of devtools tests as a
separate step.

Once this is running smoothly on buildbot, and the test flakiness results are
showing up OK, then:

2. CL to move all the appropriate devtools tests

[0]
https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_...

[Dirk Pranke, 2016-10-31 23:14:51]

lgtm w/ nit.

https://codereview.chromium.org/2454323004/diff/20001/third_party/WebKit/Tool...
File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py
(right):

https://codereview.chromium.org/2454323004/diff/20001/third_party/WebKit/Tool...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py:922: return
custom_layout_tests_dir
You can change this to 

def layout_tests_dir(self):
    return self.get_option('layout_tests_directory',
self._webkit_finder.layout_tests_dir())

[chenwilliam, 2016-11-01 18:28:35]

Thanks Dirk for reviewing. I added a response to your comment.

Pavel - PTAL.

https://codereview.chromium.org/2454323004/diff/20001/third_party/WebKit/Tool...
File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py
(right):

https://codereview.chromium.org/2454323004/diff/20001/third_party/WebKit/Tool...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py:922: return
custom_layout_tests_dir
On 2016/10/31 23:14:50, Dirk Pranke wrote:
> You can change this to 
> 
> def layout_tests_dir(self):
>     return self.get_option('layout_tests_directory',
> self._webkit_finder.layout_tests_dir())
> 

I tried it, but it doesn't seem to work because optparse by default sets an
unseen flag as None so then self.get_option's call to getattr doesn't result in
default_value since technically "layout_tests_directory" exists on self._options
(as None). I might be overlooking something but it seems like in practice the
default_value in get_option doesn't get used?

[Dirk Pranke, 2016-11-01 19:04:23]

On 2016/11/01 18:28:35, chenwilliam wrote:
> Thanks Dirk for reviewing. I added a response to your comment.
> 
> Pavel - PTAL.
> 
>
https://codereview.chromium.org/2454323004/diff/20001/third_party/WebKit/Tool...
> File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py
> (right):
> 
>
https://codereview.chromium.org/2454323004/diff/20001/third_party/WebKit/Tool...
> third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py:922:
return
> custom_layout_tests_dir
> On 2016/10/31 23:14:50, Dirk Pranke wrote:
> > You can change this to 
> > 
> > def layout_tests_dir(self):
> >     return self.get_option('layout_tests_directory',
> > self._webkit_finder.layout_tests_dir())
> > 
> 
> I tried it, but it doesn't seem to work because optparse by default sets an
> unseen flag as None so then self.get_option's call to getattr doesn't result
in
> default_value since technically "layout_tests_directory" exists on
self._options
> (as None). I might be overlooking something but it seems like in practice the
> default_value in get_option doesn't get used?

Oh, yeah, that could be. Never mind then :).

[chenwilliam, 2016-11-03 21:13:21]

Patchset #3 (id:40001) has been deleted

[chenwilliam, 2016-11-04 18:16:13]

The CQ bit was checked by chenwilliam@chromium.org

[chenwilliam, 2016-11-04 18:16:13]

The patchset sent to the CQ was uploaded after l-g-t-m from dpranke@chromium.org
Link to the patchset: https://codereview.chromium.org/2454323004/#ps80001
(title: "Undo non test harness changes")

[commit-bot: I haz the power, 2016-11-04 18:17:04]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[chenwilliam, 2016-11-04 18:17:46]

Description was changed from

==========
DevTools: Setup test harness to move devtools layout tests

This lays the foundation for moving the inspector* layout
tests to within /Source/devtools. This only includes 8 tests so the diff
is manageable but I've ran this across all inspector tests (~1100) and all 
but ~12 tests are passing.

- Includes several types of inspector tests, but not inspector-protocol
which are backend tests and will not be moved
- Includes TestExpectations, SlowTests, and VirtualTestSuites for inspector
tests
- Moves the apache config to within Tools/Scripts (easier to re-use for devtools
tests)

The plan is to follow-up with a CL that moves all of the inspector
tests and ensures the bots run the inspector test step.

BUG=660477
==========

to

==========
Move apache config for layout tests to within Tools/Scripts

This makes it easier to re-use for devtools tests.

BUG=660477
==========

[chenwilliam, 2016-11-04 18:20:17]

Description was changed from

==========
Move apache config for layout tests to within Tools/Scripts

This makes it easier to re-use for devtools tests.

BUG=660477
==========

to

==========
Make layout test harness extensible with custom layout tests directory

- Moves apache config for layout tests to within Tools/Scripts
- This will make it easier to re-use for devtools tests.

BUG=660477
==========

[chenwilliam, 2016-11-04 18:21:51]

Per Pavel's suggestion, I've narrowed the scope of the CL to just the test
harness changes and will have a separate CL for the devtools-specific test
changes.

[commit-bot: I haz the power, 2016-11-04 19:25:29]

Description was changed from

==========
Make layout test harness extensible with custom layout tests directory

- Moves apache config for layout tests to within Tools/Scripts
- This will make it easier to re-use for devtools tests.

BUG=660477
==========

to

==========
Make layout test harness extensible with custom layout tests directory

- Moves apache config for layout tests to within Tools/Scripts
- This will make it easier to re-use for devtools tests.

BUG=660477
==========

[commit-bot: I haz the power, 2016-11-04 19:25:31]

Committed patchset #4 (id:80001)

[Dirk Pranke, 2016-11-04 19:29:37]

(still lgtm).

[commit-bot: I haz the power, 2016-11-04 20:00:08]

Description was changed from

==========
Make layout test harness extensible with custom layout tests directory

- Moves apache config for layout tests to within Tools/Scripts
- This will make it easier to re-use for devtools tests.

BUG=660477
==========

to

==========
Make layout test harness extensible with custom layout tests directory

- Moves apache config for layout tests to within Tools/Scripts
- This will make it easier to re-use for devtools tests.

BUG=660477

Committed: https://crrev.com/98e38c15f03f244abf103caa9abd6da265cda794
Cr-Commit-Position: refs/heads/master@{#429973}
==========

[commit-bot: I haz the power, 2016-11-04 20:00:09]

Patchset 4 (id:??) landed as
https://crrev.com/98e38c15f03f244abf103caa9abd6da265cda794
Cr-Commit-Position: refs/heads/master@{#429973}