Make run-webkit-tests work with virtual parent LayoutTest directories

Make run-webkit-tests work with virtual parent LayoutTest directories

This patch makes possible to run multiple test suites with the same 
prefix.

BUG=707973
Review-Url: https://codereview.chromium.org/2795793003
Cr-Commit-Position: refs/heads/master@{#463114}
Committed: https://chromium.googlesource.com/chromium/src/+/22a541ff6954534ba227c393e329938cbac2253a

[Gleb Lanbin, 2017-04-03 22:58:04]

Description was changed from

==========
Make run-webkit-tests work with virtual parent LayoutTest directories

BUG=707973
==========

to

==========
Make run-webkit-tests work with virtual parent LayoutTest directories

This patch makes possible to run multiple test suites with the same 
prefix.

BUG=707973
==========

[Gleb Lanbin, 2017-04-03 22:58:44]

glebl@chromium.org changed reviewers:
+ dpranke@chromium.org, qyearsley@chromium.org

[Gleb Lanbin, 2017-04-03 22:58:49]

The CQ bit was checked by glebl@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-03 22:59:14]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-03 23:29:55]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-03 23:29:55]

Dry run: Try jobs failed on following builders:
  linux_chromium_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[Dirk Pranke, 2017-04-04 01:29:09]

I'll defer this one to qyearsley@.

[Gleb Lanbin, 2017-04-04 22:28:33]

The CQ bit was checked by glebl@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-04 22:29:46]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-04 23:42:07]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-04 23:42:08]

Dry run: Try jobs failed on following builders:
  win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[Gleb Lanbin, 2017-04-05 22:09:39]

The CQ bit was checked by glebl@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-05 22:10:23]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[qyearsley, 2017-04-05 22:47:07]

Thanks for doing this, I definitely agree that this will make it easier to run
groups of virtual tests.

I have a nitpicky concern about the behavior of Ports.tests when given a path
that's contains the start of a directory name but not the whole directory path,
and wanted to know what you think.

https://codereview.chromium.org/2795793003/diff/20001/third_party/WebKit/Tool...
File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py
(right):

https://codereview.chromium.org/2795793003/diff/20001/third_party/WebKit/Tool...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py:1509: if
any(p.startswith(suite.name) for p in paths) or any(suite.name.startswith(p) for
p in paths):
This is a bit difficult to think about without thinking of examples, for me...

So, suppose suites = [
  VirtualTestSuite(prefix='layout_ng', base='fast/block/basic',
args=['----enable-blink-features=LayoutNG']),
  VirtualTestSuite(prefix='layout_ng', base='fast/block/margin-collapse',
args=['----enable-blink-features=LayoutNG']),
] and paths = ['virtual/layout_ng/fast']

The the names of the suites are 

  virtual/layout_ng/fast/block/basic, and
  virtual/layout_ng/fast/block/margin-collapse

then any(p.startswith(suite.name) for p in paths) is False because both of the
suite names are longer than the given path, but any(suite.name.startswith(p) for
p in paths) is True because both of the suite names start with the given path.

So, this seems correct. :-)

https://codereview.chromium.org/2795793003/diff/20001/third_party/WebKit/Tool...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py:1513:
tests.append(test)
I *think* that we might want to change this bit as well (see comment below in
tests)

if any(test == p or
       test.startswith(p + '/') or
       test.startswith(p) and p.endswith('/')) for p in paths)

Although that makes it more complex and uglier.

What do you think?

https://codereview.chromium.org/2795793003/diff/20001/third_party/WebKit/Tool...
File
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py
(right):

https://codereview.chromium.org/2795793003/diff/20001/third_party/WebKit/Tool...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py:452:
self.assertIn('virtual/virtual_passes/passes_two/test-virtual-passes.html',
tests)
I think that this behavior is subtly inconsistent with the way that Port.tests
works for "real tests"... if the input paths do not have a "*", then the last
part of the path can't normally be just part of a directory name.

For example, right now there are the directories "fast/inline" and
"fast/inline-block". But if I run `run-webkit-tests fast/inline`, then only
those in "fast/inline" are run, and not those in "fast/inline-block".

So I think to test the desired behavior, we might want to do:

    tests = port.tests(['virtual/virtual_passes'])
    self.assertIn('virtual/virtual_passes/passes/test-virtual-passes.html',
tests)
    self.assertIn('virtual/virtual_passes/passes_two/test-virtual-passes.html',
tests)

and/or we might want to add the extra test as
tests.add('passes/two/test-virtual-passes.html') and then:

    tests = port.tests(['virtual/virtual_passes/passes'])
    self.assertIn('virtual/virtual_passes/passes/test-virtual-passes.html',
tests)
    self.assertIn('virtual/virtual_passes/passes/two/test-virtual-passes.html',
tests)

Does that make sense?

[commit-bot: I haz the power, 2017-04-05 23:21:05]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-05 23:21:06]

Dry run: This issue passed the CQ dry run.

[Gleb Lanbin, 2017-04-06 17:27:32]

glebl@chromium.org changed reviewers:
- dpranke@chromium.org

[Gleb Lanbin, 2017-04-06 17:27:33]

thanks, PTAL

https://codereview.chromium.org/2795793003/diff/20001/third_party/WebKit/Tool...
File third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py
(right):

https://codereview.chromium.org/2795793003/diff/20001/third_party/WebKit/Tool...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py:1509: if
any(p.startswith(suite.name) for p in paths) or any(suite.name.startswith(p) for
p in paths):
On 2017/04/05 22:47:07, qyearsley wrote:
> This is a bit difficult to think about without thinking of examples, for me...
> 
> So, suppose suites = [
>   VirtualTestSuite(prefix='layout_ng', base='fast/block/basic',
> args=['----enable-blink-features=LayoutNG']),
>   VirtualTestSuite(prefix='layout_ng', base='fast/block/margin-collapse',
> args=['----enable-blink-features=LayoutNG']),
> ] and paths = ['virtual/layout_ng/fast']
> 
> The the names of the suites are 
> 
>   virtual/layout_ng/fast/block/basic, and
>   virtual/layout_ng/fast/block/margin-collapse
> 
> then any(p.startswith(suite.name) for p in paths) is False because both of the
> suite names are longer than the given path, but any(suite.name.startswith(p)
for
> p in paths) is True because both of the suite names start with the given path.
> 
> So, this seems correct. :-)

Acknowledged.

https://codereview.chromium.org/2795793003/diff/20001/third_party/WebKit/Tool...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base.py:1513:
tests.append(test)
On 2017/04/05 22:47:07, qyearsley wrote:
> I *think* that we might want to change this bit as well (see comment below in
> tests)
> 
> if any(test == p or
>        test.startswith(p + '/') or
>        test.startswith(p) and p.endswith('/')) for p in paths)
> 
> Although that makes it more complex and uglier.
> 
> What do you think?

Done.

https://codereview.chromium.org/2795793003/diff/20001/third_party/WebKit/Tool...
File
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py
(right):

https://codereview.chromium.org/2795793003/diff/20001/third_party/WebKit/Tool...
third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/port/base_unittest.py:452:
self.assertIn('virtual/virtual_passes/passes_two/test-virtual-passes.html',
tests)
On 2017/04/05 22:47:07, qyearsley wrote:
> I think that this behavior is subtly inconsistent with the way that Port.tests
> works for "real tests"... if the input paths do not have a "*", then the last
> part of the path can't normally be just part of a directory name.
> 
> For example, right now there are the directories "fast/inline" and
> "fast/inline-block". But if I run `run-webkit-tests fast/inline`, then only
> those in "fast/inline" are run, and not those in "fast/inline-block".
> 
> So I think to test the desired behavior, we might want to do:
> 
>     tests = port.tests(['virtual/virtual_passes'])
>     self.assertIn('virtual/virtual_passes/passes/test-virtual-passes.html',
> tests)
>    
self.assertIn('virtual/virtual_passes/passes_two/test-virtual-passes.html',
> tests)
> 
> and/or we might want to add the extra test as
> tests.add('passes/two/test-virtual-passes.html') and then:
> 
>     tests = port.tests(['virtual/virtual_passes/passes'])
>     self.assertIn('virtual/virtual_passes/passes/test-virtual-passes.html',
> tests)
>    
self.assertIn('virtual/virtual_passes/passes/two/test-virtual-passes.html',
> tests)
> 
> Does that make sense?

Done.

[Gleb Lanbin, 2017-04-06 17:28:55]

The CQ bit was checked by glebl@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-06 17:29:27]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[qyearsley, 2017-04-06 17:37:22]

LGTM, thanks

[Gleb Lanbin, 2017-04-06 17:40:38]

The CQ bit was unchecked by glebl@chromium.org

[Gleb Lanbin, 2017-04-06 17:40:42]

The CQ bit was checked by glebl@chromium.org

[commit-bot: I haz the power, 2017-04-06 17:42:05]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-06 19:38:46]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-06 19:38:50]

Try jobs failed on following builders:
  win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[Gleb Lanbin, 2017-04-06 19:47:25]

The CQ bit was checked by glebl@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-06 19:48:02]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-06 21:59:07]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-06 21:59:08]

Dry run: Try jobs failed on following builders:
  win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[Gleb Lanbin, 2017-04-06 22:56:36]

The CQ bit was checked by glebl@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-06 22:57:13]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[Gleb Lanbin, 2017-04-07 00:04:50]

The CQ bit was unchecked by glebl@chromium.org

[Gleb Lanbin, 2017-04-07 00:05:00]

The CQ bit was checked by glebl@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-07 00:05:24]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-07 02:21:15]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-07 02:21:16]

Dry run: Try jobs failed on following builders:
  win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[Gleb Lanbin, 2017-04-07 16:38:36]

The CQ bit was checked by glebl@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-07 16:39:07]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-07 18:50:32]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-07 18:50:33]

Dry run: Try jobs failed on following builders:
  win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[Gleb Lanbin, 2017-04-08 00:09:36]

The CQ bit was checked by glebl@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-08 00:10:25]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-08 01:57:26]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-08 01:57:26]

Dry run: This issue passed the CQ dry run.

[Gleb Lanbin, 2017-04-08 02:53:12]

The CQ bit was checked by glebl@chromium.org

[Gleb Lanbin, 2017-04-08 02:53:13]

The patchset sent to the CQ was uploaded after l-g-t-m from
qyearsley@chromium.org
Link to the patchset: https://codereview.chromium.org/2795793003/#ps60001
(title: "fix test for win_chromium_rel_ng")

[commit-bot: I haz the power, 2017-04-08 02:53:23]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-08 02:57:09]

CQ is committing da patch.

Bot data: {"patchset_id": 60001, "attempt_start_ts": 1491619992777310,
"parent_rev": "6ff361f672aceea93e56d10189c7a2f150dcf415", "commit_rev":
"22a541ff6954534ba227c393e329938cbac2253a"}

[commit-bot: I haz the power, 2017-04-08 02:57:58]

Description was changed from

==========
Make run-webkit-tests work with virtual parent LayoutTest directories

This patch makes possible to run multiple test suites with the same 
prefix.

BUG=707973
==========

to

==========
Make run-webkit-tests work with virtual parent LayoutTest directories

This patch makes possible to run multiple test suites with the same 
prefix.

BUG=707973

Review-Url: https://codereview.chromium.org/2795793003
Cr-Commit-Position: refs/heads/master@{#463114}
Committed:
https://chromium.googlesource.com/chromium/src/+/22a541ff6954534ba227c393e329...
==========

[commit-bot: I haz the power, 2017-04-08 02:57:59]

Committed patchset #4 (id:60001) as
https://chromium.googlesource.com/chromium/src/+/22a541ff6954534ba227c393e329...