Virtual test suites for slimming paint phase 2

Virtual test suites for slimming paint phase 2

These suites runs about 1000 layout tests as smoking tests for
spv2.

BUG=524134
Committed: https://src.chromium.org/viewvc/blink?view=rev&revision=201328

R=chrishtr@chromium.org, joelo@chromium.org

Committed: https://src.chromium.org/viewvc/blink?view=rev&revision=201339

[Xianzhu, 2015-08-26 16:43:18]

wangxianzhu@chromium.org changed reviewers:
+ chrishtr@chromium.org, pdr@chromium.org

[Xianzhu, 2015-08-26 16:43:19]

I think the virtual test suites added in this CL will cover the most important
parts of spv2. 

However for now, tt may not help much on coverage for slimming paint v2 because
so many tests are marked as Failure. Most of them are dumping layer trees which
will contain only one root layer and lack some expected information (e.g. layer
geometries, layer properties, etc.) with spv2.

Perhaps the we need the following steps to make these tests useful for spv2:
1. Dump simple layer trees and/or the property trees;
2. Dump result of cc layerization.

[chrishtr, 2015-08-26 16:49:02]

On 2015/08/26 at 16:43:19, wangxianzhu wrote:
> I think the virtual test suites added in this CL will cover the most important
parts of spv2. 
> 
> However for now, tt may not help much on coverage for slimming paint v2
because so many tests are marked as Failure. Most of them are dumping layer
trees which will contain only one root layer and lack some expected information
(e.g. layer geometries, layer properties, etc.) with spv2.
> 
> Perhaps the we need the following steps to make these tests useful for spv2:
> 1. Dump simple layer trees and/or the property trees;
> 2. Dump result of cc layerization.

Given the above comments, let's start with just a set of smoke tests, perhaps
just LayoutTests/paint, and focus on adding useful simple layer & property tree
information before adding more tests.

The proposed directories you have in this patch also add up to 2500 tests, which
is too many.

[Xianzhu, 2015-08-26 17:05:33]

On 2015/08/26 16:49:02, chrishtr wrote:
> On 2015/08/26 at 16:43:19, wangxianzhu wrote:
> > I think the virtual test suites added in this CL will cover the most
important
> parts of spv2. 
> > 
> > However for now, tt may not help much on coverage for slimming paint v2
> because so many tests are marked as Failure. Most of them are dumping layer
> trees which will contain only one root layer and lack some expected
information
> (e.g. layer geometries, layer properties, etc.) with spv2.
> > 
> > Perhaps the we need the following steps to make these tests useful for spv2:
> > 1. Dump simple layer trees and/or the property trees;
> > 2. Dump result of cc layerization.
> 
> Given the above comments, let's start with just a set of smoke tests, perhaps
> just LayoutTests/paint, and focus on adding useful simple layer & property
tree
> information before adding more tests.
>

SGTM.

> The proposed directories you have in this patch also add up to 2500 tests,
which
> is too many.

Not that many:
- compositing: 692
- fast/repaint: 434
- paint: 36

[chrishtr, 2015-08-26 17:07:31]

On 2015/08/26 at 17:05:33, wangxianzhu wrote:
> On 2015/08/26 16:49:02, chrishtr wrote:
> > On 2015/08/26 at 16:43:19, wangxianzhu wrote:
> > > I think the virtual test suites added in this CL will cover the most
important
> > parts of spv2. 
> > > 
> > > However for now, tt may not help much on coverage for slimming paint v2
> > because so many tests are marked as Failure. Most of them are dumping layer
> > trees which will contain only one root layer and lack some expected
information
> > (e.g. layer geometries, layer properties, etc.) with spv2.
> > > 
> > > Perhaps the we need the following steps to make these tests useful for
spv2:
> > > 1. Dump simple layer trees and/or the property trees;
> > > 2. Dump result of cc layerization.
> > 
> > Given the above comments, let's start with just a set of smoke tests,
perhaps
> > just LayoutTests/paint, and focus on adding useful simple layer & property
tree
> > information before adding more tests.
> >
> 
> SGTM.
> 
> > The proposed directories you have in this patch also add up to 2500 tests,
which
> > is too many.
> 
> Not that many:
> - compositing: 692
> - fast/repaint: 434
> - paint: 36

You're right, my crude estimate over-counted.

[Xianzhu, 2015-08-26 17:12:13]

Patchset #4 (id:60001) has been deleted

[Xianzhu, 2015-08-26 17:18:18]

Fwiw, I used 'run-webkit-tests --dry-run <directory>'.

Deleted Patch Set 4 which added the whole compositing/ directory.

How about Patch Set 3 (suites for compositing/repaint, fast/repaint and paint)
which added about 500 tests? The results of repaint tests are more meaningful
than the pure layer tree results because repaint tests generate object-based
invalidation information. Some of them reproduce crbug.com/524236.

[chrishtr, 2015-08-26 18:28:02]

chrishtr@chromium.org changed reviewers:
+ joelo@chromium.org

[chrishtr, 2015-08-26 18:28:02]

+joelo

Joel, do you think virtual test suites are performant enough at this point to
support 500 tests?

[joelo, 2015-08-26 19:04:59]

On 2015/08/26 18:28:02, chrishtr wrote:
> +joelo
> 
> Joel, do you think virtual test suites are performant enough at this point to
> support 500 tests?

lgtm 

Yes, in general adding virtual test suites should not add much more overhead
other than increasing cycle times. The only thing to look out for would be to
check that the runtime of each run-webkit-test shard is relatively balanced,
since each virtual test suite runs on a different shard. In this case the test
suites being added take up a relatively small percentage of the runtime so I
wouldn't worry about it:
http://test-results.appspot.com/dashboards/treemap.html#treemapfocus=AllTests...

That said, we are currently tackling http://crbug.com/524248 where we have bot
exceptions due to content shells not being cleaned up properly. I would keep an
eye on things since we spawn a new content shell for each virtual test suite,
but should be okay since you are only adding 3 suites.

[chrishtr, 2015-08-26 19:56:58]

lgtm

[Xianzhu, 2015-08-27 17:11:19]

The CQ bit was checked by wangxianzhu@chromium.org

[Xianzhu, 2015-08-27 17:11:19]

The patchset sent to the CQ was uploaded after l-g-t-m from joelo@chromium.org,
chrishtr@chromium.org
Link to the patchset: https://codereview.chromium.org/1307293002/#ps130001
(title: " ")

[commit-bot: I haz the power, 2015-08-27 17:11:31]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1307293002/130001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1307293002/130001

[commit-bot: I haz the power, 2015-08-27 18:08:35]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-08-27 18:08:36]

Try jobs failed on following builders:
  linux_chromium_rel_ng on tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[Xianzhu, 2015-08-27 18:13:20]

The CQ bit was checked by wangxianzhu@chromium.org

[Xianzhu, 2015-08-27 18:13:21]

The patchset sent to the CQ was uploaded after l-g-t-m from joelo@chromium.org,
chrishtr@chromium.org
Link to the patchset: https://codereview.chromium.org/1307293002/#ps150001
(title: " ")

[commit-bot: I haz the power, 2015-08-27 18:13:34]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1307293002/150001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1307293002/150001

[commit-bot: I haz the power, 2015-08-27 19:25:46]

Committed patchset #8 (id:150001) as
https://src.chromium.org/viewvc/blink?view=rev&revision=201328

[Xianzhu, 2015-08-27 19:38:58]

Sorry, forgot to update the number in CL description. It should be "about 500
layout tests". 1000 included other compositing/ tests.

[Xianzhu, 2015-08-27 20:53:16]

Committed patchset #9 (id:170001) manually as 201339 (presubmit successful).