Replace flaky test, wrangle with harness/infra

Replace flaky test, wrangle with harness/infra

BUG=638621

[chcunningham, 2016-10-22 01:16:03]

The CQ bit was checked by chcunningham@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-10-22 01:16:43]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[chcunningham, 2016-10-22 01:21:13]

chcunningham@chromium.org changed reviewers:
+ dpranke@chromium.org

[chcunningham, 2016-10-22 01:21:13]

Hey Dirk, I'm attempting to create a new testharness test that also does pixel
testing. But I think what I really have here is a Frankenstein thing. Has this
been done before? A better way?

[commit-bot: I haz the power, 2016-10-22 02:48:36]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-10-22 02:48:36]

Dry run: Try jobs failed on following builders:
  mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

[chcunningham, 2016-10-24 20:37:33]

On 2016/10/22 01:21:13, chcunningham wrote:
> Hey Dirk, I'm attempting to create a new testharness test that also does pixel
> testing. But I think what I really have here is a Frankenstein thing. Has this
> been done before? A better way?

To explain more: Its probably best to just ignore the code here completely and
address the question: How do I take screenshots in a testHarness test?

I've attempted to use window.testRunner, but it seems that is actually part of
pre-testHarness.js infrastructure. Looking at the logic in single_test_runner.py
[0], I see that having any image output is a condition of NOT being a
testHarness.js test. So I suspect I'm in uncharted territory. 

So the next question is, if I want to do this, should I use testRunner to dump
the pixels and just refactor the logic in single_test_runner to allow pixels
with testHarness tests?

https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitp...

[Dirk Pranke, 2016-10-26 18:40:54]

On 2016/10/24 20:37:33, chcunningham wrote:
> On 2016/10/22 01:21:13, chcunningham wrote:
> > Hey Dirk, I'm attempting to create a new testharness test that also does
pixel
> > testing. But I think what I really have here is a Frankenstein thing. Has
this
> > been done before? A better way?
> 
> To explain more: Its probably best to just ignore the code here completely and
> address the question: How do I take screenshots in a testHarness test?
> 
> I've attempted to use window.testRunner, but it seems that is actually part of
> pre-testHarness.js infrastructure. Looking at the logic in
single_test_runner.py
> [0], I see that having any image output is a condition of NOT being a
> testHarness.js test. So I suspect I'm in uncharted territory. 
> 
> So the next question is, if I want to do this, should I use testRunner to dump
> the pixels and just refactor the logic in single_test_runner to allow pixels
> with testHarness tests?
> 
>
https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitp...

testharness.js is the framework we get from the W3C, and it (intentionally) does
not expose a way
to get screenshots, because screenshots require manual review.

I think you roughly have three options: 

1) Use the js-test framework instead.

2) Split this test into two: one that does the asserting, and one that just
produces the image.

3) Try to convince people that we should have a non-standard extension to
testharness or get
the W3C to buy in to this extension.

I don't think option (3) is a very good idea, but it's possible there might be
support for it from others.
If you can make things work with js-test that's probably they way to go.

Does that help? (And sorry for the delay, I was out for a few days).

[chcunningham, 2016-10-28 17:11:19]

On 2016/10/26 18:40:54, Dirk Pranke wrote:
> On 2016/10/24 20:37:33, chcunningham wrote:
> > On 2016/10/22 01:21:13, chcunningham wrote:
> > > Hey Dirk, I'm attempting to create a new testharness test that also does
> pixel
> > > testing. But I think what I really have here is a Frankenstein thing. Has
> this
> > > been done before? A better way?
> > 
> > To explain more: Its probably best to just ignore the code here completely
and
> > address the question: How do I take screenshots in a testHarness test?
> > 
> > I've attempted to use window.testRunner, but it seems that is actually part
of
> > pre-testHarness.js infrastructure. Looking at the logic in
> single_test_runner.py
> > [0], I see that having any image output is a condition of NOT being a
> > testHarness.js test. So I suspect I'm in uncharted territory. 
> > 
> > So the next question is, if I want to do this, should I use testRunner to
dump
> > the pixels and just refactor the logic in single_test_runner to allow pixels
> > with testHarness tests?
> > 
> >
>
https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitp...
> 
> testharness.js is the framework we get from the W3C, and it (intentionally)
does
> not expose a way
> to get screenshots, because screenshots require manual review.
> 
> I think you roughly have three options: 
> 
> 1) Use the js-test framework instead.
> 
> 2) Split this test into two: one that does the asserting, and one that just
> produces the image.
> 
> 3) Try to convince people that we should have a non-standard extension to
> testharness or get
> the W3C to buy in to this extension.
> 
> I don't think option (3) is a very good idea, but it's possible there might be
> support for it from others.
> If you can make things work with js-test that's probably they way to go.
> 
> Does that help? (And sorry for the delay, I was out for a few days).

Thanks, this is insightful. I think I need to take option 1. The primary
function of this test is to verify media controls painted correctly via a
screenshot.

> because screenshots require manual review.

I'm confused. Our js-tests seem to compare the image to expected pixels in an
automated fashion. When do we do manual review?

Also, I keep seeing that js-tests is the past, test-harness the future. I
haven't seen any mention of actually deprecating js-tests... is that the
longterm plan? I know we have lots of tests that primarily verify via
screenshots, so just wondering about their future.

[Dirk Pranke, 2016-10-28 18:56:05]

On 2016/10/28 17:11:19, chcunningham wrote:
> On 2016/10/26 18:40:54, Dirk Pranke wrote:
> > On 2016/10/24 20:37:33, chcunningham wrote:
> > > On 2016/10/22 01:21:13, chcunningham wrote:
> > > > Hey Dirk, I'm attempting to create a new testharness test that also does
> > pixel
> > > > testing. But I think what I really have here is a Frankenstein thing.
Has
> > this
> > > > been done before? A better way?
> > > 
> > > To explain more: Its probably best to just ignore the code here completely
> and
> > > address the question: How do I take screenshots in a testHarness test?
> > > 
> > > I've attempted to use window.testRunner, but it seems that is actually
part
> of
> > > pre-testHarness.js infrastructure. Looking at the logic in
> > single_test_runner.py
> > > [0], I see that having any image output is a condition of NOT being a
> > > testHarness.js test. So I suspect I'm in uncharted territory. 
> > > 
> > > So the next question is, if I want to do this, should I use testRunner to
> dump
> > > the pixels and just refactor the logic in single_test_runner to allow
pixels
> > > with testHarness tests?
> > > 
> > >
> >
>
https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitp...
> > 
> > testharness.js is the framework we get from the W3C, and it (intentionally)
> does
> > not expose a way
> > to get screenshots, because screenshots require manual review.
> > 
> > I think you roughly have three options: 
> > 
> > 1) Use the js-test framework instead.
> > 
> > 2) Split this test into two: one that does the asserting, and one that just
> > produces the image.
> > 
> > 3) Try to convince people that we should have a non-standard extension to
> > testharness or get
> > the W3C to buy in to this extension.
> > 
> > I don't think option (3) is a very good idea, but it's possible there might
be
> > support for it from others.
> > If you can make things work with js-test that's probably they way to go.
> > 
> > Does that help? (And sorry for the delay, I was out for a few days).
> 
> Thanks, this is insightful. I think I need to take option 1. The primary
> function of this test is to verify media controls painted correctly via a
> screenshot.
> 
> > because screenshots require manual review.
> 
> I'm confused. Our js-tests seem to compare the image to expected pixels in an
> automated fashion. When do we do manual review?

When the initial version of a baseline gets checked in or when baselines are
updated
(as part of code reviews). The day-to-day execution of the suite is then
automated.

The w3c doesn't have an equivalent of baselines or repeated executions, so it's
just "manual review" to them.

> Also, I keep seeing that js-tests is the past, test-harness the future. I
> haven't seen any mention of actually deprecating js-tests... is that the
> longterm plan? I know we have lots of tests that primarily verify via
> screenshots, so just wondering about their future.

I don't believe we have any expectation of being able to remove js-tests.
It's not actually deprecated, just not preferred.

 Put differently, you should use testharness tests or reftests if you can, 
but those two will never work for everything, and it seems somewhat less
likely that we'll ever get to a standardized w3c framework for the other uses
(though maybe we'll see something someday based around selenium/webdriver).