Ignore a11y failures in extension browser_tests that enter dev mode.

Ignore a11y failures in extension browser_tests that enter dev mode.

BUG=463245
Committed: https://crrev.com/a695de78dc3236b3d4b6914a037287979a50b29b
Cr-Commit-Position: refs/heads/master@{#327178}

[hcarmona, 2015-04-25 00:22:45]

hcarmona@chromium.org changed reviewers:
+ dbeam@chromium.org, kalman@chromium.org, rdevlin.cronin@chromium.org,
thakis@chromium.org

[hcarmona, 2015-04-25 00:22:46]

CL to ignore a11y failures introduced in
https://codereview.chromium.org/1090763003/

[Devlin, 2015-04-27 15:49:25]

lgtm, but you'll need an OWNERS stamp.

https://codereview.chromium.org/1104883002/diff/1/chrome/browser/ui/webui/ext...
File chrome/browser/ui/webui/extensions/extension_settings_browsertest.js
(right):

https://codereview.chromium.org/1104883002/diff/1/chrome/browser/ui/webui/ext...
chrome/browser/ui/webui/extensions/extension_settings_browsertest.js:123: * mode
is enabled.
nit: I'd comment on why we ignore it, and why it's probably okay (and maybe do
the same briefly in the CL description).  (i.e., it fails because of an overlap,
but that overlap is only true during a short transition, so by the end it would
pass, if it could).

[Dan Beam, 2015-04-27 15:55:01]

what? why? this is a regression then...

[Devlin, 2015-04-27 16:00:47]

On 2015/04/27 15:55:01, Dan Beam wrote:
> what? why? this is a regression then...

I'm don't think this test ever passed un-flakily with a11y audits, and, to my
naive eyes, the error seems like it's just complaining that (*before* the
transition), the dev controls are buried - which is true and intentional.  Of
course, if there's a way around this that doesn't involve just suppressing
audits here, that would be preferable. :)

[Dan Beam, 2015-04-27 16:13:40]

On 2015/04/27 16:00:47, Devlin wrote:
> On 2015/04/27 15:55:01, Dan Beam wrote:
> > what? why? this is a regression then...
> 
> I'm don't think this test ever passed un-flakily with a11y audits, and, to my
> naive eyes, the error seems like it's just complaining that (*before* the
> transition), the dev controls are buried - which is true and intentional.  Of
> course, if there's a way around this that doesn't involve just suppressing
> audits here, that would be preferable. :)

the way is to make the default state for hidden controls... hidden/untabbled (or
aria-hidden).

[Dan Beam, 2015-04-27 16:14:10]

On 2015/04/27 16:13:40, Dan Beam wrote:
> On 2015/04/27 16:00:47, Devlin wrote:
> > On 2015/04/27 15:55:01, Dan Beam wrote:
> > > what? why? this is a regression then...
> > 
> > I'm don't think this test ever passed un-flakily with a11y audits, and, to
my
> > naive eyes, the error seems like it's just complaining that (*before* the
> > transition), the dev controls are buried - which is true and intentional. 
Of
> > course, if there's a way around this that doesn't involve just suppressing
> > audits here, that would be preferable. :)
> 
> the way is to make the default state for hidden controls... hidden/untabbled
(or
> aria-hidden).

untabable* (if that's a word)

[Nico, 2015-04-27 17:23:23]

On 2015/04/27 15:55:01, Dan Beam wrote:
> what? why? this is a regression then...

See https://codereview.chromium.org/1090763003/#msg37 and
https://codereview.chromium.org/1090763003/#msg40 . It started failing in
official builds after https://codereview.chromium.org/1090763003/ . The failing
test was added in that CL, so it couldn't have been passing before (if that's
what the question is about)

[hcarmona, 2015-04-27 17:31:56]

On 2015/04/27 16:14:10, Dan Beam wrote:
> On 2015/04/27 16:13:40, Dan Beam wrote:
> > On 2015/04/27 16:00:47, Devlin wrote:
> > > On 2015/04/27 15:55:01, Dan Beam wrote:
> > > > what? why? this is a regression then...
> > > 
> > > I'm don't think this test ever passed un-flakily with a11y audits, and, to
> my
> > > naive eyes, the error seems like it's just complaining that (*before* the
> > > transition), the dev controls are buried - which is true and intentional. 
> Of
> > > course, if there's a way around this that doesn't involve just suppressing
> > > audits here, that would be preferable. :)
> > 
> > the way is to make the default state for hidden controls... hidden/untabbled
> (or
> > aria-hidden).
> 
> untabable* (if that's a word)

The controls are appropriately tabable/nontabable whne they're visible
or hidden. We have issues when transitioning.

I've tried speeding up the transition (0ms !important), but the
transition still happens. It should take 0ms but on some trybots this
apparently takes longer. Even worse is that we don't get a
webkitTransitionEnd event for a 0ms transition.

[Dan Beam, 2015-04-27 17:43:55]

On 2015/04/27 17:31:56, Hector Carmona wrote:
> On 2015/04/27 16:14:10, Dan Beam wrote:
> > On 2015/04/27 16:13:40, Dan Beam wrote:
> > > On 2015/04/27 16:00:47, Devlin wrote:
> > > > On 2015/04/27 15:55:01, Dan Beam wrote:
> > > > > what? why? this is a regression then...
> > > > 
> > > > I'm don't think this test ever passed un-flakily with a11y audits, and,
to
> > my
> > > > naive eyes, the error seems like it's just complaining that (*before*
the
> > > > transition), the dev controls are buried - which is true and
intentional. 
> > Of
> > > > course, if there's a way around this that doesn't involve just
suppressing
> > > > audits here, that would be preferable. :)
> > > 
> > > the way is to make the default state for hidden controls...
hidden/untabbled
> > (or
> > > aria-hidden).
> > 
> > untabable* (if that's a word)
> 
> The controls are appropriately tabable/nontabable whne they're visible
> or hidden. We have issues when transitioning.
> 
> I've tried speeding up the transition (0ms !important), but the
> transition still happens. It should take 0ms but on some trybots this
> apparently takes longer. Even worse is that we don't get a
> webkitTransitionEnd event for a 0ms transition.

so is the problem that we can't choose a reliable timeout length that's
guaranteed to work?

[hcarmona, 2015-04-27 17:48:15]

On 2015/04/27 17:43:55, Dan Beam wrote:
> On 2015/04/27 17:31:56, Hector Carmona wrote:
> > On 2015/04/27 16:14:10, Dan Beam wrote:
> > > On 2015/04/27 16:13:40, Dan Beam wrote:
> > > > On 2015/04/27 16:00:47, Devlin wrote:
> > > > > On 2015/04/27 15:55:01, Dan Beam wrote:
> > > > > > what? why? this is a regression then...
> > > > > 
> > > > > I'm don't think this test ever passed un-flakily with a11y audits,
and,
> to
> > > my
> > > > > naive eyes, the error seems like it's just complaining that (*before*
> the
> > > > > transition), the dev controls are buried - which is true and
> intentional. 
> > > Of
> > > > > course, if there's a way around this that doesn't involve just
> suppressing
> > > > > audits here, that would be preferable. :)
> > > > 
> > > > the way is to make the default state for hidden controls...
> hidden/untabbled
> > > (or
> > > > aria-hidden).
> > > 
> > > untabable* (if that's a word)
> > 
> > The controls are appropriately tabable/nontabable whne they're visible
> > or hidden. We have issues when transitioning.
> > 
> > I've tried speeding up the transition (0ms !important), but the
> > transition still happens. It should take 0ms but on some trybots this
> > apparently takes longer. Even worse is that we don't get a
> > webkitTransitionEnd event for a 0ms transition.
> 
> so is the problem that we can't choose a reliable timeout length that's
> guaranteed to work?

We can increase the timeout, but how big is big enough? Before the
previous CL, there was a disabled test with a 4000ms timeout, and that
was STILL flaky when it was disabled.

[Dan Beam, 2015-04-27 17:54:40]

On 2015/04/27 17:48:15, Hector Carmona wrote:
> On 2015/04/27 17:43:55, Dan Beam wrote:
> > On 2015/04/27 17:31:56, Hector Carmona wrote:
> > > On 2015/04/27 16:14:10, Dan Beam wrote:
> > > > On 2015/04/27 16:13:40, Dan Beam wrote:
> > > > > On 2015/04/27 16:00:47, Devlin wrote:
> > > > > > On 2015/04/27 15:55:01, Dan Beam wrote:
> > > > > > > what? why? this is a regression then...
> > > > > > 
> > > > > > I'm don't think this test ever passed un-flakily with a11y audits,
> and,
> > to
> > > > my
> > > > > > naive eyes, the error seems like it's just complaining that
(*before*
> > the
> > > > > > transition), the dev controls are buried - which is true and
> > intentional. 
> > > > Of
> > > > > > course, if there's a way around this that doesn't involve just
> > suppressing
> > > > > > audits here, that would be preferable. :)
> > > > > 
> > > > > the way is to make the default state for hidden controls...
> > hidden/untabbled
> > > > (or
> > > > > aria-hidden).
> > > > 
> > > > untabable* (if that's a word)
> > > 
> > > The controls are appropriately tabable/nontabable whne they're visible
> > > or hidden. We have issues when transitioning.
> > > 
> > > I've tried speeding up the transition (0ms !important), but the
> > > transition still happens. It should take 0ms but on some trybots this
> > > apparently takes longer. Even worse is that we don't get a
> > > webkitTransitionEnd event for a 0ms transition.
> > 
> > so is the problem that we can't choose a reliable timeout length that's
> > guaranteed to work?
> 
> We can increase the timeout, but how big is big enough? Before the
> previous CL, there was a disabled test with a 4000ms timeout, and that
> was STILL flaky when it was disabled.

it seems like if any time is flaky, all times would be flaky.  we should just
ensureTransitionEndEvent(el, 0);

[hcarmona, 2015-04-27 17:58:46]

On 2015/04/27 17:54:40, Dan Beam wrote:
> On 2015/04/27 17:48:15, Hector Carmona wrote:
> > On 2015/04/27 17:43:55, Dan Beam wrote:
> > > On 2015/04/27 17:31:56, Hector Carmona wrote:
> > > > On 2015/04/27 16:14:10, Dan Beam wrote:
> > > > > On 2015/04/27 16:13:40, Dan Beam wrote:
> > > > > > On 2015/04/27 16:00:47, Devlin wrote:
> > > > > > > On 2015/04/27 15:55:01, Dan Beam wrote:
> > > > > > > > what? why? this is a regression then...
> > > > > > > 
> > > > > > > I'm don't think this test ever passed un-flakily with a11y audits,
> > and,
> > > to
> > > > > my
> > > > > > > naive eyes, the error seems like it's just complaining that
> (*before*
> > > the
> > > > > > > transition), the dev controls are buried - which is true and
> > > intentional. 
> > > > > Of
> > > > > > > course, if there's a way around this that doesn't involve just
> > > suppressing
> > > > > > > audits here, that would be preferable. :)
> > > > > > 
> > > > > > the way is to make the default state for hidden controls...
> > > hidden/untabbled
> > > > > (or
> > > > > > aria-hidden).
> > > > > 
> > > > > untabable* (if that's a word)
> > > > 
> > > > The controls are appropriately tabable/nontabable whne they're visible
> > > > or hidden. We have issues when transitioning.
> > > > 
> > > > I've tried speeding up the transition (0ms !important), but the
> > > > transition still happens. It should take 0ms but on some trybots this
> > > > apparently takes longer. Even worse is that we don't get a
> > > > webkitTransitionEnd event for a 0ms transition.
> > > 
> > > so is the problem that we can't choose a reliable timeout length that's
> > > guaranteed to work?
> > 
> > We can increase the timeout, but how big is big enough? Before the
> > previous CL, there was a disabled test with a 4000ms timeout, and that
> > was STILL flaky when it was disabled.
> 
> it seems like if any time is flaky, all times would be flaky.  we should just
> ensureTransitionEndEvent(el, 0);

That's the equivalent of setting a timeout because there won't be a
webkitTransitionEnd for a 0ms transition and if we use a non-zero
transition it will still be equivalent to a timeout on a slow machine
where the timeout will happen while the transition is still in progress.

[Dan Beam, 2015-04-27 18:27:41]

let's just focus on fixing this for a little while longer.

i think we can modify the behavior for testing (e.g. disable transitions,
dispatch synthetic events if required) and/or change when the a11y audit runs.

[Nico, 2015-04-27 21:46:48]

(what's the status here? the official-test-running bots remain red.)

[Dan Beam, 2015-04-27 22:22:45]

lgtm if you want to commit while we deflake

[hcarmona, 2015-04-27 22:27:42]

The CQ bit was checked by hcarmona@chromium.org

[commit-bot: I haz the power, 2015-04-27 22:28:22]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1104883002/1

[Nico, 2015-04-27 22:58:57]

I just noticed that this test is failing on non-official regular trybots too:
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...

So it's good to see this CL in the cq.

[commit-bot: I haz the power, 2015-04-27 23:47:22]

Committed patchset #1 (id:1)

[commit-bot: I haz the power, 2015-04-27 23:49:12]

Patchset 1 (id:??) landed as
https://crrev.com/a695de78dc3236b3d4b6914a037287979a50b29b
Cr-Commit-Position: refs/heads/master@{#327178}