Add a script to test Chrome with the NVDA screen reader.

Add a script to test Chrome with the NVDA screen reader.

This script automates testing Chrome with NVDA to help catch
regressions. Eventually maybe this could be run on bots, but
right now it requires several manual steps to set up, but still
saves valuable manual testing time.

BUG=389622
NOTRY=true

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=283699

[dmazzoni, 2014-06-27 17:55:15]

Suggestions for where to put this code are welcome, this is just the first idea
I had.

[David Tseng, 2014-06-30 16:55:29]

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/REA...
File tools/accessibility/nvda/README.txt (right):

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/REA...
tools/accessibility/nvda/README.txt:1: This directory contains semi-automated
tests of Chrome with
Add OWNERS file in this directory?

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/REA...
tools/accessibility/nvda/README.txt:33: just use the stable version instead,
from http://www.nvaccess.org/download/
Do they have a tracking bug?

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
File tools/accessibility/nvda/nvda_chrome_tests.py (right):

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
tools/accessibility/nvda/nvda_chrome_tests.py:6: """Semi-automated tests of
Chrome with NVDA.
What does it mean to be semi automated?

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
tools/accessibility/nvda/nvda_chrome_tests.py:75: time.sleep(1)
Is this needed? Looks flakey.

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
tools/accessibility/nvda/nvda_chrome_tests.py:93: time.sleep(1)
Ditto.

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
tools/accessibility/nvda/nvda_chrome_tests.py:94: 
Wrt ordering, do we ever want to test Chrome initial launch with NVDA already
enabled?

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
tools/accessibility/nvda/nvda_chrome_tests.py:112: print 'Killing Nvda
subprocess'
nit: NVDA

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
tools/accessibility/nvda/nvda_chrome_tests.py:115: print 'Nvda already died.'
Ditto.

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
tools/accessibility/nvda/nvda_chrome_tests.py:125: [u'Google Chrome', u'window',
u'slash']
Any chance NVDA could just use json for the log format?

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
tools/accessibility/nvda/nvda_chrome_tests.py:142: time.sleep(1)
Going to be flakey; any way we could just listen to events from NVDA?

[dmazzoni, 2014-07-15 06:09:58]

No more flaky sleep calls! Ready for another look.

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/REA...
File tools/accessibility/nvda/README.txt (right):

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/REA...
tools/accessibility/nvda/README.txt:1: This directory contains semi-automated
tests of Chrome with
On 2014/06/30 16:55:28, David Tseng wrote:
> Add OWNERS file in this directory?

Done.

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/REA...
tools/accessibility/nvda/README.txt:33: just use the stable version instead,
from http://www.nvaccess.org/download/
On 2014/06/30 16:55:28, David Tseng wrote:
> Do they have a tracking bug?

It's already merged into the stable branch, so I changed it to specify the exact
version number of the next stable version to use.

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
File tools/accessibility/nvda/nvda_chrome_tests.py (right):

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
tools/accessibility/nvda/nvda_chrome_tests.py:6: """Semi-automated tests of
Chrome with NVDA.
On 2014/06/30 16:55:28, David Tseng wrote:
> What does it mean to be semi automated?

Clarified in comments. I basically mean that it's not really set up to run on a
build bot, you have to manually kick off the tests, but they're automated
otherwise.

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
tools/accessibility/nvda/nvda_chrome_tests.py:75: time.sleep(1)
On 2014/06/30 16:55:28, David Tseng wrote:
> Is this needed? Looks flakey.

OK, I successfully removed this and replaced it with a WaitForSpeech helper
function.

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
tools/accessibility/nvda/nvda_chrome_tests.py:94: 
On 2014/06/30 16:55:29, David Tseng wrote:
> Wrt ordering, do we ever want to test Chrome initial launch with NVDA already
> enabled?

Sure. My guess is that opening a new browser window would cover 90% of the
possible issues with opening Chrome from scratch so I'll probably start there.

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
tools/accessibility/nvda/nvda_chrome_tests.py:112: print 'Killing Nvda
subprocess'
On 2014/06/30 16:55:29, David Tseng wrote:
> nit: NVDA

Done.

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
tools/accessibility/nvda/nvda_chrome_tests.py:115: print 'Nvda already died.'
On 2014/06/30 16:55:29, David Tseng wrote:
> Ditto.

Done.

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
tools/accessibility/nvda/nvda_chrome_tests.py:125: [u'Google Chrome', u'window',
u'slash']
On 2014/06/30 16:55:28, David Tseng wrote:
> Any chance NVDA could just use json for the log format?

Happy to ask - but this seems robust enough for what we need.

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
tools/accessibility/nvda/nvda_chrome_tests.py:142: time.sleep(1)
On 2014/06/30 16:55:29, David Tseng wrote:
> Going to be flakey; any way we could just listen to events from NVDA?

I think I've removed the flakiness. I really like keeping these really
high-level and just focusing on key events and speech, not digging into anything
lower-level than that if possible.

[dmazzoni, 2014-07-15 15:41:18]

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
File tools/accessibility/nvda/nvda_chrome_tests.py (right):

https://codereview.chromium.org/331363004/diff/1/tools/accessibility/nvda/nvd...
tools/accessibility/nvda/nvda_chrome_tests.py:94: 
On 2014/07/15 06:09:58, dmazzoni wrote:
> On 2014/06/30 16:55:29, David Tseng wrote:
> > Wrt ordering, do we ever want to test Chrome initial launch with NVDA
already
> > enabled?

Oh, I almost forgot - one big reason I did it in this order is because NVDA gave
us a command-line argument that tells it to pay attention to just one process id
and ignore any others. That helps cut down on flakiness a lot, because otherwise
it would announce any notifications or events from other applications. if we
started NVDA and then Chrome, the test has the potential to be a bit flakier.

[David Tseng, 2014-07-15 21:34:45]

lgtm

Just a few questions.

https://codereview.chromium.org/331363004/diff/40001/tools/accessibility/nvda...
File tools/accessibility/nvda/nvda_chrome_tests.py (right):

https://codereview.chromium.org/331363004/diff/40001/tools/accessibility/nvda...
tools/accessibility/nvda/nvda_chrome_tests.py:2: # Copyright (c) 2014 The
Chromium Authors. All rights reserved.
nit: I think the (c) isn't needed.

https://codereview.chromium.org/331363004/diff/40001/tools/accessibility/nvda...
tools/accessibility/nvda/nvda_chrome_tests.py:155: return result
Do you want/care to clear the speech log? (so that you don't end up parsing the
entire contents every time you wait).

https://codereview.chromium.org/331363004/diff/40001/tools/accessibility/nvda...
tools/accessibility/nvda/nvda_chrome_tests.py:175: if (lines[-len(expected):] ==
expected):
What if some other speech slips in right after your expected text?

[dmazzoni, 2014-07-17 06:52:43]

https://codereview.chromium.org/331363004/diff/40001/tools/accessibility/nvda...
File tools/accessibility/nvda/nvda_chrome_tests.py (right):

https://codereview.chromium.org/331363004/diff/40001/tools/accessibility/nvda...
tools/accessibility/nvda/nvda_chrome_tests.py:2: # Copyright (c) 2014 The
Chromium Authors. All rights reserved.
On 2014/07/15 21:34:45, David Tseng wrote:
> nit: I think the (c) isn't needed.

Done.

https://codereview.chromium.org/331363004/diff/40001/tools/accessibility/nvda...
tools/accessibility/nvda/nvda_chrome_tests.py:155: return result
On 2014/07/15 21:34:45, David Tseng wrote:
> Do you want/care to clear the speech log? (so that you don't end up parsing
the
> entire contents every time you wait).

I'm not sure how to do this easily without potentially breaking NVDA, since it
has the file open for writing. I could save the byte number, I guess.

In practice, each test is pretty small and this takes negligible time, so I'm
not going to worry about it for now. We can optimize it later when we have a
longer test.

https://codereview.chromium.org/331363004/diff/40001/tools/accessibility/nvda...
tools/accessibility/nvda/nvda_chrome_tests.py:175: if (lines[-len(expected):] ==
expected):
On 2014/07/15 21:34:45, David Tseng wrote:
> What if some other speech slips in right after your expected text?

Currently these tests really want to assert that it was the last thing spoken,
otherwise they could get confused when the same utterance repeats more than
once. However, if we encounter flakiness due to unrelated speech, we can address
that later. So far that hasn't happened.

[dmazzoni, 2014-07-17 06:54:29]

The CQ bit was checked by dmazzoni@chromium.org

[commit-bot: I haz the power, 2014-07-17 07:00:09]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/dmazzoni@chromium.org/331363004/60001

[commit-bot: I haz the power, 2014-07-17 07:48:42]

Change committed as 283699