Record screenshots on failures on swarming bots.

Record screenshots on failures on swarming bots.

[mikecase (-- gone --), 2017-03-01 18:36:30]

mikecase@chromium.org changed reviewers:
+ bpastene@chromium.org, jbudorick@chromium.org

[mikecase (-- gone --), 2017-03-01 18:36:31]

Question, where is the best place to add test arguments for swarming bots.

For example, I wanted to add this --screenshot-dir argument for only
instrumentation tests run on swarming bots and I didn't see a good place.

Also, about this CL, the screenshots will get uploaded to Google Storage, and
then the links to the images will be automatically added to results_details. So
it doesnt matter that they are only saved to a temp directory.

[jbudorick, 2017-03-01 20:19:21]

https://codereview.chromium.org/2721323003/diff/1/build/android/test_runner.py
File build/android/test_runner.py (right):

https://codereview.chromium.org/2721323003/diff/1/build/android/test_runner.p...
build/android/test_runner.py:118: group.add_argument(
Why is this moving to the generic arguments when it's only supported for
instrumentation tests?

https://codereview.chromium.org/2721323003/diff/1/tools/mb/mb.py
File tools/mb/mb.py (right):

https://codereview.chromium.org/2721323003/diff/1/tools/mb/mb.py#newcode1094
tools/mb/mb.py:1094: '--screenshot-dir', '$(mktemp -d)',
I don't think we ought to pass this for anything but instrumentation tests...

[bpastene, 2017-03-01 20:35:54]

https://codereview.chromium.org/2721323003/diff/1/tools/mb/mb.py
File tools/mb/mb.py (right):

https://codereview.chromium.org/2721323003/diff/1/tools/mb/mb.py#newcode1094
tools/mb/mb.py:1094: '--screenshot-dir', '$(mktemp -d)',
On 2017/03/01 20:19:21, jbudorick wrote:
> I don't think we ought to pass this for anything but instrumentation tests...

You could add the arg to all instrumentation tests in the testing/buildbot json
specs like:
https://chromium.googlesource.com/chromium/src/+/a515dc389dc5e4a6ecf5583f3569...

 It'd be pretty tedious and painful, but it should work.

[jbudorick, 2017-03-01 20:45:39]

https://codereview.chromium.org/2721323003/diff/1/tools/mb/mb.py
File tools/mb/mb.py (right):

https://codereview.chromium.org/2721323003/diff/1/tools/mb/mb.py#newcode1094
tools/mb/mb.py:1094: '--screenshot-dir', '$(mktemp -d)',
On 2017/03/01 20:35:54, bpastene wrote:
> On 2017/03/01 20:19:21, jbudorick wrote:
> > I don't think we ought to pass this for anything but instrumentation
tests...
> 
> You could add the arg to all instrumentation tests in the testing/buildbot
json
> specs like:
>
https://chromium.googlesource.com/chromium/src/+/a515dc389dc5e4a6ecf5583f3569...
> 
>  It'd be pretty tedious and painful, but it should work.

Ideally, this would be less tedious once we're generating the test spec, but
yeah, that's kinda what I was thinking.

[mikecase (-- gone --), 2017-03-02 19:45:24]

mikecase@chromium.org changed reviewers:
+ yolandyan@chromium.org
- bpastene@chromium.org

[mikecase (-- gone --), 2017-03-02 19:45:24]

Changing approach since I feel like it is silly to have bots have to specify
--screenshot-dir=tempdir when that tempdir is only used within the test runner
script.

Instead, adding code that if screenshot-dir is NOT specified, but we still want
to upload the screenshots, we create a tempdir in that case. This CL will make
most bots take a screenshot on failed instrumentation tests and upload them.
Those running result detail steps will have links to those screenshots on the
Buildbot page.

[mikecase (-- gone --), 2017-05-02 21:06:04]

Going to merge this CL with the reland of the change this depended on. Closing