Modify telemetry tests to write test-results-compatible json files.

Modify telemetry tests to write test-results-compatible json files.

This will allow us to be able to intelligently de-apply patches,
re-run the tests, and compare the results on the bot. A follow-on
patch will also enable uploading the results to
test-results.appspot.com

R=dtu@chromium.org
BUG=323212
Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=284871

[Dirk Pranke, 2014-07-22 22:45:25]

Here's all that it should take to be able to write out the test results in a
format that the bots can understand and parse to be able to deal with failures
at tip-of-tree.

We will need to land a separate patch in the recipes to pass the right flags and
do the right thing.

This patch as-is will not upload the results to the flakiness dashboard; a
trivial follow-on patch will enable that.

I considered implementing this as a separate output formatting class, but given
that we're only doing stuff at the end of the run rather than with each test, it
didn't seem to me like the right way to go. Let me know if you would prefer
something like that.

[ojan, 2014-07-22 22:52:13]

Can this go in the infa.git repo? That way it could share code with
mojo/tools/run_mojo_python_tests.py.

[Dirk Pranke, 2014-07-22 22:56:23]

On 2014/07/22 22:52:13, ojan-only-code-yellow-reviews wrote:
> Can this go in the infa.git repo? That way it could share code with
> mojo/tools/run_mojo_python_tests.py.

What I actually want to do is land the revised test runner I'm working on in
infra (or buildtools), and use that test runner from here. That test runner will
replace run_mojo_python_tests and test-webkitpy (which are simpler than
telemetry's tests and run-webkit-tests, which will only re-use some of this
code).

I posted this as-is so that we weren't blocked on the revised test runner in
order to get telemetry working now (discussed this w/ dtu and tonyg face-to-face
already).

Sound okay?

[ojan, 2014-07-22 23:18:05]

non-owner lgtm

[dtu, 2014-07-22 23:52:04]

https://chromiumcodereview.appspot.com/407243004/diff/20001/tools/telemetry/t...
File tools/telemetry/telemetry/unittest/json_results.py (right):

https://chromiumcodereview.appspot.com/407243004/diff/20001/tools/telemetry/t...
tools/telemetry/telemetry/unittest/json_results.py:11: #
src/mojo/tools/run/mojo_python_tests.py, and also duplicates logic
nit: tools/run_mojo

https://chromiumcodereview.appspot.com/407243004/diff/20001/tools/telemetry/t...
tools/telemetry/telemetry/unittest/json_results.py:81: def _AllTestNames(suite):
For Telemetry TestResult objects in particular:

return set(result.successes) | set(test for test, _ in result.skipped) |
_FailedTestNames(result)

Then you don't need to access the suite at all.

But it's fine to leave as-is if you want to minimize changes from mojo.

https://chromiumcodereview.appspot.com/407243004/diff/20001/tools/telemetry/t...
tools/telemetry/telemetry/unittest/json_results.py:92: def
_FailedTestNames(result):
return set(test.id() for test, _ in result.failures + result.errors)

https://chromiumcodereview.appspot.com/407243004/diff/20001/tools/telemetry/t...
tools/telemetry/telemetry/unittest/json_results.py:112: def _UnitTestName(test):
This is just test.id().

[dtu, 2014-07-22 23:54:10]

lgtm in case you want to punt on refactorings because there's a new test runner
coming.

[Dirk Pranke, 2014-07-23 00:13:40]

https://chromiumcodereview.appspot.com/407243004/diff/20001/tools/telemetry/t...
File tools/telemetry/telemetry/unittest/json_results.py (right):

https://chromiumcodereview.appspot.com/407243004/diff/20001/tools/telemetry/t...
tools/telemetry/telemetry/unittest/json_results.py:11: #
src/mojo/tools/run/mojo_python_tests.py, and also duplicates logic
On 2014/07/22 23:52:04, dtu wrote:
> nit: tools/run_mojo

Done.

https://chromiumcodereview.appspot.com/407243004/diff/20001/tools/telemetry/t...
tools/telemetry/telemetry/unittest/json_results.py:81: def _AllTestNames(suite):
On 2014/07/22 23:52:04, dtu wrote:
> For Telemetry TestResult objects in particular:
> 
> return set(result.successes) | set(test for test, _ in result.skipped) |
> _FailedTestNames(result)
> 
> Then you don't need to access the suite at all.
> 
> But it's fine to leave as-is if you want to minimize changes from mojo.

Yeah, I think this (and test.id()) were new in unittest2. 

I'm not sure if we actually want the 'skipped' tests; I'll have to think about
that a bit, they might actually need the 'SKIP' annotation instead.

https://chromiumcodereview.appspot.com/407243004/diff/20001/tools/telemetry/t...
tools/telemetry/telemetry/unittest/json_results.py:92: def
_FailedTestNames(result):
On 2014/07/22 23:52:04, dtu wrote:
> return set(test.id() for test, _ in result.failures + result.errors)

Yeah, see comment above. Will fix.

https://chromiumcodereview.appspot.com/407243004/diff/20001/tools/telemetry/t...
tools/telemetry/telemetry/unittest/json_results.py:112: def _UnitTestName(test):
On 2014/07/22 23:52:04, dtu wrote:
> This is just test.id().

Yeah, also see comment above. Will fix.

[Dirk Pranke, 2014-07-23 01:41:40]

https://codereview.chromium.org/407243004/diff/20001/tools/telemetry/telemetr...
File tools/telemetry/telemetry/unittest/json_results.py (right):

https://codereview.chromium.org/407243004/diff/20001/tools/telemetry/telemetr...
tools/telemetry/telemetry/unittest/json_results.py:81: def _AllTestNames(suite):
Actually, for this one I'm going to leave the code as-is for consistency.

[Dirk Pranke, 2014-07-23 01:41:45]

The CQ bit was checked by dpranke@chromium.org

[commit-bot: I haz the power, 2014-07-23 01:43:07]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/dpranke@chromium.org/407243004/40001

[commit-bot: I haz the power, 2014-07-23 06:48:00]

Change committed as 284871