Add support for uploading results to the flakiness dashboard

Add support for uploading results to the flakiness dashboard.

R=ojan@chromium.org, dtu@chromium.org
BUG=323212
Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=288551

[Dirk Pranke, 2014-08-07 23:41:49]

I think this is ready for review. I attempted to test it locally, and my dev
appengine instance of the test result server keeled over on a bug, but it
"looked" like the data being uploaded was correct. I can test further if someone
is willing to look at the app server :).

AFAIK we don't have a shared test instance that has a separate database; I
didn't want to test against the live instance, though we can certainly do that.
It's not like we have any valid results for this step yet that would get
polluted.

In order for this to actually do something, you need to add a bunch of command
line args to run_tests to ensure that everything gets populated properly, e.g.:

./run_tests --test-results-server=test-results.appspot.com
--master-name=chromium.webkit --builder-name "Linux Tests" --test-type
"telemetry_unittests" --metadata chromium_revision=288100 --metadata
blink_revision=179736 --metadata build_number=1001

https://codereview.chromium.org/456483002/diff/40001/tools/telemetry/telemetr...
File tools/telemetry/telemetry/unittest/json_results.py (right):

https://codereview.chromium.org/456483002/diff/40001/tools/telemetry/telemetr...
tools/telemetry/telemetry/unittest/json_results.py:197: while retry_count <= 3:
Ojan, do you think we still need to support retries for uploads?

[ojan, 2014-08-08 18:35:22]

lgtm

There's test-results-test.appspot.com for testing. For some reason the frontends
are hitting the resource limit when they're basically doing nothing. I think
it's updatebuilders is having issues. We probably just need to push a new server
and it'll fix itself once the quota resets.

https://codereview.chromium.org/456483002/diff/40001/tools/telemetry/telemetr...
File tools/telemetry/telemetry/unittest/json_results.py (right):

https://codereview.chromium.org/456483002/diff/40001/tools/telemetry/telemetr...
tools/telemetry/telemetry/unittest/json_results.py:29:
parser.add_option("--test-results-server", default="",
Should we default to test-results.appspot.com? See next comment below.

https://codereview.chromium.org/456483002/diff/40001/tools/telemetry/telemetr...
tools/telemetry/telemetry/unittest/json_results.py:42: if
(args.test_results_server and
I'd like to find ways to make the bot actually fail to run if it's not
configured correctly to upload. Increasingly we'll be depending on this data as
the source of truth for whether a test run failed and which test failed, so
uploading is not optional.

https://codereview.chromium.org/456483002/diff/40001/tools/telemetry/telemetr...
tools/telemetry/telemetry/unittest/json_results.py:197: while retry_count <= 3:
No idea. We don't really track how often this happens and results in the file
successfully being uploaded. I've only seen the cases where it retries 3 times
and still fails. I'd leave it out for now and add it back in if we find it's a
problem in practice.

[Dirk Pranke, 2014-08-08 19:46:30]

> There's test-results-test.appspot.com for testing. 

Ah, for some reasons I thought that used the same underlying data source. 

Okay, I'll run a test against that instance as well.

https://codereview.chromium.org/456483002/diff/40001/tools/telemetry/telemetr...
File tools/telemetry/telemetry/unittest/json_results.py (right):

https://codereview.chromium.org/456483002/diff/40001/tools/telemetry/telemetr...
tools/telemetry/telemetry/unittest/json_results.py:29:
parser.add_option("--test-results-server", default="",
On 2014/08/08 18:35:21, ojan-only-code-yellow-reviews wrote:
> Should we default to test-results.appspot.com? See next comment below.

We need some mechanism to indicate "please upload the results"; currently that
is done by passing a value for this. If we used a default value, we'd need some
other sort of flag.

(The behavior in this patch matches what we do in run-webkit-tests).

https://codereview.chromium.org/456483002/diff/40001/tools/telemetry/telemetr...
tools/telemetry/telemetry/unittest/json_results.py:42: if
(args.test_results_server and
On 2014/08/08 18:35:21, ojan-only-code-yellow-reviews wrote:
> I'd like to find ways to make the bot actually fail to run if it's not
> configured correctly to upload. Increasingly we'll be depending on this data
as
> the source of truth for whether a test run failed and which test failed, so
> uploading is not optional.

Throwing a parser error will cause the step to fail, so that's probably good
enough.

https://codereview.chromium.org/456483002/diff/40001/tools/telemetry/telemetr...
tools/telemetry/telemetry/unittest/json_results.py:197: while retry_count <= 3:
On 2014/08/08 18:35:21, ojan-only-code-yellow-reviews wrote:
> No idea. We don't really track how often this happens and results in the file
> successfully being uploaded. I've only seen the cases where it retries 3 times
> and still fails. I'd leave it out for now and add it back in if we find it's a
> problem in practice.

Okay, I was reasonably well inclined to do that, as well, so we'll give that a
shot. Obviously it's easy enough to add a retry if it turns out we need it.

[Dirk Pranke, 2014-08-08 20:06:24]

The CQ bit was checked by dpranke@chromium.org

[Dirk Pranke, 2014-08-08 20:07:03]

Ah, I see now what you mean by 'hitting the resource limit'. I'll go w/ the
untested route for now and we'll see how it goes.

[commit-bot: I haz the power, 2014-08-08 20:07:11]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/dpranke@chromium.org/456483002/60001

[dtu, 2014-08-08 21:09:08]

lgtm

[commit-bot: I haz the power, 2014-08-09 02:07:01]

FYI, CQ is re-trying this CL (attempt #1).
The failing builders are:
  win_chromium_x64_rel on tryserver.chromium.win
(http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_...)

[commit-bot: I haz the power, 2014-08-09 10:22:43]

Change committed as 288551