Updating isolate scripts to read in chartjson telemetry results

testing/scripts/run_telemetry_benchmark_as_googletest.py

Index: testing/scripts/run_telemetry_benchmark_as_googletest.py
diff --git a/testing/scripts/run_telemetry_benchmark_as_googletest.py b/testing/scripts/run_telemetry_benchmark_as_googletest.py
index 387c5c8da95993f984f20522253b471eca93a58d..fe7bcd06d5c41adc5be2870c9e14feaa521d3c03 100755
--- a/testing/scripts/run_telemetry_benchmark_as_googletest.py
+++ b/testing/scripts/run_telemetry_benchmark_as_googletest.py
@@ -62,6 +62,10 @@ def main():
     tempfile_dir = tempfile.mkdtemp('telemetry')
     valid = True
     failures = []
+    chartjson = False
+    if '--output-format=chartjson' in rest_args:
+      chartjson = True
+    chartresults = None
     try:
       rc = common.run_command([sys.executable] + rest_args + [
         '--output-dir', tempfile_dir,
@@ -74,6 +78,11 @@ def main():
         if value['type'] == 'failure':
           failures.append(results['pages'][str(value['page_id'])]['name'])
       valid = bool(rc == 0 or failures)
+      # If we have also output chartjson read it in and return it
+      if chartjson:
+        chart_tempfile_name = os.path.join(tempfile_dir, 'results-chart.json')
+        with open(chart_tempfile_name) as f:
+          chartresults = json.load(f)
     except Exception:
       traceback.print_exc()
       valid = False
@@ -85,10 +94,17 @@ def main():
       if rc == 0:
         rc = 1  # Signal an abnormal exit.
 
-    json.dump({
-        'valid': valid,
-        'failures': failures,
-    }, args.isolated_script_test_output)
+    if chartjson:
+      json.dump({
+          'valid': valid,
+          'failures': failures,
+          'chartjson': chartresults
+      }, args.isolated_script_test_output)
+    else:
+      json.dump({
+          'valid': valid,
+          'failures': failures
+      }, args.isolated_script_test_output)

[Ken Russell (switch to Gerrit), 2016/09/12 19:31:42]

Rather than overloading the meaning of --isolated-script-test-output, I think it
is a better direction to add a new command line argument
--isolated-script-chartjson-output taking a separate file to which the chartjson
results are written. This simplified JSON format is mainly used to display
results on the waterfall, but it's known that we are going to need to output
more "full" JSON results for the flakiness dashboard. It would be better to
start extending the contract between recipes and IsolatedScriptTest now. Do you
agree?

[eyaich1, 2016/09/13 11:53:41]

Good to know there is another use case coming down the pipe.  Adding another
output was one of my options, but I wasn't sure if I was still going to need the
regular json output or not.  My plan was to make the output just the chartjson
once I concluded that.  Therefore, I was going to iterate on a solution once I
got something up on the waterfall and figure out if I needed both from there.

I am open to going down this route first though, I don't think it was that much
more work, I just chose this route while I was still in investigation phase.  I
will ping this CL again when changes are made.

     return rc
 
   finally: