Upgrade v8.todomvc{,-ignition} to TBMv2

Upgrade v8.todomvc{,-ignition} to TBMv2

BUG=593406, 593793
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq

Committed: https://crrev.com/5800dfe3f534cd97ff433501b3e45c56831d2f2b
Cr-Commit-Position: refs/heads/master@{#385381}

[jochen (gone - plz use gerrit), 2016-04-03 12:06:14]

Description was changed from

==========
Upgrade v8.todomvc{,-ignition} to TBMv2

BUG=
==========

to

==========
Upgrade v8.todomvc{,-ignition} to TBMv2

BUG=
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq
==========

[eakuefner, 2016-04-04 22:28:34]

eakuefner@chromium.org changed reviewers:
+ eakuefner@chromium.org, sullivan@chromium.org

[eakuefner, 2016-04-04 22:28:35]

+sullivan for question about alerting.

Jochen, once https://codereview.chromium.org/1860753003 lands and Catapult rolls
into Chromium, you should be able to have this work. I'll keep an eye and try to
ping this CL once that's done.

Annie, once Jochen does this, the metric names reported by these benchmarks will
change. Do you think alerting will need to be modified as a result?

[jochen (gone - plz use gerrit), 2016-04-05 01:43:45]

On 2016/04/04 at 22:28:35, eakuefner wrote:
> +sullivan for question about alerting.
> 
> Jochen, once https://codereview.chromium.org/1860753003 lands and Catapult
rolls into Chromium, you should be able to have this work. I'll keep an eye and
try to ping this CL once that's done.
> 
> Annie, once Jochen does this, the metric names reported by these benchmarks
will change. Do you think alerting will need to be modified as a result?

We don't have monitoring on for those benchmarks yet

[sullivan, 2016-04-05 01:47:02]

On 2016/04/05 01:43:45, jochen wrote:
> On 2016/04/04 at 22:28:35, eakuefner wrote:
> > +sullivan for question about alerting.
> > 
> > Jochen, once https://codereview.chromium.org/1860753003 lands and Catapult
> rolls into Chromium, you should be able to have this work. I'll keep an eye
and
> try to ping this CL once that's done.
> > 
> > Annie, once Jochen does this, the metric names reported by these benchmarks
> will change. Do you think alerting will need to be modified as a result?
> 
> We don't have monitoring on for those benchmarks yet

When you're ready to turn it on, use the "report issue" menu from the perf
dashboard.

[jochen (gone - plz use gerrit), 2016-04-05 04:56:19]

Description was changed from

==========
Upgrade v8.todomvc{,-ignition} to TBMv2

BUG=
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq
==========

to

==========
Upgrade v8.todomvc{,-ignition} to TBMv2

BUG=593406,593793
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq
==========

[jochen (gone - plz use gerrit), 2016-04-05 06:03:54]

@eakuefner, I now get the following error with ToT catapult:

Traceback (most recent call last):
  RunBenchmark at
blink/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:311
    benchmark.ShouldTearDownStateAfterEachStoryRun())
  Run at
/blink/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:224
    _RunStoryAndProcessErrorIfNeeded(story, results, state, test)
  _RunStoryAndProcessErrorIfNeeded at
/blink/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:86
    test.Measure(state.platform, results)
  Measure at
/blink/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:284
    self._ComputeTimelineBasedMetric(results, trace_value)
  _ComputeTimelineBasedMetric at
/blink/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:314
    translate_common_values.TranslateScalarValue(d, page))
  TranslateScalarValue at
/blink/src/third_party/catapult/telemetry/telemetry/value/translate_common_values.py:34
    return scalar.ScalarValue.FromDict(scalar_value, {page.id: page})
  FromDict at
/blink/src/third_party/catapult/telemetry/telemetry/value/scalar.py:88
    return ScalarValue(**kwargs)
  __init__ at
/blink/src/third_party/catapult/telemetry/telemetry/value/scalar.py:28
    none_values.ValidateNoneValueReason(value, none_value_reason)
  ValidateNoneValueReason at
/blink/src/third_party/catapult/telemetry/telemetry/value/none_values.py:22
    raise NoneValueMissingReason()
NoneValueMissingReason

and the benchmark results are incomplete:

RESULT v8_compile_ignition_cpu_self_count: Polymer= 0 unitless
RESULT v8_compile_ignition_cpu_self_sum: Polymer= 0 ms
RESULT v8_execution_cpu_self_avg: Polymer= 0.769891089109 ms
RESULT v8_execution_cpu_self_count: Polymer= 101 unitless
RESULT v8_execution_cpu_self_sum: Polymer= 77.759 ms
RESULT v8_execution_cpu_total_avg: Polymer= 1.52465346535 ms
RESULT v8_execution_cpu_total_count: Polymer= 101 unitless
RESULT v8_execution_cpu_total_sum: Polymer= 153.99 ms
RESULT v8_execution_wall_self_avg: Polymer= 0.809653465347 ms
RESULT v8_execution_wall_self_count: Polymer= 101 unitless
RESULT v8_execution_wall_self_sum: Polymer= 81.775 ms
RESULT v8_execution_wall_total_avg: Polymer= 1.57826732673 ms
RESULT v8_execution_wall_total_count: Polymer= 101 unitless
RESULT v8_execution_wall_total_sum: Polymer= 159.405 ms
RESULT v8_parse_lazy_cpu_self_avg: Polymer= 0.0512570921986 ms
RESULT v8_parse_lazy_cpu_self_count: Polymer= 564 unitless
RESULT v8_parse_lazy_cpu_self_sum: Polymer= 28.909 ms
RESULT v8_parse_lazy_wall_self_avg: Polymer= 0.052329787234 ms
RESULT v8_parse_lazy_wall_self_count: Polymer= 564 unitless
RESULT v8_parse_lazy_wall_self_sum: Polymer= 29.514 ms

makes me suspect that it's choking on v8_compile_ignition_cpu_self_avg

[jochen (gone - plz use gerrit), 2016-04-05 06:04:05]

@eakuefner, I now get the following error with ToT catapult:

Traceback (most recent call last):
  RunBenchmark at
blink/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:311
    benchmark.ShouldTearDownStateAfterEachStoryRun())
  Run at
/blink/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:224
    _RunStoryAndProcessErrorIfNeeded(story, results, state, test)
  _RunStoryAndProcessErrorIfNeeded at
/blink/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:86
    test.Measure(state.platform, results)
  Measure at
/blink/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:284
    self._ComputeTimelineBasedMetric(results, trace_value)
  _ComputeTimelineBasedMetric at
/blink/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:314
    translate_common_values.TranslateScalarValue(d, page))
  TranslateScalarValue at
/blink/src/third_party/catapult/telemetry/telemetry/value/translate_common_values.py:34
    return scalar.ScalarValue.FromDict(scalar_value, {page.id: page})
  FromDict at
/blink/src/third_party/catapult/telemetry/telemetry/value/scalar.py:88
    return ScalarValue(**kwargs)
  __init__ at
/blink/src/third_party/catapult/telemetry/telemetry/value/scalar.py:28
    none_values.ValidateNoneValueReason(value, none_value_reason)
  ValidateNoneValueReason at
/blink/src/third_party/catapult/telemetry/telemetry/value/none_values.py:22
    raise NoneValueMissingReason()
NoneValueMissingReason

and the benchmark results are incomplete:

RESULT v8_compile_ignition_cpu_self_count: Polymer= 0 unitless
RESULT v8_compile_ignition_cpu_self_sum: Polymer= 0 ms
RESULT v8_execution_cpu_self_avg: Polymer= 0.769891089109 ms
RESULT v8_execution_cpu_self_count: Polymer= 101 unitless
RESULT v8_execution_cpu_self_sum: Polymer= 77.759 ms
RESULT v8_execution_cpu_total_avg: Polymer= 1.52465346535 ms
RESULT v8_execution_cpu_total_count: Polymer= 101 unitless
RESULT v8_execution_cpu_total_sum: Polymer= 153.99 ms
RESULT v8_execution_wall_self_avg: Polymer= 0.809653465347 ms
RESULT v8_execution_wall_self_count: Polymer= 101 unitless
RESULT v8_execution_wall_self_sum: Polymer= 81.775 ms
RESULT v8_execution_wall_total_avg: Polymer= 1.57826732673 ms
RESULT v8_execution_wall_total_count: Polymer= 101 unitless
RESULT v8_execution_wall_total_sum: Polymer= 159.405 ms
RESULT v8_parse_lazy_cpu_self_avg: Polymer= 0.0512570921986 ms
RESULT v8_parse_lazy_cpu_self_count: Polymer= 564 unitless
RESULT v8_parse_lazy_cpu_self_sum: Polymer= 28.909 ms
RESULT v8_parse_lazy_wall_self_avg: Polymer= 0.052329787234 ms
RESULT v8_parse_lazy_wall_self_count: Polymer= 564 unitless
RESULT v8_parse_lazy_wall_self_sum: Polymer= 29.514 ms

makes me suspect that it's choking on v8_compile_ignition_cpu_self_avg

[eakuefner, 2016-04-05 06:15:41]

On 2016/04/05 at 06:04:05, jochen wrote:
> @eakuefner, I now get the following error with ToT catapult:
> 
> Traceback (most recent call last):
>   RunBenchmark at
blink/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:311
>     benchmark.ShouldTearDownStateAfterEachStoryRun())
>   Run at
/blink/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:224
>     _RunStoryAndProcessErrorIfNeeded(story, results, state, test)
>   _RunStoryAndProcessErrorIfNeeded at
/blink/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:86
>     test.Measure(state.platform, results)
>   Measure at
/blink/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:284
>     self._ComputeTimelineBasedMetric(results, trace_value)
>   _ComputeTimelineBasedMetric at
/blink/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:314
>     translate_common_values.TranslateScalarValue(d, page))
>   TranslateScalarValue at
/blink/src/third_party/catapult/telemetry/telemetry/value/translate_common_values.py:34
>     return scalar.ScalarValue.FromDict(scalar_value, {page.id: page})
>   FromDict at
/blink/src/third_party/catapult/telemetry/telemetry/value/scalar.py:88
>     return ScalarValue(**kwargs)
>   __init__ at
/blink/src/third_party/catapult/telemetry/telemetry/value/scalar.py:28
>     none_values.ValidateNoneValueReason(value, none_value_reason)
>   ValidateNoneValueReason at
/blink/src/third_party/catapult/telemetry/telemetry/value/none_values.py:22
>     raise NoneValueMissingReason()
> NoneValueMissingReason
> 
> and the benchmark results are incomplete:
> 
> RESULT v8_compile_ignition_cpu_self_count: Polymer= 0 unitless
> RESULT v8_compile_ignition_cpu_self_sum: Polymer= 0 ms
> RESULT v8_execution_cpu_self_avg: Polymer= 0.769891089109 ms
> RESULT v8_execution_cpu_self_count: Polymer= 101 unitless
> RESULT v8_execution_cpu_self_sum: Polymer= 77.759 ms
> RESULT v8_execution_cpu_total_avg: Polymer= 1.52465346535 ms
> RESULT v8_execution_cpu_total_count: Polymer= 101 unitless
> RESULT v8_execution_cpu_total_sum: Polymer= 153.99 ms
> RESULT v8_execution_wall_self_avg: Polymer= 0.809653465347 ms
> RESULT v8_execution_wall_self_count: Polymer= 101 unitless
> RESULT v8_execution_wall_self_sum: Polymer= 81.775 ms
> RESULT v8_execution_wall_total_avg: Polymer= 1.57826732673 ms
> RESULT v8_execution_wall_total_count: Polymer= 101 unitless
> RESULT v8_execution_wall_total_sum: Polymer= 159.405 ms
> RESULT v8_parse_lazy_cpu_self_avg: Polymer= 0.0512570921986 ms
> RESULT v8_parse_lazy_cpu_self_count: Polymer= 564 unitless
> RESULT v8_parse_lazy_cpu_self_sum: Polymer= 28.909 ms
> RESULT v8_parse_lazy_wall_self_avg: Polymer= 0.052329787234 ms
> RESULT v8_parse_lazy_wall_self_count: Polymer= 564 unitless
> RESULT v8_parse_lazy_wall_self_sum: Polymer= 29.514 ms
> 
> makes me suspect that it's choking on v8_compile_ignition_cpu_self_avg

Yes, I think the problem here is that we don't initialize the average to 0
because that's incorrect if you take sum(l)/len(l) to be the average of a list.

Technically this should be a NoneValue, and the reason would be that you can't
average an empty list.

I'll fix this tomorrow, thanks for reporting.

[eakuefner, 2016-04-05 21:02:15]

On 2016/04/05 at 06:15:41, eakuefner wrote:
> On 2016/04/05 at 06:04:05, jochen wrote:
> > @eakuefner, I now get the following error with ToT catapult:
> > 
> > Traceback (most recent call last):
> >   RunBenchmark at
blink/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:311
> >     benchmark.ShouldTearDownStateAfterEachStoryRun())
> >   Run at
/blink/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:224
> >     _RunStoryAndProcessErrorIfNeeded(story, results, state, test)
> >   _RunStoryAndProcessErrorIfNeeded at
/blink/src/third_party/catapult/telemetry/telemetry/internal/story_runner.py:86
> >     test.Measure(state.platform, results)
> >   Measure at
/blink/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:284
> >     self._ComputeTimelineBasedMetric(results, trace_value)
> >   _ComputeTimelineBasedMetric at
/blink/src/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py:314
> >     translate_common_values.TranslateScalarValue(d, page))
> >   TranslateScalarValue at
/blink/src/third_party/catapult/telemetry/telemetry/value/translate_common_values.py:34
> >     return scalar.ScalarValue.FromDict(scalar_value, {page.id: page})
> >   FromDict at
/blink/src/third_party/catapult/telemetry/telemetry/value/scalar.py:88
> >     return ScalarValue(**kwargs)
> >   __init__ at
/blink/src/third_party/catapult/telemetry/telemetry/value/scalar.py:28
> >     none_values.ValidateNoneValueReason(value, none_value_reason)
> >   ValidateNoneValueReason at
/blink/src/third_party/catapult/telemetry/telemetry/value/none_values.py:22
> >     raise NoneValueMissingReason()
> > NoneValueMissingReason
> > 
> > and the benchmark results are incomplete:
> > 
> > RESULT v8_compile_ignition_cpu_self_count: Polymer= 0 unitless
> > RESULT v8_compile_ignition_cpu_self_sum: Polymer= 0 ms
> > RESULT v8_execution_cpu_self_avg: Polymer= 0.769891089109 ms
> > RESULT v8_execution_cpu_self_count: Polymer= 101 unitless
> > RESULT v8_execution_cpu_self_sum: Polymer= 77.759 ms
> > RESULT v8_execution_cpu_total_avg: Polymer= 1.52465346535 ms
> > RESULT v8_execution_cpu_total_count: Polymer= 101 unitless
> > RESULT v8_execution_cpu_total_sum: Polymer= 153.99 ms
> > RESULT v8_execution_wall_self_avg: Polymer= 0.809653465347 ms
> > RESULT v8_execution_wall_self_count: Polymer= 101 unitless
> > RESULT v8_execution_wall_self_sum: Polymer= 81.775 ms
> > RESULT v8_execution_wall_total_avg: Polymer= 1.57826732673 ms
> > RESULT v8_execution_wall_total_count: Polymer= 101 unitless
> > RESULT v8_execution_wall_total_sum: Polymer= 159.405 ms
> > RESULT v8_parse_lazy_cpu_self_avg: Polymer= 0.0512570921986 ms
> > RESULT v8_parse_lazy_cpu_self_count: Polymer= 564 unitless
> > RESULT v8_parse_lazy_cpu_self_sum: Polymer= 28.909 ms
> > RESULT v8_parse_lazy_wall_self_avg: Polymer= 0.052329787234 ms
> > RESULT v8_parse_lazy_wall_self_count: Polymer= 564 unitless
> > RESULT v8_parse_lazy_wall_self_sum: Polymer= 29.514 ms
> > 
> > makes me suspect that it's choking on v8_compile_ignition_cpu_self_avg
> 
> Yes, I think the problem here is that we don't initialize the average to 0
because that's incorrect if you take sum(l)/len(l) to be the average of a list.
> 
> Technically this should be a NoneValue, and the reason would be that you can't
average an empty list.
> 
> I'll fix this tomorrow, thanks for reporting.

Jochen, FYI https://codereview.chromium.org/1857003003 has landed in Catapult
which should fix this problem.

If there are any more issues please let me know!

[jochen (gone - plz use gerrit), 2016-04-05 21:15:12]

now it works, cool!

can you please review this CL?

[eakuefner, 2016-04-05 21:38:28]

lgtm

[jochen (gone - plz use gerrit), 2016-04-05 23:44:17]

The CQ bit was checked by jochen@chromium.org

[commit-bot: I haz the power, 2016-04-05 23:44:39]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1857613002/1
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1857613002/1

[commit-bot: I haz the power, 2016-04-06 03:51:59]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-04-06 03:52:00]

Try jobs failed on following builders:
  android_s5_perf_cq on tryserver.chromium.perf (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.perf/builders/android_s5_perf_...)

[jochen (gone - plz use gerrit), 2016-04-06 04:03:25]

The CQ bit was checked by jochen@chromium.org

[jochen (gone - plz use gerrit), 2016-04-06 04:03:42]

The CQ bit was unchecked by jochen@chromium.org

[commit-bot: I haz the power, 2016-04-06 04:03:49]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1857613002/1
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1857613002/1

[jochen (gone - plz use gerrit), 2016-04-06 04:03:58]

Description was changed from

==========
Upgrade v8.todomvc{,-ignition} to TBMv2

BUG=593406,593793
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq
==========

to

==========
Upgrade v8.todomvc{,-ignition} to TBMv2

BUG=593406,593793
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq
==========

[jochen (gone - plz use gerrit), 2016-04-06 04:04:00]

The CQ bit was checked by jochen@chromium.org

[commit-bot: I haz the power, 2016-04-06 04:04:25]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1857613002/1
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1857613002/1

[commit-bot: I haz the power, 2016-04-06 04:09:31]

Committed patchset #1 (id:1)

[commit-bot: I haz the power, 2016-04-06 04:11:00]

Description was changed from

==========
Upgrade v8.todomvc{,-ignition} to TBMv2

BUG=593406,593793
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq
==========

to

==========
Upgrade v8.todomvc{,-ignition} to TBMv2

BUG=593406,593793
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:winx64_10_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq;tryserver.chromium.perf:linux_perf_cq

Committed: https://crrev.com/5800dfe3f534cd97ff433501b3e45c56831d2f2b
Cr-Commit-Position: refs/heads/master@{#385381}
==========

[commit-bot: I haz the power, 2016-04-06 04:11:01]

Patchset 1 (id:??) landed as
https://crrev.com/5800dfe3f534cd97ff433501b3e45c56831d2f2b
Cr-Commit-Position: refs/heads/master@{#385381}