Add metrics for Web Components

Add metrics for Web Components

Add basic metrics for features used in Web Components.

BUG=chromium:703996

[hayato, 2017-03-22 06:03:30]

Description was changed from

==========
WIP: Web Componennts metrics

BUG=catapult:#
==========

to

==========
WIP: Web Componennts metrics

BUG=chromium:703996
==========

[hayato, 2017-03-22 06:10:20]

WIP:

[hayato, 2017-03-22 06:18:58]

Description was changed from

==========
WIP: Web Componennts metrics

BUG=chromium:703996
==========

to

==========
Add metrics for Web Components

Add basic metrics for features used in Web Components.

BUG=chromium:703996
==========

[hayato, 2017-03-22 06:26:50]

hayato@chromium.org changed reviewers:
+ nednguyen@google.com

[hayato, 2017-03-22 06:26:50]

nednguyen@, could you early check this WIP patch so that it would meet
catapult's requirements?

Since I have just started to learn how telemetry works, and this is the first
patch in catapult repository for me, I appreciate any feedback so that this
patch would become ready to be landed.

I have tried to make this CL smallest as I can. I'll add more things in another
CL, hopefully.

FYI. Blink side's WIP patch is here: https://codereview.chromium.org/2703443004

[nednguyen, 2017-03-22 13:23:53]

On 2017/03/22 06:26:50, hayato wrote:
> nednguyen@, could you early check this WIP patch so that it would meet
> catapult's requirements?
> 
> Since I have just started to learn how telemetry works, and this is the first
> patch in catapult repository for me, I appreciate any feedback so that this
> patch would become ready to be landed.
> 
> I have tried to make this CL smallest as I can. I'll add more things in
another
> CL, hopefully.
> 
> FYI. Blink side's WIP patch is here:
https://codereview.chromium.org/2703443004

Sorry, looking at your CL, it seems to me that you just want to sum up the
duration of a bunch of trace events? 

If so, this should be using the blink perf harness + the tracing feature
proposed in
https://groups.google.com/a/chromium.org/d/msg/platform-architecture-dev/5POw...
instead of TBMv2 metric.

[hayato, 2017-03-23 04:05:48]

Hi Ned, thank you for taking a look.

Sorry, I should have been more explicit about this WIP CL. This CL is not the
final result. My plan is to add more complex metrics.

For example,

- I would like to add more page sets so that we can measure these metrics for a
bunch of sites in real worlds easily.
- I would like to add more stories, like a button click, for some sites, to know
how it affects style recalc or something.
- I would like to add more advanced(?) metrics which is not simply the result of
a trace event.

Though these are not done yet, it looks TBMv2 meets our requirements.

Thus, I appreciate if you would check using TBMv2 for our purpose is a preferred
way or not.

If it sounds good, I will continue to implement more metrics using the feature
of TBMv2.
I would like to send a review incrementally to avoid a huge patch, and would
like to get early feedback to know my direction is wrong or not.

Does my plan sound good from the view of catapult's expert?

[nednguyen, 2017-03-23 14:23:53]

Description was changed from

==========
Add metrics for Web Components

Add basic metrics for features used in Web Components.

BUG=chromium:703996
==========

to

==========
Add metrics for Web Components

Add basic metrics for features used in Web Components.

BUG=chromium:703996
==========

[nednguyen, 2017-03-23 14:23:53]

nednguyen@google.com changed reviewers:
+ tdresser@chromium.org

[nednguyen, 2017-03-23 14:28:53]

On 2017/03/23 04:05:48, hayato wrote:
> Hi Ned, thank you for taking a look.
> 
> Sorry, I should have been more explicit about this WIP CL. This CL is not the
> final result. My plan is to add more complex metrics.
> 
> For example,
> 
> - I would like to add more page sets so that we can measure these metrics for
a
> bunch of sites in real worlds easily.
> - I would like to add more stories, like a button click, for some sites, to
know
> how it affects style recalc or something.
> - I would like to add more advanced(?) metrics which is not simply the result
of
> a trace event.
> 
> Though these are not done yet, it looks TBMv2 meets our requirements.
> 
> Thus, I appreciate if you would check using TBMv2 for our purpose is a
preferred
> way or not.
> 
> If it sounds good, I will continue to implement more metrics using the feature
> of TBMv2.
> I would like to send a review incrementally to avoid a huge patch, and would
> like to get early feedback to know my direction is wrong or not.
> 
> Does my plan sound good from the view of catapult's expert?

Thanks for your explanation. Our recommendation is:
1) For simple metric (like computing time duration of trace events) & test, you
should use blink_perf testing. You have full control over the page structure,
thus making fixing issues when the test breakage much easier. I think there is
nothing us from doing things like adding button click to tests using blink perf
harness?

2) For complicated metrics & tests, you're in the realm of performance
integration testing. For this, our recommendation is that your metrics should
fit into a class of user facing metrics we have: loading, responsiveness,
power/cpu,...
(https://docs.google.com/document/d/1oiy0_ych1v2ADhyG_QW7Ps4BNER2ShlJjx2zCbVzV...
for the list)l

For each of those core metric, we have a suite of tests that you can add your
test case to. Be aware that we are in the middle of refactoring our harness, so
your timeline will be affected if you choose this route.

[hayato, 2017-03-24 06:47:31]

Thank you.


On 2017/03/23 at 14:28:53, nednguyen wrote:

> Thanks for your explanation. Our recommendation is:
> 1) For simple metric (like computing time duration of trace events) & test,
you should use blink_perf testing. You have full control over the page
structure, thus making fixing issues when the test breakage much easier. I think
there is nothing us from doing things like adding button click to tests using
blink perf harness?
> 

Yeah, we have been using blink_perf testing to measure Web Components's
performance for years, as micro benchmarks. We continue to use blink_perf when
it is suitable. That does not change.
What I would like to do in the next is to measure the performance in
applications in real world.


> 2) For complicated metrics & tests, you're in the realm of performance
integration testing. For this, our recommendation is that your metrics should
fit into a class of user facing metrics we have: loading, responsiveness,
power/cpu,...
(https://docs.google.com/document/d/1oiy0_ych1v2ADhyG_QW7Ps4BNER2ShlJjx2zCbVzV...
for the list)l
> 
> For each of those core metric, we have a suite of tests that you can add your
test case to. Be aware that we are in the middle of refactoring our harness, so
your timeline will be affected if you choose this route.

Ah, the document looks useful. Let me take a deeper look later. So, unless there
is an absolute need for adding new *aspect* of metrics (new html file), it is
recommended to update the existing *.html file, right?
I, rather, just thought that updating existing metrics should be avoided because
I might distract existing users of the metrics, but that does not look true. If
it it is okay to update the existing metrics which fit our purpose, let me do
that.

I might ask more advice later. Thanks.

[nednguyen, 2017-03-24 08:45:13]

On 2017/03/24 06:47:31, hayato wrote:
> Thank you.
> 
> 
> On 2017/03/23 at 14:28:53, nednguyen wrote:
> 
> > Thanks for your explanation. Our recommendation is:
> > 1) For simple metric (like computing time duration of trace events) & test,
> you should use blink_perf testing. You have full control over the page
> structure, thus making fixing issues when the test breakage much easier. I
think
> there is nothing us from doing things like adding button click to tests using
> blink perf harness?
> > 
> 
> Yeah, we have been using blink_perf testing to measure Web Components's
> performance for years, as micro benchmarks. We continue to use blink_perf when
> it is suitable. That does not change.
> What I would like to do in the next is to measure the performance in
> applications in real world.
> 
> 
> > 2) For complicated metrics & tests, you're in the realm of performance
> integration testing. For this, our recommendation is that your metrics should
> fit into a class of user facing metrics we have: loading, responsiveness,
> power/cpu,...
>
(https://docs.google.com/document/d/1oiy0_ych1v2ADhyG_QW7Ps4BNER2ShlJjx2zCbVzV...
> for the list)l
> > 
> > For each of those core metric, we have a suite of tests that you can add
your
> test case to. Be aware that we are in the middle of refactoring our harness,
so
> your timeline will be affected if you choose this route.
> 
> Ah, the document looks useful. Let me take a deeper look later. So, unless
there
> is an absolute need for adding new *aspect* of metrics (new html file), it is
> recommended to update the existing *.html file, right?

Correct. Coming up with new real world metric is not easy, so at least we should
make sure that they fit into our narrative of measuring real world performance.

> I, rather, just thought that updating existing metrics should be avoided
because
> I might distract existing users of the metrics, but that does not look true.
If
> it it is okay to update the existing metrics which fit our purpose, let me do
> that.
> 
> I might ask more advice later. Thanks.

[hayato, 2017-04-18 03:37:19]

no land; just for sharing purpose

[hayato, 2017-04-18 03:37:20]

The CQ bit was checked by hayato@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-04-18 03:37:29]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-04-18 03:39:38]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-04-18 03:39:38]

Dry run: Try jobs failed on following builders:
  Catapult Presubmit on master.tryserver.client.catapult (JOB_FAILED,
https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Pr...)