[tools/perf] Add memory.top_10_mobile_stress benchmark

[tools/perf] Add memory.top_10_mobile_stress benchmark

For reproducibility, soon we'll tear down the state between page
sets on all benchmarks by default at:
https://codereview.chromium.org/2099233002/
(also see attached bug for context)

This would leave uncovered the case of leaks and crashes in the browser
after prolonged use.

For that we plan to enable memory.long_running_dual_browser_test,
which runs for 60 iterations with no browser restarts, but is currently
blocked because it would produce too much data
(crbug.com/623015).

Thus, in the meantime, we add this "realistic" clone (no state tearing
down) of memory.top_10_mobile to keep some minimum of coverage.

BUG=625657
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq

Committed: https://crrev.com/ec3a082e76e4cc16c3b1281cd20b7ba6105d65fe
Cr-Commit-Position: refs/heads/master@{#403700}

[perezju, 2016-07-04 12:28:34]

Description was changed from

==========
[Telemetry] Add memory.top_10_mobile_stress benchmark

Run top 10 mobile page set without closing/restarting the browser. This
benchmark is intended to stress-test the browser, catching memory leaks or
possible crashes after interacting with the browser for a period of time.

Note: this is currently identical to memory.top_10_mobile, but that will
change after landing: https://codereview.chromium.org/2099233002/

BUG=623015,catapult:#2294
==========

to

==========
[Telemetry] Add memory.top_10_mobile_stress benchmark

Run top 10 mobile page set without closing/restarting the browser. This
benchmark is intended to stress-test the browser, catching memory leaks or
possible crashes after interacting with the browser for a period of time.

Note: this is currently identical to memory.top_10_mobile, but that will
change after landing: https://codereview.chromium.org/2099233002/

BUG=623015,catapult:#2294
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq
==========

[perezju, 2016-07-04 12:33:04]

perezju@chromium.org changed reviewers:
+ primiano@chromium.org, skyostil@chromium.org

[perezju, 2016-07-04 12:33:04]

Primiano: This would be the benchmark as we discussed this morning.

Sami: hope you can rubberstamp.

[Sami, 2016-07-04 13:05:19]

lgtm if Primiano is happy.

[Primiano Tucci (use gerrit), 2016-07-04 13:46:32]

primiano@chromium.org changed reviewers:
+ petrcermak@chromium.org

[Primiano Tucci (use gerrit), 2016-07-04 13:46:32]

On 2016/07/04 13:05:19, Sami wrote:
> lgtm if Primiano is happy.

I defer to Peter to make sure he is aware and this is aligned with all his
plans. RS-LGTM from my side.

I just think we need to add a bit more context in the commit message explaining
that:
- As per conversation in (link to your thread) we decided to tear down the state
on each page, for sake of reproducibility yada yada.
- This however would leave uncovered the case of leaks in the browser.
- We plan to have a longer stress test (link to the discussion if public) but
right now that is blocked on (whatever bug) because would produce too much data.
- In the meantime then we have this to keep a minimum coverage on longer pages

[perezju, 2016-07-04 14:29:45]

Description was changed from

==========
[Telemetry] Add memory.top_10_mobile_stress benchmark

Run top 10 mobile page set without closing/restarting the browser. This
benchmark is intended to stress-test the browser, catching memory leaks or
possible crashes after interacting with the browser for a period of time.

Note: this is currently identical to memory.top_10_mobile, but that will
change after landing: https://codereview.chromium.org/2099233002/

BUG=623015,catapult:#2294
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq
==========

to

==========
[Telemetry] Add memory.top_10_mobile_stress benchmark

For reproducibility, soon we'll tear down the state between page
sets on all benchmarks by default at:
https://codereview.chromium.org/2099233002/
(also see attached bug for context)

This would leave uncovered the case of leaks and crashes in the browser
after prolonged use.

For that we plan to enable memory.long_running_dual_browser_test,
which runs for 60 iterations with no browser restarts, but is currently
blocked because it would produce too much data
(crbug.com/623015).

Thus, in the meantime, we add this "realistic" clone (no state tearing
down) of memory.top_10_mobile to keep some minimum of coverage.

BUG=625657
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq
==========

[perezju, 2016-07-04 14:31:16]

On 2016/07/04 13:46:32, Primiano Tucci wrote:
> On 2016/07/04 13:05:19, Sami wrote:
> > lgtm if Primiano is happy.
> 
> I defer to Peter to make sure he is aware and this is aligned with all his
> plans. RS-LGTM from my side.
> 
> I just think we need to add a bit more context in the commit message
explaining
> that:
> - As per conversation in (link to your thread) we decided to tear down the
state
> on each page, for sake of reproducibility yada yada.
> - This however would leave uncovered the case of leaks in the browser.
> - We plan to have a longer stress test (link to the discussion if public) but
> right now that is blocked on (whatever bug) because would produce too much
data.
> - In the meantime then we have this to keep a minimum coverage on longer pages

Agreed. I filed crbug.com/625657 to hold more of the context of these changes,
and updated the commit message as you suggested.

[petrcermak, 2016-07-04 14:41:14]

LGTM with one comment.

Thanks,
Petr

https://codereview.chromium.org/2115393003/diff/1/tools/perf/benchmarks/memor...
File tools/perf/benchmarks/memory_infra.py (right):

https://codereview.chromium.org/2115393003/diff/1/tools/perf/benchmarks/memor...
tools/perf/benchmarks/memory_infra.py:77: # TODO(bashi): Workaround for
http://crbug.com/532075.
http://crbug.com/532075 is apparently fixed, so you shouldn't need this anymore.
If you do, please re-open the bug.

[petrcermak, 2016-07-04 14:42:20]

One more comment: s|[Telemetry]|[tools/perf]| in the description and title.

[perezju, 2016-07-04 14:44:06]

On 2016/07/04 14:41:14, petrcermak wrote:
> LGTM with one comment.
> 
> Thanks,
> Petr
> 
>
https://codereview.chromium.org/2115393003/diff/1/tools/perf/benchmarks/memor...
> File tools/perf/benchmarks/memory_infra.py (right):
> 
>
https://codereview.chromium.org/2115393003/diff/1/tools/perf/benchmarks/memor...
> tools/perf/benchmarks/memory_infra.py:77: # TODO(bashi): Workaround for
> http://crbug.com/532075.
> http://crbug.com/532075 is apparently fixed, so you shouldn't need this
anymore.
> If you do, please re-open the bug.

OK. I'll remove the decorator from this new benchmark and, if it doesn't fail,
I'll remove it from other benchmarks on a follow up CL.

[perezju, 2016-07-04 16:36:29]

Description was changed from

==========
[Telemetry] Add memory.top_10_mobile_stress benchmark

For reproducibility, soon we'll tear down the state between page
sets on all benchmarks by default at:
https://codereview.chromium.org/2099233002/
(also see attached bug for context)

This would leave uncovered the case of leaks and crashes in the browser
after prolonged use.

For that we plan to enable memory.long_running_dual_browser_test,
which runs for 60 iterations with no browser restarts, but is currently
blocked because it would produce too much data
(crbug.com/623015).

Thus, in the meantime, we add this "realistic" clone (no state tearing
down) of memory.top_10_mobile to keep some minimum of coverage.

BUG=625657
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq
==========

to

==========
[tools/perf] Add memory.top_10_mobile_stress benchmark

For reproducibility, soon we'll tear down the state between page
sets on all benchmarks by default at:
https://codereview.chromium.org/2099233002/
(also see attached bug for context)

This would leave uncovered the case of leaks and crashes in the browser
after prolonged use.

For that we plan to enable memory.long_running_dual_browser_test,
which runs for 60 iterations with no browser restarts, but is currently
blocked because it would produce too much data
(crbug.com/623015).

Thus, in the meantime, we add this "realistic" clone (no state tearing
down) of memory.top_10_mobile to keep some minimum of coverage.

BUG=625657
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq
==========

[perezju, 2016-07-04 16:36:35]

The CQ bit was checked by perezju@chromium.org

[perezju, 2016-07-04 16:36:35]

The patchset sent to the CQ was uploaded after l-g-t-m from
primiano@chromium.org, skyostil@chromium.org, petrcermak@chromium.org
Link to the patchset: https://codereview.chromium.org/2115393003/#ps20001
(title: "remove decorator")

[commit-bot: I haz the power, 2016-07-04 16:36:54]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-07-04 17:39:43]

Description was changed from

==========
[tools/perf] Add memory.top_10_mobile_stress benchmark

For reproducibility, soon we'll tear down the state between page
sets on all benchmarks by default at:
https://codereview.chromium.org/2099233002/
(also see attached bug for context)

This would leave uncovered the case of leaks and crashes in the browser
after prolonged use.

For that we plan to enable memory.long_running_dual_browser_test,
which runs for 60 iterations with no browser restarts, but is currently
blocked because it would produce too much data
(crbug.com/623015).

Thus, in the meantime, we add this "realistic" clone (no state tearing
down) of memory.top_10_mobile to keep some minimum of coverage.

BUG=625657
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq
==========

to

==========
[tools/perf] Add memory.top_10_mobile_stress benchmark

For reproducibility, soon we'll tear down the state between page
sets on all benchmarks by default at:
https://codereview.chromium.org/2099233002/
(also see attached bug for context)

This would leave uncovered the case of leaks and crashes in the browser
after prolonged use.

For that we plan to enable memory.long_running_dual_browser_test,
which runs for 60 iterations with no browser restarts, but is currently
blocked because it would produce too much data
(crbug.com/623015).

Thus, in the meantime, we add this "realistic" clone (no state tearing
down) of memory.top_10_mobile to keep some minimum of coverage.

BUG=625657
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq
==========

[commit-bot: I haz the power, 2016-07-04 17:39:44]

Committed patchset #2 (id:20001)

[commit-bot: I haz the power, 2016-07-04 17:39:45]

CQ bit was unchecked.

[commit-bot: I haz the power, 2016-07-04 17:41:01]

Description was changed from

==========
[tools/perf] Add memory.top_10_mobile_stress benchmark

For reproducibility, soon we'll tear down the state between page
sets on all benchmarks by default at:
https://codereview.chromium.org/2099233002/
(also see attached bug for context)

This would leave uncovered the case of leaks and crashes in the browser
after prolonged use.

For that we plan to enable memory.long_running_dual_browser_test,
which runs for 60 iterations with no browser restarts, but is currently
blocked because it would produce too much data
(crbug.com/623015).

Thus, in the meantime, we add this "realistic" clone (no state tearing
down) of memory.top_10_mobile to keep some minimum of coverage.

BUG=625657
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq
==========

to

==========
[tools/perf] Add memory.top_10_mobile_stress benchmark

For reproducibility, soon we'll tear down the state between page
sets on all benchmarks by default at:
https://codereview.chromium.org/2099233002/
(also see attached bug for context)

This would leave uncovered the case of leaks and crashes in the browser
after prolonged use.

For that we plan to enable memory.long_running_dual_browser_test,
which runs for 60 iterations with no browser restarts, but is currently
blocked because it would produce too much data
(crbug.com/623015).

Thus, in the meantime, we add this "realistic" clone (no state tearing
down) of memory.top_10_mobile to keep some minimum of coverage.

BUG=625657
CQ_EXTRA_TRYBOTS=tryserver.chromium.perf:android_s5_perf_cq;tryserver.chromium.perf:mac_retina_perf_cq

Committed: https://crrev.com/ec3a082e76e4cc16c3b1281cd20b7ba6105d65fe
Cr-Commit-Position: refs/heads/master@{#403700}
==========

[commit-bot: I haz the power, 2016-07-04 17:41:01]

Patchset 2 (id:??) landed as
https://crrev.com/ec3a082e76e4cc16c3b1281cd20b7ba6105d65fe
Cr-Commit-Position: refs/heads/master@{#403700}