Disable failing telemetry unit tests on chromeos.

Disable failing telemetry unit tests on chromeos.

BUG=483212
Committed: https://crrev.com/dfca3a35a4689831793b1ac8b0bd5f6727d2bcd6
Cr-Commit-Position: refs/heads/master@{#328416}

[achuithb, 2015-04-30 22:14:16]

The CQ bit was checked by achuith@chromium.org to run a CQ dry run

[achuithb, 2015-04-30 22:15:40]

achuith@chromium.org changed reviewers:
+ cylee@chromium.org, tbarzic@chromium.org

[achuithb, 2015-04-30 22:15:41]

Toni, Cheng-Yu please take a look

[commit-bot: I haz the power, 2015-04-30 22:15:49]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1115383002/20001

[achuithb, 2015-05-01 04:38:47]

The CQ bit was checked by achuith@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2015-05-01 04:39:21]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1115383002/100001

[commit-bot: I haz the power, 2015-05-01 05:39:03]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-05-01 05:39:05]

Dry run: Try jobs failed on following builders:
  linux_perf_bisect on tryserver.chromium.perf (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisec...)

[cylee1, 2015-05-04 11:17:00]

Thanks for taking care of it.

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/benchmarks/...
File tools/perf/benchmarks/blink_style.py (right):

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/benchmarks/...
tools/perf/benchmarks/blink_style.py:10: @benchmark.Disabled('reference',
'chromeos')
Why disable performance test ? I thought you're disabling failing unit test in
the CL.
Also the test runs successfully in my local test environment ?

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/benchmarks/...
File tools/perf/benchmarks/v8.py (right):

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/benchmarks/...
tools/perf/benchmarks/v8.py:25: @benchmark.Disabled('win', 'reference',
'chromeos')
same question here.

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/profile_cre...
File tools/perf/profile_creators/cookie_profile_extender.py (right):

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/profile_cre...
tools/perf/profile_creators/cookie_profile_extender.py:7: try:
Please leave some comment/reference here so people know why it needs a
try-catch.

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/profile_cre...
File tools/perf/profile_creators/cookie_profile_extender_unittest.py (right):

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/profile_cre...
tools/perf/profile_creators/cookie_profile_extender_unittest.py:11: sqlite3 =
None
How does it different from just "pass" since nobody can reference it anyway

BTW this is a workaround, I'm wondering if the root cause out-dated python
version ?
sqlite3 should be available after python 2.5
https://docs.python.org/2/library/sqlite3.html
If so we should try to make trybot use newer python ?

[cylee1, 2015-05-04 11:46:37]

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/benchmarks/...
File tools/perf/benchmarks/blink_style.py (right):

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/benchmarks/...
tools/perf/benchmarks/blink_style.py:10: @benchmark.Disabled('reference',
'chromeos')
On 2015/05/04 11:16:59, cylee1 wrote:
> Why disable performance test ? I thought you're disabling failing unit test in
> the CL.
> Also the test runs successfully in my local test environment ?

BTW, just found that the blink_style.top_25 fails on a new chunk of code added
on 5/1
https://codereview.chromium.org/1118593002
so I guess it's not problem on our side.

[achuithb, 2015-05-04 19:48:00]

Cheng-Yu, I looked at one of the test runs and disabled all the failing tests. I
also ran this locally and disabled everything that failed.

I think it's ok to be somewhat aggressive about disabling tests - we still have
to go through the tests and re-enable them one by one after figuring out why
they are failing. But I think the first priority is to get the bot green.

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/benchmarks/...
File tools/perf/benchmarks/blink_style.py (right):

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/benchmarks/...
tools/perf/benchmarks/blink_style.py:10: @benchmark.Disabled('reference',
'chromeos')
On 2015/05/04 11:46:36, cylee1 wrote:
> On 2015/05/04 11:16:59, cylee1 wrote:
> > Why disable performance test ? I thought you're disabling failing unit test
in
> > the CL.
> > Also the test runs successfully in my local test environment ?
> 
> BTW, just found that the blink_style.top_25 fails on a new chunk of code added
> on 5/1
> https://codereview.chromium.org/1118593002
> so I guess it's not problem on our side.
> 

Done.

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/benchmarks/...
File tools/perf/benchmarks/v8.py (right):

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/benchmarks/...
tools/perf/benchmarks/v8.py:25: @benchmark.Disabled('win', 'reference',
'chromeos')
On 2015/05/04 11:16:59, cylee1 wrote:
> same question here.

this failed on one of the test runs

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/profile_cre...
File tools/perf/profile_creators/cookie_profile_extender.py (right):

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/profile_cre...
tools/perf/profile_creators/cookie_profile_extender.py:7: try:
On 2015/05/04 11:16:59, cylee1 wrote:
> Please leave some comment/reference here so people know why it needs a
> try-catch.

Done.

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/profile_cre...
File tools/perf/profile_creators/cookie_profile_extender_unittest.py (right):

https://codereview.chromium.org/1115383002/diff/100001/tools/perf/profile_cre...
tools/perf/profile_creators/cookie_profile_extender_unittest.py:11: sqlite3 =
None
On 2015/05/04 11:16:59, cylee1 wrote:
> How does it different from just "pass" since nobody can reference it anyway
> 
> BTW this is a workaround, I'm wondering if the root cause out-dated python
> version ?
> sqlite3 should be available after python 2.5
> https://docs.python.org/2/library/sqlite3.html
> If so we should try to make trybot use newer python ?

Done.

This test is running on the device, and sqlite3 is not present. We would have to
update python packages to make this available.

[cylee1, 2015-05-04 20:40:57]

On 2015/05/04 19:48:00, achuithb wrote:
> Cheng-Yu, I looked at one of the test runs and disabled all the failing tests.
I
> also ran this locally and disabled everything that failed.
> 
> I think it's ok to be somewhat aggressive about disabling tests - we still
have
> to go through the tests and re-enable them one by one after figuring out why
> they are failing. But I think the first priority is to get the bot green.

OK just one ask: could you separate perf failure from unit test failure?
I hope crbug/483212 is to address unit test only (as the description says!)

The other reason is that telemetry is somehow unstable on CrOS, so even a perf
test which runs successfully most of the time can fail from time to time
randomly.
I'm not sure what's the case for v8.py . If you see it constantly fails, please
file a separate bug for it, just like what we did for other failing perf tests.
If it just fails for once, I'd rather keep it in the watch list and disable it
later when we see it fails frequently.

[achuithb, 2015-05-05 00:12:05]

On 2015/05/04 20:40:57, cylee1 wrote:
> On 2015/05/04 19:48:00, achuithb wrote:
> > Cheng-Yu, I looked at one of the test runs and disabled all the failing
tests.
> I
> > also ran this locally and disabled everything that failed.
> > 
> > I think it's ok to be somewhat aggressive about disabling tests - we still
> have
> > to go through the tests and re-enable them one by one after figuring out why
> > they are failing. But I think the first priority is to get the bot green.
> 
> OK just one ask: could you separate perf failure from unit test failure?
> I hope crbug/483212 is to address unit test only (as the description says!)
> 

The bug is really to get the bots green. The bots run both telemetry unit tests
and perf unit tests.

> The other reason is that telemetry is somehow unstable on CrOS, so even a perf
> test which runs successfully most of the time can fail from time to time
> randomly.

There is no value in a flaky test right now. These bots are completely ignored
because they have been red for months. They need to be made green and stay green
if the cros ui gardeners are to take any interest in them. It's better to get
coverage for the passing 800+ tests rather than worry about some 10-20 tests
that are flaky.

> I'm not sure what's the case for v8.py . If you see it constantly fails,
please
> file a separate bug for it, just like what we did for other failing perf
tests.
> If it just fails for once, I'd rather keep it in the watch list and disable it
> later when we see it fails frequently.

I disagree with this strategy. Let's get the bots green first and then enable
tests and monitor them.

[achuithb, 2015-05-05 00:14:04]

On 2015/05/04 20:40:57, cylee1 wrote:
> On 2015/05/04 19:48:00, achuithb wrote:
> > Cheng-Yu, I looked at one of the test runs and disabled all the failing
tests.
> I
> > also ran this locally and disabled everything that failed.
> > 
> > I think it's ok to be somewhat aggressive about disabling tests - we still
> have
> > to go through the tests and re-enable them one by one after figuring out why
> > they are failing. But I think the first priority is to get the bot green.
> 
> OK just one ask: could you separate perf failure from unit test failure?
> I hope crbug/483212 is to address unit test only (as the description says!)

I've created a separate CL: https://codereview.chromium.org/1123843002/

[cylee1, 2015-05-05 06:10:31]

The CQ bit was checked by cylee@chromium.org

[cylee1, 2015-05-05 06:10:32]

lgtm

[commit-bot: I haz the power, 2015-05-05 06:11:17]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1115383002/140001

[commit-bot: I haz the power, 2015-05-05 06:11:21]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-05-05 06:11:22]

No LGTM from a valid reviewer yet. Only full committers are accepted.
Even if an LGTM may have been provided, it was from a non-committer,
_not_ a full super star committer.
See http://www.chromium.org/getting-involved/become-a-committer
Note that this has nothing to do with OWNERS files.

[achuithb, 2015-05-05 07:02:22]

Toni - could you please take a look?

[tbarzic, 2015-05-05 13:26:43]

On 2015/05/05 07:02:22, achuithb wrote:
> Toni - could you please take a look?

lgtm

[achuithb, 2015-05-05 15:25:16]

The CQ bit was checked by achuith@chromium.org

[commit-bot: I haz the power, 2015-05-05 15:26:26]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1115383002/140001

[commit-bot: I haz the power, 2015-05-05 15:33:40]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-05-05 15:33:42]

Try jobs failed on following builders:
  mac_chromium_compile_dbg_ng on tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_comp...)
  mac_chromium_rel_ng on tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

[achuithb, 2015-05-05 17:23:54]

The CQ bit was checked by achuith@chromium.org

[achuithb, 2015-05-05 17:23:54]

The patchset sent to the CQ was uploaded after l-g-t-m from cylee@chromium.org,
tbarzic@chromium.org
Link to the patchset: https://codereview.chromium.org/1115383002/#ps50011
(title: "rebase")

[commit-bot: I haz the power, 2015-05-05 17:25:20]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1115383002/50011

[commit-bot: I haz the power, 2015-05-05 17:52:46]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-05-05 17:52:48]

Try jobs failed on following builders:
  win_perf_bisect on tryserver.chromium.perf (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.perf/builders/win_perf_bisect/...)

[achuithb, 2015-05-05 19:11:49]

The CQ bit was checked by achuith@chromium.org

[commit-bot: I haz the power, 2015-05-05 19:13:30]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1115383002/50011

[commit-bot: I haz the power, 2015-05-05 19:22:51]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-05-05 19:22:52]

Try jobs failed on following builders:
  linux_perf_bisect on tryserver.chromium.perf (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisec...)

[achuithb, 2015-05-05 19:30:25]

The CQ bit was checked by achuith@chromium.org

[achuithb, 2015-05-05 19:30:26]

The patchset sent to the CQ was uploaded after l-g-t-m from cylee@chromium.org,
tbarzic@chromium.org
Link to the patchset: https://codereview.chromium.org/1115383002/#ps170001
(title: "rebase")

[commit-bot: I haz the power, 2015-05-05 19:31:04]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1115383002/170001

[commit-bot: I haz the power, 2015-05-05 19:44:22]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-05-05 19:44:23]

Try jobs failed on following builders:
  linux_perf_bisect on tryserver.chromium.perf (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisec...)

[achuithb, 2015-05-05 20:21:22]

The CQ bit was checked by achuith@chromium.org

[achuithb, 2015-05-05 20:21:23]

The patchset sent to the CQ was uploaded after l-g-t-m from cylee@chromium.org,
tbarzic@chromium.org
Link to the patchset: https://codereview.chromium.org/1115383002/#ps190001
(title: "revert devtools_client_backend_unittest.py")

[commit-bot: I haz the power, 2015-05-05 20:23:03]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1115383002/190001

[achuithb, 2015-05-05 20:29:54]

The CQ bit was unchecked by achuith@chromium.org

[commit-bot: I haz the power, 2015-05-05 20:31:51]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-05-05 20:31:53]

Try jobs failed on following builders:
  linux_perf_bisect on tryserver.chromium.perf (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.perf/builders/linux_perf_bisec...)

[achuithb, 2015-05-05 22:03:26]

Patchset #11 (id:190001) has been deleted

[achuithb, 2015-05-05 22:03:37]

The CQ bit was checked by achuith@chromium.org

[commit-bot: I haz the power, 2015-05-05 22:05:23]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1115383002/170001

[commit-bot: I haz the power, 2015-05-05 22:12:08]

Committed patchset #10 (id:170001)

[commit-bot: I haz the power, 2015-05-05 22:13:12]

Patchset 10 (id:??) landed as
https://crrev.com/dfca3a35a4689831793b1ac8b0bd5f6727d2bcd6
Cr-Commit-Position: refs/heads/master@{#328416}