Chromium Code Reviews
|
|
Chromium Code Reviews
Issue
235573010:
Add telemetry tests to Android tester. (Closed)
|
Created:
6 years, 8 months ago by navabi Modified:
6 years, 8 months ago CC:
chromium-reviews, klundberg+watch_chromium.org, bulach+watch_chromium.org, yfriedman+watch_chromium.org, ilevy-cc_chromium.org Base URL:
svn://svn.chromium.org/chrome/trunk/src Visibility:
Public. |
DescriptionAdd telemetry tests to Android tester.
BUG=337829
Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=265800
Patch Set 1 #
Total comments: 3
Patch Set 2 : Split suites and put in alphabetic order. #
Total comments: 1
Patch Set 3 : Install apk before running the telemetry tests. #Patch Set 4 : Install ChromeShell not ContentShell for telemetry tests. #Patch Set 5 : Run telemetry tests on single device. #Patch Set 6 : Enable perf for now. #Patch Set 7 : Fix presubmit error by running script and letting it fail. #Messages
Total messages: 37 (0 generated)
Woot! lgtm To confirm that I understand what this config does, will it add it to the main waterfall + cq without master restarts? https://codereview.chromium.org/235573010/diff/1/build/android/buildbot/bb_de... File build/android/buildbot/bb_device_steps.py (right): https://codereview.chromium.org/235573010/diff/1/build/android/buildbot/bb_de... build/android/buildbot/bb_device_steps.py:80: VALID_TESTS = set(['chromedriver', 'gpu', 'ui', 'unit', 'telemetry_tests', Looks like these were alphabetical. Should we maintain that? https://codereview.chromium.org/235573010/diff/1/build/android/buildbot/bb_de... build/android/buildbot/bb_device_steps.py:500: ('telemetry_tests', RunTelemetryTests), ditto
https://codereview.chromium.org/235573010/diff/1/build/android/buildbot/bb_de... File build/android/buildbot/bb_device_steps.py (right): https://codereview.chromium.org/235573010/diff/1/build/android/buildbot/bb_de... build/android/buildbot/bb_device_steps.py:80: VALID_TESTS = set(['chromedriver', 'gpu', 'ui', 'unit', 'telemetry_tests', On 2014/04/12 01:37:29, tonyg wrote: > Looks like these were alphabetical. Should we maintain that? Yes we should. Will fix. And I realized I should split them into telemetry_unittests and telemetry_perf_unittests so that the bb_device_steps.py can run them individually with the test filter.
On 2014/04/12 01:37:29, tonyg wrote: > Woot! lgtm > > To confirm that I understand what this config does, will it add it to the main > waterfall + cq without master restarts? Yes for main waterfall, but not on the try bots triggered by the cq. I will add it to those too (i.e. for try-tests-dbg in bb_run_bot.py). Will not require a master restart.
https://codereview.chromium.org/235573010/diff/20001/build/android/buildbot/b... File build/android/buildbot/bb_run_bot.py (right): https://codereview.chromium.org/235573010/diff/20001/build/android/buildbot/b... build/android/buildbot/bb_run_bot.py:203: ('try-tests', 'main-tests'), I lied. This will also run on the try bots triggered by the CQ because of line 203. It says copy whatever we do for the main tester for testers.
The CQ bit was checked by navabi@google.com
CQ is trying da patch. Follow status at https://chromium-status.appspot.com/cq/navabi@google.com/235573010/20001
On 2014/04/12 16:44:03, I haz the power (commit-bot) wrote: > CQ is trying da patch. Follow status at > https://chromium-status.appspot.com/cq/navabi@google.com/235573010/20001 We should see the telemetry tests on the android_dbg_triggered tests. It will show us which tests from the telemetry suite fail, so that we can disable them, then trigger another android try job and have it land.
The CQ bit was unchecked by commit-bot@chromium.org
Retried try job too often on chromium_presubmit for step(s) presubmit http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=chromium_p...
On 2014/04/12 18:07:12, I haz the power (commit-bot) wrote: > Retried try job too often on chromium_presubmit for step(s) presubmit > http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=chromium_p... Putting the tests in alphabetical order means that telemetry tests would run before the ui unittests which means that ChromeShell is not installed. We can still just call the run_tests scripts, but we need to install the APK before we do.
On 2014/04/13 17:22:31, navabi wrote: > On 2014/04/12 18:07:12, I haz the power (commit-bot) wrote: > > Retried try job too often on chromium_presubmit for step(s) presubmit > > > http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=chromium_p... > > Putting the tests in alphabetical order means that telemetry tests would run > before the ui unittests which means that ChromeShell is not installed. We can > still just call the run_tests scripts, but we need to install the APK before we > do. It looks as though the scripts do not work on multiple devices. I have also tried locally testing on a single device and I get a CloudStorageError when the script is trying to determine the list of available browsers. Please refer the bug for the details.
On 2014/04/14 14:44:03, navabi wrote: > On 2014/04/13 17:22:31, navabi wrote: > > On 2014/04/12 18:07:12, I haz the power (commit-bot) wrote: > > > Retried try job too often on chromium_presubmit for step(s) presubmit > > > > > > http://build.chromium.org/p/tryserver.chromium/buildstatus?builder=chromium_p... > > > > Putting the tests in alphabetical order means that telemetry tests would run > > before the ui unittests which means that ChromeShell is not installed. We can > > still just call the run_tests scripts, but we need to install the APK before > we > > do. > > It looks as though the scripts do not work on multiple devices. I have also > tried locally testing on a single device and I get a CloudStorageError when the > script is trying to determine the list of available browsers. Please refer the > bug for the details. PTAL. The CloudStorageError is not a problem. The bots have the appropriate permissions. The only problem is that telemetry tests need a device to run on, otherwise there is an error (see android_dbg_triggered_tests try job failure in previous patches). This CL fixes that by running on one attached device.
telemetry_unittests took 22mins. We can't enable the test suite if it takes that long. Perhaps timeouts need to be decreased. I've fired off a new try job to see if this is consistent.
lgtm
The CQ bit was unchecked by commit-bot@chromium.org
Try jobs failed on following builders: tryserver.chromium on chromium_presubmit
Those failing tests are disabled now and this looks good to go now. Just needs an owner (and maybe enough time for the lkgr to pick up my disabled test patch).
On 2014/04/23 02:16:40, tonyg wrote: > Those failing tests are disabled now and this looks good to go now. Just needs > an owner (and maybe enough time for the lkgr to pick up my disabled test patch). I think it already picked up your disabled test patch, because it looks like the try bot picked it up and disabled the failing MeasurementUnitTest http://build.chromium.org/p/tryserver.chromium/builders/android_dbg_triggered... Adding yfriedman for OWNERS lgtm.
On 2014/04/23 15:18:46, navabi wrote: rs lgtm
The CQ bit was checked by navabi@google.com
CQ is trying da patch. Follow status at https://chromium-status.appspot.com/cq/navabi@google.com/235573010/100001
The CQ bit was unchecked by commit-bot@chromium.org
Try jobs failed on following builders: tryserver.chromium on chromium_presubmit
lgtm
The CQ bit was unchecked by commit-bot@chromium.org
Try jobs failed on following builders: tryserver.chromium on linux_chromium_chromeos_rel
The CQ bit was checked by tonyg@chromium.org
On 2014/04/23 22:27:08, I haz the power (commit-bot) wrote: > Try jobs failed on following builders: > tryserver.chromium on linux_chromium_chromeos_rel browser_test flake
CQ is trying da patch. Follow status at https://chromium-status.appspot.com/cq/navabi@google.com/235573010/120001
The CQ bit was unchecked by commit-bot@chromium.org
Try jobs failed on following builders: tryserver.chromium on win_chromium_rel
On 2014/04/23 23:04:35, I haz the power (commit-bot) wrote: > Try jobs failed on following builders: > tryserver.chromium on win_chromium_rel More flake :/
The CQ bit was checked by tonyg@chromium.org
CQ is trying da patch. Follow status at https://chromium-status.appspot.com/cq/navabi@google.com/235573010/120001
Message was sent while issue was closed.
Change committed as 265800 |
