Roll out framework change & update gpu tests to conform to the API changes

Roll out framework change & update gpu tests to conform to the API changes

The changes include in this roll:
https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/2242f1ab32bac4e1884b..42d9cffaa797712c58d7b5fb

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC-Failures-due-to-DEPS-rolls

BUG=636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2663813003
Cr-Commit-Position: refs/heads/master@{#452269}
Committed: https://chromium.googlesource.com/chromium/src/+/ecdf404b0186581b08828eb1db2caa3b88b336f9

[nednguyen, 2017-01-30 21:41:44]

Description was changed from

==========
[Telemetry] Disable
browser_test_runner_unittest.TestLoadAllTestModules.testLoadAllTestsInModule

BUG=chromium:636153
==========

to

==========
[Telemetry] Disable
browser_test_runner_unittest.TestLoadAllTestModules.testLoadAllTestsInModule

BUG=chromium:636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

[nednguyen, 2017-01-31 13:56:55]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-01-31 13:57:02]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[nednguyen, 2017-01-31 13:57:24]

Patchset #1 (id:1) has been deleted

[nednguyen, 2017-01-31 13:57:42]

Patchset #1 (id:20001) has been deleted

[nednguyen, 2017-01-31 13:59:30]

Description was changed from

==========
[Telemetry] Disable
browser_test_runner_unittest.TestLoadAllTestModules.testLoadAllTestsInModule

BUG=chromium:636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

to

==========
Manually roll src/third_party/catapult/6bc0354c3..34e664ad6a6e3 (1 commit).

https://chromium.googlesource.com/external/github.com/catapult-project/catapu...




Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC...


TBR=catapult-sheriff@chromium.org

BUG=chromium:636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel:master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

[nednguyen, 2017-01-31 14:01:26]

Description was changed from

==========
Manually roll src/third_party/catapult/6bc0354c3..34e664ad6a6e3 (1 commit).

https://chromium.googlesource.com/external/github.com/catapult-project/catapu...




Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC...


TBR=catapult-sheriff@chromium.org

BUG=chromium:636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel:master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

to

==========
Manually roll src/third_party/catapult/6bc0354c3..34e664ad6a6e3 (1 commit).

https://chromium.googlesource.com/external/github.com/catapult-project/catapu...

git log 6bc0354c3..34e664ad6a6 --date=short --no-merges --format='%ad %ae %s'
2017-01-31 nednguyen@google.com [Telemetry] Migrate browser_test_runner to use
typ as the test runner


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC...

This roll also includes change to update gpu tests to conform to changes
to browser_test_runner harness. (details in crbug.com/636153)

TBR=catapult-sheriff@chromium.org, kbr@chromium.org

BUG=636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel:master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

[nednguyen, 2017-01-31 14:01:33]

The CQ bit was unchecked by nednguyen@google.com

[nednguyen, 2017-01-31 14:01:33]

The CQ bit was checked by nednguyen@google.com

[commit-bot: I haz the power, 2017-01-31 14:01:34]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-31 14:01:35]

Dry run: Your CL can not be processed by CQ because of:
 * Failed to parse additional trybots

[nednguyen, 2017-01-31 14:01:38]

The CQ bit was unchecked by nednguyen@google.com

[nednguyen, 2017-01-31 14:01:41]

The CQ bit was checked by nednguyen@google.com

[commit-bot: I haz the power, 2017-01-31 14:01:54]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-31 14:01:55]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-31 14:01:56]

Your CL can not be processed by CQ because of:
 * Failed to parse additional trybots

[nednguyen, 2017-01-31 15:03:01]

The CQ bit was checked by nednguyen@google.com

[commit-bot: I haz the power, 2017-01-31 15:03:17]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-31 15:03:18]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-31 15:03:19]

Your CL can not be processed by CQ because of:
 * Failed to parse additional trybots

[nednguyen, 2017-01-31 15:30:38]

Description was changed from

==========
Manually roll src/third_party/catapult/6bc0354c3..34e664ad6a6e3 (1 commit).

https://chromium.googlesource.com/external/github.com/catapult-project/catapu...

git log 6bc0354c3..34e664ad6a6 --date=short --no-merges --format='%ad %ae %s'
2017-01-31 nednguyen@google.com [Telemetry] Migrate browser_test_runner to use
typ as the test runner


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC...

This roll also includes change to update gpu tests to conform to changes
to browser_test_runner harness. (details in crbug.com/636153)

TBR=catapult-sheriff@chromium.org, kbr@chromium.org

BUG=636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel:master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

to

==========
Manually roll src/third_party/catapult/6bc0354c3..34e664ad6a6e3 (1 commit).

https://chromium.googlesource.com/external/github.com/catapult-project/catapu...

git log 6bc0354c3..34e664ad6a6 --date=short --no-merges --format='%ad %ae %s'
2017-01-31 nednguyen@google.com [Telemetry] Migrate browser_test_runner to use
typ as the test runner


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC...

This roll also includes change to update gpu tests to conform to changes
to browser_test_runner harness. (details in crbug.com/636153)

TBR=catapult-sheriff@chromium.org, kbr@chromium.org

BUG=636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

[nednguyen, 2017-01-31 15:30:42]

The CQ bit was checked by nednguyen@google.com

[commit-bot: I haz the power, 2017-01-31 15:31:01]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-31 15:31:03]

There were warnings when CQ was processing your CL:
 * CQ is not running the android_optional_gpu_tests_rel trybot, per your
CQ_INCLUDE_TRYBOTS flag request. That bot was already specified

[commit-bot: I haz the power, 2017-01-31 15:32:28]

There were warnings when CQ was processing your CL:
 * CQ is not running the android_optional_gpu_tests_rel trybot, per your
CQ_INCLUDE_TRYBOTS flag request. That bot was already specified

[commit-bot: I haz the power, 2017-01-31 16:01:40]

There were warnings when CQ was processing your CL:
 * CQ is not running the android_optional_gpu_tests_rel trybot, per your
CQ_INCLUDE_TRYBOTS flag request. That bot was already specified

[nednguyen, 2017-01-31 16:03:06]

Description was changed from

==========
Manually roll src/third_party/catapult/6bc0354c3..34e664ad6a6e3 (1 commit).

https://chromium.googlesource.com/external/github.com/catapult-project/catapu...

git log 6bc0354c3..34e664ad6a6 --date=short --no-merges --format='%ad %ae %s'
2017-01-31 nednguyen@google.com [Telemetry] Migrate browser_test_runner to use
typ as the test runner


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC...

This roll also includes change to update gpu tests to conform to changes
to browser_test_runner harness. (details in crbug.com/636153)

TBR=catapult-sheriff@chromium.org, kbr@chromium.org

BUG=636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

to

==========
Manually roll src/third_party/catapult/6bc0354c3..34e664ad6a6e3 (1 commit).

https://chromium.googlesource.com/external/github.com/catapult-project/catapu...

git log 6bc0354c3..34e664ad6a6 --date=short --no-merges --format='%ad %ae %s'
2017-01-31 nednguyen@google.com [Telemetry] Migrate browser_test_runner to use
typ as the test runner


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC...

This roll also includes change to update gpu tests to conform to changes
to browser_test_runner harness. (details in crbug.com/636153)

TBR=catapult-sheriff@chromium.org, kbr@chromium.org

BUG=636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

[commit-bot: I haz the power, 2017-01-31 16:52:56]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-31 16:52:57]

Try jobs failed on following builders:
  mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

[nednguyen, 2017-01-31 17:05:20]

On 2017/01/31 16:52:57, commit-bot: I haz the power wrote:
> Try jobs failed on following builders:
>   mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED,
>
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

Sadly that this is causing timeout on webgl_conformance. This is due to the fact
that typ invokes teardownClass after every test run, which causes the browser to
be restarted every time between tests.

+Dirk: can we modify typ to not invoke teardownClass at least for the case of
single process? (browser_test_runner is currently using typ in single process
mode by setting runner.jobs = 1)

[nednguyen, 2017-01-31 17:05:41]

nednguyen@google.com changed reviewers:
+ dpranke@chromium.org

[nednguyen, 2017-02-06 19:54:17]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-02-06 19:55:02]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-06 19:57:30]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-06 19:57:31]

Dry run: Try jobs failed on following builders:
  ios-device on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-device/builds...)
  ios-device-xcode-clang on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-device-xcode-...)
  ios-simulator on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator/bui...)
  ios-simulator-xcode-clang on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/ios-simulator-xco...)

[nednguyen, 2017-02-06 22:08:35]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-02-06 22:09:41]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[nednguyen, 2017-02-06 23:37:12]

Description was changed from

==========
Manually roll src/third_party/catapult/6bc0354c3..34e664ad6a6e3 (1 commit).

https://chromium.googlesource.com/external/github.com/catapult-project/catapu...

git log 6bc0354c3..34e664ad6a6 --date=short --no-merges --format='%ad %ae %s'
2017-01-31 nednguyen@google.com [Telemetry] Migrate browser_test_runner to use
typ as the test runner


Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC...

This roll also includes change to update gpu tests to conform to changes
to browser_test_runner harness. (details in crbug.com/636153)

TBR=catapult-sheriff@chromium.org, kbr@chromium.org

BUG=636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

to

==========
Test typ framework change (Not for landing)

COMMIT=false
BUG=636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

[nednguyen, 2017-02-06 23:37:12]

nednguyen@google.com changed reviewers:
+ kbr@chromium.org

[commit-bot: I haz the power, 2017-02-06 23:56:33]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-06 23:56:34]

Dry run: Try jobs failed on following builders:
  win_optional_gpu_tests_rel on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_optional_gpu_...)

[nednguyen, 2017-02-07 01:16:17]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-02-07 01:17:16]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-07 02:03:32]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-07 02:03:34]

Dry run: Try jobs failed on following builders:
  linux_optional_gpu_tests_rel on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_optional_...)

[nednguyen, 2017-02-07 11:28:35]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-02-07 11:28:48]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-07 12:30:41]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-07 12:30:42]

Dry run: This issue passed the CQ dry run.

[nednguyen, 2017-02-07 12:44:46]

On 2017/01/31 17:05:20, nednguyen wrote:
> On 2017/01/31 16:52:57, commit-bot: I haz the power wrote:
> > Try jobs failed on following builders:
> >   mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED,
> >
>
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)
> 
> Sadly that this is causing timeout on webgl_conformance. This is due to the
fact
> that typ invokes teardownClass after every test run, which causes the browser
to
> be restarted every time between tests.
> 
> +Dirk: can we modify typ to not invoke teardownClass at least for the case of
> single process? (browser_test_runner is currently using typ in single process
> mode by setting runner.jobs = 1)

All gpu tests are passing now! Though I am not sure whether we should fix the
"TearDownProcess fails silently" first before landing the change. What do you
think, Ken?

[Ken Russell (switch to Gerrit), 2017-02-07 18:37:14]

On 2017/02/07 12:44:46, nednguyen wrote:
> On 2017/01/31 17:05:20, nednguyen wrote:
> > On 2017/01/31 16:52:57, commit-bot: I haz the power wrote:
> > > Try jobs failed on following builders:
> > >   mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED,
> > >
> >
>
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)
> > 
> > Sadly that this is causing timeout on webgl_conformance. This is due to the
> fact
> > that typ invokes teardownClass after every test run, which causes the
browser
> to
> > be restarted every time between tests.
> > 
> > +Dirk: can we modify typ to not invoke teardownClass at least for the case
of
> > single process? (browser_test_runner is currently using typ in single
process
> > mode by setting runner.jobs = 1)
> 
> All gpu tests are passing now! Though I am not sure whether we should fix the
> "TearDownProcess fails silently" first before landing the change. What do you
> think, Ken?

That's great news, Ned!

I don't actually see that silent failure in your tryjobs. Where is it?

There's one remaining problem: the time it took to run each test isn't in the
JSON output. This will break the re-sharding pipeline (using
src/content/test/gpu/gather_swarming_json_results.py). Here's a sample
invocation of this script:

./content/test/gpu/gather_swarming_json_results.py --waterfall
tryserver.chromium.linux --bot linux_optional_gpu_tests_rel --build 5836 --step
webgl2_conformance_tests --output webgl2-times.json

I can update the script to select the right entries out of the trie and merge it
properly (it's broken after the typ switchover), but we need the per-test run
times in the JSON file. How can we get them?

[nednguyen, 2017-02-07 18:44:52]

On 2017/02/07 18:37:14, Ken Russell wrote:
> On 2017/02/07 12:44:46, nednguyen wrote:
> > On 2017/01/31 17:05:20, nednguyen wrote:
> > > On 2017/01/31 16:52:57, commit-bot: I haz the power wrote:
> > > > Try jobs failed on following builders:
> > > >   mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED,
> > > >
> > >
> >
>
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)
> > > 
> > > Sadly that this is causing timeout on webgl_conformance. This is due to
the
> > fact
> > > that typ invokes teardownClass after every test run, which causes the
> browser
> > to
> > > be restarted every time between tests.
> > > 
> > > +Dirk: can we modify typ to not invoke teardownClass at least for the case
> of
> > > single process? (browser_test_runner is currently using typ in single
> process
> > > mode by setting runner.jobs = 1)
> > 
> > All gpu tests are passing now! Though I am not sure whether we should fix
the
> > "TearDownProcess fails silently" first before landing the change. What do
you
> > think, Ken?
> 
> That's great news, Ned!
> 
> I don't actually see that silent failure in your tryjobs. Where is it?

https://build.chromium.org/p/tryserver.chromium.win/builders/win_optional_gpu...

> 
> There's one remaining problem: the time it took to run each test isn't in the
> JSON output. This will break the re-sharding pipeline (using
> src/content/test/gpu/gather_swarming_json_results.py). Here's a sample
> invocation of this script:
> 
> ./content/test/gpu/gather_swarming_json_results.py --waterfall
> tryserver.chromium.linux --bot linux_optional_gpu_tests_rel --build 5836
--step
> webgl2_conformance_tests --output webgl2-times.json
> 
> I can update the script to select the right entries out of the trie and merge
it
> properly (it's broken after the typ switchover), but we need the per-test run
> times in the JSON file. How can we get them?

+Dirk: typ doesn't support outputing per-test run time yet.

[Dirk Pranke, 2017-02-07 22:05:16]

On 2017/02/07 18:44:52, nednguyen wrote:
> +Dirk: typ doesn't support outputing per-test run time yet.

We've talked about this a couple of times in the past. The standard
"JSON Results File Format" doesn't actually contain a test duration field.

You can also get typ to output a trace file today that does contain the
per-test timing.

Or, it would be easy enough to add the field to the JSON format.

[Ken Russell (switch to Gerrit), 2017-02-08 03:20:34]

On 2017/02/07 22:05:16, Dirk Pranke wrote:
> On 2017/02/07 18:44:52, nednguyen wrote:
> > +Dirk: typ doesn't support outputing per-test run time yet.
> 
> We've talked about this a couple of times in the past. The standard
> "JSON Results File Format" doesn't actually contain a test duration field.
> 
> You can also get typ to output a trace file today that does contain the
> per-test timing.
> 
> Or, it would be easy enough to add the field to the JSON format.

Thanks for the feedback. I think we should add the field to the JSON format.
Filed http://crbug.com/689788 about this and blocked http://crbug.com/636153 on
it.

[nednguyen, 2017-02-16 22:15:33]

Description was changed from

==========
Test typ framework change (Not for landing)

COMMIT=false
BUG=636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

to

==========
Roll out framework change & update gpu tests to conform to the API changes

The changes include in this roll:
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC...

BUG=636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

[nednguyen, 2017-02-16 22:19:22]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[nednguyen, 2017-02-16 22:19:36]

On 2017/02/08 03:20:34, Ken Russell wrote:
> On 2017/02/07 22:05:16, Dirk Pranke wrote:
> > On 2017/02/07 18:44:52, nednguyen wrote:
> > > +Dirk: typ doesn't support outputing per-test run time yet.
> > 
> > We've talked about this a couple of times in the past. The standard
> > "JSON Results File Format" doesn't actually contain a test duration field.
> > 
> > You can also get typ to output a trace file today that does contain the
> > per-test timing.
> > 
> > Or, it would be easy enough to add the field to the JSON format.
> 
> Thanks for the feedback. I think we should add the field to the JSON format.
> Filed http://crbug.com/689788 about this and blocked http://crbug.com/636153
on
> it.

This is now ready to land. Triggering a dry run.

[commit-bot: I haz the power, 2017-02-16 22:20:29]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-16 22:55:21]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-16 22:55:22]

Dry run: Try jobs failed on following builders:
  android_optional_gpu_tests_rel on master.tryserver.chromium.android
(JOB_FAILED,
https://build.chromium.org/p/tryserver.chromium.android/builders/android_opti...)
  win_optional_gpu_tests_rel on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_optional_gpu_...)

[Ken Russell (switch to Gerrit), 2017-02-16 23:36:12]

On 2017/02/16 22:19:36, nednguyen wrote:
> On 2017/02/08 03:20:34, Ken Russell wrote:
> > On 2017/02/07 22:05:16, Dirk Pranke wrote:
> > > On 2017/02/07 18:44:52, nednguyen wrote:
> > > > +Dirk: typ doesn't support outputing per-test run time yet.
> > > 
> > > We've talked about this a couple of times in the past. The standard
> > > "JSON Results File Format" doesn't actually contain a test duration field.
> > > 
> > > You can also get typ to output a trace file today that does contain the
> > > per-test timing.
> > > 
> > > Or, it would be easy enough to add the field to the JSON format.
> > 
> > Thanks for the feedback. I think we should add the field to the JSON format.
> > Filed http://crbug.com/689788 about this and blocked http://crbug.com/636153
> on
> > it.
> 
> This is now ready to land. Triggering a dry run.

Looks like browser_test_runner's interface changed since the last time a dry run
was done?

Traceback (most recent call last):
  <module> at /b/s/w/iroBYPCt/content/test/gpu/run_gpu_integration_test.py:46
    sys.exit(main())
  main at /b/s/w/iroBYPCt/content/test/gpu/run_gpu_integration_test.py:31
    gpu_project_config.CONFIG, rest_args)
TypeError: Run() takes exactly 3 arguments (2 given)

[nednguyen, 2017-02-16 23:39:50]

On 2017/02/16 23:36:12, Ken Russell wrote:
> On 2017/02/16 22:19:36, nednguyen wrote:
> > On 2017/02/08 03:20:34, Ken Russell wrote:
> > > On 2017/02/07 22:05:16, Dirk Pranke wrote:
> > > > On 2017/02/07 18:44:52, nednguyen wrote:
> > > > > +Dirk: typ doesn't support outputing per-test run time yet.
> > > > 
> > > > We've talked about this a couple of times in the past. The standard
> > > > "JSON Results File Format" doesn't actually contain a test duration
field.
> > > > 
> > > > You can also get typ to output a trace file today that does contain the
> > > > per-test timing.
> > > > 
> > > > Or, it would be easy enough to add the field to the JSON format.
> > > 
> > > Thanks for the feedback. I think we should add the field to the JSON
format.
> > > Filed http://crbug.com/689788 about this and blocked
http://crbug.com/636153
> > on
> > > it.
> > 
> > This is now ready to land. Triggering a dry run.
> 
> Looks like browser_test_runner's interface changed since the last time a dry
run
> was done?
> 
> Traceback (most recent call last):
>   <module> at /b/s/w/iroBYPCt/content/test/gpu/run_gpu_integration_test.py:46
>     sys.exit(main())
>   main at /b/s/w/iroBYPCt/content/test/gpu/run_gpu_integration_test.py:31
>     gpu_project_config.CONFIG, rest_args)
> TypeError: Run() takes exactly 3 arguments (2 given)

Not sure what's going on with the Android recipe, the desktop bot are running
fine. Lemme retry it

[nednguyen, 2017-02-16 23:39:59]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-02-16 23:41:54]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[nednguyen, 2017-02-16 23:43:59]

On 2017/02/16 23:39:50, nednguyen wrote:
> On 2017/02/16 23:36:12, Ken Russell wrote:
> > On 2017/02/16 22:19:36, nednguyen wrote:
> > > On 2017/02/08 03:20:34, Ken Russell wrote:
> > > > On 2017/02/07 22:05:16, Dirk Pranke wrote:
> > > > > On 2017/02/07 18:44:52, nednguyen wrote:
> > > > > > +Dirk: typ doesn't support outputing per-test run time yet.
> > > > > 
> > > > > We've talked about this a couple of times in the past. The standard
> > > > > "JSON Results File Format" doesn't actually contain a test duration
> field.
> > > > > 
> > > > > You can also get typ to output a trace file today that does contain
the
> > > > > per-test timing.
> > > > > 
> > > > > Or, it would be easy enough to add the field to the JSON format.
> > > > 
> > > > Thanks for the feedback. I think we should add the field to the JSON
> format.
> > > > Filed http://crbug.com/689788 about this and blocked
> http://crbug.com/636153
> > > on
> > > > it.
> > > 
> > > This is now ready to land. Triggering a dry run.
> > 
> > Looks like browser_test_runner's interface changed since the last time a dry
> run
> > was done?
> > 
> > Traceback (most recent call last):
> >   <module> at
/b/s/w/iroBYPCt/content/test/gpu/run_gpu_integration_test.py:46
> >     sys.exit(main())
> >   main at /b/s/w/iroBYPCt/content/test/gpu/run_gpu_integration_test.py:31
> >     gpu_project_config.CONFIG, rest_args)
> > TypeError: Run() takes exactly 3 arguments (2 given)
> 
> Not sure what's going on with the Android recipe, the desktop bot are running
> fine. Lemme retry it

OOps, I also just realize that Dirk forgot to roll out typ change in catapult/

[Ken Russell (switch to Gerrit), 2017-02-17 00:04:08]

On 2017/02/16 23:43:59, nednguyen wrote:
> On 2017/02/16 23:39:50, nednguyen wrote:
> > On 2017/02/16 23:36:12, Ken Russell wrote:
> > > On 2017/02/16 22:19:36, nednguyen wrote:
> > > > On 2017/02/08 03:20:34, Ken Russell wrote:
> > > > > On 2017/02/07 22:05:16, Dirk Pranke wrote:
> > > > > > On 2017/02/07 18:44:52, nednguyen wrote:
> > > > > > > +Dirk: typ doesn't support outputing per-test run time yet.
> > > > > > 
> > > > > > We've talked about this a couple of times in the past. The standard
> > > > > > "JSON Results File Format" doesn't actually contain a test duration
> > field.
> > > > > > 
> > > > > > You can also get typ to output a trace file today that does contain
> the
> > > > > > per-test timing.
> > > > > > 
> > > > > > Or, it would be easy enough to add the field to the JSON format.
> > > > > 
> > > > > Thanks for the feedback. I think we should add the field to the JSON
> > format.
> > > > > Filed http://crbug.com/689788 about this and blocked
> > http://crbug.com/636153
> > > > on
> > > > > it.
> > > > 
> > > > This is now ready to land. Triggering a dry run.
> > > 
> > > Looks like browser_test_runner's interface changed since the last time a
dry
> > run
> > > was done?
> > > 
> > > Traceback (most recent call last):
> > >   <module> at
> /b/s/w/iroBYPCt/content/test/gpu/run_gpu_integration_test.py:46
> > >     sys.exit(main())
> > >   main at /b/s/w/iroBYPCt/content/test/gpu/run_gpu_integration_test.py:31
> > >     gpu_project_config.CONFIG, rest_args)
> > > TypeError: Run() takes exactly 3 arguments (2 given)
> > 
> > Not sure what's going on with the Android recipe, the desktop bot are
running
> > fine. Lemme retry it
> 
> OOps, I also just realize that Dirk forgot to roll out typ change in catapult/

The stack trace I posted above was from the Mac Optional GPU trybot:
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu...

[commit-bot: I haz the power, 2017-02-17 00:11:01]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-17 00:11:02]

Dry run: Try jobs failed on following builders:
  android_optional_gpu_tests_rel on master.tryserver.chromium.android
(JOB_FAILED,
https://build.chromium.org/p/tryserver.chromium.android/builders/android_opti...)

[nednguyen, 2017-02-17 15:36:45]

Description was changed from

==========
Roll out framework change & update gpu tests to conform to the API changes

The changes include in this roll:
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC...

BUG=636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

to

==========
Roll out framework change & update gpu tests to conform to the API changes

The changes include in this roll:
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC...

BUG=636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

[nednguyen, 2017-02-17 15:36:52]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-02-17 15:37:05]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-17 16:10:24]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-17 16:10:25]

Dry run: Try jobs failed on following builders:
  linux_optional_gpu_tests_rel on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_optional_...)

[nednguyen, 2017-02-17 16:16:42]

On 2017/02/17 00:04:08, Ken Russell wrote:
> On 2017/02/16 23:43:59, nednguyen wrote:
> > On 2017/02/16 23:39:50, nednguyen wrote:
> > > On 2017/02/16 23:36:12, Ken Russell wrote:
> > > > On 2017/02/16 22:19:36, nednguyen wrote:
> > > > > On 2017/02/08 03:20:34, Ken Russell wrote:
> > > > > > On 2017/02/07 22:05:16, Dirk Pranke wrote:
> > > > > > > On 2017/02/07 18:44:52, nednguyen wrote:
> > > > > > > > +Dirk: typ doesn't support outputing per-test run time yet.
> > > > > > > 
> > > > > > > We've talked about this a couple of times in the past. The
standard
> > > > > > > "JSON Results File Format" doesn't actually contain a test
duration
> > > field.
> > > > > > > 
> > > > > > > You can also get typ to output a trace file today that does
contain
> > the
> > > > > > > per-test timing.
> > > > > > > 
> > > > > > > Or, it would be easy enough to add the field to the JSON format.
> > > > > > 
> > > > > > Thanks for the feedback. I think we should add the field to the JSON
> > > format.
> > > > > > Filed http://crbug.com/689788 about this and blocked
> > > http://crbug.com/636153
> > > > > on
> > > > > > it.
> > > > > 
> > > > > This is now ready to land. Triggering a dry run.
> > > > 
> > > > Looks like browser_test_runner's interface changed since the last time a
> dry
> > > run
> > > > was done?
> > > > 
> > > > Traceback (most recent call last):
> > > >   <module> at
> > /b/s/w/iroBYPCt/content/test/gpu/run_gpu_integration_test.py:46
> > > >     sys.exit(main())
> > > >   main at
/b/s/w/iroBYPCt/content/test/gpu/run_gpu_integration_test.py:31
> > > >     gpu_project_config.CONFIG, rest_args)
> > > > TypeError: Run() takes exactly 3 arguments (2 given)
> > > 
> > > Not sure what's going on with the Android recipe, the desktop bot are
> running
> > > fine. Lemme retry it
> > 
> > OOps, I also just realize that Dirk forgot to roll out typ change in
catapult/
> 
> The stack trace I posted above was from the Mac Optional GPU trybot:
>
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu...

Hmhh, this is still failing on extra gpu bots

[Ken Russell (switch to Gerrit), 2017-02-17 18:04:23]

On 2017/02/17 16:16:42, nednguyen wrote:
> On 2017/02/17 00:04:08, Ken Russell wrote:
> > On 2017/02/16 23:43:59, nednguyen wrote:
> > > On 2017/02/16 23:39:50, nednguyen wrote:
> > > > On 2017/02/16 23:36:12, Ken Russell wrote:
> > > > > On 2017/02/16 22:19:36, nednguyen wrote:
> > > > > > On 2017/02/08 03:20:34, Ken Russell wrote:
> > > > > > > On 2017/02/07 22:05:16, Dirk Pranke wrote:
> > > > > > > > On 2017/02/07 18:44:52, nednguyen wrote:
> > > > > > > > > +Dirk: typ doesn't support outputing per-test run time yet.
> > > > > > > > 
> > > > > > > > We've talked about this a couple of times in the past. The
> standard
> > > > > > > > "JSON Results File Format" doesn't actually contain a test
> duration
> > > > field.
> > > > > > > > 
> > > > > > > > You can also get typ to output a trace file today that does
> contain
> > > the
> > > > > > > > per-test timing.
> > > > > > > > 
> > > > > > > > Or, it would be easy enough to add the field to the JSON format.
> > > > > > > 
> > > > > > > Thanks for the feedback. I think we should add the field to the
JSON
> > > > format.
> > > > > > > Filed http://crbug.com/689788 about this and blocked
> > > > http://crbug.com/636153
> > > > > > on
> > > > > > > it.
> > > > > > 
> > > > > > This is now ready to land. Triggering a dry run.
> > > > > 
> > > > > Looks like browser_test_runner's interface changed since the last time
a
> > dry
> > > > run
> > > > > was done?
> > > > > 
> > > > > Traceback (most recent call last):
> > > > >   <module> at
> > > /b/s/w/iroBYPCt/content/test/gpu/run_gpu_integration_test.py:46
> > > > >     sys.exit(main())
> > > > >   main at
> /b/s/w/iroBYPCt/content/test/gpu/run_gpu_integration_test.py:31
> > > > >     gpu_project_config.CONFIG, rest_args)
> > > > > TypeError: Run() takes exactly 3 arguments (2 given)
> > > > 
> > > > Not sure what's going on with the Android recipe, the desktop bot are
> > running
> > > > fine. Lemme retry it
> > > 
> > > OOps, I also just realize that Dirk forgot to roll out typ change in
> catapult/
> > 
> > The stack trace I posted above was from the Mac Optional GPU trybot:
> >
>
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu...
> 
> Hmhh, this is still failing on extra gpu bots

Talking with hinoka@, something is stuck, and the fix for
http://crbug.com/693296 hasn't rolled out yet.
Updates will be posted on that bug. Working on getting it fixed.

[Ken Russell (switch to Gerrit), 2017-02-17 19:36:23]

On 2017/02/17 18:04:23, Ken Russell wrote:
> On 2017/02/17 16:16:42, nednguyen wrote:
> > On 2017/02/17 00:04:08, Ken Russell wrote:
> > > On 2017/02/16 23:43:59, nednguyen wrote:
> > > > On 2017/02/16 23:39:50, nednguyen wrote:
> > > > > On 2017/02/16 23:36:12, Ken Russell wrote:
> > > > > > On 2017/02/16 22:19:36, nednguyen wrote:
> > > > > > > On 2017/02/08 03:20:34, Ken Russell wrote:
> > > > > > > > On 2017/02/07 22:05:16, Dirk Pranke wrote:
> > > > > > > > > On 2017/02/07 18:44:52, nednguyen wrote:
> > > > > > > > > > +Dirk: typ doesn't support outputing per-test run time yet.
> > > > > > > > > 
> > > > > > > > > We've talked about this a couple of times in the past. The
> > standard
> > > > > > > > > "JSON Results File Format" doesn't actually contain a test
> > duration
> > > > > field.
> > > > > > > > > 
> > > > > > > > > You can also get typ to output a trace file today that does
> > contain
> > > > the
> > > > > > > > > per-test timing.
> > > > > > > > > 
> > > > > > > > > Or, it would be easy enough to add the field to the JSON
format.
> > > > > > > > 
> > > > > > > > Thanks for the feedback. I think we should add the field to the
> JSON
> > > > > format.
> > > > > > > > Filed http://crbug.com/689788 about this and blocked
> > > > > http://crbug.com/636153
> > > > > > > on
> > > > > > > > it.
> > > > > > > 
> > > > > > > This is now ready to land. Triggering a dry run.
> > > > > > 
> > > > > > Looks like browser_test_runner's interface changed since the last
time
> a
> > > dry
> > > > > run
> > > > > > was done?
> > > > > > 
> > > > > > Traceback (most recent call last):
> > > > > >   <module> at
> > > > /b/s/w/iroBYPCt/content/test/gpu/run_gpu_integration_test.py:46
> > > > > >     sys.exit(main())
> > > > > >   main at
> > /b/s/w/iroBYPCt/content/test/gpu/run_gpu_integration_test.py:31
> > > > > >     gpu_project_config.CONFIG, rest_args)
> > > > > > TypeError: Run() takes exactly 3 arguments (2 given)
> > > > > 
> > > > > Not sure what's going on with the Android recipe, the desktop bot are
> > > running
> > > > > fine. Lemme retry it
> > > > 
> > > > OOps, I also just realize that Dirk forgot to roll out typ change in
> > catapult/
> > > 
> > > The stack trace I posted above was from the Mac Optional GPU trybot:
> > >
> >
>
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu...
> > 
> > Hmhh, this is still failing on extra gpu bots
> 
> Talking with hinoka@, something is stuck, and the fix for
> http://crbug.com/693296 hasn't rolled out yet.
> Updates will be posted on that bug. Working on getting it fixed.

Recipes have been rolled. Retrying the optional tryservers.

[Ken Russell (switch to Gerrit), 2017-02-21 02:20:13]

The failures on linux_optional_gpu_tests_rel of:
WebglConformance_conformance2_textures_canvas_sub_rectangle_tex_2d_r11f_g11f_b10f_rgb_half_float

appear to be a real failure, somehow caused by the change to typ. Perhaps the
new harness runs the tests in a different order and that provokes a bug.
http://crbug.com/694359 has been filed to track the bug and
https://codereview.chromium.org/2703123003/ has been filed to suppress it.

[nednguyen, 2017-02-21 23:55:41]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-02-21 23:56:42]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[nednguyen, 2017-02-21 23:58:18]

On 2017/02/21 02:20:13, Ken Russell wrote:
> The failures on linux_optional_gpu_tests_rel of:
>
WebglConformance_conformance2_textures_canvas_sub_rectangle_tex_2d_r11f_g11f_b10f_rgb_half_float
> 
> appear to be a real failure, somehow caused by the change to typ. Perhaps the
> new harness runs the tests in a different order and that provokes a bug.
> http://crbug.com/694359 has been filed to track the bug and
> https://codereview.chromium.org/2703123003/ has been filed to suppress it.

Rolling DEPS again, running the try job

[Ken Russell (switch to Gerrit), 2017-02-22 00:37:56]

LGTM!

[Ken Russell (switch to Gerrit), 2017-02-22 00:41:45]

https://codereview.chromium.org/2663813003/diff/200001/content/test/gpu/gpu_t...
File content/test/gpu/gpu_tests/gpu_integration_test_unittest.py (right):

https://codereview.chromium.org/2663813003/diff/200001/content/test/gpu/gpu_t...
content/test/gpu/gpu_tests/gpu_integration_test_unittest.py:264: def
load_tests(loader, tests, pattern):
Hmm. Does this test run correctly as part of gpu_unittests after the typ
cutover?

[nednguyen, 2017-02-22 00:46:06]

https://codereview.chromium.org/2663813003/diff/200001/content/test/gpu/gpu_t...
File content/test/gpu/gpu_tests/gpu_integration_test_unittest.py (right):

https://codereview.chromium.org/2663813003/diff/200001/content/test/gpu/gpu_t...
content/test/gpu/gpu_tests/gpu_integration_test_unittest.py:264: def
load_tests(loader, tests, pattern):
On 2017/02/22 00:41:45, Ken Russell wrote:
> Hmm. Does this test run correctly as part of gpu_unittests after the typ
> cutover?

Yes it does. In the typ change, I have logic that does basically:

def LoadAllTestsInModule:
   if test is in invoked in integration harness:
      generate & return actual tests
   else:
      return []

[Ken Russell (switch to Gerrit), 2017-02-22 00:57:20]

https://codereview.chromium.org/2663813003/diff/200001/content/test/gpu/gpu_t...
File content/test/gpu/gpu_tests/gpu_integration_test_unittest.py (right):

https://codereview.chromium.org/2663813003/diff/200001/content/test/gpu/gpu_t...
content/test/gpu/gpu_tests/gpu_integration_test_unittest.py:264: def
load_tests(loader, tests, pattern):
On 2017/02/22 00:46:06, nednguyen wrote:
> On 2017/02/22 00:41:45, Ken Russell wrote:
> > Hmm. Does this test run correctly as part of gpu_unittests after the typ
> > cutover?
> 
> Yes it does. In the typ change, I have logic that does basically:
> 
> def LoadAllTestsInModule:
>    if test is in invoked in integration harness:
>       generate & return actual tests
>    else:
>       return []

It looks like something's wrong. See this dry run result:
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel...

specifically for telemetry_gpu_unittests:
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel...

Right now (before this CL), when running ./content/test/gpu/run_unittests.py ,
there are lines in the output like:
[0/33]
gpu_tests.gpu_integration_test_unittest.GpuIntegrationTestUnittest.testIntegrationUnittestWithBrowserCrashUponStart

But none of the tests from gpu_integration_test_unittest.py are running in that
test step any more.

[commit-bot: I haz the power, 2017-02-22 01:59:49]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-22 01:59:50]

Dry run: Try jobs failed on following builders:
  cast_shell_linux on master.tryserver.chromium.linux (JOB_TIMED_OUT, no build
URL)
  chromeos_amd64-generic_chromium_compile_only_ng on
master.tryserver.chromium.linux (JOB_TIMED_OUT, no build URL)
  chromium_presubmit on master.tryserver.chromium.linux (JOB_TIMED_OUT, no build
URL)
  linux_chromium_chromeos_ozone_rel_ng on master.tryserver.chromium.linux
(JOB_TIMED_OUT, no build URL)
  linux_chromium_compile_dbg_ng on master.tryserver.chromium.linux
(JOB_TIMED_OUT, no build URL)
  linux_chromium_rel_ng on master.tryserver.chromium.linux (JOB_TIMED_OUT, no
build URL)
  linux_chromium_tsan_rel_ng on master.tryserver.chromium.linux (JOB_TIMED_OUT,
no build URL)

[nednguyen, 2017-02-22 02:32:11]

Patchset #10 (id:220001) has been deleted

[nednguyen, 2017-02-22 02:33:50]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-02-22 02:35:00]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-22 03:46:39]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-22 03:46:41]

Dry run: Try jobs failed on following builders:
  win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)
  win_chromium_x64_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_...)

[nednguyen, 2017-02-22 15:31:43]

The CQ bit was checked by nednguyen@google.com to run a CQ dry run

[commit-bot: I haz the power, 2017-02-22 15:32:15]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[nednguyen, 2017-02-22 15:32:44]

On 2017/02/22 00:57:20, Ken Russell wrote:
>
https://codereview.chromium.org/2663813003/diff/200001/content/test/gpu/gpu_t...
> File content/test/gpu/gpu_tests/gpu_integration_test_unittest.py (right):
> 
>
https://codereview.chromium.org/2663813003/diff/200001/content/test/gpu/gpu_t...
> content/test/gpu/gpu_tests/gpu_integration_test_unittest.py:264: def
> load_tests(loader, tests, pattern):
> On 2017/02/22 00:46:06, nednguyen wrote:
> > On 2017/02/22 00:41:45, Ken Russell wrote:
> > > Hmm. Does this test run correctly as part of gpu_unittests after the typ
> > > cutover?
> > 
> > Yes it does. In the typ change, I have logic that does basically:
> > 
> > def LoadAllTestsInModule:
> >    if test is in invoked in integration harness:
> >       generate & return actual tests
> >    else:
> >       return []
> 
> It looks like something's wrong. See this dry run result:
>
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel...
> 
> specifically for telemetry_gpu_unittests:
>
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel...
> 
> Right now (before this CL), when running ./content/test/gpu/run_unittests.py ,
> there are lines in the output like:
> [0/33]
>
gpu_tests.gpu_integration_test_unittest.GpuIntegrationTestUnittest.testIntegrationUnittestWithBrowserCrashUponStart
> 
> But none of the tests from gpu_integration_test_unittest.py are running in
that
> test step any more.

Ken, PTAL

This time, most of the change are in:
content/test/gpu/gpu_tests/gpu_integration_test_unittest.py
content/test/gpu/unittest_data/integration_tests.py

[commit-bot: I haz the power, 2017-02-22 16:44:59]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-22 16:45:01]

Dry run: This issue passed the CQ dry run.

[Ken Russell (switch to Gerrit), 2017-02-22 21:55:11]

Thanks Ned for making the gpu_integration_test_unittests work again in the new
harness. LGTM

[nednguyen, 2017-02-22 22:07:38]

The CQ bit was checked by nednguyen@google.com

[nednguyen, 2017-02-22 22:07:39]

The patchset sent to the CQ was uploaded after l-g-t-m from kbr@chromium.org
Link to the patchset: https://codereview.chromium.org/2663813003/#ps280001
(title: "Roll DEPS to 42d9cffaa797712c58d7b5fb59820f0aa3e82138")

[nednguyen, 2017-02-22 22:08:06]

Description was changed from

==========
Roll out framework change & update gpu tests to conform to the API changes

The changes include in this roll:
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC...

BUG=636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

to

==========
Roll out framework change & update gpu tests to conform to the API changes

The changes include in this roll:
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC...

BUG=636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

[commit-bot: I haz the power, 2017-02-22 22:08:31]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-22 23:28:54]

CQ is committing da patch.

Bot data: {"patchset_id": 280001, "attempt_start_ts": 1487801258639680,
"parent_rev": "a755006832e23bbdbf38c3fa692e502ef4d67616", "commit_rev":
"ecdf404b0186581b08828eb1db2caa3b88b336f9"}

[commit-bot: I haz the power, 2017-02-22 23:29:31]

Description was changed from

==========
Roll out framework change & update gpu tests to conform to the API changes

The changes include in this roll:
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC...

BUG=636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
==========

to

==========
Roll out framework change & update gpu tests to conform to the API changes

The changes include in this roll:
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, see:
http://www.chromium.org/developers/tree-sheriffs/sheriff-details-chromium#TOC...

BUG=636153
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2663813003
Cr-Commit-Position: refs/heads/master@{#452269}
Committed:
https://chromium.googlesource.com/chromium/src/+/ecdf404b0186581b08828eb1db2c...
==========

[commit-bot: I haz the power, 2017-02-22 23:29:33]

Committed patchset #12 (id:280001) as
https://chromium.googlesource.com/chromium/src/+/ecdf404b0186581b08828eb1db2c...