buildbot: Run the webkit layout tests under swarming on RandomOrder bots.

buildbot: Run the webkit layout tests under swarming on RandomOrder bots.

This CL enables the RandomOrder FYI bots to run the webkit layout tests on
swarming. It does this by;

 * Creating a dummy webkit_layout_tests_exparchive build target which tells
   recipes to use 'isolate exparchive' for uploading (rather than 'isolate
   batcharchive'). See;
    - https://chromium-review.googlesource.com/c/436324/
    - https://chromium-review.googlesource.com/c/422460/

 * Updates the testing/buildbot/chromium.fyi.json to include the RandomOrder
   bots.

This CL is based on Dirk's CL at http://crrev.com/2595793002#ps40001

BUG=524758
NOTRY=true

Review-Url: https://codereview.chromium.org/2678153003
Cr-Commit-Position: refs/heads/master@{#450911}
Committed: https://chromium.googlesource.com/chromium/src/+/0e6baaf2652e33f99bb396851b2036fff99b7834

[mithro, 2017-02-07 18:04:37]

Description was changed from

==========
Run the webkit layout tests under swarming on RandomOrder bots.

BUG=524758
==========

to

==========
Run the webkit layout tests under swarming on RandomOrder bots.

This CL enables the RandomOrder FYI bots to run the webkit layout tests on
swarming. It does this by;

 * Creating a dummy webkit_layout_tests_exparchive build target which tells
recipes to use 'isolate exparchive' for uploading (rather than 'isolate
batcharchive'). See https://chromium-review.googlesource.com/c/436324/ and
https://chromium-review.googlesource.com/c/422460/

 * Updates the testing/buildbot/chromium.fyi.json to include the RandomOrder
bots.

BUG=524758
==========

[mithro, 2017-02-07 18:04:37]

tansell@chromium.org changed reviewers:
+ jeffcarp@chromium.org, mcgreevy@chromium.org, qyearsley@chromium.org

[mithro, 2017-02-07 18:05:14]

Description was changed from

==========
Run the webkit layout tests under swarming on RandomOrder bots.

This CL enables the RandomOrder FYI bots to run the webkit layout tests on
swarming. It does this by;

 * Creating a dummy webkit_layout_tests_exparchive build target which tells
recipes to use 'isolate exparchive' for uploading (rather than 'isolate
batcharchive'). See https://chromium-review.googlesource.com/c/436324/ and
https://chromium-review.googlesource.com/c/422460/

 * Updates the testing/buildbot/chromium.fyi.json to include the RandomOrder
bots.

BUG=524758
==========

to

==========
Run the webkit layout tests under swarming on RandomOrder bots.

This CL enables the RandomOrder FYI bots to run the webkit layout tests on
swarming. It does this by;

 * Creating a dummy webkit_layout_tests_exparchive build target
   which tells recipes to use 'isolate exparchive' for uploading
   (rather than 'isolate batcharchive'). See; 
    - https://chromium-review.googlesource.com/c/436324/
    - https://chromium-review.googlesource.com/c/422460/

 * Updates the testing/buildbot/chromium.fyi.json to include 
   the RandomOrder bots.

BUG=524758
==========

[mithro, 2017-02-07 18:05:42]

Description was changed from

==========
Run the webkit layout tests under swarming on RandomOrder bots.

This CL enables the RandomOrder FYI bots to run the webkit layout tests on
swarming. It does this by;

 * Creating a dummy webkit_layout_tests_exparchive build target
   which tells recipes to use 'isolate exparchive' for uploading
   (rather than 'isolate batcharchive'). See; 
    - https://chromium-review.googlesource.com/c/436324/
    - https://chromium-review.googlesource.com/c/422460/

 * Updates the testing/buildbot/chromium.fyi.json to include 
   the RandomOrder bots.

BUG=524758
==========

to

==========
buildbot: Run the webkit layout tests under swarming on RandomOrder bots.

This CL enables the RandomOrder FYI bots to run the webkit 
layout tests on swarming. It does this by;

 * Creating a dummy webkit_layout_tests_exparchive build target
   which tells recipes to use 'isolate exparchive' for uploading
   (rather than 'isolate batcharchive'). See; 
    - https://chromium-review.googlesource.com/c/436324/
    - https://chromium-review.googlesource.com/c/422460/

 * Updates the testing/buildbot/chromium.fyi.json to include 
   the RandomOrder bots.

BUG=524758
==========

[mithro, 2017-02-07 18:08:22]

Hi everyone,

This CL is not 100% ready for review, but wanted to send it out for people to
start looking at anyway. 

My biggest question is if we need testing/scripts/run_isolated_script_test.py
script, previously I had just been using
//testing/scripts/run_telemetry_as_googletest.py and I really want to stop the
proliferation of wrapper scripts.

Myself and Jeff will probably be working on this today (and hopefully look at
landing).

Thanks!

Tim 'mithro' Ansell

[mithro, 2017-02-07 18:52:42]

The CQ bit was checked by tansell@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-02-07 18:53:14]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[mithro, 2017-02-07 19:56:11]

Description was changed from

==========
buildbot: Run the webkit layout tests under swarming on RandomOrder bots.

This CL enables the RandomOrder FYI bots to run the webkit 
layout tests on swarming. It does this by;

 * Creating a dummy webkit_layout_tests_exparchive build target
   which tells recipes to use 'isolate exparchive' for uploading
   (rather than 'isolate batcharchive'). See; 
    - https://chromium-review.googlesource.com/c/436324/
    - https://chromium-review.googlesource.com/c/422460/

 * Updates the testing/buildbot/chromium.fyi.json to include 
   the RandomOrder bots.

BUG=524758
==========

to

==========
buildbot: Run the webkit layout tests under swarming on RandomOrder bots.

This CL enables the RandomOrder FYI bots to run the webkit layout tests on
swarming. It does this by;

 * Creating a dummy webkit_layout_tests_exparchive build target which tells
   recipes to use 'isolate exparchive' for uploading (rather than 'isolate
   batcharchive'). See;
    - https://chromium-review.googlesource.com/c/436324/
    - https://chromium-review.googlesource.com/c/422460/

 * Updates the testing/buildbot/chromium.fyi.json to include the RandomOrder
   bots.

This CL is based on Dirk's CL at http://crrev.com/2595793002#ps40001

BUG=524758
==========

[mithro, 2017-02-07 20:07:02]

The CQ bit was unchecked by tansell@chromium.org

[mithro, 2017-02-07 20:38:34]

The CQ bit was checked by tansell@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-02-07 20:39:28]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-07 21:47:23]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-07 21:47:25]

Dry run: This issue passed the CQ dry run.

[jeffcarp, 2017-02-10 02:38:36]

https://codereview.chromium.org/2678153003/diff/40001/testing/scripts/run_iso...
File testing/scripts/run_isolated_script_test.py (right):

https://codereview.chromium.org/2678153003/diff/40001/testing/scripts/run_iso...
testing/scripts/run_isolated_script_test.py:72: del env['GTEST_SHARD_INDEX']
run-webkit-tests already understands GTEST_TOTAL_SHARDS and GTEST_SHARD_INDEX:
https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitp...

Unless there's another reason this logic is here, those args could be passed
through to run-webkit-tests.

[mithro, 2017-02-14 00:53:32]

tansell@chromium.org changed reviewers:
+ dpranke@chromium.org

[mithro, 2017-02-14 01:20:29]

I need an OWNERS review, adding Dirk.

[Dirk Pranke, 2017-02-14 01:30:40]

lgtm

[Dirk Pranke, 2017-02-14 01:31:15]

To address jeffcarp's comment: I really dislike using env vars for things and
would much rather stomp those out as early as possible, rather than passing them
through.

[mithro, 2017-02-14 03:41:29]

The CQ bit was checked by tansell@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-02-14 03:42:23]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-14 05:27:48]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-14 05:27:49]

Dry run: This issue passed the CQ dry run.

[mithro, 2017-02-14 18:57:53]

tansell@chromium.org changed reviewers:
+ phajdan.jr@chromium.org

[mithro, 2017-02-14 18:57:54]

Hi Dirk & Pawel,

I created https://codereview.chromium.org/2681643007/ on top of this change to
verify that this change would work using the the trybots (I'm also testing on my
local fake buildbot).

This correctly ran the webkit_layout_tests_exparchive on swarming. However, it
seems to *also* be running the local webkit_tests. At first I thought it was
because of a dependency analysis stuff but I think it is because of special
cases in the chromium recipe.

I think I need to modify scripts/slave/recipe_modules/chromium_tests/api.py in
recipies here ->
https://chromium.googlesource.com/chromium/tools/build/+/master/scripts/slave...

Any thoughts before I go ahead an do that?

Tim 'mithro' Ansell

[Dirk Pranke, 2017-02-15 00:56:05]

On 2017/02/14 18:57:54, mithro wrote:
> Hi Dirk & Pawel,
> 
> I created https://codereview.chromium.org/2681643007/ on top of this change to
> verify that this change would work using the the trybots (I'm also testing on
my
> local fake buildbot).
> 
> This correctly ran the webkit_layout_tests_exparchive on swarming. However, it
> seems to *also* be running the local webkit_tests. At first I thought it was
> because of a dependency analysis stuff but I think it is because of special
> cases in the chromium recipe.
> 
> I think I need to modify scripts/slave/recipe_modules/chromium_tests/api.py in
> recipies here ->
>
https://chromium.googlesource.com/chromium/tools/build/+/master/scripts/slave...
> 
> Any thoughts before I go ahead an do that?

What are you suggesting we modify? We need to keep running the non-swarmed
version of those tests until we're ready to switch
to the swarmed ones.

For your testing purposes, I'd expect you can mostly just run both sets of
tests?

[mithro, 2017-02-15 01:02:40]

On 2017/02/15 00:56:05, Dirk Pranke wrote:
> On 2017/02/14 18:57:54, mithro wrote:
> > Hi Dirk & Pawel,
> > 
> > I created https://codereview.chromium.org/2681643007/ on top of this change
to
> > verify that this change would work using the the trybots (I'm also testing
on
> my
> > local fake buildbot).
> > 
> > This correctly ran the webkit_layout_tests_exparchive on swarming. However,
it
> > seems to *also* be running the local webkit_tests. At first I thought it was
> > because of a dependency analysis stuff but I think it is because of special
> > cases in the chromium recipe.
> > 
> > I think I need to modify scripts/slave/recipe_modules/chromium_tests/api.py
in
> > recipies here ->
> >
>
https://chromium.googlesource.com/chromium/tools/build/+/master/scripts/slave...
> > 
> > Any thoughts before I go ahead an do that?
> 
> What are you suggesting we modify? We need to keep running the non-swarmed
> version of those tests until we're ready to switch
> to the swarmed ones.
> 
> For your testing purposes, I'd expect you can mostly just run both sets of
> tests?

Sorry if I wasn't clear, local webkit_tests *includes* the webkit_layout_tests
run locally. We don't really want to be running the layout tests locally and on
swarming at the same time?

My suggestion is that we add a property when set disables adding the
self.steps.BlinkTest() at
https://chromium.googlesource.com/chromium/tools/build/+/master/scripts/slave...

Tim 'mithro' Ansell

[Dirk Pranke, 2017-02-15 01:09:23]

On 2017/02/15 01:02:40, mithro wrote:
> On 2017/02/15 00:56:05, Dirk Pranke wrote:
> > On 2017/02/14 18:57:54, mithro wrote:
> > > Hi Dirk & Pawel,
> > > 
> > > I created https://codereview.chromium.org/2681643007/ on top of this
change
> to
> > > verify that this change would work using the the trybots (I'm also testing
> on
> > my
> > > local fake buildbot).
> > > 
> > > This correctly ran the webkit_layout_tests_exparchive on swarming.
However,
> it
> > > seems to *also* be running the local webkit_tests. At first I thought it
was
> > > because of a dependency analysis stuff but I think it is because of
special
> > > cases in the chromium recipe.
> > > 
> > > I think I need to modify
scripts/slave/recipe_modules/chromium_tests/api.py
> in
> > > recipies here ->
> > >
> >
>
https://chromium.googlesource.com/chromium/tools/build/+/master/scripts/slave...
> > > 
> > > Any thoughts before I go ahead an do that?
> > 
> > What are you suggesting we modify? We need to keep running the non-swarmed
> > version of those tests until we're ready to switch
> > to the swarmed ones.
> > 
> > For your testing purposes, I'd expect you can mostly just run both sets of
> > tests?
> 
> Sorry if I wasn't clear, local webkit_tests *includes* the webkit_layout_tests
> run locally. We don't really want to be running the layout tests locally and
on
> swarming at the same time?

It actually can be useful to run them at the same time for testing purposes
(i.e., so you can see what fails in which case).

> 
> My suggestion is that we add a property when set disables adding the
> self.steps.BlinkTest() at
>
https://chromium.googlesource.com/chromium/tools/build/+/master/scripts/slave...

I think it's fine to add some sort of switch somewhere to control whether you
want to run things or not, but it's not 100%
clear to me what that switch would look like. 

Ideally we'd have something like a blink_tests() generator that would read a
`blink_tests` entry from the JSON file and a list
of directories to trigger on, and move the CHROMIUM_BLINK_TESTS_PATHS src-side
as a result, so we had better control
over this generally.

Would you be willing to take a stab at that, even though it's more work?

[mithro, 2017-02-16 00:33:31]

The CQ bit was checked by tansell@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-02-16 00:35:07]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[mithro, 2017-02-16 00:54:12]

The CQ bit was unchecked by tansell@chromium.org

[mithro, 2017-02-16 01:02:59]

The CQ bit was checked by tansell@chromium.org

[mithro, 2017-02-16 01:03:00]

The patchset sent to the CQ was uploaded after l-g-t-m from dpranke@chromium.org
Link to the patchset: https://codereview.chromium.org/2678153003/#ps120001
(title: "Rebase onto master.")

[commit-bot: I haz the power, 2017-02-16 01:05:38]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-16 02:38:56]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-16 02:38:57]

Try jobs failed on following builders:
  chromeos_daisy_chromium_compile_only_ng on master.tryserver.chromium.linux
(JOB_TIMED_OUT, no build URL)
  linux_chromium_asan_rel_ng on master.tryserver.chromium.linux (JOB_TIMED_OUT,
no build URL)
  linux_chromium_chromeos_rel_ng on master.tryserver.chromium.linux
(JOB_TIMED_OUT, no build URL)
  linux_chromium_compile_dbg_ng on master.tryserver.chromium.linux
(JOB_TIMED_OUT, no build URL)
  linux_chromium_rel_ng on master.tryserver.chromium.linux (JOB_TIMED_OUT, no
build URL)
  linux_chromium_tsan_rel_ng on master.tryserver.chromium.linux (JOB_TIMED_OUT,
no build URL)

[mithro, 2017-02-16 02:44:44]

The CQ bit was checked by tansell@chromium.org

[commit-bot: I haz the power, 2017-02-16 02:46:11]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-16 02:51:48]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-16 02:51:49]

Exceeded global retry quota

[mithro, 2017-02-16 03:03:33]

The CQ bit was checked by tansell@chromium.org

[commit-bot: I haz the power, 2017-02-16 03:05:17]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-16 03:08:32]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-16 03:08:33]

Try jobs failed on following builders:
  chromium_presubmit on master.tryserver.chromium.linux (JOB_TIMED_OUT, no build
URL)

[mithro, 2017-02-16 03:45:10]

The CQ bit was checked by tansell@chromium.org

[commit-bot: I haz the power, 2017-02-16 03:45:51]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-16 05:34:59]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-16 05:35:00]

Try jobs failed on following builders:
  chromium_presubmit on master.tryserver.chromium.linux (JOB_TIMED_OUT, no build
URL)

[mithro, 2017-02-16 05:37:45]

The CQ bit was checked by tansell@chromium.org

[commit-bot: I haz the power, 2017-02-16 05:38:56]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-16 07:49:54]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-16 07:49:55]

Try jobs failed on following builders:
  linux_chromium_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[mithro, 2017-02-16 07:50:28]

The CQ bit was checked by tansell@chromium.org

[commit-bot: I haz the power, 2017-02-16 07:51:08]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[mithro, 2017-02-16 09:02:37]

Description was changed from

==========
buildbot: Run the webkit layout tests under swarming on RandomOrder bots.

This CL enables the RandomOrder FYI bots to run the webkit layout tests on
swarming. It does this by;

 * Creating a dummy webkit_layout_tests_exparchive build target which tells
   recipes to use 'isolate exparchive' for uploading (rather than 'isolate
   batcharchive'). See;
    - https://chromium-review.googlesource.com/c/436324/
    - https://chromium-review.googlesource.com/c/422460/

 * Updates the testing/buildbot/chromium.fyi.json to include the RandomOrder
   bots.

This CL is based on Dirk's CL at http://crrev.com/2595793002#ps40001

BUG=524758
==========

to

==========
buildbot: Run the webkit layout tests under swarming on RandomOrder bots.

This CL enables the RandomOrder FYI bots to run the webkit layout tests on
swarming. It does this by;

 * Creating a dummy webkit_layout_tests_exparchive build target which tells
   recipes to use 'isolate exparchive' for uploading (rather than 'isolate
   batcharchive'). See;
    - https://chromium-review.googlesource.com/c/436324/
    - https://chromium-review.googlesource.com/c/422460/

 * Updates the testing/buildbot/chromium.fyi.json to include the RandomOrder
   bots.

This CL is based on Dirk's CL at http://crrev.com/2595793002#ps40001

BUG=524758
NOTRY=true
==========

[mithro, 2017-02-16 09:02:46]

The CQ bit was unchecked by tansell@chromium.org

[mithro, 2017-02-16 09:03:57]

The CQ bit was checked by tansell@chromium.org

[commit-bot: I haz the power, 2017-02-16 09:04:46]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-16 09:14:59]

CQ is committing da patch.

Bot data: {"patchset_id": 120001, "attempt_start_ts": 1487235837961770,
"parent_rev": "af126ad510eb5f6ba94fa96ef44bb49a576eaff6", "commit_rev":
"0e6baaf2652e33f99bb396851b2036fff99b7834"}

[commit-bot: I haz the power, 2017-02-16 09:15:36]

Description was changed from

==========
buildbot: Run the webkit layout tests under swarming on RandomOrder bots.

This CL enables the RandomOrder FYI bots to run the webkit layout tests on
swarming. It does this by;

 * Creating a dummy webkit_layout_tests_exparchive build target which tells
   recipes to use 'isolate exparchive' for uploading (rather than 'isolate
   batcharchive'). See;
    - https://chromium-review.googlesource.com/c/436324/
    - https://chromium-review.googlesource.com/c/422460/

 * Updates the testing/buildbot/chromium.fyi.json to include the RandomOrder
   bots.

This CL is based on Dirk's CL at http://crrev.com/2595793002#ps40001

BUG=524758
NOTRY=true
==========

to

==========
buildbot: Run the webkit layout tests under swarming on RandomOrder bots.

This CL enables the RandomOrder FYI bots to run the webkit layout tests on
swarming. It does this by;

 * Creating a dummy webkit_layout_tests_exparchive build target which tells
   recipes to use 'isolate exparchive' for uploading (rather than 'isolate
   batcharchive'). See;
    - https://chromium-review.googlesource.com/c/436324/
    - https://chromium-review.googlesource.com/c/422460/

 * Updates the testing/buildbot/chromium.fyi.json to include the RandomOrder
   bots.

This CL is based on Dirk's CL at http://crrev.com/2595793002#ps40001

BUG=524758
NOTRY=true

Review-Url: https://codereview.chromium.org/2678153003
Cr-Commit-Position: refs/heads/master@{#450911}
Committed:
https://chromium.googlesource.com/chromium/src/+/0e6baaf2652e33f99bb396851b20...
==========

[commit-bot: I haz the power, 2017-02-16 09:15:39]

Committed patchset #7 (id:120001) as
https://chromium.googlesource.com/chromium/src/+/0e6baaf2652e33f99bb396851b20...