Use swarming in create_pagesets CT task

Use swarming in create_pagesets CT task

Tested with:
$ go run go/worker_scripts/create_pagesets/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local
$ go run go/master_scripts/create_pagesets_on_workers/main.go --logtostderr --emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing"

Verified that the above created 1000 page sets:
$ gsutil -m ls -l gs://cluster-telemetry/swarming/page_sets/Dummy1k/*/*.py | grep -v TOTAL | wc -l
1000


BUG=skia:5292
Committed: https://skia.googlesource.com/buildbot/+/53db4d396e321807de6259bb0117328b448dbd18

[rmistry, 2016-05-18 13:43:47]

Description was changed from

==========
Use swarming in create_pagesets CT task

BUG=skia:5292
==========

to

==========
Use swarming in create_pagesets CT task

Tested with:
$ go run go/worker_scripts/create_pagesets/main.go --logtostderr --start_range=1
--num=10 --pageset_type=Dummy1k --local
$ go run go/master_scripts/create_pagesets_on_workers/main.go --logtostderr
--emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing"


BUG=skia:5292
==========

[rmistry, 2016-05-18 14:17:06]

Description was changed from

==========
Use swarming in create_pagesets CT task

Tested with:
$ go run go/worker_scripts/create_pagesets/main.go --logtostderr --start_range=1
--num=10 --pageset_type=Dummy1k --local
$ go run go/master_scripts/create_pagesets_on_workers/main.go --logtostderr
--emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing"


BUG=skia:5292
==========

to

==========
Use swarming in create_pagesets CT task

Tested with:
$ go run go/worker_scripts/create_pagesets/main.go --logtostderr --start_range=1
--num=10 --pageset_type=Dummy1k --local
$ go run go/master_scripts/create_pagesets_on_workers/main.go --logtostderr
--emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing"

Verified that the above created 1000 page sets:
$ gsutil -m ls -l gs://cluster-telemetry/swarming/page_sets/Dummy1k/*/*.py |
grep -v TOTAL | wc -l
1000


BUG=skia:5292
==========

[rmistry, 2016-05-18 14:17:35]

rmistry@google.com changed reviewers:
+ benjaminwagner@google.com

[rmistry, 2016-05-18 14:53:57]

The CQ bit was checked by rmistry@google.com to run a CQ dry run

[commit-bot: I haz the power, 2016-05-18 14:54:09]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1990873002/40001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1990873002/40001

[commit-bot: I haz the power, 2016-05-18 14:56:58]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-05-18 14:56:59]

Dry run: This issue passed the CQ dry run.

[dogben, 2016-05-18 17:38:26]

lgtm

https://codereview.chromium.org/1990873002/diff/40001/ct/go/master_scripts/cr...
File ct/go/master_scripts/create_pagesets_on_workers/main.go (right):

https://codereview.chromium.org/1990873002/diff/40001/ct/go/master_scripts/cr...
ct/go/master_scripts/create_pagesets_on_workers/main.go:112: pathToIsolates :=
filepath.Join(filepath.Dir((filepath.Dir(filepath.Dir(filepath.Dir(currentFile))))),
"isolates")
nit: extra parentheses

https://codereview.chromium.org/1990873002/diff/40001/ct/go/util/util.go
File ct/go/util/util.go (right):

https://codereview.chromium.org/1990873002/diff/40001/ct/go/util/util.go#newc...
ct/go/util/util.go:403: func ArchiveTriggerCollectSwarmingTask(s
*swarming.SwarmingClient, taskNames, genJSONs []string, hardTimeout, ioTimeout
time.Duration) error {
Seems that taskNames is only used to verify tasksToHashes, but you could use
genJSONs for that purpose.

If you remove taskNames here, you can also simplify the code in
create_pagesets_on_workers.

https://codereview.chromium.org/1990873002/diff/40001/ct/go/util/util.go#newc...
ct/go/util/util.go:430: var pathToPyFiles string
nit: just return rather than assigning to pathToPyFiles?

https://codereview.chromium.org/1990873002/diff/40001/ct/go/worker_scripts/cr...
File ct/go/worker_scripts/create_pagesets/main.go (right):

https://codereview.chromium.org/1990873002/diff/40001/ct/go/worker_scripts/cr...
ct/go/worker_scripts/create_pagesets/main.go:73: endRange := *startRange +
(skutil.MinInt(*num, numPages) - 1)
Does this want to be
skutil.MinInt(*startRange + *num - 1, numPages)

[rmistry, 2016-05-19 12:43:37]

Patchset #4 (id:60001) has been deleted

[rmistry, 2016-05-19 12:50:15]

Patchset #4 (id:80001) has been deleted

[rmistry, 2016-05-19 12:59:13]

Patchset #4 (id:100001) has been deleted

[rmistry, 2016-05-19 12:59:50]

I also refactored the swarming code into util.go to be used from all master
scripts.

https://codereview.chromium.org/1990873002/diff/40001/ct/go/master_scripts/cr...
File ct/go/master_scripts/create_pagesets_on_workers/main.go (right):

https://codereview.chromium.org/1990873002/diff/40001/ct/go/master_scripts/cr...
ct/go/master_scripts/create_pagesets_on_workers/main.go:112: pathToIsolates :=
filepath.Join(filepath.Dir((filepath.Dir(filepath.Dir(filepath.Dir(currentFile))))),
"isolates")
On 2016/05/18 17:38:25, Ben Wagner wrote:
> nit: extra parentheses

Done.

https://codereview.chromium.org/1990873002/diff/40001/ct/go/util/util.go
File ct/go/util/util.go (right):

https://codereview.chromium.org/1990873002/diff/40001/ct/go/util/util.go#newc...
ct/go/util/util.go:403: func ArchiveTriggerCollectSwarmingTask(s
*swarming.SwarmingClient, taskNames, genJSONs []string, hardTimeout, ioTimeout
time.Duration) error {
On 2016/05/18 17:38:26, Ben Wagner wrote:
> Seems that taskNames is only used to verify tasksToHashes, but you could use
> genJSONs for that purpose.
> 
> If you remove taskNames here, you can also simplify the code in
> create_pagesets_on_workers.

Keeping the taskNames check because of all the new code moved into this function
now.

https://codereview.chromium.org/1990873002/diff/40001/ct/go/util/util.go#newc...
ct/go/util/util.go:430: var pathToPyFiles string
On 2016/05/18 17:38:26, Ben Wagner wrote:
> nit: just return rather than assigning to pathToPyFiles?

Done.

https://codereview.chromium.org/1990873002/diff/40001/ct/go/worker_scripts/cr...
File ct/go/worker_scripts/create_pagesets/main.go (right):

https://codereview.chromium.org/1990873002/diff/40001/ct/go/worker_scripts/cr...
ct/go/worker_scripts/create_pagesets/main.go:73: endRange := *startRange +
(skutil.MinInt(*num, numPages) - 1)
On 2016/05/18 17:38:26, Ben Wagner wrote:
> Does this want to be
> skutil.MinInt(*startRange + *num - 1, numPages)

Yes that looks right. Thanks!

[rmistry, 2016-05-19 13:07:02]

The CQ bit was checked by rmistry@google.com to run a CQ dry run

[commit-bot: I haz the power, 2016-05-19 13:07:07]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1990873002/120001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1990873002/120001

[commit-bot: I haz the power, 2016-05-19 13:09:58]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-05-19 13:09:59]

Dry run: This issue passed the CQ dry run.

[dogben, 2016-05-19 14:27:22]

lgtm

https://codereview.chromium.org/1990873002/diff/40001/ct/go/util/util.go
File ct/go/util/util.go (right):

https://codereview.chromium.org/1990873002/diff/40001/ct/go/util/util.go#newc...
ct/go/util/util.go:403: func ArchiveTriggerCollectSwarmingTask(s
*swarming.SwarmingClient, taskNames, genJSONs []string, hardTimeout, ioTimeout
time.Duration) error {
On 2016/05/19 12:59:50, rmistry wrote:
> On 2016/05/18 17:38:26, Ben Wagner wrote:
> > Seems that taskNames is only used to verify tasksToHashes, but you could use
> > genJSONs for that purpose.
> > 
> > If you remove taskNames here, you can also simplify the code in
> > create_pagesets_on_workers.
> 
> Keeping the taskNames check because of all the new code moved into this
function
> now.

I don't understand.

[rmistry, 2016-05-19 14:41:30]

Patchset #6 (id:130008) has been deleted

[rmistry, 2016-05-19 14:42:20]

https://codereview.chromium.org/1990873002/diff/40001/ct/go/util/util.go
File ct/go/util/util.go (right):

https://codereview.chromium.org/1990873002/diff/40001/ct/go/util/util.go#newc...
ct/go/util/util.go:403: func ArchiveTriggerCollectSwarmingTask(s
*swarming.SwarmingClient, taskNames, genJSONs []string, hardTimeout, ioTimeout
time.Duration) error {
On 2016/05/19 14:27:22, Ben Wagner wrote:
> On 2016/05/19 12:59:50, rmistry wrote:
> > On 2016/05/18 17:38:26, Ben Wagner wrote:
> > > Seems that taskNames is only used to verify tasksToHashes, but you could
use
> > > genJSONs for that purpose.
> > > 
> > > If you remove taskNames here, you can also simplify the code in
> > > create_pagesets_on_workers.
> > 
> > Keeping the taskNames check because of all the new code moved into this
> function
> > now.
> 
> I don't understand.

Done.

[dogben, 2016-05-19 16:15:41]

lgtm

Still LGTM.

[rmistry, 2016-05-19 16:16:50]

The CQ bit was checked by rmistry@google.com

[commit-bot: I haz the power, 2016-05-19 16:16:59]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1990873002/170001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1990873002/170001

[commit-bot: I haz the power, 2016-05-19 16:19:58]

Description was changed from

==========
Use swarming in create_pagesets CT task

Tested with:
$ go run go/worker_scripts/create_pagesets/main.go --logtostderr --start_range=1
--num=10 --pageset_type=Dummy1k --local
$ go run go/master_scripts/create_pagesets_on_workers/main.go --logtostderr
--emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing"

Verified that the above created 1000 page sets:
$ gsutil -m ls -l gs://cluster-telemetry/swarming/page_sets/Dummy1k/*/*.py |
grep -v TOTAL | wc -l
1000


BUG=skia:5292
==========

to

==========
Use swarming in create_pagesets CT task

Tested with:
$ go run go/worker_scripts/create_pagesets/main.go --logtostderr --start_range=1
--num=10 --pageset_type=Dummy1k --local
$ go run go/master_scripts/create_pagesets_on_workers/main.go --logtostderr
--emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing"

Verified that the above created 1000 page sets:
$ gsutil -m ls -l gs://cluster-telemetry/swarming/page_sets/Dummy1k/*/*.py |
grep -v TOTAL | wc -l
1000


BUG=skia:5292

Committed:
https://skia.googlesource.com/buildbot/+/53db4d396e321807de6259bb0117328b448d...
==========

[commit-bot: I haz the power, 2016-05-19 16:20:00]

Committed patchset #6 (id:170001) as
https://skia.googlesource.com/buildbot/+/53db4d396e321807de6259bb0117328b448d...