Chromium Code Reviews
|
|
Chromium Code Reviews
Issue
1988133002:
Use swarming in capture_archives CT task and capture archives in parallel (Closed)
|
Created:
4 years, 7 months ago by rmistry Modified:
4 years, 7 months ago Reviewers:
dogben CC:
reviews_skia.org Base URL:
https://skia.googlesource.com/buildbot@ct-4-create_pagesets Target Ref:
refs/heads/master Project:
skiabuildbot Visibility:
Public. |
DescriptionUse swarming in capture_archives CT task and capture archives in parallel.
Tested with:
$ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local
$ go run go/master_scripts/capture_archives_on_workers/main.go --logtostderr --emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing"
Verified that the above created 913 archives:
$ gsutil -m ls -l gs://cluster-telemetry/swarming/webpage_archives/Dummy1k/*/*.json | grep -v TOTAL | wc -l
861 (with 100 per machine and 5 goroutines took 20m)
913 (with 10 per machine and 1 goroutines took 20m)
944 (with 10 per machine and 1 goroutines with retries took 25m)
BUG=skia:5293
BUG=skia:5317
Committed: https://skia.googlesource.com/buildbot/+/3e56252be203f9a5e5cdb7167cc4d6eab7a51c95
Patch Set 1 : Initial upload #Patch Set 2 : Start Xvfb #Patch Set 3 : Remove TODOs #Patch Set 4 : Try 10 per machine and no parallelism #
Total comments: 5
Patch Set 5 : Fix errcheck #Patch Set 6 : Use new swarming method in util.go #Patch Set 7 : Added retries to capture archives #
Total comments: 2
Patch Set 8 : Comment #
Depends on Patchset: Dependent Patchsets: Messages
Total messages: 39 (21 generated)
Description was changed from ========== Use swarming in capture_archives CT task Tested with: Verified that the above created x archives: BUG=skia:5293 ========== to ========== Use swarming in capture_archives CT task Tested with: Verified that the above created x archives: BUG=skia:5293 BUG=skia:5317 ==========
Description was changed from ========== Use swarming in capture_archives CT task Tested with: Verified that the above created x archives: BUG=skia:5293 BUG=skia:5317 ========== to ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: Verified that the above created x archives: BUG=skia:5293 BUG=skia:5317 ==========
Description was changed from ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: Verified that the above created x archives: BUG=skia:5293 BUG=skia:5317 ========== to ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k Verified that the above created x archives: BUG=skia:5293 BUG=skia:5317 ==========
Description was changed from ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k Verified that the above created x archives: BUG=skia:5293 BUG=skia:5317 ========== to ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local Verified that the above created x archives: BUG=skia:5293 BUG=skia:5317 ==========
Description was changed from ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local Verified that the above created x archives: BUG=skia:5293 BUG=skia:5317 ========== to ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local $ go run go/master_scripts/capture_archives_on_workers/main.go --logtostderr --emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing" Verified that the above created x archives: BUG=skia:5293 BUG=skia:5317 ==========
Description was changed from ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local $ go run go/master_scripts/capture_archives_on_workers/main.go --logtostderr --emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing" Verified that the above created x archives: BUG=skia:5293 BUG=skia:5317 ========== to ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local $ go run go/master_scripts/capture_archives_on_workers/main.go --logtostderr --emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing" Verified that the above created x archives: $ gsutil -m ls -l gs://cluster-telemetry/swarming/webpage_archives/Dummy1k/*/*.json | grep -v TOTAL | wc -l 861 BUG=skia:5293 BUG=skia:5317 ==========
Description was changed from ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local $ go run go/master_scripts/capture_archives_on_workers/main.go --logtostderr --emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing" Verified that the above created x archives: $ gsutil -m ls -l gs://cluster-telemetry/swarming/webpage_archives/Dummy1k/*/*.json | grep -v TOTAL | wc -l 861 BUG=skia:5293 BUG=skia:5317 ========== to ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local $ go run go/master_scripts/capture_archives_on_workers/main.go --logtostderr --emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing" Verified that the above created x archives: $ gsutil -m ls -l gs://cluster-telemetry/swarming/webpage_archives/Dummy1k/*/*.json | grep -v TOTAL | wc -l 861 (with 100 per machine and 5 goroutines took 20m) BUG=skia:5293 BUG=skia:5317 ==========
Description was changed from ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local $ go run go/master_scripts/capture_archives_on_workers/main.go --logtostderr --emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing" Verified that the above created x archives: $ gsutil -m ls -l gs://cluster-telemetry/swarming/webpage_archives/Dummy1k/*/*.json | grep -v TOTAL | wc -l 861 (with 100 per machine and 5 goroutines took 20m) BUG=skia:5293 BUG=skia:5317 ========== to ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local $ go run go/master_scripts/capture_archives_on_workers/main.go --logtostderr --emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing" Verified that the above created x archives: $ gsutil -m ls -l gs://cluster-telemetry/swarming/webpage_archives/Dummy1k/*/*.json | grep -v TOTAL | wc -l 861 (with 100 per machine and 5 goroutines took 20m) 913 (with 10 per machine and 1 goroutines took 20m) BUG=skia:5293 BUG=skia:5317 ==========
rmistry@google.com changed reviewers: + benjaminwagner@google.com
https://codereview.chromium.org/1988133002/diff/60001/ct/go/worker_scripts/ca... File ct/go/worker_scripts/capture_archives/main.go (right): https://codereview.chromium.org/1988133002/diff/60001/ct/go/worker_scripts/ca... ct/go/worker_scripts/capture_archives/main.go:25: WORKER_POOL_SIZE = 1 Will continue to play with this number to find the sweet spot.
Description was changed from ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local $ go run go/master_scripts/capture_archives_on_workers/main.go --logtostderr --emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing" Verified that the above created x archives: $ gsutil -m ls -l gs://cluster-telemetry/swarming/webpage_archives/Dummy1k/*/*.json | grep -v TOTAL | wc -l 861 (with 100 per machine and 5 goroutines took 20m) 913 (with 10 per machine and 1 goroutines took 20m) BUG=skia:5293 BUG=skia:5317 ========== to ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local $ go run go/master_scripts/capture_archives_on_workers/main.go --logtostderr --emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing" Verified that the above created 913 archives: $ gsutil -m ls -l gs://cluster-telemetry/swarming/webpage_archives/Dummy1k/*/*.json | grep -v TOTAL | wc -l 861 (with 100 per machine and 5 goroutines took 20m) 913 (with 10 per machine and 1 goroutines took 20m) BUG=skia:5293 BUG=skia:5317 ==========
The CQ bit was checked by rmistry@google.com to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1988133002/60001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1988133002/60001
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: Try jobs failed on following builders: Infra-PerCommit-Trybot on client.skia.fyi (JOB_FAILED, http://build.chromium.org/p/client.skia.fyi/builders/Infra-PerCommit-Trybot/b...)
The CQ bit was checked by rmistry@google.com to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1988133002/80001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1988133002/80001
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
lgtm https://codereview.chromium.org/1988133002/diff/60001/ct/go/master_scripts/ca... File ct/go/master_scripts/capture_archives_on_workers/main.go (right): https://codereview.chromium.org/1988133002/diff/60001/ct/go/master_scripts/ca... ct/go/master_scripts/capture_archives_on_workers/main.go:92: // Instantiate the swarming client. Seems capture_archives and create_pagesets are very similar... Is it likely that other swarming tasks will be the same? Why not a generic "trigger CT swarming" script? https://codereview.chromium.org/1988133002/diff/60001/ct/go/master_scripts/ca... ct/go/master_scripts/capture_archives_on_workers/main.go:112: pathToIsolates := filepath.Join(filepath.Dir((filepath.Dir(filepath.Dir(filepath.Dir(currentFile))))), "isolates") nit: extra parens
The CQ bit was checked by rmistry@google.com to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1988133002/100001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1988133002/100001
https://codereview.chromium.org/1988133002/diff/60001/ct/go/master_scripts/ca... File ct/go/master_scripts/capture_archives_on_workers/main.go (right): https://codereview.chromium.org/1988133002/diff/60001/ct/go/master_scripts/ca... ct/go/master_scripts/capture_archives_on_workers/main.go:92: // Instantiate the swarming client. On 2016/05/18 18:59:14, Ben Wagner wrote: > Seems capture_archives and create_pagesets are very similar... Is it likely that > other swarming tasks will be the same? Why not a generic "trigger CT swarming" > script? Done. https://codereview.chromium.org/1988133002/diff/60001/ct/go/master_scripts/ca... ct/go/master_scripts/capture_archives_on_workers/main.go:112: pathToIsolates := filepath.Join(filepath.Dir((filepath.Dir(filepath.Dir(filepath.Dir(currentFile))))), "isolates") On 2016/05/18 18:59:14, Ben Wagner wrote: > nit: extra parens Done.
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
The CQ bit was checked by rmistry@google.com to run a CQ dry run
Dry run: CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1988133002/120001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1988133002/120001
Added retries @patchset7 to see if it helps increase the count of captured archives.
The CQ bit was unchecked by commit-bot@chromium.org
Dry run: This issue passed the CQ dry run.
lgtm https://codereview.chromium.org/1988133002/diff/120001/ct/go/worker_scripts/c... File ct/go/worker_scripts/capture_archives/main.go (right): https://codereview.chromium.org/1988133002/diff/120001/ct/go/worker_scripts/c... ct/go/worker_scripts/capture_archives/main.go:147: glog.Warning("Encountered an error. Retrying...") nit: include error in log?
https://codereview.chromium.org/1988133002/diff/120001/ct/go/worker_scripts/c... File ct/go/worker_scripts/capture_archives/main.go (right): https://codereview.chromium.org/1988133002/diff/120001/ct/go/worker_scripts/c... ct/go/worker_scripts/capture_archives/main.go:147: glog.Warning("Encountered an error. Retrying...") On 2016/05/19 14:29:38, Ben Wagner wrote: > nit: include error in log? Done.
Description was changed from ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local $ go run go/master_scripts/capture_archives_on_workers/main.go --logtostderr --emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing" Verified that the above created 913 archives: $ gsutil -m ls -l gs://cluster-telemetry/swarming/webpage_archives/Dummy1k/*/*.json | grep -v TOTAL | wc -l 861 (with 100 per machine and 5 goroutines took 20m) 913 (with 10 per machine and 1 goroutines took 20m) BUG=skia:5293 BUG=skia:5317 ========== to ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local $ go run go/master_scripts/capture_archives_on_workers/main.go --logtostderr --emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing" Verified that the above created 913 archives: $ gsutil -m ls -l gs://cluster-telemetry/swarming/webpage_archives/Dummy1k/*/*.json | grep -v TOTAL | wc -l 861 (with 100 per machine and 5 goroutines took 20m) 913 (with 10 per machine and 1 goroutines took 20m) 944 (with 10 per machine and 1 goroutines with retries took 25m) BUG=skia:5293 BUG=skia:5317 ==========
On 2016/05/19 14:47:04, rmistry wrote: > https://codereview.chromium.org/1988133002/diff/120001/ct/go/worker_scripts/c... > File ct/go/worker_scripts/capture_archives/main.go (right): > > https://codereview.chromium.org/1988133002/diff/120001/ct/go/worker_scripts/c... > ct/go/worker_scripts/capture_archives/main.go:147: glog.Warning("Encountered an > error. Retrying...") > On 2016/05/19 14:29:38, Ben Wagner wrote: > > nit: include error in log? > > Done. The retries seem to have helped a lot. It captured 944 archives instead of the previous 913 archives).
lgtm Still LGTM.
The CQ bit was checked by rmistry@google.com
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1988133002/140001 View timeline at https://chromium-cq-status.appspot.com/patch-timeline/1988133002/140001
Message was sent while issue was closed.
Description was changed from ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local $ go run go/master_scripts/capture_archives_on_workers/main.go --logtostderr --emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing" Verified that the above created 913 archives: $ gsutil -m ls -l gs://cluster-telemetry/swarming/webpage_archives/Dummy1k/*/*.json | grep -v TOTAL | wc -l 861 (with 100 per machine and 5 goroutines took 20m) 913 (with 10 per machine and 1 goroutines took 20m) 944 (with 10 per machine and 1 goroutines with retries took 25m) BUG=skia:5293 BUG=skia:5317 ========== to ========== Use swarming in capture_archives CT task and capture archives in parallel. Tested with: $ go run go/worker_scripts/capture_archives/main.go --logtostderr --start_range=1 --num=10 --pageset_type=Dummy1k --local $ go run go/master_scripts/capture_archives_on_workers/main.go --logtostderr --emails="rmistry@google.com" --pageset_type="Dummy1k" --run_id="testing" Verified that the above created 913 archives: $ gsutil -m ls -l gs://cluster-telemetry/swarming/webpage_archives/Dummy1k/*/*.json | grep -v TOTAL | wc -l 861 (with 100 per machine and 5 goroutines took 20m) 913 (with 10 per machine and 1 goroutines took 20m) 944 (with 10 per machine and 1 goroutines with retries took 25m) BUG=skia:5293 BUG=skia:5317 Committed: https://skia.googlesource.com/buildbot/+/3e56252be203f9a5e5cdb7167cc4d6eab7a5... ==========
Message was sent while issue was closed.
Committed patchset #8 (id:140001) as https://skia.googlesource.com/buildbot/+/3e56252be203f9a5e5cdb7167cc4d6eab7a5... |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
