Add script for triggering Cluster Telemetry jobs using local scripts

Add script for triggering Cluster Telemetry jobs using local scripts

BUG=skia:3979
Committed: https://skia.googlesource.com/skia/+/8cd8f9429a8a7d5c09c48f7cc9ab34073e01d945

[borenet, 2015-06-26 19:58:44]

borenet@google.com changed reviewers:
+ rmistry@google.com

[borenet, 2015-06-26 19:58:45]

Not quite ready for review.  Requires some changes in this script or on the
server side.

https://codereview.chromium.org/1214843002/diff/1/tools/lua/ngrams.lua
File tools/lua/ngrams.lua (right):

https://codereview.chromium.org/1214843002/diff/1/tools/lua/ngrams.lua#newcode6
tools/lua/ngrams.lua:6: -- $ lua tools/lua/ngrams_aggregate.lua
Everything after '&&' gets stripped out by the server, so I removed it.  Turns
out, that's not the only problem.  All instances of '+' also get stripped out. 
I think it's time to consider base64 encoding.  That's super easy in python:
https://docs.python.org/2/library/base64.html, and it looks like it's also
possible in javascript if we wanted to make the web interface a little more
bulletproof: https://developer.mozilla.org/en-US/docs/Web/API/WindowBase64/btoa

https://codereview.chromium.org/1214843002/diff/1/trigger_ct_lua
File trigger_ct_lua (right):

https://codereview.chromium.org/1214843002/diff/1/trigger_ct_lua#newcode21
trigger_ct_lua:21: 'c94a028ff836f8f0af41ec33ceb1f4bc140841bf')
I just hard-coded this for now. Ideally the default would be "most recent".  Not
sure how we'd choose a build otherwise.

https://codereview.chromium.org/1214843002/diff/1/trigger_ct_lua#newcode58
trigger_ct_lua:58: '(on https://valentine.corp.google.com/): ')
This kills the automation.  An alternative is to add a --password_file flag, or
even better would be to support OAuth2, which would have the useful side effect
that we wouldn't have to pass in or figure out the user.

[rmistry, 2015-06-26 20:16:01]

https://codereview.chromium.org/1214843002/diff/1/tools/lua/ngrams.lua
File tools/lua/ngrams.lua (right):

https://codereview.chromium.org/1214843002/diff/1/tools/lua/ngrams.lua#newcode6
tools/lua/ngrams.lua:6: -- $ lua tools/lua/ngrams_aggregate.lua
On 2015/06/26 at 19:58:44, borenet wrote:
> Everything after '&&' gets stripped out by the server, so I removed it.  Turns
out, that's not the only problem.  All instances of '+' also get stripped out. 
I think it's time to consider base64 encoding.  That's super easy in python:
https://docs.python.org/2/library/base64.html, and it looks like it's also
possible in javascript if we wanted to make the web interface a little more
bulletproof: https://developer.mozilla.org/en-US/docs/Web/API/WindowBase64/btoa

You could try escaping it to make it work. I agree that base64 is the right way
to do this. I made you the owner of skia-tree-status-staging which is where I
stage changes to the appengine code. Feel free to make these server side changes
yourself if you prefer.
I hesitate to make many changes right now because the polymerized UI is going to
replace the whole frontend soon.

https://codereview.chromium.org/1214843002/diff/1/trigger_ct_lua
File trigger_ct_lua (right):

https://codereview.chromium.org/1214843002/diff/1/trigger_ct_lua#newcode21
trigger_ct_lua:21: 'c94a028ff836f8f0af41ec33ceb1f4bc140841bf')
On 2015/06/26 at 19:58:45, borenet wrote:
> I just hard-coded this for now. Ideally the default would be "most recent". 
Not sure how we'd choose a build otherwise.

Yes it should be simple to fix this on the frontend to use the latest build.

https://codereview.chromium.org/1214843002/diff/1/trigger_ct_lua#newcode58
trigger_ct_lua:58: '(on https://valentine.corp.google.com/): ')
On 2015/06/26 at 19:58:45, borenet wrote:
> This kills the automation.  An alternative is to add a --password_file flag,
or even better would be to support OAuth2, which would have the useful side
effect that we wouldn't have to pass in or figure out the user.

Yes all of this sounds good :)

[borenet, 2015-06-29 16:27:28]

Okay, fixed most of the issues.  Once
https://codereview.chromium.org/1214113004/ lands, this should work.  My only
remaining big concern is choosing which build to use.  We should either
automatically use the most recent, or we should grab a list of the most recent
builds from the server and present the options to the user.

[borenet, 2015-06-29 18:16:52]

Patch set 4 uses the get_skp_repos endpoint to decide which SKP repository to
use.  Depends on https://codereview.chromium.org/1214213002/

[rmistry, 2015-06-29 18:44:04]

https://codereview.chromium.org/1214843002/diff/50001/tools/lua/trigger_ct_lua
File tools/lua/trigger_ct_lua (right):

https://codereview.chromium.org/1214843002/diff/50001/tools/lua/trigger_ct_lu...
tools/lua/trigger_ct_lua:51: parser.add_argument('script', help='Lua script to
run')
script -> --script ?

https://codereview.chromium.org/1214843002/diff/50001/tools/lua/trigger_ct_lu...
tools/lua/trigger_ct_lua:52: parser.add_argument('--aggregator',
help='Aggregator script')
Specify and check that script and aggregator are required flags?

https://codereview.chromium.org/1214843002/diff/50001/tools/lua/trigger_ct_lu...
tools/lua/trigger_ct_lua:54: parser.add_argument('--email', help='Email address
to send results')
Email address to send results. Will use the output of "git config user.email" if
not specified.

https://codereview.chromium.org/1214843002/diff/50001/tools/lua/trigger_ct_lu...
tools/lua/trigger_ct_lua:56: help='File in which the CT password is stored.')
File in which CT password is stored. Script will prompt for the password if not
specified.

https://codereview.chromium.org/1214843002/diff/50001/tools/lua/trigger_ct_lu...
tools/lua/trigger_ct_lua:58: help='Which set of SKPs to use, eg. "10k", "all"')
all -> All
(This is something I have regretted for a long time. I should have called it 1M)

[borenet, 2015-06-29 19:11:53]

https://codereview.chromium.org/1214843002/diff/50001/tools/lua/trigger_ct_lua
File tools/lua/trigger_ct_lua (right):

https://codereview.chromium.org/1214843002/diff/50001/tools/lua/trigger_ct_lu...
tools/lua/trigger_ct_lua:51: parser.add_argument('script', help='Lua script to
run')
On 2015/06/29 18:44:04, rmistry wrote:
> script -> --script ?

Done. That changes it to optional, so I added 'required=True'.

https://codereview.chromium.org/1214843002/diff/50001/tools/lua/trigger_ct_lu...
tools/lua/trigger_ct_lua:52: parser.add_argument('--aggregator',
help='Aggregator script')
On 2015/06/29 18:44:04, rmistry wrote:
> Specify and check that script and aggregator are required flags?

Done for --script.  --aggregator is intended to be optional.

https://codereview.chromium.org/1214843002/diff/50001/tools/lua/trigger_ct_lu...
tools/lua/trigger_ct_lua:54: parser.add_argument('--email', help='Email address
to send results')
On 2015/06/29 18:44:04, rmistry wrote:
> Email address to send results. Will use the output of "git config user.email"
if
> not specified.

Done.

https://codereview.chromium.org/1214843002/diff/50001/tools/lua/trigger_ct_lu...
tools/lua/trigger_ct_lua:56: help='File in which the CT password is stored.')
On 2015/06/29 18:44:04, rmistry wrote:
> File in which CT password is stored. Script will prompt for the password if
not
> specified.

Done.

https://codereview.chromium.org/1214843002/diff/50001/tools/lua/trigger_ct_lu...
tools/lua/trigger_ct_lua:58: help='Which set of SKPs to use, eg. "10k", "all"')
On 2015/06/29 18:44:04, rmistry wrote:
> all -> All
> (This is something I have regretted for a long time. I should have called it
1M)

Done.

[rmistry, 2015-06-29 19:46:04]

lgtm

[borenet, 2015-06-29 19:46:22]

The CQ bit was checked by borenet@google.com

[commit-bot: I haz the power, 2015-06-29 19:46:33]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1214843002/70001

[commit-bot: I haz the power, 2015-06-29 19:54:34]

Committed patchset #5 (id:70001) as
https://skia.googlesource.com/skia/+/8cd8f9429a8a7d5c09c48f7cc9ab34073e01d945