BuildBucket-based build triggering

BuildBucket-based build triggering

If bucket is specified explicitly, or the triggering build was scheduled
through buildbucket, STEP_TRIGGER annotations schedule a build on
buildbucket.

R=stip@chromium.org
BUG=449218
Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=294333

[nodir, 2015-03-03 01:11:21]

Patchset #1 (id:1) has been deleted

[nodir, 2015-03-03 01:18:52]

Patchset #1 (id:20001) has been deleted

[nodir, 2015-03-03 01:18:57]

Patchset #1 (id:40001) has been deleted

[nodir, 2015-03-03 01:19:06]

PTAL

[nodir, 2015-03-03 01:23:08]

Example of log:

@@@STEP_TRIGGER@{"bucket": "test", "builderNames": ["test builder"], "changes":
[], "properties": null}@@@

@@@STEP_CURSOR trigger test builder@@@

@@@STEP_CURSOR trigger test builder@@@

Will wait till async operations complete:
* Triggering build(s) on test builder
BuildBucket.put API response: {
    "kind": "buildbucket#resourcesItem", 
    "etag": "\"hG6O3EN3PvwddEuR-5vfnJdD8mk/42rb2GSvE-lCKe4GF15PKmrDZHA\"", 
    "build": {
        "status": "SCHEDULED", 
        "created_ts": "1425344034555390", 
        "status_changed_ts": "1425344034555420", 
        "result_details_json": "null", 
        "bucket": "test", 
        "created_by":
"user:1037694916935-05jtf550tv8fihsqujvmhkn3bnb73tdp@developer.gserviceaccount.com",

        "updated_ts": "1425344034555370", 
        "parameters_json": "{\"builder_name\": \"test builder\", \"changes\":
[], \"properties\": {}}", 
        "id": "9052412169580940032"
    }
}

--------------------------------------------------------------------------------
started: Mon Mar  2 16:53:53 2015
ended: Mon Mar  2 16:53:55 2015
duration: 1 secs
status: SUCCESS
status reason:

[nodir, 2015-03-03 17:09:10]

Patchset #2 (id:80001) has been deleted

[nodir, 2015-03-03 17:09:48]

Added build_url to the log

[ghost stip (do not use), 2015-03-04 23:07:31]

lgtm with some changes

https://chromiumcodereview.appspot.com/968053003/diff/60001/scripts/master/bu...
File scripts/master/buildbucket/__init__.py (right):

https://chromiumcodereview.appspot.com/968053003/diff/60001/scripts/master/bu...
scripts/master/buildbucket/__init__.py:28: from . import trigger
have we always used relative imports here? seems weird

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/b...
File scripts/master/buildbucket/client.py (right):

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/b...
scripts/master/buildbucket/client.py:29: def
get_default_buildbucktet_hostname(master):
get_default_buildbucket_hostname

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/b...
scripts/master/buildbucket/client.py:32: else BUILDBUCKET_HOSTNAME_TESTING)
what happens when a master is started as part of masters_test? will it load?

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/b...
scripts/master/buildbucket/client.py:42: hostname = hostname or
get_default_buildbucktet_hostname(master)
same here

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/b...
File scripts/master/buildbucket/trigger.py (right):

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/b...
scripts/master/buildbucket/trigger.py:57: 'Neither bucket_name is specified, nor
the build is from buildbucket')
this causes an exception in the build and doesn't kill the master, right?

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/c...
File scripts/master/chromium_step.py (right):

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/c...
scripts/master/chromium_step.py:1203: d =
self.triggerBuildsAsBuildsets(builder_names, properties, changes)
can you rename this to triggerBuildsLocally() instead?

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/c...
scripts/master/chromium_step.py:1317: if not self.bb_triggering_service:
ok, so the service is only instantiated when something is actually triggered

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/c...
scripts/master/chromium_step.py:1334: def triggerBuildsAsBuildsets(
triggerBuildsLocally

[nodir, 2015-03-04 23:51:26]

The CQ bit was checked by nodir@chromium.org

[nodir, 2015-03-04 23:51:26]

The patchset sent to the CQ was uploaded after l-g-t-m from stip@chromium.org
Link to the patchset: https://chromiumcodereview.appspot.com/968053003/#ps120001
(title: "re stip")

[nodir, 2015-03-04 23:52:10]

Thanks

https://chromiumcodereview.appspot.com/968053003/diff/60001/scripts/master/bu...
File scripts/master/buildbucket/__init__.py (right):

https://chromiumcodereview.appspot.com/968053003/diff/60001/scripts/master/bu...
scripts/master/buildbucket/__init__.py:28: from . import trigger
On 2015/03/04 23:07:30, stip wrote:
> have we always used relative imports here? seems weird

I understand why using absolute imports is better: if your the importing modules
moves, all imports would have to be modified. This is not the case for
inseparable files, like all files in the buildbucket module. They are likely to
be moved together than independently.

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/b...
File scripts/master/buildbucket/client.py (right):

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/b...
scripts/master/buildbucket/client.py:29: def
get_default_buildbucktet_hostname(master):
On 2015/03/04 at 23:07:30, stip wrote:
> get_default_buildbucket_hostname

Done

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/b...
scripts/master/buildbucket/client.py:32: else BUILDBUCKET_HOSTNAME_TESTING)
On 2015/03/04 at 23:07:30, stip wrote:
> what happens when a master is started as part of masters_test? will it load?

Tests don't fail: before it reaches this point, it will probably complain that
the service account file is not found
(/creds/service_account/service-account-X.json). If it is there, then it will
try to peek builds from cr-buildbucket-test.appspot.com

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/b...
scripts/master/buildbucket/client.py:42: hostname = hostname or
get_default_buildbucktet_hostname(master)
On 2015/03/04 at 23:07:30, stip wrote:
> same here

Thanks

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/b...
File scripts/master/buildbucket/trigger.py (right):

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/b...
scripts/master/buildbucket/trigger.py:57: 'Neither bucket_name is specified, nor
the build is from buildbucket')
On 2015/03/04 at 23:07:30, stip wrote:
> this causes an exception in the build and doesn't kill the master, right?

it doesn't kill the master, but calls errback. In general, it is hard to kill
master, unhandled exceptions are normally thrown to the log.

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/c...
File scripts/master/chromium_step.py (right):

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/c...
scripts/master/chromium_step.py:1203: d =
self.triggerBuildsAsBuildsets(builder_names, properties, changes)
On 2015/03/04 at 23:07:31, stip wrote:
> can you rename this to triggerBuildsLocally() instead?

Done

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/c...
scripts/master/chromium_step.py:1317: if not self.bb_triggering_service:
On 2015/03/04 at 23:07:30, stip wrote:
> ok, so the service is only instantiated when something is actually triggered

Yes, when something is actually triggered through buildbucket

https://chromiumcodereview.appspot.com/968053003/diff/100001/scripts/master/c...
scripts/master/chromium_step.py:1334: def triggerBuildsAsBuildsets(
On 2015/03/04 at 23:07:30, stip wrote:
> triggerBuildsLocally

Done

[nodir, 2015-03-04 23:52:35]

Could you please also review this final piece, it is trivial:
https://codereview.chromium.org/973703002

[commit-bot: I haz the power, 2015-03-04 23:53:09]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/968053003/120001

[commit-bot: I haz the power, 2015-03-04 23:56:57]

Committed patchset #3 (id:120001) as
http://src.chromium.org/viewvc/chrome?view=rev&revision=294333