Migrate to the Google Cloud SDK.

Migrate to the Google Cloud SDK.

The Cloud SDK replaces the App Engine SDK, and we want to migrate to be able to
use the updated documentation and take advantage of new features only available
using the Cloud SDK.

The new SDK has a workspace configuration feature, so after installing, you need
to set the project ID in order to deploy. Running the deployment script will
tell you this as well.
  $ gcloud config set project chromeperf

Running `./bin/deploy` no longer deploys cron, index, and task queue configs.
You must pass the .yaml file(s) as a parameter. Deploying pinpoint is also now
`./bin/deploy pinpoint.yaml`. This is more explicit than the old behavior, and
allows you to update the pinpoint indexes/queues without deploying dashboard.

Everything else works the same as before.


BUG=catapult:#1407

Review-Url: https://codereview.chromium.org/2701113002
Committed: https://chromium.googlesource.com/external/github.com/catapult-project/catapult/+/2242f1ab32bac4e1884b8b3c4de55c787bc5798a

[dtu, 2017-02-18 02:36:02]

dtu@chromium.org changed reviewers:
+ simonhatch@chromium.org, sullivan@chromium.org

[dtu, 2017-02-18 02:37:45]

Description was changed from

==========
Migrate to the Google Cloud SDK.

The Cloud SDK replaces the App Engine SDK, and we want to migrate to be able to
use the updated documentation and take advantage of new features only available
using the Cloud SDK.

The new SDK has a workspace configuration feature, so after installing, you need
to set the project ID in order to deploy. Running the deployment script will
tell you this as well.
  $ gcloud config set project chromeperf

Running `./bin/deploy` no longer deploys index and task queue configurations.
You must pass the .yaml file(s) as a parameter. Deploying pinpoint is also now
`./bin/deploy pinpoint.yaml`. This is more explicit than the old behavior, and
allows you to update the pinpoint indexes/queues without deploying dashboard.

Everything else works the same as before.


BUG=catapult:#1407
==========

to

==========
Migrate to the Google Cloud SDK.

The Cloud SDK replaces the App Engine SDK, and we want to migrate to be able to
use the updated documentation and take advantage of new features only available
using the Cloud SDK.

The new SDK has a workspace configuration feature, so after installing, you need
to set the project ID in order to deploy. Running the deployment script will
tell you this as well.
  $ gcloud config set project chromeperf

Running `./bin/deploy` no longer deploys cron, index, and task queue configs.
You must pass the .yaml file(s) as a parameter. Deploying pinpoint is also now
`./bin/deploy pinpoint.yaml`. This is more explicit than the old behavior, and
allows you to update the pinpoint indexes/queues without deploying dashboard.

Everything else works the same as before.


BUG=catapult:#1407
==========

[dtu, 2017-02-18 03:06:40]

Whoops, hold on, looks like there's still a dependency on the old SDK in
appengine_config.py.

[eakuefner, 2017-02-18 04:08:08]

eakuefner@chromium.org changed reviewers:
+ eakuefner@chromium.org

[eakuefner, 2017-02-18 04:08:08]

One high-level question I have is, should we use the dependency manager to
bootstrap gcloud? I'm fine either way, but it's just a possibility we should
consider. I know gcloud generally requires an installation script to run, but if
it's possible to install the SDK in a self-contained way (as we do with
node/npm) then it might be worth it.

[perezju, 2017-02-20 10:22:26]

perezju@chromium.org changed reviewers:
+ perezju@chromium.org

[perezju, 2017-02-20 10:22:26]

https://codereview.chromium.org/2701113002/diff/1/catapult_build/appengine_de...
File catapult_build/appengine_deploy.py (right):

https://codereview.chromium.org/2701113002/diff/1/catapult_build/appengine_de...
catapult_build/appengine_deploy.py:28: script_path =
_FindScriptInPath(os.path.join('..', 'lib', 'gcloud.py'))
This seems odd. The error message bellow suggests that users are responsible for
making sure that "gcloud.py" is in the PATH; why are we trying to find it in
"../lib/" (and relative to what?)

[sullivan, 2017-02-20 14:46:18]

On 2017/02/18 04:08:08, eakuefner wrote:
> One high-level question I have is, should we use the dependency manager to
> bootstrap gcloud? I'm fine either way, but it's just a possibility we should
> consider. I know gcloud generally requires an installation script to run, but
if
> it's possible to install the SDK in a self-contained way (as we do with
> node/npm) then it might be worth it.

We considered it in the design doc and talked about it in our dashboard sync.
The big problem is that the cloud sdk has an install step with side effects that
are largely unknown and subject to change. You are just supposed to run a binary
that sets it all up. My understanding is that we'd prefer not to have anything
with these type of side effects, but in this case it's unavoidable, so we
require users do it manually to understand that they need to install software to
make this work.

Dave, I think you should probably update the doc and link it in your CL:
https://docs.google.com/document/d/1GLaXuQsc0r6K6WfFk0LC7nGVO-1aExt4PUsBCRXQg...

[sullivan, 2017-02-20 14:58:36]

https://codereview.chromium.org/2701113002/diff/1/catapult_build/appengine_de...
File catapult_build/appengine_deploy.py (right):

https://codereview.chromium.org/2701113002/diff/1/catapult_build/appengine_de...
catapult_build/appengine_deploy.py:30: print 'This script requires the Google
Cloud SDK to be in PATH.'
Can this also link to instructions for installing Google Cloud SDK?

https://codereview.chromium.org/2701113002/diff/1/dashboard/docs/getting-set-...
File dashboard/docs/getting-set-up.md (right):

https://codereview.chromium.org/2701113002/diff/1/dashboard/docs/getting-set-...
dashboard/docs/getting-set-up.md:25: code to be deployed and runs `gcloud app
deploy`. Note that this
Isn't it now `gcloud app deploy <filename.yaml>`? Shouldn't this section also
explain the different yaml files and when you would want to deploy which?

[shatch, 2017-02-20 21:31:01]

https://codereview.chromium.org/2701113002/diff/1/catapult_build/appengine_de...
File catapult_build/appengine_deploy.py (right):

https://codereview.chromium.org/2701113002/diff/1/catapult_build/appengine_de...
catapult_build/appengine_deploy.py:33: subprocess.call([sys.executable,
script_path, 'app', 'deploy',
I forget if we discussed this in last week's sync, should we add --quiet to skip
the prompt?

Also should --no-promote be default? The current deploy script doesn't
auto-promote the deployed version, you typically deploy and then go migrate it.

Lastly, how does "gcloud config set project chromeperf" work with multiple
projects? Like if I wanted to deploy firefighter, do I have to reset the
project? Should we just pass --project=chromeperf by default or set
CLOUDSDK_CORE_PROJECT=chromeperf?

https://codereview.chromium.org/2701113002/diff/1/catapult_build/appengine_de...
catapult_build/appengine_deploy.py:34: '--version', _VersionName()] + args)
I seemed to need to add cwd=temp_dir otherwise my deployed version was a bit
busted.

[dtu, 2017-02-22 01:09:41]

https://codereview.chromium.org/2701113002/diff/1/catapult_build/appengine_de...
File catapult_build/appengine_deploy.py (right):

https://codereview.chromium.org/2701113002/diff/1/catapult_build/appengine_de...
catapult_build/appengine_deploy.py:28: script_path =
_FindScriptInPath(os.path.join('..', 'lib', 'gcloud.py'))
On 2017/02/20 10:22:26, perezju wrote:
> This seems odd. The error message bellow suggests that users are responsible
for
> making sure that "gcloud.py" is in the PATH; why are we trying to find it in
> "../lib/" (and relative to what?)

Done. I've changed it to look for 'gcloud.cmd' on Windows and 'gcloud'
otherwise. Originally, I had it call the Python executable
(google-cloud-sdk/lib/gcloud.py instead of google-cloud-sdk/bin/gcloud) to avoid
dealing with the platform differences.

https://codereview.chromium.org/2701113002/diff/1/catapult_build/appengine_de...
catapult_build/appengine_deploy.py:30: print 'This script requires the Google
Cloud SDK to be in PATH.'
On 2017/02/20 14:58:36, sullivan wrote:
> Can this also link to instructions for installing Google Cloud SDK?

Done.

https://codereview.chromium.org/2701113002/diff/1/catapult_build/appengine_de...
catapult_build/appengine_deploy.py:33: subprocess.call([sys.executable,
script_path, 'app', 'deploy',
On 2017/02/20 21:31:01, shatch wrote:
> I forget if we discussed this in last week's sync, should we add --quiet to
skip
> the prompt?
> 
> Also should --no-promote be default? The current deploy script doesn't
> auto-promote the deployed version, you typically deploy and then go migrate
it.
> 
> Lastly, how does "gcloud config set project chromeperf" work with multiple
> projects? Like if I wanted to deploy firefighter, do I have to reset the
> project? Should we just pass --project=chromeperf by default or set
> CLOUDSDK_CORE_PROJECT=chromeperf?

--quiet: No, deployment is fast enough that it should be fine with the prompt.
And I like the clarity of seeing exactly what you're deploying.

--no-promote: Done. Sorry, I thought that was the default behavior. They keep
changing it.

--project=chromeperf: gcloud has the ability to have multiple configs and switch
between them, though it is sort of cumbersome. Firefighter is going to go away
eventually, so we won't run into project conflicts often. I think hard-coding
the project ID would be a bad open source practice -- anyone who uses this
project would have to have a local branch with their own project ID.

https://codereview.chromium.org/2701113002/diff/1/catapult_build/appengine_de...
catapult_build/appengine_deploy.py:34: '--version', _VersionName()] + args)
On 2017/02/20 21:31:01, shatch wrote:
> I seemed to need to add cwd=temp_dir otherwise my deployed version was a bit
> busted.

Done.

https://codereview.chromium.org/2701113002/diff/1/dashboard/docs/getting-set-...
File dashboard/docs/getting-set-up.md (right):

https://codereview.chromium.org/2701113002/diff/1/dashboard/docs/getting-set-...
dashboard/docs/getting-set-up.md:25: code to be deployed and runs `gcloud app
deploy`. Note that this
On 2017/02/20 14:58:36, sullivan wrote:
> Isn't it now `gcloud app deploy <filename.yaml>`? Shouldn't this section also
> explain the different yaml files and when you would want to deploy which?

Done. (`/bin/deploy` with no parameters deploys the default service.)

[shatch, 2017-02-22 01:32:28]

lgtm on my end

[sullivan, 2017-02-22 14:43:35]

lgtm

[dtu, 2017-02-22 17:24:01]

The CQ bit was checked by dtu@chromium.org

[commit-bot: I haz the power, 2017-02-22 17:24:08]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-22 17:27:37]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-22 17:27:38]

Try jobs failed on following builders:
  Catapult Mac Tryserver on master.tryserver.client.catapult (JOB_FAILED,
https://build.chromium.org/p/tryserver.client.catapult/builders/Catapult%20Ma...)

[dtu, 2017-02-22 19:42:18]

The CQ bit was checked by dtu@chromium.org

[dtu, 2017-02-22 19:42:18]

The patchset sent to the CQ was uploaded after l-g-t-m from
sullivan@chromium.org, simonhatch@chromium.org
Link to the patchset: https://codereview.chromium.org/2701113002/#ps40001
(title: "Work for both old and new SDKs.")

[commit-bot: I haz the power, 2017-02-22 19:42:26]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-22 20:04:11]

CQ is committing da patch.

Bot data: {"patchset_id": 40001, "attempt_start_ts": 1487792538337760,
"parent_rev": "8c5f79979fb59652f150d78bdb2b78c51f2d12fd", "commit_rev":
"2242f1ab32bac4e1884b8b3c4de55c787bc5798a"}

[commit-bot: I haz the power, 2017-02-22 20:04:14]

Description was changed from

==========
Migrate to the Google Cloud SDK.

The Cloud SDK replaces the App Engine SDK, and we want to migrate to be able to
use the updated documentation and take advantage of new features only available
using the Cloud SDK.

The new SDK has a workspace configuration feature, so after installing, you need
to set the project ID in order to deploy. Running the deployment script will
tell you this as well.
  $ gcloud config set project chromeperf

Running `./bin/deploy` no longer deploys cron, index, and task queue configs.
You must pass the .yaml file(s) as a parameter. Deploying pinpoint is also now
`./bin/deploy pinpoint.yaml`. This is more explicit than the old behavior, and
allows you to update the pinpoint indexes/queues without deploying dashboard.

Everything else works the same as before.


BUG=catapult:#1407
==========

to

==========
Migrate to the Google Cloud SDK.

The Cloud SDK replaces the App Engine SDK, and we want to migrate to be able to
use the updated documentation and take advantage of new features only available
using the Cloud SDK.

The new SDK has a workspace configuration feature, so after installing, you need
to set the project ID in order to deploy. Running the deployment script will
tell you this as well.
  $ gcloud config set project chromeperf

Running `./bin/deploy` no longer deploys cron, index, and task queue configs.
You must pass the .yaml file(s) as a parameter. Deploying pinpoint is also now
`./bin/deploy pinpoint.yaml`. This is more explicit than the old behavior, and
allows you to update the pinpoint indexes/queues without deploying dashboard.

Everything else works the same as before.


BUG=catapult:#1407

Review-Url: https://codereview.chromium.org/2701113002
Committed:
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...
==========

[commit-bot: I haz the power, 2017-02-22 20:04:15]

Committed patchset #3 (id:40001) as
https://chromium.googlesource.com/external/github.com/catapult-project/catapu...