deploytool: Add README.md, migrate docs to it.

deploytool/README.md

Index: deploytool/README.md
diff --git a/deploytool/README.md b/deploytool/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..82aa4dd71b4cdc5d18edf6d23a5106eabac76b21
--- /dev/null
+++ b/deploytool/README.md
@@ -0,0 +1,478 @@
+The LUCI cloud services deployment tool
+=======================================
+
+The LUCI project hosts code on Google AppEngine, Google Container Engine (via
+[Kubernetes](http://kubernetes.io/)), and several other locations. The code
+layout is complex and modular, which is oftentimes at odds with the layout
+requirements of supported deployment tools such as `gcloud` and `docker`. The
+`luci_deploy` attempts to smooth that over by providing:
+
+* A common declarative deployment configuration language.
+* Hermetic, pinned source checkouts.
+* A single tool capable of organizing and executing deployment programs.
+* A utility that facilitates deployment management given its understanding of
+  the deployment parameters.
+
+While much of the underlying code in the deployment tool is cross-platform, it
+is specifically targeting Linux support. Any deployment from other platforms is,
+at this time, unsupported.
+
+The deployment tool doesn't reimplement all of the functionality of existing
+tools; instead, it manages configurations and layouts and employs existing
+tooling to perform the actual deployment operations. Some tools that it uses
+are:
+* [gcloud](https://cloud.google.com/sdk/gcloud/), the Google Cloud SDK tool.
+* [kubectl](http://kubernetes.io/docs/user-guide/kubectl-overview/), the
+  Kubernetes control tool.
+* [aedeploy](https://godoc.org/google.golang.org/appengine/cmd/aedeploy),
+  a tool which collects `GOPATH` packages for Docker deployment.
+* [docker](https://www.docker.com/), a container management system used by
+  Kubernetes.
+
+Deployment configuration for a given deployable component is stored in two
+places:
+
+1. Generic component configuration and requirements are stored alongside the
+   component in its source repository.
+1. Deployment-specific parameters are stored in a separate repository, and
+   project the generic component layout into a product space.
+
+The component-specific configuration details which resources the component
+needs, which services it employs, and its relationship with other components.
+The deployment projection then takes that generic configuration and applies it
+to a set of specific deployment parameters. For example:
+
+* One deployment may project a component into a production environment,
+  allocating expensive CPU resources.
+* A development deployment may project the same component into a development
+  environment.
+* A staging deployment may project the same component into a staging
+  enviornment alongside other components from other projects.
+
+
+## Overview and Terminology
+
+Deployment configuration uses the following concepts:
+
+* A **source** is a specific named repository checked out at a specific
+  revision.
+* A **source group** is a collection of **source** repositories.
+* A **component** is a single buildable and deployable entity. Its configuration
+  resides in a specific **source**.
+* An **application** is a collection of deployable **component** pointers. Each
+  pointer is a subpath to the **component** configuration within its **source**.
+* A **resource** is a global cloud resource (e.g., cloud project, Kubernetes
+  cluster, etc.).
+* A **deployment** binds an **application** to a **source group** (and,
+  consequently, a specific set of **sources**) and a set of **resources**.
+
+During deployment, several operations are performed:
+
+* The **working directory** is used to contain files generated and managed by
+  the `luci_deploy`.
+* The **checkout** is a read-only directory containing all of the **sources**
+  checked out at their specific revisions and initialized.
+* The **staging** directory is a constructed environment containing deployment
+  and component sources and generated files organized such that deployment tools
+  can operate on them. See [Staging](#staging) for more information.
+
+A user will first perform a `checkout`, which loads all of the configured
+**sources** at their specified revisions into the working directory. Afterwards,
+the user performs operations on the checkout:
+* `deploy`, which deploy the configured **deployments** and their
+  composite **components**.
+* `manage`, which offers component management utilities based on the
+  deployment configuration.
+
+
+## Deployment Layout
+
+The deployment layout is a filesystem-based configuration structure that
+defines the parameters of the configured deployments. Unless stated otherwise,
+all layout component names may only contain:
+- Alphanumeric characters (`[0-9a-zA-Z]`)
+- A hyphen (`-`).
+
+Configuration files are text-encoded protobufs. All protobufs used by the
+`luci_deploy` are defined in its [api directory](api/deploy). Text protobuf
+configuration files have the ".cfg" suffix. The protobuf messages used in the
+layout are defined in [config.proto](api/deploy/config.proto).
+
+The default deployment layout consists of a root `layout.cfg` file and several
+subdirectories defining **source groups**, **applications**, and
+**deployments**. Note that the layout depicted below is the *default* layout;
+the specific config directory paths may be overridden in `layout.cfg`.
+
+```
+/layout.cfg
+/sources/
+  <source-group-name>/     (Defines a source group).
+    <source-name>.cfg      (Defines a source within the source group).
+
+/applications/
+  <application-name>.cfg   (Defines a application).
+
+/deployments/
+  <deployment-name>.cfg    (Defines a deployment).
+```
+
+All configuration files 
+* `layout.cfg`, a text protobuf file containing a `Layout` message.
+* `<source-group-name>`, which defines a source group.
+* `<source-name>.cfg`, which defines a `Source` message within its
+  **source group** parent directory.
+* `<application-name>.cfg`, which defines an `Application` message.
+* `<deployment-name>.cfg`, which defines a `Deployment` message.
+
+## Sources
+
+**Sources** define a fully-specified repository in which **component**
+configuration and data are stored. `luci_deploy` will manage the source
+checkouts configured in the deployment layout.
+
+A **source** is a file located within a source group directory named
+`<source-name>.cfg`. The source is specified using the `Source` text protobuf,
+defined in [config.proto](api/deploy/config.proto). Each **source** name is
+unique within its **source group**, but **source** names can (and likely will)
+be re-used across other **source groups** to enable deployment tracks (see
+[Source Group Versioning](#source-group-versioning)).
+
+Each **source** checkout is read-only, meaning that after the `checkout`
+operation is complete, no other deployment operations will modify its contents.
+
+### Source Initialization
+
+**Sources** may require additional post-checkout steps to render them usable.
+Such sources can offer initialization scripts that will be run by `luci_deploy`
+during the `checkout` phase after the source has been checked out.
+
+Initialization scripts are specified by adding a `SourceLayout` text protobuf
+file called `luci-deploy.cfg`, to the root of the **source**. If present,
+this file will be interpreted, and any initialization options present will be
+included as part of the **source**'s checkout operation.
+
+The `SourceLayout` protobuf is defined in
+[checkout.proto](api/deploy/checkout.proto).
+
+Note that the **source**'s definition *must* have `run_scripts` set to true in
+order for initialization scripts to be executed. This is a security precaution
+to prevent untrusted **source** repositories from adding and executing
+`luci_deploy` commands without the user's permission.
+
+### Source Group Versioning
+
+**Applications** bind components to source *name*. Deployments then bind those
+**applications** to **source groups**. Therefore, the specific **source** that
+is used in a deployment is determined by the **deployment**'s
+choice of **source group**.
+
+An example is a directory layout with two source groups, one named `canary` and
+one named `production`.
+
+```
+sources/canary/base.cfg
+sources/production/base.cfg
+```
+
+The source named `base` is defined in both **source groups** is then used in
+an **application**. At this point, the **application's** specific **source**
+is not known, since the choice of *which* `base` to use requires a **source
+group** to be selected. In other words, the **application** is bound to
+`sources/*/base.cfg`, and the choice of which `*` to use is left to a specific
+**deployment**.
+
+A *canary* deployment can bind the **application** to the `canary` **source
+group**, causing its **components** to be loaded from the version of `base`
+defined in `sources/canary/base.cfg`. A *production* deployment can bind the
+same **application** to the `production` **source group**, causing its
+**components** to be loaded from `sources/production/base.cfg`.
+
+### Source-Relative Paths
+
+Paths referenced within a source are referenced using "source-relative paths".
+These paths are:
+* Operating system independent, as they all use "/" as their delimiter.
+* Either **absolute** (starting with a "/") or **relative** (not starting with
+  a "/"). Relative paths are relative to the configuration file in which they
+  are specified.
+
+## Checkout
+
+Prior to deployment, a user must sync the checked out sources with the
+configured sources using the `checkout` sub-command. This checks out *all*
+sources defined in the deployment layout and runs any configured initialization
+scripts to initialize them.
+
+A `checkout` is a manual operation. If the configured **sources** change, a new
+`checkout` operation must be performed to sync the on-disk checkout with the
+configured sources.
+
+Performing a checkout is simple:
+
+```shell
+$ luci_deploy checkout
+```
+
+### Local Checkout
+
+It is oftentimes useful for a user to stage (for testing) or deploy (for triage)
+code from their local checkout. Rather than editing the **source** files to use
+`file:///` URLs, the user may define a set of repository overrides in their
+[User Configuration](#user-configuration). Running the `checkout` sub-command
+with the `--local` flag will cause the checkout to prefer the user configuration
+repositroies to those configured in the **sources**.
+
+```shell
+$ luci_deploy checkout --local
+```
+
+**NOTE**: The checkout will continue to use the local overrides until the
+`checkout` sub-command is re-run without the `--local` flag. However, also note
+that a checkout containing a local override is considered *tainted*.
+
+## Deployment
+
+Deployment is accessed using the `deploy` sub-command. A user may deploy a
+single component or a full deployment. All deployment operations are run in
+stages, and, within each stage, in parallel.
+
+```shell
+# Deploy all Components within a Deployment.
+$ luci_deploy deploy mydeployment
+
+# Deploy a specific set of Components.
+$ luci_deploy deploy mydeployment/component-a mydeployment/component-b
+```
+
+The `deploy` operation consists of several sub-stages:
+1. `stage`, where deployable **components** are generated from their sources and
+  configurations. See [Staging](#staging) for more information.
+1. `localbuild`, where any local build operations are performed on the staged
+  **components**. For some component types, this doubles as a sanity check prior
+  to engaging remote services.
+1. `push`, where **components** are uploaded to their remote platforms.
+1. `commit`, where the uploaded **components** are activated and become live.
+
+For testing and debugging purposes, a deployment can be stopped prior to a full
+commit by using the `--stage` parameter. For example, to assert that all
+**components** in a deployment can be successfully staged and locally built, a
+user may run:
+
+```shell
+$ luci_deploy deploy --stage=localbuild mydeployment
+```
+
+## Component Configuration
+
+A **component** is a single buildable/deployable unit. **Components** are
+defined in **application** configuration files, and are specified as paths to
+`Component` messages within a **source**. `Component` messages are defined in
+[component.proto](api/deploy/component.proto).
+
+**Components** are intentionally defined within a **source**, as opposed to the
+deployment layout, because their specific composition and resource requirements
+will be versioned alongside their deployable code. For example, different
+versions of a Google AppEngine app may define different datastore indexes based
+on the underlying functionality of their code.
+
+## User Configuration
+
+The user may include a configuration file in their home directory at
+`~/.luci_deploy.cfg`. If present, this file will be loaded alongside the
+layout configuration and incorporated into `luci_deploy` behavior.
+
+The user configuration file is a `UserConfig` text protobuf defined in
+[userconfig.proto](api/deploy/userconfig.proto).
+
+## Staging
+
+A staging space is created for each deployable Component. Staging offers
+`luci_deploy` an isolated canvas with which it can the filesystem layouts for
+that Component's actual deployment tooling to operate. All file operations,
+generation, and structuring are performed in the staging state such that the
+resulting staging directory is available for tooling or humans to use.
+
+The staging space for a given Component depends on that Component's type.
+A staging layout is intended to be human-navigatable while conforming to any
+layout requirements imposed by that Component's deployment tooling.
+
+One goal that is enforced is that the actual checkout directories used by
+any given Component are considered read-only. This means that staging for
+Components which require generated files to exist alongside source must
+copy or mirror that source elsewhere. Some of the more convoluted aspects of
+staging layouts are the result of this requirement.
+
+### AppEngine
+
+AppEngine deployments are composed of two sets of information:
+* Individual AppEngine modules, including the default module.
+* AppEngine Project-wide globals such as Index, Cron, Dispatch, and Queue
+  settings.
+
+The individual Components are staged and deployed independently. The globals
+are composed of their respective settings in each individual Component
+associated with the AppEngine project regardless of whether that Component is
+actually being deployed. The aggregate globals are re-asserted once per
+cloud project at the end of module deployment.
+
+References to static content are flattened into static directories within the
+staging area. The generated YAML files are configured to point to this
+flattened space regardless of the original static content's location within
+the source.
+
+#### dev_appserver
+
+Moving Component configuration into protobufs and constructing composite
+GOPATH and generated configuration files prevents AppEngine tooling from
+working out of the box in the source repository.
+
+The offered solution is to manually stage the Components under test, then
+run tooling against the staged Component directories. The user may optionally
+install the staged paths (GOPATH, etc.) or use their default environment's
+paths.
+
+One downside to this solution is that staging operates on a snapshot of the
+repository, meaning that changes to the repository won't be reflected in the
+staged environment. The user can address this by either manually re-staging
+the Deployment when a file changes. The user may also structure their
+application such that the staged content doesn't change frequently (e.g.,
+for Go, have a simple entry point that immediately imports the main app
+logic). Specific structures depend on the type of Component and how it is
+staged (see below).
+
+In the future, a command to bootstrap "dev_appserver" through the staged
+environment would be useful.
+
+#### Go on Classic AppEngine
+
+Go Classic AppEngine Components are deployed using the `appcfg.py` tool,
+which is the fastest available method to deploy such applications.
+
+Because the "app.yaml" file must exist alongside the deployed source, a
+stub entry point is generated in the staging area. This stub simply imports
+the actual entry point package.
+
+The Component's Sources which declare GOPATH presence are combined in a
+virutal GOPATH within the Component's staging area. This GOPATH is then
+installed and `appcfg.py`'s "update" method is invoked to upload the
+Component.
+
+#### Go on Managed VM
+
+Go AppEngine Managed VMs use a set of tools for deployment:
+* `aedeploy`, an AppEngine project tool which copies the various referenced
+  sources across GOPATH entries into a single GOPATH hierarchy for Docker
+  isolation.
+* `gcloud`, which engages the remote AppEngine service and offers deployment
+  utility.
+* Behind the scenes, `gcloud` uses `docker` to build the actual deployed
+  image from the source (`aedeploy` target) and assembled GOPATH.
+
+Because the "app.yaml" file must exist alongside the entry point code, the
+contents of the entry package are copied (via symlink) into a generated
+entry point package in the staging area.
+
+The Component's Sources which declare GOPATH presence are combined in a
+virutal GOPATH within the Component's staging area. This GOPATH is then
+collapsed by `aedeploy` when the Managed VM image is built.
+
+ NOTE: because the entry point is actually a clone of the entry point
+ package, "internal/" imports will not work. This can be easily worked around
+ by having the entry point import another non-internal package within the
+ project, and having that package act as the actual entry point.
+
+#### Static Content Module
+
+The `luci_deploy` supports the concept of a static content module. This is an
+AppEngine module whose sole purpose is to, via AppEngine handler definitions,
+map to uploaded static content. The utility of a static module is that it can
+be effortlessly updated without impacting actual AppEngine runtime processes.
+Static modules can be used in conjunction with module-referencing handlers in
+the default AppEngine module to create the effect of the default module
+actually hosting the static content.
+
+Static content modules are staged like other AppEngine modules, only with
+no running code.
+
+### Container Engine / Kubernetes
+
+The `luci_deploy` supports depoying services to Google Container Engine, which
+is backed by Kubernetes. A project's Container Engine configuration consists
+of a series of Container Engine clusters, each of which hosts a series of
+homogenous machines. Kubernetes Pods, each of which are composed of one or
+more Kubernetes Components (i.e., Docker images), are deployed to one or more
+Google Container Engine Clusters.
+
+The Deployment Component defines a series of Container Engine Pods, an
+amalgam of a Kubernetes Pod definition and Container Engine requirements of
+that Kubernetes Pod. Each Kubernetes Pod's Component is built in its own
+staging area and deployed as part of that Pod to one or more Container Engine
+clusters.
+
+The Build phase of Container Engine deployment constructs a local Docker
+image of each Kubernetes Component. The Push phase pushes those images to
+the remote Docker image service. The Commit phase enacts the generated
+Kubernetes/ configuration which references those images on the Container
+Engine configuration.
+
+Container Engine management is done in two layers: firstly, the Container
+Engine configuration is managed with `gcloud` commands. This configures:
+
+* Which managed Clusters exist on Google Container Engine.
+* What scopes, system specs, and node count each Cluster has.
+
+Within a Cluster, several managed Kubernetes pods are deployed. The
+deployment is done using the `kubectl` tool, selecting the cluster using
+the "--context" flag to select the `gcloud`-generated Kubernetes context.
+
+Each `luci_deploy` Component (Kubernetes Pod, at this level) is managed as a
+Kubernetes Deployment (http://kubernetes.io/docs/user-guide/deployments/). A
+`luci_deploy`-managed Deployment will have the following metadata annotations:
+
+* "luci.managedBy", set to "luci-deploytool"
+* "luci.deploytool/version" set to the `deploytool` Deployment's version
+  string.
+* "luci.deploytool/sourceVersion" set to the revision of the Deployment
+  Component's source.
+
+The Kubernetes Deployment for a Component will be named
+"<project>--<component>".
+
+`luci_deploy`-driven Kubernetes depoyment is fairly straightforward:
+* Use "kubectl get deployments/<name>" to get the current Deployment state.
+* If there is a current Deployment,
+  * If its "luci.managedBy" annotation doesn't equal "luci-deploytool",
+    fail. The user must manually correct this situation.
+  * Check if the "luci.deploytool/version" matches the container version.
+    If it does, succeed.
+* Create/update the Deployment's configuration using "kubectl apply".
+
+#### Go Container Engine Pod Components
+
+Go Container Engine Pods use a set of tools for deployment:
+* `aedeploy`, an AppEngine project tool which copies the various referenced
+  sources across GOPATH entries into a single GOPATH hierarchy for Docker
+  isolation.
+* `gcloud`, which engages the remote AppEngine service and offers deployment
+  utility.
+* `docker`, which is used to build and manage the Docker images.
+
+The Component's Sources which declare GOPATH presence are combined in a
+virutal GOPATH within the Component's staging area. This GOPATH is then
+collapsed by `aedeploy` when the Docker image is built.
+
+The actual Component is built directly from the Source using `aedeploy` and
+`docker build`. This is acceptable, since this is a read-only operation and
+will not modify the Source.
+
+## To Do
+
+Following are some ideas of "planned" features that would be useful to add
+to `luci_deploy`:
+* Add a "manage" command to manage a specific Deployment and/or Component.
+  Each invocation would load special set of subcommands based on that
+  resource:
+  * If the resource is a Deployment, query status?
+  * Other common automatable management macros.
+* Offer a `dev_appserver` fallthrough to create a staging area and
+  bootstrap `dev_appserver` for the named staged AppEngine Components.