| OLD | NEW |
|---|
| (Empty) |
| 1 # Bootstrapping the Chromium Infra Repo | |
| 2 | |
| 3 The infra/infra repo uses python [wheel files][1], [virtualenv][2] and [pip][3] | |
| 4 to manage dependencies. The process for bootstrapping these is contained | |
| 5 entirely within the bootstrap directory. | |
| 6 | |
| 7 1: http://legacy.python.org/dev/peps/pep-0427/ | |
| 8 2: https://github.com/pypa/virtualenv | |
| 9 3: https://github.com/pypa/pip | |
| 10 | |
| 11 | |
| 12 ## TL;DR - Workflows | |
| 13 | |
| 14 ### Setting up the env with already-built-deps | |
| 15 gclient sync # OR | |
| 16 gclient runhooks | |
| 17 | |
| 18 ### Adding a new dep | |
| 19 Say we want to add a stock `my_pkg` python package at version 1.2.3 | |
| 20 | |
| 21 ``` | |
| 22 # If it comes from a tarball | |
| 23 $ ./bootstrap/ingest_source.py <tarball> | |
| 24 ... | |
| 25 deadbeefdeadbeefdeadbeefdeadbeef.tar.gz | |
| 26 ``` | |
| 27 | |
| 28 # If it comes from a repo | |
| 29 # File a ticket to have it mirrored (no matter what VCS!) to | |
| 30 # `chromium.googlesource.com/infra/third_party/my_pkg` | |
| 31 # Grab the git commit hash of the commit to build: | |
| 32 # badc0ffeebadc0ffeebadc0ffeebadc0 | |
| 33 | |
| 34 Then add the actual dep | |
| 35 | |
| 36 ``` | |
| 37 $ vim bootstrap/deps.pyl # add a new entry (see `deps.pyl` section) | |
| 38 ... | |
| 39 'my_pkg' : { | |
| 40 'version': '1.2.3', | |
| 41 'build': 0, # This is the first build | |
| 42 'gs': 'deadbeefdeadbeefdeadbeefdeadbeef.tar.gz', # if tarball | |
| 43 'rev': 'badc0ffeebadc0ffeebadc0ffeebadc0', # if repo | |
| 44 } | |
| 45 ... | |
| 46 ``` | |
| 47 | |
| 48 Then build it | |
| 49 | |
| 50 ``` | |
| 51 $ ./bootstrap/build_deps.py | |
| 52 # builds and uploads my_pkg-1.2.3-0_deadbeef...-....whl to google storage | |
| 53 ``` | |
| 54 | |
| 55 If your dep is not pure-python, you will have to run `build_deps.py` for each | |
| 56 platform. | |
| 57 | |
| 58 | |
| 59 ### If your dep needs special treatment | |
| 60 Do everything in the 'Adding a new dep' section, but before running | |
| 61 `build_deps.py`, add a file `bootstrap/custom_builds/{wheel package name}.py`. | |
| 62 This file is expected to implement: | |
| 63 | |
| 64 `def Build(source_path, wheelhouse_path)` | |
| 65 | |
| 66 See 'custom builds' for more detail. | |
| 67 | |
| 68 | |
| 69 ## `bootstrap.py` (a.k.a. "I just want a working infra repo!") | |
| 70 Run `gclient runhooks`. Under the hood, this is running | |
| 71 `./bootstrap/bootstrap.py --deps_file bootstrap/deps.pyl`. | |
| 72 | |
| 73 This creates a virtualenv called `{repo_root}/ENV` with all the deps contained | |
| 74 in `bootstrap/deps.pyl`. You must be online, or must already have the wheels for | |
| 75 your system cached in `{repo_root}/.wheelcache`. | |
| 76 | |
| 77 If you already have an ENV, `bootstrap.py` will check the manifest in ENV to | |
| 78 see if it matches `deps.pyl` (i.e. the diff is zero). If it's not, then ENV | |
| 79 will be re-created _from scratch_. | |
| 80 | |
| 81 `{repo_root}/run.py` will automatically use the environment ENV. It is an error | |
| 82 to use run.py without first setting up ENV. | |
| 83 | |
| 84 | |
| 85 ## `deps.pyl` | |
| 86 This file is a python dictionary containing the exact versions of all python | |
| 87 module dependencies. These versions are the standard upstream package versions | |
| 88 (e.g. '0.8.0'), plus the commit hash or sha1.{ext} of an ingested source bundle | |
| 89 (see `inject_source.py`). | |
| 90 | |
| 91 The format of this file is `{'package_name': <values>}`. This file is a python | |
| 92 [ast literal][4], so comments are allowed and encouraged. | |
| 93 | |
| 94 Values are: | |
| 95 * version: The pip version of the module | |
| 96 * build: An integer representing which build of this version/hash. If you | |
| 97 modify the _way_ that a requirement is built, but not the source hash, you | |
| 98 can bump the build number to get a new pinned dependency. | |
| 99 | |
| 100 And either: | |
| 101 * rev: The revision or sha1 of the source for this module. | |
| 102 * The repo is | |
| 103 'git+https://chromium.googlesource.com/infra/third_party/{package_name}' | |
| 104 * gs: '{sha1}.{ext}' indicates file | |
| 105 'gs://chrome-infra-wheelhouse/sources/{sha1}.{ext}'. The sha1 will be | |
| 106 checked against the content of the file. | |
| 107 | |
| 108 And optionally: | |
| 109 * implicit: A boolean indicating that this dep should only be installed as a | |
| 110 dependency of some other dep. For example, you want package A, which | |
| 111 depends on package Z, but you don't really care about Z. You should mark | |
| 112 Z as 'implicit' to allow it to be pinned correctly, but not to | |
| 113 deliberately install it. | |
| 114 | |
| 115 4: https://docs.python.org/2/library/ast.html#ast.literal_eval | |
| 116 | |
| 117 | |
| 118 ## `ingest_source.py` | |
| 119 Some python modules don't have functional python repos (i.e. ones that pip | |
| 120 can natively clone+build), and thus ship their source in tarballs. To ingest | |
| 121 such a tarball into the infra google storage bucket, use | |
| 122 `ingest_source.py /path/to/archive`. | |
| 123 This will print the value for the 'gs' key for a `deps.pyl` entry. | |
| 124 | |
| 125 | |
| 126 ## `build_deps.py` / rolling deps | |
| 127 Any time a new dependency/version is introduced into `deps.pyl`, you must run | |
| 128 `build_deps.py`. If the dependency is a pure-python dependency (i.e. no compiled | |
| 129 extensions), you only need to run it once on CPython 2.7. You can tell that it's | |
| 130 a pure python module by looking at the name of the wheel file. For example: | |
| 131 | |
| 132 requests-2.3.0-py2.py3-none-any.whl | |
| 133 | |
| 134 Is compatible with Python 2 and Python 3 (py2.py3) any python ABI (none), and | |
| 135 any OS platform (any). | |
| 136 | |
| 137 If the module does contain compiled extensions, you must run `build_deps.py` on | |
| 138 the following systems (all with CPython 2.7): | |
| 139 * OS X 10.9 - `x86_64` | |
| 140 * Windows 7 - `x86_64` | |
| 141 * Linux - `x86_64` | |
| 142 | |
| 143 TODO(iannucci): Add job to build wheels on all appropriate systems. | |
| 144 | |
| 145 Once a wheel is sucessfully built, it is uploaded to | |
| 146 `gs://chrome-python-wheelhouse/wheels` if it is not there already. | |
| 147 | |
| 148 Running `build_deps.py` will only attempt to build dependencies which are | |
| 149 missing for the current platform. | |
| 150 | |
| 151 `build_deps.py` assumes that it can find `gsutil` on PATH, so go ahead and | |
| 152 install it appropriately for whichever platform you're on. | |
| 153 | |
| 154 | |
| 155 ## custom builds | |
| 156 Sometimes building a wheel is a bit trickier than `pip wheel {repo}@{hash}`. In | |
| 157 order to support this, add a script named `custom_builds/{name}.py`. This module | |
| 158 should have a function defined like: | |
| 159 | |
| 160 `def Build(source_path, wheelhouse_path)` | |
| 161 | |
| 162 Where `source_path` is a string path to the checked-out / unpacked source code, | |
| 163 and `wheelhouse_path` is a string path where `build_deps.py` expects to find | |
| 164 a .whl file after Build completes. | |
| 165 | |
| 166 | |
| 167 ## rolling the version of wheel | |
| 168 Since wheel is a package needed to build the wheels, it has a slightly different | |
| 169 treatment. To roll wheel, bump the version in deps.pyl, and then run | |
| 170 `bootstrap_wheel_wheel.sh` | |
| 171 to build and upload the wheel for 'wheel' pinned at the version in `deps.pyl`. | |
| 172 | |
| 173 Once you do that, `build_deps.py` will continue working as expected. | |
| 174 | |
| 175 | |
| 176 ## Building deps on Windows | |
| 177 TODO(iannucci): actually implement this | |
| 178 | |
| 179 Windows builds require a slightly more care when building, due to the | |
| 180 complexities of getting a compile environment. To this effect, `build_deps.py` | |
| 181 relies on the `depot_tools/win_toolchain` functionality to get a hermetic | |
| 182 windows compiler toolchain. This should not be an issue for chromium devs | |
| 183 working on windows, since they should already have this installed by compiling | |
| 184 chromium, but it's something to be aware of. | |
| 185 | |
| 186 | |
| 187 ## modified (non-upstream) deps | |
| 188 If it is necessary to roll a patched version of a library, we should branch it | |
| 189 in the infra googlesource mirror. This branch should be named `{version}-cr`, | |
| 190 and will build packages whose version is `{version}.{cr_version}` (e.g. modify | |
| 191 setup.py on this branch to add an additional component to the version field). | |
| 192 | |
| 193 For example, given the package `jane` at version `2.1.3`, we would create | |
| 194 a branch `2.1.3-cr`. On this branch we would commit any changes necessary to | |
| 195 `2.1.3`, and would adjust the version number in the builds to be e.g. `2.1.3.0`. | |
| OLD | NEW |
|---|
Chromium Code Reviews
Issue 775763005:
Added bootstrap.rst (Closed)
Base URL: https://chromium.googlesource.com/infra/infra.git@master