| OLD | NEW |
| 1 # Gsubtreed | 1 # Gsubtreed |
| 2 | 2 |
| 3 #### (git subtree daemon) | 3 #### (git subtree daemon) |
| 4 | 4 |
| 5 Gsubtreed is a daemon which mirrors subtrees from a large git repo to their | 5 Gsubtreed is a daemon which mirrors subtrees from a large git repo to their |
| 6 own smaller repos. It is typically run for medium-term intervals (e.g. 10 min), | 6 own smaller repos. It is typically run for medium-term intervals (e.g. 10 min), |
| 7 restarting after every interval. While it's running it has a short (e.g. 5s) | 7 restarting after every interval. While it's running it has a short (e.g. 5s) |
| 8 poll+process cycle for the repo that it's mirroring. | 8 poll+process cycle for the repo that it's mirroring. |
| 9 | 9 |
| 10 ## Setup: | 10 ## Setup: |
| 11 | 11 |
| 12 1. Push a file named `config.json` to the `refs/gsubtreed-config/main` ref of | 12 1. Push a file named `config.json` to the `refs/gsubtreed-config/main` ref of |
| 13 the repo you intend to mirror. Its contents is determined by | 13 the repo you intend to mirror. Its contents is determined by |
| 14 [`GsubtreedConfigRef`][1]. You must at least set the `enabled_paths` | 14 [`GsubtreedConfigRef`][1]. You must at least set the `enabled_paths` |
| 15 portion. | 15 portion. |
| 16 * You should create a new line of history (e.g. `git checkout --orphan`) | 16 * You should create a new line of history (e.g. `git checkout --orphan`) |
| 17 to avoid having unecessary extra files in this ref (as opposed to | 17 to avoid having unecessary extra files in this ref (as opposed to |
| 18 branching it off of another ref like `master`). Note that you'll | 18 branching it off of another ref like `master`). Note that you'll |
| 19 probably also want to do a `git rm -rf` after you check out the orphan | 19 probably also want to do a `git rm -rf` after you check out the orphan |
| 20 ref to clear the index+working copy. | 20 ref to clear the index+working copy. |
| 21 |
| 22 ```sh |
| 23 git checkout --orphan gsubtreed-config/main |
| 24 git rm -rf * |
| 25 vim config.json ## Edit the file |
| 26 git commit -am 'Created gsubtreed config' |
| 27 git push remote refs/gsubtreed-config/main |
| 28 ``` |
| 29 |
| 21 1. (optional) If you have existing git mirrors (probably mirrored via git-svn), | 30 1. (optional) If you have existing git mirrors (probably mirrored via git-svn), |
| 22 disable the mirroring service for them. If not, create the mirrors. By | 31 disable the mirroring service for them. If not, create the mirrors. By |
| 23 default if gsubtreed mirrors the path `bob` in `https://.../repo`, it will | 32 default if gsubtreed mirrors the path `bob` in `https://.../repo`, it will |
| 24 expect to find the mirror repo at `https://.../repo/bob`. You can change | 33 expect to find the mirror repo at `https://.../repo/bob`. You can change |
| 25 this with the `base_url` parameter in the config. | 34 this with the `base_url` parameter in the config. |
| 26 * If you do have existing mirrors, you will also need to bootstrap them. | 35 * If you do have existing mirrors, you will also need to bootstrap them. |
| 27 gsubtreed uses the mirror repos to store state about what it has already | 36 gsubtreed uses the mirror repos to store state about what it has already |
| 28 processed. It stores state as commit footers (e.g. `Cr-Mirrored-From`) | 37 processed. It stores state as commit footers (e.g. `Cr-Mirrored-From`) |
| 29 The existing mirrors don't have this state so we have to cheat. | 38 The existing mirrors don't have this state so we have to cheat. |
| 30 | 39 |
| (...skipping 20 matching lines...) Expand all Loading... |
| 51 this directly with the `git update-ref` command after fetching the | 60 this directly with the `git update-ref` command after fetching the |
| 52 existing notes from the mirror. | 61 existing notes from the mirror. |
| 53 | 62 |
| 54 1. Once your mirrors are in a good state (either empty or primed with | 63 1. Once your mirrors are in a good state (either empty or primed with |
| 55 `extra_footers`), you should be able to run: `run.py | 64 `extra_footers`), you should be able to run: `run.py |
| 56 infra.services.gsubtreed <repo url>` and it should Just Work. | 65 infra.services.gsubtreed <repo url>` and it should Just Work. |
| 57 | 66 |
| 58 1. At this point you should set up a new bot on the [infra cron waterfall][2] | 67 1. At this point you should set up a new bot on the [infra cron waterfall][2] |
| 59 to run this on a regular interval. | 68 to run this on a regular interval. |
| 60 | 69 |
| 70 * Request a new VM for the bot - |
| 71 [sample ticket](http://crbug.com/626818) (internal) |
| 72 |
| 73 * Use existing gsubtreed configs as templates in `master.cfg` and |
| 74 `slaves.cfg` to set it up; remember to add the new bot to the |
| 75 `Periodic` scheduler together with the other gsubtreed bots |
| 76 |
| 77 * Push the appropriate credentials to the VM - |
| 78 [see an example CL](http://go/chromerev/469017013) (internal). |
| 79 |
| 61 ## Usage: | 80 ## Usage: |
| 62 | 81 |
| 63 run.py infra.services.gsubtreed <repo_url> | 82 run.py infra.services.gsubtreed <repo_url> |
| 64 | 83 |
| 65 ## GsubtreedConfigRef fields | 84 ## GsubtreedConfigRef fields |
| 66 | 85 |
| 67 * `interval` *number*: The time in seconds between iterations of gsubtreed. | 86 * `interval` *number*: The time in seconds between iterations of gsubtreed. |
| 68 Each iteration will fetch from the main repo, and try to push to all of the | 87 Each iteration will fetch from the main repo, and try to push to all of the |
| 69 subtree repos. | 88 subtree repos. |
| 70 | 89 |
| (...skipping 22 matching lines...) Expand all Loading... |
| 93 base_url was `https://example.com`, then it would mirror `path/to/foo` to | 112 base_url was `https://example.com`, then it would mirror `path/to/foo` to |
| 94 `https://example.com/bar`, instead of `https://example.com/path/to/foo`. | 113 `https://example.com/bar`, instead of `https://example.com/path/to/foo`. |
| 95 | 114 |
| 96 * `path_extra_push` *{string: [string]}*: A dictionary mapping `enabled_path` | 115 * `path_extra_push` *{string: [string]}*: A dictionary mapping `enabled_path` |
| 97 to a list of full_git_repo_urls. Any time we find changes in the | 116 to a list of full_git_repo_urls. Any time we find changes in the |
| 98 enabled_path, we'll also push those subtree commits to all the git repos in | 117 enabled_path, we'll also push those subtree commits to all the git repos in |
| 99 full_git_repo_urls. | 118 full_git_repo_urls. |
| 100 | 119 |
| 101 [1]: ./gsubtreed.py#32 | 120 [1]: ./gsubtreed.py#32 |
| 102 [2]: http://build.chromium.org/p/chromium.infra.cron | 121 [2]: http://build.chromium.org/p/chromium.infra.cron |
| OLD | NEW |