OLD | NEW |
1 # Gsubtreed | 1 # Gsubtreed |
2 | 2 |
3 #### (git subtree daemon) | 3 #### (git subtree daemon) |
4 | 4 |
5 Gsubtreed is a daemon which mirrors subtrees from a large git repo to their | 5 Gsubtreed is a daemon which mirrors subtrees from a large git repo to their |
6 own smaller repos. It is typically run for medium-term intervals (e.g. 10 min), | 6 own smaller repos. It is typically run for medium-term intervals (e.g. 10 min), |
7 restarting after every interval. While it's running it has a short (e.g. 5s) | 7 restarting after every interval. While it's running it has a short (e.g. 5s) |
8 poll+process cycle for the repo that it's mirroring. | 8 poll+process cycle for the repo that it's mirroring. |
9 | 9 |
10 ## Setup: | 10 ## Setup: |
11 | 11 |
12 1. Push a file named `config.json` to the `refs/gsubtreed-config/main` ref of | 12 1. Push a file named `config.json` to the `refs/gsubtreed-config/main` ref of |
13 the repo you intend to mirror. Its contents is determined by | 13 the repo you intend to mirror. Its contents is determined by |
14 [`GsubtreedConfigRef`][1]. You must at least set the `enabled_paths` | 14 [`GsubtreedConfigRef`][1]. You must at least set the `enabled_paths` |
15 portion. | 15 portion. |
16 * You should create a new line of history (e.g. `git checkout --orphan`) | 16 * You should create a new line of history (e.g. `git checkout --orphan`) |
17 to avoid having unecessary extra files in this ref (as opposed to | 17 to avoid having unecessary extra files in this ref (as opposed to |
18 branching it off of another ref like `master`). Note that you'll | 18 branching it off of another ref like `master`). Note that you'll |
19 probably also want to do a `git rm -rf` after you check out the orphan | 19 probably also want to do a `git rm -rf` after you check out the orphan |
20 ref to clear the index+working copy. | 20 ref to clear the index+working copy. |
| 21 |
| 22 ```sh |
| 23 git checkout --orphan gsubtreed-config/main |
| 24 git rm -rf * |
| 25 vim config.json ## Edit the file |
| 26 git commit -am 'Created gsubtreed config' |
| 27 git push remote refs/gsubtreed-config/main |
| 28 ``` |
| 29 |
21 1. (optional) If you have existing git mirrors (probably mirrored via git-svn), | 30 1. (optional) If you have existing git mirrors (probably mirrored via git-svn), |
22 disable the mirroring service for them. If not, create the mirrors. By | 31 disable the mirroring service for them. If not, create the mirrors. By |
23 default if gsubtreed mirrors the path `bob` in `https://.../repo`, it will | 32 default if gsubtreed mirrors the path `bob` in `https://.../repo`, it will |
24 expect to find the mirror repo at `https://.../repo/bob`. You can change | 33 expect to find the mirror repo at `https://.../repo/bob`. You can change |
25 this with the `base_url` parameter in the config. | 34 this with the `base_url` parameter in the config. |
26 * If you do have existing mirrors, you will also need to bootstrap them. | 35 * If you do have existing mirrors, you will also need to bootstrap them. |
27 gsubtreed uses the mirror repos to store state about what it has already | 36 gsubtreed uses the mirror repos to store state about what it has already |
28 processed. It stores state as commit footers (e.g. `Cr-Mirrored-From`) | 37 processed. It stores state as commit footers (e.g. `Cr-Mirrored-From`) |
29 The existing mirrors don't have this state so we have to cheat. | 38 The existing mirrors don't have this state so we have to cheat. |
30 | 39 |
(...skipping 20 matching lines...) Expand all Loading... |
51 this directly with the `git update-ref` command after fetching the | 60 this directly with the `git update-ref` command after fetching the |
52 existing notes from the mirror. | 61 existing notes from the mirror. |
53 | 62 |
54 1. Once your mirrors are in a good state (either empty or primed with | 63 1. Once your mirrors are in a good state (either empty or primed with |
55 `extra_footers`), you should be able to run: `run.py | 64 `extra_footers`), you should be able to run: `run.py |
56 infra.services.gsubtreed <repo url>` and it should Just Work. | 65 infra.services.gsubtreed <repo url>` and it should Just Work. |
57 | 66 |
58 1. At this point you should set up a new bot on the [infra cron waterfall][2] | 67 1. At this point you should set up a new bot on the [infra cron waterfall][2] |
59 to run this on a regular interval. | 68 to run this on a regular interval. |
60 | 69 |
| 70 * Request a new VM for the bot - |
| 71 [sample ticket](http://crbug.com/626818) (internal) |
| 72 |
| 73 * Use existing gsubtreed configs as templates in `master.cfg` and |
| 74 `slaves.cfg` to set it up; remember to add the new bot to the |
| 75 `Periodic` scheduler together with the other gsubtreed bots |
| 76 |
| 77 * Push the appropriate credentials to the VM - |
| 78 [see an example CL](http://go/chromerev/469017013) (internal). |
| 79 |
61 ## Usage: | 80 ## Usage: |
62 | 81 |
63 run.py infra.services.gsubtreed <repo_url> | 82 run.py infra.services.gsubtreed <repo_url> |
64 | 83 |
65 ## GsubtreedConfigRef fields | 84 ## GsubtreedConfigRef fields |
66 | 85 |
67 * `interval` *number*: The time in seconds between iterations of gsubtreed. | 86 * `interval` *number*: The time in seconds between iterations of gsubtreed. |
68 Each iteration will fetch from the main repo, and try to push to all of the | 87 Each iteration will fetch from the main repo, and try to push to all of the |
69 subtree repos. | 88 subtree repos. |
70 | 89 |
(...skipping 22 matching lines...) Expand all Loading... |
93 base_url was `https://example.com`, then it would mirror `path/to/foo` to | 112 base_url was `https://example.com`, then it would mirror `path/to/foo` to |
94 `https://example.com/bar`, instead of `https://example.com/path/to/foo`. | 113 `https://example.com/bar`, instead of `https://example.com/path/to/foo`. |
95 | 114 |
96 * `path_extra_push` *{string: [string]}*: A dictionary mapping `enabled_path` | 115 * `path_extra_push` *{string: [string]}*: A dictionary mapping `enabled_path` |
97 to a list of full_git_repo_urls. Any time we find changes in the | 116 to a list of full_git_repo_urls. Any time we find changes in the |
98 enabled_path, we'll also push those subtree commits to all the git repos in | 117 enabled_path, we'll also push those subtree commits to all the git repos in |
99 full_git_repo_urls. | 118 full_git_repo_urls. |
100 | 119 |
101 [1]: ./gsubtreed.py#32 | 120 [1]: ./gsubtreed.py#32 |
102 [2]: http://build.chromium.org/p/chromium.infra.cron | 121 [2]: http://build.chromium.org/p/chromium.infra.cron |
OLD | NEW |