Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(28)

Side by Side Diff: infra/services/gsubtreed/README.md

Issue 2145753008: Update gsubtreed docs with more details. (Closed) Base URL: https://chromium.googlesource.com/infra/infra.git@master
Patch Set: Created 4 years, 5 months ago
Use n/p to move between diff chunks; N/P to move between comments. Draft comments are only viewable by you.
Jump to:
View unified diff | Download patch
« no previous file with comments | « no previous file | no next file » | no next file with comments »
Toggle Intra-line Diffs ('i') | Expand Comments ('e') | Collapse Comments ('c') | Show Comments Hide Comments ('s')
OLDNEW
1 # Gsubtreed 1 # Gsubtreed
2 2
3 #### (git subtree daemon) 3 #### (git subtree daemon)
4 4
5 Gsubtreed is a daemon which mirrors subtrees from a large git repo to their 5 Gsubtreed is a daemon which mirrors subtrees from a large git repo to their
6 own smaller repos. It is typically run for medium-term intervals (e.g. 10 min), 6 own smaller repos. It is typically run for medium-term intervals (e.g. 10 min),
7 restarting after every interval. While it's running it has a short (e.g. 5s) 7 restarting after every interval. While it's running it has a short (e.g. 5s)
8 poll+process cycle for the repo that it's mirroring. 8 poll+process cycle for the repo that it's mirroring.
9 9
10 ## Setup: 10 ## Setup:
11 11
12 1. Push a file named `config.json` to the `refs/gsubtreed-config/main` ref of 12 1. Push a file named `config.json` to the `refs/gsubtreed-config/main` ref of
13 the repo you intend to mirror. Its contents is determined by 13 the repo you intend to mirror. Its contents is determined by
14 [`GsubtreedConfigRef`][1]. You must at least set the `enabled_paths` 14 [`GsubtreedConfigRef`][1]. You must at least set the `enabled_paths`
15 portion. 15 portion.
16 * You should create a new line of history (e.g. `git checkout --orphan`) 16 * You should create a new line of history (e.g. `git checkout --orphan`)
17 to avoid having unecessary extra files in this ref (as opposed to 17 to avoid having unecessary extra files in this ref (as opposed to
18 branching it off of another ref like `master`). Note that you'll 18 branching it off of another ref like `master`). Note that you'll
19 probably also want to do a `git rm -rf` after you check out the orphan 19 probably also want to do a `git rm -rf` after you check out the orphan
20 ref to clear the index+working copy. 20 ref to clear the index+working copy.
21
22 ```sh
23 git checkout --orphan gsubtreed-config/main
24 git rm -rf *
25 vim config.json ## Edit the file
26 git commit -am 'Created gsubtreed config'
27 git push remote refs/gsubtreed-config/main
28 ```
29
21 1. (optional) If you have existing git mirrors (probably mirrored via git-svn), 30 1. (optional) If you have existing git mirrors (probably mirrored via git-svn),
22 disable the mirroring service for them. If not, create the mirrors. By 31 disable the mirroring service for them. If not, create the mirrors. By
23 default if gsubtreed mirrors the path `bob` in `https://.../repo`, it will 32 default if gsubtreed mirrors the path `bob` in `https://.../repo`, it will
24 expect to find the mirror repo at `https://.../repo/bob`. You can change 33 expect to find the mirror repo at `https://.../repo/bob`. You can change
25 this with the `base_url` parameter in the config. 34 this with the `base_url` parameter in the config.
26 * If you do have existing mirrors, you will also need to bootstrap them. 35 * If you do have existing mirrors, you will also need to bootstrap them.
27 gsubtreed uses the mirror repos to store state about what it has already 36 gsubtreed uses the mirror repos to store state about what it has already
28 processed. It stores state as commit footers (e.g. `Cr-Mirrored-From`) 37 processed. It stores state as commit footers (e.g. `Cr-Mirrored-From`)
29 The existing mirrors don't have this state so we have to cheat. 38 The existing mirrors don't have this state so we have to cheat.
30 39
(...skipping 20 matching lines...) Expand all
51 this directly with the `git update-ref` command after fetching the 60 this directly with the `git update-ref` command after fetching the
52 existing notes from the mirror. 61 existing notes from the mirror.
53 62
54 1. Once your mirrors are in a good state (either empty or primed with 63 1. Once your mirrors are in a good state (either empty or primed with
55 `extra_footers`), you should be able to run: `run.py 64 `extra_footers`), you should be able to run: `run.py
56 infra.services.gsubtreed <repo url>` and it should Just Work. 65 infra.services.gsubtreed <repo url>` and it should Just Work.
57 66
58 1. At this point you should set up a new bot on the [infra cron waterfall][2] 67 1. At this point you should set up a new bot on the [infra cron waterfall][2]
59 to run this on a regular interval. 68 to run this on a regular interval.
60 69
70 * Request a new VM for the bot -
71 [sample ticket](http://crbug.com/626818) (internal)
72
73 * Use existing gsubtreed configs as templates in `master.cfg` and
74 `slaves.cfg` to set it up; remember to add the new bot to the
75 `Periodic` scheduler together with the other gsubtreed bots
76
77 * Push the appropriate credentials to the VM -
78 [see an example CL](http://go/chromerev/469017013) (internal).
79
61 ## Usage: 80 ## Usage:
62 81
63 run.py infra.services.gsubtreed <repo_url> 82 run.py infra.services.gsubtreed <repo_url>
64 83
65 ## GsubtreedConfigRef fields 84 ## GsubtreedConfigRef fields
66 85
67 * `interval` *number*: The time in seconds between iterations of gsubtreed. 86 * `interval` *number*: The time in seconds between iterations of gsubtreed.
68 Each iteration will fetch from the main repo, and try to push to all of the 87 Each iteration will fetch from the main repo, and try to push to all of the
69 subtree repos. 88 subtree repos.
70 89
(...skipping 22 matching lines...) Expand all
93 base_url was `https://example.com`, then it would mirror `path/to/foo` to 112 base_url was `https://example.com`, then it would mirror `path/to/foo` to
94 `https://example.com/bar`, instead of `https://example.com/path/to/foo`. 113 `https://example.com/bar`, instead of `https://example.com/path/to/foo`.
95 114
96 * `path_extra_push` *{string: [string]}*: A dictionary mapping `enabled_path` 115 * `path_extra_push` *{string: [string]}*: A dictionary mapping `enabled_path`
97 to a list of full_git_repo_urls. Any time we find changes in the 116 to a list of full_git_repo_urls. Any time we find changes in the
98 enabled_path, we'll also push those subtree commits to all the git repos in 117 enabled_path, we'll also push those subtree commits to all the git repos in
99 full_git_repo_urls. 118 full_git_repo_urls.
100 119
101 [1]: ./gsubtreed.py#32 120 [1]: ./gsubtreed.py#32
102 [2]: http://build.chromium.org/p/chromium.infra.cron 121 [2]: http://build.chromium.org/p/chromium.infra.cron
OLDNEW
« no previous file with comments | « no previous file | no next file » | no next file with comments »

Powered by Google App Engine
This is Rietveld 408576698