Add git subtree daemon service.

Add git subtree daemon service.

R=agable@chromium.org, vadimsh@chromium.org
BUG=370554
Committed: https://chromium.googlesource.com/infra/infra/+/1f057878467d0fd78a06d28fe97f5c6e5ef8a2e9

[Vadim Sh., 2014-08-17 19:37:59]

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
File infra/services/gsubtreed/gsubtreed.py (right):

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/gsubtreed.py:40: #  
refs/subtree-processed/b/foo/-/heads/master    <- ancestor tag of master
Will this refs be created on a first use or some of the should be created
manually?

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/gsubtreed.py:46: 'base_url': None,
Document.

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/gsubtreed.py:66: base_url = fake_base_repo_url or
config['base_url']
can't you mock config['base_url'] in tests to get rid of fake_base_repo_url?

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/gsubtreed.py:71:
subtree_repo.reify(share_from=origin_repo)
what if repo at base_url is not initially dervied from origin_repo? Is it safe
to share objects that way?

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/gsubtreed.py:84: cur_tree = synthed.commit.data.tree
is it None if ref doesn't exist yet?

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/gsubtreed.py:108: ('Cr-Mirrored-Commit', [commit.hsh]),
It will also inherit Cr-Commit-Position, right?

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/gsubtreed.py:123:
subtree_repo.fast_forward_push(subtree_repo_push)
TODO: push to subtree repos in parallel. This can be done safely I think, and
it'll cut a lot of time when many subtree are mirrored.

I think process_path can return all pushes it wants to make and inner_loop would
run them in parallel at the end.

Though since subtreed consumers are not latency sensitive it doesn't really
matter...

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
File infra/services/gsubtreed/test/gsubtreed_test.py (right):

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/test/gsubtreed_test.py:35: # TODO(iannucci): Make these
first class data library objects
huh? what exactly?

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/test/gsubtreed_test.py:94: import traceback
move it to where all other imports are

[iannucci, 2014-08-17 22:35:30]

ptal

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
File infra/services/gsubtreed/gsubtreed.py (right):

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/gsubtreed.py:40: #  
refs/subtree-processed/b/foo/-/heads/master    <- ancestor tag of master
On 2014/08/17 19:37:59, Vadim Sh. wrote:
> Will this refs be created on a first use or some of the should be created
> manually?

It'll do a push... normally that would create it, but if it doesn't have
permissions.... It's designed to execute assuming those refs aren't there.

This one is a lot easier than gnumbd, because when it processes some commit X,
it will always produce Y. There's no funny business with the clock, etc. here.

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/gsubtreed.py:46: 'base_url': None,
On 2014/08/17 19:37:59, Vadim Sh. wrote:
> Document.

Done.

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/gsubtreed.py:66: base_url = fake_base_repo_url or
config['base_url']
On 2014/08/17 19:37:59, Vadim Sh. wrote:
> can't you mock config['base_url'] in tests to get rid of fake_base_repo_url?

Did a thing. ptal.

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/gsubtreed.py:71:
subtree_repo.reify(share_from=origin_repo)
On 2014/08/17 19:37:59, Vadim Sh. wrote:
> what if repo at base_url is not initially dervied from origin_repo? Is it safe
> to share objects that way?

Yes, it's safe. Object sharing only means that it'll look up in origin_repo for
a given object before it requests it from the remote.

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/gsubtreed.py:84: cur_tree = synthed.commit.data.tree
On 2014/08/17 19:37:59, Vadim Sh. wrote:
> is it None if ref doesn't exist yet?

It's INVALID (and thus synthed.commit.data.tree is also INVALID)

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/gsubtreed.py:108: ('Cr-Mirrored-Commit', [commit.hsh]),
On 2014/08/17 19:37:59, Vadim Sh. wrote:
> It will also inherit Cr-Commit-Position, right?

Yes, this is just adding footers to whatever's there (unless they already have
these particular footers... which would be... silly).

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/gsubtreed.py:123:
subtree_repo.fast_forward_push(subtree_repo_push)
On 2014/08/17 19:37:59, Vadim Sh. wrote:
> TODO: push to subtree repos in parallel. This can be done safely I think, and
> it'll cut a lot of time when many subtree are mirrored.
> 
> I think process_path can return all pushes it wants to make and inner_loop
would
> run them in parallel at the end.
> 
> Though since subtreed consumers are not latency sensitive it doesn't really
> matter... 

Yeah I thought about it. I'll add a comment, but I'd rather optimize later. The
way the origin push happens right now, that would be a bottleneck, though as
long as I did all the pushes in order (subtree first, then origin), then I could
batch the origin pushes.

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
File infra/services/gsubtreed/test/gsubtreed_test.py (right):

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/test/gsubtreed_test.py:35: # TODO(iannucci): Make these
first class data library objects
On 2014/08/17 19:37:59, Vadim Sh. wrote:
> huh? what exactly?

Oh, oops. That was from before I moved all the stuff from gnumbd_test into
testing_support.

https://codereview.chromium.org/477623003/diff/40001/infra/services/gsubtreed...
infra/services/gsubtreed/test/gsubtreed_test.py:94: import traceback
On 2014/08/17 19:37:59, Vadim Sh. wrote:
> move it to where all other imports are

I have no idea why I did this... :/

[Vadim Sh., 2014-08-18 18:01:51]

lgtm

[iannucci, 2014-08-18 18:19:33]

The CQ bit was checked by iannucci@chromium.org

[commit-bot: I haz the power, 2014-08-18 18:20:08]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/iannucci@chromium.org/477623003/60001

[commit-bot: I haz the power, 2014-08-18 18:20:19]

Committed patchset #4 (60001) as 1f057878467d0fd78a06d28fe97f5c6e5ef8a2e9