Add update.py to easily update the imported in-tree copy of Crashpad

third_party/crashpad/update.py

+#!/usr/bin/env python
+# coding: utf-8
+
+# Copyright 2015 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import argparse
+import os
+import re
+import subprocess
+import sys
+import textwrap
+
+
+def main(args):
+    parser = argparse.ArgumentParser(
+        description='Update the in-tree copy of an imported project')
+    parser.add_argument(
+        '--project',
+        help='The imported project name, used for the git remote name',
+        metavar='PROJECT')
+    parser.add_argument(
+        '--repository',
+        default='https://chromium.googlesource.com/crashpad/crashpad',
+        help='The imported project\'s remote fetch URL',
+        metavar='URL')
+    parser.add_argument(
+        '--subtree',
+        default='third_party/crashpad/crashpad',
+        help='The imported project\'s location in this project\'s tree',
+        metavar='PATH')
+    parser.add_argument(
+        '--update-to',
+        help='What to update the imported project to',
+        metavar='COMMITISH')
+    parser.add_argument(
+        '--readme',
+        help='The README.chromium file describing the imported project',
+        metavar='FILE',
+        dest='readme_path')
+    parser.add_argument(
+        '--keep-remote',
+        default='default',
+        choices=['default', 'no', 'yes'],
+        help='Whether to keep the imported project\'s remote when done')
+    parsed = parser.parse_args(args)
+
+    original_head = \

[Primiano Tucci (use gerrit), 2015/12/10 09:31:21]

I think our coding style is more in favor of parentheses rather than
backslashes.

"Do not use backslash line continuation." from
https://google.github.io/styleguide/pyguide.html#Line_length

[Mark Mentovai, 2015/12/10 15:13:55]

Primiano Tucci wrote:
Done.

+        subprocess.check_output(['git', 'rev-parse', 'HEAD']).rstrip()
+
+    # Read the README, because that’s what it’s for. Extract a few things from
+    # it, and save it to be able to update it later.
+    readme_path = parsed.readme_path or \
+        os.path.join(os.path.dirname(__file__ or '.'), 'README.chromium')
+    readme_content_old = open(readme_path).read()
+
+    project_name_match = re.search(
+        r'^Name:\s+(.*)$', readme_content_old, re.MULTILINE)
+    project_name = project_name_match.group(1)
+
+    project_shortname_match = re.search(
+        r'^Short Name:\s+(.*)$', readme_content_old, re.MULTILINE)
+    project_shortname = project_shortname_match.group(1)
+
+    # Extract the original commit hash from the README.
+    revision_match = re.search(r'^Revision:\s+([0-9a-fA-F]{40})($|\s)',
+                               readme_content_old,
+                               re.MULTILINE)
+    revision_old = revision_match.group(1)
+
+    project = parsed.project or project_shortname
+
+    added_remote = False
+    try:
+        # The remote might already exist. Check for it first, and make sure the

[Primiano Tucci (use gerrit), 2015/12/10 09:31:21]

you don't need to add the remote here.
I think what you want to do is just a 
git fetch https://chromium.googlesource.com/blabla.git

that will fetch the packs and give you the head into the FETCH_HEAD name

Essentially looks like you can get rid of lines 74-89 and just pass the url to
git fetch

[Mark Mentovai, 2015/12/10 15:13:55]

Primiano Tucci wrote:
That’s nicer. Done.

Let’s say that hypothetically, I wanted this script’s --update-to to accept a
remote ref. I know that I can say “git fetch repository_url ref_name” and
FETCH_HEAD will be set up to point to the ref. But how do I tell whether
--update-to was a hash or a ref? What does git do here? Does it treat any string
of 4 or more hex digits as a hash, and treat anything else as a ref? Does it do
anything to resolve ambiguity between a ref named abcd and a hash that starts
with abcd? Is there a way to get git to resolve the difference for me?

[Primiano Tucci (use gerrit), 2015/12/10 15:26:13]

git rev-parse FETCH_HEAD will always give you a sha, regardless of what was the
original input

[Mark Mentovai, 2015/12/10 15:31:28]

Sure, but if I say update.py --update-to=6789abcd, I’m expecting to update to a
hash, so I can do “git fetch repository_url” and then “git cherry-pick
whatever..6789abcd”. But if I say update.py --update-to=some_ref, I’d want to do
“git fetch repository_url some_ref” and then “git cherry-pick
whatever..FETCH_HEAD”. I’m asking if there’s some standard git way to
disambiguate between --update-to=6789abcd and --update-to=some_ref in light of
the fact that 6789abcd might be a ref instead of a commit hash.

If not, I can just split it into --update-to-hash and --update-to-ref.

[Mark Mentovai, 2015/12/10 15:50:01]

I wrote:
Oh, another thing I was wondering: is there a way to verify that a hash came
from a given remote (or git fetch?) I know about git rev-list but that seems
like not the best way to do it. Also, it won’t catch a hash that came from
non-master but did come from the remote.

[Primiano Tucci (use gerrit), 2015/12/10 15:53:57]

Ahh that's the part I was missing. I think the answer is easier than expected.
Just always do git fetch remote ref_or_sha1 and always use FETCH_HEAD (even in
the case of a sha1).

Fetching a SHA1 is not 100% officialy supported by the git-scm server, but it is
definitely a thing with GoB and hence the googlesource.com server, which is
probably what you really care about.

+        # URL matches what was specified.
+        remote_url = subprocess.check_output(
+            ['git', 'config', '--local', 'remote.' + project + '.url']).rstrip()
+        if remote_url != parsed.repository:
+            raise Exception(
+                'repository mismatch', parsed.repository, remote_url)
+    except subprocess.CalledProcessError as e:
+        if e.returncode == 1:
+            # The remote doesn’t exist. Add it.
+            subprocess.check_call(
+                ['git', 'remote', 'add', project, parsed.repository])
+            added_remote = True
+
+    subprocess.check_call(['git', 'fetch', project])
+
+    commitish_new = parsed.update_to or project + '/master'

[Primiano Tucci (use gerrit), 2015/12/10 09:31:21]

and this would become FETCH_HEAD

[Mark Mentovai, 2015/12/10 15:13:55]

Primiano Tucci wrote:
Done.

+    update_range = revision_old + '..' + commitish_new
+
+    # This cherry-picks each change in the window from the upstream project into
+    # the current branch.
+    subprocess.check_call(['git',

[Primiano Tucci (use gerrit), 2015/12/10 09:31:21]

Not necessary for this CL: I think you want to be a bit more conservative here.
There is a chance that this cherry-pick will fail because of merge conflicts, in
which case cherry-pick will return with !0 retcode (and leave your repo dirty).
I think what you want to do here is:
 - If you detect non-zero retcode you want to print a message saying: "Looks
like you have merge conflicts. Open a new terminal, solve them, and then do a
git cherry-pick --continue. Hit enter when done"
 - If the user aborts (KeyboardInterrupt) instead you want to fire a git
cherry-pick --abort to leave the repo in a clean state.

[Mark Mentovai, 2015/12/10 15:16:01]

Primiano Tucci wrote:
I’ll do this in a follow-up unless you’re slow to review. :)

[Mark Mentovai, 2015/12/10 20:08:50]

Did this now.

+                           'cherry-pick',
+                           '--keep-redundant',

[Primiano Tucci (use gerrit), 2015/12/10 09:31:21]

Isn't this --keep-redundant-commits (with -commits at the end?)

[Mark Mentovai, 2015/12/10 15:13:55]

Primiano Tucci wrote:
It is, but --keep-redundant worked too. Done.
https://chromium.googlesource.com/crashpad/crashpad/+/7efdc94f5982a1f9654c53f...
is already in Chromium’s local copy. Without --keep-redundant-commits, the
cherry-pick stops and prompts:

---
The previous cherry-pick is now empty, possibly due to conflict resolution.
If you wish to commit it anyway, use:

    git commit --allow-empty

If you wish to skip this commit, use:

    git reset

Then "git cherry-pick --continue" will resume cherry-picking
the remaining commits.
---

--allow-empty would be fine too because I’m squashing anyway, but
--keep-redundant-commits seems more in line with the spirit of what’s being
done. It does work correctly to update through 7efdc94f5982.

[Primiano Tucci (use gerrit), 2015/12/10 15:26:13]

ah ok. I never figured out how git really does patch detection and when decides
to just bail out :)

+                           '--strategy=subtree',
+                           '-Xsubtree=' + parsed.subtree,
+                           '-x',
+                           update_range])
+
+    # Make a nice commit message. Start with the full commit hash.
+    revision_new = subprocess.check_output(
+        ['git', 'rev-parse', commitish_new]).rstrip()
+    new_message = 'Update ' + project_name + ' to ' + revision_new + '\n\n'
+
+    # Get an abbreviated hash and subject line for each commit in the window,
+    # sorted in chronological order. Wrap everything to 72 characters, with a
+    # hanging indent.
+    log_lines = subprocess.check_output(['git',
+                                         '-c',
+                                         'core.abbrev=12',
+                                         'log',
+                                         '--abbrev-commit',
+                                         '--pretty=oneline',
+                                         '--reverse',
+                                         update_range]).splitlines(False)
+    wrapper = textwrap.TextWrapper(width=72, subsequent_indent = ' ' * 13)
+    for line in log_lines:
+        # Strip trailing periods from subjects.

[Primiano Tucci (use gerrit), 2015/12/10 09:31:21]

such classy :)

[Mark Mentovai, 2015/12/10 15:13:55]

Primiano Tucci wrote:
Thanks! :)

+        if line.endswith('.'):
+            line = line[:-1]
+
+        # If any subjects have what look like commit hashes in them, truncate
+        # them to 12 characters.
+        line = re.sub(r'(\s)([0-9a-fA-F]{12})([0-9a-fA-F]{28})($|\s)',

[Primiano Tucci (use gerrit), 2015/12/10 09:31:21]

Hmm dunno about this. This would make copy / grepping into another git log a bit
of a headache.
I find myself often searching for full text in git logs to find matching
commits.
I'd probably leave these as they are

[Mark Mentovai, 2015/12/10 15:13:55]

Primiano Tucci wrote:
72’s already very restrictive, and 13 of those are already being used up by the
commit’s own hash. If these guys were allowed to remain at their full
40-character lengths, it’d only leave 19 more characters on the line for prose.
The wrapping gets pretty nasty pretty quickly, and this’ll be just about
guaranteed to wrap which will interfere with clean copy/paste/grep anyway.

Since I’m abbreviating this commit’s hash to 12, I think it’s reasonable to
abbreviate another has that it references to the same length.

+                      r'\1\2\4',
+                      line)
+
+        new_message += '\n'.join(wrapper.wrap(line)) + '\n'
+
+    # Update the README with the new hash.
+    readme_content_new = re.sub(
+        r'^(Revision:\s+)([0-9a-fA-F]{40})($|\s.*?$)',
+        r'\g<1>' + revision_new,
+        readme_content_old,
+        1,
+        re.MULTILINE)
+
+    # If the in-tree copy has no changes relative to the upstream, clear the
+    # “Local Modifications” section of the README.
+    has_local_modifications = True
+    try:
+        subprocess.check_call(['git',
+                               'diff-tree',
+                               '--quiet',
+                               commitish_new,
+                               'HEAD:' + parsed.subtree])
+        has_local_modifications = False
+        readme_content_new = re.sub(r'\nLocal Modifications:\n.*$',
+                                    '\nLocal Modifications:\nNone.',
+                                    readme_content_new,
+                                    1)
+    except subprocess.CalledProcessError, e:
+        # If the return code is 1, swallow the exception. There are local
+        # modifications.
+        if e.returncode != 1:
+            raise
+
+    # This soft-reset causes all of the cherry-picks to show up as staged,
+    # which will have the effect of squashing them along with the README update
+    # when committed below.
+    subprocess.check_call(['git', 'reset', '--soft', original_head])
+
+    # Remove the remote if desired.
+    if parsed.keep_remote == 'no' or \
+        (parsed.keep_remote == 'default' and added_remote):
+        subprocess.check_call(['git', 'remote', 'remove', project])
+
+    # Write the new README.
+    open(readme_path, 'w').write(readme_content_new)
+
+    # Commit everything.
+    subprocess.check_call(['git', 'add', readme_path])
+    subprocess.check_call(['git', 'commit', '--message=' + new_message])
+
+    if has_local_modifications:
+        print 'Remember to check the Local Modifications section in ' + \
+            readme_path
+
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main(sys.argv[1:]))