Add support for running git in tests

pkg/analysis_server/test/stress/utilities/git.dart

+// Copyright (c) 2015, the Dart project authors.  Please see the AUTHORS file
+// for details. All rights reserved. Use of this source code is governed by a
+// BSD-style license that can be found in the LICENSE file.
+
+/**
+ * Support for interacting with a git repository.
+ */
+library analysis_server.test.stress.utilities.git;
+
+import 'dart:convert';
+import 'dart:io';
+
+import 'package:path/path.dart' as path;
+
+/**
+ * A representation of the differences between two blobs.
+ */
+class BlobDiff {
+  /**
+   * The regular expression used to identify the beginning of a hunk.
+   */
+  static final RegExp hunkHeaderRegExp =
+      new RegExp(r'@@ -([0-9]+)(?:,[0-9]+)? \+([0-9]+)(?:,[0-9]+)? @@');
+
+  /**
+   * A list of the hunks in the diff.
+   */
+  List<DiffHunk> hunks = <DiffHunk>[];
+
+  /**
+   * Initialize a newly created blob diff by parsing the result of the git diff
+   * command (the [input]).

[Paul Berry, 2015/11/17 19:49:31]

Git diff has a lot of options controlling its output format.  We should probably
add some clarifying text specifying what set of options need to be used to
create the diff in order for it to be compatible with BlobDiff.

For instance, the diff can't have any context lines.

[Brian Wilkerson, 2015/11/18 17:36:46]

I don't really want to deeply document it everywhere, so I took a slightly
different tack.
Actually, I think we correctly ignore context lines.

[Paul Berry, 2015/11/18 18:03:19]

We ignore them, but that won't give correct results for two reasons: (1) the
numbers that appear above the diff hunk (on the line beginning "@@") include
context lines, so if we don't account for that we'll apply the diff in the wrong
place.  (2) if two hunks are closer together than the number of context lines,
Git will combine them, and we won't handle that properly.  E.g.:

@@ -100,5 +100,5 @@
 context line
-removed line
+added line
 context line
-removed line
+added line
 context line

+   */
+  BlobDiff(List<String> input) {
+    _parseInput(input);
+  }
+
+  /**
+   * Parse the result of the git diff command (the [input]).
+   */
+  void _parseInput(List<String> input) {
+    for (String line in input) {
+      _parseLine(line);
+    }
+  }
+
+  /**
+   * Parse a single [line] from the result of the git diff command.
+   */
+  void _parseLine(String line) {
+    DiffHunk currentHunk = hunks.isEmpty ? null : hunks.last;
+    if (line.startsWith('@@')) {
+      Match match = hunkHeaderRegExp.matchAsPrefix(line);
+      int srcLine = int.parse(match.group(1));
+      int dstLine = int.parse(match.group(2));
+      hunks.add(new DiffHunk(srcLine, dstLine));
+    } else if (currentHunk != null && line.startsWith('+')) {
+      currentHunk.addLines.add(line.substring(1));
+    } else if (currentHunk != null && line.startsWith('-')) {
+      currentHunk.removeLines.add(line.substring(1));
+    }

[Paul Berry, 2015/11/17 19:49:31]

Diff uses special formatting when the end of the file lacks a newline.  Consider
adding code to support this.

[Brian Wilkerson, 2015/11/18 17:36:46]

I'll need a pointer to understand what needs to happen.

[Paul Berry, 2015/11/18 18:03:19]

Unfortunately I don't have a pointer to give you (I don't know if there is one).
 But I think if you just try diffing files that do/don't end in a newline it
should be pretty obvious.

+  }
+}
+
+/**
+ * A representation of the differences between two commits.
+ */
+class CommitDelta {
+  /**
+   * The length (in characters) of a SHA.
+   */
+  static final int SHA_LENGTH = 40;
+
+  /**
+   * The code-point for a colon (':').
+   */
+  static final int COLON = ':'.codeUnitAt(0);
+
+  /**
+   * The code-point for a nul character.
+   */
+  static final int NUL = 0;
+
+  /**
+   * The code-point for a tab.
+   */
+  static final int TAB = '\t'.codeUnitAt(0);
+
+  /**
+   * The repository from which the commits were taken.
+   */
+  final GitRepository repository;
+
+  /**
+   * The records of the files that were changed.
+   */
+  final List<DiffRecord> diffRecords = <DiffRecord>[];
+
+  /**
+   * Initialize a newly created representation of the differences between two
+   * commits. The differences are computed by parsing the result of a git diff
+   * command (the [diffResults]).

[Paul Berry, 2015/11/17 19:49:31]

As with BlobDiff, it would be nice to clarify what git options are required to
format the diff in the way that CommitDelta expects.

[Brian Wilkerson, 2015/11/18 17:36:46]

Ditto

+   */
+  CommitDelta(this.repository, String diffResults) {
+    _parseInput(diffResults);
+  }
+
+  /**
+   * Return `true` if there are differences.
+   */
+  bool get hasDiffs => diffRecords.isNotEmpty;
+
+  /**
+   * Return the absolute paths of all of the files in this commit whose name
+   * matches the given [fileName].
+   */
+  Iterable<String> filesMatching(String fileName) {
+    return diffRecords
+        .where((DiffRecord record) => record.isFor(fileName))
+        .map((DiffRecord record) => record.srcPath);
+  }
+
+  /**
+   * Remove any diffs for files that are either (a) outside the given
+   * [analysisRoots], or (b) are files that are not being analyzed by the server.
+   */
+  void filterDiffs(List<String> analysisRoots) {

[Paul Berry, 2015/11/17 19:49:30]

This method seems out of place, since it is the only method in the file that
encodes analyzer-specific semantics.  Everything else is general-purpose enough
to be used by any app that wants to interact with git.

Is there some way we could generalize this method so that the analyzer-specific
logic moves to the caller?

[Brian Wilkerson, 2015/11/18 17:36:46]

I can rename the parameter; it's just a list of directory paths (they wouldn't
have to be analysis roots). I can also pass in the list of glob patterns. Or
just pass in glob patters. I'll do this in a future CL.

+    diffRecords.retainWhere((DiffRecord record) {
+      String filePath = record.srcPath ?? record.dstPath;
+      for (String analysisRoot in analysisRoots) {
+        if (path.isWithin(analysisRoot, filePath)) {
+          // TODO(brianwilkerson) Generalize this by asking the server for the
+          // list of glob patterns of files to be analyzed (once there's an API
+          // for doing so).
+          if (filePath.endsWith('.dart') ||
+              filePath.endsWith('.html') ||
+              filePath.endsWith('.htm') ||
+              filePath.endsWith('.analysisOptions')) {
+            return true;
+          }
+        }
+      }
+      return false;
+    });
+  }
+
+  /**
+   * Return the index of the first nul character in the given [string] that is
+   * at or after the given [start] index.
+   */
+  int _findEnd(String string, int start) {
+    int length = string.length;
+    int end = start;
+    while (end < length && string.codeUnitAt(end) != NUL) {
+      end++;
+    }
+    return end;
+  }
+
+  /**
+   * Return the result of converting the given [relativePath] to an absolute
+   * path. The path is assumed to be relative to the root of the repository.
+   */
+  String _makeAbsolute(String relativePath) {
+    return path.join(repository.path, relativePath);
+  }
+
+  /**
+   * Parse all of the diff records in the given [input].
+   */
+  void _parseInput(String input) {
+    int length = input.length;
+    int start = 0;
+    while (start < length) {
+      start = _parseRecord(input, start);
+    }
+  }
+
+  /**
+   * Parse a single record from the given [input], assuming that the record
+   * starts at the given [startIndex].
+   *
+   * Each record is formatted as a sequence of fields. The fields are, from the
+   * left to the right:
+   *
+   * 1. a colon.
+   * 2. mode for "src"; 000000 if creation or unmerged.
+   * 3. a space.
+   * 4. mode for "dst"; 000000 if deletion or unmerged.
+   * 5. a space.
+   * 6. sha1 for "src"; 0{40} if creation or unmerged.
+   * 7. a space.
+   * 8. sha1 for "dst"; 0{40} if creation, unmerged or "look at work tree".
+   * 9. a space.
+   * 10. status, followed by optional "score" number.
+   * 11. a tab or a NUL when -z option is used.
+   * 12. path for "src"
+   * 13. a tab or a NUL when -z option is used; only exists for C or R.
+   * 14. path for "dst"; only exists for C or R.
+   * 15. an LF or a NUL when -z option is used, to terminate the record.
+   */
+  int _parseRecord(String input, int startIndex) {
+    // Skip the first five fields.
+    startIndex += 15;
+    // 6

[Paul Berry, 2015/11/17 19:49:31]

What do these numbers mean?

[Brian Wilkerson, 2015/11/18 17:36:46]

They're field numbers. Added to the comments to make it more clear.

+    String srcSha = input.substring(startIndex, startIndex + SHA_LENGTH);
+    startIndex += SHA_LENGTH + 1;
+    // 8
+    String dstSha = input.substring(startIndex, startIndex + SHA_LENGTH);
+    startIndex += SHA_LENGTH + 1;
+    // 10
+    int endIndex = _findEnd(input, startIndex);
+    String status = input.substring(startIndex, endIndex);
+    startIndex = endIndex + 1;
+    // 12
+    endIndex = _findEnd(input, startIndex);
+    String srcPath = _makeAbsolute(input.substring(startIndex, endIndex));
+    startIndex = endIndex + 1;
+    // 14
+    String dstPath = null;
+    if (status.startsWith('C') || status.startsWith('R')) {
+      endIndex = _findEnd(input, startIndex);
+      dstPath = _makeAbsolute(input.substring(startIndex, endIndex));
+    }
+    // Create the record.
+    diffRecords.add(
+        new DiffRecord(repository, srcSha, dstSha, status, srcPath, dstPath));
+    return endIndex + 1;
+  }
+}
+
+/**
+ * A representation of the history of a Git repository.
+ */
+class CommitHistory {

[Paul Berry, 2015/11/17 19:49:31]

Since git histories frequently have branches, we should clarify that we're not
trying to represent the full branching structure, but merely a linear subset of
the commit history.  Maybe rename this to "LinearCommitHistory"?

[Brian Wilkerson, 2015/11/18 17:36:46]

Done

+  /**
+   * The repository whose history is being represented.
+   */
+  final GitRepository repository;
+
+  /**
+   * The id's (SHA's) of the commits in the repository, with the most recent
+   * commit being first and the oldest commit being last.
+   */
+  final List<String> commitIds;
+
+  /**
+   * Initialize a commit history for the given [repository] to have the given
+   * [commitIds].
+   */
+  CommitHistory(this.repository, this.commitIds);
+
+  /**
+   * Return an iterator that can be used to iterate over this commit history.
+   */
+  CommitHistoryIterator iterator() {
+    return new CommitHistoryIterator(this);
+  }
+}
+
+/**
+ * An iterator over the history of a Git repository.
+ */
+class CommitHistoryIterator {

[Paul Berry, 2015/11/17 19:49:31]

Similar concern here.

[Brian Wilkerson, 2015/11/18 17:36:46]

Done

+  /**
+   * The commit history being iterated over.
+   */
+  final CommitHistory history;
+
+  /**
+   * The index of the current commit in the list of [commitIds].
+   */
+  int currentCommit;
+
+  /**
+   * Initialize a newly created iterator to iterate over the commits with the
+   * given [commitIds];
+   */
+  CommitHistoryIterator(this.history) {
+    currentCommit = history.commitIds.length;
+  }
+
+  /**
+   * Return the SHA1 of the commit after the current commit (the 'dst' of the
+   * [next] diff).
+   */
+  String get dstCommit => history.commitIds[currentCommit - 1];
+
+  /**
+   * Return the SHA1 of the current commit (the 'src' of the [next] diff).
+   */
+  String get srcCommit => history.commitIds[currentCommit];
+
+  /**
+   * Advance to the next commit in the history. Return `true` if it is safe to
+   * ask for the [next] diff.
+   */
+  bool moveNext() {
+    if (currentCommit <= 1) {
+      return false;
+    }
+    currentCommit--;
+    return true;
+  }
+
+  /**
+   * Return the difference between the current commit and the commit that
+   * followed it.
+   */
+  CommitDelta next() => history.repository.getCommitDiff(srcCommit, dstCommit);
+}
+
+/**
+ * Representation of a single diff hunk.
+ */
+class DiffHunk {
+  /**
+   * The index of the first line that was changed in the src.

[Paul Berry, 2015/11/17 19:49:31]

Since diffSrcLine and diffDstLine are public, their doc comments should
explicitly specify that they follow the conventions used by Git (subtracting 1
if there are no lines on the src/destination side of the diff, respectively).

[Brian Wilkerson, 2015/11/18 17:36:46]

Done

+   */
+  int diffSrcLine;
+
+  /**
+   * The index of the first line that was changed in the dst.
+   */
+  int diffDstLine;
+
+  /**
+   * A list of the individual lines that were removed from the src.
+   */
+  List<String> removeLines = <String>[];
+
+  /**
+   * A list of the individual lines that were added to the dst.
+   */
+  List<String> addLines = <String>[];
+
+  /**
+   * Initialize a newly created hunk. The lines will be

[Paul Berry, 2015/11/17 19:49:31]

Looks like you forgot to finish this comment.

[Brian Wilkerson, 2015/11/18 17:36:46]

Yep. Thanks.

+   */
+  DiffHunk(this.diffSrcLine, this.diffDstLine);
+
+  /**
+   * Return the index of the first line that was changed in the dst.

[Paul Berry, 2015/11/17 19:49:31]

Similarly, the doc comment here should explicitly specify that this getter
*doesn't* follow the convention used by git.

[Brian Wilkerson, 2015/11/18 17:36:46]

Done

+   */
+  int get dstLine {
+    // The diff command numbers lines starting at 1, but it subtracts 1 from
+    // dstLine if there are no lines on the destination side of the hunk. We
+    // convert it into something reasonable.
+    return addLines.isEmpty ? diffDstLine : diffDstLine - 1;
+  }
+
+  /**
+   * Return the index of the first line that was changed in the src.
+   */
+  int get srcLine {
+    // The diff command numbers lines starting at 1, but it subtracts 1 from
+    // srcLine if there are no lines on the source side of the hunk. We convert
+    // it into something reasonable.
+    return removeLines.isEmpty ? diffSrcLine : diffSrcLine - 1;
+  }
+}
+
+/**
+ * A representation of a single line (record) from a raw diff.
+ */
+class DiffRecord {
+  /**
+   * The repository containing the file(s) that were modified.
+   */
+  final GitRepository repository;
+
+  /**
+   * The SHA1 of the blob in the src.
+   */
+  final String srcBlob;
+
+  /**
+   * The SHA1 of the blob in the dst.
+   */
+  final String dstBlob;
+
+  /**
+   * The status of the change. Valid values are:
+   * * A: addition of a file
+   * * C: copy of a file into a new one
+   * * D: deletion of a file
+   * * M: modification of the contents or mode of a file
+   * * R: renaming of a file
+   * * T: change in the type of the file
+   * * U: file is unmerged (you must complete the merge before it can be committed)
+   * * X: "unknown" change type (most probably a bug, please report it)
+   *
+   * Status letters C and R are always followed by a score (denoting the
+   * percentage of similarity between the source and target of the move or
+   * copy), and are the only ones to be so.
+   */
+  final String status;
+
+  /**
+   * The path of the src.
+   */
+  final String srcPath;
+
+  /**
+   * The path of the dst if this was either a copy or a rename operation.
+   */
+  final String dstPath;
+
+  /**
+   * Initialize a newly created diff record.
+   */
+  DiffRecord(this.repository, this.srcBlob, this.dstBlob, this.status,
+      this.srcPath, this.dstPath);
+
+  /**
+   * Return `true` if this record represents a file that was added.
+   */
+  bool get isAddition => status == 'A';

[Paul Berry, 2015/11/17 19:49:31]

Rather than having a bunch of "is" getters that are mutually exclusive by
convention, I'd prefer to have a single "kind" getter that returns an enum. 
That way clients clients can process a diff record using a switch statement
rather than having to do a bunch of cascaded "if" tests, and then they get the
benefit of the "missing case clause" warnings if they forget to handle one of
the possibilities.

[Brian Wilkerson, 2015/11/18 17:36:46]

I think you can do that today:

    switch (record.status.substring(0, 1)) {
      case 'A':
        ...

If it becomes useful we could add a getter to return the first character.

+
+  /**
+   * Return `true` if this record represents a file that was copied.
+   */
+  bool get isCopy => status.startsWith('C');
+
+  /**
+   * Return `true` if this record represents a file that was deleted.
+   */
+  bool get isDeletion => status == 'D';
+
+  /**
+   * Return `true` if this record represents a file that was modified.
+   */
+  bool get isModification => status == 'M';
+
+  /**
+   * Return `true` if this record represents a file that was renamed.
+   */
+  bool get isRename => status.startsWith('R');
+
+  /**
+   * Return `true` if this record represents an entity whose type was changed
+   * (for example, from a file to a directory).
+   */
+  bool get isTypeChange => status == 'T';
+
+  /**
+   * Return a representation of the individual blobs within this diff.
+   */
+  BlobDiff getBlobDiff() => repository.getBlobDiff(srcBlob, dstBlob);
+
+  /**
+   * Return `true` if this diff applies to a file with the given name.
+   */
+  bool isFor(String fileName) =>
+      (srcPath != null && fileName == path.basename(srcPath)) ||
+          (dstPath != null && fileName == path.basename(dstPath));
+
+  @override
+  String toString() => srcPath ?? dstPath;
+}
+
+/**
+ * A representation of a git repository.
+ */
+class GitRepository {
+  /**
+   * The absolute path of the directory containing the repository.
+   */
+  final String path;
+
+  /**
+   * Initialize a newly created repository to represent the git repository at
+   * the given [path].
+   */
+  GitRepository(this.path);
+
+  /**
+   * Checkout the given [commit] from the repository. This is done by running
+   * the command `git checkout <sha>`.
+   */
+  void checkout(String commit) {
+    _run('checkout', commit);
+  }
+
+  /**
+   * Return details about the differences between the two blobs identified by
+   * the SHA1 of the [srcBlob] and the SHA1 of the [dstBlob]. This is done by
+   * running the command `git diff <blob> <blob>`.
+   */
+  BlobDiff getBlobDiff(String srcBlob, String dstBlob) {
+    ProcessResult result = _run('diff', srcBlob, dstBlob);
+    List<String> diffResults = LineSplitter.split(result.stdout).toList();
+    return new BlobDiff(diffResults);
+  }
+
+  /**
+   * Return details about the differences between the two commits identified by
+   * the [srcCommit] and [dstCommit]. This is done by running the command
+   * `git diff --raw --no-abbrev --no-renames -z <sha> <sha>`.
+   */
+  CommitDelta getCommitDiff(String srcCommit, String dstCommit) {
+    // Consider --find-renames instead of --no-renames if rename information is
+    // desired.
+    ProcessResult result = _run('diff', '--raw', '--no-abbrev', '--no-renames',
+        '-z', srcCommit, dstCommit);
+    return new CommitDelta(this, result.stdout);
+  }
+
+  /**
+   * Return a representation of the history of this repository. This is done by
+   * running the command `git rev-list --first-parent HEAD`.
+   */
+  CommitHistory getCommitHistory() {
+    ProcessResult result = _run('rev-list', '--first-parent', 'HEAD');
+    List<String> commitIds = LineSplitter.split(result.stdout).toList();
+    return new CommitHistory(this, commitIds);
+  }
+
+  /**
+   * Synchronously run the given [executable] with the given [arguments]. Return
+   * the result of running the process.
+   */
+  ProcessResult _run(String arg1,
+      [String arg2,

[Paul Berry, 2015/11/17 19:49:31]

These optional args, and the if tree below, seem like a lot of effort to go to
just so that we can get behavior that's reminiscent of variable length arguments
in Java.  Why not just do:

ProcessResult _run(List<String> arguments) {
  return Process.runSync(...);
}

And then add "[" and "]" characters at the call sites, e.g.:

ProcessResult result = _run(['rev-list', '--first-parent', 'HEAD'])

[Brian Wilkerson, 2015/11/18 17:36:46]

Just following a pattern I saw somewhere else. I've change it.

+      String arg3,
+      String arg4,
+      String arg5,
+      String arg6,
+      String arg7]) {
+    List<String> arguments = <String>[];
+    arguments.add(arg1);
+    if (arg2 != null) {
+      arguments.add(arg2);
+      if (arg3 != null) {
+        arguments.add(arg3);
+        if (arg4 != null) {
+          arguments.add(arg4);
+          if (arg5 != null) {
+            arguments.add(arg5);
+            if (arg6 != null) {
+              arguments.add(arg6);
+              if (arg7 != null) {
+                arguments.add(arg7);
+              }
+            }
+          }
+        }
+      }
+    }
+    return Process.runSync('git', arguments,
+        stderrEncoding: UTF8, stdoutEncoding: UTF8, workingDirectory: path);
+  }
+}