Split run_tool.py into run_tool.py, extract_edits.py and apply_edits.py

tools/clang/scripts/extract_edits.py

+#!/usr/bin/env python
+# Copyright (c) 2016 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+"""Script to extract edits from clang tool output.
+
+Clang tools are expected to emit edits like this:

[dcheng, 2016/12/27 07:30:14]

Not all clang tools, just rewriter tools, right?

[Łukasz Anforowicz, 2016/12/27 22:33:26]

If a clang tool emits edits, then it is a rewriter tool + the edits should look
like documented below.  I tried to tweak the comment to make it clearer.

+    ...
+    ==== BEGIN EDITS ====
+    <edit1>
+    <edit2>
+    ...
+    ==== END EDITS ====
+    ...
+    ...
+    ==== BEGIN EDITS ====
+    <yet another edit1>
+    <yet another edit2>
+    ...
+    ==== END EDITS ====
+    ...
+
+This python script extracts and dedupes lines between the begin/end marker
+lines.
+
+This python script is mainly needed on Windows.
+On unix this script can be replaced with running sed as follows:
+
+    $ cat run_tool.out \
+        | sed '/^==== BEGIN EDITS ====$/,/^==== END EDITS ====$/{//!b};d'
+        | sort | uniq
+"""
+
+
+import sys
+
+
+def main():
+  unique_lines = set()
+  inside_marker_lines = False
+  for line in sys.stdin:
+    line = line.rstrip("\n\r")
+    if line == '==== BEGIN EDITS ====':
+      inside_marker_lines = True
+      continue
+    if line == '==== END EDITS ====':
+      inside_marker_lines = False
+      continue
+    if inside_marker_lines and line not in unique_lines:
+      unique_lines.add(line)

[danakj, 2016/12/23 15:45:06]

jw why bother with this, it will make this tool slower, and we already have
de-duping in the apply script?

[dcheng, 2016/12/27 07:30:14]

The apply script reads all the edits before deduping. It could be a fairly
decent memory savings from dropping repeated edits in things like headers that
are included a lot.

[Łukasz Anforowicz, 2016/12/27 22:33:26]

I haven't made a change here, but I am open to removing the deduping above:

1. For a big-scale refactoring, one would move the whole, unfiltered output from
all platforms onto a linux box + deduped via sort -u.  For small-scale,
single-platform refactoring, maybe the deduping doesn't matter that much.

2. I like the simplification offered by not deduping:
   - Potential/not-yet-measured performance gains.
   - Possibly making the script reusable in other (hypothetical?) scenarios (if
in the future the begin/end markers are configurable via a parameter?).

[dcheng, 2016/12/28 19:01:36]

I don't feel strongly about this, but one easy way to figure out how many lines
are deduped, right?

[Łukasz Anforowicz, 2016/12/28 19:33:07]

I am sure that this is lots and lots of lines.  I don't know if sort -u would be
more efficient at removing the duplicates.  All in all - I think it is fine to
land as is and we can revisit later if needed.

+      print(line)

[dcheng, 2016/12/27 07:30:14]

For consistency, no ()

[Łukasz Anforowicz, 2016/12/27 22:33:26]

Done.

+  return 0
+
+
+if __name__ == '__main__':
+  sys.exit(main())