Split run_tool.py into run_tool.py, extract_edits.py and apply_edits.py

docs/clang_tool_refactoring.md

 # Clang Tool Refactoring
 
 [TOC]
 
 ## Introduction
 
 Clang tools can help with global refactorings of Chromium code. Clang tools can
 take advantage of clang's AST to perform refactorings that would be impossible
 with a traditional find-and-replace regexp:
 
@@ skipping 51 matching lines @@
        ...
 
 ==== END EDITS ====
 ```
 
 The header and footer are required. Each line between the header and footer
 represents one edit. Fields are separated by `:::`, and the first field must
 be `r` (for replacement). In the future, this may be extended to handle header
 insertion/removal. A deletion is an edit with no replacement text.
 
-The edits are applied by [`run_tool.py`](#Running), which understands certain
+The edits are applied by [`apply_edits.py`](#Running), which understands certain
 conventions:
 
 *   The tool should munge newlines in replacement text to `\0`. The script
     knows to translate `\0` back to newlines when applying edits.
 *   When removing an element from a 'list' (e.g. function parameters,
     initializers), the tool should emit a deletion for just the element. The
     script understands how to extend the deletion to remove commas, etc. as
     needed.
 
 TODO: Document more about `SourceLocation` and how spelling loc differs from
@@ skipping 28 matching lines @@
 It is important to use --bootstrap as there appear to be [bugs](https://crbug.com/580745)
 in the clang library this script produces if you build it with gcc, which is the default.
 
 ## Running
 First, build all Chromium targets to avoid failures due to missing dependencies
 that are generated as part of the build:
 
 ```shell
 ninja -C out/Debug  # For non-Windows
 ninja -d keeprsp -C out/Debug  # For Windows
+
+# experimental alternative:
+$gen_targets = $(ninja -C out/gn -t targets all \
+                 | grep '^gen/[^: ]*\.[ch][pc]*:' \
+                 | cut -f 1 -d :`)
+ninja -C out/Debug $gen_targets
 ```
 
 On Windows, generate the compile DB first, and after making any source changes.
 Then omit the `--generate-compdb` in later steps.
 
 ```shell
 tools/clang/scripts/generate_win_compdb.py out/Debug
 ```
 
-Then run the actual tool:
+Then run the actual clang tool to generate a list of edits:
 
 ```shell
 tools/clang/scripts/run_tool.py <toolname> \
   --generate-compdb
-  out/Debug <path 1> <path 2> ...
+  out/Debug <path 1> <path 2> ... >/tmp/list-of-edits.debug
 ```
 
 `--generate-compdb` can be omitted if the compile DB was already generated and
 the list of build flags and source files has not changed since generation.
 
 `<path 1>`, `<path 2>`, etc are optional arguments to filter the files to run
-the tool across. This is helpful when sharding global refactorings into smaller
+the tool against. This is helpful when sharding global refactorings into smaller
 chunks. For example, the following command will run the `empty_string` tool
-across just the files in `//base`:
+against just the `.c`, `.cc`, `.cpp`, `.m`, `.mm` files in `//net`.  Note that
+the filtering is not applied to the *output* of the tool - the tool can emit
+edits that apply to files outside of `//cc` (i.e. edits that apply to headers
+from `//base` that got included by source files in `//cc`).
+
+Note that some header files might only be included from generated files
+(e.g. from only from some `.cpp` files under out/Debug/gen).  To make sure
+that contents of such header files are processed by the tool, the tool

[dcheng, 2016/12/28 19:01:36]

the tool => run_tool.py (there's too many different tools here, that the use of
the generic 'tool' label is confusing)

[Łukasz Anforowicz, 2016/12/28 19:33:07]

Done (but I think it is more correct to say "clang tool" here, not
"run_tool.py").

+needs to be run against the generated files.  The only way to accomplish this
+today is to pass `--all` switch to `run_tool.py` - this will run the tool
+against all the sources from the compilation database.
 
 ```shell
 tools/clang/scripts/run_tool.py empty_string  \
   --generated-compdb \
-  out/Debug base
+  out/Debug net >/tmp/list-of-edits.debug
 ```
 
+Finally, apply the edits as follows:
+
+```shell
+cat /tmp/list-of-edits.debug \
+  | tools/clang/scripts/extract_edits.py \
+  | tools/clang/scripts/apply_edits.py out/Debug <path 1> <path 2> ...
+```
+
+The tool will only apply edits to files actually under control of `git`.

[dcheng, 2016/12/28 19:01:36]

The tool => apply_edits.py

[Łukasz Anforowicz, 2016/12/28 19:33:07]

Done.

+`<path 1>`, `<path 2>`, etc are optional arguments to further filter the files
+that the edits are applied to.  Note that semantics of these filters is
+distinctly different from the arguments of `run_tool.py` filters - one set of
+filters controls which files are edited, the other set of filters controls which
+files the clang tool is run against.
+
 ## Debugging
 Dumping the AST for a file:
 
 ```shell
 clang++ -cc1 -ast-dump foo.cc
 ```
 
 Using `clang-query` to dynamically test matchers (requires checking out
 and building [clang-tools-extras](https://github.com/llvm-mirror/clang-tools-extra)):
 
@@ skipping 18 matching lines @@
 
 ```shell
 tools/clang/scripts/test_tool.py <tool name>
 ```
 
 The name of the tool binary and the subdirectory for the tool in
 `//tools/clang` must match. The test runner finds all files that match the
 pattern `//tools/clang/<tool name>/tests/*-original.cc`, runs the tool across
 those files, and compared it to the `*-expected.cc` version. If there is a
 mismatch, the result is saved in `*-actual.cc`.