Split run_tool.py into run_tool.py, extract_edits.py and apply_edits.py

docs/clang_tool_refactoring.md

Index: docs/clang_tool_refactoring.md
diff --git a/docs/clang_tool_refactoring.md b/docs/clang_tool_refactoring.md
index 40a8dae053463b6ef894eb8111fe905a4fe3c8c8..0e1edc80081e6072dbcddc6f4208e8f912af449e 100644
--- a/docs/clang_tool_refactoring.md
+++ b/docs/clang_tool_refactoring.md
@@ -69,7 +69,7 @@ represents one edit. Fields are separated by `:::`, and the first field must
 be `r` (for replacement). In the future, this may be extended to handle header
 insertion/removal. A deletion is an edit with no replacement text.
 
-The edits are applied by [`run_tool.py`](#Running), which understands certain
+The edits are applied by [`apply_edits.py`](#Running), which understands certain
 conventions:
 
 *   The tool should munge newlines in replacement text to `\0`. The script
@@ -118,6 +118,12 @@ that are generated as part of the build:
 ```shell
 ninja -C out/Debug  # For non-Windows
 ninja -d keeprsp -C out/Debug  # For Windows
+
+# experimental alternative:
+$gen_targets = $(ninja -C out/gn -t targets all \
+                 | grep '^gen/[^: ]*\.[ch][pc]*:' \
+                 | cut -f 1 -d :`)
+ninja -C out/Debug $gen_targets
 ```
 
 On Windows, generate the compile DB first, and after making any source changes.
@@ -127,28 +133,53 @@ Then omit the `--generate-compdb` in later steps.
 tools/clang/scripts/generate_win_compdb.py out/Debug
 ```
 
-Then run the actual tool:
+Then run the actual clang tool to generate a list of edits:
 
 ```shell
 tools/clang/scripts/run_tool.py <toolname> \
   --generate-compdb
-  out/Debug <path 1> <path 2> ...
+  out/Debug <path 1> <path 2> ... >/tmp/list-of-edits.debug
 ```
 
 `--generate-compdb` can be omitted if the compile DB was already generated and
 the list of build flags and source files has not changed since generation.
 
 `<path 1>`, `<path 2>`, etc are optional arguments to filter the files to run
-the tool across. This is helpful when sharding global refactorings into smaller
+the tool against. This is helpful when sharding global refactorings into smaller
 chunks. For example, the following command will run the `empty_string` tool
-across just the files in `//base`:
+against just the `.c`, `.cc`, `.cpp`, `.m`, `.mm` files in `//net`.  Note that
+the filtering is not applied to the *output* of the tool - the tool can emit
+edits that apply to files outside of `//cc` (i.e. edits that apply to headers
+from `//base` that got included by source files in `//cc`).
+
+Note that some header files might only be included from generated files
+(e.g. from only from some `.cpp` files under out/Debug/gen).  To make sure
+that contents of such header files are processed by the tool, the tool

[dcheng, 2016/12/28 19:01:36]

the tool => run_tool.py (there's too many different tools here, that the use of
the generic 'tool' label is confusing)

[Łukasz Anforowicz, 2016/12/28 19:33:07]

Done (but I think it is more correct to say "clang tool" here, not
"run_tool.py").

+needs to be run against the generated files.  The only way to accomplish this
+today is to pass `--all` switch to `run_tool.py` - this will run the tool
+against all the sources from the compilation database.
 
 ```shell
 tools/clang/scripts/run_tool.py empty_string  \
   --generated-compdb \
-  out/Debug base
+  out/Debug net >/tmp/list-of-edits.debug
+```
+
+Finally, apply the edits as follows:
+
+```shell
+cat /tmp/list-of-edits.debug \
+  | tools/clang/scripts/extract_edits.py \
+  | tools/clang/scripts/apply_edits.py out/Debug <path 1> <path 2> ...
 ```
 
+The tool will only apply edits to files actually under control of `git`.

[dcheng, 2016/12/28 19:01:36]

The tool => apply_edits.py

[Łukasz Anforowicz, 2016/12/28 19:33:07]

Done.

+`<path 1>`, `<path 2>`, etc are optional arguments to further filter the files
+that the edits are applied to.  Note that semantics of these filters is
+distinctly different from the arguments of `run_tool.py` filters - one set of
+filters controls which files are edited, the other set of filters controls which
+files the clang tool is run against.
+
 ## Debugging
 Dumping the AST for a file: