Enable the ninja parsing code all the time.

isolate_driver: Enable ninja parsing code all the time.

Split the ninja parsing code in its own file so that isolate_driver is focused
on .isolate management.

- This removes the need to specify the majority of binary runtime dependencies.
  Sadly, this means a few unrelated binaries are now pulled in, for example d8,
  libvpx_obj_int_extract and tls_edit. That's a maintenance/performance trade
  off.
- Removed some dependencies for unit_tests.isolate to ensure this works.

R=vadimsh@chromium.org
BUG=368543
Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=269140

[M-A Ruel, 2014-05-07 16:15:49]

I'm double checking with:
python -c "import json; print
'\n'.join(sorted(json.load(open('out/Release/unit_tests.isolated'))['files']))"

The new parser catches many more dependencies.

[Vadim Sh., 2014-05-07 18:08:33]

Mostly documentation related nits.

https://codereview.chromium.org/270333002/diff/20001/chrome/unit_tests.isolate
File chrome/unit_tests.isolate (right):

https://codereview.chromium.org/270333002/diff/20001/chrome/unit_tests.isolat...
chrome/unit_tests.isolate:51: '<(PRODUCT_DIR)/locales/fr.pak',
side note: why only 'fr'? :) it looks suspicious :)

https://codereview.chromium.org/270333002/diff/20001/chrome/unit_tests.isolat...
chrome/unit_tests.isolate:107: #         '<(PRODUCT_DIR)/chrome_elf.dll',
Why not just remove it? We use source control system after all...

Or at least add a comment describing why these deps are commented out.

https://codereview.chromium.org/270333002/diff/20001/tools/ninja_parser.py
File tools/ninja_parser.py (right):

https://codereview.chromium.org/270333002/diff/20001/tools/ninja_parser.py#ne...
tools/ninja_parser.py:203: """Loads build.ninja and the tree of .ninja files in
build_dir.
Doc string: What does it return?

https://codereview.chromium.org/270333002/diff/20001/tools/ninja_parser.py#ne...
tools/ninja_parser.py:222: """Finds all the links from 'target' to
'search_for'."""
Doc string: What is 'link' exactly? List of targets?

https://codereview.chromium.org/270333002/diff/20001/tools/ninja_parser.py#ne...
tools/ninja_parser.py:228: """Processes the dependency list with OS specific
rules."""
Doc string: what does it do exactly? Nor name, nor "processes the dependency
list" doesn't hint on why this function exists...

Also mention somewhere that it works only after ninja built all the targets
(since it checks X file flag). I think it is worth to mention this assumption in
module-level doc string. i.e. when I see 'ninja_parser' my first assumption
would be that it works purely on top of ninja files, not the build output
directory.

[Vadim Sh., 2014-05-07 18:08:33]

Mostly documentation related nits.

https://codereview.chromium.org/270333002/diff/20001/chrome/unit_tests.isolate
File chrome/unit_tests.isolate (right):

https://codereview.chromium.org/270333002/diff/20001/chrome/unit_tests.isolat...
chrome/unit_tests.isolate:51: '<(PRODUCT_DIR)/locales/fr.pak',
side note: why only 'fr'? :) it looks suspicious :)

https://codereview.chromium.org/270333002/diff/20001/chrome/unit_tests.isolat...
chrome/unit_tests.isolate:107: #         '<(PRODUCT_DIR)/chrome_elf.dll',
Why not just remove it? We use source control system after all...

Or at least add a comment describing why these deps are commented out.

https://codereview.chromium.org/270333002/diff/20001/tools/ninja_parser.py
File tools/ninja_parser.py (right):

https://codereview.chromium.org/270333002/diff/20001/tools/ninja_parser.py#ne...
tools/ninja_parser.py:203: """Loads build.ninja and the tree of .ninja files in
build_dir.
Doc string: What does it return?

https://codereview.chromium.org/270333002/diff/20001/tools/ninja_parser.py#ne...
tools/ninja_parser.py:222: """Finds all the links from 'target' to
'search_for'."""
Doc string: What is 'link' exactly? List of targets?

https://codereview.chromium.org/270333002/diff/20001/tools/ninja_parser.py#ne...
tools/ninja_parser.py:228: """Processes the dependency list with OS specific
rules."""
Doc string: what does it do exactly? Nor name, nor "processes the dependency
list" doesn't hint on why this function exists...

Also mention somewhere that it works only after ninja built all the targets
(since it checks X file flag). I think it is worth to mention this assumption in
module-level doc string. i.e. when I see 'ninja_parser' my first assumption
would be that it works purely on top of ninja files, not the build output
directory.

[M-A Ruel, 2014-05-07 19:39:08]

https://codereview.chromium.org/270333002/diff/20001/chrome/unit_tests.isolate
File chrome/unit_tests.isolate (right):

https://codereview.chromium.org/270333002/diff/20001/chrome/unit_tests.isolat...
chrome/unit_tests.isolate:51: '<(PRODUCT_DIR)/locales/fr.pak',
On 2014/05/07 18:08:34, Vadim Sh. wrote:
> side note: why only 'fr'? :) it looks suspicious :)

Russians are welcome to add theirs if needed.

https://codereview.chromium.org/270333002/diff/20001/chrome/unit_tests.isolat...
chrome/unit_tests.isolate:107: #         '<(PRODUCT_DIR)/chrome_elf.dll',
On 2014/05/07 18:08:34, Vadim Sh. wrote:
> Why not just remove it? We use source control system after all...
> 
> Or at least add a comment describing why these deps are commented out.

Added back, it was an error on my part.

https://codereview.chromium.org/270333002/diff/20001/tools/ninja_parser.py
File tools/ninja_parser.py (right):

https://codereview.chromium.org/270333002/diff/20001/tools/ninja_parser.py#ne...
tools/ninja_parser.py:203: """Loads build.ninja and the tree of .ninja files in
build_dir.
On 2014/05/07 18:08:34, Vadim Sh. wrote:
> Doc string: What does it return?

Done.

https://codereview.chromium.org/270333002/diff/20001/tools/ninja_parser.py#ne...
tools/ninja_parser.py:222: """Finds all the links from 'target' to
'search_for'."""
On 2014/05/07 18:08:34, Vadim Sh. wrote:
> Doc string: What is 'link' exactly? List of targets?

Done.

https://codereview.chromium.org/270333002/diff/20001/tools/ninja_parser.py#ne...
tools/ninja_parser.py:228: """Processes the dependency list with OS specific
rules."""
On 2014/05/07 18:08:34, Vadim Sh. wrote:
> Doc string: what does it do exactly? Nor name, nor "processes the dependency
> list" doesn't hint on why this function exists...
> 
> Also mention somewhere that it works only after ninja built all the targets
> (since it checks X file flag). I think it is worth to mention this assumption
in
> module-level doc string. i.e. when I see 'ninja_parser' my first assumption
> would be that it works purely on top of ninja files, not the build output
> directory.

Done.

[Vadim Sh., 2014-05-07 19:41:07]

lgtm

[M-A Ruel, 2014-05-07 19:42:28]

The CQ bit was checked by maruel@chromium.org

[commit-bot: I haz the power, 2014-05-07 19:43:25]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/maruel@chromium.org/270333002/40001

[commit-bot: I haz the power, 2014-05-07 23:52:15]

FYI, CQ is re-trying this CL (attempt #1).
Please consider checking whether the failures are real,
and report flakes to chrome-troopers@google.com.
The failing builders are:
  android_dbg_triggered_tests on tryserver.chromium
  mac_chromium_rel on tryserver.chromium
  win_chromium_x64_rel on tryserver.chromium

[commit-bot: I haz the power, 2014-05-08 04:31:07]

FYI, CQ is re-trying this CL (attempt #1).
Please consider checking whether the failures are real,
and report flakes to chrome-troopers@google.com.
The failing builders are:
  android_dbg_triggered_tests on tryserver.chromium

[M-A Ruel, 2014-05-08 16:41:06]

The CQ bit was unchecked by maruel@chromium.org

[M-A Ruel, 2014-05-08 16:42:59]

The CQ bit was checked by maruel@chromium.org

[commit-bot: I haz the power, 2014-05-08 16:44:02]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/maruel@chromium.org/270333002/40001

[commit-bot: I haz the power, 2014-05-09 01:43:41]

Change committed as 269140

[falken, 2014-05-09 03:12:34]

On 2014/05/09 01:43:41, I haz the power (commit-bot) wrote:
> Change committed as 269140

I think this may have caused various tests to fail on Linux Tests

Log:
http://build.chromium.org/p/chromium.linux/builders/Linux%20Tests/builds/10820

Snippet:
/usr/bin/python
/mnt/data/b/build/slave/Linux_Tests/build/src/tools/swarming_client/isolate.py
run --isolated
/mnt/data/b/build/slave/Linux_Tests/build/src/out/Release/interactive_ui_tests.isolated
-v -- --no-cr --test-launcher-bot-mode
--gtest_output=xml:/mnt/data/b/build/slave/Linux_Tests/build/gtest-results/interactive_ui_tests/interactive_ui_tests.xml
--test-launcher-summary-output=/tmp/tmpldQYtj
[------ Swarming Error ------]
A .isolate file is required.
Traceback (most recent call last):
  File
"/mnt/data/b/build/slave/Linux_Tests/build/src/tools/swarming_client/isolate.py",
line 1516, in main
    return dispatcher.execute(OptionParserIsolate(version=__version__), argv)
  File
"/mnt/data/b/build/slave/Linux_Tests/build/src/tools/swarming_client/third_party/depot_tools/subcommand.py",
line 242, in execute
    return command(parser, args[1:])
  File
"/mnt/data/b/build/slave/Linux_Tests/build/src/tools/swarming_client/isolate.py",
line 1251, in CMDrun
    options, os.getcwd(), None, options.skip_refresh)
  File
"/mnt/data/b/build/slave/Linux_Tests/build/src/tools/swarming_client/isolate.py",
line 863, in load_complete_state
    raise ExecutionError('A .isolate file is required.')
ExecutionError: A .isolate file is required.

[falken, 2014-05-09 03:30:26]

A revert of this CL has been created in
https://codereview.chromium.org/272113002/ by falken@chromium.org.

The reason for reverting is: Speculative revert. Maybe the cause of various bot
failures involving "A .isolate file is required"

Logs:
http://build.chromium.org/p/chromium.linux/builders/Linux%20Tests/builds/10820
http://build.chromium.org/p/chromium.win/builders/Vista%20Tests%20%283%29/bui...

Snippet from Linux Tests log:
/usr/bin/python
/mnt/data/b/build/slave/Linux_Tests/build/src/tools/swarming_client/isolate.py
run --isolated
/mnt/data/b/build/slave/Linux_Tests/build/src/out/Release/interactive_ui_tests.isolated
-v -- --no-cr --test-launcher-bot-mode
--gtest_output=xml:/mnt/data/b/build/slave/Linux_Tests/build/gtest-results/interactive_ui_tests/interactive_ui_tests.xml
--test-launcher-summary-output=/tmp/tmpldQYtj
[------ Swarming Error ------]
A .isolate file is required.
Traceback (most recent call last):
  File
"/mnt/data/b/build/slave/Linux_Tests/build/src/tools/swarming_client/isolate.py",
line 1516, in main
    return dispatcher.execute(OptionParserIsolate(version=__version__), argv)
  File
"/mnt/data/b/build/slave/Linux_Tests/build/src/tools/swarming_client/third_party/depot_tools/subcommand.py",
line 242, in execute
    return command(parser, args[1:])
  File
"/mnt/data/b/build/slave/Linux_Tests/build/src/tools/swarming_client/isolate.py",
line 1251, in CMDrun
    options, os.getcwd(), None, options.skip_refresh)
  File
"/mnt/data/b/build/slave/Linux_Tests/build/src/tools/swarming_client/isolate.py",
line 863, in load_complete_state
    raise ExecutionError('A .isolate file is required.')
ExecutionError: A .isolate file is required..

[falken, 2014-05-09 07:16:50]

The revert seems to have healed the bots.