Make landmines work on local builds too

Make landmines work on local builds too

Moves (some of) gyp environment setup out of gyp_chromium into separate
module, and shares that between gyp_chromium and landmines.py.

landmines.py is added as the first entry in DEPS hooks so that it can
clobber the entire build directory before running other hooks that
extract/generate into the build dir.

Reland with fix for ios, and for clean pull.

R=iannucci@chromium.org
BUG=400011
Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=289099

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=289546

[scottmg, 2014-08-12 17:22:12]

I tried to make the smallest change possible that I think works. If it seems OK,
we can move more out of gyp_chromium into gyp_environment incrementally.

https://codereview.chromium.org/457003004/diff/1/build/gyp_chromium
File build/gyp_chromium (right):

https://codereview.chromium.org/457003004/diff/1/build/gyp_chromium#newcode271
build/gyp_chromium:271: supplemental_includes = GetSupplementalFiles()
All this stuff could/should be moved to gyp_environment.py too, but
landmine_utils.py already doesn't see these (as they go into command line
arguments, not GYP_DEFINES, e.g.) so it's no worse than it was before. (And I'm
a little scared to move this stuff around because there's so many ways to break
people's random setups and no tests or bots.)

[iannucci, 2014-08-12 18:34:09]

lgtm, probably worth a PSA + CQ.

https://codereview.chromium.org/457003004/diff/20001/build/gyp_environment.py
File build/gyp_environment.py (right):

https://codereview.chromium.org/457003004/diff/20001/build/gyp_environment.py...
build/gyp_environment.py:35: return vs2013_runtime_dll_dirs
weird that this has a side effect and also returns these dirs. should probably
be in docstring

https://codereview.chromium.org/457003004/diff/20001/build/landmines.py
File build/landmines.py (right):

https://codereview.chromium.org/457003004/diff/20001/build/landmines.py#newcode7
build/landmines.py:7: This script runs every build as the first hook. If it
detects that the build
(See DEPS)

https://codereview.chromium.org/457003004/diff/20001/build/landmines.py#newco...
build/landmines.py:38:
'/b/build/slave/ios_rel_device/build/src/xcodebuild/Release-iphoneos'
these should probably change

https://codereview.chromium.org/457003004/diff/20001/build/landmines.py#newco...
build/landmines.py:40: Keep this function in sync with
tools/build/scripts/slave/compile.py
this may need to be done

[scottmg, 2014-08-12 18:50:44]

https://codereview.chromium.org/457003004/diff/20001/build/gyp_environment.py
File build/gyp_environment.py (right):

https://codereview.chromium.org/457003004/diff/20001/build/gyp_environment.py...
build/gyp_environment.py:35: return vs2013_runtime_dll_dirs
On 2014/08/12 18:34:08, iannucci wrote:
> weird that this has a side effect and also returns these dirs. should probably
> be in docstring

Yeah, removed from here. gyp_chromium just does it again now to get those dirs
(more localized).

https://codereview.chromium.org/457003004/diff/20001/build/landmines.py
File build/landmines.py (right):

https://codereview.chromium.org/457003004/diff/20001/build/landmines.py#newcode7
build/landmines.py:7: This script runs every build as the first hook. If it
detects that the build
On 2014/08/12 18:34:08, iannucci wrote:
> (See DEPS)

Done.

https://codereview.chromium.org/457003004/diff/20001/build/landmines.py#newco...
build/landmines.py:38:
'/b/build/slave/ios_rel_device/build/src/xcodebuild/Release-iphoneos'
On 2014/08/12 18:34:08, iannucci wrote:
> these should probably change

Done.

https://codereview.chromium.org/457003004/diff/20001/build/landmines.py#newco...
build/landmines.py:40: Keep this function in sync with
tools/build/scripts/slave/compile.py
On 2014/08/12 18:34:09, iannucci wrote:
> this may need to be done

Yeah, there's a bunch of stuff in there alright. I think options.clobber is...
"fine". I guess that'll happen from the buildbot ui still. The landmines-based
setting of clobber won't ever happen now (because .landmines moved and there's
no .landmines_triggered) so I'll just remove that code in a follow up. (Sound
right-ish?)

[scottmg, 2014-08-12 18:51:08]

The CQ bit was checked by scottmg@chromium.org

[iannucci, 2014-08-12 18:52:10]

On 2014/08/12 18:50:44, scottmg wrote:
> https://codereview.chromium.org/457003004/diff/20001/build/gyp_environment.py
> File build/gyp_environment.py (right):
> 
>
https://codereview.chromium.org/457003004/diff/20001/build/gyp_environment.py...
> build/gyp_environment.py:35: return vs2013_runtime_dll_dirs
> On 2014/08/12 18:34:08, iannucci wrote:
> > weird that this has a side effect and also returns these dirs. should
probably
> > be in docstring
> 
> Yeah, removed from here. gyp_chromium just does it again now to get those dirs
> (more localized).
> 
> https://codereview.chromium.org/457003004/diff/20001/build/landmines.py
> File build/landmines.py (right):
> 
>
https://codereview.chromium.org/457003004/diff/20001/build/landmines.py#newcode7
> build/landmines.py:7: This script runs every build as the first hook. If it
> detects that the build
> On 2014/08/12 18:34:08, iannucci wrote:
> > (See DEPS)
> 
> Done.
> 
>
https://codereview.chromium.org/457003004/diff/20001/build/landmines.py#newco...
> build/landmines.py:38:
> '/b/build/slave/ios_rel_device/build/src/xcodebuild/Release-iphoneos'
> On 2014/08/12 18:34:08, iannucci wrote:
> > these should probably change
> 
> Done.
> 
>
https://codereview.chromium.org/457003004/diff/20001/build/landmines.py#newco...
> build/landmines.py:40: Keep this function in sync with
> tools/build/scripts/slave/compile.py
> On 2014/08/12 18:34:09, iannucci wrote:
> > this may need to be done
> 
> Yeah, there's a bunch of stuff in there alright. I think options.clobber is...
> "fine". I guess that'll happen from the buildbot ui still. The landmines-based
> setting of clobber won't ever happen now (because .landmines moved and there's
> no .landmines_triggered) so I'll just remove that code in a follow up. (Sound
> right-ish?)

sgtm

[commit-bot: I haz the power, 2014-08-12 18:53:26]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/scottmg@chromium.org/457003004/40001

[commit-bot: I haz the power, 2014-08-12 22:48:35]

FYI, CQ is re-trying this CL (attempt #1).
The failing builders are:
  android_aosp on tryserver.chromium.linux
(http://build.chromium.org/p/tryserver.chromium.linux/builders/android_aosp/bu...)
  mac_chromium_rel_swarming on tryserver.chromium.mac
(http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)
  win_chromium_x64_rel on tryserver.chromium.win
(http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_...)

[scottmg, 2014-08-12 22:52:28]

Committed patchset #3 manually as 289099 (presubmit successful).

[iannucci, 2014-08-13 01:59:58]

A revert of this CL (patchset #3) has been created in
https://codereview.chromium.org/469623002/ by iannucci@chromium.org.

The reason for reverting is: Apparently this requires win_toolchain.json to
exist, but I'm not sure how it's supposed to get there (as seen on a clobber
build):


Traceback (most recent call last):
  File "src/build/landmines.py", line 132, in <module>
    sys.exit(main())
  File "src/build/landmines.py", line 119, in main
    gyp_environment.SetEnvironment()
  File "C:\b\build\slave\win_trunk\build\src\build\gyp_environment.py", line 33,
in SetEnvironment
    vs_toolchain.SetEnvironmentAndGetRuntimeDllDirs()
  File "C:\b\build\slave\win_trunk\build\src\build\vs_toolchain.py", line 33, in
SetEnvironmentAndGetRuntimeDllDirs
    with open(json_data_file, 'r') as tempf:
IOError: [Errno 2] No such file or directory:
'C:\\b\\build\\slave\\win_trunk\\build\\src\\build\\win_toolchain.json'.

[scottmg, 2014-08-13 17:43:16]

The CQ bit was checked by scottmg@chromium.org

[commit-bot: I haz the power, 2014-08-13 17:44:24]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/scottmg@chromium.org/457003004/80001

[commit-bot: I haz the power, 2014-08-13 23:48:01]

FYI, CQ is re-trying this CL (attempt #1).
The failing builders are:
  android_aosp on tryserver.chromium.linux
(http://build.chromium.org/p/tryserver.chromium.linux/builders/android_aosp/bu...)
  android_dbg on tryserver.chromium.linux
(http://build.chromium.org/p/tryserver.chromium.linux/builders/android_dbg/bui...)
  mac_chromium_rel_swarming on tryserver.chromium.mac
(http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

[commit-bot: I haz the power, 2014-08-13 23:56:59]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2014-08-13 23:57:00]

Try jobs failed on following builders:
  mac_chromium_rel_swarming on tryserver.chromium.mac
(http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

[scottmg, 2014-08-14 00:36:08]

The CQ bit was checked by scottmg@chromium.org

[commit-bot: I haz the power, 2014-08-14 00:44:53]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/scottmg@chromium.org/457003004/80001

[commit-bot: I haz the power, 2014-08-14 14:03:52]

Committed patchset #4 (80001) as 289546