Allow using goma on Windows with symbol_level == 2

Allow using goma on Windows with symbol_level == 2

Traditionally goma for Chrome has been less useful on Windows than on
other platforms because it was incompatible with full debug information.
Building with goma requires using /Z7 instead of /Zi, and this causes
the linker's memory usage and runtime to blow up as all of the debug
information is merged.

However, /debug:fastlink makes this work. Because it doesn't merge all
of the debug information it makes goma and /Z7 practical. Full release
component builds can be done in less than fifteen minutes, with
incremental builds taking just a few seconds. Without goma a full
release component build of Chrome can easily take 40+ minutes, even on a
Z840.

Goma's speedup comes from massively parallelizing the compile phase,
however even with /debug:fastlink the linking phases are longer with the
/Z7 switch that is required by goma. A /debug:fastlink of chrome.dll in
a component build goes from ~32 seconds to ~110 seconds on a Z620 when
/Z7 (goma) is selected. This penalty will be reduced by VC++ 2017 which
claims 30% speed improvements on /debug:fastlink.

So, if you frequently need to do full relinks then goma may still be a
bad choice. However for most scenarios this change should make goma a
good choice for component builds of Chrome, even with full debug
information enabled. To make use of this ability you need to explicitly
specify the switches below - symbol_level will otherwise default to 1
when use_goma == true.

    use_goma = true
    is_win_fastlink = true
    symbol_level = 2

The fastlink PDBs work well for most scenarios but there are a few
scenarios (rare for most people) where they do not work:
- Copying PDBs to another machine (fails due to references to the .obj
files)
- Reporting on all symbols/globals with SymbolSort or similar fails
- ETW tracing fails
- VS heap profiling fails
Ideally mspdbcmf.exe would let us create "normal" PDBs when needed but
in practice this appears to be unusable with /Z7 created PDBs.

R=scottmg@chromium.org

Review-Url: https://codereview.chromium.org/2661023010
Cr-Commit-Position: refs/heads/master@{#447892}
Committed: https://chromium.googlesource.com/chromium/src/+/4a3cadb2d998fb8f477d0ebc7ff13619b123a805

[brucedawson, 2017-02-01 21:34:21]

The CQ bit was checked by brucedawson@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-02-01 21:34:41]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[brucedawson, 2017-02-01 21:37:03]

Description was changed from

==========
Allow using goma on Windows with symbol_level == 2

Traditionally goma for Chrome has been less useful on Windows than on
other platforms because it was incompatible with full debug information.
Building with goma requires using /Z7 instead of /Zi, and this causes
the linker's memory usage and runtime to blow up as all of the debug
information is merged.

However, /debug:fastlink makes this work. Because it doesn't merge all
of the debug information it makes goma and /Z7 practical. Full release
component builds can be done in less than fifteen minutes, with
incremental builds taking just a few seconds. Without goma a full
release component build of Chrome can easily take 40+ minutes, even on a
Z840.

Goma's speedup comes from massively parallelizing the compile phase,
however even with /debug:fastlink the linking phases are longer with the
/Z7 switch that is required by goma. A /debug:fastlink of chrome.dll in
a component build goes from ~32 seconds to ~110 seconds on a Z620 when
/Z7 (goma) is selected. This penalty will be reduced by VC++ 2017 which
claims 30% speed improvements on /debug:fastlink.

So, if you frequently need to do full relinks then goma may still be a
bad choice. However for most scenarios this change should make goma a
good choice for component builds of Chrome, even with full debug
information enabled. To make use of this ability you need to explicitly
specify the switches below - symbol_level will otherwise default to 1
when use_goma == true.

    use_goma = true
    is_win_fastlink = true
    symbol_level = 2

R=scottmg@chromium.org
==========

to

==========
Allow using goma on Windows with symbol_level == 2

Traditionally goma for Chrome has been less useful on Windows than on
other platforms because it was incompatible with full debug information.
Building with goma requires using /Z7 instead of /Zi, and this causes
the linker's memory usage and runtime to blow up as all of the debug
information is merged.

However, /debug:fastlink makes this work. Because it doesn't merge all
of the debug information it makes goma and /Z7 practical. Full release
component builds can be done in less than fifteen minutes, with
incremental builds taking just a few seconds. Without goma a full
release component build of Chrome can easily take 40+ minutes, even on a
Z840.

Goma's speedup comes from massively parallelizing the compile phase,
however even with /debug:fastlink the linking phases are longer with the
/Z7 switch that is required by goma. A /debug:fastlink of chrome.dll in
a component build goes from ~32 seconds to ~110 seconds on a Z620 when
/Z7 (goma) is selected. This penalty will be reduced by VC++ 2017 which
claims 30% speed improvements on /debug:fastlink.

So, if you frequently need to do full relinks then goma may still be a
bad choice. However for most scenarios this change should make goma a
good choice for component builds of Chrome, even with full debug
information enabled. To make use of this ability you need to explicitly
specify the switches below - symbol_level will otherwise default to 1
when use_goma == true.

    use_goma = true
    is_win_fastlink = true
    symbol_level = 2

R=scottmg@chromium.org
==========

[brucedawson, 2017-02-01 21:40:05]

I noticed that the clang team was using /Z7 + goma + /debug:fastlink to get fast
clang-cl builds and I realized there was nothing preventing us from doing the
same.

It's not perfect (full links are still a bit slower) but incremental linking +
/debug:fastlink seems to make full debug information plus goma practical.

The only big risk I can see is increased demand for goma resources - is that a
potential concern ukai@?

I plan to do more testing and then share updated recommendations on
chromium-dev@

[commit-bot: I haz the power, 2017-02-01 21:42:27]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-01 21:42:28]

Dry run: Try jobs failed on following builders:
  win_chromium_compile_dbg_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_comp...)
  win_chromium_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[brucedawson, 2017-02-01 21:44:44]

Huh. Several build failures. These may be uncovering existing problems or maybe
I need to tweak my assert.

[brucedawson, 2017-02-01 21:52:01]

The CQ bit was checked by brucedawson@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-02-01 21:52:28]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[scottmg, 2017-02-01 21:56:43]

https://codereview.chromium.org/2661023010/diff/20001/build/config/compiler/B...
File build/config/compiler/BUILD.gn (right):

https://codereview.chromium.org/2661023010/diff/20001/build/config/compiler/B...
build/config/compiler/BUILD.gn:1585: cflags = [ "/Z7" ]  # Debug information in
the .obj files.
Can you not just set is_win_fastlink here instead of asserting in the other
location? (That's really a question, I don't know if it'll work or not.)

[grt (UTC plus 2), 2017-02-01 22:06:37]

https://codereview.chromium.org/2661023010/diff/20001/build/config/compiler/B...
File build/config/compiler/BUILD.gn (right):

https://codereview.chromium.org/2661023010/diff/20001/build/config/compiler/B...
build/config/compiler/BUILD.gn:1585: cflags = [ "/Z7" ]  # Debug information in
the .obj files.
On 2017/02/01 21:56:43, scottmg wrote:
> Can you not just set is_win_fastlink here instead of asserting in the other
> location? (That's really a question, I don't know if it'll work or not.)

1) You are awesome, Bruce. I was just searching for goma advice and came across
this CL.
2) I like the idea of flipping one switch to get sane behavior rather than
instructions like "set A and B to get cool feature". I believe that other GN
knobs work this way, so I hope it's possible here.

[brucedawson, 2017-02-01 22:08:55]

On 2017/02/01 21:56:43, scottmg wrote:
>
https://codereview.chromium.org/2661023010/diff/20001/build/config/compiler/B...
> File build/config/compiler/BUILD.gn (right):
> 
>
https://codereview.chromium.org/2661023010/diff/20001/build/config/compiler/B...
> build/config/compiler/BUILD.gn:1585: cflags = [ "/Z7" ]  # Debug information
in
> the .obj files.
> Can you not just set is_win_fastlink here instead of asserting in the other
> location? (That's really a question, I don't know if it'll work or not.)

Good question. Yes, that works. So, the question becomes what is the least
surprising thing to do. Currently enabling goma sets symbol_level to 1, which is
one level of surprise, done in aid of avoiding a bad experience. We could make
it so that goma automatically sets is_win_fastlink if the user sets symbol_level
to 2, and we could also make it so that goma sets symbol_level to 2 if the user
sets is_win_fastlink.

Automatically increasing the symbol_level probably just adds confusion, so I'm
not actually proposing that. I do sort of wish that we hadn't started lowering
symbol_level for goma builds, but I'm not sure we should change that now.

The one downside to automatically enabling is_win_fastlink is that it will
silently create PDBs which are slightly less capable - you can't run symbolsort
or static_initializers reports on them, you can't use Visual Studio's memory
profiler with them, and you can't move them to other machines. So I *think* we
don't want to automatically turn it on, but I'm prepared to be convinced
otherwise.

[brucedawson, 2017-02-01 22:13:15]

> 1) You are awesome, Bruce. I was just searching for goma advice and came
across
> this CL.

Thanks! I'm not sure why it took me six months to think of this.

> 2) I like the idea of flipping one switch to get sane behavior rather than
> instructions like "set A and B to get cool feature". I believe that other GN
> knobs work this way, so I hope it's possible here.

We *can* make it so that use_goma leaves symbol_level alone and sets
is_win_fastlink. This will work. I think most builders set symbol_level == 1 so
it shouldn't affect them.

The only question is whether we should. The (minor) downsides to is_win_fastlink
make me slightly worried about automatically turning it on.

[scottmg, 2017-02-01 22:24:34]

On 2017/02/01 22:08:55, brucedawson (slow) wrote:
> On 2017/02/01 21:56:43, scottmg wrote:
> >
>
https://codereview.chromium.org/2661023010/diff/20001/build/config/compiler/B...
> > File build/config/compiler/BUILD.gn (right):
> > 
> >
>
https://codereview.chromium.org/2661023010/diff/20001/build/config/compiler/B...
> > build/config/compiler/BUILD.gn:1585: cflags = [ "/Z7" ]  # Debug information
> in
> > the .obj files.
> > Can you not just set is_win_fastlink here instead of asserting in the other
> > location? (That's really a question, I don't know if it'll work or not.)
> 
> Good question. Yes, that works. So, the question becomes what is the least
> surprising thing to do. Currently enabling goma sets symbol_level to 1, which
is
> one level of surprise, done in aid of avoiding a bad experience. We could make
> it so that goma automatically sets is_win_fastlink if the user sets
symbol_level
> to 2, and we could also make it so that goma sets symbol_level to 2 if the
user
> sets is_win_fastlink.
> 
> Automatically increasing the symbol_level probably just adds confusion, so I'm
> not actually proposing that. I do sort of wish that we hadn't started lowering
> symbol_level for goma builds, but I'm not sure we should change that now.
> 
> The one downside to automatically enabling is_win_fastlink is that it will
> silently create PDBs which are slightly less capable - you can't run
symbolsort
> or static_initializers reports on them, you can't use Visual Studio's memory
> profiler with them, and you can't move them to other machines. So I *think* we
> don't want to automatically turn it on, but I'm prepared to be convinced
> otherwise.

I agree it's a bit sticky.

I think I'd go with removing the win/goma special cases that turns down symbols
(so win-goma would mostly get symbol_level=2), since that special case seems
more confusing than the possibility of moving builds amongst machines or running
(relatively obscure) pdb tools.

I guess looking at it another way, the default without goma is "symbols". So
now, with goma the default would be "symbols (that you can't move to another
machine)", vs. "not really getting symbols at all". So that seems closer to the
defaults/expected in other cases/platforms.

But I'm OK with whichever way you prefer. I can't use win-goma anyway. :)

[brucedawson, 2017-02-01 22:26:28]

Good way of looking at it Scott. I'll check in with the other dev who
collaborated on the original goma/symbol change.

Possible concrete example: crbug.com/659439 (coincidentally popping up today)
seems to be happening because we are using goma and symbol_level 2 on a builder.
I think that the correct fix is to go to symbol_level 1 but auto-enabling
is_win_fastlink would have auto-fixed it in the other way.

On the other hand, it would have auto-fixed it.

[brucedawson, 2017-02-01 22:28:35]

brucedawson@chromium.org changed reviewers:
+ dpranke@chromium.org

[brucedawson, 2017-02-01 22:29:58]

dpranke@ - you landed crrev.com/2296953002 that tweaked my change to have goma
imply symbol_level == 1. What do you think about this discussion?

Crucial context: with is_win_fastlink it seems to work quite well to have goma
and symbol_level==2.

[commit-bot: I haz the power, 2017-02-02 00:11:45]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-02 00:11:46]

Dry run: Try jobs failed on following builders:
  android_n5x_swarming_rel on master.tryserver.chromium.android (JOB_FAILED,
https://build.chromium.org/p/tryserver.chromium.android/builders/android_n5x_...)

[brucedawson, 2017-02-02 00:15:38]

On 2017/02/01 22:29:58, brucedawson (slow) wrote:
> dpranke@ - you landed crrev.com/2296953002 that tweaked my change to have goma
> imply symbol_level == 1. What do you think about this discussion?
> 
> Crucial context: with is_win_fastlink it seems to work quite well to have goma
> and symbol_level==2.

Other details: I have a goma component build that I am trying to profile with
ETW and it is working badly. If I try to convert chrome.dll.pdb to a
non-fastlink PDB with mspdbcmf.exe it takes forever (32 minutes and counting).
If I try to load the trace into WPA so that it can read the PDB directly it
crashes.

Sigh...

I'll report these bugs but they have me leaning towards not defaulting to
is_win_fastlink.

[Dirk Pranke, 2017-02-02 01:49:18]

I do care about this!

What's the perf difference if you set is_win_fastlink=true on a non-goma build
(either or both of debug and release)? Does it speed things up regardless of
symbol level (maybe has no effect if symbol_level=0)?

Is the downside to fastlink the fact that we'd have to copy the object files
around in order to get debug information? Are there other downsides?

Generally, GN args should work so that developers (not bots) have to set the
fewest flags to get the sanest defaults. If I was right above, then since most
developers aren't copying files around, maybe is_win_fastlink should just be on
by default?

[brucedawson, 2017-02-02 01:54:28]

On 2017/02/02 01:49:18, Dirk Pranke wrote:
> I do care about this!
> 
> What's the perf difference if you set is_win_fastlink=true on a non-goma build
> (either or both of debug and release)? Does it speed things up regardless of
> symbol level (maybe has no effect if symbol_level=0)?
> 
> Is the downside to fastlink the fact that we'd have to copy the object files
> around in order to get debug information? Are there other downsides?
> 
> Generally, GN args should work so that developers (not bots) have to set the
> fewest flags to get the sanest defaults. If I was right above, then since most
> developers aren't copying files around, maybe is_win_fastlink should just be
on
> by default?

The non-portability of fastlink PDBs is one problem, and I think it is one that
is rarely hit. It matters for official builds, of course, but not often
otherwise.
The other failings of fastlink PDBs probably hit me more than everybody else on
the team combined - they cause problems for ETW, for VS heap profiling, and for
analyzing global variables.

The ETW problem is probably the most serious. It looks like it "just doesn't
work (tm)" with /Z7 and fastlink - I had to redo my build to get traces. But
maybe that is rare enough that fastlink should still be the default.

I'd say we should proceed towards that but proceed cautiously. Maybe land this
change as-is and have people start using it. If it doesn't break anything, then
go further.

I haven't tested fastlink speed on other symbol levels so I don't know if it
helps. I would expect minimal benefit.

[Dirk Pranke, 2017-02-02 22:29:05]

lgtm as-is, then, and we can continue to iterate.

[brucedawson, 2017-02-02 22:32:10]

Description was changed from

==========
Allow using goma on Windows with symbol_level == 2

Traditionally goma for Chrome has been less useful on Windows than on
other platforms because it was incompatible with full debug information.
Building with goma requires using /Z7 instead of /Zi, and this causes
the linker's memory usage and runtime to blow up as all of the debug
information is merged.

However, /debug:fastlink makes this work. Because it doesn't merge all
of the debug information it makes goma and /Z7 practical. Full release
component builds can be done in less than fifteen minutes, with
incremental builds taking just a few seconds. Without goma a full
release component build of Chrome can easily take 40+ minutes, even on a
Z840.

Goma's speedup comes from massively parallelizing the compile phase,
however even with /debug:fastlink the linking phases are longer with the
/Z7 switch that is required by goma. A /debug:fastlink of chrome.dll in
a component build goes from ~32 seconds to ~110 seconds on a Z620 when
/Z7 (goma) is selected. This penalty will be reduced by VC++ 2017 which
claims 30% speed improvements on /debug:fastlink.

So, if you frequently need to do full relinks then goma may still be a
bad choice. However for most scenarios this change should make goma a
good choice for component builds of Chrome, even with full debug
information enabled. To make use of this ability you need to explicitly
specify the switches below - symbol_level will otherwise default to 1
when use_goma == true.

    use_goma = true
    is_win_fastlink = true
    symbol_level = 2

R=scottmg@chromium.org
==========

to

==========
Allow using goma on Windows with symbol_level == 2

Traditionally goma for Chrome has been less useful on Windows than on
other platforms because it was incompatible with full debug information.
Building with goma requires using /Z7 instead of /Zi, and this causes
the linker's memory usage and runtime to blow up as all of the debug
information is merged.

However, /debug:fastlink makes this work. Because it doesn't merge all
of the debug information it makes goma and /Z7 practical. Full release
component builds can be done in less than fifteen minutes, with
incremental builds taking just a few seconds. Without goma a full
release component build of Chrome can easily take 40+ minutes, even on a
Z840.

Goma's speedup comes from massively parallelizing the compile phase,
however even with /debug:fastlink the linking phases are longer with the
/Z7 switch that is required by goma. A /debug:fastlink of chrome.dll in
a component build goes from ~32 seconds to ~110 seconds on a Z620 when
/Z7 (goma) is selected. This penalty will be reduced by VC++ 2017 which
claims 30% speed improvements on /debug:fastlink.

So, if you frequently need to do full relinks then goma may still be a
bad choice. However for most scenarios this change should make goma a
good choice for component builds of Chrome, even with full debug
information enabled. To make use of this ability you need to explicitly
specify the switches below - symbol_level will otherwise default to 1
when use_goma == true.

    use_goma = true
    is_win_fastlink = true
    symbol_level = 2

The fastlink PDBs work well for most scenarios but there are a few
scenarios (rare for most people) where they do not work:
- Copying PDBs to another machine (fails due to references to the .obj
files)
- Reporting on all symbols/globals with SymbolSort or similar fails
- ETW tracing fails
- VS heap profiling fails
Ideally mspdbcmf.exe would let us create "normal" PDBs when needed but
in practice this appears to be unusable with /Z7 created PDBs.

R=scottmg@chromium.org
==========

[brucedawson, 2017-02-02 22:32:14]

The CQ bit was checked by brucedawson@chromium.org

[commit-bot: I haz the power, 2017-02-02 22:33:17]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-03 00:08:23]

CQ is committing da patch.

Bot data: {"patchset_id": 20001, "attempt_start_ts": 1486074734707660,
"parent_rev": "701ccf95a0ec04657b59ad240e28be4085fe033f", "commit_rev":
"4a3cadb2d998fb8f477d0ebc7ff13619b123a805"}

[commit-bot: I haz the power, 2017-02-03 00:08:54]

Description was changed from

==========
Allow using goma on Windows with symbol_level == 2

Traditionally goma for Chrome has been less useful on Windows than on
other platforms because it was incompatible with full debug information.
Building with goma requires using /Z7 instead of /Zi, and this causes
the linker's memory usage and runtime to blow up as all of the debug
information is merged.

However, /debug:fastlink makes this work. Because it doesn't merge all
of the debug information it makes goma and /Z7 practical. Full release
component builds can be done in less than fifteen minutes, with
incremental builds taking just a few seconds. Without goma a full
release component build of Chrome can easily take 40+ minutes, even on a
Z840.

Goma's speedup comes from massively parallelizing the compile phase,
however even with /debug:fastlink the linking phases are longer with the
/Z7 switch that is required by goma. A /debug:fastlink of chrome.dll in
a component build goes from ~32 seconds to ~110 seconds on a Z620 when
/Z7 (goma) is selected. This penalty will be reduced by VC++ 2017 which
claims 30% speed improvements on /debug:fastlink.

So, if you frequently need to do full relinks then goma may still be a
bad choice. However for most scenarios this change should make goma a
good choice for component builds of Chrome, even with full debug
information enabled. To make use of this ability you need to explicitly
specify the switches below - symbol_level will otherwise default to 1
when use_goma == true.

    use_goma = true
    is_win_fastlink = true
    symbol_level = 2

The fastlink PDBs work well for most scenarios but there are a few
scenarios (rare for most people) where they do not work:
- Copying PDBs to another machine (fails due to references to the .obj
files)
- Reporting on all symbols/globals with SymbolSort or similar fails
- ETW tracing fails
- VS heap profiling fails
Ideally mspdbcmf.exe would let us create "normal" PDBs when needed but
in practice this appears to be unusable with /Z7 created PDBs.

R=scottmg@chromium.org
==========

to

==========
Allow using goma on Windows with symbol_level == 2

Traditionally goma for Chrome has been less useful on Windows than on
other platforms because it was incompatible with full debug information.
Building with goma requires using /Z7 instead of /Zi, and this causes
the linker's memory usage and runtime to blow up as all of the debug
information is merged.

However, /debug:fastlink makes this work. Because it doesn't merge all
of the debug information it makes goma and /Z7 practical. Full release
component builds can be done in less than fifteen minutes, with
incremental builds taking just a few seconds. Without goma a full
release component build of Chrome can easily take 40+ minutes, even on a
Z840.

Goma's speedup comes from massively parallelizing the compile phase,
however even with /debug:fastlink the linking phases are longer with the
/Z7 switch that is required by goma. A /debug:fastlink of chrome.dll in
a component build goes from ~32 seconds to ~110 seconds on a Z620 when
/Z7 (goma) is selected. This penalty will be reduced by VC++ 2017 which
claims 30% speed improvements on /debug:fastlink.

So, if you frequently need to do full relinks then goma may still be a
bad choice. However for most scenarios this change should make goma a
good choice for component builds of Chrome, even with full debug
information enabled. To make use of this ability you need to explicitly
specify the switches below - symbol_level will otherwise default to 1
when use_goma == true.

    use_goma = true
    is_win_fastlink = true
    symbol_level = 2

The fastlink PDBs work well for most scenarios but there are a few
scenarios (rare for most people) where they do not work:
- Copying PDBs to another machine (fails due to references to the .obj
files)
- Reporting on all symbols/globals with SymbolSort or similar fails
- ETW tracing fails
- VS heap profiling fails
Ideally mspdbcmf.exe would let us create "normal" PDBs when needed but
in practice this appears to be unusable with /Z7 created PDBs.

R=scottmg@chromium.org

Review-Url: https://codereview.chromium.org/2661023010
Cr-Commit-Position: refs/heads/master@{#447892}
Committed:
https://chromium.googlesource.com/chromium/src/+/4a3cadb2d998fb8f477d0ebc7ff1...
==========

[commit-bot: I haz the power, 2017-02-03 00:09:00]

Committed patchset #2 (id:20001) as
https://chromium.googlesource.com/chromium/src/+/4a3cadb2d998fb8f477d0ebc7ff1...

[xdai1, 2017-02-03 01:09:04]

A revert of this CL (patchset #2 id:20001) has been created in
https://codereview.chromium.org/2669373002/ by xdai@chromium.org.

The reason for reverting is: It broke amd64-generic Trusty informational build
at generate_build_files:
ERROR at //build/config/compiler/compiler.gni:115:3: Assertion failed.
  assert(is_win_fastlink || !use_goma,
  ^-----
Goma builds that use symbol_level 2 must use is_win_fastlink.
See //BUILD.gn:11:1: whence it was imported.
import("//build/config/compiler/compiler.gni")
^--------------------------------------------
GN gen failed: 1
step returned non-zero exit code: 1
@@@STEP_FAILURE@@@

See crbug.com/688203 for more info..