Allow using goma on Windows with symbol_level == 2

Allow using goma on Windows with symbol_level == 2

Relanding with fix for amd64-generic Trusty build failure. Previous
version was crrev.com/2661023010.

Traditionally goma for Chrome has been less useful on Windows than on
other platforms because it was incompatible with full debug information.
Building with goma requires using /Z7 instead of /Zi, and this causes
the linker's memory usage and runtime to blow up as all of the debug
information is merged.

However, /debug:fastlink makes this work. Because it doesn't merge all
of the debug information it makes goma and /Z7 practical. Full release
component builds can be done in less than fifteen minutes, with
incremental builds taking just a few seconds. Without goma a full
release component build of Chrome can easily take 40+ minutes, even on a
Z840.

Goma's speedup comes from massively parallelizing the compile phase,
however even with /debug:fastlink the linking phases are longer with the
/Z7 switch that is required by goma. A /debug:fastlink of chrome.dll in
a component build goes from ~32 seconds to ~110 seconds on a Z620 when
/Z7 (goma) is selected. This penalty will be reduced by VC++ 2017 which
claims 30% speed improvements on /debug:fastlink.

So, if you frequently need to do full relinks then goma may still be a
bad choice. However for most scenarios this change should make goma a
good choice for component builds of Chrome, even with full debug
information enabled. To make use of this ability you need to explicitly
specify the switches below - symbol_level will otherwise default to 1
when use_goma == true.

    use_goma = true
    is_win_fastlink = true
    symbol_level = 2

In addition, these two settings are strongly recommended if you want the
fastest possible builds:

    is_component_build = true
    target_cpu="x86" - to ensure incremental linking always works

The fastlink PDBs work well for most scenarios but there are a few
scenarios where they do not work:
- Copying PDBs to another machine (fails due to references to the .obj
files)
- Reporting on all symbols/globals with SymbolSort or windbg's "x"
command or similar will report nothing
- ETW tracing fails
- VS heap profiling fails

Ideally mspdbcmf.exe would let us create "normal" PDBs when needed but
in practice this appears to be unusable with /Z7 created PDBs.

BUG=630074, 688203
Review-Url: https://codereview.chromium.org/2702203004
Cr-Commit-Position: refs/heads/master@{#452171}
Committed: https://chromium.googlesource.com/chromium/src/+/77bdd81b8517ef0166ba14549545fa1f27d9bfec

[brucedawson, 2017-02-21 22:31:14]

The CQ bit was checked by brucedawson@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-02-21 22:33:31]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-22 00:37:29]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-22 00:37:30]

Dry run: Try jobs failed on following builders:
  chromeos_amd64-generic_chromium_compile_only_ng on
master.tryserver.chromium.linux (JOB_TIMED_OUT, no build URL)
  chromium_presubmit on master.tryserver.chromium.linux (JOB_TIMED_OUT, no build
URL)
  linux_chromium_chromeos_ozone_rel_ng on master.tryserver.chromium.linux
(JOB_TIMED_OUT, no build URL)
  linux_chromium_chromeos_rel_ng on master.tryserver.chromium.linux
(JOB_TIMED_OUT, no build URL)
  linux_chromium_compile_dbg_ng on master.tryserver.chromium.linux
(JOB_TIMED_OUT, no build URL)
  linux_chromium_rel_ng on master.tryserver.chromium.linux (JOB_TIMED_OUT, no
build URL)
  linux_chromium_tsan_rel_ng on master.tryserver.chromium.linux (JOB_TIMED_OUT,
no build URL)

[brucedawson, 2017-02-22 01:17:03]

The CQ bit was checked by brucedawson@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-02-22 01:18:58]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[brucedawson, 2017-02-22 01:20:57]

Description was changed from

==========
Allow using goma on Windows with symbol_level == 2

Relanding with fix for amd64-generic Trusty build failure

Traditionally goma for Chrome has been less useful on Windows than on
other platforms because it was incompatible with full debug information.
Building with goma requires using /Z7 instead of /Zi, and this causes
the linker's memory usage and runtime to blow up as all of the debug
information is merged.

However, /debug:fastlink makes this work. Because it doesn't merge all
of the debug information it makes goma and /Z7 practical. Full release
component builds can be done in less than fifteen minutes, with
incremental builds taking just a few seconds. Without goma a full
release component build of Chrome can easily take 40+ minutes, even on a
Z840.

Goma's speedup comes from massively parallelizing the compile phase,
however even with /debug:fastlink the linking phases are longer with the
/Z7 switch that is required by goma. A /debug:fastlink of chrome.dll in
a component build goes from ~32 seconds to ~110 seconds on a Z620 when
/Z7 (goma) is selected. This penalty will be reduced by VC++ 2017 which
claims 30% speed improvements on /debug:fastlink.

So, if you frequently need to do full relinks then goma may still be a
bad choice. However for most scenarios this change should make goma a
good choice for component builds of Chrome, even with full debug
information enabled. To make use of this ability you need to explicitly
specify the switches below - symbol_level will otherwise default to 1
when use_goma == true.

    use_goma = true
    is_win_fastlink = true
    symbol_level = 2

In addition, these two settings are strongly recommended if you want the
fastest possible builds:

    is_component_build = true
    target_cpu="x86" - to ensure incremental linking always works

The fastlink PDBs work well for most scenarios but there are a few
scenarios where they do not work:
- Copying PDBs to another machine (fails due to references to the .obj
files)
- Reporting on all symbols/globals with SymbolSort or windbg's "x"
command or similar will report nothing
- ETW tracing fails
- VS heap profiling fails
Ideally mspdbcmf.exe would let us create "normal" PDBs when needed but
in practice this appears to be unusable with /Z7 created PDBs.

R=scottmg@chromium.org
BUG=688203
==========

to

==========
Allow using goma on Windows with symbol_level == 2

Relanding with fix for amd64-generic Trusty build failure

Traditionally goma for Chrome has been less useful on Windows than on
other platforms because it was incompatible with full debug information.
Building with goma requires using /Z7 instead of /Zi, and this causes
the linker's memory usage and runtime to blow up as all of the debug
information is merged.

However, /debug:fastlink makes this work. Because it doesn't merge all
of the debug information it makes goma and /Z7 practical. Full release
component builds can be done in less than fifteen minutes, with
incremental builds taking just a few seconds. Without goma a full
release component build of Chrome can easily take 40+ minutes, even on a
Z840.

Goma's speedup comes from massively parallelizing the compile phase,
however even with /debug:fastlink the linking phases are longer with the
/Z7 switch that is required by goma. A /debug:fastlink of chrome.dll in
a component build goes from ~32 seconds to ~110 seconds on a Z620 when
/Z7 (goma) is selected. This penalty will be reduced by VC++ 2017 which
claims 30% speed improvements on /debug:fastlink.

So, if you frequently need to do full relinks then goma may still be a
bad choice. However for most scenarios this change should make goma a
good choice for component builds of Chrome, even with full debug
information enabled. To make use of this ability you need to explicitly
specify the switches below - symbol_level will otherwise default to 1
when use_goma == true.

    use_goma = true
    is_win_fastlink = true
    symbol_level = 2

In addition, these two settings are strongly recommended if you want the
fastest possible builds:

    is_component_build = true
    target_cpu="x86" - to ensure incremental linking always works

The fastlink PDBs work well for most scenarios but there are a few
scenarios where they do not work:
- Copying PDBs to another machine (fails due to references to the .obj
files)
- Reporting on all symbols/globals with SymbolSort or windbg's "x"
command or similar will report nothing
- ETW tracing fails
- VS heap profiling fails

Ideally mspdbcmf.exe would let us create "normal" PDBs when needed but
in practice this appears to be unusable with /Z7 created PDBs.

R=scottmg@chromium.org
BUG=630074,688203
==========

[brucedawson, 2017-02-22 01:39:46]

Description was changed from

==========
Allow using goma on Windows with symbol_level == 2

Relanding with fix for amd64-generic Trusty build failure

Traditionally goma for Chrome has been less useful on Windows than on
other platforms because it was incompatible with full debug information.
Building with goma requires using /Z7 instead of /Zi, and this causes
the linker's memory usage and runtime to blow up as all of the debug
information is merged.

However, /debug:fastlink makes this work. Because it doesn't merge all
of the debug information it makes goma and /Z7 practical. Full release
component builds can be done in less than fifteen minutes, with
incremental builds taking just a few seconds. Without goma a full
release component build of Chrome can easily take 40+ minutes, even on a
Z840.

Goma's speedup comes from massively parallelizing the compile phase,
however even with /debug:fastlink the linking phases are longer with the
/Z7 switch that is required by goma. A /debug:fastlink of chrome.dll in
a component build goes from ~32 seconds to ~110 seconds on a Z620 when
/Z7 (goma) is selected. This penalty will be reduced by VC++ 2017 which
claims 30% speed improvements on /debug:fastlink.

So, if you frequently need to do full relinks then goma may still be a
bad choice. However for most scenarios this change should make goma a
good choice for component builds of Chrome, even with full debug
information enabled. To make use of this ability you need to explicitly
specify the switches below - symbol_level will otherwise default to 1
when use_goma == true.

    use_goma = true
    is_win_fastlink = true
    symbol_level = 2

In addition, these two settings are strongly recommended if you want the
fastest possible builds:

    is_component_build = true
    target_cpu="x86" - to ensure incremental linking always works

The fastlink PDBs work well for most scenarios but there are a few
scenarios where they do not work:
- Copying PDBs to another machine (fails due to references to the .obj
files)
- Reporting on all symbols/globals with SymbolSort or windbg's "x"
command or similar will report nothing
- ETW tracing fails
- VS heap profiling fails

Ideally mspdbcmf.exe would let us create "normal" PDBs when needed but
in practice this appears to be unusable with /Z7 created PDBs.

R=scottmg@chromium.org
BUG=630074,688203
==========

to

==========
Allow using goma on Windows with symbol_level == 2

Relanding with fix for amd64-generic Trusty build failure

Traditionally goma for Chrome has been less useful on Windows than on
other platforms because it was incompatible with full debug information.
Building with goma requires using /Z7 instead of /Zi, and this causes
the linker's memory usage and runtime to blow up as all of the debug
information is merged.

However, /debug:fastlink makes this work. Because it doesn't merge all
of the debug information it makes goma and /Z7 practical. Full release
component builds can be done in less than fifteen minutes, with
incremental builds taking just a few seconds. Without goma a full
release component build of Chrome can easily take 40+ minutes, even on a
Z840.

Goma's speedup comes from massively parallelizing the compile phase,
however even with /debug:fastlink the linking phases are longer with the
/Z7 switch that is required by goma. A /debug:fastlink of chrome.dll in
a component build goes from ~32 seconds to ~110 seconds on a Z620 when
/Z7 (goma) is selected. This penalty will be reduced by VC++ 2017 which
claims 30% speed improvements on /debug:fastlink.

So, if you frequently need to do full relinks then goma may still be a
bad choice. However for most scenarios this change should make goma a
good choice for component builds of Chrome, even with full debug
information enabled. To make use of this ability you need to explicitly
specify the switches below - symbol_level will otherwise default to 1
when use_goma == true.

    use_goma = true
    is_win_fastlink = true
    symbol_level = 2

In addition, these two settings are strongly recommended if you want the
fastest possible builds:

    is_component_build = true
    target_cpu="x86" - to ensure incremental linking always works

The fastlink PDBs work well for most scenarios but there are a few
scenarios where they do not work:
- Copying PDBs to another machine (fails due to references to the .obj
files)
- Reporting on all symbols/globals with SymbolSort or windbg's "x"
command or similar will report nothing
- ETW tracing fails
- VS heap profiling fails

Ideally mspdbcmf.exe would let us create "normal" PDBs when needed but
in practice this appears to be unusable with /Z7 created PDBs.

BUG=630074,688203
==========

[brucedawson, 2017-02-22 01:39:46]

brucedawson@chromium.org changed reviewers:
+ dpranke@chromium.org
- scottmg@chromium.org

[brucedawson, 2017-02-22 01:40:02]

The CQ bit was checked by brucedawson@chromium.org to run a CQ dry run

[brucedawson, 2017-02-22 01:41:22]

Hey, I'm back and I thought I'd try this change again.

The previous version (crrev.com/2661023010) failed because the assert was not
scoped to is_win only. Oops. It passed the try bots because none of them set
symbol_level = 1.

I also tweaked a comment and the CL description.

[commit-bot: I haz the power, 2017-02-22 01:41:27]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[brucedawson, 2017-02-22 01:41:42]

Description was changed from

==========
Allow using goma on Windows with symbol_level == 2

Relanding with fix for amd64-generic Trusty build failure

Traditionally goma for Chrome has been less useful on Windows than on
other platforms because it was incompatible with full debug information.
Building with goma requires using /Z7 instead of /Zi, and this causes
the linker's memory usage and runtime to blow up as all of the debug
information is merged.

However, /debug:fastlink makes this work. Because it doesn't merge all
of the debug information it makes goma and /Z7 practical. Full release
component builds can be done in less than fifteen minutes, with
incremental builds taking just a few seconds. Without goma a full
release component build of Chrome can easily take 40+ minutes, even on a
Z840.

Goma's speedup comes from massively parallelizing the compile phase,
however even with /debug:fastlink the linking phases are longer with the
/Z7 switch that is required by goma. A /debug:fastlink of chrome.dll in
a component build goes from ~32 seconds to ~110 seconds on a Z620 when
/Z7 (goma) is selected. This penalty will be reduced by VC++ 2017 which
claims 30% speed improvements on /debug:fastlink.

So, if you frequently need to do full relinks then goma may still be a
bad choice. However for most scenarios this change should make goma a
good choice for component builds of Chrome, even with full debug
information enabled. To make use of this ability you need to explicitly
specify the switches below - symbol_level will otherwise default to 1
when use_goma == true.

    use_goma = true
    is_win_fastlink = true
    symbol_level = 2

In addition, these two settings are strongly recommended if you want the
fastest possible builds:

    is_component_build = true
    target_cpu="x86" - to ensure incremental linking always works

The fastlink PDBs work well for most scenarios but there are a few
scenarios where they do not work:
- Copying PDBs to another machine (fails due to references to the .obj
files)
- Reporting on all symbols/globals with SymbolSort or windbg's "x"
command or similar will report nothing
- ETW tracing fails
- VS heap profiling fails

Ideally mspdbcmf.exe would let us create "normal" PDBs when needed but
in practice this appears to be unusable with /Z7 created PDBs.

BUG=630074,688203
==========

to

==========
Allow using goma on Windows with symbol_level == 2

Relanding with fix for amd64-generic Trusty build failure. Previous
version was crrev.com/2661023010.

Traditionally goma for Chrome has been less useful on Windows than on
other platforms because it was incompatible with full debug information.
Building with goma requires using /Z7 instead of /Zi, and this causes
the linker's memory usage and runtime to blow up as all of the debug
information is merged.

However, /debug:fastlink makes this work. Because it doesn't merge all
of the debug information it makes goma and /Z7 practical. Full release
component builds can be done in less than fifteen minutes, with
incremental builds taking just a few seconds. Without goma a full
release component build of Chrome can easily take 40+ minutes, even on a
Z840.

Goma's speedup comes from massively parallelizing the compile phase,
however even with /debug:fastlink the linking phases are longer with the
/Z7 switch that is required by goma. A /debug:fastlink of chrome.dll in
a component build goes from ~32 seconds to ~110 seconds on a Z620 when
/Z7 (goma) is selected. This penalty will be reduced by VC++ 2017 which
claims 30% speed improvements on /debug:fastlink.

So, if you frequently need to do full relinks then goma may still be a
bad choice. However for most scenarios this change should make goma a
good choice for component builds of Chrome, even with full debug
information enabled. To make use of this ability you need to explicitly
specify the switches below - symbol_level will otherwise default to 1
when use_goma == true.

    use_goma = true
    is_win_fastlink = true
    symbol_level = 2

In addition, these two settings are strongly recommended if you want the
fastest possible builds:

    is_component_build = true
    target_cpu="x86" - to ensure incremental linking always works

The fastlink PDBs work well for most scenarios but there are a few
scenarios where they do not work:
- Copying PDBs to another machine (fails due to references to the .obj
files)
- Reporting on all symbols/globals with SymbolSort or windbg's "x"
command or similar will report nothing
- ETW tracing fails
- VS heap profiling fails

Ideally mspdbcmf.exe would let us create "normal" PDBs when needed but
in practice this appears to be unusable with /Z7 created PDBs.

BUG=630074,688203
==========

[Dirk Pranke, 2017-02-22 01:47:57]

lgtm

[brucedawson, 2017-02-22 02:02:03]

The CQ bit was unchecked by brucedawson@chromium.org

[brucedawson, 2017-02-22 02:02:07]

The CQ bit was checked by brucedawson@chromium.org

[commit-bot: I haz the power, 2017-02-22 02:03:25]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-22 03:43:28]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-02-22 03:43:29]

Try jobs failed on following builders:
  cast_shell_linux on master.tryserver.chromium.linux (JOB_TIMED_OUT, no build
URL)
  chromeos_amd64-generic_chromium_compile_only_ng on
master.tryserver.chromium.linux (JOB_TIMED_OUT, no build URL)
  chromeos_daisy_chromium_compile_only_ng on master.tryserver.chromium.linux
(JOB_TIMED_OUT, no build URL)
  linux_chromium_asan_rel_ng on master.tryserver.chromium.linux (JOB_TIMED_OUT,
no build URL)
  linux_chromium_chromeos_ozone_rel_ng on master.tryserver.chromium.linux
(JOB_TIMED_OUT, no build URL)
  linux_chromium_chromeos_rel_ng on master.tryserver.chromium.linux
(JOB_TIMED_OUT, no build URL)
  linux_chromium_compile_dbg_ng on master.tryserver.chromium.linux
(JOB_TIMED_OUT, no build URL)
  linux_chromium_rel_ng on master.tryserver.chromium.linux (JOB_TIMED_OUT, no
build URL)
  linux_chromium_tsan_rel_ng on master.tryserver.chromium.linux (JOB_TIMED_OUT,
no build URL)

[brucedawson, 2017-02-22 17:42:51]

The CQ bit was checked by brucedawson@chromium.org

[commit-bot: I haz the power, 2017-02-22 17:43:30]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-02-22 19:57:33]

CQ is committing da patch.

Bot data: {"patchset_id": 40001, "attempt_start_ts": 1487785371837330,
"parent_rev": "717789e00f9ff34efe3a2ca23ebd69e802d8b6ec", "commit_rev":
"77bdd81b8517ef0166ba14549545fa1f27d9bfec"}

[commit-bot: I haz the power, 2017-02-22 19:58:10]

Description was changed from

==========
Allow using goma on Windows with symbol_level == 2

Relanding with fix for amd64-generic Trusty build failure. Previous
version was crrev.com/2661023010.

Traditionally goma for Chrome has been less useful on Windows than on
other platforms because it was incompatible with full debug information.
Building with goma requires using /Z7 instead of /Zi, and this causes
the linker's memory usage and runtime to blow up as all of the debug
information is merged.

However, /debug:fastlink makes this work. Because it doesn't merge all
of the debug information it makes goma and /Z7 practical. Full release
component builds can be done in less than fifteen minutes, with
incremental builds taking just a few seconds. Without goma a full
release component build of Chrome can easily take 40+ minutes, even on a
Z840.

Goma's speedup comes from massively parallelizing the compile phase,
however even with /debug:fastlink the linking phases are longer with the
/Z7 switch that is required by goma. A /debug:fastlink of chrome.dll in
a component build goes from ~32 seconds to ~110 seconds on a Z620 when
/Z7 (goma) is selected. This penalty will be reduced by VC++ 2017 which
claims 30% speed improvements on /debug:fastlink.

So, if you frequently need to do full relinks then goma may still be a
bad choice. However for most scenarios this change should make goma a
good choice for component builds of Chrome, even with full debug
information enabled. To make use of this ability you need to explicitly
specify the switches below - symbol_level will otherwise default to 1
when use_goma == true.

    use_goma = true
    is_win_fastlink = true
    symbol_level = 2

In addition, these two settings are strongly recommended if you want the
fastest possible builds:

    is_component_build = true
    target_cpu="x86" - to ensure incremental linking always works

The fastlink PDBs work well for most scenarios but there are a few
scenarios where they do not work:
- Copying PDBs to another machine (fails due to references to the .obj
files)
- Reporting on all symbols/globals with SymbolSort or windbg's "x"
command or similar will report nothing
- ETW tracing fails
- VS heap profiling fails

Ideally mspdbcmf.exe would let us create "normal" PDBs when needed but
in practice this appears to be unusable with /Z7 created PDBs.

BUG=630074,688203
==========

to

==========
Allow using goma on Windows with symbol_level == 2

Relanding with fix for amd64-generic Trusty build failure. Previous
version was crrev.com/2661023010.

Traditionally goma for Chrome has been less useful on Windows than on
other platforms because it was incompatible with full debug information.
Building with goma requires using /Z7 instead of /Zi, and this causes
the linker's memory usage and runtime to blow up as all of the debug
information is merged.

However, /debug:fastlink makes this work. Because it doesn't merge all
of the debug information it makes goma and /Z7 practical. Full release
component builds can be done in less than fifteen minutes, with
incremental builds taking just a few seconds. Without goma a full
release component build of Chrome can easily take 40+ minutes, even on a
Z840.

Goma's speedup comes from massively parallelizing the compile phase,
however even with /debug:fastlink the linking phases are longer with the
/Z7 switch that is required by goma. A /debug:fastlink of chrome.dll in
a component build goes from ~32 seconds to ~110 seconds on a Z620 when
/Z7 (goma) is selected. This penalty will be reduced by VC++ 2017 which
claims 30% speed improvements on /debug:fastlink.

So, if you frequently need to do full relinks then goma may still be a
bad choice. However for most scenarios this change should make goma a
good choice for component builds of Chrome, even with full debug
information enabled. To make use of this ability you need to explicitly
specify the switches below - symbol_level will otherwise default to 1
when use_goma == true.

    use_goma = true
    is_win_fastlink = true
    symbol_level = 2

In addition, these two settings are strongly recommended if you want the
fastest possible builds:

    is_component_build = true
    target_cpu="x86" - to ensure incremental linking always works

The fastlink PDBs work well for most scenarios but there are a few
scenarios where they do not work:
- Copying PDBs to another machine (fails due to references to the .obj
files)
- Reporting on all symbols/globals with SymbolSort or windbg's "x"
command or similar will report nothing
- ETW tracing fails
- VS heap profiling fails

Ideally mspdbcmf.exe would let us create "normal" PDBs when needed but
in practice this appears to be unusable with /Z7 created PDBs.

BUG=630074,688203

Review-Url: https://codereview.chromium.org/2702203004
Cr-Commit-Position: refs/heads/master@{#452171}
Committed:
https://chromium.googlesource.com/chromium/src/+/77bdd81b8517ef0166ba14549545...
==========

[commit-bot: I haz the power, 2017-02-22 19:58:11]

Committed patchset #3 (id:40001) as
https://chromium.googlesource.com/chromium/src/+/77bdd81b8517ef0166ba14549545...