Use DWARF 3 on android, instead of DWARF 4.

Use DWARF 3 on android, instead of DWARF 4.

Breakpad dump_syms cannot currently parse DWARF 4 binaries correctly,
and omits some information in the generated breakpad symbols (such as
parameter lists of functions). This problem is significantly worse when
using symbol_level=1, as with -g1 in DWARF 4, Breakpad has even more
difficulty.

DWARF 4 has been the default since gcc 4.8. Explicitly switch back to
DWARF 3 for now until Breakpad is able to parse DWARF 4 correctly.

BUG=638485
Committed: https://crrev.com/2e5abf9dbb5947b21921e2b46dba8f44d0db4e7c
Cr-Commit-Position: refs/heads/master@{#416614}

[Torne, 2016-08-26 13:45:26]

torne@chromium.org changed reviewers:
+ mark@chromium.org, primiano@chromium.org, scottmg@chromium.org

[Torne, 2016-08-26 13:45:27]

Not sure what other effects this might have - does this work with debug fission?
Should we restrict this only to Android? (breakpad is affected on linux as well,
but the bug that motivates this change about wanting to use -g1 to avoid
exceeding 4GB binaries is only relevant for android)

The alternative is to just fix breakpad to support DWARF4 - I had a go at doing
this but it's hard to tell whether it's working correctly/sufficient (there are
still weird differences in the breakpad output between a -g1 and -g2 build).

[Torne, 2016-08-26 13:45:32]

The CQ bit was checked by torne@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-08-26 13:45:45]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[Primiano Tucci (use gerrit), 2016-08-26 13:59:25]

I dunno anything about Debug Fission but some of the doc in
https://gcc.gnu.org/wiki/DebugFission seems to refer to dwarf-4.

For the sake of consistency I'd probably try to land this change as-is and send
a PSA to chromium-dev. if anybody finds the debug fission is broken we can fall
back and add the -gdwarf opt only to the else branch of use_debug_fission.

LGTM but I'm not an owner

[Primiano Tucci (use gerrit), 2016-08-26 14:03:15]

CC-ing bunch of folks who might know about dwarf and debug fission

[commit-bot: I haz the power, 2016-08-26 14:22:39]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-08-26 14:22:40]

Dry run: Try jobs failed on following builders:
  win_chromium_compile_dbg_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_comp...)

[Mark Mentovai, 2016-08-26 20:13:31]

I support this, but the trybots don’t in the current state. It seems that at the
very least, we need to separate out -gdwarf-3 so that it only applies to host
and normal-target compilations, but not to NaCl-target compilations.

[Torne, 2016-08-31 12:25:05]

The CQ bit was checked by torne@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-08-31 12:25:16]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-08-31 14:26:00]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-08-31 14:26:01]

Dry run: Try jobs failed on following builders:
  linux_chromium_asan_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
  linux_chromium_chromeos_ozone_rel_ng on master.tryserver.chromium.linux
(JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
  linux_chromium_chromeos_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
  linux_chromium_clobber_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
  linux_chromium_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[Torne, 2016-08-31 14:54:01]

Separated it out to not apply to nacl compilation and most of the bots are now
happy, but a bunch of bots died with exceptions and it doesn't appear to be
anything to do with my change.. any idea?

[Primiano Tucci (use gerrit), 2016-08-31 15:00:27]

go/mabug? Not sure how to get access to goma logs on those bots. I'd ask a
trooper

On Wed, Aug 31, 2016 at 3:54 PM <torne@chromium.org> wrote:

> Separated it out to not apply to nacl compilation and most of the bots are
> now
> happy, but a bunch of bots died with exceptions and it doesn't appear to be
> anything to do with my change.. any idea?
>
> https://codereview.chromium.org/2285723002/
>

-- 
You received this message because you are subscribed to the Google Groups
"Chromium-reviews" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to chromium-reviews+unsubscribe@chromium.org.

[Torne, 2016-08-31 15:03:12]

On 2016/08/31 15:00:27, Primiano Tucci wrote:
> go/mabug? Not sure how to get access to goma logs on those bots. I'd ask a
> trooper

They didn't all fail during compilation, and most of them just cut off logs with
no error; it looks rather like the machines rebooted, but on five at once? :)

[Torne, 2016-08-31 15:03:38]

The CQ bit was checked by torne@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-08-31 15:03:58]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-08-31 17:04:48]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-08-31 17:04:49]

Dry run: Try jobs failed on following builders:
  linux_chromium_chromeos_ozone_rel_ng on master.tryserver.chromium.linux
(JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
  linux_chromium_chromeos_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
  linux_chromium_clobber_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
  linux_chromium_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[Torne, 2016-08-31 17:14:46]

Nope, still dying on retries. This is very weird..

[Primiano Tucci (use gerrit), 2016-08-31 17:41:34]

On 2016/08/31 17:14:46, Torne wrote:
> Nope, still dying on retries. This is very weird..

Maybe that takes extra linker effort (read memory) and ld is OOM-ing?

[Torne, 2016-09-01 10:22:13]

torne@chromium.org changed reviewers:
+ rsesek@chromium.org

[Torne, 2016-09-01 10:22:14]

OK, it's just hitting the two hour tryjob timeout, because changing a global
compiler flag here is invalidating every object file and gets no hits in goma :/

I think skipping the tryjobs is the only easy answer there. This can't really
cause anything to fail to compile?

I've tested debug fission on linux and it doesn't seem to have any issue doing
this with -gdwarf3.

So: the one remaining concern is "will this cause all the signatures in crash to
change". rsesek@ might know but is OOO this week.

[Torne, 2016-09-02 18:18:15]

On 2016/09/01 10:22:14, Torne wrote:
> OK, it's just hitting the two hour tryjob timeout, because changing a global
> compiler flag here is invalidating every object file and gets no hits in goma
:/
> 
> I think skipping the tryjobs is the only easy answer there. This can't really
> cause anything to fail to compile?
> 
> I've tested debug fission on linux and it doesn't seem to have any issue doing
> this with -gdwarf3.
> 
> So: the one remaining concern is "will this cause all the signatures in crash
to
> change". rsesek@ might know but is OOO this week.

update: we checked on the behaviour of crash and it explicitly ignores
parameters when generating signatures. It's possible a small number of
signatures will change for other reasons, but the vast majority should be the
same.

We've also talked to infra about the CQ timeout here; they're considering how to
fix this in future, but the plan is to land this change NOTRY=true over the
weekend and revert right away if anything looks bad.

[Torne, 2016-09-02 18:18:52]

Description was changed from

==========
Use DWARF 3 on linux/android, instead of DWARF 4.

Breakpad dump_syms cannot currently parse DWARF 4 binaries correctly,
and omits some information in the generated breakpad symbols (such as
parameter lists of functions). This problem is significantly worse when
using symbol_level=1, as with -g1 in DWARF 4, Breakpad has even more
difficulty.

DWARF 4 has been the default since gcc 4.8. Explicitly switch back to
DWARF 3 for now until Breakpad is able to parse DWARF 4 correctly.

BUG=638485
==========

to

==========
Use DWARF 3 on linux/android, instead of DWARF 4.

Breakpad dump_syms cannot currently parse DWARF 4 binaries correctly,
and omits some information in the generated breakpad symbols (such as
parameter lists of functions). This problem is significantly worse when
using symbol_level=1, as with -g1 in DWARF 4, Breakpad has even more
difficulty.

DWARF 4 has been the default since gcc 4.8. Explicitly switch back to
DWARF 3 for now until Breakpad is able to parse DWARF 4 correctly.

BUG=638485
NOTRY=true
==========

[scottmg, 2016-09-02 18:20:01]

On 2016/09/01 10:22:14, Torne wrote:
> OK, it's just hitting the two hour tryjob timeout, because changing a global
> compiler flag here is invalidating every object file and gets no hits in goma
:/
> 
> I think skipping the tryjobs is the only easy answer there. This can't really
> cause anything to fail to compile?
> 
> I've tested debug fission on linux and it doesn't seem to have any issue doing
> this with -gdwarf3.
> 
> So: the one remaining concern is "will this cause all the signatures in crash
to
> change". rsesek@ might know but is OOO this week.

I don't know if this will change signatures either. Maybe better to wait until
Tuesday? But if you're not concerned about that, the content of the change lgtm.

[Primiano Tucci (use gerrit), 2016-09-02 18:43:58]

On 2016/09/02 18:20:01, scottmg wrote:
> On 2016/09/01 10:22:14, Torne wrote:
> > OK, it's just hitting the two hour tryjob timeout, because changing a global
> > compiler flag here is invalidating every object file and gets no hits in
goma
> :/
> > 
> > I think skipping the tryjobs is the only easy answer there. This can't
really
> > cause anything to fail to compile?
> > 
> > I've tested debug fission on linux and it doesn't seem to have any issue
doing
> > this with -gdwarf3.
> > 
> > So: the one remaining concern is "will this cause all the signatures in
crash
> to
> > change". rsesek@ might know but is OOO this week.
> 
> I don't know if this will change signatures either. Maybe better to wait until
> Tuesday? But if you're not concerned about that, the content of the change
lgtm.

Nico pointed out that clang rolls are effectively clobbers with no goma cache,
but they always go through trybots.
let's benchmark locally without goma first?

Also, we might want to do this only on Android. he explained me that most clang
debugging stuff in clang is usually tested only against the latest dwarf
versons, so dwarf3 might cause some subtle issue.
Maybe we could make this Android only (where clang is not the default yet)?

[Torne, 2016-09-05 15:19:59]

Local experimentation unfortunately makes it look like:

1) dwarf3 building takes significantly longer
2) it might be that it takes longer because the binaries are bigger

Messing with it a bit more to check for sure, but unfortunately this looks like
we may not be able to use this as a short-term solution, and may need breakpad
updated ASAP.

[Torne, 2016-09-05 17:09:12]

Okay, so some experimentation has resulted in "there's an argument parsing bug
in gcc", which I think I was accidentally avoiding by having the arguments in a
different order when originally testing this locally. Specifically:

-g1 -gdwarf-<anything>   = ignores -g1 and just generates -g2 output
-gdwarf-<anything> -g1   = generates -g1 output as expected

but this *only* applies to -g1. -g2 and -g3 work identically either way around.

So, the build is extremely slow because I'm forcing the trybot to do -g2 level
output rather than the trybot's default -g1, and so it does a huge amount of
extra IO for the much larger binaries and times out. :(

So, the fix is to swap the order, with a BIG COMMENT explaining that the world
is just terrible. I'm also going to restrict this to Android only, to avoid
disrupting linux as previously discussed. New patch shortly, whcih should
hopefully pass tryjobs.

[Torne, 2016-09-05 17:18:42]

Okay, new CL uploaded, folks can take a look while I do another dry run :)

[Torne, 2016-09-05 17:18:52]

The CQ bit was checked by torne@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-09-05 17:18:59]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-09-05 17:21:29]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-09-05 17:21:30]

Dry run: This issue passed the CQ dry run.

[Torne, 2016-09-05 17:23:48]

Description was changed from

==========
Use DWARF 3 on linux/android, instead of DWARF 4.

Breakpad dump_syms cannot currently parse DWARF 4 binaries correctly,
and omits some information in the generated breakpad symbols (such as
parameter lists of functions). This problem is significantly worse when
using symbol_level=1, as with -g1 in DWARF 4, Breakpad has even more
difficulty.

DWARF 4 has been the default since gcc 4.8. Explicitly switch back to
DWARF 3 for now until Breakpad is able to parse DWARF 4 correctly.

BUG=638485
NOTRY=true
==========

to

==========
Use DWARF 3 on linux/android, instead of DWARF 4.

Breakpad dump_syms cannot currently parse DWARF 4 binaries correctly,
and omits some information in the generated breakpad symbols (such as
parameter lists of functions). This problem is significantly worse when
using symbol_level=1, as with -g1 in DWARF 4, Breakpad has even more
difficulty.

DWARF 4 has been the default since gcc 4.8. Explicitly switch back to
DWARF 3 for now until Breakpad is able to parse DWARF 4 correctly.

BUG=638485
==========

[Torne, 2016-09-05 17:24:22]

Whoops, still had a NOTRY=true in there because I was way too optimistic about
my own testing to believe the CL was wrong :p

another dry run..

[Torne, 2016-09-05 17:24:34]

The CQ bit was checked by torne@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-09-05 17:24:39]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[Torne, 2016-09-05 17:24:45]

Description was changed from

==========
Use DWARF 3 on linux/android, instead of DWARF 4.

Breakpad dump_syms cannot currently parse DWARF 4 binaries correctly,
and omits some information in the generated breakpad symbols (such as
parameter lists of functions). This problem is significantly worse when
using symbol_level=1, as with -g1 in DWARF 4, Breakpad has even more
difficulty.

DWARF 4 has been the default since gcc 4.8. Explicitly switch back to
DWARF 3 for now until Breakpad is able to parse DWARF 4 correctly.

BUG=638485
==========

to

==========
Use DWARF 3 on android, instead of DWARF 4.

Breakpad dump_syms cannot currently parse DWARF 4 binaries correctly,
and omits some information in the generated breakpad symbols (such as
parameter lists of functions). This problem is significantly worse when
using symbol_level=1, as with -g1 in DWARF 4, Breakpad has even more
difficulty.

DWARF 4 has been the default since gcc 4.8. Explicitly switch back to
DWARF 3 for now until Breakpad is able to parse DWARF 4 correctly.

BUG=638485
==========

[commit-bot: I haz the power, 2016-09-05 18:45:17]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-09-05 18:45:18]

Dry run: This issue passed the CQ dry run.

[Torne, 2016-09-06 10:29:52]

OK, this now doesn't murder the cq (yay!) and only affects Android. Thanks to
the various people who chimed in to point out I was going to be taking way too
big a risk ;)

[Primiano Tucci (use gerrit), 2016-09-06 10:35:48]

ok this looks better to me. re-LGTM.
So far we have one gold-linker bug, one GCC frontend bug and still counting. I'm
thrilled for what is coming next :/

[Torne, 2016-09-06 10:59:03]

The CQ bit was checked by torne@chromium.org

[Torne, 2016-09-06 10:59:03]

The patchset sent to the CQ was uploaded after l-g-t-m from scottmg@chromium.org
Link to the patchset: https://codereview.chromium.org/2285723002/#ps40001
(title: "Restrict to just android, and fix arg order")

[commit-bot: I haz the power, 2016-09-06 10:59:15]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-09-06 11:02:00]

Description was changed from

==========
Use DWARF 3 on android, instead of DWARF 4.

Breakpad dump_syms cannot currently parse DWARF 4 binaries correctly,
and omits some information in the generated breakpad symbols (such as
parameter lists of functions). This problem is significantly worse when
using symbol_level=1, as with -g1 in DWARF 4, Breakpad has even more
difficulty.

DWARF 4 has been the default since gcc 4.8. Explicitly switch back to
DWARF 3 for now until Breakpad is able to parse DWARF 4 correctly.

BUG=638485
==========

to

==========
Use DWARF 3 on android, instead of DWARF 4.

Breakpad dump_syms cannot currently parse DWARF 4 binaries correctly,
and omits some information in the generated breakpad symbols (such as
parameter lists of functions). This problem is significantly worse when
using symbol_level=1, as with -g1 in DWARF 4, Breakpad has even more
difficulty.

DWARF 4 has been the default since gcc 4.8. Explicitly switch back to
DWARF 3 for now until Breakpad is able to parse DWARF 4 correctly.

BUG=638485
==========

[commit-bot: I haz the power, 2016-09-06 11:02:02]

Committed patchset #3 (id:40001)

[commit-bot: I haz the power, 2016-09-06 11:04:05]

Description was changed from

==========
Use DWARF 3 on android, instead of DWARF 4.

Breakpad dump_syms cannot currently parse DWARF 4 binaries correctly,
and omits some information in the generated breakpad symbols (such as
parameter lists of functions). This problem is significantly worse when
using symbol_level=1, as with -g1 in DWARF 4, Breakpad has even more
difficulty.

DWARF 4 has been the default since gcc 4.8. Explicitly switch back to
DWARF 3 for now until Breakpad is able to parse DWARF 4 correctly.

BUG=638485
==========

to

==========
Use DWARF 3 on android, instead of DWARF 4.

Breakpad dump_syms cannot currently parse DWARF 4 binaries correctly,
and omits some information in the generated breakpad symbols (such as
parameter lists of functions). This problem is significantly worse when
using symbol_level=1, as with -g1 in DWARF 4, Breakpad has even more
difficulty.

DWARF 4 has been the default since gcc 4.8. Explicitly switch back to
DWARF 3 for now until Breakpad is able to parse DWARF 4 correctly.

BUG=638485

Committed: https://crrev.com/2e5abf9dbb5947b21921e2b46dba8f44d0db4e7c
Cr-Commit-Position: refs/heads/master@{#416614}
==========

[commit-bot: I haz the power, 2016-09-06 11:04:06]

Patchset 3 (id:??) landed as
https://crrev.com/2e5abf9dbb5947b21921e2b46dba8f44d0db4e7c
Cr-Commit-Position: refs/heads/master@{#416614}

[Torne, 2016-09-06 14:43:44]

Filed https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77497 about the unexpected
argument-order dependency here :)

[Lei Zhang, 2016-09-06 15:37:17]

On 2016/09/06 14:43:44, Torne wrote:
> Filed https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77497 about the unexpected
> argument-order dependency here :)

Did you file a Breakpad bug to support DWARF 4? e.g. we also need it to dump
symbols for system libraries on modern Linux distros.
https://bugs.chromium.org/p/google-breakpad/issues/list

[Torne, 2016-09-08 15:04:12]

On 2016/09/06 15:37:17, Lei Zhang wrote:
> On 2016/09/06 14:43:44, Torne wrote:
> > Filed https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77497 about the
unexpected
> > argument-order dependency here :)
> 
> Did you file a Breakpad bug to support DWARF 4? e.g. we also need it to dump
> symbols for system libraries on modern Linux distros.
> https://bugs.chromium.org/p/google-breakpad/issues/list

I've just filed https://bugs.chromium.org/p/google-breakpad/issues/detail?id=715
for this.