Trial: build trunk with Oilpan everywhere.

Trial: build trunk with Oilpan everywhere.

NOT.FOR.LANDING.

[sof, 2015-11-27 06:28:00]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-11-27 06:28:20]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/1
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/1

[commit-bot: I haz the power, 2015-11-27 07:45:40]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-11-27 07:45:41]

Dry run: Try jobs failed on following builders:
  win_chromium_rel_ng on tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[sof, 2015-11-28 19:56:55]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-11-28 19:57:05]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/20001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/20001

[sof, 2015-11-28 21:01:41]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-11-28 21:02:17]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/40001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/40001

[commit-bot: I haz the power, 2015-11-28 22:26:09]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-11-28 22:26:09]

Dry run: Try jobs failed on following builders:
  linux_chromium_rel_ng on tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[sof, 2015-11-29 08:40:05]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-11-29 08:40:19]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/60001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/60001

[commit-bot: I haz the power, 2015-11-29 09:53:38]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-11-29 09:53:38]

Dry run: Try jobs failed on following builders:
  win_chromium_rel_ng on tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[sof, 2015-11-29 14:34:30]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-11-29 14:34:36]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/80001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/80001

[commit-bot: I haz the power, 2015-11-29 15:57:02]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-11-29 15:57:02]

Dry run: Try jobs failed on following builders:
  linux_chromium_rel_ng on tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[sof, 2015-11-29 19:57:22]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-11-29 19:57:35]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/100001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/100001

[commit-bot: I haz the power, 2015-11-29 21:07:37]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-11-29 21:07:38]

Dry run: Try jobs failed on following builders:
  win_chromium_rel_ng on tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[haraken, 2015-11-30 01:48:46]

haraken@chromium.org changed reviewers:
+ haraken@chromium.org

[haraken, 2015-11-30 01:48:47]

https://codereview.chromium.org/1482683002/diff/100001/third_party/WebKit/Sou...
File third_party/WebKit/Source/core/css/CSSDefaultStyleSheets.cpp (right):

https://codereview.chromium.org/1482683002/diff/100001/third_party/WebKit/Sou...
third_party/WebKit/Source/core/css/CSSDefaultStyleSheets.cpp:50:
DEFINE_STATIC_LOCAL(OwnPtrWillBePersistent<CSSDefaultStyleSheets>,
cssDefaultStyleSheets, (adoptPtrWillBeNoop(new CSSDefaultStyleSheets)));

What happens if we insert WTF_ANNOTATE_SCOPED_MEMORY_LEAK into
DEFINE_STATIC_LOCAL (and its family)? I guess it would be reasonable to mark all
DEFINE_STATIC_LOCALs as leaking.

https://codereview.chromium.org/1482683002/diff/100001/third_party/WebKit/Sou...
File third_party/WebKit/Source/core/frame/RemoteDOMWindow.h (right):

https://codereview.chromium.org/1482683002/diff/100001/third_party/WebKit/Sou...
third_party/WebKit/Source/core/frame/RemoteDOMWindow.h:86:
RawPtrWillBeWeakMember<RemoteFrame> m_frame;

I'm wondering why this needs to be a weak member.

[haraken, 2015-11-30 01:49:30]

haraken@chromium.org changed reviewers:
+ oilpan-reviews@chromium.org

[sof, 2015-11-30 06:25:47]

https://codereview.chromium.org/1482683002/diff/100001/third_party/WebKit/Sou...
File third_party/WebKit/Source/core/frame/RemoteDOMWindow.h (right):

https://codereview.chromium.org/1482683002/diff/100001/third_party/WebKit/Sou...
third_party/WebKit/Source/core/frame/RemoteDOMWindow.h:86:
RawPtrWillBeWeakMember<RemoteFrame> m_frame;
On 2015/11/30 01:48:47, haraken wrote:
> 
> I'm wondering why this needs to be a weak member.

The RemoteDOMWindow will turn into the global object (and have a wrapper holding
it alive) in some unit tests (frame "swap" tests involving remote frames.)

So once the RemoteFrame becomes detached, the WebRemoteFrameImpl object just
drops the reference to it. But it is still kept alive via the wrapper holding
onto the RemoteDOMWindow and consequently not finalized (which will take down
the wrapper & the RemoteDOMWindow.) Break the undesirable dependency by making
this weak.

A one line fix to a leak that it took some time to understand; I hope we don't
have to mirror the explicit protocol between LocalFrame <-> LocalDOMWindow as
far as destruction steps are concerned. But I have my doubts.

[sof, 2015-11-30 21:53:39]

hope that makes some sense..

https://codereview.chromium.org/1482683002/diff/100001/third_party/WebKit/Sou...
File third_party/WebKit/Source/core/css/CSSDefaultStyleSheets.cpp (right):

https://codereview.chromium.org/1482683002/diff/100001/third_party/WebKit/Sou...
third_party/WebKit/Source/core/css/CSSDefaultStyleSheets.cpp:50:
DEFINE_STATIC_LOCAL(OwnPtrWillBePersistent<CSSDefaultStyleSheets>,
cssDefaultStyleSheets, (adoptPtrWillBeNoop(new CSSDefaultStyleSheets)));
On 2015/11/30 01:48:47, haraken wrote:
> 
> What happens if we insert WTF_ANNOTATE_SCOPED_MEMORY_LEAK into
> DEFINE_STATIC_LOCAL (and its family)? I guess it would be reasonable to mark
all
> DEFINE_STATIC_LOCALs as leaking.

I think it will mostly be non-harmful, with or without Oilpan, but it isn't
sufficient to solve the problem for Oilpan.

The reason why is that Oilpan heap pages are registered root regions to LSan,
meaning that the leak detector will scan those regions for pointers into other
heaps (e.g., PartitionAlloc's) and check if those pointed-to objects are
LSan-controlled allocations. If they are, we've got a leak.

Which means it will be able to scan some of the stylesheet heap objects that
instance() retains and refers to, and from those locate their off-heap &
LSan-controlled allocations. So, if we just use WTF_ANNOTATE_SCOPED_MEMORY_LEAK
when instantiating instance(), only the allocations that the constructor makes
will be exempted from LSan's consideration. e.g., what
instance()::m_defaultSheet initially allocates will not be seen as leaking.

However, any allocation made subsequently and then attached to instance() will
not be exempted from LSan leak detection & will be seen as leaking. e.g.,
instance()::m_defaultViewSourceStyle and all it points will leak if it is ever
lazily created.

Which means we have to turn off LSan whenever we create objects and update a
global singleton Persistent<> to refer to those. Hence all the uses of
WTF_ANNOTATE_SCOPED_MEMORY_LEAK here. Which is a brittle arrangement.

We sort of want LSan to flood fill kIgnored tags from the global
cache/singleton, but that is not what LSan (also) does.

iow, I don't have a good solution or approach to suggest atm.

[haraken, 2015-12-01 01:52:45]

haraken@chromium.org changed reviewers:
+ kcc@chromium.org, thakis@chromium.org

[haraken, 2015-12-01 01:52:45]

On 2015/11/30 21:53:39, sof wrote:
> hope that makes some sense..
> 
>
https://codereview.chromium.org/1482683002/diff/100001/third_party/WebKit/Sou...
> File third_party/WebKit/Source/core/css/CSSDefaultStyleSheets.cpp (right):
> 
>
https://codereview.chromium.org/1482683002/diff/100001/third_party/WebKit/Sou...
> third_party/WebKit/Source/core/css/CSSDefaultStyleSheets.cpp:50:
> DEFINE_STATIC_LOCAL(OwnPtrWillBePersistent<CSSDefaultStyleSheets>,
> cssDefaultStyleSheets, (adoptPtrWillBeNoop(new CSSDefaultStyleSheets)));
> On 2015/11/30 01:48:47, haraken wrote:
> > 
> > What happens if we insert WTF_ANNOTATE_SCOPED_MEMORY_LEAK into
> > DEFINE_STATIC_LOCAL (and its family)? I guess it would be reasonable to mark
> all
> > DEFINE_STATIC_LOCALs as leaking.
> 
> I think it will mostly be non-harmful, with or without Oilpan, but it isn't
> sufficient to solve the problem for Oilpan.
> 
> The reason why is that Oilpan heap pages are registered root regions to LSan,
> meaning that the leak detector will scan those regions for pointers into other
> heaps (e.g., PartitionAlloc's) and check if those pointed-to objects are
> LSan-controlled allocations. If they are, we've got a leak.
> 
> Which means it will be able to scan some of the stylesheet heap objects that
> instance() retains and refers to, and from those locate their off-heap &
> LSan-controlled allocations. So, if we just use
WTF_ANNOTATE_SCOPED_MEMORY_LEAK
> when instantiating instance(), only the allocations that the constructor makes
> will be exempted from LSan's consideration. e.g., what
> instance()::m_defaultSheet initially allocates will not be seen as leaking.
> 
> However, any allocation made subsequently and then attached to instance() will
> not be exempted from LSan leak detection & will be seen as leaking. e.g.,
> instance()::m_defaultViewSourceStyle and all it points will leak if it is ever
> lazily created.
> 
> Which means we have to turn off LSan whenever we create objects and update a
> global singleton Persistent<> to refer to those. Hence all the uses of
> WTF_ANNOTATE_SCOPED_MEMORY_LEAK here. Which is a brittle arrangement.
> 
> We sort of want LSan to flood fill kIgnored tags from the global
> cache/singleton, but that is not what LSan (also) does.
> 
> iow, I don't have a good solution or approach to suggest atm.

Thanks for the clarification.

Would there be any way to annotate that the raw pointers of the static
persistent handles are not a root region?

[Nico, 2015-12-01 01:55:18]

fwiw, if you can think of an lsan runtime change that might help, then getting
that deployed everywhere is just a clang roll away after it lands in the
compiler-rt repo (where the lsan runtime lives)

[sof, 2015-12-01 06:33:50]

On 2015/12/01 01:52:45, haraken wrote:
> On 2015/11/30 21:53:39, sof wrote:
> > hope that makes some sense..
> > 
> >
>
https://codereview.chromium.org/1482683002/diff/100001/third_party/WebKit/Sou...
> > File third_party/WebKit/Source/core/css/CSSDefaultStyleSheets.cpp (right):
> > 
> >
>
https://codereview.chromium.org/1482683002/diff/100001/third_party/WebKit/Sou...
> > third_party/WebKit/Source/core/css/CSSDefaultStyleSheets.cpp:50:
> > DEFINE_STATIC_LOCAL(OwnPtrWillBePersistent<CSSDefaultStyleSheets>,
> > cssDefaultStyleSheets, (adoptPtrWillBeNoop(new CSSDefaultStyleSheets)));
> > On 2015/11/30 01:48:47, haraken wrote:
> > > 
> > > What happens if we insert WTF_ANNOTATE_SCOPED_MEMORY_LEAK into
> > > DEFINE_STATIC_LOCAL (and its family)? I guess it would be reasonable to
mark
> > all
> > > DEFINE_STATIC_LOCALs as leaking.
> > 
> > I think it will mostly be non-harmful, with or without Oilpan, but it isn't
> > sufficient to solve the problem for Oilpan.
> > 
> > The reason why is that Oilpan heap pages are registered root regions to
LSan,
> > meaning that the leak detector will scan those regions for pointers into
other
> > heaps (e.g., PartitionAlloc's) and check if those pointed-to objects are
> > LSan-controlled allocations. If they are, we've got a leak.
> > 
> > Which means it will be able to scan some of the stylesheet heap objects that
> > instance() retains and refers to, and from those locate their off-heap &
> > LSan-controlled allocations. So, if we just use
> WTF_ANNOTATE_SCOPED_MEMORY_LEAK
> > when instantiating instance(), only the allocations that the constructor
makes
> > will be exempted from LSan's consideration. e.g., what
> > instance()::m_defaultSheet initially allocates will not be seen as leaking.
> > 
> > However, any allocation made subsequently and then attached to instance()
will
> > not be exempted from LSan leak detection & will be seen as leaking. e.g.,
> > instance()::m_defaultViewSourceStyle and all it points will leak if it is
ever
> > lazily created.
> > 
> > Which means we have to turn off LSan whenever we create objects and update a
> > global singleton Persistent<> to refer to those. Hence all the uses of
> > WTF_ANNOTATE_SCOPED_MEMORY_LEAK here. Which is a brittle arrangement.
> > 
> > We sort of want LSan to flood fill kIgnored tags from the global
> > cache/singleton, but that is not what LSan (also) does.
> > 
> > iow, I don't have a good solution or approach to suggest atm.
> 
> Thanks for the clarification.
> 
> Would there be any way to annotate that the raw pointers of the static
> persistent handles are not a root region?

CSSDefaultStyleSheets.cpp:parseUASheet() is an example of how to do that with
WTF_ANNOTATE_LEAKING_PTR().

The problem is that the ignorance of that object isn't transitive, so the
objects it refers to are still part of a root region and end up being scanned by
the leak detector pass.

And, funnily enough, if you remove the Oilpan PageMemory regions as root regions
altogether, it still reports the leaks. So, reachable somehow still.

[sof, 2015-12-01 06:38:44]

On 2015/12/01 01:55:18, Nico wrote:
> fwiw, if you can think of an lsan runtime change that might help, then getting
> that deployed everywhere is just a clang roll away after it lands in the
> compiler-rt repo (where the lsan runtime lives)

Thanks, appreciate that. But I suspect there's an element of "select() is
broken" [1] here, me not understanding important aspects of LSan. ..a bit of
printf() debugging might just help :)

1:
http://blog.codinghorror.com/the-first-rule-of-programming-its-always-your-fa...

[sof, 2015-12-01 16:52:08]

The latest patchset has the only approach I think of that could work for
combining Oilpan heaps and LSan -- keep track of the Persistent<>s kept in
static references and release those before doing the final round of GCs. 

The patchset isn't complete nor polished, but thought it would be worthwhile to
get some feedback first before getting it all ready (if that's deemed worthy.)

[sof, 2015-12-01 16:52:16]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-12-01 16:52:43]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/120001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/120001

[sof, 2015-12-01 16:54:21]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-12-01 16:54:52]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/140001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/140001

[commit-bot: I haz the power, 2015-12-01 18:11:03]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-12-01 18:11:04]

Dry run: Try jobs failed on following builders:
  win_chromium_x64_rel_ng on tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_...)

[sof, 2015-12-01 22:27:45]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-12-01 22:31:02]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/180001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/180001

[commit-bot: I haz the power, 2015-12-01 23:44:01]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-12-01 23:44:01]

Dry run: Try jobs failed on following builders:
  win8_chromium_ng on tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win8_chromium_ng/...)
  win_chromium_rel_ng on tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[haraken, 2015-12-02 02:17:07]

Thanks for being persistent on this! The approach looks reasonable to me.

Do we still need WTF_ANNOTATE_SCOPED_MEMORY_LEAK? If we release all persistent
handles and then trigger GCs before LSan runs, I guess we no longer need to
worry about objects reachable from the persistent handles...

[sof, 2015-12-02 06:33:40]

On 2015/12/02 02:17:07, haraken wrote:
> Thanks for being persistent on this! The approach looks reasonable to me.
> 
> Do we still need WTF_ANNOTATE_SCOPED_MEMORY_LEAK? If we release all persistent
> handles and then trigger GCs before LSan runs, I guess we no longer need to
> worry about objects reachable from the persistent handles...

Yes, until proven otherwise, none will be needed.

Hiding away the registration details of these static PersistentNodes in
DEFINE_STATIC_LOCAL() et al, is the main part needing work here.

[haraken, 2015-12-02 07:28:58]

On 2015/12/02 06:33:40, sof wrote:
> On 2015/12/02 02:17:07, haraken wrote:
> > Thanks for being persistent on this! The approach looks reasonable to me.
> > 
> > Do we still need WTF_ANNOTATE_SCOPED_MEMORY_LEAK? If we release all
persistent
> > handles and then trigger GCs before LSan runs, I guess we no longer need to
> > worry about objects reachable from the persistent handles...
> 
> Yes, until proven otherwise, none will be needed.
> 
> Hiding away the registration details of these static PersistentNodes in
> DEFINE_STATIC_LOCAL() et al, is the main part needing work here.

I think the approach is reasonable.

[sof, 2015-12-02 14:26:15]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-12-02 14:26:57]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/200001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/200001

[sof, 2015-12-02 14:35:04]

On 2015/12/02 07:28:58, haraken wrote:
> On 2015/12/02 06:33:40, sof wrote:
> > On 2015/12/02 02:17:07, haraken wrote:
> > > Thanks for being persistent on this! The approach looks reasonable to me.
> > > 
> > > Do we still need WTF_ANNOTATE_SCOPED_MEMORY_LEAK? If we release all
> persistent
> > > handles and then trigger GCs before LSan runs, I guess we no longer need
to
> > > worry about objects reachable from the persistent handles...
> > 
> > Yes, until proven otherwise, none will be needed.
> > 
> > Hiding away the registration details of these static PersistentNodes in
> > DEFINE_STATIC_LOCAL() et al, is the main part needing work here.
> 
> I think the approach is reasonable.

It sort of works out, I think, able to retire the suppressions. But I need to
think through the registration parts, not sure the right tradeoffs have been
made.

[commit-bot: I haz the power, 2015-12-02 14:41:00]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-12-02 14:41:01]

Dry run: Try jobs failed on following builders:
  linux_chromium_compile_dbg_32_ng on tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[sof, 2015-12-02 14:51:31]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-12-02 14:52:03]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/220001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/220001

[commit-bot: I haz the power, 2015-12-02 15:53:01]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-12-02 15:53:01]

Dry run: Try jobs failed on following builders:
  mac_chromium_rel_ng on tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

[sof, 2015-12-02 15:54:12]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-12-02 15:54:43]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/240001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/240001

[sof, 2015-12-02 16:44:54]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-12-02 16:45:17]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/260001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/260001

[commit-bot: I haz the power, 2015-12-02 18:13:50]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-12-02 18:13:51]

Dry run: Try jobs failed on following builders:
  linux_android_rel_ng on tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_android_r...)

[sof, 2015-12-02 20:08:44]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-12-02 20:09:44]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/280001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/280001

[commit-bot: I haz the power, 2015-12-02 20:26:01]

Dry run: Try jobs failed on following builders:
  linux_chromium_compile_dbg_32_ng on tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[commit-bot: I haz the power, 2015-12-02 20:27:14]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-12-02 20:27:15]

Dry run: Try jobs failed on following builders:
  linux_chromium_compile_dbg_32_ng on tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[sof, 2015-12-03 08:09:25]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-12-03 08:09:43]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/300001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/300001

[commit-bot: I haz the power, 2015-12-03 09:09:20]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-12-03 09:09:21]

Dry run: Try jobs failed on following builders:
  linux_chromium_asan_rel_ng on tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[sof, 2015-12-03 10:07:36]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-12-03 10:07:49]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/320001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/320001

[sof, 2015-12-03 11:05:32]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-12-03 11:06:15]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/340001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/340001

[commit-bot: I haz the power, 2015-12-03 12:23:00]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-12-03 12:23:01]

Dry run: Try jobs failed on following builders:
  win_chromium_rel_ng on tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_...)

[sof, 2015-12-03 14:28:23]

The CQ bit was checked by sigbjornf@opera.com to run a CQ dry run

[commit-bot: I haz the power, 2015-12-03 14:28:39]

Dry run: CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1482683002/360001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1482683002/360001

[commit-bot: I haz the power, 2015-12-03 15:35:13]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-12-03 15:35:14]

Dry run: Try jobs failed on following builders:
  linux_chromium_asan_rel_ng on tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)

[sof, 2015-12-03 16:24:48]

I think I'm ready here for the most part; split out the main chunk of it as
https://codereview.chromium.org/1491253004/