Revert of Verify if GPU message pump is signaled when it hangs in WaitForWork

Revert of Verify if GPU message pump is signaled when it hangs in WaitForWork (patchset #3 id:40001 of https://codereview.chromium.org/2077613002/ )

Reason for revert:
This check isn't needed anymore and in practice it is negative in 100% of crash dumps.
After some additional research I realized that that was
a false negative and that this check doesn't work as
expected with auto-reset events. I confirmed that an
auto-reset event gets promptly reset back to non-signaled 
when it gets signaled as long as there is at least one
thread already waiting on it. That is the case even when
when the target thread is never scheduled to run. The
check would work with a manual-reset event but apparently 
it is useless in the case of an auto-reset event.

Original issue's description:
> Verify if GPU message pump is signaled when it hangs in WaitForWork
> 
> This is a diagnostic change. The code introduced in this
> change runs only when GPU process is about to terminate
> with a deliberate crash.
> 
> We a getting a number of crashes triggered by GPU hang in
> MessagePumpForGpu::WaitForWork. There is already some
> instrumentation that indicates that:
> a) MessagePumpForGpu::WaitForWork is sitting in
> MsgWaitForMultipleObjectsEx for longer than 15 seconds
> b) MessagePumpForGpu::ScheduleWork is called after
> WaitForWork enters the wait, sometimes several seconds
> after, and SetEvent must be called (at least we grab the
> timestamp right before calling SetEvent method)
> c) The event is set but it doesn't wake up the wait
> 
> While it is possible for a thread that is awaken to not
> be immediately scheduled by the OS, it is hard to imagine
> that going up for 15+ seconds. So the theory is that the
> event handle might be recycled and there might be some
> other code that has closed but hasn't nulled out the
> old handle. Since the event is auto-reset, when there are
> multiple waiters only one of them would be awaken and
> reset the event, and the other one would
> just continue waiting. So if the other code is somehow
> still waiting on its closed and now recycled handle, that
> would explain the hang.
> 
> This code would allow GPU watchdog to check whether
> the event was set at the time of the crash. This would
> give us a clue of whether the situation described above
> is actually happening.
> 
> Extra bonus: the investigation would also explain if
> Renderer hangs in MessagePumpDefault::Run are caused by
> the same issue.
> 
> BUG=620904
> 
> Committed: https://crrev.com/702c0f481843035dd46c6a6a256cbe65dda8629c
> Cr-Commit-Position: refs/heads/master@{#400871}

TBR=kbr@chromium.org,dcheng@chromium.org,brucedawson@chromium.org,jbauman@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG=620904

[stanisc, 2016-10-06 20:37:06]

Created Revert of Verify if GPU message pump is signaled when it hangs in
WaitForWork

[stanisc, 2016-10-06 20:41:26]

The CQ bit was checked by stanisc@chromium.org

[commit-bot: I haz the power, 2016-10-06 20:42:14]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-10-06 20:46:35]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-10-06 20:46:36]

Try jobs failed on following builders:
  android_clang_dbg_recipe on master.tryserver.chromium.android (JOB_FAILED,
https://build.chromium.org/p/tryserver.chromium.android/builders/android_clan...)
  android_compile_dbg on master.tryserver.chromium.android (JOB_FAILED,
https://build.chromium.org/p/tryserver.chromium.android/builders/android_comp...)
  android_n5x_swarming_rel on master.tryserver.chromium.android (JOB_FAILED,
https://build.chromium.org/p/tryserver.chromium.android/builders/android_n5x_...)
  linux_android_rel_ng on master.tryserver.chromium.android (JOB_FAILED,
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_androi...)
  cast_shell_linux on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/cast_shell_linu...)
  chromeos_amd64-generic_chromium_compile_only_ng on
master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/chromeos_amd64-...)
  chromeos_daisy_chromium_compile_only_ng on master.tryserver.chromium.linux
(JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/chromeos_daisy_...)
  chromeos_x86-generic_chromium_compile_only_ng on
master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/chromeos_x86-ge...)
  linux_chromium_asan_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
  linux_chromium_chromeos_compile_dbg_ng on master.tryserver.chromium.linux
(JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
  linux_chromium_chromeos_ozone_rel_ng on master.tryserver.chromium.linux
(JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
  linux_chromium_chromeos_rel_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
  linux_chromium_compile_dbg_ng on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
  mac_chromium_compile_dbg_ng on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_comp...)
  mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

[Ken Russell (switch to Gerrit), 2016-10-07 02:58:06]

+vmiura

Are you planning to rebase this and land it, or do this under a different CL?

vmiura's been investigating GPU process hangs under http://crbug.com/609252 ;
the current hypothesis is that a memory stomp is corrupting the observer list,
causing a key notification to the watchdog to be dropped.

[stanisc, 2016-10-07 04:06:20]

On 2016/10/07 02:58:06, Ken Russell wrote:
> +vmiura
> 
> Are you planning to rebase this and land it, or do this under a different CL?
> 
> vmiura's been investigating GPU process hangs under http://crbug.com/609252 ;
> the current hypothesis is that a memory stomp is corrupting the observer list,
> causing a key notification to the watchdog to be dropped.

I created a new CL - https://codereview.chromium.org/2393333002/. I'll submit it
tomorrow.

[Ken Russell (switch to Gerrit), 2016-10-07 04:06:55]

On 2016/10/07 04:06:20, stanisc wrote:
> On 2016/10/07 02:58:06, Ken Russell wrote:
> > +vmiura
> > 
> > Are you planning to rebase this and land it, or do this under a different
CL?
> > 
> > vmiura's been investigating GPU process hangs under http://crbug.com/609252
;
> > the current hypothesis is that a memory stomp is corrupting the observer
list,
> > causing a key notification to the watchdog to be dropped.
> 
> I created a new CL - https://codereview.chromium.org/2393333002/. I'll submit
it
> tomorrow.

OK. I'll close this.

[Ken Russell (switch to Gerrit), 2016-10-07 04:06:59]

Description was changed from

==========
Revert of Verify if GPU message pump is signaled when it hangs in WaitForWork
(patchset #3 id:40001 of https://codereview.chromium.org/2077613002/ )

Reason for revert:
This check isn't needed anymore and in practice it is negative in 100% of crash
dumps.
After some additional research I realized that that was
a false negative and that this check doesn't work as
expected with auto-reset events. I confirmed that an
auto-reset event gets promptly reset back to non-signaled 
when it gets signaled as long as there is at least one
thread already waiting on it. That is the case even when
when the target thread is never scheduled to run. The
check would work with a manual-reset event but apparently 
it is useless in the case of an auto-reset event.

Original issue's description:
> Verify if GPU message pump is signaled when it hangs in WaitForWork
> 
> This is a diagnostic change. The code introduced in this
> change runs only when GPU process is about to terminate
> with a deliberate crash.
> 
> We a getting a number of crashes triggered by GPU hang in
> MessagePumpForGpu::WaitForWork. There is already some
> instrumentation that indicates that:
> a) MessagePumpForGpu::WaitForWork is sitting in
> MsgWaitForMultipleObjectsEx for longer than 15 seconds
> b) MessagePumpForGpu::ScheduleWork is called after
> WaitForWork enters the wait, sometimes several seconds
> after, and SetEvent must be called (at least we grab the
> timestamp right before calling SetEvent method)
> c) The event is set but it doesn't wake up the wait
> 
> While it is possible for a thread that is awaken to not
> be immediately scheduled by the OS, it is hard to imagine
> that going up for 15+ seconds. So the theory is that the
> event handle might be recycled and there might be some
> other code that has closed but hasn't nulled out the
> old handle. Since the event is auto-reset, when there are
> multiple waiters only one of them would be awaken and
> reset the event, and the other one would
> just continue waiting. So if the other code is somehow
> still waiting on its closed and now recycled handle, that
> would explain the hang.
> 
> This code would allow GPU watchdog to check whether
> the event was set at the time of the crash. This would
> give us a clue of whether the situation described above
> is actually happening.
> 
> Extra bonus: the investigation would also explain if
> Renderer hangs in MessagePumpDefault::Run are caused by
> the same issue.
> 
> BUG=620904
> 
> Committed: https://crrev.com/702c0f481843035dd46c6a6a256cbe65dda8629c
> Cr-Commit-Position: refs/heads/master@{#400871}

TBR=kbr@chromium.org,dcheng@chromium.org,brucedawson@chromium.org,jbauman@chr...
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG=620904
==========

to

==========
Revert of Verify if GPU message pump is signaled when it hangs in WaitForWork
(patchset #3 id:40001 of https://codereview.chromium.org/2077613002/ )

Reason for revert:
This check isn't needed anymore and in practice it is negative in 100% of crash
dumps.
After some additional research I realized that that was
a false negative and that this check doesn't work as
expected with auto-reset events. I confirmed that an
auto-reset event gets promptly reset back to non-signaled 
when it gets signaled as long as there is at least one
thread already waiting on it. That is the case even when
when the target thread is never scheduled to run. The
check would work with a manual-reset event but apparently 
it is useless in the case of an auto-reset event.

Original issue's description:
> Verify if GPU message pump is signaled when it hangs in WaitForWork
> 
> This is a diagnostic change. The code introduced in this
> change runs only when GPU process is about to terminate
> with a deliberate crash.
> 
> We a getting a number of crashes triggered by GPU hang in
> MessagePumpForGpu::WaitForWork. There is already some
> instrumentation that indicates that:
> a) MessagePumpForGpu::WaitForWork is sitting in
> MsgWaitForMultipleObjectsEx for longer than 15 seconds
> b) MessagePumpForGpu::ScheduleWork is called after
> WaitForWork enters the wait, sometimes several seconds
> after, and SetEvent must be called (at least we grab the
> timestamp right before calling SetEvent method)
> c) The event is set but it doesn't wake up the wait
> 
> While it is possible for a thread that is awaken to not
> be immediately scheduled by the OS, it is hard to imagine
> that going up for 15+ seconds. So the theory is that the
> event handle might be recycled and there might be some
> other code that has closed but hasn't nulled out the
> old handle. Since the event is auto-reset, when there are
> multiple waiters only one of them would be awaken and
> reset the event, and the other one would
> just continue waiting. So if the other code is somehow
> still waiting on its closed and now recycled handle, that
> would explain the hang.
> 
> This code would allow GPU watchdog to check whether
> the event was set at the time of the crash. This would
> give us a clue of whether the situation described above
> is actually happening.
> 
> Extra bonus: the investigation would also explain if
> Renderer hangs in MessagePumpDefault::Run are caused by
> the same issue.
> 
> BUG=620904
> 
> Committed: https://crrev.com/702c0f481843035dd46c6a6a256cbe65dda8629c
> Cr-Commit-Position: refs/heads/master@{#400871}

TBR=kbr@chromium.org,dcheng@chromium.org,brucedawson@chromium.org,jbauman@chr...
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG=620904
==========