[Findit] Make Findit more robust.

tools/findit/crash_utils.py

Index: tools/findit/crash_utils.py
diff --git a/tools/findit/crash_utils.py b/tools/findit/crash_utils.py
index 6be058b0823fa8f635b13090e169362a875d7b79..a3af854106a5d56aac52991fc0c2a959e8fb1fba 100644
--- a/tools/findit/crash_utils.py
+++ b/tools/findit/crash_utils.py
@@ -2,6 +2,7 @@
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
+import atexit
 import cgi
 import ConfigParser
 import json
@@ -20,10 +21,39 @@ MAX_THREAD_NUMBER = 10
 TASK_QUEUE = None
 
 
+def SignalWorkerThreads():
+  global TASK_QUEUE
+  if not TASK_QUEUE:
+    return
+
+  for i in range(MAX_THREAD_NUMBER):
+    TASK_QUEUE.put(None)
+
+  # Give worker threads a chance to exit.
+  # Workaround the harmless bug in python 2.7 below.
+  time.sleep(1)
+  print '%s is quitting' % threading.currentThread().name
+
+
+atexit.register(SignalWorkerThreads)
+
+
 def Worker():
   global TASK_QUEUE
   while True:
-    function, args, kwargs, result_semaphore = TASK_QUEUE.get()

[stgao, 2014/09/05 08:33:18]

The exception is as below:

Exception in thread worker_9 (most likely raised during interpreter shutdown):  

Traceback (most recent call last):                                              

  File "/usr/lib/python2.7/threading.py", line 808, in __bootstrap_inner        

  File "/usr/lib/python2.7/threading.py", line 761, in run                      

  File "/mnt/scratch0/clusterfuzz/src/tools/findit/crash_utils.py", line 26, in
Worker
  File "/usr/lib/python2.7/Queue.py", line 168, in get                          

  File "/usr/lib/python2.7/threading.py", line 332, in wait                     

<type 'exceptions.TypeError'>: 'NoneType' object is not callable  

+    try:
+      task = TASK_QUEUE.get()
+      if not task:
+        print '%s is quitting' % threading.currentThread().name

[stgao, 2014/09/05 08:33:18]

I will revert the print before committing.
But I'd like to show this fix will work, because the order of quitting is as
below:

worker_3 is quitting
 worker_2 is quitting
 worker_7 is quitting
worker_1 is quitting
worker_6 is quitting
 worker_8 is quitting
worker_5 is quitting
worker_9 is quitting
 worker_0 is quitting
worker_4 is quitting
Dummy-1 is quitting

+        return
+    except TypeError:
+      # According to http://bugs.python.org/issue14623, this is a harmless bug
+      # in python 2.7 which won't be fixed.
+      # The exception is raised on daemon threads when python interpreter is
+      # shutting down.
+      return
+
+    function, args, kwargs, result_semaphore = task
     try:
       function(*args, **kwargs)
     except: