elf_symbolizer: Use a process for max 4000 lookups and then restart

build/android/pylib/symbols/elf_symbolizer.py

Index: build/android/pylib/symbols/elf_symbolizer.py
diff --git a/build/android/pylib/symbols/elf_symbolizer.py b/build/android/pylib/symbols/elf_symbolizer.py
index 90e99e5a2757770b30d8d3a89d0a34b589212780..a5761d1e0404f7c166d721ab85e2b66d00240e13 100644
--- a/build/android/pylib/symbols/elf_symbolizer.py
+++ b/build/android/pylib/symbols/elf_symbolizer.py
@@ -15,6 +15,13 @@ import sys
 import threading
 
 
+# addr2line builds a possibly infinite memory cache that can exhaust
+# the computer's memory if allowed to grow for too long. This constant
+# controls how many lookups we do before restarting the process. 4000
+# gives near peak performance without extreme memory usage.
+ADDR2LINE_RECYCLE_LIMIT = 4000
+
+
 class ELFSymbolizer(object):
   """An uber-fast (multiprocessing, pipelined and asynchronous) ELF symbolizer.
 
@@ -117,6 +124,7 @@ class ELFSymbolizer(object):
     # Essentially, this drains all the addr2line(s) out queues.
     for a2l_to_purge in self._a2l_instances:
       a2l_to_purge.ProcessAllResolvedSymbolsInQueue()
+      a2l_to_purge.RecycleIfNecessary()
 
     # Find the best instance according to this logic:
     # 1. Find an existing instance with the shortest queue.
@@ -184,6 +192,10 @@ class ELFSymbolizer(object):
       # separate field because turned out to be a perf hot-spot.
       self.queue_size = 0
 
+      # Keep track of the number of symbols a process has processed to
+      # avoid a single process growing too big and using all the memory.
+      self._processed_symbols_count = 0

[Primiano Tucci (use gerrit), 2014/06/05 08:57:45]

Move this hunk in _RestartAddr2LineProcess, so you don't need to reset
_processed_symbols_count = 0 twice (here and on line 272), but just on _Restart

[Daniel Bratell, 2014/06/05 09:07:52]

It made pylint unhappy:
W0201:348,6:ELFSymbolizer.Addr2Line._RestartAddr2LineProcess: Attribute
'_processed_symbols_count' defined outside __init__

I agree that setting it inside _Restart is good though. Setting it only when
restarted through the recycler was a bug.

[Primiano Tucci (use gerrit), 2014/06/05 12:57:18]

Oh right, you definitely need to define it in __init__ (but should be reset in
restart).
 Good point

+
       # Objects required to handle the addr2line subprocess.
       self._proc = None  # Subprocess.Popen(...) instance.
       self._thread = None  # Threading.thread instance.
@@ -251,6 +263,16 @@ class ELFSymbolizer(object):
           break
         self._ProcessSymbolOutput(lines)
 
+    def RecycleIfNecessary(self):
+      """Restarts the process if it has been used for too long.
+
+      A long running addr2line process will consume excessive amounts
+      of memory without any gain in performance."""
+      if self._processed_symbols_count >= ADDR2LINE_RECYCLE_LIMIT:
+        self._processed_symbols_count = 0
+        self._RestartAddr2LineProcess()
+
+
     def Terminate(self):
       """Kills the underlying addr2line process.
 
@@ -297,6 +319,7 @@ class ELFSymbolizer(object):
         if not innermost_sym_info:
           innermost_sym_info = sym_info
 
+      self._processed_symbols_count += 1
       self._symbolizer.callback(innermost_sym_info, callback_arg)
 
     def _RestartAddr2LineProcess(self):