[Findit] Enable project classifier and component classifier

appengine/findit/crash/component_classifier.py

Index: appengine/findit/crash/component_classifier.py
diff --git a/appengine/findit/crash/component_classifier.py b/appengine/findit/crash/component_classifier.py
new file mode 100644
index 0000000000000000000000000000000000000000..366a37e67f2b1010e9e46bb096acbda395f1525f
--- /dev/null
+++ b/appengine/findit/crash/component_classifier.py
@@ -0,0 +1,78 @@
+# Copyright 2016 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+import copy
+import re
+
+from crash.classifier import Classifier
+from model.crash.crash_config import CrashConfig
+
+
+class ComponentClassifier(Classifier):
+  """Determines the component of a crash.
+
+  For example: ['Blink>DOM', 'Blink>HTML'].
+  """
+
+  def __init__(self):
+    super(ComponentClassifier, self).__init__()
+
+    def _GetCompiledConfig(config):
+      config = copy.copy(config)

[stgao, 2016/05/17 21:40:57]

deepcopy

[Sharu Jiang, 2016/05/20 23:16:33]

This is not necessary any more.

+      config['top_n'] = int(config['top_n'])

[stgao, 2016/05/17 21:40:57]

Why this is needed?

[Sharu Jiang, 2016/05/20 23:16:33]

Oops...

+      path_function_component = []
+
+      for path, function, component in config['path_function_component']:

[stgao, 2016/05/17 21:40:57]

Could this be cached in the CrashConfig instance?

This will be executed for each analyzed crash. The overall overhead might be
high.

[Sharu Jiang, 2016/05/20 23:16:33]

I do a CrashConfig.Get() for every crash, so I don't think cache it in the
instance would help, or you mean the instance in datastore? then yes, I changed
this to a BlobProperty there:), but I still need to decompress and unpickle for
every crashes, but the overhead shouldn't be big I think. 

Another option is to cache compiled settings as a class variable of CrashConfig
class, and every time we click save button, it will update this class variable,
but this would cause race condition, and I don't think it's very necessary.

[stgao, 2016/05/21 00:54:03]

On appengine, NDB has an app-instance cache for NDB data, before it reaches to
memcache or NDB.
A non-transactional Model.get() is not necessarily to retrieve data from NDB.
I don't quite understand how this will work.

To be more specific,  my suggestion is something like this (cache in an
instance):

@property
def processed_data(self):
   if not self.cached_data:
      # build the pattern and cache it.
      self.cached_data = ...
   return self.cached_data

This is an optimization to avoid overhead of computing the pattern, and also
save memory for string operations. We have a lot of OOMs recently.

[Sharu Jiang, 2016/05/23 23:54:50]

Done.

+        path_function_component.append((
+            re.compile(path),
+            re.compile(function) if function else None,
+            component))
+      config['path_function_component'] = path_function_component
+
+      return config
+
+    self.config = _GetCompiledConfig(CrashConfig.Get().component_classifier)
+
+  def GetClassFromStackFrame(self, frame):
+    """Gets the component from file path and function of a frame."""
+    for path_regex, function_regex, component in self.config[
+        'path_function_component']:
+      path_match = path_regex.match(frame.dep_path + frame.file_path)
+      if not path_match:
+        continue
+
+      if not function_regex:
+        return component
+
+      function_match = function_regex.match(frame.function)
+      if function_match:
+        return component
+
+    return ''
+
+  def GetClassFromResult(self, result):
+    """Gets the component from a result.
+
+    Note that Findit assumes files that the culprit result touched come from
+    the same component.
+    """
+    if result.file_to_stack_infos:
+      # A file in culprit result should always have its stack_info, namely a
+      # list of (frame, callstack_priority) pairs.
+      frame, _ = result.file_to_stack_infos.values()[0][0]
+      return self.GetClassFromStackFrame(frame)
+
+    return ''
+
+  def Classify(self, results, crash_stack):
+    """Classifies project of a crash.
+
+    Args:
+      results (list of Result): Culprit results.
+      crash_stack (CallStack): The callstack that caused the crash.
+
+    Returns:
+      List of top 2 components.
+    """
+    return self._Classify(results, crash_stack, self.config['top_n'], 2)