[Findit] scripts to calculate confidence level of Findit results.

appengine/findit/model/cl_confidence.py

Index: appengine/findit/model/cl_confidence.py
diff --git a/appengine/findit/model/cl_confidence.py b/appengine/findit/model/cl_confidence.py
new file mode 100644
index 0000000000000000000000000000000000000000..205c141810fdc757d70ba2ddaaa1226de87a50f8
--- /dev/null
+++ b/appengine/findit/model/cl_confidence.py
@@ -0,0 +1,91 @@
+# Copyright 2016 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+from google.appengine.ext import ndb
+
+
+class CLConfidence(ndb.Model):

[stgao, 2016/10/06 01:57:59]

Should we versionize this model to keep track of improvement over time?

[chanli, 2016/10/07 00:01:12]

That's already taken care of with the original implementation. Versionize it for
caching.

+  """Stores confidence data of different types of suspected CLs.
+
+  Confidence data includes confidence scores and the numbers we used to get
+  those scores.
+
+  The types of suspected CLs are:
+  1. CLs for compile failures found by Heuristic approach.
+      a. The score has been further refined by the top score of hints.
+  2. CLs for compile failures found by Try Job approach.
+  3. CLs for compile failures found by both approaches.
+  4. CLs for test failures found by Heuristic approach.
+      a. The score has been further refined by the top score of hints.
+  5. CLs for test failures found by Try Job approach.
+  6. CLs for test failures found by both approaches.
+  """
+
+  # Start date of querying suspected CLs.
+  # Note: the confidence scores will be for all the CLs up until end_date,
+  # not just for CLs from start date to end_date.
+  start_date = ndb.DateTimeProperty(indexed=False)
+
+  # End date of querying suspected CLs.
+  end_date = ndb.DateTimeProperty(indexed=True)
+

[stgao, 2016/10/06 01:57:59]

Add another field for when the confidence data is updated?

[chanli, 2016/10/07 00:01:12]

It seems not useful for me. I think end_date is sufficient. If you can think of
any use of updated_time, I can add it.

+  # Confidence scores for CLs for compile failures found by Heuristic approach.
+  # A dict like below:
+  # {
+  #     '1': {
+  #         'correct': 30,
+  #         'total': 40,
+  #         'confidence': 0.75
+  #     },

[stgao, 2016/10/06 01:57:59]

Can we use another model to record the data? And use LocalStructedProperty
instead of pure JsonProperty?

[chanli, 2016/10/07 00:01:12]

Done.

+  #     ...
+  # }
+  compile_heuristic = ndb.JsonProperty(indexed=False, compressed=True)
+
+  # Confidence score for CLs for compile failures found by Try Job approach.
+  # A dict like below:
+  # {
+  #   'correct': 96,
+  #   'total': 100,
+  #   'confidence': 0.96
+  # }
+  compile_try_job = ndb.JsonProperty(indexed=False, compressed=True)
+
+  # Confidence score for CLs for compile failures found by both approaches.
+  # A dict like below:
+  # {
+  #   'correct': 20,
+  #   'total': 20,
+  #   'confidence': 1.00
+  # }
+  compile_heuristic_try_job = ndb.JsonProperty(indexed=False, compressed=True)
+
+  # Confidence scores for CLs for test failures found by Heuristic approach.
+  # A dict like below:
+  # {
+  #     '1': {
+  #         'correct': 30,
+  #         'total': 40,
+  #         'confidence': 0.75
+  #     },
+  #     ...
+  # }
+  test_heuristic = ndb.JsonProperty(indexed=False, compressed=True)
+
+  # Confidence score for CLs for test failures found by Try Job approach.
+  # A dict like below:
+  # {
+  #   'correct': 96,
+  #   'total': 100,
+  #   'confidence': 0.96
+  # }
+  test_try_job = ndb.JsonProperty(indexed=False, compressed=True)
+
+  # Confidence score for CLs for test failures found by both approaches.
+  # A dict like below:
+  # {
+  #   'correct': 20,
+  #   'total': 20,
+  #   'confidence': 1.00
+  # }
+  test_heuristic_try_job = ndb.JsonProperty(indexed=False, compressed=True)