Waterfall components of regression range finder.

appengine/findit/waterfall/flake/initialize_flake_pipeline.py

+# Copyright 2014 The Chromium Authors. All rights reserved.

[chanli, 2016/07/19 20:48:27]

2014 -> 2016

[caiw, 2016/07/20 18:26:47]

Done.

+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+from datetime import datetime
+import logging

[chanli, 2016/07/19 20:48:27]

Remove this import

[lijeffrey, 2016/07/19 22:44:10]

If you run gpylint <path to python file> it should point out unused imports like
this

[caiw, 2016/07/20 18:26:47]

Done.

+
+from google.appengine.ext import ndb
+
+from common import appengine_util
+from common import constants
+from model import analysis_status
+from model.flake.flake_swarming_task import FlakeSwarmingTask
+from model.flake.master_flake_analysis import MasterFlakeAnalysis
+from waterfall.flake.recursive_flake_pipeline import RecursiveFlakePipeline
+
+
+@ndb.transactional
+def NeedANewAnalysis(
+    master_name, builder_name, build_number, step_name, test_name):
+  """Checks status of analysis for the test and decides if a new one is needed.
+
+  A MasterFlakeAnalysis entity for the given build will be created if none
+  exists. When a new analysis is needed, this function will create and
+  save a MasterFlakeAnalysis entity to the datastore, or it will
+  reset the existing one but still keep the result of last analysis.
+
+  Returns:
+    True if an analysis is needed, otherwise False.
+  """
+  analysis = MasterFlakeAnalysis.Get(master_name, builder_name, build_number,
+                                     step_name, test_name)
+
+  if not analysis:
+    analysis = MasterFlakeAnalysis.Create(
+        master_name, builder_name, build_number, step_name, test_name)
+    analysis.status = analysis_status.PENDING
+    analysis.put()
+    return True
+  elif (analysis.status == analysis_status.COMPLETED or
+        analysis.status == analysis_status.PENDING or
+        analysis.status == analysis_status.RUNNING):
+    return False
+  else:

[chanli, 2016/07/19 20:48:28]

First: Maybe you can create the Reset method and remove the TODO?

Second: Here you just delete the entity right? But in docstring you said the
result of last analysis will be kept?

[caiw, 2016/07/20 18:26:47]

So this is only if the status of the last analysis is an error.  If that's the
case, I wanted to delete it and just run it again.  The todo is just a reminder
to move this logic into a reset method - I was hoping to do that after we got
the CL committed. 

[chanli, 2016/07/21 20:34:09]

Acknowledged.

+    # TODO(caiw): Reset method.
+    MasterFlakeAnalysis.Get(
+        master_name, builder_name, build_number,
+        step_name, test_name).key.delete()
+    analysis = MasterFlakeAnalysis.Create(
+        master_name, builder_name, build_number, step_name, test_name)
+    analysis.status = analysis_status.PENDING
+    analysis.put()
+    return True
+
+# Unused arguments - pylint: disable=W0612, W0613
+def ScheduleAnalysisIfNeeded(master_name, builder_name, build_number, step_name,
+                             test_name, force=False,

[chanli, 2016/07/19 20:48:28]

Is force saved for later change?

[caiw, 2016/07/20 18:26:47]

Yes - I think later we will need to figure out who has the permissions to run
these things?  I am not completely clear on how it works.

[chanli, 2016/07/21 20:34:09]

I think we can add force when we want to do it? 

[caiw, 2016/07/22 21:46:42]

So should we do it now? 

[chanli, 2016/07/26 17:41:05]

I would say add it in a separated CL, even the change is small.

[caiw, 2016/07/26 20:19:44]

Acknowledged.

+                             queue_name=constants.DEFAULT_QUEUE):
+  """Schedules an analysis if needed and returns the MasterFlakeAnalysis.
+
+  When the build failure was already analyzed and a new analysis is scheduled,
+  the returned WfAnalysis will still have the result of last completed analysis.
+
+  Args:
+    master_name (str): The master name of the failed test
+    builder_name (str): The builder name of the failed test
+    build_number (int): The build number of the failed test
+    step_name (str): The name of the test suite
+    test_name (str): The single test we are checking
+  Returns:
+    A MasterFlakeAnalysis instance.
+  """
+  if NeedANewAnalysis(
+      master_name, builder_name, build_number, step_name, test_name):
+    pipeline_job = RecursiveFlakePipeline(
+        master_name, builder_name, build_number, step_name, test_name,
+        master_build_number=build_number)
+    pipeline_job.target = appengine_util.GetTargetNameForModule(
+        constants.WATERFALL_BACKEND)
+    pipeline_job.start(queue_name=queue_name)
+  return MasterFlakeAnalysis.Get(
+      master_name, builder_name, build_number, step_name, test_name)