Waterfall components of regression range finder.

appengine/findit/waterfall/flake/recursive_flake_pipeline.py

Index: appengine/findit/waterfall/flake/recursive_flake_pipeline.py
diff --git a/appengine/findit/waterfall/flake/recursive_flake_pipeline.py b/appengine/findit/waterfall/flake/recursive_flake_pipeline.py
new file mode 100644
index 0000000000000000000000000000000000000000..24d7cc097cd90f94d866de88a1c3b946d6999717
--- /dev/null
+++ b/appengine/findit/waterfall/flake/recursive_flake_pipeline.py
@@ -0,0 +1,57 @@
+# Copyright 2015 The Chromium Authors. All rights reserved.

[chanli, 2016/07/19 20:48:28]

2015 -> 2016

[caiw, 2016/07/20 18:26:47]

Done.

+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+from datetime import datetime
+import logging

[chanli, 2016/07/19 20:48:28]

Remove this import?

[caiw, 2016/07/20 18:26:47]

Done.

+
+from common import appengine_util
+from common import constants
+from common.pipeline_wrapper import BasePipeline
+
+from model import analysis_status
+from model.flake.master_flake_analysis import MasterFlakeAnalysis
+from waterfall.trigger_flake_swarming_task_pipeline import (
+    TriggerFlakeSwarmingTaskPipeline)
+from waterfall.process_flake_swarming_task_result_pipeline import (
+    ProcessFlakeSwarmingTaskResultPipeline)
+
+
+class RecursiveFlakePipeline(BasePipeline):
+
+  # Arguments number differs from overridden method - pylint: disable=W0221
+  def run(self, master_name, builder_name, run_build_number, step_name,
+          test_name, master_build_number, queue_name=constants.DEFAULT_QUEUE):
+    # Call trigger pipeline (flake style)

[lijeffrey, 2016/07/19 22:44:10]

nit: comments end with .

[caiw, 2016/07/20 18:26:47]

Done.

+    task_id = yield TriggerFlakeSwarmingTaskPipeline(master_name, builder_name,
+                          run_build_number, step_name, [test_name])
+    # Pass the trigger pipeline into a process pipeline
+    test_result_future = yield ProcessFlakeSwarmingTaskResultPipeline(
+        master_name, builder_name, run_build_number,
+        step_name, task_id, master_build_number, test_name)
+    yield NextBuildNumberPipeline(
+        master_name, builder_name, master_build_number,
+        step_name, test_name, test_result_future, queue_name)
+
+class NextBuildNumberPipeline(BasePipeline):
+  # Arguments number differs from overridden method - pylint: disable=W0221
+  # Unused argument - pylint: disable=W0613
+  def run(self, master_name, builder_name, master_build_number, step_name,
+          test_name, test_result_future, queue_name):
+    # Get MasterFlakeAnalysis success list corresponding to parameters

[chanli, 2016/07/19 20:48:28]

Nit: Gets ... and add '.' at the end.

+    master = MasterFlakeAnalysis.Get(master_name, builder_name,
+                                     master_build_number, step_name, test_name)
+    # Figure out what build_number we should call, if any
+    # This is a placeholder for testing:
+    next_run = False
+    if len(master.build_numbers) < 10:

[lijeffrey, 2016/07/19 22:44:10]

It looks like 10 is used twice and not sure if it's related to line 49. If it
is, move it to a constant at the top of the file.

If 10 is subject to change I'd make it configurable and add a todo here that
this should be set in Findit's config (see model/wf_config.py and
waterfall/waterfall_config.py if you're curious)

[caiw, 2016/07/20 18:26:47]

So they aren't related, and also the algorithm (which is currently in
development in a googledoc somewhere if you want to take a look!) doesn't use
this same logic of checking if we did x number of swarming reruns and then stop,
so I think it's not worth it to set it in the config since the next CL I submit
will probably remove this anyways.

+      #TODO(caiw): Develop algorithm to optimize this.

[lijeffrey, 2016/07/19 22:44:10]

nit: space after #

[caiw, 2016/07/20 18:26:47]

Done.

+      next_run = min(master.build_numbers) - 10
+    if next_run:
+      pipeline_job = RecursiveFlakePipeline(
+          master_name, builder_name, next_run, step_name, test_name,
+          master_build_number)
+      #pylint: disable=W0201
+      pipeline_job.target = appengine_util.GetTargetNameForModule(
+          constants.WATERFALL_BACKEND)
+      pipeline_job.start(queue_name=queue_name)