Raise Exception instead of failure when some steps fail.

Raise Exception instead of failure when some steps fail.

BUG=skia:2000
R=epoger@google.com, rmistry@google.com

Committed: https://skia.googlesource.com/buildbot/+/9623fb0

[borenet, 2014-02-21 21:55:55]

I didn't include the mass of factory configuration files:

@@ -1045,6 +1060,7 @@
                 'description': 'UploadBenchResults',
                 'doStepIf': <Object>,
                 'env': {},
+                'exception_on_failure': <Object>,
                 'flunkOnFailure': <Object>,
                 'get_props_from_stdout': None,
                 'haltOnFailure': <Object>,

x1000000000

[rmistry, 2014-02-24 14:35:24]

Could you explain the context behind this? When would this be required?

https://codereview.chromium.org/175523003/diff/70001/master/skia_master_scrip...
File master/skia_master_scripts/commands.py (right):

https://codereview.chromium.org/175523003/diff/70001/master/skia_master_scrip...
master/skia_master_scripts/commands.py:89:
exception_on_failure=exception_on_failure)
What is the difference between exception_on_failures and flunk_on_failure?

[borenet, 2014-02-24 14:42:55]

The idea is to better distinguish between different types of failures.  Here
I've made the steps which should never fail due to a Skia code change turn
purple instead of red when they fail.  This indicates either a transient failure
or something which needs attention from the infra team.  For example, if
UploadGMResults fails due to a network or codesite issue, the build shouldn't go
red.  Instead, it should go purple.

Relevant bug: https://code.google.com/p/skia/issues/detail?id=2000

https://codereview.chromium.org/175523003/diff/70001/master/skia_master_scrip...
File master/skia_master_scripts/commands.py (right):

https://codereview.chromium.org/175523003/diff/70001/master/skia_master_scrip...
master/skia_master_scripts/commands.py:89:
exception_on_failure=exception_on_failure)
On 2014/02/24 14:35:24, rmistry wrote:
> What is the difference between exception_on_failures and flunk_on_failure?

flunk_on_failure means that if this step fails, the build as a whole fails. 
exception_on_failure means that if this step fails, the master throws an
exception, causing the build to go purple instead of red.

[rmistry, 2014-02-24 14:47:54]

https://codereview.chromium.org/175523003/diff/70001/master/skia_master_scrip...
File master/skia_master_scripts/skia_build_step.py (right):

https://codereview.chromium.org/175523003/diff/70001/master/skia_master_scrip...
master/skia_master_scripts/skia_build_step.py:32: this step fails.
Can we add here something like:
If there is a failure then the build will turn purple instead of red. Should be
used when a buildstep's failure is typically transient or infra related.

[rmistry, 2014-02-24 14:48:36]

LGTM

On 2014/02/24 14:42:55, borenet wrote:
> The idea is to better distinguish between different types of failures.  Here
> I've made the steps which should never fail due to a Skia code change turn
> purple instead of red when they fail.  This indicates either a transient
failure
> or something which needs attention from the infra team.  For example, if
> UploadGMResults fails due to a network or codesite issue, the build shouldn't
go
> red.  Instead, it should go purple.

Makes sense. Could you please add the above explanation to the CL's description.

> 
> Relevant bug: https://code.google.com/p/skia/issues/detail?id=2000
> 
>
https://codereview.chromium.org/175523003/diff/70001/master/skia_master_scrip...
> File master/skia_master_scripts/commands.py (right):
> 
>
https://codereview.chromium.org/175523003/diff/70001/master/skia_master_scrip...
> master/skia_master_scripts/commands.py:89:
> exception_on_failure=exception_on_failure)
> On 2014/02/24 14:35:24, rmistry wrote:
> > What is the difference between exception_on_failures and flunk_on_failure?
> 
> flunk_on_failure means that if this step fails, the build as a whole fails. 
> exception_on_failure means that if this step fails, the master throws an
> exception, causing the build to go purple instead of red.

[borenet, 2014-02-24 15:05:46]

Uploaded patch set 4.  Pulling in Elliot for discussion.  In short, I'm
wondering how many of these steps should cause the build to halt.  This is kind
of unfortunate because I'd like the build to go purple if UploadGMResults fails
but I'd still like the subsequent steps to run before the build stops.

https://codereview.chromium.org/175523003/diff/70001/master/skia_master_scrip...
File master/skia_master_scripts/skia_build_step.py (right):

https://codereview.chromium.org/175523003/diff/70001/master/skia_master_scrip...
master/skia_master_scripts/skia_build_step.py:32: this step fails.
On 2014/02/24 14:47:54, rmistry wrote:
> Can we add here something like:
> If there is a failure then the build will turn purple instead of red. Should
be
> used when a buildstep's failure is typically transient or infra related.

Done. Note that this does cause the build as a whole to stop.

[borenet, 2014-02-24 16:13:27]

(Actually adding Elliot)

[epoger, 2014-02-24 18:18:43]

LGTM

On 2014/02/24 15:05:46, borenet wrote:
> Uploaded patch set 4.  Pulling in Elliot for discussion.  In short, I'm
> wondering how many of these steps should cause the build to halt.  This is
kind
> of unfortunate because I'd like the build to go purple if UploadGMResults
fails
> but I'd still like the subsequent steps to run before the build stops.

Discussed live... for the most part, it's fine for the build to halt when we see
one of these failures.

[borenet, 2014-02-24 21:06:46]

Committed patchset #5 manually as r9623fb0 (presubmit successful).