Add a retry to Chromoting tests on Linux builders.

Add a retry to Chromoting tests on Linux builders.
Retry tests that fail once.

BUG=

Committed: https://crrev.com/2a6cb4b21b972471e7f6dd19374d3e1cefd73eda
Cr-Commit-Position: refs/heads/master@{#376833}

[anandc, 2016-02-19 23:09:15]

Description was changed from

==========
Add a retry to Chromoting tests on Linux builders.
Retry tests that fail once.

BUG=
==========

to

==========
Add a retry to Chromoting tests on Linux builders.
Retry tests that fail once.

BUG=
==========

[anandc, 2016-02-19 23:16:07]

Patchset #1 (id:1) has been deleted

[anandc1, 2016-02-19 23:22:27]

anandc@google.com changed reviewers:
+ joedow@chromium.org

[anandc1, 2016-02-19 23:22:27]

Hi Joe, PTAL.

The changes were tested on Swarming bots. Sample result for browser-tests (which
are failing due to known bug):
https://chromium-swarm.appspot.com/user/task/2d149967d0262510

Sample result for integration/test-driver test, with a forced failure to test
this change:
https://chromium-swarm.appspot.com/user/task/2d14a56662b95c10
Sample result for integration test with no other changes, showing success:
https://chromium-swarm.appspot.com/user/task/2d14a2ebb8f1a610

[joedow, 2016-02-20 17:08:38]

https://codereview.chromium.org/1718733002/diff/20001/testing/chromoting/brow...
File testing/chromoting/browser_tests_launcher.py (left):

https://codereview.chromium.org/1718733002/diff/20001/testing/chromoting/brow...
testing/chromoting/browser_tests_launcher.py:96: # Sometimes, during execution
of browser-tests, a browser instance is
I think this logic is important to keep.  In the long term we will remove the
generic retry logic but this was something you worked on quite a bit to track
down and it is likely to remain a problem (since it was never root caused).  I'd
hate to remove the retry logic and then start hitting his failure again.

It should be too hard to distinguish between this failure and some other
failure, the retry logic is ultimately the same but we would keep this
protection for later.

If you ultimately decide to rip this code out, then please remove the vars at
the top of the file that were referenced here but are no longer used.

https://codereview.chromium.org/1718733002/diff/20001/testing/chromoting/chro...
File testing/chromoting/chromoting_test_driver_launcher.py (right):

https://codereview.chromium.org/1718733002/diff/20001/testing/chromoting/chro...
testing/chromoting/chromoting_test_driver_launcher.py:46: while retries <=
MAX_RETRIES:
Can you add a TODO/bug to track the removal of the retry logic?  I don't think
we want to keep this for more than a few months as we stabilize the waterfall.

[anandc1, 2016-02-22 20:58:08]

Thanks Joe. PTAL at patch-set #2.

https://codereview.chromium.org/1718733002/diff/20001/testing/chromoting/brow...
File testing/chromoting/browser_tests_launcher.py (left):

https://codereview.chromium.org/1718733002/diff/20001/testing/chromoting/brow...
testing/chromoting/browser_tests_launcher.py:96: # Sometimes, during execution
of browser-tests, a browser instance is
On 2016/02/20 17:08:38, joedow wrote:
> I think this logic is important to keep.  In the long term we will remove the
> generic retry logic but this was something you worked on quite a bit to track
> down and it is likely to remain a problem (since it was never root caused). 
I'd
> hate to remove the retry logic and then start hitting his failure again.
> 
> It should be too hard to distinguish between this failure and some other
> failure, the retry logic is ultimately the same but we would keep this
> protection for later.
> 
> If you ultimately decide to rip this code out, then please remove the vars at
> the top of the file that were referenced here but are no longer used.

Done. Logging a line indicating what type of failure this potentially was and
retrying for any failure, up to MAX_RETRIES.

https://codereview.chromium.org/1718733002/diff/20001/testing/chromoting/chro...
File testing/chromoting/chromoting_test_driver_launcher.py (right):

https://codereview.chromium.org/1718733002/diff/20001/testing/chromoting/chro...
testing/chromoting/chromoting_test_driver_launcher.py:46: while retries <=
MAX_RETRIES:
On 2016/02/20 17:08:38, joedow wrote:
> Can you add a TODO/bug to track the removal of the retry logic?  I don't think
> we want to keep this for more than a few months as we stabilize the waterfall.

Done.

[joedow, 2016-02-22 21:52:15]

lgtm

https://codereview.chromium.org/1718733002/diff/40001/testing/chromoting/chro...
File testing/chromoting/chromoting_test_driver_launcher.py (right):

https://codereview.chromium.org/1718733002/diff/40001/testing/chromoting/chro...
testing/chromoting/chromoting_test_driver_launcher.py:46: # TODO(jamiewalch):
Remove this retry-logic once http://crbug/570840 is fixed.
I think this should be TODO(anandc): blah...

The TODO alias is for the person who wrote the TODO, not necessarily who is
going to address it.

[anandc1, 2016-02-22 21:56:08]

https://codereview.chromium.org/1718733002/diff/40001/testing/chromoting/chro...
File testing/chromoting/chromoting_test_driver_launcher.py (right):

https://codereview.chromium.org/1718733002/diff/40001/testing/chromoting/chro...
testing/chromoting/chromoting_test_driver_launcher.py:46: # TODO(jamiewalch):
Remove this retry-logic once http://crbug/570840 is fixed.
On 2016/02/22 21:52:15, joedow wrote:
> I think this should be TODO(anandc): blah...
> 
> The TODO alias is for the person who wrote the TODO, not necessarily who is
> going to address it.

I see. Thanks. Done.

[anandc1, 2016-02-22 21:56:23]

The CQ bit was checked by anandc@google.com

[anandc1, 2016-02-22 21:56:23]

The patchset sent to the CQ was uploaded after l-g-t-m from joedow@chromium.org
Link to the patchset: https://codereview.chromium.org/1718733002/#ps60001
(title: "Update TODO to anandc from jamiewalch.")

[commit-bot: I haz the power, 2016-02-22 21:56:39]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1718733002/60001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1718733002/60001

[commit-bot: I haz the power, 2016-02-22 22:41:35]

Description was changed from

==========
Add a retry to Chromoting tests on Linux builders.
Retry tests that fail once.

BUG=
==========

to

==========
Add a retry to Chromoting tests on Linux builders.
Retry tests that fail once.

BUG=
==========

[commit-bot: I haz the power, 2016-02-22 22:41:36]

Committed patchset #3 (id:60001)

[commit-bot: I haz the power, 2016-02-22 22:42:50]

Description was changed from

==========
Add a retry to Chromoting tests on Linux builders.
Retry tests that fail once.

BUG=
==========

to

==========
Add a retry to Chromoting tests on Linux builders.
Retry tests that fail once.

BUG=

Committed: https://crrev.com/2a6cb4b21b972471e7f6dd19374d3e1cefd73eda
Cr-Commit-Position: refs/heads/master@{#376833}
==========

[commit-bot: I haz the power, 2016-02-22 22:42:51]

Patchset 3 (id:??) landed as
https://crrev.com/2a6cb4b21b972471e7f6dd19374d3e1cefd73eda
Cr-Commit-Position: refs/heads/master@{#376833}