Reformat bisect results output.

Reformat bisect results output.

Reformat the bisect results output to give detailed information regarding the bisect job.
Includes clear instructions to repro steps.

BUG=383669, 383935
NOTRY=true

Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=282735

[prasadv, 2014-07-10 21:31:59]

This is how it looks now!

===== BISECT JOB RESULTS =====
Status: Successful with low confidence and warnings.

Test Command: python tools/perf/run_benchmark -v --browser=release
robohornet_pro
Test Metric: Total/Total
Relative Change: 1.23% (+/-1.26%)
Estimated Confidence: 59%

WARNINGS:
  !!! Confidence is less than 100%. There could be other candidates for this
regression. Try bisecting again with increased repeat_count or on a sub-metric
that shows the regression more clearly.

===== SUSPECTED CL(s) =====
Subject : Handle redirects with NULL headers in PluginURLFetcher
Author  : davidben@chromium.org
Link    : http://src.chromium.org/viewvc/chrome?view=revision&revision=278883
Date    : Sat, 21 Jun 2014 01:00:33 +0000

===== TESTED COMMITS =====
       Depot                                      Commit SHA                    
              Mean      Std. Error     State    

      chromium      
http://src.chromium.org/viewvc/chrome?view=revision&revision=278883    3264.00  
    +-8.00     Suspected CL

      chromium      
http://src.chromium.org/viewvc/chrome?view=revision&revision=278882    3304.50  
   +-40.50         Good    

Average build time : 0:01:56
Average test time  : 0:00:25

==== INSTRUCTIONS TO REPRODUCE ====
To run locally:
$$ python tools/perf/run_benchmark -v --browser=release robohornet_pro
Also consider passing --profiler=list to see available profilers.

To reproduce on Performance trybot:
1. Create new git branch or check out existing branch.
2. Edit tools/run-perf-test.cfg (instructions in file) or
third_party/WebKit/Tools/run-perf-test.cfg.
  a) Take care to strip any src/ directories from the head of relative path
names.
  b) On desktop, only --browser=release is supported, on android
--browser=android-chromium-testshell.
  c) Test command to use: $ python tools/perf/run_benchmark -v --browser=release
robohornet_pro
Also consider passing --profiler=list to see available profilers.
3. Upload your patch. --bypass-hooks is necessary to upload the changes you
committed locally to run-perf-test.cfg.
   Note: *DO NOT* commit run-perf-test.cfg changes to the project repository.
   $ git cl upload --bypass-hooks
4. Send your try job to the tryserver. [Please make sure to use appropriate bot
to reproduce]
   $ git cl try -m tryserver.chromium.perf -b <bot>

For more details please visit 
https://sites.google.com/a/chromium.org/dev/developers/performance-try-bots

===== THANK YOU FOR CHOOSING BISECT AIRLINES =====
Visit http://www.chromium.org/developers/core-principles for Chrome's policy
on perf regressions.
Contact chrome-perf-dashboard-team with any questions or suggestions about
bisecting.
                    .------.
      .---.         \       \==)
      |PERF\         \       \
      |     ---------'-------'-----------.
      . 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 `-.
      \______________.-------._______________)
                    /       /
                   /       /
                  /       /==)
                 ._______.

[chromium-reviews, 2014-07-10 21:38:35]

oops! I think output got messed up due to gmail compose auto-format.

Attached a file with results output for your reference


On Thu, Jul 10, 2014 at 2:31 PM, <prasadv@chromium.org> wrote:

> Reviewers: simonhatch, tonyg, qyearsley,
>
> Message:
> This is how it looks now!
>
> ===== BISECT JOB RESULTS =====
> Status: Successful with low confidence and warnings.
>
> Test Command: python tools/perf/run_benchmark -v --browser=release
> robohornet_pro
> Test Metric: Total/Total
> Relative Change: 1.23% (+/-1.26%)
> Estimated Confidence: 59%
>
> WARNINGS:
>   !!! Confidence is less than 100%. There could be other candidates for
> this
> regression. Try bisecting again with increased repeat_count or on a
> sub-metric
> that shows the regression more clearly.
>
> ===== SUSPECTED CL(s) =====
> Subject : Handle redirects with NULL headers in PluginURLFetcher
> Author  : davidben@chromium.org
> Link    : http://src.chromium.org/viewvc/chrome?view=revision&
> revision=278883
> Date    : Sat, 21 Jun 2014 01:00:33 +0000
>
> ===== TESTED COMMITS =====
>        Depot                                      Commit SHA
>               Mean      Std. Error     State
>
>       chromium
> http://src.chromium.org/viewvc/chrome?view=revision&revision=278883
>  3264.00
>     +-8.00     Suspected CL
>
>       chromium
> http://src.chromium.org/viewvc/chrome?view=revision&revision=278882
>  3304.50
>    +-40.50         Good
>
> Average build time : 0:01:56
> Average test time  : 0:00:25
>
> ==== INSTRUCTIONS TO REPRODUCE ====
> To run locally:
> $$ python tools/perf/run_benchmark -v --browser=release robohornet_pro
> Also consider passing --profiler=list to see available profilers.
>
> To reproduce on Performance trybot:
> 1. Create new git branch or check out existing branch.
> 2. Edit tools/run-perf-test.cfg (instructions in file) or
> third_party/WebKit/Tools/run-perf-test.cfg.
>   a) Take care to strip any src/ directories from the head of relative path
> names.
>   b) On desktop, only --browser=release is supported, on android
> --browser=android-chromium-testshell.
>   c) Test command to use: $ python tools/perf/run_benchmark -v
> --browser=release
> robohornet_pro
> Also consider passing --profiler=list to see available profilers.
> 3. Upload your patch. --bypass-hooks is necessary to upload the changes you
> committed locally to run-perf-test.cfg.
>    Note: *DO NOT* commit run-perf-test.cfg changes to the project
> repository.
>    $ git cl upload --bypass-hooks
> 4. Send your try job to the tryserver. [Please make sure to use
> appropriate bot
> to reproduce]
>    $ git cl try -m tryserver.chromium.perf -b <bot>
>
> For more details please visit
> https://sites.google.com/a/chromium.org/dev/developers/
> performance-try-bots
>
> ===== THANK YOU FOR CHOOSING BISECT AIRLINES =====
> Visit http://www.chromium.org/developers/core-principles for Chrome's
> policy
> on perf regressions.
> Contact chrome-perf-dashboard-team with any questions or suggestions about
> bisecting.
>                     .------.
>       .---.         \       \==)
>       |PERF\         \       \
>       |     ---------'-------'-----------.
>       . 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 `-.
>       \______________.-------._______________)
>                     /       /
>                    /       /
>                   /       /==)
>                  ._______.
>
>
> Description:
> Reformat bisect results output.
>
> Reformat the bisect results output to give detailed information regarding
> the
> bisect job.
> Includes clear instructions to repro steps.
>
> BUG=383669,383935
> NOTRY=true
>
> Please review this at https://codereview.chromium.org/388623002/
>
> SVN Base: svn://svn.chromium.org/chrome/trunk/src
>
> Affected files (+124, -40 lines):
>   M tools/bisect-perf-regression.py
>
>
>

To unsubscribe from this group and stop receiving emails from it, send an email
to chromium-reviews+unsubscribe@chromium.org.

[shatch, 2014-07-10 23:03:09]

lg2m but I think Quinten had a bunch of suggestions from the bug.

Were you going to explore also generating the cfg file for them?

https://codereview.chromium.org/388623002/diff/1/tools/bisect-perf-regression.py
File tools/bisect-perf-regression.py (right):

https://codereview.chromium.org/388623002/diff/1/tools/bisect-perf-regression...
tools/bisect-perf-regression.py:208: REPRO_STEPS_LOCAL ="""
nit: space here and on the thankyou

https://codereview.chromium.org/388623002/diff/1/tools/bisect-perf-regression...
tools/bisect-perf-regression.py:3306: print RESULTS_BANNER % (
I like your use of keyword arguments for the other output changes, maybe here
too for consistency?

https://codereview.chromium.org/388623002/diff/1/tools/bisect-perf-regression...
tools/bisect-perf-regression.py:3503: print '  !!! %s' % w
Vaguely recall from Jeff's doc that he wanted to get rid of the multiple !'s?

[prasadv, 2014-07-11 00:34:54]

ptal,
regarding generating the cfg file for them, I think this can be done as a part
of dashboard changes and append it to results output.

https://codereview.chromium.org/388623002/diff/1/tools/bisect-perf-regression.py
File tools/bisect-perf-regression.py (right):

https://codereview.chromium.org/388623002/diff/1/tools/bisect-perf-regression...
tools/bisect-perf-regression.py:208: REPRO_STEPS_LOCAL ="""
On 2014/07/10 23:03:09, shatch wrote:
> nit: space here and on the thankyou

Done.

https://codereview.chromium.org/388623002/diff/1/tools/bisect-perf-regression...
tools/bisect-perf-regression.py:3306: print RESULTS_BANNER % (
On 2014/07/10 23:03:09, shatch wrote:
> I like your use of keyword arguments for the other output changes, maybe here
> too for consistency?

Done.

https://codereview.chromium.org/388623002/diff/1/tools/bisect-perf-regression...
tools/bisect-perf-regression.py:3503: print '  !!! %s' % w
On 2014/07/10 23:03:09, shatch wrote:
> Vaguely recall from Jeff's doc that he wanted to get rid of the multiple !'s?

Done.

[shatch, 2014-07-11 17:09:05]

On 2014/07/11 00:34:54, prasadv wrote:
> ptal,
> regarding generating the cfg file for them, I think this can be done as a part
> of dashboard changes and append it to results output.
> 

Ah ok!

>
https://codereview.chromium.org/388623002/diff/1/tools/bisect-perf-regression.py
> File tools/bisect-perf-regression.py (right):
> 
>
https://codereview.chromium.org/388623002/diff/1/tools/bisect-perf-regression...
> tools/bisect-perf-regression.py:208: REPRO_STEPS_LOCAL ="""
> On 2014/07/10 23:03:09, shatch wrote:
> > nit: space here and on the thankyou
> 
> Done.
> 
>
https://codereview.chromium.org/388623002/diff/1/tools/bisect-perf-regression...
> tools/bisect-perf-regression.py:3306: print RESULTS_BANNER % (
> On 2014/07/10 23:03:09, shatch wrote:
> > I like your use of keyword arguments for the other output changes, maybe
here
> > too for consistency?
> 
> Done.
> 
>
https://codereview.chromium.org/388623002/diff/1/tools/bisect-perf-regression...
> tools/bisect-perf-regression.py:3503: print '  !!! %s' % w
> On 2014/07/10 23:03:09, shatch wrote:
> > Vaguely recall from Jeff's doc that he wanted to get rid of the multiple
!'s?
> 
> Done.

lgtm, wait for Quinten as well since he had a few ideas listed in the bug

[prasadv, 2014-07-11 18:51:25]

Hi Quinten, PTAL

thanks

[qyearsley, 2014-07-11 18:55:34]

On 2014/07/11 18:51:25, prasadv wrote:
> Hi Quinten, PTAL
> 
> thanks

LGTM since my comments regarding the "BISECT JOB RESULTS" section were
addressed, although I still feel that the "INSTRUCTIONS TO REPRODUCE" and "THANK
YOU" sections make it too long over all; if you submit this now then maybe we'll
want to change that later?

[prasadv, 2014-07-11 23:20:55]

On 2014/07/11 18:55:34, qyearsley wrote:
> On 2014/07/11 18:51:25, prasadv wrote:
> > Hi Quinten, PTAL
> > 
> > thanks
> 
> LGTM since my comments regarding the "BISECT JOB RESULTS" section were
> addressed, although I still feel that the "INSTRUCTIONS TO REPRODUCE" and
"THANK
> YOU" sections make it too long over all; if you submit this now then maybe
we'll
> want to change that later?

I spoke with Jeff about this, and based on feedback from users we can change
this.

[prasadv, 2014-07-11 23:21:09]

The CQ bit was checked by prasadv@chromium.org

[commit-bot: I haz the power, 2014-07-11 23:23:09]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/prasadv@chromium.org/388623002/20001

[commit-bot: I haz the power, 2014-07-11 23:37:21]

Change committed as 282735

[Andrew Hayden (chromium.org), 2014-07-15 10:24:21]

https://codereview.chromium.org/388623002/diff/20001/tools/bisect-perf-regres...
File tools/bisect-perf-regression.py (right):

https://codereview.chromium.org/388623002/diff/20001/tools/bisect-perf-regres...
tools/bisect-perf-regression.py:235: ===== THANK YOU FOR CHOOSING BISECT
AIRLINES =====
For posterity, this made my perf-sheriffing day.