rebaseline_server: add --compare-configs option

rebaseline_server: add --compare-configs option

This allows us to compare GMs between configs across all builders, so we can see the largest deviations between raster and GPU renderings.

BUG=skia:1919
NOTREECHECKS=True
NOTRY=True

Committed: http://code.google.com/p/skia/source/detail?r=13991

[epoger, 2014-03-27 21:31:06]

Ready for review at patchset 1!

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/compare...
File gm/rebaseline_server/compare_configs_test.py (right):

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/compare...
gm/rebaseline_server/compare_configs_test.py:4: Copyright 2014 Google Inc.
rietveld is showing diffs against compare_to_expectations.py, a similar file.

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/compare...
File gm/rebaseline_server/compare_to_expectations.py (right):

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/compare...
gm/rebaseline_server/compare_to_expectations.py:55: class
ExpectationComparisons(results.BaseComparisons):
I moved the common code (used by both ExpectationComparisons and
ConfigComparisons) into a shared BaseComparisons class in results.py.

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/server.py
File gm/rebaseline_server/server.py (right):

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/server....
gm/rebaseline_server/server.py:101: CONFIG_PAIRS_TO_COMPARE = [('8888', 'gpu')]
Someday the pairs will be configurable from the command line, but for now, I
just picked a reasonable pair to compare.

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/server....
gm/rebaseline_server/server.py:161: """Creates an index file linking to all
results available from this server.
I used to have an index.html file checked in (you'll see it deleted in this CL),
but now that the results may or may not include config comparisons, index.html
needs to be generated differently depending on which results are included.

[epoger, 2014-03-27 21:48:09]

The server is running at http://172.24.179.183:8888/ ; feel free to explore the
results.

It took 30 minutes to download/generate all the 8888-vs-gpu diffs.  I'm not yet
sure why it takes so long.  That's why it's turned off by default, for now. :-)

Also--the Housekeeper-PerCommit-Trybot run looks fine.  The error has nothing to
do with this change.

[rmistry, 2014-03-28 18:51:39]

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/compare...
File gm/rebaseline_server/compare_configs.py (right):

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/compare...
gm/rebaseline_server/compare_configs.py:50: by _ignore_builder().
Change to results._ignore_builder() ?

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/compare...
File gm/rebaseline_server/compare_configs_test.py (right):

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/compare...
gm/rebaseline_server/compare_configs_test.py:4: Copyright 2014 Google Inc.
On 2014/03/27 21:31:07, epoger wrote:
> rietveld is showing diffs against compare_to_expectations.py, a similar file.

Acknowledged.

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/results.py
File gm/rebaseline_server/results.py (right):

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/results...
gm/rebaseline_server/results.py:68: """ Base class for generating summary of
comparisons between two image sets.
Nit: Extra space before 'Base', I mention this because in other places there is
no space here.

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/results...
gm/rebaseline_server/results.py:87: """ Package the results of some/all tests as
a complete response_dict.
Nit: Extra space before 'Package'.

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/results...
gm/rebaseline_server/results.py:151: ('ASAN' in builder))
Should we create a global list of BUILDERS_TO_IGNORE and then loop over that
here?
Reason is it will be more noticable at the top and you can reference skia:2036
there.

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/server.py
File gm/rebaseline_server/server.py (right):

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/server....
gm/rebaseline_server/server.py:101: CONFIG_PAIRS_TO_COMPARE = [('8888', 'gpu')]
On 2014/03/27 21:31:07, epoger wrote:
> Someday the pairs will be configurable from the command line, but for now, I
> just picked a reasonable pair to compare.

Acknowledged.

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/server....
gm/rebaseline_server/server.py:161: """Creates an index file linking to all
results available from this server.
On 2014/03/27 21:31:07, epoger wrote:
> I used to have an index.html file checked in (you'll see it deleted in this
CL),
> but now that the results may or may not include config comparisons, index.html
> needs to be generated differently depending on which results are included.

Acknowledged.

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/server....
gm/rebaseline_server/server.py:196: summary_type, summary_type))
Instead of writing raw HTML what do you think about using a template and writing
only the parts you part about using django or something similar.
Write a todo/bug if you do not want to do this as part of this CL?

[epoger, 2014-03-28 19:25:54]

Thanks, Ravi.  Please see patchset 2.

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/compare...
File gm/rebaseline_server/compare_configs.py (right):

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/compare...
gm/rebaseline_server/compare_configs.py:50: by _ignore_builder().
On 2014/03/28 18:51:39, rmistry wrote:
> Change to results._ignore_builder() ?

Changed to BaseComparisons._ignore_builder()

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/results.py
File gm/rebaseline_server/results.py (right):

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/results...
gm/rebaseline_server/results.py:68: """ Base class for generating summary of
comparisons between two image sets.
On 2014/03/28 18:51:39, rmistry wrote:
> Nit: Extra space before 'Base', I mention this because in other places there
is
> no space here.

Fixed throughout CL.

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/results...
gm/rebaseline_server/results.py:87: """ Package the results of some/all tests as
a complete response_dict.
On 2014/03/28 18:51:39, rmistry wrote:
> Nit: Extra space before 'Package'.

Done.

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/results...
gm/rebaseline_server/results.py:151: ('ASAN' in builder))
On 2014/03/28 18:51:39, rmistry wrote:
> Should we create a global list of BUILDERS_TO_IGNORE and then loop over that
> here?
> Reason is it will be more noticable at the top and you can reference skia:2036
> there.

Took a stab at it, let me know what you think.

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/server.py
File gm/rebaseline_server/server.py (right):

https://codereview.chromium.org/215503002/diff/1/gm/rebaseline_server/server....
gm/rebaseline_server/server.py:196: summary_type, summary_type))
On 2014/03/28 18:51:39, rmistry wrote:
> Instead of writing raw HTML what do you think about using a template and
writing
> only the parts you part about using django or something similar.
> Write a todo/bug if you do not want to do this as part of this CL?

Created a TODO.

[rmistry, 2014-03-31 14:46:35]

LGTM

[epoger, 2014-03-31 14:56:30]

The CQ bit was checked by epoger@google.com

[commit-bot: I haz the power, 2014-03-31 14:56:37]

CQ is trying da patch. Follow status at
https://skia-tree-status.appspot.com/cq/epoger@google.com/215503002/40001

[epoger, 2014-03-31 15:17:12]

The CQ bit was unchecked by epoger@google.com

[epoger, 2014-03-31 15:17:27]

The CQ bit was checked by epoger@google.com

[commit-bot: I haz the power, 2014-03-31 15:17:28]

CQ is trying da patch. Follow status at
https://skia-tree-status.appspot.com/cq/epoger@google.com/215503002/40001

[commit-bot: I haz the power, 2014-03-31 15:18:01]

Change committed as 13991