Real world impact script: scalable manual rendering QA

Real world impact script: scalable manual rendering QA

Layout Tests can tell you whether rendering has changed, but it's often
hard to determine whether a subtle/controversial change is beneficial
without additional context.

This script aims to provide that context. It takes screenshots of 1000s
of sites, both before and after applying the patch being evaluated,
then diffs each pair of screenshots and sorts them by greatest
difference in rendering, in an HTML report. A human reviewer can then
manually skim through this to review the most impacted sites, rather
than having to browse random sites to see what their patch changed.

This is a script I wrote at the start of the year (but hadn't got round
to uploading a patch, as it used to depend on proprietary binaries;
I fixed that dependency by writing a replacement: crrev.com/67973005).

The script isn't yet smart enough to build content_shell itself, so you
run it in several steps:

1. Build content_shell in out/Release, without the controversial patch.
2. Run: real_world_impact.py before [num sites to test (default 1000)]
3. Apply the controversial patch, and rebuild content_shell in out/Release.
4. Run: real_world_impact.py after [num sites to test (default 1000)]
5. Run: real_world_impact.py compare [num sites to test (default 1000)]

In practice I use the following bash function to drive it:
http://pastebin.com/yJ7KAGdX

BUG=135823

[johnme, 2013-12-19 19:27:10]

There are many things that could be improved with this patch:
- Use proper chromium coding style
- Refactor out the embedded HTML(?)
- Use web-page-replay rather than wget --page-requisites
- Use a page-set that includes non-homepages, and less popular sites, rather
than just the first few Alexa homepages.
- Have the python script do the builds for you, rather than relying on an
external bash script (pastebin.com/yJ7KAGdX) to drive them.

On the other hand, I haven't got much time to spend on this right now, and
there's value in getting a form of this checked in so people can start using it,
then evolving it incrementally.

So, what would be reasonable minimum requirements for getting this checked in?

[pdr., 2014-01-06 23:27:32]

Wow, this is fantastic. I agree that we should get this in :)

I had difficulty getting this to run on OSX but maybe we can iterate on that.
Could you add a test of the basic functionality so the bots will keep us from
breaking this?

https://codereview.chromium.org/112423006/diff/1/tools/real_world_impact/real...
File tools/real_world_impact/real_world_impact.py (right):

https://codereview.chromium.org/112423006/diff/1/tools/real_world_impact/real...
tools/real_world_impact/real_world_impact.py:52: USAGE
Please list the command line options here

[skobes, 2014-03-07 21:43:33]

Hey John, how do you feel about submitting this?  It would be great to have it
in the tree so we can iterate on it, fix issues etc.

[pdr., 2014-03-13 22:23:08]

On 2014/03/07 21:43:33, skobes wrote:
> Hey John, how do you feel about submitting this?  It would be great to have it
> in the tree so we can iterate on it, fix issues etc.

Ping? Abarth would like to use this as well, and I'd like to fix a few bugs :)

[pdr., 2014-03-22 04:09:12]

I've uploaded this for review at https://codereview.chromium.org/209393002