rebaseline_server: multithreaded loading/diffing of images

rebaseline_server: multithreaded loading/diffing of images

BUG=skia:2414
NOTRY=True

Committed: http://code.google.com/p/skia/source/detail?r=14184

[epoger, 2014-04-11 21:11:34]

Ready for review at patchset 1.

This reduces the time it takes to launch rebaseline_server by about 60% in my
tests.

[rmistry, 2014-04-14 17:59:06]

LGTM

https://codereview.chromium.org/235923002/diff/1/gm/rebaseline_server/imagedi...
File gm/rebaseline_server/imagediffdb.py (right):

https://codereview.chromium.org/235923002/diff/1/gm/rebaseline_server/imagedi...
gm/rebaseline_server/imagediffdb.py:350: # and just waste a little time in the
process. Nothing will break.
Can we use semaphores/locks here? Based on your tests how many
global_file_collisions show up?

Also, you said it runs about 60% faster, does that mean it now takes about 10-15
mins during startup (with gpu cpu comparison)?

[epoger, 2014-04-14 18:10:51]

Thanks, Ravi!

https://codereview.chromium.org/235923002/diff/1/gm/rebaseline_server/imagedi...
File gm/rebaseline_server/imagediffdb.py (right):

https://codereview.chromium.org/235923002/diff/1/gm/rebaseline_server/imagedi...
gm/rebaseline_server/imagediffdb.py:350: # and just waste a little time in the
process. Nothing will break.
On 2014/04/14 17:59:06, rmistry wrote:
> Can we use semaphores/locks here? Based on your tests how many
> global_file_collisions show up?

In my tests, I didn't see any global_file_collisions yet.  Thus, I concluded we
were better off not incurring the cost of an additional lock.  (Given that the
downside is not disastrous, just takes some extra time.)

> Also, you said it runs about 60% faster, does that mean it now takes about
10-15
> mins during startup (with gpu cpu comparison)?

Well..... no.  I haven't tried with the gpu/cpu comparison yet.

This is the command I was running, on my Ubiquity instance:

$ rm -rf gm/rebaseline_server/static/generated-images/ &&
gm/rebaseline_server/server.py --export --actuals-repo ""

(Clear out disk cache to force download of actual/expected images for all with
differing checksums, but don't update the actuals checksums from skia-autogen so
we have consistent results.)

It consistenly took ~139 seconds before this change, and ~58 seconds after.

I'm sure there is plenty more speedup to be found.  I will keep looking...

[epoger, 2014-04-14 18:12:59]

The CQ bit was checked by epoger@google.com

[commit-bot: I haz the power, 2014-04-14 18:13:15]

CQ is trying da patch. Follow status at
https://skia-tree-status.appspot.com/cq/epoger@google.com/235923002/1

[epoger, 2014-04-14 18:14:56]

The CQ bit was unchecked by epoger@google.com

[epoger, 2014-04-14 18:15:11]

The CQ bit was checked by epoger@google.com

[commit-bot: I haz the power, 2014-04-14 18:15:19]

CQ is trying da patch. Follow status at
https://skia-tree-status.appspot.com/cq/epoger@google.com/235923002/1

[commit-bot: I haz the power, 2014-04-14 18:15:32]

Change committed as 14184

[epoger, 2014-04-15 18:49:53]

A revert of this CL has been created in
https://codereview.chromium.org/239623002/ by epoger@google.com.

The reason for reverting is: Caused
https://code.google.com/p/skia/issues/detail?id=2423 ('"Deadline exceeded" error
when connecting to live rebaseline_server').  Reverting until I can figure it
out..