skimage self-tests: confirm correctness of bitmap hash values

skimage self-tests: confirm correctness of bitmap hash values

R=scroggo@google.com

Committed: https://code.google.com/p/skia/source/detail?r=9591

[scroggo, 2013-06-13 16:17:36]

https://codereview.chromium.org/16866010/diff/2001/tools/tests/skimage/input/...
File tools/tests/skimage/input/README (right):

https://codereview.chromium.org/16866010/diff/2001/tools/tests/skimage/input/...
tools/tests/skimage/input/README:1: Resources needed for skimage self-tests go
here.
Need these resources be different from the top level resources?

https://codereview.chromium.org/16866010/diff/2001/tools/tests/skimage_self_t...
File tools/tests/skimage_self_test.py (right):

https://codereview.chromium.org/16866010/diff/2001/tools/tests/skimage_self_t...
tools/tests/skimage_self_test.py:53: "json-selftest", "results.json")
Does json-selftest exist?

[epoger, 2013-06-13 16:24:47]

https://codereview.chromium.org/16866010/diff/2001/tools/tests/skimage/input/...
File tools/tests/skimage/input/README (right):

https://codereview.chromium.org/16866010/diff/2001/tools/tests/skimage/input/...
tools/tests/skimage/input/README:1: Resources needed for skimage self-tests go
here.
On 2013/06/13 16:17:37, scroggo wrote:
> Need these resources be different from the top level resources?

A fair point.  Initially, I hadn't even noticed that this used images pulled
from trunk/resources.

Here's why I am doing this at all: I want to start using "skimage
--createExpectationsPath" on the bots, to convert our expectations from images
to JSON format.  (See Step 1b in
https://goto.google.com/ChecksumTransitionDetail )

Before I do that, I want to make sure that the skimage tool is correctly
generating bitmap hashes from image files.

Currently, the skimage self-test confirms that the bitmap hashes are generated
CONSISTENTLY, but not that they are generated CORRECTLY.  And having a
particular set of input images (not shared with other tests, which may need
their own images) for this test is key.

Tell you what... I'm going to add more to this CL in a little bit... let's see
if you think it looks good at that point.

https://codereview.chromium.org/16866010/diff/2001/tools/tests/skimage_self_t...
File tools/tests/skimage_self_test.py (right):

https://codereview.chromium.org/16866010/diff/2001/tools/tests/skimage_self_t...
tools/tests/skimage_self_test.py:53: "json-selftest", "results.json")
On 2013/06/13 16:17:37, scroggo wrote:
> Does json-selftest exist?

Yes, just doesn't show up in the diff.

[epoger, 2013-06-13 17:09:20]

The codereview site is choking on this because of the PNG files in the CL (see
https://code.google.com/p/chromium/issues/detail?id=169600 ), so here's some raw
diff data... 

Leon, please take a look and let me know what you think.


$ svn diff
M       tools/tests/skimage_self_test.py
A       tools/tests/skimage/output-actual
A       tools/tests/skimage/output-actual/README
A       tools/tests/skimage/output-actual/create-expectations
D       tools/tests/skimage/README
A       tools/tests/skimage/input
A       tools/tests/skimage/input/README
A       tools/tests/skimage/input/images-with-known-hashes
A      
tools/tests/skimage/input/images-with-known-hashes/1209453360120438698.png
A      
tools/tests/skimage/input/images-with-known-hashes/12927999507540085554.png
A       tools/tests/skimage/output-expected
A       tools/tests/skimage/output-expected/create-expectations
A      
tools/tests/skimage/output-expected/create-expectations/expectations.json
A       tools/tests/skimage/output-expected/README



$ gcl changes
--- Changelist move:
M      tools/tests/skimage_self_test.py
A      tools/tests/skimage/output-actual
A      tools/tests/skimage/output-actual/README
D      tools/tests/skimage/README
A      tools/tests/skimage/input
A      tools/tests/skimage/input/README
A      tools/tests/skimage/output-expected
A      tools/tests/skimage/output-expected/README
A      tools/tests/skimage/output-actual/create-expectations
A      tools/tests/skimage/input/images-with-known-hashes
A     
tools/tests/skimage/input/images-with-known-hashes/1209453360120438698.png
A     
tools/tests/skimage/input/images-with-known-hashes/12927999507540085554.png
A      tools/tests/skimage/output-expected/create-expectations
A      tools/tests/skimage/output-expected/create-expectations/expectations.json


$ svn diff
Index: tools/tests/skimage_self_test.py
===================================================================
--- tools/tests/skimage_self_test.py	(revision 9583)
+++ tools/tests/skimage_self_test.py	(working copy)
@@ -5,6 +5,7 @@
 
 # Self-test for skimage.
 
+import filecmp
 import os
 import subprocess
 import sys
@@ -30,6 +31,13 @@
             return binary_full_path
     raise BinaryNotFoundException
 
+# Quit early if two files have different content.
+def DieIfFilesMismatch(expected, actual):
+    if not filecmp.cmp(expected, actual):
+        print 'Error: file mismatch! expected=%s , actual=%s' % (
+            expected, actual)
+        exit(1)
+
 def main():
     # Use the directory of this file as the out directory
     file_dir = os.path.abspath(os.path.dirname(__file__))
@@ -40,24 +48,29 @@
     skimage_binary = PickBinaryPath(trunk_dir)
     print "Running " + skimage_binary
 
-    # Run skimage twice, first to create an expectations file, and then
-    # comparing to it.
+    # Generate an expectations file from known images.
+    images_dir = os.path.join(file_dir, "skimage", "input",
+                              "images-with-known-hashes")
+    expectations_path = os.path.join(file_dir, "skimage", "output-actual",
+                                     "create-expectations",
"expectations.json")
+    subprocess.check_call([skimage_binary, "--readPath", images_dir,
+                           "--createExpectationsPath", expectations_path])
 
-    # Both commands will run the binary, reading from resources.
-    cmd_line = [skimage_binary]
-    resources_dir = os.path.join(trunk_dir, 'resources')
-    cmd_line.extend(["-r", resources_dir])
+    # Make sure the expectations file was generated correctly.
+    golden_expectations = os.path.join(file_dir, "skimage", "output-expected",
+                                       "create-expectations",
+                                       "expectations.json")
+    DieIfFilesMismatch(expected=golden_expectations, actual=expectations_path)
 
-    # Create the expectations file
-    results_file = os.path.join(file_dir, "skimage", "self_test_results.json")
-    create_expectations_cmd = cmd_line + ["--createExpectationsPath",
-                                          results_file]
-    subprocess.check_call(create_expectations_cmd)
+    # Tell skimage to read back the expectations file it just wrote, and
+    # confirm that the images in images_dir match it.
+    subprocess.check_call([skimage_binary, "--readPath", images_dir,
+                           "--readExpectationsPath", expectations_path])
 
-    # Now read from the expectations file
-    check_expectations_cmd = cmd_line + ["--readExpectationsPath",
-                                         results_file]
-    subprocess.check_call(check_expectations_cmd)
+    # TODO(scroggo): Add a test that compares expectations and image files that
+    # are known to NOT match, and make sure it returns an error.
+
+    # Done with all tests.
     print "Self tests succeeded!"
 
 if __name__ == "__main__":
Index: tools/tests/skimage/output-actual/README
===================================================================
--- tools/tests/skimage/output-actual/README	(revision 0)
+++ tools/tests/skimage/output-actual/README	(revision 0)
@@ -0,0 +1 @@
+Actual output of skimage self-tests goes here.
Index: tools/tests/skimage/README
===================================================================
--- tools/tests/skimage/README	(revision 9583)
+++ tools/tests/skimage/README	(working copy)
@@ -1 +0,0 @@
-This directory is for the output of skimage_self_test.
Index:
tools/tests/skimage/input/images-with-known-hashes/1209453360120438698.png
===================================================================
Cannot display: file marked as a binary type.
svn:mime-type = application/octet-stream

Property changes on:
tools/tests/skimage/input/images-with-known-hashes/1209453360120438698.png
___________________________________________________________________
Added: svn:mime-type
   + application/octet-stream

Index:
tools/tests/skimage/input/images-with-known-hashes/12927999507540085554.png
===================================================================
Cannot display: file marked as a binary type.
svn:mime-type = application/octet-stream

Property changes on:
tools/tests/skimage/input/images-with-known-hashes/12927999507540085554.png
___________________________________________________________________
Added: svn:mime-type
   + application/octet-stream

Index: tools/tests/skimage/input/README
===================================================================
--- tools/tests/skimage/input/README	(revision 0)
+++ tools/tests/skimage/input/README	(revision 0)
@@ -0,0 +1 @@
+Resources needed for skimage self-tests go here.
Index: tools/tests/skimage/output-expected/README
===================================================================
--- tools/tests/skimage/output-expected/README	(revision 0)
+++ tools/tests/skimage/output-expected/README	(revision 0)
@@ -0,0 +1 @@
+Expected output of skimage self-tests goes here.
Index: tools/tests/skimage/output-expected/create-expectations/expectations.json
===================================================================
---
tools/tests/skimage/output-expected/create-expectations/expectations.json	(revision
0)
+++
tools/tests/skimage/output-expected/create-expectations/expectations.json	(revision
0)
@@ -0,0 +1,22 @@
+{
+   "actual-results" : {
+      "failed" : null,
+      "failure-ignored" : null,
+      "no-comparison" : null,
+      "succeeded" : null
+   },
+   "expected-results" : {
+      "1209453360120438698.png" : {
+         "allowed-digests" : [
+            [ "bitmap-64bitMD5", 1209453360120438698 ]
+         ],
+         "ignore-failure" : false
+      },
+      "12927999507540085554.png" : {
+         "allowed-digests" : [
+            [ "bitmap-64bitMD5", 12927999507540085554 ]
+         ],
+         "ignore-failure" : false
+      }
+   }
+}

[scroggo, 2013-06-13 17:45:53]

lgtm

[epoger, 2013-06-13 19:12:10]

Committed patchset #3 manually as r9591 (presubmit successful).