Create a self test for skimage.

tools/tests/skimage/self_test.py

+#!/usr/bin/env python

[scroggo, 2013/05/09 15:50:43]

I noticed that gm's self test has a note that it should be written in python, so
this is written in python.

[epoger, 2013/05/09 16:35:50]

Yes, good idea to do it that way in the first place. :-)

Ideally, I think it would be good to (eventually) call this test from
https://code.google.com/p/skia/source/browse/trunk/tools/tests/run.sh , so that
developers can easily run self-tests for *everything* in the tools directory.

+# Copyright (c) 2013 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+# Self-test for skimage.
+
+import os
+import subprocess
+
+# Use the directory of this file as the out directory

[epoger, 2013/05/09 16:35:50]

I think it would be better to write test files into a separate directory, so
that any given directory includes EITHER svn-controlled source code OR
disposable test files, but not BOTH.  Like how
https://code.google.com/p/skia/source/browse/trunk/tools/tests/run.sh writes
into trunk/tools/tests/skdiff and trunk/tools/tests/benchgraphs

[scroggo, 2013/05/09 19:05:59]

Done.

+out_dir = os.path.dirname(__file__)
+out_dir = os.path.abspath(out_dir)
+
+# Move up to trunk
+old_working_dir = os.getcwd()
+os.chdir(out_dir)
+os.chdir('../../../')
+
+# Find the binary
+skimage_binary = None
+xcode_path = "xcodebuild/Debug/skimage"
+gcc_path = "out/Debug/skimage"

[scroggo, 2013/05/09 15:50:43]

It looks like gm's self test assumes you have already built gm. I've followed
that convention, although I ran into a funny situation on my linux machine where
I had already built skimage, but not since the program had been updated.

I was tempted to attempt to build, but it occurred to me that building might
require different commands on different machines, and some may be set up for
ninja/some other build process.

[epoger, 2013/05/09 16:35:50]

Yeah, it's a tough call, because individual developers (or buildbots) may want
to build the code in different ways.  On the other hand, I have seen people get
confused because they made a code change and the self-test results did not
change, because they forgot to rebuild the code.

I think the way you have it is good, at least for now.

+
+if (os.path.exists(gcc_path)):

[epoger, 2013/05/09 16:35:50]

Good idea to go looking for the binary in different places on different
platforms!

Here's another approach which would be more flexible, and would allow for either
Debug or Release builds...

# Ordered list of paths where we look for the skimage binary.
# Paths that are earlier in the list are preferred.
POSSIBLE_BINARY_PATHS = [
  'xcodebuild/Debug/skimage',
  'xcodebuild/Release/skimage',
  'out/Debug/skimage',
  'out/Release/skimage',
]

skimage_binary = PickFirstFileThatExists(POSSIBLE_BINARY_PATHS)

... and PickFirstFileThatExists() iterates over the list, returning the first
one that refers to an existing file.

A downside of this would be: if the user had built both Debug and Release
versions, he wouldn't know which one was run.

[scroggo, 2013/05/09 19:05:59]

I have added a function to find the binary. I also print out which binary is
used to alleviate any confusion.

+    skimage_binary = gcc_path
+elif (os.path.exists(xcode_path)):
+    skimage_binary = xcode_path
+
+resultsFile = None
+
+if (skimage_binary != None):
+    try:
+        # Run skimage twice, first to create an expectations file, and then
+        # comparing to it.
+        # Both commands will run the binary, reading from resources.
+        cmd_line = [skimage_binary]
+        cmd_line.extend(["-r", "resources/CMYK.jpg", "resources/plane.png"])

[epoger, 2013/05/09 16:35:50]

Alternatively, you could assemble these paths as absolute paths, and thus
eliminate the need for chdir()

[scroggo, 2013/05/09 19:05:59]

Done.

+        # Create the expectations file
+        create_expectations_cmd = cmd_line + ["--createExpectationsPath", out_dir]
+        subprocess.check_call(create_expectations_cmd)

[scroggo, 2013/05/09 15:57:29]

There are some commands for calling programs more safely in the buildbot
directory. Should those be available for me to use here?

[epoger, 2013/05/09 16:35:50]

Unfortunately, no.  Most developers will have checked out trunk/ but not
buildbot/ .

The buildbots definitely check out *both*, though, so if there are scripts we
want to be able to use from either trunk/ or buildbot/, we should consider
moving them into trunk/ .

[scroggo, 2013/05/09 19:05:59]

That sounds reasonable to me. Probably outside of the scope of this CL, but
where within trunk/ would they live? Should there be a scripts directory in
trunk?

[epoger, 2013/05/09 19:19:39]

Definitely outside the scope of this CL.  Please coordinate with Ravi and/or
Eric as far as the directory goes.

+        # Now read from the expectations file
+        resultsFile = os.path.join(out_dir, "results.json")
+        check_expectations_cmd = cmd_line + ["--readExpectationsPath", resultsFile]

[epoger, 2013/05/09 16:35:50]

Great!  This is definitely a good start... but, if I understand correctly, this
proves that *any given revision* of skimage reads and writes consistent
expectations.  It does NOT prove that the results are the same as they were
before any local changes.

Please consider adding tests that will exercise that, in the future, if it's not
terribly hard to do.  It can be tricky, because the typical way to do that is to
check expected results into SVN, and then you have to worry about results that
differ by platform, rebaselining the expected results when appropriate, etc.

[scroggo, 2013/05/09 19:05:59]

Definitely the long term plan is to compare against previous revisions, but that
will be trickier. Is there a way to merge gm_expectations files from different
platforms or will I need to keep separate versions?

[epoger, 2013/05/09 19:19:39]

To some extent, we may be able to have expectations that work across platforms. 
The bitmap hash values in GM self-test work cross-platform, for example.

But for encoders/decoders that yield different pixel values on different
platforms, I guess we'll need separate baselines.  Blah.

Yes, it's for another day. :-)

+        subprocess.check_call(check_expectations_cmd)
+        print "Self tests succeeded!"
+    except subprocess.CalledProcessError as e:
+        print skimage_binary + " failed with error code " + str(e.returncode)
+        print "Self tests failed!"

[epoger, 2013/05/09 16:35:50]

It would be better if the process exited with a nonzero return value when the
self-tests fail, for uses like this:

run_self_test_1 && run_self_test_2 && run_self_test_3 && echo "it all worked!"

[scroggo, 2013/05/09 19:05:59]

Done.

+else:
+    print "Could not find skimage binary!"
+    print "Did you forget to build tools project?"
+    print "Self tests failed"
+
+# Delete the results file, which is no longer needed.
+if (resultsFile != None and os.path.exists(resultsFile)):
+    os.remove(resultsFile)
+
+# Return to the original working directory.
+os.chdir(old_working_dir)