Add new testing framework along with a few micro-tests.

util/do_tests.sh

+#!/bin/sh
+#

[Stefan Reinauer, 2010/11/23 19:05:30]

With other things living in util/ too and the tests becoming ubiquitous and
countless over times, maybe the tests (and do_tests.sh) should go to their own
tests/ or testing/ directory?

[dhendrix, 2010/11/24 02:48:14]

I tend to agree. It would certainly make it easier to do use wildcards to
specify which tests to copy to my target machines and run :-)

We brought up an idea for organization on the e-mail thread -- Perhaps we can
have a directories for "default" tests (safe, easy to run, maybe a good
candidate for a Makefile hook) and "unsafe" tests which would require special
attention from the user.

That would make it easier to glob a desired subset of tests and help casual
users avoid bricking their system.

+# Copyright (C) 2010 Google Inc.
+# Written by David Hendricks for Google Inc.
+#

[Stefan Reinauer, 2010/11/23 19:05:30]

It might also be nice to have a Makefile hook to run the tests. Not sure if this
makes sense with the environment variables required.

+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+#
+# This is a simple test harness which contains setup and cleanup code common to 
+# all tests. The syntax for this script is:
+# do_tests.sh --options <test1> <test2> ... <testN>
+#
+# Arguments which are files are assumed to be tests which are run after common
+# setup routines and before common cleanup routines.
+#
+
+export EXIT_SUCCESS=0
+export EXIT_FAILURE=1
+export DEBUG=1
+
+show_help()
+{
+        echo " \
+        Usage: 
+        do_test.sh [OPTION] test1.sh test2.sh ... testN.sh

[Stefan Reinauer, 2010/11/23 19:05:30]

Given that no tests are invoked outside of do_test.sh (is this true?) I think we
could drop the .sh suffix.

[hailfinger, 2010/11/23 20:05:34]

The .sh suffix makes it obvious that we're dealing with shell scripts, and you
can actually use
find -name "*.sh"

[Stefan Reinauer, 2010/11/23 21:14:14]

Not sure if there is a benefit in "knowing" these are shell scripts unless you
want to change them, in which case you will sure enough find out by opening them
in your editor.
The find does something useful (though not intuitive) until some other utility
ends up in util with a .sh ending. Maybe we should rather call the tests
test_<something>

[dhendrix, 2010/11/24 02:48:14]

I like the idea of placing the tests in a subdir... that addresses all concerns
and doesn't require a naming convention.

+
+        Environment variables:
+        FLASHROM: Path to the Flashrom binary to test
+        FLASHROM_PARAM: Parameters to pass in
+
+        OPTIONS
+        -h or --help
+            Display this message.
+
+        Arguments with one or two leading dashes are processed as options.
+        Arguments which are filenames are processed as test cases. Names of test
+        cases may not contain spaces.
+        "
+
+        exit $EXIT_SUCCESS
+}
+
+msg_dbg() {
+        if [ ${DEBUG} -eq 1 ]; then
+                echo "$@"
+        fi
+}
+
+# Parse command-line
+OPTIONS=""
+TESTS=""
+for ARG in $@; do
+        if echo "${ARG}" | grep "^-.*$" >/dev/null
+        then
+                OPTIONS=${OPTIONS}" "${ARG}
+        elif [ -f ${ARG} ]; then
+                TESTS=${TESTS}" "${ARG}
+        fi
+done
+
+for ARG in ${OPTIONS}; do
+        case ${ARG} in 
+                -h|--help) 
+                        show_help;
+                        shift;; 
+        esac; 
+done
+
+#
+# Setup Routine
+#
+
+# The copy of flashrom to test. If unset, we'll assume the user wants to test
+# a newly built flashrom binary in the parent directory (this script should
+# reside in flashrom/util).
+if [ -z "${FLASHROM}" ] ; then
+        FLASHROM="../flashrom"

[Stefan Reinauer, 2010/11/23 19:05:30]

It would be nice to be able to run the tests from the top level directory, too.
This could be autodetected, and a variable TESTDIR could point to the tests so
util/do_tests test1 test2 test3 would start the tests from util/ (or testing/ or
whatever)

[hailfinger, 2010/11/23 20:05:34]

Yes, I'll probably start the tests from the top level dir as well. It's just a
lot more convenient.

[dhendrix, 2010/11/24 02:48:14]

Done.

+fi
+echo "testing flashrom binary: ${FLASHROM}"
+
+OLDDIR=$(pwd)
+
+# test data location
+TMPDIR=$(mktemp -d -t flashrom_test.XXXXXXXXXX)
+export TMPDIR="${TMPDIR}"
+if [ "$?" != "0" ] ; then
+        echo "Could not create temporary directory"
+        exit ${EXIT_FAILURE}
+fi
+
+which flashrom > /dev/null
+#if [ "$?" != "0" ] ; then
+if [ ${?} -ne 0 ] ; then
+        echo "Please install a stable version of flashrom in your path."
+        echo "This will be used to compare the test flashrom binary and "
+        echo "restore your firmware image at the end of the test."
+        exit ${EXIT_FAILURE}
+fi
+
+# Copy the test case files and flashrom to temporary directory.
+cp "${FLASHROM}" "${TMPDIR}/"
+for TEST in ${TESTS}; do
+        cp ${TEST} "${TMPDIR}/"
+done
+
+cd "${TMPDIR}"
+echo "Running test in ${TMPDIR}"
+
+# Make a backup
+echo "Reading BIOS image"
+BACKUP="backup.bin"
+flashrom ${FLASHROM_PARAM} -r "$BACKUP" > /dev/null
+echo "Original image saved as ${BACKUP}"
+export BACKUP=${BACKUP}
+
+#
+# Execute test cases
+#
+rc=${EXIT_SUCCESS}
+msg_dbg "Test cases: ${TESTS}"
+for TEST in ${TESTS}; do
+        msg_dbg "Running test: \"${TEST}\""
+        ./${TEST}
+        if [ ${?} -ne ${EXIT_SUCCESS} ]; then
+                rc=${EXIT_FAILURE}
+                break
+        fi
+done
+
+#
+# Cleanup Routine
+#
+if [ ${rc} -ne ${EXIT_SUCCESS} ] ; then
+        echo "Result: FAILED"
+else
+        echo "Result: PASSED"
+fi
+
+echo "restoring original image using system's flashrom"
+flashrom ${FLASHROM_PARAM} -w "$BACKUP" #FIXME: add this back in

[hailfinger, 2010/11/23 20:05:34]

What should be added back in?

[dhendrix, 2010/11/24 02:48:14]

stale comment -- removed.

+echo "test files remain in ${TMPDIR}"
+cd "${OLDDIR}"
+exit ${rc}