add and enable validator tests for x86

buildbot/buildbot_spec2k.sh

Index: buildbot/buildbot_spec2k.sh
diff --git a/buildbot/buildbot_spec2k.sh b/buildbot/buildbot_spec2k.sh
index 2e903f9d9045863b6c8463ba4658c50ff501639e..db4db0276f78b6105bb415d3ee587d803f6a93c4 100755
--- a/buildbot/buildbot_spec2k.sh
+++ b/buildbot/buildbot_spec2k.sh
@@ -11,19 +11,19 @@ set -o errexit
 # SCRIPT CONFIG
 ######################################################################
 
-CLOBBER=${CLOBBER:-yes}
-SCONS_TRUSTED="./scons --mode=opt-host -j8"
-SCONS_NACL="./scons --mode=opt-host,nacl -j8"
-SPEC_HARNESS=${SPEC_HARNESS:-${HOME}/cpu2000-redhat64-ia32}/
+readonly CLOBBER=${CLOBBER:-yes}
+readonly SCONS_TRUSTED="./scons --mode=opt-host -j8"
+readonly SCONS_NACL="./scons --mode=opt-host,nacl -j8"
+readonly SPEC_HARNESS=${SPEC_HARNESS:-${HOME}/cpu2000-redhat64-ia32}/
 
-TRYBOT_TESTS="176.gcc 179.art 181.mcf 197.parser 252.eon 254.gap"
-TRYBOT_TRANSLATOR_TESTS="176.gcc"
-TRYBOT_X86_64_ZERO_BASED_SANDBOX_TESTS="176.gcc"
+readonly TRYBOT_TESTS="176.gcc 179.art 181.mcf 197.parser 252.eon 254.gap"
+readonly TRYBOT_TRANSLATOR_TESTS="176.gcc"
+readonly TRYBOT_X86_64_ZERO_BASED_SANDBOX_TESTS="176.gcc"
 
 readonly BUILDBOT_PNACL="buildbot/buildbot_pnacl.sh"
 readonly UP_DOWN_LOAD="buildbot/file_up_down_load.sh"
 
-SPEC_BASE="tests/spec2k"
+readonly SPEC_BASE="tests/spec2k"
 readonly ARCHIVE_NAME=$(${SPEC_BASE}/run_all.sh GetTestArchiveName)
 
 readonly NAME_ARM_TRY_UPLOAD=$(${BUILDBOT_PNACL} NAME_ARM_TRY_UPLOAD)
@@ -31,8 +31,12 @@ readonly NAME_ARM_TRY_DOWNLOAD=$(${BUILDBOT_PNACL} NAME_ARM_TRY_DOWNLOAD)
 readonly NAME_ARM_UPLOAD=$(${BUILDBOT_PNACL} NAME_ARM_UPLOAD)
 readonly NAME_ARM_DOWNLOAD=$(${BUILDBOT_PNACL} NAME_ARM_DOWNLOAD)
 
+readonly QEMU_TOOL="$(pwd)/toolchain/linux_arm-trusted/run_under_qemu_arm"
+
+readonly CANNED_NEXE_REV=1001
+
 # If true, terminate script when first error is encountered.
-FAIL_FAST=${FAIL_FAST:-false}
+readonly FAIL_FAST=${FAIL_FAST:-false}
 RETCODE=0
 
 # Print the number of tests being run for the buildbot status output
@@ -151,6 +155,69 @@ download-test-binaries() {
   popd
 }
 
+download-validator-test-nexes() {
+  local arch="$1"
+  echo "@@@BUILD_STEP validator test download@@@"
+  ${UP_DOWN_LOAD} DownloadArchivedNexes ${CANNED_NEXE_REV} \

[jvoung (off chromium), 2012/11/15 19:10:48]

Did you want to reuse tools/canned_nexe_tool.sh instead?

+      "${arch}_giant" giant_nexe.tar.bz2
+  # This generates "CannedNexes/" in the current directory
+ rm -rf CannedNexes

[jvoung (off chromium), 2012/11/15 19:10:48]

indentation

[robertm, 2012/11/15 19:33:31]

Done.

+ tar jxf  giant_nexe.tar.bz2
+}
+
+get-validator() {
+  local arch="$1"
+  if [[ ${arch} == "x86-32" ]] ; then
+      echo "$(pwd)/scons-out/opt-linux-x86-32/staging/ncval"

[jvoung (off chromium), 2012/11/15 19:14:19]

Also, would the dfa-validator devs be interested in using this?  I think they
had some scripts to run benchmarks offline, but not actually get them benchmarks
tracked on the bots.

[robertm, 2012/11/15 19:33:31]

absolutely, will send email to khim about it

On 2012/11/15 19:14:19, jvoung (chromium) wrote:

+  elif [[ ${arch} == "x86-64" ]] ; then
+      echo "$(pwd)/scons-out/opt-linux-x86-64/staging/ncval"
+  elif [[ ${arch} == "arm" ]] ; then
+      echo "$(pwd)/scons-out/opt-linux-arm/staging/arm-ncval-core"
+  else
+      echo "ERROR: unknown arch"
+  fi
+}
+
+LogTimeHelper() {
+  # This format is recognized by the buildbot system
+  echo "RESULT $1_$2: $3= $(bc) secs"

[jvoung (off chromium), 2012/11/15 19:10:48]

What does the invocation of bc do here?
Oh... I see from below.

Why don't you compute the value within LogTimedRun, then just pass the value
into LogTimeHelper instead of splitting the calculation up between the two
functions?

[robertm, 2012/11/15 19:33:31]

I eliminated the mess with the temp files.
so the output of /usr/bin/time is piped straight to
LogTimeHelper that means that LogTimedRun does not
see the output of "time".

On 2012/11/15 19:10:48, jvoung (chromium) wrote:

+}
+
+LogTimedRun() {
+  local graph=$1
+  local benchmark=$2
+  local variant=$3
+  shift 3
+  # S: system mode CPU-seconds used by the process
+  # U: user mode CPU-seconds  used by the process
+  # We add a plus sign inbetween so that we can pipe the output to "bc"
+  /usr/bin/time -f "%U + %S" \
+      --output >(LogTimeHelper ${graph} ${benchmark} ${variant}) \

[jvoung (off chromium), 2012/11/15 21:00:43]

Does --output need an argument?

[robertm, 2012/11/15 21:12:37]

added a comment

On 2012/11/15 21:00:43, jvoung (chromium) wrote:

+      "$@"
+}
+
+measure-validator-speed() {

[jvoung (off chromium), 2012/11/15 21:00:43]

Perhaps add a cross reference to the canned-nexe tool to indicate how to
regenerate the nexes if validation rules change.

[robertm, 2012/11/15 21:12:37]

I added a comment before:
readonly CANNED_NEXE_REV=1001

On 2012/11/15 21:00:43, jvoung (chromium) wrote:

+  local arch="$1"
+  local validator=$(get-validator ${arch})
+  echo "@@@BUILD_STEP validator speed test@@@"

[jvoung (off chromium), 2012/11/15 19:10:48]

maybe print the ${arch} too inside the BUILD_STEP

[robertm, 2012/11/15 19:33:31]

Done.

+  if [[ ! -e ${validator} ]] ; then
+    echo "ERROR: missing validator executable: ${validator}"
+    handle-error
+    return
+  fi
+
+  if [[ ${arch} == "arm" && $(uname -p) != arm* ]] ; then
+    validator="${QEMU_TOOL} ${validator}"

[jvoung (off chromium), 2012/11/15 19:10:48]

Do we really want to run this in qemu, or can we do this on the hardware bot
only, as "pnacl-trybot-arm-hw" does?

Maybe add the step to pnacl-arm-hw too.

[robertm, 2012/11/15 19:33:31]

The qemu part is useful when I manually tests this.
I definitely want to enable this for arm-hw 
but I have some feeling it might break or could be flake.
So, I first want to get it in for x86 and make sure it works
there which is more predictable.


On 2012/11/15 19:10:48, jvoung (chromium) wrote:

+  fi
+
+  for nexe in CannedNexes/* ; do
+    ls --size --block-size=1 ${nexe}
+    LogTimedRun  "validationtime" $(basename ${nexe}) "canned" \
+        ${validator} ${nexe}
+    echo "next"

[jvoung (off chromium), 2012/11/15 19:10:48]

why not echo "validating ${nexe}" at the beginning, instead of echo'ing "next"

[robertm, 2012/11/15 19:33:31]

that was debugging leftovers - remvoed

+  done
+}
+
 ######################################################################
 # NOTE: trybots only runs a subset of the the spec2k tests
 # TODO: elminate this long running bot in favor per arch sharded bots
@@ -246,6 +313,7 @@ pnacl-x8664() {
   build-tests "${setups}" all 1 3
   run-tests "${setups}" all 1 3
   pnacl-x86-64-zero-based-sandbox
+  measure-validator-speed x86-64
 }
 
 pnacl-x8632() {
@@ -257,6 +325,7 @@ pnacl-x8632() {
                 SetupPnaclTranslatorX8632Opt"
   build-tests "${setups}" all 1 3
   run-tests "${setups}" all 1 3
+  measure-validator-speed x86-32
 }
 
 nacl-x8632() {
@@ -266,6 +335,7 @@ nacl-x8632() {
                 SetupNaclX8632Opt"
   build-tests "${setups}" all 1 3
   run-tests "${setups}" all 1 3
+  measure-validator-speed x86-32
 }
 
 nacl-x8664() {
@@ -275,6 +345,7 @@ nacl-x8664() {
                 SetupNaclX8664Opt"
   build-tests "${setups}" all 1 3
   run-tests "${setups}" all 1 3
+  measure-validator-speed x86-64
 }