New LLVM merging bots

1) Add a new mode to utman for downloading "upstream" repository, which contains both LLVM and LLVM-GCC.

2) Rename buildbot_pnacl2.sh to buildbot_spec2k.sh, along with some cleanup.

2) Setup two new builders "pnacl-merge-fast" and "pnacl-merge-slow" for testing LLVM merging.

pnacl-merge-fast:
  Perform merge, rebuild PNaCl, run a series of quick tests.
  If all the quick tests pass, commit the merge.
  This bot is not yet enabled, pending a merge to tip.

pnacl-merge-slow:
  Using the last committed merge result, run all tests, including browser tests and Spec2K.
  This keeps track of the total status of the merge.
Committed: http://src.chromium.org/viewvc/native_client?view=rev&revision=6239

[pdox, 2011-07-28 11:46:11]

PTAL

[jvoung - send to chromium..., 2011-07-28 23:48:54]

http://codereview.chromium.org/7524023/diff/6001/buildbot/buildbot_pnacl_merg...
File buildbot/buildbot_pnacl_merge.sh (right):

http://codereview.chromium.org/7524023/diff/6001/buildbot/buildbot_pnacl_merg...
buildbot/buildbot_pnacl_merge.sh:29: echo @@@BUILD_STEP clobber@@@
should this check if the clobber env var is set?

http://codereview.chromium.org/7524023/diff/6001/buildbot/buildbot_pnacl_merg...
buildbot/buildbot_pnacl_merge.sh:46: ${UTMAN} everything-translator
this leaves "sdk" around for spec testing right?

http://codereview.chromium.org/7524023/diff/6001/buildbot/buildbot_pnacl_merg...
buildbot/buildbot_pnacl_merge.sh:57: slow-bot() {
Comment on difference between slow-bot and fast-bot. Slow bot doesn't seem to do
any merging (doesn't run MERGE_TOOL or commit-and-push)? I.e., the difference is
more than just the speed / amount of testing which can be an important point.

http://codereview.chromium.org/7524023/diff/6001/buildbot/buildbot_pnacl_merg...
buildbot/buildbot_pnacl_merge.sh:89: ${MERGE_TOOL} auto
is there a change to merge tool to add an "auto" function?

Should that be included in the CL?

http://codereview.chromium.org/7524023/diff/6001/buildbot/buildbot_spec2k.sh
File buildbot/buildbot_spec2k.sh (right):

http://codereview.chromium.org/7524023/diff/6001/buildbot/buildbot_spec2k.sh#...
buildbot/buildbot_spec2k.sh:142: build-and-run-some SetupPnaclTranslatorX8632Opt
"${TRANSLATOR_TESTS}"
TRANSLATOR_TESTS -> TRYBOT_TRANSLATOR_TESTS

http://codereview.chromium.org/7524023/diff/6001/tools/llvm/utman.sh
File tools/llvm/utman.sh (right):

http://codereview.chromium.org/7524023/diff/6001/tools/llvm/utman.sh#newcode670
tools/llvm/utman.sh:670: hg-checkout ${REPO_UPSTREAM} ${TC_SRC_UPSTREAM}
${UPSTREAM_REV}
Just noticed, but should all the ${TC_SRC...} have quotes around them?

[pdox, 2011-07-29 00:05:55]

http://codereview.chromium.org/7524023/diff/6001/buildbot/buildbot_pnacl_merg...
File buildbot/buildbot_pnacl_merge.sh (right):

http://codereview.chromium.org/7524023/diff/6001/buildbot/buildbot_pnacl_merg...
buildbot/buildbot_pnacl_merge.sh:29: echo @@@BUILD_STEP clobber@@@
CLOBBER is probably not useful here.

http://codereview.chromium.org/7524023/diff/6001/buildbot/buildbot_pnacl_merg...
buildbot/buildbot_pnacl_merge.sh:46: ${UTMAN} everything-translator
yes, but that's not critical.

scons uses its own SDK anyway.
And the spec2k script rebuilds the SDK.

http://codereview.chromium.org/7524023/diff/6001/buildbot/buildbot_pnacl_merg...
buildbot/buildbot_pnacl_merge.sh:57: slow-bot() {
On 2011/07/28 23:48:54, jvoung wrote:
> Comment on difference between slow-bot and fast-bot. Slow bot doesn't seem to
do
> any merging (doesn't run MERGE_TOOL or commit-and-push)? I.e., the difference
is
> more than just the speed / amount of testing which can be an important point.

Done.

http://codereview.chromium.org/7524023/diff/6001/buildbot/buildbot_pnacl_merg...
buildbot/buildbot_pnacl_merge.sh:89: ${MERGE_TOOL} auto
yes, added. It is not quite ready, but 90% there. (another CL may be needed to
iron out the bugs)

http://codereview.chromium.org/7524023/diff/6001/buildbot/buildbot_spec2k.sh
File buildbot/buildbot_spec2k.sh (right):

http://codereview.chromium.org/7524023/diff/6001/buildbot/buildbot_spec2k.sh#...
buildbot/buildbot_spec2k.sh:142: build-and-run-some SetupPnaclTranslatorX8632Opt
"${TRANSLATOR_TESTS}"
oh, good catch

http://codereview.chromium.org/7524023/diff/6001/tools/llvm/utman.sh
File tools/llvm/utman.sh (right):

http://codereview.chromium.org/7524023/diff/6001/tools/llvm/utman.sh#newcode670
tools/llvm/utman.sh:670: hg-checkout ${REPO_UPSTREAM} ${TC_SRC_UPSTREAM}
${UPSTREAM_REV}
On 2011/07/28 23:48:54, jvoung wrote:
> Just noticed, but should all the ${TC_SRC...} have quotes around them?

Done.

[jvoung - send to chromium..., 2011-07-29 00:38:32]

http://codereview.chromium.org/7524023/diff/10004/tools/llvm/merge-tool.sh
File tools/llvm/merge-tool.sh (right):

http://codereview.chromium.org/7524023/diff/10004/tools/llvm/merge-tool.sh#ne...
tools/llvm/merge-tool.sh:37: readonly MASTER_LLVM=${MASTER_LLVM:-}
So, on the bots these values are set in .bashrc or something like that? It is
not specified in the invocations found in buildbot_pnacl_merge.sh

Are they "TC_SRC" directories, or are they URLs? Could prefix it with "TC_SRC"
or something if it is a directory of sources.

http://codereview.chromium.org/7524023/diff/10004/tools/llvm/merge-tool.sh#ne...
tools/llvm/merge-tool.sh:170: StepBanner "Updating hg file list"
${stepid} ?

http://codereview.chromium.org/7524023/diff/10004/tools/llvm/merge-tool.sh#ne...
tools/llvm/merge-tool.sh:177: hg-commit "${TC_SRC_UPSTREAM}" "Updating vendor to
r${MERGE_REVISION}"
Might be better to just $(get-merge-revision) wherever you need it instead of
making it a global and relying on it being set before you get to this point.

http://codereview.chromium.org/7524023/diff/10004/tools/llvm/merge-tool.sh#ne...
tools/llvm/merge-tool.sh:208: StepBanner "final-commit" "Committing and pushing
merge"
Could print the MERGE_REVISION somewhere before all the interactive stuff.

[pdox, 2011-07-29 00:59:58]

http://codereview.chromium.org/7524023/diff/10004/tools/llvm/merge-tool.sh
File tools/llvm/merge-tool.sh (right):

http://codereview.chromium.org/7524023/diff/10004/tools/llvm/merge-tool.sh#ne...
tools/llvm/merge-tool.sh:37: readonly MASTER_LLVM=${MASTER_LLVM:-}
They are the upstream SVN directories, not part of the toolchain source tree.

I'm not sure where they are going to come from. I am waiting on BradN to setup
the new waterfall, which triggers on LLVM svn revisions. Most likely, the
buildbot slave will download the svn repositories and keep it up to date. And
then MASTER_LLVM, MASTER_LLVM_GCC can be hard-coded to a fixed location in
buildbot_pnacl_merge.sh (or PWD?).

I've added a comment to clarify.

http://codereview.chromium.org/7524023/diff/10004/tools/llvm/merge-tool.sh#ne...
tools/llvm/merge-tool.sh:170: StepBanner "Updating hg file list"
On 2011/07/29 00:38:32, jvoung wrote:
> ${stepid} ?

Done.

http://codereview.chromium.org/7524023/diff/10004/tools/llvm/merge-tool.sh#ne...
tools/llvm/merge-tool.sh:177: hg-commit "${TC_SRC_UPSTREAM}" "Updating vendor to
r${MERGE_REVISION}"
Technically, I always want to run get-merge-revision at least once, since it
does a sanity check to make sure the revision numbers match.

I've added the setting of MERGE_REVISION to check-svn-repo, and arranged for
check-svn-repo to run before the main command.

http://codereview.chromium.org/7524023/diff/10004/tools/llvm/merge-tool.sh#ne...
tools/llvm/merge-tool.sh:208: StepBanner "final-commit" "Committing and pushing
merge"
done, I think? Added the banner to check-svn-repos

[jvoung - send to chromium..., 2011-07-29 01:18:13]

LGTM