Dynamically pick build directory on the slave side.

Dynamically pick build directory on the slave side.

Ignore the --build-dir flag passed in from the server. From what I can tell,
this is always one of src/build, src/out or src/xcodebuild (except for one FYI
bot which uses src/chrome, but that's on FYI).

All scripts that call this are below chromium/ except extract_build.py, but
that's only called by Tester bots, and only chromium bots use that slave type.

From what I can tell, this change shouldn't affect behavior on any waterfall.

BUG=294387
Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=229801

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=230011

Committed: http://src.chromium.org/viewvc/chrome?view=rev&revision=230050

[M-A Ruel, 2013-10-21 13:12:20]

lgtm, thanks for doing this.

https://codereview.chromium.org/31543002/diff/50001/build/scripts/slave/build...
File build/scripts/slave/build_directory.py (right):

https://codereview.chromium.org/31543002/diff/50001/build/scripts/slave/build...
build/scripts/slave/build_directory.py:19: pass
optional: You could put file_a_stat = 0 here, saving 3 lines total.

or use a local function, it'll be even shorter at 7 lines total.

def getmtime(f):
  try:
    return os.path.getmtime(f)
  except OSError:
    return 0

return getmtime(file_a) > getmtime(file_b)

https://codereview.chromium.org/31543002/diff/50001/build/scripts/slave/build...
build/scripts/slave/build_directory.py:58: # TODO(thakis): Remove parameters of
this function, rename it to
I prefer the TODO comment to be inside the function, not outside.

https://codereview.chromium.org/31543002/diff/50001/build/scripts/slave/build...
build/scripts/slave/build_directory.py:62: Assumes that the current working
directory is the checkout root.
Add an empty line above.

https://codereview.chromium.org/31543002/diff/50001/build/scripts/slave/build...
build/scripts/slave/build_directory.py:77: raise Exception('Unexpected platform
' + sys.platform)
s/Exception/NotImplementedError/
and use '... %s' % sys.platform

[commit-bot: I haz the power, 2013-10-21 14:11:00]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/thakis@chromium.org/31543002/50001

[commit-bot: I haz the power, 2013-10-21 14:11:48]

Presubmit check for 31543002-50001 failed and returned exit status 1.

Running presubmit commit checks ...

** Presubmit Warnings **
Found line ending with white spaces in:

***************
build/scripts/slave/build_directory.py:47
***************

Presubmit checks took 45.7s to calculate.

[Nico, 2013-10-21 14:14:08]

https://codereview.chromium.org/31543002/diff/50001/build/scripts/slave/build...
File build/scripts/slave/build_directory.py (right):

https://codereview.chromium.org/31543002/diff/50001/build/scripts/slave/build...
build/scripts/slave/build_directory.py:19: pass
On 2013/10/21 13:12:20, M-A Ruel wrote:
> optional: You could put file_a_stat = 0 here, saving 3 lines total.
> 
> or use a local function, it'll be even shorter at 7 lines total.
> 
> def getmtime(f):
>   try:
>     return os.path.getmtime(f)
>   except OSError:
>     return 0
> 
> return getmtime(file_a) > getmtime(file_b)

Done.

https://codereview.chromium.org/31543002/diff/50001/build/scripts/slave/build...
build/scripts/slave/build_directory.py:58: # TODO(thakis): Remove parameters of
this function, rename it to
On 2013/10/21 13:12:20, M-A Ruel wrote:
> I prefer the TODO comment to be inside the function, not outside.

Done.

https://codereview.chromium.org/31543002/diff/50001/build/scripts/slave/build...
build/scripts/slave/build_directory.py:62: Assumes that the current working
directory is the checkout root.
On 2013/10/21 13:12:20, M-A Ruel wrote:
> Add an empty line above.

Done.

https://codereview.chromium.org/31543002/diff/50001/build/scripts/slave/build...
build/scripts/slave/build_directory.py:77: raise Exception('Unexpected platform
' + sys.platform)
On 2013/10/21 13:12:20, M-A Ruel wrote:
> s/Exception/NotImplementedError/
> and use '... %s' % sys.platform

Done.

[commit-bot: I haz the power, 2013-10-21 14:15:00]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/thakis@chromium.org/31543002/140001

[commit-bot: I haz the power, 2013-10-21 14:15:51]

Change committed as 229801

[commit-bot: I haz the power, 2013-10-22 02:09:36]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/thakis@chromium.org/31543002/180001

[commit-bot: I haz the power, 2013-10-22 02:10:43]

Presubmit check for 31543002-180001 failed and returned exit status 1.

Running presubmit commit checks ...

** Presubmit ERRORS **
tests/masters_test.py failed

=== Error running master.chromium.webrtc === 
2013-10-21 19:10:09-0700 [-] Starting factory
<buildbot.status.web.baseweb.RotateLogSite instance at 0x411b9e0>
2013-10-21 19:10:09-0700 [-] Setting up http.log rotating 10 files of 1000000
bytes each
2013-10-21 19:10:09-0700 [-] WebStatus using
(/b/commit-queue/workdir/tools/build/masters/master.chromium/public_html)
2013-10-21 19:10:10-0700 [-] adding IStatusReceiver <WebStatus on port tcp:8254
at 0x40f7ef0>
2013-10-21 19:10:10-0700 [-] buildbot.status.web.baseweb.RotateLogSite starting
on 8254
2013-10-21 19:10:10-0700 [-] Starting factory
<buildbot.status.web.baseweb.RotateLogSite instance at 0x2b88d8062320>
2013-10-21 19:10:10-0700 [-] Setting up http.log rotating 10 files of 1000000
bytes each
2013-10-21 19:10:10-0700 [-] WebStatus using
(/b/commit-queue/workdir/tools/build/masters/master.chromium/public_html)
2013-10-21 19:10:10-0700 [-] removing 0 old schedulers, updating 0, and adding 8
2013-10-21 19:10:15-0700 [-] Unhandled Error
	Traceback (most recent call last):
	Failure: twisted.internet.defer.FirstError: FirstError[#7, [Failure instance:
Traceback: <class 'sqlalchemy.exc.OperationalError'>: (OperationalError)
database is locked u'INSERT INTO schedulers (name, state, class_name) VALUES (?,
?, ?)' ('chromium_cros', '{}', 'buildbot.schedulers.basic.Scheduler')
	/usr/lib/python2.7/threading.py:524:__bootstrap
	/usr/lib/python2.7/threading.py:551:__bootstrap_inner
	/usr/lib/python2.7/threading.py:504:run
	--- <exception caught here> ---
	/b/commit-queue/workdir/tools/build/third_party/twisted_10_2/twisted/python/threadpool.py:207:_worker
	/b/commit-queue/workdir/tools/build/third_party/twisted_10_2/twisted/python/context.py:59:callWithContext
	/b/commit-queue/workdir/tools/build/third_party/twisted_10_2/twisted/python/context.py:37:callWithContext
	/b/commit-queue/workdir/tools/build/third_party/buildbot_8_4p1/buildbot/db/pool.py:112:thd
	/b/commit-queue/workdir/tools/build/third_party/buildbot_8_4p1/buildbot/db/schedulers.py:165:thd
	/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/base.py:1358:execute
	/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/base.py:1491:_execute_clauseelement
	/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/base.py:1599:_execute_context
	/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/base.py:1592:_execute_context
	/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/default.py:325:do_execute
	]]


=== Error running master.chromium.webrtc.fyi === 
2013-10-21 19:10:09-0700 [-] WebStatus using
(/b/commit-queue/workdir/tools/build/masters/master.chromium/public_html)
2013-10-21 19:10:09-0700 [-] adding IStatusReceiver <WebStatus on port tcp:8256
at 0x315ae60>
2013-10-21 19:10:09-0700 [-] buildbot.status.web.baseweb.RotateLogSite starting
on 8256
2013-10-21 19:10:09-0700 [-] Starting factory
<buildbot.status.web.baseweb.RotateLogSite instance at 0x32299e0>
2013-10-21 19:10:09-0700 [-] Setting up http.log rotating 10 files of 1000000
bytes each
2013-10-21 19:10:09-0700 [-] WebStatus using
(/b/commit-queue/workdir/tools/build/masters/master.chromium/public_html)
2013-10-21 19:10:09-0700 [-] removing 0 old schedulers, updating 0, and adding
17
2013-10-21 19:10:14-0700 [-] win_periodic_scheduler: missed scheduled build
time, so building immediately
2013-10-21 19:10:14-0700 [-] linux_periodic_scheduler: missed scheduled build
time, so building immediately
2013-10-21 19:10:14-0700 [-] Unhandled Error
	Traceback (most recent call last):
	Failure: twisted.internet.defer.FirstError: FirstError[#6, [Failure instance:
Traceback: <class 'sqlalchemy.exc.OperationalError'>: (OperationalError)
database is locked u'INSERT INTO schedulers (name, state, class_name) VALUES (?,
?, ?)' ('win_webrtc_trunk_scheduler', '{}',
'buildbot.schedulers.basic.Scheduler')
	/usr/lib/python2.7/threading.py:524:__bootstrap
	/usr/lib/python2.7/threading.py:551:__bootstrap_inner
	/usr/lib/python2.7/threading.py:504:run
	--- <exception caught here> ---
	/b/commit-queue/workdir/tools/build/third_party/twisted_10_2/twisted/python/threadpool.py:207:_worker
	/b/commit-queue/workdir/tools/build/third_party/twisted_10_2/twisted/python/context.py:59:callWithContext
	/b/commit-queue/workdir/tools/build/third_party/twisted_10_2/twisted/python/context.py:37:callWithContext
	/b/commit-queue/workdir/tools/build/third_party/buildbot_8_4p1/buildbot/db/pool.py:112:thd
	/b/commit-queue/workdir/tools/build/third_party/buildbot_8_4p1/buildbot/db/schedulers.py:165:thd
	/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/base.py:1358:execute
	/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/base.py:1491:_execute_clauseelement
	/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/base.py:1599:_execute_context
	/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/base.py:1592:_execute_context
	/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/default.py:325:do_execute
	]]


=== Error running master.client.dynamorio === 
2013-10-21 19:10:00-0700 [-] Setting up http.log rotating 10 files of 1000000
bytes each
2013-10-21 19:10:00-0700 [-] WebStatus using
(/b/commit-queue/workdir/tools/build/masters/master.client.dynamorio/public_html)
2013-10-21 19:10:00-0700 [-] adding IStatusReceiver
<buildbot.status.mail.MailNotifier instance at 0x40b13f8>
2013-10-21 19:10:00-0700 [-] removing 0 old schedulers, updating 0, and adding 3
2013-10-21 19:10:00-0700 [-] periodic_stable: missed scheduled build time, so
building immediately
2013-10-21 19:10:04-0700 [-] adding 1 new changesources, removing 0
2013-10-21 19:10:04-0700 [-] configuration update complete
2013-10-21 19:10:05-0700 [-] while initializing Nightly 'dr-nightly'
	Traceback (most recent call last):
	  File "/usr/lib/python2.7/threading.py", line 524, in __bootstrap
	    self.__bootstrap_inner()
	  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
	    self.run()
	  File "/usr/lib/python2.7/threading.py", line 504, in run
	    self.__target(*self.__args, **self.__kwargs)
	--- <exception caught here> ---
	  File
"/b/commit-queue/workdir/tools/build/third_party/twisted_10_2/twisted/python/threadpool.py",
line 207, in _worker
	    result = context.call(ctx, function, *args, **kwargs)
	  File
"/b/commit-queue/workdir/tools/build/third_party/twisted_10_2/twisted/python/context.py",
line 59, in callWithContext
	    return self.currentContext().callWithContext(ctx, func, *args, **kw)
	  File
"/b/commit-queue/workdir/tools/build/third_party/twisted_10_2/twisted/python/context.py",
line 37, in callWithContext
	    return func(*args,**kw)
	  File
"/b/commit-queue/workdir/tools/build/third_party/buildbot_8_4p1/buildbot/db/pool.py",
line 112, in thd
	    rv = callable(conn, *args, **kwargs)
	  File
"/b/commit-queue/workdir/tools/build/third_party/buildbot_8_4p1/buildbot/db/schedulers.py",
line 98, in thd
	    conn.execute(q)
	  File
"/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/base.py",
line 1358, in execute
	    params)
	  File
"/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/base.py",
line 1491, in _execute_clauseelement
	    compiled_sql, distilled_params
	  File
"/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/base.py",
line 1599, in _execute_context
	    context)
	  File
"/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/base.py",
line 1592, in _execute_context
	    context)
	  File
"/b/commit-queue/workdir/tools/build/third_party/sqlalchemy_0_7_1/sqlalchemy/engine/default.py",
line 325, in do_execute
	    cursor.execute(statement, parameters)
	sqlalchemy.exc.OperationalError: (OperationalError) database is locked u'DELETE
FROM scheduler_changes WHERE scheduler_changes.schedulerid = ?' (2,)


=== Error running master.client.nacl === 
2013-10-21 19:10:00-0700 [-] Starting factory
<buildbot.status.web.baseweb.RotateLogSite instance at 0x3fa5ab8>
2013-10-21 19:10:00-0700 [-] Setting up http.log rotating 10 files of 1000000
bytes each
2013-10-21 19:10:00-0700 [-] WebStatus using
(/b/commit-queue/workdir/tools/build/masters/master.chromium/public_html)
2013-10-21 19:10:00-0700 [-] adding IStatusReceiver <WebStatus on port tcp:8230
at 0x3d1a518>
2013-10-21 19:10:00-0700 [-] buildbot.status.web.baseweb.RotateLogSite starting
on 8230
2013-10-21 19:10:00-0700 [-] Starting factory
<buildbot.status.web.baseweb.RotateLogSite instance at 0x4090b00>
2013-10-21 19:10:00-0700 [-] Setting up http.log rotating 10 files of 1000000
bytes each
2013-10-21 19:10:00-0700 [-] WebStatus using
(/b/commit-queue/workdir/tools/build/masters/master.chromium/public_html)
2013-10-21 19:10:00-0700 [-] removing 0 old schedulers, updating 0, and adding 4
2013-10-21 19:10:05-0700 [-] Unhandled Error
	Traceback (most recent call last):
	Failure: twisted.internet.defer.FirstError: FirstError[#1, [Failure instance:
Traceback: <class 'sqlalchemy.exc.OperationalError'>: (OperationalError)
database is locked u'INSERT INTO schedulers (name, state, class_name) VALUES (?,
?, ?)' ('arm_dbg_hw_tests', '{}', 'buildbot.schedulers.triggerable.Triggerable')
	/usr/lib/python2.7/threading.py:524:__bootstrap
	/usr/lib/python2.7/threading.py:551:__bootstrap_inner
	/usr/lib/python2.7/threading.py:504:run
	--- <exception caught here> ---
	/b/commit-queue/workdir/tools/build/third_party/twisted_10_2/twisted/python/threadpool.py:207:_worker
	/b/commit-queue/workdir/tools/build/third_party/twisted_10_2/twisted/python/context.py:59:callWithContext
	/b/c…
(message too large)

[commit-bot: I haz the power, 2013-10-22 02:13:06]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/thakis@chromium.org/31543002/180001

[commit-bot: I haz the power, 2013-10-22 02:14:01]

Change committed as 230011

[cmp, 2013-10-22 02:21:18]

Robbie, can you take a look at this, too?

[iannucci, 2013-10-22 02:31:55]

lgtm2

https://codereview.chromium.org/31543002/diff/180001/build/scripts/slave/buil...
File build/scripts/slave/build_directory.py (right):

https://codereview.chromium.org/31543002/diff/180001/build/scripts/slave/buil...
build/scripts/slave/build_directory.py:65: if sys.platform == 'cygwin' or
sys.platform.startswith('win'):
nit: if sys.platform.startswith(('win', 'cygwin'))

[commit-bot: I haz the power, 2013-10-22 04:42:52]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/thakis@chromium.org/31543002/180001

[commit-bot: I haz the power, 2013-10-22 04:43:43]

Change committed as 230050

[Nico, 2013-10-24 04:07:48]

Retrying this again at https://codereview.chromium.org/38873003