[Android] Add a configurable environment for devil/. (RELAND)

[Android] Add a configurable environment for devil/. (RELAND)

This is a reland of https://codereview.chromium.org/1316413003/

BUG=476719
Committed: https://crrev.com/06869eebcab4b41d21e96ea2df732a2b40914fe4
Cr-Commit-Position: refs/heads/master@{#355683}

[jbudorick, 2015-10-12 22:50:17]

jbudorick@chromium.org changed reviewers:
+ aiolos@chromium.org, mikecase@chromium.org, nednguyen@google.com

[nednguyen, 2015-10-13 01:52:58]

On 2015/10/12 22:50:17, jbudorick wrote:

I defer reviewing this to the other two reviewers.

[aiolos (Not reviewing), 2015-10-13 13:29:17]

lgtm

https://codereview.chromium.org/1399273002/diff/20001/build/android/gyp/get_d...
File build/android/gyp/get_device_configuration.py (right):

https://codereview.chromium.org/1399273002/diff/20001/build/android/gyp/get_d...
build/android/gyp/get_device_configuration.py:20: BUILD_ANDROID_DIR =
os.path.abspath(
Is this supposed to be here? It's already defined in bulid_device, and I don't
see a reason for it to be in the middle of the imports.

[jbudorick, 2015-10-13 13:40:48]

https://codereview.chromium.org/1399273002/diff/20001/build/android/gyp/get_d...
File build/android/gyp/get_device_configuration.py (right):

https://codereview.chromium.org/1399273002/diff/20001/build/android/gyp/get_d...
build/android/gyp/get_device_configuration.py:20: BUILD_ANDROID_DIR =
os.path.abspath(
On 2015/10/13 13:29:17, aiolos wrote:
> Is this supposed to be here? It's already defined in bulid_device, and I don't
> see a reason for it to be in the middle of the imports.

I put it here to add to sys.path and promptly forgot to do so :/

It works as is because build_device adds //build/android to the sys.path, but I
would rather not depend on that. Added sys.path.append here. BUILD_ANDROID_DIR
should now be handled consistently throughout //build/android/gyp

[jbudorick, 2015-10-13 13:41:21]

The CQ bit was checked by jbudorick@chromium.org

[jbudorick, 2015-10-13 13:41:21]

The patchset sent to the CQ was uploaded after l-g-t-m from aiolos@chromium.org
Link to the patchset: https://codereview.chromium.org/1399273002/#ps40001
(title: "Consistent handling of BUILD_ANDROID_DIR in build/android/gyp/")

[commit-bot: I haz the power, 2015-10-13 13:43:44]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1399273002/40001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1399273002/40001

[commit-bot: I haz the power, 2015-10-13 14:30:59]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-10-13 14:30:59]

Try jobs failed on following builders:
  linux_android_rel_ng on tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_android_r...)

[aiolos (Not reviewing), 2015-10-13 18:27:54]

Still lgtm

https://codereview.chromium.org/1399273002/diff/20001/build/android/gyp/get_d...
File build/android/gyp/get_device_configuration.py (right):

https://codereview.chromium.org/1399273002/diff/20001/build/android/gyp/get_d...
build/android/gyp/get_device_configuration.py:20: BUILD_ANDROID_DIR =
os.path.abspath(
On 2015/10/13 13:40:48, jbudorick wrote:
> On 2015/10/13 13:29:17, aiolos wrote:
> > Is this supposed to be here? It's already defined in bulid_device, and I
don't
> > see a reason for it to be in the middle of the imports.
> 
> I put it here to add to sys.path and promptly forgot to do so :/
> 
> It works as is because build_device adds //build/android to the sys.path, but
I
> would rather not depend on that. Added sys.path.append here. BUILD_ANDROID_DIR
> should now be handled consistently throughout //build/android/gyp

Acknowledged.

[jbudorick, 2015-10-21 03:21:44]

aiolos: ptal, particularly at the new changes in tools/telemetry/

[aiolos (Not reviewing), 2015-10-21 04:25:32]

I haven't looked in depth at the devil changes, but I have a couple of comments
for the Telemetry ones.

https://codereview.chromium.org/1399273002/diff/120001/tools/telemetry/teleme...
File tools/telemetry/telemetry/internal/binary_dependencies.json (right):

https://codereview.chromium.org/1399273002/diff/120001/tools/telemetry/teleme...
tools/telemetry/telemetry/internal/binary_dependencies.json:32:
"../../../../third_party/android_tools/sdk"
Telemetry isn't allowed to depend on the chromium repo here, but you can add
them into the tools/perf/chrome_telemetry_build/binary_dependencies.json file
instead. The one draw back from that is that those files aren't available
outside of a chromium repo. Our work around for that is to have a pre-built
binary that gets uploaded to cloud storage, and then it's information gets added
to this file.

https://codereview.chromium.org/1399273002/diff/120001/tools/telemetry/teleme...
File tools/telemetry/telemetry/internal/util/binary_manager.py (right):

https://codereview.chromium.org/1399273002/diff/120001/tools/telemetry/teleme...
tools/telemetry/telemetry/internal/util/binary_manager.py:38:
devil_env.config.InitializeFromDm(_dependency_manager)
Oh, interesting. I had assumed you would just call devil_env.config.Initialize()
from here. I don't mind Telemetry tracking the dependencies in the
binary_manager itself, so that you'd only have the one code path to keep things
simpler/contained in devil.

[jbudorick, 2015-10-21 15:13:01]

Working on a revised version. (again)

https://codereview.chromium.org/1399273002/diff/120001/tools/telemetry/teleme...
File tools/telemetry/telemetry/internal/binary_dependencies.json (right):

https://codereview.chromium.org/1399273002/diff/120001/tools/telemetry/teleme...
tools/telemetry/telemetry/internal/binary_dependencies.json:32:
"../../../../third_party/android_tools/sdk"
On 2015/10/21 04:25:32, aiolos wrote:
> Telemetry isn't allowed to depend on the chromium repo here, but you can add
> them into the tools/perf/chrome_telemetry_build/binary_dependencies.json file
> instead. The one draw back from that is that those files aren't available
> outside of a chromium repo. Our work around for that is to have a pre-built
> binary that gets uploaded to cloud storage, and then it's information gets
added
> to this file.

I had planned on doing this separately, but between this comment & primiano's
memtrack CL I think I have a better idea.

https://codereview.chromium.org/1399273002/diff/120001/tools/telemetry/teleme...
File tools/telemetry/telemetry/internal/util/binary_manager.py (right):

https://codereview.chromium.org/1399273002/diff/120001/tools/telemetry/teleme...
tools/telemetry/telemetry/internal/util/binary_manager.py:38:
devil_env.config.InitializeFromDm(_dependency_manager)
On 2015/10/21 04:25:32, aiolos wrote:
> Oh, interesting. I had assumed you would just call
devil_env.config.Initialize()
> from here. I don't mind Telemetry tracking the dependencies in the
> binary_manager itself, so that you'd only have the one code path to keep
things
> simpler/contained in devil.

I implemented this in at least three different ways before settling on the
current version, but I think I like that suggestion better.

[jbudorick, 2015-10-22 01:47:27]

This is ready for review again. I'm testing out its impact on cycle time,
though, and may have a few minor tweaks accordingly.

[aiolos (Not reviewing), 2015-10-22 02:37:45]

lgtm

https://codereview.chromium.org/1399273002/diff/180001/build/android/devil/de...
File build/android/devil/devil_env.py (right):

https://codereview.chromium.org/1399273002/diff/180001/build/android/devil/de...
build/android/devil/devil_env.py:89: arch = device.product_cpu_abi if device
else 'any'
To be clear "android_any" is used for any dependency that runs on the host
instead of the device, correct? Is it used for anything else?

[jbudorick, 2015-10-22 02:43:17]

https://codereview.chromium.org/1399273002/diff/180001/build/android/devil/de...
File build/android/devil/devil_env.py (right):

https://codereview.chromium.org/1399273002/diff/180001/build/android/devil/de...
build/android/devil/devil_env.py:89: arch = device.product_cpu_abi if device
else 'any'
On 2015/10/22 02:37:45, aiolos wrote:
> To be clear "android_any" is used for any dependency that runs on the host
> instead of the device, correct? Is it used for anything else?

Exactly. It isn't used for anything else.

[aiolos (Not reviewing), 2015-10-22 02:46:03]

https://codereview.chromium.org/1399273002/diff/180001/build/android/devil/de...
File build/android/devil/devil_env.py (right):

https://codereview.chromium.org/1399273002/diff/180001/build/android/devil/de...
build/android/devil/devil_env.py:89: arch = device.product_cpu_abi if device
else 'any'
On 2015/10/22 02:43:17, jbudorick wrote:
> On 2015/10/22 02:37:45, aiolos wrote:
> > To be clear "android_any" is used for any dependency that runs on the host
> > instead of the device, correct? Is it used for anything else?
> 
> Exactly. It isn't used for anything else.

nit: I'd suggest changing it to 'host' instead of 'any' or clarity, but lgtm
either way.

[aiolos (Not reviewing), 2015-10-22 02:46:40]

On 2015/10/22 02:46:03, aiolos wrote:
>
https://codereview.chromium.org/1399273002/diff/180001/build/android/devil/de...
> File build/android/devil/devil_env.py (right):
> 
>
https://codereview.chromium.org/1399273002/diff/180001/build/android/devil/de...
> build/android/devil/devil_env.py:89: arch = device.product_cpu_abi if device
> else 'any'
> On 2015/10/22 02:43:17, jbudorick wrote:
> > On 2015/10/22 02:37:45, aiolos wrote:
> > > To be clear "android_any" is used for any dependency that runs on the host
> > > instead of the device, correct? Is it used for anything else?
> > 
> > Exactly. It isn't used for anything else.
> 
> nit: I'd suggest changing it to 'host' instead of 'any' or clarity, but lgtm
> either way.

s/or/for/

[jbudorick, 2015-10-22 23:55:04]

The CQ bit was checked by jbudorick@chromium.org

[jbudorick, 2015-10-22 23:55:05]

The patchset sent to the CQ was uploaded after l-g-t-m from aiolos@chromium.org
Link to the patchset: https://codereview.chromium.org/1399273002/#ps220001
(title: "android_any -> android_host")

[commit-bot: I haz the power, 2015-10-22 23:55:48]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1399273002/220001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1399273002/220001

[commit-bot: I haz the power, 2015-10-23 00:06:12]

Committed patchset #12 (id:220001)

[commit-bot: I haz the power, 2015-10-23 00:07:24]

Patchset 12 (id:??) landed as
https://crrev.com/06869eebcab4b41d21e96ea2df732a2b40914fe4
Cr-Commit-Position: refs/heads/master@{#355683}

[rmistry, 2015-10-23 13:28:14]

On 2015/10/23 00:07:24, commit-bot: I haz the power wrote:
> Patchset 12 (id:??) landed as
> https://crrev.com/06869eebcab4b41d21e96ea2df732a2b40914fe4
> Cr-Commit-Position: refs/heads/master@{#355683}


Cluster Telemetry (https://www.chromium.org/developers/cluster-telemetry)
parallel linux runs have started failing in the last ~20 hours with:


********************************************************************************
(CRITICAL) 2015-10-23 06:05:40,062 timeout_retry._LogLastException:120 
Exception on thread TimeoutThread-2-for-MainThread (attempt 2 of 3)
(CRITICAL) 2015-10-23 06:05:40,062 timeout_retry._LogLastException:121 
********************************************************************************
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124 
Traceback (most recent call last):
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
"/b/storage/chromium/src/build/android/devil/utils/timeout_retry.py", line 161,
in Run
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
error_log_func=error_log_func)
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
"/b/storage/chromium/src/build/android/devil/utils/reraiser_thread.py", line
186, in JoinAll
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
self._JoinAll(watcher, timeout)
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
"/b/storage/chromium/src/build/android/devil/utils/reraiser_thread.py", line
158, in _JoinAll
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
thread.ReraiseIfException()
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
"/b/storage/chromium/src/build/android/devil/utils/reraiser_thread.py", line 81,
in run
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
self._ret = self._func(*self._args, **self._kwargs)
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
"/b/storage/chromium/src/build/android/devil/utils/timeout_retry.py", line 154,
in <lambda>
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
"/b/storage/chromium/src/build/android/devil/android/decorators.py", line 44, in
impl
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
return f(*args, **kwargs)
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
"/b/storage/chromium/src/build/android/devil/android/sdk/adb_wrapper.py", line
94, in _RunAdbCmd
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
timeout_retry.CurrentTimeoutThreadGroup().GetRemainingTime())
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
"/b/storage/chromium/src/build/android/devil/utils/cmd_helper.py", line 264, in
GetCmdStatusAndOutputWithTimeout
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
stderr=subprocess.STDOUT)
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
"/b/storage/chromium/src/build/android/devil/utils/cmd_helper.py", line 95, in
Popen
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
preexec_fn=lambda: signal.signal(signal.SIGPIPE, signal.SIG_DFL))
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
"/usr/lib/python2.7/subprocess.py", line 679, in __init__
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
errread, errwrite)
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
"/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
raise child_exception
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124  OSError:
[Errno 26] Text file busy
(CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:125  



CT runs some benchmarks in parallel, what *I think* is happening is they are
trying to touch the same file (on the Android device) which fails. Note: These
runs are testing the chromium build on the Linux platform NOT on the attached
Android device, which is why I am surprised why the state of the Android device
matters.


Please investigate / revert / something else.

[rmistry, 2015-10-23 13:32:34]

On 2015/10/23 13:28:14, rmistry wrote:
> On 2015/10/23 00:07:24, commit-bot: I haz the power wrote:
> > Patchset 12 (id:??) landed as
> > https://crrev.com/06869eebcab4b41d21e96ea2df732a2b40914fe4
> > Cr-Commit-Position: refs/heads/master@{#355683}
> 
> 
> Cluster Telemetry (https://www.chromium.org/developers/cluster-telemetry)
> parallel linux runs have started failing in the last ~20 hours with:
> 
> 
>
********************************************************************************
> (CRITICAL) 2015-10-23 06:05:40,062 timeout_retry._LogLastException:120 
> Exception on thread TimeoutThread-2-for-MainThread (attempt 2 of 3)
> (CRITICAL) 2015-10-23 06:05:40,062 timeout_retry._LogLastException:121 
>
********************************************************************************
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124 
> Traceback (most recent call last):
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
> "/b/storage/chromium/src/build/android/devil/utils/timeout_retry.py", line
161,
> in Run
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
> error_log_func=error_log_func)
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
> "/b/storage/chromium/src/build/android/devil/utils/reraiser_thread.py", line
> 186, in JoinAll
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
> self._JoinAll(watcher, timeout)
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
> "/b/storage/chromium/src/build/android/devil/utils/reraiser_thread.py", line
> 158, in _JoinAll
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
> thread.ReraiseIfException()
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
> "/b/storage/chromium/src/build/android/devil/utils/reraiser_thread.py", line
81,
> in run
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
> self._ret = self._func(*self._args, **self._kwargs)
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
> "/b/storage/chromium/src/build/android/devil/utils/timeout_retry.py", line
154,
> in <lambda>
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
> child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
> "/b/storage/chromium/src/build/android/devil/android/decorators.py", line 44,
in
> impl
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
> return f(*args, **kwargs)
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
> "/b/storage/chromium/src/build/android/devil/android/sdk/adb_wrapper.py", line
> 94, in _RunAdbCmd
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
> timeout_retry.CurrentTimeoutThreadGroup().GetRemainingTime())
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
> "/b/storage/chromium/src/build/android/devil/utils/cmd_helper.py", line 264,
in
> GetCmdStatusAndOutputWithTimeout
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
> stderr=subprocess.STDOUT)
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
> "/b/storage/chromium/src/build/android/devil/utils/cmd_helper.py", line 95, in
> Popen
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
> preexec_fn=lambda: signal.signal(signal.SIGPIPE, signal.SIG_DFL))
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
> "/usr/lib/python2.7/subprocess.py", line 679, in __init__
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
> errread, errwrite)
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124    File
> "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124     
> raise child_exception
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:124 
OSError:
> [Errno 26] Text file busy
> (CRITICAL) 2015-10-23 06:05:40,063 timeout_retry._LogLastException:125  
> 
> 
> 
> CT runs some benchmarks in parallel, what *I think* is happening is they are
> trying to touch the same file (on the Android device) which fails. Note: These
> runs are testing the chromium build on the Linux platform NOT on the attached
> Android device, which is why I am surprised why the state of the Android
device
> matters.
> 
> 
> Please investigate / revert / something else.

Filed https://code.google.com/p/skia/issues/detail?id=4506

[jbudorick, 2015-10-23 13:40:49]

A revert of this CL (patchset #12 id:220001) has been created in
https://codereview.chromium.org/1415413005/ by jbudorick@chromium.org.

The reason for reverting is: broke cluster telemetry:
https://code.google.com/p/skia/issues/detail?id=4506.