Overhaul Android support and make Swarming bot use python-adb

Overhaul Android support and make Swarming bot use python-adb

- Embed adb and its dependency libusb1. Remove all other dependencies. In
  particular, there is no dependency on the 'adb' executable, which is awesome.
  TODO(maruel): M2Crypto
- Update os_utilities.py to report android devices.
- Add tools/android/51-android.rules as a drop-in file to put in
  /etc/udev/rules.d/.

R=vadimsh@chromium.org
BUG=

Committed: https://github.com/luci/luci-py/commit/43605c224e3aa1afa0f382200b4f46f51492987d

[M-A Ruel, 2015-08-20 20:59:24]

Patchset #1 (id:1) has been deleted

[M-A Ruel, 2015-08-20 21:13:56]

Patchset #1 (id:20001) has been deleted

[M-A Ruel, 2015-08-20 22:02:49]

Patchset #1 (id:40001) has been deleted

[M-A Ruel, 2015-08-20 22:10:51]

Patchset #1 (id:60001) has been deleted

[M-A Ruel, 2015-08-20 22:12:53]

Patchset #1 (id:80001) has been deleted

[Vadim Sh., 2015-08-20 22:40:09]

this all looks very hacky :( os_utilities.py is on a path of being a big mess...
I mean, keeping state in globals, loading modules on the fly, "popping"
dimensions, etc.

If we aren't going to run non-android tasks on android swarming bot, I'd prefer
it to be somewhat more structured, e.g. there's some big switch somewhere that
transforms a bot to an android bot or something (instead of bunch of if's in
os_utilities.py).

Crypto stuff + python is also making me very sad. It is another signal to
consider including CIPD support (or any other binary package delivery mechanism)
into Swarming bot, so that bot can bootstrap binary dependencies when needed
(then we can package the signer as simple go binary and shell out to it).

I'm not going to block this CL, just expressing my general grumpiness regarding
the trajectory of swarming bot code.

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
File appengine/swarming/swarming_bot/adb (right):

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/adb:1: ../../third_party/python-adb/adb
why symlinks instead of sys.path hack? sys.path hack is more conventional..
Other third_party libs are added via sys.path.

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
File appengine/swarming/swarming_bot/os_utilities.py (right):

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1223: def
initialize_android(pub_key, priv_key):
how do you feel about moving it to "android_utilities.py"? os_utilities is
already huge.

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1227: ~/.android/adbkey.pub.
is it true only for chrome-infra bots?

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1235: # Workaround pylint lint
errors.
just disable them, it will look less weird

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1252: path =
os.path.expanduser('~/.android/adbkey')
is it intentionally different from priv_key? What is priv_key then? Add a doc
string for priv_key and pub_key.

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1343:
subprocess.call(['killall', 'adb'])
killall --exact adb

(In general I'm not a fan of blindly killing processes)

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1368: # holds an handle to the
USB ports or group acl (plugdev) hasn't been
typo: a handle

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1370: handle.Open()
maybe at least log exception here and ignore this device?

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1374: handle, banner='swarming',
rsa_keys=_ADB_KEYS, auth_timeout_ms=100)
100ms? Is it really that fast?

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1403: device.Close()
catch and log exceptions here? (so that all devices are closed even if one
failed)

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1898:
close_devices_android(devices)
put close_devices_android(...) in finally section. (Because "Caller MUST call
close_devices_android")

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/too...
File appengine/swarming/tools/android/51-android.rules (right):

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/too...
appengine/swarming/tools/android/51-android.rules:4: #Google
Google what?

[M-A Ruel, 2015-08-21 23:37:27]

Patchset #2 (id:120001) has been deleted

[M-A Ruel, 2015-08-21 23:37:39]

Code is live on chromium-swarm-android

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
File appengine/swarming/swarming_bot/adb (right):

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/adb:1: ../../third_party/python-adb/adb
On 2015/08/20 22:40:08, Vadim Sh. wrote:
> why symlinks instead of sys.path hack? sys.path hack is more conventional..
> Other third_party libs are added via sys.path.

Done.

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
File appengine/swarming/swarming_bot/os_utilities.py (right):

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1223: def
initialize_android(pub_key, priv_key):
On 2015/08/20 22:40:09, Vadim Sh. wrote:
> how do you feel about moving it to "android_utilities.py"? os_utilities is
> already huge.

Done.

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1227: ~/.android/adbkey.pub.
On 2015/08/20 22:40:08, Vadim Sh. wrote:
> is it true only for chrome-infra bots?

It's created by adb.

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1235: # Workaround pylint lint
errors.
On 2015/08/20 22:40:09, Vadim Sh. wrote:
> just disable them, it will look less weird

Not necessary anymore.

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1252: path =
os.path.expanduser('~/.android/adbkey')
On 2015/08/20 22:40:08, Vadim Sh. wrote:
> is it intentionally different from priv_key? What is priv_key then? Add a doc
> string for priv_key and pub_key.

The private and public keys for adb are stored in separate files.
The private key is stored in a standard format but the public key has its own
encoding.

The adb package allows multiple keys. We'd have one fleet wise and a backup key
of adb, so users that authenticated with their key will still work just fine.

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1343:
subprocess.call(['killall', 'adb'])
On 2015/08/20 22:40:08, Vadim Sh. wrote:
> killall --exact adb
> 
> (In general I'm not a fan of blindly killing processes)

Added --exact. But that's adb, it needs to be killed afaik.

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1368: # holds an handle to the
USB ports or group acl (plugdev) hasn't been
On 2015/08/20 22:40:09, Vadim Sh. wrote:
> typo: a handle

Done.

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1370: handle.Open()
On 2015/08/20 22:40:09, Vadim Sh. wrote:
> maybe at least log exception here and ignore this device?

Done.

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1374: handle, banner='swarming',
rsa_keys=_ADB_KEYS, auth_timeout_ms=100)
On 2015/08/20 22:40:08, Vadim Sh. wrote:
> 100ms? Is it really that fast?

Yes.

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1403: device.Close()
On 2015/08/20 22:40:09, Vadim Sh. wrote:
> catch and log exceptions here? (so that all devices are closed even if one
> failed)

I don't know which exception type would be thrown (thanks python) and I don't
expect exceptions to be thrown here.

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:1898:
close_devices_android(devices)
On 2015/08/20 22:40:08, Vadim Sh. wrote:
> put close_devices_android(...) in finally section. (Because "Caller MUST call
> close_devices_android")

Done.

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/too...
File appengine/swarming/tools/android/51-android.rules (right):

https://codereview.chromium.org/1306633002/diff/100001/appengine/swarming/too...
appengine/swarming/tools/android/51-android.rules:4: #Google
On 2015/08/20 22:40:09, Vadim Sh. wrote:
> Google what?

added

# Google Vendor ID

[Vadim Sh., 2015-08-21 23:50:27]

lgtm

https://codereview.chromium.org/1306633002/diff/140001/appengine/swarming/swa...
File appengine/swarming/swarming_bot/os_utilities.py (right):

https://codereview.chromium.org/1306633002/diff/140001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:728: def
get_dimensions_all_devices_android(cmds):
what is "cmds"? Why not "devices"?

[M-A Ruel, 2015-08-22 01:38:05]

https://codereview.chromium.org/1306633002/diff/140001/appengine/swarming/swa...
File appengine/swarming/swarming_bot/os_utilities.py (right):

https://codereview.chromium.org/1306633002/diff/140001/appengine/swarming/swa...
appengine/swarming/swarming_bot/os_utilities.py:728: def
get_dimensions_all_devices_android(cmds):
On 2015/08/21 23:50:27, Vadim Sh. wrote:
> what is "cmds"? Why not "devices"?

Yeah cmds is a bit obscure. it's a dict of devices, anyhow I've renamed to
devices for now. I may refactor this part later.

[M-A Ruel, 2015-08-22 12:47:15]

The CQ bit was checked by maruel@chromium.org

[M-A Ruel, 2015-08-22 12:47:15]

The patchset sent to the CQ was uploaded after l-g-t-m from vadimsh@chromium.org
Link to the patchset: https://codereview.chromium.org/1306633002/#ps160001
(title: "Use devices instead of cmds")

[commit-bot: I haz the power, 2015-08-22 12:47:22]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1306633002/160001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1306633002/160001

[commit-bot: I haz the power, 2015-08-22 12:50:54]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-08-22 12:50:55]

Try jobs failed on following builders:
  Luci-py Presubmit on master.tryserver.infra (JOB_FAILED,
https://build.chromium.org/p/tryserver.infra/builders/Luci-py%20Presubmit/bui...)

[M-A Ruel, 2015-08-22 12:57:22]

On 2015/08/22 12:50:55, commit-bot: I haz the power wrote:
> Try jobs failed on following builders:
>   Luci-py Presubmit on master.tryserver.infra (JOB_FAILED,
>
https://build.chromium.org/p/tryserver.infra/builders/Luci-py%20Presubmit/bui...)

Oops, I thought I had migrated the libusb1 symlink but I forgot, fixing.

[M-A Ruel, 2015-08-24 13:23:03]

Redid the layout so libusb1 is a relative import.

[M-A Ruel, 2015-08-24 20:58:51]

On 2015/08/24 13:23:03, M-A Ruel wrote:
> Redid the layout so libusb1 is a relative import.

Sorry for the nagging but if you want to take a look at the hack I added in
python-adb so I could package python-libusb1 properly, that would be
appreciated. The nice result is that I don't have to hack sys.path.

[Vadim Sh., 2015-08-24 21:34:25]

lgtm

[M-A Ruel, 2015-08-24 22:01:49]

The CQ bit was checked by maruel@chromium.org

[commit-bot: I haz the power, 2015-08-24 22:01:58]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1306633002/200001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1306633002/200001

[commit-bot: I haz the power, 2015-08-24 22:03:38]

Committed patchset #5 (id:200001) as
https://github.com/luci/luci-py/commit/43605c224e3aa1afa0f382200b4f46f51492987d