Install sdk extras packages using android to prompt for ToS.

Install sdk extras packages using android to prompt for ToS.

BUG=350151
Committed: https://crrev.com/cc789e8e3697aa4875fb459f4878fcb9f66d09d8
Cr-Commit-Position: refs/heads/master@{#310837}

[navabi, 2015-01-07 03:23:20]

navabi@google.com changed reviewers:
+ aurimas@chromium.org, cjhopman@chromium.org, feng@chromium.org,
nyquist@chromium.org

[navabi, 2015-01-07 03:25:38]

Let me know what you think about the general approach of using the DEPS file to
determine what packages to download.

https://codereview.chromium.org/838733002/diff/1/build/get_sdk_extras_package...
File build/get_sdk_extras_packages.py (right):

https://codereview.chromium.org/838733002/diff/1/build/get_sdk_extras_package...
build/get_sdk_extras_packages.py:10: def get_sdk_extras(hooks):
Need some comments here about how I am looking in the dictionary defined by the
DEPS file to find the hook with the name 'sdkextras', and from that taking the
list of arguments to the download_sdk_extras.py script (e.g.
google_google_play_services_21.0.zip) and from that determining the name of the
package (e.g. google play services).

The name will be used by sdk/tools/android update to install the corresponding
package.

[cjhopman, 2015-01-07 03:57:50]

https://codereview.chromium.org/838733002/diff/1/build/get_sdk_extras_package...
File build/get_sdk_extras_packages.py (right):

https://codereview.chromium.org/838733002/diff/1/build/get_sdk_extras_package...
build/get_sdk_extras_packages.py:10: def get_sdk_extras(hooks):
On 2015/01/07 03:25:38, navabi wrote:
> Need some comments here about how I am looking in the dictionary defined by
the
> DEPS file to find the hook with the name 'sdkextras', and from that taking the
> list of arguments to the download_sdk_extras.py script (e.g.
> google_google_play_services_21.0.zip) and from that determining the name of
the
> package (e.g. google play services).
> 
> The name will be used by sdk/tools/android update to install the corresponding
> package.

What about maintaining the list of these packages in a separate (json?) file?
and then read that file for this step and pass it as an argument to
download_sdk_extras.py in DEPS?

You could try doing them as more structured data instead of just a zip path...
something like:

{
  'dir_name': 'google',
  'package': 'google_play_services',
  'version': '1.0.0',
  'zip': 'google_google_play_services_1.0.0.zip',
}

but I don't know if that would make these scripts and such any simpler or not.

[navabi, 2015-01-07 16:41:18]

https://codereview.chromium.org/838733002/diff/1/build/get_sdk_extras_package...
File build/get_sdk_extras_packages.py (right):

https://codereview.chromium.org/838733002/diff/1/build/get_sdk_extras_package...
build/get_sdk_extras_packages.py:10: def get_sdk_extras(hooks):
On 2015/01/07 03:57:49, cjhopman wrote:
> On 2015/01/07 03:25:38, navabi wrote:
> > Need some comments here about how I am looking in the dictionary defined by
> the
> > DEPS file to find the hook with the name 'sdkextras', and from that taking
the
> > list of arguments to the download_sdk_extras.py script (e.g.
> > google_google_play_services_21.0.zip) and from that determining the name of
> the
> > package (e.g. google play services).
> > 
> > The name will be used by sdk/tools/android update to install the
corresponding
> > package.
> 
> What about maintaining the list of these packages in a separate (json?) file?
> and then read that file for this step and pass it as an argument to
> download_sdk_extras.py in DEPS?
> 
> You could try doing them as more structured data instead of just a zip path...
> something like:
> 
> {
>   'dir_name': 'google',
>   'package': 'google_play_services',
>   'version': '1.0.0',
>   'zip': 'google_google_play_services_1.0.0.zip',
> }
> 
> but I don't know if that would make these scripts and such any simpler or not.

That's a good suggestion, because I don't like that i have to do a 'python -c'
in the install-build-deps-android.sh just to get the package name. For that
reason, I'm thinking of just using a text file that is easily grep'able in bash
rather than a json file, which I will not know how to read from bash without
again doing a 'python -c' (which would at least be an easier 'python -c' command
than the one in this patch).

[nyquist, 2015-01-07 17:37:32]

So, I talked to Xavier on the Android SDK team, and he suggested we'd use the
command line tool to download the latest version of 'Android Repository' and
'Google Repository'.

You would then get updated versions of the folders:
/extras/google/m2repository
/extras/android/m2repository

Those folders are two different full maven repositories which we can use to
depend on. All versions of all the packages are there, which makes it easier to
strip out parts of Google Play Services that we do not need, and we can specify
which version.

There are a few ways of doing this:
1. We could create som pom.xml parser and automatically generate GYP-targets
that mimic them, or do this on the fly during build time.
2. We could manually follow the dependencies and just update the
google_play_services GYP target to include all the dependencies it needs all the
way down.

In this new world there are aar files instead of .jar files, but they are still
.zip files that we should be able to use on the classpath I hope.

[cjhopman, 2015-01-07 22:00:46]

On 2015/01/07 17:37:32, nyquist wrote:
> So, I talked to Xavier on the Android SDK team, and he suggested we'd use the
> command line tool to download the latest version of 'Android Repository' and
> 'Google Repository'.
> 
> You would then get updated versions of the folders:
> /extras/google/m2repository
> /extras/android/m2repository
> 
> Those folders are two different full maven repositories which we can use to
> depend on. All versions of all the packages are there, which makes it easier
to
> strip out parts of Google Play Services that we do not need, and we can
specify
> which version.
> 
> There are a few ways of doing this:
> 1. We could create som pom.xml parser and automatically generate GYP-targets
> that mimic them, or do this on the fly during build time.
> 2. We could manually follow the dependencies and just update the
> google_play_services GYP target to include all the dependencies it needs all
the
> way down.
> 
> In this new world there are aar files instead of .jar files, but they are
still
> .zip files that we should be able to use on the classpath I hope.

That sounds like something that would be good to put in the bug. My
understanding is that this approach that navabi@ is taking now is just
short-term to unblock other stuff.

[navabi, 2015-01-07 22:16:24]

> That sounds like something that would be good to put in the bug. My
> understanding is that this approach that navabi@ is taking now is just
> short-term to unblock other stuff.

That is correct, but if the clank team figures out how to use the aar files
instead of jar files in this new world, I could easily switch the downloading of
google_play_services to instead download the Android and Google repos.
Otherwise, we can continue with this approach, and make the switch later. From
my discussion with nyquist@, it seems the issue is that the way we are currently
doing it, we can't guarantee install-build-deps-android.sh always installs the
same version of the extras packages that we have in google storage, as the
packages are not backwards compatible.

[nyquist, 2015-01-07 22:25:16]

Discussed with cjhopman@ offline. Let's land this with the current approach
first to unblock people, and then see afterwards what we can do for a long-term
approach.

[Feng Qian, 2015-01-07 22:27:52]

On 2015/01/07 22:25:16, nyquist wrote:
> Discussed with cjhopman@ offline. Let's land this with the current approach
> first to unblock people, and then see afterwards what we can do for a
long-term
> approach.

Thanks you guys for working on the issue to unblock upstreaming!

[navabi, 2015-01-07 23:47:29]

On 2015/01/07 03:57:50, cjhopman wrote:
>
https://codereview.chromium.org/838733002/diff/1/build/get_sdk_extras_package...
> File build/get_sdk_extras_packages.py (right):
> 
>
https://codereview.chromium.org/838733002/diff/1/build/get_sdk_extras_package...
> build/get_sdk_extras_packages.py:10: def get_sdk_extras(hooks):
> On 2015/01/07 03:25:38, navabi wrote:
> > Need some comments here about how I am looking in the dictionary defined by
> the
> > DEPS file to find the hook with the name 'sdkextras', and from that taking
the
> > list of arguments to the download_sdk_extras.py script (e.g.
> > google_google_play_services_21.0.zip) and from that determining the name of
> the
> > package (e.g. google play services).
> > 
> > The name will be used by sdk/tools/android update to install the
corresponding
> > package.
> 
> What about maintaining the list of these packages in a separate (json?) file?
> and then read that file for this step and pass it as an argument to
> download_sdk_extras.py in DEPS?

I went ahead and changed this to use a json file. That way we can do a simple
"python -c 'import json; json.load...'" command in install build deps to
determine what package(s) to download. Please review that CL first. Once that
lands, I'll fix this.

[navabi, 2015-01-08 21:23:41]

navabi@google.com changed reviewers:
+ friedman@chromium.org

[navabi, 2015-01-08 21:23:42]

PTAL. Added friedman@ for all things bash.

[friedman1, 2015-01-08 22:06:41]

friedman@google.com changed reviewers:
+ friedman@google.com

[friedman1, 2015-01-08 22:06:42]

https://codereview.chromium.org/838733002/diff/20001/build/install-build-deps...
File build/install-build-deps-android.sh (right):

https://codereview.chromium.org/838733002/diff/20001/build/install-build-deps...
build/install-build-deps-android.sh:90:
package_num=$(../third_party/android_tools/sdk/tools/android list sdk \
Move the pipes to the prev line so the next line is just the binary:

  package_num=$(../third_party/android_tools/sdk/tools/android list sdk | \
    grep -i "$package," | \
    awk '/^[ ]*[0-9]*- / {gsub("-",""); print $1}')

[navabi, 2015-01-09 01:14:52]

https://codereview.chromium.org/838733002/diff/20001/build/install-build-deps...
File build/install-build-deps-android.sh (right):

https://codereview.chromium.org/838733002/diff/20001/build/install-build-deps...
build/install-build-deps-android.sh:90:
package_num=$(../third_party/android_tools/sdk/tools/android list sdk \
On 2015/01/08 22:06:41, friedman1 wrote:
> Move the pipes to the prev line so the next line is just the binary:
> 
>   package_num=$(../third_party/android_tools/sdk/tools/android list sdk | \
>     grep -i "$package," | \
>     awk '/^[ ]*[0-9]*- / {gsub("-",""); print $1}')

Done.

[cjhopman, 2015-01-09 02:55:01]

lgtm

[friedman1, 2015-01-09 18:44:35]

lgtm

[nyquist, 2015-01-09 18:50:21]

lgtm

[navabi, 2015-01-09 19:22:26]

The CQ bit was checked by navabi@google.com

[commit-bot: I haz the power, 2015-01-09 19:23:07]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/838733002/40001

[commit-bot: I haz the power, 2015-01-09 20:07:22]

Committed patchset #3 (id:40001)

[commit-bot: I haz the power, 2015-01-09 20:08:39]

Patchset 3 (id:??) landed as
https://crrev.com/cc789e8e3697aa4875fb459f4878fcb9f66d09d8
Cr-Commit-Position: refs/heads/master@{#310837}