[Android] Rework multidex and enable multidex for unit_tests_apk.

[Android] Rework multidex and enable multidex for unit_tests_apk.

This allows multidex to be used in release builds.

BUG=272790
Committed: https://crrev.com/f743bef34d49987c0f1f69e2f0bb6cb8beb03bf5
Cr-Commit-Position: refs/heads/master@{#369715}

[jbudorick, 2016-01-12 03:34:47]

jbudorick@chromium.org changed reviewers:
+ agrieve@chromium.org, dpranke@chromium.org, thakis@chromium.org,
yfriedman@chromium.org

[jbudorick, 2016-01-12 03:34:48]

The unit_tests_apk target is currently over 63k java methods in both debug and
release builds, and the upcoming GMS update pushes it over the dex limit. :(

ptal:
yfriedman: base/android/, base/test/android/
agrieve: build/
thakis: base/{BUILD.gn,base.gyp}, chrome/
dpranke: testing/

thanks!

[Nico, 2016-01-12 03:42:12]

should some of the gni changes in https://codereview.chromium.org/1451483002/ be
undone, or are they still needed?

https://codereview.chromium.org/1581563003/diff/1/base/base.gyp
File base/base.gyp (left):

https://codereview.chromium.org/1581563003/diff/1/base/base.gyp#oldcode1492
base/base.gyp:1492: 'additional_gcc_preprocess_options': [
can this
https://code.google.com/p/chromium/codesearch#chromium/src/build/android/java...
go now?

[agrieve, 2016-01-12 03:48:28]

https://codereview.chromium.org/1581563003/diff/1/build/config/android/rules.gni
File build/config/android/rules.gni (right):

https://codereview.chromium.org/1581563003/diff/1/build/config/android/rules....
build/config/android/rules.gni:1497: _srcjar_deps += [
":${_template_name}__multidex_config_java" ]
I think this adds ChromiumMultidex.java to *every* android_apk. Not sure we want
that... If we want this to work so that the file is only included when base_java
is included, then it'll need to add logic to write_build_config.py I think. 

I think maybe the best thing to do is add support for generating a BuildConfig
class like Gradle does. I could certainly help with that. WDYT?

[jbudorick, 2016-01-12 04:48:12]

thakis: the gni changes from https://codereview.chromium.org/1451483002/ are
still needed.

https://codereview.chromium.org/1581563003/diff/1/base/base.gyp
File base/base.gyp (left):

https://codereview.chromium.org/1581563003/diff/1/base/base.gyp#oldcode1492
base/base.gyp:1492: 'additional_gcc_preprocess_options': [
On 2016/01/12 03:42:12, Nico wrote:
> can this
>
https://code.google.com/p/chromium/codesearch#chromium/src/build/android/java...
> go now?

it can. removed.

https://codereview.chromium.org/1581563003/diff/1/build/config/android/rules.gni
File build/config/android/rules.gni (right):

https://codereview.chromium.org/1581563003/diff/1/build/config/android/rules....
build/config/android/rules.gni:1497: _srcjar_deps += [
":${_template_name}__multidex_config_java" ]
Didn't do anything here yet pending further discussion.

On 2016/01/12 03:48:28, agrieve wrote:
> I think this adds ChromiumMultidex.java to *every* android_apk. Not sure we
want
> that... If we want this to work so that the file is only included when
base_java
> is included, then it'll need to add logic to write_build_config.py I think. 

It does, though ChromiumMultiDex.java is now just a single static function that
imports nothing. There are a few APKs that don't appear to depend on base_java
(mostly customtabs related), but it's definitely an edge case at the moment.

> 
> I think maybe the best thing to do is add support for generating a BuildConfig
> class like Gradle does. I could certainly help with that. WDYT?

I'm not particularly familiar with Gradle, but at a glance that seems reasonable
to me. What would we put in there? MultiDex + NativeLibraries + ?

[agrieve, 2016-01-12 20:25:46]

lgtm

https://codereview.chromium.org/1581563003/diff/1/build/config/android/rules.gni
File build/config/android/rules.gni (right):

https://codereview.chromium.org/1581563003/diff/1/build/config/android/rules....
build/config/android/rules.gni:1497: _srcjar_deps += [
":${_template_name}__multidex_config_java" ]
On 2016/01/12 04:48:11, jbudorick wrote:
> Didn't do anything here yet pending further discussion.
> 
> On 2016/01/12 03:48:28, agrieve wrote:
> > I think this adds ChromiumMultidex.java to *every* android_apk. Not sure we
> want
> > that... If we want this to work so that the file is only included when
> base_java
> > is included, then it'll need to add logic to write_build_config.py I think. 
> 
> It does, though ChromiumMultiDex.java is now just a single static function
that
> imports nothing. There are a few APKs that don't appear to depend on base_java
> (mostly customtabs related), but it's definitely an edge case at the moment.
> 
> > 
> > I think maybe the best thing to do is add support for generating a
BuildConfig
> > class like Gradle does. I could certainly help with that. WDYT?
> 
> I'm not particularly familiar with Gradle, but at a glance that seems
reasonable
> to me. What would we put in there? MultiDex + NativeLibraries + ?

Ironically, the best docs I could find are from buck:
https://buckbuild.com/rule/android_build_config.html

For gradle / ant apps, the build injects a class called
YOUR_APP_PACKAGE.BuildConfig, which by default just has DEBUG = true / false. We
could add this and extend it to also include ENABLE_MULTIDEX.

The annoying thing about it though is that to access the var from base/, you'd
need to extract the app's package from a Context and use reflection to find the
BuildConfig class.

So... perhaps it isn't actually any better.

Other ideas:
1. Use a <meta-data> in the AndroidManifest.xml instead. Not sure if for some
reason this is slow to access at runtime.
2. Use a class that you know will never end up in the primary dex file, try to
access it, and if it fails you know you need multidex.

I don't think I like either idea better than just adding a class to every apk...
So, lgtm and I'll continue to think about this.

[Nico, 2016-01-12 20:29:27]

rs-lgtm

[Yaron, 2016-01-12 22:43:22]

lgtm

[dgn, 2016-01-14 14:32:22]

Description was changed from

==========
[Android] Rework multidex and enable multidex for unit_tests_apk.

This allows multidex to be used in release builds.

BUG=272790
==========

to

==========
[Android] Rework multidex and enable multidex for unit_tests_apk.

This allows multidex to be used in release builds.

BUG=272790
==========

[jbudorick, 2016-01-14 14:38:28]

jbudorick@chromium.org changed reviewers:
+ phajdan.jr@chromium.org

[jbudorick, 2016-01-14 14:38:28]

+phajdan for testing/ owners

[Paweł Hajdan Jr., 2016-01-14 15:43:14]

testing LGTM

[jbudorick, 2016-01-14 23:36:24]

The CQ bit was checked by jbudorick@chromium.org

[jbudorick, 2016-01-14 23:36:25]

The patchset sent to the CQ was uploaded after l-g-t-m from
yfriedman@chromium.org, thakis@chromium.org, agrieve@chromium.org,
phajdan.jr@chromium.org
Link to the patchset: https://codereview.chromium.org/1581563003/#ps40001
(title: "rebase")

[commit-bot: I haz the power, 2016-01-14 23:37:51]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1581563003/40001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1581563003/40001

[jbudorick, 2016-01-15 01:42:17]

The CQ bit was unchecked by jbudorick@chromium.org

[jbudorick, 2016-01-15 05:07:46]

The CQ bit was checked by jbudorick@chromium.org

[jbudorick, 2016-01-15 05:07:46]

The patchset sent to the CQ was uploaded after l-g-t-m from
yfriedman@chromium.org, thakis@chromium.org, agrieve@chromium.org,
phajdan.jr@chromium.org
Link to the patchset: https://codereview.chromium.org/1581563003/#ps60001
(title: "don't generate ChromiumMultiDex.java for instrumentation test apks")

[commit-bot: I haz the power, 2016-01-15 05:07:59]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1581563003/60001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1581563003/60001

[commit-bot: I haz the power, 2016-01-15 05:35:19]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-01-15 05:35:19]

Try jobs failed on following builders:
  linux_android_rel_ng on tryserver.chromium.android (JOB_FAILED,
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_androi...)

[jbudorick, 2016-01-15 06:19:37]

The CQ bit was checked by jbudorick@chromium.org

[jbudorick, 2016-01-15 06:19:38]

The patchset sent to the CQ was uploaded after l-g-t-m from
yfriedman@chromium.org, thakis@chromium.org, agrieve@chromium.org,
phajdan.jr@chromium.org
Link to the patchset: https://codereview.chromium.org/1581563003/#ps80001
(title: "fix gyp")

[commit-bot: I haz the power, 2016-01-15 06:19:46]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1581563003/80001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1581563003/80001

[commit-bot: I haz the power, 2016-01-15 06:40:40]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-01-15 06:40:41]

Try jobs failed on following builders:
  android_clang_dbg_recipe on tryserver.chromium.android (JOB_FAILED,
https://build.chromium.org/p/tryserver.chromium.android/builders/android_clan...)

[jbudorick, 2016-01-15 08:54:03]

The CQ bit was checked by jbudorick@chromium.org

[jbudorick, 2016-01-15 08:54:03]

The patchset sent to the CQ was uploaded after l-g-t-m from
yfriedman@chromium.org, thakis@chromium.org, agrieve@chromium.org,
phajdan.jr@chromium.org
Link to the patchset: https://codereview.chromium.org/1581563003/#ps120001
(title: "fix junit tests")

[commit-bot: I haz the power, 2016-01-15 08:54:26]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1581563003/120001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1581563003/120001

[commit-bot: I haz the power, 2016-01-15 09:59:19]

Description was changed from

==========
[Android] Rework multidex and enable multidex for unit_tests_apk.

This allows multidex to be used in release builds.

BUG=272790
==========

to

==========
[Android] Rework multidex and enable multidex for unit_tests_apk.

This allows multidex to be used in release builds.

BUG=272790
==========

[commit-bot: I haz the power, 2016-01-15 09:59:20]

Committed patchset #7 (id:120001)

[commit-bot: I haz the power, 2016-01-15 10:00:12]

Description was changed from

==========
[Android] Rework multidex and enable multidex for unit_tests_apk.

This allows multidex to be used in release builds.

BUG=272790
==========

to

==========
[Android] Rework multidex and enable multidex for unit_tests_apk.

This allows multidex to be used in release builds.

BUG=272790

Committed: https://crrev.com/f743bef34d49987c0f1f69e2f0bb6cb8beb03bf5
Cr-Commit-Position: refs/heads/master@{#369715}
==========

[commit-bot: I haz the power, 2016-01-15 10:00:13]

Patchset 7 (id:??) landed as
https://crrev.com/f743bef34d49987c0f1f69e2f0bb6cb8beb03bf5
Cr-Commit-Position: refs/heads/master@{#369715}

[vasilii, 2016-01-15 10:17:59]

A revert of this CL (patchset #7 id:120001) has been created in
https://codereview.chromium.org/1587253003/ by vasilii@chromium.org.

The reason for reverting is: Broke compilation on Android 

https://build.chromium.org/p/chromium/builders/Android/builds/50665

FAILED: cd ../../components; python ../build/android/gyp/apk_obfuscate.py
--configuration-name Release --android-sdk
/b/build/slave/Android/build/src/third_party/android_tools/sdk//platforms/android-23
--android-sdk-tools
/b/build/slave/Android/build/src/third_party/android_tools/sdk//build-tools/23.0.1
--android-sdk-jar
/b/build/slave/Android/build/src/third_party/android_tools/sdk//platforms/android-23/android.jar
"--input-jars-paths=/b/build/slave/Android/build/src/third_party/android_tools/sdk//extras/android/support/multidex/library/libs/android-support-multidex.jar
\"../out/Release/lib.java/jsr_305_javalib.jar\"
\"../out/Release/lib.java/base_java.jar\"
/b/build/slave/Android/build/src/third_party/android_tools/sdk//extras/android/support/annotations/android-support-annotations.jar
\"../out/Release/lib.java/cronet_api.jar\"
\"../out/Release/lib.java/url_java.jar\"
\"../out/Release/lib.java/net_java.jar\"
\"../out/Release/lib.java/cronet_java.jar\"
\"../out/Release/lib.java/chromium_apk_cronet_perf_test_apk.jar\""
"--proguard-configs=cronet/android/proguard.cfg
\"../out/Release/cronet_perf_test_apk/proguard.txt\"" --test-jar-path
../out/Release/test.lib.java/CronetPerfTest.jar --obfuscated-jar-path
../out/Release/cronet_perf_test_apk/obfuscated.jar --proguard-jar-path
../third_party/proguard/lib/proguard.jar --stamp
../out/Release/cronet_perf_test_apk/obfuscate.stamp --testapp --proguard-enabled
Traceback (most recent call last):
  File "../build/android/gyp/apk_obfuscate.py", line 185, in <module>
    sys.exit(main(sys.argv[1:]))
  File "../build/android/gyp/apk_obfuscate.py", line 167, in main
    DoProguard(options)
  File "../build/android/gyp/apk_obfuscate.py", line 124, in DoProguard
    proguard.CheckOutput()
  File
"/b/build/slave/Android/build/src/build/android/gyp/util/proguard_util.py", line
180, in CheckOutput
    stderr_filter=stderr_filter)
  File "/b/build/slave/Android/build/src/build/android/gyp/util/build_utils.py",
line 162, in CheckOutput
    raise CalledProcessError(cwd, args, stdout + stderr)
util.build_utils.CalledProcessError: Command failed: ( cd
/b/build/slave/Android/build/src/components; java -jar
../third_party/proguard/lib/proguard.jar -forceprocessing -libraryjars
/b/build/slave/Android/build/src/third_party/android_tools/sdk//platforms/android-23/android.jar
-injars
/b/build/slave/Android/build/src/third_party/android_tools/sdk//extras/android/support/multidex/library/libs/android-support-multidex.jar:../out/Release/lib.java/jsr_305_javalib.jar:../out/Release/lib.java/base_java.jar:/b/build/slave/Android/build/src/third_party/android_tools/sdk//extras/android/support/annotations/android-support-annotations.jar:../out/Release/lib.java/cronet_api.jar:../out/Release/lib.java/url_java.jar:../out/Release/lib.java/net_java.jar:../out/Release/lib.java/cronet_java.jar:../out/Release/lib.java/chromium_apk_cronet_perf_test_apk.jar
-include cronet/android/proguard.cfg -include
../out/Release/cronet_perf_test_apk/proguard.txt -outjars
../out/Release/cronet_perf_test_apk/obfuscated.jar -dump
../out/Release/cronet_perf_test_apk/obfuscated.jar.dump -printseeds
../out/Release/cronet_perf_test_apk/obfuscated.jar.seeds -printusage
../out/Release/cronet_perf_test_apk/obfuscated.jar.usage -printmapping
../out/Release/cronet_perf_test_apk/obfuscated.jar.mapping )
Warning: org.chromium.base.multidex.ChromiumMultiDexInstaller: can't find
referenced class org.chromium.base.multidex.ChromiumMultiDex
Warning: org.chromium.base.multidex.ChromiumMultiDexInstaller: can't find
referenced class org.chromium.base.multidex.ChromiumMultiDex
Warning: there were 2 unresolved references to classes or interfaces.
         You may need to specify additional library jars (using '-libraryjars').
Error: Please correct the above warnings first..