Make javac and jar a single build action

Make javac and jar a single build action

This allows us to more accurately specify the outputs of each action (we
can't specify the .class file outputs easily because determing them
essentially requires compiling the .java files).

The lint action still operates directly on the .class files, so we continue to
support specifying a directory for the .class files (but in a very simple to
remove way).

BUG=359249
Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=279152

[cjhopman, 2014-06-13 21:33:44]

newt: *

[newt (away), 2014-06-16 18:48:47]

https://codereview.chromium.org/328893003/diff/40001/build/android/gyp/jar.py
File build/android/gyp/jar.py (right):

https://codereview.chromium.org/328893003/diff/40001/build/android/gyp/jar.py...
build/android/gyp/jar.py:1: #!/usr/bin/env python
do we still use jar.py's main function anywhere?

https://codereview.chromium.org/328893003/diff/40001/build/java.gypi
File build/java.gypi (right):

https://codereview.chromium.org/328893003/diff/40001/build/java.gypi#newcode72
build/java.gypi:72: 'classes_dir': '<(intermediate_dir)/classes/1/',
why /1/ ?

[cjhopman, 2014-06-23 14:48:43]

https://codereview.chromium.org/328893003/diff/40001/build/java.gypi
File build/java.gypi (right):

https://codereview.chromium.org/328893003/diff/40001/build/java.gypi#newcode72
build/java.gypi:72: 'classes_dir': '<(intermediate_dir)/classes/1/',
On 2014/06/16 18:48:46, newt wrote:
> why /1/ ?
Done. I was just confirming to myself that none of the steps below had
constructed the classes dir from <(intermediate_dir) themselves.

[newt (away), 2014-06-23 15:41:56]

lgtm

[cjhopman, 2014-06-23 15:44:44]

The CQ bit was checked by cjhopman@chromium.org

[commit-bot: I haz the power, 2014-06-23 15:45:43]

CQ is trying da patch. Follow status at
 https://chromium-status.appspot.com/cq/cjhopman@chromium.org/328893003/60001

[cjhopman, 2014-06-23 16:08:51]

https://codereview.chromium.org/328893003/diff/40001/build/android/gyp/jar.py
File build/android/gyp/jar.py (right):

https://codereview.chromium.org/328893003/diff/40001/build/android/gyp/jar.py...
build/android/gyp/jar.py:1: #!/usr/bin/env python
On 2014/06/16 18:48:46, newt wrote:
> do we still use jar.py's main function anywhere?
Sorry forgot to reply to this.

Yes, components/cronet uses it (they do some weird stuff that I should figure
out).

[commit-bot: I haz the power, 2014-06-23 18:42:18]

Change committed as 279152

[Nico, 2014-06-23 21:31:46]

All my android try results (e.g.
http://build.chromium.org/p/tryserver.chromium/builders/android_dbg/builds/19...
) come back with stacks like

FAILED: cd ../../build/android/tests/multiple_proguards; python
../../../../build/android/gyp/dex.py
"--dex-path=../../../../out/Debug/multiple_proguards_test_apk/classes.dex"
"--android-sdk-tools=/mnt/scratch0/b_used/build/slave/android/build/src/third_party/android_tools/sdk//build-tools/19.0.0"
"--configuration-name=Debug" "--proguard-enabled=true"
"--proguard-enabled-input-path=../../../../out/Debug/multiple_proguards_test_apk/obfuscated.jar"
"--no-locals=0" ../../../../out/Debug/lib.java/guava_javalib.dex.jar
../../../../out/Debug/lib.java/jsr_305_javalib.dex.jar
../../../../out/Debug/lib.java/base_java.dex.jar
../../../../out/Debug/lib.java/chromium_apk_multiple_proguards_test_apk.jar
Traceback (most recent call last):
  File "../../../../build/android/gyp/dex.py", line 65, in <module>
    sys.exit(main())
  File "../../../../build/android/gyp/dex.py", line 61, in main
    DoDex(options, paths)
  File "../../../../build/android/gyp/dex.py", line 29, in DoDex
    force=not os.path.exists(options.dex_path))
  File
"/mnt/scratch0/b_used/build/slave/android/build/src/build/android/gyp/util/md5_check.py",
line 30, in CallAndRecordIfStale
    function()
  File "../../../../build/android/gyp/dex.py", line 25, in <lambda>
    lambda: build_utils.CheckOutput(dex_cmd, print_stderr=False),
  File
"/mnt/scratch0/b_used/build/slave/android/build/src/build/android/gyp/util/build_utils.py",
line 139, in CheckOutput
    raise CalledProcessError(cwd, args, stdout + stderr)
util.build_utils.CalledProcessError: Command failed: ( cd
/mnt/scratch0/b_used/build/slave/android/build/src/build/android/tests/multiple_proguards;
/mnt/scratch0/b_used/build/slave/android/build/src/third_party/android_tools/sdk//build-tools/19.0.0/dx
--dex --force-jumbo --output
../../../../out/Debug/multiple_proguards_test_apk/classes.dex
../../../../out/Debug/lib.java/guava_javalib.dex.jar
../../../../out/Debug/lib.java/jsr_305_javalib.dex.jar
../../../../out/Debug/lib.java/base_java.dex.jar
../../../../out/Debug/lib.java/chromium_apk_multiple_proguards_test_apk.jar )

UNEXPECTED TOP-LEVEL EXCEPTION:
com.android.dx.cf.iface.ParseException: class name (dummy/DummyActivity) does
not match path (2/dummy/DummyActivity.class)
	at com.android.dx.cf.direct.DirectClassFile.parse0(DirectClassFile.java:520)
	at com.android.dx.cf.direct.DirectClassFile.parse(DirectClassFile.java:406)
	at
com.android.dx.cf.direct.DirectClassFile.parseToInterfacesIfNecessary(DirectClassFile.java:388)
	at com.android.dx.cf.direct.DirectClassFile.getMagic(DirectClassFile.java:251)
	at com.android.dx.command.dexer.Main.processClass(Main.java:665)
	at com.android.dx.command.dexer.Main.processFileBytes(Main.java:634)
	at com.android.dx.command.dexer.Main.access$600(Main.java:78)
	at com.android.dx.command.dexer.Main$1.processFileBytes(Main.java:572)
	at
com.android.dx.cf.direct.ClassPathOpener.processArchive(ClassPathOpener.java:284)
	at
com.android.dx.cf.direct.ClassPathOpener.processOne(ClassPathOpener.java:166)
	at com.android.dx.cf.direct.ClassPathOpener.process(ClassPathOpener.java:144)
	at com.android.dx.command.dexer.Main.processOne(Main.java:596)
	at com.android.dx.command.dexer.Main.processAllFiles(Main.java:498)
	at com.android.dx.command.dexer.Main.runMonoDex(Main.java:264)
	at com.android.dx.command.dexer.Main.run(Main.java:230)
	at com.android.dx.command.dexer.Main.main(Main.java:199)
	at com.android.dx.command.Main.main(Main.java:103)
...while parsing 2/dummy/DummyActivity.class


Is this due to this change?

[cjhopman, 2014-06-23 21:38:08]

On 2014/06/23 21:31:46, Nico (away) wrote:
> All my android try results (e.g.
>
http://build.chromium.org/p/tryserver.chromium/builders/android_dbg/builds/19...
> ) come back with stacks like
> 
> FAILED: cd ../../build/android/tests/multiple_proguards; python
> ../../../../build/android/gyp/dex.py
> "--dex-path=../../../../out/Debug/multiple_proguards_test_apk/classes.dex"
>
"--android-sdk-tools=/mnt/scratch0/b_used/build/slave/android/build/src/third_party/android_tools/sdk//build-tools/19.0.0"
> "--configuration-name=Debug" "--proguard-enabled=true"
>
"--proguard-enabled-input-path=../../../../out/Debug/multiple_proguards_test_apk/obfuscated.jar"
> "--no-locals=0" ../../../../out/Debug/lib.java/guava_javalib.dex.jar
> ../../../../out/Debug/lib.java/jsr_305_javalib.dex.jar
> ../../../../out/Debug/lib.java/base_java.dex.jar
> ../../../../out/Debug/lib.java/chromium_apk_multiple_proguards_test_apk.jar
> Traceback (most recent call last):
>   File "../../../../build/android/gyp/dex.py", line 65, in <module>
>     sys.exit(main())
>   File "../../../../build/android/gyp/dex.py", line 61, in main
>     DoDex(options, paths)
>   File "../../../../build/android/gyp/dex.py", line 29, in DoDex
>     force=not os.path.exists(options.dex_path))
>   File
>
"/mnt/scratch0/b_used/build/slave/android/build/src/build/android/gyp/util/md5_check.py",
> line 30, in CallAndRecordIfStale
>     function()
>   File "../../../../build/android/gyp/dex.py", line 25, in <lambda>
>     lambda: build_utils.CheckOutput(dex_cmd, print_stderr=False),
>   File
>
"/mnt/scratch0/b_used/build/slave/android/build/src/build/android/gyp/util/build_utils.py",
> line 139, in CheckOutput
>     raise CalledProcessError(cwd, args, stdout + stderr)
> util.build_utils.CalledProcessError: Command failed: ( cd
>
/mnt/scratch0/b_used/build/slave/android/build/src/build/android/tests/multiple_proguards;
>
/mnt/scratch0/b_used/build/slave/android/build/src/third_party/android_tools/sdk//build-tools/19.0.0/dx
> --dex --force-jumbo --output
> ../../../../out/Debug/multiple_proguards_test_apk/classes.dex
> ../../../../out/Debug/lib.java/guava_javalib.dex.jar
> ../../../../out/Debug/lib.java/jsr_305_javalib.dex.jar
> ../../../../out/Debug/lib.java/base_java.dex.jar
> ../../../../out/Debug/lib.java/chromium_apk_multiple_proguards_test_apk.jar )
> 
> UNEXPECTED TOP-LEVEL EXCEPTION:
> com.android.dx.cf.iface.ParseException: class name (dummy/DummyActivity) does
> not match path (2/dummy/DummyActivity.class)
> 	at com.android.dx.cf.direct.DirectClassFile.parse0(DirectClassFile.java:520)
> 	at com.android.dx.cf.direct.DirectClassFile.parse(DirectClassFile.java:406)
> 	at
>
com.android.dx.cf.direct.DirectClassFile.parseToInterfacesIfNecessary(DirectClassFile.java:388)
> 	at
com.android.dx.cf.direct.DirectClassFile.getMagic(DirectClassFile.java:251)
> 	at com.android.dx.command.dexer.Main.processClass(Main.java:665)
> 	at com.android.dx.command.dexer.Main.processFileBytes(Main.java:634)
> 	at com.android.dx.command.dexer.Main.access$600(Main.java:78)
> 	at com.android.dx.command.dexer.Main$1.processFileBytes(Main.java:572)
> 	at
>
com.android.dx.cf.direct.ClassPathOpener.processArchive(ClassPathOpener.java:284)
> 	at
> com.android.dx.cf.direct.ClassPathOpener.processOne(ClassPathOpener.java:166)
> 	at com.android.dx.cf.direct.ClassPathOpener.process(ClassPathOpener.java:144)
> 	at com.android.dx.command.dexer.Main.processOne(Main.java:596)
> 	at com.android.dx.command.dexer.Main.processAllFiles(Main.java:498)
> 	at com.android.dx.command.dexer.Main.runMonoDex(Main.java:264)
> 	at com.android.dx.command.dexer.Main.run(Main.java:230)
> 	at com.android.dx.command.dexer.Main.main(Main.java:199)
> 	at com.android.dx.command.Main.main(Main.java:103)
> ...while parsing 2/dummy/DummyActivity.class
> 
> 
> Is this due to this change?

afaict, that build synced to 279105 and this landed as 279152

[cjhopman, 2014-06-23 22:12:10]

On 2014/06/23 21:38:08, cjhopman wrote:
> On 2014/06/23 21:31:46, Nico (away) wrote:
> > All my android try results (e.g.
> >
>
http://build.chromium.org/p/tryserver.chromium/builders/android_dbg/builds/19...
> > ) come back with stacks like
> > 
> > FAILED: cd ../../build/android/tests/multiple_proguards; python
> > ../../../../build/android/gyp/dex.py
> > "--dex-path=../../../../out/Debug/multiple_proguards_test_apk/classes.dex"
> >
>
"--android-sdk-tools=/mnt/scratch0/b_used/build/slave/android/build/src/third_party/android_tools/sdk//build-tools/19.0.0"
> > "--configuration-name=Debug" "--proguard-enabled=true"
> >
>
"--proguard-enabled-input-path=../../../../out/Debug/multiple_proguards_test_apk/obfuscated.jar"
> > "--no-locals=0" ../../../../out/Debug/lib.java/guava_javalib.dex.jar
> > ../../../../out/Debug/lib.java/jsr_305_javalib.dex.jar
> > ../../../../out/Debug/lib.java/base_java.dex.jar
> > ../../../../out/Debug/lib.java/chromium_apk_multiple_proguards_test_apk.jar
> > Traceback (most recent call last):
> >   File "../../../../build/android/gyp/dex.py", line 65, in <module>
> >     sys.exit(main())
> >   File "../../../../build/android/gyp/dex.py", line 61, in main
> >     DoDex(options, paths)
> >   File "../../../../build/android/gyp/dex.py", line 29, in DoDex
> >     force=not os.path.exists(options.dex_path))
> >   File
> >
>
"/mnt/scratch0/b_used/build/slave/android/build/src/build/android/gyp/util/md5_check.py",
> > line 30, in CallAndRecordIfStale
> >     function()
> >   File "../../../../build/android/gyp/dex.py", line 25, in <lambda>
> >     lambda: build_utils.CheckOutput(dex_cmd, print_stderr=False),
> >   File
> >
>
"/mnt/scratch0/b_used/build/slave/android/build/src/build/android/gyp/util/build_utils.py",
> > line 139, in CheckOutput
> >     raise CalledProcessError(cwd, args, stdout + stderr)
> > util.build_utils.CalledProcessError: Command failed: ( cd
> >
>
/mnt/scratch0/b_used/build/slave/android/build/src/build/android/tests/multiple_proguards;
> >
>
/mnt/scratch0/b_used/build/slave/android/build/src/third_party/android_tools/sdk//build-tools/19.0.0/dx
> > --dex --force-jumbo --output
> > ../../../../out/Debug/multiple_proguards_test_apk/classes.dex
> > ../../../../out/Debug/lib.java/guava_javalib.dex.jar
> > ../../../../out/Debug/lib.java/jsr_305_javalib.dex.jar
> > ../../../../out/Debug/lib.java/base_java.dex.jar
> > ../../../../out/Debug/lib.java/chromium_apk_multiple_proguards_test_apk.jar
)
> > 
> > UNEXPECTED TOP-LEVEL EXCEPTION:
> > com.android.dx.cf.iface.ParseException: class name (dummy/DummyActivity)
does
> > not match path (2/dummy/DummyActivity.class)
> > 	at
com.android.dx.cf.direct.DirectClassFile.parse0(DirectClassFile.java:520)
> > 	at com.android.dx.cf.direct.DirectClassFile.parse(DirectClassFile.java:406)
> > 	at
> >
>
com.android.dx.cf.direct.DirectClassFile.parseToInterfacesIfNecessary(DirectClassFile.java:388)
> > 	at
> com.android.dx.cf.direct.DirectClassFile.getMagic(DirectClassFile.java:251)
> > 	at com.android.dx.command.dexer.Main.processClass(Main.java:665)
> > 	at com.android.dx.command.dexer.Main.processFileBytes(Main.java:634)
> > 	at com.android.dx.command.dexer.Main.access$600(Main.java:78)
> > 	at com.android.dx.command.dexer.Main$1.processFileBytes(Main.java:572)
> > 	at
> >
>
com.android.dx.cf.direct.ClassPathOpener.processArchive(ClassPathOpener.java:284)
> > 	at
> >
com.android.dx.cf.direct.ClassPathOpener.processOne(ClassPathOpener.java:166)
> > 	at
com.android.dx.cf.direct.ClassPathOpener.process(ClassPathOpener.java:144)
> > 	at com.android.dx.command.dexer.Main.processOne(Main.java:596)
> > 	at com.android.dx.command.dexer.Main.processAllFiles(Main.java:498)
> > 	at com.android.dx.command.dexer.Main.runMonoDex(Main.java:264)
> > 	at com.android.dx.command.dexer.Main.run(Main.java:230)
> > 	at com.android.dx.command.dexer.Main.main(Main.java:199)
> > 	at com.android.dx.command.Main.main(Main.java:103)
> > ...while parsing 2/dummy/DummyActivity.class
> > 
> > 
> > Is this due to this change?
> 
> afaict, that build synced to 279105 and this landed as 279152

Hm.

So, on android_clang_dbg:
http://build.chromium.org/p/tryserver.chromium/builders/android_clang_dbg/?nu...

The first such failure was 154943
http://build.chromium.org/p/tryserver.chromium/builders/android_clang_dbg/bui...
which was before this change even landed.... but it was after this changes
tryjob ran on that bot.

I'm going to revert this locally and see if I then get that failure.

[cjhopman, 2014-06-23 22:36:49]

I didn't reproduce this, but it is definitely due to this change. Syncing past
this and then back behind it can definitely cause that failure.

https://codereview.chromium.org/328893003/diff/60001/build/java_apk.gypi
File build/java_apk.gypi (right):

https://codereview.chromium.org/328893003/diff/60001/build/java_apk.gypi#newc...
build/java_apk.gypi:97: 'classes_dir': '<(intermediate_dir)/classes/2',
This is the bad line. We generate java files into /2. This means that during
this build we will delete everything in classes/2/ but not in classes/. So now,
classes/ contains duplicates of all the class files.

Then, when syncing back the other way, if no .java files change, we won't delete
this directory, but we will jar it. This happily grabs both copies of the .class
files and then fails dexing.

This is one of the problems with the previous way that we did javac+jar (we
leave random .class files all over the place that we have to be careful with
cleaning/moving). The new way (that this change does most of the work for) does
not have that problem. The .class files are generated in a temporary directory
and the output is a .jar of just the .class files from this build.