|
|
Created:
5 years, 8 months ago by Ian Cullinan Modified:
5 years, 7 months ago CC:
chromium-reviews, klundberg+watch_chromium.org, yfriedman+watch_chromium.org, jbudorick+watch_chromium.org Base URL:
https://chromium.googlesource.com/chromium/src.git@master Target Ref:
refs/pending/heads/master Project:
chromium Visibility:
Public. |
DescriptionActually use --extra-r-text-files in process_resources.py
Generate extra packages' R.java files only containing the resources
listed in their R.txt files. This significantly reduces the number of
DEX field IDs taken up by resources.
BUG=480036
R=cjhopman
Committed: https://crrev.com/a92cf235709397c737d5d9fdbadfb62e497c0334
Cr-Commit-Position: refs/heads/master@{#328565}
Patch Set 1 #Patch Set 2 : GN #
Total comments: 2
Patch Set 3 : Don't FindInDirectory R.txt #Patch Set 4 : Handle missing R.txt files #Patch Set 5 : Generate R.onResourcesLoaded() callback for --shared-resources #Patch Set 6 : Use correct R class in SyncCustomizationFragmentTest #
Total comments: 1
Messages
Total messages: 45 (10 generated)
PTAL
https://codereview.chromium.org/1104703003/diff/20001/build/android/gyp/proce... File build/android/gyp/process_resources.py (right): https://codereview.chromium.org/1104703003/diff/20001/build/android/gyp/proce... build/android/gyp/process_resources.py:112: r_txt_files = build_utils.FindInDirectory(r_dir, 'R.txt') Below you find the R.txt by just doing os.path.join(r_dir, 'R.txt'). why not do that here too?
https://codereview.chromium.org/1104703003/diff/20001/build/android/gyp/proce... File build/android/gyp/process_resources.py (right): https://codereview.chromium.org/1104703003/diff/20001/build/android/gyp/proce... build/android/gyp/process_resources.py:112: r_txt_files = build_utils.FindInDirectory(r_dir, 'R.txt') On 2015/04/27 18:34:47, cjhopman wrote: > Below you find the R.txt by just doing os.path.join(r_dir, 'R.txt'). why not do > that here too? Acknowledged.
lgtm
The CQ bit was checked by cullinan@amazon.com
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1104703003/30006
The CQ bit was unchecked by commit-bot@chromium.org
Try jobs failed on following builders: linux_chromium_gn_chromeos_rel on tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_...)
Looks like aapt doesn't generate R.java/R.txt when there are no resources. The current code deals with this via an early return from CreateExtraRJavaFiles if len(java_files) != 1. I've added checking for the existence of R.txt; PTAnotherL. Thanks, Ian
lgtm
The CQ bit was checked by cullinan@amazon.com
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1104703003/50001
The CQ bit was unchecked by commit-bot@chromium.org
Try jobs failed on following builders: android_compile_dbg on tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/android_compile...) linux_android_rel_ng on tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_android_r...)
Hi cjhopman, A couple more changes required for system_webview_apk: 1. Actually passing --custom-package through to process_resources.py in java_apk.gypi (the correct R.java was previously getting generated only accidentally via --extra-res-packages). 2. Generating the non-final fields and onResourcesLoaded() needed by library APKs. The Java-generating code was getting rather unwieldy so I turned it into a Jinja template which is shorter and hopefully easier to understand. PTAL (again, sorry) Thanks, Ian
lgtm
The CQ bit was checked by cullinan@amazon.com
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1104703003/70001
The CQ bit was unchecked by commit-bot@chromium.org
Try jobs failed on following builders: android_compile_dbg on tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/android_compile...) linux_android_rel_ng on tryserver.chromium.linux (JOB_FAILED, http://build.chromium.org/p/tryserver.chromium.linux/builders/linux_android_r...)
cullinan@amazon.com changed reviewers: + yfriedman@chromium.org
Hi Yaron, PTAL at chrome/android/sync_shell/javatests/src/chromium/chrome/browser/sync/SyncCustomizationFragmentTest.java As far as I can tell, this is the only file in Chromium that uses a resource ID from a package further down the dependency graph than that resource comes from (thus breaking the build when the rest of this CL is applied). Thanks, Ian
On 2015/05/01 22:46:28, Ian Cullinan wrote: > Hi Yaron, > > PTAL at > chrome/android/sync_shell/javatests/src/chromium/chrome/browser/sync/SyncCustomizationFragmentTest.java > > > As far as I can tell, this is the only file in Chromium that uses a resource ID > from a package further down the dependency graph than that resource comes from > (thus breaking the build when the rest of this CL is applied). > > Thanks, > Ian lgtm Sorry I missed it cause it was in my +watch list.
The CQ bit was checked by cullinan@amazon.com
The patchset sent to the CQ was uploaded after l-g-t-m from cjhopman@chromium.org Link to the patchset: https://codereview.chromium.org/1104703003/#ps90001 (title: "Use correct R class in SyncCustomizationFragmentTest")
CQ is trying da patch. Follow status at https://chromium-cq-status.appspot.com/patch-status/1104703003/90001
On 2015/05/06 16:56:13, Yaron wrote: > Sorry I missed it cause it was in my +watch list. No worries; thanks for the review.
Message was sent while issue was closed.
Committed patchset #6 (id:90001)
Message was sent while issue was closed.
Patchset 6 (id:??) landed as https://crrev.com/a92cf235709397c737d5d9fdbadfb62e497c0334 Cr-Commit-Position: refs/heads/master@{#328565}
Message was sent while issue was closed.
On 2015/05/06 18:34:30, I haz the power (commit-bot) wrote: > Patchset 6 (id:??) landed as > https://crrev.com/a92cf235709397c737d5d9fdbadfb62e497c0334 > Cr-Commit-Position: refs/heads/master@{#328565} This seems to have broken some of the internal Chromium-Android builds on our team---for example, strings that are part of our APK target cannot be found when compiling Java source files that reference them, R.mipmap.* not available, etc. What part of the Chromium-Android build system is typically generating the R.txt flies?
Message was sent while issue was closed.
On 2015/05/06 19:58:41, gunsch wrote: > On 2015/05/06 18:34:30, I haz the power (commit-bot) wrote: > > Patchset 6 (id:??) landed as > > https://crrev.com/a92cf235709397c737d5d9fdbadfb62e497c0334 > > Cr-Commit-Position: refs/heads/master@{#328565} > > This seems to have broken some of the internal Chromium-Android builds on our > team---for example, strings that are part of our APK target cannot be found when > compiling Java source files that reference them, R.mipmap.* not available, etc. > What part of the Chromium-Android build system is typically generating the R.txt > flies? aapt generates the R.txt files (see process_resources.py probably). I expect that if compile is failing, that there are references to resources using the wrong R class (and so need fixes like SyncCustomizationFragmentTest.java here).
Message was sent while issue was closed.
msw@chromium.org changed reviewers: + msw@chromium.org
Message was sent while issue was closed.
This also broke our FYI bot: http://build.chromium.org/p/chromium.mojo/builders/Chromium%20Mojo%20Android/... I'm not sure what needs to be updated (yet), and I would appreciate your help/guidance. I still see over 160 references to "org.chromium.chrome.R" in the codebase. Do all need to be updated to "org.chromium.chrome.shell.R"? $ git gs org.chromium.chrome.R | wc -l 162
Message was sent while issue was closed.
On 2015/05/06 22:02:57, msw wrote: > This also broke our FYI bot: > http://build.chromium.org/p/chromium.mojo/builders/Chromium%20Mojo%20Android/... > > I'm not sure what needs to be updated (yet), and I would appreciate your > help/guidance. I still see over 160 references to "org.chromium.chrome.R" in the > codebase. Do all need to be updated to "org.chromium.chrome.shell.R"? > $ git gs org.chromium.chrome.R | wc -l > 162 DOes your rule that includes build/java_apk.gypi also specify an R_package? If so, you need to make sure that imports of resources for that apk import resources from that package
Message was sent while issue was closed.
On 2015/05/06 22:08:51, Yaron wrote: > On 2015/05/06 22:02:57, msw wrote: > > This also broke our FYI bot: > > > http://build.chromium.org/p/chromium.mojo/builders/Chromium%20Mojo%20Android/... > > > > I'm not sure what needs to be updated (yet), and I would appreciate your > > help/guidance. I still see over 160 references to "org.chromium.chrome.R" in > the > > codebase. Do all need to be updated to "org.chromium.chrome.shell.R"? > > $ git gs org.chromium.chrome.R | wc -l > > 162 > > DOes your rule that includes build/java_apk.gypi also specify an R_package? If > so, you need to make sure that imports of resources for that apk import > resources from that package I only see "build/java_apk.gypi" mentioned in gyp files, and I'm building GN only targets, like mojo/runner:mojo_runner_apk: https://code.google.com/p/chromium/codesearch#chromium/src/mojo/runner/BUILD.... Should this CL have updated the GN apk target rules, like it did for GYP in build/java_apk.gypi?
Message was sent while issue was closed.
On 2015/05/06 22:15:42, msw wrote: > On 2015/05/06 22:08:51, Yaron wrote: > > On 2015/05/06 22:02:57, msw wrote: > > > This also broke our FYI bot: > > > > > > http://build.chromium.org/p/chromium.mojo/builders/Chromium%20Mojo%20Android/... > > > > > > I'm not sure what needs to be updated (yet), and I would appreciate your > > > help/guidance. I still see over 160 references to "org.chromium.chrome.R" in > > the > > > codebase. Do all need to be updated to "org.chromium.chrome.shell.R"? > > > $ git gs org.chromium.chrome.R | wc -l > > > 162 > > > > DOes your rule that includes build/java_apk.gypi also specify an R_package? If > > so, you need to make sure that imports of resources for that apk import > > resources from that package > > I only see "build/java_apk.gypi" mentioned in gyp files, and I'm building GN > only targets, like mojo/runner:mojo_runner_apk: > https://code.google.com/p/chromium/codesearch#chromium/src/mojo/runner/BUILD.... > Should this CL have updated the GN apk target rules, like it did for GYP in > build/java_apk.gypi? References to org.chromium.chrome.R are fine as long as the resources used are in the chrome resources (and not a higher-level's resources like chrome_shell). That mojo failure looks like something else is going wrong, though. The failure is in processing resources, not compiling java.
Message was sent while issue was closed.
On 2015/05/06 22:15:42, msw wrote: > On 2015/05/06 22:08:51, Yaron wrote: > > On 2015/05/06 22:02:57, msw wrote: > > > This also broke our FYI bot: > > > > > > http://build.chromium.org/p/chromium.mojo/builders/Chromium%20Mojo%20Android/... > > > > > > I'm not sure what needs to be updated (yet), and I would appreciate your > > > help/guidance. I still see over 160 references to "org.chromium.chrome.R" in > > the > > > codebase. Do all need to be updated to "org.chromium.chrome.shell.R"? > > > $ git gs org.chromium.chrome.R | wc -l > > > 162 > > > > DOes your rule that includes build/java_apk.gypi also specify an R_package? If > > so, you need to make sure that imports of resources for that apk import > > resources from that package > > I only see "build/java_apk.gypi" mentioned in gyp files, and I'm building GN > only targets, like mojo/runner:mojo_runner_apk: > https://code.google.com/p/chromium/codesearch#chromium/src/mojo/runner/BUILD.... > Should this CL have updated the GN apk target rules, like it did for GYP in > build/java_apk.gypi? This change did update the gn apk rules (see changes in build/config/android/*.gni).
Message was sent while issue was closed.
Yeah, it looks like this breaks gn apks that have no resources. I'd revert it. https://codereview.chromium.org/1104703003/diff/90001/build/android/gyp/proce... File build/android/gyp/process_resources.py (right): https://codereview.chromium.org/1104703003/diff/90001/build/android/gyp/proce... build/android/gyp/process_resources.py:384: r_text_path = os.path.join(gen_dir, 'R.txt') This needs to handle the case when aapt doesn't generate an R.txt. I think we should still create an R.txt because the caller expects that as an output (and may have specified it as an output to ninja), but just make an empty one. That will then mean that we may get a bunch of empty-ish R.java files, and maybe we need to handle that differently.
Message was sent while issue was closed.
A revert of this CL (patchset #6 id:90001) has been created in https://codereview.chromium.org/1129103002/ by cjhopman@chromium.org. The reason for reverting is: Breaks gn apks with no resources..
Message was sent while issue was closed.
On 2015/05/07 00:33:34, cjhopman wrote: > A revert of this CL (patchset #6 id:90001) has been created in > https://codereview.chromium.org/1129103002/ by mailto:cjhopman@chromium.org. > > The reason for reverting is: Breaks gn apks with no resources.. So I patched this into our internal build to try to resolve build issues. There are a couple layers of issues: 1) In our case, the suggestion that the resource was using the wrong package was incorrect. Targets where resources are pulled in from the java_apk.gypi target rather than a java.gypi target break at build time. For example, our target looks like this: { 'target_name': 'cast_shell_internal_apk', 'type': 'none', 'dependencies': [ ... ], 'variables': { 'apk_name': 'CastShellInternal', ... 'has_java_resources': 1, 'resource_dir': 'shell/android/apk/res', 'R_package': 'com.google.android.apps.mediashell', 'additional_res_dirs': ['../shell/android/apk/res'], 'additional_res_packages': ['org.chromium.chromecast.shell'], ... }, 'includes': [ '../../build/java_apk.gypi' ], }, # end of target 'cast_shell_internal_apk' which fails to build with this CL patched in. However, if we extract the "has_java_resources" bit to a target that instead includes "build/java.gypi", it builds again. (Might be worth resolving next go-round of this feature). 2) Even when we *do* make the above change, the APK fails at runtime with resources not bundled in ("Failed resolution of: Lcom/android/tv/settings/R$color;"). Note that this is from a different package than the APK's R_package, but I believe currently gets bundled in with other resources. I'm happy to help reproduce/dig in further for next go-round, preferably before submitting :)
Message was sent while issue was closed.
On 2015/05/07 18:06:41, gunsch wrote: > On 2015/05/07 00:33:34, cjhopman wrote: > > A revert of this CL (patchset #6 id:90001) has been created in > > https://codereview.chromium.org/1129103002/ by mailto:cjhopman@chromium.org. > > > > The reason for reverting is: Breaks gn apks with no resources.. > > So I patched this into our internal build to try to resolve build issues. There > are a couple layers of issues: > > 1) In our case, the suggestion that the resource was using the wrong package was > incorrect. Targets where resources are pulled in from the java_apk.gypi target > rather than a java.gypi target break at build time. For example, our target > looks like this: > > { > 'target_name': 'cast_shell_internal_apk', > 'type': 'none', > 'dependencies': [ > ... > ], > 'variables': { > 'apk_name': 'CastShellInternal', > ... > 'has_java_resources': 1, > 'resource_dir': 'shell/android/apk/res', > 'R_package': 'com.google.android.apps.mediashell', > 'additional_res_dirs': ['../shell/android/apk/res'], > 'additional_res_packages': ['org.chromium.chromecast.shell'], > ... > }, > 'includes': [ '../../build/java_apk.gypi' ], > }, # end of target 'cast_shell_internal_apk' > > which fails to build with this CL patched in. However, if we extract the > "has_java_resources" bit to a target that instead includes "build/java.gypi", it > builds again. (Might be worth resolving next go-round of this feature). I expect that you are referring to resources in org.chromium.chromecast.shell.R instead of com.google.android.apps.mediashell.R. This change breaks the way that you are using additional_res_packages (because you have no corresponding additional_r_txt_file) and so no resources will be created in that package. I believe that you could split this up into something like cast_shell_resources resource_dir shell/android/apk/res R_package org.chromium.chromecast.shell deps [ <whatever resources deps are needed here> ] include java.gypi cast_shell_internal_apk resource_dir build/android/ant/empty/res R_package com.google.android.apps.mediashell deps [ <whatever>, cast_shell_resources ] include java_apk.gypi and get the old behavior. Alternatively, we could introduce a way to specify that you want the resources to be in multiple packages... but that's really complicated if we want to support it on non-apks. Supporting it just on apks wouldn't be too bad. > > 2) Even when we *do* make the above change, the APK fails at runtime with > resources not bundled in ("Failed resolution of: > Lcom/android/tv/settings/R$color;"). Note that this is from a different package > than the APK's R_package, but I believe currently gets bundled in with other > resources. I'm happy to help reproduce/dig in further for next go-round, > preferably before submitting :) That's a strange way for it to fail. Looking at Android's way of doing this (https://android.googlesource.com/platform/tools/base/+/master/sdklib/src/main...) it looks like when they don't have any values for one of the inner types, they still create the class, but it's just empty. Here it looks like we're only creating an inner class (like R.color) if it has some value in it.
Message was sent while issue was closed.
On 2015/05/07 18:21:40, cjhopman wrote: > On 2015/05/07 18:06:41, gunsch wrote: > > 2) Even when we *do* make the above change, the APK fails at runtime with > > resources not bundled in ("Failed resolution of: > > Lcom/android/tv/settings/R$color;"). Note that this is from a different > package > > than the APK's R_package, but I believe currently gets bundled in with other > > resources. I'm happy to help reproduce/dig in further for next go-round, > > preferably before submitting :) > > That's a strange way for it to fail. Looking at Android's way of doing this > (https://android.googlesource.com/platform/tools/base/+/master/sdklib/src/main...) > it looks like when they don't have any values for one of the inner types, they > still create the class, but it's just empty. Here it looks like we're only > creating an inner class (like R.color) if it has some value in it. I don't read the Android code that way - SymbolLoader populates the table that SymbolWriter is looping over from lines in R.txt files, and since each line in an R.txt file has a value it's impossible to have a class without any values. I suspect I've screwed up something more subtle - the fact that the failure only shows up at runtime implies that there's code building against a com.android.tv.setttings.R class that has more inner classes than the version that ends up in the APK. Another thought that comes to mind is that it might have something to do with the resolution of one resource referenced from another (something like "@com.android.tv.settings:color/foo" in a layout in a package that doesn't have a dependency on the target that produces com.android.tv.settings). I would appreciate any help you can give in reproducing this error.
Message was sent while issue was closed.
On 2015/05/07 19:23:44, Ian Cullinan wrote: > On 2015/05/07 18:21:40, cjhopman wrote: > > On 2015/05/07 18:06:41, gunsch wrote: > > > 2) Even when we *do* make the above change, the APK fails at runtime with > > > resources not bundled in ("Failed resolution of: > > > Lcom/android/tv/settings/R$color;"). Note that this is from a different > > package > > > than the APK's R_package, but I believe currently gets bundled in with other > > > resources. I'm happy to help reproduce/dig in further for next go-round, > > > preferably before submitting :) > > > > That's a strange way for it to fail. Looking at Android's way of doing this > > > (https://android.googlesource.com/platform/tools/base/+/master/sdklib/src/main...) > > it looks like when they don't have any values for one of the inner types, they > > still create the class, but it's just empty. Here it looks like we're only > > creating an inner class (like R.color) if it has some value in it. > > I don't read the Android code that way - SymbolLoader populates the table that > SymbolWriter is looping over from lines in R.txt files, and since each line in > an R.txt file has a value it's impossible to have a class without any values. I yeah, you're right about that (I had a couple variables swapped in my head). > suspect I've screwed up something more subtle - the fact that the failure only > shows up at runtime implies that there's code building against a > com.android.tv.setttings.R class that has more inner classes than the version > that ends up in the APK. > > Another thought that comes to mind is that it might have something to do with > the resolution of one resource referenced from another (something like > "@com.android.tv.settings:color/foo" in a layout in a package that doesn't have > a dependency on the target that produces com.android.tv.settings). > > I would appreciate any help you can give in reproducing this error. For the cast internal failures, they'll be really difficult for us to reproduce. I'd suggest just adding a flag to java_apk.gypi that allows disabling of the resource filtering and then I can try to help figure out what's going wrong with the internal stuff, but it won't block landing this change.
Message was sent while issue was closed.
On 2015/05/07 19:45:26, cjhopman wrote: > On 2015/05/07 19:23:44, Ian Cullinan wrote: > > On 2015/05/07 18:21:40, cjhopman wrote: > > > On 2015/05/07 18:06:41, gunsch wrote: > > > > 2) Even when we *do* make the above change, the APK fails at runtime with > > > > resources not bundled in ("Failed resolution of: > > > > Lcom/android/tv/settings/R$color;"). Note that this is from a different > > > package > > > > than the APK's R_package, but I believe currently gets bundled in with > other > > > > resources. I'm happy to help reproduce/dig in further for next go-round, > > > > preferably before submitting :) > > > > > > That's a strange way for it to fail. Looking at Android's way of doing this > > > > > > (https://android.googlesource.com/platform/tools/base/+/master/sdklib/src/main...) > > > it looks like when they don't have any values for one of the inner types, > they > > > still create the class, but it's just empty. Here it looks like we're only > > > creating an inner class (like R.color) if it has some value in it. > > > > I don't read the Android code that way - SymbolLoader populates the table that > > SymbolWriter is looping over from lines in R.txt files, and since each line in > > an R.txt file has a value it's impossible to have a class without any values. > I > > yeah, you're right about that (I had a couple variables swapped in my head). > > > suspect I've screwed up something more subtle - the fact that the failure only > > shows up at runtime implies that there's code building against a > > com.android.tv.setttings.R class that has more inner classes than the version > > that ends up in the APK. > > > > Another thought that comes to mind is that it might have something to do with > > the resolution of one resource referenced from another (something like > > "@com.android.tv.settings:color/foo" in a layout in a package that doesn't > have > > a dependency on the target that produces com.android.tv.settings). > > > > I would appreciate any help you can give in reproducing this error. > > > For the cast internal failures, they'll be really difficult for us to reproduce. > I'd suggest just adding a flag to java_apk.gypi that allows disabling of the > resource filtering and then I can try to help figure out what's going wrong with > the internal stuff, but it won't block landing this change. Oh, it could also be related to the use of additional_res_packages without adding something to addition_r_text_files. We assume that these lists go together. Adding to one but not the other will throw off the mapping between package and text file.
Message was sent while issue was closed.
|