Chromium Code Reviews
chromiumcodereview-hr@appspot.gserviceaccount.com (chromiumcodereview-hr) | Please choose your nickname with Settings | Help | Chromium Project | Gerrit Changes | Sign out
(51)

Issue 1000163003: Generate the icu data binaries at compile time instead of checking in binaries

Created:
5 years, 9 months ago by Daniel Bratell
Modified:
5 years, 8 months ago
CC:
chromium-reviews, jochen (gone - plz use gerrit), Andrew Hayden (chromium.org)
Base URL:
https://chromium.googlesource.com/chromium/deps/icu.git@master
Target Ref:
refs/heads/master
Visibility:
Public.

Description

Historically someone (jshin) has generated all data that people could possibly need and checked it into the repository. This has made it easy to use the data as long as it contained what you needed but as soon as you needed something different it was suddenly quite tricky. By generating it during build time it can be better tuned for each product and it will be possible to experiment and work on it without being the icu specialist. It will also be possible to merge different branches with different changes to the data file composition, the git repositories will stop growing so rapidly (only master in icu.git is 0.5 GB already and that is steadily increasing). Downside of this is that we will have to maintain more gyp/gn files and it will cost some compilation time when building from scratch. Added compilation time right now is about 15s for tools, 15s for data in Linux. 40s for tools, 35s for data in Windows. Since this is 10-20% of the total binary, it is still compiled much faster than most of the project so I think it's a reasonable (but unwanted) cost. * filter_data_for_size.py conditionally removes parts of the icu data to make the generated data smaller than the default 10 MB. * icu_tools.gyp builds (via ninja) the various tools that build the icu data files. * icu_tools.gypi lists the files involved. * icu_data.gyp builds (via ninja) the data files (about 500) and packages them. * icu_data.gypi lists most of the data files (some one-off files are mentioned just in icu_data.gyp) * build_icu_data.py was a proof of concept and reverse engineering of the Makefile system (not to be included). * write_icudata_lst.py creates the package instructions by being fed file lists from gyp/ninja. * write_res_index.py creates res_index.txt files that contain language lists for each resource category. * run_without_cygwin.py is a helper script to make icu tools work in Windows v8 standalone which has a cygwin at the start of the path. * gn_data_build_system/BUILD.gn is the gn script to build the icu data. * gn_data_build_system/run.py is the necessary wrapper so that gn can execute native tools. * gn_tool_build_system/BUILD.gn is the gn script to build the icu tools. BUG=424192

Patch Set 1 #

Patch Set 2 : Working in gyp Windows too. #

Patch Set 3 : Dependency problems fixed. #

Patch Set 4 : Updated to icu 5.4 and got rid of branch changes. #

Patch Set 5 : Now with gn #

Patch Set 6 : Dumping gn WIP #

Patch Set 7 : Now with working gn #

Patch Set 8 : Hooked into the real build system. #

Patch Set 9 : Filtering of data to make the Android icu data smaller. #

Patch Set 10 : Stripping in python instead of sh+sed #

Patch Set 11 : Filtering data #

Patch Set 12 : Android Android Android #

Patch Set 13 : Fixing pool.res dependencies and adding documentation. #

Patch Set 14 : Making icu work in v8-standalone. Not all the way there. #

Patch Set 15 : Including missing file #

Patch Set 16 : Now working in v8 standalone for Windows. #

Patch Set 17 : Removing and moving thinsg. #

Patch Set 18 : And (skeleton) unittest. #

Patch Set 19 : Making BUILD.gn simpler (smaller) #

Patch Set 20 : Polishing and making BUILD.gn smaller #

Total comments: 1

Patch Set 21 : Fixed warnings in cross compiling #

Unified diffs Side-by-side diffs Delta from patch set Stats (+6441 lines, -237 lines) Patch
M BUILD.gn View 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 1 chunk +4 lines, -61 lines 0 comments Download
M README.chromium View 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 4 chunks +121 lines, -72 lines 0 comments Download
M android/patch_locale.sh View 1 2 3 4 5 6 7 8 9 10 3 chunks +3 lines, -2 lines 0 comments Download
A build_tools/PRESUBMIT.py View 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 1 chunk +32 lines, -0 lines 0 comments Download
A build_tools/copy_rename_file.py View 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 chunk +4 lines, -0 lines 0 comments Download
A build_tools/filter_data_for_size.py View 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 1 chunk +449 lines, -0 lines 0 comments Download
A build_tools/filter_data_for_size_unittest.py View 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 1 chunk +33 lines, -0 lines 0 comments Download
A build_tools/run_without_cygwin.py View 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 chunk +32 lines, -0 lines 0 comments Download
A build_tools/write_icudata_lst.py View 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 chunk +77 lines, -0 lines 0 comments Download
A build_tools/write_icupkg_inc.py View 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 chunk +53 lines, -0 lines 0 comments Download
A build_tools/write_res_index.py View 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 1 chunk +44 lines, -0 lines 0 comments Download
A gn_data_build_system/BUILD.gn View 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 1 chunk +966 lines, -0 lines 0 comments Download
A gn_data_build_system/run.py View 1 2 3 4 5 1 chunk +23 lines, -0 lines 0 comments Download
A gn_tool_build_system/BUILD.gn View 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 1 chunk +147 lines, -0 lines 0 comments Download
M icu.gyp View 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 3 chunks +128 lines, -102 lines 0 comments Download
A icu_data.gyp View 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 1 chunk +1907 lines, -0 lines 0 comments Download
A icu_data.gypi View 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 1 chunk +2136 lines, -0 lines 0 comments Download
A icu_tools.gyp View 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 1 chunk +181 lines, -0 lines 0 comments Download
A icu_tools.gypi View 1 2 3 1 chunk +101 lines, -0 lines 0 comments Download

Messages

Total messages: 6 (4 generated)
Daniel Bratell
Since the code now works(*), I think it's best to involve the code review tool ...
5 years, 8 months ago (2015-04-09 14:05:49 UTC) #4
Andrew Hayden (chromium.org)
5 years, 8 months ago (2015-04-13 13:20:29 UTC) #6
https://codereview.chromium.org/1000163003/diff/380001/gn_data_build_system/B...
File gn_data_build_system/BUILD.gn (right):

https://codereview.chromium.org/1000163003/diff/380001/gn_data_build_system/B...
gn_data_build_system/BUILD.gn:128: if (put_icu_data_on_diet) {
Cute :)

Powered by Google App Engine
This is Rietveld 408576698