Generate the icu data binaries at compile time instead of checking in binaries

README.chromium

 Name: icu
 URL: http://site.icu-project.org/
 Version: 54.1
 License: MIT
 Security Critical: yes
 
 Description:
 This directory contains the source code of ICU 54.1 for C/C++.
 
 
@@ skipping 45 matching lines @@
 
    - Apply patches/khmer-dictbe.patch and put in a smaller Khmer dictionary
      (source/data/brkitr/khmerdict.txt) obtained from
      http://bugs.icu-project.org/trac/ticket/9451
 
    - Add several common Chinese words that were dropped previously to
      source/data/cjdict/brkitr/cjdict.txt
      patch: patches/cjdict.patch
      upstream bug: http://bugs.icu-project.org/trac/ticket/10888
 
-
-   - android/brkitr.patch (to be applied for Android build only) :
-       Do not use the C+J dictionary for Chinese/Japanese segmentation
-       to reduce the data size. Adjust word.txt and a few other files.
-
-   - source/data/brkitr/word_ja.txt (used only on Android)
-       Added for Japanese-specific word-breaking without the C+J dictionary.
-
 4. Converter changes :
 
   - convrtrs.txt : Replaced the original by our own that only lists encodings
     and aliases required by the WHATWG Encoding spec plus a few extra (see
     the file as to why).
 
   - Add source/data/mappings/ucmlocal.txt : to list only converters we need.
 
   - Add new tables per the WHATWG encoding standards for EUC-JP,
     Shift_JIS, Big5 (Big5+Big5HKSCS), EUC-KR and all the single byte encodings.
@@ skipping 37 matching lines @@
         ExemplarCharacters, LocaleScript, layout, and the name of the
         language for a locale in its native language.
       c. Remove the legacy Chinese character set-based collation
          (big5han/gb2312han) that don't make any sense and nobdoy uses.
 
   - Add tg.txt, ckb.txt, and ku.txt to source/data/{locale,lang}
     with the minimal locale data necessary for spellchecker and
     and language menus. Also change the English display name
     for ckb to 'Kurdish (Arabic)'.
 
-  - android/patch_locale.sh (to be run for Android build only):
-      a. Make changes to source/data/{region,lang} to exclude these data
-         except the language and script names of zh_Hans and zh_Hant.
-      b. Remove exemplar cities in timezone data (data/zone).
-      c. Keep only the minimal calendar data in data/locales.
-      d. Include currency display names for a smaller subset of currencies.
-      e. Minimize the locale data for 9 locales to which Chrome on Android
-         is not localized.
-
 6. Timezone data update
   - Grab the latest version of the following timezone data files and
     put them in source/data/misc.
 
      metaZones.txt
      timezoneTypes.txt
      windowsZones.txt
      zoneinfo64.txt
 
    As of January 2015, the latest version is 2015a and the above files
@@ skipping 26 matching lines @@
       Remove unnecessary resources : unames, collator rule source
   - patches/pkg_gen.patch :
     upstream bug (fixed in the upstream RC 55)
       http://bugs.icu-project.org/trac/ticket/10572
   - patches/data.build.win.patch :
       Windows-only data build patch.
   - patches/data_symb.patch :
       Put ICU_DATA_ENTRY_POINT(icudtXX_dat) in common when we use
       the icu data file or icudt.dll
 
-9. Pre-built data files are checked in with the following steps on Linux:
-
-    a. Make a icu data build directory outside the Chromium source tree
-       and cd to that directory.
-    b. Run
-
-      ${CHROME_ICU_TREE_TOP}/source/runConfigureICU Linux --disable-layout
-
-    c. Run 'make'
-    d. 'make' will fail  when pkgdata looks for css3transform.res. Edit
-      data/out/tmp/icudata.lst to replace 'css3transform.res' with 'root.res'.
-    (see http://bugs.icu-project.org/trac/ticket/10570 ) and run 'make' again.
-
-
-    - source/data/in/icudtl.dat : Built on Linux with all the patches
-      above applied. icudt54l.dat is generated in
-      {BUILD_DIR_ROOT}/data/out/tmp and copied to the above location with a
-      version number (54) dropped.
-
-
-    - {mac,linux}/icudtl_dat.S : Built on Linux with all the
-      patches above (except android/brkitr.patch) applied and checked in.
-      This file will be generated in {BUILD_DIR_ROOT}/data/out/tmp as
-      icudt54l_dat.S, but '54' is dropped while copying.
-
-      mac/icudtl_dat.S is identical to linux/icudtl_dat.S except for
-      the header portion. With "linux/icudtl_dat.S" in its place,
-      run scripts/make_mac_assembly.sh to generate it.
-
-    - android/icudtl_dat.S : Built on Linux with all the patches above and
-      android/brkitr.patch applied and android/patch_locale.sh executed.
-      '54' is dropped from the name generated in the build tree.
-
-    - android/icudtl.dat : Generated as icudt54l.dat in
-      {BUILD_DIR_ROOT}/data/out/tmp along with icudt54l_dat.S and
-      copied to the above location with '54' dropped in its name.
-
-    - windows/icudt.dll (by default, we set icu_use_icu_data_flag to 1
-      and don't use this file.)
-
-      a. check out a clean copy of icu54  from the upstream on Windows
-         outside the Chrome tree.
-
-        $ svn export --native-eol LF http://source.icu-project.org/repos/icu/icu/tags/release-54-1 ${SEPARATE_ICU_ROOT}/icu54
-
-      b. copy ${CHROME_ICU_ROOT}/source/data/in/icudtl.dat to
-         ${SEPARATE_ICU_ROOT}/source/data/in/icudt54l.dat
-      c. copy ${CHROME_ICU_ROOT}/source/data/makedata.mak to
-         ${SEPARATE_ICU_ROOT}/source/data/makedata.mak
-      c. In Visual Studio, open source/allinone/allinone.sln solution
-         in ${SEPARATE_ICU_ROOT}
-      d. Build 'makedata' target
-      e. icudt54.dll will be generated in ${SEPARATE_ICU_ROOT}/bin
-      f. Copy that icudt54.dll to ${CHROME_ICU_ROOT}/windows/icudt.dll
-         and check that in.
+#9. Pre-built data files are checked in with the following steps on Linux:
+#
+#    a. Make a icu data build directory outside the Chromium source tree
+#       and cd to that directory.
+#    b. Run
+#
+#      ${CHROME_ICU_TREE_TOP}/source/runConfigureICU Linux --disable-layout
+#
+#    c. Run 'make'
+#    d. 'make' will fail  when pkgdata looks for css3transform.res. Edit
+#      data/out/tmp/icudata.lst to replace 'css3transform.res' with 'root.res'.
+#    (see http://bugs.icu-project.org/trac/ticket/10570 ) and run 'make' again.
+#
+#
+#    - source/data/in/icudtl.dat : Built on Linux with all the patches
+#      above applied. icudt54l.dat is generated in
+#      {BUILD_DIR_ROOT}/data/out/tmp and copied to the above location with a
+#      version number (54) dropped.
+#
+#
+#    - {mac,linux}/icudtl_dat.S : Built on Linux with all the
+#      patches above (except android/brkitr.patch) applied and checked in.
+#      This file will be generated in {BUILD_DIR_ROOT}/data/out/tmp as
+#      icudt54l_dat.S, but '54' is dropped while copying.
+#
+#      mac/icudtl_dat.S is identical to linux/icudtl_dat.S except for
+#      the header portion. With "linux/icudtl_dat.S" in its place,
+#      run scripts/make_mac_assembly.sh to generate it.
+#
+#    - android/icudtl_dat.S : Built on Linux with all the patches above and
+#      android/brkitr.patch applied and android/patch_locale.sh executed.
+#      '54' is dropped from the name generated in the build tree.
+#
+#    - android/icudtl.dat : Generated as icudt54l.dat in
+#      {BUILD_DIR_ROOT}/data/out/tmp along with icudt54l_dat.S and
+#      copied to the above location with '54' dropped in its name.
+#
+#    - windows/icudt.dll (by default, we set icu_use_icu_data_flag to 1
+#      and don't use this file.)
+#
+#      a. check out a clean copy of icu54  from the upstream on Windows
+#         outside the Chrome tree.
+#
+#        $ svn export --native-eol LF http://source.icu-project.org/repos/icu/icu/tags/release-54-1 ${SEPARATE_ICU_ROOT}/icu54
+#
+#      b. copy ${CHROME_ICU_ROOT}/source/data/in/icudtl.dat to
+#         ${SEPARATE_ICU_ROOT}/source/data/in/icudt54l.dat
+#      c. copy ${CHROME_ICU_ROOT}/source/data/makedata.mak to
+#         ${SEPARATE_ICU_ROOT}/source/data/makedata.mak
+#      c. In Visual Studio, open source/allinone/allinone.sln solution
+#         in ${SEPARATE_ICU_ROOT}
+#      d. Build 'makedata' target
+#      e. icudt54.dll will be generated in ${SEPARATE_ICU_ROOT}/bin
+#      f. Copy that icudt54.dll to ${CHROME_ICU_ROOT}/windows/icudt.dll
+#         and check that in.
 
 10. Apply the following patches for regex
    - patches/regex.patch (a combined patch of 3 revisions below)
    - upstream bugs (fixed in the upstream 55 RC)
      http://bugs.icu-project.org/trac/ticket/11370 (r36723:36724)
      http://bugs.icu-project.org/trac/ticket/11369 (r36726:36727)
      http://bugs.icu-project.org/trac/ticket/11371 (r36800:36801)
 
 11. Fix bugs in locid (getBaseName / thread safety).
    - patches/locid.patch
    - upstream bugs (fixed in the upstream 55 RC)
      http://bugs.icu-project.org/trac/ticket/11421
      http://bugs.icu-project.org/trac/ticket/11547
 
 12. Fix bugs in BiDi
    - patches/bidi.patch
    - upstream bugs (fixed in the upstream 55 RC)
      http://bugs.icu-project.org/trac/ticket/11177
      http://bugs.icu-project.org/trac/ticket/11451
 
 13. Fix a data race in cmemory
    - patches/cmemory.patch
    - upstream bug (fixed in the upstream 55 RC)
      http://www.icu-project.org/trac/ticket/11538
+
+
+General information for maintenance
+-----------------------------------
+
+ * Build system in gyp/gn + ninja to generate the data packages:
+   We build the necessary tools (genrb, makeconv, icupkg, ...) and then
+   run those tools on all data files we want to generate.
+
+   This process is slightly complicated because some data files we want to
+   modify (shrink) before the build them. This is implemented with a filter
+   program that takes files that we want to modify and put them in a modified
+   shape in a special directory.
+
+   The whole chain is (dir names can vary):
+   source/data/*                 <- raw/pure data
+       out/gen/tmp_icudt54l/*    <- filtered/stripped data
+           out/gen/icudt54l/*    <- compiled data
+              out/icudt54l.dat   <- packaged data
+
+   This multi step chain means that files are named twice or three times in
+   the build system with different paths (this might change if we can drop
+   gyp) so to add or remove one data chunk it will have to be removed
+   from these lists in icu_data.gypi:
+     *_raw_source
+     *_filtered_source
+     *_generated
+   where * is something like icu_curr_res or icu_lang_res.
+
+   Some special files are also handled specifically in the filtering system
+   and in the build system.
+
+ * Build system for tools: makeconv, pkgdata, icupkg, genrb, gendict,
+   genbrk, ...
+
+   These are pretty straight forward. ICU is built as a static lib and used by
+   the tools that often are just a single file implementing main() and some
+   utility code. Whatever works for building the rest of ICU typically works
+   for building the data tools.
+
+ * Filtering tool (filter_data_for_size.py):
+   Based on:
+   - android/patch_locale.sh (to be run for Android build only):
+      a. Make changes to source/data/{region,lang} to exclude these data
+         except the language and script names of zh_Hans and zh_Hant.
+      b. Remove exemplar cities in timezone data (data/zone).
+      c. Keep only the minimal calendar data in data/locales.
+      d. Include currency display names for a smaller subset of currencies.
+      e. Minimize the locale data for 9 locales to which Chrome on Android
+         is not localized.
+
+   - android/brkitr.patch (to be applied for Android build only) :
+       Do not use the C+J dictionary for Chinese/Japanese segmentation
+       to reduce the data size. Adjust word.txt and a few other files.
+
+   - source/data/brkitr/word_ja.txt (used only on Android)
+       Added for Japanese-specific word-breaking without the C+J dictionary.
+
+ * The build system is not warning clean. In particular:
+   - ICU tools warn about empty input files. - Check --quiet flags?
+   - The gyp Makefile generator generates duplicate rules (affects
+     v8-standalone) - Reported as gyp issue 484.
+
+ * Updating to a newer ICU: Diff all Makefile.* and *.mk files before
+   and after and mimic necessary changes (if any) in BUILD.gn, icu_data.gyp
+   and icu_data.gypi.