Make it possible to read CLD data from a file

third_party/cld_2/cld_2.gyp

Index: third_party/cld_2/cld_2.gyp
diff --git a/third_party/cld_2/cld_2.gyp b/third_party/cld_2/cld_2.gyp
index 9b7fbdfcadca6b5dbc74ef30df9c46e8635f2822..f0552a7798ad54ced044f68d870f23b5edf9eda9 100644
--- a/third_party/cld_2/cld_2.gyp
+++ b/third_party/cld_2/cld_2.gyp
@@ -17,13 +17,20 @@
 {
   'targets': [
     {
-      'target_name': 'cld_2',
-      'type': 'static_library',
+      'target_name': 'cld2_dynamic_data_tool',
+      'type': 'executable',
       'include_dirs': [
         'src/internal',
         'src/public',
       ],
       'sources': [
+        'src/internal/cld2_dynamic_data.h',

[palmer, 2014/03/24 19:01:47]

How is this code getting tested? Is there a fuzzer? Et c.

[Andrew Hayden (chromium.org), 2014/03/26 15:34:11]

The testing is done via unit tests in CLD2, which were modified to fully
exercise the dynamic data use cases. All unit tests in CLD2 can be built in
either dynamic or static mode. Of course, they all pass.

There's no fuzzer. My stance so far has been that...
1. On Android, this file will be part of a signed APK that we generated and that
lives in the application's private directory. Therefore it is secure, and
shouldn't be any more of a security concern than any other data files we might
have.
2. On other platforms, the file is as well protected as the OS makes it, neither
more or less secure than anything else Chromium deals with.

It's not abundantly clear to me what a fuzzer would be able to do here, since
the data within the frozen structures is all critical to the correct functioning
of the code (e.g., shifts and offsets in arrays of characters). Fuzzing anything
in the data set would immediately and completely break the code in weird ways.
IANAE on this, but if you have docs on chromium fuzzers that I could somehow
adapt I'd be happy to take a look.

I guess my point is that right now I think we're as (in)secure as we are with
non-dynamic CLD1; anyone able to access the filesystem can corrupt the CLD data,
be that within the readonly block in the .so or within the standalone data file.
CLD has not and does not have any hardening against attacks of this type.

+        'src/internal/cld2_dynamic_data.cc',
+        'src/internal/cld2_dynamic_data_extractor.h',
+        'src/internal/cld2_dynamic_data_extractor.cc',
+        'src/internal/cld2_dynamic_data_loader.h',
+        'src/internal/cld2_dynamic_data_loader.cc',
+        'src/internal/cld2_dynamic_data_tool.cc',
         'src/internal/cld2_generated_cjk_compatible.cc',
         'src/internal/cld2_generated_deltaoctachrome0122.cc',
         'src/internal/cld2_generated_distinctoctachrome0122.cc',
@@ -96,5 +103,109 @@
         }],
       ],
     },
+    {
+      'target_name': 'cld_2',
+      'type': 'static_library',
+      'include_dirs': [
+        'src/internal',
+        'src/public',
+      ],
+      'sources': [
+        'src/internal/cld2tablesummary.h',
+        'src/internal/cldutil.cc',
+        'src/internal/cldutil.h',
+        'src/internal/cldutil_shared.cc',
+        'src/internal/cldutil_shared.h',
+        'src/internal/compact_lang_det.cc',
+        'src/internal/compact_lang_det_hint_code.cc',
+        'src/internal/compact_lang_det_hint_code.h',
+        'src/internal/compact_lang_det_impl.cc',
+        'src/internal/compact_lang_det_impl.h',
+        'src/internal/debug.h',
+        'src/internal/debug_empty.cc',
+        'src/internal/fixunicodevalue.cc',
+        'src/internal/fixunicodevalue.h',
+        'src/internal/generated_distinct_bi_0.cc',
+        'src/internal/generated_entities.cc',
+        'src/internal/generated_language.cc',
+        'src/internal/generated_language.h',
+        'src/internal/generated_ulscript.cc',
+        'src/internal/generated_ulscript.h',
+        'src/internal/getonescriptspan.cc',
+        'src/internal/getonescriptspan.h',
+        'src/internal/integral_types.h',
+        'src/internal/lang_script.cc',
+        'src/internal/lang_script.h',
+        'src/internal/langspan.h',
+        'src/internal/offsetmap.cc',
+        'src/internal/offsetmap.h',
+        'src/internal/port.h',
+        'src/internal/scoreonescriptspan.cc',
+        'src/internal/scoreonescriptspan.h',
+        'src/internal/stringpiece.h',
+        'src/internal/tote.cc',
+        'src/internal/tote.h',
+        'src/internal/utf8prop_lettermarkscriptnum.h',
+        'src/internal/utf8repl_lettermarklower.h',
+        'src/internal/utf8scannot_lettermarkspecial.h',
+        'src/internal/utf8statetable.cc',
+        'src/internal/utf8statetable.h',
+        'src/public/compact_lang_det.h',
+        'src/public/encodings.h',
+      ],
+      'conditions': [
+        ['OS=="win"', {
+          'msvs_disabled_warnings': [4267],
+        }],
+        # Dynamic or vanilla mode
+        ['cld_dynamic==1',
+          {
+            # Enable dynamic mode, which allows CLD to read its data from a file
+            # instead of requiring the data to be linked into the library.
+            # You must generate the data file and check it in using the
+            # cld2_dynamic_data_tool.
+            # build/common.gypi takes care of setting -D CLD2_MODE_DYNAMIC
+            'sources+': [
+              'src/internal/cld2_dynamic_data.h',
+              'src/internal/cld2_dynamic_data.cc',
+              'src/internal/cld2_dynamic_data_loader.h',
+              'src/internal/cld2_dynamic_data_loader.cc',
+            ],
+          },
+          {
+            # Else it's regular mode, so link in all the data blobs directly.
+            'sources+': [
+              'src/internal/cld2_generated_cjk_compatible.cc',
+              'src/internal/cld2_generated_deltaoctachrome0122.cc',
+              'src/internal/cld2_generated_distinctoctachrome0122.cc',
+              'src/internal/cld_generated_cjk_delta_bi_4.cc',
+              'src/internal/cld_generated_cjk_uni_prop_80.cc',
+              'src/internal/cld_generated_score_quad_octa_0122_2.cc',
+              'src/internal/generated_distinct_bi_0.cc',
+            ],
+            'conditions': [
+              ['cld2_table_size==0', {
+                # Small table sizes
+                'sources+': [
+                  'src/internal/cld2_generated_quadchrome0122_16.cc',
+                 ],
+              }],
+              ['cld2_table_size==1', {
+                # Medium table sizes
+                'sources+': [
+                  'src/internal/cld2_generated_quadchrome0122_19.cc',
+                 ],
+              }],
+              ['cld2_table_size==2', {
+                # Large table sizes
+                'sources+': [
+                  'src/internal/cld2_generated_quadchrome0122_2.cc',
+                 ],
+              }],
+            ],
+          },
+        ],
+      ],
+    },
   ],
 }