Import SQLite 3.8.7.4.

third_party/sqlite/src/ext/fts3/README.tokenizers

 
 1. FTS3 Tokenizers
 
   When creating a new full-text table, FTS3 allows the user to select
   the text tokenizer implementation to be used when indexing text
   by specifying a "tokenize" clause as part of the CREATE VIRTUAL TABLE
   statement:
 
     CREATE VIRTUAL TABLE <table-name> USING fts3(
       <columns ...> [, tokenize <tokenizer-name> [<tokenizer-args>]]
     );
 
   The built-in tokenizers (valid values to pass as <tokenizer name>) are
-  "simple" and "porter".
+  "simple", "porter" and "unicode".
 
   <tokenizer-args> should consist of zero or more white-space separated
   arguments to pass to the selected tokenizer implementation. The 
   interpretation of the arguments, if any, depends on the individual 
   tokenizer.
 
 2. Custom Tokenizers
 
   FTS3 allows users to provide custom tokenizer implementations. The 
   interface used to create a new tokenizer is defined and described in 
@@ skipping 99 matching lines @@
       
         sqlite3_bind_text(pStmt, 1, zName, -1, SQLITE_STATIC);
         if( SQLITE_ROW==sqlite3_step(pStmt) ){
           if( sqlite3_column_type(pStmt, 0)==SQLITE_BLOB ){
             memcpy(pp, sqlite3_column_blob(pStmt, 0), sizeof(*pp));
           }
         }
       
         return sqlite3_finalize(pStmt);
       }