Adding break iterator support to the i18n api extension....

src/extensions/experimental/break-iterator.cc

+// Copyright 2011 the V8 project authors. All rights reserved.
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions are
+// met:
+//
+//     * Redistributions of source code must retain the above copyright
+//       notice, this list of conditions and the following disclaimer.
+//     * Redistributions in binary form must reproduce the above
+//       copyright notice, this list of conditions and the following
+//       disclaimer in the documentation and/or other materials provided
+//       with the distribution.
+//     * Neither the name of Google Inc. nor the names of its
+//       contributors may be used to endorse or promote products derived
+//       from this software without specific prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+#include "break-iterator.h"
+
+#include "unicode/brkiter.h"
+#include "unicode/locid.h"
+#include "unicode/uloc.h"
+#include "unicode/rbbi.h"
+
+namespace v8 {
+namespace internal {
+
+v8::Persistent<v8::ObjectTemplate> BreakIterator::break_iterator_template_;
+
+// Unpacks break iterator object from corresponding JavaScript object.
+static icu::BreakIterator* UnpackBreakIterator(v8::Handle<v8::Object> obj) {
+  return static_cast<icu::BreakIterator*>(obj->GetPointerFromInternalField(0));
+}
+
+// Release memory we allocated for the BreakIterator once the JS object that
+// holds the pointer gets garbage collected.
+static void DeleteBreakIterator(v8::Persistent<v8::Value> object,
+                                void* param) {
+  delete UnpackBreakIterator(v8::Persistent<v8::Object>::Cast(object));
+}
+
+// Assigns new text to the iterator.
+static v8::Handle<v8::Value> BreakIteratorAssign(const v8::Arguments& args) {
+  if (args.Length() != 1 || !args[0]->IsString()) {
+    return v8::ThrowException(v8::Exception::SyntaxError(
+        v8::String::New("Text input is required.")));
+  }
+  
+  icu::BreakIterator* break_iterator = UnpackBreakIterator(args.Holder());
+
+  v8::Local<v8::String> text_value = args[0]->ToString();
+  UnicodeString text(*v8::String::Utf8Value(text_value),
+                     text_value->Utf8Length());

[jungshik at Google, 2011/02/26 00:31:38]

Chrome's ICU didn't define a macro (U_CHARSET_IS_UTF8) to get ICU to assume that
|char *| is UTF-8. Perhaps, we'll in the future.

At least for now (and perhaps in the future, too because we don't control how
other embedders will build ICU),  you should explicitly specify the encoding for
UnicodeString ctor. You didn't get caught with this because you tested it on Mac
(or Linux under UTF-8 locale). Under non-UTF-8 locale on Linux or Windows, your
test would have failed. 

You have three alternatives:

1. use UnicodeString:: fromUTF8 
2. use a ctor accepting the codepage name : pass "UTF-8"
3. use a ctor accepting UChar* instead of char*. 

#1 is faster than #2. 

[Nebojša Ćirić, 2011/02/26 01:13:17]

Done.

+  
+  break_iterator->setText(text);
+
+  return v8::Undefined();
+}
+
+// Moves iterator to the beginning of the string and returns new position.
+static v8::Handle<v8::Value> BreakIteratorFirst(const v8::Arguments& args) {
+  icu::BreakIterator* break_iterator = UnpackBreakIterator(args.Holder());
+
+  return v8::Int32::New(break_iterator->first());
+}
+
+// Moves iterator to the next position and returns it.
+static v8::Handle<v8::Value> BreakIteratorNext(const v8::Arguments& args) {
+  icu::BreakIterator* break_iterator = UnpackBreakIterator(args.Holder());
+
+  return v8::Int32::New(break_iterator->next());

[jungshik at Google, 2011/02/26 00:31:38]

When there's no more item, ICU's next() will returb UBRK_DONE (or sth like
that). How would you signal that to your API users? If you want to use '-1' for
that, perhaps you need to check the return value from bi->next() and return '-1'
explicitly for UBRK_DONE. 

[Nebojša Ćirić, 2011/02/26 00:47:45]

UBRK_DONE is actually int32_t and is represented as -1, so this works as
intended, and ICU is probably not going to change public API any time soon. I
would leave it as is.

On 2011/02/26 00:31:38, Jungshik Shin wrote:

+}
+
+// Returns current iterator's current position.
+static v8::Handle<v8::Value> BreakIteratorCurrent(const v8::Arguments& args) {
+  icu::BreakIterator* break_iterator = UnpackBreakIterator(args.Holder());
+
+  return v8::Int32::New(break_iterator->current());
+}
+
+// Returns type of the item from current position.
+// This call is only valid for word break iterators. Others just return 0.
+static v8::Handle<v8::Value> BreakIteratorBreakType(
+    const v8::Arguments& args) {
+  icu::BreakIterator* break_iterator = UnpackBreakIterator(args.Holder());
+
+  // TODO(cira): Remove cast once ICU fixes base BreakIterator class.
+  int32_t status =
+      static_cast<RuleBasedBreakIterator*>(break_iterator)->getRuleStatus();
+  // Keep return values in sync with JavaScript BreakType enum.
+  if (status >= UBRK_WORD_NONE && status < UBRK_WORD_NONE_LIMIT) {
+    return v8::Int32::New(UBRK_WORD_NONE);
+  } else if (status >= UBRK_WORD_NUMBER && status < UBRK_WORD_NUMBER_LIMIT) {
+    return v8::Int32::New(UBRK_WORD_NUMBER);
+  } else if (status >= UBRK_WORD_LETTER && status < UBRK_WORD_LETTER_LIMIT) {
+    return v8::Int32::New(UBRK_WORD_LETTER);
+  } else if (status >= UBRK_WORD_KANA && status < UBRK_WORD_KANA_LIMIT) {
+    return v8::Int32::New(UBRK_WORD_KANA);
+  } else if (status >= UBRK_WORD_IDEO && status < UBRK_WORD_IDEO_LIMIT) {
+    return v8::Int32::New(UBRK_WORD_IDEO);

[jungshik at Google, 2011/02/26 00:31:38]

We need to give a bit more thought on what types to expose to Javascript. 

BTW, line/setnence breakers can also have rule_status (
http://icu-project.org/apiref/icu4c/ubrk_8h.html#ad03d8e27f121bcf11eaed0a2887...
) although they may not be very useful. 

[Nebojša Ćirić, 2011/02/26 00:47:45]

We can take them out at any time - it's still experimental.

On 2011/02/26 00:31:38, Jungshik Shin wrote:

+  } else {
+    return v8::Int32::New(-1);
+  }
+}
+
+v8::Handle<v8::Value> BreakIterator::JSBreakIterator(
+    const v8::Arguments& args) {
+  // No need to check args here, we do it in JavaScript.
+
+  const char* locale = *v8::String::Utf8Value(args[0]->ToString());
+  icu::Locale icu_locale(locale);
+
+  UErrorCode status = U_ZERO_ERROR;
+  icu::BreakIterator* break_iterator = NULL;
+  const char* type = *v8::String::Utf8Value(args[1]->ToString());
+  if (!strcmp(type, "character")) {
+    break_iterator =
+        icu::BreakIterator::createCharacterInstance(icu_locale, status);
+  } else if (!strcmp(type, "word")) {
+    break_iterator =
+        icu::BreakIterator::createWordInstance(icu_locale, status);
+  } else if (!strcmp(type, "sentence")) {
+    break_iterator =
+        icu::BreakIterator::createSentenceInstance(icu_locale, status);
+  } else if (!strcmp(type, "line")) {
+    break_iterator =
+        icu::BreakIterator::createLineInstance(icu_locale, status);
+  } else {
+    return v8::ThrowException(v8::Exception::SyntaxError(
+        v8::String::New("Invalid iterator type.")));
+  }
+
+  if (U_FAILURE(status)) {
+    delete break_iterator;
+    return v8::ThrowException(v8::Exception::Error(
+        v8::String::New("Failed to create break iterator.")));
+  }
+  
+  v8::HandleScope handle_scope;
+
+  if (break_iterator_template_.IsEmpty()) {
+    v8::Handle<v8::ObjectTemplate> raw_template(v8::ObjectTemplate::New());
+    raw_template->SetInternalFieldCount(1);
+    raw_template->Set(v8::String::New("assign"),

[jungshik at Google, 2011/02/26 00:31:38]

How about 'adopt' or 'adoptText'? 'assign' is kinda misleading (assigning
'string' to 'break iterator' ?) 

[Nebojša Ćirić, 2011/02/26 00:47:45]

I'll fix that.

On 2011/02/26 00:31:38, Jungshik Shin wrote:

[Nebojša Ćirić, 2011/02/26 01:13:17]

Done.

+                      v8::FunctionTemplate::New(BreakIteratorAssign));
+    raw_template->Set(v8::String::New("first"),
+                      v8::FunctionTemplate::New(BreakIteratorFirst));
+    raw_template->Set(v8::String::New("next"),
+                      v8::FunctionTemplate::New(BreakIteratorNext));
+    raw_template->Set(v8::String::New("current"),
+                      v8::FunctionTemplate::New(BreakIteratorCurrent));
+    raw_template->Set(v8::String::New("breakType"),
+                      v8::FunctionTemplate::New(BreakIteratorBreakType));

[jungshik at Google, 2011/02/26 00:31:38]

One  method is missing;  a method that returns the word at the current position.
 A customer of this API can certainly find that out, but wouldn't it be nice to
offer ourselves?  

Yeah, we discussed this and I said that next() is sufficient without nextWord().


Alternatives:

1. Do nothing. Customers can figure that out
2. Add currentWord() method (either in C++ or JavaScript). 
3. Add nextWord() (and remove next())

What do you say? 

 

[Nebojša Ćirić, 2011/02/26 00:47:45]

JavaScript has two (actually 3, but one is deprecated) methods to slice a
string:

slice(from, to) and substring(from, to). So simple prev_pos and pos with
slice/substring give you that new method.

I think we should keep the API minimal, and I don't think it's going to put any
burden on the users given the existing JS apis.

On 2011/02/26 00:31:38, Jungshik Shin wrote:

+    break_iterator_template_ =
+        v8::Persistent<v8::ObjectTemplate>::New(raw_template);
+    // Make template weak so we can delete iterator once GC kicks in.
+    break_iterator_template_.MakeWeak(NULL, DeleteBreakIterator);
+  }
+
+  //Create an empty object wrapper.
+  v8::Handle<v8::Object> wrapper = break_iterator_template_->NewInstance();
+  // Set break iterator as internal field of the resulting JS object.
+  wrapper->SetPointerInInternalField(0, break_iterator);
+
+  return handle_scope.Close(wrapper);
+}
+
+} }  // namespace v8::internal