Adapt String interface for Utf16.

sdk/lib/core/string.dart

Index: sdk/lib/core/string.dart
diff --git a/sdk/lib/core/string.dart b/sdk/lib/core/string.dart
index 7f2dcbd48b4fc51306c3fcf896c0c1c91a876b14..9e2a2c6fd3da5ded0f3e5bc1a410dda8e7eec11d 100644
--- a/sdk/lib/core/string.dart
+++ b/sdk/lib/core/string.dart
@@ -6,24 +6,41 @@ part of dart.core;
 
 /**
  * The String class represents character strings. Strings are

[Lasse Reichstein Nielsen, 2013/02/01 12:44:10]

Newlines after first sentence.
"strings" is not a common word. It's a "sequences of characters".

[floitsch, 2013/02/01 20:21:15]

Done.

- * immutable. A string is represented by a list of 32-bit Unicode
- * scalar character codes accessible through the [charCodeAt] or the
- * [charCodes] method.
+ * immutable. A string is represented by a list of 16-bit Unicode

[erikcorry, 2013/02/01 09:42:50]

Unicode code units -> Unicode UTF-16 code units

[Lasse Reichstein Nielsen, 2013/02/01 12:44:10]

"list" means something else in Dart. Again "sequence" might be appropriate.

[floitsch, 2013/02/01 20:21:15]

Done.

[floitsch, 2013/02/01 20:21:15]

Done.

+ * code units accessible through the [codeUnitAt] or the
+ * [codeUnits] members.

[erikcorry, 2013/02/01 09:42:50]

Also accessible with []

[Lasse Reichstein Nielsen, 2013/02/01 12:44:10]

That returns a String, not a code unit.

[floitsch, 2013/02/01 20:21:15]

reworded. PTAL.

+ *
+ * Strings are encoded in Utf16. Decoding Utf16, which combines

[erikcorry, 2013/02/01 09:42:50]

Correct spelling is "UTF-16".  Our style guide and the limitations of valid
identifiers mean it has to be called Utf16 in code, but in comments we can write
it correctly.

[Lasse Reichstein Nielsen, 2013/02/01 12:44:10]

"The characters of a string are encoded as UTF-16 code units."
The string itself is not encoded, it just is. Their content is stored in encoded
form.

[floitsch, 2013/02/01 20:21:15]

Done.

[floitsch, 2013/02/01 20:21:15]

Done.

+ * surrogate pairs, yields Unicode code-points. Following a similar

[Lasse Reichstein Nielsen, 2013/02/01 12:44:10]

No dash in "code points". It's two words.

[floitsch, 2013/02/01 20:21:15]

Done.

+ * terminology as Go we call Unicode code-points "runes". The 32-bit

[erikcorry, 2013/02/01 09:42:50]

as -> to

[Lasse Reichstein Nielsen, 2013/02/01 12:44:10]

"we call X Y" puts two names right next to each other. How about "we use the
name Y for X".

[floitsch, 2013/02/01 20:21:15]

Done.

+ * rune value is accessible through the [runes] getter.
  */
 abstract class String implements Comparable, Pattern {
   /**
    * Allocates a new String for the specified [charCodes].
+   *
+   * The [charCodes] can be code-units or runes. If a char-code value is

[erikcorry, 2013/02/01 09:42:50]

code units -> UTF-16 code units

[floitsch, 2013/02/01 20:21:15]

Done.

+   * 16-bit it is copied verbatim. If it is greater than 16 bits it is
+   * decomposed into a surrogate pair.
+   */
+  external factory String.fromCharCodes(Iterable<int> charCodes);
+
+  /**
+   * *Deprecated*. Use [String.fromCharCode] instead.
    */
-  external factory String.fromCharCodes(List<int> charCodes);
+  factory String.character(int charCode) => new String.fromCharCode(charCode);
 
   /**
    * Allocates a new String for the specified [charCode].
    *
-   * The built string is of [length] one, if the [charCode] lies inside the
-   * basic multilingual plane (plane 0). Otherwise the [length] is 2 and
-   * the code units form a surrogate pair.
+   * The built string is of [length] one, if the [charCode] is less than

[Lasse Reichstein Nielsen, 2013/02/01 12:44:10]

"The new string contains a single code unit if the [charCode] can be represented
in 16 bits. Otherwise .."
(The current text compares values to bit-lengths).

[floitsch, 2013/02/01 20:21:15]

Done.

+   * 16 bits. Otherwise the [length] is 2 and the code units form a surrogate
+   * pair.
+   *
+   * It is allowed (though generally discouraged) to create a String with only
+   * one half of a surrogate pair.
    */
-  factory String.character(int charCode) {
+  factory String.fromCharCode(int charCode) {
     List<int> charCodes = new List<int>.fixedLength(1, fill: charCode);
     return new String.fromCharCodes(charCodes);
   }
@@ -35,17 +52,39 @@ abstract class String implements Comparable, Pattern {
 
   /**
    * Gets the scalar character code at the given [index].
+   *
+   * *This method is deprecated. Please use [codeUnitAt] instead.*
    */
   int charCodeAt(int index);
 
   /**
+   * Returns the code-unit (16-bit) at the given [index].

[erikcorry, 2013/02/01 09:42:50]

16-bit -> 16 bit UTF-16

[floitsch, 2013/02/01 20:21:15]

Done.

+   */
+  int codeUnitAt(int index);
+
+  /**
    * The length of the string.
+   *
+   * Returns the number of 16-bit code units in this string. The number

[erikcorry, 2013/02/01 09:42:50]

code units -> UTF-16 code units

[floitsch, 2013/02/01 20:21:15]

Done.

+   * of [runes] might be less, if the string contains characters outside
+   * the basic multilingual plane (plane 0).
    */
   int get length;
 
   /**
-   * Returns whether the two strings are equal. This method compares
-   * each individual scalar character codes of the strings.
+   * Returns whether the two strings are equal.
+   *
+   * This method compares each individual code unit of the strings. It does not
+   * check for Unicode equivalence. For example the two following strings both
+   * represent the string "Amélie" but, due to their different encoding will
+   * not return equal.
+   *
+   *     "Am\xe9lie"
+   *     "Ame\u{301}lie"
+   *
+   * In the first string the "é" is encoded as a single unicode code unit,
+   * whereas the second string encodes it as "e" with the combining
+   * accent character "◌́".
    */
   bool operator ==(String other);

[Lasse Reichstein Nielsen, 2013/02/01 12:44:10]

Shouldn't this be 
 bool operator==(Object other)
to be compatible with Object.operator==?

[floitsch, 2013/02/01 20:21:15]

yes. Changed to "var".

 
@@ -147,11 +186,28 @@ abstract class String implements Comparable, Pattern {
   /**
    * Splits the string around matches of [pattern]. Returns
    * a list of substrings.
+   *
+   * Splitting with an empty string pattern (`""`) splits at code unit

[erikcorry, 2013/02/01 09:42:50]

code unit -> UTF-16 code unit

[floitsch, 2013/02/01 20:21:15]

Done.

+   * boundaries and not at rune boundaries. The following two expressions
+   * are hence equivalent:
+   *
+   *     string.split("")
+   *     string.codeUnits.map((unit) => new String.character(unit))
+   *
+   * Unless it guaranteed that the string is in the basic multilingual plane
+   * (meaning that a code-unit represents a rune) it is often better to

[erikcorry, 2013/02/01 09:42:50]

a -> each

[floitsch, 2013/02/01 20:21:15]

Done.

+   * map the runes instead:
+   *
+   *     string.runes.map((rune) => new String.character(rune))
    */
   List<String> split(Pattern pattern);
 
   /**
-   * Returns a list of the characters of this string.
+   * Returns a list of the individual code-units characters of this string.
+   *
+   * *Deprecated*
+   * If you want to split on code-unit boundaries, use [split]. If you
+   * want to split on rune boundaries, use [runes] and map the result.

[erikcorry, 2013/02/01 09:42:50]

I feel this comment would benefit from an example:
Iterable<String> characters = string.runes.map((c) => new
String.fromCharCode(c))

[floitsch, 2013/02/01 20:21:15]

Shouldn't be necessary since the method is going away soon. But done.

    */
   List<String> splitChars();
 
@@ -173,11 +229,30 @@ abstract class String implements Comparable, Pattern {
                        String onNonMatch(String nonMatch)});
 
   /**
-   * Returns a list of the scalar character codes of this string.
+   * Returns a list of 16-bit code-units of this string.

[erikcorry, 2013/02/01 09:42:50]

code-units -> UTF-16 code units

[floitsch, 2013/02/01 20:21:15]

Done.

+   *
+   * *This getter is deprecated. Use [codeUnits] instead.*
    */
   List<int> get charCodes;
 
   /**
+   * Returns an iterable of the 16-bit code-units of this string.

[erikcorry, 2013/02/01 09:42:50]

And here

[floitsch, 2013/02/01 20:21:15]

Done.

+   */
+  // TODO(floitsch): should it return a list?
+  // TODO(floitsch): make it a bidirectional iterator.
+  Iterable<int> get codeUnits;
+
+  /**
+   * Returns an iterable of Unicode code-points of this string.
+   *
+   * If the string contains surrogate pairs, they will be combined and returned
+   * as one integer by this iterator. Unmatched surrogate halves are treated
+   * like valid 16-bit code-units.
+   */
+  // TODO(floitsch): make it a bidirectional iterator.

[Lasse Reichstein Nielsen, 2013/02/01 12:44:10]

Let's make it a Runes class with extra functionality on top of being a
bidi-iterable.

[floitsch, 2013/02/01 20:21:15]

Changed TODO.
I will see that I can commit this CL eventually, and you can then replace the
"Unimplemented" with the class you wrote.

+  Iterable<int> get runes;
+
+  /**
    * If this string is not already all lower case, returns a new string
    * where all characters  are made lower case. Returns [:this:] otherwise.

[erikcorry, 2013/02/01 09:42:50]

double space

[Lasse Reichstein Nielsen, 2013/02/01 12:44:10]

You need to say how upper-casing is done. ASCII only or Unicode case folding? In
the latter case, in which locale?
Does it normalize the result as well, or will it never do that?

And we really need:
 int compareIgnoreCase(String other);
too (which would use Unicode canonicalization instead of case folding).

[floitsch, 2013/02/01 20:21:15]

Done.

[floitsch, 2013/02/01 20:21:15]

Same as JavaScript: using the locale-independent Unicode mapping.
Added TODO.

    */