- Represent strings internally in UTF-16 format, this makes it

lib/compiler/implementation/string_validator.dart

Index: lib/compiler/implementation/string_validator.dart
===================================================================
--- lib/compiler/implementation/string_validator.dart	(revision 14156)
+++ lib/compiler/implementation/string_validator.dart	(working copy)
@@ -99,7 +99,9 @@
     for(Iterator<int> iter = string.iterator(); iter.hasNext; length++) {
       index++;
       int code = iter.next();
+      bool verify_code = false;

[ngeoffray, 2012/10/27 09:02:00]

verify_code -> verifyCode

       if (code == $BACKSLASH) {
+        verify_code = true;
         if (quoting.raw) continue;
         containsEscape = true;
         if (!iter.hasNext) {
@@ -170,7 +172,7 @@
       }
       // This handles both unescaped characters and the value of unicode

[ahe, 2012/10/28 12:07:02]

I don't think this comment is correct any more.

       // escapes.
-      if (!isUnicodeScalarValue(code)) {
+      if (verify_code && !isUnicodeScalarValue(code)) {

[ahe, 2012/10/28 12:07:02]

This doesn't look right to me. If we are switching to strings being UTF-16
encoded strings, then we shouldn't be checking for valid Unicode scalar values,
and we should allow users to write strings like this:

"\uD801\uDC12" to create a string composed of two unicode code points that make
up the single character U+10412.

[cshapiro, 2012/10/29 06:37:17]

The specification for string literals was deliberately left the same when the
specification was changed.  Part of the spirit of the change was to avoid
exposing the details of the encoding to the user wherever possible.  I believe
ES6 prescribes similarly and is where our \u{N+} syntax came from.

[ahe, 2012/10/29 08:11:07]

If the underlying string representation is UTF-16, and this is exposed to users,
then users need to be able to create a string literal in UTF-16 notation.

By itself, either of these options are simple:

* Dart strings are a sequence of valid Unicode scalar values.

* Dart strings are a sequence of Unicode code points.

* Dart strings are a sequence of UTF-16 code units.

But you're mixing them all up. I think that is highly confusing and I strongly
urge you to reconsider this design.

[siva, 2012/10/31 01:21:26]

I have decided to drop these changes made to dart2js in this CL. It is clear
that I do not understand the architecture enough, so I have created bug 6418 to
track the two tests that fail in dart2js when run with a VM that has the change
to represent strings as UTf-16.
I have changed the status files to reflect bug 6418 for the two tests so that
the buildbots remain green.

On 2012/10/29 08:11:07, ahe wrote:

         stringParseError(
             "Invalid Unicode scalar value U+${code.toRadixString(16)}",
             token, index);