- Represent strings internally in UTF-16 format, this makes it

lib/compiler/implementation/string_validator.dart

 // Copyright (c) 2012, the Dart project authors.  Please see the AUTHORS file
 // for details. All rights reserved. Use of this source code is governed by a
 // BSD-style license that can be found in the LICENSE file.
 
 // Check the validity of string literals.
 
 library stringvalidator;
 
 import "dart2jslib.dart";
 import "tree/tree.dart";
@@ skipping 81 matching lines @@
                             StringQuoting quoting) {
     // We only need to check for invalid x and u escapes, for line
     // terminators in non-multiline strings, and for invalid Unicode
     // scalar values (either directly or as u-escape values).
     int length = 0;
     int index = startOffset;
     bool containsEscape = false;
     for(Iterator<int> iter = string.iterator(); iter.hasNext; length++) {
       index++;
       int code = iter.next();
+      bool verify_code = false;

[ngeoffray, 2012/10/27 09:02:00]

verify_code -> verifyCode

       if (code == $BACKSLASH) {
+        verify_code = true;
         if (quoting.raw) continue;
         containsEscape = true;
         if (!iter.hasNext) {
           stringParseError("Incomplete escape sequence",token, index);
           return null;
         }
         index++;
         code = iter.next();
         if (code == $x) {
           for (int i = 0; i < 2; i++) {
@@ skipping 48 matching lines @@
                 stringParseError("Invalid character in escape sequence",
                                  token, index);
                 return null;
               }
               value = value * 16 + hexDigitValue(code);
             }
           }
           code = value;
         }
       }
       // This handles both unescaped characters and the value of unicode

[ahe, 2012/10/28 12:07:02]

I don't think this comment is correct any more.

       // escapes.
-      if (!isUnicodeScalarValue(code)) {
+      if (verify_code && !isUnicodeScalarValue(code)) {

[ahe, 2012/10/28 12:07:02]

This doesn't look right to me. If we are switching to strings being UTF-16
encoded strings, then we shouldn't be checking for valid Unicode scalar values,
and we should allow users to write strings like this:

"\uD801\uDC12" to create a string composed of two unicode code points that make
up the single character U+10412.

[cshapiro, 2012/10/29 06:37:17]

The specification for string literals was deliberately left the same when the
specification was changed.  Part of the spirit of the change was to avoid
exposing the details of the encoding to the user wherever possible.  I believe
ES6 prescribes similarly and is where our \u{N+} syntax came from.

[ahe, 2012/10/29 08:11:07]

If the underlying string representation is UTF-16, and this is exposed to users,
then users need to be able to create a string literal in UTF-16 notation.

By itself, either of these options are simple:

* Dart strings are a sequence of valid Unicode scalar values.

* Dart strings are a sequence of Unicode code points.

* Dart strings are a sequence of UTF-16 code units.

But you're mixing them all up. I think that is highly confusing and I strongly
urge you to reconsider this design.

[siva, 2012/10/31 01:21:26]

I have decided to drop these changes made to dart2js in this CL. It is clear
that I do not understand the architecture enough, so I have created bug 6418 to
track the two tests that fail in dart2js when run with a VM that has the change
to represent strings as UTf-16.
I have changed the status files to reflect bug 6418 for the two tests so that
the buildbots remain green.

On 2012/10/29 08:11:07, ahe wrote:

         stringParseError(
             "Invalid Unicode scalar value U+${code.toRadixString(16)}",
             token, index);
         return null;
       }
     }
     // String literal successfully validated.
     if (quoting.raw || !containsEscape) {
       // A string without escapes could just as well have been raw.
       return new DartString.rawString(string, length);
     }
     return new DartString.escapedString(string, length);
   }
 }