Experimental lexer: fix internalization and allocation of literals.

Experimental lexer: fix internalization and allocation of literals.

BUG=
R=dcarney@chromium.org, marja@chromium.org

Committed: https://code.google.com/p/v8/source/detail?r=18680

[ulan, 2014-01-17 17:41:21]

PTAL. This doesn't optimize allocation of two-byte literals that have only ascii
chars.

[dcarney, 2014-01-18 08:39:16]

lgtm

[marja, 2014-01-20 07:54:32]

https://codereview.chromium.org/140913009/diff/50001/src/lexer/experimental-s...
File src/lexer/experimental-scanner.cc (right):

https://codereview.chromium.org/140913009/diff/50001/src/lexer/experimental-s...
src/lexer/experimental-scanner.cc:68: if (*cursor >= unibrow::Latin1::kMaxChar)
return true;
Wut? true and false reversed maybe?

https://codereview.chromium.org/140913009/diff/50001/src/lexer/experimental-s...
src/lexer/experimental-scanner.cc:206: int from = literal->offset;
Can you add a comment here (or to a more suitable file) describing why and how.
I suppose NewRawOneByte can cause GC, and this works because we only get the
offset before the possible GC, and then after GC read the actual data from the
string....

https://codereview.chromium.org/140913009/diff/50001/src/lexer/experimental-s...
File src/lexer/experimental-scanner.h (right):

https://codereview.chromium.org/140913009/diff/50001/src/lexer/experimental-s...
src/lexer/experimental-scanner.h:261: if (!next_literal_->is_in_buffer)
next_literal_->beg_pos = -1;
Shouldn't we reset is_in_buffer here too? Why not?

[ulan, 2014-01-20 09:15:00]

https://codereview.chromium.org/140913009/diff/50001/src/lexer/experimental-s...
File src/lexer/experimental-scanner.cc (right):

https://codereview.chromium.org/140913009/diff/50001/src/lexer/experimental-s...
src/lexer/experimental-scanner.cc:68: if (*cursor >= unibrow::Latin1::kMaxChar)
return true;
On 2014/01/20 07:54:33, marja wrote:
> Wut? true and false reversed maybe?
This should be correct. In a two-byte string,  a substring that contains a
non-ascii character can be referenced as (offset, length) pair, i.e. it is a
real substring of the source. A substring that contains only ascii characters is
currently copied into a one-byte buffer (we do not implement yet that
optimization that handles ascii substrings of a two-byte string).

https://codereview.chromium.org/140913009/diff/50001/src/lexer/experimental-s...
src/lexer/experimental-scanner.cc:206: int from = literal->offset;
On 2014/01/20 07:54:33, marja wrote:
> Can you add a comment here (or to a more suitable file) describing why and
how.
> I suppose NewRawOneByte can cause GC, and this works because we only get the
> offset before the possible GC, and then after GC read the actual data from the
> string....

Done.

https://codereview.chromium.org/140913009/diff/50001/src/lexer/experimental-s...
File src/lexer/experimental-scanner.h (right):

https://codereview.chromium.org/140913009/diff/50001/src/lexer/experimental-s...
src/lexer/experimental-scanner.h:261: if (!next_literal_->is_in_buffer)
next_literal_->beg_pos = -1;
On 2014/01/20 07:54:33, marja wrote:
> Shouldn't we reset is_in_buffer here too? Why not?
beg_pos == -1 indicates an invalid buffer, for invalid buffer we do not look at
is_in_buffer.

[marja, 2014-01-20 09:17:45]

lgtm

[ulan, 2014-01-20 09:25:33]

Committed patchset #4 manually as r18680 (presubmit successful).