Merge preparser Scanner with main JavaScript scanner.

Merge preparser Scanner with main JavaScript scanner. 
Optimize scanning of keywords.

[Søren Thygesen Gjesse, 2010-11-19 08:17:20]

LGTM

http://codereview.chromium.org/5188009/diff/1/src/parser.cc
File src/parser.cc (right):

http://codereview.chromium.org/5188009/diff/1/src/parser.cc#newcode4639
src/parser.cc:4639: // Create a Scannerfor the preparser to use as input, and
preparse the source.
Scannerfor -> Scanner for

http://codereview.chromium.org/5188009/diff/1/src/parser.cc#newcode4644
src/parser.cc:4644: int kLiteralFlags) {
kLiteralFlags -> literal_flags.

http://codereview.chromium.org/5188009/diff/1/src/parser.cc#newcode4644
src/parser.cc:4644: int kLiteralFlags) {
Instead of passing flags could the PartialParserRecorder have a method for
requesting this - or does that not make any sense?

http://codereview.chromium.org/5188009/diff/1/src/parser.cc#newcode4673
src/parser.cc:4673: const int kNoLiteralsFlag = 0;
Shouldn't this constant be part of the enum in the JavaScriptScanner?

http://codereview.chromium.org/5188009/diff/1/src/scanner-base.h
File src/scanner-base.h (right):

http://codereview.chromium.org/5188009/diff/1/src/scanner-base.h#newcode339
src/scanner-base.h:339: // More specialized literal scope.
More specialized - in what sense?

http://codereview.chromium.org/5188009/diff/1/src/scanner.h
File src/scanner.h (right):

http://codereview.chromium.org/5188009/diff/1/src/scanner.h#newcode120
src/scanner.h:120: static const int kAllLiteralsFlag = kLiteralNumber |
kLiteralString
This constant should also be in the enum.

[Lasse Reichstein, 2010-11-19 08:51:24]

http://codereview.chromium.org/5188009/diff/1/src/parser.cc
File src/parser.cc (right):

http://codereview.chromium.org/5188009/diff/1/src/parser.cc#newcode4639
src/parser.cc:4639: // Create a Scannerfor the preparser to use as input, and
preparse the source.
On 2010/11/19 08:17:21, Søren Gjesse wrote:
> Scannerfor -> Scanner for

Done.

http://codereview.chromium.org/5188009/diff/1/src/parser.cc#newcode4644
src/parser.cc:4644: int kLiteralFlags) {
On 2010/11/19 08:17:21, Søren Gjesse wrote:
> kLiteralFlags -> literal_flags.

Done.

http://codereview.chromium.org/5188009/diff/1/src/parser.cc#newcode4644
src/parser.cc:4644: int kLiteralFlags) {
I'm afraid it doesn't make any sense :)
The PartialParserRecorder is not the one deciding which literals to record in
the scanner. The PartialParserRecorder has some needs for literals (the
CompleteParserRecorder needs strings and identifiers recorded in order to make
preparser data for symbols, the PartialParserRecorder doesn't), but the scanner
is also used by the real parser which doesn't have a PartialParserRecorder.

I still don't *like* this approach, and I'll try to find a better one where we
don't have to make a test for each character that is scanned.
And perhaps without using utf-8 as intermediate storage.

http://codereview.chromium.org/5188009/diff/1/src/parser.cc#newcode4673
src/parser.cc:4673: const int kNoLiteralsFlag = 0;
I was split about that. Should the LiteralType be the type of elements in a set,
or the type of the set itself. I went for the former, so the full and empty set
are defined separatly.  

Let's add the full and empty set of literal types to the type.

http://codereview.chromium.org/5188009/diff/1/src/scanner-base.h
File src/scanner-base.h (right):

http://codereview.chromium.org/5188009/diff/1/src/scanner-base.h#newcode339
src/scanner-base.h:339: // More specialized literal scope.
Elaborated.

http://codereview.chromium.org/5188009/diff/1/src/scanner.h
File src/scanner.h (right):

http://codereview.chromium.org/5188009/diff/1/src/scanner.h#newcode120
src/scanner.h:120: static const int kAllLiteralsFlag = kLiteralNumber |
kLiteralString
Done.