Add separate scanner only intended for preparsing.

Add separate scanner only intended for preparsing.

Committed: http://code.google.com/p/v8/source/detail?r=5837

[Søren Thygesen Gjesse, 2010-11-17 10:37:45]

LGTM

http://codereview.chromium.org/5063003/diff/1/src/parser.cc
File src/parser.cc (right):

http://codereview.chromium.org/5063003/diff/1/src/parser.cc#newcode4671
src/parser.cc:4671: }
How about using

if ... else if ... else if ... else if ... else ...

To indicate that we will never execute more than one?

http://codereview.chromium.org/5063003/diff/1/src/prescanner.h
File src/prescanner.h (right):

http://codereview.chromium.org/5063003/diff/1/src/prescanner.h#newcode72
src/prescanner.h:72: template <typename UTF16Buffer, typename UTF8Buffer>
Please explain these template parameters a bit. It is rather nasty to have these
all over the place just to declare two member variables of these types. Also
they are both named XXXBuffer, but one is for input and the other is for some
output and their "interfaces" are quite different.

http://codereview.chromium.org/5063003/diff/1/src/prescanner.h#newcode121
src/prescanner.h:121: Location location() const { return current_.location; }
Maybe add a separate comment for peek_location.

http://codereview.chromium.org/5063003/diff/1/src/prescanner.h#newcode349
src/prescanner.h:349: void Scanner<UTF16Buffer, UTF8Buffer>::AddChar(uc32 c) {
AddChar -> AddLiteralChar?

[Lasse Reichstein, 2010-11-17 13:08:39]

http://codereview.chromium.org/5063003/diff/1/src/parser.cc
File src/parser.cc (right):

http://codereview.chromium.org/5063003/diff/1/src/parser.cc#newcode4671
src/parser.cc:4671: }
Done.

http://codereview.chromium.org/5063003/diff/1/src/prescanner.h
File src/prescanner.h (right):

http://codereview.chromium.org/5063003/diff/1/src/prescanner.h#newcode72
src/prescanner.h:72: template <typename UTF16Buffer, typename UTF8Buffer>
Agree.
The long term plan is to have a stand-alone preparser that doesn't depend on the
V8 runtime system. Having these types as templates is a step on the way,
allowing this file to not depend on V8, but still be usable inside V8.
I'll try to move the UTF8Buffer to a shared file in a later iteration, since
there is only one implementation, and it only depends on unibrow and utils.

I haven't yet decided whether to keep the UTF16Buffer as a template type or to
have a purely virtual superclass for it in a V8-independent file. 

The names of the classes suck, but there is no reason why the template
parameters have to have the same names. I'll rename them to be more distinct.

http://codereview.chromium.org/5063003/diff/1/src/prescanner.h#newcode121
src/prescanner.h:121: Location location() const { return current_.location; }
Done.

http://codereview.chromium.org/5063003/diff/1/src/prescanner.h#newcode349
src/prescanner.h:349: void Scanner<UTF16Buffer, UTF8Buffer>::AddChar(uc32 c) {
Done.
Also in scanner.h/.cc.