Add very basic CSS3 Syntax compatible tokenizer

Add very basic CSS3 Syntax compatible tokenizer

DO NOT COMMIT.

This doesn't actually work yet.

This is just to save my work and give others a heads-up
on what this might look like.

BUG=330389

[eseidel, 2014-01-01 08:20:32]

This is just a skeleton and doesn't actually work yet.  No clue when/if I'll
find time to take it further.

This is able to parse a single identifier token.

[abarth-chromium, 2014-01-01 18:47:51]

https://codereview.chromium.org/123053002/diff/30002/Source/core/css/parser/C...
File Source/core/css/parser/CSSToken.h (right):

https://codereview.chromium.org/123053002/diff/30002/Source/core/css/parser/C...
Source/core/css/parser/CSSToken.h:87: CSSToken(CSSTokenType);
Please mark one-argument constructors explicit

https://codereview.chromium.org/123053002/diff/30002/Source/core/css/parser/N...
File Source/core/css/parser/NewCSSTokenizer.cpp (right):

https://codereview.chromium.org/123053002/diff/30002/Source/core/css/parser/N...
Source/core/css/parser/NewCSSTokenizer.cpp:45:
m_string.append(kEndOfFileMarker);
Hum...  String::append is monstrously slow...

https://codereview.chromium.org/123053002/diff/30002/Source/core/css/parser/N...
Source/core/css/parser/NewCSSTokenizer.cpp:124: if (isCSSSpace(cc)) {
I bet it's faster to implement this if-cascade using a lookup table.

https://codereview.chromium.org/123053002/diff/30002/Source/core/css/parser/N...
Source/core/css/parser/NewCSSTokenizer.cpp:321: result.append(consume());
Please use StringBuilder rather than String.  String::append is so bad we should
remove it from the API.

https://codereview.chromium.org/123053002/diff/30002/Source/core/css/parser/N...
Source/core/css/parser/NewCSSTokenizer.cpp:340: String hexChars;
StringBuilder

https://codereview.chromium.org/123053002/diff/30002/Source/core/css/parser/N...
Source/core/css/parser/NewCSSTokenizer.cpp:345: } while (consumedHexDigits < 6
&& isASCIIHexDigit(cc));
You can reserve capacity 6 in the StringBuilder to avoid reallocs.

https://codereview.chromium.org/123053002/diff/30002/Source/core/css/parser/N...
Source/core/css/parser/NewCSSTokenizer.cpp:347: UChar codePoint =
hexChars.toUIntStrict(&ok, 16);
Oh, actually, you don't need to malloc at all in this function.  Just make a
UChar[] on the stack and then call the underlying conversion function instead of
mallocing up a string.

https://codereview.chromium.org/123053002/diff/30002/Source/core/css/parser/N...
File Source/core/css/parser/NewCSSTokenizer.h (right):

https://codereview.chromium.org/123053002/diff/30002/Source/core/css/parser/N...
Source/core/css/parser/NewCSSTokenizer.h:59: String m_string;
I would have expected you'd want to templatize this class based on the CharType
and have two copies of the CSS parser in the binary: one for UChar and one for
LChar.  Taking the is8Bit branch for every peek is going to be expensive.

[eseidel, 2014-01-01 19:31:54]

Sorry. I should have been more clear.  There is no chance in hell this patch is
fast.  It was attempting to see if it was possible to write a CSS Parser from
the CSS Parser spec (which I'm now doubting).  I've not yet attempted to write a
fast one.

[Yoav Weiss, 2014-02-18 22:26:27]

I started implementing a thread-safe MQ parser based on this CL. It is not yet
ready for an official review (so not yet published for review), but a preview is
at https://codereview.chromium.org/171383002