[WIP] Streaming CSS parser

[WIP] Streaming CSS parser

This patch introduces a CSSParserTokenStream class, which is a lazily
tokenized list of CSSParserTokens. It has a similar interface to
CSSParserTokenRange, but as it lazily tokenizes it allows us to get the
character offset of where we've tokenized up to. Instead of doing an
up-front tokenization of entire stylesheets, we have it now interleaved
with parsing. This does *not* make the parser interruptible.

Lazy parsing now just stores the start offset for declaration list
instead of a vector of CSSParserTokens. A function on CSSTokenizer is
added to efficiently skip over a block to support this, without needing
to actually tokenize. Most of the complexity in this comes from url
tokens, which have interesting error recovery. The empty-block
optimization in lazy parsing is removed, although I suspect it is
generally not useful as empty blocks are only going to be used as
sentinel values. We could re-add this later by seeing the number of
characters if needed (the minimum required is 3, e.g. "x:0"), although
it's slightly awkward as we would skip the block but then have to rewind
to tokenize it.

The CSSParserObserverWrapper class, which was used to store character
offset information about tokens and comment locations, is removed since
we can now extract the information directly from stream objects. This
means anywhere that requires this offset information now needs to
operate on stream objects instead of ranges. This requires being careful
when calling peek()/atEnd() as token lookahead advances the current
offset.

For at-rules, we now need to pass in the offsets where the preludes
start. This is so we can retain the generic at-rule parsing logic as
independent of the individual at-rule logic. For style rules we now take
a stream from the start of the selector, instead of a range and a stream
for the block, as the observer requires callbacks for the selector
structure. The callbacks for @import rules now also include the imported
url (i.e. contain the entire prelude) for simplicity.

This patch removes some of the tracing metrics we have around
tokenization and parsing. As these are now interleaved, we can no longer
have separate measurements for tokenization and parsing. We also lose
the information about number of tokens as when lazy parsing is enabled
we will skip tokenization inside style rule declaration blocks.

************* TODO: Work out what to do with blink_style bucketing

This patch greatly reduces the memory requirements of stylesheet parsing
as we discard tokens after parsing them. We currently will allocate a
large contiguous chunk of memory for storing tokens (can be up to a few
MB for large web properties). With this patch, the vector usually caps
at 128 tokens or less.

The two changes in InspectorStyleSheet are due to slight changes in how
we call the CSSParserObserver. Firstly we now call it for style rules
even when the selector is invalid (startRuleHeader, observeSelector,
endRuleHeader, but no start/endRuleBlock). Secondly we now nest keyframe
rules inside the @keyframes rules (endRuleBlock for @keyframes is called
after after the nested blocks).

************* TODO: Investigate and comment on performance
************* TODO: Comment on custom property stuff

Some more details are available in the design doc:
https://docs.google.com/document/d/125OYuPzEXLziVNzzgClkRdggS1iyGEqC75c_H61rPJM/edit

BUG=661854

[Timothy Loh, 2016-11-23 05:02:51]

The CQ bit was checked by timloh@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-11-23 05:03:08]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-11-23 05:10:14]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-11-23 05:10:14]

Dry run: Try jobs failed on following builders:
  chromium_presubmit on master.tryserver.chromium.linux (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.linux/builders/chromium_presub...)

[Timothy Loh, 2016-12-05 02:15:13]

Description was changed from

==========
wip, not for review

BUG=
==========

to

==========
wip, not for review

- Introduce CSSParserTokenStream class
- offset()
- Lazy parsing
- Observer ickiness, offset before/after comments
-- Selectors
- Start/end offset for preludes
- Tracing metrics
- Performance implication
- Duplicated consumeDeclarationList for @apply until we fix custom property
input preservation
- CSSTokenizer::skipToBlockEnd()
-- url tokens :\

BUG=
==========

[Timothy Loh, 2016-12-05 02:51:31]

Description was changed from

==========
wip, not for review

- Introduce CSSParserTokenStream class
- offset()
- Lazy parsing
- Observer ickiness, offset before/after comments
-- Selectors
- Start/end offset for preludes
- Tracing metrics
- Performance implication
- Duplicated consumeDeclarationList for @apply until we fix custom property
input preservation
- CSSTokenizer::skipToBlockEnd()
-- url tokens :\

BUG=
==========

to

==========
wip, not for review

- Introduce CSSParserTokenStream class
- offset()
- Lazy parsing
- Observer ickiness, offset before/after comments
-- Selectors
- Start/end offset for preludes
- Tracing metrics
-- Token count no longer available if we use lazy parsing, so not hooked up . .
.
- Performance implication
- Duplicated consumeDeclarationList for @apply until we fix custom property
input preservation
- CSSTokenizer::skipToBlockEnd()
-- url tokens :\

BUG=
==========

[Timothy Loh, 2016-12-05 05:45:48]

Description was changed from

==========
wip, not for review

- Introduce CSSParserTokenStream class
- offset()
- Lazy parsing
- Observer ickiness, offset before/after comments
-- Selectors
- Start/end offset for preludes
- Tracing metrics
-- Token count no longer available if we use lazy parsing, so not hooked up . .
.
- Performance implication
- Duplicated consumeDeclarationList for @apply until we fix custom property
input preservation
- CSSTokenizer::skipToBlockEnd()
-- url tokens :\

BUG=
==========

to

==========
wip, not for review

- Introduce CSSParserTokenStream class
- offset()
- Lazy parsing
-- empty block optimisation
- Observer ickiness, offset before/after comments
-- Selectors
- Start/end offset for preludes
- Tracing metrics
-- Token count no longer available if we use lazy parsing, so not hooked up . .
.
- Performance implication
- Duplicated consumeDeclarationList for @apply until we fix custom property
input preservation
- CSSTokenizer::skipToBlockEnd()
-- url tokens :\

BUG=
==========

[Timothy Loh, 2016-12-05 06:30:18]

Description was changed from

==========
wip, not for review

- Introduce CSSParserTokenStream class
- offset()
- Lazy parsing
-- empty block optimisation
- Observer ickiness, offset before/after comments
-- Selectors
- Start/end offset for preludes
- Tracing metrics
-- Token count no longer available if we use lazy parsing, so not hooked up . .
.
- Performance implication
- Duplicated consumeDeclarationList for @apply until we fix custom property
input preservation
- CSSTokenizer::skipToBlockEnd()
-- url tokens :\

BUG=
==========

to

==========
wip, not for review

This patch introduces a CSSParserTokenStream class, which is a lazily
tokenized list of CSSParserTokens. It has a similar interface to
CSSParserTokenRange, but as it lazily tokenizes it allows us to get the
character offset of where we've tokenized up to. Instead of doing an
up-front tokenization of entire stylesheets, we have it now interleaved
with parsing.

Lazy parsing now just stores the start offset for declaration list
instead of a vector of CSSParserTokens. A function on CSSTokenizer is
added to efficiently skip over a block to support this, without needing
to actually tokenize. Most of the complexity in this comes from url
tokens, which have interesting error recovery. The empty-block
optimization in lazy parsing is removed, although I suspect it is
generally not useful as empty blocks are only going to be used as
sentinel values.

The CSSParserObserverWrapper class, which was used to store character
offset information about tokens and comment locations, is removed since
we can now extract the information directly from stream objects. This
means anywhere that requires this offset information now needs to
operate on stream objects instead of ranges. This requires being careful
when calling peek()/atEnd() as token lookahead advances the current
offset.

For at-rules, we now need to pass in the offsets where the preludes
start. This is so we can retain the generic at-rule parsing logic as
independent of the individual at-rule logic. For style rules we now take
a stream from the start of the selector, instead of a range and a stream
for the block, as the observer requires callbacks for the selector
structure.

This patch removes some of the tracing metrics we have around
tokenization and parsing. As these are now interleaved, we can no longer
have separate measurements for tokenization and parsing. We also lose
the information about number of tokens as when lazy parsing is enabled
we will skip tokenization inside style rule declaration blocks.

This patch greatly reduces the memory requirements of stylesheet parsing
as we discard tokens after parsing them. We currently will allocate a
large contiguous chunk of memory for storing tokens (can be up to a few
MB for large web properties). With this patch, the vector usually caps
at 128 tokens or less.

- Performance implication
- Duplicated consumeDeclarationList for @apply until we fix custom property
input preservation
- Custom property stuff

[DOC LINK]

BUG=
==========

[Timothy Loh, 2016-12-05 06:30:38]

Description was changed from

==========
wip, not for review

This patch introduces a CSSParserTokenStream class, which is a lazily
tokenized list of CSSParserTokens. It has a similar interface to
CSSParserTokenRange, but as it lazily tokenizes it allows us to get the
character offset of where we've tokenized up to. Instead of doing an
up-front tokenization of entire stylesheets, we have it now interleaved
with parsing.

Lazy parsing now just stores the start offset for declaration list
instead of a vector of CSSParserTokens. A function on CSSTokenizer is
added to efficiently skip over a block to support this, without needing
to actually tokenize. Most of the complexity in this comes from url
tokens, which have interesting error recovery. The empty-block
optimization in lazy parsing is removed, although I suspect it is
generally not useful as empty blocks are only going to be used as
sentinel values.

The CSSParserObserverWrapper class, which was used to store character
offset information about tokens and comment locations, is removed since
we can now extract the information directly from stream objects. This
means anywhere that requires this offset information now needs to
operate on stream objects instead of ranges. This requires being careful
when calling peek()/atEnd() as token lookahead advances the current
offset.

For at-rules, we now need to pass in the offsets where the preludes
start. This is so we can retain the generic at-rule parsing logic as
independent of the individual at-rule logic. For style rules we now take
a stream from the start of the selector, instead of a range and a stream
for the block, as the observer requires callbacks for the selector
structure.

This patch removes some of the tracing metrics we have around
tokenization and parsing. As these are now interleaved, we can no longer
have separate measurements for tokenization and parsing. We also lose
the information about number of tokens as when lazy parsing is enabled
we will skip tokenization inside style rule declaration blocks.

This patch greatly reduces the memory requirements of stylesheet parsing
as we discard tokens after parsing them. We currently will allocate a
large contiguous chunk of memory for storing tokens (can be up to a few
MB for large web properties). With this patch, the vector usually caps
at 128 tokens or less.

- Performance implication
- Duplicated consumeDeclarationList for @apply until we fix custom property
input preservation
- Custom property stuff

[DOC LINK]

BUG=
==========

to

==========
wip, not for review

This patch introduces a CSSParserTokenStream class, which is a lazily
tokenized list of CSSParserTokens. It has a similar interface to
CSSParserTokenRange, but as it lazily tokenizes it allows us to get the
character offset of where we've tokenized up to. Instead of doing an
up-front tokenization of entire stylesheets, we have it now interleaved
with parsing.

Lazy parsing now just stores the start offset for declaration list
instead of a vector of CSSParserTokens. A function on CSSTokenizer is
added to efficiently skip over a block to support this, without needing
to actually tokenize. Most of the complexity in this comes from url
tokens, which have interesting error recovery. The empty-block
optimization in lazy parsing is removed, although I suspect it is
generally not useful as empty blocks are only going to be used as
sentinel values.

The CSSParserObserverWrapper class, which was used to store character
offset information about tokens and comment locations, is removed since
we can now extract the information directly from stream objects. This
means anywhere that requires this offset information now needs to
operate on stream objects instead of ranges. This requires being careful
when calling peek()/atEnd() as token lookahead advances the current
offset.

For at-rules, we now need to pass in the offsets where the preludes
start. This is so we can retain the generic at-rule parsing logic as
independent of the individual at-rule logic. For style rules we now take
a stream from the start of the selector, instead of a range and a stream
for the block, as the observer requires callbacks for the selector
structure.

This patch removes some of the tracing metrics we have around
tokenization and parsing. As these are now interleaved, we can no longer
have separate measurements for tokenization and parsing. We also lose
the information about number of tokens as when lazy parsing is enabled
we will skip tokenization inside style rule declaration blocks.

This patch greatly reduces the memory requirements of stylesheet parsing
as we discard tokens after parsing them. We currently will allocate a
large contiguous chunk of memory for storing tokens (can be up to a few
MB for large web properties). With this patch, the vector usually caps
at 128 tokens or less.

- Performance implication
- Duplicated consumeDeclarationList for @apply until we fix custom property
input preservation
- Custom property stuff

[DOC LINK]

BUG=
==========

[Timothy Loh, 2016-12-05 06:52:33]

Description was changed from

==========
wip, not for review

This patch introduces a CSSParserTokenStream class, which is a lazily
tokenized list of CSSParserTokens. It has a similar interface to
CSSParserTokenRange, but as it lazily tokenizes it allows us to get the
character offset of where we've tokenized up to. Instead of doing an
up-front tokenization of entire stylesheets, we have it now interleaved
with parsing.

Lazy parsing now just stores the start offset for declaration list
instead of a vector of CSSParserTokens. A function on CSSTokenizer is
added to efficiently skip over a block to support this, without needing
to actually tokenize. Most of the complexity in this comes from url
tokens, which have interesting error recovery. The empty-block
optimization in lazy parsing is removed, although I suspect it is
generally not useful as empty blocks are only going to be used as
sentinel values.

The CSSParserObserverWrapper class, which was used to store character
offset information about tokens and comment locations, is removed since
we can now extract the information directly from stream objects. This
means anywhere that requires this offset information now needs to
operate on stream objects instead of ranges. This requires being careful
when calling peek()/atEnd() as token lookahead advances the current
offset.

For at-rules, we now need to pass in the offsets where the preludes
start. This is so we can retain the generic at-rule parsing logic as
independent of the individual at-rule logic. For style rules we now take
a stream from the start of the selector, instead of a range and a stream
for the block, as the observer requires callbacks for the selector
structure.

This patch removes some of the tracing metrics we have around
tokenization and parsing. As these are now interleaved, we can no longer
have separate measurements for tokenization and parsing. We also lose
the information about number of tokens as when lazy parsing is enabled
we will skip tokenization inside style rule declaration blocks.

This patch greatly reduces the memory requirements of stylesheet parsing
as we discard tokens after parsing them. We currently will allocate a
large contiguous chunk of memory for storing tokens (can be up to a few
MB for large web properties). With this patch, the vector usually caps
at 128 tokens or less.

- Performance implication
- Duplicated consumeDeclarationList for @apply until we fix custom property
input preservation
- Custom property stuff

[DOC LINK]

BUG=
==========

to

==========
wip, not for review

This patch introduces a CSSParserTokenStream class, which is a lazily
tokenized list of CSSParserTokens. It has a similar interface to
CSSParserTokenRange, but as it lazily tokenizes it allows us to get the
character offset of where we've tokenized up to. Instead of doing an
up-front tokenization of entire stylesheets, we have it now interleaved
with parsing.

Lazy parsing now just stores the start offset for declaration list
instead of a vector of CSSParserTokens. A function on CSSTokenizer is
added to efficiently skip over a block to support this, without needing
to actually tokenize. Most of the complexity in this comes from url
tokens, which have interesting error recovery. The empty-block
optimization in lazy parsing is removed, although I suspect it is
generally not useful as empty blocks are only going to be used as
sentinel values. We could re-add this later by seeing the number of
characters if needed (the minimum required is 3, e.g. "x:0").

The CSSParserObserverWrapper class, which was used to store character
offset information about tokens and comment locations, is removed since
we can now extract the information directly from stream objects. This
means anywhere that requires this offset information now needs to
operate on stream objects instead of ranges. This requires being careful
when calling peek()/atEnd() as token lookahead advances the current
offset.

For at-rules, we now need to pass in the offsets where the preludes
start. This is so we can retain the generic at-rule parsing logic as
independent of the individual at-rule logic. For style rules we now take
a stream from the start of the selector, instead of a range and a stream
for the block, as the observer requires callbacks for the selector
structure.

This patch removes some of the tracing metrics we have around
tokenization and parsing. As these are now interleaved, we can no longer
have separate measurements for tokenization and parsing. We also lose
the information about number of tokens as when lazy parsing is enabled
we will skip tokenization inside style rule declaration blocks.

This patch greatly reduces the memory requirements of stylesheet parsing
as we discard tokens after parsing them. We currently will allocate a
large contiguous chunk of memory for storing tokens (can be up to a few
MB for large web properties). With this patch, the vector usually caps
at 128 tokens or less.

- Performance implication
- Duplicated consumeDeclarationList for @apply until we fix custom property
input preservation
- Custom property stuff

[DOC LINK]

BUG=
==========

[Timothy Loh, 2016-12-05 07:06:46]

The CQ bit was checked by timloh@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-12-05 07:07:02]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-12-05 08:19:24]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-12-05 08:19:24]

Dry run: Try jobs failed on following builders:
  win_chromium_x64_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_...)

[Timothy Loh, 2016-12-06 05:28:43]

Description was changed from

==========
wip, not for review

This patch introduces a CSSParserTokenStream class, which is a lazily
tokenized list of CSSParserTokens. It has a similar interface to
CSSParserTokenRange, but as it lazily tokenizes it allows us to get the
character offset of where we've tokenized up to. Instead of doing an
up-front tokenization of entire stylesheets, we have it now interleaved
with parsing.

Lazy parsing now just stores the start offset for declaration list
instead of a vector of CSSParserTokens. A function on CSSTokenizer is
added to efficiently skip over a block to support this, without needing
to actually tokenize. Most of the complexity in this comes from url
tokens, which have interesting error recovery. The empty-block
optimization in lazy parsing is removed, although I suspect it is
generally not useful as empty blocks are only going to be used as
sentinel values. We could re-add this later by seeing the number of
characters if needed (the minimum required is 3, e.g. "x:0").

The CSSParserObserverWrapper class, which was used to store character
offset information about tokens and comment locations, is removed since
we can now extract the information directly from stream objects. This
means anywhere that requires this offset information now needs to
operate on stream objects instead of ranges. This requires being careful
when calling peek()/atEnd() as token lookahead advances the current
offset.

For at-rules, we now need to pass in the offsets where the preludes
start. This is so we can retain the generic at-rule parsing logic as
independent of the individual at-rule logic. For style rules we now take
a stream from the start of the selector, instead of a range and a stream
for the block, as the observer requires callbacks for the selector
structure.

This patch removes some of the tracing metrics we have around
tokenization and parsing. As these are now interleaved, we can no longer
have separate measurements for tokenization and parsing. We also lose
the information about number of tokens as when lazy parsing is enabled
we will skip tokenization inside style rule declaration blocks.

This patch greatly reduces the memory requirements of stylesheet parsing
as we discard tokens after parsing them. We currently will allocate a
large contiguous chunk of memory for storing tokens (can be up to a few
MB for large web properties). With this patch, the vector usually caps
at 128 tokens or less.

- Performance implication
- Duplicated consumeDeclarationList for @apply until we fix custom property
input preservation
- Custom property stuff

[DOC LINK]

BUG=
==========

to

==========
wip, not for review

This patch introduces a CSSParserTokenStream class, which is a lazily
tokenized list of CSSParserTokens. It has a similar interface to
CSSParserTokenRange, but as it lazily tokenizes it allows us to get the
character offset of where we've tokenized up to. Instead of doing an
up-front tokenization of entire stylesheets, we have it now interleaved
with parsing.

Lazy parsing now just stores the start offset for declaration list
instead of a vector of CSSParserTokens. A function on CSSTokenizer is
added to efficiently skip over a block to support this, without needing
to actually tokenize. Most of the complexity in this comes from url
tokens, which have interesting error recovery. The empty-block
optimization in lazy parsing is removed, although I suspect it is
generally not useful as empty blocks are only going to be used as
sentinel values. We could re-add this later by seeing the number of
characters if needed (the minimum required is 3, e.g. "x:0").

The CSSParserObserverWrapper class, which was used to store character
offset information about tokens and comment locations, is removed since
we can now extract the information directly from stream objects. This
means anywhere that requires this offset information now needs to
operate on stream objects instead of ranges. This requires being careful
when calling peek()/atEnd() as token lookahead advances the current
offset.

For at-rules, we now need to pass in the offsets where the preludes
start. This is so we can retain the generic at-rule parsing logic as
independent of the individual at-rule logic. For style rules we now take
a stream from the start of the selector, instead of a range and a stream
for the block, as the observer requires callbacks for the selector
structure.

This patch removes some of the tracing metrics we have around
tokenization and parsing. As these are now interleaved, we can no longer
have separate measurements for tokenization and parsing. We also lose
the information about number of tokens as when lazy parsing is enabled
we will skip tokenization inside style rule declaration blocks.

This patch greatly reduces the memory requirements of stylesheet parsing
as we discard tokens after parsing them. We currently will allocate a
large contiguous chunk of memory for storing tokens (can be up to a few
MB for large web properties). With this patch, the vector usually caps
at 128 tokens or less.

The two changes in InspectorStyleSheet are due to slight changes in how
we call the CSSParserObserver. Firstly we now call it for style rules
even when the selector is invalid (startRuleHeader, observeSelector,
endRuleHeader, but no start/endRuleBlock). Secondly we now nest keyframe
rules inside the @keyframes rules (endRuleBlock for @keyframes is called
after after the nested blocks).

- Performance implication
- Duplicated consumeDeclarationList for @apply until we fix custom property
input preservation
- Custom property stuff

[DOC LINK]

BUG=
==========

[Timothy Loh, 2016-12-06 06:57:49]

csharrison: This isn't quite done yet, but if you have a spare moment and it
isn't too much hassle it'd be great if you could try get performance numbers for
this. I'll probably profile later to try and get small improvements. My quick
non-scientific tests suggested this regresses stylesheet parsing by maybe ~10%,
but I haven't tested with lazy parsing. Perhaps if this further improves lazy
parsing we could accept a small regression for the non-lazy case?

[Charlie Harrison, 2016-12-06 21:14:52]

csharrison@chromium.org changed reviewers:
+ csharrison@chromium.org

[Charlie Harrison, 2016-12-06 21:14:53]

Awesome work! I just sent you a brief manual test I did with the 2x2 modes of
streaming x lazy and saw only improvement or status quo with lazy parsing, but
did observe regressions in normal mode.

We should try to get in a place where we can land this without significant
regressions though, as we don't want to force ourselves to launch lazy parsing
before we are ready.

Please let me know if you'd like some help profiling or doing a pass over the
code. I am curious why the regression is so prominent. Perhaps we are losing
cache locality of tokenizing when we interleave parsing? A quick glance at
profiler results was not super illuminating, but I remain optimistic.

[Timothy Loh, 2016-12-09 04:44:34]

The CQ bit was checked by timloh@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-12-09 04:44:54]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-12-09 06:01:02]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-12-09 06:01:03]

Dry run: Try jobs failed on following builders:
  mac_chromium_rel_ng on master.tryserver.chromium.mac (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_...)

[Timothy Loh, 2016-12-12 06:50:38]

Description was changed from

==========
wip, not for review

This patch introduces a CSSParserTokenStream class, which is a lazily
tokenized list of CSSParserTokens. It has a similar interface to
CSSParserTokenRange, but as it lazily tokenizes it allows us to get the
character offset of where we've tokenized up to. Instead of doing an
up-front tokenization of entire stylesheets, we have it now interleaved
with parsing.

Lazy parsing now just stores the start offset for declaration list
instead of a vector of CSSParserTokens. A function on CSSTokenizer is
added to efficiently skip over a block to support this, without needing
to actually tokenize. Most of the complexity in this comes from url
tokens, which have interesting error recovery. The empty-block
optimization in lazy parsing is removed, although I suspect it is
generally not useful as empty blocks are only going to be used as
sentinel values. We could re-add this later by seeing the number of
characters if needed (the minimum required is 3, e.g. "x:0").

The CSSParserObserverWrapper class, which was used to store character
offset information about tokens and comment locations, is removed since
we can now extract the information directly from stream objects. This
means anywhere that requires this offset information now needs to
operate on stream objects instead of ranges. This requires being careful
when calling peek()/atEnd() as token lookahead advances the current
offset.

For at-rules, we now need to pass in the offsets where the preludes
start. This is so we can retain the generic at-rule parsing logic as
independent of the individual at-rule logic. For style rules we now take
a stream from the start of the selector, instead of a range and a stream
for the block, as the observer requires callbacks for the selector
structure.

This patch removes some of the tracing metrics we have around
tokenization and parsing. As these are now interleaved, we can no longer
have separate measurements for tokenization and parsing. We also lose
the information about number of tokens as when lazy parsing is enabled
we will skip tokenization inside style rule declaration blocks.

This patch greatly reduces the memory requirements of stylesheet parsing
as we discard tokens after parsing them. We currently will allocate a
large contiguous chunk of memory for storing tokens (can be up to a few
MB for large web properties). With this patch, the vector usually caps
at 128 tokens or less.

The two changes in InspectorStyleSheet are due to slight changes in how
we call the CSSParserObserver. Firstly we now call it for style rules
even when the selector is invalid (startRuleHeader, observeSelector,
endRuleHeader, but no start/endRuleBlock). Secondly we now nest keyframe
rules inside the @keyframes rules (endRuleBlock for @keyframes is called
after after the nested blocks).

- Performance implication
- Duplicated consumeDeclarationList for @apply until we fix custom property
input preservation
- Custom property stuff

[DOC LINK]

BUG=
==========

to

==========
wip, not for review

This patch introduces a CSSParserTokenStream class, which is a lazily
tokenized list of CSSParserTokens. It has a similar interface to
CSSParserTokenRange, but as it lazily tokenizes it allows us to get the
character offset of where we've tokenized up to. Instead of doing an
up-front tokenization of entire stylesheets, we have it now interleaved
with parsing.

Lazy parsing now just stores the start offset for declaration list
instead of a vector of CSSParserTokens. A function on CSSTokenizer is
added to efficiently skip over a block to support this, without needing
to actually tokenize. Most of the complexity in this comes from url
tokens, which have interesting error recovery. The empty-block
optimization in lazy parsing is removed, although I suspect it is
generally not useful as empty blocks are only going to be used as
sentinel values. We could re-add this later by seeing the number of
characters if needed (the minimum required is 3, e.g. "x:0").

The CSSParserObserverWrapper class, which was used to store character
offset information about tokens and comment locations, is removed since
we can now extract the information directly from stream objects. This
means anywhere that requires this offset information now needs to
operate on stream objects instead of ranges. This requires being careful
when calling peek()/atEnd() as token lookahead advances the current
offset.

For at-rules, we now need to pass in the offsets where the preludes
start. This is so we can retain the generic at-rule parsing logic as
independent of the individual at-rule logic. For style rules we now take
a stream from the start of the selector, instead of a range and a stream
for the block, as the observer requires callbacks for the selector
structure. The callbacks for @import rules now also include the imported
url (i.e. contain the entire prelude) for simplicity.

This patch removes some of the tracing metrics we have around
tokenization and parsing. As these are now interleaved, we can no longer
have separate measurements for tokenization and parsing. We also lose
the information about number of tokens as when lazy parsing is enabled
we will skip tokenization inside style rule declaration blocks.

This patch greatly reduces the memory requirements of stylesheet parsing
as we discard tokens after parsing them. We currently will allocate a
large contiguous chunk of memory for storing tokens (can be up to a few
MB for large web properties). With this patch, the vector usually caps
at 128 tokens or less.

The two changes in InspectorStyleSheet are due to slight changes in how
we call the CSSParserObserver. Firstly we now call it for style rules
even when the selector is invalid (startRuleHeader, observeSelector,
endRuleHeader, but no start/endRuleBlock). Secondly we now nest keyframe
rules inside the @keyframes rules (endRuleBlock for @keyframes is called
after after the nested blocks).

- Performance implication
- Custom property stuff

[DOC LINK]

BUG=
==========

[Timothy Loh, 2016-12-15 04:41:37]

Description was changed from

==========
wip, not for review

This patch introduces a CSSParserTokenStream class, which is a lazily
tokenized list of CSSParserTokens. It has a similar interface to
CSSParserTokenRange, but as it lazily tokenizes it allows us to get the
character offset of where we've tokenized up to. Instead of doing an
up-front tokenization of entire stylesheets, we have it now interleaved
with parsing.

Lazy parsing now just stores the start offset for declaration list
instead of a vector of CSSParserTokens. A function on CSSTokenizer is
added to efficiently skip over a block to support this, without needing
to actually tokenize. Most of the complexity in this comes from url
tokens, which have interesting error recovery. The empty-block
optimization in lazy parsing is removed, although I suspect it is
generally not useful as empty blocks are only going to be used as
sentinel values. We could re-add this later by seeing the number of
characters if needed (the minimum required is 3, e.g. "x:0").

The CSSParserObserverWrapper class, which was used to store character
offset information about tokens and comment locations, is removed since
we can now extract the information directly from stream objects. This
means anywhere that requires this offset information now needs to
operate on stream objects instead of ranges. This requires being careful
when calling peek()/atEnd() as token lookahead advances the current
offset.

For at-rules, we now need to pass in the offsets where the preludes
start. This is so we can retain the generic at-rule parsing logic as
independent of the individual at-rule logic. For style rules we now take
a stream from the start of the selector, instead of a range and a stream
for the block, as the observer requires callbacks for the selector
structure. The callbacks for @import rules now also include the imported
url (i.e. contain the entire prelude) for simplicity.

This patch removes some of the tracing metrics we have around
tokenization and parsing. As these are now interleaved, we can no longer
have separate measurements for tokenization and parsing. We also lose
the information about number of tokens as when lazy parsing is enabled
we will skip tokenization inside style rule declaration blocks.

This patch greatly reduces the memory requirements of stylesheet parsing
as we discard tokens after parsing them. We currently will allocate a
large contiguous chunk of memory for storing tokens (can be up to a few
MB for large web properties). With this patch, the vector usually caps
at 128 tokens or less.

The two changes in InspectorStyleSheet are due to slight changes in how
we call the CSSParserObserver. Firstly we now call it for style rules
even when the selector is invalid (startRuleHeader, observeSelector,
endRuleHeader, but no start/endRuleBlock). Secondly we now nest keyframe
rules inside the @keyframes rules (endRuleBlock for @keyframes is called
after after the nested blocks).

- Performance implication
- Custom property stuff

[DOC LINK]

BUG=
==========

to

==========
[WIP] Streaming CSS parser

This patch introduces a CSSParserTokenStream class, which is a lazily
tokenized list of CSSParserTokens. It has a similar interface to
CSSParserTokenRange, but as it lazily tokenizes it allows us to get the
character offset of where we've tokenized up to. Instead of doing an
up-front tokenization of entire stylesheets, we have it now interleaved
with parsing. This does *not* make the parser interruptible.

Lazy parsing now just stores the start offset for declaration list
instead of a vector of CSSParserTokens. A function on CSSTokenizer is
added to efficiently skip over a block to support this, without needing
to actually tokenize. Most of the complexity in this comes from url
tokens, which have interesting error recovery. The empty-block
optimization in lazy parsing is removed, although I suspect it is
generally not useful as empty blocks are only going to be used as
sentinel values. We could re-add this later by seeing the number of
characters if needed (the minimum required is 3, e.g. "x:0"), although
it's slightly awkward as we would skip the block but then have to rewind
to tokenize it.

The CSSParserObserverWrapper class, which was used to store character
offset information about tokens and comment locations, is removed since
we can now extract the information directly from stream objects. This
means anywhere that requires this offset information now needs to
operate on stream objects instead of ranges. This requires being careful
when calling peek()/atEnd() as token lookahead advances the current
offset.

For at-rules, we now need to pass in the offsets where the preludes
start. This is so we can retain the generic at-rule parsing logic as
independent of the individual at-rule logic. For style rules we now take
a stream from the start of the selector, instead of a range and a stream
for the block, as the observer requires callbacks for the selector
structure. The callbacks for @import rules now also include the imported
url (i.e. contain the entire prelude) for simplicity.

This patch removes some of the tracing metrics we have around
tokenization and parsing. As these are now interleaved, we can no longer
have separate measurements for tokenization and parsing. We also lose
the information about number of tokens as when lazy parsing is enabled
we will skip tokenization inside style rule declaration blocks.

************* TODO: Work out what to do with blink_style bucketing

This patch greatly reduces the memory requirements of stylesheet parsing
as we discard tokens after parsing them. We currently will allocate a
large contiguous chunk of memory for storing tokens (can be up to a few
MB for large web properties). With this patch, the vector usually caps
at 128 tokens or less.

The two changes in InspectorStyleSheet are due to slight changes in how
we call the CSSParserObserver. Firstly we now call it for style rules
even when the selector is invalid (startRuleHeader, observeSelector,
endRuleHeader, but no start/endRuleBlock). Secondly we now nest keyframe
rules inside the @keyframes rules (endRuleBlock for @keyframes is called
after after the nested blocks).

************* TODO: Investigate and comment on performance
************* TODO: Comment on custom property stuff

Some more details are available in the design doc:
https://docs.google.com/document/d/125OYuPzEXLziVNzzgClkRdggS1iyGEqC75c_H61rP...

BUG=661854
==========

[Timothy Loh, 2016-12-15 05:09:26]

The CQ bit was checked by timloh@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2016-12-15 05:09:46]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[Timothy Loh, 2016-12-15 05:29:33]

timloh@chromium.org changed reviewers:
+ alancutter@chromium.org

[Timothy Loh, 2016-12-15 05:29:34]

There's still some stuff missing but I think this is ready for review now.
Mainly what's left is to investigate performance, fix up a few tests and work
out how to proceed with the telemetry benchmarks.

[commit-bot: I haz the power, 2016-12-15 06:22:26]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-12-15 06:22:27]

Dry run: Try jobs failed on following builders:
  win_chromium_x64_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_...)

[Charlie Harrison, 2016-12-22 15:47:27]

FYI I will probably not have time to review this until after the new year
holiday.

[Timothy Loh, 2017-01-06 04:06:30]

The CQ bit was checked by timloh@chromium.org to run a CQ dry run

[commit-bot: I haz the power, 2017-01-06 04:06:48]

Dry run: CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2017-01-06 05:12:20]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2017-01-06 05:12:21]

Dry run: Try jobs failed on following builders:
  win_chromium_x64_rel_ng on master.tryserver.chromium.win (JOB_FAILED,
http://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_x64_...)

[Charlie Harrison, 2017-01-09 21:35:07]

Generally looks good, though I am still learning this code.

https://codereview.chromium.org/2503683003/diff/260001/third_party/WebKit/Sou...
File third_party/WebKit/Source/core/css/parser/CSSParserTokenStream.cpp (right):

https://codereview.chromium.org/2503683003/diff/260001/third_party/WebKit/Sou...
third_party/WebKit/Source/core/css/parser/CSSParserTokenStream.cpp:13: //
TODO(timloh): Should be using CSSTokenier::skipToBlockEnd() instead.
s/CSSTokenier/CSSTokenizer

https://codereview.chromium.org/2503683003/diff/260001/third_party/WebKit/Sou...
third_party/WebKit/Source/core/css/parser/CSSParserTokenStream.cpp:16: if
(m_tokenizer.m_tokens.size() == m_currentIndex) {
could these conditions be re-written in terms of hasLookedAhead?

https://codereview.chromium.org/2503683003/diff/260001/third_party/WebKit/Sou...
third_party/WebKit/Source/core/css/parser/CSSParserTokenStream.cpp:28: const
CSSParserToken& CSSParserTokenStream::peekInternal() {
If tokenizeSingle() returned a token, this could be rewritten to:

if (hasLookedAhead)
  return m_tokenizer.m_tokens[m_currentIndex]
return m_tokenizer.tokenizeSingle()


Assuming tokenizeSingle just returns the staticEOFToken on end.

https://codereview.chromium.org/2503683003/diff/260001/third_party/WebKit/Sou...
File third_party/WebKit/Source/core/css/parser/CSSParserTokenStream.h (right):

https://codereview.chromium.org/2503683003/diff/260001/third_party/WebKit/Sou...
third_party/WebKit/Source/core/css/parser/CSSParserTokenStream.h:22: enum
MakeSubStreamTag { MakeSubStream };
optional: Put this right above the constructor that needs this, so as to make it
obvious this enum's purpose.

https://codereview.chromium.org/2503683003/diff/260001/third_party/WebKit/Sou...
third_party/WebKit/Source/core/css/parser/CSSParserTokenStream.h:50:
m_tokenizer.m_tokens.remove(m_startIndex, m_currentIndex - m_startIndex);
I'll need to look at all usage, but it seems like if it is possible to be
removing in the middle of a Vector this could have bad perf consequences.

Disclaimer: I'm not seeing any of this in profiles.

https://codereview.chromium.org/2503683003/diff/260001/third_party/WebKit/Sou...
third_party/WebKit/Source/core/css/parser/CSSParserTokenStream.h:133: bool
hasLookedAhead() {
Maybe replace this method with:

DCHECK(m_currentIndex == m_tokenizer.size() || m_currentIndex ==
m_tokenizer.m_tokens.size() - 1);
return m_currentIndex != m_tokenizer.m_tokens.size();

Consider this optional if the current code is successfully optimized away
though.

https://codereview.chromium.org/2503683003/diff/260001/third_party/WebKit/Sou...
File third_party/WebKit/Source/core/css/parser/CSSSelectorParser.cpp (right):

https://codereview.chromium.org/2503683003/diff/260001/third_party/WebKit/Sou...
third_party/WebKit/Source/core/css/parser/CSSSelectorParser.cpp:162: return
CSSSelectorList::adoptSelectorVector(selectorList);
Not new code, but why do we allocate some selectors on the stack, then copy them
into a separate CSSSelectorList?

Context: I am seeing the fastMalloc (and other things) in adoptSelectorVector in
profiles as a non-significant chunk of time here.

https://codereview.chromium.org/2503683003/diff/260001/third_party/WebKit/Sou...
File third_party/WebKit/Source/core/css/parser/CSSTokenizer.cpp (left):

https://codereview.chromium.org/2503683003/diff/260001/third_party/WebKit/Sou...
third_party/WebKit/Source/core/css/parser/CSSTokenizer.cpp:34:
m_tokens.reserveInitialCapacity(string.length() / 3);
w000t!

https://codereview.chromium.org/2503683003/diff/260001/third_party/WebKit/Sou...
File third_party/WebKit/Source/core/css/parser/CSSTokenizer.cpp (right):

https://codereview.chromium.org/2503683003/diff/260001/third_party/WebKit/Sou...
third_party/WebKit/Source/core/css/parser/CSSTokenizer.cpp:26:
m_input.advance(startOffset);
It looks like advance() does not change the underlying length. Should I be
worried at the reserveInitialCapacity() call below?

https://codereview.chromium.org/2503683003/diff/260001/third_party/WebKit/Sou...
third_party/WebKit/Source/core/css/parser/CSSTokenizer.cpp:48: void
CSSTokenizer::tokenizeSingle() {
Maybe tokenizeSingle could return something useful, like whether we're finished
tokenizing, or maybe the last token tokenized?

https://codereview.chromium.org/2503683003/diff/260001/third_party/WebKit/Sou...
third_party/WebKit/Source/core/css/parser/CSSTokenizer.cpp:677: if
(isASCIIAlphaCaselessEqual(nextChar, cc))
Hm, this function has a compiler hint that the branch is likely to be true. I'm
worried that might lead to poor branch prediction. WDYT?