Introduce newline tokens into the token stream.

Introduce newline tokens into the token stream.

CL only introduces the tokens into the compressed token stream. We do not
use the newline tokens (yet).

Overhead:
snapshot: 6.4% (860k/808k)
compressed token stream (gen_snapshot): 21% (314312/259907)

BUG=
R=asiva@google.com, hausner@google.com

Committed: https://code.google.com/p/dart/source/detail?r=27677

[Michael Lippautz (Google), 2013-09-18 19:40:06]

Parser changes are only getting rid of "TokenPos() - 1". We could also loop
backwards because (I think) there can only be kNEWLINE or kRBRACE. Would rather
avoid the looping though.

The introduced overhead is slightly bigger than initially  estimated.

[hausner, 2013-09-18 22:27:47]

Generally lgtm but try to eliminate the newline counters in the scanner.

https://codereview.chromium.org/23452043/diff/1/runtime/vm/object_test.cc
File runtime/vm/object_test.cc (right):

https://codereview.chromium.org/23452043/diff/1/runtime/vm/object_test.cc#new...
runtime/vm/object_test.cc:2083: EXPECT_EQ(8, first_idx);  // Token } is first
and only token.
Why has this token index changed?

https://codereview.chromium.org/23452043/diff/1/runtime/vm/scanner.cc
File runtime/vm/scanner.cc (right):

https://codereview.chromium.org/23452043/diff/1/runtime/vm/scanner.cc#newcode895
runtime/vm/scanner.cc:895: // Account for all newlines that should be pushed
after the token, i.e., all
Why so complicated? Is it not enough to remember the line number of the previous
token and add the number of newline tokens to catch up with the line number of
the current token? No need even to remember the number of newlines before the
token, inside multiline strings, etc.

https://codereview.chromium.org/23452043/diff/1/runtime/vm/scanner.h
File runtime/vm/scanner.h (right):

https://codereview.chromium.org/23452043/diff/1/runtime/vm/scanner.h#newcode213
runtime/vm/scanner.h:213: intptr_t newline_seen_;      // Number of newlines
before current token.
num_newlines_before_ ?

[Michael Lippautz (Google), 2013-09-18 23:59:31]

Thanks!

https://codereview.chromium.org/23452043/diff/1/runtime/vm/object_test.cc
File runtime/vm/object_test.cc (right):

https://codereview.chromium.org/23452043/diff/1/runtime/vm/object_test.cc#new...
runtime/vm/object_test.cc:2083: EXPECT_EQ(8, first_idx);  // Token } is first
and only token.
On 2013/09/18 22:27:47, hausner wrote:
> Why has this token index changed?

The source text contains 2 NL, hence 7->9 (previous patchset still had a bug).
While we don't access the NL tokens (because the stream skips them), the index
still reflects the position in the stream.

https://codereview.chromium.org/23452043/diff/1/runtime/vm/scanner.cc
File runtime/vm/scanner.cc (right):

https://codereview.chromium.org/23452043/diff/1/runtime/vm/scanner.cc#newcode895
runtime/vm/scanner.cc:895: // Account for all newlines that should be pushed
after the token, i.e., all
On 2013/09/18 22:27:47, hausner wrote:
> Why so complicated? Is it not enough to remember the line number of the
previous
> token and add the number of newline tokens to catch up with the line number of
> the current token? No need even to remember the number of newlines before the
> token, inside multiline strings, etc.

Done. Makes everything much simpler.

https://codereview.chromium.org/23452043/diff/1/runtime/vm/scanner.h
File runtime/vm/scanner.h (right):

https://codereview.chromium.org/23452043/diff/1/runtime/vm/scanner.h#newcode213
runtime/vm/scanner.h:213: intptr_t newline_seen_;      // Number of newlines
before current token.
On 2013/09/18 22:27:47, hausner wrote:
> num_newlines_before_ ?
> 

Doesn't apply anymore. Reverted it to being a flag.

[Michael Lippautz (Google), 2013-09-19 00:17:33]

Newline occurrences while generating the snapshot.

format: <occurences> NL: <#NLs>

  25275 NL: 1
   3929 NL: 2
    753 NL: 3
    347 NL: 4
    465 NL: 5
    264 NL: 6
    182 NL: 7
    195 NL: 8
    129 NL: 9
    110 NL: 10
     76 NL: 11
     52 NL: 12
     37 NL: 13
     39 NL: 14
     29 NL: 15
     13 NL: 16
     18 NL: 17
     17 NL: 18
      9 NL: 19
      8 NL: 20
     11 NL: 21
     12 NL: 22
     13 NL: 23
      4 NL: 24
      5 NL: 25
      5 NL: 26
      3 NL: 27
      3 NL: 28
     12 NL: 29
      4 NL: 30
      4 NL: 31
      1 NL: 32
      6 NL: 33
      1 NL: 34
      3 NL: 35
      2 NL: 36
      1 NL: 37
      2 NL: 38
      2 NL: 39
      1 NL: 40
      1 NL: 42
      2 NL: 44
      1 NL: 45
      1 NL: 46
      2 NL: 48
      1 NL: 50
      1 NL: 53
      2 NL: 57
      1 NL: 61
      1 NL: 63
      1 NL: 65
      1 NL: 83
      1 NL: 90
      1 NL: 152

[siva, 2013-09-19 20:07:55]

lgtm

[Michael Lippautz (Google), 2013-09-19 21:46:58]

Committed patchset #4 manually as r27677 (presubmit successful).