Stop copying tokens in incremental scanner

Stop copying tokens in incremental scanner. Once the incremental parser is no longer using the token map I can remove the tokenMap support.

R=paulberry@google.com

Committed: https://code.google.com/p/dart/source/detail?r=41913

[Brian Wilkerson, 2014-11-20 22:03:48]

brianwilkerson@google.com changed reviewers:
+ paulberry@google.com

[Paul Berry, 2014-11-20 22:41:59]

lgtm once comments are addressed.

Once this commit lands I'll take responsibility for making sure
IncrementalParser doesn't need tokenMap anymore, and removing it.

https://codereview.chromium.org/745873002/diff/20001/pkg/analyzer/lib/src/gen...
File pkg/analyzer/lib/src/generated/incremental_scanner.dart (right):

https://codereview.chromium.org/745873002/diff/20001/pkg/analyzer/lib/src/gen...
pkg/analyzer/lib/src/generated/incremental_scanner.dart:197: oldLeftToken =
oldFirst;
Instead of these four statements, I think we need to do:

oldLeftToken.setNext(oldFirst);
oldLeftToken = oldFirst;
oldFirst = oldFirst.next;
_leftToken = oldLeftToken
newFirst = newFirst.next;

This ensures that new tokens only appear in the range newFirst...newLast, which
is an invariant that I believe the incremental parser will need to rely on once
it stops using the tokenMap.

If we make this change, then I believe oldLeftToken and _leftToken will become
redundant.

https://codereview.chromium.org/745873002/diff/20001/pkg/analyzer/lib/src/gen...
pkg/analyzer/lib/src/generated/incremental_scanner.dart:208: oldRightToken =
oldLast;
Similar comment here, mutatis mutandis.

https://codereview.chromium.org/745873002/diff/20001/pkg/analyzer/lib/src/gen...
pkg/analyzer/lib/src/generated/incremental_scanner.dart:276: token.offset +=
delta;
Does the EOF token ever have precedingComments?  (i.e. because there was a
comment after the last token in the file)

[Brian Wilkerson, 2014-11-20 22:55:03]

https://codereview.chromium.org/745873002/diff/20001/pkg/analyzer/lib/src/gen...
File pkg/analyzer/lib/src/generated/incremental_scanner.dart (right):

https://codereview.chromium.org/745873002/diff/20001/pkg/analyzer/lib/src/gen...
pkg/analyzer/lib/src/generated/incremental_scanner.dart:197: oldLeftToken =
oldFirst;
> Instead of these four statements, I think we need to do...

I think we want even bigger changes. My plan is to re-order the code slightly so
that we narrow the range of new tokens before splicing them in to the old token
stream. I also want, as part of that, to be more intelligent about begin/end
token pairs. But I'd like to do that in a later CL rather than pile more work on
top of this one.

https://codereview.chromium.org/745873002/diff/20001/pkg/analyzer/lib/src/gen...
pkg/analyzer/lib/src/generated/incremental_scanner.dart:276: token.offset +=
delta;
> Does the EOF token ever have precedingComments?  (i.e. because there was a
> comment after the last token in the file)

Yes, when there are comments at the end of the file. Good catch! I'd like to fix
this in a later CL as well.

[Brian Wilkerson, 2014-11-21 17:24:10]

Committed patchset #2 (id:20001) manually as 41913 (presubmit successful).