Source positions for constructors and lots of async machinery

Source positions for constructors and lots of async machinery


- Rework token position address space.
- Use ClassifyingTokenPositions::kMethodExtractor for the token position of a method extractor.
- Plumb token positions through all of Ast Transformer.
- Plumb token positions for temporary expressions, etc in Flow Graph Builder.
- Add token positions for parts of the await machinery.
- Move ClassifyingTokenPositions into token.h.
- Remove default token position of Scanner::kNoSourcePos for many IR instructions.
- A couple of unit tests.
- Use synthetic token positions for synthetic AstNodes.
- Fix SLEB128 encoding / decoding + test
- s/Scanner::kNoSourcePos/Token::kNoSourcePos.
- Remove >= 0 and < 0 checks against token positions and use helpers instead.

R=rmacnak@google.com

Committed: https://github.com/dart-lang/sdk/commit/77101d63e3c661957b7829dd96d6b4324da0e768

[Cutch, 2016-01-13 23:32:54]

Description was changed from

==========
Source positions for constructors and lots of await machinery

BUG=
==========

to

==========
Source positions for constructors and lots of await machinery

- Use ClassifyingTokenPositions::kMethodExtractor for the token position of a
method extractor.
- Plumb token positions through all of Ast Transformer.
- Plumb token positions for temporary expressions, etc in Flow Graph Builder.
- Add token positions for parts of the await machinery.
- Move ClassifyingTokenPositions into token.h.
- Remove default token position of Scanner::kNoSourcePos for many IR
instructions.
- A couple of unit tests.
==========

[Cutch, 2016-01-13 23:32:54]

johnmccutchan@google.com changed reviewers:
+ rmacnak@google.com

[rmacnak, 2016-01-14 00:28:08]

https://codereview.chromium.org/1589643002/diff/20001/runtime/vm/flow_graph_b...
File runtime/vm/flow_graph_builder_test.cc (right):

https://codereview.chromium.org/1589643002/diff/20001/runtime/vm/flow_graph_b...
runtime/vm/flow_graph_builder_test.cc:99: void InstanceCallAt(intptr_t line,
Nit: the argument order here is different from StaticCallAt

https://codereview.chromium.org/1589643002/diff/20001/runtime/vm/flow_graph_b...
runtime/vm/flow_graph_builder_test.cc:719: "class A {"
missing new line...

https://codereview.chromium.org/1589643002/diff/20001/runtime/vm/flow_graph_b...
runtime/vm/flow_graph_builder_test.cc:732:
spt.FuzzyInstructionMatchAt("AllocateObject(A)", 5, 15);       // 'A'
...Making these lines short by 1.

https://codereview.chromium.org/1589643002/diff/20001/runtime/vm/flow_graph_b...
runtime/vm/flow_graph_builder_test.cc:762: spt.BuildGraphFor("foo");
This doesn't actually test 'await'. All that transformed code is in 'foo.<async
body>'.

[Cutch, 2016-01-14 15:25:10]

https://codereview.chromium.org/1589643002/diff/20001/runtime/vm/flow_graph_b...
File runtime/vm/flow_graph_builder_test.cc (right):

https://codereview.chromium.org/1589643002/diff/20001/runtime/vm/flow_graph_b...
runtime/vm/flow_graph_builder_test.cc:99: void InstanceCallAt(intptr_t line,
On 2016/01/14 00:28:08, rmacnak wrote:
> Nit: the argument order here is different from StaticCallAt

Done.

https://codereview.chromium.org/1589643002/diff/20001/runtime/vm/flow_graph_b...
runtime/vm/flow_graph_builder_test.cc:719: "class A {"
On 2016/01/14 00:28:08, rmacnak wrote:
> missing new line...

Done.

https://codereview.chromium.org/1589643002/diff/20001/runtime/vm/flow_graph_b...
runtime/vm/flow_graph_builder_test.cc:732:
spt.FuzzyInstructionMatchAt("AllocateObject(A)", 5, 15);       // 'A'
On 2016/01/14 00:28:08, rmacnak wrote:
> ...Making these lines short by 1.

Done.

https://codereview.chromium.org/1589643002/diff/20001/runtime/vm/flow_graph_b...
runtime/vm/flow_graph_builder_test.cc:762: spt.BuildGraphFor("foo");
On 2016/01/14 00:28:08, rmacnak wrote:
> This doesn't actually test 'await'. All that transformed code is in
'foo.<async
> body>'.

Correct, I've renamed this to be SourcePosition_Async as the generated code is
all related to the async machinery.

[Cutch, 2016-01-14 15:25:27]

Description was changed from

==========
Source positions for constructors and lots of await machinery

- Use ClassifyingTokenPositions::kMethodExtractor for the token position of a
method extractor.
- Plumb token positions through all of Ast Transformer.
- Plumb token positions for temporary expressions, etc in Flow Graph Builder.
- Add token positions for parts of the await machinery.
- Move ClassifyingTokenPositions into token.h.
- Remove default token position of Scanner::kNoSourcePos for many IR
instructions.
- A couple of unit tests.
==========

to

==========
Source positions for constructors and lots of async machinery

- Use ClassifyingTokenPositions::kMethodExtractor for the token position of a
method extractor.
- Plumb token positions through all of Ast Transformer.
- Plumb token positions for temporary expressions, etc in Flow Graph Builder.
- Add token positions for parts of the await machinery.
- Move ClassifyingTokenPositions into token.h.
- Remove default token position of Scanner::kNoSourcePos for many IR
instructions.
- A couple of unit tests.
==========

[Cutch, 2016-01-14 16:55:30]

I'm now stuck because precompilation and smart async stepping has been broken by
this CL:

$ out/DebugX64/dart --ignore-unrecognized-flags
--package-root=out/DebugX64/packages/
/usr/local/google/home/johnmccutchan/workspace/dart/sdk/tests/standalone/precompilation_dart2js_test.dart
LD_LIBRARY_PATH=null:/usr/local/google/home/johnmccutchan/workspace/dart/sdk/temp_precompilation_testhjpaCR
/usr/local/google/home/johnmccutchan/workspace/dart/sdk/out/DebugX64/dart_precompiled
--run-precompiled-snapshot ignored_script --version

runtime/vm/object.cc:6824: error: expected: obj.IsPatchClass()

Unhandled exception:
Precompiled binary failed.
#0      main
(file:///usr/local/google/home/johnmccutchan/workspace/dart/sdk/tests/standalone/precompilation_dart2js_test.dart:99:7)
#1      _startIsolate.<anonymous closure>
(dart:isolate-patch/isolate_patch.dart:259)
#2      _RawReceivePortImpl._handleMessage
(dart:isolate-patch/isolate_patch.dart:148)

$ python tools/test.py -t120 service/smart_next_test
Test configuration: none_vm_debug_x64

FAILED: none-vm debug_x64 service/smart_next_test
Expected: Pass 
Actual: RuntimeError
CommandOutput[vm]:
stdout:
** Launching out/DebugX64/dart --error_on_bad_type --error_on_bad_override
--ignore-unrecognized-flags --package-root=out/DebugX64/packages/
--enable-vm-service:0
/usr/local/google/home/johnmccutchan/workspace/dart/sdk/runtime/observatory/tests/service/smart_next_test.dart
--testee-mode
Observatory listening on http://127.0.0.1:37125
** Signaled to run test queries on 37125

Running smart_next_test.dart [1/27]
Breakpoint reached
Running smart_next_test.dart [2/27]
Checking we are at line 16
Running smart_next_test.dart [3/27]
next-sync
Running smart_next_test.dart [4/27]
Breakpoint reached
Breakpoint reached
Running smart_next_test.dart [5/27]
Checking we are at line 16
Running smart_next_test.dart [6/27]
next-async
Running smart_next_test.dart [7/27]
Breakpoint reached
Breakpoint reached
Running smart_next_test.dart [8/27]
Checking we are at line 17
** Killing script
Unexpected exception in service tests: Expected to be at line 17 but actually at
line 14
Full stack trace:
 Frame(doAsync.<doAsync_async_body> smart_next_test.dart:token(37))
 Frame(_RootZone.runUnary zone.dart:token(8965))
 Frame(_Future._propagateToListeners.handleValueCallback
future_impl.dart:token(3557))
 Frame(_Future._propagateToListeners future_impl.dart:token(4182))
 Frame(_Future._complete future_impl.dart:token(2767))
 Frame(_SyncCompleter.complete future_impl.dart:token(405))
 Frame(foo.<foo_async_body> smart_next_test.dart:token(27))
 Frame(Future.Future.microtask.<anonymous closure> future.dart:token(301))
 Frame(_microtaskLoop schedule_microtask.dart:token(157))
 Frame(_startMicrotaskLoop schedule_microtask.dart:token(184))
 Frame(_runPendingImmediateCallback isolate_patch.dart:token(531))
 Frame(_RawReceivePortImpl._handleMessage isolate_patch.dart:token(792))
 #0      stoppedAtLine.<anonymous closure>.<<anonymous closure>_async_body>
(file:///usr/local/google/home/johnmccutchan/workspace/dart/sdk/runtime/observatory/tests/service/test_helper.dart:266:7)
#1      _asyncThenWrapperHelper.<anonymous closure>
(dart:async-patch/async_patch.dart:27)
#2      _rootRunUnary (dart:async/zone.dart:914)
#3      _CustomZone.runUnary (dart:async/zone.dart:810)
#4      _Future._propagateToListeners.handleValueCallback
(dart:async/future_impl.dart:551)
#5      _Future._propagateToListeners (dart:async/future_impl.dart:637)
#6      _Future._completeWithValue (dart:async/future_impl.dart:424)
#7      _Future._asyncComplete.<anonymous closure>
(dart:async/future_impl.dart:479)
#8      _rootRun (dart:async/zone.dart:907)
#9      _CustomZone.run (dart:async/zone.dart:802)
#10     _CustomZone.runGuarded (dart:async/zone.dart:708)
#11     _CustomZone.bindCallback.<anonymous closure> (dart:async/zone.dart:733)
#12     _microtaskLoop (dart:async/schedule_microtask.dart:41)
#13     _startMicrotaskLoop (dart:async/schedule_microtask.dart:50)
#14     _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#15     _RawReceivePortImpl._handleMessage
(dart:isolate-patch/isolate_patch.dart:149)

[Cutch, 2016-01-15 21:53:01]

Description was changed from

==========
Source positions for constructors and lots of async machinery

- Use ClassifyingTokenPositions::kMethodExtractor for the token position of a
method extractor.
- Plumb token positions through all of Ast Transformer.
- Plumb token positions for temporary expressions, etc in Flow Graph Builder.
- Add token positions for parts of the await machinery.
- Move ClassifyingTokenPositions into token.h.
- Remove default token position of Scanner::kNoSourcePos for many IR
instructions.
- A couple of unit tests.
==========

to

==========
Source positions for constructors and lots of async machinery


- Rework token position address space.
- Use ClassifyingTokenPositions::kMethodExtractor for the token position of a
method extractor.
- Plumb token positions through all of Ast Transformer.
- Plumb token positions for temporary expressions, etc in Flow Graph Builder.
- Add token positions for parts of the await machinery.
- Move ClassifyingTokenPositions into token.h.
- Remove default token position of Scanner::kNoSourcePos for many IR
instructions.
- A couple of unit tests.
- Use synthetic token positions for synthetic AstNodes.
==========

[Cutch, 2016-01-15 21:53:26]

Description was changed from

==========
Source positions for constructors and lots of async machinery


- Rework token position address space.
- Use ClassifyingTokenPositions::kMethodExtractor for the token position of a
method extractor.
- Plumb token positions through all of Ast Transformer.
- Plumb token positions for temporary expressions, etc in Flow Graph Builder.
- Add token positions for parts of the await machinery.
- Move ClassifyingTokenPositions into token.h.
- Remove default token position of Scanner::kNoSourcePos for many IR
instructions.
- A couple of unit tests.
- Use synthetic token positions for synthetic AstNodes.
==========

to

==========
Source positions for constructors and lots of async machinery


- Rework token position address space.
- Use ClassifyingTokenPositions::kMethodExtractor for the token position of a
method extractor.
- Plumb token positions through all of Ast Transformer.
- Plumb token positions for temporary expressions, etc in Flow Graph Builder.
- Add token positions for parts of the await machinery.
- Move ClassifyingTokenPositions into token.h.
- Remove default token position of Scanner::kNoSourcePos for many IR
instructions.
- A couple of unit tests.
- Use synthetic token positions for synthetic AstNodes.
- Fix SLEB128 encoding / decoding.
==========

[Cutch, 2016-01-15 22:07:16]

Description was changed from

==========
Source positions for constructors and lots of async machinery


- Rework token position address space.
- Use ClassifyingTokenPositions::kMethodExtractor for the token position of a
method extractor.
- Plumb token positions through all of Ast Transformer.
- Plumb token positions for temporary expressions, etc in Flow Graph Builder.
- Add token positions for parts of the await machinery.
- Move ClassifyingTokenPositions into token.h.
- Remove default token position of Scanner::kNoSourcePos for many IR
instructions.
- A couple of unit tests.
- Use synthetic token positions for synthetic AstNodes.
- Fix SLEB128 encoding / decoding.
==========

to

==========
Source positions for constructors and lots of async machinery


- Rework token position address space.
- Use ClassifyingTokenPositions::kMethodExtractor for the token position of a
method extractor.
- Plumb token positions through all of Ast Transformer.
- Plumb token positions for temporary expressions, etc in Flow Graph Builder.
- Add token positions for parts of the await machinery.
- Move ClassifyingTokenPositions into token.h.
- Remove default token position of Scanner::kNoSourcePos for many IR
instructions.
- A couple of unit tests.
- Use synthetic token positions for synthetic AstNodes.
- Fix SLEB128 encoding / decoding + test
==========

[Cutch, 2016-01-19 18:51:31]

Description was changed from

==========
Source positions for constructors and lots of async machinery


- Rework token position address space.
- Use ClassifyingTokenPositions::kMethodExtractor for the token position of a
method extractor.
- Plumb token positions through all of Ast Transformer.
- Plumb token positions for temporary expressions, etc in Flow Graph Builder.
- Add token positions for parts of the await machinery.
- Move ClassifyingTokenPositions into token.h.
- Remove default token position of Scanner::kNoSourcePos for many IR
instructions.
- A couple of unit tests.
- Use synthetic token positions for synthetic AstNodes.
- Fix SLEB128 encoding / decoding + test
==========

to

==========
Source positions for constructors and lots of async machinery


- Rework token position address space.
- Use ClassifyingTokenPositions::kMethodExtractor for the token position of a
method extractor.
- Plumb token positions through all of Ast Transformer.
- Plumb token positions for temporary expressions, etc in Flow Graph Builder.
- Add token positions for parts of the await machinery.
- Move ClassifyingTokenPositions into token.h.
- Remove default token position of Scanner::kNoSourcePos for many IR
instructions.
- A couple of unit tests.
- Use synthetic token positions for synthetic AstNodes.
- Fix SLEB128 encoding / decoding + test
- s/Scanner::kNoSourcePos/Token::kNoSourcePos.
- Remove >= 0 and < 0 checks against token positions and use helpers instead.
==========

[rmacnak, 2016-01-19 22:52:05]

LGTM

[Cutch, 2016-01-19 23:01:14]

Description was changed from

==========
Source positions for constructors and lots of async machinery


- Rework token position address space.
- Use ClassifyingTokenPositions::kMethodExtractor for the token position of a
method extractor.
- Plumb token positions through all of Ast Transformer.
- Plumb token positions for temporary expressions, etc in Flow Graph Builder.
- Add token positions for parts of the await machinery.
- Move ClassifyingTokenPositions into token.h.
- Remove default token position of Scanner::kNoSourcePos for many IR
instructions.
- A couple of unit tests.
- Use synthetic token positions for synthetic AstNodes.
- Fix SLEB128 encoding / decoding + test
- s/Scanner::kNoSourcePos/Token::kNoSourcePos.
- Remove >= 0 and < 0 checks against token positions and use helpers instead.
==========

to

==========
Source positions for constructors and lots of async machinery


- Rework token position address space.
- Use ClassifyingTokenPositions::kMethodExtractor for the token position of a
method extractor.
- Plumb token positions through all of Ast Transformer.
- Plumb token positions for temporary expressions, etc in Flow Graph Builder.
- Add token positions for parts of the await machinery.
- Move ClassifyingTokenPositions into token.h.
- Remove default token position of Scanner::kNoSourcePos for many IR
instructions.
- A couple of unit tests.
- Use synthetic token positions for synthetic AstNodes.
- Fix SLEB128 encoding / decoding + test
- s/Scanner::kNoSourcePos/Token::kNoSourcePos.
- Remove >= 0 and < 0 checks against token positions and use helpers instead.

R=rmacnak@google.com

Committed:
https://github.com/dart-lang/sdk/commit/77101d63e3c661957b7829dd96d6b4324da0e768
==========

[Cutch, 2016-01-19 23:01:15]

Committed patchset #9 (id:160001) manually as
77101d63e3c661957b7829dd96d6b4324da0e768 (presubmit successful).