observatory/vm: add support for computing/passing end tokens on classes

observatory: add class view based on end token position
vm: add method to compute end token positions

BUG=
R=johnmccutchan@google.com

Committed: https://code.google.com/p/dart/source/detail?r=37964

[hausner, 2014-06-30 18:54:19]

We use classes for many things in the VM, e.g. function signatures. Those
classes don't have a textual representation. You should at least set the end
token position to be equal to the token position.

Also, are we sure we want to add this field? It costs 4 bytes for each class
object, even those that are not classes defined in Dart code. Can the
observatory not figure out the end of the class by "reading" the source and just
skip to the end brace?

[Cutch, 2014-06-30 19:45:55]

On 2014/06/30 18:54:19, hausner wrote:
> We use classes for many things in the VM, e.g. function signatures. Those
> classes don't have a textual representation. You should at least set the end
> token position to be equal to the token position.
> 
> Also, are we sure we want to add this field? It costs 4 bytes for each class
> object, even those that are not classes defined in Dart code. Can the
> observatory not figure out the end of the class by "reading" the source and
just
> skip to the end brace?

We can scan the script in Observatory but I'd rather just get this over the
wire. We keep the end token position for each Function object (Function
instances far outnumber Class instances). Should we remove the end token
position from RawFunction to really save space?

[hausner, 2014-06-30 20:58:45]

The end token position for functions is used in more performance-critical paths,
so computing it on demand does not work well.

[Michael Lippautz (Google), 2014-07-01 23:04:34]

PTAL. as discussed offline we re-parse on the client side.

[hausner, 2014-07-01 23:49:07]

I don't think this works :(

https://codereview.chromium.org/361743002/diff/60001/runtime/bin/vmservice/cl...
File runtime/bin/vmservice/client/lib/src/service/object.dart (right):

https://codereview.chromium.org/361743002/diff/60001/runtime/bin/vmservice/cl...
runtime/bin/vmservice/client/lib/src/service/object.dart:1106: class ClassParser
{
When we spoke offline I was under the impression that you had access to the
tokenized script and would not have to parse text. This one-off parser looks
incorrect to me, e.g. does not handle all whitespace, does not handle string
interpolation etc. Writing a full tokenizer in Dart is not what I expected you'd
have to do.

Can you have native functions here? If so, you could write a native function
that computes the end token position of a class using the token stream of the VM
Script object.

[Michael Lippautz (Google), 2014-07-02 00:02:24]

On 2014/07/01 23:49:07, hausner wrote:
> I don't think this works :(
> 
>
https://codereview.chromium.org/361743002/diff/60001/runtime/bin/vmservice/cl...
> File runtime/bin/vmservice/client/lib/src/service/object.dart (right):
> 
>
https://codereview.chromium.org/361743002/diff/60001/runtime/bin/vmservice/cl...
> runtime/bin/vmservice/client/lib/src/service/object.dart:1106: class
ClassParser
> {
> When we spoke offline I was under the impression that you had access to the
> tokenized script and would not have to parse text. This one-off parser looks
> incorrect to me, e.g. does not handle all whitespace, does not handle string
> interpolation etc. Writing a full tokenizer in Dart is not what I expected
you'd
> have to do.
> 
> Can you have native functions here? If so, you could write a native function
> that computes the end token position of a class using the token stream of the
VM
> Script object.

You are right about the whitespace and interpolation issues, but I am kind of
stuck then..

* we should not trigger a computation remotely through the vm service
* we should not transfer token streams
* we should not save the end token in the vm
* no re-parsing raw dart on client

One of those has to be relaxed or it just won't work :)

[Michael Lippautz (Google), 2014-07-02 21:10:43]

On 2014/07/02 00:02:24, Michael Lippautz wrote:
> On 2014/07/01 23:49:07, hausner wrote:
> > I don't think this works :(
> > 
> >
>
https://codereview.chromium.org/361743002/diff/60001/runtime/bin/vmservice/cl...
> > File runtime/bin/vmservice/client/lib/src/service/object.dart (right):
> > 
> >
>
https://codereview.chromium.org/361743002/diff/60001/runtime/bin/vmservice/cl...
> > runtime/bin/vmservice/client/lib/src/service/object.dart:1106: class
> ClassParser
> > {
> > When we spoke offline I was under the impression that you had access to the
> > tokenized script and would not have to parse text. This one-off parser looks
> > incorrect to me, e.g. does not handle all whitespace, does not handle string
> > interpolation etc. Writing a full tokenizer in Dart is not what I expected
> you'd
> > have to do.
> > 
> > Can you have native functions here? If so, you could write a native function
> > that computes the end token position of a class using the token stream of
the
> VM
> > Script object.
> 
> You are right about the whitespace and interpolation issues, but I am kind of
> stuck then..
> 
> * we should not trigger a computation remotely through the vm service
> * we should not transfer token streams
> * we should not save the end token in the vm
> * no re-parsing raw dart on client
> 
> One of those has to be relaxed or it just won't work :)

PTAL: end token pos is now computed in the vm

[Cutch, 2014-07-02 21:12:01]

lgtm

[Michael Lippautz (Google), 2014-07-02 23:23:03]

Committed patchset #6 manually as r37964 (presubmit successful).