Extract computing UnlinkedUnit(s) and cache them in ByteStore.

pkg/front_end/lib/src/incremental/unlinked_unit.dart

+// Copyright (c) 2017, the Dart project authors.  Please see the AUTHORS file
+// for details. All rights reserved. Use of this source code is governed by a
+// BSD-style license that can be found in the LICENSE file.
+
+import 'dart:typed_data';
+
+import 'package:front_end/src/base/api_signature.dart';
+import 'package:front_end/src/fasta/parser/listener.dart' show Listener;
+import 'package:front_end/src/fasta/parser/parser.dart' show Parser, optional;
+import 'package:front_end/src/fasta/parser/top_level_parser.dart';
+import 'package:front_end/src/fasta/scanner.dart';
+import 'package:front_end/src/fasta/scanner/token_constants.dart'
+    show STRING_TOKEN;
+import 'package:front_end/src/fasta/source/directive_listener.dart';
+import 'package:front_end/src/incremental/format.dart';
+
+/// Compute the [UnlinkedUnitBuilder] for the [content].
+UnlinkedUnitBuilder computeUnlinkedUnit(List<int> salt, List<int> content) {
+  // Scan the content.
+  ScannerResult scanResult = _scan(content);
+  Token token = scanResult.tokens;
+
+  // Parse directives.
+  var listener = new DirectiveListener();
+  new TopLevelParser(listener).parseUnit(token);

[Paul Berry, 2017/06/13 21:49:46]

It seems wasteful that we call `parseUnit` here and on line 29.  Could we
instead modify _BodySkippingParser so that it records directives?

[scheglov, 2017/06/13 22:03:21]

Well, yes, we waste something. But in general TopLevelParser in very cheap - it
does not parse class members and bodies. I'd like to land it as is and then make
some measurements for how long it takes to parse here and below. If it is
expensive, we will look how to optimize it.

+
+  // Parse to record function bodies.
+  var parser = new _BodySkippingParser();
+  parser.parseUnit(token);
+
+  ApiSignature apiSignature = new ApiSignature();
+  apiSignature.addBytes(salt);
+
+  // Iterate over tokens and skip bodies.
+  Iterator<_BodyRange> bodyIterator = parser.bodyRanges.iterator;
+  bodyIterator.moveNext();
+  for (; token.kind != EOF_TOKEN; token = token.next) {
+    // Move to the body range that ends after the token.
+    while (bodyIterator.current != null &&
+        bodyIterator.current.last < token.charOffset) {
+      bodyIterator.moveNext();
+    }
+    // If the current body range starts before or at the token, skip it.
+    if (bodyIterator.current != null &&
+        bodyIterator.current.first <= token.charOffset) {
+      continue;
+    }
+    // The token is outside of a function body, add it.
+    apiSignature.addString(token.lexeme);
+  }
+
+  return new UnlinkedUnitBuilder(
+      apiSignature: apiSignature.toByteList(),
+      imports: listener.imports.map(_toUnlinkedNamespaceDirective).toList(),
+      exports: listener.exports.map(_toUnlinkedNamespaceDirective).toList(),
+      parts: listener.parts.toList(),
+      hasMixinApplication: parser.hasMixin);
+}
+
+/// Exclude all `native 'xyz';` token sequences.
+void _excludeNativeClauses(Token token) {

[Paul Berry, 2017/06/13 21:49:46]

This seems like an unnecessarily fragile hack.  Can't we rely on the existing
parser support for native clauses?

[scheglov, 2017/06/13 22:03:21]

Unfortunately my understanding is that position of Fasta is that native clauses
are not a part of the language, so they are not supported. It requires handling
[codeExpectedFunctionBody] in listener. I thought that for my usecase this rough
handling of native clauses is good enough.

[Paul Berry, 2017/06/14 12:20:52]

Ok, I will have a discussion with Peter about this.

[Siggi Cherem (dart-lang), 2017/06/14 22:37:38]

FWIW - in our perf scripts we take the other approach:
https://github.com/dart-lang/sdk/blob/master/pkg/front_end/tool/fasta_perf.da...
(add logic in the directive-listener to handle native clauses).

It is handled only for the VM, and doing something more general requires that we
delegate to a target-implementation like we do in the regular compiles in fasta

[scheglov, 2017/06/14 23:20:34]

OK, I see how this is done for parsing directives.
Does not quite work for parsing and skipping bodies.
I've opened an issue against Fasta / Parser.
https://github.com/dart-lang/sdk/issues/29879

+  for (; token.kind != EOF_TOKEN; token = token.next) {
+    if (optional('native', token) &&
+        token.next.kind == STRING_TOKEN &&
+        optional(';', token.next.next)) {
+      token.previous.next = token.next.next;
+    }
+  }
+}
+
+/// Scan the content of the file.
+ScannerResult _scan(List<int> content) {
+  var zeroTerminatedBytes = new Uint8List(content.length + 1);
+  zeroTerminatedBytes.setRange(0, content.length, content);
+  ScannerResult result = scan(zeroTerminatedBytes);
+  _excludeNativeClauses(result.tokens);
+  return result;
+}
+
+/// Convert [NamespaceCombinator] into [UnlinkedCombinatorBuilder].
+UnlinkedCombinatorBuilder _toUnlinkedCombinator(NamespaceCombinator c) =>
+    new UnlinkedCombinatorBuilder(isShow: c.isShow, names: c.names.toList());
+
+/// Convert [NamespaceDirective] into [UnlinkedNamespaceDirectiveBuilder].
+UnlinkedNamespaceDirectiveBuilder _toUnlinkedNamespaceDirective(
+        NamespaceDirective directive) =>
+    new UnlinkedNamespaceDirectiveBuilder(
+        uri: directive.uri,
+        combinators: directive.combinators.map(_toUnlinkedCombinator).toList());
+
+/// The char range of a function body.
+class _BodyRange {
+  /// The char offset of the first token in the range.
+  final int first;
+
+  /// The char offset of the last token in the range.
+  final int last;
+
+  _BodyRange(this.first, this.last);
+
+  @override
+  String toString() => '[$first, $last]';
+}
+
+/// The [Parser] that skips function bodies and remembers their token ranges.
+class _BodySkippingParser extends Parser {
+  bool hasMixin = false;
+  final List<_BodyRange> bodyRanges = [];
+
+  _BodySkippingParser() : super(new Listener());
+
+  @override
+  Token parseFunctionBody(Token token, bool isExpression, bool allowAbstract) {

[ahe, 2017/08/07 14:05:13]

This feature is already provided by ClassMemberParser. If it doesn't do exactly
what you need, I think we should extend it.

Generally, it would be great if we could avoid subclassing Parser outside
pkg/front_end/lib/src/fasta/parser/. This makes it easier to make parser
changes.

+    if (identical('{', token.lexeme)) {
+      Token close = skipBlock(token);
+      bodyRanges.add(new _BodyRange(token.charOffset, close.charOffset));
+      return close;
+    }
+    return super.parseFunctionBody(token, isExpression, allowAbstract);
+  }
+
+  Token parseMixinApplication(Token token) {

[ahe, 2017/08/07 14:05:13]

You should be able to collect this information via a listener.

+    hasMixin = true;
+    return super.parseMixinApplication(token);
+  }
+}