Adding package:code_transformers for unifying common transformers code

pkg/code_transformers/lib/src/resolver_impl.dart

Index: pkg/code_transformers/lib/src/resolver_impl.dart
diff --git a/pkg/code_transformers/lib/src/resolver_impl.dart b/pkg/code_transformers/lib/src/resolver_impl.dart
new file mode 100644
index 0000000000000000000000000000000000000000..5918c95933ee4e2aa5c48472aa5b2fb94bc0336a
--- /dev/null
+++ b/pkg/code_transformers/lib/src/resolver_impl.dart
@@ -0,0 +1,492 @@
+// Copyright (c) 2014, the Dart project authors.  Please see the AUTHORS file
+// for details. All rights reserved. Use of this source code is governed by a
+// BSD-style license that can be found in the LICENSE file.
+
+library code_transformer.src.resolver_impl;
+
+import 'dart:async';
+import 'package:analyzer/src/generated/ast.dart';
+import 'package:analyzer/src/generated/element.dart';
+import 'package:analyzer/src/generated/engine.dart';
+import 'package:analyzer/src/generated/error.dart';
+import 'package:analyzer/src/generated/java_io.dart';
+import 'package:analyzer/src/generated/parser.dart' show Parser;
+import 'package:analyzer/src/generated/scanner.dart';
+import 'package:analyzer/src/generated/sdk.dart' show DartSdk;
+import 'package:analyzer/src/generated/sdk_io.dart' show DirectoryBasedDartSdk;
+import 'package:analyzer/src/generated/source.dart';
+import 'package:barback/barback.dart';
+import 'package:path/path.dart' as path;
+import 'package:source_maps/refactor.dart';
+import 'package:source_maps/span.dart' show SourceFile, Span;
+
+import 'resolver.dart';
+
+/// Resolves and updates an AST based on Barback-based assets.
+///
+/// This also provides a handful of useful APIs for traversing and working
+/// with the resolved AST.
+class ResolverImpl implements Resolver {
+  /// Cache of all asset sources currently referenced.
+  final Map<AssetId, _AssetBasedSource> sources =
+      <AssetId, _AssetBasedSource>{};
+  /// The Dart entry point file where parsing begins.

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

nit: add empty line above comment

[blois, 2014/02/19 21:42:58]

Done.

+  final AssetId entryPoint;
+
+  final AnalysisContext _context =
+      AnalysisEngine.instance.createAnalysisContext();
+
+  /// Transform for which this is currently updating, or null when not updating.
+  Transform _currentTransform;
+  /// The currently resolved library, or null if unresolved.

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

same here and below

[blois, 2014/02/19 21:42:58]

Done.

+  LibraryElement _entryLibrary;
+  /// Handler for all Dart SDK (dart:) sources.
+  DirectoryBasedDartSdk _dartSdk;
+
+  /// Creates a resolver that will resolve the Dart code starting at
+  /// [entryPoint].
+  ///
+  /// [sdkDir] is the root directory of the Dart SDK, for resolving dart:
+  /// imports.
+  ResolverImpl(this.entryPoint, String sdkDir, {AnalysisOptions options}) {
+    if (options == null) {
+      options = new AnalysisOptionsImpl()
+        ..cacheSize = 256 // # of sources to cache ASTs for.

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

can we make it bigger :)?

[blois, 2014/02/19 21:42:58]

I actually just cloned this from Angular's extractor- can make it bigger but
it's just a perf optimization (AST will be re-generated if it is needed after
purged). Default is 64.

+        ..preserveComments = false
+        ..analyzeFunctionBodies = true;

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

should this be an option too? Seems like a common use for the resolver can be
without this

[blois, 2014/02/19 21:42:58]

Added, but the options themselves can be provided as well.

[Siggi Cherem (dart-lang), 2014/02/19 22:36:06]

Ah - good point, I overlooked that if options is provided this codepath is not
reached, feel free to keep or remove it if you like. 

+    }
+    _context.analysisOptions = options;
+
+    _dartSdk = new _DirectoryBasedDartSdkProxy(new JavaFile(sdkDir));
+    _dartSdk.context.analysisOptions = options;
+
+    _context.sourceFactory = new SourceFactory.con2([
+        new DartUriResolverProxy(_dartSdk),
+        new _AssetUriResolver(this)]);
+  }
+
+  LibraryElement get entryLibrary => _entryLibrary;
+
+
+  /// Update the status of all the sources referenced by the entryPoint and
+  /// update the resolved library.
+  ///
+  /// This will be invoked automatically by [ResolverTransformer]. Only one
+  /// transformer may update this at a time.
+  Future updateSources(Transform transform) {
+    if (_currentTransform != null) {
+      throw new StateError('Cannot be accessed by concurrent transforms');
+    }
+    _currentTransform = transform;
+    // Clear this out and update once all asset changes have been processed.
+    _entryLibrary = null;
+
+    // Basic approach is to start at the first file, update it's contents
+    // and see if it changed, then walk all files accessed by it.
+    var visited = new Set<AssetId>();
+    var toVisit = new Set<AssetId>();
+    var changedSources = <Source>[];
+    var addedSources = <Source>[];
+    var removedSources = <Source>[];
+    toVisit.add(entryPoint);
+
+    Future visitNext() {
+      if (toVisit.length == 0) return null;
+
+      var assetId = toVisit.first;

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

might be cheaper to use a queue instead of a Set for toVisit

[blois, 2014/02/19 21:42:58]

Since order is not critical better to optimize for the contains check?

[Siggi Cherem (dart-lang), 2014/02/19 22:36:06]

oh - I think you can remove the check for 'toVisit.contains' and instead return
quickly here if it happened to have been visited:

  var assetId = toVisit.removeFirst();
  if (visited.contains(assetId)) return null;
  visited.add(assetId);

[blois, 2014/02/20 01:03:12]

Ended up switching over to FutureGroup which allows removal of the toVisit set
completely.

+      toVisit.remove(assetId);
+      visited.add(assetId);
+
+      return transform.readInputAsString(assetId).then((contents) {
+        var source = sources[assetId];
+        if (source == null) {
+          source = new _AssetBasedSource(assetId, this);
+          sources[assetId] = source;
+          addedSources.add(source);
+        }
+        var changed = source.updateContents(contents);
+        if (changed) {
+          changedSources.add(source);
+        }
+
+        for (var id in source.dependentAssets) {
+          if (!visited.contains(id) && !toVisit.contains(id)) {
+            toVisit.add(id);
+          }
+        }
+
+        return visitNext();

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

FYI - this is fine, but it might be worth using FutureGroup in the future for
this (I have similar code in polymer I'd like to update).

FutureGroup works great for this kind of work where you dynamically and
asynchronously enqueue more work to do.

[blois, 2014/02/19 21:42:58]

Would prefer to leave as-is for now, though switching to FutureGroup in the
future may allow IO to be parallelized in the future.

+      }, onError: (e) {
+        removedSources.add(sources[assetId]);
+        return visitNext();
+      });
+    }
+
+    // Once we have all asset sources updated with the new contents then
+    // resolve everything.
+    return new Future(visitNext).then((_) {
+      ChangeSet changeSet = new ChangeSet();

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

-> var

[blois, 2014/02/19 21:42:58]

Done.

+
+      // Add all sources which were not reached to the removed set.
+      removedSources.addAll(new Set.from(sources.keys)
+          .difference(visited)
+          .map((id) => sources[id]));
+
+      // Clear out removed sources from our cache.
+      removedSources.map((source) => source.assetId).forEach(sources.remove);
+
+      addedSources.forEach(changeSet.added);
+      changedSources.forEach(changeSet.changed);
+      removedSources.forEach(changeSet.removed);
+
+      // Update the analyzer context with the latest sources
+      _context.applyChanges(changeSet);
+      // Resolve the AST
+      _entryLibrary = _context.computeLibraryElement(sources[entryPoint]);
+
+      _currentTransform = null;
+    });
+  }
+
+  Iterable<LibraryElement> get libraries => entryLibrary.visibleLibraries;
+
+  LibraryElement getLibraryByName(String libraryName) =>
+      libraries.firstWhere((l) => l.name == libraryName, orElse: () => null);
+
+  LibraryElement getLibraryByUri(Uri uri) =>
+      libraries.firstWhere((l) => getImportUri(l) == uri, orElse: () => null);
+
+  ClassElement getType(String typeName) {
+    var dotIndex = typeName.lastIndexOf('.');
+    var libraryName = dotIndex == -1 ? '' : typeName.substring(0, dotIndex);
+
+    var className = dotIndex == -1 ?
+        typeName : typeName.substring(dotIndex + 1);
+
+    for (var lib in libraries.where((l) => l.name == libraryName)) {
+      var type = lib.getType(className);
+      if (type != null) return type;
+    }
+    return null;
+  }
+
+  Element getLibraryVariable(String variableName) {
+    var dotIndex = variableName.lastIndexOf('.');
+    var libraryName = dotIndex == -1 ? '' : variableName.substring(0, dotIndex);
+
+    var name = dotIndex == -1 ?
+        variableName : variableName.substring(dotIndex + 1);
+
+    return libraries.where((lib) => lib.name == libraryName)
+        .expand((lib) => lib.units)
+        .expand((unit) => unit.topLevelVariables)
+        .firstWhere((variable) => variable.name == name,
+            orElse: () => null);
+  }
+
+  Element getLibraryFunction(String fnName) {
+    var dotIndex = fnName.lastIndexOf('.');
+    var libraryName = dotIndex == -1 ? '' : fnName.substring(0, dotIndex);
+
+    var name = dotIndex == -1 ?
+        fnName : fnName.substring(dotIndex + 1);
+
+    return libraries.where((lib) => lib.name == libraryName)
+        .expand((lib) => lib.units)
+        .expand((unit) => unit.functions)
+        .firstWhere((fn) => fn.name == name,
+            orElse: () => null);
+  }
+
+  Uri getImportUri(LibraryElement lib, {AssetId from}) {
+    var source = lib.source;
+    if (source is _AssetBasedSource) {
+      var id = source.assetId;
+
+      if (!id.path.startsWith('lib/')) {
+        // Cannot do absolute imports of non lib-based assets.
+        if (from == null) return null;
+
+        if (id.package != from.package) return null;
+        return new Uri(
+            path:  path.relative(id.path, from: path.dirname(from.path)));

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

remove extra space

[blois, 2014/02/19 21:42:58]

Done.

+      }
+
+      return Uri.parse('package:${id.package}/${id.path.substring(4)}');
+    } else if (source is _DartSourceProxy) {
+      return source.uri;
+    }
+    // Should not be able to encounter any other source types.
+    throw new StateError('Unable to resolve URI for ${source.runtimeType}');
+  }
+
+  AssetId getSourceAssetId(Element element) {
+    var source = element.source;
+    if (source is _AssetBasedSource) return source.assetId;
+    return null;
+  }
+
+  Span getSourceSpan(Element element) {
+    var assetId = getSourceAssetId(element);
+    if (assetId == null) return null;
+
+    var sourceFile = new SourceFile.text(assetId.path,

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

let's change this first argument to match what we do in
polymer/src/build/common.dart#spanUrlFor(id, ...)

Basically, it's useful to have the package name in the url if you are processing
an element that it's not in the same package as the primary input. More details
on the #spanUrlFor comment here:
https://code.google.com/p/dart/source/browse/branches/bleeding_edge/dart/pkg/...

[blois, 2014/02/19 21:42:58]

Done.

+        sources[assetId].contents);
+    return sourceFile.span(element.node.offset, element.node.end);
+  }
+
+  TextEditTransaction createTextEditTransaction(Element element) {
+    if (element.source is! _AssetBasedSource) return null;
+
+    _AssetBasedSource source = element.source;
+    // Cannot modify assets in other packages.
+    if (source.assetId.package != entryPoint.package) return null;
+
+    var sourceFile = new SourceFile.text(source.assetId.path, source.contents);

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

same here

[blois, 2014/02/19 21:42:58]

Done.

+    return new TextEditTransaction(source.contents, sourceFile);
+  }
+}
+
+/// Implementation of Analyzer's Source for Barback based assets.
+class _AssetBasedSource extends Source {
+  /// Asset ID where this source can be found.
+  final AssetId assetId;
+  /// The resolver this is being used in.

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

similar style nit here: add empty line above each comment block

[blois, 2014/02/19 21:42:58]

Done.

+  final ResolverImpl _resolver;
+  /// Cache of dependent asset IDs, to avoid re-parsing the AST.
+  Iterable<AssetId> _dependentAssets;
+  /// The current revision of the file, incremented only when file changes.
+  int _revision = 0;
+  /// The file contents.
+  String _contents;
+
+  _AssetBasedSource(this.assetId, this._resolver);
+
+  /// Update the contents of this file with [contents].
+  ///
+  /// Returns true if the contents of this asset have changed.
+  bool updateContents(String contents) {
+    if (contents != _contents) {
+      _contents = contents;
+      ++_revision;

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

_revision++?

It's style nit, I think we use i++ more than ++i, it probably will read better
because of the _ too.

[blois, 2014/02/19 21:42:58]

Old habits die hard. Prefix operator is sometimes faster (when dealing with C++
iterators).

[Siggi Cherem (dart-lang), 2014/02/19 22:36:06]

Yeah, I recall that was the case in C++, good thing that's no longer true in
Dart :)

+      // Invalidate the imports so we only parse the AST when needed.
+      _dependentAssets = null;
+      return true;
+    }
+    return false;
+  }
+
+  /// Contents of the file.
+  String get contents => _contents;
+
+  /// Logger for the current transform.
+  ///
+  /// Only valid while the resolver is updating assets.
+  TransformLogger get _logger => _resolver._currentTransform.logger;
+
+  /// Gets all imports/parts/exports which resolve to assets (non-Dart files).
+  Iterable<AssetId> get dependentAssets {

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

since you already have a cache of _AssetBasedSource up in the ResolverImpl, it
might be worth storing the dependentAssets explicitly. Then invalidate the cache
when you call updateContents and find that the contents actually changed.

+    // Use the cached imports if we have them.
+    if (_dependentAssets != null) return _dependentAssets;
+
+    var errorListener = new _ErrorCollector();
+    var reader = new CharSequenceReader(contents);
+    var scanner = new Scanner(null, reader, errorListener);
+    var token = scanner.tokenize();
+    var parser = new Parser(null, errorListener);
+
+    var compilationUnit = parser.parseCompilationUnit(token);
+
+    // Walk the AST looking for import/export/part directives.
+    _dependentAssets = compilationUnit.directives
+        .where((d) => (d is ImportDirective || d is PartDirective ||
+            d is ExportDirective))
+        .map((d) => _resolve(assetId, d.uri.stringValue,
+            _logger, _getSpan(d)))
+        .where((id) => id != null);
+
+    return _dependentAssets;
+  }
+
+  bool exists() => true;
+
+  bool operator ==(Object other) =>
+      other is _AssetBasedSource && assetId == other.assetId;
+
+  int get hashCode => assetId.hashCode;
+
+  void getContents(Source_ContentReceiver receiver) {
+    receiver.accept(contents, modificationStamp);
+  }
+
+  String get encoding =>
+      "${uriKind.encoding}${assetId.package}/${assetId.path}";
+
+  String get fullName => assetId.toString();
+
+  int get modificationStamp => _revision;
+
+  String get shortName => path.basename(assetId.path);
+
+  UriKind get uriKind {
+    if (assetId.path.startsWith('lib/')) return UriKind.PACKAGE_URI;
+    return UriKind.FILE_URI;
+  }
+
+  bool get isInSystemLibrary => false;
+
+  Source resolveRelative(Uri relativeUri) {
+    var id = _resolve(assetId, relativeUri.toString(), _logger, null);
+    if (id == null) return null;
+
+    // The entire AST should have been parsed and loaded at this point.
+    var source = _resolver.sources[id];
+    if (source == null) {
+      _logger.error('Could not load asset $id');
+    }
+    return source;
+  }
+
+  /// For logging errors.
+  Span _getSpan(ASTNode node) => _sourceFile.span(node.offset, node.end);
+  /// For logging errors.
+  SourceFile get _sourceFile => new SourceFile.text(assetId.path, contents);

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

same here: use something like spanUriFor(...)

[blois, 2014/02/19 21:42:58]

Done.

+}
+
+/// Implementation of Analyzer's UriResolver for Barback based assets.
+class _AssetUriResolver implements UriResolver {
+  final ResolverImpl _resolver;
+  _AssetUriResolver(this._resolver);
+
+  Source resolveAbsolute(ContentCache contentCache, Uri uri) {
+    var assetId = _resolve(null, uri.toString(), logger, null);
+    var source = _resolver.sources[assetId];
+    /// All resolved assets should be available by this point.
+    if (source == null) {
+      logger.error('Unable to find asset for "$uri"');
+    }
+    return source;
+  }
+
+  Source fromEncoding(ContentCache contentCache, UriKind kind, Uri uri) =>
+      throw new UnsupportedError('fromEncoding is not supported');
+
+  Uri restoreAbsolute(Source source) =>
+      throw new UnsupportedError('restoreAbsolute is not supported');
+
+  TransformLogger get logger => _resolver._currentTransform.logger;
+}
+
+
+/// Dart SDK which wraps all Dart sources to ensure they are tracked with Uris.
+///
+/// Just a simple wrapper to make it easy to make sure that all sources we
+/// encounter are either _AssetBasedSource or _DartSourceProxy.
+class _DirectoryBasedDartSdkProxy extends DirectoryBasedDartSdk {
+  _DirectoryBasedDartSdkProxy(JavaFile sdkDirectory) : super(sdkDirectory);
+
+  Source mapDartUri(String dartUri) =>
+      _DartSourceProxy.wrap(super.mapDartUri(dartUri), Uri.parse(dartUri));
+}
+
+
+/// Dart SDK resolver which wraps all Dart sources to ensure they are tracked
+/// with URIs.
+class DartUriResolverProxy implements DartUriResolver {
+  final DartUriResolver _proxy;
+  DartUriResolverProxy(DirectoryBasedDartSdk sdk) :
+      _proxy = new DartUriResolver(sdk);
+
+  Source resolveAbsolute(ContentCache contentCache, Uri uri) =>
+    _DartSourceProxy.wrap(_proxy.resolveAbsolute(contentCache, uri), uri);
+
+  DartSdk get dartSdk => _proxy.dartSdk;
+
+  Source fromEncoding(ContentCache contentCache, UriKind kind, Uri uri) =>
+      throw new UnsupportedError('fromEncoding is not supported');
+
+  Uri restoreAbsolute(Source source) =>
+      throw new UnsupportedError('restoreAbsolute is not supported');
+}
+
+/// Source file for dart: sources which track the sources with dart: URIs.
+///
+/// This is primarily to support [Resolver.getImportUri] for Dart SDK (dart:)
+/// based libraries.
+class _DartSourceProxy implements Source {
+  /// Absolute URI which this source can be imported from
+  final Uri uri;
+  /// Underlying source object.

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

+line

[blois, 2014/02/19 21:42:58]

Done.

+  final Source _proxy;
+
+  _DartSourceProxy(this._proxy, this.uri);
+
+  /// Ensures that [source] is a _DartSourceProxy.
+  static _DartSourceProxy wrap(Source source, Uri uri) {
+    if (source == null || source is _DartSourceProxy) return source;
+    return new _DartSourceProxy(source, uri);
+  }
+
+  Source resolveRelative(Uri relativeUri) {
+    // Assume that the type can be accessed via this URI, since these
+    // should only be parts for dart core files.
+    return wrap(_proxy.resolveRelative(relativeUri), uri);
+  }
+
+  bool exists() => _proxy.exists();
+
+  bool operator ==(Object other) =>
+    (other is _DartSourceProxy && _proxy == other._proxy);
+
+  int get hashCode => _proxy.hashCode;
+
+  void getContents(Source_ContentReceiver receiver) {
+    _proxy.getContents(receiver);
+  }
+
+  String get encoding => _proxy.encoding;
+
+  String get fullName => _proxy.fullName;
+
+  int get modificationStamp => _proxy.modificationStamp;
+
+  String get shortName => _proxy.shortName;
+
+  UriKind get uriKind => _proxy.uriKind;
+
+  bool get isInSystemLibrary => _proxy.isInSystemLibrary;
+}
+
+
+class _ErrorCollector extends AnalysisErrorListener {
+  final errors = <AnalysisError>[];
+  onError(error) => errors.add(error);
+}
+
+/// Get an asset ID for a URL relative to another source asset.
+AssetId _resolve(AssetId source, String url, TransformLogger logger,

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

this seems similar to what we have in common, is the idea to unify them? I'm ok
not unifying them now, but maybe add a TODO?

[blois, 2014/02/19 21:42:58]

It's actually slightly different- this goes from a dart: URI to an asset ID, the
one in common goes from an asset URI to an asset ID.

+    Span span) {
+  if (url == null || url == '') return null;
+  var urlBuilder = path.url;
+  var uri = Uri.parse(url);
+
+  if (uri.scheme == 'package') {
+    var segments = new List.from(uri.pathSegments);
+    var package = segments[0];
+    segments[0] = 'lib';
+    return new AssetId(package, segments.join(urlBuilder.separator));
+  }
+  // Dart SDK libraries do not have assets.
+  if (uri.scheme == 'dart') {

[Siggi Cherem (dart-lang), 2014/02/14 02:13:37]

style nit: fast-exit style

if (uri.scheme == 'dart') return null;

[blois, 2014/02/19 21:42:58]

Done.

+    return null;
+  }
+
+  if (uri.host != '' || uri.scheme != '' || urlBuilder.isAbsolute(url)) {
+    logger.error('absolute paths not allowed: "$url"', span: span);
+    return null;
+  }
+
+  var targetPath = urlBuilder.normalize(
+      urlBuilder.join(urlBuilder.dirname(source.path), url));
+  return new AssetId(source.package, targetPath);
+}