Add support for SDK outline in KernelDriver.

pkg/front_end/lib/src/incremental/file_state.dart

 // Copyright (c) 2017, the Dart project authors.  Please see the AUTHORS file
 // for details. All rights reserved. Use of this source code is governed by a
 // BSD-style license that can be found in the LICENSE file.
 
 import 'dart:async';
 import 'dart:convert';
 import 'dart:typed_data';
 
 import 'package:convert/convert.dart';
 import 'package:crypto/crypto.dart';
@@ skipping 263 matching lines @@
   _FileSystemView _fileSystemView;
 
   /// Mapping from import URIs to corresponding [FileState]s. For example, this
   /// may contain an entry for `dart:core`.
   final Map<Uri, FileState> _uriToFile = {};
 
   /// Mapping from file URIs to corresponding [FileState]s. This map should only
   /// contain `file:*` URIs as keys.
   final Map<Uri, FileState> _fileUriToFile = {};
 
+  /// If `true`, then files with the `dart` scheme should be skipped.
+  /// We do this when we use SDK outline instead of compiling SDK sources.
+  bool skipSdkLibraries = false;
+
   FileSystemState(this._byteStore, this.fileSystem, this.uriTranslator,
       this._salt, this._newFileFn);
 
   /// Return the [FileSystem] that is backed by this [FileSystemState].  The
   /// files in this [FileSystem] always have the same content as the
   /// corresponding [FileState]s, thus avoiding race conditions when a file
   /// is updated on the actual file system.
   FileSystem get fileSystemView {
     return _fileSystemView ??= new _FileSystemView(this);
   }
@@ skipping 32 matching lines @@
     urisToRemove.forEach(_uriToFile.remove);
     fileUrisToRemove.forEach(_fileUriToFile.remove);
     return filesToRemove;
   }
 
   /// Return the [FileState] for the given [absoluteUri], or `null` if the
   /// [absoluteUri] cannot be resolved into a file URI.
   ///
   /// The returned file has the last known state since it was last refreshed.
   Future<FileState> getFile(Uri absoluteUri) async {
+    // We don't need to process SDK libraries if we have SK outline.

[Siggi Cherem (dart-lang), 2017/08/05 03:22:47]

nit: "SK" => "an SDK"

[scheglov, 2017/08/06 21:04:06]

Done.

+    if (skipSdkLibraries && absoluteUri.isScheme('dart')) {

[Siggi Cherem (dart-lang), 2017/08/05 03:22:47]

rather than skipping any `dart:*` library, could we instead store which
libraries to skip?

I am thinking about this because dart:vmservice is not part of the vm's
platform.dill, but it is compiled separately. I don't how whether that comes up
in practice though, so it might be OK to have that as a TODO.

Alex - can you clarify if that's a scenario we need to support?

[scheglov, 2017/08/06 21:04:06]

I'm adding TODO for now.

[aam, 2017/08/07 14:41:34]

Good point! Yes, it seems it won't work with how vmservice is compiled here: 
https://github.com/dart-lang/sdk/blob/master/tools/patch_sdk.dart#L145.

[Siggi Cherem (dart-lang), 2017/08/07 15:29:52]

Konstantin: I was thinking more about this over the weekend, and I think pushing
this further will help make the public API more intuitive and possibly make this
code more agnostic of the SDK.

In particular, I'm thinking that we should allow users to provide a
`CompilerOptions` with `sdkSummary` `inputSummaries` or `linkedDependencies`,
and we will use all those .dill files here. Then, assuming we use the
program.libraries directly, we will be able to ignore any of the import-Uris
here in getFile.

What I especially like about it is that we no longer have a special rule about
the "SDK", we simply have a special rule to attach an existing outline/dill file
and treat it as if all the sources mentioned there are never modified. The logic
to combine all those .dill files together will be exactly the same we use in
`generateKernel` today.

WDYT?

Alex - just to clarify, I assume in patch_sdk.dart we will continue to use the
regular batch compiler and not the incremental compiler. Do we expect to build
vmservice also in kernel-service, or only ahead of time when building the sdk
like we do now?

[aam, 2017/08/07 15:42:11]

Ah, true, I don't see patch_sdk.dart using incremental compiler in the nearest
future.

[scheglov, 2017/08/07 15:51:25]

The first problem is that we need to make sure that signatures of results
continue depending on all inputs, including input kernels. It is convenient to
do when you have bytes of these kernels, so you can mix them into the salt, and
load libraries from them. OTOH, Library or Program objects from CompilerOptions
don't allow this.

Them main question is however whether Kernel Driver is going to be used in this
way. My understanding is that it will be used as Analysis Driver is used - in
interactive environment, where the driver manages its dependencies and cached
results. For batch compilation, with external dependencies management, there is
the batch compiler.

[scheglov, 2017/08/07 15:51:25]

So, is there still a reason to change the way `dart` libraries are skipped?

[aam, 2017/08/07 16:04:38]

Not sure, about short term, but in long term, to me it sounds surprising  that
the only way to compile dart:vmservice will be with batch compiler, not with
IKG. I would expect that both batch compiler and 1st run of IKG behave similarly
on the same input.

[Siggi Cherem (dart-lang), 2017/08/07 16:27:42]

So basically we need ProcessedOptions to expose the bytes or the URI (and not
just the parsed program) for all these input .dill files, correct?

Note that we need to do this already (I believe): after this CL we need to
expose a way to specify the sdk outline in the IKG API. I thought we will use
CompilerOptions.sdkSummary to specify the URI location, just like we do in all
other APIs, and make ProcessedOptions expose to you internally anything else
that you need.
Good question - I don't know yet. Alex's new use case seems to be somewhere in
between: a batch compiler with the ability to support hot-reload.

Even without this answer, I think there is also value in having all of our
compiler options working consistently across our different APIs. Maybe this will
become more clear when we add the IKG API to expose this change in kernel
driver. Given how the logic to support it is practically the same, I am inclined
to support it as well.
 If we don't allow providing inputSummary and linkedDependency as flags, then
let's report an error for those in the IKG saying that they are not supported at
this time. 

Independently of what we do in IKG - I thought you preferred to make
kernel-driver more agnostic of the SDK. If so, just saying there is a list of
bytes that we load in order, and any library from those is not compiled from
sources, seems to me like a clean way to achieve that.

+      return null;
+    }
+
     // Resolve the absolute URI into the absolute file URI.
     Uri fileUri;
     if (absoluteUri.isScheme('file')) {
       fileUri = absoluteUri;
     } else {
       fileUri = uriTranslator.translate(absoluteUri);
       if (fileUri == null) return null;
     }
 
     FileState file = _uriToFile[absoluteUri];
@@ skipping 165 matching lines @@
       cycle.libraries.add(node.file);
       fileToCycleMap[node.file] = cycle;
     }
     topologicallySortedCycles.add(cycle);
   }
 
   _LibraryNode getNode(FileState file) {
     return nodesOfFiles.putIfAbsent(file, () => new _LibraryNode(this, file));
   }
 }