Make pumpEventQueue() private and explain why it is used in 'results'.

pkg/analyzer/lib/src/dart/analysis/driver.dart

 // Copyright (c) 2016, the Dart project authors.  Please see the AUTHORS file
 // for details. All rights reserved. Use of this source code is governed by a
 // BSD-style license that can be found in the LICENSE file.
 
 import 'dart:async';
 import 'dart:collection';
 import 'dart:convert';
 
 import 'package:analyzer/dart/ast/ast.dart';
 import 'package:analyzer/dart/ast/token.dart';
@@ skipping 213 matching lines @@
    * More than one result might be produced for the same file, even if the
    * client does not change the state of the files.
    *
    * Results might be produced even for files that have never been added
    * using [addFile], for example when [getResult] was called for a file.
    */
   Stream<AnalysisResult> get results async* {
     try {
       PerformanceLogSection analysisSection = null;
       while (true) {
-        await pumpEventQueue(100);
+        // Pump the event queue to allow IO and other asynchronous data
+        // processing while analysis is active. For example Analysis Server
+        // need to be able to process update content or set priority requests

[Brian Wilkerson, 2016/10/31 20:41:04]

"need" --> "needs"
"update content" --> "updateContent"
"set priority" --> "setPriorityFiles"

[scheglov, 2016/10/31 20:48:14]

Done.

+        // while background analysis is in progress.
+        //
+        // The number of pumpings is arbitrary, might be changed if we see that
+        // analysis or other data processing tasks are starving. Ideally we
+        // would need to be able to set priority of (continuous) asynchronous
+        // tasks.
+        await _pumpEventQueue(128);

[Florian Schneider, 2016/10/31 20:41:01]

dbc: This seems arbitrary. Why not wait on the completion of the analysis
results directly?

[scheglov, 2016/10/31 20:48:14]

Yes, the number is arbitrary.

The question does not make sense to me :-(

What I would like to wait is any other asynchronous processing that requires CPU
attention, i.e. does not sleep or wait for data to arrive.

[Florian Schneider, 2016/10/31 22:16:50]

What happens if this number is too low, or too high?

What I meant was that instead of pumpEventQueue, await on an the future with the
result: see my CL https://codereview.chromium.org/2059503002/ which fixed issue
#26556.

I'm asking because I replaced several uses of pumpEventQueue in tests in the
past to fix timing-dependent analyzer tests failing because some task takes a
bit longer (e.g. running in debug mode). Last time I checked there were still
tests that are timing-dependent. 

[Paul Berry, 2016/10/31 22:40:33]

Unfortunately this is a different situation from the aforementioned CL.  In this
CL, what's happening is that there isn't any particular result that we're
waiting for.  Instead, the situation is that we are trying to do a long-running
computation in such a way that it appears to the user to be a background
process, while allowing other code to use the same event queue to quickly handle
foreground requests from the user.  It would be really cool if there was a way
for this code to say to the event loop "yield to all other events in the queue,
and any other events they may spawn, until the event queue is completely empty. 
Then schedule me to continue executing".  But I don't know of any way to do
that.  So as a poor man's substitute, we are pumping the event queue a large
number of times in the hopes that any higher priority events get completely
taken care of (or go into a state where they are waiting for I/O or timers)
before we try to do more time consuming analysis.

As I mentioned in a previous email, in the long run I would like us to fix this
by moving the time-consuming analysis to its own isolate, so that the main
isolate can respond quickly to user requests.  But it is going to take time to
rework the code to make that possible, so in the short term pumping the event
queue is the best solution we've come up with.

+
         await _hasWork.signal;
 
         if (analysisSection == null) {
           analysisSection = _logger.enter('Analyzing');
         }
 
         // Verify all changed files one at a time.
         if (_changedFiles.isNotEmpty) {
           String path = _removeFirst(_changedFiles);
           _verifyApiSignatureOfChangedFile(path);
@@ skipping 125 matching lines @@
     var completer = new Completer<AnalysisResult>();
     _requestedFiles
         .putIfAbsent(path, () => <Completer<AnalysisResult>>[])
         .add(completer);
     _transitionToAnalyzing();
     _hasWork.notify();
     return completer.future;
   }
 
   /**
-   * Returns a [Future] that completes after pumping the event queue [times]
-   * times. By default, this should pump the event queue enough times to allow
-   * any code to run, as long as it's not waiting on some external event.
-   */
-  Future pumpEventQueue([int times = 5000]) {
-    if (times == 0) return new Future.value();
-    // We use a delayed future to allow microtask events to finish. The
-    // Future.value or Future() constructors use scheduleMicrotask themselves and
-    // would therefore not wait for microtask callbacks that are scheduled after
-    // invoking this method.
-    return new Future.delayed(Duration.ZERO, () => pumpEventQueue(times - 1));
-  }
-
-  /**
    * Remove the file with the given [path] from the list of files to analyze.
    *
    * The [path] must be absolute and normalized.
    *
    * The results of analysis of the file might still be produced by the
    * [results] stream. The driver will try to stop producing these results,
    * but does not guarantee this.
    */
   void removeFile(String path) {
     _explicitFiles.remove(path);
@@ skipping 314 matching lines @@
       // resolution results are invalid.
       if (oldSignature != null && oldSignature != newSignature) {
         _logger.writeln('API signatures mismatch found for $newFile');
         _dependencySignatureMap.clear();
         _filesToAnalyze.addAll(_explicitFiles);
       }
     });
   }
 
   /**
+   * Returns a [Future] that completes after performing [times] pumpings of
+   * the event queue.
+   */
+  static Future _pumpEventQueue(int times) {
+    if (times == 0) {
+      return new Future.value();
+    }
+    return new Future.delayed(Duration.ZERO, () => _pumpEventQueue(times - 1));
+  }
+
+  /**
    * Remove and return the first item in the given [set].
    */
   static Object/*=T*/ _removeFirst/*<T>*/(LinkedHashSet<Object/*=T*/ > set) {
     Object/*=T*/ element = set.first;
     set.remove(element);
     return element;
   }
 }
 
 /**
@@ skipping 484 matching lines @@
       }
     }
     for (UnlinkedExportPublic export in unit.publicNamespace.exports) {
       referenced.exported.add(export.uri);
     }
     return referenced;
   }
 
   _ReferencedUris._();
 }