Assorted fixes to async: yield may suspend in async* methods; yield* skips emptysequences; definiti…

docs/language/dartLangSpec.tex

 \documentclass{article}
 \usepackage{epsfig}
 \usepackage{color}
 \usepackage{dart}
 \usepackage{bnf}
 \usepackage{hyperref}
 \usepackage{lmodern}
 \newcommand{\code}[1]{{\sf #1}}
 \title{Dart Programming Language  Specification \\
 {\large Version 1.9}}
-%\author{The Dart Team}
 
 % For information about Location Markers (and in particular the
 % commands \LMHash and \LMLabel), see the long comment at the
 % end of this file.
 
 \begin{document}
 \maketitle
 \tableofcontents
 
 
@@ skipping 11 matching lines @@
 
 \section{Conformance}
 \LMLabel{ecmaConformance}
 
 \LMHash{}
 A conforming  implementation of the Dart programming language must provide and support all the  APIs (libraries, types, functions, getters, setters, whether top-level, static, instance or local) mandated in this specification. 
 
 \LMHash{}
 A conforming implementation is permitted to provide additional APIs, but not additional syntax.
 
-% A claim of conformance with this Ecma Standard shall specify?
-
 \section{Normative References}
 \LMLabel{ecmaNormativeReferences}
 
 \LMHash{}
 The following referenced documents are indispensable for the application of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies.
 
 \begin{enumerate}
 \item
 The Unicode Standard, Version 5.0, as amended by Unicode 5.1.0, or successor.
 \item
@@ skipping 3030 matching lines @@
 
 $(T_1$ $a_1, \ldots, T_n$ $a_n, [T_{n+1}$ $x_{n+1} = d_1, \ldots,  T_{n+k}$ $x_{n+k} = d_k]) => e$ 
 is 
 
 $(T_1 \ldots, T_n, [T_{n+1}$ $x_{n+1}, \ldots, T_{n+k}$ $x_{n+k}]) \rightarrow T_0$, where $T_0$ is the static type of $e$. 
 
 \LMHash{}
 The static type of a function literal of the form 
 
 $(T_1$ $a_1, \ldots, T_n$ $a_n, [T_{n+1}$ $x_{n+1} = d_1, \ldots,  T_{n+k}$ $x_{n+k} = d_k])$ \ASYNC{} $=> e$
-is $(T_1 \ldots, T_n, [T_{n+1}$ $x_{n+1}, \ldots, T_{n+k}$ $x_{n+k}]) \rightarrow Future<flatten(T_0)>$, where $T_0$ is the static type of $e$ and  $flatten(T) = flatten(S)$ if $T = Future<S>$, and $T$ otherwise. 
+is $(T_1 \ldots, T_n, [T_{n+1}$ $x_{n+1}, \ldots, T_{n+k}$ $x_{n+k}]) \rightarrow Future<flatten(T_0)>$, where $T_0$ is the static type of $e$ and  $flatten(T)$ is defined as follows:
+
+ If $T = Future<S>$ then $flatten(T) = flatten(S)$. 
+ 
+ Otherwise if $T <: Future$ then let $S$ be a type such that $T << Future<S>$ and for all $R$, if $T << Future<R>$ then $S << R$.  
+
+\rationale{
+This ensures that $Future<S>$ is the most specific instantiation of \cd{Future} that is a super type of $T$.
+}
+
+Then $flatten(T) =  S$.  
+
+In any other circumstance, $flatten(T) = T$.
+
+
 
 \rationale{
 We collapse multiple layers of futures into one. If $e$ evaluates to a future $f$, the future will not invoke its \code{then()} callback until f completes to a non-future value, and so the result of an await is never a future, and the result of an async function will never have type \code{Future$<X>$} where $X$ itself is an invocation of \code{Future}. 
+
+The  exception to that would be a type $X$ that extended or implemented \code{Future}. In that case, only one unwrapping takes place. As an example of why this is done, consider
+
+\cd{\CLASS{} C$<$T$>$  \IMPLEMENTS{}  Future$<$C$<$C$<$T$>>>$ \ldots }
+
+Here, a naive definition of $flatten$ diverges; there is not even a fixed point. A more sophisticated definition of $flatten$ is possible, but the existing rule deals with most realistic examples while remaining relatively simple to understand.
+
 }
 
 
 \LMHash{}
 The static type of a function literal of the form 
 
 $(T_1$ $a_1, \ldots, T_n$ $a_n, \{T_{n+1}$ $x_{n+1} : d_1, \ldots,  T_{n+k}$ $x_{n+k} : d_k\}) => e$ 
 is 
 
 $(T_1 \ldots, T_n, \{T_{n+1}$ $x_{n+1}, \ldots, T_{n+k}$ $x_{n+k}\}) \rightarrow T_0$, where $T_0$ is the static type of $e$. 
@@ skipping 412 matching lines @@
 }
 
 \LMHash{}
 The iterable implementation must comply with the contract of \code{Iterable} and should not take any steps identified as exceptionally efficient in that contract.
 
 \commentary {
 The contract explicitly mentions a number of situations where certain iterables could be more efficient than normal. For example, by precomputing their length. Normal iterables must iterate over their elements to determine their length. This is certainly true in the case of a synchronous generator, where each element is computed by a function. It would not be acceptable to pre-compute the results of the generator and cache them, for example.
 }
 
 \LMHash{}
-When iteration over the iterable is started, by getting an iterator $j$ from the iterable and calling \code{moveNext()} on it, execution of the body of $f$ will begin. When $f$ terminates, $j$ is positioned after its last element, so that its current value is \NULL{} and the current call to \code{moveNext()} on $j$ returns false, as will all further calls.
+When iteration over the iterable is started, by getting an iterator $j$ from the iterable and calling \code{moveNext()}, execution of the body of $f$ will begin. When $f$ terminates, $j$ is positioned after its last element, so that its current value is \NULL{} and the current call to \code{moveNext()} on $j$ returns false, as will all further calls.
 
 Each iterator starts a separate computation. If the \SYNC* function is impure, the sequence of values yielded by each iterator may differ. 
 
 \commentary{
 One can derive more than one iterator from a given iterable.   Note that operations on the iterable itself can create distinct iterators. An example would be \code{length}.  It is conceivable that different iterators might yield sequences of different length. The same care needs to be taken when writing \SYNC* functions as when
 writing an \code{Iterator} class. In particular, it should handle multiple
 simultaneous iterators gracefully. If the iterator depends on external state
 that might change, it should check that the state is still valid after every
 yield (and maybe throw a \code{ConcurrentModificationError} if it isn't).
 }
@@ skipping 2306 matching lines @@
 \begin{grammar}
 {\bf breakStatement:}
      \BREAK{} identifier? `{\escapegrammar ;}'
     .
  \end{grammar}
  
 \LMHash{}
 Let $s_b$ be a \BREAK{} statement. If $s_b$ is of the form  \code{\BREAK{} $L$;}, then let $s_E$ be the the innermost labeled statement with label $L$ enclosing $s_b$. If $s_b$ is of the form \code{\BREAK{};},  then let $s_E$ be the the innermost  \DO{} (\ref{do}), \FOR{} (\ref{for}), \SWITCH{} (\ref{switch}) or \WHILE{} (\ref{while}) statement enclosing  $s_b$. It is a compile-time error if no such statement $s_E$ exists within the innermost function in which  $s_b$ occurs.  Furthermore, let $s_1, \ldots, s_n$ be those \TRY{} statements that are both enclosed in $s_E$ and that enclose  $s_b$, and that have a \FINALLY{} clause. Lastly, let $f_j$ be the \FINALLY{} clause of $s_j, 1 \le j \le n$.   Executing  $s_b$ first executes $f_1, \ldots,  f_n$ in innermost-clause-first  order and then terminates $s_E$. 
 
 \LMHash{}
 If $s_E$ is an asynchronous for loop (\ref{asynchronousFor-in}), its associated stream subscription is canceled. Furthermore, let $a_k$ be the set of asynchronous for loops  and yield-each statements (\ref{yieldEach}) enclosing $s_b$ that are enclosed in $s_E , 1 \le k \le m$.   The stream subscriptions associated with $a_j$ are canceled, $1 \le j \le m$. 

[Lasse Reichstein Nielsen, 2015/03/19 09:14:57]

Just noticed: When the stream subscription is canceled, the call to
subscription.cancel may return a future. 
If so, the implementation should wait for that future to complete before
continuing with the code outside the loop.

If the stream is created using async*, cancel always returns a future containing
the result of the stream cleanup (execution of the finally blocks that the yield
transferred control to when it noticed that it was canceled). Manually created
streams may also return a future that tells you when clean-up is done.
It is important to wait for that future before continuing.

Also, when canceling multiple subscriptions, they should be canceled from the
inside out, and each one should be waited on before the next one is canceled.

Same goes for continue and the handlers hit by return/throw.

 
 
 
 \subsection{ Continue}
 \LMLabel{continue}
 
 \LMHash{}
 The {\em continue statement} consists of the reserved word \CONTINUE{} and an optional label (\ref{labels}). 
 
 \begin{grammar}
@@ skipping 37 matching lines @@
 Next, $o$ is added to the iterable or stream associated with the immediately enclosing function. 
 
 \LMHash{}
 If the enclosing function $m$ is marked \ASYNC* and the stream $u$ associated with $m$ has been canceled, then let $c$ be the \FINALLY{} clause (\ref{try}) of the innermost enclosing try-finally statement, if any. If $c$ is defined, let $h$ be the handler induced by $c$. If $h$ is defined, control is transferred to $h$. If $h$ is undefined, the immediately enclosing function terminates.
 
 \rationale{
 The stream associated with an asynchronous generator could be canceled by any code with a reference to that stream at any point where the generator was passivated.  Such a cancellation constitutes an irretrievable error for the generator. At this point, the only plausible action for the generator is to clean up after itself via its \FINALLY{} clauses.
 }
 
 \LMHash{}
+Otherwise, if the enclosing function $m$ is marked \ASYNC* (\ref{functions}) then the enclosing function may suspend.
+
+\rationale {
+If a \YIELD{} occurred inside an infinite loop and the enclosing function never suspended, there might not be an opportunity for consumers of the  enclosing stream to run and access the data in the stream.  The stream might then accumulate an unbounded number of elements. Such a situation is untenable. Therefore, we allow the enclosing function to be suspended when a new value is added to its associated stream. However, it is not essential (and in fact, can be quite costly) to suspend the function on every \YIELD{}. The implementation is free to decide how often to suspend the enclosing function. The only requirement is that consumers are not blocked indefinitely.
+}
+
+
+\LMHash{}
 If the enclosing function $m$ is marked \SYNC* (\ref{functions}) then:
 \begin{itemize}
 \item
-Execution of the function $m$ immediately enclosing $s$ is suspended until the method \code{moveNext()} is invoked upon the iterator used to initiate the current invocation of $m$. 
+Execution of the function $m$ immediately enclosing $s$ is suspended until the nullary method \code{moveNext()} is invoked upon the iterator used to initiate the current invocation of $m$. 
 \item
 The current call to \code{moveNext()} returns \TRUE.
 \end{itemize}
 
 \LMHash{}
 It is a compile-time error if a yield statement appears in a function that is not a generator function.
 
 \LMHash{}
 Let $T$ be the static type of $e$ and let $f$ be the immediately enclosing function.  It is a static type warning if either:
 \begin{itemize}
@@ skipping 10 matching lines @@
 \LMHash{}
  The {\em yield-each statement} adds a series of values to the result of a generator function (\ref{functions}).
  
  \begin{grammar}
 {\bf yieldEachStatement:}
    \YIELD* expression `{\escapegrammar ;}'
       .
 \end{grammar}
 
 \LMHash{}
-Execution of a statement s of the form \code{\YIELD* $e$;}  proceeds as follows:
+Execution of a statement $s$ of the form \code{\YIELD* $e$;}  proceeds as follows:
 
 \LMHash{}
-First, the expression $e$ is evaluated to an object $o$. If the immediately enclosing function $m$ is synchronous, then it is a dynamic error if the class of $o$ does not implement \code{Iterable}.  If $m$ asynchronous, then it is a dynamic error if the class of $o$ does not implement \code{Stream}. Next, for each element $x$ of $o$:
+First, the expression $e$ is evaluated to an object $o$. 
+
+\LMHash{}
+If the immediately enclosing function $m$ is marked \SYNC* (\ref{functions}), then:
+\begin{enumerate}
+\item It is a dynamic error if the class of $o$ does not implement \code{Iterable}.  Otherwise
+\item The method \cd{iterator} is invoked upon $o$ returning an object $i$. 
+\item \label{moveNext} The \cd{moveNext} method of $i$ is invoked on it with no arguments. If \cd{moveNext} returns \FALSE{} execution of $s$ is complete. Otherwise
+\item The getter \cd{current} is invoked on $i$. If the invocation raises an exception $ex$, execution of $s$ throws $ex$. Otherwise, the result $x$ of the getter invocation is added to the iterable associated with $m$.
+Execution of the function $m$ immediately enclosing $s$ is suspended until the nullary method \code{moveNext()} is invoked upon the iterator used to initiate the current invocation of $m$, at which point execution of $s$ continues at \ref{moveNext}. 
+\item
+The current call to \code{moveNext()} returns \TRUE.
+\end{enumerate}
+
+\LMHash{}
+If $m$ is marked \ASYNC* (\ref{functions}), then:
+\begin{itemize}
+\item  It is a dynamic error if the class of $o$ does not implement \code{Stream}. Otherwise
+\item For each element $x$ of $o$:
 \begin{itemize}
 \item
-If $m$ is marked \ASYNC* (\ref{functions}) and the stream $u$ associated with $m$ has been paused,  then execution of $m$ is suspended until $u$ is resumed or canceled.
+If the stream $u$ associated with $m$ has been paused,  then execution of $m$ is suspended until $u$ is resumed or canceled.
+ \item
+If the stream $u$ associated with $m$ has been canceled, then let $c$ be the \FINALLY{} clause (\ref{try}) of the innermost enclosing try-finally statement, if any. If $c$ is defined,  let $h$ be the handler induced by $c$. If $h$ is defined, control is transferred to $h$. If $h$ is undefined, the immediately enclosing function terminates.
 \item
- $x$ is added to the iterable or stream associated with $m$ in the order it appears in $o$.  
- \item
-If $m$ is marked \ASYNC* and the stream $u$ associated with $m$ has been canceled, then let $c$ be the \FINALLY{} clause (\ref{try}) of the innermost enclosing try-finally statement, if any. If $c$ is defined,  let $h$ be the handler induced by $c$. If $h$ is defined, control is transferred to $h$. If $h$ is undefined, the immediately enclosing function terminates.
+Otherwise,  $x$ is added to the stream associated with $m$ in the order it appears in $o$.  The function $m$ may suspend.
+\end{itemize}
+\item If the stream $o$ is done, execution of $s$ is complete. 
 \end{itemize}
 
-\LMHash{}
-If the enclosing function is marked \SYNC* (\ref{functions}) then:
-\begin{itemize}
-\item
-Execution of the function $m$ immediately enclosing $s$ is suspended until the method \code{moveNext()} is invoked upon the iterator used to initiate the current invocation of $m$. 
-\item
-The current call to \code{moveNext()} returns \TRUE.
-\end{itemize}
 
 \LMHash{}
 It is a compile-time error if a yield-each statement appears in a function that is not a generator function.
 
 \LMHash{}
 Let $T$ be the static type of $e$ and let $f$ be the immediately enclosing function.  It is a static type warning if $T$ may not be assigned to the declared return type of $f$.  If $f$ is synchronous it is a static  type warning if $T$ may not be assigned to \code{Iterable}.  If $f$ is asynchronous it is a static  type warning if $T$ may not be assigned to \code{Stream}. 
 
 
 \subsection{ Assert}
 \LMLabel{assert}
@@ skipping 1402 matching lines @@
 
 The invariant that each normative paragraph is associated with a line
 containing the text \LMHash{} should be maintained.  Extra occurrences
 of \LMHash{} can be added if needed, e.g., in order to make
 individual \item{}s in itemized lists addressable.  Each \LM.. command
 must occur on a separate line.  \LMHash{} must occur immediately
 before the associated paragraph, and \LMLabel must occur immediately
 after the associated \section{}, \subsection{} etc.
 
 ----------------------------------------------------------------------