tc39/proposal-async-iteration

test262 tests

domenic opened this issue · 47 comments

domenic commented

One of the requirements for stage 4 is test262 tests. Besides being a process requirement, it's also great for interoperability.

Are there people that are willing to help work on this? I know @caitp has been working on an implementation in V8, which comes with its own tests presumably in mjsunit format; apparently it's now possible to write test262 tests directly in V8 for later upstreaming, which would be one route.

Alternately, the community and/or myself could help port @caitp's tests and write our own. @benjamingr seems to have volunteered :)

One valuable thing we can do besides even writing tests is come up with a test plan. That is, go through the spec and write down all the testable things and edge case behaviors that we want to be sure to hit when writing tests. Just getting that done alone will help the test writers a lot.

benjamingr commented

I'd love to help with this but I'm not really sure how to get started.

gskachkov commented

+1 I would like to participate also.

domenic commented

I'd love to help with this but I'm not really sure how to get started.

I think a test plan would be the most valuable thing to get started. If you really want to dive in and start writing tests, you can use e.g. https://github.com/tc39/test262/pull/479/files as a guide.

  • 👍1
caitp commented

Here's a starter test plan informed by my experience prototyping. There's plenty of stuff missing, though.

  • Symbol.asyncIterator
    1. test exposed on Symbol with correct property descriptor
  • AsyncGeneratorFunction builtin
    1. not exposed on global
    2. exposed via .constructor with correct property descriptor, etc
    3. has the right length value
    4. tests relating to CreateDynamicFunction
    5. tests relating to Function.prototype.toString() (similar to other tests upstream in test262 --- I really don't want to be the one to write these ones :))
  • For-Await-OF / GetIterator
    1. Test Symbol.asyncIterator is invoked if present on an object during for-await-of (only caller of GetIterator(iterable, async)
    2. Test restriction that Symbol.asyncIterator must return an Object else throw a TypeError
    3. Test that, in the absence of a a Symbol.asyncIterator property (property is undefined or null), Symbol.iterator is invoked instead (and again, if absent, a TypeError is thrown).
    4. Test that in the absence of an asyncIterator symbol, the iterator returned is always an instance of Async-from-Sync Iterator, with [[Prototype]] == %AsyncFromSyncIteratorPrototype%, regardless of what iterator type is returned (see Async-from-Sync iterator section)
  • Async-from-Sync Iterator
    1. Unfortunately, short of for-await-of, there isn't a way to observe access to the Async-from-Sync iterator. If no method for programmatically creating an Async-from-Sync iterator is introduced, we need to expose VM hooks for doing it (similar to $.detachArrayBuffer (tc39/test262#795)).
    2. Test that .next(), .throw() and .return() return Promise instances, which are eventually resolved with expected values.
    3. Test corner cases. In normal Iterators, .return() and .throw() are optional, and for-of loops invoke them conditionally if present. Async-from-Sync iterators always expose them, so we want to check behaviours of when the sync iterator provides them, doesn't provide them, or provides misbehaving versions of them. I don't have a complete list of these corner cases in mind, but I am happy to help get someone started (or finish it myself), just ping me on mozilla IRC in #jslang, or in #v8 on freenode.
    4. Test that, If the value of a sync iterator is a Promise, the resolved value of the Promise becomes the value of the unwrapped iterator result
    5. Similarly, test that if the promise is rejected, the returned Promise from the async-from-sync iterator method is also rejected (Ping @domenic: I think currently Async-from-Sync Iterators just eat these rejections and aren't specified to reject the returned Promise (https://tc39.github.io/proposal-async-iteration/#sec-async-iterator-value-unwrap-functions), but that seems like an oversight. Edit: this seems to work as expected, currently, although I have no idea why it works for rejected Promises, but not resolved Promises. Edit2: I think either A) the spec was not actually eating rejections here and the default promise rejection handler does the right thing, or B) there's a bug in v8 re the default Promise rejection handler, that happens to make things work slightly better/faster :) But we'll look at that elsewhere)
  • Await Expressions in Async Generators
    1. Test that the Generator is not resumed when an Await is in progress (Await must block resumption, otherwise resuming from the Await can have the Generator at an unexpected continuation, or even crash if the VM has "invalid jump table index" assertions like V8 does!)
    2. Test that invalid optimizations (such as https://bugs.chromium.org/p/v8/issues/detail?id=5691, https://bugs.chromium.org/p/v8/issues/detail?id=5694#c3) are not present when evaluating AwaitExpressions.
  • Async Generators
    1. Test that it's possible to call .next(), .resume() and .throw() during execution of an async generator, and those requests will be honoured in order (order of returned Promise resolution for those requests should match the order the requests were made)
    2. Test that once the async generator is closed, it is not possible to resume it any longer

There's a lot of stuff I've left out of this list:

  • yield* in async generators
  • Prototype inheritance of builtin AsyncIterators
  • For-await-of being allowed in "ordinary" Async Functions
  • More in-depth testing of Async Generator resume mechanics (similar to the Sync Iterator tests)
  • 👍3
benjamingr commented

Awesome, thanks! @domenic @caitp I'll take a look this Friday and I'll ensure I know what I'm doing that I'll let you know if I get stuck.

domenic commented

Unfortunately, short of for-await-of, there isn't a way to observe access to the Async-from-Sync iterator. If no method for programmatically creating an Async-from-Sync iterator is introduced, we need to expose VM hooks for doing it (similar to $.detachArrayBuffer

I don't think we want to add this kind of direct test to test262. Instead, we only want to test the aspects that are observable from the language, i.e. for-await-of and yield*. If some implementation were able to implement those without every creating a concrete Async-from-Sync iterator class, then that's great, and should be allowed! We shouldn't write test262 tests that they would start failing.

Maybe we should even mention in the spec the aspects of Async-from-Sync iterator that are unobservable. It sounds like there are a lot.

benjamingr commented

Ok, I have a few volunteers and we'll meet and discuss this this Friday.

caitp commented

I think it's worth having tests upstream wrt interactions between the async iterator and sync iterator (and whoever is calling the iterator methods), because those are observable. It's awkward to only test it in for-await-of, especially if eventually there is another user of GetIterator(async) user.

domenic commented

It may be awkward, but I think it's worth writing awkward tests to allow implementations to optimize fully.

caitp commented

well, by "VM Hooks" I'm not necessarily saying "expose Async-from-Sync Iterator to normal JS", I'm saying "allow VM test harnesses to provide a way to perform this thing, even if normal JS code can't"

I'm absolutely in favour of minimizing the cost of the runtime Async-from-Sync iterator proxy, and if subclassing it or manually creating it, or modifying its prototype chain is out of the question, that works for me

domenic commented

Yeah, but my point is that some implementations might not even be able to expose it, if they've optimized it away super-extensively, and so we shouldn't write tests that those implementations are unable to pass (since they are unable to provide $.createAsyncFromSyncIterator at all).

thefourtheye commented

Could someone please change @caitp's test plan, a check-list? We can tick them whenever tests land.

benjamingr commented

Here's a link to @thefourtheye 's work which we'll build on on Friday (if he won't finish it by then :P)

#71

caitp commented

I'm going to start writing these tests today, as I believe my prototype has become feature complete today.

benjamingr commented

Great, it would be very helpful if you kept a list (like thefourtheye suggested) of tests you've already written and tests you'd like help with - I have 5 hours blocked in my schedule to work on this on Friday and thefourtheye already did some work.

caitp commented

Runtime

  • %AsyncGeneratorFunction%.[[Extensible]] === true
  • %AsyncGeneratorFunction%.[[Call]] exists (is callable)
  • %AsyncGeneratorFunction%.[[Construct]] exists (is a constructor)
  • %AsyncGeneratorFunction%.[[Prototype]] === %FunctionPrototype%
  • %AsyncGeneratorFunction%.name === "AsyncGeneratorFunction" ({ [[Writable]]: false, [[Enumerable]]: false, [[Configurable]]: true }).
  • %AsyncGeneratorFunction%.prototype === %AsyncGenerator% ({ [[Writable]]: false, [[Enumerable]]: false, [[Configurable]]: false })
  • %AsyncGeneratorFunction%.length === 1 ({ [[Writable]]: false, [[Enumerable]]: false, [[Configurable]]: true }).
  • %AsyncGenerator%.constructor === %AsyncGeneratorFunction% ({ [[Writable]]: false, [[Enumerable]]: false, [[Configurable]]: true }).
  • %AsyncGenerator%.prototype === %AsyncGeneratorPrototype% ({ [[Writable]]: false, [[Enumerable]]: false, [[Configurable]]: true }).
  • %AsyncGenerator%[@@toStringTag] === "AsyncGeneratorFunction" ({ [[Writable]]: false, [[Enumerable]]: false, [[Configurable]]: true }).
  • %AsyncGeneratorPrototype%.[[Prototype]] === %AsyncIteratorPrototype%.
  • %AsyncGeneratorPrototype%.[[Extensible]] === true.
  • %AsyncGeneratorPrototype%.constructor === %AsyncGenerator% ({ [[Writable]]: false, [[Enumerable]]: false, [[Configurable]]: true }).
  • %AsyncGeneratorPrototype%.next.[[Call]] exists (is callable).
  • %AsyncGeneratorPrototype%.next.[[Construct]] does not exist (is not a constructor).
  • %AsyncGeneratorPrototype%.next() throws a TypeError when receiver is not an AsyncGenerator instance.
  • %AsyncGeneratorPrototype%.next() returns an instance of %Promise%.
  • %AsyncGeneratorPrototype%.return.[[Call]] exists (is callable).
  • %AsyncGeneratorPrototype%.return.[[Construct]] does not exist (is not a constructor).
  • %AsyncGeneratorPrototype%.return() throws a TypeError when receiver is not an AsyncGenerator instance.
  • %AsyncGeneratorPrototype%.return() returns an instance of %Promise%.
  • %AsyncGeneratorPrototype%.throw.[[Call]] exists (is callable).
  • %AsyncGeneratorPrototype%.throw.[[Construct]] does not exist (is not a constructor).
  • %AsyncGeneratorPrototype%.throw throws a TypeError when receiver is not an AsyncGenerator instance.
  • %AsyncGeneratorPrototype%.throw() returns an instance of %Promise%.
  • %AsyncGeneratorPrototype%[@@toStringTag] === "AsyncGenerator" ({ [[Writable]]: false, [[Enumerable]]: false, [[Configurable]]: true }).
  • AsyncGeneratorDeclaration.prototype.[[Prototype]] === %AsyncGeneratorPrototype.
  • AsyncGeneratorExpression.prototype.[[Prototype]] === %AsyncGeneratorPrototype.
  • AsyncGeneratorMethod.prototype.[[Prototype]] === %AsyncGeneratorPrototype.
  • AsyncGeneratorDeclaration().[[Prototype]] === %AsyncGeneratorPrototype.
  • AsyncGeneratorExpression().[[Prototype]] === %AsyncGeneratorPrototype.
  • AsyncGeneratorMethod().[[Prototype]] === %AsyncGeneratorPrototype.
  • Promises returned from async iterator protocol methods of %AsyncGeneratorPrototype% are resolved in FIFO order.
  • AsyncGeneratorExpression YieldExpression with an AwaitExpression as its operand, where the AwaitExpression's operand waits for a Promise or Promise subclass (and invokes .then()).
  • AsyncGeneratorExpression YieldExpression with an AwaitExpression as its operand, where the AwaitExpression's operand waits for a custom non-Promise Object with a then method (and then method is invoked)
  • AsyncGeneratorExpression YieldExpression with an AwaitExpression as its operand, where the AwaitExpression's operand waits for a non-thenable.
  • AsyncGeneratorExpression YieldExpression with a YieldExpression as its operand

Grammar

  • AsyncGeneratorDeclaration throw a SyntaxError if BindingIdentifier is "await" and function is nested inside of an AsyncFunctionDeclaration, AsyncFunctionExpression or AsyncMethod, AsyncGeneratorDeclaration, AsyncGeneratorExpression or AsyncGeneratorMethod,
  • AsyncGeneratorDeclaration throw a SyntaxError if BindingIdentifier is "yield" and function is nested inside of a GeneratorDeclaration, GeneratorExpression, GeneratorMethod, AsyncFunctionExpression or AsyncMethod.
  • AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains yield (invalid BindingIdentifier)
  • AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains await (invalid BindingIdentifier)
  • AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains AwaitExpression
  • AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains SuperCall
  • AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains SuperProperty
  • AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains YieldExpression
  • AsyncGeneratorDeclaration throw a SyntaxError if AsyncGeneratorBody contains SuperCall
  • AsyncGeneratorDeclaration throw a SyntaxError if AsyncGeneratorBody contains SuperProperty
  • AsyncGeneratorDeclaration throw a SyntaxError if ContainsUseStrict of AsyncGeneratorBody is true and IsSimpleParameterList of FormalParameters is false.
  • AsyncGeneratorDeclaration throw a SyntaxError if any element of the BoundNames of FormalParameters also occurs in the LexicallyDeclaredNames of AsyncGeneratorBody.
  • AsyncGeneratorExpression throw a SyntaxError if BindingIdentifier is "await".
  • AsyncGeneratorExpression throw a SyntaxError if BindingIdentifier is "yield".
  • AsyncGeneratorExpression throw a SyntaxError if FormalParameters contains yield (invalid BindingIdentifier)
  • AsyncGeneratorExpression throw a SyntaxError if FormalParameters contains await (invalid BindingIdentifier)
  • AsyncGeneratorExpression throw a SyntaxError if FormalParameters contains BindingIdentifier
    arguments in strict mode
  • AsyncGeneratorExpression throw a SyntaxError if function BindingIdentifier is
    arguments in strict mode
  • AsyncGeneratorExpression throw a SyntaxError if function BindingIdentifier is
    eval in strict mode
  • AsyncGeneratorExpression throw a SyntaxError if FormalParameters contains BindingIdentifier eval in strict mode
  • AsyncGeneratorExpression throw a SyntaxError if FormalParameters contains AwaitExpression
  • AsyncGeneratorExpression throw a SyntaxError if FormalParameters contains SuperCall
  • AsyncGeneratorExpression throw a SyntaxError if FormalParameters contains SuperProperty
  • AsyncGeneratorExpression throw a SyntaxError if FormalParameters contains YieldExpression
  • AsyncGeneratorExpression throw a SyntaxError if AsyncGeneratorBody contains SuperCall
  • AsyncGeneratorExpression throw a SyntaxError if AsyncGeneratorBody contains SuperProperty
  • AsyncGeneratorExpression throw a SyntaxError if ContainsUseStrict of AsyncGeneratorBody is true and IsSimpleParameterList of FormalParameters is false.
  • AsyncGeneratorExpression throw a SyntaxError if any element of the BoundNames of FormalParameters also occurs in the LexicallyDeclaredNames of AsyncGeneratorBody. (let variables)
  • AsyncGeneratorExpression throw a SyntaxError if any element of the BoundNames of FormalParameters also occurs in the LexicallyDeclaredNames of AsyncGeneratorBody. (const variables)
  • AsyncGeneratorExpression throw a SyntaxError if LabelIdentifier is yield (invalid LabelIdentifier in [+Yield] production)
  • AsyncGeneratorExpression throw a SyntaxError if LabelIdentifier is await (invalid LabelIdentifier in [+Await] production)
  • AsyncGeneratorExpression throw an early ReferenceError if AsyncGeneratorExpression is the target of an assignment operation or count operation
  • AsyncGeneratorExpression throw a SyntaxError if there is a line terminator between yield and *.
  • AsyncGeneratorExpression YieldExpression parsed as an ExpressionStatement
  • AsyncGeneratorExpression YieldExpression with automatic semicolon insertion between yield and operand yields undefined
  • AsyncGeneratorExpression yield* YieldExpression does not insert semicolon if line terminator between * and operand.
  • AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains yield (invalid BindingIdentifier)
  • AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains await (invalid BindingIdentifier)
  • AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains duplicate entries.
  • AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains AwaitExpression
  • AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains YieldExpression.
  • AsyncGeneratorMethod throw a SyntaxError if HasDirectSuper of AsyncGeneratorMethod is true.
  • AsyncGeneratorMethod throw a SyntaxError if ContainsUseStrict of AsyncGeneratorBody is true and IsSimpleParameterList of UniqueFormalParameters is false.
  • AsyncGeneratorMethod throw a SyntaxError if any element of the BoundNames of UniqueFormalParameters also occurs in the LexicallyDeclaredNames of AsyncGeneratorBody.
caitp commented

There's a bunch of things to check off (I know some of these have been covered already).

There's a ton more stuff to add to the list, though.

domenic commented

I wonder if there's a better way to organize this than GitHub checklists, given that those require write access to check. Trello board? Separate shared repo? I dunno, suggestions welcome.

  • 👍1
benjamingr commented

Going to start working on this now. Separate repo and trello board sound good.

benjamingr commented

WIP - doing (updating as I go):

  • %AsyncGeneratorPrototype%.next() throws a TypeError when receiver is not an AsyncGenerator instance.
  • Added tests for Symbol.asyncIterator
  • Added tests under built-ins
domenic commented

%AsyncGeneratorPrototype%.next() throws a TypeError when receiver is not an AsyncGenerator instance.

This seems to be incorrect per spec actually. It should reject with a TypeError.

sashakru commented

WIP - doing (updating as I go):

AsyncGeneratorDeclaration throw a SyntaxError if AsyncGeneratorBody contains SuperProperty
AsyncGeneratorDeclaration throw a SyntaxError if AsyncGeneratorBody contains SuperCall
AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains YieldExpression
AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains SuperProperty
AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains SuperCall
AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains AwaitExpression
AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains await (invalid BindingIdentifier)
AsyncGeneratorDeclaration throw a SyntaxError if FormalParameters contains yield (invalid BindingIdentifier)

benjamingr commented

@domenic ptal at https://github.com/caitp/test262/pull/2/files and let me know if what we're doing makes sense.

benjamingr commented

This seems to be incorrect per spec actually. It should reject with a TypeError.

Would you prefer it if we wrote tests for the current spec and update when it updates or would you prefer it if we test it rejects?

Also, I'm on IRC if you want more direct comm

domenic commented

I don't understand the question? The current spec says it rejects.

benjamingr commented

@domenic roger, will fix.

Question about: AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains duplicate entries.:

async functions don't do this outside of strict mode - do async generators throw in sloppy mode?

caitp commented

Yeah, you're right, should be written as rejects promise

Would you prefer it if we wrote tests for the current spec and update when it updates or would you prefer it if we test it rejects?

The current spec says to reject, not throw, and it's likely to stay that way. In async functions/generators, pretty much any instance of "throw" really means reject, other than for early errors

caitp commented

async functions don't do this outside of strict mode - do async generators throw in sloppy mode?

These are specific to the Method versions, which always have strict formal parameters whether in strict mode or not

  • 👍1
benjamingr commented

@caitp can you please take a look at the start of the work (as a PR to your fork of test262) and let us know if we're on the right track or not?

benjamingr commented

@domenic are async generators subclasses of AsyncFunction or Function? From the spec it looks like they're a direct subclass of Function and I'm wondering if that's intentional (just making sure).

domenic commented

Function. That's intentional.

  • 👍1
sashakru commented

Hey,

how can I test this one: AsyncGeneratorDeclaration throw a SyntaxError if BindingIdentifier is "yield" and function is nested inside of a GeneratorDeclaration, GeneratorExpression, GeneratorMethod, AsyncFunctionExpression or AsyncMethod? Is there any specific folder to put this tests?

Linkgoron commented

Hi,

Added tests for:
AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains yield (invalid BindingIdentifier)
AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains await (invalid BindingIdentifier)
AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains duplicate entries.
AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains AwaitExpression
AsyncGeneratorMethod throw a SyntaxError if UniqueFormalParameters contains YieldExpression.
AsyncGeneratorMethod throw a SyntaxError if HasDirectSuper of AsyncGeneratorMethod is true.

benjamingr commented

We're done for now - @thefourtheye might add more tests later today. All collaborators (me, @sashakru , @Linkgoron and @thefourtheye signed the CLA) and I'll rebase after today so it's all neatly done in a single PR (to @caitp's own PR for test262 for easy validation)

littledan commented

Great to see more contributors to test262 here!

benjamingr commented
littledan commented

Separate PRs seem like a good idea, for licensing at the very least. Let me know if you need any help getting that set up. Looks like there's still a bunch more checkmarks unchecked, so if you're available for it, then just plugging away sounds great.

caitp commented

I think the checklist is pretty incomplete, anyways :( there's not really much mention of for-await-of or it's interactions with rejected promises or AsyncIteratorClose.

I'm sure there's a ton of stuff left to add.
Has anybody hosted a separate checklist that more people have write access to? That would make this easier to track.

benjamingr commented

I'll do more work on Friday to ensure that our PRs are split into separate items - then I'll look at the tests for iterators and for async functions and see what's tested. Hopefully I'll be able to figure it out.

littledan commented

I don't think it's necessary to split the PRs up to be small. The more important thing is that the PRs are filed against the upstream test262, rather than Caitlin's repository. It's fine for a commit to have tons of tests in it or just one test.

arai-a commented

started writing tests for yield* in async generator
tc39/test262#844

domenic commented

@benjamingr @thefourtheye would you mind submitting your tests to the test262 repository directly?

@arai-a sorry for missing that ping. I will give feedback on the tests on Monday!

arai-a commented

another small chunk for abrupt cases in yield* in async generator.
tc39/test262#873

arai-a commented

added another chunk in yield* tc39/test262#883

jburton-scottlogic commented

Was a list ever created to track the remaining tests?

  • 👍2