Include success and error outcomes in GraphQL Coverage
Closed this issue · 8 comments
In contrast to OpenAPI where we can declare possible outcomes in the schema itself (e.g. 200, 201, 302, 400, 412, default), graphQL responses have either a “happy” outcome with the data property populated (and errors being null) XOR an error outcome where data is null and errors is populated.
We can enhance graphql support in webtau by saying the set of “declared queries” consists of both the happy and error outcomes of all queries declared in the schema.
Recording results would have to inspect the response body and determine happiness based on the data/errors properties.
100% test coverage then means that every declared query in our schema has been hit with at least 1 successful and 1 error outcome.
Alternative:
In addition to graphQLSkippedQueries and graphQLCoveredQueries, we can introduce ReportCustomData for graphQLCoveredQueriesWithData and graphQLCoveredQueriesWithErrors. That would give us the opportunity to introduce something similar to "branch coverage" vs "method coverage" where we can add covered happy paths = graphQLCoveredQueriesWithData / declaredQueries and covered error paths = graphQLCoveredQueriesWithErrors / declaredQueries
I like that we keep thinking about coverage! We need to push it forward.
Will this be solved by finally implementing schema coverage as opposite to operations coverage?
Will this be solved by finally implementing schema coverage as opposite to operations coverage?
Sadly not because GraphQL doesn't provide a mechanism for modelling errors.
Do you have a preference for option 1 (declared including success + error) or option 2 (additional custom data)?
The longer I think about it, the more I prefer the alternative with additional custom data.
I definitely prefer option 2.
I may not fully follow it all yet. Will we be able to introduce similar approach to open api? I think one if the ideas we considered is to merge data or inner structures to collect coverage.
doing "the same" for OpenAPI is trickier because statuses are not either happy or failure (data vs error), but we have a range of status codes. Can we definitively say what is a success case (e.g. status code <400) and what is an error (e.g. >=400)?
The 3xx range is grey zone.
Interesting. Errors in graphQL can also be different. Server error. Auth error. Basically even more errors than open API since it is completely up to a server to implement.
taking a crack at this now