Live/end-to-end test framework and tests for all providers
abayer opened this issue · 5 comments
I'm not comfortable with the fact that our tests for "send a payload to the provider and get a payload in return" are not really testing anything but that we're sending the payload we expect to send and that we can parse the return payload we expect to get back. I'd really like to see live/end-to-end tests, verifying not just that our code will create a payload we expect when we, say, create a pull request, but that the payload is valid for the provider and that we really do get back the response we'd expect. These wouldn't be tests we'd necessarily run all the time, since we'd need accounts, creds, servers for some of the providers, etc, and we'd have to create/manipulate real resources in those providers, but I think there'd be a ton of value in knowing that we're not just guessing that we're generating the right payload and that we know what to do with the response.
Some ideas for what the live tests should do - with checks to make sure we only run things that we expect to be supported by a particular provider, and with a simple service listening for webhooks.
- Find an org
- List members of an org
- Find an existing user
- Check if a user is a member of an org
- List repos in an org
- Using an existing repo within an org (until we eventually add create/fork a repo!)...
- List hooks
- Create a hook to trigger on all events.
- Push a new commit to master and push a new tag to the repo and make sure we get the right webhook events back
- List commits and tags to find the new commit and tag
- Create a new issue in the repo, check webhook events, and then using that issue...
- List/find the new issue
- Add/list/delete a label to the issue, check webhook events
- Delete the label from the issue, check webhook events
- Add/list/delete a comment to the issue, check webhook events
- Assign the issue, check webhook events
- Unassign the issue, check webhook events
- Close the issue, check webhook events
- Push a new branch to the repo, open a PR using that branch, check webhook events
- Using that PR, add/list/delete a comment, check webhook events
- Add/delete a label, check webhook events
- Add/list/delete a review, check webhook events
- List PRs, find this one
- List changes in the PR
- Push a new commit to the PR, check webhook events
- Merge the PR, check webhook events
- Create another PR
- Close the PR, check webhook events (we maybe should have a reopen, but I'm not sure if that's a common enough API to be worth having and/or simplifying the test)
- Delete the webhook we added earlier
I'm probably missing some little bits here and there, but that should cover nearly all of what we have currently, and we don't have webhook event support for all of those things in every provider, but hey. Obviously as we add more functionality (at some point in the non-distant future, I'll be adding "update title/description/target branch of a PR", e.g.), we add more to the live tests, but I think this is a good start. Also a lot of work, but yeah. =)
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle rotten
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://jenkins-x.io/community.
/close
@jenkins-x-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue with/reopen.
Mark the issue as fresh with/remove-lifecycle rotten.
Provide feedback via https://jenkins-x.io/community.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository.