Create automated testing
SocksTheWolf opened this issue · 9 comments
It would be great to have a series of unit tests built in that could be ran and validated against. I'm not aware of any python unit test frameworks and their functionalities, so this would require a bit of research.
Scope of task is as follows:
- Find unit test library to use
- Create bot in discord developer center for just testing
- Implement multiple unit test cases (see below)
- Create testing folder for handling said tests
Tests to run:
- Command sent for banning
- Command sent for unban
- Banning in a server
- Banning without permissions
- Invalid input checking
- more as necessary
On finish of this task, a creation of a new task to have the system automate a test run on python commits/merge requests should be made.
This task should wait until #3 is completed, and probably should wait until #28 is too
We’ll want to have a special discord key for this version of the bot so that tests can be run. Can confirm bans by requesting audit log permissions.
Looking online, pytest seems the most popular for unit testing frameworks. They have a feature called fixtures that we can use to share a single instance of the discord connection throughout the test https://docs.pytest.org/en/6.2.x/fixture.html#scope-sharing-fixtures-across-classes-modules-packages-or-session. It should be easy to integrate into a pipeline script too.
I'm trying out dpytest which mocks all the backend stuff which would save us time from needing to setup and maintain a discord server for ci testing. Unfortunately, it does not seem like it has support for commands. I sent a "/activate" message and it was unable to trigger the command code in Main.
Edit: I completely forgot the main reason to switch to commands was to remove message parsing so the mock messages not working makes sense
avaankha/dpytest-example
With dpytest, I was able to write a test for DiscordBot.PrepareBan() with DEVELOPMENT_MODE set to true. dpytest will not work with DEVELOPEMENT_MODE set to false since there are a few http calls that are not mocked/supported. Also, dpytest needs the bot to have member intents/permissions to properly setup test users in the mocked environment
avaankha/pytest
Here's a branch with only pytest testing BotDatabase.AddBan(). It's more of an unit test so we'd need a few together to cover the test cases above
Overall, I think it will be very difficult to write a complete end-to-end automated test to cover the mentioned test cases above. We could extract the commands call via http sniffers as shown here and then use them in our testing but it seems very hacky how-to-invoke-a-discord-bot-command-via-rest-api
I'll take a look in more detail later, thank you for getting started on this!
I think setting up a secondary bot that is just for testing makes sense, we could potentially run it as a github action then to verify features are working well.
Sounds good. I'm going to leave those two pytest branches for now since they're unit tests rather than functional tests that we want. I'll start looking into the discord command http calls and see what I can do to automate them being called. Once I figure that out I'm sure the rest of the test script should be easy.
I've hit a roadblock with this. While I was able to verify that it was possible to trigger a command through a http request, (https://discord.com/api/v9/interactions endpoint) it was only with a user auth token. Replacing it with a bot token returned "401: Unauthorized". I've searched online and I don't think anyone has tried this out. The bot triggering the command has admin permissions and I've been trying the global unrestricted commands "activate/deactivate" but it doesn't get past authentication. I've opened up a discord support ticket to see if there's any special permissions I need to set for it. Hopefully I get some answers soon.
In the meantime, should we try a different approach with automated testing? I can try to use the built-in mocking library and see if I can setup an automated test case for the command functions like ScamBan and ScamUnban. Would that be a valid alternative?
Hey sorry for disappearing for a month. IRL work got in the way. For the Bot token issue above, Discord support hasn't given me an answer so I think that approach won't work.
For the mocking approach. I did not realize that the decorators convert the command function into a Command object. I originally thought I could mock the interaction and targetid and just call ScamBan/ScamUnban directly but that is not the case.
With the mocking/patching approach, I have been able to create a test for DiscordBot.PrepareBan() and DiscordBot.PrepareUnban() functions without needing to elevate privileges like with dpytest. My current big function test covers the following scenarios
- When calling PrepareBan() with an id that hasn't been banned, Guild.ban for each guild set in the db is called with the id and the result is BanLookup.Banned
- When calling PrepareBan() with an id that already has been banned, Guild.ban is not called and the result is BanLookup.Duplicate
- When Calling PrepareUnban() with an id that has been banned, Guild.unban for each guild set in the db is called and the result is BanLookup.Unbanned
- When Calling PrepareUnban() with an id that has been unbanned, Guild.unban is not called and the result is BanLookup.NotExist
You can see this automated test in my pytest branch if you're interested. I'll be working on creating a test script for the activate command as it looks very complex and a core part of the bot.