Training the object detector
Closed this issue · 4 comments
Hi,
This project is pretty interesting. I was wondering how you obtained the labeled image data set and perform the training to get the object detector model. As there are multiple objects that could be found on any given screen capture throughout the game play, I don't have much idea on how to accomplish it. The riot developer API also doesn't seem to have the functionality for you to query useful information on the map at any given time. Would you mind sharing some insights?
Hi,
thank you for your interest in the project!
The object detection is using the tensorflow object detection API: https://github.com/tensorflow/models/tree/master/research/object_detection
And you are right there is unfortunately no official API to interact with the game. So the LeagueAI uses object detection to detect 3 different objects in the game (The champion Vayne, enemy minions and enemy towers). When the position of the playercharacter is determined a grid is put on top of the game just to simplify the environment for decision making.
To train the object detector I just spent some (or let's say a lot of) time recording the objects I wanted to learn in the game, export single frames and labeled them by hand using this program: https://github.com/tzutalin/labelImg. Then i retrained an existing tensorflow object detection model with my labeled images. The learned model is included in the repository: https://github.com/Oleffa/LeagueAI/blob/master/LeagueAI_v3.zip
This could still be improved because when working on this project I was running the game + the LeagueAI on my small notebook, so I had to use some faster but not so precise models. With a powerful machine one could retrain a better model with the images to improve detection performance. Also it is always possible to add more objects to the detector. Like own minions or more different champions to give the decision making more options to work with.
I can really recommend this tutorial to get started with the tensorflow object detection API: https://pythonprogramming.net/introduction-use-tensorflow-object-detection-api-tutorial/
Currently the program is not reading information from the minimap and just tries to push through the mid lane until it sees enemies and tries to kite/kill them and to not die. For more info on the decision making I recommend reading a bit in the report that is also included in the repo. About reading the minimap I have seen an interesting post on reddit that used a similar approach as me and trained a detector to detect champion portraits on the minimap and track champions:
https://www.reddit.com/r/leagueoflegends/comments/7sg1yn/imagine_if_you_had_a_robot_that_could/
I thought of implementing something like this at some point.
Hope this helps!
Best Regards
Thanks a lot for the info. I guess I will not be able to avoid the painful process of collecting images for training if I need to detect more objects in the game. How many labeled images did you end up using for the training if you don't mind me asking?
No problem! I think I had about 500 images in the end with some containing multiple different objects, some containing only one. But as I said I retrained the ssd_mobilenet_v1_coco model which already had quite good results if the minions/champions were not occluded or something. In another project (not related to League) where I had a powerful computer with a GTX 1070 I used this one: faster_rcnn_resnet101_coco which performed really well with only 200 images for 5 different objects.
My recommendation for getting the images from the game is to use the Windows Game Bar to record a custom game where you can control the scenarios you want to learn the objects from and then split the video in a lot of single frames. Then remove the ones you think are not containing valuable new information to teach to the detector. I wrote a python script to export frames from a video: https://github.com/Oleffa/Python_Videoprocessing_Toolbox/blob/master/pyFrameexporter.py
Then learn the hotkeys for labelImg, put on a good TV show and start labeling.
That's awesome. I think I have enough information to kick start and fool around. Thanks a lot for your help mate!