In a shocking turn of events this approach actually works!
It really, really, works.
To qualify the above statement, after two plays of 1-1 as training data, and banning nofun from pressing no buttons at all, it managed to kill the first goomba (in a way that I hadn't done in the training sessions) and jump over the first pipe.
- If the screenshot isn't changing try a new input
- Use TAS recordings as training data
- Use pipes and a server-client approach to avoid loading the bot each time (for speed)
- Try a different game
- Add instructions, in case I'm not the only one in the world interested in this
- I have no idea how to (properly) use scikit-learn, maybe some outside help would be good here
Its original purpose of this program was to see how far a minimal almost-brute-force approach could get in Super Mario Bros. for the NES in a reasonable amount of time (see the 0.1 release). The answer is level 4-4.
The scope has since been expanded based on feedback and the author's boundless ambition.
In its current state the program is brutally general. It doesn't rely on any particular game, or even particular system (any emulator supporting Lua should do). In essence it's a lua-bash-python glue-ball which records screenshots and input from an emulator run, simplifies them and uses them to machine-learn to play the game, producing a function which guesses suitable inputs given a screenshot. It is then able to feed these guesses back into the emulator and so on, in effect playing the game as it learned to do.
The project was inspired by tom7's playfun and learnfun. This approach differs in that playfun plays like one might imagine a child plays; watching, learning and trying, while nofun plays as an actual child plays i.e. holding B and right and tapping A frantically.
The current goal is to get nofun to actually play them games by machine learning from good input (generated by hand, the old brute-force version and maybe even TASes), and hopefully get the machine-learning into such a state that it plays the game better than terribly, finishing 1-1 would be nice.
The advantage of the current approach over the old one is that nofun should be able to play without savestates and memory access, all things going well it may even manage to play in real-time on a console, but that is rather optimistic.