PyWinAssistant is capable to fully play interactive fiction games

[henyckma]

Open for example "Eat Me" a text-based game and watch the results. Only ask the assistant "Play the game" or something like that.

https://ifarchive.org/if-archive/games/competition2017/Eat%20Me/Eat%20Me/

2023 December Demo:

`Improved goal statement: "Start and engage with the interactive fiction game 'Eat Me' in the Google Chrome browser on Windows."

Current status of the system:

• The Google Chrome browser is open with a tab titled "Eat Me."
• The URL indicates that the content is hosted on the archive.org website, within a path related to a competition from 2017 for a game named "Eat Me."
• There is a text prompt displayed with the question "Have you played interactive fiction before?" and a cursor indicating that input is expected from the user.
• There are no other visible tabs or pop-ups that would impede interaction with the game.
• The browser window is maximized, providing full view of the interactive fiction game.

To proceed with playing the interactive fiction game, the user would typically type a response into the text prompt and press enter to interact with the game.`

Visualization of thought output:

{"actions": [{"act": "click_element", "step": "Click on the text prompt where the question 'Have you played interactive fiction before?' is displayed.", "additional_info": "x=459, y=180"}, {"act": "text_entry", "step": "Type 'yes' or 'no' depending on your experience with interactive fiction games."}, {"act": "press_key", "step": "Press 'Enter' to submit the response."}, {"act": "time_sleep", "step": "Wait for the game to process the input and continue with the next part of the story."}, {"act": "press_key", "step": "Press 'Enter' to proceed with the game after reading the resulting text."}]}

Several steps later while playing:

`Improved goal statement: "Navigate and interact within the text adventure game 'Eat Me' in Chrome as if embodying an advanced artificial being with creative problem-solving capabilities."

Current status: A text adventure game titled "Eat Me" is open in Google Chrome, specifically on a web page hosted on archive.org. The player is currently in a "Dungeon (south cell)", with actions such as "eat cake", "examine manacles", "look around", "examine chains", and "eat chains" recently performed in the game. The text indicates that the player's character is bound by "cacao manacles" but has made progress by eating the chains that are now almost broken. The game is awaiting the next command from the player.`

Visualization of thought output:

{"actions": [{"act": "click_element", "step": "Click on the text entry field where commands are inputted for 'Eat Me'.", "additional_info": "x=708, y=994"}, {"act": "text_entry", "step": "Enter 'examine manacles' to gather more information about the cacao manacles binding the character."}, {"act": "press_key", "step": "Press 'Enter' to submit the command in the game."}, {"act": "time_sleep", "step": "Wait for the game to process the command and provide a response."}, {"act": "click_element", "step": "Click on the text entry field again to enter a new command.", "additional_info": "x=708, y=994"}, {"act": "text_entry", "step": "Type in 'eat manacles' to attempt to free the character from the cacao manacles by eating them."}, {"act": "press_key", "step": "Press 'Enter' to submit the 'eat manacles' command."}, {"act": "time_sleep", "step": "Wait for the game to update the scenario after the command is executed."}, {"act": "click_element", "step": "Click once more on the text entry field to continue the interaction.", "additional_info": "x=708, y=994"}, {"act": "text_entry", "step": "Input 'look around' to observe the new state of the dungeon and identify other possible actions."}, {"act": "press_key", "step": "Hit 'Enter' to send the 'look around' command and receive an updated description of the surroundings."}]}

Another implementation I made can also somewhat play minesweeper (depends on the game/app used):