This project showcases an experimental mob session on 21.03.24 using the open-source tool Pythagoras (formerly known as gptpilot), aimed at exploring the capabilities of AI coding agents in a collaborative environment. Our goal was to assess the practicality and efficiency of such technology in real-world coding scenarios, particularly in developing a simple game.
The motivation for this experiment was to understand the potential of AI-assisted coding, specifically with the GPT-4 API, in accelerating and enhancing the development process. By setting a challenge to create a game based on a vague requirement - "a snake that eats" - we embarked on a journey to discover how AI could handle specification refinement, question posing for clarification, and iterative coding with minimal human intervention.
- Requirement Gathering: Using Pythagoras, we provided a broad description of our desired game. The tool impressively parsed through this vague input to ask meaningful questions, helping us refine our requirements into a clear specification document.
- Iteration One: The first output was a playable snake game controllable via cursor keys. This initial success, after one troubleshooting turn, showcased the AI's capability to generate functional code based on refined specifications.
- Iteration Two: Required two manual fixes to address issues not caught in the first iteration. This highlighted a potential limitation in developing dynamic applications like games, where manual iterations and testing are crucial in polishing the final product.
- Costs & Efficiency: The total cost for this experiment, utilizing the GPT-4 API, was around $9.
- Application Suitability: Pythagoras seems to have a stronger affinity for developing form-based applications over dynamic games, likely due to the structured nature of such apps.
- Regression & Testing: Like in traditional coding, we observed regression issues after several iterations. This underscores the importance of a Test-Driven Development (TDD) cycle, which could significantly enhance the AI's efficiency in future iterations by preventing regression.
- Potential & Limitations: Despite some limitations, the experiment underscored the impressive capabilities of AI coding agents. It offers a glimpse into how future generations of LLMs could revolutionize software development, particularly for small-scale projects.
- Incorporate TDD: Adopting a Test-Driven Development approach from the start could improve code quality and reduce regression issues.
- Expand Application Types: Experiment with different types of applications to better understand the AI's versatility and limitations.
- Further Iterations: Continue refining the game with additional features, such as levels, speed variations, and leaderboards, to explore the AI's capability in handling more complex requirements.
Our exploration with Pythagoras provided valuable insights into the future of AI-assisted coding. While it showcased the potential for creating basic applications, it also highlighted the importance of human oversight, especially in the development of dynamic applications. As AI technology evolves, its integration into the software development lifecycle promises to open new horizons for efficiency and creativity.
For companies and developers, now is the time to explore and adapt to AI coding agents. Gaining practical experience and familiarizing oneself with the principles of AI-assisted coding can provide a significant competitive advantage. As we transition to more advanced LLMs, those with hands-on experience will find themselves at the forefront of a revolution in software development. The leap from GPT-4 to GPT-5 and beyond is expected to bring about substantial improvements in AI's ability to understand complex requirements, manage larger projects, and significantly reduce development time and costs.