Join our community Discord: AI Stack Devs
![Screen Shot 2023-08-14 at 10 01 00 AM](https://private-user-images.githubusercontent.com/3489963/260520547-a4c91f17-23ed-47ec-8c4e-9f9a8505057d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIyOTk2NTAsIm5iZiI6MTcyMjI5OTM1MCwicGF0aCI6Ii8zNDg5OTYzLzI2MDUyMDU0Ny1hNGM5MWYxNy0yM2VkLTQ3ZWMtOGM0ZS05ZjlhODUwNTA1N2QucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDczMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA3MzBUMDAyOTEwWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MGNjY2E0YTYxYmFmNmUzMTg1MWVlNzMwN2RlY2MyNmZlZDM4MzhmYzIzOGU2MDNlODZmZjAxZjFjYzhkNTdlNiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.yr-OHBmECbkLjvvl6gGbI2VRrcFrWcViMxWAlmr4pjg)
AI Town is a virtual town where AI characters live, chat and socialize.
This project is a deployable starter kit for easily building and customizing your own version of AI town. Inspired by the research paper Generative Agents: Interactive Simulacra of Human Behavior.
The primary goal of this project, beyond just being a lot of fun to work on, is to provide a platform with a strong foundation that is meant to be extended. The back-end engine natively supports shared global state, transactions, and a journal of all events so should be suitable for everything from a simple project to play around with to a scalable, multi-player game. A secondary goal is to make a JS/TS framework available as most simulators in this space (including the original paper above) are written in Python.
- Game engine & Database: Convex
- VectorDB: Pinecone
- Auth: Clerk
- Text model: OpenAI
- Deployment: Fly
- Pixel Art Generation: Replicate, Fal.ai
- Background Music Generation: Replicate using MusicGen
git clone https://github.com/a16z-infra/ai-town.git
cd ai-town
npm install
npm run dev
npm run dev
will fail asking for environment variables.
Enter them in the environment variables on your Convex dashboard to proceed.
You can get there via npx convex dashboard
or https://dashboard.convex.dev
See below on how to get the various environment variables.
a. Set up Clerk
- Go to https://dashboard.clerk.com/ and click on "Add Application"
- Name your application and select the sign-in providers you would like to offer users
- Create Application
- Add
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY
andCLERK_SECRET_KEY
to.env.local
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=pk_***
CLERK_SECRET_KEY=sk_***
- Go to JWT Templates and create a new Convex Template.
- Copy the JWKS endpoint URL for use below.
b. OpenAI API key
Visit https://platform.openai.com/account/api-keys to get your OpenAI API key if you're using OpenAI for your language model.
c. Pinecone API keys
- Create a Pinecone index by visiting https://app.pinecone.io/ and click on "Create Index"
- Give it an index name (this will be the environment variable
PINECONE_INDEX_NAME
) - Fill in Dimension as
1536
- Once the index is successfully created, click on "API Keys" on the left side nav and create an API key: copy "Environment" value to
PINECONE_ENVIRONMENT
variable, and "Value" toPINECONE_API_KEY
d. Replicate API key (Optional)
For Daily background music generation, create an account at Replicate and create a token in your Profile's API Token page.
Add the token as REPLICATE_API_TOKEN
in yout .env.local
e. Add secrets to the convex dashboard
npx convex dashboard
Go to "settings" and add the following environment variables. CLERK_ISSUER_URL
should be the URL from the JWKS endpoint.
OPENAI_API_KEY sk-*******
CLERK_ISSUER_URL https://****
PINECONE_API_KEY ********
PINECONE_ENVIRONMENT us****
PINECONE_INDEX_NAME ********
REPLICATE_API_TOKEN **** #optional
To run both the front and and back end:
npm run dev
You can now visit http://localhost:[PORT_NUMBER]
If you'd rather run the frontend in a separate terminal from Convex (which syncs your backend functions as they're saved), you can run these two commands:
npm run dev:frontend
npm run dev:backend
See package.json for details, but dev:backend runs npx convex dev
*Note: The simulation will pause after 5 minutes if the window is idle.
Loading the page will unpause it. If you want to run the world without the
browser, you can comment-out the heartbeat check in convex/engine.ts
To add a new world, seed it, and start it running
Note: you can add --no-push
to run these commands without first syncing
the functions. If you already have npm run dev
running, this will be faster.
If you remove it, it'll push up the latest version of code before running the
command.
npx convex run init:reset
To go one iteration at a time, you can create a world with
npx convex run --no-push init:resetFrozen
# for each iteration
npx convex run --no-push engine:tick '{"worldId":"<your world id>","noSchedule":true}'
To freeze the back end, in case of too much activity
npx convex run --no-push engine:freezeAll
# when ready to rerun (defaults to latest world)
npx convex run --no-push engine:unfreeze
To clear all databases
Many options:
- Go to the dashboard
npx convex dashboard
and clear tables from there. - Adjust the variables in
crons.ts
to automatically clear up space from old journal and memory entries. - Run
npx convex run --no-push testing:debugClearAll
to wipe all the tables. - As a fallback, if things are stuck, you can check out the
origin/reset-town
git branch. Doingnpm run dev
from there will clear your schema, stop your functions, and allow you to delete your tables in the dashboard.
To delete all vectors from the Pinecone index, you can run:
npx convex run --no-push lib/pinecone:deleteAllVectors
NOTE: If you share this index between dev & prod, or between projects, it will wipe them all out. You generally don't need to be deleting vectors from Pinecone, as each query is indexed on the userId, which is unique between worlds and backend instances.
To Snoop on messages
Run the following in a side terminal
npx convex run testing:listMessages --no-push --watch
Or to watch one player's state:
npx convex run testing:latestPlayer --no-push --watch
See more functions in testing.ts
.
-
Register an account on fly.io and then install flyctl
-
If you are using Github Codespaces: You will need to install flyctl and authenticate from your codespaces cli by running
fly auth login
. -
Run
npx convex deploy
to deploy your dev environment to prod environment. Make sure you copy over all secrets to Convex's prod environment -
Run
fly launch
under project root. This will generate afly.toml
that includes all the configurations you will need -
Modify generated
fly.toml
to includeNEXT_PUBLIC_*
during build time for NextJS to access client side.
[build]
[build.args]
NEXT_PUBLIC_CLERK_SIGN_IN_URL="/sign-in"
NEXT_PUBLIC_CLERK_SIGN_UP_URL="/sign-up"
NEXT_PUBLIC_CLERK_AFTER_SIGN_IN_URL="/"
NEXT_PUBLIC_CLERK_AFTER_SIGN_UP_URL="/"
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY="pk_*****"
NEXT_PUBLIC_CONVEX_URL="https://*******.convex.cloud"
- Modify fly.io's generated
Dockerfile
to include new ENV variables right aboveRUN npm run build
ARG NEXT_PUBLIC_CLERK_SIGN_IN_URL
ARG NEXT_PUBLIC_CLERK_SIGN_UP_URL
ARG NEXT_PUBLIC_CLERK_AFTER_SIGN_IN_URL
ARG NEXT_PUBLIC_CLERK_AFTER_SIGN_UP_URL
ARG NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY
ARG NEXT_PUBLIC_CONVEX_URL
# Build application
RUN npm run build
- Run
fly deploy --ha=false
to deploy the app. The --ha flag makes sure fly only spins up one instance, which is included in the free plan. - Run
fly scale memory 512
to scale up the fly vm memory for this app. - Create a new file
.env.prod
locally and fill in all the production-environment secrets. Remember to updateNEXT_PUBLIC_CLERK_PUBLISHABLE_KEY
andCLERK_SECRET_KEY
by copying secrets from Clerk's production instance -cat .env.prod | fly secrets import
to upload secrets. Also remember to updateCONVEX_DEPLOYMENT
andNEXT_PUBLIC_CONVEX_URL
-- both of them should now point to Convex's prod environment.
Before you can run the app, you will need to make sure the convex functions are deployed to its production environment.
- Run
npx convex deploy
to deploy the convex functions to production - Go to convex dashboard, select prod
- Navigate to Functions on the left side nav, click on testing -> debugClearAll, click on run function on the top right
- Then click on init - init function, run this function.
NOTE: every time you change character data, you should re-run npx convex run testing:debugClearAll --no-push
and then npm run dev
to re-upload everything to Convex. This is because character data is sent to Convex on the initial load. However, beware that npx convex run testing:debugClearAll --no-push
WILL wipe all of your data, including your vector store.
- Create your own characters and stories: All characters and stories, as well as their spritesheet references are stored in data.ts. You can start by changing character descriptions.
- Updating spritesheets: in
data.ts
, you will see this code:
{
name: 'f1',
textureUrl: '/assets/32x32folk.png',
spritesheetData: f1SpritesheetData,
speed: 0.1,
},...
You should find a sprite sheet for your character, and define sprite motion / assets in the corresponding file (in the above example, f1SpritesheetData
was defined in f1.ts)
- Update the background (environment):
convex/maps/firstmap.ts
is where the map gets loaded. The easiest way to export a tilemap is by using Tiled -- Tiled exports tilemaps as a CSV and you can convert CSV to a 2d array accepted by firstmap.ts - Change the background music by modifying the prompt in
convex/lib/replicate.ts
- Change how often to generate new music at
convex/crons.ts
by modifying thegenerate new background music
job
- All interactions, background music and rendering on the component in the project are powered by PixiJS.
- Tilesheet:
- https://opengameart.org/content/16x16-game-assets by George Bailey
- https://opengameart.org/content/16x16-rpg-tileset by hilau
- We used https://github.com/pierpo/phaser3-simple-rpg for the original POC of this project. We have since re-wrote the whole app, but appreciated the easy starting point
- Original assets by ansimuz
- The UI is based on original assets by Mounir Tohami