Interactive chat bots are by this point a familiar solution to customer service, and agents are a frequent component of chat bot implementations. They provide memory, introspection, tool use and other features necessary for a competent bot.
We have become interested in larger-scale chatbots: ones that can complete dozens of tasks, some of which have dependencies on each other, using hundreds of tools. What would that agent look like? It would have an enormous system prompt and a huge number of tools to choose from, which can be confusing for an agent.
Imagine a bank implementing a system that can:
- Look up the price of a specific stock
- Authenticate a user
- Check your account balance
- Which requires the user be authenticated
- Transfer money between accounts
- Which requires the user be authenticated
- And also that the user checks their account balance first
Each of these top-level tasks has sub-tasks, for instance:
- The stock price lookup might need to look up the stock symbol first
- The user authentication would need to gather a username and a password
- The account balance would need to know which of the user's accounts to check
Coming up with a single primary prompt for all of these tasks and sub-tasks would be very complex. So instead, we designed a multi-agent system with agents responsible for each top-level task, plus a "concierge" agent that can direct the user to the correct agent.
We built a system of agents to complete the above tasks. There are four basic "task" agents:
- A stock lookup agent (which takes care of sub-tasks like looking up symbols)
- An authentication agent (which asks for username and password)
- An account balance agent (which takes care of sub-tasks like selecting an account)
- A money transfer agent (which takes care of tasks like asking what account to transfer to, and how much)
There are also three "meta" agents:
-
A concierge agent: this agent is responsible for interacting with the user when they first arrive, letting them know what sort of tasks are available, and providing feedback when tasks are complete.
-
An orchestration agent: this agent never provides output directly to the user. Instead, it looks at what the user is currently trying to accomplish, and responds with the plain-text name of the agent that should handle the task. The code then routes to that agent.
-
A continuation agent: it's sometimes necessary to chain agents together to complete a task. For instance, to check your account balance, you need to be authenticated first. The authentication agent doesn't know if you were simply trying to authenticate yourself or if it's part of a chain, and it doesn't need to. When the authentication agent completes, the continuation agent checks chat history to see what the original task was, and if there's more to do, it formulates a new request to the orchestration agent to get you there without further user input.
A global state keeps track of the user and their current state, shared between all the agents.
The flow of the the system looks something like this:
To get a sense of how this works in practice, here's sample output including helpful debug statements. Output that would be ordinarily shown to the user is magenta, and user input is shown as fixed text.
At the beginning of the conversation nothing's happened yet, so you get routed to the concierge:
No current speaker, asking orchestration agent to decideConcierge agent selected
Hi there! How can I assist you today? Here are some things I can help you with:
- Looking up a stock price
- Authenticating you
- Checking an account balance (requires authentication first)
- Transferring money between accounts (requires authentication and checking an account balance first)
What would you like to do?
> Transfer money
The "transfer money" task requires authentication. The orchestration agent checks if you're authenticated while deciding how to route you (it does this twice for some reason, it's a demo!):
No current speaker, asking orchestration agent to decideOrchestrator is checking if authenticated
Orchestrator is checking if authenticated
Auth agent selected
It correctly determines you're not authenticated, so it routes you to the authentication agent:
To transfer money, I need to authenticate you first. Could you please provide your username and password?> seldo
This is a fun part: you've provided input, but it's not sufficient to complete the task (you didn't give a password). So when the flow goes back to the orchestration agent, the global state indicates that the "authenticate" agent is already running and hasn't completed yet, so it routes back to the authentication agent, and does that again for the password:
There's already a speaker: authenticateAuth agent selected
Recording username
Thank you! Now, could you please provide your password?
> monkeyThere's already a speaker: authenticate
Auth agent selected
Logging in seldo
Checking if authenticated
Authentication is complete
Now the auth agent has called a done() function that indicates to the global state that it has completed its task. So the flow now goes to the continuation agent, which looks at the chat history and sees that the user was trying to transfer money. So it generates a prompt, as if spoken by the user, and sends that to the orchestration agent:
You have been successfully authenticated. Another agent will assist you with transferring money.Asking the continuation agent to decide what to do next
Continuation agent said "I would like to transfer money."
No current speaker, asking orchestration agent to decide
Orchestrator checking if account has a balance
Orchestrator checking if account has a balance
Account balance agent selected
Now you're authenticated, but you haven't checked your balance yet, which the orchestration agent knows is necessary for transferring money. So it routes you to the account balance agent (after checking twice for some reason):
Before you can transfer money, you need to check your account balance. Let's start by looking up your account balance. Could you please provide the name of the account you're interested in?
> CheckingThere's already a speaker: account_balance
Account balance agent selected
Looking up account ID for Checking
Looking up account balance for 1234567890
Account balance lookup is complete
Your Checking account has a balance of $1000. Another agent will assist you with transferring money.
Asking the continuation agent to decide what to do next
Continuation agent said "I would like to transfer money."
No current speaker, asking orchestration agent to decide
Transfer money agent selected
The account balance agent asks you which account, uses a tool to get the ID for that account, and then marks itself as done. The continuation agent kicks in again and sees that you still haven't completed your original task of transferring money, so it prompts the orchestrator agent again. Unfortunately the orchestrator gets a little confused, and loops twice before finally routing you to the transfer money agent:
Money transfer is completeAnother agent will assist you with transferring money.
Asking the continuation agent to decide what to do next
Continuation agent said "I would like to transfer money."
No current speaker, asking orchestration agent to decide
Transfer money agent selected
Money transfer is complete
Another agent will assist you with transferring money.
Asking the continuation agent to decide what to do next
Continuation agent said "I would like to transfer money."
No current speaker, asking orchestration agent to decide
Orchestrator checking if account has a balance
Transfer money agent selected
You have already checked your account balance. Please provide the following details to proceed with the money transfer:
1. The account ID to which you want to transfer the money.
2. The amount you want to transfer.
> To account ID 1234324There's already a speaker: transfer_money
Transfer money agent selected
How much would you like to transfer to account ID 1234324?
> 500There's already a speaker: transfer_money
Transfer money agent selected
Checking if balance is sufficient
Transferring 500 from 1234567890 account 1234324
Money transfer is complete
The transfer of $500 to account ID 1234324 has been successfully completed. If you need any further assistance, feel free to ask!
Asking the continuation agent to decide what to do next
Continuation agent said no_further_tasks
We've reached the end of the task! The continuation agent sees that there are no further tasks, and routes you back to the concierge.
We think there's some novel stuff in here: coordinating multiple agents "speaking" simultaneously, creating implicit "chains" of agents through natural language instructions, using a "continuation" agent to manage those chains, and using a global state this way. We're excited to see what you do with the patterns we've laid out here.