This tool allows you to chat with your agent locally in order to create a faster feedback loop for developers developing an extension. Debug mode is enabled by default so that you can see clearer information around what exactly is getting parsed successfully.
The different SSE events in the agent protocol that the CLI gives debug output for are:
Note: This tool does not handle the payload verification process. To use this tool to validate your events, please temporarily disable payload verification for local testing and re-enable when completed.
- Authenticate with GitHub CLI OAuth app
gh auth login --web -h github.com
- Install / upgrade extension
gh extension install github.com/copilot-extensions/gh-debug-cli
- See more info about the cli tool
gh debug-cli chat -h
- Run the following command
gh debug-cli chat -hto see the different flags that it takes in.
> gh debug-cli chat -h
This cli tool allows you to debug your agent by chatting with it locally.
Usage:
[flags]
Flags:
-h, --help help for this command
--log-level DEBUG Log level to help debug events. Supported types are DEBUG, `TRACE`, `NONE`. `DEBUG` returns general logs. `TRACE` prints the raw http response. (default "DEBUG")
--token string GitHub token for chat authentication (optional)
--url string url to chat with your agent (default "http://localhost:8080")
--username string username to display in chat (default "sparklyunicorn")
The token noted in the flag above is used to authenticate against the provided LLM. If you are using a different service, then this token is not needed. Generate the user-to-server token by creating a GitHub Applicatiion and then following the using the device flow to generate a user access token to generate the token.
- You can alternatively set these flags as environment variables (in all caps) so you don't need to pass them in every time. The only "required" one to get this up and running is the url for your agent
export URL="http://localhost:8080/agent/blackbeard"
- When you run the CLI, you will see any flags that were previously set in your environment variables as the output.
> gh debug-cli chat
Setting url to http://localhost:8080/agents/blackbeard
Start typing to chat with your assistant...
sparklyunicorn:
- Type something to simulate chatting with your assistant.
> gh debug-cli chat
Setting url to http://localhost:8080/agents/blackbeard
Start typing to chat with your assistant...
sparklyunicorn: hello
assistant: Ahoy, @monalisa! A jolly good day to ye, me heartie. How can ol' Blackbeard be of service to ye today?
Huzzah! You successfully received a message!
╔═══════════╤════════════════════════════════════════════════════════════════╗
║ Role │ Content ║
╟━━━━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢
║ assistant │ [condensed] Ahoy, @monalisa! A jolly good day to ye, me hearti ║
╟━━━━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢
║ Parsed message data ║
╚═══════════╧════════════════════════════════════════════════════════════════╝
sparklyunicorn:
- To debug your SSE events, you can set up a key word that your assistant uses to send you a specific type of event. My blackbeard agent allows me to send a keyword "confirmation", and here I can see the debug output on what is parsed from the SSE event
> sparklyunicorn: confirmation
assistant: Arrr, @monalisa! I be ready and waitin' for yer confirmation. Be ye ready to set sail on this treacherous journey and receive a custom limerick 'bout petals? Aye or nay, let me know yer decision, and I'll be at yer service.
Huzzah! You successfully received a message!
╔═══════════╤════════════════════════════════════════════════════════════════╗
║ Role │ Content ║
╟━━━━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢
║ assistant │ [condensed] Arrr, @monalisa! I be ready and waitin' for yer co ║
╟━━━━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢
║ Parsed message data ║
╚═══════════╧════════════════════════════════════════════════════════════════╝
Huzzah! You successfully received a confirmation!
╔══════════════╤═════════════════════════════════════════════════════╗
║ Key │ Value ║
╟━━━━━━━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢
║ type │ action ║
║ title │ Be ye sure ye want a custom limerick 'bout petals ? ║
║ message │ Arrr, this here action be irreversible, matey! ║
║ confirmation │ map[id:123] ║
╟━━━━━━━━━━━━━━┼━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╢
║ Parsed confirmation data ║
╚══════════════╧═════════════════════════════════════════════════════╝
Be ye sure ye want a custom limerick 'bout petals ?
Arrr, this here action be irreversible, matey!
Reply: [y/N]
- If I got a bad confirmation, it would look something like this
> sparklyunicorn: bad confirmation
Alas...The following is not a valid confirmation:
["conf"]
assistant: Avast, @monalisa! Me apologies if I didn't quite understand yer request. Pray tell, could ye please clarify what be wrong with the confirmation? I be here to assist ye, me matey!
- And if debug mode was set to false, then I would only see the confirmation prompt itself.
gh debug-cli chat --log-level none
Setting url to http://localhost:8080/agents/blackbeard
Start typing to chat with your assistant...
sparklyunicorn: confirmation
assistant: Ahoy, @monalisa! Ye be seekin' confirmation, me hearty. Are ye sure ye want a custom limerick 'bout petals? This here action be irreversible, matey!
Be ye sure ye want a custom limerick 'bout petals ?
Arrr, this here action be irreversible, matey!
Reply: [y/N]
- Currently, the supported event types for debug mode are references, errors, and confirmations! Have fun chatting with your assistant!
-
To quickly parse an agent response by running command
gh debug-cli stream --file test.txt -
This tool will take llm streaming response and parse it to make it more readable
- In this example, if a file test.txt holds the following streamed response. Then will return the response "A closure in JavaScript is a function that retains access... " This will make repsonse more readable.
example of .txt file
data: {"choices":[{"delta":{"content":"A closure in JavaScript "}}],"created":1727120830,"id":"chatcmpl-AAjJW0Nz9E2Gu1P6YQMFqqmn10mdR","model":"gpt-4o-2024-05-13","system_fingerprint":"fp_80a1bad4c7"}
data: {"choices":[{"delta":{"content":"is a function that retains access "}}],"created":1727120831,"id":"chatcmpl-AAjJW0Nz9E2Gu1P6YQMFqqmn10mdR","model":"gpt-4o-2024-05-13","system_fingerprint":"fp_80a1bad4c7"}
data: {"choices":[{"delta":{"content":"to its lexical scope, even "}}],"created":1727120832,"id":"chatcmpl-AAjJW0Nz9E2Gu1P6YQMFqqmn10mdR","model":"gpt-4o-2024-05-13","system_fingerprint":"fp_80a1bad4c7"}
data: {"choices":[{"delta":{"content":"when the function is executed "}}],"created":1727120833,"id":"chatcmpl-AAjJW0Nz9E2Gu1P6YQMFqqmn10mdR","model":"gpt-4o-2024-05-13","system_fingerprint":"fp_80a1bad4c7"}
data: {"choices":[{"delta":{"content":"outside that scope. "}}],"created":1727120834,"id":"chatcmpl-AAjJW0Nz9E2Gu1P6YQMFqqmn10mdR","model":"gpt-4o-2024-05-13","system_fingerprint":"fp_80a1bad4c7"}