How to count WebSearchTool() Calls when using Agents as Tools
Closed this issue · 5 comments
When using agents as tools, is there a way to determine how many times WebSearchTool() has been called?
As per the docs:
The new_items property contains the new items generated during the run.
...
ToolCallItem indicates that the LLM invoked a tool.
So, you can maybe iterate over the new_items and count the occurrences of a ToolCallItem with type ResponseFunctionWebSearch.
Not sure if there is a better way.
Edit:
Okay, did some more digging into the code; In test_agent_hooks.py, there is AgentHooksForTests class defined. So, you may follow the same approach and create a custom agent hooks class for your agent, which counts the occurrences of WebSearchTool().
You can learn more about AgentHooks in the Lifecycle docs.
So, you can maybe iterate over the new_items and count the occurrences of a ToolCallItem with type
The new_items do not contain tool calls made by agents as tools, if I am not missing anything.
Regarding custom agent hooks, I’ve tried that already; however, I don’t see any WebSearch tool calls being made by the agent, even though the output suggests it did access the internet.
I’m attaching a screenshot of the web-search related trace I see in LangFuse.
@seratch, could you please look into the above? Specifically, is web-search not appearing as a function call the expected behaviour?
When you run a built-in tool like web search, those tools are called as part of your Responses API call. Thus, it won't be shown like a function tool call, which can come right after a Response API call. So, this is an expected behavior. If you want to know how many times those built-in tools are used on the Responses API side, you can check the result.new_items like suggested above (thanks for sharing the info!).
Here is a simple example code I took from examples dir and customized a bit:
import asyncio
from agents import Agent, Runner, WebSearchTool, trace
async def main():
agent = Agent(
name="Web searcher",
instructions="You are a helpful agent.",
tools=[WebSearchTool(user_location={"type": "approximate", "city": "New York"})],
)
with trace("Web search example"):
result = await Runner.run(
agent,
"search the web for 'local sports news' and give me 1 interesting update in a sentence.",
)
for item in result.new_items:
print(item)
print(result.final_output)
if __name__ == "__main__":
asyncio.run(main())This code prints:
ToolCallItem(agent=Agent(name='Web searcher', handoff_description=None, tools=[WebSearchTool(user_location={'type': 'approximate', 'city': 'New York'}, filters=None, search_context_size='medium')], mcp_servers=[], mcp_config={}, instructions='You are a helpful agent.', prompt=None, handoffs=[], model=None, model_settings=ModelSettings(temperature=None, top_p=None, frequency_penalty=None, presence_penalty=None, tool_choice=None, parallel_tool_calls=None, truncation=None, max_tokens=None, reasoning=None, verbosity=None, metadata=None, store=None, include_usage=None, response_include=None, top_logprobs=None, extra_query=None, extra_body=None, extra_headers=None, extra_args=None), input_guardrails=[], output_guardrails=[], output_type=None, hooks=None, tool_use_behavior='run_llm_again', reset_tool_choice=True), raw_item=ResponseFunctionWebSearch(id='ws_68c3cadb7a2881a381f295579b2b8065084d28d80737be82', action=ActionSearch(query='local sports news', type='search', sources=None), status='completed', type='web_search_call'), type='tool_call_item')
MessageOutputItem(agent=Agent(name='Web searcher', handoff_description=None, tools=[WebSearchTool(user_location={'type': 'approximate', 'city': 'New York'}, filters=None, search_context_size='medium')], mcp_servers=[], mcp_config={}, instructions='You are a helpful agent.', prompt=None, handoffs=[], model=None, model_settings=ModelSettings(temperature=None, top_p=None, frequency_penalty=None, presence_penalty=None, tool_choice=None, parallel_tool_calls=None, truncation=None, max_tokens=None, reasoning=None, verbosity=None, metadata=None, store=None, include_usage=None, response_include=None, top_logprobs=None, extra_query=None, extra_body=None, extra_headers=None, extra_args=None), input_guardrails=[], output_guardrails=[], output_type=None, hooks=None, tool_use_behavior='run_llm_again', reset_tool_choice=True), raw_item=ResponseOutputMessage(id='msg_68c3cadd20c081a39bb9324775e939b4084d28d80737be82', content=[ResponseOutputText(annotations=[AnnotationURLCitation(end_index=402, start_index=275, title='New York Liberty close regular season with win over Chicago Sky, 91-86', type='url_citation', url='https://www.netsdaily.com/nyliberty/98853/liberty-vs-sky-91-86-breanna-stewart-angel-reese?utm_source=openai')], text='Here’s one notable update from New York local sports news, in one sentence:\n\nThe New York Liberty wrapped up their 2025 WNBA regular season with a 91–86 victory over the Chicago Sky—fueling playoff momentum with standout performances from Breanna Stewart and Emma Meesseman. ([netsdaily.com](https://www.netsdaily.com/nyliberty/98853/liberty-vs-sky-91-86-breanna-stewart-angel-reese?utm_source=openai))', type='output_text', logprobs=[])], role='assistant', status='completed', type='message'), type='message_output_item')
Here’s one notable update from New York local sports news, in one sentence:
The New York Liberty wrapped up their 2025 WNBA regular season with a 91–86 victory over the Chicago Sky—fueling playoff momentum with standout performances from Breanna Stewart and Emma Meesseman. ([netsdaily.com](https://www.netsdaily.com/nyliberty/98853/liberty-vs-sky-91-86-breanna-stewart-angel-reese?utm_source=openai))Hope this helps. If everything is fine so far, could you close this issue? Whenever you get a specific question, you can create a new issue for it.
@seratch, thank you, my question, however, was specifically about counting web search calls when using agents as tools, because results.new_items doesn’t include tool calls made by agents-as-tools.
However, by running an agent as a function tool (as mentioned here: #551), I was able to access the web search calls made by the tool agent.