TypeError: unsupported operand type(s) for +: 'NoneType' and 'str' in EmpowerFunctionsCompletionHandler

[nreck]

When interacting with the Empower Functions model, any input that does not trigger a function call results in a TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'. The model works fine when processing function calls like retrieving the weather, but fails for general conversation responses.

Environment:

• Model: empower-dev/llama3-empower-functions-small-gguf
• File: ggml-model-Q4_K_M.gguf
• Platform: MacBook M2 Pro, 32GB RAM
• Python version: 3.11.3

Steps to Reproduce:

• Initialize the model with the given settings.
• Ask a question that triggers a function call (e.g., "How is the weather in Copenhagen?").
• Ask a general question that does not trigger a function call (e.g., "Do they have many animals there?").

Expected Behavior:
The model should handle non-function call responses without raising an error, providing a coherent response.

Actual Behavior:
The model raises a TypeError:

Traceback (most recent call last): File "/path/to/script.py", line 93, in <module> chat() File "/path/to/script.py", line 67, in chat result = llm.create_chat_completion( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/path/to/llama_cpp/llama.py", line 1731, in create_chat_completion return handler( ^^^^^^^^ File "/path/to/empower_functions/chat_handler.py", line 144, in __call__ generated["choices"][0]["text"] = thinking + content[3:] ~~~~~~~~~^~~~~~~~~~~~~ TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

Code snippet:

import requests
from empower_functions import EmpowerFunctionsCompletionHandler
from llama_cpp.llama_tokenizer import LlamaHFTokenizer
from llama_cpp import Llama

# Function to get the current weather using OpenWeather API
def get_current_weather(location):
    api_key = "xxx"  # Replace with your actual API key
    base_url = "http://api.openweathermap.org/data/2.5/weather"
    params = {
        "q": location,
        "appid": api_key,
        "units": "metric"
    }
    response = requests.get(base_url, params=params)
    data = response.json()

    if response.status_code == 200:
        temp = data["main"]["temp"]
        return f"The current temperature in {location} is {temp}°C."
    else:
        return f"Could not get the weather for {location}. Please check the location name."

# Initialize the LLM
llm = Llama.from_pretrained(
    repo_id="empower-dev/llama3-empower-functions-small-gguf",
    filename="ggml-model-Q4_K_M.gguf",
    chat_format="llama-3",
    chat_handler=EmpowerFunctionsCompletionHandler(),
    tokenizer=LlamaHFTokenizer.from_pretrained("empower-dev/llama3-empower-functions-small-gguf"),
    n_gpu_layers=0
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city e.g. Copenhagen"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

def chat():
    messages = [
        {"role": "system", "content": "You are a helpful assistant capable of chatting and retrieving weather informaition when asked. Use the tool <get_current_weather> to get the current weather of a location whenever appropriate."}
    ]

    while True:
        user_input = input("You: ")
        if user_input.lower() in ["exit", "quit"]:
            break

        messages.append({"role": "user", "content": user_input})

        result = llm.create_chat_completion(
            messages=messages,
            tools=tools,
            tool_choice="auto",
            max_tokens=128
        )

        # Parse the result to extract tool calls
        response_message = result["choices"][0]["message"]
        assistant_content = response_message.get("content", None)
        tool_calls = response_message.get("tool_calls", [])

        # Handle the tool calls
        for call in tool_calls:
            if call["function"]["name"] == "get_current_weather":
                location = json.loads(call["function"]["arguments"])["location"]
                weather_info = get_current_weather(location)
                print(f"Assistant: {weather_info}")
                messages.append({"role": "assistant", "content": weather_info})
                break  # Assuming only one tool call per message

        if assistant_content:
            print(f"Assistant: {assistant_content}")
            messages.append({"role": "assistant", "content": assistant_content})

if __name__ == "__main__":
    chat()

[dyang415]

Hi @nreck thanks for reporting! It's a new bug that we introduced with https://github.com/empower-ai/empower-functions/blob/main/docs/inference/built-in-cot.md. We have just pushed the fix and it's available in the latest pip package: https://github.com/empower-ai/empower-functions/blob/main/docs/inference/built-in-cot.md.

Apologies for the inconvenience and please let us know if you have any other issues or questions. Thanks!