Salesforce

/

Llama-xLAM-2-8b-fc-r

like 63

Follow

Salesforce AI Research 2.36k

I try to get streamed response but I get this error:

File "/secondary/thies/Llama-xLAM-2-8b-fc-r/xlam_tool_call_parser.py", line 146, in extract_tool_calls_streaming
  function_name = current_tool_call.get("name")

Client:

import openai
import json

# Configure the client to use your local vLLM endpoint
client = openai.OpenAI(
    base_url="http://localhost:8000/v1",  # Default vLLM server URL
    api_key="empty"  # Can be any string
)

# Define a tool/function
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "The unit of temperature to return"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Create a chat completion
response = client.chat.completions.create(
    model="/secondary/thies/Llama-xLAM-2-8b-fc-r/",  # Model name doesn't matter, vLLM uses the served model
    messages=[
        {"role": "system", "content": "You are a helpful assistant that can use tools."},
        {"role": "user", "content": "What's the weather like in San Francisco?"}
    ],
    tools=tools,
    tool_choice="auto",
    stream=True
)

# Print the response
print("Assistant's response:")
print(list(response))

Is there a way to get this working?