Spaces:
Paused
Paused
File size: 3,619 Bytes
1f77df0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 |
# Tool Calling Guide
Your ExCom AI deployment supports tool calling! However, there's a quirk with vLLM that requires a workaround.
## The Issue
vLLM requires `--enable-auto-tool-choice` and `--tool-call-parser` flags to accept the `tool_choice: "auto"` parameter. Since Qwen 2.5 has native tool calling built into the model, we don't use these flags.
**Result**: LangChain's default agent framework sends `tool_choice: "auto"` which vLLM rejects with a 400 error.
## Solution: Use OpenAI SDK Directly
I've created `simple_tool_chat.py` which uses the OpenAI SDK directly and doesn't send `tool_choice`.
### Installation
```bash
pip install openai
```
### Usage
```bash
python simple_tool_chat.py
```
### Example Session
```
You: What is 15 * 23 + 100?
🔧 Calling tool: calculator({'expression': '15 * 23 + 100'})
Assistant: The result is 445.
You: What's the weather in Paris and what time is it?
🔧 Calling tool: get_weather({'city': 'Paris'})
🔧 Calling tool: get_current_time({})
Assistant: The weather in Paris is 18°C and sunny. The current time is 2025-10-09 18:30:45.
```
## How It Works
1. **No tool_choice parameter** - We don't send `tool_choice` at all
2. **Qwen decides naturally** - The model's training handles when to use tools
3. **OpenAI SDK** - Direct HTTP calls to your vLLM endpoint
4. **Multi-turn** - Maintains conversation history for context
## Using with Your Own Code
```python
from openai import OpenAI
client = OpenAI(
base_url="https://plarnholt-excom-ai-demo.hf.space/v1",
api_key="not-needed"
)
# Define your tools
tools = [{
"type": "function",
"function": {
"name": "my_tool",
"description": "What it does",
"parameters": {
"type": "object",
"properties": {
"param": {"type": "string"}
}
}
}
}]
# Call without tool_choice parameter
response = client.chat.completions.create(
model="excom-ai",
messages=[{"role": "user", "content": "Use my tool"}],
tools=tools,
temperature=0.4
# NOTE: No tool_choice parameter!
)
# Check for tool calls
if response.choices[0].message.tool_calls:
for tool_call in response.choices[0].message.tool_calls:
print(f"Tool: {tool_call.function.name}")
print(f"Args: {tool_call.function.arguments}")
```
## Adding Custom Tools
Edit `simple_tool_chat.py`:
```python
# 1. Add tool definition to 'tools' list
{
"type": "function",
"function": {
"name": "my_custom_tool",
"description": "What it does",
"parameters": {
"type": "object",
"properties": {
"param": {"type": "string", "description": "Param description"}
},
"required": ["param"]
}
}
}
# 2. Add implementation
def my_custom_tool(param: str) -> str:
# Your logic here
return "result"
# 3. Add to dispatcher
def execute_tool(tool_name: str, arguments: dict) -> str:
# ... existing tools ...
elif tool_name == "my_custom_tool":
return my_custom_tool(arguments["param"])
```
## Troubleshooting
**Error: "auto" tool choice requires --enable-auto-tool-choice**
- You're using LangChain's agent framework
- Solution: Use `simple_tool_chat.py` instead
**Tool calls not working**
- Make sure your Space is running: https://huggingface.co/spaces/plarnholt/excom-ai-demo
- Check that you're not sending `tool_choice` parameter
- Verify tools are properly formatted (see OpenAI docs)
**500 Internal Server Error**
- Space might be sleeping - make a request to wake it up
- Check Space logs for errors
|