ibm-granite

/

granite-3.0-8b-instruct

like 203

Follow

IBM Granite 3.38k

Additional Comment:

Hello,
I wanted to clarify that while the issue was submitted with an English prompt for your convenience, the actual tests were conducted using a Korean prompt. The English prompt in the issue description is a direct translation of the Korean prompt to make it easier for your team to review.

Here is the original Korean prompt I used during testing:

<|start_of_role|>available_tools<|end_of_role|>
[
    {
        "name": "calculator.add",
        "description": "두 숫자의 합을 계산합니다.",
        "parameters": {
            "type": "object",
            "properties": {
                "num1": {
                    "type": "integer",
                    "description": "첫 번째 숫자"
                },
                "num2": {
                    "type": "integer",
                    "description": "두 번째 숫자"
                }
            },
            "required": [
                "num1",
                "num2"
            ]
        }
    }
]
<|end_of_text|>
<|start_of_role|>system<|end_of_role|>AI는 제공된 도구만 사용하여 질문에 답변합니다.<|end_of_text|>
<|start_of_role|>user<|end_of_role|>123 + 456은 얼마인가요?<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>

The results I received using this Korean prompt are consistent with the issue described. Thank you for your attention, and please let me know if additional details or clarification are needed!

Additional Comment:

Hello,
I would like to provide additional details regarding the issue. I conducted tests using both English and Korean prompts based on the instructions from the IBM Granite Function Calling tutorial (link).

Here are the results:

English Prompt Test

Prompt:

<|start_of_role|>available_tools<|end_of_role|>
[
    {
        "name": "get_stock_price",
        "description": "Retrieve the current price of a given stock ticker.",
        "parameters": {
            "type": "object",
            "properties": {
                "ticker": {
                    "type": "string",
                    "description": "The stock ticker symbol (e.g., AAPL for Apple Inc.)."
                }
            },
            "required": ["ticker"]
        }
    }
]
<|end_of_text|>
<|start_of_role|>system<|end_of_role|>The AI must only use the provided tools to respond.<|end_of_text|>
<|start_of_role|>user<|end_of_role|>What is the current price of IBM stock?<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>

Response:

<|start_of_role|>assistant<|end_of_role|><|tool_call|>{"name": "get_stock_price", "arguments": {"ticker": "IBM"}}<|end_of_text|>

Korean Prompt Test

Prompt:

<|start_of_role|>available_tools<|end_of_role|>
[
    {
        "name": "get_stock_price",
        "description": "주어진 주식 심볼의 현재 가격을 조회합니다.",
        "parameters": {
            "type": "object",
            "properties": {
                "ticker": {
                    "type": "string",
                    "description": "주식 심볼 (예: AAPL은 Apple Inc.)"
                }
            },
            "required": ["ticker"]
        }
    }
]
<|end_of_text|>
<|start_of_role|>system<|end_of_role|>AI는 제공된 도구만 사용하여 질문에 답변해야 합니다.<|end_of_text|>
<|start_of_role|>user<|end_of_role|>IBM의 현재 주가는 얼마인가요?<|end_of_text|>
<|start_of_role|>assistant<|end_of_role|>

Response:

<|start_of_role|>assistant<|end_of_role|>{
  "tool": {
    "name": "get_stock_price",
    "arguments": {
      "ticker": "IBM"
    }
  }
}<|end_of_text|>

Observations

1. English Prompt:

   • The <|tool_call|> special token was correctly included in the model’s response.
   • The response followed the expected format, indicating proper behavior.

2. Korean Prompt:

   • The <|tool_call|> special token was not included in the model’s response.
   • Instead, the response was generated in a plain JSON format without the special token.

3. Hypothesis:

   • The difference in behavior may be due to the model’s training data being more optimized for English prompts.
   • Korean prompts may not have sufficient training examples for <|tool_call|> generation.

Questions:

1. Does the Granite model currently support <|tool_call|> generation for non-English prompts, particularly Korean?
2. If support for non-English prompts is limited, are there plans to improve this functionality in the future?
3. Are there any specific adjustments or configurations I can make to ensure <|tool_call|> is generated for Korean prompts?

Thank you for your assistance! Please let me know if additional details or clarification are needed.