Tool Calling: How an LLM Actually "Uses" Tools

An LLM can't run code. It can't access the internet. It can't read files. It can't call APIs.

All it can do is predict the next token.

So how does "tool calling" work? How does ChatGPT search the web, run Python code, or generate images? The answer is surprisingly simple — and once you understand it, you'll see through the magic of every AI product on the market.

The LLM doesn't call tools. It asks the system to call tools. It writes a structured request — like filling out a form — and the surrounding system actually executes it.

The Fundamental Trick

Here's the entire concept in one diagram:

User: "What's the weather in Mumbai?"

↓
    ┌─────────┐
    │   LLM   │ → Generates: {"name": "get_weather", "arguments": {"city": "Mumbai"}}
    └─────────┘
         ↓
    ┌──────────────┐
    │ Your Backend  │ → Actually calls the weather API
    └──────────────┘
         ↓
    ┌─────────┐
    │   LLM   │ → Reads the result, generates a human-friendly answer
    └─────────┘
         ↓
    "It's 32°C in Mumbai with high humidity."

The LLM's contribution is deciding to call the tool and formatting the request correctly. Your code does the actual execution. Then the LLM reads the result and weaves it into its response.

It's like a manager who writes memos but never touches a keyboard. The LLM writes the request; the system does the work.

How Tool Definitions Work

Before the LLM can use tools, you have to tell it what tools exist. This is done through tool definitions — JSON schemas that describe each function:

json

{

"type": "function",

"function": {

"name": "get_weather",
    "description": "Get the current weather for a city",
    "parameters": {
      "type": "object",
      "properties": {
        "city": {
          "type": "string",
          "description": "The city name, e.g., 'Mumbai' or 'New York'"
        },
        "units": {
          "type": "string",
          "enum": ["celsius", "fahrenheit"],
          "description": "Temperature units"
        }
      },
      "required": ["city"]
    }

}

}

This schema is injected into the system prompt. The LLM sees it and knows: "I have a function called get_weather that takes a city name and optional units."

What the LLM Actually Sees

When you send tools with your API call, the LLM's context looks something like this:

[System] You are a helpful assistant. You have access to these tools:

- get_weather(city: string, units?: "celsius""fahrenheit")

Get the current weather for a city

• search_web(query: string, num_results?: integer)

Search the internet and return results

[User] What's the weather in Mumbai?

The model then decides: "I should call get_weather with city='Mumbai'" and generates a structured output instead of a text response.

🔧

The API Flow (Step by Step)

Here's exactly what happens in an OpenAI API call with tools:

Step 1: You Send the Request

python

response = openai.chat.completions.create(
model="gpt-4o",
    messages=[
        {"role": "user", "content": "What's the weather in Mumbai?"}
    ],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"},
                    "units": {"type": "string", "enum": ["celsius", "fahrenheit"]}
                },
                "required": ["city"]
            }
        }
    }]

)

Step 2: The LLM Responds with a Tool Call (Not Text)

python

response.choices[0].message:

{

"role": "assistant",
    "content": null,  # No text! Just a tool call
    "tool_calls": [{
        "id": "call_abc123",
        "type": "function",
        "function": {
            "name": "get_weather",
            "arguments": '{"city": "Mumbai", "units": "celsius"}'
        }
    }]

}

Notice: content is null. The LLM didn't write a response — it wrote a function call. The arguments field is a JSON string that the LLM generated token by token.

Step 3: Your Code Executes the Tool

python

You parse the tool call and actually run it

import requests

def get_weather(city, units="celsius"):
resp = requests.get(f"https://api.weather.com/v1/{city}")
    return resp.json()
result = get_weather("Mumbai", "celsius")
result = {"temp": 32, "humidity": 78, "condition": "Partly Cloudy"}

Step 4: You Send the Result Back

python

Add the assistant's tool call message and the tool result

messages.append(response.choices[0].message)

messages.append({

"role": "tool",
    "tool_call_id": "call_abc123",
    "content": '{"temp": 32, "humidity": 78, "condition": "Partly Cloudy"}'

})

Call the API again

final_response = openai.chat.completions.create(
model="gpt-4o",
    messages=messages,
    tools=tools

)

Step 5: The LLM Gives a Human-Readable Answer

"It's currently 32°C in Mumbai with 78% humidity and partly cloudy skies.

You might want to carry an umbrella just in case!"

Two LLM calls for one tool use. That's the cost. First call to decide + format the tool call. Second call to interpret the result.

📊

Multiple Tool Calls

Modern models can call multiple tools in a single turn:

User: "Compare the weather in Mumbai and Delhi"

LLM responds with TWO tool calls:

→ get_weather(city="Mumbai")
→ get_weather(city="Delhi")

This is called parallel tool calling. Instead of calling Mumbai, waiting, then calling Delhi, the LLM requests both at once. Your backend executes them simultaneously, sends both results back, and the LLM synthesizes the answer.

python

The response might contain multiple tool_calls:

{

"tool_calls": [
        {"id": "call_1", "function": {"name": "get_weather", "arguments": '{"city": "Mumbai"}'}},
        {"id": "call_2", "function": {"name": "get_weather", "arguments": '{"city": "Delhi"}'}}
    ]

}

OpenAI vs Anthropic: Different Approaches

Both support tool calling, but the implementation differs:

OpenAI Anthropic (Claude)

Anthropic's Format

python

Anthropic tool call response

{

"role": "assistant",
    "content": [
        {"type": "text", "text": "Let me check the weather for you."},
        {
            "type": "tool_use",
            "id": "toolu_abc123",
            "name": "get_weather",
            "input": {"city": "Mumbai", "units": "celsius"}
        }
    ]

}

Sending result back (Anthropic)

{

"role": "user",
    "content": [{
        "type": "tool_result",
        "tool_use_id": "toolu_abc123",
        "content": '{"temp": 32, "humidity": 78}'
    }]

}

Key difference: Anthropic includes text alongside tool calls (the model can "think aloud" before calling a tool), and results go back as a user message with a tool_result block. OpenAI uses a dedicated tool role.

The JSON Generation Problem

Here's something most tutorials don't mention: the LLM generates the tool call arguments token by token, just like regular text. It's not running a JSON serializer — it's predicting characters:

Token 1: {

Token 2: "

Token 3: city

Token 4: "

Token 5: :

Token 6: "

Token 7: Mum

Token 8: bai

Token 9: "

Token 10: }

This means the JSON can be malformed. The model might:

Miss a closing brace

Add a trailing comma

Use single quotes instead of double quotes

Include a field that doesn't exist in the schema

Modern models (GPT-4o, Claude 3.5+) are very reliable at generating valid JSON, but it's never 100% guaranteed. That's why production systems always wrap tool call parsing in try/catch blocks.

Structured Outputs (The Fix)

OpenAI introduced Structured Outputs to guarantee valid JSON:

python

response = openai.chat.completions.create(
model="gpt-4o",
    messages=messages,
    tools=tools,
    response_format={"type": "json_schema", "json_schema": schema}

)

With structured outputs, the model uses constrained decoding — at each token position, it can only generate tokens that would result in valid JSON matching the schema. Invalid tokens get zero probability. This guarantees 100% valid output.

🔬

Tool Calling vs Function Calling

You'll see both terms used interchangeably. The history:

June 2023: OpenAI introduces "function calling" — functions parameter, function_call in response

November 2023: OpenAI renames it to "tool calling" — tools parameter, tool_calls in response

Why? Functions are just one type of tool. Future tools might include "code_interpreter", "file_search", "browser" — things that aren't simple function calls

The old functions parameter still works but is deprecated. Always use tools.

🛡️

How Many Tools Can You Define?

There's no hard limit, but there are practical constraints:

Tools Defined Impact

Every tool definition adds tokens to the prompt. 20 tools with detailed descriptions might add 2,000-3,000 tokens. At scale, that's real cost.

Pro tip: If you have many tools, use a two-phase approach:

First, ask the LLM to categorize the user's request

Then, load only the relevant subset of tools for the actual call

📦

Try It Yourself

python

Minimal tool-calling example (OpenAI)

from openai import OpenAI

import json

client = OpenAI()
tools = [{
"type": "function",
    "function": {
        "name": "calculate",
        "description": "Evaluate a mathematical expression",
        "parameters": {
            "type": "object",
            "properties": {
                "expression": {
                    "type": "string",
                    "description": "Math expression like '2 + 2' or 'sqrt(144)'"
                }
            },
            "required": ["expression"]
        }
    }

}]

messages = [{"role": "user", "content": "What's 47 * 89 + 123?"}]

First call — LLM decides to use calculator

response = client.chat.completions.create(
model="gpt-4o", messages=messages, tools=tools

)

msg = response.choices[0].message

if msg.tool_calls:

# Execute the tool
    expr = json.loads(msg.tool_calls[0].function.arguments)["expression"]
    result = eval(expr)  # Don't use eval in production!

# Send result back
    messages.append(msg)
    messages.append({
        "role": "tool",
        "tool_call_id": msg.tool_calls[0].id,
        "content": str(result)
    })

# Second call — LLM interprets result
    final = client.chat.completions.create(
        model="gpt-4o", messages=messages, tools=tools
    )
    print(final.choices[0].message.content)

🚀

Practical Takeaways

Tool calling is a structured output trick — the LLM generates JSON instead of text, your code executes it

Two API calls per tool use — one to decide + call, one to interpret the result

The LLM generates JSON token by token — it can fail, always handle parsing errors

Descriptions matter more than names — the LLM reads descriptions to decide when to use a tool

Parallel tool calls save time — modern models can request multiple tools in one turn

Use structured outputs for guaranteed valid JSON — constrained decoding eliminates parsing failures

🔄

What's Next?

Episode 31: ReAct Pattern — The agent loop from Episode 29 has a name: ReAct (Reasoning + Acting). Think → Act → Observe. It's the foundational pattern behind every AI agent, and understanding it will change how you build with LLMs.

← Previous

Ep 29: What Is an Agent

Next →

Ep 31: ReAct Pattern

Next: Episode 31 — ReAct Pattern

Every AI agent you've used — Cursor, Claude Code, Perplexity, Devin — runs the same pattern under the hood.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.