Routing agent tools with JEV: 74% fewer prompt tokens

At a glance

The problem: every tool, in every call

Tool-calling agents work by sending the model a description of every tool it is allowed to use. That is fine with three tools. It gets expensive when an agent grows into a real assistant with ten, thirty, or a hundred integrations.

Every one of those tool schemas (name, description, and typed parameters) is sent as input tokens on every single request, even when the user only wants the weather. You pay for that context each time, and a long tool list also gives the model more ways to pick the wrong one.

We wanted to know how much of that cost we could remove without hurting accuracy.

The idea: choose the tool first

Instead of handing GPT-4o all the tools, we added a routing step in front of it using JEV, the fast "System One" classification model from TypeSafe AI. JEV answers typed questions about a piece of text. Here, the question is simply: which of these tools does this request need?

1. JEV reads the user's request and the one-line description of each tool, and returns a single tool name.
2. We build the GPT-4o agent with only that tool.
3. The agent fills in the arguments and calls it, exactly as before.

GPT-4o still does the work it is good at (understanding the request and producing correct arguments). It just no longer reads ten tool schemas to do it.

The setup

We gave the agent ten tools covering a typical assistant's range: weather, currency conversion, flights, stock prices, email, calendar, translation, web search, a calculator, and news. Each is a regular LangChain tool. The implementations are stubs because we are measuring routing and token usage, not the APIs behind them.

@tool
def get_weather(city: str) -> str:
    """Get the current weather for a city."""
    return f"It's sunny and 22°C in {city}."
 
 
@tool
def send_email(to: str, subject: str, body: str) -> str:
    """Send an email to a recipient with a subject and body."""
    return f"Email '{subject}' sent to {to}."
 
 
# ...eight more tools: currency, flights, stocks, calendar,
# translation, web search, calculator, and news.
 
tools_dict = {t.name: t for t in [get_weather, convert_currency, ...]}

The baseline: one agent, all ten tools

This is how most agents are built today. The model receives every tool schema and decides for itself.

def run_agent_without_jev(user_query: str):
    # The agent sees ALL tool schemas and has to pick one itself
    agent = create_agent(model=model, tools=list(tools_dict.values()))
    result = agent.invoke({"messages": [("user", user_query)]})
    return tool_called(result), result

With JEV: classify, then call with one tool

The tool descriptions double as JEV's selection criteria, so there is nothing extra to maintain. When a tool is added, JEV can route to it automatically.

from typesafe_sdk import Choice, TypeSafeClient
 
classifier = TypeSafeClient(
    api_key=os.environ.get("OPEN_ROUTER_API_KEY"),
    base_url="https://openrouter.ai/api",
)
 
 
def run_agent_with_jev(user_query: str):
    # Map tool names to their descriptions for JEV's criteria
    tool_criteria = {name: t.description for name, t in tools_dict.items()}
 
    response = classifier.system_one(
        state=user_query,
        questions={
            # The criteria keys are the options JEV chooses between
            "selected_tool": Choice(
                criteria=tool_criteria,
                instructions=(
                    "Select the single most appropriate tool needed to handle"
                    " this request."
                ),
            ),
        },
    )
    chosen_key = response.choices["selected_tool"].choice
 
    # Build an agent loaded with ONLY the one relevant tool schema
    filtered_agent = create_agent(model=model, tools=[tools_dict[chosen_key]])
    result = filtered_agent.invoke({"messages": [("user", user_query)]})
    return chosen_key, result, response.usage

The answer comes back typed: response.choices["selected_tool"].choice is always one of the tool names we passed in, so there is no free-text parsing and no risk of an invented tool name.

Measuring it fairly

We wrapped each run in LangChain's usage callback and priced GPT-4o at its OpenRouter list price ($2.50 per million input tokens, $10.00 per million output tokens). JEV's own tokens are counted separately, so the GPT-4o numbers compare like for like.

INPUT_PRICE_PER_M = 2.50
OUTPUT_PRICE_PER_M = 10.00
 
 
@contextmanager
def track_usage():
    """Collect token usage and gpt-4o cost for every LangChain model call inside."""
    report = UsageReport()
    with get_usage_metadata_callback() as cb:
        yield report
    for usage in cb.usage_metadata.values():
        report.prompt_tokens += usage["input_tokens"]
        report.completion_tokens += usage["output_tokens"]
        report.total_cost += (
            usage["input_tokens"] * INPUT_PRICE_PER_M
            + usage["output_tokens"] * OUTPUT_PRICE_PER_M
        ) / 1_000_000

We also checked which tool each agent actually called, not just which one JEV suggested, by reading the tool calls from the agent's messages.

Results

We ran eight everyday requests through both versions, each one aimed at a different tool.

Accuracy held. JEV picked the right tool for all eight requests, the same as GPT-4o choosing from the full list. That includes the one we expected to be tricky: "What's happening in the AI industry this week?" could plausibly go to web search, and both versions correctly chose news.

Prompt tokens fell by 74%. On average GPT-4o read 172 tokens instead of 668. Almost all of the difference is the nine tool schemas it no longer has to see.

GPT-4o cost fell by 57%. The saving on cost is smaller than on tokens because output tokens (the agent's tool call and reply) are unchanged and cost four times as much as input tokens. Across a million requests like these, that is roughly $2,180 versus $930 in GPT-4o spend.

The trade-offs

A routing step is not free, and it is worth being clear about what it costs.

• Latency went up by about 0.4 seconds. JEV's classification took 0.85 seconds on average. The smaller agent call got about 0.4 seconds faster, but not enough to cover the extra round trip. For a chat assistant that is barely noticeable. For a voice agent, where every half second matters, we would run the classification in parallel with other work or cache routes for common requests.
• JEV has its own token bill. Each routing call used about 620 JEV tokens, priced separately and not included in the GPT-4o numbers above. Check JEV's current pricing for your volume before counting the net savings.
• One tool per request. We asked JEV for a single tool. Requests that need two steps ("check the weather in Berlin and email it to Nino") need either a multi-select question or a fallback to the full toolset.

What we'd do in production

1. Add a fallback. If the router's choice fails (the agent doesn't call the tool, or the tool errors), retry once with the full tool list. You keep the savings on the common path and lose nothing on the hard cases.
2. Route to groups, not single tools. With large catalogues, picking a category (for example "calendar and email") and loading that small group handles multi-step requests while still cutting most of the context.
3. Expect bigger savings with more tools. This benchmark used ten short tool definitions. Real integrations often have longer descriptions and more parameters, and agents routinely carry dozens of them. The baseline cost grows with every tool you add; the routed cost does not.
4. Measure on your own traffic. Eight requests show the shape of the result, not a guarantee. Before rolling out, we replay a sample of real conversations through both paths and compare tool accuracy, cost, and latency side by side.

The takeaway

Most of what an agent pays for on a typical request is context it never uses. A cheap classification step in front of the expensive model removed three quarters of GPT-4o's input tokens and more than half its cost on this workload, with no loss in accuracy and a small, manageable latency cost.

If your agents are carrying a long list of tools, talk to us. Routing is one of the first optimisations we look at.