finance
One Product, Four LLMs: How We're Choosing Inference Models for Injectionator
The Question We Should Have Asked Sooner
I was building a demo — one of Injectionator’s batteries-included applications — when my AI coding assistant did something unexpected. It wired up an inference provider I hadn’t considered. Not a wrong choice. Just a different one. And it worked.
That triggered a question: are we using the right model? We’d picked a well-known commercial model early on and never revisited the decision. It was good enough. It handled our use cases. We moved on to harder problems.
But “good enough” is a dangerous place to stop when you’re building a product that calls an LLM on every single request. So I started digging. And the more I dug, the more I realised we weren’t just asking the wrong answer — we were asking the wrong question. The question isn’t “which model should we use?” It’s “which models should we use, for what, and why?”
When we mapped our actual inference needs, we found we didn’t have one. We had four. Each with wildly different requirements for cost, speed, quality, and capabilities. Treating them as a single decision was leaving both money and performance on the table.
This post is about the process we built to make those decisions properly. We haven’t finished making them yet — that’s Part 2. This is the framework.
What an Inference Model Actually Does (and Doesn’t)
Before we get into model selection, it’s worth clearing up something that tripped us up early on. It’s about what an LLM actually does when your application “calls a tool.”
An LLM is a stateless function. Tokens in, tokens out. It doesn’t search the internet. It doesn’t query your database. It doesn’t send emails. It doesn’t call tools. When people say a model “called a tool,” what actually happened is more mundane: the model produced structured text that suggested a tool call. The application sitting in front of the model parsed that text, executed the actual function, and fed the result back as more tokens. The model never touched your API keys, your network, or your data. The application did. The model had influence over what happened. It didn’t have access.
This matters more than it sounds like it should.
“Tool calling support” on a model card means two things: the model was trained to produce well-formed structured output matching a tool-call schema, and the provider’s API supports the request/response format for passing tool definitions and receiving invocations. That’s it. It’s a training quality attribute — how reliably does this model produce parseable tool invocations? — not a capability toggle you flip on.
And here’s the practical punchline: tool-calling reliability varies wildly between models. A model that tops general benchmarks might produce malformed tool calls a significant percentage of the time. Another model, lower on the leaderboard, might nail your specific tool schemas every time. You can’t read this off a comparison chart. You have to test it with your tools, your schemas, your prompts. This becomes a selection criterion in its own right — and one that most evaluation frameworks miss entirely.
One Product, Four Inference Needs
Most AI products treat model selection as a single decision. “We use Model X.” Ship it. Move on.
But if your product does multiple things with an LLM, each thing might have completely different requirements. We found four.
Classification inspectors. We have an inspector that uses an LLM to classify whether user input is a prompt injection attempt. The model never sees the raw input — just an abstracted intent schema, following what we call the Intent-Gated Reconstruction pattern. This needs to be fast. It’s on the hot path of our inspection pipeline, and it has a sub-500ms budget. It needs to be cheap — it runs on every single scan. And it needs to be accurate — false positives break user trust, false negatives miss attacks. What it doesn’t need is tool calling, long context windows, or conversational ability. This is a narrow classification task at high volume, and it’s our most price-sensitive scenario by far.
Batteries-included applications. Our HR Chatbot demo is a fully functional LLM application with tool calling. It looks up employee records, searches the web, sends emails, updates records. The model needs to produce reliable, well-formed tool invocations and carry on natural conversation with a user. Latency matters — it’s user-facing — but not as critically as the inspector. Cost matters at scale, but quality matters more. A chatbot that calls the wrong tool is worse than one that costs a bit more per request but gets it right.
Red team attack generation. Our Attackbot generates adversarial prompts for testing. It needs to be creative, understand injection techniques deeply, and produce diverse attacks that probe different vulnerabilities. Speed doesn’t matter — this isn’t real-time. Cost barely matters — the volume is low. This is the one scenario where “use the biggest, smartest model you can get” is genuinely the right answer.
The proxy pass-through. In back-proxy mode, Injectionator sits between the user’s application and whatever LLM they’re already using. We don’t choose this model — they do. We just inspect the traffic. But this means our inspection overhead needs to be tiny relative to the model call it’s wrapping. If their model call takes two seconds and we add three seconds of inspection, we’ve made their application worse. This isn’t a model choice for us — it’s an architectural constraint that makes scenario one even harder.
Four scenarios. Four different sets of requirements. The cheapest model for classification would be terrible at generating attacks. The best attack generator would be absurdly wasteful for classification. There is no single right answer here, and pretending there is means you’re optimising for one scenario at the expense of the other three.
The Framework: Five Dimensions
For each inference scenario, we evaluate candidate models across five dimensions. Not all dimensions matter equally for every scenario — that’s the whole point.
Cost per request, not cost per token. Raw token pricing is misleading. What matters is what a single request actually costs you, given your real token profile. A model with cheap input tokens but that needs five times more output tokens to complete your task is more expensive than one with pricier tokens that’s more concise. The real formula is simple: (avg_input_tokens × input_price) + (avg_output_tokens × output_price). Multiply by volume for monthly cost. When we ran this calculation across candidate models for the same classification task, we found cost differences of up to 95%. Whether the cheaper model actually does the job is a separate question — but you need to know the cost shape before you can make that tradeoff.
Latency. There are two flavours and they matter in different contexts. Time to first token is what drives perceived responsiveness in a chat interface — it’s the moment the user sees something happening. Total generation time is what matters in a pipeline — it’s eating into a timeout budget. Some providers offer dramatically lower latency than others for the same model, because they’ve invested in specialised inference hardware. Provider choice is a latency decision, not just a model decision.
Quality for your task. General benchmarks tell you how a model performs on average across generic tasks. They tell you almost nothing about how well it classifies injection attempts in your specific prompt format, or how reliably it invokes your particular tool schemas. The only way to know is to test with your data. For us, that means running our benchmark payload suite against candidate models and measuring detection accuracy directly. A model that scores lower on public leaderboards might score higher on our narrow classification task. Don’t trust leaderboards. Trust your own evals.
Capabilities as hard filters. Some requirements are binary. If you need tool calling, models without it are eliminated — full stop. If you need 100K+ context for RAG, models with 8K windows are out. If you need vision, that’s a filter. Apply capability filters first to narrow the field, then evaluate what’s left on cost, latency, and quality. And remember: “supports tool calling” means the model is eligible for testing, not that it’s good at it. Eligibility and quality are different things.
Reliability and availability. Uptime, rate limits, geographic distribution, error rates under load. Most teams evaluate this last. We think it should come earlier. For a security product, a down inspector doesn’t mean a degraded user experience — it means unprotected traffic flowing through. That’s a security incident. We weight reliability more heavily than most products would. Fallback chains — if provider A is down, route to provider B with a compatible model — are an architectural decision you make up front, not a patch you add after your first outage.
Not every dimension matters equally for every scenario. For our classifier, cost and latency dominate. For the chatbot, capabilities and quality dominate. For attack generation, quality dominates and nothing else really matters. Part of the framework is deciding which dimensions to optimise for in each case — and being explicit about which ones you’re willing to compromise on.
What We’re Testing
We’ve mapped the landscape of available models across several providers — both hardware-specialised providers focused on inference speed and platform providers offering a broad range of models. The map is surprisingly large. It’s also shifting constantly, which is both an opportunity and an operational challenge.
For classification, we found models that could be dramatically cheaper than our current default — up to 95% less per request. But we don’t know yet if they hit our accuracy bar. That’s what the benchmark suite is for, and we’re running it. A model that saves 95% but misses 5% more attacks isn’t a saving — it’s a liability.
For the chatbot, tool calling reliability is the gating question. We’re testing specific models against our four tools — employee lookup, web search, send email, update records — to see which ones produce reliable, parseable invocations consistently. The difference between a model that gets tool calls right 95% of the time and one that gets them right 99% of the time is the difference between a demo that mostly works and a product you can trust.
For attack generation, we’re prioritising the smartest models available regardless of cost. Volume is low, so even an expensive model is cheap at this scale. Creativity and depth of understanding matter more than anything else here.
And we’re exploring the multi-provider pattern — routing different scenarios to different providers rather than picking one provider for everything. Some people call this “inference arbitrage.” We just call it using the right tool for the job.
We don’t have answers yet. We have a method and initial data. Part 2 will have the answers.
The Bigger Picture: Knowing What Your AI Costs
Once you’re routing requests to multiple models at different price points, you need to know what things cost. Not at the end of the month from a bill. In real time, per request, per user.
If your product calls LLMs on behalf of users — and users can make choices that affect which models get called, like enabling specific inspectors or configuring different modes — they deserve to see the cost impact of those choices. This is an emerging discipline. Some people call it “FinOps for AI” or “inference economics.” We think it’s going to become table stakes for any product with LLM calls in the loop.
We’re building towards this for Injectionator. More on that in a future part of this series.
What’s Next
One product, four inference needs, five evaluation dimensions, and a process — not a decision tree that gives you an answer, but a method for finding your own.
This is Part 1. We’ve shared the framework and the thinking. In Part 2, we’ll share what we actually chose for each scenario, and what surprised us. After that: how we wired up multiple providers (the integration patterns that worked and the ones that didn’t), and what we learned about cost once real traffic started flowing.
If you’re building AI applications and wrestling with these same questions — or if you’ve already made your model decisions and want to compare notes — get in touch.
Steve Chambers is the founder of Viewyonder, a GenAI consultancy helping teams build reliable AI systems. He’s currently building Injectionator, a security tool for AI applications, and learning more about inference economics than he expected to. If your team is making model selection decisions and wants a framework to work from, get in touch.