Vertical Small LLMs: A Deep Dive
How a fine-tuned 3 billion parameter model beat today's frontier models at one job, and how to build one this weekend.
Introduction
I shipped a new feature. It reads a support ticket, chooses the right tag, and routes the request to the right person. The simplest way to build this is to call a frontier model through an API.
Yet this approach doesn't scale, because of API costs, latency, and data sensitivity risks. Plus, the model isn't solving a hard reasoning problem. Instead, it's making the same decision a thousand times a day. So I tried a "boring" alternative approach.
I fine-tuned a 3 billion parameter model on my desk in 74 minutes. On one legal-labeling task, it scored 81.71%, ahead of Claude Sonnet 4.6 at 77% and GPT-5.5 at 76.7%.
In other words, the small model isn't smarter than a frontier model. But it's a specialist, and it does one narrow task extremely well. Onward.
SLM vs. LLM at a Glance
Before going deeper, here's how the two approaches compare across the dimensions that actually matter when you're deciding which one to build on.
| Vertical small model | Frontier large model | |
|---|---|---|
| Architecture | 1-8 billion parameters, small enough to run on hardware you already own | Tens of billions to over a trillion parameters, run in cloud data centers |
| Task complexity | Excels at one narrow, repeated decision | Handles broad, multi-step reasoning across many domains |
| Long context recall | Typically shorter context windows; recall can degrade on very long inputs | Generally stronger long-context recall, built for large documents and conversations |
| Latency and cost | Roughly 10-50ms locally, no per-token API fee | Roughly 200ms-2s+ round trip, billed per token |
| Deployment and privacy | Runs on your own infrastructure; data never leaves | Data is sent to a third-party API |
What Makes Small AI Different?
The idea comes down to two words: small and vertical.
A small language model usually has between 1 and 8 billion parameters. That's small enough to run on hardware you already own. A 3 billion parameter model can run on a gaming GPU, and many models in this range even run on a modern laptop. Frontier models, by contrast, are much larger. They run in cloud data centers, and you access them through an API.
A vertical model is trained for one domain. Instead of becoming good at every task, it becomes extremely good at one, for example banking support, contract review, or medical Q&A.
Think of it like hiring people. A frontier model is a general practitioner who knows a little about many subjects. A vertical small model is like a heart surgeon. You wouldn't ask a surgeon to file your taxes, and you wouldn't ask a general practitioner to perform heart surgery. Each is built for a different job. The same idea applies to AI.
If your company's workflow keeps making the same decision over and over, a vertical small model can outperform a much larger general-purpose model on that specific task.
Important: small doesn't mean weak. It just means focused.
Why Companies Build Their Own Small Models
Three reasons keep coming up: cost, speed, and data privacy.
Running your own small model cuts inference costs dramatically since there's no per-token API bill. It responds locally in tens of milliseconds instead of waiting on a network round trip. And because the data never leaves your infrastructure, you sidestep the privacy risk of sending sensitive requests to a third-party server.
How Small Models Work
Building a small model is usually much simpler than most people expect. You don't train a model from scratch. Instead, you start with a small open base model and improve it using one or more of these techniques.
Retrieval-Augmented Generation (RAG). Think of RAG as an open-book exam instead of a closed-book one. Rather than relying only on what it memorized during training, the model looks up your company's latest docs, policies, or FAQs the moment someone asks a question. Update the source documents and the model's knowledge updates instantly, no retraining required.
Fine-tuning (LoRA). If RAG is an open-book exam, fine-tuning is on-the-job training. You show the model hundreds or thousands of examples of the exact task, label this ticket, flag this clause, answer it this way, and it adjusts to match. LoRA is the efficient version: instead of retraining the whole model, you train a small set of extra weights on top of it. That's how a 3 billion parameter model gets fine-tuned on a desktop in a little over an hour instead of needing a data center.
Distillation. Distillation is apprenticeship. A large, expensive frontier model generates thousands of high-quality answers for your specific task, and your small model trains on those answers. It never has to be as broad as the teacher, it just has to imitate the teacher on the one job that matters.
Quantization. Quantization puts the model on a diet. Weights are normally stored as 32-bit or 16-bit numbers; quantization compresses them to 8-bit or 4-bit. You lose a small, usually negligible amount of precision, but you cut memory and compute needs dramatically, often the difference between needing a GPU and running fine on a laptop.
Putting It Together
A typical production setup looks like this:
- A small model receives the request.
- RAG provides the latest company information.
- A LoRA fine-tune teaches the model the specific task.
- Quantization reduces the hardware needed to run it.
- If the request is too difficult, it's forwarded to a larger frontier model.
This approach gives you the speed and low cost of a small model while having a powerful model available for the few requests that need it.
Where Small Models Fall Short
Small models are powerful, but they're not the right tool for every job.
They struggle with complex tasks. Vertical small models are specialists, and specialists don't generalize. Ask your 3B legal-labeling model a multi-step reasoning question, a creative writing task, or anything that leans on broad world knowledge, and it falls apart fast. It hasn't learned to reason widely, it's learned to pattern-match one job extremely well. If the task genuinely needs chained reasoning or knowledge outside its training distribution, a small model is the wrong tool.
They can still hallucinate. Fine-tuning narrows a model's behavior, but it doesn't remove its tendency to make things up. A small model can state a wrong answer just as confidently as a large one, especially on inputs near the edge of its training distribution. This is exactly why routing and confidence thresholds matter, you want a way to catch these cases before a user sees them.
Fine-tuning has trade-offs. Every point you gain on your task can cost you somewhere else:
- It gets much better at your task, but often measurably worse at general knowledge and reasoning outside it
- It can lose some of the base model's built-in safety behavior, since fine-tuning reshapes its response patterns
- It can overfit to the phrasing of your training examples, making it brittle on inputs worded even slightly differently
- It stops being reusable as a general-purpose assistant, so you typically can't repurpose the same checkpoint for other tasks
The fix isn't avoiding fine-tuning, it's knowing you're trading breadth for depth, and keeping a broader model around for anything outside that narrow lane.
Benchmarks don't tell the whole story. Sometimes a model scores well because it's seen questions similar to the benchmark during training, not because it truly understands the task. When researchers evaluate the same model on completely new data, the scores can drop significantly. That's why the 81.71% number earlier in this post only means something because it was measured on a held-out set the model never saw during training, and it's why you should be skeptical of any benchmark claim that doesn't say the same.
Combining Small and Large Models
The goal is not to make the small model do everything. Instead, the goal is to use the correct model for each request, roughly 90% to the small model, and the complex or hard 10% to an LLM. Let's dive in.
Model routing. The simplest approach: every request goes to the cheap, small model first. If the model is confident, it returns its answer immediately. If confidence falls below a threshold, the request is forwarded to a frontier model instead. This gets you roughly 95% of the quality of a frontier model while reducing inference costs by roughly 85%.
Speculative decoding. A second optimization technique is speculative decoding. The small model generates a draft response first. Instead of generating the entire answer from scratch, the frontier model verifies the draft and corrects it if necessary. Because checking a draft is faster than writing from scratch, speculative decoding can improve speed by around 3-4x while producing the same final output.
On-device AI. Apple Intelligence uses a small model running directly on the device for everyday tasks. Only requests that need more reasoning or external knowledge are sent to larger models in the cloud.
Measure before you ship. None of these techniques matter if you don't evaluate. Don't rely on public datasets, build your own held-out test using real-world examples from your app, and make sure the model has never seen those examples during training. This is the only reliable way to know whether your model is actually improving, or simply performing well on familiar benchmarks.
Where Small Models Work Best
Support tickets. This is the clearest case. Ticket routing and tagging is the same decision made thousands of times a day, with a small, well-defined label set. There's no deep reasoning involved, just consistent pattern recognition. A model fine-tuned on your actual historical tickets learns your team's categories, terminology, and edge cases far better than a frontier model working off a generic system prompt.
Contract review. Contract review is repetitive the same way. Most contracts in a given industry reuse the same clause structures, indemnification, termination, liability caps. A vertical model trained on thousands of "flag this clause" examples gets very good at spotting the handful of patterns that actually matter, and it can run privately on your own infrastructure, which matters a lot when the documents are confidential.
Regulated industries. Healthcare, finance, and legal all share the same constraint: the data usually can't leave your infrastructure. That rules out most frontier APIs for a huge share of tasks right out of the gate. A small model you host yourself sidesteps the problem entirely, nothing about a patient record or a financial statement ever touches a third-party server. Combine that with the fact that regulated workflows tend to be narrow and repetitive by nature, and you get a near-perfect fit for a vertical small model.
Why Run Your Own Model
At this point you might be asking why bother at all, why not just call an API and move on? Honestly, for most things, you should. Not every task deserves this level of investment. But once a workflow becomes routine, once you're making the same decision hundreds or thousands of times a day, the economics flip. You stop paying for intelligence you don't need, you stop shipping sensitive data to someone else's servers, and you stop being at the mercy of someone else's rate limits and price changes. Owning the model for that one job turns a recurring cost into a one-time investment.
That said, this post only scratches the surface. If vertical small models are new to you, here's what's worth digging into next:
- Evaluation design, building a held-out test set that actually predicts real-world performance, not just benchmark performance
- Data collection, where the labeled examples for fine-tuning actually come from, and how much you really need
- Model selection, which open base models are worth starting from, and how to pick one for your task
- Serving infrastructure, what it takes to run a fine-tuned model in production, from a single GPU box to a small fleet
- Monitoring drift, how to tell when your specialist model has started slipping, and when it's time to retrain
For now: if your product has a workflow that makes the same decision over and over, that's your best candidate for a weekend project.
Frequently Asked Questions
What is a vertical small language model?
A vertical small language model is a language model, typically 1-8 billion parameters, that has been fine-tuned on data from a single domain or task rather than trained to handle general-purpose requests. It trades broad capability for deep, reliable performance on one narrow job.
Do I need a data center to fine-tune a small model?
No. Techniques like LoRA fine-tuning let you adapt a small open base model on a single consumer or gaming GPU, often in under two hours, without touching the majority of the model's original weights.
Can a small model really beat a frontier model?
On a narrow, well-defined task with enough labeled examples, yes. A vertical small model isn't more intelligent than a frontier model, but on the one task it was built for, it can outperform a much larger general-purpose model, as long as it's evaluated on data it has genuinely never seen.
What's the biggest risk of building a small model in-house?
Overfitting to your training data and benchmarks that don't reflect real usage. Always evaluate on a held-out set of real, unseen examples before trusting the model in production.
Conclusion
A vertical small model won't replace a frontier model, and it isn't trying to. It's built to do one job, the job your workflow repeats thousands of times a day, faster, cheaper, and more privately than an API call ever could. Pair it with a frontier model as a fallback for the hard cases, measure it honestly on data it has never seen, and you get most of the benefit of a large model with almost none of the cost.
If you're weighing whether a vertical small model makes sense for your workflow, get in touch with our team to talk through your specific use case.
