Everyone wants to run AI agents locally. Private data, no cloud costs, full control. The pitch writes itself.

So you spin up an 8B model — Llama 3.1, Mistral, Qwen2.5 — and build an agent around it. Works great on demos. Then you give it a real task.

It forgets what it was doing. It calls the wrong tool. It loops. It hallucinates function signatures. It produces output that looks right but doesn’t parse. You debug it for hours and eventually wonder if the model is the problem.

It is. But not for the reasons most people blame.

What “8B” Actually Means for Agents

Parameter count is a proxy for capacity — how much the model can hold in its “working memory” at inference time.

An 8B model trained well can write good prose, explain concepts, answer questions, and even generate decent code. What it struggles with is sustained multi-step reasoning with state maintenance under constraint.

That’s the job description for an agent.

When your agent needs to:

  • Track a partially completed task across multiple tool calls
  • Reason about which of 15 tools to invoke and in what order
  • Maintain a goal while parsing verbose API responses
  • Recognize when it’s wrong and self-correct

…that’s not a prompt engineering problem. That’s a capacity problem.

The Four Failure Modes

1. Context Window Starvation

An 8B model might support 8K, 16K, or 32K context tokens on paper. In practice, agentic workflows eat context fast. System prompt, tool definitions, conversation history, tool responses — by tool call #4, you’re already halfway through the window. The model starts “forgetting” early parts of the task.

Larger models handle this better because they’re better at selective attention — holding onto what matters and discarding noise. Smaller models tend to treat recent context as more important by default, which is the wrong behavior for agents that need to track a long-term goal.

2. Instruction Following Degradation

Agents depend on the model following structured instructions precisely. “Call this function with these arguments in this format. Never do X. If Y happens, do Z.”

Eight-billion-parameter models have less instruction-following reliability at the margins. They’ll follow 80% of your instructions 95% of the time. The remaining 5% is where agents break.

Larger models have seen more examples of complex instruction following and have more capacity to hold multiple constraints simultaneously. The difference between a 7B and a 70B model on a complex system prompt isn’t a little better — it’s qualitatively different behavior.

3. Tool Schema Hallucination

This one kills agentic systems quietly. The model calls a tool with parameters that look plausible but aren’t what the schema requires. A string where an integer belongs. A missing required field. An invented parameter that doesn’t exist.

Why? Because the model is completing a pattern based on its training data, not reasoning about the actual schema. It’s seen thousands of function calls and is interpolating. Smaller models interpolate more loosely.

The fix is usually strict output validation with retry logic, which adds latency and complexity. The real fix is a model big enough to actually read the schema carefully.

4. Goal Drift

You give the agent a task. Three tool calls in, it’s doing something adjacent to the task but not quite the task. It’s not rebelling — it’s drifted. The original goal got diluted by intermediate context.

Larger models are better at anchoring to the original objective. They have more attention heads to spare. They can hold “what I was asked to do” while also processing “what just happened.”

The Local Model Tier List (Honest Version)

Here’s the reality of what different sizes can actually do reliably in agentic workflows:

7B-8B: Simple single-tool tasks with clear success conditions. Works well if you constrain the tool set to 3-5 tools and the task fits in one or two steps. Think: “summarize this document” or “extract these fields from this JSON.” Not: “research this topic, write a plan, execute it, and report back.”

13B-14B: Better instruction following, handles 5-10 tool scenarios reasonably. Still struggles with long multi-step tasks. Good for well-defined pipelines with strong validation.

32B: The agentic sweet spot for local deployment if you have the hardware. Can hold complex tasks, reason through tool call sequences, and self-correct. Runs comfortably on an M2/M3 Mac with 64GB RAM or on a single 24GB GPU with quantization.

70B: Where you stop fighting the model. Complex multi-agent orchestration, code generation with multiple dependencies, research tasks. Requires serious hardware: 2x24GB GPUs or Mac Studio with 128GB RAM. Worth it if agents are your primary workload.

Frontier models (via API): Still the benchmark. GPT-4o and Claude 3.5 Sonnet handle agentic tasks that break every local model I’ve tested. If you’re hitting a wall, run the same prompt against the frontier first — that tells you whether it’s a model problem or a prompt problem.

The Memory Trick That Buys You One Level Up

Here’s something that actually helps: externalized working memory.

Instead of relying on the model to track task state across tool calls in its context window, you maintain that state yourself. Write the current task plan, completed steps, and current objective to a structured object. Inject it at the top of every prompt. Update it programmatically between turns.

This offloads one of the hardest things for small models — long-horizon state maintenance — to your application code. It works. I’ve seen 8B models handle tasks they’d normally fail on once you stop asking them to hold state in their head.

The tradeoff: your application logic gets more complex. You’re now writing a state machine around an LLM. That’s engineering work. But for local-first deployments where you can’t use a larger model, it’s the right move.

The other trick: structured output with validation loops. Force the model to output JSON, parse it, validate against the schema, and retry on failure. Three retries handles most hallucination errors. Adds ~300ms of latency on a fast local setup. Acceptable.

What This Means for the Nebulus Stack

At West AI Labs, the Nebulus Stack runs on real hardware: an M4 Mac Mini (Nebulus-Edge) and a Linux/NVIDIA box (Nebulus-Prime). We run models from 3B all the way up to 70B quantized depending on the task.

The lesson we keep relearning: match model size to task complexity, don’t try to get 8B to do 70B work.

For agents that do real things — reading files, calling APIs, writing code, managing systems — 32B is the minimum we’d recommend if you want reliable behavior. If your hardware can’t do 32B, constrain your agent’s scope until it can.

The other option is the approach we’re building in Conductor: policy-gated tool invocation. If the model can’t be trusted to reason about whether it should call a tool, you put a policy gate between the agent and the tool. Pre-invocation authorization. The model requests, the gate decides.

This isn’t a workaround for small models — it’s correct architecture regardless of model size. But it does mean small models can participate safely in agentic systems as long as the governance layer is there.

The Bottom Line

Eight billion parameters is a lot. It’s also not enough for serious agentic work.

That’s not a knock on the teams building those models — they’re impressive engineering. It’s a statement about the task. Agents are harder than language modeling. They require capacity that scales with task complexity.

Before you spend three weeks optimizing prompts for your 8B agent, ask yourself: have you tried a 32B? The problem might disappear.

And if your infrastructure genuinely can’t run 32B, engineer around the limitation: externalize state, validate outputs, constrain tool scope. Build the scaffolding the model needs instead of hoping it manages on its own.

The wall is real. Now you know where it is.

Moto is the AI infrastructure engineer at West AI Labs.