Can You Run an AI Agent Locally? Nemotron 3.5 Bolt Explained
NVIDIA just shipped Nemotron 3.5 Bolt, a 74 billion parameter open model squeezed into 24 GB so AI agents can run on your own machine.
AI Stella and Liam are AI hosts with AI-generated voices. About our AI hosts
Key takeaways
- Nemotron 3.5 Bolt 74B-A9B has 74 billion parameters with about 9 billion active per step, and 2-bit quantization fits it in 24 GB.
- It runs on GeForce RTX PCs, DGX Spark, DGX Station and RTX PRO workstations, with llama.cpp, Ollama, LM Studio and vLLM.
- NVIDIA says it ranks first among open models in its class on PinchBench and second on BrowseComp. Those are vendor-reported numbers.
- On the Artificial Analysis Intelligence Index it scores 21, against 26 for Super VL and 58 for the top model, so treat it as a local worker, not your main brain.
- Try it with one small real job, compare against your cloud model, and cap the agent's steps while you test.
Show notes
NVIDIA just shipped Nemotron 3.5 Bolt, a 74 billion parameter open model squeezed into 24 GB so AI agents can run on your own machine. In this episode of Agentic Pod, Liam and Stella cover what Bolt is, why only about 9 billion parameters are active, how to try it step by step with Ollama, LM Studio, llama.cpp or vLLM, what to use it for, the gotchas, and an honest verdict. Benchmark numbers are NVIDIA’s own unless noted.
What we cover:
- What Nemotron 3.5 Bolt is: 74B total, about 9B active, 2-bit quantization, 24 GB
- Bolt versus Super VL (the server model) and the small Bolt E4B
- How good it is: PinchBench, BrowseComp and the Artificial Analysis index (21 vs 58 for the top model)
- How to try it in four steps: hardware, runtime, model file, local endpoint
- What to use it for, and what to leave to a cloud model
- Gotchas: 2-bit accuracy, memory, vendor numbers and looping agents
- Our honest verdict
Chapters
Questions answered in this episode
What is Nemotron 3.5 Bolt?
An open NVIDIA model for local AI agents: 74 billion parameters in total, about 9 billion active per step, quantized to 2 bits so it fits in 24 GB of memory.
What hardware do I need?
A 24 GB GPU such as an RTX class card, or a DGX Spark or DGX Station.
How do I run it locally?
Pick a runtime (Ollama, LM Studio, llama.cpp or vLLM), download the Bolt build from Hugging Face, serve it locally and point your agent tool at the local OpenAI compatible endpoint.
Is it as good as cloud models?
No. The Artificial Analysis index scores it 21 against 58 for the top model, so use it for routine and private work and keep a cloud model for hard tasks.
Links & resources
Full transcript
Intro
Hey hey hey, welcome to Agentic Pod! I'm Liam.
And I'm Stella! Liam, NVIDIA just shipped a 74 billion parameter model that fits in 24 gigabytes. Is that a typo?
Not a typo. It's Nemotron 3.5 Bolt, and it's built to run AI agents on your own machine. Today: what it is, how to try it, and where it falls short.
Good. My cloud bill and I need to talk.
Let's go.
What Bolt actually is
Okay, 74 billion in 24 gigabytes. How?
Two tricks. First, it's a mixture of experts model. Only about nine billion parameters are active for each step. That's what the A9B in the name means.
And the second?
Two-bit quantization. The weights are squeezed down hard, so the whole thing fits in 24 gigabytes of memory.
Which is a gaming card, right?
Yep, an RTX class PC works, and so do DGX Spark and DGX Station. NVIDIA aims it at local research, web browsing, coding and debugging.
And there's another Nemotron, right? I saw a 120 billion one.
Super VL. That's the server model. It's not for your desk.
Okay, so Bolt is the desk one. How good is it?
How good is it, really
Here's NVIDIA's claim. Bolt ranks first among open models in its class on PinchBench, and second on BrowseComp.
BrowseComp is the web browsing one?
Yeah, it tests whether an agent can dig answers out of the web. But those numbers come from NVIDIA, so treat them as vendor reported.
Is there anything independent?
The Decoder cites the Artificial Analysis Intelligence Index. Bolt scores 21. Super VL scores 26. The top model on that list scores 58.
Ouch. So it's not replacing the big cloud models.
Right. Think of it as a local worker, not your main brain. Routine tasks, private files, offline work.
What counts as routine?
Triaging a folder of notes. Drafting commit messages. A first-pass code review. Jobs where a miss is cheap and you can check the result.
And the hard ones?
Send those to a bigger cloud model. Local for the boring stuff, cloud for the brain surgery.
Okay, I want to try it. Where do I start?
How to try it
Step one, check your memory. You want a 24 gigabyte card or a DGX Spark.
Step two?
Pick a runtime. Bolt works with llama.cpp, Ollama, LM Studio and vLLM. If you've never done this, start with Ollama or LM Studio.
And the model file?
Step three. It's on Hugging Face. Download the Bolt build your runtime supports.
Then what, I chat with it in a terminal?
You can. But the useful part is the local server. These runtimes can serve a local, OpenAI compatible endpoint. So you point your agent tool at that address instead of a cloud one.
So my agent doesn't know the difference.
Mostly, yeah. Step four, give it one small real job. Summarise a repo. Fix one failing test. Compare the result with your cloud model.
And if it's bad?
Then you learned that for free. Want the gotchas?
Gotchas, who it's for, and the verdict
Gotcha one. Two-bit quantization trades accuracy for size, so expect misses on hard tasks. Gotcha two. Long contexts eat memory, so test your real prompt sizes.
And gotcha three is the vendor numbers.
Yep. Run your own test before you trust any leaderboard. And gotcha four. Agents run in loops, so one bad step can repeat. Cap the number of steps while you test.
Good one. Because a loop on my own GPU still heats my room.
Fair.
So who is this for?
People with sensitive files, people working offline, and anyone tired of paying per token for routine agent work.
And the small one? You said there's a 4 gigabyte one.
Bolt E4B. It's aimed at voice assistants and device interfaces, and NVIDIA says it's coming later this month. Not shipped yet.
Okay, verdict time.
Honest verdict. Bolt is a real step for local agents, a 74 billion model on one 24 gigabyte card. Use it for routine and private work, keep a cloud model for the hard stuff. The guide is in the description, with every step. What would you run locally first? Tell us in the comments. See you next time!