The best open-source tools to run LLMs locally in 2026
The best open-source tools to run LLMs locally right now are llama.cpp, vLLM and Ollama, out of 32 we have reviewed. Each is ranked by our rating, with its price, our verdict and who it suits.
By ReviewRamble editors · Updated Nov 18, 2026 · How we review
How we ranked: by our rating out of 5, from reviews built on each product's own pricing, documentation and policies plus independent benchmarks where they exist, each checked on the date shown in the review. We take no affiliate commission.
- 1
If an open model runs on a laptop, it almost certainly runs through llama.cpp, and nothing else covers as much hardware.
Best for: Developers who want direct control over running open models locally, on edge devices or on modest servers.
Price: Free and open source under the MIT licence. No paid edition.
Read our llama.cpp review4.8/5 - 2
Still the engine to put an open model behind a multi-user API, provided someone on the team owns its upgrades and its network exposure.
Best for: Teams serving open models to many concurrent users on GPU servers.
Price: Free and open source under the Apache 2.0 licence. You pay for the GPUs it runs on.
Read our vLLM review4.7/5 - 3
Local models in one command, with defaults (a 4,096-token context, one request at a time, no login) that suit a laptop and nothing bigger.
Best for: Developers and individuals who want open models running locally with minimal setup.
Price: Open source (MIT) and free to run locally. Optional Ollama Cloud: Free with starter credits; Pro $20/month ($60 credits); Max $100/month ($300 credits); Team $500/month ($1,000 shared credits, unlimited users); Enterprise custom.
Read our Ollama review4.6/5 - 4
Kubernetes sized for a handful of GPU boxes or a fleet of edge sites, though on a single machine Docker Compose still does the job with less to maintain.
Best for: Small teams running inference or agent services across a few GPU machines or many edge sites who want Kubernetes without a platform team.
Price: Free and open source under Apache 2.0, with no paid edition. Commercial support comes through SUSE Rancher Prime (up to five years per release); prices are quoted on request.
Read our K3s review4.4/5 - 5
MLX turns a high-memory Mac into a credible box for local inference and LoRA tuning; off Apple silicon it is a port, not a reason to switch.
Best for: Mac-based developers and researchers who want native speed and Python control on Apple silicon.
Price: Free and open source under the MIT license.
Read our MLX review4.4/5 - 6
SGLang is built for the agent and RL workloads that resend the same long prompts, and it belongs behind a locked-down network.
Best for: Teams serving open models with heavy prompt reuse, such as agents and multi-turn chat.
Price: Free and open source under the Apache 2.0 license.
Read our SGLang review4.4/5 - 7
audio.cpp is doing for local speech what llama.cpp did for chat models, and four months in it already runs more than 110 model families.
Best for: Engineers who self-host several speech or audio models and want one native runtime and HTTP API instead of a Python environment per model.
Price: Free under Apache 2.0. No paid edition. Model weights keep their own licences, many of them non-commercial.
Read our audio.cpp review4.3/5 - 8
A forecasting model that tops the leaderboards by Google's own count, with one trap: the 3.0 weights are for research only, so production users either stay on Apache-licensed 2.5 or pay Google through BigQuery.
Best for: Analysts and data teams who need decent forecasts across many series quickly, and who can either live with TimesFM 2.5 or already run on BigQuery.
Price: Code and weights up to 2.5 are free under Apache 2.0. TimesFM 3.0 weights are free for non-commercial use only; commercial use goes through BigQuery ML, billed at BigQuery rates, with token-based pricing for 3.0 from 1 December 2026.
Read our TimesFM review4.3/5 - 9
A disk-streaming C engine that makes 744B-parameter open models run on a desktop, slowly, with an honest local stand-in for the Jev decision API on the side.
Best for: Enthusiasts and developers who want to run frontier-size open MoE models, or prototype Jev-style decisions, on their own machine.
Price: Free and open source (Apache 2.0). Each model keeps its own licence.
Read our colibri review4.2/5 - 10
Cog is the shortest road from a research repo to a container with a typed HTTP API, and the obvious choice if Replicate is where you will host, though its API documents no authentication of its own and its interface is still moving.
Best for: ML engineers and researchers who need a model to run reliably in a GPU container, especially those publishing to Replicate.
Price: Free and open source under Apache 2.0. Running images on Replicate is billed separately on Replicate's per-second hardware or per-output pricing.
Read our Cog review4.1/5 - 11
ds4 gets frontier-class open models running at usable speed on a single big Mac or DGX Spark by refusing to be general, which is both why it is fast and why you may lose your model on the next commit.
Best for: Developers with a 96 GB-plus Mac, a DGX Spark or a Strix Halo box who want DeepSeek V4 Flash or GLM 5.3 Flash running locally behind their coding agent.
Price: Free and open source (MIT). No hosted or paid tier; you supply the hardware and download the weights.
Read our DwarfStar (ds4) review4.1/5 - 12
A genuinely clever way to run a 125B-parameter model on a 12 GB graphics card, as long as you have 64 GB of RAM and can live with a one-person project that changes weekly.
Best for: Owners of a 12 to 24 GB graphics card and 64 GB or more of RAM who want a large open model for coding agents on their own machine.
Price: Free and open source under the MIT licence; funded by donations. Model files carry their own licences (Qwen3.8-Flash-Next uses the Qwen Community License 1.0).
Read our Strata review4.0/5 - 13
TensorRT-LLM is where new NVIDIA silicon gets its first serious optimisations, and the price is a long CVE list and a stable release cadence measured in seasons.
Best for: Teams running large NVIDIA GPU fleets who want maximum throughput per GPU.
Price: Free and open source under the Apache 2.0 license (some bundled components carry their own licenses).
Read our TensorRT-LLM review4.0/5 - 14
Careful research code that gets honest probabilities out of the open model you already run, with leaderboard results the board's maintainers measured themselves.
Best for: ML engineers who already serve an open model on vLLM and want calibrated classification and routing decisions without training a separate classifier.
Price: Free and open source under Apache-2.0, including the Tacit model weights. You supply the GPU.
Read our AnyJev review3.9/5 - 15
Foundry Local is a tidy way to ship a small model inside a desktop app, but its curated catalogue, telemetry on by default and single-user design make it a poor substitute for Ollama on a workstation.
Best for: Developers shipping desktop or edge apps that need a local model on users' own machines without an API key or cloud bill.
Price: Free. SDKs under MIT; the CLI under Microsoft's own licence terms; each model keeps its own licence. No Azure subscription or per-token charge.
Read our Foundry Local review3.9/5 - 16
The simplest way to hand someone a working local model, but less capable than the llama.cpp it wraps.
Best for: People who need to share or run a local model with zero setup, including offline.
Price: Free and open source under the Apache 2.0 licence, with MIT-licensed changes to llama.cpp.
Read our llamafile review3.9/5 - 17
The Mac model server to choose when several agents share one small model, with measured gains and an unusually frank list of where it is not faster.
Best for: Mac developers who run several coding agents or sub-agents against a local model and want them to finish faster.
Price: Free and open source (Apache 2.0). Each model keeps its own licence.
Read our Locals review3.9/5 - 18
A serious attempt at turning a drawer of mismatched GPUs into one endpoint, still rough at the edges and honest that it is experimental.
Best for: Homelab owners and small teams with several GPU machines who want one local endpoint and are comfortable running experimental distributed software.
Price: Free and open source (Apache-2.0). No paid edition; you supply the hardware.
Read our Mesh LLM review3.9/5 - 19
VibeVoice is now a strong open transcription line with a small real-time voice attached, and the long-form TTS that made it famous is the one part you can no longer run from Microsoft's code.
Best for: Teams that self-host transcription of long multi-speaker audio and want one maintained repo for batch, streaming and CPU recognition, plus a small real-time English voice.
Price: Free under the MIT licence; weights free on Hugging Face. VibeVoice-ASR is also in Azure AI Foundry Labs. No paid edition.
Read our VibeVoice review3.9/5 - 20
DSpine is a real speed-up for anyone serving Qwen3.8-27B on SGLang, and of no use yet to anyone serving anything else.
Best for: Teams serving Qwen3.8-27B or Qwen3-8B on SGLang who want more tokens per GPU without changing outputs.
Price: Free under Apache 2.0, code and draft weights. You need your own GPU running the target model.
Read our DSpine review3.8/5 - 21
bitnet.cpp makes ternary models genuinely fast on a CPU, but with so few models built for it, it is a research runtime rather than a general one.
Best for: Researchers and embedded engineers testing native 1-bit models on CPU-only hardware.
Price: Free and open source under the MIT licence. No paid edition or hosted API.
Read our bitnet.cpp review3.6/5 - 22
A rare look inside a production LLM service, but for now it is a reference engine for one model on two chip families, not something to put behind an API.
Best for: Inference engineers running DeepSeek-V4.1-Flash at cluster scale who want DeepSeek's kernels and settings as a reference.
Price: Free and open source under the Apache 2.0 licence. You pay for Blackwell GPUs or Ascend NPUs.
Read our DeepInfer review3.6/5 - 23
A careful, well-documented way to fit a 35B model on a 24 GB Mac or a phone, but it runs only two preview models of its makers' choosing and nearly every speed figure is their own.
Best for: Apple Silicon owners with 16 to 24 GB who want a 35B-class model locally, and mobile developers testing on-device chat on recent phones, who accept preview models.
Price: Free and open source under Apache 2.0, models included. No paid edition.
Read our edge0 review3.6/5 - 24
It really does run an 80B model on a 32 GB laptop, but the big margins over llama.cpp are the author's own and the whole thing is one person's experiment.
Best for: x86-64 users who want a mixture-of-experts model larger than their RAM, such as Qwen3-Next-80B on a 32 GB laptop, and accept experimental software.
Price: Free and open source under GPL-3.0 or later. No paid edition.
Read our Janas review3.6/5 - 25
Thoughtful plumbing around small decision models that only become accurate once you fine-tune them on your own labels.
Best for: Developers who will fine-tune a small decision model on their own data and want a ready router, server and training loop.
Price: Free and open source (Apache 2.0), including the model weights.
Read our Laya review3.6/5 - 26
An impressive feat of packing that runs a 284B model on a 64 GB desktop at about 9 tokens a second, in file formats nothing else can read and with a real loss in quality.
Best for: Enthusiasts with a 64 GB AVX-512 desktop who want to run DeepSeek-V4-Flash 284B entirely on the CPU and accept a single-project file format.
Price: Free and open source (Apache 2.0). Each model keeps its own licence.
Read our llama-modchip review3.6/5 - 27
A smart idea with real speed behind it on some machines, but the 2x headline is the company's own measurement and has not held up on every user's hardware.
Best for: Developers on Apple silicon or recent NVIDIA cards who want a fast local model behind a coding agent with little setup.
Price: Free and open source under Apache-2.0. No paid edition is documented; model licences are separate.
Read our Magnitude review3.6/5 - 28
Moshi still shows what a truly conversational voice model feels like, but a 2024 English-only model that wants a 24 GB GPU is a research platform, not a voice agent to ship.
Best for: Speech researchers and voice-interface engineers who want to study or extend a full-duplex spoken dialogue model on their own GPUs.
Price: Free: code under MIT and Apache 2.0, weights under CC-BY 4.0. No paid edition.
Read our Moshi review3.6/5 - 29
The Whisper package is still the shortest path from an audio file to a transcript, but for any real volume faster-whisper or whisper.cpp do the same job in a fraction of the time.
Best for: Developers and researchers who want the reference Whisper implementation for small jobs, teaching or checking another runtime's output.
Price: Free and open source under the MIT licence, code and weights alike. No paid edition.
Read our Whisper (openai/whisper) review3.6/5 - 30
A careful paper with runnable code behind it, but until someone outside the lab reproduces it on more than two Qwen models, treat the 3x throughput as a lead to test, not a setting to ship.
Best for: Inference researchers and serving engineers with Blackwell GPUs who want to test 4-bit KV cache on long-context workloads.
Price: Free under the MIT licence for original code; some kernel helpers fall under BSD 3-Clause and NVIDIA's CUTLASS DSL licence. You need a B200 or B300 GPU.
Read our BitDecoding-FP4 review3.4/5 - 31
ChatTTS still gives a research voice agent convincing laughs and pauses in Chinese and English, but its non-commercial weights, deliberate audio degradation and a model untouched since 2024 rule it out for products.
Best for: Researchers and hobbyists studying conversational speech in Chinese or English who will never put the output in a commercial product.
Price: Free. Code is AGPLv3+; the model weights are CC BY-NC 4.0, so no commercial use. No paid or commercial licence is published.
Read our ChatTTS review3.4/5 - 32
AirLLM really does fit giant models into a few gigabytes of GPU memory, but at seconds to minutes per token it is a way to run a model once, not to chat with it.
Best for: Researchers and hobbyists who need to run or fine-tune a model far larger than their GPU for a handful of prompts and can wait minutes for each answer.
Price: Free and open source under Apache-2.0; no paid tier.
Read our AirLLM review3.3/5
Sources
The main official source behind each ranked review; every review lists its full set.
- llama.cpp: llama.cpp on GitHub
- vLLM: vLLM on GitHub
- Ollama: Ollama on GitHub
- K3s: k3s-io/k3s on GitHub
- MLX: MLX on GitHub
- SGLang: SGLang on GitHub
- audio.cpp: 0xShug0/audio.cpp on GitHub
- TimesFM: google-research/timesfm on GitHub
- colibri: colibri on GitHub
- Cog: replicate/cog on GitHub
- DwarfStar (ds4): antirez/ds4 on GitHub
- Strata: Niko1221/Strata on GitHub
- TensorRT-LLM: TensorRT-LLM on GitHub
- AnyJev: nokia-applied-research/AnyJev on GitHub
- Foundry Local: GitHub: microsoft/Foundry-Local
- llamafile: llamafile on GitHub
- Locals: Locals on GitHub
- Mesh LLM: Mesh-LLM/mesh-llm on GitHub
- VibeVoice: microsoft/VibeVoice on GitHub
- DSpine: GitHub: Tencent/DSpine
- bitnet.cpp: microsoft/BitNet on GitHub
- DeepInfer: DeepInfer on GitHub
- edge0: Edge0-AI/edge0 on GitHub (README, benchmarks, roadmap)
- Janas: prabanta-dev/janas on GitHub (README and benchmark methodology)
- Laya: Laya on GitHub
- llama-modchip: llama-modchip on GitHub
- Magnitude: magnitudedev/magnitude on GitHub
- Moshi: kyutai-labs/moshi on GitHub
- Whisper (openai/whisper): openai/whisper on GitHub
- BitDecoding-FP4: BitDecoding-FP4 on GitHub
- ChatTTS: 2noise/ChatTTS on GitHub
- AirLLM: lyogavin/airllm on GitHub
More guides
- Best open-source AI agent frameworks
- Best open-source RAG tools
- Best open-source LLM fine-tuning tools
- Best open-source AI chat apps
- Best open-source AI coding agents
- Best open-source image and video AI tools
- Best open-source LLM evaluation tools
- Best open-source LLM gateways and observability tools
- Best Claude Code plugins and agent skills
- Best open-source tools for coding agents
- Best free courses for learning AI