Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

The best open-source tools to run LLMs locally in 2026

The best open-source tools to run LLMs locally right now are llama.cpp, vLLM and Ollama, out of 32 we have reviewed. Each is ranked by our rating, with its price, our verdict and who it suits.

By ReviewRamble editors · Updated Nov 18, 2026 · How we review

How we ranked: by our rating out of 5, from reviews built on each product's own pricing, documentation and policies plus independent benchmarks where they exist, each checked on the date shown in the review. We take no affiliate commission.

  1. 1

    If an open model runs on a laptop, it almost certainly runs through llama.cpp, and nothing else covers as much hardware.

    Best for: Developers who want direct control over running open models locally, on edge devices or on modest servers.

    Price: Free and open source under the MIT licence. No paid edition.

    Read our llama.cpp review
    4.8/5
  2. 2

    Still the engine to put an open model behind a multi-user API, provided someone on the team owns its upgrades and its network exposure.

    Best for: Teams serving open models to many concurrent users on GPU servers.

    Price: Free and open source under the Apache 2.0 licence. You pay for the GPUs it runs on.

    Read our vLLM review
    4.7/5
  3. 3

    Local models in one command, with defaults (a 4,096-token context, one request at a time, no login) that suit a laptop and nothing bigger.

    Best for: Developers and individuals who want open models running locally with minimal setup.

    Price: Open source (MIT) and free to run locally. Optional Ollama Cloud: Free with starter credits; Pro $20/month ($60 credits); Max $100/month ($300 credits); Team $500/month ($1,000 shared credits, unlimited users); Enterprise custom.

    Read our Ollama review
    4.6/5
  4. 4

    Kubernetes sized for a handful of GPU boxes or a fleet of edge sites, though on a single machine Docker Compose still does the job with less to maintain.

    Best for: Small teams running inference or agent services across a few GPU machines or many edge sites who want Kubernetes without a platform team.

    Price: Free and open source under Apache 2.0, with no paid edition. Commercial support comes through SUSE Rancher Prime (up to five years per release); prices are quoted on request.

    Read our K3s review
    4.4/5
  5. 5

    MLX turns a high-memory Mac into a credible box for local inference and LoRA tuning; off Apple silicon it is a port, not a reason to switch.

    Best for: Mac-based developers and researchers who want native speed and Python control on Apple silicon.

    Price: Free and open source under the MIT license.

    Read our MLX review
    4.4/5
  6. 6

    SGLang is built for the agent and RL workloads that resend the same long prompts, and it belongs behind a locked-down network.

    Best for: Teams serving open models with heavy prompt reuse, such as agents and multi-turn chat.

    Price: Free and open source under the Apache 2.0 license.

    Read our SGLang review
    4.4/5
  7. 7

    audio.cpp is doing for local speech what llama.cpp did for chat models, and four months in it already runs more than 110 model families.

    Best for: Engineers who self-host several speech or audio models and want one native runtime and HTTP API instead of a Python environment per model.

    Price: Free under Apache 2.0. No paid edition. Model weights keep their own licences, many of them non-commercial.

    Read our audio.cpp review
    4.3/5
  8. 8

    A forecasting model that tops the leaderboards by Google's own count, with one trap: the 3.0 weights are for research only, so production users either stay on Apache-licensed 2.5 or pay Google through BigQuery.

    Best for: Analysts and data teams who need decent forecasts across many series quickly, and who can either live with TimesFM 2.5 or already run on BigQuery.

    Price: Code and weights up to 2.5 are free under Apache 2.0. TimesFM 3.0 weights are free for non-commercial use only; commercial use goes through BigQuery ML, billed at BigQuery rates, with token-based pricing for 3.0 from 1 December 2026.

    Read our TimesFM review
    4.3/5
  9. 9

    A disk-streaming C engine that makes 744B-parameter open models run on a desktop, slowly, with an honest local stand-in for the Jev decision API on the side.

    Best for: Enthusiasts and developers who want to run frontier-size open MoE models, or prototype Jev-style decisions, on their own machine.

    Price: Free and open source (Apache 2.0). Each model keeps its own licence.

    Read our colibri review
    4.2/5
  10. 10

    Cog is the shortest road from a research repo to a container with a typed HTTP API, and the obvious choice if Replicate is where you will host, though its API documents no authentication of its own and its interface is still moving.

    Best for: ML engineers and researchers who need a model to run reliably in a GPU container, especially those publishing to Replicate.

    Price: Free and open source under Apache 2.0. Running images on Replicate is billed separately on Replicate's per-second hardware or per-output pricing.

    Read our Cog review
    4.1/5
  11. 11

    ds4 gets frontier-class open models running at usable speed on a single big Mac or DGX Spark by refusing to be general, which is both why it is fast and why you may lose your model on the next commit.

    Best for: Developers with a 96 GB-plus Mac, a DGX Spark or a Strix Halo box who want DeepSeek V4 Flash or GLM 5.3 Flash running locally behind their coding agent.

    Price: Free and open source (MIT). No hosted or paid tier; you supply the hardware and download the weights.

    Read our DwarfStar (ds4) review
    4.1/5
  12. 12

    A genuinely clever way to run a 125B-parameter model on a 12 GB graphics card, as long as you have 64 GB of RAM and can live with a one-person project that changes weekly.

    Best for: Owners of a 12 to 24 GB graphics card and 64 GB or more of RAM who want a large open model for coding agents on their own machine.

    Price: Free and open source under the MIT licence; funded by donations. Model files carry their own licences (Qwen3.8-Flash-Next uses the Qwen Community License 1.0).

    Read our Strata review
    4.0/5
  13. 13

    TensorRT-LLM is where new NVIDIA silicon gets its first serious optimisations, and the price is a long CVE list and a stable release cadence measured in seasons.

    Best for: Teams running large NVIDIA GPU fleets who want maximum throughput per GPU.

    Price: Free and open source under the Apache 2.0 license (some bundled components carry their own licenses).

    Read our TensorRT-LLM review
    4.0/5
  14. 14

    Careful research code that gets honest probabilities out of the open model you already run, with leaderboard results the board's maintainers measured themselves.

    Best for: ML engineers who already serve an open model on vLLM and want calibrated classification and routing decisions without training a separate classifier.

    Price: Free and open source under Apache-2.0, including the Tacit model weights. You supply the GPU.

    Read our AnyJev review
    3.9/5
  15. 15

    Foundry Local is a tidy way to ship a small model inside a desktop app, but its curated catalogue, telemetry on by default and single-user design make it a poor substitute for Ollama on a workstation.

    Best for: Developers shipping desktop or edge apps that need a local model on users' own machines without an API key or cloud bill.

    Price: Free. SDKs under MIT; the CLI under Microsoft's own licence terms; each model keeps its own licence. No Azure subscription or per-token charge.

    Read our Foundry Local review
    3.9/5
  16. 16

    The simplest way to hand someone a working local model, but less capable than the llama.cpp it wraps.

    Best for: People who need to share or run a local model with zero setup, including offline.

    Price: Free and open source under the Apache 2.0 licence, with MIT-licensed changes to llama.cpp.

    Read our llamafile review
    3.9/5
  17. 17

    The Mac model server to choose when several agents share one small model, with measured gains and an unusually frank list of where it is not faster.

    Best for: Mac developers who run several coding agents or sub-agents against a local model and want them to finish faster.

    Price: Free and open source (Apache 2.0). Each model keeps its own licence.

    Read our Locals review
    3.9/5
  18. 18

    A serious attempt at turning a drawer of mismatched GPUs into one endpoint, still rough at the edges and honest that it is experimental.

    Best for: Homelab owners and small teams with several GPU machines who want one local endpoint and are comfortable running experimental distributed software.

    Price: Free and open source (Apache-2.0). No paid edition; you supply the hardware.

    Read our Mesh LLM review
    3.9/5
  19. 19

    VibeVoice is now a strong open transcription line with a small real-time voice attached, and the long-form TTS that made it famous is the one part you can no longer run from Microsoft's code.

    Best for: Teams that self-host transcription of long multi-speaker audio and want one maintained repo for batch, streaming and CPU recognition, plus a small real-time English voice.

    Price: Free under the MIT licence; weights free on Hugging Face. VibeVoice-ASR is also in Azure AI Foundry Labs. No paid edition.

    Read our VibeVoice review
    3.9/5
  20. 20

    DSpine is a real speed-up for anyone serving Qwen3.8-27B on SGLang, and of no use yet to anyone serving anything else.

    Best for: Teams serving Qwen3.8-27B or Qwen3-8B on SGLang who want more tokens per GPU without changing outputs.

    Price: Free under Apache 2.0, code and draft weights. You need your own GPU running the target model.

    Read our DSpine review
    3.8/5
  21. 21

    bitnet.cpp makes ternary models genuinely fast on a CPU, but with so few models built for it, it is a research runtime rather than a general one.

    Best for: Researchers and embedded engineers testing native 1-bit models on CPU-only hardware.

    Price: Free and open source under the MIT licence. No paid edition or hosted API.

    Read our bitnet.cpp review
    3.6/5
  22. 22

    A rare look inside a production LLM service, but for now it is a reference engine for one model on two chip families, not something to put behind an API.

    Best for: Inference engineers running DeepSeek-V4.1-Flash at cluster scale who want DeepSeek's kernels and settings as a reference.

    Price: Free and open source under the Apache 2.0 licence. You pay for Blackwell GPUs or Ascend NPUs.

    Read our DeepInfer review
    3.6/5
  23. 23

    A careful, well-documented way to fit a 35B model on a 24 GB Mac or a phone, but it runs only two preview models of its makers' choosing and nearly every speed figure is their own.

    Best for: Apple Silicon owners with 16 to 24 GB who want a 35B-class model locally, and mobile developers testing on-device chat on recent phones, who accept preview models.

    Price: Free and open source under Apache 2.0, models included. No paid edition.

    Read our edge0 review
    3.6/5
  24. 24

    It really does run an 80B model on a 32 GB laptop, but the big margins over llama.cpp are the author's own and the whole thing is one person's experiment.

    Best for: x86-64 users who want a mixture-of-experts model larger than their RAM, such as Qwen3-Next-80B on a 32 GB laptop, and accept experimental software.

    Price: Free and open source under GPL-3.0 or later. No paid edition.

    Read our Janas review
    3.6/5
  25. 25

    Thoughtful plumbing around small decision models that only become accurate once you fine-tune them on your own labels.

    Best for: Developers who will fine-tune a small decision model on their own data and want a ready router, server and training loop.

    Price: Free and open source (Apache 2.0), including the model weights.

    Read our Laya review
    3.6/5
  26. 26

    An impressive feat of packing that runs a 284B model on a 64 GB desktop at about 9 tokens a second, in file formats nothing else can read and with a real loss in quality.

    Best for: Enthusiasts with a 64 GB AVX-512 desktop who want to run DeepSeek-V4-Flash 284B entirely on the CPU and accept a single-project file format.

    Price: Free and open source (Apache 2.0). Each model keeps its own licence.

    Read our llama-modchip review
    3.6/5
  27. 27

    A smart idea with real speed behind it on some machines, but the 2x headline is the company's own measurement and has not held up on every user's hardware.

    Best for: Developers on Apple silicon or recent NVIDIA cards who want a fast local model behind a coding agent with little setup.

    Price: Free and open source under Apache-2.0. No paid edition is documented; model licences are separate.

    Read our Magnitude review
    3.6/5
  28. 28

    Moshi still shows what a truly conversational voice model feels like, but a 2024 English-only model that wants a 24 GB GPU is a research platform, not a voice agent to ship.

    Best for: Speech researchers and voice-interface engineers who want to study or extend a full-duplex spoken dialogue model on their own GPUs.

    Price: Free: code under MIT and Apache 2.0, weights under CC-BY 4.0. No paid edition.

    Read our Moshi review
    3.6/5
  29. 29

    The Whisper package is still the shortest path from an audio file to a transcript, but for any real volume faster-whisper or whisper.cpp do the same job in a fraction of the time.

    Best for: Developers and researchers who want the reference Whisper implementation for small jobs, teaching or checking another runtime's output.

    Price: Free and open source under the MIT licence, code and weights alike. No paid edition.

    Read our Whisper (openai/whisper) review
    3.6/5
  30. 30

    A careful paper with runnable code behind it, but until someone outside the lab reproduces it on more than two Qwen models, treat the 3x throughput as a lead to test, not a setting to ship.

    Best for: Inference researchers and serving engineers with Blackwell GPUs who want to test 4-bit KV cache on long-context workloads.

    Price: Free under the MIT licence for original code; some kernel helpers fall under BSD 3-Clause and NVIDIA's CUTLASS DSL licence. You need a B200 or B300 GPU.

    Read our BitDecoding-FP4 review
    3.4/5
  31. 31

    ChatTTS still gives a research voice agent convincing laughs and pauses in Chinese and English, but its non-commercial weights, deliberate audio degradation and a model untouched since 2024 rule it out for products.

    Best for: Researchers and hobbyists studying conversational speech in Chinese or English who will never put the output in a commercial product.

    Price: Free. Code is AGPLv3+; the model weights are CC BY-NC 4.0, so no commercial use. No paid or commercial licence is published.

    Read our ChatTTS review
    3.4/5
  32. 32

    AirLLM really does fit giant models into a few gigabytes of GPU memory, but at seconds to minutes per token it is a way to run a model once, not to chat with it.

    Best for: Researchers and hobbyists who need to run or fine-tune a model far larger than their GPU for a handful of prompts and can wait minutes for each answer.

    Price: Free and open source under Apache-2.0; no paid tier.

    Read our AirLLM review
    3.3/5

Sources

The main official source behind each ranked review; every review lists its full set.

More guides