ggml-org/llama.cpp

What it does

It loads a model stored in the GGUF file format and runs it on the processor, with optional help from a graphics card. There is a command-line tool for a single prompt or an interactive chat, and a local server that answers over HTTP with a familiar request format, so existing client code can point at it. Quantisation is the central idea: a model's numbers are stored at reduced precision, which cuts the memory needed by a large factor in exchange for some accuracy, and several levels are available. It supports Apple Silicon, CUDA, Vulkan, ROCm and plain processors, and it will split a model between graphics memory and main memory when the card is too small. A library interface lets other programs embed it, which is what most of the local-model applications do. It does not have a model catalogue or a graphical interface, it does not train or fine-tune models, and it will not choose a model for you.

Why it matters

Who it suits. Anyone who wants a language model running on hardware they control, with no outside service in the path, and who is willing to read about file formats and memory. Also anyone building a product that needs local inference inside it, since the library is the point. Skip it if you want to type one command and chat, because the friendlier wrappers exist for exactly that reason.

Verdict. This is the foundation, and that is both the recommendation and the warning. Edition 42 covered ollama/ollama, Edition 52 covered janhq/jan and mudler/LocalAI, and the easy experience those give you is built on the work in this repository. Come here when you want control the wrappers do not give you: a specific quantisation, a build tuned to your hardware, or an embedded library rather than a background service. The project moves very fast, with tagged builds most days, and the newest tag could not be read from the releases page today — code was last pushed on 5 October 2026, so currency is not in question. Build from a tag rather than from the tip if you want the same behaviour next month. For serving many users at once on server graphics cards, vllm-project/vllm is the better-suited engine.