Bring your own model: how Gufo extensions work

As of November 2026, Gufo serves eight models on one machine: three text generators, an embedding model, speech recognition, speech synthesis, image generation, and video generation. Each one ships kernels hand-tuned for AMD Strix Halo, its own numerical tests and, for the text models, its own speculative decoding and cache integration. Until recently, none of them had an enforced boundary with the rest of Gufo.

An extension is that boundary: a model is one directory with its sources, kernels, tests and a manifest, and the build wires it in without anyone editing a shared file. Anyone can build, try and share a model, or a quantized variant of one, from their own repository. In short: bring your own model. Section 6 shows how.

This post explains how that works. The first half is about ideas and doesn’t assume you know Gufo. The second half goes into how it’s built and how to write one.

Contents

1. Why extensions at all
2. Where the extension ends and Gufo begins
3. One directory per extension
4. Picking the code for a file
5. Compiled in, not loaded
6. What ships, and what you can bring
7. A version you can depend on
8. Writing an extension
9. How it compares
10. References

1. Why extensions at all

Before extensions, a new model was never just its own directory. It also meant edits to files every other model shared: the CLI, the server, the build, CI and the test registries. That cost landed on two groups of people.

Fig. 1 · Adding one model, before extensions

1. A new model arrives as one directory: sources, kernels, tests and docs.2. Its runner joins a 4,070-line file, its tool-call markers join a 4,191-line file, and two core enums gain a value each.3. The build, the CI workflows, the Nix files and the two functional registries each carry its name by hand.4. Reviewing a new model means reviewing the core — and a fork re-merges every one of these hunks against upstream, forever.

Maintainers stop being the bottleneck

Gufo has a small team and an opinionated principle: support only the best model that Strix Halo can run for each use case. Enforcing this rule keeps the bar for official models high; unfortunately, it is also why a few models and quantizations requested by the community have not been upstreamed.

Gufo extensions can remove this friction. A community extension is independent of upstream Gufo and its maintainers. Its author can publish and promote it on their own. This means that maintainers can focus on improving the common layers of the inference engine (caching, scheduling, etc.) and the official models.

Nobody has to keep a fork alive

A fork is cheap on the first day but gets expensive every day after. With Gufo changing daily, the fork’s author either keeps resolving conflicts or sticks with an old Gufo that misses every later update.

An extension lives in a standalone repository and doesn’t need any file upstreamed. To make this work, Gufo introduced a versioned extension API, and your extension states the lowest API revision it needs (section 7).

2. Where the extension ends and Gufo begins

Before drawing a boundary, Gufo had to say what sits on each side of it. The split follows one rule: the extension provides primitives, the core keeps policy.

• The extension owns execution, tokenization, chat templates, tool-call syntax, speculative decoding, input preprocessing, and the model’s part of what identifies a cached state.
• The core keeps HTTP and OpenAI semantics, scheduling, cache retention and memory budgets, diagnostics and the test harness.

The line between them sits at the level of a step. Generating text is a loop. First the model reads the whole prompt in one pass, called prefill. Then it produces the answer one token at a time, and each new token is one decode step. Other capabilities have loops of the same shape: an embedding model works through batches of texts, a speech model through chunks of audio.

The core owns the loop: it decides when the next step runs and which requests share it. The model owns what happens inside one step. So the interface is a handful of calls like “prefill these 2,000 tokens for this conversation” or “produce the next token for these three conversations”, and the model runs all of its layers and GPU kernels behind each call, fused and ordered however it likes.

Fig. 2 · The boundary sits at the step

1. Generating text is a loop: one prefill pass over the whole prompt, then one decode step for every new token.2. The core owns the loop: it decides when the next step runs and which conversations share it.3. Each step is one call across the interface: “prefill these 2,000 tokens for A”, “produce the next token for A, B and C”.4. Behind each call the model runs all of its layers and GPU kernels, fused and ordered however it likes.

On the extension’s side, the unit is a provider: one implementation of one capability. A capability is a kind of work the server can do, each with its own HTTP endpoints and its own interface: text generation, embeddings, speech recognition, speech synthesis, image generation, video generation and decisions, which covers classifiers that answer with a label instead of generated output.

A provider brings four things:

• An ID, such as flash-next-q4, which users and logs can name.
• Artifact keys, which declare the files it claims, such as gguf:qwen4exp for a GGUF whose architecture is qwen4exp. The build uses them to catch two extensions claiming the same files (section 4).
• A probe, a cheap check that reads a file’s metadata and answers, without touching the GPU, whether this provider is the right code to run that file. The core asks the providers when it has to decide which model a file belongs to (section 4).
• A factory, which builds the running model once the probe has said yes.

An extension holds one or more providers. Most hold one; some hold several. The LFM3 community extension holds four: text, speech recognition, speech synthesis and decisions, all built on the same backbone, so they share code and ship together.

3. One directory per extension

An extension is a directory:

src/models/qwen38_flash_next/
  extension.json      facts: name, version, providers, keys, options, targets
  extension.cmake     build logic: libraries, sources, HIP flags
  extension.cpp       entry point for the whole extension
  provider.cpp        entry point for each provider: probe, factory, command drivers
  ...                 model sources, kernels, cache adapter
  functional/         fragments for the functional test registries

Two files describe the extension to the tools around it, and they’re split by what they hold. The facts go in extension.json, the manifest, which describes the extension the way a package manifest describes a package. Being plain JSON, it can be read without running a build: by CMake at configure time, Nix at evaluation time, CI scripts and the Python test harness. It says:

• which keys the model claims;
• which command-line options it adds, and to which commands;
• which libraries it builds, and what each one needs before it can be built (hip for AMD GPU code, testing for test targets, or a specific platform).

The options bullet is what makes the command line belong to the models. The server reserves only its own names, such as host settings, sampling and command controls; every other option comes from a provider. --mtp-model exists because Flash-Next’s manifest declares it, for serve, bench and prompt. The accepted values of --speculative are off plus whatever the built providers declare, so a build with DeepSeek offers dspark and one without doesn’t.

The build logic goes in extension.cmake, because only CMake can run it. It’s written with a few helpers that give every extension the same compiler warnings, runtime checks and GPU settings.

From those files the build generates two things:

• A registry: a C++ file that calls each extension’s entry point in a fixed order, so the list of models in a binary is something you can read.
• A constants header: the manifest’s IDs, keys and options as C++ values, so code and manifest can’t drift apart.

Fig. 3 · One directory, two manifests

1. One directory holds the model: two manifests, the entry points, the model sources and its functional test fragments.2. extension.json states the facts — keys, options, libraries and what each needs, production files — and CMake, Nix and CI read it without compiling anything.3. The build writes one registry per executable: plain calls to every entry point, in a fixed order.4. A generated constants header carries the IDs, keys and options as C++ values, so code and manifest can’t drift apart.

4. Picking the code for a file

When you start gufo serve llm --model model.gguf, the core has to choose a provider. It does that without loading weights or touching the GPU.

The core opens the file’s metadata (for a GGUF, general.architecture and the tensor names and shapes) and hands it to each registered text provider’s probe, one at a time, in the registry’s fixed order. Each probe answers yes or no from metadata alone. The first yes wins, and that provider’s factory builds the model. Artifact keys play no part here: they’re for the build-time check below.

Fig. 4 · Selection: ask each probe in order

1. The core reads the artifact identity without touching the GPU: general.architecture = qwen4exp.2. Every registered text provider is a candidate, asked in the registry’s fixed order.3. Each probe answers from metadata alone. deepseek4 and qwen27-q4 say no; flash-next-q4 says yes, and the first yes wins.4. No yes at all is an error. A community extension that claims a key, option or model ID an official one owns is stopped earlier, at configure time.

If no probe says yes, the server stops with “No text provider found for model architecture”. A model Gufo doesn’t know never gets answered by one it does.

“First yes wins” would be a weak rule if two probes could both say yes by accident. Gufo guards the case that matters most at configure time: a community extension taking over a file that an official extension, one shipped in Gufo’s releases (section 6), already handles. Keys are declared in the manifests, so CMake sees every claim before anything is compiled, and it rejects a community extension that claims a key, option name or model ID an official one owns. Two community extensions, or two official ones, may share a key, and then the probes, in order, decide.

5. Compiled in, not loaded

There were four ways to make a model arrive as a separate unit:

What an extension can touch

Compiling someone else’s source into the server means deciding what that code is allowed to touch. The answer is the public interface: the headers under include/gufo/ext/, backed by one library, gufo_ext_runtime, that every extension links against. These headers are the only Gufo code an extension may include. They come in three kinds:

• one header per capability: the interfaces a model implements;
• the registry: what the generated registration code calls;
• the host services an extension borrows instead of reimplementing, such as memory allocation under the server’s budgets, transfers, sampling state, logging and file formats.

Shared GPU building blocks, such as quantized matrix multiplication, HIP graphs and weight upload, live in include/gufo/ext/experimental/. They’re part of the public interface, so an extension may use them, but they carry no compatibility promise (section 7).

Fig. 5 · The link graph

The boundary runs in both directions: server code may not include model code, and model code may not include server code. For in-tree extensions, a CI script fails the build if such an include appears. External extensions are held at build time: they see only the public headers, and the build checks what they include and link against an allowlist (section 8). A few exceptions remain on the server side, such as the HTTP layer still linking a Qwen library for tool-call parsing, and they’re being moved out.

6. What ships, and what you can bring

What gets built depends on who builds it.

If you compile Gufo yourself, the build discovers every extension in the tree, plus any directories you point it at, and compiles every one your platform supports. A new model directory is picked up without anyone editing a list.

An official release is the opposite: it ships only the models the maintainers have qualified. The Nix release package, the container images and the Windows ZIP all read one list, the release lock extensions.lock:

{
  "schema": 1,
  "extensions": ["deepseek_v4_flash", "embedding_gemma2", "minimax_h3", "qwen",
                 "qwen3_asr", "qwen3_tts", "qwen38_flash_next", "qwen_image_21"]
}

The extensions in the lock are the official extensions; every other one, wherever it lives, is a community extension. Adding a directory therefore can’t change what users download; shipping a model is a one-line change to the lock. The lock is validated in every build, so a stale name fails a development configure too.

This is where bring your own model becomes concrete. Models outside the repository use the same directory layout. Point CMake at one with GUFO_EXTENSION_DIRS, or pass it to the Nix package with gufo.override { extensions = [ ... ]; }, and it builds with zero changes to Gufo. Community extensions published on GitHub are listed in the extension catalogue, next to the official ones.

Two community extensions show what that buys. Fish Audio Drama-3 brings a new speech synthesis model, and Flash-Next GSQ a quantized variant of an official model. Neither needed a maintainer to say yes. Both live in their authors’ repositories, and both update against a new Gufo with a rebuild instead of a rebase.

gufo diagnose --section extensions lists everything that was compiled in, with its version and source, so a bug report says exactly what ran.

7. A version you can depend on

An extension in someone else’s repository survives Gufo releases only if the interface changes predictably. The extension API therefore has its own version, separate from Gufo’s, in the form 0.MINOR.REVISION. The leading 0 means the API isn’t frozen yet, and unlike semver, the minor number is the breaking one: a revision may only add things, while a minor bump may break extensions. An extension declares the lowest revision it needs, such as "api_version": ">= 0.1.12" for LFM3, and its minor number must equal the host’s, so an extension written for 0.1 won’t build against 0.2.

Nobody edits that number by hand. Every change to the public headers, the manifest schema or the CMake helpers needs a short change record in include/gufo/ext/changes/, and CMake derives the revision from the number of records. A CI check fails any pull request that changes the API without adding a record, and records, once merged, can’t be edited. The API is at 0.1.16 today, so LFM3 asks for a version from four records ago.

New hooks start in experimental/, and new features arrive as optional interfaces with defaults rather than new mandatory methods. The promise is source compatibility: rebuild against a new Gufo, and an extension that uses only the public headers outside experimental/ compiles. Binary compatibility isn’t promised.

The safety net is the official extensions. All eight live in the Gufo repository and use the same public headers, so a change that breaks extensions breaks an official one first, in the maintainers’ CI, before it reaches anyone else’s repository. The Linux kernel makes a similar bargain: whoever changes an interface fixes its in-tree users in the same change.

8. Writing an extension

This section is for readers who want to write or change an extension. It sketches the shape of the work; the extension guide has the current details.

From manifest to running server

Every time Gufo is configured, the build takes all extensions through the same steps:

1. Discover every manifest in the tree, plus any directory you point it at.
2. Validate each one, even for extensions that won’t be built, so their metadata can still be reported.
3. Select what to build: everything the platform supports, an explicit list, or the release lock, always with dependencies.
4. Check collisions between official and community extensions (section 4).
5. Build each selected extension, checking external ones against the include and link allowlist (section 5).
6. Generate the constants headers and the registry.

At start-up, the registry calls each entry point in its fixed order, and from then on the CLI, the server and the diagnostics all see the same list of providers.

Two kinds of entry point

An extension has one entry point for itself and one for each provider. A simplified manifest shows both (the guide has the full schema):

{
  "name": "qwen38_flash_next",
  "api_version": ">= 0.1.0",
  "entry": "qwen38_flash_next_register",
  "providers": [{
    "id": "flash-next-q4",
    "capability": "text_generation",
    "keys": ["gguf:qwen4exp"],
    "target": "gufo_qwen38_flash_next_hip",
    "entry": "qwen38_flash_next_provider_register"
  }]
}

The extension’s entry always runs. A provider’s entry lives in that provider’s own library, named by target (here the GPU one), and runs only if that library was built; otherwise the registry records the provider as unavailable, with a reason. That’s how a CPU-only build can still list a GPU model, and say why it can’t run it, without linking any of its code.

A provider entry is short: it registers the metadata generated from the manifest, then the provider’s functions: its probe, its factory, and the drivers for commands like bench, prompt and chat.

Adding a model

Start from the extension template in the Gufo repository, rename its extension, provider and key, and follow the extension guide. The template only registers metadata; the work is in implementing the capability interface and proving it. The conformance guide lists what that means for each capability: independent tests for configuration, tokenizer and operators, CPU-against-GPU comparisons, session isolation, streaming, cancellation, and cache histories for models that support continuation. A model that only answers one prompt isn’t qualified.

9. How it compares

This section compares design choices, not quality. Gufo’s setting is one machine, one model per process, eight hand-tuned models and a small maintainer team, where “whoever changes the interface fixes the in-tree users” is a feature.

Where Gufo’s choices match the others, and where they differ:

• Models outside the Gufo repository. vLLM and SGLang reached the same conclusion for their Python ecosystems: a new model shouldn’t need the maintainers’ review to exist.
• Source, not binary. Python gives vLLM and SGLang runtime loading for free. A C++ engine would need a frozen binary interface for it, and Gufo’s text interface changed in 12 commits in the month before the work started.
• A generated registry, not self-registration. Models that sign themselves up can fail quietly: llama.cpp’s dynamically loaded backends have shipped in builds that report “no backends are loaded”. Gufo’s registry is a file with a fixed order.
• Collisions caught at configure time. Modular MAX lets a custom architecture silently replace a built-in one with the same name. Gufo rejects that before compiling anything.
• A boundary at the step. vLLM and SGLang call back into Python model code at every layer of the network, which suits their setting. Gufo’s models run each step as one optimized block of GPU work, so the boundary is crossed once per step.
• A version derived from records. Triton and ONNX Runtime freeze C interfaces for large ecosystems. Gufo isn’t ready to freeze, so it versions source compatibility instead.

Gufo is behind where the others are strongest: vLLM and SGLang load a new model without a rebuild, in any Python environment, and llama.cpp’s ecosystem of forks is exactly the audience binary plugins would serve. Both paths stay open: a C plugin layer and models as data could each be built on the same provider interface later.

References

The extension system combines prior art from other engines and projects with decisions of its own. Thanks to the authors and maintainers of the work below; each entry says what it contributed.

• vLLM plugin system: registration by architecture name through a small entry point whose stability is promised in writing.
• SGLang: support new models: one discoverable entry symbol per external package.
• Modular MAX: serve custom architectures: matching on checkpoint identity, and the contrast case of silent override that Gufo’s configure-time collision check rules out.
• Triton Inference Server C API: version negotiation and a coarse execute boundary, the shape kept in reserve for a future plugin layer.
• ONNX Runtime OrtApiBase: append-only, versioned function tables, the reference pattern for a stable binary interface.
• ggml dynamic backend loading (llama.cpp #10469) and llama-cpp-python #2069: the “no backends are loaded” failure that decided generated registration over static constructors or runtime discovery.
• Godot: what is GDExtension: modules compiled into the engine first, runtime-loaded libraries later, the order Gufo follows.
• Zephyr modules: a declarative metadata file plus CMake build logic per out-of-tree module, proven at large scale.
• Linux kernel: stable-api-nonsense: no stable internal API, with in-tree users fixed in the same change, the bargain behind keeping the official models in-tree as canaries.