Skip to content

Adding a backend

Introduction

Lemonade provides a marketplace of backends for users and developers to experiment with, engine authors to gather community feedback, and software vendors to integrate into their own applications. This guide explains the development process of adding a backend, including both the policy and mechanical aspects.

Policy

This policy extends the spec-driven development policy with specifics for backends. Each backend submitted must include a user story, a hardware-specific test plan, and maintenance plan that maps into one of our support tiers.

User Story

What should the release notes say about the new backend?

Test Plan

All backends must run an integration smoketest in CI that exercises all features of that backend (e.g., endpoints, system compatibility detection, etc.). Most backends require access to specific GPU or NPU hardware, which must be provided by self-hosted runners.

As of October 1, 2026, most existing backends run their tests on a custom self-hosted lab provided by AMD, which has reached capacity. After October 1, 2026 all new self-hosted CI jobs that have hardware supported by AMD DevLab Dispatch must run their Lemonade CI checks in AMD DevLab Dispatch. The existing backend tests hosted in the lab should also migrate in due time. CI requirements that cannot be serviced by DevLab Dispatch should be discussed on an per-backend basis in an RFC.

Support Tier

All backends must be categorized into one of the following support tiers. These tiers are assigned in the code and propagate throughout the documentation, server endpoints, CLI, and GUI to set expectations for downstream usage.

Definitions

Item Description Cadence
Updates New releases of the upstream engine, which many engines ship quarterly, monthly, weekly, or more often. How often these updates are pushed into Lemonade.
Feedback Discord messages and GitHub Pull Requests (e.g., contributing fixes, features), Issues (e.g., bug reports, feature requests), and Discussions (e.g., RFCs, Q&A) from the ecosystem. How often this feedback is triaged and acted upon, as well as the extent to which feedback is acted upon.

Support Tiers

Tier Recommended for Production? Maintained By Update Cadence Feedback Cadence Sandbox Required?
Core Yes A specific Lemonade maintainer Same week as the upstream release Triaged daily; high-severity feedback acted upon the same week No
Experimental No A specific Lemonade maintainer Maintainer's discretion Triaged weekly; acted upon at the maintainer's discretion No
Guest No; use at your own risk A contributor, without any support guarantee No guarantee No guarantee Industry standard backends may run un-sandboxed; all others must be sandboxed

Mechanics

Lemonade backends are self-describing. A backend declares what it is in a plain-data descriptor and implements how it runs in a server class, and both live together in the backend's own folder. A registry collects every descriptor, and the router, the CLI, /system-info, and the generated docs all read it — so there are no scattered if (recipe == "...") sites to update.

Adding a backend is one folder plus three small appends:

You edit What goes there
CMakeLists.txt → LEMON_BACKENDS one line: "<recipe>\|<stem>"
src/cpp/include/lemon/backends/<stem>/<stem>.h the descriptor (header-only inline const)
src/cpp/include/lemon/backends/<stem>/<stem>_server.h the WrappedServer subclass + create() declaration
src/cpp/server/backends/<stem>/<stem>_server.cpp the implementation + create() definition
src/cpp/resources/backend_versions.json version pin(s) — skip if there's no downloaded binary (e.g. cloud)
src/cpp/resources/server_models.json the models

No router edits, no CLI edits, no doc edits, no support-matrix edits.

That covers the backend's wiring. You still write an integration test (Testing). If the backend introduces a new endpoint or capability instead of serving one that already exists, you also write that endpoint, its router plumbing, and its API docs (Adding a new endpoint or capability).

Everything for one backend lives in lemon::backends::<stem>. The descriptor is header-only so it links into both the lemonade CLI and lemond; the server class and create() are server-only (compiled into lemond).

The descriptor — <stem>/<stem>.h

Plain data. The single object the registry, CLI, /system-info, and docs all read.

#pragma once
#include "lemon/backends/backend_descriptor.h"

namespace lemon { namespace backends { namespace myrecipe {

inline const BackendDescriptor descriptor = {
    /*recipe*/          "myrecipe",
    /*display_name*/    "My Backend",
    /*binary*/          "my-server",        // "" = no subprocess (e.g. cloud)
    /*config_section*/  "myrecipe",         // defaults to recipe
    /*default_device*/  DEVICE_GPU,
    /*slot_policy*/     SlotPolicy::Standard,
    /*selectable_backend*/ false,           // true auto-exposes "<recipe>_backend" + "--<recipe>"
    /*uses_ctx_size*/   true,               // opt in to the shared ctx_size option
    /*dynamic_models*/  false,              // true = models discovered at runtime (cloud)
    /*options*/ {                           // backend-specific knobs (common ones are automatic)
        {"myrecipe_args", "--myrecipe-args", "", "ARGS", "Custom args to pass", "My Options"},
    },
    /*support*/ {                           // OS / device families ({} = no local gating)
        {"cpu", BackendTier::Experimental, BackendFormat::Native, {"linux", "windows"}, {{"cpu", {"x86_64"}}}},
    },
    /*about*/ {                             // one entry per backend named in `support`
        {"cpu", {"One or two sentences on what this backend is for.",
                 /*author*/ "upstream-org", /*publisher*/ "lemonade-sdk",
                 /*license*/ "MIT", /*project_url*/ "https://github.com/upstream-org/my-engine"}},
    },
    /*supported_modes*/ {"chat"},           // every mode you serve, most-default first
    /*required_checkpoints*/ {"main"},      // unconditional files; conditional ones checked in load()
    /*default_capabilities*/ {},            // non-mode labels stamped on a bare model
};

}}}  // namespace lemon::backends::myrecipe

SlotPolicy controls accelerator sharing: Standard (counts toward LRU slots), ExclusiveNpu (evicts all NPU servers first), CoexistByType (one per model type), ExclusiveHost (shares the machine with no other local model: loading it evicts every other local model, and loading another local model evicts it), Unmetered (never counted, never auto-evicted — cloud).

Each support row names its backend's tier (BackendTier::Core, BackendTier::Experimental or BackendTier::Guest) and format (BackendFormat::Native, BackendFormat::Python or BackendFormat::Container) right after the backend name, so a row without them does not compile. Rows for the same backend on different OSes can carry different labels; /system-info reports the labels of the row for the current OS. Tiers are defined in Support Tier and formats in Backend Tiers and Formats.

about tells users what each backend is and where it comes from; lemonade backends info and the app's backend cards show it. The author wrote the engine or fork, and the publisher builds the artifact Lemonade installs. They differ when, for example, lemonade-sdk builds ggml-org's llama.cpp for ROCm, or Donato Capitella (kyuz0) packages a community fork as a container image. Leave license empty only when the upstream project declares none. For a backend that runs a binary Lemonade did not install, such as llamacpp:system, describe the upstream project and set publisher to who supplies the binary (for example, "Your system"). BackendContractsTest fails when a backend in support has no about entry, or an entry is missing its description, author, publisher or an https:// project URL.

supported_modes is the backend's contract with the model registry. A model whose labels name no mode is stamped with supported_modes[0]; a model whose labels name a mode absent from the list is refused by POST /pull with a 400 rather than failing later at inference time. BackendModeContractTest fails the build if this list disagrees with the capability interfaces your server class implements.

An options entry is a user knob unless it sets user_configurable = false, marking a value that is not supposed to be changed by the user. The option still resolves at load time and stays settable through the API and config.json.

The server class + factory — <stem>/<stem>_server.{h,cpp}

The server class is a WrappedServer subclass. Implement load() (Starting the server process) and only the capability interfaces you serve (ITranscriptionServer, IImageServer, ITextToSpeechServer, …). WrappedServer provides default "unsupported" chat_completion/completion/responses, so a non-chat backend does not stub them. Alongside it, a free create() builds the instance.

<stem>_server.h:

#pragma once
#include "lemon/backends/backend_registry.h"   // BackendContext
#include "lemon/wrapped_server.h"

namespace lemon { namespace backends {

class MyServer : public WrappedServer {
    // load(), the capability methods you serve …
};

namespace myrecipe {
std::unique_ptr<WrappedServer> create(const BackendContext& ctx);  // server-only
// Lets BackendModeContractTest check supported_modes against the real class.
constexpr uint32_t capabilities() { return capability_mask_of<MyServer>(); }
}

}}  // namespace lemon::backends

<stem>_server.cpp:

#include "lemon/backends/myrecipe/myrecipe_server.h"
// … MyServer method definitions …

namespace lemon { namespace backends { namespace myrecipe {
std::unique_ptr<WrappedServer> create(const BackendContext& ctx) {
    return std::make_unique<MyServer>(ctx.log_level, ctx.model_manager, ctx.backend_manager);
}
}}}  // namespace lemon::backends::myrecipe

Starting the server process

load() builds a ServerCommand and passes it to start_server() together with the ServerProcess that runs it:

void MyServer::load(const std::string& model_name, const ModelInfo& model_info,
                    const RecipeOptions& options, bool do_not_upgrade) {
    backend_manager_->install_backend(myrecipe::spec()->recipe, "cpu");
    port_ = choose_port();

    ServerCommand command;
    command.program = BackendUtils::get_backend_binary_path(*myrecipe::spec(), "cpu");
    command.args = {"-m", model_info.resolved_path(), "--port", std::to_string(port_)};
    command.port = port_;
    start_server(std::make_unique<NativeProcess>(ProcessOutput{is_debug(), false}), command);
}

start_server() launches the process and polls command.ready_endpoint until it answers, then starts the backend watchdog on the same endpoint. It throws, with the process stopped, when the process exits, the load is cancelled, or the timeout passes first. These settings cover the differences between servers:

Setting Effect
command.env Extra environment variables for the server process.
command.ready_endpoint The path polled for readiness and by the watchdog. Defaults to /health; set it when the server has a different always-on route (FLM uses /api/tags).
start_server(process, command, timeout_seconds) Startup timeout in seconds. Defaults to 600.
ProcessOutput{inherit, filter_health_logs} inherit copies the server's output into the Lemonade log. filter_health_logs drops the server's health-check request lines from that output.
NativeProcess(output, working_dir) Working directory for the server process. Defaults to the Lemonade process's own.

The default unload() calls stop_server(). Override it when the backend owns resources beyond the process, such as files it created (vLLM's ROCm shim directory), and call stop_server() from the override. The router destroys the server object right after unload(), so an override cleans up external resources only. The WrappedServer destructor stops the server too, so a backend writes a destructor only for its own resources (whisper.cpp removes its temp directory).

Register it: one line

set(LEMON_BACKENDS
    ...
    "myrecipe|myrecipe"   # "<recipe>|<stem>"
)

The foreach in CMakeLists.txt compiles <stem>/<stem>_server.cpp and regenerates the registry headers, binding <stem>::descriptor to <stem>::create.

What you get for free

  • Standard options: merge_args, auto_evict, evict_idle_timeout, downsize_idle_timeout, evict_weight_factor, pinned. ctx_size is opt-in via uses_ctx_size.
  • Generated CLI flags for every descriptor option with a cli_flag, plus --<recipe> when selectable_backend = true.
  • Install/download via the backend's BackendSpec (binary + install params).
  • /system-info recipes entry (display name, options schema, support matrix).
  • Generated docs — your backend appears automatically in backends-reference.md, the README "Supported Configurations" matrix, and the multi-model NPU-exclusivity list. A CI job (backend-docs-drift) fails if the committed docs are stale. The descriptor's modality, experimental, web_display_name, and each support row's device_summary supply the editorial bits the matrix needs.

Testing

The descriptor wires your backend in, but nothing exercises it. Write an integration test in test/; these run against a live server. test/server_sd.py is the model to follow: it pulls the model, calls the endpoint on each backend variant, and asserts the response is real output (a valid RIFF or PNG header and a substantial byte count) rather than an empty body or a backend error served with a success status. Test the failure paths too: a missing required field, and an error raised by the backend.

  • Existing modality: the endpoint already has a test (another image, transcription, or LLM backend). Add your recipe as a --wrapped-server case to that script instead of writing a new one.
  • New modality: add test/server_<modality>.py modeled on server_sd.py, and a test_models entry under the modality in test/utils/capabilities.py.

Add the test to CI in both matrices of .github/workflows/cpp_server_build_test_release.yml, the Windows block and the Linux block, with one row per backend variant. These matrices run in the merge queue, not on PR pushes — label your PR ci:backends so the new rows execute before merge instead of failing for the first time in the queue (see the testing guide):

- name: <modality>-<recipe>
  script: server_<modality>.py
  extra_args: "--wrapped-server <recipe>"
  backends: "vulkan rocm"
  runner: [Linux, X64, vulkan, rocm, lemon-prod]

If your backend uses ROCm via TheRock, add its recipe to _THEROCK_RECIPES in test/utils/server_base.py. Otherwise a cold runner folds the one-time TheRock download into the first request's timeout instead of TIMEOUT_ROCM_INSTALL, and the ROCm job flakes.

Adding a new endpoint or capability

Serving an existing capability interface (ITranscriptionServer, IImageServer, ITextToSpeechServer) needs nothing beyond the folder above. The endpoint, router plumbing, and API docs already exist, and the endpoint's test already covers any backend that serves it.

A new capability, a request type with no existing endpoint, is a larger change. The descriptor cannot generate the request path, so you also edit the router and server:

You add Where
Capability interface I<Thing>Server src/cpp/include/lemon/server_capabilities.h
A ModelType value and its label mapping src/cpp/include/lemon/model_types.h. Add it to Router::get_pinned_model_counts so loaded models of the new type are counted.
Router method that dynamic_casts to your interface and dispatches src/cpp/server/router.h, src/cpp/server/router.cpp
A route class, usually a ModelRoute One file under src/cpp/server/routes/<page>/<section>/, plus one line in that folder's routes.cpp. Prefixes::Quad registers it under /v1/, which the docs and examples use (see Compatibility with v2026.42 and Prior).
API documentation The route's RouteSpec. Run docs/tools/gen_api_boilerplate.py and commit docs/api/ and docs/tools/api_examples_cache.json.

The route's folder picks its docs page: routes/openai/ for an OpenAI-compatible endpoint, a new top-level folder when you mirror another server's standard, or routes/lemonade/ for a Lemonade-specific endpoint. See the HTTP Server spec for the route classes and the generated docs.

Escape hatches

Need Hook
Device depends on the chosen backend variant (whisper npu vs cpu) override WrappedServer::effective_device(opts)
Eviction rule depends on the variant override WrappedServer::effective_slot_policy(opts)
Availability decided at runtime (cloud creds) override WrappedServer::availability()
Conditional / grouped checkpoints (sd-cpp flux, whisper npu_cache) validate in load(); list only unconditional files in required_checkpoints
Custom per-model fields without editing ModelInfo read model_info.extra<T>("my_field", fallback) (populated from unknown server_models.json keys)
Models supplied at runtime, not from server_models.json set dynamic_models = true and provide them in the class (see cloud's discover_models())
Per-create setup before load (cloud's provider) do it in create()

The simplest end-to-end example

Moonshine is the minimal case: a single descriptor option, no backend selection, CPU-only, one capability interface. See src/cpp/server/backends/moonshine/ and include/lemon/backends/moonshine/.

Note: collections (collection.omni) are orchestrator-driven, not WrappedServer subprocesses, and are the one explicit exception to this model.