Container Backends Spec

• Summary
• Design Philosophy
• Scope
• New Backend Descriptor Labels
• Class Architecture
• Command Contract
• Setup by Install Type
• Image Updates
• Appendix A: Ramalama and AI Cockpit Conventions, and Where Lemonade Differs
• Appendix B: Halogen Launch Commands

Summary

This spec describes how we will add new container backends to Lemonade.

A container backend is a Lemonade backend whose executable is a digest-pinned OCI image instead of a downloaded binary.

• Install, load, unload, /system-info, benchmarking and the backend manager work unchanged.
• The one new host requirement is a container tool on Linux: Podman first, then Docker, each verified by a socket probe. Users will be guided to install Podman if needed.

Design Philosophy

Lemonade follows Ramalama's container conventions wherever they apply, and AI Cockpit's wherever the choice belongs to this hardware. A dagger (†) marks an item taken from Ramalama, a double dagger (‡) one taken from AI Cockpit, and a section mark (§) one both already share, on the item or at the end of a table cell for the whole cell. Appendix A sources each against Ramalama v0.24.0 and AI Cockpit v2026.9.19.1535.

Scope

• Platform. Linux-only, GPU-only. Single GPU.
• Image sources.
• Donato's Docker Hub account for rocmfpx, nathanw, and ds4.
• Peonist's ghcr.io account for Halogen.
• Models. Downloaded by Lemonade and mounted read-only. Mounts and privileges are Lemonade's to set.
• Packaging.
• The PPA, the GitHub release .deb and .rpm, source builds and the embedded SDK tarball will natively support the container backends.

New Backend Descriptor Labels

Users and clients will be able to assess critical information about each backend at a glance using these two new labels.

Its tier is one of:

1. core: supported by Lemonade's maintainers for production use.
2. community: provided by Lemonade's maintainers, but not supported for production use.
3. experimental: developed in the community and listed in Lemonade. Use at your own risk.

Its format is one of:

1. native: backend is a compiled executable binary.
2. python: backend is implemented in Python and packaged for portability with a Python interpreter inside.
3. container: backend is implemented as an OCI container, deploys as a sandboxed image, and requires a pre-installed container tool (Podman or Docker).

Tier and format are set per backend. Examples: - llamacpp:rocm: core, native - vllm:rocm: community, python - halogen:rocm: experimental, container

Lemonade's GUI and CLI will display a disclaimer the first time the user attempts to install an experimental backend. Models specific to an experimental backend should not be displayed in /v1/models?show_all=true until the backend has been installed.

Note: we can keep the experimental boolean label in the descriptor to avoid a breaking change.

Class Architecture

WrappedServer is Lemonade's main primitive: a subclass per inference engine, each spawning a server subprocess and proxying HTTP to it. This spec adds the concept of a container backend to the existing concept of a native backend. This section describes the class architecture that enables native and container backends to share as many concepts as possible, while diverging where necessary.

flowchart TD
  R["Router"]
  W["WrappedServer subclass<br/>(one per inference engine)"]
  NP["NativeProcess"]
  CP["ContainerProcess"]
  PM["ProcessManager"]
  CM["ContainerManager<br/>(global for lemond)"]
  HP(["server process<br/>on the host"])
  CL(["podman run or docker run<br/>(child of lemond)"])
  SV(["supervisor<br/>conmon or containerd-shim"])
  KP(["server process<br/>in a container"])
  R ==>|"load(model, options)"| W
  W ==>|"ServerCommand + binary path"| NP
  W ==>|"ServerCommand + pinned image"| CP
  NP ==> PM
  PM ==>|"start_process()"| HP
  CP ==>|"ContainerRunSpec"| CM
  CM ==>|"start_process()"| CL
  CL ==>|"run request"| SV
  SV ==>|"starts"| KP
  W -.->|"HTTP over loopback"| HP
  W -.->|"HTTP to the published port<br/>or container address"| KP

Rectangles are classes, rounded boxes are OS processes. Thick edges carry the command down the stack; dotted edges are the steady-state request path.

Container Backends

A container backend runs its server from an OCI image pinned by digest. Three pieces define it:

• Its BackendDescriptor names the image and the container's permissions (see containers Descriptor Field).
• Its WrappedServer subclass passes engine settings as environment variables (see Engine Settings).
• backend_versions.json pins the image's digest (see Version Pins).

containers Descriptor Field

Each BackendDescriptor contains the information needed to install and launch that backend, organized in a standardized format. BackendDescriptor gets a new struct, ContainerPolicy, that holds a container backend's image, and the device nodes and Linux permissions its container gets:

Halogen's BackendDescriptor declares one container backend, rocm, for Strix Halo (gfx1151):

/*support*/ {
    {"rocm", {"linux"}, {{"amd_gpu", {"gfx1151"}}}, /*device_summary*/ "AMD Strix Halo"},
},
/*containers*/ {
    {"rocm", {
        /*image*/   "ghcr.io/peonist-ai/halogen-flash-server",
        /*devices*/ {"/dev/dri", "/dev/kfd"},
        /*cap_add*/ {},
    }},
},

Engine Settings

A container backend's WrappedServer subclass puts engine settings in the env field of its ServerCommand, the struct that holds the program a backend runs, its flags, environment variables, model files, port and ready endpoint. Halogen's WrappedServer sets these variables:

Version Pins

A container backend's entry in backend_versions.json is one string, <tag>@<digest>. Lemonade pulls the digest, and the tag records where the digest came from.

For example, the llamacpp entry holds a native pin and a container pin side by side:

"llamacpp": {
  "vulkan": "<release tag>",
  "nathanw": "vulkan-radv-performance@sha256:<digest>"
}

ServerProcess

ServerProcess is a base class that holds one running backend server. WrappedServer owns one while a model is loaded, under its process mutex. Unload, the watchdog and a load timeout all end it with stop(). After start, WrappedServer polls the command's ready endpoint at the address the process reports. Native backends use the child class NativeProcess, and container backends use ContainerProcess:

ContainerManager

ContainerManager is the single object in lemond that runs podman or docker. It is the container counterpart of ProcessManager. It:

• picks the container tool (see Tool Choice and Install Commands)
• builds the podman run or docker run command in one function, which holds every Podman and Docker difference § (see Command Contract)
• runs, stops, inspects and sweeps containers by name and label †; the sweep runs at startup and includes stopped containers
• checks prerequisites before a load and gives each failure its fix (see Setup Assistant)
• pulls and removes images for install and uninstall

Command Contract

Lemonade starts each container backend with one podman run or docker run command, built by ContainerManager::run_command(). The command has three parts, in order: run options, the image, and the program to run inside the container. Every load logs the full command.

Run Options

Run options are the flags between run and the image.

These options are the same for Podman and Docker:

When the container tool is Podman, lemond connects to the server at 127.0.0.1:<port>, and the command adds:

When the container tool is Docker, lemond connects to the server at the container's address on its private network, and the command adds:

Image

The image reference is <image>@<digest>, from ContainerPolicy.image and the digest in the backend's pin (see Version Pins). Install pulls it, so a load never downloads anything.

Program

The program is the server the container runs, with its arguments, such as llama-server -m /mnt/models/Qwen3-4B-Q4_K_M.gguf --ctx-size 8192 --port 8001 --host 0.0.0.0. The WrappedServer subclass builds it in ServerCommand the same way as for a native launch, with two differences:

• Model paths point to the mounted files under /mnt/models.
• --host is 0.0.0.0 instead of 127.0.0.1, so lemond can reach the server from outside the container.

Complete Example

Suppose lemond runs under the user's own account on a native host with rootless Podman, and loads the model Qwen3-4B-GGUF on llamacpp:nathanw with ctx_size set to 8192. ContainerProcess runs two commands. The first creates the container's private network, and the second starts the server:

podman network create --internal \
  --label ai.lemonade \
  lemonade-llamacpp-nathanw-Qwen3-4B-GGUF

podman run \
  --rm \
  --init \
  --name lemonade-llamacpp-nathanw-Qwen3-4B-GGUF \
  --label ai.lemonade \
  --label ai.lemonade.recipe=llamacpp \
  --label ai.lemonade.backend=nathanw \
  --label ai.lemonade.model=Qwen3-4B-GGUF \
  --label ai.lemonade.port=8001 \
  --cap-drop=all \
  --security-opt=no-new-privileges \
  --pull=never \
  --network=lemonade-llamacpp-nathanw-Qwen3-4B-GGUF \
  --device /dev/dri \
  --group-add keep-groups \
  -v /home/alice/.cache/huggingface/hub/models--unsloth--Qwen3-4B-GGUF/snapshots/a1b2c3d4e5f67890a1b2c3d4e5f67890a1b2c3d4/Qwen3-4B-Q4_K_M.gguf:/mnt/models/Qwen3-4B-Q4_K_M.gguf:ro,z \
  --env HOME=/tmp \
  -p 127.0.0.1:8001:8001 \
  docker.io/kyuz0/amd-strix-halo-toolboxes@sha256:<digest> \
  llama-server \
    -m /mnt/models/Qwen3-4B-Q4_K_M.gguf \
    --ctx-size 8192 \
    --port 8001 \
    --host 0.0.0.0 \
    --jinja \
    --metrics \
    --parallel 1

Setup by Install Type

A container backend loads only when lemond can reach Podman or Docker, and that tool can open the GPU's device nodes. This section specifies, for each install type, what starts the containers, what the installer sets up, and what the setup assistant checks. The install types are:

• lemond Started by the User
• lemond as a System Service

Container backends are hidden on Windows and macOS.

Setup Assistant

The setup assistant tells the user which setup step is missing and how to fix it. While one of its checks fails, the container backend's state is action_required, with two fields:

• message: the check's "Fails when" text.
• action: the commands that fix it.

/system-info reports both fields. The Backend Manager panel in the desktop and web apps shows message, with a help button that copies action, and lemonade backends prints both. Checks rerun on each /system-info request, so the backend changes to installable as soon as a fix takes effect.

Each install type below lists its checks in the order they run, in one table per container tool. The action cells are the user's setup steps.

Tool Choice and Install Commands

Lemonade uses Podman when it is installed, and Docker otherwise §. When neither tool is installed, the setup assistant tells the user to install Podman. The install command it gives comes from the host's /etc/os-release: Lemonade matches ID, then each word of ID_LIKE, against this table:

lemond Started by the User

The user starts lemond as their own account, through any of:

• a systemctl --user unit, lemond-user.service, which runs with the user's own rights, as a shell does. Rootless Podman needs the setuid newuidmap/newgidmap helpers and the user's full subordinate ID range, and in a user manager each of systemd's sandboxing options takes one away: the seccomp-based ones (NoNewPrivileges, RestrictNamespaces, LockPersonality, RestrictRealtime) set the no-new-privileges flag, and the mount-based ones (PrivateTmp, ProtectSystem) confine the service to a user namespace that maps only the user's own UID.
• a shell, including a lemond built from source and run from its build directory
• an app that embeds lemond

What the user does:

• With Podman:
• Installs Podman.
• Adds their account to video and render, if it is not in both.
• Logs out and back in.
• With Docker:
• Adds their account to the docker group.
• Logs out and back in.

Containers are started by:

When the container tool is Podman, the checks are:

When the container tool is Docker, the checks are:

lemond as a System Service

The Debian (.deb/PPA) and Fedora (.rpm) packages install lemond as lemond.service, a systemd system service under the lemonade account. The Debian packages build from contrib/debian, the Fedora .rpms for Fedora 43 and 44 from src/cpp/CPackRPM.cmake.

What the user does:

• With Podman: nothing; podman installs with the package.
• With Docker:
• Runs sudo usermod -aG docker lemonade.
• Runs sudo systemctl restart lemond.

What the package does:

Containers are started by:

When the container tool is Podman, the checks are:

When the container tool is Docker, the checks are:

lemond.service and rootless Podman

lemond runs podman as a child process, the same way it runs every native backend. The unit is configured to give rootless Podman what it needs:

• It lets lemond create user namespaces and run the setuid newuidmap/newgidmap helpers that rootless Podman relies on, by running without NoNewPrivileges and RestrictNamespaces. The unit keeps LockPersonality, RestrictRealtime, ProtectSystem, ProtectHome and PrivateTmp: systemd applies them while it still runs as root, before it switches to the lemonade account, so they leave the no-new-privileges flag unset and the account's user namespace unchanged.
• It points XDG_RUNTIME_DIR at the unit's existing RuntimeDirectory (Environment=XDG_RUNTIME_DIR=%t/lemonade), giving Podman a runtime directory without a login session.

Rootless Podman also needs the lemonade account to own a subordinate UID/GID range: a block of host IDs that the container's own users map onto. useradd allocates one only for regular accounts, and the packages create lemonade as a system account (useradd -r), so each package's install script allocates the range right after its useradd:

• When /etc/subuid and /etc/subgid both already have a lemonade entry, the script leaves them as they are, so upgrades and reinstalls keep the existing range.
• Otherwise it reads both files, takes the highest end (start plus count) of every range in them, with a minimum of 100000, and allocates the 65,536 IDs from there to whichever file lacks a lemonade entry, with usermod --add-subuids <start>-<start+65535> lemonade and usermod --add-subgids <start>-<start+65535> lemonade. The range sits past every account's existing block, and useradd allocates later accounts past it in turn.
• When /etc/nsswitch.conf has a subid: line, the host gets subordinate IDs from a central directory such as FreeIPA or SSSD, and the script leaves the allocation to the host's administrator. The setup assistant's "Subordinate IDs allocated" check reports a missing range.

lemond starts, stops and sweeps the containers as children in its own cgroup, so stopping lemond.service tears down its containers with it.

SELinux Check

A container backend whose devices include /dev/kfd runs one more check before the checks for its install type. The SELinux policy on some Fedora releases lets a container read and write /dev/kfd but not map it, so ROCm fails at its first GPU operation. Hosts without SELinux, such as Ubuntu, have no /sys/fs/selinux and pass the check:

Halogen Kernel Check

halogen:rocm runs two more checks before the checks for its install type. Halogen needs Linux 6.18.4 or newer, built with CONFIG_HSA_AMD_SVM. lemond checks for both on the KFD node whose gfx_target_version is 110501, reading its properties under /sys/devices/virtual/kfd/kfd/topology/nodes. A kernel too old to report cwsr_size or ctl_stack_size fails the second check:

Image Updates

A new workflow, .github/workflows/validate_containers.yml, keeps every container backend on the newest build of its tag. It follows the pattern of validate_llamacpp.yml: it runs weekly, every Sunday at 18:00 UTC, and opens its pull request only after the new pins pass validation on the self-hosted runners.

The workflow changes only the digest in each pin. A maintainer sets the tag by hand to the tag AI Toolbox Cockpit runs for the same engine on the same GPU, such as vulkan-radv-performance for llamacpp:nathanw and latest for Halogen. This will be one of the first jobs to take advantage of Devlab Dispatch, which provides Strix Halo 128 GB Linux runners that can handle Halogen and DS4 models.

Appendix A: Ramalama and AI Cockpit Conventions, and Where Lemonade Differs

Ramalama (Red Hat) has run llama.cpp in Podman and Docker containers across Fedora, RHEL, Ubuntu, macOS and WSL for two years. AI Toolbox Cockpit (Donato Capitella) is the terminal app that launches these same images, Halogen included, and is where each engine's device access comes from. Ramalama answers how to run a model in a container at all; AI Cockpit answers what this hardware in particular needs. Every row is read from source (Ramalama at v0.24.0, AI Cockpit at v2026.9.19.1535), not from either README, one of which overstates its defaults (last row). Each †, ‡ or § in the body points at a row here. The last column says whether Lemonade is the same as, stricter than or looser than each, and why.

What AI Cockpit gives an operator that Lemonade, with this spec implemented, still would not:

• Three engines Lemonade has no recipe for at all: vLLM on Strix Halo and GB10, carrying the image's own per-model launch recipe; ComfyUI image workflows; and R9V for the R9700.
• Non-AMD images: NVIDIA GB10 (llama.cpp CUDA 13, DS4 CUDA 13, vLLM nightly) and Intel Arc B70 (SYCL and Vulkan). This spec is one AMD GPU on Linux.
• More than one GPU, and more than one host: DS4 coordinator and worker roles with tensor parallelism over TCP or RoCE, InfiniBand passed through automatically wherever /dev/infiniband exists, and R9V across two R9700s.
• A shell inside the toolbox: create, update, enter and delete Toolbx and Distrobox containers. Lemonade only ever starts a server, so the image's compilers, profilers and CLI tools stay out of reach.
• Launch settings someone has already tested: recommended_use profiles naming the platform and model they were validated on, benchmark-derived batch and ubatch values keyed to the job that produced them, and a warning before launch when you deviate from them.
• Integrity and licensing at download time: SHA256 per file, the model's license shown in the download confirmation, and multi-step preparation such as R9V's PLE extraction.

Out of scope but not precluded: Ramalama emits Quadlet, Kubernetes and Compose files from the same run spec. ContainerRunSpec is plain data, so a later emitter needs no backend changes, which is the property Mario asked the design to keep.

Appendix B: Halogen Launch Commands

This appendix shows the exact commands Lemonade and AI Toolbox Cockpit run to launch Halogen, the most scrutinized engine, so that reviewers can compare them flag by flag. Both launch the same model: Qwen3.8-Flash-Next W4B with the quality overlay, which is Lemonade's Qwen3.8-Flash-Next-Halogen and AI Cockpit's qwen38-flash-next-w4b-quality bundle. Both use rootless Podman, each tool's default settings, and a user named alice.

Lemonade's command is the one this spec defines: Command Contract applied to Halogen's ContainerPolicy and version pin in Container Backends, with the model files and variables that Halogen's ServerCommand names. AI Cockpit's command is the output of its build_server_cmd() at commit 57346d8 with its Server Mode defaults.

Each row holds the options that serve one purpose. Where the options differ, the last column says whether Lemonade's are stricter, looser or the same as AI Cockpit's, and why. <snapshot> stands for /home/alice/.cache/huggingface/hub/models--peonist-ai--halogen-qwen3.8-flash-next/snapshots/ac23b1b223b4e9192d27c22367d4dbacf2b595ef: