Backends¶

A backend is how a model actually runs. nereid works out which one a model needs from the contents of its folder, or you can just say so with backend: in nereid.yaml.

This table is the canonical list of backends — other pages link here rather than repeat it, and CI checks that every registered backend appears below (see contributing). Add a row here when you add a backend.

Backend	Feature	Model folder	Heavy dependency	Isolation
Torch (TorchScript)	`torch` (default)	`.pt` + textproto	libtorch (via `tch`)	in-process
Python	`python` (default)	`main.py` + `requirements.txt`	a per-model virtualenv (built at startup)	subprocess
ONNX	`onnx`	`.onnx` + textproto	ONNX Runtime (via `ort`)	in-process
TensorFlow	`tensorflow`	SavedModel + textproto	libtensorflow	in-process
C++ subprocess	`cpp`	`main.cpp` + textproto	a C++ compiler (at runtime)	subprocess
Compile-time C++	`cxx`	textproto only (code compiled in)	the `cxx-models` crate (linked in)	in-process

The "heavy dependency" column is what a backend pulls in. For the compiled-in engines that's a linked native library; for Python it's not a crate dependency at all, but a per-model virtualenv built from requirements.txt — which, depending on what the model imports, can be the heaviest runtime cost of the lot.

You only build the ones you want. The default build keeps torch + python; select an exact set with ./build.sh --backends … (or, driving Cargo directly, --no-default-features --features …), and an ONNX-only server never links libtorch. See Building & running.

Torch (TorchScript `.pt`)¶

A folder with one .pt file (a TorchScript export) and a model_inference.textproto. The model is loaded into libtorch through tch and run in process. Single- and multi-tensor models both work, across the libtorch dtypes (FP16/32/64, INT8/16/32/64, UINT8, BOOL, BF16), and each model's device (cpu, cuda, or cuda:<index>) picks the CPU or a particular GPU.

Python (`main.py`)¶

A folder with main.py + requirements.txt. At startup nereid builds a virtualenv for the model from requirements.txt (and reuses it if one is already there), then runs main.py as a subprocess per request. The model reads its input tensor on stdin and writes a typed output tensor to a scratch file — the subprocess tensor contract has the details. Since it runs out of process, a model that crashes takes down only itself.

ONNX¶

A folder with one .onnx file + textproto, run in process on ONNX Runtime via the ort crate. If a model's device is cuda, ort's CUDA execution provider is selected for it. This path covers the full KServe dtype set, including UINT16/32/64, which the Torch backend can't (libtorch has no kind for them).

TensorFlow¶

A folder with a SavedModel (saved_model.pb + variables/) + textproto, run on libtensorflow via the tensorflow crate. The SavedModel signature defaults to serving_default, which you can override per model with signature: in nereid.yaml. GPU support needs the libtensorflow GPU build, and BF16 isn't available on this path.

C++ subprocess (`cpp`)¶

A folder with a main.cpp + textproto. This backend works like the Python one, just for C++: the server compiles main.cpp into a model executable at startup — the same idea as building a Python model's virtualenv — and runs it as a subprocess per request. It speaks the same subprocess tensor contract as Python (raw tensor on stdin, a framed tensor back), so there's no unsafe FFI and no rebuilding the server to add a model. Because it runs out of process, a model that crashes takes down only itself.

If one c++ -O2 -std=c++17 main.cpp -o model invocation isn't enough — extra sources, link flags — a folder can ship an executable build.sh instead, or just a prebuilt model binary. It needs a C++ compiler on PATH at runtime, since that's when the compile happens, and is served over the ModelInfer path, single-tensor for now. See ml-backends/cppadd (output = input + 1).

Compile-time C++ (`cxx`)¶

The other way to serve C++, at the opposite trade-off: instead of a subprocess, the model's C++ is linked into the server through the cxx crate and run in process, with a boundary the compiler checks rather than one you hand-write. The C++ lives in the cxx-models crate (crates/cxx-models/, meant to be vendored as a git submodule or workspace member): implement the nereid::Model interface, register it by name, and rebuild with --features cxx.

Because the code is compiled in and keyed by name, a model's folder holds nothing but a model_inference.textproto, and you select it with backend: "cxx" — it's the one backend that can't be auto-detected, since there's no file to detect (see the registry's auto_detect: false below). It's served over the ModelInfer path, single-tensor for now. The cost is that adding or changing a model recompiles the server; what you buy is direct in-process interop with no process or loader machinery. See crates/cxx-models for the cxxadd example and ml-backends/cxxadd for its model folder.

Choosing a backend¶

If a folder matches exactly one backend, that's the one you get. Set backend: in nereid.yaml when you want to be explicit, or when a folder ships files for more than one backend and the server can't pick for you:

models:
  - name: "my_onnx_model"
    device: "cuda"
    queue_capacity: 16
    backend: "onnx"    # python | torch (or rust) | onnx | tensorflow | cpp | cxx

A model whose files need a backend the server wasn't built with fails at startup and tells you which --features to rebuild with. It is never quietly mislabeled or skipped, which matters more than it sounds: a model that silently runs on the wrong engine is a much worse day than one that refuses to start.

How the server finds a backend¶

Nothing in the server core knows the backends above exist. There's no enum of backend kinds and no match that dispatches to them, which is deliberate — every one of those would be a central file you'd have to edit to add a backend.

Instead each backend lives in its own folder under src/backends/<name>/ and submits a registration at link time:

inventory::submit! {
    BackendRegistration {
        name: "tensorflow",             // the `backend:` value in nereid.yaml
        version: "0.1.0",               // this backend's own version
        aliases: &[],                   // other accepted spellings
        describes: "a SavedModel (saved_model.pb + variables/) + model_inference.textproto",
        auto_detect: true,              // false = only selectable by declaring it
        detect,                         // does this folder look like my model?
        load,                           // build the backend, or say which feature is missing
    }
}

The core iterates those registrations to detect and load, so it never names a backend. Detection is pure file inspection with no dependency on the engine itself, which means the registry is complete even when a backend's feature is switched off — a .pt folder in an ONNX-only build still gets a precise "rebuild with --features torch" instead of a confusing "no backend matches this folder".

Two fields are worth calling out. version is the backend's own version rather than the server's, so a backend that evolves on its own schedule can say where it is; bump the major when a revision changes the folder shape, the contract, or what a model has to declare. It's reported in the startup log for every model loaded, so you can trace a deployment back to the exact revision that served it. auto_detect: false is for a backend whose code is compiled into the server rather than sitting on disk — there's no file signature to look for, so it's selectable only by naming it in nereid.yaml.

Adding your own backend¶

Drop a folder into src/backends/ with two files in it:

mod.rs — the detection predicate and the inventory::submit! above. This is always compiled, and must not depend on the engine.
imp.rs — the engine itself, behind #[cfg(feature = "...")], implementing the Backend trait (platform(), infer(), and optionally checkpoint_stream()).

Then build. build.rs globs the subfolders of src/backends/ and emits the module declarations, so there is no mod line to add, no enum arm, no detection list, and no registration call anywhere else in the tree. Because a backend is just a directory, it can be a git submodule pointing at your own repository, and one that hasn't been initialized yet (so it has no mod.rs) is skipped rather than breaking the build.

Which discovered backends get compiled in is a separate question from Cargo features, because a feature can only name a backend that Cargo.toml already lists — which an out-of-tree backend, by definition, doesn't. So the build also takes a selection by name pattern, from $NEREID_BACKENDS or a backends.conf file:

NEREID_BACKENDS="onnx,tensorflow" cargo build --no-default-features --features onnx,tensorflow
NEREID_BACKENDS="!torch"          cargo build      # everything discovered except torch
NEREID_BACKENDS="*,!vendor-*"     cargo build      # drop a family of vendored backends

Patterns are separated by commas or newlines, * matches any run of characters, a leading ! excludes (and beats any include), and # starts a comment in the file. Leave it unset and you get everything that was discovered; whatever the selection drops is printed as a build warning, so a missing backend is never a mystery. The two knobs compose rather than overlap: the pattern decides which backend folders are compiled in at all, and the feature decides whether an in-tree backend's engine and its heavy dependency come along with it.

One caveat, since it will come up the first time you edit one of these files: cargo fmt and rust-analyzer both walk the module tree, and that tree stops at the generated include! that wires in src/backends/. CI therefore runs rustfmt on those files directly in addition to cargo fmt --all --check. rust-analyzer does run build scripts and should resolve the generated modules, though its support for include!-wired modules can be flaky.

Contributing¶

Backend facts are spread across the docs — the table at the top of this page, the diagram on the overview, the folder contract in Model contract. To keep those from drifting, there is one canonical source and everything else links to it: the table at the top of this page. When you add (or rename) a backend, update that table; the other pages carry prose that references it rather than duplicating the rows.

scripts/check_backend_docs.sh guards the one place duplication is unavoidable — that the docs and the code agree on which backends exist. It reads the registered name of every backend from src/backends/*/mod.rs and fails if any is missing from the table above, so registering a backend without documenting it breaks CI (it runs in the Docs workflow). Run it locally with ./scripts/check_backend_docs.sh.