KServe v2 compatibility

nereid exposes inference.GRPCInferenceService, the KServe v2 inference protocol over gRPC, on the same bind address as its native service. KServe v2 is an open standard rather than any one server's protocol, so anything that speaks it can drive nereid without code changes — a stock NVIDIA tritonclient among them, since Triton implements the same protocol.

Wire parity, not authoring parity

The package, service name, RPC names, and message field numbers in proto/grpc_service.proto are vendored verbatim from the KServe v2 spec (via Triton's copy of it), so the wire format is byte-compatible. That's parity on the wire and nothing more: models are still written against nereid's own contracts (a .pt file, a main.py, a SavedModel, …), not against any other server's model-repository layout.

Implemented RPCs

ServerLive, ServerReady, ModelReady, ServerMetadata, ModelMetadata, unary ModelInfer, streaming ModelStreamInfer, and RepositoryIndex.

  • Datatypes are the KServe fixed-width kinds: FP16/32/64, INT8/16/32/64, UINT8, BOOL, BF16, plus UINT16/32/64 on the native (ONNX/TensorFlow) path.
  • Inputs may be sent as raw_input_contents (raw little-endian bytes, preferred) or, for FP32, the typed contents.
  • nereid serves a single implicit model version, "1".
  • RepositoryIndex — the model-discovery call a client makes on connect (CMSSW's SONIC TritonService, for one, uses it to learn which models a server offers). nereid answers with one entry per model in nereid.yaml, in that order, each at version "1" and state READY: every model is loaded before the server listens, so there is no other state to report, and the ready: true filter therefore excludes nothing. nereid serves exactly one model repository, named after server.ml_backends_path as written in nereid.yaml; a request may leave repository_name empty or name that repository, and any other name is rejected with NOT_FOUND.

Not implemented (deferred): Torch UINT16/32/64 and BYTES; the HTTP/REST /v2 mirror; Prometheus metrics; and the model load/unload, config and statistics RPCs.

Verifying compatibility

It would be easy to convince ourselves of compatibility by having nereid's own client talk to nereid's own server, but that only proves the two agree with each other. So the check uses a stock tritonclient, built from Triton's proto stubs rather than our vendored copy, which makes it a real cross-implementation test. The committed checker scripts/triton_compat_check.py runs it across the example models:

pip install -r scripts/requirements.txt
# with a nereid.yaml exposing pymul, pyaddint, rustint, multi and model3, and the server running:
python scripts/triton_compat_check.py --url 127.0.0.1:50051 \
    --model pymul:mul --model pyaddint:addint --model rustint:addint64 --model model3 \
    --multi multi --stream pymul
# -> ... rustint: output == input+1 (int64) ✓ ... multi (sum, prod) == (a+b, a*b) ✓ ... TRITON_COMPAT_OK

Poking it with grpcurl

The native Nereid service is easy to hit directly:

# Health check
grpcurl -plaintext -import-path ./proto -proto inference.proto -d '{}' \
    '[::1]:50051' inference.Nereid/HealthCheck

# List configured models
grpcurl -plaintext -import-path ./proto -proto inference.proto -d '{}' \
    '[::1]:50051' inference.Nereid/ViewModels