Triton Inference Server
hosomaki.triton
Inference server from NVIDIA hosting models from several frameworks — TensorRT, PyTorch, ONNX Runtime, Python — behind one HTTP and gRPC endpoint, with dynamic batching and per-model metrics.
A Docker Compose file for Triton Inference Server alone, generated from the catalog and checked by the safety inspector. Sign in and use Kitchen Credits to activate it.
Uses the Medusa cart and payment-session path. This test control does not create an order or capture payment.
Triton reads a model repository in which each model has its own versioned directory and configuration file, loading a backend appropriate to each format so one server can answer for a TensorRT engine, a TorchScript module and an ONNX graph at once. Dynamic batching groups arriving requests to raise device utilisation, concurrent model instances let several copies share a GPU, and ensembles chain models into a pipeline resolved server-side rather than by the caller. Load, latency and queue time are exported per model in Prometheus format. A first deployment mounts a repository containing one model and confirms the server reports it ready.
You know it worked when
- The server reports ready on its health endpoint.
- The repository index lists each expected model in a ready state.
- An inference request over HTTP returns output tensors of the declared shape.
- The same request over gRPC returns an identical result.
- The metrics endpoint reports request counts and queue times per model.
Known sharp edges
- The image is pulled from NVIDIA's NGC registry, which may require an account and accepted terms before the pull succeeds.
- The model repository layout is strict, and a misnamed directory or missing configuration file causes the model to be skipped rather than reported as an error.
- The image is several gigabytes compressed and first pull time dominates setup.
- Batching and instance counts are set in each model's configuration, not by the server, so defaults leave most device capacity unused.