Capability overview
NVIDIA GPU inference systems
- Entry type
- Capability overview
- Business
- Accelerated Inference
- Project family
- NVIDIA GPU inference systems
- Evidence
- No customer or result claims
Context
The problem this work addresses
A trained model still has to run: fast enough for its users, efficiently enough to operate and without losing the quality the application depends on.
Typical use cases
- Deploying and serving models for an AI application
- Understanding workload behaviour under realistic traffic
- Optimising response time and throughput against cost and quality
Typical applications of this kind of system — not a list of delivered projects.
What Smriti engineers
Our contribution
The parts of the system we design, build and evaluate — beyond the platforms and models we build with.
Deployment and serving
Putting models into service on NVIDIA GPU-based systems.Workload analysis
Profiling how requests, batches and models behave together.Performance optimisation
Tuning the execution layer within the application's requirements.Measurement
Response time, throughput, resource use and model quality, reported with conditions.
System shape
How a system in this family is typically shaped
Layers run from the people who use the system down to the foundations it runs on. Each tag shows who is responsible for that layer.
Technologies we build with
- NVIDIA GPUs
NVIDIA inference software — such as TensorRT, TensorRT LLM, Dynamo-Triton, Dynamo or NIM microservices — is selected per workload and named on a project only when it is used.
- L1Smriti engineers
AI application
Requests, latency targets and quality needs
- L2Smriti engineers
Serving layer
Routing, batching and scaling
- L3Selected per workload
Optimised model execution
Inference software selected per workload
- L4NVIDIA provides
NVIDIA GPUs
Accelerated hardware
- Engineered by Smriti
- Provided by NVIDIA
- Inference software selected per workload and tuned by us
Accelerated Inference