Accelerated Inference
Intelligence needs more than a trained model.
Smriti AI Labs engineers NVIDIA GPU-based inference systems for AI applications.
- 01
Application requests
Traffic with its own shape: request sizes, concurrency and response-time targets.
Defined by the application
Measured here Response time
- 02
Serving layer
- Routing to a suitable model instance
- Batching to keep each GPU usefully busy
- Scaling as demand rises and falls
Engineered by Smriti
Measured here Throughput
- 03
Optimised model execution
- Model configured for the workload
- Inference software chosen per workload
- Quality checked after each change
Selected per workload
Measured here Model quality
- 04
NVIDIA GPUs
Accelerated hardware the model runs on.
NVIDIA hardware
Measured here Resource use
Responses return through the serving layer to the application.
The execution layer
We build the application—and engineer how its intelligence runs.
We work on the execution layer: deploying models, understanding workload behaviour and refining how models run within the needs of the application.
A trained model still has to run: fast enough for the people using it, efficiently enough to operate and without losing the quality the application depends on.
Deploying models
Putting models into service on NVIDIA GPU-based systems, with the serving layer the application needs.Understanding workload behaviour
Profiling how requests, batches and models behave together under realistic traffic.Refining how models run
Tuning configuration and execution within the application's requirements for response time, throughput, resource use and quality.
Measurement
Four measures of performance
Response time
How long a request takes, as the application experiences it. For streamed output, the time to the first part of a response matters too.
How we report it
As percentiles at a stated load, because an average hides the slow requests people notice.
Throughput
How much work the system completes in a given time, in requests or tokens per second.
How we report it
At a stated response-time target. Throughput without one is easy to inflate.
Resource use
What it takes to run the workload: GPU memory, compute and the capacity needed for the expected load.
How we report it
Alongside the hardware and configuration that produced it.
Model quality
Whether outputs are still good enough for the task after the model has been optimised to run faster or leaner.
How we report it
On a task-specific evaluation set, before and after each change.
The measures pull against each other
Larger batches can raise throughput and lengthen response time. Lower numerical precision can reduce resource use, and has to be checked against quality. We optimise within the application’s requirements, not for a single number.
Reporting
How we report performance
Distributions, not averages
Response time is reported as percentiles at a stated load, because averages hide the slow requests.Quality alongside speed
An optimisation is reported together with its effect on model quality for the task.Like-for-like comparisons
Before-and-after results share the same workload and measurement conditions, or they are not compared.
| ModelModel name, version and source | HardwareGPU type, count and memory; host details where relevant | ConfigurationPrecision, batching policy, inference software and its settings | WorkloadRequest mix, input and output lengths, concurrency | Measurement conditionsLoad pattern, warm-up, duration, percentiles reported and the quality test set | ResultEach measure with its unit, read together with every column before it |
|---|---|---|---|---|---|
Status No public measurements are published yet. Results are published only with all of the conditions above: model, hardware, configuration, workload and measurement conditions. Never as a standalone number. | |||||
Implementation options, selected per workload
NVIDIA inference software
NVIDIA TensorRT
An SDK for high-performance inference that optimises trained models to run on NVIDIA GPUs.
Where it can fit: Model executionNVIDIA TensorRT LLM
An open-source library for optimising large language model inference on NVIDIA GPUs.
Where it can fit: Model executionNVIDIA Dynamo-Triton
Formerly NVIDIA Triton Inference Server
An open-source inference server for deploying models behind a consistent serving interface.
Where it can fit: Serving layerNVIDIA Dynamo
An open-source framework for distributed inference serving, able to run with engines such as TensorRT LLM, vLLM and SGLang.
Where it can fit: Serving layerNVIDIA NIM microservices
Containers for self-hosted GPU inference, part of NVIDIA AI Enterprise.
Where it can fit: Packaged serving
Ownership
What we engineer, and what the platform provides
NVIDIA provides the GPUs and the inference software we select from. We engineer how a model is served, configured and measured for a specific application.
Models may be our own, open, third-party or the customer’s. Where a model belongs to someone else, we deploy it with permission and within its licence terms.
- L1Smriti engineers
AI application
The product, its users and its response-time and quality targets
- L2Smriti engineers
Serving layer
Routing, batching and scaling
- L3Smriti engineers
Model configuration and optimisation
How the model is prepared and run for the workload
- L4Smriti engineers
Measurement
Response time, throughput, resource use and model quality
- L5Depends on the model
Models
Our own, open, third-party or customer models, each used within its terms
- L6NVIDIA provides
NVIDIA inference software
Selected per workload from the implementation options
- L7NVIDIA provides
NVIDIA GPUs
Accelerated hardware
- Engineered by Smriti
- Provided by NVIDIA
- Models: ownership depends on the model and its licence
Future direction
Infrastructure direction
Two longer-term directions sit beside our current inference engineering. They describe where we intend to build, not something available today.
We do not operate a self-service GPU cloud today. There is no console to sign in to and no capacity to reserve.
- Future direction
Dedicated low-latency cloud infrastructure
Infrastructure designed around the latency needs of the AI systems we build. A future direction, not a current service.
- Future direction
Smriti-owned GPU cloud capacity
Future GPU capacity of our own. We do not operate a self-service GPU cloud today; our current inference work is engineering on NVIDIA GPU-based systems.
- What exists today
- Engineering of NVIDIA GPU-based inference systems for AI applications: deployment, serving and performance optimisation.
- What does not
- A self-service GPU cloud, a cloud console, or GPU capacity to rent or reserve.
Accelerated Inference