NCA Inference Optimization & TensorRT 5 — Questions and Answers
Question 1: What is 'sparsity' acceleration in TensorRT, and which GPU architecture first introduced hardware support for it?
- Exploiting 2:4 structured sparsity in weight tensors for 2× speedup, first supported on Ampere GPUs (Correct answer)
- Removing zero-valued activations dynamically at runtime, introduced on Volta GPUs
- Pruning convolutional filters automatically during TensorRT build, available since Pascal
- Eliminating redundant batch normalization layers, enabled on Turing GPUs
Correct answer: Exploiting 2:4 structured sparsity in weight tensors for 2× speedup, first supported on Ampere GPUs
NVIDIA's 2:4 structured sparsity (2 zeros per 4 weights) is decoded in hardware by Ampere Tensor Cores, delivering up to 2× speedup for sparse models.
Question 2: When deploying TensorRT engines on NVIDIA Jetson edge devices, which runtime library is used to execute the engine without the full TensorRT SDK?
- TensorRT Runtime (libnvinfer.so) with only the runtime components, no builder required (Correct answer)
- CUDA Lite runtime packaged specifically for Jetson
- TensorFlow Lite delegate for NVIDIA hardware
- The full TensorRT SDK must always be present on the target device
Correct answer: TensorRT Runtime (libnvinfer.so) with only the runtime components, no builder required
Only the TensorRT runtime library (libnvinfer.so) is needed on the deployment device; the builder SDK is only required on the machine that creates the engine.
Question 3: Which TensorRT calibration algorithm is most suitable when calibration data is plentiful and accuracy must be maximized?
- IInt8EntropyCalibrator2 (Correct answer)
- IInt8MinMaxCalibrator
- IInt8LegacyCalibrator
- IInt8PercentileCalibrator
Correct answer: IInt8EntropyCalibrator2
IInt8EntropyCalibrator2 uses KL-divergence minimization to find optimal INT8 scale factors and is NVIDIA's recommended default for most vision models.
Question 4: What does setting 'builder_config.set_flag(trt.BuilderFlag.REFIT)' enable for a TensorRT engine?
- Updating model weights in the compiled engine without rebuilding it from scratch (Correct answer)
- Allowing the engine to adjust layer precision dynamically at runtime
- Enabling hot-swapping between multiple engines sharing the same GPU memory
- Forcing all layers to be refactored into TensorRT's native op set during build
Correct answer: Updating model weights in the compiled engine without rebuilding it from scratch
The REFIT flag builds the engine so its weights can be updated later via an IRefitter, avoiding costly full rebuilds when only weights change.
Question 5: In TensorRT, what is the function of 'mark_output()' when building a network via the Network Definition API?
- It designates a specific tensor as a network output so TensorRT includes it in the engine's binding list (Correct answer)
- It freezes the tensor's precision to FP32 to ensure output correctness
- It signals the end of the network graph and triggers optimization
- It marks a tensor to be copied back to the host automatically after each inference
Correct answer: It designates a specific tensor as a network output so TensorRT includes it in the engine's binding list
mark_output() tells TensorRT which tensors should be accessible as outputs through the engine's bindings during inference.
Question 6: What is the impact of enabling 'DLA (Deep Learning Accelerator)' offloading in TensorRT on Jetson platforms?
- Eligible layers run on the dedicated low-power DLA core, freeing the GPU for other tasks and reducing power consumption (Correct answer)
- All network layers are migrated to the DLA, removing the need for GPU memory allocation
- DLA offloading doubles INT8 throughput by running layers simultaneously on both DLA and GPU
- It converts the model to a DLA-native format incompatible with standard TensorRT engines
Correct answer: Eligible layers run on the dedicated low-power DLA core, freeing the GPU for other tasks and reducing power consumption
TensorRT can partition supported layers to run on the Jetson DLA, a fixed-function accelerator that is more power-efficient than the GPU for inference.
Question 7: Which metric does TensorRT's throughput mode ('maximize throughput' flag in trtexec) optimize for, and how does it differ from latency mode?
- Throughput mode maximizes inferences per second by running multiple concurrent streams, while latency mode minimizes time-to-result for a single request (Correct answer)
- Throughput mode increases batch size automatically, while latency mode reduces it to 1
- Throughput mode uses FP16 exclusively; latency mode uses FP32 for precision
- Throughput mode disables INT8 calibration; latency mode enables it for speed
Correct answer: Throughput mode maximizes inferences per second by running multiple concurrent streams, while latency mode minimizes time-to-result for a single request
Maximizing throughput runs inference with multiple concurrent CUDA streams to keep the GPU fully utilized, whereas minimizing latency focuses on the fastest single-request response time.
NCA Inference Optimization & TensorRT 5
What is 'sparsity' acceleration in TensorRT, and which GPU architecture first introduced hardware support for it?