NCA Inference Optimization & TensorRT 3 — Questions and Answers
Question 1: What is 'layer fusion' in TensorRT and why does it improve performance?
- Combining multiple adjacent operations into a single kernel to reduce memory bandwidth and kernel launch overhead (Correct answer)
- Merging model weights across layers to reduce parameter count
- Replacing convolutional layers with fully-connected layers for speed
- Sharing GPU memory buffers between unrelated networks
Correct answer: Combining multiple adjacent operations into a single kernel to reduce memory bandwidth and kernel launch overhead
Layer fusion merges operations like Conv+BN+ReLU into one optimized kernel, eliminating intermediate memory reads/writes and launch overhead.
Question 2: When using TensorRT's IExecutionContext for inference, which method launches asynchronous GPU execution?
- execute_async_v2(bindings, stream_handle) (Correct answer)
- run_inference(inputs)
- engine.forward(tensors)
- context.execute_sync(bindings)
Correct answer: execute_async_v2(bindings, stream_handle)
execute_async_v2 enqueues the inference work into a specified CUDA stream, enabling overlap with other GPU or CPU work.
Question 3: In the context of TensorRT deployment, what does 'binding' refer to?
- A GPU memory buffer associated with a named network input or output tensor (Correct answer)
- A CPU-side handle that links model weights to inference context
- A TensorRT plugin that binds custom layers to built-in operations
- A CUDA event used to synchronize between streams
Correct answer: A GPU memory buffer associated with a named network input or output tensor
Bindings are GPU device pointers mapped to each named input/output tensor that must be provided when calling execute.
Question 4: Which NVIDIA tool is commonly used to profile TensorRT engine latency and GPU utilization during inference?
- Nsight Systems
- CUDA-GDB
- nvprof (legacy) or Nsight Systems (Correct answer)
- cuDNN Bench
Correct answer: nvprof (legacy) or Nsight Systems
nvprof (legacy) and its modern replacement Nsight Systems both capture CUDA kernel timelines and GPU metrics for TensorRT workloads.
Question 5: What is the primary purpose of TensorRT plugins?
- To implement custom layers or operations that TensorRT does not natively support (Correct answer)
- To add new calibration algorithms for INT8 quantization
- To enable multi-GPU data parallelism within a single engine
- To serialize engines in a platform-independent format
Correct answer: To implement custom layers or operations that TensorRT does not natively support
TensorRT plugins let developers integrate custom CUDA kernels for operations outside TensorRT's built-in layer library.
Question 6: What does enabling 'builder.fp16_mode = True' (or the FP16 flag in BuilderConfig) allow TensorRT to do?
- Automatically convert eligible layers to FP16 precision to leverage Tensor Core acceleration (Correct answer)
- Force all layers to run in FP16 regardless of accuracy impact
- Enable mixed-precision only for convolutional layers
- Disable FP32 fallback for unsupported operations
Correct answer: Automatically convert eligible layers to FP16 precision to leverage Tensor Core acceleration
Enabling FP16 lets TensorRT select FP16 kernels for layers where it can maintain sufficient accuracy, while falling back to FP32 elsewhere.
Question 7: In TensorRT, what is the significance of the 'opt' shape in an IOptimizationProfile compared to 'min' and 'max'?
- The opt shape is the most common expected input size and is used as the primary target for kernel auto-tuning (Correct answer)
- The opt shape is the maximum memory allocation reserved at engine creation
- The opt shape defines the shape used exclusively during INT8 calibration
- The opt shape sets a hard limit that inference inputs cannot exceed
Correct answer: The opt shape is the most common expected input size and is used as the primary target for kernel auto-tuning
TensorRT tunes kernel selection specifically for the opt shape, so choosing it to match the most frequent inference batch size yields the best real-world performance.
NCA Inference Optimization & TensorRT 3
What is 'layer fusion' in TensorRT and why does it improve performance?