NCA Inference Optimization & TensorRT 4 — Questions and Answers
Question 1: Which TensorRT feature allows different network segments to run at different precisions (e.g., some layers in INT8, others in FP16)?
- Mixed-precision execution via per-layer precision constraints (Correct answer)
- Separate engine compilation per precision mode
- Precision cascading through nested engine calls
- cuDNN mixed-mode fallback
Correct answer: Mixed-precision execution via per-layer precision constraints
TensorRT supports setting per-layer precision and data-type constraints, enabling a single engine to mix INT8, FP16, and FP32 layers.
Question 2: What is the recommended approach to converting a PyTorch model for TensorRT deployment?
- Export to ONNX first, then parse with TensorRT's ONNX parser (Correct answer)
- Use torch.jit.trace and pass the ScriptModule directly to TensorRT
- Rewrite the model in CUDA C++ before importing into TensorRT
- Save model weights as NumPy arrays and load them via the TensorRT network API
Correct answer: Export to ONNX first, then parse with TensorRT's ONNX parser
The standard workflow is PyTorch → ONNX export → TensorRT ONNX parser, which preserves the computational graph in an interoperable format.
Question 3: How does TensorRT handle operations not supported by its ONNX parser during model import?
- It raises an error and requires the user to implement a custom TensorRT plugin for that operation (Correct answer)
- It silently skips unsupported operations and continues parsing
- It automatically converts unsupported ops to the nearest equivalent built-in layer
- It falls back to running unsupported ops on the CPU
Correct answer: It raises an error and requires the user to implement a custom TensorRT plugin for that operation
Unsupported ONNX ops cause a parse failure; the developer must register a TensorRT plugin that implements the missing operation on the GPU.
Question 4: What is 'kernel auto-tuning' in TensorRT?
- During engine build, TensorRT benchmarks multiple CUDA kernel implementations for each layer and selects the fastest for the target GPU (Correct answer)
- Automatic rewriting of CUDA kernels to reduce register usage at runtime
- Dynamic kernel recompilation triggered when input shapes change during inference
- Selection of cuDNN algorithms at runtime based on live GPU load
Correct answer: During engine build, TensorRT benchmarks multiple CUDA kernel implementations for each layer and selects the fastest for the target GPU
TensorRT's build phase benchmarks candidate kernel implementations for each layer on the actual target GPU and permanently selects the fastest one.
Question 5: Which scenario best justifies using TensorRT's 'strongly typed' mode rather than default precision selection?
- When specific layers must remain in FP32 for numerical correctness and automatic precision selection would convert them (Correct answer)
- When the model contains only convolutional layers and FP16 is always acceptable
- When targeting INT8-only deployment on edge devices without calibration data
- When building engines for multiple GPU architectures simultaneously
Correct answer: When specific layers must remain in FP32 for numerical correctness and automatic precision selection would convert them
Strongly typed mode forces TensorRT to respect explicit per-layer type constraints, preventing automatic precision downcasting for numerically sensitive layers.
Question 6: What is the purpose of 'trtexec', the command-line tool included with TensorRT?
- Benchmarking inference throughput/latency and building/testing engines without writing code (Correct answer)
- Compiling CUDA kernels from ONNX models directly to PTX
- Managing multiple TensorRT engines across a cluster of GPU servers
- Converting TensorRT plan files to ONNX for framework re-import
Correct answer: Benchmarking inference throughput/latency and building/testing engines without writing code
trtexec is NVIDIA's reference benchmarking tool that builds TensorRT engines from ONNX models and measures throughput and latency with configurable precision and batch sizes.
Question 7: In multi-stream TensorRT inference, what is the main benefit of using multiple CUDA streams?
- Overlapping data transfers and kernel execution across batches to maximize GPU utilization (Correct answer)
- Distributing a single inference request across multiple GPU SMs simultaneously
- Enabling the same engine to run on multiple GPUs without modification
- Reducing INT8 quantization error by averaging results across streams
Correct answer: Overlapping data transfers and kernel execution across batches to maximize GPU utilization
Multiple CUDA streams allow prefetching next-batch data while the GPU executes inference on the current batch, hiding PCIe transfer latency and improving throughput.
NCA Inference Optimization & TensorRT 4
Which TensorRT feature allows different network segments to run at different precisions (e.g., some layers in INT8, others in FP16)?