NCA Inference Optimization & TensorRT 2 — Questions and Answers
Question 1: Which TensorRT API call finalizes the network definition and produces an optimized inference engine?
- builder.build_cuda_engine(network) (Correct answer)
- network.finalize()
- runtime.deserialize_cuda_engine()
- parser.parse_from_file()
Correct answer: builder.build_cuda_engine(network)
builder.build_cuda_engine(network) compiles and optimizes the network into a TensorRT engine ready for inference.
Question 2: What does TensorRT's 'workspace size' configuration control during the engine build process?
- Maximum GPU memory the optimizer may use for algorithm selection (Correct answer)
- The size of the input tensor batch
- Amount of RAM reserved for calibration data
- Number of concurrent inference streams
Correct answer: Maximum GPU memory the optimizer may use for algorithm selection
Workspace size limits the scratch GPU memory TensorRT can allocate when evaluating candidate layer implementations during optimization.
Question 3: In TensorRT, what is the purpose of the IOptimizationProfile when using dynamic shapes?
- It defines min, opt, and max dimensions so the engine can be tuned across a range of input shapes (Correct answer)
- It sets the maximum batch size for static-shape engines
- It specifies which GPU layers are eligible for INT8 quantization
- It controls the memory pool allocated per CUDA stream
Correct answer: It defines min, opt, and max dimensions so the engine can be tuned across a range of input shapes
An IOptimizationProfile provides the min/opt/max shape bounds that TensorRT uses to auto-tune kernels for dynamic-shape networks.
Question 4: Which TensorRT precision mode typically offers the best throughput on Ampere and later GPUs while maintaining acceptable accuracy for most vision tasks?
- FP16 (Correct answer)
- FP32
- INT8
- TF32
Correct answer: FP16
FP16 leverages Tensor Core acceleration on Ampere GPUs and usually provides near-FP32 accuracy with roughly 2× throughput improvement.
Question 5: What is the role of a calibration dataset in TensorRT INT8 quantization?
- It is used to compute activation range statistics that determine the optimal quantization scale factors (Correct answer)
- It fine-tunes the model weights for lower precision
- It validates the final INT8 engine accuracy against FP32
- It sets the workspace memory limit for the builder
Correct answer: It is used to compute activation range statistics that determine the optimal quantization scale factors
INT8 calibration runs representative data through the network to collect per-layer activation distributions used to derive scale factors.
Question 6: When serializing a TensorRT engine with engine.serialize(), what format is the output?
- A binary blob (plan file) specific to the GPU and TensorRT version used during the build (Correct answer)
- A portable ONNX model file
- A JSON configuration that can be reloaded on any GPU
- A CUDA PTX assembly file
Correct answer: A binary blob (plan file) specific to the GPU and TensorRT version used during the build
Serialized TensorRT engines are binary plan files that are GPU-architecture-specific and not portable across different GPU generations or TensorRT versions.
Question 7: Which flag in TensorRT's BuilderConfig enables layer-level profiling to identify per-layer latency bottlenecks?
- PROFILING_VERBOSITY set to LAYER_NAMES_ONLY or DETAILED (Correct answer)
- STRICT_TYPES flag
- INT8 calibration mode
- MAX_WORKSPACE_SIZE flag
Correct answer: PROFILING_VERBOSITY set to LAYER_NAMES_ONLY or DETAILED
Setting ProfilingVerbosity to LAYER_NAMES_ONLY or DETAILED causes TensorRT to record per-layer timing data accessible via the IExecutionContext profiler.
NCA Inference Optimization & TensorRT 2
Which TensorRT API call finalizes the network definition and produces an optimized inference engine?