NCA Inference Optimization & TensorRT 3

Question 1

What is 'layer fusion' in TensorRT and why does it improve performance?

Accepted Answer

Combining multiple adjacent operations into a single kernel to reduce memory bandwidth and kernel launch overhead

Answer

Layer fusion merges operations like Conv+BN+ReLU into one optimized kernel, eliminating intermediate memory reads/writes and launch overhead.

Question 2

When using TensorRT's IExecutionContext for inference, which method launches asynchronous GPU execution?

Accepted Answer

execute_async_v2(bindings, stream_handle)

Answer

execute_async_v2 enqueues the inference work into a specified CUDA stream, enabling overlap with other GPU or CPU work.

Question 3

In the context of TensorRT deployment, what does 'binding' refer to?

Accepted Answer

A GPU memory buffer associated with a named network input or output tensor

Answer

Bindings are GPU device pointers mapped to each named input/output tensor that must be provided when calling execute.

Question 4

Which NVIDIA tool is commonly used to profile TensorRT engine latency and GPU utilization during inference?

Accepted Answer

nvprof (legacy) or Nsight Systems

Answer

nvprof (legacy) and its modern replacement Nsight Systems both capture CUDA kernel timelines and GPU metrics for TensorRT workloads.

Question 5

What is the primary purpose of TensorRT plugins?

Accepted Answer

To implement custom layers or operations that TensorRT does not natively support

Answer

TensorRT plugins let developers integrate custom CUDA kernels for operations outside TensorRT's built-in layer library.

(NCA) NVIDIA-Certified Associate Practice Test