NCA Tensor Core & Mixed Precision 4

Question 1

When does NVIDIA recommend keeping a FP32 master copy of weights during mixed precision training?

Accepted Answer

Always — to accumulate small gradient updates that FP16 cannot represent

Answer

FP16 cannot represent very small gradient updates, so a FP32 master copy accumulates updates precisely before copying back to FP16 for forward passes.

Question 2

Which of the following layer types typically does NOT benefit from Tensor Core acceleration and should stay in FP32?

Accepted Answer

Batch normalization

Answer

Batch normalization requires FP32 for numerical stability in variance computation; AMP keeps it in FP32 by default.

Question 3

What is the NVIDIA term for the end-to-end workflow combining cuBLAS, Tensor Cores, and AMP to maximize training throughput?

Accepted Answer

Automatic Mixed Precision (AMP)

Answer

NVIDIA's Automatic Mixed Precision (AMP) framework orchestrates FP16/BF16 computation, gradient scaling, and FP32 accumulation for Tensor Core throughput.

Question 4

On an A100, what peak Tensor Core throughput (TFLOPS) can be achieved for dense FP16 matrix operations?

Accepted Answer

312 TFLOPS

Answer

The A100 delivers approximately 312 TFLOPS of dense FP16 Tensor Core performance (624 TFLOPS with sparsity).

Question 5

Which CUDA compute capability version is the minimum required to use Tensor Cores?

Accepted Answer

7.0

Answer

Tensor Cores were introduced with compute capability 7.0 (Volta architecture, V100).

(NCA) NVIDIA-Certified Associate Practice Test

(NCA) NVIDIA-Certified Associate Practice Test

NCA Tensor Core & Mixed Precision 4