NCA Tensor Core & Mixed Precision 4 — Questions and Answers
Question 1: When does NVIDIA recommend keeping a FP32 master copy of weights during mixed precision training?
- Never — FP16 weights are sufficient for all cases
- Always — to accumulate small gradient updates that FP16 cannot represent (Correct answer)
- Only for the first epoch
- Only when batch size exceeds 256
Correct answer: Always — to accumulate small gradient updates that FP16 cannot represent
FP16 cannot represent very small gradient updates, so a FP32 master copy accumulates updates precisely before copying back to FP16 for forward passes.
Question 2: Which of the following layer types typically does NOT benefit from Tensor Core acceleration and should stay in FP32?
- Linear layers with large matrix multiplications
- Batch normalization (Correct answer)
- Convolutional layers with aligned dimensions
- Attention score computation
Correct answer: Batch normalization
Batch normalization requires FP32 for numerical stability in variance computation; AMP keeps it in FP32 by default.
Question 3: What is the NVIDIA term for the end-to-end workflow combining cuBLAS, Tensor Cores, and AMP to maximize training throughput?
- RAPIDS AI
- Deep Learning Accelerator (DLA)
- Automatic Mixed Precision (AMP) (Correct answer)
- Multi-Process Service (MPS)
Correct answer: Automatic Mixed Precision (AMP)
NVIDIA's Automatic Mixed Precision (AMP) framework orchestrates FP16/BF16 computation, gradient scaling, and FP32 accumulation for Tensor Core throughput.
Question 4: On an A100, what peak Tensor Core throughput (TFLOPS) can be achieved for dense FP16 matrix operations?
- 19.5 TFLOPS
- 77.6 TFLOPS
- 312 TFLOPS (Correct answer)
- 624 TFLOPS
Correct answer: 312 TFLOPS
The A100 delivers approximately 312 TFLOPS of dense FP16 Tensor Core performance (624 TFLOPS with sparsity).
Question 5: Which CUDA compute capability version is the minimum required to use Tensor Cores?
- 6.0
- 7.0 (Correct answer)
- 8.0
- 5.3
Correct answer: 7.0
Tensor Cores were introduced with compute capability 7.0 (Volta architecture, V100).
Question 6: In FlashAttention, how do Tensor Cores contribute to efficiency improvements in transformer self-attention?
- By skipping the softmax computation entirely
- By tiling QK^T and V multiplications to maximize Tensor Core utilization while minimizing HBM accesses (Correct answer)
- By storing attention weights in registers instead of HBM
- By converting attention to a sparse matrix format
Correct answer: By tiling QK^T and V multiplications to maximize Tensor Core utilization while minimizing HBM accesses
FlashAttention tiles the attention computation into blocks sized for Tensor Core operations, reducing HBM reads/writes while maximizing arithmetic intensity.
Question 7: What happens when torch.backends.cuda.matmul.allow_tf32 = True is set on an Ampere GPU?
- All matrix multiplications use FP64 accumulation
- Matrix multiplications use TF32 Tensor Cores instead of FP32 CUDA cores (Correct answer)
- Only convolutional layers switch to TF32
- The setting is ignored on Ampere; only Hopper supports it
Correct answer: Matrix multiplications use TF32 Tensor Cores instead of FP32 CUDA cores
Setting allow_tf32=True lets PyTorch dispatch FP32 matrix multiplications to TF32 Tensor Cores on Ampere, trading minor precision for significant speedup.
NCA Tensor Core & Mixed Precision 4
When does NVIDIA recommend keeping a FP32 master copy of weights during mixed precision training?