NCA Tensor Core & Mixed Precision 3

Question 1

Ampere Tensor Cores added support for which sparse computation technique that can double effective throughput?

Accepted Answer

2:4 structured sparsity

Answer

Ampere introduced 2:4 structured sparsity, where exactly 2 of every 4 weights are zero, doubling throughput via the Sparsity feature.

Question 2

When using GradScaler with AMP in PyTorch, what problem does it solve?

Accepted Answer

Gradient underflow (vanishing gradients) in FP16

Answer

GradScaler scales the loss before backward pass to prevent FP16 gradients from underflowing to zero during backpropagation.

Question 3

Which precision format introduced in Hopper GPUs is specifically optimized for transformer inference?

Accepted Answer

FP8

Answer

Hopper (H100) added FP8 precision (E4M3 and E5M2 variants) Tensor Core support, targeting high-throughput transformer inference.

Question 4

What does TF32 (TensorFloat-32) format use compared to standard FP32?

Accepted Answer

Same exponent, reduced mantissa (10 bits)

Answer

TF32 keeps FP32's 8-bit exponent but reduces the mantissa to 10 bits, matching FP16 precision while retaining FP32 dynamic range.

Question 5

In multi-GPU training, which NVIDIA library uses Tensor Cores for the all-reduce collective communication pattern?

Accepted Answer

NCCL

Answer

NCCL (NVIDIA Collective Communications Library) implements optimized all-reduce operations that leverage NVLink bandwidth, though compute remains on Tensor Cores via training kernels.

(NCA) NVIDIA-Certified Associate Practice Test

(NCA) NVIDIA-Certified Associate Practice Test

NCA Tensor Core & Mixed Precision 3