NCA Tensor Core & Mixed Precision 5 — Questions and Answers
Question 1: In quantization-aware training (QAT), how do Tensor Cores enable INT8 inference after FP16/FP32 training?
- INT8 weights are upcast to FP16 before Tensor Core operations
- Tensor Cores support INT8 GEMM natively, allowing trained FP32 models to be calibrated and deployed in INT8 (Correct answer)
- QAT requires CPU-based quantization post-training
- INT8 inference bypasses Tensor Cores entirely
Correct answer: Tensor Cores support INT8 GEMM natively, allowing trained FP32 models to be calibrated and deployed in INT8
Turing and later Tensor Cores support INT8 GEMM operations, so QAT calibrates scale factors allowing FP32-trained models to run natively in INT8 on Tensor Cores.
Question 2: What is 'loss scaling' and why is it critical specifically for FP16 Tensor Core training?
- Dividing the learning rate by the loss value to stabilize convergence
- Multiplying the loss by a large scalar so small gradients don't underflow to zero in FP16 (Correct answer)
- Normalizing loss across multiple GPUs
- Scaling weights by the loss magnitude each epoch
Correct answer: Multiplying the loss by a large scalar so small gradients don't underflow to zero in FP16
FP16's minimum positive value (~6×10⁻⁸) can cause gradient underflow, so the loss is scaled up before backward pass to shift gradients into representable FP16 range.
Question 3: Which metric best indicates that Tensor Cores are being heavily utilized in a training workload?
- High GPU memory bandwidth utilization
- High sm__pipe_tensor_cycles_active percentage in Nsight Compute (Correct answer)
- High PCIe transfer rate
- High SM occupancy alone
Correct answer: High sm__pipe_tensor_cycles_active percentage in Nsight Compute
sm__pipe_tensor_cycles_active measures the fraction of cycles where Tensor Core pipelines are active, directly indicating utilization.
Question 4: Why does using a batch size of 32 instead of 33 often improve Tensor Core efficiency?
- Odd batch sizes cause CUDA kernel launch errors
- 32 is a multiple of 8, aligning the batch dimension with Tensor Core tile requirements (Correct answer)
- The OS scheduler prefers power-of-two process counts
- cuDNN only supports even batch sizes
Correct answer: 32 is a multiple of 8, aligning the batch dimension with Tensor Core tile requirements
Tensor Core tiles require dimensions aligned to multiples of 8 (or 16); batch size 32 aligns perfectly while 33 requires padding that wastes cycles.
Question 5: On Hopper H100, what is the purpose of the Transformer Engine that builds on Tensor Cores?
- A separate CPU coprocessor for transformer pre-processing
- Hardware and software that automatically selects FP8/FP16/BF16 per layer to maximize throughput with minimal accuracy loss (Correct answer)
- A fixed-function unit for tokenization and embedding lookup
- A scheduler that routes transformer workloads to specific SMs
Correct answer: Hardware and software that automatically selects FP8/FP16/BF16 per layer to maximize throughput with minimal accuracy loss
H100's Transformer Engine dynamically selects the optimal precision (FP8, FP16, BF16) per transformer layer while maintaining accuracy via scaling factors.
Question 6: In a convolutional neural network, which dimension alignment most critically affects Tensor Core utilization for NHWC input tensors?
- N (batch)
- H (height)
- W (width)
- C (channels) (Correct answer)
Correct answer: C (channels)
For NHWC convolutions, the channel dimension C must be a multiple of 8 (FP16) or 16 (INT8) for Tensor Core tiles to align without padding waste.
Question 7: Which NVIDIA container provides a pre-configured environment with AMP, cuBLAS, and cuDNN optimized for Tensor Core training?
- NVIDIA Driver Container
- NGC Deep Learning Containers (e.g., nvcr.io/nvidia/pytorch) (Correct answer)
- CUDA Base Image
- NVIDIA GRID vGPU Container
Correct answer: NGC Deep Learning Containers (e.g., nvcr.io/nvidia/pytorch)
NGC (NVIDIA GPU Cloud) Deep Learning containers ship with optimized cuBLAS, cuDNN, and framework AMP support pre-configured for Tensor Core workloads.
NCA Tensor Core & Mixed Precision 5
In quantization-aware training (QAT), how do Tensor Cores enable INT8 inference after FP16/FP32 training?