NCA Multi-GPU Computing & Scaling 4 — Questions and Answers
Question 1: What is 'expert parallelism' in the context of Mixture-of-Experts (MoE) models on multi-GPU systems?
- Distributing different expert sub-networks across GPUs so each GPU hosts a subset of experts (Correct answer)
- Assigning expert human reviewers to validate GPU outputs
- Running the same expert network on all GPUs simultaneously for redundancy
- Using specialized GPUs with higher memory bandwidth for attention layers only
Correct answer: Distributing different expert sub-networks across GPUs so each GPU hosts a subset of experts
Expert parallelism places different MoE expert layers on different GPUs, with a routing mechanism directing tokens to the correct GPU.
Question 2: What problem does 'activation recomputation' (gradient checkpointing) solve in multi-GPU training?
- It reduces GPU memory usage by recomputing activations during the backward pass instead of storing them (Correct answer)
- It eliminates gradient divergence between GPU ranks
- It speeds up AllReduce by caching intermediate gradients
- It prevents NaN loss values during mixed-precision training
Correct answer: It reduces GPU memory usage by recomputing activations during the backward pass instead of storing them
Gradient checkpointing trades compute for memory by discarding intermediate activations and recomputing them during backpropagation.
Question 3: In NVIDIA's Megatron-LM framework, what combination of parallelism is referred to as '3D parallelism'?
- Data parallelism + tensor parallelism + pipeline parallelism (Correct answer)
- Spatial parallelism + temporal parallelism + depth parallelism
- Model parallelism + expert parallelism + quantization
- CPU parallelism + GPU parallelism + NVLink parallelism
Correct answer: Data parallelism + tensor parallelism + pipeline parallelism
3D parallelism combines data, tensor, and pipeline parallelism to scale training across thousands of GPUs efficiently.
Question 4: What is the significance of the 'ring-AllReduce' algorithm for multi-GPU gradient synchronization?
- It arranges GPUs in a logical ring so bandwidth scales with GPU count rather than being limited by a single bottleneck (Correct answer)
- It uses a central parameter server to aggregate all gradients
- It sends gradients directly to the CPU for averaging across nodes
- It reduces GPU memory by storing only ring-buffer segments of gradients
Correct answer: It arranges GPUs in a logical ring so bandwidth scales with GPU count rather than being limited by a single bottleneck
Ring-AllReduce distributes the reduction workload evenly so total communication volume is independent of the number of GPUs, maximizing bandwidth utilization.
Question 5: When using mixed-precision training (FP16/BF16 + FP32) in a multi-GPU setting, where are the FP32 master weights typically stored?
- Only on GPU 0 as a parameter server
- On each GPU alongside the FP16/BF16 working copies (Correct answer)
- Exclusively in CPU RAM to save GPU memory
- Distributed across NVLink cache with no full copy on any GPU
Correct answer: On each GPU alongside the FP16/BF16 working copies
Each GPU maintains its own FP32 master copy for numerically stable gradient accumulation while using FP16/BF16 for forward and backward compute.
Question 6: What is the primary purpose of 'sequence parallelism' in transformer model training?
- To distribute the sequence (token) dimension of activations across GPUs to reduce per-GPU memory (Correct answer)
- To process multiple independent sentences in parallel with data parallelism
- To pipeline sequence-to-sequence translation across GPU stages
- To replicate the entire sequence on each GPU for faster attention computation
Correct answer: To distribute the sequence (token) dimension of activations across GPUs to reduce per-GPU memory
Sequence parallelism splits long activation tensors along the sequence length dimension across GPUs, reducing the per-GPU memory needed for long-context models.
Question 7: Which NVIDIA tool is used to profile collective communication operations and identify AllReduce bottlenecks in multi-GPU training?
- Nsight Systems (Correct answer)
- CUDA Debugger (cuda-gdb)
- cuSolver
- TensorRT Profiler
Correct answer: Nsight Systems
Nsight Systems provides timeline views of NCCL operations, GPU kernels, and CPU activity, making it possible to identify AllReduce stalls and overlap inefficiencies.
NCA Multi-GPU Computing & Scaling 4
What is 'expert parallelism' in the context of Mixture-of-Experts (MoE) models on multi-GPU systems?