NCA Multi-GPU Computing & Scaling 4

Question 1

What is 'expert parallelism' in the context of Mixture-of-Experts (MoE) models on multi-GPU systems?

Accepted Answer

Distributing different expert sub-networks across GPUs so each GPU hosts a subset of experts

Answer

Expert parallelism places different MoE expert layers on different GPUs, with a routing mechanism directing tokens to the correct GPU.

Question 2

What problem does 'activation recomputation' (gradient checkpointing) solve in multi-GPU training?

Accepted Answer

It reduces GPU memory usage by recomputing activations during the backward pass instead of storing them

Answer

Gradient checkpointing trades compute for memory by discarding intermediate activations and recomputing them during backpropagation.

Question 3

In NVIDIA's Megatron-LM framework, what combination of parallelism is referred to as '3D parallelism'?

Accepted Answer

Data parallelism + tensor parallelism + pipeline parallelism

Answer

3D parallelism combines data, tensor, and pipeline parallelism to scale training across thousands of GPUs efficiently.

Question 4

What is the significance of the 'ring-AllReduce' algorithm for multi-GPU gradient synchronization?

Accepted Answer

It arranges GPUs in a logical ring so bandwidth scales with GPU count rather than being limited by a single bottleneck

Answer

Ring-AllReduce distributes the reduction workload evenly so total communication volume is independent of the number of GPUs, maximizing bandwidth utilization.

Question 5

When using mixed-precision training (FP16/BF16 + FP32) in a multi-GPU setting, where are the FP32 master weights typically stored?

Accepted Answer

On each GPU alongside the FP16/BF16 working copies

Answer

Each GPU maintains its own FP32 master copy for numerically stable gradient accumulation while using FP16/BF16 for forward and backward compute.

(NCA) NVIDIA-Certified Associate Practice Test

(NCA) NVIDIA-Certified Associate Practice Test

NCA Multi-GPU Computing & Scaling 4