NCA Multi-GPU Computing & Scaling 5 — Questions and Answers
Question 1: What is 'ZeRO' (Zero Redundancy Optimizer) and which parallel training problem does it primarily address?
- A loss-scaling algorithm that prevents gradient underflow in FP16 training
- A memory optimization technique that partitions optimizer states, gradients, and parameters across data-parallel ranks (Correct answer)
- A communication protocol replacing NCCL for multi-node clusters
- A GPU power management policy that zeros idle GPU clocks
Correct answer: A memory optimization technique that partitions optimizer states, gradients, and parameters across data-parallel ranks
ZeRO (from DeepSpeed) eliminates memory redundancy in data-parallel training by sharding optimizer states, gradients, and optionally parameters across GPUs.
Question 2: In a multi-GPU inference deployment, what technique reduces latency by splitting a single large request's tensor operations across multiple GPUs simultaneously?
- Batched inference with data parallelism
- Tensor parallelism for inference (Correct answer)
- Pipeline parallelism with micro-batch scheduling
- Post-training quantization
Correct answer: Tensor parallelism for inference
Tensor parallelism for inference partitions weight matrices across GPUs so a single request's matrix multiplications run in parallel, reducing per-request latency.
Question 3: What is the effect of increasing the number of GPUs in a data-parallel setup if the per-GPU batch size is kept constant?
- Effective global batch size increases, which may require learning rate scaling (Correct answer)
- Model accuracy automatically improves proportionally
- Per-GPU memory usage decreases because gradients are shared
- Training throughput stays flat because AllReduce becomes the bottleneck immediately
Correct answer: Effective global batch size increases, which may require learning rate scaling
Keeping per-GPU batch size fixed while adding GPUs scales the global batch size, which typically requires linear or square-root learning rate scaling to maintain convergence.
Question 4: What does GPUDirect RDMA enable in a multi-node GPU cluster?
- Direct data transfer between a GPU and a remote network adapter (InfiniBand/RoCE) without CPU involvement (Correct answer)
- GPU-to-GPU communication exclusively over PCIe within a single node
- Automatic NVLink topology configuration across nodes
- Remote access to GPU kernels via the network for distributed debugging
Correct answer: Direct data transfer between a GPU and a remote network adapter (InfiniBand/RoCE) without CPU involvement
GPUDirect RDMA allows network adapters to directly read from and write to GPU memory, removing CPU copies from the inter-node communication path.
Question 5: When training on multiple nodes with InfiniBand networking, which software layer typically sits between NCCL and the physical network to provide RDMA-capable transport?
- OpenMPI
- UCX (Unified Communication X) (Correct answer)
- CUDA Streams
- cuDNN backend
Correct answer: UCX (Unified Communication X)
UCX is a communication framework that NCCL and MPI use to abstract RDMA transports like InfiniBand verbs and RoCE, enabling high-performance multi-node collectives.
Question 6: In large-scale multi-GPU training, what is the purpose of 'overlap of communication and computation'?
- To run AllReduce on gradients of earlier layers while later layers are still computing their forward pass (Correct answer)
- To send model weights to CPU while GPUs compute loss functions
- To overlap disk I/O with GPU kernel launches for faster checkpoint loading
- To pipeline batch preprocessing on the CPU while the GPU runs inference
Correct answer: To run AllReduce on gradients of earlier layers while later layers are still computing their forward pass
Overlapping AllReduce with backward computation hides communication latency by transmitting ready gradients while remaining backward kernels continue on the GPU.
Question 7: What is a key limitation of naive pipeline parallelism that the '1F1B' (one-forward-one-backward) schedule addresses?
- Excessive NVLink bandwidth use during the fill phase
- High memory usage from storing all micro-batch activations simultaneously during the fill phase (Correct answer)
- Inability to use FP16 precision across pipeline stages
- Gradient divergence caused by different micro-batch orderings
Correct answer: High memory usage from storing all micro-batch activations simultaneously during the fill phase
1F1B interleaves forward and backward passes of micro-batches so that activations from completed backward passes are freed before new forward passes accumulate them, bounding peak memory.
NCA Multi-GPU Computing & Scaling 5
What is 'ZeRO' (Zero Redundancy Optimizer) and which parallel training problem does it primarily address?