NCA Multi-GPU Computing & Scaling 3 — Questions and Answers
Question 1: In pipeline parallelism, what is a 'micro-batch' used for?
- A smaller batch processed by a single GPU layer to keep all pipeline stages busy (Correct answer)
- A batch of data sent directly to the CPU for preprocessing
- A compressed representation of gradients shared via NCCL
- A unit of memory allocated per NVLink lane
Correct answer: A smaller batch processed by a single GPU layer to keep all pipeline stages busy
Splitting a mini-batch into micro-batches allows pipeline stages to overlap computation and reduces the pipeline bubble.
Question 2: What does the term 'pipeline bubble' refer to in multi-GPU pipeline parallelism?
- Network congestion between GPU nodes
- Idle time at pipeline stages during forward and backward pass transitions (Correct answer)
- Memory overflow when activations exceed GPU VRAM
- The latency introduced by NVLink arbitration
Correct answer: Idle time at pipeline stages during forward and backward pass transitions
A pipeline bubble is unavoidable idle time that occurs at the start and end of each mini-batch pass when some stages have no work to do.
Question 3: Which NVIDIA framework feature allows automatic parallelization of PyTorch models across multiple GPUs without manual layer partitioning?
- CUDA Streams
- torch.nn.parallel.DistributedDataParallel (DDP) (Correct answer)
- cuSPARSE
- NCCL Broadcast only
Correct answer: torch.nn.parallel.DistributedDataParallel (DDP)
DDP wraps a model and handles gradient synchronization across GPUs automatically using NCCL as the backend.
Question 4: What is the purpose of gradient compression techniques (e.g., PowerSGD) in large-scale multi-GPU training?
- To reduce the size of gradient tensors communicated during AllReduce and lower communication overhead (Correct answer)
- To increase the learning rate dynamically based on GPU count
- To prune low-magnitude weights from the model during training
- To convert gradients from FP32 to INT8 for inference
Correct answer: To reduce the size of gradient tensors communicated during AllReduce and lower communication overhead
Gradient compression approximates gradient tensors with lower-rank representations, reducing the communication bandwidth needed for AllReduce.
Question 5: In a multi-node multi-GPU setup, what is the function of a 'rendezvous' backend (e.g., etcd, Redis) in PyTorch distributed training?
- It stores model checkpoints across nodes
- It coordinates process discovery so all ranks can find each other at startup (Correct answer)
- It compresses gradients before AllReduce
- It manages GPU power states across the cluster
Correct answer: It coordinates process discovery so all ranks can find each other at startup
The rendezvous backend provides a shared key-value store that distributed processes use to discover peers and synchronize initialization.
Question 6: What NVIDIA GPU feature enables peer-to-peer memory access between GPUs on the same node without routing data through the CPU?
- Unified Virtual Addressing (UVA) with GPUDirect P2P (Correct answer)
- CUDA Managed Memory
- NVLink Compression
- ECC Memory
Correct answer: Unified Virtual Addressing (UVA) with GPUDirect P2P
GPUDirect Peer-to-Peer allows GPUs to directly read and write each other's memory over PCIe or NVLink, bypassing the CPU and system memory.
Question 7: Which metric best quantifies communication efficiency in an AllReduce operation across multiple GPUs?
- Peak TFLOPS per GPU
- Bus bandwidth utilization relative to theoretical peak interconnect bandwidth (Correct answer)
- GPU memory clock frequency
- Number of CUDA cores per GPU
Correct answer: Bus bandwidth utilization relative to theoretical peak interconnect bandwidth
AllReduce efficiency is measured by how close the actual communication throughput is to the theoretical peak bandwidth of the interconnect (NVLink or PCIe).
NCA Multi-GPU Computing & Scaling 3
In pipeline parallelism, what is a 'micro-batch' used for?