NCA Multi-GPU Computing & Scaling 2 — Questions and Answers
Question 1: What is the primary role of NVIDIA NVLink in a multi-GPU system compared to PCIe?
- NVLink provides higher bandwidth GPU-to-GPU interconnect than PCIe (Correct answer)
- NVLink replaces the CPU-to-GPU connection entirely
- NVLink is used only for power delivery between GPUs
- NVLink reduces GPU memory capacity to improve speed
Correct answer: NVLink provides higher bandwidth GPU-to-GPU interconnect than PCIe
NVLink delivers significantly higher bidirectional bandwidth between GPUs than PCIe, enabling faster peer-to-peer data transfers.
Question 2: In data parallelism across multiple GPUs, which operation is performed after each training batch to synchronize model parameters?
- Weight pruning
- Gradient averaging (AllReduce) (Correct answer)
- Activation checkpointing
- Tensor transposition
Correct answer: Gradient averaging (AllReduce)
AllReduce aggregates gradients from all GPUs and distributes the averaged result so each GPU updates identically.
Question 3: Which NVIDIA software library provides optimized collective communication primitives (AllReduce, Broadcast, etc.) for multi-GPU and multi-node training?
- cuDNN
- NCCL (NVIDIA Collective Communications Library) (Correct answer)
- cuBLAS
- TensorRT
Correct answer: NCCL (NVIDIA Collective Communications Library)
NCCL implements ring-based and tree-based collective operations optimized for NVIDIA GPU interconnects.
Question 4: What does 'weak scaling' measure in a multi-GPU benchmark?
- Performance when problem size is fixed and GPU count increases
- Performance when problem size grows proportionally with GPU count (Correct answer)
- The ratio of GPU idle time to active compute time
- Memory bandwidth utilization per GPU
Correct answer: Performance when problem size grows proportionally with GPU count
Weak scaling evaluates whether throughput remains constant as both problem size and GPU count increase proportionally.
Question 5: In tensor parallelism, how are transformer model layers typically partitioned across GPUs?
- Each GPU holds a full copy of every layer
- Weight matrices are split along one dimension so GPUs compute partial results (Correct answer)
- Different transformer blocks are assigned to different GPUs sequentially
- Attention heads are replicated identically on every GPU
Correct answer: Weight matrices are split along one dimension so GPUs compute partial results
Tensor parallelism shards weight matrices across GPUs so each device computes a column- or row-parallel portion of the matrix multiply.
Question 6: What is a key advantage of using NVSwitch in an HGX or DGX system?
- It eliminates the need for a CPU in AI inference
- It enables all-to-all GPU communication at full NVLink bandwidth without bottlenecking through a single switch port (Correct answer)
- It replaces DRAM with HBM for all GPUs simultaneously
- It provides PCIe Gen 5 lanes to each GPU independently
Correct answer: It enables all-to-all GPU communication at full NVLink bandwidth without bottlenecking through a single switch port
NVSwitch creates a non-blocking fabric so every GPU can communicate with every other GPU at full NVLink speed simultaneously.
Question 7: Which scaling strategy is most appropriate when a single GPU cannot hold the entire model due to memory constraints?
- Data parallelism
- Model parallelism (pipeline or tensor) (Correct answer)
- Batch size reduction only
- Gradient checkpointing without any parallelism
Correct answer: Model parallelism (pipeline or tensor)
Model parallelism partitions model layers or tensors across multiple GPUs, allowing models larger than a single GPU's memory to be trained.
NCA Multi-GPU Computing & Scaling 2
What is the primary role of NVIDIA NVLink in a multi-GPU system compared to PCIe?