NCA Multi-GPU Computing & Scaling 2

Question 1

What is the primary role of NVIDIA NVLink in a multi-GPU system compared to PCIe?

Accepted Answer

NVLink provides higher bandwidth GPU-to-GPU interconnect than PCIe

Answer

NVLink delivers significantly higher bidirectional bandwidth between GPUs than PCIe, enabling faster peer-to-peer data transfers.

Question 2

In data parallelism across multiple GPUs, which operation is performed after each training batch to synchronize model parameters?

Accepted Answer

Gradient averaging (AllReduce)

Answer

AllReduce aggregates gradients from all GPUs and distributes the averaged result so each GPU updates identically.

Question 3

Which NVIDIA software library provides optimized collective communication primitives (AllReduce, Broadcast, etc.) for multi-GPU and multi-node training?

Accepted Answer

NCCL (NVIDIA Collective Communications Library)

Answer

NCCL implements ring-based and tree-based collective operations optimized for NVIDIA GPU interconnects.

Question 4

What does 'weak scaling' measure in a multi-GPU benchmark?

Accepted Answer

Performance when problem size grows proportionally with GPU count

Answer

Weak scaling evaluates whether throughput remains constant as both problem size and GPU count increase proportionally.

Question 5

In tensor parallelism, how are transformer model layers typically partitioned across GPUs?

Accepted Answer

Weight matrices are split along one dimension so GPUs compute partial results

Answer

Tensor parallelism shards weight matrices across GPUs so each device computes a column- or row-parallel portion of the matrix multiply.

(NCA) NVIDIA-Certified Associate Practice Test

(NCA) NVIDIA-Certified Associate Practice Test

NCA Multi-GPU Computing & Scaling 2