NCA Multi-GPU Computing & Scaling 3

Question 1

In pipeline parallelism, what is a 'micro-batch' used for?

Accepted Answer

A smaller batch processed by a single GPU layer to keep all pipeline stages busy

Answer

Splitting a mini-batch into micro-batches allows pipeline stages to overlap computation and reduces the pipeline bubble.

Question 2

What does the term 'pipeline bubble' refer to in multi-GPU pipeline parallelism?

Accepted Answer

Idle time at pipeline stages during forward and backward pass transitions

Answer

A pipeline bubble is unavoidable idle time that occurs at the start and end of each mini-batch pass when some stages have no work to do.

Question 3

Which NVIDIA framework feature allows automatic parallelization of PyTorch models across multiple GPUs without manual layer partitioning?

Accepted Answer

torch.nn.parallel.DistributedDataParallel (DDP)

Answer

DDP wraps a model and handles gradient synchronization across GPUs automatically using NCCL as the backend.

Question 4

What is the purpose of gradient compression techniques (e.g., PowerSGD) in large-scale multi-GPU training?

Accepted Answer

To reduce the size of gradient tensors communicated during AllReduce and lower communication overhead

Answer

Gradient compression approximates gradient tensors with lower-rank representations, reducing the communication bandwidth needed for AllReduce.

Question 5

In a multi-node multi-GPU setup, what is the function of a 'rendezvous' backend (e.g., etcd, Redis) in PyTorch distributed training?

Accepted Answer

It coordinates process discovery so all ranks can find each other at startup

Answer

The rendezvous backend provides a shared key-value store that distributed processes use to discover peers and synchronize initialization.

(NCA) NVIDIA-Certified Associate Practice Test