NCA Multi-GPU Computing & Scaling 5

Question 1

What is 'ZeRO' (Zero Redundancy Optimizer) and which parallel training problem does it primarily address?

Accepted Answer

A memory optimization technique that partitions optimizer states, gradients, and parameters across data-parallel ranks

Answer

ZeRO (from DeepSpeed) eliminates memory redundancy in data-parallel training by sharding optimizer states, gradients, and optionally parameters across GPUs.

Question 2

In a multi-GPU inference deployment, what technique reduces latency by splitting a single large request's tensor operations across multiple GPUs simultaneously?

Accepted Answer

Tensor parallelism for inference

Answer

Tensor parallelism for inference partitions weight matrices across GPUs so a single request's matrix multiplications run in parallel, reducing per-request latency.

Question 3

What is the effect of increasing the number of GPUs in a data-parallel setup if the per-GPU batch size is kept constant?

Accepted Answer

Effective global batch size increases, which may require learning rate scaling

Answer

Keeping per-GPU batch size fixed while adding GPUs scales the global batch size, which typically requires linear or square-root learning rate scaling to maintain convergence.

Question 4

What does GPUDirect RDMA enable in a multi-node GPU cluster?

Accepted Answer

Direct data transfer between a GPU and a remote network adapter (InfiniBand/RoCE) without CPU involvement

Answer

GPUDirect RDMA allows network adapters to directly read from and write to GPU memory, removing CPU copies from the inter-node communication path.

Question 5

When training on multiple nodes with InfiniBand networking, which software layer typically sits between NCCL and the physical network to provide RDMA-capable transport?

Accepted Answer

UCX (Unified Communication X)

Answer

UCX is a communication framework that NCCL and MPI use to abstract RDMA transports like InfiniBand verbs and RoCE, enabling high-performance multi-node collectives.

(NCA) NVIDIA-Certified Associate Practice Test

(NCA) NVIDIA-Certified Associate Practice Test

NCA Multi-GPU Computing & Scaling 5