NCA AI & Deep Learning Frameworks 3 — Questions and Answers
Question 1: Which NVIDIA tool is used to profile GPU kernel performance and identify bottlenecks in deep learning workloads?
- NVIDIA-SMI
- Nsight Systems / Nsight Compute (Correct answer)
- cuSPARSE
- CUDA GDB
Correct answer: Nsight Systems / Nsight Compute
Nsight Systems provides system-level profiling while Nsight Compute offers detailed GPU kernel analysis for optimizing deep learning workloads.
Question 2: What is the key advantage of using JAX over standard NumPy for deep learning research?
- Built-in support for CUDA streams
- Automatic differentiation and JIT compilation via XLA for GPU/TPU acceleration (Correct answer)
- Native integration with NVIDIA cuDNN
- Support for dynamic computation graphs only
Correct answer: Automatic differentiation and JIT compilation via XLA for GPU/TPU acceleration
JAX provides `grad()` for automatic differentiation and `jit()` for XLA-based JIT compilation, enabling high-performance research on accelerators.
Question 3: In PyTorch's Distributed Data Parallel (DDP), how are gradients synchronized across GPUs?
- A parameter server collects and redistributes gradients
- All-reduce operations synchronize gradients after the backward pass (Correct answer)
- Only the master GPU computes gradients and broadcasts them
- Gradients are averaged on CPU before the optimizer step
Correct answer: All-reduce operations synchronize gradients after the backward pass
DDP uses all-reduce (typically via NCCL) to average gradients across all GPUs simultaneously, making it more efficient than parameter server approaches.
Question 4: What does NVIDIA TensorRT primarily optimize when deploying deep learning models?
- Training throughput by fusing optimizer steps
- Inference latency and throughput through layer fusion, precision calibration, and kernel auto-tuning (Correct answer)
- Multi-GPU gradient synchronization
- Data pipeline preprocessing speed
Correct answer: Inference latency and throughput through layer fusion, precision calibration, and kernel auto-tuning
TensorRT optimizes trained models for deployment by fusing layers, selecting optimal kernels, and applying INT8/FP16 quantization to maximize inference performance.
Question 5: Which framework introduced the concept of 'eager execution' as its default mode, executing operations immediately without building a static graph?
- Caffe2
- Theano
- TensorFlow 2.x (Correct answer)
- MXNet
Correct answer: TensorFlow 2.x
TensorFlow 2.x made eager execution the default, allowing operations to run immediately like NumPy, which greatly simplified debugging and experimentation.
Question 6: What is the purpose of NVIDIA's Apex library in PyTorch mixed-precision training?
- Providing distributed training primitives for multi-node clusters
- Enabling automatic mixed precision (AMP) with FP16/FP32 for faster training with less memory (Correct answer)
- Optimizing data loading pipelines for GPU transfer
- Implementing custom CUDA kernels for transformer layers
Correct answer: Enabling automatic mixed precision (AMP) with FP16/FP32 for faster training with less memory
Apex provides AMP utilities including `amp.initialize()` and `amp.scale_loss()` to use FP16 computations while maintaining FP32 master weights for stability.
Question 7: In a convolutional neural network, what does the 'receptive field' of a neuron refer to?
- The number of filters in a convolutional layer
- The region of the input image that influences a particular neuron's output (Correct answer)
- The spatial dimensions of feature maps after pooling
- The total number of trainable parameters in the network
Correct answer: The region of the input image that influences a particular neuron's output
The receptive field is the spatial extent of the input that a neuron 'sees,' which grows deeper in the network through stacked convolutions.
NCA AI & Deep Learning Frameworks 3
Which NVIDIA tool is used to profile GPU kernel performance and identify bottlenecks in deep learning workloads?