NCA CUDA Programming & Parallel Computing — Questions and Answers

Question 1: What programming language is commonly used for CUDA development?

Python only.
CUDA C/C++ (Correct answer)
Java.
Fortran.

Correct answer: CUDA C/C++

CUDA C/C++ is the most commonly used programming language for CUDA development. It extends standard C/C++ with keywords and APIs that allow developers to define and launch functions (kernels) to be executed on the GPU. This integration enables seamless interaction between CPU host code and GPU device code, making it accessible for C/C++ programmers to leverage GPU acceleration.

Question 2: Which CUDA memory type is shared between threads in the same block?

Global memory.
Shared memory (Correct answer)
Constant memory.
Register memory.

Correct answer: Shared memory

Shared memory is a fast, on-chip memory that is explicitly managed by the programmer and shared among all threads within the same thread block. It provides very low-latency access, significantly faster than global memory, and is used for inter-thread communication and data sharing within a block. This allows threads to cooperate efficiently by sharing intermediate results without accessing slower global memory.

Question 3: What is a CUDA kernel?

A CPU function.
A GPU function executed in parallel (Correct answer)
A memory allocation routine.
A graphics shader.

Correct answer: A GPU function executed in parallel

In CUDA, a kernel is a function written in CUDA C/C++ that is executed on the GPU device by many threads in parallel. When a kernel is launched from the CPU (host), it specifies how many threads and blocks should execute it. Each thread then executes the kernel function independently, often operating on a different piece of data.

Question 4: What does occupancy mean in CUDA programming?

The percentage of GPU clock speed.
Ratio of active warps to max warps (Correct answer)
The amount of memory used.
Number of threads per block.

Correct answer: Ratio of active warps to max warps

Occupancy in CUDA programming refers to the ratio of the number of active warps on a Streaming Multiprocessor (SM) to the maximum number of warps that the SM can support. High occupancy indicates that the SM is effectively utilized, as it has many warps ready to execute, which helps hide memory latency and keep the execution units busy. Achieving optimal occupancy is crucial for maximizing GPU performance.

Question 5: Which CUDA API function launches kernels?

cudaLaunch().
kernel<<<grid, block>>> (Correct answer)
cudaStart().
kernelLaunch() function.

Correct answer: kernel<<<grid, block>>>

In CUDA, kernels are launched using a special syntax called the "execution configuration" or "chevron syntax": `kernel_name<<<gridDim, blockDim>>>(arguments)`. `gridDim` specifies the dimensions of the grid of thread blocks, and `blockDim` specifies the dimensions of each thread block. This syntax is fundamental for defining how a kernel will be executed across the GPU's parallel architecture.

Question 6: What is warp in CUDA terminology?

A single thread.
Group of 32 threads (Correct answer)
A GPU core.
A block of memory.

Correct answer: Group of 32 threads

In CUDA terminology, a warp is a fundamental unit of execution consisting of 32 threads. These 32 threads execute the same instruction in lockstep, making them highly efficient for parallel processing. The GPU hardware schedules and executes warps, not individual threads, which is a key aspect of its SIMT (Single Instruction, Multiple Thread) architecture.

Question 7: Which type of memory has the highest latency in CUDA?

Register memory.
Shared memory.
Global memory (Correct answer)
Texture memory.

Correct answer: Global memory

Global memory has the highest latency among the commonly used CUDA memory types. It resides off-chip in the device's DRAM and is accessible by all threads across all blocks, but accessing it is significantly slower than on-chip memories like registers or shared memory. Minimizing global memory accesses and maximizing data reuse in faster memories is a critical optimization strategy in CUDA programming.

Question 8: What does CUDA streams enable?

Sequential execution only.
Concurrent execution (Correct answer)
Memory allocation.
Kernel compilation.

Correct answer: Concurrent execution

CUDA streams enable concurrent execution of multiple kernels and host-device memory transfers. A stream is a sequence of operations that execute in issue-order on the device. By using multiple streams, developers can overlap computations with data transfers or execute independent kernels simultaneously, thereby improving the overall utilization of the GPU and reducing the total execution time.

Question 9: How does CUDA handle synchronization between threads?

Using mutexes only.
Using __syncthreads() (Correct answer)
Threads run independently.
No synchronization is possible.

Correct answer: Using __syncthreads()

In CUDA, `__syncthreads()` is a barrier synchronization primitive used to ensure that all threads within the same thread block have completed their operations up to that point before any thread proceeds further. This is essential for correctness when threads within a block need to share data via shared memory or depend on each other's results. It does not synchronize threads across different blocks.

NCA CUDA Programming & Parallel Computing

What programming language is commonly used for CUDA development?