CUDA Programming & Parallel Computing Flashcards

What programming language is commonly used for CUDA development?

Answer: CUDA C/C++

CUDA C/C++ is the most commonly used programming language for CUDA development. It extends standard C/C++ with keywords and APIs that allow developers to define and launch functions (kernels) to be executed on the GPU. This integration enables seamless interaction between CPU host code and GPU device code, making it accessible for C/C++ programmers to leverage GPU acceleration.

Which CUDA memory type is shared between threads in the same block?

Answer: Shared memory

Shared memory is a fast, on-chip memory that is explicitly managed by the programmer and shared among all threads within the same thread block. It provides very low-latency access, significantly faster than global memory, and is used for inter-thread communication and data sharing within a block. This allows threads to cooperate efficiently by sharing intermediate results without accessing slower global memory.

What is a CUDA kernel?

Answer: A GPU function executed in parallel

In CUDA, a kernel is a function written in CUDA C/C++ that is executed on the GPU device by many threads in parallel. When a kernel is launched from the CPU (host), it specifies how many threads and blocks should execute it. Each thread then executes the kernel function independently, often operating on a different piece of data.

What does occupancy mean in CUDA programming?

Answer: Ratio of active warps to max warps

Occupancy in CUDA programming refers to the ratio of the number of active warps on a Streaming Multiprocessor (SM) to the maximum number of warps that the SM can support. High occupancy indicates that the SM is effectively utilized, as it has many warps ready to execute, which helps hide memory latency and keep the execution units busy. Achieving optimal occupancy is crucial for maximizing GPU performance.

Which CUDA API function launches kernels?

Answer: kernel>>

In CUDA, kernels are launched using a special syntax called the "execution configuration" or "chevron syntax": `kernel_name>>(arguments)`. `gridDim` specifies the dimensions of the grid of thread blocks, and `blockDim` specifies the dimensions of each thread block. This syntax is fundamental for defining how a kernel will be executed across the GPU's parallel architecture.

What is warp in CUDA terminology?

Answer: Group of 32 threads

In CUDA terminology, a warp is a fundamental unit of execution consisting of 32 threads. These 32 threads execute the same instruction in lockstep, making them highly efficient for parallel processing. The GPU hardware schedules and executes warps, not individual threads, which is a key aspect of its SIMT (Single Instruction, Multiple Thread) architecture.

Which type of memory has the highest latency in CUDA?

Answer: Global memory

Global memory has the highest latency among the commonly used CUDA memory types. It resides off-chip in the device's DRAM and is accessible by all threads across all blocks, but accessing it is significantly slower than on-chip memories like registers or shared memory. Minimizing global memory accesses and maximizing data reuse in faster memories is a critical optimization strategy in CUDA programming.

What does CUDA streams enable?

Answer: Concurrent execution

CUDA streams enable concurrent execution of multiple kernels and host-device memory transfers. A stream is a sequence of operations that execute in issue-order on the device. By using multiple streams, developers can overlap computations with data transfers or execute independent kernels simultaneously, thereby improving the overall utilization of the GPU and reducing the total execution time.

How does CUDA handle synchronization between threads?

Answer: Using __syncthreads()

In CUDA, `__syncthreads()` is a barrier synchronization primitive used to ensure that all threads within the same thread block have completed their operations up to that point before any thread proceeds further. This is essential for correctness when threads within a block need to share data via shared memory or depend on each other's results. It does not synchronize threads across different blocks.