Kernels, Thread Hierarchy & Host/Device Memory

Why GPUs, Why CUDA

CUDA is NVIDIA's platform for running general-purpose computation on GPUs, not just graphics. CPUs optimize for fast sequential execution with few powerful cores; GPUs trade that for thousands of simpler cores built for data-parallel work — the same operation applied across massive amounts of data, like matrix math.

A Kernel

__global__ void addVectors(float* a, float* b, float* c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) c[i] = a[i] + b[i];
}

// launch: gridDim blocks, each with blockDim threads
addVectors<<<numBlocks, threadsPerBlock>>>(d_a, d_b, d_c, n);

A kernel (marked __global__) runs on the GPU, executed in parallel by many threads — each thread typically computes its own index from blockIdx/threadIdx/blockDim and processes one element.

Thread Hierarchy: Thread → Block → Grid

A kernel launch specifies a grid of thread blocks, each block containing many threads. This hierarchy maps onto the GPU's physical execution units and lets the same kernel code scale across different GPU sizes without rewriting it.

Host and Device Memory

float *d_a;
cudaMalloc(&d_a, n * sizeof(float));
cudaMemcpy(d_a, h_a, n * sizeof(float), cudaMemcpyHostToDevice);
// ... launch kernel using d_a ...
cudaMemcpy(h_result, d_c, n * sizeof(float), cudaMemcpyDeviceToHost);
cudaFree(d_a);

Host (CPU) and device (GPU) typically have separate physical memory; data must be explicitly allocated and transferred with cudaMemcpy — or, more conveniently but sometimes less predictably fast, via unified memory (cudaMallocManaged) which lets the runtime migrate data automatically.