Originally designed for the video-game market and the handling of screen graphics, GPUs are massively parallel multiprocessors equipped with large fast access on-board memory. Unlike CPUs, much more silicon area is devoted to computing units than to control structures (\figurename~\ref{img:cpu-gpu}). The computing power of GPUs arises from the large number of processing cores installed on the device, rather than from the chip clock speed (as for CPUs). Graphic card devices are indeed used to execute highly parallelized tasks.\\

\begin{figure}
\centering
\def\svgwidth{\textwidth}
\input{Images/pdf/gpu_algo/Cpu-gpu-tex.pdf_tex}
\caption{Compared to the CPU, the GPU devotes more transistors to data processing \cite{Nvi11b}.}\label{img:cpu-gpu}
\end{figure}

The trigger algorithm and the test framework I will describe in Section~\ref{sect:test-data-format} use the CUDA (Compute Unified Device Architecture) toolkit\footnote{\url{http://www.nvidia.com/object/cuda_home_new.html}}. CUDA is a platform for parallel programming and computing developed by NVIDIA\footnote{\url{http://www.nvidia.com/}}, compatible with GeForce, Quadro and Tesla GPUs. This platform exposes GPUs for computing just like any usual processor, through CUDA-accelerated libraries and extensions to the most popular programming languages.\\

A set of C/\C++ libraries enables heterogeneous programming and provides straightforward APIs\footnote{Application Programming Interface} for device and memory management. GPUs can be embedded in the PC motherboard, or on the CPU die, or reside in dedicated \textit{graphics cards} connected to the host PC via PCI Express links, as in this work. Hereinafter, I will call \textit{host} the CPU and its memory, and \textit{device} the GPU. Serial functions, decorated with the \texttt{\_\_host\_\_} prefix, are coded in standard C and execute on the processor; the host can call a \texttt{\_\_device\_\_} function (\textit{kernel}) at any moment, that will run on the GPU.\\

A few definitions should be given before discussing the GPU architecture:
\begin{itemize}
\item[\textbf{Thread:}] the smallest sequence of instruction that can be run independently.%a basic element of the data to be processed.
\item[\textbf{Warp:}] the minimum \textit{work group size}, i.e. the maximum number of threads that can execute the same instruction simultaneously, in SIMD mode (\textit{Single Instruction -- Multiple Data}), within a single CUDA multiprocessor. Currently, all NVIDIA GPUs feature warps of 32 threads.
\item[\textbf{Block:}] the basic element of a GPU program. The threads of a block execute concurrently on one multiprocessor, and multiple blocks can run at the same time. New blocks are launched on the free multiprocessors, as the previous ones terminate.
\end{itemize}

The building block of the GPU architecture is the Streaming Multiprocessor (SM), which hosts a number of single-precision CUDA cores (see Appendix~\ref{app:devicequery}) executing identical sets of instructions, and a block of high-speed on-chip memory. Up to 24 warps can be active at the same time on a single SM, depending on their memory usage and on the number of registers available on the SM. %The scheduler will distribute the jobs to SM cores, and queue the instructions according to this scheme. Threads are synchronized after each warp terminates.\\
Each block of threads can be scheduled in any order on any of the available SMs, allowing for program scalability: devices with more SMs automatically outperform older GPUs, as demonstrated in \figurename~\ref{img:grid} \cite{Nvi11b}. One of the most important characteristics of the CUDA architecture, indeed, is that kernels are scalable across any number of parallel SMs to adapt to different hardware.\\

The CUDA runtime manages the number of blocks processed simultaneously by the GPU SMs, that is closely linked to the availability of hardware resources.\\

\begin{figure}
\centering
\includegraphics[width=\textwidth]{Images/pdf/gpu_algo/grid-sm}
\caption{Concurrent execution of thread blocks on devices with a different number of multiprocessors: 2 SMs (left) and 4 SMs (right) \cite{cudabasics}.}\label{img:grid}
\end{figure}

A kernel is launched with a call
\begin{lstlisting}[fontadjust]
mykernel<<<numberOfBlocks,threadsPerBlock,sharedMemorySize,streamID>>>(input* somestuff, output* someresults);
\end{lstlisting}
where:
\begin{itemize}
\item the triple angle brackets denote a call to a \textit{device} function by a \textit{host} function;
\item the kernel is only executed by the stream \texttt{streamID} (see Section~\ref{sect:streams});
\item the input parameters point to the device memory, that must be allocated before the kernel call;
\item \texttt{mykernel} is executed at the same time by a grid of \texttt{numberOfBlocks} independent blocks;
\item each block is split into \texttt{threadsPerBlock} threads that can synchronize and communicate to each other through access to the shared memory;
\item a portion of shared memory (see Section~\ref{sect:memory}) of size \texttt{sharedMemorySize} is allocated for each block.
\end{itemize}

The same kernel is executed by all the threads. Data-parallel programming maps data elements to parallel processing threads: inside the kernel function, jobs can be distributed to the blocks and split between the threads of a block by means of the built-in three-dimensional indices \texttt{blockIdx} and \texttt{threadIdx}. This provides a natural way to invoke computation across the elements of a multidimensional domain, such as matrices and vectors \cite{Nvi11b}. Individual threads executing in a warp are free to execute independently via data-dependent conditionals branches. In this case, all the branched paths are scheduled serially. Threads can be synchronized, i.e. can execute a common instruction at the same time, through dedicated CUDA functions, and automatically synchronize upon convergence to the same instruction branch.\\

%, as sketched in \figurename~\ref{img:grid}.

\subsection{CUDA memory hierarchy}\label{sect:memory}
The CUDA architecture features a hierarchy of memory spaces that are accessible in different scopes.
%CUDA threads may access multiple memory spaces. 
All threads have read/write access to a large memory space called \textit{global} (some~\si{\giga\byte}), persistent across subsequent kernel launches within the same application. A different space, residing on each Streaming Multiprocessor, is devoted to independent portions of memory -- referred to as \textit{shared memory} -- visible to all the threads of the same block. The lifetime of the shared memory coincides with the execution time of the block. Finally, each thread has access to a small private local memory space \cite{Nvi11b}.


\subsection{Streams and concurrency}\label{sect:streams}
Since the development of the Fermi\footnote{\url{http://www.nvidia.com/object/fermi-architecture.html}} architecture, NVIDIA GPUs can execute up to three \textit{streams} at the same time, allowing to concurrently manage processing and data transfers between host and GPU of different data sets.\\

A \textbf{CUDA stream} is a sequence of operations that execute in issue-order on the GPU \cite{nv-conc}. CUDA operations from different streams may run concurrently and be interleaved. As an example, the concurrent execution of a small kernel in up to four streams may use as much as possible of the GPU computing resources at once. The maximum number of kernels that a device can run at the same time is four \cite{nv-conc2}.\\

A typical example of concurrent execution of streams emerges from the possible overlapping of \textit{host to device} and \textit{device to host} memory copies and of the execution of a device kernel. We have adopted this solution, often referred to as ``3-way concurrency'', for this project. \figurename~\ref{img:concurrency} explains the concurrent execution of such three streams.

\begin{figure}
\centering
\def\svgwidth{0.5\textwidth}\input{Images/pdf/Appendices/streams_concurrency.pdf_tex}
\caption{\textit{3-way concurrency} demonstrated for a four-step program \cite{nv-conc}. At the beginning, data is copied from the host memory into the device memory (\textit{HD}). As the copy is executed, the stream is free to undertake another operation. At the second step, one stream executes the kernel (\textit{K}), and another one concurrently copies more data into the device memory. At the following step, as soon as the previous two operations are concluded, three streams can run at the same time: the first one pulls the results of the previous execution of the kernel from the device, and stores them back into the host memory (\textit{DH}); the second executes the kernel on the last data copied into the device memory; the third pushes some more data into the device memory.}\label{img:concurrency}
\end{figure}