\subsection{Test framework, data format and input}\label{sect:test-data-format}
A framework was built by F. Pantaleo \cite{gtcfelice}, that consists of a multi-threaded software platform designed in order to optimize the execution latency of a given set of instructions. The platform executes on the CPU as host for GPU-based applications.\\

The main components are four:
\begin{itemize}
\item Network communication manager
\item Process scheduler
\item Device kernel (on the GPU)
\item Trigger monitor
\end{itemize}

Network communication is handled by means of a modified API for the ``Direct NIC Access'' (DNA) driver\footnote{\url{http://www.ntop.org/products/pf_ring/dna/}}. This driver represents an alternative to the usual ethernet drivers, and it exposes the memory buffers stored on the Network Interface Controller (NIC) board to the user space. This way, data can be accessed directly from the NIC buffers, without the need of copying them to the host RAM first: this results in a lower and much more stable latency.\\

This part of the framework allows the user to use a large number of network interfaces. A thread is spawned for each interface, to handle the network communication.\\

%The framework includes a scheduler in charge of handling a smart queue, managing multiple ``producer'' and asynchronous ``consumer'' processes. In fact, data produced by those threads running the network communication is accumulated by independent threads. The buffer is copied to the device memory only when a predefined size is reached or when a ``possible timeout'' flag is raised. Here, the GPU kernel (that is the space where the trigger function resides) is executed, and the results are copied back to the host memory. The full pipeline is managed by three independent software threads.\\
The framework includes a scheduler in charge of handling a smart queue, managing multiple ``producer'' and asynchronous ``consumer'' processes. Data produced by threads running the network communication is accumulated in host memory buffers. Each buffer is copied to the device memory only when a predefined size is reached or when a ``timeout'' flag is raised: in order to sustain the throughput, the host waits for a good number of MTP packets within a given maximum time window. After that, the GPU kernel (where the trigger algorithm resides) is executed, and the results are copied back to the host memory. The full pipeline is managed by three independent software threads.\\

Finally, information about the status of the network communication and, possibly, real-time trigger results are displayed by a monitor refreshing every 5~\si{\second}. The data shown include the number of UDP\footnote{User Datagram Protocol} packets read from each interface, the saturation of the network links and the number of MTPs processed by the GPU.\\%GPU Multi-Trigger Primitives (GMTP) processed.\\

%Like some of the sub-detectors, the firmware of the readout system for NA62 is not completely ready yet, as its design needs to match the detectors output. 
The firmware of the readout system for NA62 is still under development. 
For the tests reported in 	\chaptername~\ref{ch:tests} we assumed that the TEL62 boards reading out the signals from the photomultipliers of the RICH will also take care of the conversion between the PMT channel IDs and their 8-\si{\bit} coordinates. A simple look-up table will be implemented for this purpose on the board firmware.\\

I customized the Montecarlo simulation and event reconstruction framework provided by NA62 to prepare six different sets of data:
\begin{itemize}
\item $K^+ \rightarrow \pi^+ \nu \bar{\nu}$ events selected with the standard \textit{signal} and \textit{L0} cuts;
\item $K^+ \rightarrow \pi^+ \nu \bar{\nu}$ events selected with the standard \textit{signal} cuts and relaxed hit multiplicity $5 \leq n \leq 64$ on the RICH;
\item $K^+ \rightarrow \pi^+ \pi^0$ events selected with the standard \textit{L0} cuts;
\item $K^+ \rightarrow \pi^+ \pi^0$ events with hit multiplicity $5 \leq n \leq 64$ on the RICH;
\item $K^+ \rightarrow \pi^+ \pi^+ \pi^-$ events selected with the standard \textit{L0} cuts;
\item $K^+ \rightarrow \pi^+ \pi^+ \pi^-$ events with hit multiplicity $5 \leq n \leq 64$ on the RICH.
\end{itemize}
Each data set was divided in files containing 256 events each.\\

In addition, we supposed that the geometric correction related to the tilts of the RICH mirrors (Eqn.~\ref{eqn:correct-tilt}) would also be computed by the TEL62 boards, before data are sent to the GPU. As a consequence, the lattice introduced in Section~\ref{sect:data-format} had to be modified in order to avoid the repositioning of hits on ``phantom'' lattice vertices with negative $x$ coordinates. Therefore, the whole frame was shifted by 85 units towards the positive side of the $x$ axis, i.e. the 8-\si{\bit} coordinates were computed as
\begin{align}
&x'(x) = \text{round} \left( n_x \;\frac{x - x_{min}}{x_{max} - x_{min}} \right) + 85\label{eqn:xshift}
\end{align}
instead of using Eqns.~\ref{eqn:8bit-x} and~\ref{eqn:8bit-y}.\\

%---------------------------------------------------------\\
%
%%An \textit{online trigger} software should be left running continuously, and provide real-time calculus in correspondence to any data input.\\
%
%A framework was built by F. Pantaleo, which is triggered by UDP packets sent to the PC via ethernet. The ``TEL62'' boards in use at NA62 indeed communicate with the PC farm via UDP packets. Each packet may contain data from a fixed number of events: we chose a direct correspondence 
%\begin{align}
%&1 \mbox{ UDP packet} = 1 \mbox{ event}
%\end{align}

%The UDP packets sent to the PC by the TEL62 boards contain all the relevant data we need for the analysis. Each packet consists of a header and a data container with the structure shown in \figurename~\ref{img:dataformat}. These data are prepared by the TEL62 boards, that manipulates the timestamp, to produce UDP data packets. The UDP packets are finally sent to the NIC.\\
The UDP packets sent to the PC by the TEL62 boards contain all the relevant data we need for the analysis. Each packet consists of a header and a data container with the structure shown in \figurename~\ref{img:dataformat}. These data are prepared by the TEL62 boards as UDP packets and sent to the NIC.\\

\begin{figure}
\centering
\def\svgwidth{\textwidth}
\input{Images/pdf/gpu_algo/DataFormats-2-tex.pdf_tex}
\caption{Structure of the data files produced by the TEL62 boards. The top panel shows a GPU multi-trigger primitive packet. The bottom panel shows how data are packed for each event.}\label{img:dataformat}
\end{figure}

%----------RIFARE DA QUI IN GIU' -------------\\
%
%An experimental data structure was used for GPU-based trigger applications. During the data flow, the TEL62 boards store the events information into two files:
%\begin{itemize}
%\item a ``data'' file containing events information;
%\item a file containing one word for each event in the `0x0100\#\#\#\#' format, where \#\#\#\# is the number of words in the data file for that event.
%\end{itemize}
%The following box shows the data files structure:
%\begin{colbox}{F8E0E0}
%\begin{align*}
%&\begin{array}{ccc}
%1^{\rm st} \text{ word:}\quad & & \underbrace{\;\text{event number}\;}_{32\, \si{\bit}}\\
%2^{\rm nd} \text{ word:}\quad & & \underbrace{\;\text{event timestamp}\;}_{32\, \si{\bit}}\\
%3^{\rm rd} \text{ word:}\quad & & \underbrace{\;\text{reserved} \ll 24\;}_{8\, \si{\bit}} + \underbrace{\;\text{event fine time} \ll 16}_{8\, \si{\bit}\;} + \underbrace{\;N_{\rm hits}\;}_{16\, \si{\bit}}\\
%4^{\rm th} \text{ word:}\quad & & \underbrace{\;(\text{hit}_2).y \ll 24\;}_{8\, \si{\bit}} + \underbrace{\;(\text{hit}_2).x \ll 16}_{8\, \si{\bit}\;} + \underbrace{\;(\text{hit}_1).y \ll 8\;}_{8\, \si{\bit}} + \underbrace{\;(\text{hit}_1).\;}_{8\, \si{\bit}}\\
%\vdots & & \vdots\\
%n^{\rm th} \text{ word:}\quad & & \underbrace{\;(\text{hit}_{2n}).y \ll 24\;}_{8\, \si{\bit}} + \underbrace{\;(\text{hit}_{2n}).x \ll 16}_{8\, \si{\bit}\;} + \underbrace{\;(\text{hit}_{2n-1}).y \ll 8\;}_{8\, \si{\bit}} + \underbrace{\;(\text{hit}_{2n-1}).x\;}_{8\, \si{\bit}}
%\end{array}
%\end{align*}
%\end{colbox}
%
%If the number of hits included in one event is odd, the 16 rightmost bits of the last word are filled with zeros.
%Each line of each file consists of exactly one 32-\si{\bit} word in hexadecimal format.\\
%
%----------RIFARE DA QUI IN SU -------------\\
%

%\subsection{Memory usage}
%
%DA SCRIVERE 
%While the TEL62 board feeds data to the PC, on an event-by-event basis, data are stored on the PC RAM. While events accumulate, the CPU performs the selection of the triplets: for each hit read from the packet, its coordinates are read, and the hit is put into a triplet in case its coordinates satisfy the requirements described in Section~\ref{sect:triplets}.\\
%
%As the number of accumulated events becomes 256 (this number was selected analysing the computing speed and memory usage of the program), three arrays are fed into the GPU global memory:
%\begin{itemize}
%\item $x$ coordinates of the hits
%\item $y$ coordinates of the hits
%\item an array of 12 elements containing the indexes of the hits chosen for the triplets
%\end{itemize}
%The first two arrays are ordered in a way that the correspondence between $x$ and $y$ coordinates of each hit is preserved.\\
%
%The \textit{device} sector of the trigger program is organized in 1024 blocks of 32 threads each. One event is expected to be analysed by 4 blocks simultaneously, one for each triplet.\\

%Each block is given access to the portion of memory containing the data relative to the event assigned to it. The blocks execute hermetically, and their threads can only access the \textit{shared} memory assigned to the block. 
%After the execution of the kernel, results are copied back into the global memory of the device, and then again to the PC RAM, becoming available to the user.\\
%It is convenient to define a set of constants before going through the implementation of the algorithm:
%
%
%...\\
%
%...\\
%
%A structure \texttt{Element} is allocated in the global memory of the device, that contains hits information and the triplets for each event of the stream.
%\begin{lstlisting}[fontadjust]
%typedef struct {
%	uint8_t id_x[MAXHITS*MAXEVENTS];
%	uint8_t id_y[MAXHITS*MAXEVENTS];
%	int triplet[4*3*MAXEVENTS];
%	uint16_t length[MAXEVENTS];
%	uint16_t actualsize;
%} Element;
%\end{lstlisting}
%The member \texttt{actualsize} contains the actual number of events stored: it may differ from \texttt{MAXEVENTS} if a timeout occurs during the accumulation of events, i.e. if the CPU takes too much time to read and store \texttt{MAXEVENTS} events.\\
%
%We also create a structure \texttt{UTILITY} in the global memory, that hosts variables accessed during the execution of each block.
%\begin{lstlisting}[fontadjust]
%typedef struct {
%	int8_t x_candidate[4*MAXHITS*MAXEVENTS];
%	int8_t y_candidate[4*MAXHITS*MAXEVENTS];
%	float xHit[4*MAXHITS*MAXEVENTS];
%	float yHit[4*MAXHITS*MAXEVENTS];
%	float u[4*MAXHITS*MAXEVENTS];
%	float v[4*MAXHITS*MAXEVENTS];
%	float u2[4*MAXHITS*MAXEVENTS];
%	float v2[4*MAXHITS*MAXEVENTS];
%	float z[4*MAXHITS*MAXEVENTS];
%	float z2[4*MAXHITS*MAXEVENTS];
%	float uz[4*MAXHITS*MAXEVENTS];
%	float vz[4*MAXHITS*MAXEVENTS];
%	float uv[4*MAXHITS*MAXEVENTS];
%} UTILITY;
%\end{lstlisting}
%
%Finally, memory is allocated for the results of the multi-ring fitting procedure:
%\begin{lstlisting}[fontadjust]
%typedef struct {
%	uint16_t actualsize;
%	float xCenter[MAXEVENTS*4];
%	float yCenter[MAXEVENTS*4];
%	float radius[MAXEVENTS*4];
%	int nHitsPerCandidate[MAXEVENTS*4];
%} Result;
%\end{lstlisting}

\subsection{Data stream flow and triplet forming}

It is convenient to define some constants before going through the implementation of the algorithm:
\begin{lstlisting}[fontadjust]
#define MAXHITS (int) 32
#define MAXEVENTS (int) 256
\end{lstlisting}
Here, \texttt{MAXHITS} represents the maximum number of hits allowed per event. %The value is set to 32 to emulate events that have already passed the standard L0 selection.
This value was initially set to 32 in order to analyse events that have already passed the standard online selection\footnote{In \chaptername~\ref{ch:tests} I will also test the possibility of increasing (actually doubling) this limit: if the GPU-based trigger proves efficient enough in the high-multiplicity region, the multiplicity cut could be eliminated from the hardware L0 trigger.}. 
Input from the NIC board is continuously read; the number of events packed in a UDP packet is read from the UDP header. Event data is written into \texttt{Element} structures, defined below, that are copied to the local memory of the GPU when a predefined size (\texttt{MAXEVENTS}) or data accumulation time has been reached. I will discuss  the choice of \texttt{MAXEVENTS}=256 in \chaptername~\ref{sect:improve}.
\begin{lstlisting}[fontadjust]
typedef struct {
	uint8_t x[MAXHITS*MAXEVENTS];
	uint8_t y[MAXHITS*MAXEVENTS];
	int triplet[4*3*MAXEVENTS];
	uint16_t length[MAXEVENTS];
	uint16_t actualsize;
} Element;
\end{lstlisting}
In particular, the \texttt{x[]} and \texttt{y[]} arrays are filled with the coordinates of \textit{all} the events of the batch, i.e. of the set of \texttt{MAXEVENTS} events processed concurrently:
\begin{align}
&x[\;] = \underbrace{x_0 \, x_1 \, x_2 \,\cdots\, x_{31}}_{\text{evt } 0}\, \underbrace{x_{32}\, \cdots\, x_{63}}_{\text{evt } 1}\, x_{64}\, \cdots\\
&y[\;] = \underbrace{y_0 \, y_1 \, y_2 \,\cdots\, y_{31}}_{\text{evt } 0}\, \underbrace{y_{32}\, \cdots\, y_{63}}_{\text{evt } 1}\, y_{64}\, \cdots
\end{align}
The data format is fixed and independent from the actual hit multiplicity: if an event counts less than \texttt{MAXHITS} hits, some locations are left empty. 
Let us also define an array of indices that will be used later:
\begin{align}
&I[\;] = \underbrace{0 \, 1 \, 2 \,\cdots\, {31}}_{\text{evt } 0}\, \underbrace{{32}\, \cdots\, {63}}_{\text{evt } 1}\, {64}\, \cdots
\end{align}
This way, the $x$ and $y$ coordinates of the $n$-th hit in the scope of the current event can be accessed through the same index: $x_n = x\big[I[n]\big]$, and similarly $y_n = y\big[I[n]\big]$.\\

While the 8-\si{\bit} coordinates of hit PMTs are read and stored in \texttt{x} and \texttt{y} arrays, the host program also checks if hits can be assigned to a triplet.
%
%\begin{align}
%&x[\;] = \underbrace{x_0 \, x_1 \, x_2 \,\cdots\, x_{31}}_{\text{evt } 0}\, \underbrace{x_{32}\, \cdots\, x_{63}}_{\text{evt } 1}\, x_{64}\, \cdots\\
%&y[\;] = \underbrace{y_0 \, y_1 \, y_2 \,\cdots\, y_{31}}_{\text{evt } 0}\, \underbrace{y_{32}\, \cdots\, y_{63}}_{\text{evt } 1}\, y_{64}\, \cdots\\
%&I[\;] = \underbrace{0 \, 1 \, 2 \,\cdots\, {31}}_{\text{evt } 0}\, \underbrace{{32}\, \cdots\, {63}}_{\text{evt } 1}\, {64}\, \cdots
%\end{align}
%This way, $x$ and $y$ coordinates of each hit can be accessed through the same index: $x_n = x[I_n]$, and similarly $y_n = y[I_n]$.\\
For each event, in fact, a \texttt{triplet[]} array is defined, that hosts the indices of the hits selected for the initialisation of the pattern recognition step:
\begin{align}
&\mathtt{triplet}[\;] = \underbrace{t_0\, t_1 \, t_2}_{\text{XMAX}}\, \underbrace{t_3\, t_4 \, t_5}_{\text{XMIN}}\, \underbrace{t_6\, t_7 \, t_8}_{\text{YMIN}}\, \underbrace{t_9\, t_{10} \, t_{11}}_{\text{YMAX}}\, & \left(\forall\text{ event}\right)
\end{align}
The first element of the \texttt{triplet[]} array is initialised with the index of the first hit. The indexes of those subsequent hits whose coordinates satisfy the conditions discussed in Section~\ref{sect:triplets} are then written into the other elements. 
As an example, the following code shows how the \texttt{XMIN} portion of the triplet is populated.
\lstset{breaklines=true, style=customcpp,basicstyle=\scriptsize\ttfamily}
\begin{lstlisting}[fontadjust]
#define THD (int) 8
#define sq_dist(j,k) ( pow(x[I[j]] - x[triplet[k]],2) 
                      + pow(y[I[j]] - y[triplet[k]],2) )

//i = index of the i-th hit of the current event
//I[i] = index of the i-th hit of the batch
if (i > 2) {	
	if (x[I[i]] < x[triplet[2]] && sq_dist(i,1) > THD
	    && sq_dist(i,0) > THD) {
		if (x[i] > x[triplet[1]]) triplet[2]= I[i];
		else {
			if (x[i] > x[triplet[0]]) {
				triplet[2] = triplet[1];
				triplet[1] = I[i];
			}
			else {
				triplet[2] = triplet[1];
				triplet[1] = triplet[0];
				triplet[0] = I[i];
			}
		}	
	}
}
else if (i == 0) {
	triplet[i] = I[i];
}
else if (i == 1 && sq_dist(i,0) > THD) {
	if (x[I[i]] < triplet[0]) {
		triplet[i] = triplet[0];
		triplet[0] = I[i];
	}
	else triplet[i] = I[i];
}
else if (i == 2 && sq_dist(i,1) > THD
         && sq_dist(i,0) > THD) {
	if (x[I[i]] < triplet[1]) {
		if (x[I[i]] < triplet[0]) {
			triplet[i] = triplet[1];
			triplet[1] = triplet[0];
			triplet[0] = I[i];
		}
		else {
			triplet[i] = triplet[1];
			triplet[1] = I[i];
		}
	}
	else triplet[i] = I[i];
}
\end{lstlisting}
Similarly, \texttt{XMAX}, \texttt{YMIN} and \texttt{YMAX} portions are populated. We obtain a 12-elements array $\{t_n\}$ such that:
\begin{align}
&x[t_0] \leq x[t_1] \leq x[t_2] \leq x[t_3] \leq x[t_4] \leq x[t_5]\\
&y[t_6] \leq y[t_7] \leq y[t_8] \leq y[t_9] \leq y[t_{10}] \leq y[t_{11}]
\end{align}

These operations are carried out once per event. The indices used in the above code only fit the first event of the stream, while for the subsequent events the triplet index is transformed as $t_n \longrightarrow t_n + 12 k$, where $k$ is the event index. The array containing triplets will then have the form
\begin{align}
&\mathtt{triplet}[\;] = \underbrace{t_0 \,\cdots\, t_{11}}_{\text{evt } 0}\, \underbrace{t_{12}\, \cdots\, t_{23}}_{\text{evt } 1}\, t_{24}\, \cdots
\end{align}

The operations described so far are executed on an event-by-event basis by the host program, running on the CPU, while event data are read from the NIC memory. This way, the population of the \texttt{triplet[]} array is interlaced with the readout and accumulation of UDP packets, so that smaller sets of instructions need to be processed by the GPU cores.\\
%, after reading the event from the network. In principle, this way a slightly higher level of concurrency is reached. In fact, the forming of triplets is interlaced with the readout and accumulation of the packets coming from the TEL62 board, and thus it is performed in parallel to the execution of the GPU kernel, hardly increasing the latency of the whole process.\\

When the event batch is ready, data is copied to the global memory of the GPU, where the kernel is executed, and the results are copied back to the host memory by means of dedicated CUDA functions:
\begin{lstlisting}
cudaMemcpyAsync((void*)gpu_mem, (void*)host_mem, sizeof(Element), cudaMemcpyHostToDevice, stream_id);

multiring<<<blocksPerGrid, threadsPerBlock, smemSize, stream_id>>>(gpu_mem, gpu_results, utils);

cudaMemcpyAsync((void*)host_results, (void*)gpu_results,sizeof(Result), cudaMemcpyDeviceToHost, stream_id);
\end{lstlisting}
where \texttt{stream\_id} refers to the running stream, \texttt{gpu\_mem} is a pointer to an \texttt{Element} of data stored in the global memory, \texttt{gpu\_results} points to the global memory allocated for the \texttt{Result} structure, and \texttt{utils} provides temporary storage in a \texttt{UTILITY} structure for variables that need to be accessed while executing the circle fitting function:
\begin{lstlisting}[fontadjust]
typedef struct {
  int16_t x_candidate[4*MAXHITS*MAXEVENTS];
  int16_t y_candidate[4*MAXHITS*MAXEVENTS];
  float xHit[4*MAXHITS*MAXEVENTS];
  float yHit[4*MAXHITS*MAXEVENTS];
  float xm[4*MAXEVENTS];
  float ym[4*MAXEVENTS];
  float u[4*MAXHITS*MAXEVENTS];
  float v[4*MAXHITS*MAXEVENTS];
  float u2[4*MAXHITS*MAXEVENTS];
  float v2[4*MAXHITS*MAXEVENTS];
  float z[4*MAXHITS*MAXEVENTS];
  float z2[4*MAXHITS*MAXEVENTS];
  float uz[4*MAXHITS*MAXEVENTS];
  float vz[4*MAXHITS*MAXEVENTS];
  float uv[4*MAXHITS*MAXEVENTS];
  float zav[4*MAXEVENTS];
  float z2av[4*MAXEVENTS];
  float u2av[4*MAXEVENTS];
  float v2av[4*MAXEVENTS];
  float uvav[4*MAXEVENTS];
  float uzav[4*MAXEVENTS];
  float vzav[4*MAXEVENTS];
} UTILITY;

typedef struct {
  uint16_t actualsize;
  float xCenter[MAXEVENTS*4];
  float yCenter[MAXEVENTS*4];
  float radius[MAXEVENTS*4];
  int nHitsPerCandidate[MAXEVENTS*4];
  int nHitsPerEvent[MAXEVENTS];
} Result;
\end{lstlisting}%\B\B
In the above code, \texttt{u} and \texttt{v} are the hit coordinates relative to the hits center of gravity frame; the other \texttt{UTILITY} variables are polynomial combinations of these two, that are used to compute the best ring parameters (see Appendix~\ref{app:taub-code} for the actual CUDA implementation of the ring fit).\\

The kernel call parameters are defined as
\begin{lstlisting}[fontadjust]
int threads = MAXHITS;
const unsigned int eventsPerBlock = 1;
const unsigned int tripletsPerEvent = 4;
dim3 threadsPerBlock(threads,1,1);
int blocksPerGrid = ceil(MAXEVENTS*tripletsPerEvent
                         / (float)eventsPerBlock);
int smemSize = 7*sizeof(int)+MAXHITS*sizeof(float);
\end{lstlisting}
so that 4 blocks execute for each event, each one handling a pattern recognition stage initialised with a different triplet, and the corresponding single ring fitting stage. Each block contains a number of threads equal to the maximum number of hits per event. Finally, \texttt{MAXEVENTS} events are evaluated concurrently. The amount of shared memory that has to be allocated will be discussed later.\\
%
%...\\
%
%...\\
%
%A copy of the array \texttt{triplets} will be created in the shared memory of the device, for faster access. 



\subsection{Implementation of the kernel}
The \texttt{multiring} kernel consists of 3 main subsets of instructions.

\begin{enumerate}
\item An array containing the integer $x$ and $y$ coordinates of the hits forming the current triplet (of dimension \texttt{6*sizeof(int)}) is created in the shared memory, visible to all the threads of the block.
\item Each thread evaluates the Ptolemy's theorem on a different hit of the event, and if it stands it copies the hit coordinates to the \texttt{UTILITY} structure. A counter, stored in the shared memory (\texttt{1*sizeof(int)}), is incremented every time a hit is accepted by this pattern recognition procedure.
\item A semi-parallelized version of the Taubin ring fit algorithm, discussed in Appendix~\ref{app:fits}, is executed, and the \texttt{Result} structure is populated.
\end{enumerate}

\lstset{breaklines=true, style=customcpp,basicstyle=\scriptsize\ttfamily}
\begin{lstlisting}[fontadjust]
__device__ inline int Ptolemy(const uint8_t& x, const uint8_t& y,
                              const int* triplet);

__global__ void multiring(Element* Event, Result* Result, UTILITY* utils)
{  
  //Allocate block shared memory
  __shared__ int triplet_s[6];
  __shared__ unsigned int length;
  __shared__ float reduction[32];
  
  //Distribute jobs to blocks and threads
  unsigned int eventIdx = blockIdx.x/4;
  unsigned int tripletIdx = blockIdx.x%4;
  unsigned int hitIdx = threadIdx.x + eventIdx * MAXHITS;
  unsigned int utilsIdx = MAXHITS*blockIdx.x + threadIdx.x;
  unsigned int elemIdx = 12*eventIdx+threadIdx.x
                         +3*tripletIdx;
  
  //Copy the current triplet in shared memory
  if(threadIdx.x < 3) {
      triplet_s[2*threadIdx.x] = Event->x[Event->triplet[elemIdx]];
      triplet_s[2*threadIdx.x+1] = Event->y[Event->triplet[elemIdx]];
    }
  __syncthreads();
  
  //Pattern recognition: copy 'good' hits
  if (Ptolemy(Event->x[hitIdx],Event->y[hitIdx], triplet_s) 
     && (Event->x[hitIdx] != 0) && (Event->y[hitIdx] != 0)) {
      utils->x_candidate[utilsIdx] = Event->x[hitIdx];
      utils->y_candidate[utilsIdx] = Event->y[hitIdx];
    } else {
      utils->x_candidate[utilsIdx] = 0;
      utils->y_candidate[utilsIdx] = 0;
    }
  __syncthreads();
  
  //Compute number of hit candidates per ring
  length = 0;
  if(utils->x_candidate[utilsIdx] != 0) atomicAdd(&length, 1);
  __syncthreads();
  
  //Fill some results
  if (threadIdx.x == 0) {
      Result->actualsize = Event->actualsize;
      Result->hitsPerEvent[eventIdx] = Event->length[eventIdx];
      Result->nHitsPerCandidate[blockIdx.x] = length;
    }
  __syncthreads();

  //Execute ring fit
  Taubin(utils, reduction, Result);
}
\end{lstlisting}\lstset{breaklines=true, style=customcpp}\B\B

The code implementing the \texttt{Taubin} function is available in Appendix~\ref{app:taub-code}. This function finds the parameters of the best fitting circle in the re\-pa\-ra\-me\-tri\-zed frame introduced in Section~\ref{sect:data-format}, also reverting the arbitrary 85-units shift applied to Montecarlo-generated data (see Eqn.~\ref{eqn:xshift}). The results are then converted back to the laboratory frame.\\

Several arrays need to be reduced\footnote{Those array operations that return a result with a smaller rank are referred to as ``reductions''. In this case, the sum of the array elements is computed (rank 0), in order to find their average.}: to achieve this, a modified version of the \texttt{reduce\_kernel} function provided by the CUDA Data Parallel Primitives Library (CUDPP)\footnote{\url{https://code.google.com/p/cudpp/}} is used, that allows to compute the sum of an array in a small number of steps, as sketched and explained in \figurename~\ref{img:reduct}. All the threads of the block participate to the reduction, and therefore a portion of shared memory of size \texttt{MAXHITS*sizeof(float)} is allocated.\\

Both the kernel and the triplet finding procedures have been designed together with F. Pantaleo.

\begin{figure}
\centering
%\def\svgwidth{\textwidth}
%\input{Images/pdf/gpu_algo/reduction-tex.pdf_tex}
\includegraphics[width=\textwidth]{Images/pdf/gpu_algo/reduction-full}
\caption{Parallel reduction of arrays within a single thread block \cite{reduct}. The algorithm recursively reduces the output size, by computing the local sum of 2 array elements. Results are stored in the shared memory, visible to all the threads. A sequential addressing approach is used for memory access, in order to prevent memory bank conflicts. This way, the complexity of the algorithm is $\mathcal{O}(\log_2 N)$ where $N$ is the size of the array.}\label{img:reduct}
\end{figure}