Optimal FPGA memory allocation for image processing

Abstract

In the field of computer vision, Field Programmable Gate Array (FGPA) limited de on-chip memory is difficult to meet the power, size and other requirements. To address this phenomenon, the study constructs a partitioning algorithm to achieve a balance between energy consumption and resource utilisation based on the analysis of memory resource allocation, overall power consumption and resource utilisation from the perspective of image processing technology. The power consumption of the balancing algorithm is lower compared to the optimised utilisation algorithm HLS tool, with both Block Ramdom Access Memory (BRAM) power consumption taking the value of 0.005; the dynamic power consumption takes the value range of 0.014–0.082. Compared to the High Level Synthesis (HLS) tool, the overall power consumption of the balancing algorithm and the optimised utilisation algorithm is significantly lower, with the values of 0.251 and 0.252 respectively, both with a reduction rate of approximately 30%. The accuracy rate of the proposed memory optimisation allocation algorithm is the highest among the four memory optimisation allocation algorithms and strategies on all three types of target scales. FPGA memory optimisation allocation strategy can guarantee to have lower power consumption while satisfying the same resource occupancy, and the model has in-depth application value in visual image vision technology.

Keywords

FPGA CLAHE image CAMShift partitioning algorithm

1. Introduction

Computer vision is involved in the fields of unmanned supermarkets, face recognition, driverless cars and so on, but these application scenarios will be affected by power consumption and cost, and it is difficult to complete image data processing through the traditional ordinary processor platform. To solve this problem, field programmable gate array (FPGA), as an integrated circuit, is also known as flowing solid because it can configure corresponding hardware circuits according to the different needs of designers and users [1, 2]. The design process of FPGA mainly includes device configuration, verification, design implementation, synthesis, design input, etc. it has the characteristics of high reliability, high integration and fast speed. This technology has obvious advantages in image processing. However, the on-chip memory resource of FPGA is about 6MB, and images are usually stored through external memory. Because the off-chip memory has high access latency, this will greatly reduce the performance [3, 4]. At the same time, external memory will further improve the economic cost, power consumption and size of the system. At present, the main research on the application of computer vision at home and abroad focuses on reducing power consumption and improving performance, but these research results are manifested in power consumption or performance, it is difficult to analyze the two performances at the same time, and the optimal solution under balanced conditions has not been achieved. Aiming at these problems, the research innovatively proposes to achieve efficient image processing through CAMShift (Continuous Adaptive Mean Shift) algorithm, and gives a balanced partition algorithm of FPGA on-chip memory, giving play to the advantages of strong reliability, high integration, fast speed, etc., in order to minimize the power consumption of image processing applications and the occupation of FPGA memory resources. The research content consists of the following four parts. The second part summarizes the research status of domestic and foreign scholars on the optimization of FPGA memory allocation and image processing efficiency in recent years. The third part focuses on the FPGA memory balance partition algorithm and CAMShift algorithm in image processing technology. The fourth part verifies the image processing technology through simulation experiments, and analyzes the results. The fifth part summarizes the research results, and gives the existing problems and prospects for the future.

2. Review of the literature

Park et al. [5] proposed an FPGA-based hardware accelerator design for handwritten Korean character recognition applications in order to solve the current problem that deep convolutional neural networks are difficult to achieve in real time through computer software, using techniques such as memory access optimization, computational unit parallelism, data conversion, and recognition time of 11.19 ms per character through Xilinx FPGA accelerator images. 11.19 ms, which greatly improves the recognition efficiency and also has significant advantages in terms of energy efficiency. Lai et al. [6] studied the impact of FPGA memory partition on power consumption, and used data placement to achieve parallel access, thereby minimizing communication power consumption. Chetan et al. [7] in addressing the important contributions of current matrices in quantum mechanics, image processing, and solving circuits, proposed the use of model-based FPGAs design to design and evaluate different floating-point matrix multiplication architectures/methods and floating-point matrix solving inversions. The model was implemented in hardware via a ZED board on a Zynq 7000 FPGA and the performance test results showed that the approach is highly feasible. Wu et al. [8] addressed the need for the current existence of multi-FPGA acceleration and proposed a database for efficient large-scale graphics processing in FPGA-accelerated data centres with minimal hardware engineering effort is a state-of-the-art graphics accelerator, and results on a 32-node Microsoft Catapult-type data centre show that the proposed communication library has strong data processing capabilities in a central data processor and is extremely scalable. Yan et al. [9] researchers and team designed an FPGA accelerator for graphics attention networks that eliminates the need for digital signal processors and large amounts of on-chip The performance evaluation was completed on a Wave F10A motherboard with an Intel Arria 10 GX1150 and 16 GB DDR3 memory, and the test results showed that the proposed accelerator runs faster and consumes less energy in the central data processor.

Wang et al. [10] designed two harmonic summation methods to address the problems of irregularity and low continuity in FPGAs’ access to a large number of points in the off-chip memory, namely, processing the input signals directly without storage, storing the intermediate data in the off-chip memory, and using an open computing language to implement them. The proposed preloading method that preloads all necessary points while reordering the input signals is faster than all other methods and has lower energy consumption. Menasri et al. [11] scholars used parallel and pipelined structures in order to improve the throughput of the CABAC decoder and proposed an algorithmic architecture equation for implementing the CABAC decoder on FPGAs. Test results showed that the implementation could handle 2.2 bins/cycle and exhibit a high throughput of 271.678 Mbins/s when operating at 123.49 MHz. Jiang et al. [12] proposed a method for implementing peak detection on FPGAs that explores the possibility of parallel architectures to accelerate peak detection and achieve system miniaturisation. The test results showed that the proposed method effectively reduced the peak detection error by 42.81% and the ranging error by 63.63% compared with the peak detection module using the Caruana method, and ensured the real-time update of the ranging results. Bao et al. [13] designed an instruction driven convolutional neural network accelerator based on FPGA, which is widely used as a platform to accelerate cellular neural networks because of its good performance, energy efficiency and flexibility, and applied it to image target detection. The performance of the accelerator and scheduling strategy was evaluated through Xilinx KU115 FPGA platform. The framework achieved 2.3 times and 50 times energy efficiency respectively. The accuracy of the fixed-point detection algorithm decreases by less than 1% compared with the floating-point algorithm. Zhao et al. [14] reorganized the memory layout at the algorithm level, and placed data through lattice based partitions accordingly by checking the calculation behavior.

Combined with the research status of many domestic and foreign scholars, it can be seen that the research on image processing technology mainly focuses on image data analysis and images, although some of the scholars’ involve FPGA memory optimisation allocation, but the research has made little progress in related studies, and FPGA memory optimisation allocation only single analysis of resource utilisation and energy consumption and other aspects. The research builds on previous research results and proposes a partitioning algorithm applied to FPGA memory optimisation allocation in anticipation of achieving efficient resource utilisation and lower energy consumption of FPGA memory, which in turn enhances the effectiveness of image processing techniques.

3. FPGA memory balancing partitioning algorithms in image processing technology

3.1 CAMShift algorithm in advanced image processing techniques

With the rapid development of computer vision technology and image processing technology, target tracking and detection in video images has a positive significance. The main methods of target tracking in images are generating method and discriminant method. The generating methods mainly include mean shift, Kalman filter, particle filter and sparse coding. CAMShift algorithm is the most commonly used mean shift target tracking algorithm, which has the following characteristics. The algorithm complexity is small; It is a parameterless algorithm, easy to integrate with other algorithms; Weighted histogram modeling is adopted, which is insensitive to small angle rotation, slight deformation and partial occlusion of the target. CAMShift algorithm is an adaptive tracking algorithm, which obtains the confidence of object position according to the color histogram of the object in the previous image. When the object position has been determined and the color histogram in the $k+$ 1 frame image is calculated, the algorithm can obtain the most likely object position of the $k+$ 1 frame image by comparing the color histogram. The implementation platform of CAMShift algorithm is Zynq7020 chip on Zedboard, which transmits data through OV7670 of external camera, and the processed data needs to be sent to the onboard ARM processor. The core of the algorithm is Mean Shift, which aims to track continuous frame images.

Figure 1.

Schematic diagram of mean shift principle.

The Mean-Shift algorithm is a mean-shift algorithm that marks a target object by the shape and size of the region of interest alone. The feature space consists of all attributes during image processing, and after the image attributes have been mapped the feature space is a dense set of data points consisting of significant feature attributes. The principle of mean shift is shown in Fig. 1. The initial window position, shape and centre of mass are first determined. Then a new window shift vector is obtained using the mean shift algorithm. Finally the mean shift vector is continuously adjusted to ensure that the probability density function value is maximised. The multivariate density kernel function $K(x)$ can estimate the distribution area of a dense point set, the kernel function is integrated in space as 1, and the expression is Eq. (1).

$\displaystyle K(x)=cK(\left\|x\right\|^{2})$ (1)

$x$ refers to random variable, $c$ is a positive real number, in general the kernel function can be expressed as a normal distribution function $K(x)$ and the kernel function profile expression $k(x)$ as Eq. (2).

$\displaystyle K\left(\frac{x-x_{i}}{h}\right)=ck\left(\left\|{\frac{x-x_{i}}{h% }}\right\|^{2}\right)$ (2)

In Eq. (3), $h$ represents the bandwidth. Since the probability density function takes extreme values in order to make the window position stable, the final displacement vector $m_{h,G}(x)$ is obtained by combining the profile of the kernel function and the gradient of the probability density function in Eq. (3).

$\displaystyle m_{h,G}(x)=\sum_{i=1}^{n}{x_{i}g\left({\left\|{\frac{x-x_{i}}{h}% }\right\|}\right)}\Big{/}\sum_{i=1}^{n}{g\left({\left\|{\frac{x-x_{i}}{h}}% \right\|}\right)}-x$ (3)

The updated centre-of-mass position is obtained by combining the initial centre-of-mass position. The principle of RGB differencing is to determine the detection area of a target based on the difference between the luminance values of pixels in the shadow and non-shadow regions of RGB space [15, 16]. G and B refer to the colours red, green and blue, respectively, where the digital image consists of these three shades of grey. Figure 2a and b refer to the RGB colour cube diagram and the pixel points of the RGB colour image, respectively. the model of HSV colour can also be referred to as the hexagonal vertebra model, with the parameters in colour being lightness, saturation and hue, respectively. This model is widely used in image editing.

Figure 2.

RGB color cube diagram and pixel diagram.

The formula for converting an RGB space image to HSV colour space is Eq. (4).

$\displaystyle H=\begin{cases}\text{undefined},&\text{if}\ m=n;\\ 60\times\frac{G-B}{m-n},&\text{if}\ (m=R)\ \text{and}\ (G\geqslant B);\\ 60\times\left(\frac{G-B}{m-n}+6\right),&\text{if}\ (m=R)\ \text{and}\ (G<B);\\ 60\times\left(\frac{B-R}{m-n}+2\right),&\text{if}\ m=G;\\ 60\times\left(\frac{R-G}{m-n}+4\right),&\text{if}\ m=B.\end{cases}$ (4)

Equation (4), $m=\max(R,G,B)$ , $n=\min(R,G,B)$ , $V=m$ . If $m=0$ , then $S=0$ , otherwise, $S=1-n/m$ . R, G, B are taken in the range of [0, 255], H, S, V are taken in the range of [0, 360], [0, 1], [0, 255] respectively. The grey image $V_{1}$ is histogram equalised to obtain the brightness $V$ , followed by a mapping to obtain a uniform grey level image 1 $V_{2}$ . The cumulative distribution function of the probability of each grey level is expressed as a change function in Eq. (5).

$\displaystyle S_{K}=T(r_{k})=\sum_{j=0}^{k}\frac{n_{j}}{n}=\sum_{j=0}^{k}\Pr(r% _{j})$ (5)

In Eq. (5), the total number of possible grey levels in the image is referred to by $L$ , the sum of pixels in the image is $n$ , the total number of pixels in the grey level is referred to by $r_{k}n_{k}$ , the transformation relationship is referred to by $T$ , the normalised grey levels of $V_{1}$ and are referred to by $V_{2}r_{k}$ and $s_{k}$ respectively, and the probability of the grey value of the first level of the original image. the HSV colour space is converted to RGB space by Eq. (6).

$\displaystyle(R,G,B)=\left\{\begin{matrix}(v,z,x),c_{1}=0;\\ (y,v,x),c_{1}=1;\\ (x,v,z),c_{1}=2;\\ (x,y,z),c_{1}=3;\\ (z,x,v),c_{1}=4;\\ (v,x,z),c_{1}=5;\\ \end{matrix}\right\}$ (6)

In Eq. (6), $C_{2}=\left[{\frac{H}{60}}\right]-C_{1}$ . For image content acquisition, the study sets up a set of feature maps located after each layer of non-linear filter combinations. If the model has $N$ different filters in the $l$ layer, then the same number of feature maps $N$ are also output in the $l$ layer. Assuming a feature map size of $M$ , the matrix $F^{l}\in R^{N_{l}\times M_{l}}$ can represent the $l$ layer feature response. When $i\in[1,N]$ , $k\in[1,M]$ , the feature output value $F_{ik}^{l}$ at position $k$ of the $i$ filter in the $l$ layer. Using the gradient descent method, the original image at a layer of the convolutional neural network corresponds to a white noise image, thus making the white noise image correspond to the original information content.

3.2 FPGA memory balancing partitioning algorithm

In order to achieve efficient image data processing, reduce energy consumption and reduce resource occupation, FPGA memory balance partition algorithm is proposed to solve this problem. The partitioning algorithm first analyses the problem of resource utilisation and energy consumption in image processing, and then presents a design approach for balancing utilisation and power consumption. For the resource utilisation problem in image processing, each image frame buffer is set to use only one possible BRAM (Block Ramdom Access Memory). The study gives four related definitions, as follows. For the possible configurations of the BRAM $i$ and the storage capacity $C$ , the expression for the vector of elements as a configuration is Eq. (7).

$\displaystyle\textit{Cfg}=\begin{pmatrix}(E_{1},F_{1})\\ (E_{2},F_{2})\\ \vdots\\ (E_{i},F_{i})\\ \end{pmatrix}=\begin{pmatrix}\textit{Cfg}_{1}\\ \textit{Cfg}_{2}\\ \vdots\\ \textit{Cfg}_{i}\\ \end{pmatrix}$ (7)

In Eq. (7), the first and second components of each element represent the width $E$ and the height $F$ of the BRAM respectively. The height and width must satisfy Eq. (8).

$\displaystyle E_{x}\times F_{x}\leqslant C,\quad\forall x\in[1,i]$ (8)

All frame images of any size have several possible Bram topologies. Although the total physical capacity is roughly the same, in some configurations additional data bits can be used as parity bits. The image frame is set as a three-dimensional array, 1 and 2 represent the size width and height respectively, and 3 refers to the pixel bit width. The mapping diagram of Bram topology based on 3D to 2D array is shown in Fig. 3.

Figure 3.

Mapping based on 3D to 2D array.

The main sequence of the mapping scheme used in the study is rows and columns. Set $x$ and $y$ to represent the width and height of the array respectively. The pixel bit width will be allocated to the width of the array, and the size width and height can be directly allocated to the height of the array, which also provides a possible reality for the normal work of computer hardware. If there is a mapping of Bw in the $y$ dimension, this will result in different bits being scattered and stored in Bram [17, 18]. In this case, it is necessary to read and write pixels in sequence and access different memory locations. Based on this, the mapping scheme needs to make the following assumptions: the size, width and height are orderly combined and allocated to the array height, while the pixel bit width is allocated to the array width, and the three-dimensional array is transformed into two-dimensional array $A2$ . There are two possibilities for configuration. The mapping scheme is defined as Eq. (9).

$\displaystyle(x,y)=m(W,H,\textit{Bw})$ (9)

The expression for a two-dimensional array is Eq. (10).

$\displaystyle\begin{cases}A2_{x,y}=A_{3y\backslash W,y\%W.x},x=Bw,y=W\times H% \\ A2_{x,y}=A_{3y\%wH,y\backslash H.x},x=Bw,y=W\times H\\ \end{cases}$ (10)

In Eq. (10), the integer division and modulus are represented by $\backslash$ and $\%$ respectively. Set definition three as follows, with two-dimensional mapped image frame sizes of $x$ and $y$ , respectively, as shown in Fig. 4. $a\times b$ The scheme of pixel assignment on individual BRAMs is set to a linear combination, see Eq. (11).

$\displaystyle p(x,y)=\textit{Cfg}*\left((a_{1},b_{1}),(a_{2},b_{2}),\ldots,(a_% {i},b_{i})\right)$ (11)

In Eq. (11), linear combinations are referred to by the designator, with one and only one possessing a non-zero component.

Figure 4.

Schematic diagram of 2D array mapping to Bram.

A two-dimensional array of size $y=W\times H$ and $x=Bw$ is mapped to $a\times b$ BRAMs with configurations of height and width $F$ and $E$ respectively. Definition IV is as follows, setting the maximum capacity of the BRAM to $C$ and the partitioning scheme to $p$ , the utilisation efficiency is calculated by Eq. (12).

$\displaystyle E=(x\times y)/(a_{p}\times b_{p}\times C)$ (12)

The efficiency problem can be understood as maximising the utilisation of the partitioning scheme. For a scale width and height of $W$ and $H$ respectively, the pixel width of the image frame is Bw and the condition to be satisfied is $\left((a\times M_{P})\geqslant x\right)\wedge\left((b\times N_{P})\geqslant y\right)$ . Assuming that the width and height of the image frame are 320 and 240, respectively, and that each pixel is 8 bits, the BRAM is configured as in Eq. (13).

$\displaystyle\textit{Cfg}=\left\{\begin{array}[]{l}(1,16384)\\ (2,8192)\\ (4,4096)\\ (9,2048)\\ (18,1027)\\ (36,512)\\ \end{array}\right\}$ (13)

Equation (13) refers to the FGRA of the Xilinx Virtex 7 series 18K-bit BRAM, where the total capacity of the BRAM. The expression for the partition scheme is Eq. (14).

$\displaystyle p\left(m(320,240,8)\right)=\textit{Cfg}*\left\{\begin{array}[]{l% }(8,8)\\ (0,0)\\ (0,0)\\ (0,0)\\ (0,0)\\ (0,0)\\ \end{array}\right\}^{T}$ (14)

In Eq. (14), the partitioning scheme occupies 64 BRAMs and has a storage efficiency of 0.5208333. This is the default method for High Level Synthesis (HLS), the configuration is $(E_{1},F_{1})=$ (1, 16384), where $b_{1}=(W\times H)/N_{1}$ . For image frames of arbitrary size, the default scheme for the HLS tool is Eq. (15).

$\displaystyle p\left(m(W,H,\textit{Bw})\right)=\textit{Cfg}*\left\{\begin{% array}[]{c}\left(\textit{Bw},2\left\lceil\log_{2}\left(\frac{W\times H}{F_{1}}% \right)\right\rceil\right)\\ (0,0)\\ (0,0)\\ (0,0)\\ (0,0)\\ (0,0)\\ \end{array}\right\}^{T}$ (15)

The upward rounding result of the 2-log operation in Eq. (15) is $2\left\lceil\log_{2}\left(\frac{W\times H}{F_{1}}\right)\right\rceil$ .

For the energy consumption of image processing technology, the static power consumption and utilization ratio of Bram show a positive proportion; The dynamic power consumption of Bram is generated through a series of operations. The data reading process is as follows: gate the clock signal; Decoding address; The data is connected to the multiplexer; Transfer data to Bram external port [19, 20]. The writing process is as follows: gate the clock signal; Transfer data to write buffer; Decoding address; Store data in random access memory (RAM) unit. In the partition presented in Eq. (14), each data is distributed among 8 BRAMs. The power consumption of the read and write operations is reduced when the partitioning is optimised by choosing a configuration scheme. Therefore, the study can be optimised by satisfying the widest configuration of BRAMs at a multiple of Bw, but the static power consumption will increase when increasing the number of BRAMs, as well as the chip and address power consumption. Therefore, the study needs to balance utilisation and power consumption. Firstly, a choice of optimisation algorithm (optimisation utilisation algorithm) is required. Secondly, there is a need to balance utilisation and power consumption (balancing algorithm). The method of comparing configuration efficiency with maximum efficiency is an extremely limited search, and Table 1 refers to the classical configuration, set 160 at the same time $\times$ 120, 320 $\times$ 240, 512 $\times$ 512, 640 $\times$ 480, 1280 $\times$ 720 five image frames.

Table 1

Configuration parameters for utilization optimization

Pixel width	160 $\times$ 120	320 $\times$ 240	512 $\times$ 512	640 $\times$ 480	1280 $\times$ 720
8	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096
10	4 $\times$ 4096	2 $\times$ 8192	2 $\times$ 8192	2 $\times$ 8192	2 $\times$ 8192
12	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096
14	18 $\times$ 1024	2 $\times$ 8192	2 $\times$ 8192	2 $\times$ 8192	2 $\times$ 8192
16	18 $\times$ 1024	18 $\times$ 1024	4 $\times$ 4096	18 $\times$ 1024	18 $\times$ 1024
18	18 $\times$ 1024	18 $\times$ 1024	18 $\times$ 1024	18 $\times$ 1024	18 $\times$ 1024
20	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096
22	9 $\times$ 2048	2 $\times$ 8192	2 $\times$ 8192	2 $\times$ 8192	2 $\times$ 8192
24	4 $\times$ 4096	4 $\times$ 4096	1 $\times$ 16384	4 $\times$ 4096	4 $\times$ 4096

Study the setting of the balance relationship between utilization and energy consumption according to user-defined trade-offs and power experience attributes, as well as power and space heuristics. The balance algorithm chooses the solution of optimizing utilization as the premise, and finds the configuration of Bram through continuous iteration. When the balance algorithm finds the first solution below the threshold, it needs to exit the process of configuration optimization and return to the last solution above the threshold.

4. Analysis of the effect of the application of optimal FPGA memory allocation in image processing technology

Table 2
Configuration parameters obtained by balancing utilization and power consumption algorithm

Pixel width	160 $\times$ 120	320 $\times$ 240	512 $\times$ 512	640 $\times$ 480	1280 $\times$ 720
8	9 $\times$ 2048	9 $\times$ 2048	9 $\times$ 2048	9 $\times$ 2048	9 $\times$ 2048
10	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096
12	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096
14	18 $\times$ 1024	18 $\times$ 1024	18 $\times$ 1024	18 $\times$ 1024	18 $\times$ 1024
16	18 $\times$ 1024	18 $\times$ 1024	18 $\times$ 1024	18 $\times$ 1024	18 $\times$ 1024
18	18 $\times$ 1024	18 $\times$ 1024	18 $\times$ 1024	18 $\times$ 1024	18 $\times$ 1024
20	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096	4 $\times$ 4096
22	9 $\times$ 2048	9 $\times$ 2048	9 $\times$ 2048	9 $\times$ 2048	9 $\times$ 2048
24	9 $\times$ 2048	9 $\times$ 2048	9 $\times$ 2048	9 $\times$ 2048	9 $\times$ 2048

Figure 5.

Test results of Bram and dynamic power consumption in sliding window.

In order to analyze the effect of FPGA memory optimization configuration scheme in image processing technology, the hardware used in the research is Xilin virtex 7 FPGA, and the power characterization tool is Xilinx power estimator. This paper studies the power consumption and utilization of the generated frame buffer configured by the partitioning algorithm, and compares it with the corresponding value of the equivalent frame buffer of FPGA memory optimization algorithm in other image processing technologies to verify the performance of the proposed FPGA memory optimization allocation partitioning algorithm. Firstly, the power consumption of each system is estimated by Xilinx power estimator. In the sliding window experiment, use 3 $\times$ 3 sliding window reads Bram power consumption. Figure 5a refers to the BRAM power consumption tested in the sliding window. For each of the five frame sizes, the BRAM power consumption for the Balanced algorithm of the Partitioning algorithm is 0.005 W. The BRAM power consumption for the Optimised utilisation algorithm of the Partitioning algorithm is 0.008 W. The BRAM power consumption for the HLS tool ranges from 0.0059 to 0.117 W. Figure 5b shows the dynamic power consumption tested in the sliding window. The dynamic power consumption of the balanced algorithm in the partitioning algorithm ranges from 0.014 to 0.082W, the dynamic power consumption of the optimised utilisation algorithm in the partitioning algorithm ranges from 0.017 to 0.086W, and the dynamic power consumption of the HLS tool ranges from 0.072 to 0.162W for the five image frame sizes. The power consumption of the balancing algorithm in the partitioning algorithm is lower at all sizes, with a dynamic power reduction rate of approximately 9% and a BRAM reduction rate of approximately 43%, compared to the HLS tool. Thus, the memory partitioning utilised by the partitioning algorithm can significantly reduce power consumption.

Table 3

Resource usage of different memory optimization algorithms/number

Algorithm	IOs	DSPs	FFS	BRAM	LUTs
HLS	65	9	6263	229	9178
Optimized	65	9	6263	146	9199
Balance	65	9	6263	146	9356
Method 1	65	9	6263	152	9368
Method 2	65	9	6263	165	9451

Figure 6.

Power consumption comparison results of all versions.

Table 2 refers to the configuration parameters obtained by the algorithm for balancing utilisation and power consumption. In comparison to the parameters obtained by the optimised utilisation algorithm, the width of most of the configuration parameters will be increased, which will reduce the power consumption, both in terms of dynamic power consumption and BRAM power consumption.

In order to analyze the energy consumption of different algorithms in amsshift algorithm of image processing technology, the size of frame image is set to 320 $\times$ 240, the pixel of each image is 24 bits, and the size of the target area is 16 $\times$ 24. The power consumption comparison results of different categories are shown in Fig. 6. Compared with the default value obtained by the HLS tool, the overall power consumption of the balance algorithm and the optimization utilization algorithm is significantly lower, with values of 0.251w and 0.252w respectively, with a reduction rate of about 30%. The low power consumption of balance algorithm and optimized utilization algorithm is mainly reflected in Bram power consumption and dynamic power consumption. The reduction rate of Bram power consumption is about 85%, and the reduction rate of dynamic power consumption is about 45%. The Bram power consumption and dynamic power consumption of HLS tool are 0.123 w and 0.229 w respectively; The Bram power consumption and dynamic power consumption of the balancing algorithm are 0.18 w and 0.124 w respectively; The Bram power consumption and dynamic power consumption of the optimized utilization algorithm are 0.18 w and 0.124 w respectively. Therefore, although the logical cost of partitioning algorithm has increased, the power consumption of memory partitioning is still lower than that of other algorithms.

Figure 7.

Accuracy comparison results under different frame image pixels.

Figure 8.

Variation rules of classification accuracy and number of image sequences under different SNR.

Table 3 refers to the resource usage and performance of different memory optimisation configuration algorithms, as specified by FringeFieldSwitching (FFS), Look-Up-Table (LUTs), Internetworking Operating System-Cisco (The comparison algorithms are the FPGA dual-port memory mapping optimization algorithm (Method 1) and the FPGA triple-addressable memory acceleration update mechanism (Method 2)). As can be seen from the table, the partitioning algorithm has a higher resource usage compared to the other two memory optimisation allocation algorithms, which may be closely related to the number of BRAMs used. The number of BRAMs used by the HLS tool is about 1.5 times higher than that of the partitioning algorithm, whereas for IOs, DSPs, FFSs and LUTs, the partitioning algorithm’s is not affected.

Experiments were conducted to compare the accuracy at different frame image pixels. The results of the HLS, partitioning algorithm, FPGA dual-port memory mapping optimisation algorithm (Method 1), and FPGA triple addressable memory accelerated update mechanism (Method 2) algorithms are shown in Fig. 7a–d. respectively. For the HLS tool, the best image frame pixels for the three target scales of general target scale, small target scale and minimum target scale are 224 dpi, 278 dpi and 254 dpi respectively, and the accuracy rates obtained are 95.8%, 86.7% and 87.5% in that order. Combined with Fig. 7b, the best frame pixels for the three target scales of general target scale, small target scale and minimum target scale can be determined as 286 dpi, 258 dpi and 246 dpi, and the accuracy rates obtained are 99.1%, 86.7% and 88.6% respectively. For methods 2 and 3, the best image frame pixels for the three target scales of general, small and minimum target scales are all around 250 dpi, but the accuracy rates of both algorithms are reduced, and the accuracy rates are taken to be in the range of 70.9% to 83.5%. Therefore, the memory optimisation allocation algorithm proposed in the study has the highest accuracy rate among the four memory optimisation allocation algorithms and strategies for all three types of target scales. Compared to Methods 1 and 2, the partitioning algorithm and has the fastest improvement in accuracy rate at the smallest target scales, providing an accuracy rate of nearly 40%. The algorithm’s accuracy rate also improved somewhat for the general and large target scales, with an improvement of approximately 20%.

The experiments further verify the accuracy of the proposed partitioning algorithm for image processing results. The variation pattern of classification accuracy and the number of image sequences under different signal-to-noise ratio (SNR) is shown in Fig. 8. As the SNR increases, the maximum number of iterations increases continuously. When the SNR values are 40 dB, 50 dB and 60 dB respectively, the corresponding maximum number of iterations are 30, 40 and 65 respectively. From the figure, we can calculate the minimum magnitude difference to complete the signal classification in the distribution network fault data, and as the SNR increases, the maximum resolution capacity of the algorithm also shows an increasing trend.

5. Conclusion

In response to the problems of high space occupation and high power consumption in advanced image processing in the FGPA platform, the study proposes an FPGA memory partitioning algorithm for image processing technology. The low power consumption of the balanced algorithm and the optimised utilisation algorithm is mainly in the BRAM power consumption and dynamic power consumption, with the reduction rate of BRAM power consumption being about 85% and dynamic power consumption being about 45% in both cases. Compared to the optimised utilisation algorithm, the balanced algorithm consumes less power at different sizes, with a dynamic power reduction rate of approximately 9% and a BRAM reduction rate of approximately 43%; compared to the HLS tool, the balanced algorithm consumes less power at different sizes, with a dynamic power reduction rate of approximately 73% and a BRAM reduction rate of approximately 94%. The best pixel values of the frame images for the partitioning algorithm at the three target scales of general, small and minimal target scales are 286 dpi, 258 dpi and 246 dpi, and the accuracy rates obtained are 99.1%, 86.7% and 88.6% in that order. When the SNR values are out of 40 dB, 50 dB and 60 dB respectively, the corresponding maximum number of iterations of the partitioning algorithm is 30, 40 and 65 respectively, and the BRAM configuration and memory power model have strong advantages over other storage optimisation configurations. The partitioning algorithm proposed in the established study is applied to the entire FPGA family, which is conducive to the expansion and development of the FPGA platform at a later stage. For some special scenes, the subjective evaluation of the processing effect cannot achieve a very ideal effect. The following research needs to apply the proposed segmentation algorithm to image tracking and target tracking to enhance the application value of the segmentation algorithm.

Footnotes

Funding

The research is supported by: the Hunan Provincial Natural Science Foundation (Grant No. 2020JJ4322).

References

Węgrzyn

Jamro

Dbrowska-Boruch

Wiatr

. Optimal reduction of number of test vectors for soft processor cores implemented in FPGAs. Electronics. 2021; 10(20): 2505-2517.

Budhkar

Absalyamov

Zois

Windh

Najjar

Tsotras

. Accelerating in-memory database selections using latency masking hardware threads. ACM T Archi Code Op. 2019; 16(2): 1-28.

Zhang

Guo

Sun

Chen

. A survey: FPGA-based dynamic scheduling of hardware tasks. Chinese J Electron. 2021; 30(6): 991-1007.

Mami

Lahbib

Mami

. A new HLS allocation algorithm for efficient DSP utilization in FPGAs. J Signal Process Sys. 2020, 92(2): 153-171.

Mittal

. A survey of FPGA-based accelerators for convolutional neural networks. Neural Comput Appl. 2020; 32(4): 1109-1139.

Lai

Ustun

Xiang

Fang

Rong

Zhang

. Programming and synthesis for software-defined FPGA acceleration: status and future prospects. ACM T Reconfig Techn. 2021; 14(4): 1-39.

Chetan

Sourabh

Lekshmi

Sudhakar

Manikandan

. Design and evaluation of floating point matrix operations for FPGA based system design. Procedia Comput Sci. 2020; 171: 959-968.

Wang

Zheng

Liao

Jin

Jiang

, et al. FDGLib: A communication library for efficient large-scale graph processing in FPGA-accelerated data centers. J Comput Sci Technol. 2021; 36(5): 1051-1070.

Yan

Tong

Zhi

. FPGAN: An FPGA accelerator for graph attention networks with software and hardware co-optimization. IEEE Access. 2020; 8: 171608-171620.

10.

Wang

Thiagaraj

Sinnen

. Harmonic-summing module of SKA on FPGA-optimizing the irregular memory accesses. IEEE T VLSI Syst. 2019; 27(3): 624-636.

11.

Menasri

Abdellah

Belhadj

Azzaz

. FPGA implementation of variable-bins HEVC CABAC decoder with path delay optimization. IET Image Process. 2019; 13(6): 954-963.

12.

Jiang

Liu

Cao

Song

. Peak detection based on FPGA using quasi-newton optimization method for femtosecond laser ranging. IEEE Access. 2020; 8(99): 47776-47786.

13.

Bao

Xie

Feng

Chang

. A power-efficient optimizing framework FPGA accelerator based on winograd for yolo. IEEE Access. 2020; 8: 94307-94317.

14.

Zhao

Feng

Sinha

Zhang

Liang

. Performance modeling and directives optimization for high-level synthesis on FPGA. IEEE T Computer Aid D. 2019; 39(7): 1428-1441.

15.

Siracusa

Delsozzo

Rabozzi

Di Tucci

. A comprehensive methodology to optimize FPGA designs via the roofline model. IEEE T Comput. 2021; 71(8): 1903-1915.

16.

Mikolajczyk

Nowicki

Bustillo

Yu Pimenovd

. Predicting tool life in turning operations using neural networks and image processing. Mech Syst Signal Pr. 2018; 104(May1): 503-513.

17.

Dinelli

Meoni

Rapuano

Pacini

. MEM-OPT: A scheduling and data re-use system to optimize on-chip memory usage for CNNs on-board FPGAs. IEEE J Em Sel Top C. 2020; 10(3): 335-347.

18.

Ghaffari

Soleimani

Capson

. Analysis and comparison of FPGA-based histogram of oriented gradients implementations. IEEE Access. 2020; 8: 79920-79934.

19.

Shao

Liu

Liao

. Processing grid-format real-world graphs on DRAM-based FPGA accelerators with application-specific caching mechanisms. ACM T Reconfig Techn. 2020; 13(3): 1-33.

20.

Nangia

Shukla

. Resource utilization optimization with design alternatives in FPGA based arithmetic logic unit architectures. Procedia Comput Sci. 2018; 132: 843-848.

Optimal FPGA memory allocation for image processing

Abstract

Keywords

1. Introduction

2. Review of the literature

3. FPGA memory balancing partitioning algorithms in image processing technology

3.1 CAMShift algorithm in advanced image processing techniques

Table 2 Configuration parameters obtained by balancing utilization and power consumption algorithm

Footnotes

Funding

References

Table 2
Configuration parameters obtained by balancing utilization and power consumption algorithm