Design study of FIT dataflow machine for high performance 3-D electrostatic field simulation

1. Introduction

The high-performance computing (HPC) technologies have been developing until now, and the peak performance of the latest supercomputer has reached to 1680 pflop/s. However, such the supercomputers are extremely huge system, which consist of over 8 million cores and need 20 MW power consumption. This means that such the HPC technologies are not suitable for product design in industry. We have been working on the development of a dedicated computer to aim to achieve a portable, low cost, low power consumption and green HPC to be used for industry applications [1–4]. In previous works, we proposed a dedicated computer for 2-D magneto-static field simulations [5–7], in which we designed hardware circuit of the BiCG-Stab matrix solver based on dataflow architecture to execute the finite integration technique (FIT) scheme in 2-D grids space. In addition, we proposed a dedicated computer for microwave simulations [8–16], in which we discussed a hardware circuits of finite-difference time-domain (FDTD) scheme based on sliced 3-D dataflow architecture for 3-D grids space. In this paper, we consider to design a 3-D FIT dataflow machine for electrostatic field simulations combing techniques of the BiCG-Stab scheme circuits and sliced 3-D grid dataflow architecture in previous works.

2. FIT scheme for 3-D electrostatic field simulation

In order to achieve high performance dedicated computer for the 3-D electrostatic fields, the use of FIT scheme is crucial to achieve highly parallel computation based on dataflow architecture. We here summarize an overview of the FIT scheme for 3-D electrostatic field simulation.

In the FIT scheme for 3-D electrostatic field simulation, the following integral form of Gauss’s law is discretized in 3-D grids space, (1) where 𝜌 is the charge density, 𝜙 is the scalar potential and ϵ is permittivity. As shown in Fig. 1, If we allocate the component of the scalar potential 𝜙 at i-th, j-th, k-th grid in all 3-D grids space, ((1)) can be expressed in discretized form as follows: (2) where, (3)

To obtain the distribution of the electrostatic potential 𝜙, which satisfy the FIT discretization (2) for all 3-D grids space simultaneously, we need to construct the FIT matrix equation as shown in Fig. 2 and solve the FIT matrix equation. Then, the boundary condition is taken to be 0 at the outer boundary which is sufficiently far away from the field source.

OPEN IN VIEWER

Fig. 1.

FIT discretization in 3-D grid space.

OPEN IN VIEWER

Fig. 2.

FIT matrix equation of (2).

For the FIT matrix equation (Fig. 2), we here consider to use the BiCG-Stab scheme which is a relatively stable matrix solver. The detail procedure of the BiCG-Stab scheme for a matrix equation Ax = b is as follows:

1. Set an initial value x₀

2. Compute initial residual r₀ = b − Ax₀

3. Set the shadow residual vector r₀^∗ s.t. (r₀, r₀^∗) ≠ 0, e.g., r₀^∗ = r₀

4. Set 𝛽₋₁ = 0, then the search direction vector p₀ = r₀

5. For n = 0, 1, 2, …, until ‖ r n + 1 ‖ ≤ ε ‖ b ‖ (ϵ = 10⁻⁹)

begin (4) end

In the BiCG-Stab scheme, before the iteration process (i)–(vii) starts, the appropriate initial values x₀ and initial residual r₀ are set. Then, the iteration process (i)–(vii) of the BiCG-Stab are repeated until the residual r _n satisfies convergence condition ‖ r n + 1 ‖ ≤ ε ‖ b ‖  (ϵ = 10⁻⁶ ∼ 10⁻⁹). The unit grid circuit of FIT dataflow machine for 3-D electrostatic fields simulation based on BiCG-Stab scheme is designed as shown in Fig. 3. All of unkown values of (p _n , Ap _n , t _n , At _n , x _n , r _n ) in the iteration process of the BiCG-Stab and the coefficient values of ( c i , j , k ( 0 ) , c i , j , k ( 1 ) , c i , j , k ( 2 ) , c i , j , k ( 3 ) , c i , j , k ( 4 ) , c i , j , k ( 5 ) , c i , j , k ( 6 ) ) of ((2)) are stored in the registers of each grids, and these registers are connected each other by arithmetic circuits to execute the iteration process (i)–(vii) in ((4)) of the BiCG-Stab scheme. For example, the circuit connection of p _n of the BiCG-Stab iteration process of (vii) in ((4)) is highlighted as shown in Fig. 3, and the logic circuit of other steps of BiCG-Stab iteration process can be implemented as same as the circuit of p _n in Fig. 3. In addition, Fig. 3 includes circuits for one row of matrix-vector multiplications Ap _n and At _n in the BiCG-Stab iteration process (i) and (iii). If the unit grid circuits of Fig. 3 are connected all over 3-D grids space, inner product calculation of (i) and (iii) in ((4)) can be executed in single clock cycle, which is the extremely high-performance computation. However, such 3-D grid circuits will result in a very large size hardware, and it is impossible to be implemented in a single LSI (Large-Scale Integrated Circuits).

OPEN IN VIEWER

Fig. 3.

Unit grid circuit of BiCG-Stab matrix solver.

In this work, we consider to use sliced 3-D dataflow architecture, which was used in 3-D FDTD dataflow machine, for implementation of the 3-D dataflow architecture machine in practical hardware size. Figure 4(a) depicts three grid circuits which consist of arithmetic grid circuits of Fig. 3 at the middle and additional two register grid circuits. The upper and lower grid circuits (register grid circuit) contain only registers for storing unknown values (p _n , Ap _n , t _n , At _n , x _n , r _n ) of the BiCG-Stab iteration process and coefficient values ( c i , j , k ( 0 ) , c i , j , k ( 1 ) , c i , j , k ( 2 ) , c i , j , k ( 3 ) , c i , j , k ( 4 ) , c i , j , k ( 5 ) , c i , j , k ( 6 ) ) of (2). We here called these circuit “arithmetic 3 grid circuit”. In Fig. 4(b), additional register grid circuits (which are same as the upper or lower grid circuit in Fig. 4(a)) are connected vertically upon the arithmetic 3 grid circuit. To connect the vertical grid circuit of Fig. 4(b) horizontally, 3-D grid circuit is constructed as in Fig. 4(c). Then, in the 3-D grid circuit of Fig. 4(c), after execution of calculation of (4) for the bottom layer register values, it is necessary to exchange register values by the upper layer register values to do shift-down operation. That is, to repeated the execution of (4) at the bottom layer and shift-down operations, the calculation of (4) can be executed for all 3-D grids. It is known that hot-spot parts of the BiCG-Stab scheme are calculations of the matrix-vector multiplications of Ap _n and At _n in (i) and (iii) of (4). In the 3-D grid circuits for the FIT scheme, a part of calculations of the Ap _n for one layer can be executed in a single clock cycle as shown Fig. 5. If we denote the number of layers of 3-D grid circuits of Fig. 4(c) as N _z , the calculation of Ap _n can be done only by 2N _z clock cycles including vertical shift operations.

OPEN IN VIEWER

Fig. 4.

Sliced 3-D dataflow architecture.

OPEN IN VIEWER

Fig. 5.

The matrix equation corresponds to 3-D grids space.

OPEN IN VIEWER

Fig. 6.

The whole configuration of FIT dataflow machine.

OPEN IN VIEWER

Fig. 7.

VHDL simulation for MASTER SCHEDULER.

OPEN IN VIEWER

Fig. 8.

VHDL simulation result for one columon of 3-D grids space.

In this section, we construct entire structure of the FIT dataflow machine for 3-D electrostatic field simulation to use the 3-D grid circuit of Fig. 4(c), and described the machine operation in particular for execution of the BiCG-Stab matrix calculation. The whole configuration of 3-D FIT dataflow machine consists of 3 parts as shown in Fig. 6, MASTER SCHEDULER, INNER PRODUCT MODULE, and FIT GRID MODULE. The FIT GRID MODULE executes the BiCG-Stab iteration process (i)–(vii) as mentioned in the previous section. The INNER PRODUCT MODULE collects the multiplications of r_{0i, j, k}r_{i, j, k}, r_{0i, j, k}Ap_{i, j, k}, t_{i, j, k}At_{i, j, k}, At_{i, j, k}At_{i, j, k} from the FIT GRID MODULE, and inner product calculations of 𝛼 _n , 𝜍 _n , 𝛽 _n are executed to be sent back to the FIT GRID MODULE. These two modules are controlled by the data strobe (DS) signals from the MASTER SCHEDULER. The circuits of the dedicated computer is designed by a hardware description language VHDL. A VHDL logic circuit simulation of MASTER SCHEDULER for single iteration process of the BiCG-Stab scheme of (4) is depicted in Fig. 7. For each calculations of (ii), (iv), (v), (vii), arithmetic calculation at the bottom layer and vertical shift operation are carried out all over the 3-D grids. Accordingly, the single BiCG-Stab iteration process for all 3-D grids space takes (2N _z × 5 + 3N _z + 48) clock cycles.

6. Numerical example

In Fig. 8, an example of the VHDL logic circuit simulation for the p _n of (vii) in (4) in the vertical grid circuit (Fig. 4(b)) is indicated. When the DS signals for p _n is high-level, the value of p _n in the computation layer is updated. On the other hand, when the DS signals for p _n is low-level, the unkown values (p _n , Ap _n , t _n , At _n , x _n , r _n ) of the BiCG-Stab iteration process and coefficients value ( c i , j , k ( 0 ) , c i , j , k ( 1 ) , c i , j , k ( 2 ) , c i , j , k ( 3 ) , c i , j , k ( 4 ) , c i , j , k ( 5 ) , c i , j , k ( 6 ) ) of each layer are shifted-down to the lower layer. We confirm that the calculation of p _n is performed normally according to the scheme of (vii) in (4), which means that the VHDL design of the part of the vertical grid circuit in the sliced 3-D dataflow machine is carried out correctly.

7. Conclusion

In this paper, we have presented the design of the FIT dataflow machine for 3-D electrostatic fields simulations. The detailed logic circuits of sliced 3-D dataflow architecture for the BiCG-Stab scheme is proposed, and a whole configuration for 3-D electrostatic fields FIT dataflow machine is discussed. The logic circuit of the vertical grid circuit of 3-D FIT dataflow architecture machine was designed by the VHDL and it was confirmed by the VHDL circuit simulation that the logic circuits for the BiCG-Stab scheme are implemented correctly. We will proceed to the VHDL logic circuit simulation for the whole 3-D FIT dataflow machine of electrostatic field simulation in near future.