Abstract
LTE-A downlink transfers data and control information from base station to mobile. To reduce the mean square error between original and estimated channel, pilot/training based channel estimation like Least Square Error (LSE) and Linear Minimum Mean Square Error (LMMSE) are ubiquitous for most wireless standards. To optimize the channel, many intelligent optimized techniques were developed. GA has no guarantee in finding global optima and high convergence time. ANN suits only linear solutions and more training period. PSO fits high dimensional space but needs more iterations. ABC has limited search space by initial solution. CS requires large resources and high computational time. To overcome these effects, an effortless Trellis Coded Firefly Optimized LMMSE based algorithm is proposed to estimate the channel. TCM has high spectral efficiency, more data rate and reduced error. FA has low complexity, easy implementation, automatic subdivision of groups to find local/global optima and ability to deal with multimodality. At SNR = 10 dB, LSE has high MSE of 10–2, LMMSE has 15.85% reduced MSE than LSE. The previous optimized methods have MSE ranging from 10–3 to 10–2 but the proposed method with 64-QAM has MSE range of 10–5 to 10–4, which is 100 times reduced.
Introduction
3GPP Release 10 LTE standard, referred to as Long Term Evolution Advanced (LTE-A), supports different new features like of wider bandwidth (carrier aggregation), advanced Multiple Input Multiple Output (MIMO) techniques, Coordinated Multipoint transmission or reception (CoMP), relaying and enhancements for Home eNodeB (HeNB) and fixed wireless customer premises equipment (CPE) RF requirements. LTE-A specification includes increasing the peak data rate and the enhancement of spectrum efficiency, latency and mobility. LTE-A physical layer (LTE-A PHY) downlink is responsible for carrying both data and control information between a base station (eNodeB) and mobile user equipment (UE) or user terminal (UT). Due to its several processing steps, the signal may be degraded and there is a need to estimate the channel characteristics at the receiver [11, 22].
Channel estimation [5, 9] is the process to estimatee the channel matrix containing the channel coefficients in an approximate manner compared to the original channel coefficients [21]. Channel estimation is classified as Pilot [15]/training based, blind and semi blind among which the training based channel estimation is more reliable and more prevalent supported for most wireless standards.
Pilot based channel estimation [15] is again divided into Least Square Error (LSE) and Linear Minimum Mean Square Error (LMMSE). LSE is simple to implement but has high Mean Square Error (MSE). LMMSE is better than LSE with reduced mean square error, but it is highly complex due to the calculation of matrix inversion each time the channel is time varying due to the fading phenomenon [20].
To mitigate the effect of fading and to reduce the channel MSE, so many intelligent or optimization methods are developed. A literature review of intelligent methods like Particle Swarm Optimization (PSO) [1, 2], Artificial Bee Colony (ABC [8]), modified Cuckoo Search (CS [16]) and improved PSO [14] are done.
In PSO [1, 2], there are three standards like standard PSO, Co-operative PSO (CPSO) and multi-objective PSO. CPSO is applied for MIMO channel estimation to obtain faster convergence and to lower overall complexity. Also, the iterations are known in prior to the receiver. PSO and GA share many similarities but the difference lies within the selection of leaders (in terms of PSO) or parents (in terms of GA) as well as the update of position and/or generation of new members, respectively. PSO is advantageous compared to GA in terms of fewer parameters, computational complexity, convergence speed, and accuracy. CPSO parameters do not need to be tuned by empirical measures, since the algorithm can directly be applied for MIMO Channel estimation.
By combining LSE and LMMSE, Artificial Bee Colony [8] (ABC) optimization is more efficient and requires less time to estimate the best channel. It is a metaheuristic algorithm developed based on the foraging behavior of honey bees. Here, ABC is applied to both LSE and LMMSE and the best channel with minimum error is selected among them.
Then, a Hybrid optimization method based on Bacterial Foraging Optimization (BFO) and Modified Cuckoo Search algorithm (MCS [16]) called (HBFOMCS [16]) is used to optimize placement of the Pilot tones that are used for Least Square (LS) channel estimation in MIMO-OFDM systems. It provides high system performance and superior search ability. It provides an alternative solution to achieve both high-performance efficiency and low-computational complexity in designing Pilots [15].
Later, a method based on improvement in Particle Swarm Optimization based was developed that was applied over Least Square estimator for transmit diversity. Here, QPSK modulation scheme is adopted with 64 sub-carriers, Additive White Gaussian Noise and channel is estimated using LSE. Channel is estimated using Adaptively Regularized Least Square method and then PSO is applied to LSE to find the optimum solution. The LSE performance is improved slightly by applying improved PSO [14] scheme.
Anyway, global optimization is challenging to solve due to its nonlinearity and multimodality. Firefly algorithm [23] (FA) is a new biologically inspired meta-heuristic optimization algorithm, which was proposed by Xin-She Yang in 2008. This algorithm is inspired by the flashing behavior of tropical fireflies. FA has excellent convergence rate and strong exploration ability.
To overcome the time consuming in generating best optimal solution and to reduce the premature convergence, a trellis coded Firefly algorithm is proposed for LMMSE based channel estimation with simplicity and fast convergence. FA is suitable for high level of noise.
Section 2 defines the system model, Section 3 deals the existing methods, Section 4 explains the proposed method, Sections 5 and 6 details the simulation results and discussion and Section 7 gives the conclusion.
System model
Figure 1 illustrates the system model used for implementation. The binary information from different number of users is sent to Trellis coded modulation [6, 17] (TCM) block.

Proposed system model.

General trellis coded modulation.
Figure 2 shows the general form of Trellis coded modulation (TCM). It has Convolutional encoder with coding rate of
TCM [6] has many advantages like, more bandwidth efficient, increases bit rate without altering the symbol rate, invariant phase, set partitioning, provides sequence coding, contains soft decision decoding and uses Euclidean distance instead of hamming distance. Here, the bits are Convolutional encoded with a code rate of 1/2 and digitally modulated by any one of the schemes like Binary Phase Shift Keying (BPSK), Quadrature Phase Shift Keying (QPSK), 16-Quadrature Amplitude Modulation (16-QAM) and 64-Quadrature Amplitude Modulation (64-QAM). Code rate as 1/2 refers to, for each bit input, the Convolutional encoder outputs two bits. For BPSK 1 bit/symbol, for QPSK 2 bits/symbol, for 16-QAM 4 bits/symbol and for 64-QAM 6 bits/symbol are transmitted.
The trellis coded symbols are sent to the layer mapping in which the modulated symbols are mapped to one/more transmission layers based on the number of transmitting antennas. The layer mapped symbols are multiplied with a Precoding matrix and value of precoding matrix elements are different for different type of modulation used.
Precoding [18] is a method that exploits the transmit diversity scheme by weighting information. The transmitter sends the coded information to the Receiver [19] in order to pre-knowledge the channel. In MU-MIMO, multiple transmitters communicate simultaneously with multiple receivers (SDMA). Figure 3 shows the 8×8 MIMO without and with Precoding.

(a) 8×8 MIMO without precoding. (b) 8×8 MIMO with precoding.
Precoding can be linear or non-linear. Linear precoding [13] achieves reasonable performance with much lower complexity which include maximum ratio transmission (MRT), zero-forcing (ZF) precoding and transmit Wiener precoding. The precoder output is given by,
In subcarrier/resource element mapping, the users’ information is allocated to different subcarriers and the allotment of user symbol in different subcarrier form a resource block. The subcarrier mapping is classified as interleaved/distributed.
Figure 4(a) shows the Interleaved subcarrier allocation scheme called interleaved – OFDMA (I-OFDMA), in which the subcarriers are allocated in sequence to the users and repeated. A user is assigned subcarriers that are uniformly distributed over a given band has systematic phase rotation [5, 12].

(a) Interleaved subcarrier mapping. (b) Localized subcarrier mapping.
Figure 4(b) shows the localized subcarrier mapping scheme called localized OFDMA (L-OFDMA), in which the subcarriers are mapped adjacent to each other. The main advantage of L-OFDMA is the feasibility of multiuser diversity, which leads to improved system capacity and/or performance [5, 12].
I-OFDMA is chosen as subcarrier type since it is slightly improved at high signal to noise ratio (SNR) with reduced bit error rate than the L-OFDMA.
The resource blocks are further processed by Inverse Fast Fourier Transform (IFFT) to convert the frequency domain signals into time domain information. After IFFT, the cyclic prefix is added to the symbols to reduce ISI. IFFT and CP Adder are called OFDM mapper in LTE-A system.
Then time varying signals from the LTE-A downlink transmitter are sent through a Rayleigh faded AWGN channel that affects the signals by multipath fading effects [7].
In LTE-A downlink receiver [19], the fading signals are received and the cyclic prefix part added in transmitter is removed. Then FFT is taken to convert time domain into frequency domain signal. The resource element block is chosen in which the corresponding user whose data has to be retrieved. Here, User 1 data is reconstructed in the receiver. User equipment is the mobile that contains the firefly optimized channel estimation to retrieve the channel coefficients [21] in an effective manner. Reverse precoding divides the received symbols by precoding matrix and the specific layer is selected from which the user 1 data can be reconstructed. Trellis decoder uses Viterbi algorithm to detect and demodulate User1 data. Though the data is demodulated effectively, it contains more Bit Error Rate (BER).
To reduce BER, there is a need to estimate the channel state information (CSI) in LTE-A system. The pilot/training based channel estimation includes Least Square Error (LSE) and Linear Minimum Mean Square Error (LMMSE). LSE does not use channel statistics and is easy to implement. But it has high MSE. LMMSE reduces the MSE but is highly complex to implement since the matrix inversion has to be taken each time the channel is varying in wireless environment [20].
For efficient channel estimation, in this paper, a trellis coded LMMSE based firefly optimized algorithm is developed to reduce the channel MSE and locate the channel coefficients [21] near to the original channel coefficients.
In LTE-A downlink [10, 22] channel processing, the reference signals are extracted and the channel estimation is done based on the cell – specific reference signals only. In mobile communication, channel estimation is hard due to the multipath fading effect and there is a need to recover the transmitted information correctly. The Pilot [15]/reference signal based channel estimation like Least square (LS) and minimum mean square error (MMSE) are vital.
LS estimation includes only transmitted and received symbols without considering channel statistics. The LS estimated channel coefficient is,
Minimum mean square error (MMSE) channel estimation is superior to LS since it involves the channel statistics to reduce the MSE. It filters the estimate by G
MMSE
. MMSE estimate is given by,
LS is appropriate for high SNR and MMSE is for low SNR. When SNR increases, signal power increases for transmitting the information. High power transmitter is hard and impractical to construct in hardware. Hence, there is a need to optimize the channel coefficients when it is getting affected by channel fading [7], noises and interferences.
The existing channel estimation and optimization methods had disadvantages like premature convergence and time consumption. To overcome these, in this paper, trellis coded firefly based LMMSE channel estimation is proposed to estimate the channel with reduced mean square error and select the optimized location of channel coefficients.
Optimization plays an important role in the area of computational intelligence to choose the best optimal solution among all possible sets of solutions. Optimization algorithms are classified as deterministic and stochastic. Deterministic algorithms are determining the same set of results when applying the same set of inputs every time the same algorithm is applied. Stochastic algorithms produce different set of results every time it takes the same set of inputs. Stochastic algorithms are in turn divided in to heuristic and metaheuristic. Heuristic algorithm takes reasonable amount of time to generate output but it does not give optimum solution. Metaheuristic algorithms use randomization and local search where randomization refers to global optimum solution.
Firefly algorithm [23] is a type of metaheuristic algorithm, developed by Dr. Xin – Shi Yang, based on the natural behavior of the fireflies. It is based on the phenomenon of bioluminescence that produces light to communicate with each other and attract other fireflies. The components of optimization problem include fitness function, set of possible solutions and optimization rule.
The proposed channel estimation algorithm uses a trellis coded firefly based LMMSE method that reduces the multipath fading effect and results in minimum MSE. Figure 5 shows the flowchart for the proposed channel estimation algorithm. At first, the channel is estimated using LSE and LMMSE. According to previous literatures, it has been proved that LMMSE is better than LSE with reduced MSE but involves inverse matrix calculation each time the channel is time varying due to fading in multipath wireless environment. Also, the position of the channel coefficients in space deviates its location due to multipath fading effect [7, 20].

Flowchart of the proposed firefly algorithm.
If the MSE of LMMSE is lesser than MSE of LSE, the firefly algorithm is applied to locate the position of channel coefficients in wireless environment near to the original channel position by altering the distance and velocity that act as light intensity of the fireflies. For a given medium, as per the involvement of inverse square law, the light intensity varies with respect to distance and is given by,
The distance in two dimensional space is given by,
The movement of a firefly i is attracted to another more attractive (brighter) firefly j is determined by,
The attractiveness is updated by,
Otherwise, a set of random channel coefficients are assumed within the local search area and their fitness values are calculated. If the MSE of any one firefly is lesser than MSE of LMMSE, the corresponding firefly is chosen to be the best channel coefficients. If MSE of all fireflies are greater than MSE of LMMSE, for every iteration, the fireflies are moved to the nearest channel coefficient location based on the light intensity. Among them, the best firefly based channel coefficient matrix with minimum MSE is selected.
The performance of the algorithm depends on success rate, number of fireflies and the number of iterations. Increasing the number of fireflies results in increase of the number of iterations. The optimum best solution is determined by using minimum number of fireflies. According to the previous analysis [23], reduction in number of fireflies, light absorption coefficient γ and randomization parameter α results in a best optimum solution.
The simulation parameters for implementation of LTE-A downlink in MATLAB are given in Table 1.
Simulation parameters used for implementation
Simulation parameters used for implementation
Due to the multipath effect, the position of channel coefficients in search space is deviated which results in high MSE. An effortless algorithm is used to reduce MSE.
Channel mean square error (MSE) is the ratio of difference of original and estimated channel to the original channel given by,
The MSE of Least Square Error Estimation is,
The MSE of Linear Minimum Mean Square Error Estimation is given by,
The mean square error of the proposed firefly based LMMSE channel estimation algorithm is,
Figure 6 shows the channel MSE for BPSK which sends one bit/symbol. At SNR = 10 dB, the channel MSE is 10–1.56 for LSE, 10–2.21 for LMMSE and 10–4.36 for the proposed channel estimation, less compared to LSE and LMMSE.

Channel mean square error for BPSK.
Figure 7 shows the channel MSE for QPSK modulation that carries 2 bits/symbol. The plot proves that the proposed method has channel MSE ranging from 10–5.2 to 10–4.2 for QPSK.

Channel mean square error for QPSK.
At SNR = 10 dB, the channel MSE is 10–1.56 for LSE, 10–2.21 for LMMSE and 10–4.36 for the proposed channel estimation which is less than LSE and LMMSE, but the performance is almost similar to the BPSK.
Figure 8 shows the channel MSE for 16-QAM modulation that carries 4 bits/symbol. The proposed method has channel MSE ranging from 10–4.95 to 10–3.9. At SNR = 10 dB, channel MSE is 10–1.86 for LSE, 10–2.5 for LMMSE and 10–3.66 for proposed method that is lower than existing methods. 16-QAM has reduced MSE than BPSK and QPSK.

Channel mean square error for 16-QAM.
Figure 9 shows the channel MSE for 64-QAM that carries 6 bits/symbol and proves that the firefly based optimization has MSE ranging from 10–4.95 to 10–4. At SNR = 10 dB, channel MSE is 10–1.97 for LSE, 10–2.61 for LMMSE and lowest MSE of 10–4.774 for the proposed method. 64-QAM has lowest MSE than other modulations.

Channel mean square error for 64-QAM.
As a result, the proposed method suits all modulation types, effective for 64-QAM with MSE range from 10–5 to 10–4 and is suitable for LTE-A.
The performance metric used to compare the existing and the proposed method is the mean square error. Figure 10 shows the performance comparison of LSE for different modulations.
Among them, 64-QAM has small MSE when using LSE based channel estimation. LSE does not include the noise statistics for channel estimation. It is simple to implement but has high MSE.

Channel MSE for LSE with different modulation techniques.
Figure 11 shows the comparison of MSE of LMMSE for different modulations that proves 64-QAM has lower MSE. To reduce the MSE and complexity further, in the proposed method, a random channel is assumed locally around original channel location for each generation/iteration and the fitness MSE is calculated. The best channel is chosen based on the minimum MSE. Hence it is easy to implement.

Channel MSE for LMMSE with different modulation techniques.
Figure 12 shows the comparison of all modulation schemes using the proposed method. A set of random solutions are assumed and fitness is calculated. Within the local search area, best channel with 64-QAM is obtained around 150–200 generations. As a result, 64-QAM trellis coded LMMSE based firefly optimied channel estimation is the best one.

Channel MSE for proposed firefly optimized method with different modulation techniques.
Scatter plot is a 2D representation of channel coefficients. Since the channel length is five, there are five points in Fig. 13. In phase component refers to real parts and quadrature phase component refers to imaginary parts of channel matrix. Red color stars denote original channel, green color stars denote LSE, blue color stars indicate LMMSE and black color stars denote the firefly optimized channel.
Figure 13 proves that the position of the firefly optimized channel matrix is nearly close to the original channel matrix and the concentric circles denote the local search area. Here, the optimization is based on displacement of point with light intensity.

Scatter plot of channel coefficients.
The channel frequency response is the fourier transform of channel impulse response and has both magnitude and phase responses.
Figure 14 shows the magnitude responses of original, LSE, LMMSE and the proposed methods. Since FFT length is 64, there are 64 points in channel frequency response. LSE has high MSE and is entirely different in magnitude response compared to original channel. LMMSE is almost similar to original channel but reduces little in magnitude due to the effect of multipath fading. To relocate the position of channel coefficients near to the original points, a Firefly algorithm is applied over LMMSE to increase the magnitude response.

Magnitude response of existing and proposed methods.

Phase response of existing and proposed methods.
Figure 15 shows the phase responses of original, LSE, LMMSE and the proposed firefly based channel estimation methods. LSE has high MSE and so it is entirely different in phase response compared to original channel. LMMSE has similar response of original channel but some phase response are missing due to multipath fading. To relocate the position of channel coefficients near to the original points, the firefly algorithm is applied over LMMSE without altering the phase response.
Thus, the magnitude and phase responses of LMMSE are almost similar to original channel but some values are missing for some points after 40. Hence, to locate the LMMSE channel estimation points near to original channel position without altering the frequency response of the channel, the proposed algorithm is employed.
Normalized frequency response is the frequency response of channel in which the normalized frequency ranges from 0 to π radians. Figure 16 shows the normalized frequency response of original channel. The magnitude response of original channel ranges from –300 dB to 0 dB and the phase response of original channel ranges from –170° to –50°.

Normalized frequency response of original channel matrix.
Figure 17 shows the normalized magnitude and phase responses of LSE are very linear but deviated from original channel with high MSE. The magnitude response of LSE ranges from –9 dB to 40 dB and the phase response of LSE estimated channel ranges from –6000° to 0°. This values are very far when compared to the original channel matrix response.

Normalized frequency response of LSE channel matrix.
Figure 18 shows the normalized magnitude and phase responses of LMMSE in which both responses are stable only after 0.4 radian/sample. The magnitude response of LMMSE ranges from –270 dB to 0 dB approximately equal to the original channel but phase response of LMMSE ranges from –2900° to –2700° which is more deviated than original channel. So LMMSE is suitable only for high frequency applications and also high complex to implement in fading environment.

Normalized frequency response of LMMSE channel matrix.
Figure 19 shows the normalized magnitude and phase responses of the proposed method in which both responses are approximating to the original channel for all frequency ranges. The magnitude response ranges from –240 dB to 0 dB approximately nearer to the original channel and the phase response ranges from –270° to –70°, nearing the original channel. So it is suitable for both low and high frequency applications and is able to locate an optimized channel coefficients using local search.

Normalized frequency response of Firefly optimized channel matrix.
Hence, the proposed algorithm shows an improved performance in achieving the normalized magnitude and phase responses almost similar to the original channel matrix with reduction in MSE.
To mitigate the multipath fading effects in wireless propagation environment, there is a need to estimate the channel in an effective manner. Although a lot of basic and intelligent optimized methods have been developed, each having some disadvantages. Genetic Algorithm (GA) has no guarantee of finding global maxima, more time taken for convergence, inefficiency, incomprehensible and hard to design a suitable fitness function [2]. Artificial Neural Network (ANN) is only suitable for simpler solutions like linear regression, has more training period and increases only few percent of accuracy [4]. Particle Swarm Optimization (PSO) limitations include easy falling into local optimum in high-dimensional space and have a low convergence rate in the iterative process.
Since the wireless channel is a large open space, the optimization is hard and sometime results in false positive results [1, 2]. Artificial Bee Colony (ABC) has limitations like lack of use of secondary information, losing relevant information sometimes, slow in sequential processing and increased computational cost [4].
TCM improves the data rate and conserves bandwidth. FA optimizes the channel effectively with reduced mean square error. LMMSE has low MSE than LSE. Combining all the three concepts in to one proposed scheme improves the effectiveness of the channel estimation. Referring Fig. 9, the qualitative analysis showed that for SNR = 10 dB, LSE has high MSE of 10–2 and simple to implement. LMMSE has 15.85% reduced MSE than LSE but high complexity due to the matrix inversion each time the channel is varying. The existing optimization schemes like GA, ANN, PSO and ABC has channel MSE ranging from 10–3 to 10–2. The proposed method has MSE ranging from 10–5 to 10–4 and has 7.89% reduced MSE than LMMSE and 23.74% reduced MSE than LSE for 64-QAM.
As per the implementations [1, 21], the technical reviews prove that any type of heuristic or metaheuristic optimization schemes are applied to OFDM and OFDMA schemes to estimate the channel. LTE-A downlink uses OFDMA to transfer data and control information from base station to user mobile. Hence, the proposed method uses Firefly Algorithm as a metaheuristic algorithm and it suits any 4 G and future wireless generations to estimate the channel.
Figures 6–12 proves that the proposed method has low MSE in the range of 10–5 to 10–4 and 64-QAM is more advantageous to use since it sends 6 bits per symbol. When the number of bits per symbol increases, the bandwidth efficiency improves and increases the channel utilization. Since the wireless channel is a high dimensional space, both local and global optimal solutions are needed to calculate and obtain the best optimal solution by estimating the MSE each time the channel is varying.
Comparing Figs. 16 and 19, the implementation proves that the channel frequency responses of original and proposed method are almost similar. If there is a phase deviation or magnitude change in the channel matrix due to the multipath fading effects, the best channel matrix is selected with minimum MSE by the proposed scheme. Since the algorithm is implemented inside mobile equipment embedded as Matlab coding, it does not alter the cost and size of the mobile. Previous optimized channel estimations involve 500 to 1000 generations to get optimal solution but the proposed method uses only 150–200 generations to get the global and local optimal solutions, reduces the computational cost and increases the speed.
The computational complexity of FA is in terms of evaluating objective functions. FA has two inner loops through population n and one outer loop for iteration i. If n is small (50) and i is large (200), the computational cost is inexpensive due to linearity in terms of i, equals to extreme case O (n2i). If n is large, one inner loop is used by ranking the attractiveness/brightness of all the fireflies using sorting algorithms and the computational complexity is O (ni log(n)) [23]. The same is applicable to the proposed scheme that reduces the computational cost and improves the computational accuracy by increasing convergence rate since optimal solution is obtained with only 200 generations/iterations.
The simulation results prove that the proposed method suits for all generations of wireless standards, low channel MSE ranging from 10–5 to 10–4, almost same frequency responses as that of original channel, achieves the optimal solution with only 200 iterations and proves 64-QAM has low MSE compared to BPSK, QPSK and 16-QAM.
Conclusion
LTE-A downlink transfers data and control signals from base station (eNodeB) to user equipment (mobile). Since the wireless channel is time varying in nature, there is a need to estimate the channel coefficients efficiently to reduce mean square error between the original and the estimated channel in the receiver. The previous analysis proves that the channel MSE of PSO range from 10–2.8 to 100, 1.9% less MSE for GA, zero MSE for ANN by appropriate training and high system performance with low computational complexity for ABC. To mitigate the disadvantages of existing optimized methods, a new intelligent technique called trellis coded LMMSE based firefly algorithm is proposed. Simulations proves that the proposed method with 64-QAM has low MSE ranging from 10–5 to 10–4 which is 23.74% less than LSE, 7.89% less than LMMSE and 1.5% MSE reduction than existing optimization schemes, low computational complexity, minimum computational cost (only 200 iterations), high speed ability to find local and global optima and suits high dimensional wireless environment. Also, it suits all frequency ranges (as LTE-A) and its future generations.
Footnotes
Acknowledgments
A special thanks to Keysight Technologies (Formerly Agilent Technologies), Bangalore for providing SYSTEMVUE software and a sincere thanks to my research guide (coauthor) for her constant support in completing this paper work.
