Abstract
BACKGROUND:
The human voice is the main feature of human communication. It is known that the brain controls the human voice. Therefore, there should be a relation between the characteristics of voice and brain activity.
OBJECTIVE:
In this research, electroencephalography (EEG) as the feature of brain activity and voice signals were simultaneously analyzed.
METHOD:
For this purpose, we changed the activity of the human brain by applying different odours and simultaneously recorded their voices and EEG signals while they read a text. For the analysis, we used the fractal theory that deals with the complexity of objects. The fractal dimension of EEG signal versus voice signal in different levels of brain activity were computed and analyzed.
RESULTS:
The results indicate that the activity of human voice is related to brain activity, where the variations of the complexity of EEG signal are linked to the variations of the complexity of voice signal. In addition, the EEG and voice signal complexities are related to the molecular complexity of applied odours.
CONCLUSION:
The employed method of analysis in this research can be widely applied to other physiological signals in order to relate the activities of different organs of human such as the heart to the activity of his brain.
Introduction
The brain is the main processing unit of the human body. It controls all actions/reactions of the human by sending impulses to different parts of the body. The voice is the main feature of communication for humans and its analysis is therefore of great importance. It is known that the human brain processes the voice.
For years, many works have investigated the activity of the human brain in different conditions. The major part of these studies analyzed how the human brain reacts to external stimuli. The works that studied the effect of visual [1, 2], auditory [3, 4], somatosensory [5], touch [6], and olfactory [7] stimuli on the human brain are worth mentioning. On the other hand, some scientists worked on the analysis of human voice, i.e. speech. The are several studies that investigated the effect of stress on human speech [8, 9], worked on the modelling of speech under different conditions [10], detected the effect of human’s emotional state on his voice [11], and performed statistical analysis on human speech sounds to predict musical universals [12].
Besides the reported works on the analysis of the activity of the brain or voice, no study investigated the relationship between brain and voice activity. Therefore, in this research, we investigated the relationship between human voice and brain activity. We aimed to make a link between voice and brain activities from a mathematical point of view. For this purpose, since EEG signal, as the feature of brain activity, and voice signal are random signals that have complex structures, we used fractal theory for the analysis. The fractal theory is a fast-growing mathematical technique that is used to define the complexity of self-similar or self-affine objects [13].
Through the years, researchers employed fractal analysis to analyze different time series and images in different areas of science and engineering. By focusing on the application of fractal theory in biomedical engineering, heart rate [14], eye movement [15, 16, 17, 18], electromyography signal (EMG) [19, 20, 21, 22, 23], magnetoencephalography (MEG) [24], face and DNA [25, 26, 27], respiration [28], s-ABR signal [29, 30], and Galvanic Skin Response (GSR) signal [31] are investigated by applying fractal analysis. Similarly, many studies employed the fractal theory to analyze human EEG signal in different conditions. The reported works that analyzed the effect of visual [32, 33] and auditory stimuli [34, 35], aging [36, 37], brain diseases [38, 39, 40] and body movements [41, 42, 43] on variations of EEG signal using fractal theory, are worth mentioning. However, the application of fractal theory in voice (i.e. speech) processing is very limited. The studies that applied fractal analysis for speech segmentation and phonetic classification [44], modelling of speech signals [45], prediction in waveform coding of speech signals [46], detection of voice disorder in patients [47, 48] and classification of speech [49] are worth mentioning.
Therefore, in this research, we used fractal theory to link the characteristic of voice to the characteristic of the brain activity by analyzing voice and EEG signals.
The rest of the paper is structured as follows: in the second section, the method of analysis based on fractal theory is presented. In the third section, the details of data collection procedure and analysis are provided. In the fourth section, the obtained results from the analysis is provided. In the fifth section, the obtained results and some future works are discussed.
Method
In this paper, we aimed to analyze the relationship between the complexity of human brain signal (EEG signal) and voice signal in different conditions. The applied conditions were the smells that were generated by fruit flavours (pineapple, banana, vanilla and lemon flavours) to stimulate subjects. For this purpose, we changed the activity of the human brain by applying different odours and analyzed whether human brain and voice signals are related when subjects read a text. Since EEG and voice signals are complex, fractal theory is employed to analyze their self-affine structures [50]. The fractal dimension is considered as the measure of the complexity of EEG and voice signals.
In general, fractal dimension can be computed using different methods that are mainly based on the entropy concept [51]. For a time series, the fractal dimension can have a value between 0 and 2. The greater value indicates a greater level of complexity [52]. The box counting method is the most famous method for computing the fractal dimension of objects. In this method, the object (i.e. signal in this research) is covered with a series of boxes in different steps, where the size of boxes is the same in each step. In each step, the algorithm counts the number of boxes that were used to completely cover the object [53]. After the algorithm repeated this procedure for different box sizes, the fractal dimension of an object is calculated based on the regression line that was fitted to log-log plot of a number of boxes (
The fractal dimension of order
where
where
We are interested in relating the human voice to the applied odours. In order to do this, we considered the complexity of odours by defining their molecular complexity. For our experiment, we chose four pleasant odours as olfactory stimuli and selected odours based on their molecular complexities. Bertz Formula (Eq. (1)) is applied for the calculation of molecular complexity that composes of two terms. The first term (
Figure 1 shows the molecular structure of selected odours. The first four selected odours include pineapple, banana, vanilla and lemon flavour, which all have increasing molecular complexities (see Table 1). As mentioned above, the odours were selected based on their complexities, which enabled us to investigate the relationship between the complexities of EEG signal, voice signal and odours.
The molecular structure of different odours (stimuli).
The chemical formula and molecular complexity of different odours (stimuli)
In different steps of the experiment, subjects were stimulated using different odours and then the relationship between the complexity of the voice signal and the complexity of the EEG signal were analyzed. The obtained result on fractal analysis of EEG and voice signals is discussed in relation to the variations of the molecular complexity of odours.
All procedures of recruiting subjects and conducting the experiment were approved by the Monash University Human Research Ethics Committee (MUHREC) under ethical number 18260. The study was carried out in accordance with the approved guidelines.
The experiment was conducted on ten healthy students (18–22 years old) from Monash University, Malaysia. We screened the subjects about their health conditions and drinking alcohol/coffee (within 48 hours prior the experiment). The screening was in the form of questions. All participants signed an informed consent form prior to the experiment.
The experiment was conducted in a quiet room in order to isolate subjects from unwanted stimuli that could affect the recorded EEG and voice signals. In the experiment, subjects were seated comfortably in front of the computer monitor. Subjects were instructed to sniff the odours and read the text from the monitor of a computer in front of them.
The collection of EEG signals was done using Muse headband with the sampling frequency of 256 Hz. It should be noted that the voice recording app on the mobile phone was applied to record the voice of subjects. For stimulation of subjects, 0.01 ml of each flavour (Fig. 2) was spilt on a small piece of paper and kept near the nose of the subjects and were then asked to sniff it while reading the text.
First, EEG and voice signals were recorded while subjects read the text from the computer monitor without sniffing an odour. Second, the subjects rested for one minute without receiving any external stimuli. Then, subjects sniffed the first odour while reading the same text, and their EEG and voice signals were recorded. After that, the subjects rested for one minute while sniffing coffee. This procedure was continued to collect EEG and voice signals from subjects in case of second, third and fourth odours, with one minute of rest between stimulations. The data collection was repeated in the second session to consider the repeatability of data.
Olfactory stimuli (fruit flavours).
Since subjects only received olfactory stimuli, the olfactory bulb of the brain was mainly stimulated and therefore only the recorded data from AF7 and AF8 channels was analyzed, as these are the closest electrodes to the olfactory bulb of the brain.
The raw EEG data was pre-processed to remove noises. For this purpose, a set of codes in MATLAB based on Butterworth filter with the frequency band of 0.5–30 Hz were written. It should be noted that due to the high sampling frequency of the recorded voice, we performed down-sampling on the voice data. For this purpose, we took a value between every 10 values. Then, we proceeded with the calculation of the fractal dimension of EEG and voice signals in MATLAB using the box counting method.
After computation of the fractal dimension for EEG and voice signals, statistical analysis had been performed on the obtained results. The effect of different stimuli on the variations of the fractal dimension of EEG and voice signals using effect size analysis was analyzed. Also, the significance of the difference between the fractal dimension of different pairs of data using posthoc Tukey test was analyzed. A significance level of 95% was considered in case of both statistical analyses.
In this section, the result of our analysis is presented. It should be noted that out of one hundred sets of data that were collected from the ten subjects, four sets of data did not fall within the proper range and were therefore excluded from the analysis.
Figure 3 shows the mean value of the variations of the fractal dimension of the EEG signal (a) and the molecular complexity of odours (b).
The fractal dimension of EEG signal and the molecular complexity of odours.
As can be seen in Fig. 3, the EEG signal has the lowest fractal dimension in case of no-odour condition. After that, the fractal dimension of the EEG signal increases when there is a move from the first odour to the fourth odour. Since fractal dimension indicates the complexity of the object, it can be concluded that the complexity of EEG signal increases with the increment of the molecular complexity of odours.
The results of effect size analysis are listed in Table 2, which indicate that the fourth odour had the greatest effect on the complexity of the EEG signal. The result of the posthoc Tukey test is also listed in Table 2. As can be seen from this table, there is no significant difference in the fractal dimension of the EEG signal between different pairs of conditions. Here, it should be noted that, to make the experiment relaxing for subjects, only a limited amount (0.01 ml) of odour was applied. Therefore, increasing the amount of odour could lead to a significant difference in the fractal dimension of the EEG signal.
The fractal dimension of the voice signal.
Figure 4 shows the variations of the fractal dimension of the voice signal. As can be seen in this figure, the voice signal has the greatest fractal dimension in case of no-odour condition. After that, the fractal dimension of voice signal decreases when there is a move from the first to the fourth odour. As mentioned before, fractal dimension indicates the complexity of the object, and it can therefore be said that the complexity of the voice signal decreases with the increment of the molecular complexity of odours (Fig. 3b). The effect size results (Table 3) indicate that the fourth odour had the greatest effect on the complexity of voice signal. However, the result of posthoc Tukey test in Table 3 indicates that there was no significant difference in the fractal dimension of the voice signal between different pairs of conditions. As mentioned before, presenting a greater volume of odours could lead to a significant effect on variations of the complexity of voice signal.
Comparing the obtained results in Fig. 4 with the variations of fractal dimension of EEG signal in Fig. 3a indicates that the variations of the complexity of voice signal are linked to the variations of the complexity of EEG signal.
Based on the obtained results, it can be said that since the brain controls the human voice, the variations in the complexity of human voice are related to the variations in the complexity of the brain signal.
In this paper, for the first time the relationship between human voice and brain (EEG) signals was analyzed in different conditions. The applied conditions include the application of different pleasant odours (using fruit flavours). Based on the obtained results, the fractal dimension of the EEG signal increases with the increment of the molecular complexity of applied odours. Similarly, the fractal dimension of the voice signal experiences greater changes with the increment of the molecular complexity of applied odours. Since fractal dimension stands for complexity, it can be said that the variations in the complexity of voice signals are related to the variations in the complexity of EEG signals as well as the molecular complexity of odours. Statistical analysis also confirmed the obtained results, where the fourth stimulus with the greatest complexity had the greatest effect on the complexity of EEG and voice signals. In addition, the obtained results showed the greater variations in the complexity of the EEG signal compared to the variations of the complexity of the voice signal.
Even though the obtained results from the posthoc Tukey test indicat an insignificant difference in the variations of the complexity of EEG and voice signals between different conditions, this result could be significant if we apply a greater amount of odours. Therefore, the variations in the complexity of the voice signal which are coupled with the variations in the complexity of the EEG signal have been successfully shown. In fact, this study is one step forward compared to the studies that only analyzed the variations of EEG or voice signals in different conditions without relating voice to brain activity.
The obtained results in this study can be elaborated by the mechanism of speech. As mentioned before, the human voice is controlled by the brain. On the other hand, different stimuli (i.e. odours in this research) change the level of brain activity. Therefore, when the level of brain activity changes, the level of voice complexity will change as well.
It should be noted that the analysis in this research can be expanded in case of subjects with speech problems in order to decode the relation between their voice and brain activities by fractal analysis of voice and EEG signals. Finding the relation between brain and voice activities in case of these patients can help physicians with finding better treatments.
The method of analysis in this research can be widely employed in order to analyze the relationship between different physiological signals and the brain activity in order to study the relationship between different organs of the human body and the brain activity. For instance, since the human heart is controlled by the brain, we can analyze the relationship between the complexity of heart rate and EEG signal in different conditions. In another example, since human eye movement is controlled by the brain, we can investigate how eye movement is linked to the brain activity by analysis of the complexity of eye movement versus EEG signal.
In another category of work, researchers can potentially generate a mathematical model between external stimuli, EEG signal and human voice (or other physiological signals). This model should be able to predict the complexity of the human voice based on the stimulus and EEG signal. For this purpose, different mathematical models [57, 58, 59] would be helpful, in particular fractional diffusion models [60]. Overall, these attempts can help scientists to understand how the human brain controls different parts of the human body.
Footnotes
Conflict of interest
None to report.
