Performance testing for different peak expiratory flow meters

Abstract

BACKGROUND:

Peak expiratory flow meters (PEFMs) have emerged as primary tools used for diagnosing and monitoring a range of respiratory diseases including asthma and chronic obstructive pulmonary disease, and the performance of these meters will thus impact disease evaluation.

OBJECTIVE:

The aim of this study was therefore to assess the technical performance of mechanical and electronic PEFMs commonly used in clinical practice.

METHODS:

The accuracy, repeatability, airflow resistance, frequency response, and linearity of five electronic and seven mechanical PEFMs were measured using a standard flow/volume simulator in accordance with nine A-waveforms and three B-waveforms defined in ISO 23747:2015 issued by the International Standards Organization (ISO).

RESULTS:

The accuracy, repeatability, linearity, airflow resistance, and frequency response pass rates for these 12 different PEFM brands were 41.67%, 75.00%, 50.00%, 75.00%, and 25.00%, respectively. Just 16.67% (2/12) of the tested PEFMs met all evaluated criteria, whereas the remaining PEFMs partially met these criteria. There were no significant differences between the two tested PEFM types in the low flow rate waveform test ( $P>$ 0.05), although there were significant differences in the medium and high flow rate waveform test ( $P<$ 0.05). In addition, the overall PEFMs test had poor accuracy and good repeatability, although most of the repeatability errors occurred in the BTPS state.

CONCLUSION:

PEFMs commonly used in clinical settings exhibit variable technical performance, and relevant departments need to strengthen PEFM quality control and management in China.

Keywords

Peak expiratory flow meter standard flow/volume simulator quality control ISO 23747:2015

1. Introduction

Peak expiratory flow meters (PEFMs) are valuable medical tools used to measure peak expiratory flow rate (PEF) [1]. First developed in a simplified form by British bioengineer Martin Wright in 1956, PEFMs have grown more advanced over time with technological development and now offer key advantages including low cost, simple structure, convenience, and ease of use that enable patients to reliably conduct self-monitoring at home. Routine PEFM-based assessment of PEF variations is now a routine method employed when diagnosing and evaluating the current severity of conditions such as bronchial asthma and chronic obstructive pulmonary disease [2]. Overall, this approach is particularly well-suited to primary screening and follow-up disease management in patients with chronic respiratory diseases diagnosed in primary medical institutions [3, 4, 5].

Currently, commercially available PEFMs are primarily divided into mechanical and electronic subtypes [6, 7], but the relative performance of these different devices remains to be rigorously assessed owing to manufacturer-specific design differences. In order to strengthen the ability of medical workers to monitor and diagnose pulmonary function and to ensure that PEFMs meet the minimum requirements for safety and performance, it is important that criteria for PEF measurement be developed. In 1994, the American Thoracic Society (ATS) first proposed a PEFM testing standard in which 26 waveforms were used to gauge the ability of a given PEFM to accurately measure PEF values [8]. However, subsequent studies noted the redundancy and inaccuracy of some of these waveforms [9], with the repeatability of some of these waveforms ultimately being shown to be impossible to achieve owing to limitations inherent in the simulator pump system necessary for PEFM testing. In light of this issue, in 2015 the ISO issued the updated “Anaesthetic and respiratory equipment – Peak expiratory flow meters for the assessment of pulmonary function in spontaneously breathing humans ISO 23747:2015” standard, in which A and B waveforms were recommended for use in PEFM evaluation [1]. These two waveforms cover a 90% confidence limit for the specific characteristics associated with PEF measurements, with the rise time (RT) and dwell time (DT) of the A-waveform at the 95th percentile and the RT and DT of the B-waveform at the 5th percentile, thus covering 90% of the characteristics of the expected patient population.

Multiple brands of PEFMs have emerged in the Chinese market, but the performance of these instruments remains to be evaluated. Therefore, this study utilized the standard flow/volume simulator to test and evaluate the performance of PEFMs available in the Chinese market according to the method of ISO 23747 standard.

2. Objects and methods

2.1 Research objects

For the present study 12 PEFMs of different brands were evaluated, including both mechanical and electronic meter types (Fig. 1). Mechanical PEFMs assessed in this study included the MINI-Wright PEFM (UK), AIRZONE PEFM (UK), Dofin USPE (China), Monaghan Truzone (USA), Philips Personalbest (USA), Koka PEF-3 (China), and Inspired Medical PEFM (China), while the electronic PEFMs included the eMINI-Wright PEFM (UK), Vitalograph Asma-1 (USA), BreathHome A1 (China), and CONTEC SP80B (China), Sonmol HCY-1 (China). All PEFMs were obtained in 2020 in a new, unused state.

Figure 1.

The tested instruments.

2.2 Experimental equipment

All experimental testing was performed using a standard flow/volume simulator (Model 1120, Hans Rudolph, USA), hereafter referred to as a simulator (Fig. 2) This simulator is equivalent to a standard air source, and can simulate different respiratory movements of the human body.

Figure 2.

Standard flow/volume simulator.

2.3 Waveform testing

When evaluating PEFMs, the ISO 23747:2015 standard employs the use of two waveforms (A-waveform and B-waveform) that cover a 90% confidence interval associated with specific PEF characteristics.

2.4 Testing approach

Prior to testing, PEFMs and the simulator were warned to 15 ${{}^{\circ}}$ C–25 ${{}^{\circ}}$ C for 15–20 minutes. The PEFM being tested was then connected to the standard simulator using a mouthpiece and a rigid smooth bore connector, with the orientation of the PEFM being adjusted in accordance with the provided operation manual and with particular attention being paid to connection tightness. Then a series of 9 A-waveforms were generated to account for different environmental conditions and conveyed to the PEFM, including 7 A-waveforms at normal ambient temperature, humidity, and atmospheric pressure (ATP state) and 2 A-waveforms at 37 ${}^{\circ}$ C, standard atmospheric pressure, and saturated water vapor (BTPS state) as follows: 100 L/min (1.67 L/s), 150 L/min (2.50 L/s), 200 L/min (3.33 L/s), 300 L/min (5.00 L/s), 450 L/min (7.50 L/s), 600 L/min (10.00 L/s), and 720 L/min (12.00 L/s) in the ATP state; 300 L/min (5.00 L/s), and 600 L/min (10.00 L/s) in the BTPS state. Each flow rate was repeated five times, with the peak pressure and PEF values being recorded for each trial.

In addition, three B-waveform gas sources with different flow rates under the ATP state were conveyed to the PEFM as follows: 200 L/min (3.33 L/s), 300 L/min (5.00 L/s), 600 L/min (10.00 L)/s). As above, testing was repeated five times, with PEF values being recorded for each trial.

2.5 Performance evaluation indicators and reference ranges

Accuracy: PEFM error ( $e_{n}$ ) was calculated for each recorded PEF ( $q_{n}$ ) based on the corresponding reference PEF ( $q_{\textit{ref},n}$ ). The acceptable reference range was defined as a maximum permissible flow rate error of $\pm$ 10 L/min ( $\pm$ 0.17 L/s) or $\pm$ 10 % of the reading, whichever is greater.

$\displaystyle e_{n}=\bar{q}_{n}-q_{\textit{ref},n}$ (1)

Repeatability: The span ( $S_{n}$ ) of PEFM readings for five recorded PEF ( $q_{n}$ ) values at each flowrate level was calculated. The acceptable reference range was defined as variations in PEFM readings at any PEF within the assessed measurement range of no more than 10 L/min (0.17 L/s) or 5% of the mean of the readings, whichever is greater.

$\displaystyle S_{n}=q_{\max,n}-q_{\min,n}$ (2)

Linearity: The error ( $e_{n}$ ) from Eq. (1) was utilized to calculate the linearity ( $\epsilon_{n}$ ) of recorded PEF ( $q_{n}$ ) values within the assessed measurement range. The acceptable reference range was defined as linearity ( $\epsilon_{n}$ ) that does not exceed 5% of the larger of the two test flowrates.

$\displaystyle\epsilon_{n}=(e_{n}-e_{n+1})\times 100/\bar{q}_{n+1}$ (3)

Flow resistance: The resistance ( $R_{n}$ ) was calculated with peak pressure ( $P_{n}$ ) at each flowrate and corresponding reference PEF ( $q_{\textit{ref},n}$ ). The acceptable reference range was defined as resistance ( $R_{n}$ ) $<$ 0.35 kPa/(L/s) or 0.006 kPa/(L/min).

$\displaystyle R_{n}=P_{n}/q_{\textit{ref},n}$ (4)

Frequency response: The PEFM frequency response ( $F_{n}$ , in %) was calculated based upon the PEFM, using the average difference between the A-waveform ( $q_{n,A}$ ) and the B-waveform ( $q_{n,B}$ ), with the result being expressed as a percentage of A-waveform readings. The acceptable range was defined as a frequency response ( $F_{n}$ ) of 12% or $<$ 15 L/min (0.25 L/s).

$\displaystyle F_{n}=100\times(\bar{q}_{n,A}-\bar{q}_{n,B})/\bar{q}_{n,A}$ (5)

2.6 Statistical methods

SPSS 25.0 was used to analyze all data, which are expressed as the mean $\pm$ standard deviation and Graphpad prism 8.0 was used for making scatter diagram. The student $t$ -test was used to compare the differences between PEFM types. $P<$ 0.05 was the threshold of significance.

Table 1
Evaluation of the technical evaluation of two different types of PEFMs

Instrument	Accuracy	Repeatability	Linearity	Flow resistance	Frequency response	Result
Mechanical
MINI-Wright	Y	Y	Y	Y	Y	Meeting
AIRZONE	N	Y	Y	Y	N	Partially meeting
Dofin-USPE	N	Y	N	N	N	Partially meeting
Monaghan-Truzone	N	Y	N	Y	N	Partially meeting
Philips-Personalbest	N	Y	Y	Y	N	Partially meeting
Koka-PEF-3	N	N	N	Y	N	Partially meeting
Inspired Medical-PEFM	Y	N	N	N	N	Partially meeting
Pass rate (%)	28.57	71.43	42.86	71.43	14.29
Electronic
eMINI-Wright	Y	Y	Y	Y	Y	Meeting
Vitalograph-Asma-1	Y	Y	N	Y	N	Partially meeting
BreathHome-A1	Y	N	Y	Y	N	Partially meeting
CONTEC-SP80B	N	Y	Y	Y	N	Partially meeting
SonmolHCY-1	N	Y	N	N	Y	Partially meeting
Pass rate (%)	60.00	80.00	60.00	80.00	40.00

Remarks: Y: passed the standard; N: failed the standard.

3. Results

In total, a simulator was used to evaluate 12 PEFM instruments in the present study. However, instrument 4 was unable to measure the A-waveform at 100 or 150 L/min, while instruments 3 and 12 were unable to measure the A-waveform at 720 L/min. Through these analyses, 104 sets of measurement data were generated and used to assess the accuracy, repeatability, linearity, flow resistance, and frequency response values for each of these PEFMs (Table 1). For these 12 instruments, the accuracy, repeatability, linearity, flow resistance, and frequency response pass rates were 41.67% (5/12), 75.00% (9/12), 50.00% (6/12), 75.00% (9/12), and 25.00% (3/12) respectively. Among the tested mechanical PEFMs, the respective accuracy, repeatability, linearity, low resistance, and frequency response pass rates were 28.57% (2/7), 71.43% (5/7), 42.86% (3/7), 71.43% (5/7), 14.29% (1/7), while for electronic PEFMs they were 60.00% (3/5), 80.00% (4/5), 60.00% (3/5), 80.00% (4/5), 40.00% (2/5), respectively. Just 16.67% (2/12) of the tested instruments successfully met all performance evaluation criteria, whereas the remaining instruments partially fulfilled these criteria. There were no significant differences between mechanical and electronic PEFMs in the low flow rate waveform test ( $P>$ 0.05), although they did differ significantly in the medium and high flow rate waveform test ( $P<$ 0.05), as shown in Table 2.

The relative errors between PEF measurements from all instruments and standard values of the A-wave were plotted in the form of an accuracy scatter plot, and the span of PEF measurements under each A-wave was plotted in the form of a repeatability scatter plot. As shown in Fig. 3, 66.35% (69/104) of the PEF accuracy points were within the acceptable range. Of these, 77.27% (34/44) of the points associated with electronic PEFMs, and 58.33% (35/60) of the points associated with mechanical PEFMs were within the acceptable range. As shown in Fig. 4, 96.15% (100/104) of the PEF repeatability points were within the acceptable range, and most of the errors occurred in the BTPS state. Of the electronic PEFMs, 97.73% (43/44) were in the acceptable range, whereas 95.00% (57/60) of the mechanical PEFMs were within the acceptable range.

4. Discussion

The accuracy and repeatability of PEFMs can effectively evaluate the severity of disease and the trend of disease progression. However, in previous studies where subjects were tested using different PEFMs, these measurements exhibited varying levels of accuracy [11, 12]. Vanzeller et al. [13] also employed a simulator to test nine PEFMs and spirometers with CE (European Conformity) marks, and found that only 22.2% of the instruments fully met the performance requirements. All prior studies have reported pronounced variability in the performance of different PEFMs.

In this study, only 16.67% (2/12) of the 12 instruments completely passed the ISO standard. Overall, 41.67% (5/12) of the instruments in this study passed the accuracy standard, and the respective pass rates for mechanical and electronic PEFMs were 28.57% (2/7) and 60.00% (3/5), with the lower pass rate for mechanical PEFMs potentially being attributable to variability with respect to the performance of sliding rod cursors, springs, and specifications. The repeatability pass rate for PEFMs was 75.00% (9/12),

Table 2
Comparison of the technical performance of mechanical and electronic PEFMs

	A 100 (L/min)		A150 (L/min)		A200 (L/min)		A300 (L/min)		A450 (L/min)		A600 (L/min)		A720 (L/min)		BTPS300 (L/min)		BTPS600 (L/min)
Mechanical
Average	102.567	$\pm$ 14.734	151.9	$\pm$ 16.338	190.229	$\pm$ 16.137	304.171	$\pm$ 26.508	457.314	$\pm$ 39.752	605.029	$\pm$ 48.287	732.867	$\pm$ 55.873	297.6	$\pm$ 25.181	598.343	$\pm$ 68.49
Accuracy
Absolute error	2.567	$\pm$ 14.734	1.9	$\pm$ 16.338	$-$ 9.771	$\pm$ 16.137	4.171	$\pm$ 26.508	7.314	$\pm$ 39.752	5.029	$\pm$ 48.287	12.867	$\pm$ 55.873	–		–
Relative error	0.451	$\pm$ 14.889	0.099	$\pm$ 11.106	$-$ 5.871	$\pm$ 8.93	0.623	$\pm$ 8.862	0.867	$\pm$ 8.706	0.193	$\pm$ 8.249	1.21	$\pm$ 7.44	–		–
Repeatability	2	$\pm$ 2.449	4.333	$\pm$ 2.944	2.644	$\pm$ 2.926	1.312	$\pm$ 1.443	0.573	$\pm$ 0.636	1.085	$\pm$ 0.795	1.387	$\pm$ 1.513	3.814	$\pm$ 2.533	4.317	$\pm$ 6.864
Linearity	0.122	$\pm$ 2.798	2.705	$\pm$ 3.626	$-$ 1.57	$\pm$ 3.679	0.282	$\pm$ 3.562	0.76	$\pm$ 2.778	$-$ 1.955	$\pm$ 3.758	–		–		–
Flow resistance	0.234	$\pm$ 0.086	0.261	$\pm$ 0.084	0.268	$\pm$ 0.078	0.285	$\pm$ 0.096	0.302	$\pm$ 0.11	0.314	$\pm$ 0.121	0.344	$\pm$ 0.143	–		–
Electronic
Average	101.589	$\pm$ 10.037	150.225	$\pm$ 10.82	199.827	$\pm$ 12.613	308.37	$\pm$ 13.924	466.058	$\pm$ 19.009	622.304	$\pm$ 26.49	749.84	$\pm$ 25.599	308.069	$\pm$ 11.3	622.563	$\pm$ 24.361
Accuracy
Absolute error	1.589	$\pm$ 10.037	0.225	$\pm$ 10.82	$-$ 0.173	$\pm$ 12.613	8.37	$\pm$ 13.924	16.058	$\pm$ 19.009	22.304	$\pm$ 26.49	29.84	$\pm$ 25.599	–		–
Relative error	0.525	$\pm$ 10.989	$-$ 0.367	$\pm$ 7.488	$-$ 0.482	$\pm$ 6.538	2.514	$\pm$ 4.628	3.288	$\pm$ 4.034	3.416	$\pm$ 4.12	3.871	$\pm$ 3.332	–		–
Repeatability	2.84	$\pm$ 2.555	2.24	$\pm$ 1.894	1.276	$\pm$ 1.264	0.815	$\pm$ 1.02	0.812	$\pm$ 1.301	0.75	$\pm$ 0.623	0.704	$\pm$ 0.41	1.665	$\pm$ 2.313	1.245	$\pm$ 1.254
Linearity	$-$ 0.594	$\pm$ 3.785	$-$ 1.078	$\pm$ 2.317	$-$ 1.989	$\pm$ 2.213	$-$ 2.009	$\pm$ 1.974	$-$ 1.981	$\pm$ 2.119	$-$ 0.491	$\pm$ 0.366	–		–		–
Flow resistance	0.098	$\pm$ 0.14	0.105	$\pm$ 0.126	0.112	$\pm$ 0.119	0.134	$\pm$ 0.117	0.167	$\pm$ 0.133	0.203	$\pm$ 0.156	0.173	$\pm$ 0.138	–		–
T ${}^{\ast}$	$-$ 0.258		$-$ 0.292		$-$ 2.662		$-$ 3.421		$-$ 5.024		$-$ 4.410		$-$ 3.461		–		–
$P^{\ast}$	0.800		0.773		0.014		0.002		0.000		0.000		0.004		–		–

Remarks: * refers to the comparison between the average of measurements.

and the electronic repeatability (80%) was similar to the mechanical repeatability (71.43%). Instrument reproducibility is more important in the context of long-term self-monitoring of patients than accuracy. Although most of the PEFMs included in this study could be used for long-term patient self-monitoring, the stability of some PEFMs was still not sufficient to meet with these requirements, with most of the errors being produced in the BTPS state, and a maximum error as high as 19% (Fig. 4), indicating that for human expiration in the same physiological state, the measurements of these PEFMs may not be sufficiently stable, potentially leading to biased physicians’ judgments regarding patient condition and thus impacting patient treatment.

Figure 3.

Accuracy of the PEFMs.

Figure 4.

Repeatability of the PEFMs.

This study found that, overall, electronic PEFMs exhibited superior performance to mechanical PEFMs, in addition to being simple to operate, easy to use, flexible, easy to carry, and capable of accurately storing numerical changes. Some devices are even capable of connecting with big data management platforms to support respiratory disease prevention and treatment such that doctors can understand the development of a given patient’s disease in real-time, although these instruments are relatively expensive. Mechanical PEFMs are cheap and easy to operate, but exhibit greater resistance than electronic PEFMs do to the influence of Vernier and springs.

In this study, only the MINI-Wright and eMINI-Wright instruments were able to satisfactorily meet patient requirements. Both of these brands are primarily prevalent in regions such as Europe and are less readily available on the Chinese market, leaving Chinese consumers to access to products that may underperform and fail to meet their needs. Therefore, significant attention should be paid by the relevant authorities to improving the marketing standards for PEFMs and ensuring that patients can access products that perform well.

In contrast to the present study, Vanzeller et al. [13] reported that the eMINI-Wright instrument did not pass the response frequency test whereas the Philips instrument met the accuracy standard. While the present study also used the ISO 23747 standard, the simulators used in this study were different from those in the prior report. It remains unclear as to whether different simulators may affect these test results. Alternatively, the performance of instruments produced by the same brand may be unstable. The present study was limited by the fact that we only tested one instrument for each brand and were thus unable to compare differences among multiple instruments from a given manufacturer. In future studies, we will compare multiple batches of instruments of the same brand, and PEFMs will be longitudinally tested over extended periods to determine whether any changes in variability or performance arise as a consequence of long-term use.

5. Conclusion

The present study highlighted certain errors in the technical performance of individual PEFMs currently available on the market, underscoring the need for further improvements in the underlying devices. These findings also underscore the need for strongest PEFM quality control and management in China. In particular, pre-factory instrument quality testing and the periodic re-inspection of end-user PEFMs have the potential to better ensure consistent and accurate PEF measurements over time.

Funding

This work was supported by the National Key Technology R&D Program (2018YFC1311901, 2016YFC1304603), the National Science & Technology Pillar Program (2015BAI12B10), and the Key Technology R&D Program of Zhejiang, China (2020C03062).

Author contributions

All authors contributed to study conception and design. Material preparation, data collection, and analyses were performed by ZW and RH. The first draft of the manuscript was written by ZW and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Ethics statement

Not applicable.

Footnotes

Acknowledgments

The authors would like to thank all reviewers who participated in the review and MJEditor (www.mjeditor.com) for their linguistic assistance during the preparation of this manuscript.

Conflict of interest

The authors declare that they have no conflict of interest.

References

Asthma group of Chinese Throacic Society. Guidelines for bronchial asthma prevent and management (2020 edition). Chinese Journal of Tuberculosis and Respiratory Diseases, 2020; 43(12): 1023-1048. doi: 10.3760/cma.j.cn112147-20200618-00721.

Global Initiative for Asthma. Global Strategy for Asthma Management and Prevention, 2021. Available from: https://www.ginasthma.org/.

Tian

Zhou

Cui

Wang

, et al. Peak expiratory flow as a screening tool to detect airflow obstruction in a primary health care setting. The International Journal of Tuberculosis and Lung Disease: The Official Journal of the International Union against Tuberculosis and Lung Disease. 2012; 16(5): 674-680. doi: 10.5588/ijtld.11.0429.

Zou

. Clinical Utilization Significance of Simple Expiration Peak Current-meter in Basic Hospital. Guide of China Medicine. 2011; 9(14): 12-13. doi: 10.3969/j.issn.1671-8194.2011.14.005.

Pulmonary Function Professional Group, Chinese Medical Association Respiratory Medicine Branch. Guidelines for Pulmonary Function Examination – Examination of Peak Expiratory Flow and Its Variability. Chinese Journal of Tuberculosis and Respiratory Diseases. 2017; 40(6): 426-430. doi: 10.3760/cma.j.issn.1001-0939.2017.06.006.

Hua

Lou

Zhou

. Dvelopment of the portable exhale volume meter. Information of Medical Equipment. 2003; 18(3): 10-11, 22. doi: 10.3969/j.issn.1674-1633.2003.03.004.

Kim

Lee

Cha

. Peak expiratory flow meter capable of spirometric test for asthma monitoring[C]. //Biomedical Engineering, 2003. IEEE EMBS Asian-Pacific Conference on. 2003: 254-255. doi: 10.1109/APBME.2003.1302680.

American Thoracic Society, Standardization of 1994 update ATS statement, Am J RespirCrit CareMed. 1995; 152: 1107-1136. doi: 10.1164/ajrccm.152.3.7663792.

Lefebvre

Vandergoten

Derom

Marchandise

Liistro

. Testing spirometers: are the standard curves of the american thoracic society sufficient? Respir Care. 2014; 59(12): 1895-1904. doi: 10.4187/respcare.02918.

10.

Anaesthetic and respiratory equipment – Peak expiratory flow meters for the assessment of pulmonary function in spontaneously breathing humans (ISO 23747: 2015): EN ISO 23747-2015. https://www.iso.org/standard/64926.html.

11.

Takara

Ruas

Pessoa

Jamami

Di Lorenzo