Development and Practical Application of a Multiple-Criteria Decision Analysis Framework on Respiratory Inhalers: Is It Always Useful in the MOH Malaysia Medicines Formulary Listing Context?

Abstract

Objectives. The current health technology assessment used to evaluate respiratory inhalers is associated with limitations that have necessitated the development of an explicit formulary decision-making framework to ensure balance between the accessibility, value, and affordability of medicines. This study aimed to develop a multiple-criteria decision analysis (MCDA) framework, apply the framework to potential and currently listed respiratory inhalers in the Ministry of Health Medicines Formulary (MOHMF), and analyze the impacts of applying the outputs, from the perspective of listing and delisting medicines in the formulary. Methods. The overall methodology of the framework development adhered to the recommendations of the ISPOR MCDA Emerging Good Practices Task Force. The MCDA framework was developed using Microsoft Excel 2010 and involved all relevant stakeholders. The framework was then applied to 27 medicines, based on data gathered from the highest levels of available published evidence, pharmaceutical companies, and professional opinions. The performance scores were analyzed using the additive model. The end values were then deliberated by an expert committee. Results. A total of eight main criteria and seven subcriteria were determined by the stakeholders. The economic criterion was weighted at 30%. Among the noneconomic criteria, “patient suitability” was weighted the highest. Based on the MCDA outputs, the expert committee recommended one potential medicine (out of three; 33%) be added to the MOHMF and one existing medicine (out of 24; 4%) be removed/delisted from the MOHMF. The other existing medicines remained unchanged. Conclusions. Although this framework was useful for deciding to add new medicines to the formulary, it appears to be less functional and impactful for the removal/delisting existing medicines from the MOHMF. The generalizability of this conclusion to other formulations remains to be confirmed.

Keywords

formulary MCDA respiratory inhaler

Introduction

Respiratory inhalers are devices designed to deliver medicines into the respiratory tract and are commonly used during the treatment and management of common respiratory diseases, such as asthma and chronic obstructive pulmonary disease (COPD), which each have a prevalence of more than 200 million cases worldwide.¹ Since the development of the metered-dose inhaler in 1956,² both the inhaler-delivered medical treatment and the inhaler device technology have undergone rapid development. In the past decade alone, at least 11 different respiratory inhaler products have been developed to improve patients’ clinical outcomes, in terms of efficacy, and medication adherence, in terms of convenience and device suitability.³

In Malaysia, health care services can be obtained from both private and public providers. Public health care services are funded through general taxation and are centrally administered by the Ministry of Health (MOH). Other ministries that also provide health care services are education and defense. The MOH is the major health care provider, and all Malaysian citizens have access to MOH-provided health care. The MOH Medicines Formulary (MOHMF) represents a major MOH initiative designed to contain the increasing costs of health care. The MOHMF panel decides which registered medicines, including respiratory inhalers, will be funded by the MOH, based on formal proposals submitted by pharmaceutical companies. As explained previously,⁴ the decision makers deliberated on the 1) informed outputs of the health technology assessment (HTA) and 2) recommendations made by experts (clinicians and pharmacists) working in the associated disease field (Therapeutic Drug Working Committee [TDWC]).

The current HTA is based on a generic set of criteria across all medicines that focuses on effectiveness, safety, economic considerations, pharmaceutical suitability, and experience of use without including a quantification of the costs and benefits as aggregated scores. Although this set of criteria may be applicable to medicines that rely on straightforward administration methods, such as oral tablets, whether the same HTA standards can be applied to respiratory inhaler-based treatments remains unclear because these treatments represent a composite between both a medicine and a device. The consideration of each patient’s ability to correctly administer a respiratory inhaler is a basic component of asthma management⁵; however, the measurement of this variable in the context of overall patient convenience and ease of use is uncertain. Thus, criteria associated with patient convenience and ease of use are often either inconsistently or indirectly considered if they are considered at all. The extent to which a respiratory inhaler should be assessed has also not been clearly established. In the absence of an explicit set of criteria, varying degrees of comprehensiveness may apply to the HTAs performed for respiratory inhalers by health technology evaluators in the Formulary Management Branch of the Pharmacy Practice & Development Division.

Although each combination between medicine and device has advantages and disadvantages, no known respiratory inhalers perform well in both the medicine and device components. In the absence of an explicit set of criteria, the transparency regarding the recommendations made by the TDWC can become a concern. Which medicinal or device-based attributes the TDWC prioritizes and how they choose to favor one formulary listing or delisting for a respiratory inhaler over another is not clear. Therefore, whether their preferences are consistent when making recommendations remains unknown.

These apparent limitations can potentially lead to a poorly informed decision-making process with regard to respiratory inhalers. Thus, an explicit framework for the evaluation of respiratory inhalers at the HTA and TDWC levels is urgently necessary to ensure a fair balance between patient access, the value of the medicine, and budget availability.

Therefore, this study aimed to develop a generic formulary decision-making framework that can be applied to all respiratory inhalers that have been indicated for maintenance therapy, regardless of their pharmacological class. This decision-making framework was designed using the multiple-criteria decision analysis (MCDA) method. An MCDA is a structured process that explicitly integrates value judgments, priorities, and preferences across relevant stakeholders. MCDA aims to support and facilitate reliable and credible decision-making processes through the consideration of multiple criteria and the relative importance attached to each criterion.^6,7 Subsequently, the present study also aimed to assess the impacts of the framework on TDWC recommendations by applying the framework to all current formulary-listed and potential (as proposed by the pharmaceutical companies) respiratory inhalers that have been indicated for the maintenance therapy of asthma or COPD.

Methodology

Development of the MCDA Framework

The overall methodology used during framework development strived to adhere to the recommendations made by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) MCDA Emerging Good Practices Task Force.⁸ All registered MOH specialists and consultants involved in respiratory practice, general practice, and family medicine, and pharmacists were invited to participate during this stage of the study as the relevant stakeholders, some of whom also participated on two expert committees (referred to as the TDWC).

Steps 1 and 2 were conducted in separate sessions, whereas Steps 3 and 4 were conducted in a single session. Stakeholders were briefed on the decision problem and the methodology before each step began. The MCDA framework was developed using Microsoft Excel 2010 (Microsoft Corporation). This study did not involve any patients; thus, it was exempted from the requirement to obtain ethics approval from the Medical Research & Ethics Committee (NMRR-18-1143-41856). The overall methodology is summarized in Figure 1, and the full elaboration of each step is available in Appendix 1.

Figure 1

A pictorial summary of the overall methodology used for MCDA framework development. Steps 1 and 2 were conducted in separate sessions. Steps 3 and 4 were conducted in a single session. Both the TDWC and all other relevant stakeholders were involved through live voting. The dashed-arrow represents the flow of other stakeholders’ feedback to the TDWC.

Steps 1 and 2: Identification and Definition of Criteria

The criteria that formed the bases of this framework were pre-identified based on the current HTA framework, previous frameworks designed for similar activities, and a rapid review of published literature related to MCDA and/or respiratory inhalers. A meeting with the TDWC was convened to present these identified criteria and to determine whether any additional criteria and/or subcriteria were necessary. The term “criteria” is henceforth defined as any criteria, either with or without associated subcriteria. Prior to finalizing the list of criteria, the criteria were screened to ensure adherence with ISPOR good practice recommendations for MCDA,⁸ which included nonredundancy and preference independence. Identified criteria were divided into two groups: economic (in monetary units, by nature) and noneconomic (in non-monetary units, by nature). The final set of criteria, as identified by the TDWCs, was validated by the remaining stakeholders via Google Forms.

During Step 2, a subsequent meeting with the TDWC was convened to define the identified criteria in Step 1 according to a range of consequences, referred to as outcomes, performance scales/levels, or criteria categories. For example, the range of consequences for the criterion “effectiveness” could be “less effective, similarly effective, or more effective,” whereas the criterion “requires priming” could be “yes or no,” and the criterion “number of daily doses” could be “1–2 or 3–4.” The decision was made not to define economic criteria according to any performance scales because defining the consequence endpoints was determined to be too challenging. For example, “cheap or expensive” could be used for the criterion “cost of medicine,” but the endpoint “expensive” is subjective and contextual. The range of consequences for each criterion was then validated by all other stakeholders via Google Forms. The final set of criteria and associated performance scales were collected for use in Steps 3 and 4.

Steps 3 and 4: Elicitation of Criteria Weights and Scoring Functions for Performance Scales

All stakeholders, including the TDWC, were invited to participate in a 1-day workshop, during which real-time voting was performed by using the Mentimeter software (Mentimeter AB, Sweden), with the use of visual aids presented using Microsoft PowerPoint 2010 (Microsoft Corporation). The workshop was moderated by one of the researchers. These steps were performed according to the methodology described in a previously published study examining off-patent pharmaceuticals,⁹ with minor adjustments made to Step 4, which involved the elicitation of scoring functions. Each step was conducted separately for economic and noneconomic criteria. The weights for economic criteria were elicited using a direct rating method, whereas the weights for noneconomic criteria were elicited using a combination of SMART (simple multi-attribute rating technique) and swing-weighting methods.¹⁰

When determining the scoring function used for each economic criterion, the stakeholders determined that no price threshold or cutoff point should be used because they deemed that no medicines should be “punished” for being expensive relative to the other members of the same pharmacologic class. Hence, an inverse linear partial value function was proposed, using the following formula:

1 - \frac{X - (X_{\min} - RM 10)}{(RM 10 + X_{\max}) - (X_{\min} - RM 10)}

where X is the cost of the medicine being evaluated, and X_min and X_max represent the respective minimum and maximum monthly cost for any medicine within the same pharmacological class as the medicine being evaluated. All costs were expressed in Malaysian Ringgit (RM) for the year 2018, at which point the conversion rate was RM 4.04 = $US1.¹¹ To accommodate the application of this framework to the evaluation of future medicines that might be cheaper than the current X_min or more expensive than the current X_max, an arbitrary estimated buffer of RM 10 was built into the high and low ends of the model. Therefore, none of the medicines were scored 0 or 1 based on this criterion in the current study.

The scoring functions for the noneconomic criteria were made flexible at the discretion of the stakeholders. A direct proportional scoring function for the multilevel performance scale of each criterion using a scale of 0 (worst performance) to 100 (best performance) was proposed to the stakeholders. However, they were also allowed to adjust the performance scale if approved by a majority vote; in this case, median voted scores were considered.

Application of the MCDA Framework

The finalized MCDA framework consisted of stakeholder-preferred criteria, definitions, weights between criteria, and scoring functions within criteria. This framework was then applied to existing medicines listed in the formulary, including all types of inhaler devices that are known to be procured by MOH facilities (N = 24; for the purpose of this study, these medicines were named according to their chemical entity, dosage strength, formulation, and device type, as sourced from MOH pharmacy database) and potential medicines not currently listed in the formulary (N = 3). These medicines are indicated for the maintenance treatment of asthma and/or COPD.

Step 1: Performance Measurement

The data used for this step were obtained from multiple parties. Pharmaceutical companies were asked to provide performance-related evidence for their respective products according to a defined set of criteria. They were also given an opportunity to revise their current product price if it was not procured via tender. Simultaneously, a literature review was conducted (through the year 2018) by the researchers to identify data related to the performance of each criterion for all medicines. The highest levels of available evidence, such as systematic review and meta-analysis, were retrieved.^12–36 Next, the collated evidence was independently reviewed and validated by two pharmacists to ensure that performance-related evidence was appropriate and applicable to the MCDA framework.

Two groups of health care professionals were invited to participate in a half-day workshop: 1) stakeholders who were involved in Steps 3 and 4 of the framework development process and 2) pharmacists who were not involved in the framework development process. These mixed participants were randomly allocated into three groups, containing at least one specialist or consultant in each group. Each group was assigned an equal number of medicines to evaluate. They were required to discuss and choose the performance level that best represented each medicine via Google Forms. All validated evidence was provided to participants for support and to ensure the highest possible consistency of evidence-based performance measurements. The weights and scorings of the scales were not disclosed to the participants to avoid biases in performance measurements, particularly those for which their expert opinions could supersede the published evidence provided. At the end of the session, the results of the performance measurement were displayed and opened to discussions among the participants. The results were considered final when a final consensus was reached regarding the performance for all criteria and for all medicines.

Step 2: Analysis of Performance

The results of the performance measurement assessment were applied to the MCDA framework. The scores for the “Cost of medicine” criterion were calculated based on the latest price, as declared by the pharmaceutical companies or based on the government tender, if applicable. Assuming that the criteria included in the MCDA framework were indeed preferentially independent, the additive model was used to aggregate the scores and weights of all criteria into an overall value estimate for each medicine.⁸

Step 3: Deliberation of the MCDA Outputs

A half-day meeting was convened to allow for the TDWC to make recommendations regarding which existing medicines should be delisted from the formulary (if any) and which potential medicine(s) should be considered for listing (if any). In addition to the values of the medicines, budgetary impact (financial consequences of listing each potential medicine) and the number of patients being treated (for existing medicines in the formulary) were presented. Both of these factors are routinely considered in the current HTA context as part of the economic consideration and as an indicator of experience of use but were not included in the final MCDA framework. These factors are considered separately because the budgetary impact is potentially dependent on the “Cost of medicine” criterion and defining the consequential endpoints for the number of patients being treated can be difficult due to presence of confounding factors. The numbers of patients currently prescribed each medicine were obtained from the MOH pharmacy database, whereas the results of the budget impact analysis were prepared by health technology evaluators. The TDWC was expected to use these data in a similar manner as in the current HTA context.

Results

Development of the MCDA Framework

A total of 126 stakeholders were invited to participate in the development of the MCDA framework (Table 1). Although this process was performed in multiple stages using various platforms, a number of stakeholders were unable to participate throughout the entire process. However, contributors from each field participated in almost every stage, as described in Table 1.

Table 1

The Number of Stakeholders Who Participated in Each Stage of MCDA Framework Development^a.

No.	Stage	Number of Stakeholders Participated, n (%)				Total Participants, n (%)
		Pulmonologist, 37 (29%)	General physician, 49 (39%)	Family physician, 15 (12%)	Pharmacist, 25 (20%)
1	Identification and definition of criteria	3 (8%)	2 (4%)	0 (0%)	1 (4%)	6 (5%)
2	Agreement on identified criteria	13 (35%)	13 (27%)	4 (27%)	11 (44%)	41 (30%)
3	Agreement on definition of criteria	11 (30%)	11 (22%)	2 (13%)	12 (48%)	36 (29%)
4	Determination of Weights of criteria Scores of performance scales	10 (27%)	7 (14%)	2 (13%)	7 (28%)	26 (21%)

MCDA, multiple-criteria decision analysis.

The number of stakeholders invited is listed below their designation.

A total of eight (one economic and seven noneconomic) main criteria and seven noneconomic subcriteria were identified by the stakeholders. The final list of identified criteria, including the assigned weights, scoring functions, and definitions used for the performance scales, are detailed in Table 2. The sum of all noneconomic criteria weights represented 70% of the total analysis, and the final framework featured a single economic criterion, the cost of medicine, which was weighted as 30% of the total analysis.

Table 2

The Identified Criteria, Showing the Elicited Weights, Definitions, and Scoring Functions. The Criteria Are Listed According to Their Descending Importance and Respective Weights. The Sum of Weights for All Criteria May Not Be 100% Due to Rounding.

No.	Criteria	Weight^a	Definition	Scoring Function
1	Cost of medicine	30%	Average monthly cost of medicine	Inverse linear partial value function
2	Patient suitability^b	22.7%	Can (or not) be used in patients with poor memory skills	60%/40%
			Can (or not) be used in patients with poor manual dexterity (arthritic, neurologic, musculoskeletal)—minimal need	60%/40%
			Can (or not) be used in patients with inability to form a good seal around the mouthpiece	70%/30%
3	Patient benefit(s) via pharmaceutical technology^c	15.2%
	3a. Device performance ^d	4.6%	Percentage of fine particle fraction (FPF) at 4 kPa; ≤40% or >40%	40%/60%
			Inhaler intrinsic resistance^e; Medium/high or Medium/low or Low	30%/40%/50%
			Inhalation flow rate-dependent FPF; Consistent or not	60%/40%
	3b. Dose frequency	3.1%	Higher than the lowest frequency of alternative from the same class	30%
			Lowest frequency compared to other alternatives within the same class	70%
	3c. Maintenance and Reliever therapy ^f	2.1%	Yes/no	70%/30%
	3d. Number of setup steps prior inhalation	1.6%	1–2/3–4/5–6/7–8/9–10	100%/80%/60%/40%/20%
	3e. Probability of making at least one critical error	1.4%	Probability expressed in percentage; 0–20/21–40/41–60/61–80/81–100	100%/80%/60%/40%/20%
	3f. Number of priming steps prior first use	1.2%	0/1–5/6–10/11–15	100%/67%/33%/0%
	3g. Dose counter	1.1%	Yes/no	50%/50%
4	Comparative effectiveness^g	10.1%	Inferior compared to alternatives within the same class	0%
			Not significantly different compared to other alternatives within the same class	25%
			Significantly more effective compared to 1/4 ≤x≤ 1/3 of alternatives within the same class	50%
			Significantly more effective compared to 1/2 ≤x≤ 2/3 of alternatives within the same class	75%
			Significantly more effective compared to all alternatives within the same class	100%
5	Safety^h	7.2%	Substantial safety concerns or significant adverse events reported compared to other alternatives	0%
			Similar safety profile compared to other alternatives	33%
			Fewer safety concerns or significant adverse events compared to other alternatives	67%
			Substantial reduction in safety concerns or adverse events compared to other alternatives	100%
6	Clinical/Practice needs^i,j	5.6%	More than one alternative treatment and same device exists for the same indication	0%
			More than one alternative treatment but different device exists for the same indication	25%
			One alternative treatment and same device exists for the same indication	50%
			One alternative treatment but different device exists for the same indication	75%
			New treatment; no other option within that class	100%
7	Availability of economic evidence^k	4.6%	No economic evidence available	0%
	Availability of economic evidence^k		Cost-utility analysis (CUA)/cost-effectiveness analysis (CEA)—international evidence	33%
			CUA/CEA adjusted to local cost data	67%
			CUA/CEA—local evidence	100%
8	Reimbursement/listing in other countries^l	4.6%	Not reimbursed/listed elsewhere	0%
	Reimbursement/listing in other countries^l		Reimbursed/listed in 1 country	33%
			Reimbursed/listed in 2–4 countries	67%
			Reimbursed/listed in >4 countries	100%

Weights elicited using the SMART and swing method.

The extent to which an inhaler device is more suitable than other alternatives for a given patient. The suitability of a patient for certain inhaler devices depends on his/her characteristic(s) and associated with the method of administration.

The extent to which the medicine has more advantage(s) in terms of pharmaceutical technology than other medicines that would benefit patients.

The scores for all three definitions were supposed to be summed prior to adjusting this subcriterion score by weight.

This performance is not measured for metered-dose inhalers and Respimat devices.

This criterion is not considered for medicines from a pharmacological class other than the fixed combination of inhaled corticosteroid and long-acting beta agonist.

Clinical effectiveness compared to other medicines within the same class, in terms of (one or more) outcome measures that are commonly used in randomized controlled trials, meta-analyses, or network meta-analyses.

Recent postmarketing/authorization safety reports: periodic safety update report (PSUR), adverse drug reaction (ADR) reports, and/or the extent to which the medicine is safer or has lower rates of adverse events than other medicines.

For a new medicine in the process of being listed in the MOH (Ministry of Health) Medicines Formulary: Whether or not there is any available medicine(s) in the formulary for the proposed indication.

This criterion is not considered for existing medicines listed in the MOH (Ministry of Health) Medicines Formulary.

Published economic evaluation that compares both cost and effectiveness between two or more medicines, from either local or nonlocal settings.

This criterion refers to whether or not a medicine is reimbursed or listed in other countries’ medicines formularies. These countries are known for their well-established systematic assessment methods for medicine reimbursement/listing, for example, Singapore, Thailand, South Korea, Taiwan, China, the United Kingdom, and Australia. Evidence of reimbursement/listing indirectly informs the support of other countries for the medicine in terms of effectiveness, safety, cost, and need.

During the final stage of the MCDA development, some criteria were deemed inapplicable to some medicines, based on a consensus opinion among stakeholders. First, the subcriterion “Maintenance and Reliever therapy” should only be applied to fixed-dose combinations of inhaled corticosteroids (ICS) and long-acting beta-agonists (ICS/LABA), as this is not an inherent property of medicines belonging to other pharmacologic classes. Therefore, for medicines other than ICS/LABA, the weight of “Maintenance and Reliever therapy” was set to zero, and consequently, the weights of the remaining subcriteria were redistributed (increased) to compensate for this loss of input; at this point, the total weight of main criterion, “Patient benefit(s) via pharmaceutical technology,” was maintained at 15.2% (Appendix 2).

The criterion “Clinical/Practice needs” was initially defined for existing medicines as the number of patients being treated and for potential medicines as the availability of other medicine(s) within the same pharmacological class as the medicine being evaluated. The stakeholders decided that the criterion “Clinical/Practice needs” should only be applied to proposed medicines and not to existing medicines due to possible confounding factors that could result in misconceptions. Therefore, for existing medicines, the weight of “Clinical/Practice needs” was set to zero, and consequently, the weights of all other noneconomic criteria were redistributed (increased) to compensate for this loss of input; at this point, the total sum of noneconomic criteria weights was maintained at 70% (Appendix 2).

The subcriterion “Device performance” was given three definitions, each associated with a different scoring function; the scores for all three definitions were supposed to be summed prior to adjusting this subcriterion score by weight. The stakeholders decided that the definition “Inhaler intrinsic resistance” should only be applied to dry powder inhalers, as this is not an inherent property of propellant-driven MDIs or soft-mist inhalers (Respimat). Therefore, the score for this subcriterion consisted only of the sum for the two remaining definitions when calculated for MDIs and Respimat (Appendix 2.

Application of the Developed MCDA Framework

A total of 12 participants were involved in measuring the performance of 27 medicines. The detailed performance scores for each criterion and for each medicine is available in Appendix 2. The values determined for the ICS (N = 9; four-single chemical entities, three types of devices, various dosage strengths), ICS/LABA (N = 11; five-double chemical entities, four types of devices, various dosage strengths), LABA (N = 2; two-single chemical entities, two types of devices), long-acting muscarinic antagonists (LAMA, N = 2; two-single chemical entities, two types of devices), and fixed-dose combination of LABA and LAMA (LABA/LAMA, N = 3; three-double chemical entities, three type of devices) groups are displayed in Figure 2.

Figure 2

The bar chart represents the total scores of medicines (N = 27) from five pharmacological classes, with the type of device in parenthesis. The dark shade represents the aggregate scores of both economic and noneconomic criteria, whereas the light shade represents the aggregate scores of noneconomic criteria only. The latter score was determined by adjusting the total weight of the noneconomic criteria to 100%. The medicines are arranged in ascending order of the total score for both economic and noneconomic criteria. Potential medicines are marked “*” to indicate the medicines proposed by the pharmaceutical companies for listing in the MOH Medicines Formulary at the time of the study.

All ICSs have been listed in the MOHMF for at least 5 years. Among the evaluated medicines, the TDWC expressed negative recommendations for Budesonide 100 µg and 200 µg Turbuhaler. As shown in Figure 2, Budesonide is available at two possible dosage strengths (100 µg and 200 µg), which can be delivered using and three possible devices (MDI, Turbuhaler, and Easyhaler) in the MOH market. The Budesonide 200 µg Turbuhaler scored the lowest due to a relatively high cost, which may also contribute to the low number of patients that have been prescribed this medicine compared with other forms. However, when the costs were not considered, the Turbuhaler fared better than the Easyhaler device due to a lower probability of critical error during medication administration. The TDWC, however, gauged that proper patient education regarding inhaler administration would result in the probability of critical error using either the Turbuhaler or Easyhaler being comparable; therefore, the TDWC did not recommend that MOH facilities procure the Budesonide 200 µg Turbuhaler any longer due to the associated higher costs. Procuring Budesonide 100 µg Turbuhaler was also not recommended for the sake of convenient dose tapering using the same type of device. The availability of this chemical entity in one type of dry powder device and one form of MDI device was considered sufficient for the TDWC. However, the TDWC did not recommend removing any of the listed chemical entities entirely from the formulary. Together with these recommendations, the TDWC also suggested that respiratory inhalers be renamed according to their device type to improve the cost-effectiveness of prescribing and procuring medicines in MOH facilities.

Of the nine ICS/LABAs listed in the formulary and the two potential ICS/LABAs, which represented different dosage strengths of existing chemical entities, “Budesonide 320 µg and Formoterol 9 µg Turbuhaler” scored the lowest. In addition to its high cost, it was not associated with any added value, based on the “Maintenance and Reliever therapy” criterion, unlike its lower dosage strength version. Furthermore, the number of patients prescribed this medicine represented less than one-tenth of the number of patients prescribed with the lower dosage-strength version; therefore, the TDWC suggested delisting this medicine from the formulary. Similar to the ICS, the TDWC did not recommend the complete removal of any listed chemical entity from the formulary. Although both strengths of “Fluticasone Propionate and Formoterol Fumarate Dihydrate” appeared to be prescribed to fewer patients and are indicated for asthma treatment only, the TDWC did not suggest removing these medicines from the formulary because they were only recently made available in MOH facilities at the time of the study. The TDWC also did not endorse the inclusion of two potential medicines in the formulary because they were relatively expensive, were associated with a wide range of uncertainty regarding budgetary impacts over the next 5 years, and were not associated with significant added value compared with other existing medicines.

Another potential medicine, a LABA/LAMA, was recommended for listing in the formulary by the TDWC. Although two existing LABA/LAMAs are already listed in the formulary, this potential medicine scored the highest. Unlike the existing available options, which are dry powder inhalers, this proposed medicine is available in soft-mist inhaler, which displayed added value and was comparatively cheaper than other medicines, with significant estimated cost savings in the next 5 years. However, the TDWC was not ready to remove either of the two existing dry powder LABA/LAMAs from the formulary; one was listed in the formulary only shortly before the study was performed, whereas the other has a long track record of being prescribed, despite scoring the lowest and being the most expensive. Therefore, the TDWC suggested reviewing this trade-off after another year.

No other recommendations were made by the TDWC due to the MCDA framework application for other medicines (N = 4).

Discussion

This study aimed to develop an MCDA framework for respiratory inhalers and applied the developed framework to all listed and potential medicines that have been indicated for the treatment of asthma or COPD. After deliberating the MCDA outputs and considering additional information that was not assessed within the framework, the TDWC made several recommendations regarding the listing, delisting, procurement, and renaming of medicines in the formulary.

MCDA Framework

A grand total of 15 criteria were identified and defined by the stakeholders for inclusion in the framework. This high number of criteria was considered to be sufficiently comprehensive to address the problem statement of the study. However, this high number of criteria represented a practical challenge for those stakeholders who were not MCDA experts during Steps 3 and 4 of the framework development stage. Therefore, a considerable amount of effort was made to ensure that all stakeholders were well-informed and briefed on MCDA before each step of the framework development process, regardless of which step(s) they participated in. To improve the understanding of the MCDA process among stakeholders, a protocol was designed as simply as possible for those areas that require considerable cognitive efforts, such as the elicitation of criteria weights and scorings functions.

The overall methods used in this study were similar to those described in a study by Inotai et al.⁹ Unlike other studies that have used the partial value function,^37,38 the current study has adopted a simpler scoring method, based on ordinal scales, to define the noneconomic criteria. Although this method is less precise because not all scales display interval properties, this limitation was deemed to be an acceptable trade-off for improved feasibility during framework development.

In the context of criteria weight elicitation, this study adopted a combination of SMART and swing weighting methods.¹⁰ This method is more feasible and less complex because it does not require the use of software and has lower resource requirements compared with other methods, such as MACBETH (measuring attractiveness by a categorical-based evaluation technique) and DCE (discrete choice experiments).⁸ Because this study was not separately funded, MACBETH and DCE were not ideal as they require funding for appropriate external expertise resources. However, one limitation to this approach was the lack of a cap on the maximum allowed difference between two criteria, which therefore resulted in the weight of the least important criterion appearing to be very small compared to the weight of the most important criterion due to the chain effect.³⁹ Biases could also be introduced to elicited weights, as the weights of all the other criteria could easily be affected even if one were quantified incorrectly in terms of its relative importance.³⁹ These errors could affect the validity of the overall value estimates and the outcomes of this study. Although no known studies have specifically examined the potential effects of using this weighting method, Van Til et al. found that the calculated weights and overall value estimates were not significantly affected by the weighting method used on an aggregated level.⁴⁰ The tested methods have a similar risk of bias to the combination of SMART and swing methods,³⁹ but uncertainty remains regarding the ability to generalize the finding due to the different natures of these weighting methods. Further investigations are warranted; nevertheless, the selected weighting method used in this study represents a carefully considered trade-off between complexity, feasibility, and bias potentiality.

To further enhance stakeholders’ understanding regarding the concepts of weighting and scoring and to preserve the validity of their professional judgments, a workshop approach was preferred to online surveys. The workshops were conducted as face-to-face events among stakeholders, where briefings and explanations were delivered, and concerns were addressed for all stakeholders simultaneously. This approach was not only more feasible and practical but reduced time and resource consumption.⁴¹ However, the use of online surveys during the identification and definition of criteria was justified because more time was necessary to consolidate the survey outcomes. The protocol was also designed to reduce the potentiality of dominance biases among stakeholders through the use of online surveys during the validation of the identified and defined criteria and the use of anonymous voting during the weighting and scoring elicitation workshop.

Based on the criteria weight elicitation exercise, stakeholders clearly valued how the characteristics of a device affected the user/patient more strongly than whether the chemical entity was effective or safe for the user/patient. This finding was similar to the results of a study performed by Marsh et al.,³⁷ in which the criteria associated with the patients’ convenience were collectively weighted more than the effectiveness and safety of respiratory inhalers. Although this further strengthens the value of the information altogether, patients’ preferences could differ from those of clinicians or pharmacists. Understanding patient preferences will, therefore, contribute to better health care decision-making processes and to the innovative research and development of devices and drug delivery mechanisms by pharmaceutical industries. Currently, no known studies have been published examining the differences between the preferences of patients and health care professionals regarding drug delivery mechanisms. Therefore, future studies are recommended to test the framework from a patient perspective to determine whether differences in the respiratory inhaler preferences exist across sociodemographic variables.

The framework development process is not without limitations. First, we were unable to assess the accuracy of the understanding and perception toward MCDA among the stakeholders, which is recommended for future studies. Second, the inconsistent number of stakeholders who participated in each stage of the framework development process may be associated with the loss of valuable thoughts or opinions. Third, despite the advantages of using online surveys, the opportunities to justify or elaborate on opinions regarding the choice of criteria and/or performance scales made by the TDWC to other stakeholders might have been compromised. The use of live sessions to collect these opinions and arrive at consensus may have facilitated better communications.

Impacts of the MCDA Framework on TDWC Recommendations

The TDWC recommended the listing of one of the three potential medicines, which was an unexpected outcome. Prior to this study, the same members of the TDWC had given positive recommendations for listing all three potential medicines, suggesting that the structured and explicit MCDA framework of criteria, weights, and scorings affected these recommendations. However, a similar effect was not observed for existing medicines. Although the TDWC previously declined to recommend the removal of any existing medicines from the formulary, only one medicine was proposed for removal (while retaining the other dosage strength of the same chemical entity) while two other medicines (specifically, two different device types that delivered same chemical entity) were no longer recommended for procurement by MOH facilities in this study. None of these recommendations were directed to the removal of any chemical entity at all available dosage strengths.

This observed aversion to recommending the removal of existing medicine(s) from the formulary implies that even with the support of an explicit framework, the resultant values do not necessarily precede the perceptions and attitudes of health care professionals, such as clinicians. As practicing health care professionals, the stakeholders maintain strongly held principles regarding “personalized medicine,”“individualized treatment,” and “mutually agreed to tailoring.”⁴² Despite the comprehensive nature of the developed MCDA framework, extra-fine properties, such as asthma or COPD phenotypes and the types of excipients, are equally essential but are not included within the framework. Furthermore, although the successful delivery of medicine into the lungs is strongly dependent on the patient’s ability to maneuver the inhaler device, not every chemical entity is available in both dry and nondry powder inhaler forms. When considering the number of patients currently prescribed existing medicines (which was an external factor presented alongside MCDA outputs), even the smallest number group of 45 patients presents a convincing case for the practical need for the medicine in question. Therefore, when considering these factors in parallel, removing any existing medicine from the formulary will limit the clinical freedom of prescribers⁴³ and hamper the ability of patients to access the best-tailored medicines, despite the MCDA outputs.

However, this paradigm shifted when the TDWC was faced with the option to support potential medicines. Although the principle of individualized treatment is still held, the medicine must demonstrate significant added value (based on the MCDA outputs) and be affordable (according to the budget impact analysis). Although no explicit budget impact threshold exists to define the affordability of listing a potential medicine, a consensus among the members of the TDWC was reached based on the budget impact analysis. Given the deliberation outcomes, the application of the MCDA framework might be more valuable and useful as a cost-containment strategy applied to potential medicines rather than for the evaluation of existing medicines.

In principle, MCDA improves the consistency of decision making through the use of an explicit and structured framework. Considering objectively, based on value estimates alone, inconsistencies among the recommendations made by the TDWC are unlikely, even if the members of the TDWC are re-exposed after some time or if a different set of TDWC members are involved. However, the consistency of recommendations would be difficult to guarantee when a mix of judgments are required from the TDWC, such as the subjective judgments regarding external factors and objective judgments in terms of value estimates. The extent of this uncertainty should be explored in future studies to further improve the reliability of the current MCDA framework.

The TDWC also suggested renaming all respiratory inhalers in the formulary according to the type of delivery device. All medicines in the formulary are currently named according to the international nonproprietary name, strength, and formulation. For respiratory inhalers, they are named without any specification of the type of device, for example, “Indacaterol Maleate 150 µg Inhalation Capsule.” However, for medicines that are no longer patented or chemical entities that can be delivered by more than one device, these names can be viewed as ambiguous. For instance, the chemical entity budesonide is currently named “Budesonide 200 µg/dose Inhalation.” Because the respiratory inhaler names are not associated with the delivery devices, any pharmaceutical company can expand the market access of their delivery device, as long as it is associated with the same chemical entity, dosage strength, and formulation that is already listed, without requiring additional review or approval. The differing values associated with various respiratory inhaler devices were acknowledged during the application of the MCDA framework and was raised as a concern by the TDWC in terms of procurement and prescribing. Therefore, the suggestion was made to rename these medicines according to the type of device used to administer them.

During the public procurement process, the name of the medicine listed in the formulary is used as the reference subject of procurement orders. Thus, any type of device available on the market for the delivery of a specific medicine can be procured by following the Guidelines and Directives of the Ministry of Finance,⁴⁴ which include those lost in MOH tendering. For example, budesonide can be obtained in devices X and Z, but only the tenderer for device X is awarded a MOH contract. Under current rules, the procurement of device Z remains possible, subject to special approval as an out-of-contract procurement, which may be requested by health care providers on the basis of valid reasons. From the prescribing view, this structure allows for any type of device available on the market to be prescribed, as long as it shares the name of the medicine listed in the formulary. All in all, these current practices will hamper the efforts to translate the value of each respiratory inhaler into a cost-containment context and represent an inefficient use of resources. Furthermore, the suggestion to rename medicines based on the delivery device does not necessarily limit the clinical freedom of prescribers because having one form each of the dry powder and MDI devices were considered to be sufficient for the TDWC.

Although the renaming of devices would address the TDWC’s concerns, this process would not be easy. The procurement process is bound by Guidelines and Directives of the Ministry of Finance,⁴⁴ which stresses that procurements must be performed fairly and without intentional bias toward any single brand or product. Respiratory inhalers are unique because each inhaler is a composite of medicine and device. Although a particular medicine can be manufactured by more than one source, each respiratory inhaler device (except for MDI) currently available on the market appears to represent an exclusive invention marketed by a single source. The naming of respiratory inhalers according to the delivery device in the formulary is, thus, considered controversial and heavily discouraged because it may represent a violation of the governed procurement policies. Therefore, for a given chemical entity, although the MCDA framework is able to identify that one inhaler device is superior compared with others, the ability to effectively improve the current practice of procuring and prescribing existing medicines in the formulary is limited.

Conclusion

An MCDA framework was developed to assess respiratory inhalers, according to stakeholders’ preferences, during which patient convenience and suitability were given the utmost consideration. Although this framework was undoubtedly useful as a cost-containment strategy for recommendations regarding the listing of new medicines in the formulary, it was less functional and impactful for the removal or delisting of existing medicines because the TDWC prioritized conserving clinical freedom for prescribers and maintaining patient access to existing medicines based on the principles associated with “personalized medicine,”“individualized treatment,” and “mutually agreed to tailoring.” This finding can propel future studies to determine more efficient approaches to the removal or delisting of existing medicines from the MOHMF. However, the generalizability of this conclusion to medicines other than respiratory inhalers is uncertain, and more studies remain necessary.

Supplemental Material

sj-doc-1-mpp-10.1177_2381468321994063 – Supplemental material for Development and Practical Application of a Multiple-Criteria Decision Analysis Framework on Respiratory Inhalers: Is It Always Useful in the MOH Malaysia Medicines Formulary Listing Context?

Supplemental material, sj-doc-1-mpp-10.1177_2381468321994063 for Development and Practical Application of a Multiple-Criteria Decision Analysis Framework on Respiratory Inhalers: Is It Always Useful in the MOH Malaysia Medicines Formulary Listing Context? by Yee Vern Yong, Siti Hajar Mahamad Dom, Nurulmaya Ahmad Sa’ad, Rosliza Lajis, Faridah Aryani Md. Yusof and Jamalul Azizi Abdul Rahaman in MDM Policy & Practice

Supplemental Material

sj-pdf-2-mpp-10.1177_2381468321994063 – Supplemental material for Development and Practical Application of a Multiple-Criteria Decision Analysis Framework on Respiratory Inhalers: Is It Always Useful in the MOH Malaysia Medicines Formulary Listing Context?

Supplemental material, sj-pdf-2-mpp-10.1177_2381468321994063 for Development and Practical Application of a Multiple-Criteria Decision Analysis Framework on Respiratory Inhalers: Is It Always Useful in the MOH Malaysia Medicines Formulary Listing Context? by Yee Vern Yong, Siti Hajar Mahamad Dom, Nurulmaya Ahmad Sa’ad, Rosliza Lajis, Faridah Aryani Md. Yusof and Jamalul Azizi Abdul Rahaman in MDM Policy & Practice

Footnotes

Acknowledgements

We would like to thank the Director General of Health Malaysia for his permission to publish this article. Many thanks to the person who wishes to remain anonymous for his tremendous technical support in the MCDA framework development. Thank you also to our colleague, Mr. Mohd Azuwan Mohd Zubir, for lending his technical expertise to the wonderful pictorial summary. We would also like to extend our gratitude to Ms. Li Yin Chong for her invaluable assistance as the secretary of TDWC Respiratory during the study period. Last but not least, thank you to the TDWC Respiratory, the TDWC General Medicine, Family Medicine Specialists, Pharmaceutical Care Branch of the Pharmacy Practice & Development Division, and the Clinical Pharmacy Working Committee (Respiratory Subspecialty) for their huge support in this study. Thank you to all who have participated in this study.

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: All authors are employees of the Ministry of Health Malaysia.

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Authors’ Note

Work was done at: Pharmacy Practice and Development Division, Ministry of Health.

Work was presented at: HTAsiaLink Conference 2019, Seoul, Korea.

ORCID iD

Yee Vern, Yong

Supplemental Material

Supplementary material for this article is available on the Medical Decision Making Policy & Practice website at .

References

World Health Organization. Chronic respiratory diseases [cited October 2, 2019]. Available from: https://www.who.int/health-topics/chronic-respiratory-diseases

Stein

Thiel

. The history of therapeutic aerosols: a chronological review. J Aerosol Med Pulm Drug Deliv. 2017;30(1):20–41.

Ibrahim

Verma

Garcia-Contreras

. Inhalation drug delivery devices: technology update. Med Devices (Auckl). 2015;8:131–9.

Shafie

Chandriah

Yong

Wan Puteh

. Health technology assessment and its use in drug policy in Malaysia. Value Health Reg Issues. 2019;18:145–50.

Malaysian Health Technology Assessment Section (MaHTAS). Clinical Practice Guidelines: Management of Asthma in Adults. Ministry of Health Malaysia; 2017.

Muhlbacher

Kaczynski

. Making good decisions in healthcare with multi-criteria decision analysis: the use, current research and future development of MCDA. Appl Health Econ Health Policy. 2016;14:29–40.

Thokala

Devlin

Marsh

, et al. Multiple criteria decision analysis for health care decision making—an introduction: report 1 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016;19(1):1–13.

Marsh

Ijzerman

Thokala

, et al. Multiple criteria decision analysis for health care decision making—emerging good practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health. 2016;19(2):125–37.

Inotai

Brixner

Maniadakis

, et al. Development of multi-criteria decision analysis (MCDA) framework for off-patent pharmaceuticals—an application on improving tender decision making in Indonesia. BMC Health Serv Res. 2018;18(1):1003.

10.

Edwards

Barron

. SMARTS and SMARTER: improved simple methods for multiattribute utility measurement. Organ Behav Hum Decision Proc. 1994;60(3):306–25.

11.

World Development Indicators. Official exchange rate (LCU per US$, period average) [cited November 5, 2020]. Available from: https://data.worldbank.org

12.

Molimard

Raherison

Lignot

, et al. Chronic obstructive pulmonary disease exacerbation and inhaler device handling: real-life assessment of 2935 patients. Eur Respir J. 2017;49(2):1601794.

13.

von Schantz

Katajavuori

Antikainen

Juppo

. Evaluation of dry powder inhalers with a focus on ease of use and user preference in inhaler-naive individuals. Int J Pharm. 2016;509(1–2):50–8.

14.

Dougall

Bolt

Semchuk

Winkel

. Inhaler assessment in COPD patients: a primer for pharmacists. Can Pharm J. 2016;149(5):268–73.

15.

Demoly

Hagedoorn

de Boer

Frijlink

. The clinical relevance of dry powder inhaler performance for drug delivery. Respir Med. 2014;108(8):1195–203.

16.

Nave

. Determination of lung deposition following inhalation of ciclesonide using different bioanalytical procedures. Bioanalysis. 2010;2(4):807–14.

17.

Hira

Koide

Nakamura

, et al. Assessment of inhalation flow patterns of soft mist inhaler co-prescribed with dry powder inhaler using inspiratory flow meter for multi inhalation devices. PLoS One. 2018;13(2):e0193082.

18.

Dalby

Eicher

Zierenberg

. Development of Respimat® Soft Mist™ inhaler and its clinical utility in respiratory disorders. Med Devices. 2011;4:145–55.

19.

Grant

Walker

Hamilton

Garrill

. The ELLIPTA^® dry powder inhaler: design, functionality, in vitro dosing performance and critical task compliance by patients and caregivers. J Aerosol Med Pulm Drug Deliv. 2015;28(6):474–85.

20.

Johal

Howald

Fischer

Marshall

Venthoye

. Fine particle profile of fluticasone propionate/formoterol fumarate versus other combination products: the DIFFUSE study. Combination Products in Therapy. 2013;3(1):39–51.

21.

Adams

Bestall

Jones

. Inhaled beclomethasone versus budesonide for chronic asthma. Cochrane Database Syst Rev. 2002;(1):CD003530.

22.

Manning

Gibson

Lasserson

. Ciclesonide versus other inhaled steroids for chronic asthma in children and adults. Cochrane Database Syst Rev. 2008;(2):CD007031.

23.

Adams

Lasserson

Cates

Jones

. Fluticasone versus beclomethasone or budesonide for chronic asthma in adults and children. Cochrane Database Syst Rev. 2007;(4):CD002310.

24.

Dal Negro

Bonadiman

Turco

. Fluticasone furoate/vilanterol 92/22µg once-a-day vs beclomethasone dipropionate/formoterol 100/6µg b.i.d.: a 12-month comparison of outcomes in mild-to-moderate asthma. Multidiscip Respir Med. 2018;13:18.

25.

Papi

Paggiaro

Nicolini

Vignola

Fabbri

; Inhaled Combination Asthma Treatment versus SYmbicort (ICAT SY) Study Group. Beclomethasone/formoterol versus budesonide/formoterol combination therapy in asthma. Eur Respir J. 2007;29(4):682–9.

26.

Papi

Paggiaro

Nicolini

Vignola

Fabbri

; ICAT SE Study Group. Beclomethasone/formoterol vs fluticasone/salmeterol inhaled combination in moderate to severe asthma. Allergy. 2007;62(10):1182–8.

27.

Lasserson

Ferrara

Casali

. Combination fluticasone and salmeterol versus fixed dose combination budesonide and formoterol for chronic asthma in adults and children. Cochrane Database Syst Rev. 2011;(12):CD004106.

28.

Svedsater

Stynes

Wex

, et al. Once-daily fluticasone furoate/vilanterol versus twice daily combination therapies in asthma-mixed treatment comparisons of clinical efficacy. Asthma Res Pract. 2016;2:4.

29.

Bodzenta-Lukaszyk

Buhl

Balint

Lomax

Spooner

Dissanayake

. Fluticasone/formoterol combination therapy versus budesonide/formoterol for the treatment of asthma: a randomized, controlled, non-inferiority trial of efficacy and safety. J Asthma. 2012;49(10):1060–70.

30.

Cates

Lasserson

. Regular treatment with formoterol and an inhaled corticosteroid versus regular treatment with salmeterol and an inhaled corticosteroid for chronic asthma: serious adverse events. Sao Paulo Med J. 2010;128(5):310–1.

31.

Bodzenta-Lukaszyk

Dymek

McAulay

Mansikka

. Fluticasone/formoterol combination therapy is as effective as fluticasone/salmeterol in the treatment of asthma, but has a more rapid onset of action: an open-label, randomized study. BMC Pulm Med. 2011;11:28.

32.

Oba

Lone

. Comparative efficacy of inhaled corticosteroid and long-acting beta agonist combinations in preventing COPD exacerbations: a Bayesian network meta-analysis. Int J Chron Obstruct Pulmon Dis. 2014;9:469–79.

33.

Donohue

Betts

, et al. Comparative efficacy of long-acting β2-agonists as monotherapy for chronic obstructive pulmonary disease: a network meta-analysis. Int J Chron Obstruct Pulmon Dis. 2017;12:367–81.

34.

Schlueter

Gonzalez-Rojas

Baldwin

Groenke

Voss

Reason

. Comparative efficacy of fixed-dose combinations of long-acting muscarinic antagonists and long-acting beta2-agonists: a systematic review and network meta-analysis. Ther Adv Respir Dis. 2016;10(2):89–104.

35.

Kew

Dahri

. Long-acting muscarinic antagonists (LAMA) added to combination long-acting beta2-agonists and inhaled corticosteroids (LABA/ICS) versus LABA/ICS for adults with asthma. Cochrane Database Syst Rev. 2016;(1):CD011721.

36.

Ismaila

Huisman

Punekar

Karabis

. Comparative efficacy of long-acting muscarinic antagonist monotherapies in COPD: a systematic review and network meta-analysis. Int J Chron Obstruct Pulmon Dis. 2015;10:2495–517.

37.

Marsh

Zaiser

Orfanos

, et al. Evaluation of COPD treatments: a multicriteria decision analysis of aclidinium and tiotropium in the United States. Value Health. 2017;20(1):132–40.

38.

Ramli

Aljunid

Sulong

Md Yusof

. National Drug Formulary review of statin therapeutic group using the multiattribute scoring tool. Ther Clin Risk Manag. 2013;9:491–504.

39.

Nemeth

Molnar

Bozoki

, et al. Comparison of weighting methods used in multicriteria decision analysis frameworks in healthcare with focus on low- and middle-income countries. J Comp Eff Res. 2019;8(4):195–204.

40.

van Til

Groothuis-Oudshoorn

Lieferink

Dolan

Goetghebeur

. Does technique matter; a pilot study exploring weighting techniques for a multi-criteria decision support framework. Cost Eff Resour Alloc. 2014;12:22.

41.

Inotai

Nguyen

Hidayat

, et al. Guidance toward the implementation of multicriteria decision analysis framework in developing countries. Expert Rev Pharmacoecon Outcomes Res. 2018;18(6):585–92.

42.

Denford

Frost

Dieppe

Cooper

Britten

. Individualisation of drug treatments for patients with long-term conditions: a review of concepts. BMJ Open. 2014;4(3):e004172.

43.

Reilly

Taylor

Webster

. General practitioners’ attitudes towards the limited list. J R Coll Gen Pract. 1986;36(285):151–2.

44.

Ministry of Finance Malaysia, Government Procurement Division. Malaysia’s Government Procurement Regime. Ministry of Finance Malaysia; 2010.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.05 MB

0.44 MB