Abstract
Objective:
To develop an improved tool with which to assess the efficacy of the Dangerous Decibels (DD) programme, delivered as a training programme. The theoretical foundations for the tool were the capabilities, opportunities, motivation-behaviour (COM-B) model, and a separate set of scales was included at the level of individual components of the programme.
Design:
The research was conducted in three phases: draft questionnaire design, Delphi review, and pilot testing.
Setting:
Primary schools in the Auckland region, Aotearoa New Zealand.
Method:
The Dangerous Decibels Assessment (DDA) questionnaire was developed. A Delphi review was conducted to assess readability, face validity, and content validity. The questionnaire was tested for age appropriateness and intelligibility using a small sample of children (n = 23) and, finally, piloted with a larger sample of children (n = 78) along with the existing assessment questionnaire.
Results:
The questionnaire achieved a consensus of >78% for readability, >78% face validity, and 100% content validity. Component and overall scores for the DDA improved significantly after the DD training programme, as did the overall score on the original assessment questionnaire. Most scales achieved acceptable internal consistency (Cronbach’s alpha value of > .7).
Conclusion:
The DDA can be used to evaluate the efficacy of the DD programme in terms of its component parts and overall. It will be useful for evaluating the future efficacy of DD training.
Keywords
Introduction
High-level noise exposure causes permanent damage to cochlear structures, including hair cells, stria vascularis, synapses, and spiral ganglia (Kujawa and Liberman, 2009). This damage leads to noise-induced hearing loss (NIHL). Children are vulnerable to NIHL (Henderson et al., 2011; Niskar et al., 2001; Su and Chan, 2017), so it is important to promote hearing health and improve hearing protective behaviour from childhood.
Hearing health promotion programmes targeting school children have been conducted to improve knowledge, attitudes, and behaviour related to noise and hearing conservation (Martin et al., 2013). More sustainable improvements in attitudes and intended behaviours have been noted in children aged 10 than among teenagers (Griest et al., 2007). Dangerous Decibels (DD) is a theory-based hearing health promotion programme designed to train children to (1) distinguish between safe and dangerous sound levels; (2) know the sources of dangerous sound and its effects on hearing; (3) know three ways to protect hearing (hearing protective strategies); and (4) understand that damage caused by loud noise is permanent but preventable (Martin, 2008; Martin et al., 2006).
The DD programme seeks to influence children through information and activities about sources of dangerous sounds, the impacts of exposure to them, and hearing protection. The training has nine “modules,” each with a specific hearing health promotion message (Table 1). The programme has been shown to be successful, with sustained improvement in knowledge, attitudes, and intended behaviours of primary-school children in relation to hearing protection from noise (Griest et al., 2007; Martin et al., 2013).
Structure of and key messages in the Dangerous Decibels programme.
A questionnaire was developed to evaluate the knowledge, attitudes, and intended behaviours (KAB questionnaire) of children trained in the DD programme (Griest et al., 2007) and has been adapted for workers from noisy industries (Reddy et al., 2017). While existing questionnaires capture the programme’s overall outcomes, the efficacy of individual modules is not measured. It has been suggested that research to improve the DD programme could be strengthened by designing a questionnaire that measures this (Welch et al., 2016).
Theoretical framework – The COM-B model
The COM-B model posits that personal capabilities, opportunities, and motivation determine the uptake of health behaviours (Michie et al., 2011). For a person to engage in a behaviour, they require psychological and physical capabilities. They also need opportunities provided by the external environment, including physical and social factors that shape their perception of the world. Motivation includes analytical decision-making, emotional responses, and habitual patterns. Capability, opportunity, and motivation work together to influence the enactment of the behaviour (West and Michie, 2020). The relationship between motivation and behaviour depends on capability and opportunity, which are both necessary for motivation to influence behaviour. In addition, capability and opportunity influence motivation: when a person knows how to do something, and the opportunity is present, they tend to be more motivated to do it. Furthermore, engaging in a behaviour enhances capability (through practice) and increases motivation to continue the behaviour in future.
The COM-B model has been applied across a range of health-promotion interventions, including hearing aid usage (Barker et al., 2016), falls prevention (Khong et al., 2018), dietary behaviour change (McEvoy et al., 2018), and vaccine uptake (Habersaat and Jackson, 2020), but has not been used in hearing health promotion. In this research, COM-B was operationalised as scales in a questionnaire to measure the overall efficacy of the DD training.
The research aimed to develop a new assessment tool that would use the COM-B model to assess the overall efficacy of the Dangerous Decibels programme and that would also provide a measure of learning within the individual modules.
Materials and methods
This research involved three steps: (1) draft questionnaire design; (2) Delphi review; and (3) pilot testing. Throughout this process, seven iterations of the questionnaire were developed (Figure 1). Ethical approval for the study was received from the Human Participants Ethics Committee at the University of Auckland (Reference: 024252).
Stages in the study.
Draft questionnaire design
A multiple-choice format was chosen for ease of responding and scoring. Two to three preliminary items were developed to cover each module independently, at an appropriate level for children aged 8 to 12 years. Items had direct questions, were as short as possible, and were simple and clear. The response options did not provide clues to the correct answer, were kept to a minimum (usually yes/no), were designed to be plausible, and avoided “all the above” and “none of the above” options (Borgers et al., 2000; de Leeuw, 2011; Fowler, 2014; Scott, 1997).
The first version of the Dangerous Decibels Assessment (DDA1) had 46 items of which 3 assessed respondents’ demographic characteristics. The final version (DDA7) retained the same demographic questions and was modified to include single-choice and multiple-choice items (one correct answer and two to three distractors), two option forced-choice items (Yes/No), and list of items that required respondents to select correct answers from several alternatives.
Three versions of the DDA7 were developed to be administered at three time points: pre-training (administered before training), post-training (administered shortly after training), and follow-up (administered 3 months after training). The full questionnaires, administration procedures, and scoring instructions are available from the corresponding author upon reasonable request.
Delphi review
A Delphi review assessed the questionnaire’s content, face validity, and readability. The Delphi process comprises rounds of feedback from a panel of experts, modifying the work to be commented upon after each round (Gill et al., 2013). This process has been extensively used to seek consensus when developing assessments (Crookes et al., 2010; Sowter et al., 2011). The DD Module-specific part of the questionnaire was sent to nine international experts on the DD programme for feedback. These experts were fluent in English and experienced in health promotion and the DD programme.
The review was conducted in three rounds, where data in which data were collected using an anonymous online Qualtrics questionnaire. Participants were asked to indicate “yes” or “no” for questions about the face validity and readability (for 8- to 12-year-old children) of each item and the overall content validity of the questionnaire. They were also asked to comment, especially if they had given a “no” response. Face validity was deemed high when the items appeared appropriate and relevant. Content validity described whether the questionnaire covered the programme’s overall objectives. Readability was assessed qualitatively through expert judgements, focusing on sentence simplicity, vocabulary load, and age appropriateness. Across three rounds, experts were shown anonymised, aggregated feedback from the previous rounds, and suggested modifications. We used a consensus threshold of 70% (Keeney et al., 2010). This level of agreement was achieved at the end of the three Delphi rounds.
Testing the questionnaire
Feasibility testing
After the first two stages of the Delphi review, the questionnaire, version DDA3, was checked for age appropriateness, readability, and intelligibility with a group of 23 children from a children’s group in Auckland aged 8–12 years. Age appropriateness assessed whether the concepts and vocabulary in the questionnaire were suitable for their developmental age; readability referred to how easily children could interpret the wording; and intelligibility assessed whether each item was comprehended as intended. All the children were trained with the DD classroom programme. The DDA3 questionnaire was administered before and after the training. Afterwards, a researcher who was not involved in delivering the DD programme conducted a guided discussion and notes were taken. Each item in the DDA3 was individually reviewed with the children, who spoke about their understanding of the item. The researcher also explored areas children found difficult to read and understand, and issues identified during the Delphi review were discussed. Version DDA4 was formulated after considering the feedback from the focus group discussion.
The DDA4 was subjected to the third round of the Delphi review, and after considering the comments, a fifth version of the questionnaire (DDA5) was formulated (Figure 1).
Adding COM-B scales
The total score provides one measure to assess the overall efficacy and effects of the training; however, it was also desirable to develop an overall measure based on the COM-B to capture changes in the underlying behavioural components, such as Capability, Opportunity, and Motivation. Therefore, a set of 12 questions was added to the DDA5 to develop the DDA6, which was administered during the pilot testing. These were aligned with the components of the COM-B and based on three scenarios that would provide evidence of intended behaviour under conditions that 8- to 12-year-old children might conceivably face.
Pilot testing
The questionnaire (DDA6) consisted of demographics, module-specific items, and COM-B scales. It was administered before and after the DD training for 78 children (49% male; median age = 9 years, age range = 8–12 years). All participants were first-language English speakers and identified as NZ Europeans (63%), Māori (18%), Chinese (9%), Samoan (3%), Indian (3%), Tongan (3%), and Other (3%). As a comparison measure, the existing KAB questionnaire (Griest et al., 2007) was also administered to 55 of the children.
Based on the responses from the pilot testing, a seventh and final version of the DDA was developed (Figure 1).
Scoring and statistical analysis
One point was given for the following responses:
Correct answer to single-choice and yes/no questions;
In lists of options, where some options were correct and some incorrect, one point was awarded for each correct option selected and each incorrect option not selected.
The DDA was designed to provide three types of information: overall training efficacy, efficacy at the level of each of the nine modules, and efficacy in terms of the parameters of the COM-B. These were: (1) The total points on the 43 items that assessed the DD training provided the “Total modular score”; (2) The total of the points from the items relating to each of the nine modules provided “Module-specific scores”; and (3) The total points for the items related to the four COM-B parameters provided the “Capability, Opportunity, Motivation, and Behaviour scores.”
Pre- and post-scores were tested for normality assumptions using the Kolmogorov–Smirnov and Shapiro–Wilk tests. The test statistics were p < .001, df = 78 for the DDA6 and p < .001, df = 55 for the KAB questionnaire, skewness = 0.139–1.722, and kurtosis = 0.003–2.181 for both questionnaires. These outcomes indicated that the data were not normally distributed, and we proceeded to analyse them using nonparametric statistical tests.
Cronbach’s alpha was calculated to assess the internal consistency of the items. The significance of differences between pre- and post-training Total modular scores, Module-specific scores, the COM-B scores, and the KAB scores were tested using the Wilcoxon signed-rank tests. Across the various measures, the maximum possible score on the scales varied, so to improve clarity and to allow comparability, data were converted to the proportion of the maximum possible (POMP) score in each scale (Cohen et al., 1999).
Results
Delphi review
Face validity and readability of the module-specific items
The face validity and readability ratings increased consistently over the rounds, and all the items reached a consensus (⩾78%) at the end of the third round.
Content validity of the module-specific items
Content validity of the module-specific items improved over the process, and the Delphi panel agreed unanimously that all the items had content validity at the end of the third round.
The changes made included simplifying the response options and changing the item format into direct questions from fill-in-the-blank items (e.g. DDA1 item: “Sound can be measured in . . . (select only one)” became DDA2 item: “What is the unit of measurement of sound?”). The DDA3 also encouraged respondents to pick a definite answer by removing “I do not know” and “Not sure” options from most items, and it was simplified by reducing the number of distractors in multiple-choice items to a minimum of 2.
Testing the questionnaire
Feasibility testing
During the focus group with 8- to 12-year-olds, two main points were raised about the format of the items: (1) participants reported that they found it challenging to differentiate between true and false response options, so these were changed into yes/no responses; (2) seven items allowed multiple responses, but since this was not clear to participants some provided only one response, even though they recognised other appropriate options. We also added specific instructions about the number of response options for those seven items. For example, item 3, “How do you protect your hearing from noise? (select all the answers you think are correct),” was changed to “What are the ways to protect your hearing? (Choose the four correct answers).”
We then sent the DDA4 for the third round of the Delphi review. Consensus was reached at this point, with some minor changes to wording, resulting in the DDA5 (Figure 1).
Pilot testing
Post-training scores on the DDA6 increased overall (x̅ = 46.15 and SD = 3.78) compared to the pre-training score of the DDA6 (x̅ = 26.89 and SD = 5.91).
Changes in scores for all modules and the COM-B components improved after training (Figures 2 and 3). Almost all participants scored the maximum possible in each scale after training, whereas before training, scores were more generally distributed, although in many cases distributions were already skewed towards the higher end of the scales. Similar improvements were observed in scores for knowledge, attitudes, and behaviours for the KAB questionnaire (Figure 4).
Histograms showing the distribution of participant scores on each of the nine modules, and the total modular score, before and after training.
Histograms showing the distribution of participant scores on the four elements of the COM-B model, before and after training.
Histograms showing the distribution of participant scores on the three scales of the KAB questionnaire, before and after training.
Cronbach’s alpha was calculated for pre- and post-training questionnaires to assess the reliability of each module-specific subscale. The majority of reported alpha values were > .7, indicating acceptable internal consistency and suggesting that items within those subscales measured related constructs. Only module 9 had a low alpha value, implying that the items within it were not intercorrelated (Table 2).
Reliability analysis for the post-training questionnaire.
Note. Qx = Question number; (1-n) = option numbers used for scoring; Qx.y = sub-question y of question x. For example, Q22.3 (1-4) refers to question 22, sub-question 3, and uses options 1 through 6. Scales with Cronbach’s alpha values greater than 0.7 are deemed to have acceptable internal reliability.
The Wilcoxon signed-rank test was conducted to compare the median pre- and post-training scores (Table 3). There were significant differences between the scores, with a higher median score post-training than pre-training for both DDA4 (for all the questions: Mdn ⩾ 1, z ⩾−7.726, p < .05) and KAB questionnaires (for all the questions: Mdn ⩾ 3, z ⩾ −6.453, and p < .05).
Scores for the nine Dangerous Decibels module scales, the COM-B Model scales for DDA6 and the KAB Questionnaire scales pre- and post-training.
Note. Sub-columns show the mean total score on all items within each scale and the POMP score at each time point. All scales differed significantly (p < .001
p < .001.
Discussion
The study aimed to develop a questionnaire to assess the efficacy of the DD classroom programme in terms of the nine modules and the COM-B model. Three slightly different versions of the questionnaire were developed to be used prior to training, soon after training, and some months after training. Module-specific questions achieved consensus for content validity, face validity, and readability in a Delphi review of DD experts. Pilot testing revealed that the DDA6 could assess the overall and modular efficacy of the training and informed the development of the final version (DDA7).
Delphi review
There was disagreement among the Delphi panel about providing specific instructions to choose the correct number of options where multiple correct answers were desired. Some panel members felt that the decision about how many answers were correct was part of the question, while others believed that children would perform better when prompted to choose the correct number of options. The literature advises that children take more time to process information than adults (Kail, 1991; Kail and Ferrer, 2007), so specific and helpful instructions should be added to the questions to maximise the responses (Borgers and Hox, 2000). Furthermore, the literature shows that direct questions should be used to avoid confusion (Scott et al., 1995). According to the published literature, there are both advantages and disadvantages to offering an “I do not know” option (Bell, 2007). On the one hand, it can provide a valid response where the absence of a reply is genuine; however, it can be a way to avoid responding (Borgers and Hox, 2000). It was decided to use it sparingly. The questionnaire was designed not to be overwhelming or overly simplistic. To achieve this, we included different question types (e.g. multiple-choice and yes/no) and carefully chosen vocabulary for questions and answers. The Delphi process and testing with a small group of children revealed the age-appropriate nature of the questions and responses.
Pilot testing
The post-training assessment revealed a ceiling effect in some questions, where children’s scores reached or approached the maximum value due to the training’s impact, resulting in low variance for those scales. This effect was attributable to the limited number of response options in the questionnaire. There is a trade-off between the suitability of the questionnaire for children in this age range and the quality of the data from an analytical perspective (Borgers et al., 2000), and the process we followed led to favouring the shorter, simpler questionnaire and practical administration.
Analysis of the Dangerous Decibels programme using the COM_B model
The COM-B offers a model for hearing protective behaviour. The underlying link between behavioural change and motivational change within the model implies that when children are more motivated, they report that they will protect their hearing from noise. Correlations between motivation and health behaviour have been observed previously (Moorman and Matulich, 1993). In the COM-B model, behaviour is reciprocally linked to capability, opportunity, and motivation, and motivation is connected with capability, opportunity, and behaviour (Michie et al., 2014).
When designing the questionnaire, generating items related to the motivation construct proved difficult. There are two types of motivation: reflective and automatic. Automatic motivation involves instinctive responses, such as removing a hand from a hot surface, whereas reflective motivation is a conscious decision. Humans evolved in relatively quiet environments, so our automatic protective mechanisms do not protect our hearing from continuous noise exposure. Protecting hearing is therefore conscious, so it relies on reflective motivation, which can be acquired through learning, such as the DD training. We used the interpretation given for motivation in the COM-B model (West and Michie, 2020) which describes motivation as a product of wants and needs arising at the particular moment when a behaviour occurs. In the questionnaire, participants were asked whether they want to protect their hearing under different circumstances (e.g. “Imagine you are playing video games with a friend. Do you want to play the game at a high volume?”).
Improved opportunities, including external environmental factors that influence behaviour, contribute to protecting children’s hearing from noise. These include the physical opportunities provided by the constructed environment and social opportunities offered by the cultural environment, which determine how individuals perceive the world. Aligning with this, the DD programme encourages children to identify the social opportunities, such as attending a rock concert and questioning them about their intention of using the opportunity to protect themselves and their friends’ hearing from the loud music.
In the DDA7 questionnaire, children were presented with scenarios related to physical opportunities, for example, taking earplugs to a fireworks display, considering a friend’s request to reduce the volume of a video game and identifying road drilling as a potential opportunity to protect hearing. Several studies have demonstrated the correlation between opportunity and behaviours in a health context (Howlett et al., 2021; Li et al., 2019), whereas others have not (Bettiga et al., 2018; Gruen et al., 2006). Nevertheless, being aware of the available opportunities, including peer behaviours, opinions, and social norms, improves hearing protection behaviour. Furthermore, the COM-B model suggests that opportunity influences the relationship between motivation and behaviour as opposed to the behaviour itself (West and Michie, 2020). This highlights the importance of creating environments and cultural contexts that encourage positive behaviours, such as hearing protective strategies taught in the DD programme (“turn it down,” “walk away,” and “protect your ears”).
The DD programme seeks to increase children’s motivation to take opportunities to protect their hearing. For example, if a child has the opportunity to use earplugs (e.g. noise exposure while carrying earplugs) but lacks motivation, they may not use them. If the child is highly motivated but earplugs are unavailable, they cannot take up the opportunity. In certain instances, opportunity can directly affect motivation; for example, if a child is exposed to a persuasive message promoting hearing protection behaviour, but has no exposure to high-level sound (i.e. no opportunity), this will decrease their motivation to engage in the behaviour.
The COM-B model also contains a feedback loop (Mayne, 2017); a change of behaviour can improve motivation. For example, good hearing protective behaviour may motivate a person to continue the behaviour to preserve their hearing in an ongoing way.
Conclusion
A new questionnaire was developed to evaluate the efficacy of the DD classroom programme by identifying key messages and devising module-specific and overall measures. The COM-B model was used as the theoretical foundation. Despite the limited scale variability caused by ceiling effects, the questionnaire was designed to be simple, concise, and suitable for use with children aged 8–12 years. The DDA serves as useful assessment tool for evaluating the programme’s efficacy, offering insights at a modular level for future programme development and providing an overall measure of COM-B in relation to preventing NIHL. The questionnaire we have developed will be useful for future research into the efficacy of DD training. For example, it could be used in research to assess the efficacy of DD training in diverse populations, to explore the stability of the training over time, and to assess how revisions of the content of the DD training may influence its efficacy.
Footnotes
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by a PhD scholarship (AHEAD/PhD/R1/AH/032) from the AHEAD project and research funding from Hearing New Zealand.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data availability statement
Due to the terms of the ethical approval received by this project, the data are not available.
AI use
Generative AI was not used in the development and writing of this paper.