Structured Multidisciplinary Work Evaluation Tool (SMET): Reliability testing of a multidisciplinary/multifactorial work questionnaire

Abstract

BACKGROUND:

Occupational health services (OHS) are rarely involved in preventive issues and systematic work environment management. The Structured Multidisciplinary Work Evaluation Tool (SMET) questionnaire was created to address the lack of multidisciplinary/multifactorial OHS tools with the aim to be used in preventive issues and systematic work environment management.

OBJECTIVES:

The aim of this study was to evaluate trustworthiness of the inter-rater reliability in the qualitative analysis of the open-ended items and intra-rater reliability of the self-estimated items in the SMET questionnaire.

METHODS:

A qualitative comparison of the inter-rater reliability in the qualitative analysis of the open-ended items was performed to evaluate trustworthiness. The intra-rater reliability of the self-estimated items in the SMET questionnaire were analysed with Elisabet Svensson method.

RESULTS:

Qualitative analysis of the open-ended items showed good trustworthiness. The self-estimated items showed a high percent agreement (PA), 0.98–0.99 in the physically, 0.99 in the environmentally and 0.98–1.0 in the psychosocially demanding items. A low degree of systematic errors and individual variability were found.

CONCLUSIONS:

The SMET questionnaire shows good trustworthiness and intra-rater reliability and can be used to follow up and evaluate work environmental interventions.

Keywords

Occupational health care pragmatic value validity work environment

Abbreviations

SMET

Structured Multidisciplinary Work Evaluation Tool

OHS

Occupational health service

SWEA

Swedish Work Environment Authority

OSHC

Occupational Safety and Health Care, Region Jönköping

OHSM

Occupational health and safety management

Percentage agreement

Relative position

Relative concentration

TEST

The first – Test – questionnaire

RETEST

The second – Retest – questionnaire

SWEM

Systematic Work Environment Management

1. Introduction

Occupational health service (OHS) is a multidisciplinary knowledge area intended to promote safe, healthy, progressive and productive work conditions [1]. An important factor for the successful development of OHS activities is that the services are based on a participative methodology, with active participation from both the employer and the employee, and risk identification from a multidisciplinary perspective. A participative attitude and multidisciplinary work (which includes organizational, technical and individually based measures) has positive effects in work environment management [2, 3].

A Swedish study was conducted whose purpose was to identify conditions and key factors for successful collaboration and use of OHS providers’ services in municipalities and counties. The main finding was that OHS were only to a limited extent consulted on preventive issues and instead utilized for reactive tasks, such as rehab and curative services, and for activities that focused on individual health promotion (e.g. non-smoking programmes) [4]. Similar findings, with OHS focus on health and lifestyle rather than working conditions, are reported from another Swedish study [5]. A consequence of this is that OHS support in and about preventive occupational health and safety management (OHSM) is limited. OHS also have limited access to the actual workplaces, and to local knowledge, experiences, activities and discussions on health and safety. Utilization of OHS as an expert resource in preventive OHSM is restricted and inefficient [4]. To ensure that OHS activities are successful, there is a need not only to identify risks, but also to follow-up the outcome of the interventions. This is in line with the Swedish work environment legislation that has pointed out the importance of evaluating working environments in a systematic way [6].

A search for a practical, useful tool for evaluating the work environment from a multidisciplinary/multifactorial perspective was carried out, and no useful instrument was found [7]. Therefore, the development of a multidisciplinary/multifactorial tool was started as collaboration between the ergonomics practitioner, occupational health nurse, industrial hygiene engineer and occupational health psychologist in the Occupational Safety and Health Centre (OSHC) in Region Jönköping County. The work model was called Structured Multidisciplinary Work Evaluation Tool (SMET). SMET is a tool by which OHS can support its customers in their OHSM, because it evaluates potential risks in the work environment and gives the workplace a basis for and support in their continuing OHSM.

The practical work with SMET consists of three main parts, the initial risk-analysis of the workplace, suitable work environmental interventions and the follow-up/evaluation.

The initial risk-analysis consists of three phases, 1. the SMET questionnaire by which the employees evaluate their work environment, 2. a semi-structured interview by which the managing directors describe their work environment and 3. objective work evaluations by the OHS at the workplace. The collected results are analysed and presented together after the completion of the workplace evaluation. In this initial part the SMET questionnaire is used to identify work related problem among the employees. In a previous study the content validity of the items in the SMET questionnaire was shown to be very good [7] which implies good possibilities to identify work related problems at the workplace. However there is some uncertainty concerning the results of the open ended items since the inter-rater reliability of the qualitative analysis in the open-ended items is not tested.

The second part is aimed at work environmental interventions conducted by the workplace management, which proceeds from the initial risk-analysis phase.

The third part is the follow-up and evaluation of the aimed work environmental interventions. This follow-up and evaluation is conducted with the self-estimating items in the SMET questionnaire. In order to evaluate the results of the work environmental interventions, investigation of the reliability of the SMET evaluation instrument is needed. The importance of evaluating reliability of tools and questionnaires, like the SMET, is supported by Kimberlin and Winterstein [8].

To continue the work of establishing the quality of the SMET questionnaire, the aim of this study was to evaluate trustworthiness of the inter-rater reliability in the qualitative analysis of the open-ended items and intra-rater reliability of the self-estimating items in the SMET questionnaire.

2. Method

2.1. The SMET questionnaire development

The SMET questionnaire was developed gradually with action theory from 2008 to 2015, while being used in practical work [7]. The SMET questionnaire consists of self-reported physical (7 items), environmental (4 items) and psychosocial workload (11 items) questions answered using a 0–10 Likert scale. Each subfield (physical, environmental and psychosocial workload) also contains an item that evaluates which of the previous items constitutes the highest work demands, and an open-ended item [7]. The items in the questionnaire are presented in (Table 1).

Table 1
The items in the SMET questionnaire

Physically demanding work items

1. Do you experience any problems associated with heavy lifting in your work?

2. Do you experience any problems associated with repetitive movements in your work?

3. Do you experience any problems associated with unilateral or fixed working positions in your work?

4. Do you experience any problems associated with uncomfortable working positions in your work?

5. Do you experience any problems associated with a high work-pace in your work?

6. Do you experience any problems associated with eyesight demands in your work?

7. Do you experience any problems associated with prolonged sitting in your work?

8. Which of the previous physically demanding work factors constitute the worst demands in your work?

9. Please give a written description of the physically demanding factors that constitute the worst demands in your work. Try to be concise.

Environmentally demanding work items

10. Do you experience any problems associated with high noise levels at your workplace?

11. Do you experience any problems associated with heat, cold, or drafts at your workplace?

12. Do you experience any problems associated with the lightning in your workplace?

13. Do you experience any problems associated with narrow spaces in your workplace?

14. Are you exposed to any chemical risks in your workplace (i.e. chemotherapy, formalin, thermosetting plastic)?

15. If you answered yes to Question 14, which substance/substances?

16. Which of the previous environmentally demanding work factors constitute the worst demands in your workplace?

17. Please give a written description of the environmentally demanding factors that constitute the worst demands in your workplace. Try to be concise.

Psychosocially demanding work items

18. Do you experience any problems associated with work routines and the distribution of tasks at your workplace?

19. Do you experience any problems associated with collaboration, and feedback at your workplace?

20. Do you experience any problems associated with support from your boss/employer?

21. Dou you experience any problems with responsibilities, rights and/or expectations at your workplace?

22. Do you experience any problems associated with your possibilities to develop in your work?

23. Do you experience any problems associated with unreasonable demands in your work?

24. Do you experience any problems associated with having control and being able to handle the psychological demands that arise?

25. Do you experience any problems associated with having no time to take breaks on an ordinary working day?

26. Do you experience any problems associated with anxiety about making serious mistakes in your work?

27. Do you experience any problems associated with anxiety about not having time to complete your work?

28. Which of the previous psychosocially demanding work factors constitute the worst demands at your workplace?

29. Please give a written description of the psychosocially demanding factors that constitute the worst demands at your workplace. Try to be concise.

30. If you think about your work satisfaction and work environment, how satisfied would you say that you are?

Physically demanding work items
1. Do you experience any problems associated with heavy lifting in your work?
2. Do you experience any problems associated with repetitive movements in your work?
3. Do you experience any problems associated with unilateral or fixed working positions in your work?
4. Do you experience any problems associated with uncomfortable working positions in your work?
5. Do you experience any problems associated with a high work-pace in your work?
6. Do you experience any problems associated with eyesight demands in your work?
7. Do you experience any problems associated with prolonged sitting in your work?
8. Which of the previous physically demanding work factors constitute the worst demands in your work?
9. Please give a written description of the physically demanding factors that constitute the worst demands in your work. Try to be concise.
Environmentally demanding work items
10. Do you experience any problems associated with high noise levels at your workplace?
11. Do you experience any problems associated with heat, cold, or drafts at your workplace?
12. Do you experience any problems associated with the lightning in your workplace?
13. Do you experience any problems associated with narrow spaces in your workplace?
14. Are you exposed to any chemical risks in your workplace (i.e. chemotherapy, formalin, thermosetting plastic)?
15. If you answered yes to Question 14, which substance/substances?
16. Which of the previous environmentally demanding work factors constitute the worst demands in your workplace?
17. Please give a written description of the environmentally demanding factors that constitute the worst demands in your workplace. Try to be concise.
Psychosocially demanding work items
18. Do you experience any problems associated with work routines and the distribution of tasks at your workplace?
19. Do you experience any problems associated with collaboration, and feedback at your workplace?
20. Do you experience any problems associated with support from your boss/employer?
21. Dou you experience any problems with responsibilities, rights and/or expectations at your workplace?
22. Do you experience any problems associated with your possibilities to develop in your work?
23. Do you experience any problems associated with unreasonable demands in your work?
24. Do you experience any problems associated with having control and being able to handle the psychological demands that arise?
25. Do you experience any problems associated with having no time to take breaks on an ordinary working day?
26. Do you experience any problems associated with anxiety about making serious mistakes in your work?
27. Do you experience any problems associated with anxiety about not having time to complete your work?
28. Which of the previous psychosocially demanding work factors constitute the worst demands at your workplace?
29. Please give a written description of the psychosocially demanding factors that constitute the worst demands at your workplace. Try to be concise.
30. If you think about your work satisfaction and work environment, how satisfied would you say that you are?

The SMET questionnaire is a valid method for evaluation of the work environment from a multidisciplinary/multifactorial perspective with good communicative and pragmatic validity, and very good content validity [7].

2.2. Reliability testing of the SMET questionnaire

The reliability and trustworthiness were tested with a mixed methods approach due to the inclusion of both self-estimating and open-ended items in the SMET questionnaire. The test groups were obtained from three work environmental management projects at a middle-sized hospital in Sweden. After approval by the heads of their clinics for participation in testing of the SMET questionnaire, the employees were given both oral and written information about the reliability and trustworthiness testing of the SMET questionnaire in their respective clinics, prior to their participation in the study. The questionnaire was administered via a web-based system (esMakerNX), and the participants’ answers were treated with confidentiality. The employees from the participating clinics were able to answer the first questionnaire (TEST) within week one and the second questionnaire (RETEST) the third week. In the qualitative analysis the responses from the TEST questionnaire were used, and the quantitative analysis was performed with both the TEST and the RETEST responses. The participants were informed that participation in the study was voluntary and unknown to the head of their clinic. Consent to participate was considered to be obtained if the participants chose to actively answer the questionnaire by following the e-mailed instructions in the information e-mail. The study has not been reviewed by an institutional review board since it is not required for this type of study in Swedish ethical regulations, however the research study protocol followed the regulations in the Helsinki Declaration [9]. The reliability testing of the SMET questionnaire is shown in detail (Fig. 1).

Fig.1

Flow chart for testing of the SMET questionnaire.

2.3. Participants

Purposeful sampling was chosen to include different professions, thus implying various work exposures to physical, environmental and psychosocial items. The participants consisted of administrators from the medical clinic, cooks/kitchen assistants from the kitchen/dining hall and nurses from the orthopaedic clinic. The workplace, gender, age, profession and number of years in the workplace of the participants who answered one or both questionnaires are presented, together with response rates (Table 2).

Table 2
Descriptions of the participants

Administrative (no of individuals) Kitchen/dining hall (no of individuals) Orthopaedic (no of individuals) Total (no of individuals)

Gender

Male 0 1 0 1

Female 19 22 26 67

Age

=<30 year 0 6 0 6

31–40 year 3 6 6 15

41–50 year 5 3 9 17

51–60 year 8 8 7 23

=>60 year 3 0 4 7

Profession

Nurse – – 11 11

Assistant Nurse – – 15 15

Administrator 19 – – 19

Cook – 6 – 6

Kitchen assistant – 17 – 17

No of years in the workplace

0–2 years 3 6 6 15

2–5 years 0 2 1 3

5–10 years 2 3 4 9

>10 years 14 12 15 41

Response rate

Invited participants 20 28 37 85

Test response 19 (95%) 23 (82%) 26 (70%) 68 (80%)

Retest response 15 (75%) 14 (50%) 16 (43%) 45 (53%)

	Administrative (no of individuals)	Kitchen/dining hall (no of individuals)	Orthopaedic (no of individuals)	Total (no of individuals)
Gender
Male	0	1	0	1
Female	19	22	26	67
Age
=<30 year	0	6	0	6
31–40 year	3	6	6	15
41–50 year	5	3	9	17
51–60 year	8	8	7	23
=>60 year	3	0	4	7
Profession
Nurse	–	–	11	11
Assistant Nurse	–	–	15	15
Administrator	19	–	–	19
Cook	–	6	–	6
Kitchen assistant	–	17	–	17
No of years in the workplace
0–2 years	3	6	6	15
2–5 years	0	2	1	3
5–10 years	2	3	4	9
>10 years	14	12	15	41
Response rate
Invited participants	20	28	37	85
Test response	19 (95%)	23 (82%)	26 (70%)	68 (80%)
Retest response	15 (75%)	14 (50%)	16 (43%)	45 (53%)

2.4. The SMET questionnaire numeric scale

The participants rated 22 of 30 items in the questionnaire using a 1–10 numeric scale. The scale was thereafter trichotomized to indicate work environment risk exposure: low risk (1–3), medium risk (4–7) and high risk (8–10). Trichotomization is frequently used by the Swedish Work Environment Authority (SWEA) [10] who has a colour code in their risk assessment (green = low risk, yellow = medium risk and red = high risk) which has inspired the trichotomization in the SMET questionnaire. Trichotomization is also used scientifically [11]. The practical implication for trichotomization is that it is commonly used in Sweden when communicating quality report results to public health institutions and in politics, which makes this method pragmatically useful regarding the SMET questionnaire. The test-retest reliability test was carried out on the three proportions of the trichotomized scale.

2.5. Trustworthiness of the open ended items

The three open-ended items that the participants answered in the SMET questionnaire were evaluated in the qualitative analysis in order to analyse the inter-rater reliability. Trustworthiness of the qualitative analysis depends on credibility, confirmability, transferability and dependability of the results [12]. The trustworthiness analysis in this project focused on credibility and dependability, since the confirmability was strengthened by the mixed-methods approach of the SMET questionnaire. Evaluation of transferability was not relevant due to the aim of this study.

2.5.1 Credibility

Factors important to successfully achieve credibility are that the qualitative analysis be conducted by individuals from the OHS to ensure experience in the knowledge area, that the qualitative analysis be carried out with a neutral attitude to the information and that the qualitative analysis be carried out in teams [12]. Two working teams, A and B, were created at the OSHC. Team A consisted of one Physiotherapist/ergonomics with 11.5 years of experience in OHS (performed >45 SMET analysis) and one Occupational therapist with 2 years of experience in OHS (performed two SMET analysis). Team B consisted of one Physiotherapist/ergonomics with 4 years of experience in OHS (performed 10 SMET analysis) and one Occupational health psychologist with 11 years of experience in OHS (performed 6 SMET analysis).

2.5.2 Dependability

Factors important to successfully achieve dependability are that the exact methods of gathering, analysing and interpreting data in qualitative research must be described [12]. The qualitative analysis of the three open-ended items was conducted separately by the two teams according to an established order in SMET, designed using the principles of the work by Langemar [13].

Reading through the material to obtain a general sense of the whole statement.

Re-reading the material to condense and discriminate meaning units.

Grouping of the meaning units according to their content.

Analysis of the content groups presented together with the results of the questionnaire as a whole.

To optimize the dependability of the qualitative analysis of the SMET questionnaire, the following questions were to be asked before the analysis was completed.

Do the headings reflect the content of the answers?

Do the headings clarify the main problems?

Do the headings cover areas for which there is no condensed content?

Is there condensed content that is used under more than one heading?

Are there any headings that only have one condensed item?

For each of the three included participatory clinics, the qualitative analysis was performed by two of the authors, who are experienced in qualitative methodology. The similarities and the differences in the analysis from the two teams for all three groups were compared to enable a conclusion about the analytical quality of the teams. The use of a work model in multiprofessional teams regarding qualitative data minimizes the risk of misinterpretation of data and increases the similarities of the analytical process [14].

2.6. Reliability of the self-estimating items

The intra-rater reliability was performed within a three-week process, a recommended procedure in test-retest [15]. The first and the third week, the participants were asked to answer the SMET questionnaire. The minimum number of participants in the test-retest was settled at 20 individuals [16]. Intra-rater reliability was assessed using Elisabet Svensson method [17]. This statistical method measures percentage agreement (PA), occurrence of systematic errors with relative position (RP) and relative concentration (RC), and individual variability with relative rank variance (RV). Deviation in RP exists when estimates in the second occasion (RETEST) are systematically rated higher or lower compared to the first occurrence (TEST). Deviation in RC is present when the rated estimates on the second occasion (RETEST) have a systematically larger or smaller dispersion or are unevenly distributed compared to the first occurrence. The deviation in RP and RC is significant when the confidence interval in each item does not cover 0. Relative rank variance (RV), which measures the individual variability, is significant when RV for each item is equal or higher than 2. The limit for significance was set at α= .05. Software used for the Elisabet Svensson method was Excel 2016, and descriptive statistics were analysed with SPSS version 22 (IBM Corp., Armonk, NY, USA).

3. Results

3.1. Trustworthiness of the qualitative analysis

The trustworthiness is presented by analysing the differences and similarities in the analyses of teams A and B. The aim was to find a concordance regarding the content groups as a whole. The presentation of the data analysis varied between the teams, as team A presented an overview of all open-ended items, and team B presented the analysis from each item separately; the potential effect of this difference in analytical process on the results is described.

3.1.1 Credibility

The analysis was made as planned, with no employee changes in the analytical teams during the process.

3.1.2 Dependability

The orthopaedic clinic analysis from both teams showed that the important quotes were similar and the headings were similar, but the teams varied in how they presented the qualitative results. Team B lacked a comprehensive summary of the results. The definition of a comprehensive summary in this case is that answers from the open-ended items in the SMET questionnaire are presented together, not divided into the subgroups of physical-, environmental and psychosocially demanding factors. The lack of a comprehensive summary in team B led to some repetition of information, and more categories than in the comprehensive summary presented by team A.

The administrative staff unit analysis was the most similar in the analysis and in the naming of the categories, but the lack of comprehensive summary in team B gave more categories than for team A. This unit has less diversity in their working tasks, which might have affected the results and influenced the similarities in a positive way.

The kitchen/dining hall has more differences regarding the analytical level, where team A drew conclusions in the categorization (cause and effect categories) and team B presented the categories as they were. The different presentation regarding analytical level was present also in this analysis, and it is unclear whether the differences would have persisted in a comprehensive summary.

To further ensure unity in the analysis, the teams were asked to perform a synthesis on a comprehensive summary level of their first results and present it as a written summary. This comprehensive summary, presented as a written summary, showed that the similarities increased between the teams, compared to the preliminary analysis.

The headings (categories) represent the content of the data, and the headings included to a large extent the same data between the two teams, but team A included more categories in their headings, while team B kept the categories more separated due to less comprehensive summary of the data. The headings cover the condensed content and clarify the main problems in a more comprehensive way after comprehensive summary. The comprehensive summary lessens the risk of repetition of condensed items or headings.

Some headings have only one condensed item, but those headings and condensed items are the same in both teams’ analyses, thus indicating the importance of the headings and the specific condensed item, as well as the condensed not fitting into another heading. Therefore, both teams show ability to ensure that important information is kept, even if there is only small amount of data.

3.2. Reliability of the self-estimating items

3.2.1 Observed agreement between test-retest for the physically demanding items in the SMET questionnaire

The intra-rater reliability of the physically demanding items shows a high PA with a low degree of systematic errors in the questions verified with the RP, RC and low individual variability (RV) (Table 3).

Table 3
Observed agreement between test-retest for the physically demanding items in the SMET questionnaire. Observed agreement in percentage (PA), relative position (RP), relative concentration (RC), relative rank variance (RV) and confidence interval (CI 95%)

Item PA RP CI 95% RC CI 95% RV

1 0.99 –0.00005 –0.0002–0.0001 –0.0001 –0.006–0.003 0

2 0.98 0.0009 –0.001–0.003 –0.001 –0.008–0.005 0

3 0.99 –0.001 –0.003–0.0008 –0.003 –0.009–0.003 0

4 0.99 –0.0009 –0.003–0.001 –0.0005 –0.005–0.006 0

5 0.98 –0.001 –0.006–0.003 –0.003 –0.009–0.002 0

6 0.98 0.0008 –0.001–0.003 –0.002 –0.01–0.005 0

7 0.99 –0.0009 –0.0002–0.0001 –0.0001 –0.004–0.004 0

Item	PA	RP	CI 95%	RC	CI 95%
1	0.99	–0.00005	–0.0002–0.0001	–0.0001	–0.006–0.003
2	0.98	0.0009	–0.001–0.003	–0.001	–0.008–0.005
3	0.99	–0.001	–0.003–0.0008	–0.003	–0.009–0.003
4	0.99	–0.0009	–0.003–0.001	–0.0005	–0.005–0.006
5	0.98	–0.001	–0.006–0.003	–0.003	–0.009–0.002
6	0.98	0.0008	–0.001–0.003	–0.002	–0.01–0.005
7	0.99	–0.0009	–0.0002–0.0001	–0.0001	–0.004–0.004

3.2.2 Observed agreement between test-retest for the environmentally demanding items in the SMET questionnaire.

The intra-rater reliability of the environmentally demanding items shows a high PA with a low degree of systematic errors in the questions verified with RP, RC and RV (Table 4).

Table 4
Observed agreement between test-retest for the environmentally demanding items in the SMET questionnaire. Observed agreement in percentage (PA), relative position (RP), relative concentration (RC), relative rank variance (RV) and confidence interval (CI 95%)

Item PA RP CI 95% RC CI 95% RV

10 0.99 0.001 –0.0009–0.003 0.001 –0.003–0.006 0

11 0.99 –0.00001 –0.003–0.003 –0.0004 –0.007–0.006 0

12 0.99 0 –0.0002–0.0002 0 –0.006–0.006 0

13 0.99 0.00005 –0.0002–0.0003 0.001 –0.004–0.007 0

Item	PA	RP	CI 95%	RC	CI 95%
10	0.99	0.001	–0.0009–0.003	0.001	–0.003–0.006
11	0.99	–0.00001	–0.003–0.003	–0.0004	–0.007–0.006
12	0.99	0	–0.0002–0.0002	0	–0.006–0.006
13	0.99	0.00005	–0.0002–0.0003	0.001	–0.004–0.007

3.2.3 Observed agreement between test-retest for the psychosocially demanding items in the SMET questionnaire

The intra-rater reliability of the psychosocially demanding items shows a high PA with a low degree of systematic errors in the questions verified with RP, RC and RV (Table 5).

Table 5
Observed agreement between test-retest for the psychosocially demanding items in the SMET questionnaire. Observed agreement in percentage (PA), relative position (RP), relative concentration (RC), relative rank variance (RV) and confidence interval (CI 95%)

Item PA RP CI 95% RC CI 95% RV

18 0.99 0.002 –0.0007–0.005 0.001 –0.005–0.008 0

19 0.99 0.0002 –0.00002–0.0005 0.006 0.0002–0.01 0

20 0.99 –0.0007 –0.003–0.001 0.006 0.002–0.01 0

21 0.99 –0.0009 –0.003–0.001 0.002 –0.001–0.006 0

22 0.99 0.0009 –0.002–0.004 –0.002 –0.008–0.003 0

23 0.99 0.0001 –0.004–0.004 0.005 –0.0006–0.01 0

24 0.99 0.0001 –0.00008–0.0004 0.003 –0.002–0.01 0

25 1.0 0.0009 –0.001–0.003 –0.0007 –0.004–0.003 0

26 0.98 0.0002 –0.003–0.003 0.006 –0.002–0.01 0

27 0.99 0.0001 –0.0001–0.0004 0.003 –0.003–0.009 0

30 0.99 –0.0001 –0.0004–0.00009 –0.004 –0.01–0.002 0

Item	PA	RP	CI 95%	RC	CI 95%
18	0.99	0.002	–0.0007–0.005	0.001	–0.005–0.008
19	0.99	0.0002	–0.00002–0.0005	0.006	0.0002–0.01
20	0.99	–0.0007	–0.003–0.001	0.006	0.002–0.01
21	0.99	–0.0009	–0.003–0.001	0.002	–0.001–0.006
22	0.99	0.0009	–0.002–0.004	–0.002	–0.008–0.003
23	0.99	0.0001	–0.004–0.004	0.005	–0.0006–0.01
24	0.99	0.0001	–0.00008–0.0004	0.003	–0.002–0.01
25	1.0	0.0009	–0.001–0.003	–0.0007	–0.004–0.003
26	0.98	0.0002	–0.003–0.003	0.006	–0.002–0.01
27	0.99	0.0001	–0.0001–0.0004	0.003	–0.003–0.009
30	0.99	–0.0001	–0.0004–0.00009	–0.004	–0.01–0.002

4. Discussion

4.1. Result discussion

The analysis shows that the trustworthiness of the open-ended items in the questionnaire is high when the qualitative analysis is presented as with a comprehensive summary of the results (without headings, categories or condensed items). The trustworthiness of the analysis is an important finding in the practical OHS work with SMET. The result shows that when compiling the results from the open-ended items during the initial risk-analysis with the SMET questionnaire, the result will have high trustworthiness regardless of which OHS representative that conduct the summary.

The results of this study also show a generally high PA and a low degree of systematic errors and individual variability for almost all the items included in the SMET questionnaire. This result prove that the instrument has a good reliability and that the questions are stable. The good intra-rater reliability is also an important finding in practical OHS work with SMET, which supports the use of the trichotomized self-estimating items in the SMET questionnaire in a follow-up and evaluation of work environmental interventions. The result shows with high probability that changes in the proportions of the trichotomized self-estimating items reflect real changes in how the employees experience their work environment.

Taken together the result of this study supports the usefulness of the SMET questionnaire in descriptions of employee's experiences of their work environment in an initial risk-analysis and in follow-up and evaluation of work environmental interventions, due to the high PA of the trichotomized self-estimating items.

The Swedish work environment authority's statute book: Systematic work environment management (AFS 2001:1) provisions, defines Systematic Work Environmental Management (SWEM) as ‘the work done by the employer to investigate, carry out and follow up activities in such a way that ill-health and accidents at work are prevented and a satisfactory working environment achieved’ [6]. The provision also states that ‘when the competence within the employer’s own activity is insufficient for SWEM or for work relating to job adaption and rehabilitation, the employer shall engage occupational health services or corresponding expert assistance from outside’ [6]. Despite this, OHS is seldom involved in proactive, all-encompassing interventions at workplaces, but often involved at the individual level [4]. The results in this and our previous study support the use of the SMET questionnaire as an OHS tool for a risk analysis of the workplace and to follow up the results of workplace interventions. Consequently, SMET might be suitable to use as an OHS tool in SWEM and the result from our study, with a study population with a wide variety of work demands, supports the usefulness of the SMET questionnaire in most workplaces.

A benefit with the SMET questionnaire is that the numbers of items in the questionnaire are limited. That makes the SMET questionnaire practically useful and easier to administer than full-scale batteries of work environmental items like the Dictionary of Occupational Titles (DOT) and Occupational Information Network (ONET) [18].

It is also important to point out that in the SMET procedure, the result from the SMET questionnaire has to be complemented with a semi-structured interview of the managing director and the ‘objective’ evaluation of the problems experienced by OHS at the workplace.

4.2. Method discussion

The self-estimating items in the SMET questionnaire have been designed with a 1 to 10 categorical scale. This scale was chosen due to its high sensitivity [19, 20]. In practical work with the SMET questionnaire the 1–10 scale is used to identify those who, both individually and on a group level, rate their work environment as either 1 (no problems) or 10 (major problems). Without the 1–10 scale in the questionnaire, there is an obvious risk that these individuals’ experiences of work-related problems may be missed, and that the individuals would, through continued work exposure, encounter a bigger risk of developing work-related illness in the future [21, 22]. In our practical work with the SMET questionnaire the trichotomization from a 1–10 scale to a 1–3 scale has been an important step, primarily when presenting the results to the executives.

The quantitative method used in the present study provides an opportunity to analyse ordinal data without the need for any specific grouping or reprocessing. In addition, valuable information is obtained, such as from the ability to detect both PA and systematic and individual estimation errors [23].

It is recommended that the number of participants be at least 20 individuals, and the time in between the measurements in test-rest be two to four weeks [15, 16]. Those requirements were principally met in the study, which indicates that the intra-rater reliability results should be reliable.

The results regarding dependability of the qualitative analysis shows that the pre-understanding of the field and the developmental process during the analysis might affect the results. It is possible that the qualitative analysis was affected by the order of the analysis: the kitchen/dining hall unit was the last unit to be analysed at the analytical session for team A, which might have caused the team to move their analysis further towards thinking in cause and effect, instead of only presenting data. Therefore, it is advisable to perform the analysis on only one unit at a time, which also is the work reality for the teams.

The differences between the teams conducting the qualitative analysis of the open-ended items led to a question regarding the need to reach a comprehensive summary, in order to provide a more comprehensive picture of the workers’ experiences of the work environment to the management of the clinic. It is possible that although both teams shared long work experience in OHS, the difference within the teams regarding experience of qualitative analysis might be important to acknowledge. It is important to give all teams education in this field to ensure methodological quality in the qualitative analysis. The credibility of qualitative studies is enhanced by a clear description of the data collection process and analysis [24], which is part of the aim of this project. The results of the qualitative analysis indicate the need for educated staff regarding qualitative analysis, but also offer a clear manual for how to perform the analysis, including a comprehensive summary and presentation of the result in a written summary. Another finding during the trustworthiness analysis is that headings with only one condensed item may be useful in order not to lose important information. This is also important to consider in practical work settings.

The mixed-methods approach of the SMET questionnaire is a benefit in the clinical usefulness of the instrument, but it also provides a continuous learning process regarding occupational health for those using the instrument.

The practical work with SMET questionnaire is strengthened by the ongoing learning process that appears during the practical use of the instrument. The practical usefulness of SMET questionnaire and the reliability of the tool demand stringent use of the steps involved in the SMET process. To this date SMET and the SMET questionnaire has been used in practical work in Sweden during the 10-year period of development of the procedure and the questionnaire. Further research with intervention studies and case studies are on-going.

5. Conclusion

The SMET questionnaire is a reliable method for evaluation of the work environment from a multidisciplinary/multifactorial perspective in the OHS reality, due to its trustworthiness when compiling the open-ended questions in the initial risk-analysis and due to its test-retest reliability in a follow-up procedure.

Author contributions

Haraldsson: Planned the study, contributed to the study design, conducted a specific literature review, wrote the introduction, method, result and discussion; approved the final manuscript version, and sent in the final manuscript.

Jonker: Planned the study, contributed to the study design, made continuous proof and participated in discussions regarding the manuscript; approved the final manuscript version.

Rolander: Planned the study, contributed to the study design, made continuous proof and participated in discussions regarding the manuscript; approved the final manuscript version.

Strengbom: Planned the study, contributed to the study design, made continuous proof and participated in discussions regarding the manuscript; approved the final manuscript version.

Areskoug Josefsson: Planned the study, contributed to the study design, made continuous proof and participated in discussions regarding the manuscript, approved the final manuscript version; scientific supervisor.

Conflict of interest

None to report.

Footnotes

Acknowledgments

Thanks to Ann Lund for all your work with SMET since the start 2008. Thanks to the SMET team (Ann Lund, Dirk Jonker, Marianne Rosgren, Erik Strengbom) for all the work that has made SMET what it is today. Thanks to our colleagues at OSHC, for all your practical work and your thoughts about how to develop SMET further. A special thanks to Marianne Rosgren, Occupational Safety and Health Care, Region Jönköping County, who made it possible to carry out this study.

References

RantanenJ %. Future perspectives in developing high-quality occupational health services for all. in International Symposium on Good Occupational Health Practice and Evaluation of Occupational Health Services. People and work; Helsinki: Finnish Institute of Occupational Health 1991.

OSHA. Work-related musculoskeletal disorders: Prevention report. European Agency for Safety and Health at Work, editor 2008.

Hagberg

, Silverstein

, Wells

, Smith

, Hendrick

, Carayon

, Pirusse

Work related musculoskeletal disorders (wmsds): A reference book for prevention. London: Taylor & Francis.; 2005.

Schmidt

, SjöströmJ %. IVL-rapport C 125 Anvandning av företagshalsovärd i kommuner och landsting [IVL-report C 125 Usage of occupational health service in municipalities and counties]. IVL Svenska Miljoinstitutet [IVL Swedish Environmental Research Institute]: 2015.

Grooten

, Muller

, Forsman

, Kjellberg

, Toomin-gas

, Bjorn Olov

, SvartengrenM %. Health risk appraisals in Swedish occupational health services. Work (Reading, Mass). 2016;55(4):849–59. PubMed PMID: 28059808. Pubmed Central PMCID: PMC5240593. Epub 2017/01/07.eng.

The Swedish Work Environment Authority. Provision AFS 2001:1, Systematic Work Environment Management, (2001).

Haraldsson

, Jonker

, Strengbom

, Areskoug-Josefsson

. Structured Multidisciplinary work Evaluation Tool: Development and validation of a multidisci-plinary work questionnaire. Work (Reading, Mass). 2016; 55(4):883–91. PubMed PMID: 28059819. Epub 2017/01/ 07. eng.

Kimberlin

, WintersteinAG %. Validity and reliability of measurement instruments used in research. American journal of health-system pharmacy : AJHP : Official Journal of the American Society of Health-System Pharmacists. 2008;65(23):2276–84. PubMed PMID: 19020196. Epub 2008/11/21. eng.

Association WM. WMA Declaration of Helsinki - Ethical Principles for Medical Research Involving Human Subjects. www.wma.net [2016-11-15].

10.

Ergonomics for the Prevention of Musculoskeletal Disorders AFS 2012:2, provision, (2012).

11.

Altman

, RoystonP %. The cost of dichotomising continuous variables. BMJ : British Medical Journal. 2006;332(7549):1080. PubMed PMID: PMC1458573.

12.

KreftingL %. Rigor in qualitative research: The assessment of trustworthiness. Am J Occup Ther. 1991;45(3):214–22. PubMed PMID: 2031523. Epub 1991/03/01. eng.

13.

LangemarP %. Qualitative science in psychology (SWE: Kval-itativ forskningsmetod i psykologi. Att lâta en varld oppna sig). Stockholm: Liber AB; 2008.

14.

Gladney

, Ayars

, Taylor

, Liehr

, MeiningerJC %. Consistency of Findings Produced by Two Mul-tidisciplinary Research Teams. Sociology. 2003;37(2): 297–313.

15.

Waltz

, Strickland

, Lenz

ER.

Measurement in nursing and health research New York: Springer; 2005.

16.

Hobart

, Cano

, Warner

, ThompsonAJ %. What sample sizes for reliability and validity studies in neurology? J Neurol. 2012;259(12):2681–94. PubMed PMID: 22729386. Epub 2012/06/26. eng.

17.

Svensson

, HolmS %. Separation of systematic and random differences in ordinal rating scales. Statistics in medicine. 1994;13(23-24):2437–53. PubMed PMID: 7701145. Epub 1994/12/15. eng.

18.

Information Technology Associated www.occupationalinfo.org [20180416].

19.

Skovlund

, Bretthauer

, Grotmol

, Larsen

, HoffG %. Sensitivity of pain rating scales in an endoscopy trial. The Clinical Journal of Pain. 2005;21(4):292–6. PubMed PMID: 15951645. Epub 2005/06/14. eng.

20.

Breivik

, Bjornsson

, SkovlundE %. A comparison of pain rating scales by sampling from clinical trial data. The Clinical Journal of Pain. 2000;16(1):22–8. PubMed PMID: 10741815. Epub 2000/03/31. eng.

21.

Coenen

, Kingma

, Boot

, Twisk

, Bongers

, van Dieen

. Cumulative low back load at work as a risk factor of low back pain: A prospective cohort study. J Occup Reha-bil. 2013;23(1):11–8. PubMed PMID: 22718286. Pubmed Central PMCID: PMC3563950. Epub 2012/06/22. eng.

22.

Gullander

, Hogh

, Hansen

, Persson

, Rugulies

, Kolstad

, Thomsen

, Willert

, Grynderup

, Mors

, Bonde

. Exposure to workplace bullying and risk of depression. J Occup Environ Med. 2014;56(12):1258–65. PubMed PMID: 25479295. Epub 2014/12/06. eng.

23.

SvenssonE %. Different ranking approaches defining association and agreemant measures of paired ordinal data. Statistics in Medicine. 2012;31(26):3104–17.

24.

Graneheim

, LundmanB %. Qualitative content analysis in nursing research: Concepts, procedures and measures to achieve trustworthiness. Nurse Education Today. 2004;24(2):105–12. PubMed PMID: 14769454. Epub 2004/02/11. eng.