How do the particular characteristics of less-educated employees with disabilities impact survey implementation?

Abstract

BACKGROUND:

Employees with disabilities make up a significant share of the working population. The group of employees covered by this study is hard to include in research and yet must be researched due to the vulnerable position it holds in the labour market. The topic is quite complex.

OBJECTIVE:

The article’s main goal is to demonstrate how to implement a survey and adapt a questionnaire for assessing competencies and motivation for training and career changes among older and less-educated employees who have disabilities.

METHODS:

In the paper, we discuss the approach to adapting a questionnaire and a survey by undertaking an extensive process of different testing and adaptation stages that is presented in the article.

RESULTS:

We highlight some obstacles that employees with disabilities face when participating in surveys due to their low literacy skills, as well as low self-esteem, accessibility issues and other general methodological issues in the context of our population. Potential solutions gathered from all phases of the adaptation process are discussed.

CONCLUSIONS:

Proper survey implementation and questionnaire modification must be ensured if researchers aim to increase the willingness of individuals with disabilities to participate in the survey and to gather quality results.

Keywords

Questionnaire adaptation survey implementation competencies assessment motivation training employment changes employees with disabilities

1 Introduction

Properly adapting a questionnaire to a specific target group entails a rigorous process. Adapting measurement instruments, particularly questionnaires and the measurement scales they contain, is not new in scientific research. Such adaptations are made for different linguistic or ethnic groups [1], specific needs [2] and target groups [3] and to simplify a questionnaire to suit the nature of the target group or increase a questionnaire’s practical usefulness [4].

In this article, we aim to show how to adapt a questionnaire for older and less educated employees with disabilities who form a highly specific research group and are the subject of various research restrictions. Akcil, for example, researched how to make learning applications on mobile devices more accessible to visually impaired learners and how to support the learning process through screen technology and interaction within the framework of social inclusion and digital inclusion. The lack of inclusive mobile learning applications for visually impaired learners and the absence of practices concerning mobile learning applications affected the sampling process [5].

Researchers have already developed and used certain adapted versions of measurement instruments for collecting data from people with disabilities or individuals with serious health problems [6]. This group of employees is typically hard to include in research and yet are a ‘must be researched’ group of employees.

The article’s primary goal is to demonstrate how to (successfully) implement surveying and adapt a questionnaire for assessing work-acquired competencies and motivating older and less-educated employees with disabilities for: i) education and training; and ii) career changes. To our knowledge, this topic has yet to be addressed in relation to this population, despite workers with disabilities accounting for a significant share of the working population.

This article also discusses theoretical perspectives of less-educated employees with disabilities. Next, we discuss the approach to adapting the questionnaire and sampling, followed by a section on the methodological characteristics of the sample with regard to questionnaire adaptation. The topic is quite complex and multidimensional. We highlight issues of workers with disabilities, such as resistance to survey participation due to low literacy skills and low self-esteem, accessibility issues and other general methodological issues in the context of our population. We raise possible issues arising from self-assessment, questionnaire length, acquiescence effect, wording, response scales, polarity of questions, and question order. Potential solutions are discussed.

2 Theoretical perspectives of less-educated employees with disabilities and the research model

2.1 Specific characteristics of less-educated employees with disabilities

Workers with disabilities are a vulnerable group of employees in the labour market. They encounter many difficulties finding and keeping jobs [7]. In addition, the share of employees who have disabilities lags behind the share of employment of individuals who do not have a disability [8], with some stating that it is declining even further [9]. The share of persons employed in different time periods is negatively linked to the degree of disability in a previous time period, whereas the share of persons not employed is positively linked to the degree of disability in an earlier period. Restrictions on health caused by disability reduce the chances of a person being employed. Yet, even when levels of general disability go down, the share of persons with disabilities who are employed does not rise [10]. If a person with disabilities is also older and less educated (above 50 years old and having at most completed elementary and vocational education levels), the situation in terms of employability, retaining employment, developing work-related competencies, and development opportunities in the work field can become even more problematic. The key factors under the control of employers and other participants which affect the job retention of persons with disabilities are job accommodation, communication among all, healthcare, professional counselling and early intervention [11]. Conversely, the most important activities for encouraging job retention and employability in this group of employees include additional training and education and employment-related changes or re-employment. The latter is particularly important in relation to the concept of decent work that has been shown to be able to increase employees’ work motivation [12]. This is especially valuable considering that research shows people with disabilities have a harder time finding and performing suitable work since the unemployment rate is higher among people with disabilities than among with people who do not have a disability [13].

Some studies report that the most frequent features or variables like age, gender, race, intelligence quotient, diagnosis and subsequent diagnosis are unrelated to the successful outcome or even completion of education and training [14], making it safe to assume that some other factors are important for ensuring training has a successful outcome. These include family circumstances, environmental expectations and motivation. On the other hand, it has been established that motivation of employees with disabilities to find and maintain their employment is one of the key factors, in relation to employment and retaining employment [15]. A similar logic also applies to employment-related changes. Motivation is one of the vital concepts of the research model and in adapting the questionnaire discussed in this article.

2.2 Research model

Apart from motivation for education and training / re-employment several key concepts are addressed in the research model and questionnaire (Fig. 1):

the perception of job security;

the match between competencies actually acquired and those required in the workplace;

expected effects of education and training/employment changes or re-employment;

health status of the worker and the disability;

physical intensity of work; and

possibility of career development.

Fig. 1

The initial research model of the concepts included in the questionnaire.

The questionnaire items are based on previously described concepts. Identification and adaptation of the items relate to several earlier surveys. No study has so far encompassed all of the mentioned topics in terms of the motivation of (less-educated and older) employees with disabilities to undertake employment-related changes on one hand and training and education on the other. The outcome is that no questionnaire adapted for the population under study currently exists for this purpose.

In the research model, it is expected that a greater perception of job security negatively affects one’s motivation for education and training/career-related changes. As a result, a job with high security will not make an employee see a need to change the current situation because the need (for job security) has already been achieved and is no longer a motivating factor [16].

Based on Vroom’s theory, the model also expects that stronger positive expected effects of one action (education and training/career-related changes) positively impact the motivation to engage in them [17].

Accordingly, it is also expected that work which is more physically intense will positively impact the motivation for education and training/re-employment. The higher the physical intensity of the current working conditions for a worker with disabilities, the more they will be motivated to take steps that may improve the situation.

The research model also assumes that a higher extent of disability affects the motivation for education and training/re-employment. A higher disability category implies greater impairments and potential difficulties with normal work in the current job compared to employees who have no disabilities. Based on similar arguments of expecting positive effects of one action on the outcome of another situation (e.g. a job better suiting their health status) [17], it is also expected that a worse health condition will improve motivation for education and training/re-employment.

The model also anticipates that a situation where an employee has a surplus of competencies over those actually required for the job will affect motivation for education and training/re-employment. Certain research shows that a competency deficit prompts an individual to eliminate that deficit [18]. Vice versa, a surplus of competencies gives a worker information that their knowledge is sufficient for the present work; therefore, it is expected they will be less motivated for additional education and training. On the other hand, a surplus of competencies also informs the employee of the potential that they can do even more, and more challenging work. Hence, they are expected to be more strongly motivated for work-related changes.

2.3 Approach to adapting the questionnaire and sampling

The initial step was a systematic review of literature published as scientific articles. To help prepare the initial version of the questionnaire, a methodical review of studies published on the subject of concern was followed by an overview of reference research on how already tested questionnaires and concept variables have been used. Based on the theory and published research, where possible, the operationalisation already in use was included in the first version of the questionnaire with some minor adaptations. Due to the research’s complexity, the model included concepts regarding the need to reduce several items (e.g. list of competencies, organisational characteristics etc.). The systematic review of the literature included the standard overview of the concepts, while the following steps are described below.

2.3.1 Sampling

Research is crucial and must be done due to the sensitiveness of the target group of employees with disabilities in the labour market. Such employees in Slovenia (where the survey was conducted) constitute a group of employees who are on average 71.4% long-term unemployed, making it very important to study ways to maintain and improve employment opportunities for people with disabilities [19].

The research relied on a case study of a large Slovenian company in which a significant share of the target population (workers with disabilities) is less educated due to the nature and content of the work (lower skilled jobs); 48.7% of workers with disabilities in the company have only an incomplete primary or primary education. At the same time, their average age is high (52.83 years, median 54.00 years, modus 57.00 years).

2.3.2 Analysis of company personnel data

We analysed anonymised employee data provided by the company whose employees with disabilities were to participate in the survey. The participating company’s management gave its consent to participate in all stages of the survey, including the pilot survey. The final decision as to whether an individual employee would participate in any stage of the research was left up to the individual and was not mandatory. The chief goal of analysing anonymised employee data was to obtain a detailed insight into the target group’s characteristics. The purpose of this research phase was to become systematically familiar with the features of that group. On this basis, we prepared the starting points for possible adaptation needs.

Another purpose was a later comparison with the realised sample. In addition to becoming acquainted with the target group’s characteristics, we confirmed the presumption that, due to the age and education structure, a comprehensive assessment of the initial questionnaire’s relevance would be required.

2.3.3 Expert evaluations

Three expert evaluations were made. One before the first pilot test and two after each pilot test. Initial questionnaire version, results and observations of the pilot tests were discussed. Based on expert evaluations adjustments were undertaken.

2.3.4 First pilot test

The first pilot test was conducted after the expert evaluation. The goal of this step was to test the questionnaire’s adequacy and prepare for the following steps.

Five people with disabilities were surveyed in the first pilot test. Employees were selected by the employer in whose company the survey was to be carried out after finalising the form of the questionnaire. The test participants were aged 48 years or older and held elementary and vocational education. All were cooperative while participating in the pilot testing, although this might be due to the selection of employees made by the employer.

2.3.5 Second pilot test

Seven employees with disabilities were surveyed in the second pilot. Test participants held an elementary and vocational education. Once again, the surveyed workers who cooperatively participated in the pilot testing showed that participation did not entail a refusal to be involved in the research as such, but arose from the specific circumstances of the target group of employees (e.g. the existence of a disability and a lower education).

In both pilot tests, the survey proceeded without interruptions by the employer, the work process, or any other disruption. Workers were encouraged to ask questions when answering the questionnaire if anything was unclear in terms of content or any other aspect of the questions (layout, form). There were not many self-initiated questions.

Further, in both pilot tests, the corresponding author discussed the process of responding to survey questions with the respondents. This was done to gather valuable information regarding the validity and relevance of individual questions and the questionnaire as a whole. By using the technique, checking the interpretations of the subjects studied [20] and internal evaluations [21], we sought to ensure the greatest possible level of validity of the questionnaire and of the survey results.

3 Methodological particularities of our sample in terms of questionnaire adaptation

In this section, we discuss those methodological particularities we believe are crucial for implementing the survey and adapting the questionnaire to older and less educated employees with disabilities: the resistance of employees with disabilities to participate in the survey due to their low literacy skills, accessibility, anonymity, and the sensitiveness of the topic. The sensitiveness and complexity of the topic have also become more important upon adoption of the General Data Protection Regulation (“GDPR”) adopted by the European Union, although this aspect will not be dealt with in this paper.

3.1 Resistance of employees with disabilities to participate in the survey due to their low literacy skills

Employees with disabilities who are in addition less educated and older (in line with activities as part of national policies in Slovenia to promote the employment of older workers, we define older workers as those aged 50 years or older) constitute a group of employees that is hard to include in any survey for various reason. Employees with disabilities are reluctant to complete paperwork that requires a certain level of literacy skills. Significant impairments in literacy skills can hinder the taking of written tests, questionnaires or surveys. Some research shows that participants with impairments, such as low literacy skills, needed an average of 7.5 reminder interventions to complete a traditional written test and 2.2 to complete the online version of the test [22]. Based on our researcher’s report, it was apparent that employees with disabilities were not used to participating in research, with some feeling uncomfortable when thinking about the true reasons for the research. It became clear in the pilot tests that the respondents did not proactively seek additional information when they did not understand something in the questionnaire, even if a problem or dilemma had arisen. Therefore, conducting the pilot surveying in smaller groups proved useful by allowing us to give enough attention to each participant.

3.2 Resistance of workers with disabilities to participate in the survey due to topic sensitivity

A subjective feeling by a respondent when answering questions on a sensitive topic can influence their answers. In particular, feeling uncomfortable can potentially have a negative effect [23]. A respondent might decide to not only not answer a question, but also to cease their survey participation due to a topic relating to the person’s condition [24] and the purpose of the research.

Another issue authors raise in relation to sensitive topics is the quality of responses [25] due to respondents’ social desirability concerns [26]. This also showcases the importance of providing anonymity to research participants. Indeed, studies show that when sensitive questions are involved, respondents give lower estimates of socially unacceptable behaviour and higher grades than actual ones for socially acceptable behaviour [27]. Studies also show the quality (honesty) of answers can be influenced by the method of performing the survey [28], with various methods allowing different levels of anonymity.

Workers with disabilities’ resistance to survey participation due to their low literacy skills and low self-esteem overlap at some stage. Yet, while the former may be problematic in conjunction with the possibility of unintentional errors (e.g. due to misunderstanding), the latter may be due to being associated with intentional mistakes (e.g. intentionally answering inaccurately in order to make a better impression), which requires a different intervention approach. These two aspects are discussed in separate subsections.

3.3 Accessibility: The company and workers

The challenge of surveying workers with disabilities is multidimensional. A precondition for identifying the target group is to possess information about the existence of a disability, something that is inherently personal and thus both protected by legislation and difficult to access. Data show more than 15% of all citizens in the European Union and around 12% – 13% of citizens in Slovenia have a disability [29]. Nevertheless, the problem lies in the dispersal of the target group of employees with disabilities, despite its magnitude.

This correlates with the need to make specific conceptual and methodological adjustments [30] as well as the cost of conducting research that addresses more hard-to-access populations [31]. It is sometimes a problem to even estimate the incidence or size of a specific population a sample of which we wish to include in a survey [32]. Researchers have tried a variety of ways to solve the problem of certain populations’ accessibility, including employees with disabilities. For example, using various methods of identification and sampling [33 –35]. In our case, the key issue was accessing employees with disabilities due to their dispersed locations. Given the company’s willingness to participate in the survey, the sampling was not an issue in our case. On the other hand, we had expected several problems in connection with physical access to employees located across Slovenia who work in shifts.

3.4 Other general methodological issues regarding the surveyed population

3.4.1 Self-assessment as a main data source

Self-assessment appears to be a useful and practical approach, often employed for the purpose of assessing competencies (36, 37), but also in connection with education [38] and the need for additional training and education [39]. Some authors argue that enlisting self-assessment (e.g. the individual level of competencies) for use in research is not the optimal method for evaluating competencies due to possible bias. Still, other authors state the opposite [40].

Possible bias in self-assessment, namely, a discrepancy between the true and the reported value, may occur intentionally or unintentionally. The cause of a deliberate error may be the lack of anonymity or the topic’s sensitiveness (fear of embarrassment), for instance [38]. Conversely, an unintentional error may occur due to an unclear question or a misunderstanding by the respondent. Both cases were discussed in connection with other particular characteristics of the target group of employees.

Managing intentional and unintentional errors is very important when it comes to competencies, which are often described in complex and abstract ways [40]. The divergence or coincidence of actual and self-assessed rates may be influenced by self-esteem and the difficulty of a specific task [41].

3.4.2 Questionnaire length

Research results concerning the impact of the length of a questionnaire on the unit response rate indicate that questionnaire length can influence the response rate or the proportion of surveys that are completed [42]. Yet findings on the impact of questionnaire length on the response rate are mixed, with some finding the length does not affect the level of response [43]. In our case, questionnaire length was problematic. The research model’s complexity was the reason for the questionnaire’s quite extensive initial version. However, the first pilot test revealed an issue of questionnaires not being completed in their entirety, especially in segments requiring greater cognitive engagement (for example, in differentiating seemingly similar questions or on open questions).

3.4.3 Acquiescence effect, wording, response scales and polarity of questions

Due to the questionnaire’s possible cognitive burden on the respondents, special emphasis must be paid to the acquiescence or halo effect; namely, when the answer to one question influences the answers to other questions which follow [44]. This sometimes means that respondents detect the principle according to which the questions were composed, and they do not read each sub-question, but only use the same answer or answering pattern.

The complexity and length of a question must also be considered regarding response quality. Question wording can influence responses [45]. Research suggests that questions should be as short as possible, within constraints defined by the survey objectives and emphasising simplicity [46]. Adequate wording and response scales must be ensured [47]. The purpose of simplification of wording may be to make the instrument more practically useful [48] or to adapt it to a specific target group [49]. Some studies reveal that simplifications do not affect the level of validity and reliability of measurement scales or questionnaires [4, 50]. Another issue arising from wording is the polarity of questions (or response scales) and the context of a question, especially where questions seem to be similar but hold a different meaning (e.g. the employee’s actual competencies vs. competencies required for the job). Those higher-level questions require explicit wording and instructions [51] to be clear, straightforward and as simple as possible.

3.4.4 Question order

Although some questionnaire changes like altered wording (as discussed previously) or the order of questions at first seem irrelevant, they may alter the response of respondents such that it does not necessarily reflect the actual difference in value [52]. However, in the case of question/item order, some research implies that item-order effects are possible, but less important than the colour of the paper used (e.g. in exams) [53]. That may depend on the topic or the content of the questions, the individual’s characteristics [45] as well as the survey mode [54].

In our case, the problem of question order arose in connection with a possible influence on the item-response rate due to the questionnaire length. The questionnaire was five pages long in its final version. It was crucial to consider for which questions we could afford to have the lowest item response and for which ones we needed the highest possible response level, if at a certain stage a respondent stopped answering the questionnaire and left it unfinished.

4 Proposed approach

In this section, we discuss the main adaptations arising from the pilot survey results in connection with the particular characteristics of our sample and methodological solutions. The questionnaire adaptation entailed several phases (see section 3). The argument supporting the adjustments and changes made to the questionnaire will not strictly follow the timeline of the phases, but the adjustment issues emphasised in the above section. It should also be mentioned that questionnaire adaptations were made in all of the phases, but most were made after the pilot testing and the expert evaluations that followed. Research [55] suggests that critical immersion, accountability and methodological flexibility in practices are effective for increasing readiness to participate and boosting the inclusion of under-studied populations in research.

4.1 Motivating the cooperation of the organisation and the supervisors

Due to the complexity and sensitiveness of both the topic and the target group of employees, a special approach is needed. Considering the locational issues, it was also very important to motivate the company’s leadership to actively participate in implementing the survey (as discussed in below).

A constructive and transparent approach to a company was very useful in terms of clarifying the planned performance of the research and its content, especially its potential benefits for the company. In this way, the researcher is able to gain the trust of the company that has agreed to participate in the research. In this respect, the important time component involves the least possible delays and procrastination.

4.2 How to motivate employees to participate (incl. anonymity and sensitiveness)

Regarding anonymity and the sensitiveness of the topic, a dilemma may arise in questions in which respondents might assume that different responses are more or less desirable from the employer’s point of view. For example, questions about how the employees assess the working conditions, motivation for individual performance etc. Research shows that socially desirable behaviour increases while answering under time pressure [56]. As a result, we ensured sufficient time was given to each respondent to answer all of the survey questions. Prosociality relates to providing anonymity due to the possibility that an employee may want to answer certain questions in a manner they consider more acceptable for the employer. The literature suggests that social desirability bias is influenced by embarrassment and satisficing in both face-to-face and self-completion modes and varies according to the chosen surveying mode. In the face-to-face mode, there is likely to be less social desirability bias due to satisficing but more social desirability bias due to embarrassment in comparison with self-completion modes [57]. In our research, another issue arises with regard to anonymity and sensitiveness. This was the problem of the spatial dispersion of jobs and, consequently, of employees with disabilities. Work locations are found throughout the country (Slovenia), employees work in shifts and there is much long-term sick leave, especially among the oldest and least educated employees with disabilities. This was, in connection with the wishes of the organisation, managed by delivering the questionnaires to employees through their direct superiors. This could be a problem because an employee may feel obliged to complete it, even if they do not want to, due to fear of being sanctioned. By delivering questionnaires to workers through their direct superiors, doubt may also arise regarding the anonymity of the answers (and thus once again the possibility of abuse), which may make them not answer totally sincerely. Another finding, for example, showed that respondents will more positively evaluate an advertisement, attractiveness and people from a country if interviewed by a person from that country in contrast to being interviewed by a person who does not come from the target country [58]. A similar result may be expected if a worker is asked about a potentially sensitive matter by an employer representative. As a result, we did not choose the face-to-face approach but instead opted for a self-administered approach. It was therefore important that, as far as possible, the research’s purpose was clearly explained in a way that is appropriate for the company, leaders and employees. A precondition for ensuring quality implementation and neutralising possible negative effects of the superiors delivering the questionnaire was to adequately prepare and inform those who were handing the questionnaires out to the employees. This included informing them about the purpose and meaning of the research along with the questionnaire content (concerning explanations given to respondents) and preparing them on how to correctly approach workers to encourage them to participate. Contact details of the corresponding author were given to the employees who were invited to make contact regarding any issue or question that may arise. Even though the organisation agreed to cooperate, each individual participant also had to give their consent for cooperation. To maximise the response rate, we followed the recommendations of other authors [59] about follow-up activities and strategies that are realistic given the available budget by persuading many of the initial non-respondents to return the questionnaires.

A special approach is required when the literacy of the target group is in issue. Although some research suggests using a self-directed, multimedia software approach for individuals with intellectual disabilities or literacy challenges [22] to ensure a greater response and more completed questionnaires, we do not believe it is appropriate for our target group due to their lower computer access. The traditional pen-and-pencil approach with additional adjustments is more practical because most of the surveyed workers do not use a computer or a smartphone in their work. We also did not have access to private contact information (via which we could have the respondents use personal devices to participate in the survey). Accordingly, we chose the option of having a physically present person (superior, HR expert or corresponding author) available to the workers for any questions and clarifications during the surveying. In addition, considering the pilot test results, the respondents were more inclined to self-administering than being questioned by an interviewer. They justified this by saying they were more relaxed while answering without supervision or being questioned by an interviewer.

4.3 General methodological issues in the context of surveying employees with disabilities

As mentioned, it was decided that the survey would use the self-assessment approach as the main data source. This is due to the participants’ spatial dispersion and the aim of providing the greatest anonymity possible since we are trying to survey a group of workers that is reluctant to participate in research and have reservations about it, while also minimising any overrating which may occur if respondents feel their anonymity is not ensured given that they are to be surveyed by an employer representative. This aims to reduce possible intentional errors. Conversely, the researcher must also seek to lower the incidence of unintended errors through clear and straightforward question wording, clear rating scales and ensuring access to the corresponding author should any additional information and clarifications be needed. Those options are discussed in other sections of the article.

4.4 Solving acquiescence effect

Accomplishing the acquiescence effect is possible using a split-ballot procedure by establishing positively- and negatively-worded questions [60]. When some statements are turned into positive and others into negative forms, then the positive and negative statements are mixed [61]. The aim is to encourage a respondent to read each and every question/sub-question. In this case, the acquiescence effect may still occur, but with fewer respondents. Researchers have also investigated whether the acquiescence effect would be reduced by giving respondents an explicit chance to say “Don’t know”, although there is little evidence to show this would reduce this particular response effect [62]. Another solution dealing with reducing the questionnaire’s cognitive intensiveness relates to simplification, clear wording and polarity of questions, as discussed below. This includes grouping and refining sets of sub-questions in question blocks to shorten the questionnaire and avoid the effect of agreement.

In our case, the acquiescence effect emerged as a possible problem in the second pilot testing when it seemed that problems may arise in answering the (longer) blocks of questions all with the same score (e.g. all 5 or all 1). Another issue referred to questions with a larger set of sub-questions, which were substantively different, but seemed similar in appearance. These questions were time-consuming and cognitively challenging because respondents did not know what to answer and had asked if the questions had mistakenly been included twice. Turning some statements into their negative forms and others into positive, and then mixing them to encourage a respondent to read every question/sub-question, proved impossible. The first reason is that interventions like extra changes made to the wording due to the particularities of the target population would further exacerbate the questionnaire’s cognitive complexity and thus did not appear to be meaningful by increasing the chances of greater bias in answering. Two of the company’s human resources specialists who had worked with the target population for at least 15 years also advised this. To our knowledge, this is inconsistent with other research. However, due to their in-depth knowledge of the target group of older and less educated employees with disabilities and in relation to the form of the questions, we took their suggestion into account. Another reason was that no claims were made in several questions/sub-questions, only the items. For example, one question read “How do you evaluate the characteristics below in your current job with the company” with examples given below like “working conditions, physical complexity of work, promotion options, job security and opportunity for additional education and training”.

In those cases, shorter sets of sub-questions and emphasis on the context in terms of polarisation seem possible solutions. Emphasising the context to which a question relates appears to be a simple, practical and successful approach that may reduce the acquiescence effect. This can be achieved by using capital letters in order to help workers/respondents avoid the feeling that they have answered the same questions twice. For some questions, we used capital letters for some words to stress specific aspects of a question.

Table 1
Example of using capital letters to emphasise the difference between two questions

Questions in the 1st example:

How do you evaluate the characteristics listed below in your current job with the company?

How important or unimportant are the characteristics listed below for you?

Characteristics (same in both cases): E.g. working conditions, physical complexity of work, promotion options, job security and opportunity for additional education and training

Questions in the 2nd example:

Below is a list of some skills and competencies. Please evaluate the level of development of individual skill and competence in your case?

Please indicate the expected level of the development of skills and competencies listed below FOR your current work in the company.

Skills and competencies (same in both cases): expertise in your field, independent work, working in a team, quick learning of new work tasks, effective working under pressure, clear expression of ideas and suggestions, a confident approach and collaboration with others, other knowledge, skills and abilities

Questions in the 1st example:
How do you evaluate the characteristics listed below in your current job with the company?
How important or unimportant are the characteristics listed below for you?
Characteristics (same in both cases): E.g. working conditions, physical complexity of work, promotion options, job security and opportunity for additional education and training
Questions in the 2nd example:
Below is a list of some skills and competencies. Please evaluate the level of development of individual skill and competence in your case?
Please indicate the expected level of the development of skills and competencies listed below FOR your current work in the company.
Skills and competencies (same in both cases): expertise in your field, independent work, working in a team, quick learning of new work tasks, effective working under pressure, clear expression of ideas and suggestions, a confident approach and collaboration with others, other knowledge, skills and abilities

4.5 Length of the questionnaire

Questionnaire length was another challenging issue due to the characteristics of our sample. The biggest problem in the two pilot testing rounds completed by less-educated, older employees with disabilities was the length of the questionnaire and the number of pages. Concentration dropped in the last third of the questionnaire. This occurred despite the fact that during the first and second pilot tests we had already shortened the questionnaire wherever possible, without obstructing the content in terms of meeting the needs of the research model/concepts.

For example, in the case of employee health status, in addition to the disability status question and the general health assessment question included in the questionnaire’s final form, we initially considered including two extra questions. However, the test survey showed it was reasonable to exclude them because the respondents did not understand them well and they were also irrelevant from the perspective of the initial research model. One of them was a question taken from the questionnaire “Analysis of the state of psychosocial risks in jobs in micro, small and medium-sized enterprises” [63] on how much employees had been absent in the last year due to sick leave. This question was excluded because at the time of the pilot testing it appeared that the respondents had difficulties answering or assessing the extent of absences within the specified time interval. The second question inquired about the current phase of the disability assessment procedure of an individual employee’s case before a disability commission. The question is interesting and would be valuable for improving work in practice by ensuring an appropriate approach to maintaining productive employment at the most suitable stage of the procedure. Still, it was not relevant to the research model. Moreover, it was impossible to capture an adequate number of respondents in each subgroup/category of phases of the disability assessment procedure to guarantee a good basis for further analysis. At the same time, the question arose as to whether the workers were even aware of the actual phase in the procedure they were currently in since the second phase of the pilot testing revealed that some workers had already become stuck while trying to determine their disability category. This is despite the fact that the category of disability is clearly stated in the decision of the disability commission.

We advise it is necessary to exclude unnecessary questions which could prolong the questionnaire or make it more cognitively challenging without adding true value for the research. In this vein, it is also useful to exclude some sub-questions from the block and merge sub-questions wherever possible. In this way, the researcher reduces fragmentation and helps the respondents avoid the feeling that they have repeatedly answered the same issue. Another goal is to shorten the questionnaire and make it more transparent and simpler to comprehend. The last two can also be achieved by merging some initially separate questions into a shorter block of questions and by excluding unnecessary, open questions. In our case, for example, open questions or questions that required descriptive answers proved to be cognitively challenging for the test respondents. Among other things, this was because they felt they had to write something, even though they had nothing to write. The result of that longer distraction was a rush to complete the following questions; accordingly, we excluded all unnecessary open questions.

On the other hand, we emphasise that shortening is only welcome in terms of efficiency. For instance, in the final questionnaire version we had the possibility to arrange all questions over 5 pages or 4 pages. In the second case, the questionnaire would be 1 page shorter, but would give the impression of greater fatigue and opacity. Considering the findings of pilot tests, we decided on a questionnaire with 5 pages.

4.5.1 Question order, wording and response scales

In both pilot tests, completing the questionnaire during the pilot testing lasted on average 30–35 minutes, where concentration dropped in the last third of the questionnaire. This was especially apparent in the first pilot test when the questionnaire was also the longest. However, when any further shortening of the questionnaire was impossible, a need to change the question order was revealed. That is, from the substantively more important ones according to the research model, i.e. motivation for training, education and employment-related changes, through to the least important ones and test variables (e.g. demography). The purpose of changing the order of the questions is to capture responses to all or most of the key variables or indicators for which the researcher needs the highest response rate possible.

The research revealed significant effects of the response scales on the responses, depending on the number categories available, as to whether they were balanced or not [64] and their format (e.g. Likert format) [65]. For less-educated and older employees with disabilities, it is necessary to pay special attention to formulating a balanced polarity of response scales with the same wording of response categories in several questions wherever possible, depending on the question content. This is one way of reducing the cognitive complexity of a questionnaire.

5 Conclusion

In surveying the less-educated population with disabilities, proper adaptation is an important goal. In the article, we demonstrated how to adapt an (extensive) questionnaire on a complex and sensitive topic for surveying older and less-educated employees with disabilities. The strategic agenda for surveying older and less-educated employees with disabilities should address the following questions:

How to motivate the organisation and the supervisors to cooperate in the survey and how to convince them of the benefits of the results for the organisation?

How to ensure anonymity and address issues of sensitivity in order to improve employees’ motivation to participate?

How to adjust general methodological issues such as self-assessment, questionnaire length, acquiescence effect, wording, response scales, polarity of questions, and question order in the context of surveying employees with disabilities?

Although some of the limitations of the study were listed along the way, we must also point out the major limitations here. One big limitation is that almost no studies in Slovenian and foreign scientific literature address the adaptation of a measurement instrument to such a narrowly defined target group, one that is difficult to directly include in research even before including additional factors (like lower education and higher age). This means that not only workers with disabilities but the company itself are not used to including a group of workers like this in research. Most research in the work environment considers the general population. As a result, little specific guidance could be found in the literature on how to persuade a company (and then the target group of employees) to adopt the most collaborative approach possible to conducting the research.

Our findings therefore rely on a combination of results from previous research where possible and, to a large extent, are the result of experience gained in several phases of questionnaire testing (as presented in the article), considering the company’s suggestions (stemming from decades of practice of employing a considerable number of employees with disabilities) and conducting the surveying.

Since only a small share of studies has sought to provide insight into the need to adapt measurement instruments and their implementation for such a narrowly defined target group, comparisons of findings with other similar target groups cannot be avoided, yet are not necessarily optimal. For example, the findings that apply in the case of adapting a measurement instrument to different cultures do not necessarily hold true in the case of lower educated and older workers with disabilities. Although the group of lower educated and older employees with disabilities seems narrowly defined, in practice it constitutes a very heterogeneous group of employees. Differences arise from the type or extent of disability and from specific factors that are hard to grasp without in-depth knowledge of a specific situation in a company. In our research, this particular situation is the fact that older age is not only important because of the age itself, but for what it might mean in a certain social situation. As concerns our research, a higher age in simple jobs, which account for a large share of employees with disabilities, may for instance also mean that these are workers who have moved from other countries for work and are at the start of their career path. They were employed in jobs that did not require a college degree or where the nature of the work did not call for much client interaction, with the outcome that, even after decades of living and working in the new country, they were still deficient in understanding and using the language of the country to which they have moved. This is an aspect of a particular organisational situation and hard to capture in the research preparation stage. In part, we tried to resolve this problem by using the most clearly understood terminology given that the need for this was partly anticipated due to the workers’ lower education and later confirmed while testing the questionnaire.

We believe that greater research effort is needed in the future to focus on approaches to study this under-researched target group. Only through constant practice will it be possible to consider the widest possible diversity of its members and identify the main way for organisations and workers with disabilities to become used to participating in research.

Conflict of interest

None to report.

References

Thitasan

, Velandia

, Howharn

, et al. Methodological Challenges in Developing a Youth Questionnaire, Life & Health Young People, for Comparative Studies in Thailand and Sweden: About Bridging the Language Gap Between Two Non-English-Speaking Countries. J Transcult Nurs . 2017;28:582–9.

Chowdhury

, Aman

, Lecavalier

, et al. Factor structure and psychometric properties of the revised Home Situations Questionnaire for autism spectrum disorder: The Home Situations Questionnaire-Autism Spectrum Disorder. Autism . 2016;20:528–37.

Spangenberg

, Schroder

, Duvenage

. A Leadership Competence Utilization Questionnaire for South African Managers: South African Journal of Psychology. Epub ahead of print 2 August 2016. DOI: 10.1177/008124639902900303.

Guan

, Seng

, Hway Ann

, et al. Factorial Validity and Reliability of the Malaysian Simplified Chinese Version of Multidimensional Scale of Perceived Social Support (MSPSS-SCV) Among a Group of University Students. Asia Pac J Public Health . 2015;27:225–31.

Akcil

. The use of mobile learning for visually impaired learners school in tolerance education contents. Qual Quant. 2017;1-14.

Post

MWM

, Gerritsen

, Leusen

NDM van

, et al. Adapting the Nottingham Health Profile for use in people with severe physical disabilities: Clinical Rehabilitation. Epub ahead of print 1 July 2016. DOI: 10.1191/026921501672698006.

Rita

Newton

, Marcus

Ormerod

, Pam

Thomas

. Disabled people’s experiences in the workplace environment in England. Equal Opportunities Int . 2007;26:610–23.

Mitchell

Loeb

, Jennifer

Madans

, Julie

Weeks

, et al. Disability and participation: assessing employment and education outcomes in the National Health Interview Survey (2010). In: Disability and Intersecting Statuses. Emerald Group Publishing Limited, pp. 171-88.

Dykema-Engblade

Amanda

, Stawiski

Sarah

. Employment and retirement concerns for persons with developmental disabilities. In: Autism and Developmental Disabilities: Current Practices and Issues. Emerald Group Publishing Limited, pp. 253-72.

10.

Tur-Sinai

. Labor-market mobility among persons with disabilities. Work . 2019;64:323–40.

11.

Lefever

, Decuman

, Perl

,et al.. The efficacy and efficiency of Disability Management in job-retention and job-reintegration. A systematic review. Work . 2018;59:501–34.

12.

Ferraro

, Moreira

, dos Santos

,et al.. Decent work, work motivation and psychological capital: An empirical research. Work . 2018;60:339–54.

13.

Zoer

, de Graaf

, Kuijer

PPFM

,et al.. Matching work capacities and demands at job placement in employees with disabilities. Work . 2012;42:205–14.

14.

Levy

, Murphy

, Levy

,et al.. Monitoring process and employment outcomes over time: The YAI employment project. J Dev Phys Disabil . 1993;5:167–79.

15.

Rose

, Saunders

, Hensel

,et al.. Factors affecting the likelihood that people with intellectual disabilities will gain employment. Journal of Intellectual Disabilities . 2005;9:9–23.

16.

Maslow

. Motivacija i ličnost. Beograd: NOLIT, 1982.

17.

Vroom

. Work and Motivation. John Wiley & Sons, 1967.

18.

Elliot

, Dweck

. Competence and Motivation: Competence as the Core of Achievement Motivation. In: Elliot

, Dweck

(eds) Handbook of Competence and Motivation. Guilford Press, 2005, pp. 3–12.

19.

Employment of disabled persons in the Republic of Slovenia. Ljubljana: Court of audit RS, 2009.

20.

Kogovšek

. Zanesljivost in veljavnost v kvalitativnem in kvantitativnem raziskovanju. Teorija in praksa . 2005;42:256–78.

21.

Ferligoj

, Leskošek

, Kogovšek

. Zanesljivost in veljavnost merjenja. Fakulteta za družbene vede, 1995.

22.

Stock

, Davies

, Wehmeyer

. Internet-Based Multimedia Tests and Surveys for Individuals with Intellectual Disabilities: Journal of Special Education Technology. 2004;19:43–7.

23.

Fendrich

, Lippert

, Johnson

. Respondent Reactions to Sensitive Questions. Journal of Empirical Research on Human Research Ethics . 2007;2:31–7.

24.

Tourangeau

, Yan

. Sensitive questions in surveys. Psychological Bulletin . 2007;133:859–83.

25.

Castro

. Inconsistent Respondents and Sensitive Questions. Field Methods . 2013;25:283–98.

26.

Näher

A-F

, Krumpal

. Asking sensitive questions: the impact of forgiving wording and question context on social desirability bias. Qual Quant . 2012;46:1601–16.

27.

Krumpal

. Determinants of social desirability bias in sensitive surveys: a literature review. Qual Quant . 2013;47:2025–47.

28.

Wolter

, Preisendörfer

. Asking Sensitive Questions: An Evaluation of the Randomized Response Technique Versus Direct Questioning Using Individual Validation Data. Sociological Methods & Research. 2013;42:321–53.

29.

Action Program for Invalids 2014–2021. Ljubljana: MDD Sž, 2014.

30.

Ruspini

. The Contribution of Longitudinal Research to the Study of Women’s Poverty. Quality & Quantity. 1999;33:323–38.

31.

Bertram

. A Pilot Study To Estimate the Cost of a Telephone Survey of Rare Populations: African Americans and Persons with Angina. Evaluation Review. Epub ahead of print 26 July 2016. DOI: 10.1177/0193841X9401800604.

32.

Lacy

. Efficiently Studying Rare Events: Case-Control Methods for Sociologists: Sociological Perspectives. Epub ahead of print 3 August 2016. DOI: 10.2307/1389496.

33.

Nunan

, Knox

. Can Search Engine Advertising Help Access Rare Samples? International Journal of Market Research. Epub ahead of print 1 July 2011. DOI: 10.2501/IJMR-53-4-523-540.

34.

Spreen

. Rare Populations, Hidden Populations, and Link-Tracing Designs: What and Why?. Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique. 1992;36:34–58.

35.

Terhanian

, Bremer

. A Smarter Way to Select Respondents for Surveys? International Journal of Market Research. Epub ahead of print 1 November 2012. DOI: 10.2501/IJMR-54-6-751-780.

36.

Cowan

, Wilson-Barnett

, Norman

. A European survey of general nurses’ self assessment of competence. Nurse Education Today . 2007;27:452–8.

37.

Darling

, Gallagher

. Using Self-Assessments in Early Intervention Training: Journal of Early Intervention. Epub ahead of print 25 July 2016. DOI: 10.1177/105381510302500306.

38.

Stufflebeam

, Wingate

. A Self-Assessment Procedure for Use in Evaluation Training: American Journal of Evaluation. Epub ahead of print 30 June 2016. DOI: 10.1177/1098214005279730.

39.

Bromley

, Boran

, Myddelton

. Investigating the baseline skills of research students using a competency-based self-assessment method: Active Learning in Higher Education. Epub ahead of print 24 July 2016. DOI: 10.1177/1469787407077992.

40.

Allen

, Van der Velden

. The role of self-assessment in measuring skills. Research Centre for Education and the Labour Market, http://roa.sbe.maastrichtuniversity.nl/roanew/wp-content/uploads/2014/02/REFLEX_Working-paper02_Role_of_Self_Assessment_of_Skills_28april2005.pdf (2005, accessed 15 May 2018).

41.

Kim

Y-H

, Kwon

, Lee

,et al.. Why Do People Overestimate or Underestimate Their Abilities? A Cross-Culturally Valid Model of Cognitive and Motivational Processes in Self-Assessment Biases. Journal of Cross-Cultural Psychology . 2016;47:1201–16.

42.

Iglesias

, Torgerson

. Does Length of Questionnaire Matter? A Randomised Trial of Response Rates to a Mailed Questionnaire. Journal of Health Services Research & Policy. 2000;5:219–21.

43.

Rudd

, Maxwell

. Mail Survey Response Rates: Effects of Questionnaire Topic and Length and Recipients’ Community. Psychological Reports . 1980;46:435–40.

44.

Calsyn

, Klinkenberg

. Response Bias in Needs Assessment Studies. Evaluation Review . 1995;19:217–25.

45.

Lee

, Schwarz

, Goldstein

. Culture-Sensitive Question Order Effects of Self-Rated Health Between Older Hispanic and Non-Hispanic Adults in the United States. J Aging Health . 2014;26:860–83.

46.

Alwin

, Beattie

. The KISS Principle in Survey Design: Question Length and Data Quality. Sociological Methodology . 2016;46:121–52.

47.

DeCastellarnau

. A classification of response scale characteristics that affect data quality: a literature review. Qual Quant. 2017;1-37.

48.

Bruno

, Close

, Switzer

,et al.. Simplified modified Rankin Scale questionnaire correlates with stroke severity. Clin Rehabil . 2013;27:724–7.

49.

Turró-Garriga

, Garre-Olmo

, López-Pousa

,et al.. Abridged Scale for the Screening Anosognosia in Patients With Dementia. J Geriatr Psychiatry Neurol . 2014;27:220–6.

50.

Javdani

, Finy

, Verona

. Evaluation of the Validity of the Multidimensional Personality Questionnaire–Simplified-Wording Form (MPQ-SF) in Adolescents With Treatment Histories. Assessment . 2014;21:352–62.

51.

Farmer

LSJ

. What Is the Question? IFLA Journal. Epub ahead of print 3 September 2016. DOI: 10.1177/0340035207076408.

52.

Rivers

, Meade

, Lou Fuller

. Examining Question and Context Effects in Organization Survey Data Using Item Response Theory. Organizational Research Methods . 2009;12:529–53.

53.

Tal

, Akers

, Hodge

. Effect of Paper Color and Question Order on Exam Performance: Teaching of Psychology. Epub ahead of print 1 January 2008. DOI: 10.1080/00986280701818482.

54.

Barnes

, Benjamin

, Banahan

III

, Fish

. The Response Effect of Question Order in Computer-Administered Questioning in the Social Sciences/:. Social Science Computer Review . 1995;13:47–53.

55.

Snow

, Hays

, Caliwagan

,et al.. Guiding principles for indigenous research practices. Action Research . 2016;14:357–75.

56.

Protzko

, Zedelius

, Schooler

. Rushing to Appear Virtuous: Time Pressure Increases Socially Desirable Responding. Psychol Sci . 2019;30:1584–91.

57.

Kaminska

, Foulsham

. Eye-tracking Social Desirability Bias. Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique. 2016;130:73–89.

58.

Dahlgren

, Hansen

. I’d rather be nice than honest: An experimental examination of social desirability bias in tourism surveys. Journal of Vacation Marketing . 2015;21:318–25.

59.

Day

, Dunt

, Day

. Maximizing Response to Surveys in Health Program Evaluation At Minimum Cost Using Multiple Methods: Mail, Telephone, and Visit. Eval Rev . 1995;19:436–50.

60.

Heilenman

. Self-assessment of second language ability: the role of response effects: Language Testing. 1990;7:174–201.

61.

Schriesheim

, Hill

. Controlling Acquiescence Response Bias by Item Reversals: The Effect on Questionnaire Validity: Educational and Psychological Measurement. 1981;41:1101–14.

62.

McCLENDON

. Acquiescence and Recency Response-Order Effects in Interview Surveys: Sociological Methods & Research. 1991;20:60–103.

63.

Kralj

, Simčič

. Analiza stanja psihosocialnih tveganj na delovnih mestih v mikro, malih in srednje velikih podjetjih. Koper: Univerza na Primorskem, Znanstveno-raziskovano središče Koper, 2011.

64.

Weiss

. Effects of Unbalanced Response Scales on Judgments of Social Stimuli: Psychological Reports. 1963;12:403–14.

65.

Cambré

, Welkenhuysen-Gybels

, Billiet

. Is It Content or Style? An Evaluation of Two Competitive Measurement Models Applied to a Balanced Set of Ethnocentrism Items: International Journal of Comparative Sociology. 2002;43:1–20.