Abstract
BACKGROUND:
Research consistently shows that Australian employment services are failing those they are intended to serve. Based on findings in other human service areas, a valid and reliable instrument to measure unemployed workers’ experiences may provide an opportunity for improvement in this sector.
OBJECTIVE:
To establish a basis for developing a suitable rating scale.
METHODS:
An exploratory factor analysis combined with qualitative cross check for face validity of an existing large survey of Australian unemployed workers.
RESULTS:
Six factors appear to be important elements of service delivery: (F1) useful and competent, (F2) client-centred, (F3) receptive to feedback, (F4) trustworthy, (F5) fair, and (F6) friendly.
CONCLUSIONS:
While each of these factors have been either described explicitly or referred to implicitly in previous studies, this study is the first to attempt to combine these factors and is a precursor to establishing a valid and reliable rating scale for use by unemployed workers in evaluating their employment service providers. At a time when Australia is exploring new approaches to employment services, such a scale using a robust set of factors may allow for the improvement of employment services and thus be held accountable to a significant stakeholder group whom they aim to serve –unemployed workers.
Introduction
Australia’s delivery of employment services for unemployed workers i . is unique in the OECD in that both mainstream employment and disability employment services are fully contracted to a mix of private and non-profit providers. The government pays providers on a predominantly ‘per-outcome’ basis and awards contracts based on the number of people providers place and keep in work [1]. The model is not without criticism. Qualitative methods such as in-depth interviews and focus groups have been widely used to study unemployed workers’ experiences of these services [2–12], and for the most part, the studies have found that unemployed workers have predominantly unsatisfactory experiences. These findings are not isolated. In 2019, an Australian Government parliamentary inquiry into mainstream employment services, which received 169 submissions and held five public hearings, concluded that employment services were failing those they are intended to serve [13].
Evidence from a diverse range of health and human service settings suggests that systematically understanding the end-user’s experience of a service can contribute to a research base for evaluating and improving service delivery design; for example: medical clinics and hospitals [14–16]; child and adolescent services [17]; and residential care [18, 19]. In many of these settings, there are now well-established consumer experience rating scales in use backed up by research evidence supporting their reliability and validity. Examples from these fields include: SERVQUAL [20], the Global Rating Scale [21] and the Hospital Consumer Assessment of Healthcare Providers and Systems survey [22]. This, however, is not the case regarding employment services.
While some employment service providers use completion surveys, and websites such as google.com and productreview.com.au have consumer reviews of employment services, these are not psychometrically validated and are not designed for quality improvement of the sector. The Department of Employment’s ii Performance Framework requires providers to have “processes in place to ensure that each participant receives a service tailored to meet their individual needs and personal goals” [23]. However, such processes are normally audited only once every two years and an auditor may choose to obtain this information entirely via file reviews, without direct feedback from unemployed workers. What appears to be lacking therefore, is a valid and reliable instrument to measure unemployed workers’ experiences with a direct and ongoing feedback mechanism to providers.
In the absence of a specific instrument, the study reported here is a preliminary step to creating a valid and reliable experience scale. Establishing such a scale at a time when Australia is exploring new approaches to service delivery design [24], may allow for quality improvement of employment services to be evaluated and to be held accountable to a significant stakeholder group whom they aim to serve –unemployed workers. The implications extend internationally as most countries examine new ways for publicly funded employment services to respond to the changing nature of work while remaining focussed on the needs of unemployed workers [25]
It is worth noting that experience is different to satisfaction. While the terms are sometimes used interchangeably, the concepts are distinct, although the nature and direction of the relationship is sometimes debated [16]. We define experience as feedback from consumers on what actually happened, such that this considers both the objective facts and their subjective views of it. It is a measure of process and is comparatively straightforward to assess. Satisfaction, on the other hand is a more difficult construct to define and reflects an interplay between expectations, experiences, outcomes, and trust in the underlying system, and is predominantly used as an outcome measure [26]. The use of satisfaction measures requires some caution, as changes in the level of satisfaction may be due to changes not just in the quality of the service, but also because of changes in the broader context that drive changes in expectations, trust and outcomes and which are outside the control of the provider [26]. While for the most part, the study reported here focusses on experiences, we acknowledge that the line between measuring experience and measuring satisfaction is sometimes blurred.
Methods
Design
The study used an exploratory factor analysis (EFA) of a pre-existing, Australian Unemployed Workers’ Union (AUWU) membership survey, which sought to gather feedback on unemployed workers’ experiences of Australian Employment Services. The AUWU (unemployedworkersunion.com) represents unemployed, underemployed, and unwaged workers, as well as all recipients of Social Security in Australia. The AUWU is a national organisation, with divisions and branches operating in every State/Territory in Australia. Membership is free and open to all. It has an occasional consultative role with the Minister for Employment. The analysis was post-hoc, in that the survey was not specifically designed for the purposes of creating a scale, it therefore contains many of the caveats associated with secondary analysis [27]. However, a relatively large sample size (N = 758) gave confidence to undertake at least a preliminary analysis and that the underlying constructs generated would be robust enough to form the basis for a subsequent prospective study.
The survey was designed to gather feedback on experiences of employment services and was structured around core activities such as appointments with the provider, establishing a job plan, determining the frequency of job search, the mandatory work preparation activities unemployed workers are expected to undertake, and the extent to which the providers offered financial support with work preparation activities. The questions were designed with reference to themes relating to choice, reciprocity, empowerment and coercion relevant to welfare conditionality studies [28].
As its name implies, EFA is an exploratory method used to generate theory. Researchers use EFA to search for a smaller set of underlying factors that are present in a larger data set [29, 30]. Although a researcher may have some conceptualization of what factors may be present in the data, such as when items are developed to measure expected constructs, EFA generally does not place too much emphasis on a priori theory [31]. Factors, therefore, are a function of the “mechanics and mathematics of the method” [32].
Materials and method
The web-based survey was conducted by the AUWU in November and December 2019 and consisted of 31 questions that used either a five-point Likert type response or a yes/no response, as well as 5 open ended questions. These open-ended questions enabled a thematic analysis of the text [33], which was used to verify the face validity of the factors identified by the EFA [34].
The survey was advertised on the Union’s website, its Facebook page, which has more than 22,000 followers and its Twitter account, which has more than 7,000 followers. Surveys were anonymous and participation was voluntary. To emphasise the anonymous nature of the survey, limited demographic data (age grouping, gender, and type of employment service) was collected. Ethics approval to utilise the data was granted by the Monash University Human Research Ethics Committee (Project reference number: 13889) as well as receiving endorsement from the National Executive Committee of the AUWU. All analyses were carried out using SPSS 25 [35].
Participants
As is common with web-based surveys, it was possible for participants to submit without completing all the survey. A total of 758 participants took the survey of which N = 553 were useable responses (i.e. completions or near completions: ≥95%). For items not completed (i.e. < 5%), mean substitution was applied. Of the 553 completions included, 37 (6.9%) participants were aged 18–25 years of age; 240 (45.0%) were 26–49 years of age; 81 (15.2%) were aged 50–55 years of age; 163 (30.6%) were aged 56–64 years of age; 7 (1.3%) 65 + years of age; and, 5 (1.0%) preferred not to say. 291 (54.6%) were female, 216 (40.5%) male, and 26 (4.9%) identified as other or preferred not to say.
Results
Factor analysis
Principal components analysis with varimax rotation to aid in the interpretation of the factors revealed the presence of a factor structure. Post analysis examination of the correlations indicated that most correlations were over the 0.30 level reflecting a suitable matrix for factoring. The Kaiser-Meyer-Olkin Measure of Sampling Adequacy reported a value greater than the desired level of 0.60 (i.e. 0.65) [36]. The Bartlett Test of Sphericity was significant (p < .001), supporting the factorability of the correlation matrices [37]. Nine factors were initially identified with the eigenvalues of the factors in each sample exceeding 1. Using Catell’s scree test [38], it was decided to retain six factors. Given that this is an exploratory study, the authors decided to err on the side of more, rather than less, factors with the expectation that subsequent research may lead to further refinement.
In applying Pallant’s [39] recommended cut-off of 0.40, an initial six-factor solution was identified as: (F(actor) 1) “useful”, (F2) “client centred”, (F3) “receptive to feedback”, (F4) “trustworthy”, (F5) “fair”, and (F6) “friendly”. Finding a clear descriptor for F4 was challenging as questions about on-line services together with questions about personal autonomy and safety do not appear to easily relate. This was, however clarified, after examining the qualitative data, which will be discussed below. The solution explained 62 percent of the variance, with details for each factor contained in Table 1. After examination of the rotated factor structures and factor loadings, reliability analysis was conducted on the six factors. This analysis yielded acceptable Chronbach’s alpha (α) for F(1) –F(5). Alpha for F(6) was lower (0.55) but it is most likely due to the fact that the data from the survey was being used in a manner that wasn’t originally intended –something already noted above.
Table1
Table1
Note: *Reverse scored.
Table 1 contains the factor loadings and percent of variance of the items for the principal component analysis; and a comparative summary of how items loaded on each factor. Where items cross loaded, allocation was made to the highest loading. Correlations among the six factors identified are contained in Table 2. All the correlations were significant suggesting therefore that the scale is robust.
Pearson’s product moment correlations among the six factors identified
Note: N varies with analysis due to missing data. **p < 0.001 (2-tailed).
The survey generated 1,280 separate comments about: Appointments (296 comments), Complaints handling (175 comments), Funding of items for employment goals (244 comments), Online services (163 comments), and General comments (402 comments). Each individual comment was thematically analysed by the first author using each of the originally named factors i.e. F1, F2, F3, F5, and F6, as a priori codes to determine if the EFA factors adequately encompassed the qualitative data and to hopefully discern a clearer underlying meaning of F4. It was also used to detect if any comments were not able to be coded in case there were a seventh (or more) factor. Samples of this analysis were cross checked by author three. The five named factors accounted for most of the comments.
A theme which did not neatly align to either F2 (client centred) or F6 (friendly) was identified and labelled as “trustworthy” (albeit negatively described in most comments). This description may not immediately align on face value with questions in Factor 4, for which labels such as “accessible”, “transparent”, “reliable” and “open” were also considered. However, these alternates were ultimately rejected in favour of “trustworthy” as the in-depth analysis of the comments about online services suggests that many concerns were related to having to transact online an exchange about employment that respondents found dissatisfying and untrustworthy, for example: “Humans are better because most have empathy, whereas webforms don’t, which is why the govt prefers them” and “The routine part of job searches is helped by the app. But background, advice, training pathways etc could be provided by a worker who cares as distinct from someone mainly focused on compliance”. Therefore, while some F4 questions appear to be about on-line services, it is more likely that F4 is detecting a factor relating to concerns about the loss of human and trusted services, for which “trustworthy” appears more encompassing.
When the comments that were attached to F1 (“useful”) were re-analysed, it appeared that there may have been two discernible themes: “useful” and “competent” and F1 was relabelled accordingly. While these concepts may be the same thing, it also might suggest the presence of two factors, rather than one, and that the survey was not sensitive enough to adequately differentiate the two.
Discussion
As described above, the EFA combined with the qualitative analysis suggests at least six underlying factors which have been labelled as: (F1) useful and competent, (F2) client centred, (F3) receptive to feedback, (F4) trustworthy, (F5) fair, and (F6) friendly, although following the qualitative analysis it is possible that the utility factor may comprise two factors: useful (F1a) and competent (F1b).
The construct of “useful and competent” (F1) describes providers assisting unemployed workers to get a job or improve their employment skills. It was the clearest factor to be identified in the EFA, accounting for more than 27 percent of the total variance. It is also consistent with findings in the qualitative literature. Bennet et al. [2] for example, report that their research participants found one of the most frustrating aspects about employment services was expecting help to get a job, but finding that this was not the case (p.60). Usefulness is alluded to in the conclusions of Davidson et al. [4]: “. . . more help with training and job referrals.” (p.6); O’Halloran et al. [8]: “. . . an employment provider to help them improve their skills and find a job.” (p.12); and Wickramasinghe and Bowman (2018): “. . . realistic pathways to sustainable employment”(p.23). It is also consistent with the intentions of the Department of Employment’s recommendations for the future of employment services [24], which uses the word “useful” as one of the key things that “jobseekers” want from improved employment services (p.2). While the qualitative analysis reported here suggested that this factor might be two factors: “useful” and “competent”, this was not detected by the EFA. However, by including a broader range of items relating to two themes in a subsequent prospective study, it may be possible to ellicit this further.
The need for services to be “client-centred” (F2) is noted in the findings of Casey (2018), Kossen and Hammer (2010), and Marston et al. (2019) and is also consistent with the Department of Employment’s future of employment services recommendations [24]. This factor refers to taking into account the needs and circumstances of the unemployed worker in developing their individual program.
While the questions that loaded for (F3) were about complaints handling mechanisms, labelling this factor as “receptive to feedback” rather than simply “complaints”, is intended to convey the purpose of a complaints mechanism –that is, something may change as a result of making a complaint. It is worth noting that the scores for these questions revealed high levels of dissatisfaction among respondents about the complaints process which suggests that simply being able to make a complaint is in itself unsufficent. Bennet et al. [2] noted that participants in their study did not have fair access to complaints (F3) and recommended the establishment of an Employment Services Ombudsman (p.35).
Recent studies regarding the difficulties of using on-line services for vulnerable groups [40, 41] give further support to the proposition that the underlying factor associated with (F4), is related to a concern about the removal of the human element of employment services rather than on-line services per se.
A consistent criticism of Australian employment services is that it has a punative approach to “mutual obligation”, that is being required to undertake certain activities such as look for work or improve one’s employability, e.g. Bennett et al. [2]; Casey [3]; Davidson et al. [4]; Wickramasinghe and Bowman [10]. However O’Halloran et al. [8] noted that unemployed workers in their study were broadly in support of the concept of mutual obligation however what was problematic was the often coercive and arbitrary way in which mutual obligations were enforced. Therefore, rather than being about the negative construct that the questions describe, that is, penalties, it was decided to focus on a positive framing and label this factor as “fair”. In other words, if employment services have to be conditional, the manner in which these conditions are applied should be considered fair.
“Friendly” (F6) might also be labelled as “rapport” although all other labels are adjectives, which would therefore create an inconsistency. Also, given it is the intention for a rating scale to have widespread accessibility, “friendly” was chosen over “positive” (and “rapport”) as the word is perhaps more widely and consistently understood. “Friendly” describes a positive relationship between the unemployed worker and the employment provider. The benefits of an effective and positive relationship with a provider is noted in longitudanal studies of long-term unemployed workers such as Dall and Danneris, [42] and Danneris and Caswell [43]. It is worth noting that in the thematic analysis, while the majority of comments were negative, any positive comments tended to aggregate around this factor.
This preliminary study points to the existence of at least six underlying factors in unemployed workers’ experiences of employment services, which are consistent with the findings of other studies, however, rigorous field research is now needed before firm conclusions can be made.
Study limitations
Several caveats to the data used in this analysis, reinforce that this study can only be considered as exploratory:
The nature of the data
While the survey questions are consistent with its initial purpose - to provide feedback to the government on the activities unemployed workers are expected to do, the design of the survey did not commence with the intention of developing a psychometrically valid scale. Many of the commonly recommended survey design elements were considered [44], such as basing the questions on focus group feedback, expert consultation, and a test run to ensure comprehensibility. However, extensive pre-testing of the survey using cognitive testing [45] did not occur, which would need to be done before proceeding to a revised version. The survey questions do, however, encompass most experiences of Australian unemployed workers and while it is likely that some questions would be re-worded, it is unlikely that the overall content would be substantially different. Furthermore, while most of the questions focussed on the processes and therefore are measures of experiences, some questions were more clearly measures of satisfaction e.g. “I would recommend my providers to others” and are therefore conceptually inconsistent with an experience rating scale and would need to be removed from subsequent versions.
Response options and survey completion
Seven of the questions were reverse coded, which can be confusing to respondents and may have led to response errors. Wietjers and Baumgartner’s [46] review of reverse coding in survey design discusses this problem in depth and while they suggest that reverse coding of questions can sometimes encourage better coverage of the domain of content of the construct of interest, they recommend sparing usage of the technique. In the event of a revised version of the survey, it is likely that the current reverse questions would be reworded.
Representativeness
Participation in the survey was voluntary and only advertised via the AUWU’s website and social media. As the AUWU’s mission is sometimes perceived to be at odds with the Australia’s employment services policy e.g. Burke [47], it may have attracted an unrepresentative and overly negative cohort. Although with a combined social media following of 29,000, albeit some will be one person with multiple accounts, this is a sizeable proportion of the estimated 579,000 people in employment services in Australia [48]. Nonetheless, with limited demographic data collected, there is no way of determining the representativeness of the sample. While this is a weakness, it is less concerning for an exploratory study. However, to improve the robustness of a subsequent survey, more demographic information would need to be collected and advertising for recruits extended beyond the AUWU.
Conclusion
The issue of providing a valid and reliable instrument to measure unemployed workers’ experiences may provide an opportunity for improvement in this sector, which makes this topic relevant. This paper presents preliminary data in achieving this aim and provides a base for the creation of a valid and reliable rating scale for employment services, with scope for refinement in future versions of the questionnaire, subject to further psychometric and field testing.
Nonetheless, these preliminary findings form a useful conceptual framework for employment providers and regulators to consider essential components from the perspective of the unemployed worker in the design and delivery of employment services both in Australian and internationally. In the times of pandemic and world economic challenges that are raising numbers of unemployed workers around the world, this will be even more relevant in research but especially in practical terms of improving those services.
Footnotes
Acknowledgments
Thanks to Hayden Patterson from the Australian Unemployed Workers Union for his support with this research.
Conflict of interest
None to report.
Disclaimer
The authors do not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and have no relevant affiliations beyond their academic appointments.
The term ‘unemployed worker’ is used throughout this report unless referring to a specific document, dataset or policy that uses an alternate term such as ‘jobseeker’
The Minister and the Federal department(s) responsible for employment services have had frequent name changes including at times, separate Ministers and Departments being responsible for mainstream and for disability employment services. For ease of communication, all will be referred to henceforth as the “Department of Employment”.