Inequality and discrimination in the online labor market: A scoping review

Abstract

Based on a comprehensive set of studies collected via five academic databases, this scoping review examines how inequality and discrimination have been studied in the context of paid online labor. We identify three approaches in the literature that aim to (1) identify participation patterns in (national) survey data, (2) examine background characteristics of online contractors based on survey or digital trace data, and (3) reveal social biases in the hiring process using experimental data. Building on Shaw and Hargittai’s pipeline of participation, we present a multi-stage model of engagement in online labor. When we map the studies across the stages, it becomes clear that the literature focuses on later stages (i.e. having been hired and received payment). Based on this analysis, future research should examine barriers to participation in earlier stages. Furthermore, we advocate for research that examines participation across multiple pipeline stages as well as for analysis of platform-level biases.

Keywords

Digital discrimination digital inequality online labor platform economy the gig economy scoping review literature review

Introduction

Inequality and discrimination in the labor market has been documented in myriad empirical studies, reviews, and meta-analyses (e.g. Bertrand and Duflo, 2017; Pager, 2007; Quillian et al., 2017). Much of this research has focused on conventional forms and arrangements of work, while work that happens on or via online platforms often remains overlooked. Based on the emerging research in this area (e.g. Gray and Suri, 2019; Poell et al., 2021; Schor, 2020), numerous factors suggest that findings from traditional labor contexts may not generalize to the online realm. Digital inequality research has found that accessing and participating in digital labor requires technical infrastructure and skills (e.g. Hargittai, 2021), which might disproportionally pose barriers to some jobseekers compared to others. Besides, online platforms rely on computational structures to organize labor at a scale that exceeds any hiring operation in the traditional labor market, causing hiring processes to look significantly different (e.g. Gray and Suri, 2019). Whereas labor relations in the traditional context primarily exist between two actors, they now involve the platform in addition to the users making the hiring decisions and the online contractors themselves.

In today’s economy, finding and performing work on the Internet is becoming increasingly prevalent: In 2016, 24% of American adults earned money for performing a job or task they had found through an online platform (Pew Research Center, 2016). Often referred to as the gig economy, the online labor market encompasses a diverse range of short-term activities, including ride hailing via Uber, renting out property via Airbnb, cloud-based freelancing via Upwork, and performing so-called “microtasks” via Amazon Mechanical Turk. Different platforms organize the work and work relations in distinct ways, leading to variance in the job-seeking process across platforms. For example, some platforms that automatically match contractors with customers allow only one of the parties the choice to accept the match, while the other party is obligated to go through with the transaction. Other platforms resemble a marketplace and grant either the customer or the contractor the opportunity to select the other party based on available profiles. As a result of the platforms’ design, these profiles might contain information not as readily available in offline hiring settings, such as reputation scores or reviews. Ultimately, the process of obtaining a job in this space is far from streamlined not only due to the number of actors involved (i.e. platform, clients, and contractors), but also due to the variation across the platforms (e.g. in terms of application process).

As online platforms mediate access to opportunities to exchange work for money, it is important to understand who is able to capitalize on the existence of these platforms. Over the last decade, multiple strands of inquiry within the scholarly, legal, and public literature have started to address (parts of) this question. By systematically gathering and analyzing studies on this topic, this scoping review aims to identify these approaches, provide an overview of their central findings, and synthesize a refocused research agenda. In an attempt to synthesize the different lines of research, we build on the pipeline of online participation inequalities (Shaw and Hargittai, 2018). Shaw and Hargittai’s (2018) model breaks down the steps one needs to take before being able to contribute online and presents the sequence of steps in the form of a pipeline. The pipeline model fits within the larger trend within digital inequality scholarship of disaggregating online participation (e.g. Helsper, 2021; Scheerder et al., 2017). The pipeline model in particular allows for the identification of barriers to participating and reaping benefits, as it aims to locate the “leakages” between the various pipeline stages. As part of this scoping review, we extend the pipeline to fit the specific context of online labor platforms and to account for the various barriers that prior literature has studied. We then map prior literature across the stages of this extended pipeline to identify the foci of prior studies and directions for future research.

This scoping review lays out what aspects of and how online labor market inequality have been studied. Based on systemically collected literature from five academic databases, we identify three approaches that prior literature has taken to investigate inequality and discrimination in the online labor market. The first approach analyzes national survey data to understand who does or does not participate on gig economy platforms. The second approach entails examination of a sample of online contractors to understand their different backgrounds and experiences. The third approach focuses on social biases in the hiring process, both on the side of individual users making hiring decisions and the algorithms powering the online labor platforms. We contribute a theory taking the form of a pipeline model, building on Shaw and Hargittai’s (2018) pipeline of online engagement, that describes how factors affecting inequality and discrimination limit success in online labor markets. We find that some stages in the pipeline (i.e. having been “hired” and having received payment) have received much more attention than other stages and identify worthwhile future directions based on this finding. Particularly, since most of the research focuses on individual-level resources and biases as a source of unequal participation, we see a need for future research to examine the specific role of the platform in facilitating inequality and discrimination. The review aims to answer the following three research questions:

RQ1: What characterizes different approaches to the study of inequality and discrimination in the online labor market?

RQ2: What does a participation pipeline look like in the context of online labor and how does prior literature map onto the various stages of the pipeline?

RQ3: What important puzzles do this leave for future research?

Background

Online labor market

Across popular media and academic literature, there are many different conceptualizations and definitions of the labor that happen on or via the Internet. The large number of different terms, all defining the bounds of online work differently, convolute the space. For example, the term “gig” in “gig economy” and “gig work” refers to compensation on a piece-rate basis (e.g. Kässi and Lehdonvirta, 2018; Woodcock and Graham, 2020), while terms like “contract work” and “online freelancing” imply that the work is on the basis of a contract of limited length (e.g. Hannák et al., 2017; Wood et al., 2018). The terms “crowd work” and “microwork” refer to a segment of online work that can be split into a set of smaller tasks (e.g. Kässi et al., 2019; Kittur et al., 2013) and the “sharing economy” refers to platforms that allow individuals to “share” (usually at a price) their assets through various arrangements (e.g. Ravenelle, 2017; Schor and Attwood-Charles, 2017). Other terms like “platform economy” and “platform labor” refer to the central role of the platforms that mediate economic activity, which only sometimes includes business-to-consumer platforms such as Netflix or Amazon (e.g. Hoang et al., 2020; Van Doorn, 2017).

For the purposes of this study, we employ Woodcock and Graham’s (2020) inclusive definition of online labor by considering “labor markets that are characterized by independent contracting that happens through, via, and on digital platforms” (p. 3). However, unlike some other characterizations of the gig economy and/or the online labor market (e.g. Vallas and Schor, 2020), we only take into account paid work¹ and thus exclude all forms of unpaid labor, such as what Duffy (2017) calls “aspirational labor.” Our definition also does not include platforms that facilitate professional networking and hiring, such as LinkedIn and Indeed.com. In categorizing online labor, we lean on the five subsections of the gig economy proposed by Vallas and Schor (2020): (1) architects and technologists of the platform, who design and maintain the digital infrastructure; (2) cloud-based consultants or freelancers; (3) gig workers whose labor is digitally mediated but often performed offline, such as delivery, ride-hail, and home repair; (4) microtaskers who perform short human intelligence tasks for which they are paid on piece-rate basis; and (5) content producers and influencers.

Digital inequality and discrimination

This review sets out to synthesize the body of literature that aims to record and understand patterns in participation in and outcomes on online labor markets. On the one hand, this includes studies taking on questions that fit into the research area of digital inequality, which describes the differences in Internet access and usage between Internet users and non-users as well as among Internet users (DiMaggio and Hargittai, 2001). The scholarship assessing these differences uses a variety of methods to identify patterns in access to, adoption of, and participation on the Internet, including surveys and interviews (e.g. Hargittai, 2021; Helsper, 2021; Scheerder et al., 2017). On the other hand, this review includes studies that might be described as falling under the umbrella term “digital discrimination,” which is a broad, less widespread term used to describe circumstances in which one social group of Internet users is treated less favorably than another (e.g. Cheng and Foley, 2017). It encompasses explicit and implicit discrimination enacted by individual Internet users against one another (e.g. through posts on social media), as well as cases of discrimination that occur due to platform or algorithm design. In the case of online labor markets, this includes biases of individual users and biases built into platforms that disproportionately harm one group of contractors. Ultimately, both these biases as well as the disparities described by digital inequality research allow one group to capitalize on their adoption and usage of a platform to a greater extent than others.

In their literature review, Schor and Attwood-Charles (2017) examine similar dynamics in the segment of the online labor market that mediates peer-to-peer sharing. They synthesize studies related to social connection, conditions for laborers, and who participate in the sharing economy. Their synthesis shows that users of the sharing economy are disproportionately people with privilege, but that the demographic composition of users has become more diverse over time. In this scoping review, we contribute to these findings in three main ways. First, we identify how researchers have approached studying inequality and discrimination in the online labor market, rather than only what they have found. Second, we consider all types of paid labor instead of merely focusing on a subsection of the gig economy (in this case, the type of labor described by the term “the sharing economy”). Finally, in this scoping review, we search for, collect, and include studies systematically. To our knowledge, there are no scoping reviews or systematic literature reviews that synthesize the literature in this subfield. Systematically collecting studies is necessary to ensure inclusion of all relevant studies. The exclusion of lines of research is a particular concern in this topic area due to the variety of research questions, methods, and terminologies used to understand related questions.

Our analysis builds on Shaw and Hargittai’s (2018) “pipeline of online participation inequalities,” a model that conceptualizes platform adoption as a multi-stage process. Shaw and Hargittai (2018) disaggregate the stages that an individual has to go through in order to contribute to Wikipedia, such as hearing of the website, visiting, and knowing it is possible to contribute. Breaking down online participation does not merely make the measure more precise, but it allows for the identification of barriers to Internet usage. Their analysis of US national survey data demonstrates that individuals indeed drop out of the pipeline at various points before contributing to Wikipedia (Shaw and Hargittai, 2018). These leakages in the pipeline point to barriers to online participation. Identifying and measuring such barriers has the potential to inform efforts to combat Internet-related inequality more effectively. In different ways (e.g. policy-, community-based), such policies would aim to fix a specific leakage in the pipeline and, ultimately, to facilitate a more egalitarian progression toward the end of the pipeline. While Shaw and Hargittai’s (2018) model is specific to Wikipedia contributions, this study considers how it applies to the process of becoming an independent contractor in the online labor market. Specifically, we extend the model by incorporating stages related not only to individuals’ adoption of the platforms, but also to their labor outcomes once they apply or make themselves otherwise available for jobs. By doing so, we account for the power that the hiring party and the platform hold in granting individuals’ opportunities. Ultimately, we map prior studies across the pipeline to identify the extent to which the stages have been studied, determining remaining puzzles for future research.

Method

Scoping review

A scoping review is a systematic literature review that examines the state of a particular research area with the aim of understanding and synthesizing research questions, methods, and approaches. It is

an ideal tool to determine the scope or coverage of a body of literature on a given topic and give clear indication of the volume of literature and studies available as well as an overview (broad or detailed) of its focus. (Munn et al., 2018: 2)

Scoping reviews are particularly well-suited for exploring and synthesizing an emerging set of literature, such as the literature examined in this article. We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) approach (Moher et al., 2015), which informed the transparent process of reporting on the method through the PRISMA flow chart (see Figure 1).

Figure 1.

PRISMA flow chart.

Data collection

We collected papers (n = 599) from five databases that cover a range of perspectives within the social and computer sciences: SCOPUS, Sociological Abstracts, Communication Source, ACM Digital Library, and the ABI/INFORM Collection. The search strings that formed the basis of data collection contained a combination of keywords related to inequality, discrimination, and bias as well as keywords related to the online labor markets. To ensure broad coverage of studies on online labor, we incorporated an array of terms often used to describe the online labor market, including “the gig economy,” “platform work,” “the sharing economy,” and “microwork.” Through an iterative trial-and-error process, we first developed a Boolean search string for SCOPUS (see Appendix 1 for the original search string) and adapted it to fit the format required for the other databases. We limited the search to title, abstract, and keywords, and automatically excluded all records with document types that were not peer-reviewed journal articles or conference proceedings.

As another form of data collection, we conducted a cited reference search as well as a citation search for the relevant papers that the database searches yielded. For the cited reference search, we entered the titles into Web of Science’s cited references feature and retrieved metadata for the records that cited any of these papers. The papers relevant to our topic (n = 45) were added. For the citation search, we went through the references cited in the papers found through the database searches and relevant papers (n = 87) were added. Relevance was determined based on the filtering criteria listed below.

Data filtering

Upon the retrieval of articles through the databases, we merged all data sets and removed duplicates, first through the deduplication function of the reference manager EndNote and then manually (total n = 150). We then filtered the articles for eligibility, as data collection based on the Boolean terms yielded papers outside the scope of this article. As a result, we filtered the data according to the following inclusion criteria:

All papers must be English-language empirical papers published in peer-reviewed journals or conference proceedings.

All papers must focus on a specific case or multiple cases of paid online labor mediated via some sort of platform or website. Other terms might be used to describe online labor, such as “the gig economy,” “sharing economy,” “crowd work,” or “microwork.”

NB: This means that we deleted all conceptual papers or articles related to customer experience. In the case that a paper looked at both customers and workers, we included the paper but only looked at the data and findings on workers.

All papers must examine patterns of access, participation, and/or labor outcomes within the pool of independent contractors in the online labor market. All papers explicitly discuss inequality, discrimination, or bias.

We evaluated each paper in the data set after deduplication based on the inclusion criteria in the order stated above. First, we screened the articles based on the titles and abstracts. Then, we retrieved and screened the full texts of the papers. As reported in the PRISMA flow chart (Figure 1), across the two moments of filtering, 52 papers were excluded based on criteria 1, followed by 382 papers excluded based on criteria 2 and 108 papers based on criteria 3. The final analyzed data set included 39 papers.

Coding

We coded the final set of articles by adopting a selection of strategies belonging to grounded theory (Corbin and Strauss, 2015). Initially, open codes were applied to all papers in MaxQDA, a computer-assisted qualitative data analysis software (CAQDAS). This involved an iterative process of comparing pieces of data across the entire data set and writing memos with initial ideas, as described by Corbin and Strauss (2015). The open codes, which included in vivo-style codes (i.e. entered as in-text comments), covered a wide range of topics. Subsequently, we combined codes within MaxQDA’s interface to identify initial patterns. We constructed an initial flexible coding system with five broad categories: (1) outcome variables according to an adaptation of Shaw and Hargittai’s (2018) pipeline of online engagement; (2) the levels of analysis at which inequality, discrimination, or bias (e.g. on the level of the individual user or the platform); (3) types of predictor variables; and (4) additional details on methods (e.g. methods, empirical site, location). After this initial round of open coding, we recoded the papers with more detail. In this second round of coding, we color-coded according to the previously mentioned categories in the coding system and recorded patterns in a spreadsheet. Through this iterative process, we developed an understanding of approaches to studying inequality, discrimination, and bias in the online labor market.

In the results section, we first provide an overview of the papers, reporting on empirical sites, methods, and findings. Afterward, in response to RQ1, we describe approaches to studying inequality and discrimination in the online labor market that we identified in the examined studies. As for RQ2, we first lay out the stages of the extended pipeline of online participation in the context of online labor and then map the studies across the pipeline. Finally, concerning RQ3, we identify gaps in the body of studies and consider possible directions for future research.

Results

Overview

As shown in Table 1, the papers in the data set primarily examine empirical sites that fall into three segments of the gig economy as categorized by Vallas and Schor (2020): cloud-based freelancers, service workers, and microtaskers. Most studies focused on service workers, who engage in labor mediated by a platform or application but performed offline. Home-sharing, especially through the popular platform Airbnb, has received disproportionate attention compared to other online or digitally mediated types of work.

Table 1.

Count of papers across categories of the gig economy (Vallas and Schor, 2020).

Category of the gig economy	Number of papers
Architects and technologists	0
Cloud-based consultants or freelancers	10
Gig workers performing offline services	21
Microtaskers	6
Influencers and content creators	0
Unspecified	2

All studies employ quantitative methods, which is consistent with expectations as these methods are well-suited to report patterns of participation and labor outcomes. Three quantitative methods are particularly salient across the data set: surveys, field and laboratory experiments, and digital trace data (see Table 2). Across these methods, the measurement of gender and race, the most common independent variables, varies in noteworthy ways. The survey studies rely on respondents to report their demographic information. In the field and laboratory experiments, researchers tend to manipulate race and gender across materials that participants are exposed to. In studies relying on digital trace data, gender and race are detected either by comparing names against a database of common female and male names (e.g. Foong et al., 2018; Galperin, 2019), via manual identification by the authors themselves, Amazon Mechanical Turk workers, or undergraduate research assistants (e.g. Hannák et al., 2017; Marchenko, 2019; Tjaden et al., 2018), or via automated identification using a web service such as genderize.io or angus.ia (e.g. Barzilay and Ben-David, 2016). Determining gender and race via manual or automated identification has clear limitations due to the reliance of human coders and human-designed programs on stereotypical identity expressions to assign gender and race, potentially resulting in an inaccurate and incomplete picture. Common in all these quantitative methods are the reductionist measures of identity categories, as the design requires researchers to collapse identities into categories, bypassing their complexity. In this way, it limits the ability of these studies to cast light on discrimination faced by individuals who do not fit the categories chosen by the researchers (e.g. trans- and non-binary people).

Table 2.

Count of papers across methods.

Method	Number of papers
Survey	8
Experiment	16
Digital trace data	11
Interviews	2
Other	2

Most papers in the data set report findings that indicate that those who enter and do well in the online labor market are likely to be wealthier, more highly educated, White, male, and younger. For example, several studies find those with a higher level of income (Eichhorn et al., 2020; Liu and Xu, 2019) and education (Hoang et al., 2020; Schor, 2017) to have higher adoption rates of various online labor platforms. Similarly, many studies found that Black online contractors charge less on home-sharing websites (Marchenko, 2019), receive lower offers for items they sell online (Ayres et al., 2015; Doleac and Stein, 2013), are rated lower and reviewed more negatively (Hannák et al., 2017). Female contractors face similar patterns when compared to their male counterparts (e.g. Barzilay and Ben-David, 2016; Foong et al., 2018; Hannák et al., 2017; Jahanbakhsh et al., 2020). Several studies find evidence of a preference for hiring domestic contractors (Lehdonvirta et al., 2014; Liang, 2017) as well as an age-based bias (Newlands and Lutz, 2020). While the overwhelming majority of studies find evidence for unequal levels of participation and labor outcomes, some studies do not (e.g. Dai and Brady, 2019). For example, a few studies find no evidence for female contractors receiving lower ratings than their male counterparts (e.g. Greenwood et al., 2020; Thebault-Spieker et al., 2017).

Different approaches to studying online labor market inequality

Broadly speaking, we identify three approaches to the study of inequality and discrimination in the gig economy. All address distinct research questions drawing on different methods and framing (for an overview, see Table 3).

Table 3.

Overview of three identified approaches.

Approach	Central research question	Method	Number of papers
1	Who does and who does not engage in online labor?	National survey	3
1	Who does and who does not engage in online labor?	Survey of a specific population	2
2	Who are online contractors?	Survey	4
2	Who are online contractors?	Digital trace data	11
3	What social biases exist in the hiring process?	Experiment (field, lab, online) examining hiring decisions	13
3	What social biases exist in the hiring process?	Experiment (simulation, online) examining platform design	3

The first approach draws on survey data to make inferences about factors impacting participation in online labor. The dependent variable in such studies is one (or multiple) form(s) of engagement with online labor platforms, such as having an account or having performed a task. Drawing on primarily national samples, which include both people who have engaged in this type of work and those who have not, allows for the comparison of the two groups in terms of their backgrounds. Non-demographic predictor variables for participation include digital skills or literacy (e.g. Eichhorn et al., 2020; Kowalczyk-Anioł et al., 2021; Liu and Xu, 2019), prior experience (e.g. Galperin, 2019), and values and attitudes such as materialism and innovativeness (Eichhorn et al., 2020; Liu and Xu, 2019). Similar to other digital inequality studies (e.g. Correa, 2010; Hargittai, 2021; Helsper, 2021), these papers often frame online participation as a set of opportunities that not all people have equal access to. These scholars thus view the inequitable distribution of resources (e.g. of technology ownership, infrastructure, skills, knowledge) as the root cause of unequal participation in online labor platforms. Edge cases of this approach include survey studies that analyze a sample of people within a specific social group to understand who engages in online labor within this group and what challenges they face (i.e. older adults (Brewer et al., 2016) and people with a disability (Zyskowski et al., 2015)).

The second approach examines online contractors to understand who makes up this group and how contractors behave differently. Some of these papers draw on survey (i.e. on a sample of online contractors) or digital trace data to describe the sociodemographic composition of contractors (e.g. Abendroth, 2021; Berde and Tőkés, 2019; Liu and Xu, 2019; Mashhadi and Chapman, 2018; Newlands and Lutz, 2020). Another subset of papers describes patterns in labor outcomes, identifying the composition of contractors that land the jobs and actually generate income. The outcome variable examined in these designs varies from binary measures of jobs obtained (e.g. Chan and Wang, 2018; Galperin, 2019; Leung and Koppman, 2018; Liang et al., 2018; Tjaden et al., 2018) to continuous measures of payments or hourly wage earned (e.g. Litman et al., 2020). Occasionally, studies focus on dependent variables related to the design of the platform or algorithms, such as position in the ranking algorithm or outcomes in terms of ratings or reviews (Hannák et al., 2017). A last subset of papers taking this second approach analyzes digital trace data to find patterns in how contractors of a particular gender (e.g. Foong et al., 2018), race (e.g. Wang et al., 2015), or national background (Lehdonvirta et al., 2014; Liang et al., 2018) set the prices of their services differently.

The third approach employs experimental methods to examine the social biases in the hiring process, which include both biases of individual users making hiring decisions as well as biases built into the algorithms powering the online labor platforms. Studies examining biases of individual users evaluate the effect of some manipulation in the background information listed on contractor profiles on the behavior of the users who function as “employers.” The majority of the papers are online experiments that scrutinize the consumption intentions of users within peer-to-peer online labor markets, such as Airbnb and Uber (e.g. Liebe and Beyer, 2020; Su and Mattila, 2020). After being exposed to a set of contractor profiles that differ in sociodemographic background (e.g. a Black vs a White contractor), participants are asked to indicate their intention of hiring the given contractors. These hiring intentions function as proxies for labor outcomes for contractors of that social background. For example, bias in the preferences of Airbnb guests might cause some hosts to receive fewer bookings compared to others. The field experimental studies taking this approach have a similar design, except that they rely on actual inquiries as the outcome variable. Such field experiments resemble studies on discrimination in the traditional labor market (see Quillian et al., 2017 for a meta-analysis of such field experiments focused on racial discrimination). The few studies examining biases built into the algorithms have a similar setup. In these (simulation-based) experimental studies, researchers manipulate a platform affordance or algorithm to understand the impact on who gets hired (e.g. Thebault-Spieker et al., 2017). Experimental studies that examine bias on the level of individual customers are, however, much more prevalent in the data set than studies examining bias on the level of the platform.

Pipeline of online participation in the labor context

In line with Shaw and Hargittai’s (2018) pipeline of online participation inequalities, we conceptualize participation in the online labor market as a multi-stage process with stages that are conditions for an individual to make an online labor platform into a source of income (see Figure 2 for a visualization of the pipeline). First, a would-be online contractor must have heard of the platform. Second, they must have an interest in adopting the platform, which requires an understanding of the benefits of doing so. Afterward, in the third stage, they must have visited the site. Fourth, they need to have created an account, and, fifth, they need to have “applied” for a job on the platform. This is an umbrella category for the actions necessary to make oneself available to be hired. Depending on the site, this might involve either attempting to obtain a task to complete, such as on Amazon Mechanical Turk, or building a profile to showcase one’s services, such as on TaskRabbit. While completing a profile on online labor platforms is a form of participation, it does not guarantee equal opportunity to generate income on these platforms as the hiring party—whether a consumer or business—impacts who ultimately does or does not gain access to jobs. Therefore, we include stages in the pipeline of participation in the online labor market to account for the power of the hiring party and the platform as a mediating agent. In the sixth stage, the contractor must have received attention on the platform (e.g. in its ranking algorithms) and from potential employers. Seventh, their participation is conditional on having received a response from their future employer. This stage corresponds to the callback stage common in field experiments examining hiring discrimination (e.g. Pager, 2007). Then, in the eighth stage, the user must have been “hired.” Only once all the prior stages have been passed can a user perform a job. Afterward, they might receive a payment as well as reviews and/or ratings depending on the specific design of a given platform. Both are forms of labor outcomes that can prove beneficial to the worker. Reviews and/or ratings might factor back into the labor outcomes, which, for example, Jahanbakhsh et al. (2020) address.

Figure 2.

Pipeline of participation in the online labor market.

While conceptualizing participation as a set of stages that users pass through increasingly gaining access is prevalent in digital inequality research more generally (e.g. Shaw and Hargittai, 2018; van Deursen and van Dijk, 2018), it is not as pervasive in the present data set. More than half of the studies (i.e. 26 out of 39) only examine one outcome variable, often a binary measure of having participated in this space or not. In addition, studies that examine multiple outcome variables do not examine these outcomes within the individual and thus cannot say how the various outcome measures are related. As others have pointed out before, this overlooks that there are precursors to platform adoption and online participation that prevent some from participating. Disaggregating the stages is important to identify and measure the barriers to online participation. In other words, it allows for the isolation of the precise points in the pipeline where users “leak” out of the pipeline. The identification and measurement of barriers in turn can inform initiatives to combat unequal participation levels. For example, a governmental campaign to educate the public on the employment opportunities available might make more people aware of the sites, leading to an increase in the first stage of the pipeline (i.e. having heard of a platform). The pipeline of online labor participation also includes stages related to labor outcomes, accounting for the power that hiring parties and platforms have in democratizing opportunities.

Prior literature mapped across the stages

Figure 2 provides a visualization of the stages with the number of studies in the data set that contain a corresponding variable. Relatively few studies considered the participation outcomes prior to “being hired.” A couple of papers incorporated outcome variables related to the first pipeline stages: having heard of or having interest in adopting a particular platform or type of work. Some scholars measured interest in adopting a platform by asking directly via a survey question or in an interview (Liu and Xu, 2019). More common, however, was to measure interest through various measures of perceived usefulness and understanding of the financial, social, or other benefits of participating (e.g. Brewer et al., 2016; Eichhorn et al., 2020). Some papers discussed the perceived benefits as measures of motivation or “motivational access” (Eichhorn et al., 2020: 8). The next two stages involve transforming that motivation into action by actually having visited the platform and having made an account. No studies in the data set incorporated outcome variables directly related to these stages. In a paper published after data collection, Shaw et al. (2022) examine both these outcome variables in the context of a four-stage pipeline model.

As for the next stage of having “applied” (i.e. either applying for tasks or building a profile), studies primarily use digital trace data to examine demographic information in profiles (e.g. Foong et al., 2018; Mashhadi and Chapman, 2018). After having made oneself available, the next precursor of participation is that the independent contractor needs to have received attention from the algorithm and potential clients, after which they need to have received a response. Hardly any studies scrutinized either of these stages. Hannák et al. (2017) study the attention that independent contractors receive by examining workers’ rankings in search results on TaskRabbit and Fiverr. Furthermore, Doleac and Stein (2013) recorded the number of responses that different sellers received for an ad selling a used iPod.

Finally, after all these conditions have been met (i.e. the first seven stages have been passed through), users can get hired. Sixteen papers employ some binary outcome variable measuring whether an individual successfully obtained a job through an online labor platform. In studies using digital trace data, being hired tends to be measured as a dichotomous variable indicating whether the hiring has taken place or not, whereas the experimental studies employ binary or continuous measures of the intention to hire. Closely related to the being hired stage of the participation pipeline is the stage of actually having performed a job. This should be a separate stage, since not all users might end up performing or completing the task or job. Outcome variables related to this stage include self-report measures in survey research (i.e. “have you ever performed a job?”) and signs of job completion captured by digital trace data. Some studies measured the performance of labor as a continuous variable, for example, by recording the number of hours worked. The last part of the online labor pipeline is actually receiving a payment as well as reviews and ratings. Most variables related to payment in the examined studies are continuous measures of payment or hourly wage (e.g. Litman et al., 2020). Similarly, few studies incorporated a binary measure of whether one had received a review or rating, while five studies considered the valence of the reviews and ratings received by participants (e.g. Hannák et al., 2017; Jahanbakhsh et al., 2020).

A few studies focus on outcome variables that do not fit the framework of the pipeline. For example, some studies include measures of “length since platform adoption” (e.g. Liu and Xu, 2019). Others incorporated a general self-report measure of “having participated” (e.g. Hoang et al., 2020). While this presumably corresponds to having made an account and actually used the platform, the imprecision of the measure makes it impossible to identify its exact location on the pipeline.

Remaining puzzles

In general, based on our comprehensive database searches, the existing body of scholarship that documents patterns of inequality and discrimination within online labor markets is sparse. While literature starts to examine disparities in access to and engagement in online labor, the topic remains largely understudied. Thus, future research should verify and expand on the findings summarized in this review, especially where findings of prior studies contradict one another (e.g. on the role of gender).

In terms of the proposed pipeline of online labor participation, the latter part of the pipeline receives much more attention compared to the earlier stages. Prior literature focuses primarily on patterns in who is being hired in the online labor market and how their payments compare. In contrast, the studies hardly incorporate measures of the earlier stages. Studying these stages is important as it might explain patterns of success in later stages. In addition, such studies are also worthwhile inquiries in their own right. Insights into who meets these conditions of participation and desired labor outcomes are valuable, for example, in designing policy interventions.

Few studies look at multiple participation measures and labor outcomes across the pipeline of participation. Even fewer studies look at multiple outcome variables as consecutive stages of one pipeline (for an exception, see Eichhorn et al., 2020 and Shaw et al., 2022). Future research should verify the pipeline model we propose in this study. Due to the limitations of existing methods and the variance among the eleven outcome variables, this might require a mixed-method approach. For example, a study could combine survey methods to examine the first half of the pipeline, measuring individuals’ familiarity with and motivations for using online labor platforms, leveraging (field) experimental methods to measure their success once they pass through the first stages. Field experiments are particularly helpful in examining the role of hiring parties and digital environments in a natural setting. The presence of a hiring party presents a structural barrier to participation, differentiating the case of online labor from other, more open forms of online engagement.

Finally, the body of literature is skewed toward examining bias at an individual level as opposed to bias at the platform level. Only a few studies examine the role and power of platform design in favoring a subset of users or facilitating the biases of individuals in hiring positions. This is a direction for future research that demands attention, as platforms have the power to exacerbate or alleviate inequality and discrimination in more systematic ways than individual users (e.g. Noble, 2018). Research scrutinizing the role of the platform might, for example, employ audit methods to examine algorithmic biases or field experiments to isolate particular design choices, such as reviews, ratings, and quality badges (e.g. Airbnb’s “Superhost” or TaskRabbit’s “Elite Tasker”). Insights into how this happens, how to design platforms to reduce the impact of individual users’ biases, and how to regulate such mechanisms should prove fruitful in ways that advance the existing literature.

Limitations

This scoping review is limited in that we made choices while designing the data collection that affected the composition of the set of examined papers. For example, we focused on patterns of inequality and discrimination, causing us to exclude related studies, for example, on the way that the online labor market exacerbates inequalities through poor labor conditions (e.g. Lehdonvirta, 2018; Wood et al., 2019) and limits workers’ ability to organize themselves (e.g. Wood et al., 2018). By focusing on patterns of participation and success, we also excluded most qualitative research with rich description and analysis, for example, of the role that power plays in the various experiences online contractors of different backgrounds have while pursuing and navigating opportunities on platforms (e.g. Sannon et al., 2022). Similarly, we chose to limit the scope of this article to paid labor to make the topic appropriate for a scoping review. However, this meant that we excluded all forms of unpaid labor, including volunteer-based content contribution (e.g. Fiers et al., 2021) and the labor of (aspiring) cultural influencers (e.g. Poell et al., 2021). As a result, in the search string that formed the basis of data collection for this review, we did not include terms such as “content creator” and “influencer.” A future scoping or systematic literature review could take a more inclusive approach to this body of literature. Furthermore, in only including papers that explicitly talked about inequality, discrimination, or bias, we might have excluded other relevant papers that did not talk about any of these terms explicitly yet still discussed patterns in participation and outcomes (e.g. Hargittai and Shaw, 2020).

Conclusion

This scoping review examines a systematically collected set of empirical studies on inequality, discrimination, and bias in online labor environments. Specifically, we focus on scholarship examining patterns in access to, participation in, and outcomes on online labor platforms. We identify three approaches to investigating the intersection of digital inequality and online labor: one that focuses on factors impacting participation, one that describes the demographic composition and behaviors of online contractors, and one that examines the bias of users and platforms in making and facilitating hiring decisions. In terms of empirical sites, there is a strong focus on digitally mediated service work, such as home-sharing and ride-hailing services.

We propose a model that disaggregates the stages that need to be traversed to participate in the online labor market successfully, which extends Shaw and Hargittai’s (2018) pipeline of online participation inequalities. The proposed model has eleven stages through which an individual progresses, from having heard of an online labor platform to receiving payment as well as reviews and ratings. Breaking down these precursors to labor outcomes is crucial, as it allows for the identification of “leakages” in the pipeline (i.e. barriers to reaching the last stage). Identifying and measuring the barriers has the potential to inform policies and programs that aim to lower them.

Mapping the studies in the data set of this scoping review across the stages of the pipeline of online labor participation shows that the latter part of the pipeline of participation has received more attention than the first half. Specifically, outcome variables related to having been hired and having received a payment are common in the scholarship. In contrast, hardly any studies examine the first seven stages and, therefore, overlook the conditions that need to be met before an individual can perform a job or task. Besides, analyses of the participation of a given individual across multiple stages of the pipeline are rare in the examined data set. This finding should inform future research, emphasizing the need for examination of participation across the pipeline of online labor participation. Furthermore, we find an empirical focus on hiring biases of individual hiring parties and a dearth of studies scrutinizing the biases built into platforms that could potentially impact patterns in labor participation and outcomes. Future research should, therefore, center on the platform and its role in facilitating or combatting inequality and discrimination in the online labor market.

Footnotes

Appendix 1 Acknowledgements

The author would like to acknowledge Siying Luo, who contributed greatly to this project as a research assistant. Sohyeon Hwang and Aaron Shaw provided invaluable support and advice in setting up the project. Iris Janssen and Evan Walters designed various iterations of Figure 2. The author is grateful for the feedback that they received from audiences at the Annual Conference of the International Communication Association and the Annual Conference of the Association of Internet Researchers as well as from members of the WebUse Project and the Community Data Science Collective, particularly Nathan TeBlunthuis, and from the reviewers of New Media Society.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Floor Fiers

Notes

Author biography

Floor Fiers is a doctoral candidate in the Media, Technology, and Society program at Northwestern University. They are a member of the Community Data Science Collective and the Web Use Project. Their research draws on mixed methods to examine questions of power, inequality, and worker resistance in the gig economy. In addition, they are interested in various forms of unpaid labor in online environments.

References

Abendroth

A-K

(2021) The gender pay gap in the platform economy: comparing the importance of market and organisational dynamics on two German crowdworking platforms. Gender a Výzkum / Gender and Research 21(2): 59–84.

Ayres

Banaji

Jolls

(2015) Race effects on eBay. The RAND Journal of Economics 46(4): 891–917.

Barzilay

Ben-David

(2016) Platform inequality: gender in the gig-economy. Seton Hall Law Review 47(2): 393–432.

Berde

Tőkés

(2019) The platform economy as a working opportunity for older people: the case of the Hungarian carpooling company Oszkár. Journal of Enterprising Communities: People and Places in the Global Economy 14(1): 76–90.

Bertrand

Duflo

(2017) Field experiments on discrimination. In: Duflo

Banerjee

(eds) Handbook of Field Experiments. Amsterdam: Elsevier, pp. 303–393.

Brewer

Morris

Piper

(2016) “Why would anybody do this?” Understanding older adults’ motivations and challenges in crowd work. In: Proceedings of the 2016 CHI conference on human factors in computing systems (CHI’16), New York, 7 May, pp. 2246–2257. New York: ACM.

Chan

Wang

(2018) Hiring preferences in online labor markets: evidence of a female hiring bias. Management Science 64(7): 2973–2994.

Cheng

Foley

(2017) The sharing economy and digital discrimination: the case of Airbnb. International Journal of Hospitality Management 70: 95–98.

Corbin

Strauss

(2015) Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. 4th ed. Los Angeles, CA: SAGE.

10.

Correa

(2010) The participation divide among “online experts”: experience, skills and psychological factors as predictors of college students’ web content creation. Journal of Computer-Mediated Communication 16: 71–92.

11.

Dai

Brady

(2019) Exploring invisible disability disclosure in the sharing economy. In: The 21st international ACM SIGACCESS conference on computers and accessibility, Pittsburgh, PA, 24 October, pp. 624–626. New York: ACM.

12.

DiMaggio

Hargittai

(2001) From the “digital divide” to “digital inequality”: studying internet use as penetration increases. Princeton: Center for Arts and Cultural Policy Studies, Woodrow Wilson School, Princeton University 4(1): 4–2.

13.

Doleac

Stein

LCD

(2013) The visible hand: race and online market outcomes. The Economic Journal 123(572): F469–F492.

14.

Duffy

(2017) (Not) Getting Paid to Do What You Love: Gender, Social Media, and Aspirational Work. New Haven, CT; London: Yale University Press.

15.

Eichhorn

Jürss

Hoffmann

(2020) Dimensions of digital inequality in the sharing economy. Information, Communication & Society 25: 395–412.

16.

Fiers

Shaw

Hargittai

(2021) Generous attitudes and online participation. Journal of Quantitative Description: Digital Media 1. DOI: 10.51685/jqd.2021.008.

17.

Foong

Vincent

Hecht

, et al. (2018) Women (still) ask for less: gender differences in hourly rate in an online labor marketplace. Proceedings of the ACM on Human-Computer Interaction 2(CSCW): 1–21.

18.

Galperin

(2019) “This Gig is not for women”: gender stereotyping in online hiring. Social Science Computer Review 39: 1089–1107.

19.

Gray

Suri

(2019) Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Boston, MA: Houghton Mifflin Harcourt. Available at: https://ghostwork.info/ (accessed 15 September 2019).

20.

Greenwood

Adjerid

Angst

, et al. (2020) How unbecoming of you: online experiments uncovering gender biases in perceptions of ridesharing performance. Journal of Business Ethics 175: 499–518.

21.

Hannák

Wagner

Garcia

, et al. (2017) Bias in online freelance marketplaces: evidence from Taskrabbit and Fiverr. In: Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing, pp. 1914–1933. Available at: https://personalization.ccs.neu.edu/static/pdf/hannak-cscw17.pdf

22.

Hargittai

Shaw

(2020) Comparing internet experiences and prosociality in Amazon Mechanical Turk and population-based survey samples. Socius. Epub ahead of print 9 March. DOI: 10.1177/2378023119889834.

23.

Hargittai

(ed.) (2021) Handbook of Digital Inequality. Cheltenham; Northampton, MA: Edward Elgar Publishing.

24.

Helsper

(2021) The Digital Disconnect: The Social Causes and Consequences of Digital Inequalities. 1st ed. Los Angeles, CA; London: SAGE.

25.

Hoang

Blank

Quan-Haase

(2020) The winners and the losers of the platform economy: who participates? Information, Communication & Society 23(5): 681–700.

26.

Jahanbakhsh

Cranshaw

Counts

, et al. (2020) An experimental study of bias in platform worker ratings: the role of performance quality and gender. In: Proceedings of the 2020 CHI conference on human factors in computing systems, New York, 25–30 April, pp. 1–13. New York: ACM.

27.

Kässi

Lehdonvirta

(2018) Online labour index: measuring the online gig economy for policy and research. Technological Forecasting and Social Change 137: 241–248.

28.

Kässi

Lehdonvirta

Dalle

J-M

(2019) Workers’ task choice heuristics as a source of emergent structure in digital microwork. Available at: https://osf.io/preprints/socarxiv/ngy46/

29.

Kittur

Nickerson

Bernstein

, et al. (2013) The future of crowd work. In: Proceedings of the 2013 conference on computer supported cooperative work (CSCW’13), New York, February, pp. 1301–1318. New York: ACM.

30.

Kowalczyk-Anioł

Łaszkiewicz

Warwas

(2021) Is the sharing economy inclusive? The age-related segmentation of Polish inhabitants from the perspective of the sharing economy in tourism. Innovation: The European Journal of Social Science Research 0(0): 1–21.

31.

Lehdonvirta

(2018) Flexibility in the gig economy: managing time on three online piecework platforms. New Technology, Work and Employment 33(1): 13–29.

32.

Lehdonvirta

Barnard

Graham

, et al. (2014) Online labour markets-levelling the playing field for international service markets? Available at: https://ora.ox.ac.uk/objects/uuid:5c2b1fc7-04c0-40d1-89a0-f9a28ffb4891

33.

Leung

Koppman

(2018) Taking a pass: how proportional prejudice and decisions not to hire reproduce gender segregation. American Journal of Sociology 124(3): 762–813.

34.

Liang

Hong

(2018) Home bias in hiring: evidence from an online labor market. In: Pacific Asia conference on information systems (PACIS), 26 June, p. 49. Available at: https://aisel.aisnet.org/pacis2018/49

35.

Liang

(2017) Knowledge sharing in online discussion threads: what predicts the ratings? In: Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing (CSCW’17), New York, 25 February, pp. 146–154. New York: ACM.

36.

Liebe

Beyer

(2020) Examining discrimination in everyday life: a stated choice experiment on racism in the sharing economy. Journal of Ethnic and Migration Studies 47: 2065–2088.

37.

Litman

Robinson

Rosen

, et al. (2020) The persistence of pay inequality: the gender pay gap in an anonymous online labor market. PLoS ONE 15(2): e0229383.

38.

Liu

(2019) Adoption of ride-sharing apps by Chinese taxi drivers and its implication for the equality and wellbeing in the sharing economy. Chinese Journal of Communication 12(1): 7–24.

39.

Marchenko

(2019) The impact of host race and gender on prices on Airbnb. Journal of Housing Economics 46: 101635.

40.

Mashhadi

Chapman

(2018) Who gets the lion’s share in the sharing economy: a case study of social inequality in AirBnB. In: Staab

Koltsova

Ignatov

(eds) Social Informatics (Lecture Notes in Computer Science). Cham: Springer, pp. 370–385.

41.

Moher

Shamseer

Clarke

, et al. (2015) Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews 4(1): 1.

42.

Munn

Peters

MDJ

Stern

, et al. (2018) Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology 18(1): 143.

43.

Newlands

Lutz

(2020) Crowdwork and the mobile underclass: barriers to participation in India and the United States. New Media & Society 23: 1341–1361.

44.

Noble

(2018) Algorithms of Oppression: How Search Engines Reinforce Racism. 1st ed. New York: NYU Press.

45.

Pager

(2007) The use of field experiments for studies of employment discrimination: contributions, critiques, and directions for the future. The ANNALS of the American Academy of Political and Social Science 609(1): 104–133.

46.

Pew Research Center (2016) Gig Work, Online Selling and Home Sharing. Washington, DC: Pew Research Center. Available at: https://www.pewresearch.org/internet/2016/11/17/gig-work-online-selling-and-home-sharing/ (accessed 23 April 2020).

47.

Poell

Nieborg

Duffy

(2021) Platforms and Cultural Production. 1st ed. Medford, MA: Polity.

48.

Quillian

Pager

Hexel

, et al. (2017) Meta-analysis of field experiments shows no change in racial discrimination in hiring over time. Proceedings of the National Academy of Sciences 114(41): 10870–10875.

49.

Ravenelle

(2017) Sharing economy workers: selling, not sharing. Cambridge Journal of Regions, Economy and Society 10(2): 281–295.

50.

Sannon

Sun

Cosley

(2022) Privacy, surveillance, and power in the Gig economy. In: CHI conference on human factors in computing systems, New Orleans, LA, 29 April–5 May, pp. 1–15. New York: ACM.

51.

Scheerder

van Deursen

van Dijk

(2017) Determinants of Internet skills, uses and outcomes. A systematic review of the second- and third-level digital divide. Telematics and Informatics 34(8): 1607–1624.

52.

Schor

(2017) Does the sharing economy increase inequality within the eighty percent? findings from a qualitative study of platform providers. Cambridge Journal of Regions, Economy and Society 10(2): 263–279.

53.

Schor

(2020) After the Gig: How the Sharing Economy Got Hijacked and How to Win It Back. 1st ed. Oakland, CA: University of California Press.

54.

Schor

Attwood-Charles

(2017) The “sharing” economy: labor, inequality, and social connection on for-profit platforms. Sociology Compass 11(8): 12493.

55.

Shaw

Hargittai

(2018) The pipeline of online participation inequalities: the case of Wikipedia editing. Journal of Communication 68(1): 143–168.

56.

Shaw

Fiers

Hargittai

(2022) Participation inequality in the gig economy. Information, Communication & Society. Epub ahead of print 4 March. DOI: 10.1080/1369118X.2022.2085611.

57.

Mattila

(2020) Does gender bias exist? The impact of gender congruity on consumer’s Airbnb booking intention and the mediating role of trust. International Journal of Hospitality Management 89: 102405.

58.

Thebault-Spieker

Kluver

Klein

, et al. (2017) Simulation experiments on (the absence of) ratings bias in reputation systems. Proceedings of the ACM on Human-Computer Interaction 1(CSCW): 1–25.

59.

Tjaden

Schwemmer

Khadjavi

(2018) Ride with me—ethnic discrimination, social markets, and the sharing economy. European Sociological Review 34(4): 418–432.

60.

Vallas

Schor

(2020) What do platforms do? Understanding the Gig economy. Annual Review of Sociology 46: 273–294.

61.

van Deursen

van Dijk

(2018) The first-level digital divide shifts from inequalities in physical access to inequalities in material access. New Media & Society 21(2): 354–375.

62.

Van Doorn

(2017) Platform labor: on the gendered and racialized exploitation of low-income service work in the “on-demand” economy. Information, Communication & Society 20(6): 898–914.

63.

Wang

Gilheany

(2015) The model minority? Not on Airbnb.com: a hedonic pricing model to quantify racial bias against Asian Americans. Technology Science. Available at: https://techscience.org/a/2015090104/ (accessed 31 October 2021).

64.

Wood

Graham

Lehdonvirta

, et al. (2019) Good gig, bad gig: autonomy and algorithmic control in the global gig economy. Work, Employment and Society 33(1): 56–75.

65.

Wood

Lehdonvirta

Graham

(2018) Workers of the internet unite? Online freelancer organisation among remote gig economy workers in six Asian and African countries. New Technology, Work and Employment 33(2): 95–112.

66.

Woodcock

Graham

(2020) The Gig Economy: A Critical Introduction. 1st ed. Cambridge; Medford, MA: Polity.

67.

Zyskowski

Morris

Bigham

, et al. (2015) Accessible crowdwork? Understanding the value in and challenge of microtask employment for people with disabilities. In: Proceedings of the 18th ACM conference on computer supported cooperative work & social computing, Vancouver, BC, Canada, 28 February, pp. 1682–1693. New York: ACM.