Who Are the Online Extremists Among Us? Sociodemographic Characteristics,Social Networking,and Online Experiences of Those Who Produce Online Hate Materials

Abstract

What are the factors associated with the production of online hate material? Past research has focused on attributes associated with seeing and being targeted by online hate material, but we know surprisingly little about the creators of such material. This study seeks to address this gap in the knowledge, using a random sample of Americans, aged 15–36. Descriptive results indicate that nearly one-fifth of our sample reported producing online material that others would likely interpret as hateful or degrading. We utilize a logistic regression to understand more about these individuals. Results indicate that men are significantly more likely than women to produce online hate material. This fits with the broader pattern of men being more apt to engage in deviant and criminal behaviors, both online and offline. Other results show that the use of particular social networking sites, such as Reddit, Tumblr, and general messaging boards, is positively related to the dissemination of hate material online. Counter to expectations, the use of first-person shooter games actually decreased the likelihood of producing hate material online. This could suggest that violent videogames serve as outlet for aggression, and not a precursor. In addition, we find that individuals who are close to an online community, or spend more time in areas populated by hate, are more inclined to produce hate material. We expected that spending more time online would correlate with the production of hate, but this turned out not to be true. In fact, spending more time online actually reduces the likelihood of doing so. This result could indicate that individuals who spend more time online are focused on a particular set of tasks, as opposed to using the Internet to disseminate hate.

Introduction

The number of people using social media has increased dramatically recently (Perrin 2015), so too has the time they spend on social media. In fact, a recent report indicates that the average American will spend over 5 years of their life on social media sites (Asano 2017). The virtues of social media are many, as it provides outlets for people to share ideas, engage in debates, access new information, and bridge communication gaps that would otherwise be hopelessly severed. Unfortunately, its pitfalls are equally copious; social media can monopolize time, facilitate the spread of disinformation, and breed content that degrades people based on their race, nationality, gender, or sexual orientation, among others. Indeed, with few barriers to entry, and little effective editorial control, social media sites allow for the easy transmission of hate speech. In turn, a growing number of individuals are being exposed to online hate (Costello et al. 2016a; Hawdon et al. 2014), often with deleterious results.

Hate speech has the potential to cause emotional and psychological harm to those who encounter it (Keipi et al. 2017; Tynes 2006; Tynes et al. 2004), and much like hate crimes, can broaden social fissures (Cowan and Mettrick 2002; Foxman and Wolf 2013; Näsi et al. 2015). In rare cases, hate speech can spur violence (Federal Bureau of Investigation 2011; Freilich et al. 2011; The New America Foundation International Security Program 2015). For instance, the shooting at Columbine High School in 1999, which claimed 13 lives, was largely planned online. The perpetrators, Eric Harris and Dylan Kleibold, used the Internet to plot their attack and communicate in chatrooms with individuals who shared their extreme views. More recently, Wade Michael Page fatally shot six and wounded four in an attack at a gurdwara in Wisconsin in 2012, Dylann Roof murdered nine people in a Charleston Church in 2016, and Jeremy Joseph Christian stabbed three men, two fatally, on a Portland train in 2017. While the motives for these attacks were unique, each perpetrator had a digital footprint. All of the perpetrators utilized social media to espouse hateful rhetoric and communicate with others who held similarly radical beliefs.

Barbarous acts of violence, like those noted above, highlight the need to understand online hate and extremism. Those who are radicalized online are typically not only passive consumers of hate material but also active producers of it as well. Also, while the overwhelming majority of those who trumpet hate online will never engage in violence, we know that some will, and they may also influence others to do so. It is therefore paramount to learn about who these people are. This study seeks to do so, using survey data of youths and young adults to assess sociodemographic traits and online routines associated with the dissemination of online hate material. Before turning to our analysis, we begin with an overview of online hate.

Online hate

Hate groups quickly recognized the value of establishing an online presence. The white supremacist group Stormfront was at the vanguard of the Internet revolution, going public in 1994 (Daniels 2008), and growing to an estimated 300,000 members by 2017 (Southern Poverty Law Center 2017). A barrage of hate groups followed, and at least 10,000 hate groups operate online today (Potok 2015). This figure is dwarfed, although, by the number of individuals—unconnected or loosely affiliated with hate groups—who espouse hate on social media sites, in the comments sections of online articles and videos, or on their own personal webpages.

The Internet is an attractive conduit for hate groups for myriad reasons. Namely, it is quicker and more efficient than previous methods of propagandizing, and it targets a broader audience that may not be familiar with the message being advertised. Underscoring the importance of the Internet to hate groups, Thomas Robb, the national Director of the Ku Klux Klan, stated that “we don't really need the media any more…the only thing we need is the Internet” (Garland 2008).

Online hate takes many forms, spewing malicious content at an array of targets. Even so, the Internet is currently dominated by far-right extremism (Hawdon et al. 2014; Potok 2015; Ratliff et al. 2015). Rightwing hate groups commonly tout beliefs rooted in white supremacy, racial purity, and male dominance. Groups seen as threatening to this ideology, such as race/ethnic minorities, immigrants, Muslims, Arabs, Jews, feminists, homosexuals, the federal government, and political liberals, are common targets of rightwing hate.

The motivations for expressing online hate remain understudied; however, it is theorized that poor macroeconomic conditions and a paucity of economic resources are antecedents to hate (Brown 1995; Kolb 1988; Turner 1997). Results concerning economic threats and hate crimes, however, are mixed (Green et al. 1998a; Green et al. 1998b, 1999). Other researchers suggest that hate often follows broad social and cultural changes that threaten the identities and values of individuals who feel that their way of life is under attack (Bettencourt et al. 2001; Huddy 2003). Still others posit that a person's online habits can encourage hateful activity. As the Internet increasingly creates personalized experiences, individuals are fed information that reifies their existing beliefs. This can create a filter bubble (Pariser 2011), whereby a user's virtual world becomes progressively narrower. In turn, this can increase the risk of exposure to extremist ideologies, which are likely precursors to disseminating such views (Costello et al. 2016a; Hawdon 2012).

Materials and Methods

Because we have a binary dependent variable, we examine factors related to producing online hate material by conducting a logistic regression. The data are taken from a sample of 900 Internet users between the ages of 15 to 36. The data were collected during the week of November 21, 2016, from demographically balanced panels of people who agreed to participate in research surveys. Survey Sample International (SSI) recruited potential participants through permission-based techniques such as random digit dialing and banner advertisements. SSI sent email invitations to a sample of panel members stratified to reflect the U.S. population on age, gender, and region. Only those aged 15–36 are included in the sample. Demographically balanced panels protect against bias in online surveys (see Evans and Mathur 2005; Wansink 2001), and similar samples have been used in numerous studies (e.g., Costello et al. 2016a; Costello et al. 2016b; Näsi et al. 2014; Näsi et al. 2015; Räsänen et al. 2016).

Dependent variable

Our dependent variable asked respondents, “Have you ever produced online material that other people would likely interpret as hateful or degrading?” A majority of respondents reported that they have not (80.2%), yet, approximately one-fifth (19.8%) said they have produced such material.

Independent variables

Sociodemographic characteristics

We control for several sociodemographic traits in this analysis, including sex, age, race/ethnicity, educational attainment, religious faith, political ideology, and economic engagement. Because rightwing extremism is the dominant form of online hate, a prototypical image of who produces such material is someone whose identity and beliefs align with the messaging of rightwing hate. More precisely, the image is that of a young, white, Christian, politically conservative, uneducated man facing economic challenges (see, for example, Eichenwald 2016; Plucinska 2015; Schuman and Krysan 1996). We examine the accuracy of that representation.

A slight majority of our sample is male (52.8%), an overwhelming majority is white (82.4%), the average respondent is 25 years old, and most report being Christian (63.5%). We measure political ideology using a 7-point scale, ranging from 1, “extremely liberal,” to 7, “extremely conservative.” A third of respondents reported being liberal or extremely liberal (33%), and one-fifth responded that they are conservative or very conservative (20.3%). Over half of respondents completed at least some college (52%). Finally, we include a measure of economic engagement that categorizes individuals who are in school or working full time as economically engaged, and those who are unemployed or only working part time as not economically engaged. Three-quarters of our sample qualifies as economically engaged (75.3%).

Online routines

In addition to sociodemographic characteristics, how individuals use the Internet may influence the likelihood of them producing online hate materials. As such, we examine their usage of social networking sites (SNS), time spent online, online social bonds, and whether they have been personally targeted by online hate material.

Because hate is commonly trafficked on SNS, we measure SNS usage with a series of dummy variables, asking respondents to indicate if they used various SNS in the past 3 months. We control for 11 of the most common responses in this analysis, including Facebook (90%), YouTube (86.7%), Instagram (73.9%), Twitter (58.2%), Snapchat (52.7%), Google+ (35.1%), first-person shooters (27.6%), Tumblr (23.2%), Reddit (12.1%), and general message boards (12.1%).

Controlling for time online is important because spending more time online may increase the likelihood of producing hateful material, simply by virtue of having more opportunities to do so. We measure time online by asking respondents to report the time per day they spend online. The variable ranges from 1, “less than one hour per day,” to 6, “ten or more hours per day.” Respondents spend between 3 and 5 h online per day, on average.

We assess online social bonds by asking respondents how close they feel to an online community to which they belong. Possible responses range from 1, “not at all close,” to 5, “very close.” Over 28% of respondents report a high level of closeness and 27.6% report a moderated level of closeness. One-fifth of respondents (19.9%) report they are “very close” to an online community, while only 10% said they are “not at all close” to such a community. We expect individuals with stronger ties to an online community to be more inclined to produce online hate. Merchants of hate can be emboldened when they are part of a group that shares and normalizes their views (Hawdon 2012; Watts 2001).

Finally, we include a measure asking respondents if they have ever been the target of hateful or degrading material online. Nearly one-third (30.7%) of our sample said they have been targeted. We expect those who have been targeted by hate to be more likely to produce hate because of the long-noted relationship between being victimized online and victimizing others (e.g., Bossler and Holt 2009; Bossler et al. 2012; Costello et al. 2016; Marcum et al. 2014).

Results

Descriptive statistics for all variables are displayed in Table 1. A correlation matrix^* suggests a lack of concern with multicollinearity, although, not surprisingly, age and education are positively correlated (0.68). A variance inflation factor (VIF) test, however, revealed a mean VIF of 1.39 and indicated that multicollinearity was not problematic.

Table 1.

Descriptive Statistics of All Variables

Variable	Mean or %	Standard deviation	Minimum value	Maximum value
Produce hate online = 1	20%	0.40	0	1
Male = 1	50%	0.50	0	1
Age	24.7	6.80	15	36
White = 1	82%	0.38	0	1
Black = 1	11%	0.31	0	1
Christian = 1	64%	0.48	0	1
Education	3.2	1.38	1	5
Political ideology	3.7	1.76	1	7
Economically engaged = 1	75%	0.43	0	1
Hours/day online	3.8	1.40	1	6
Use Facebook = 1	90%	0.30	0	1
Use YouTube = 1	87%	0.34	0	1
Use Instagram = 1	74%	0.44	0	1
Use Twitter = 1	58%	0.49	0	1
Use Snapchat = 1	53%	0.50	0	1
Use Google+ = 1	35%	0.48	0	1
Use first-person shooters = 1	28%	0.45	0	1
Use Tumblr = 1	23%	0.42	0	1
Use role-playing games = 1	21%	0.41	0	1
Use general message boards = 1	12%	0.33	0	1
Use Reddit = 1	12%	0.33	0	1
Close to online community	3.3	1.23	1	5
Been target of hate online = 1	31%	0.46	0	1

Table 2 shows the results of the logistic regression of producing online hate material. We found limited support for our hypotheses concerning sociodemographic characteristics. In line with expectations, men were 1.76 times more likely than women to produce hate material online (odds ratio [OR] = 1.76, p < 0.01). The remaining variables, however, were nonsignificant.

Table 2.

Logistic Regression Analysis of Producing Hate Material Online (Odds Ratios and Standard Errors)

Produce hate online	Odds ratio	Standard error
Male = 1	1.76^**	0.43
Age	1.00	0.02
White = 1	0.75	0.31
Black = 1	1.00	0.53
Christian = 1	1.40	0.35
Education	1.08	0.12
Political ideology	1.00	0.06
Economically engaged = 1	1.09	0.29
Hours/day online	0.78^**	0.06
Use Facebook = 1	1.03	0.43
Use YouTube = 1	0.87	0.29
Use Instagram = 1	0.99	0.32
Use Twitter = 1	1.05	0.27
Use Snapchat = 1	1.19	0.30
Use Google+ = 1	1.02	0.25
Use first-person shooters = 1	0.35^***	0.11
Use Tumblr = 1	0.57^*	0.17
Use role-playing games = 1	1.25	0.41
Use general message boards = 1	2.19^*	0.74
Use Reddit = 1	2.87^**	1.04
Close to online community	1.62^***	0.18
Been target of hate online = 1	8.48^***	1.94
LR X²	218.85
Log pseudolikelihood	278.41
N	769

p < 0.05; ^**p < 0.01; ^***p < 0.001 (two-tailed tests).

Next, we found mixed support for our hypotheses regarding Internet usage patterns and the production of online hate. First, users of Reddit (OR = 287, p < 0.01) and general message boards (OR = 2.19, p < 0.05) were both more than twice as likely to produce hate, compared to nonusers. Interestingly, first-person shooter players (OR = 0.36, p < 0.001) and Tumblr users (OR = 0.57, p < 0.05) were less likely to produce online hate. We did not find a significant relationship between the use of other SNS and the production of hate.

Furthermore, we found that, contrary to expectations, spending more time online was related to a decreased likelihood of producing online hate (OR = 0.74, p < 0.001); however, as predicted, being close to an online community was associated with producing online hate (OR = 1.62, p < 0.001). Likewise, those who have been targeted by hate online were more than eight times likely to be purveyors of online hate than those who were not targeted (8.48, p < 0.001).

Discussion

We set out to uncover correlates of producing online hate material. Understanding who produces online hate is imperative, as online hate material as well as violent crimes with a digital element are both increasing. While these trends are not necessarily linked, and most online hate merchants will never engage in offline violence, there is growing concern over online radicalization. So much so, in fact, that governments around the world are investing large sums of money into programs aimed at online deradicalization.

Of particular note, we found little support for our hypotheses regarding sociodemographic characteristics and the production of hate. In fact, the only significant finding was that men are more likely than women to produce online hate. Evidence suggests that women may be more involved in supporting and encouraging violent extremism than is often assumed, but we did not find this to be the case. Indeed, the complex role that women play in radicalization and their involvement in violent extremism are often underestimated (see Carter 2013; Poloni-Staudinger and Ortbalas 2014), but we find that men are more likely to be involved in producing online hate. This fits with a larger pattern of men committing criminal acts—ranging from petty street crime to acts of terrorism—far more frequently than women. In addition, rightwing hate material not only routinely takes aim at women, particularly feminists, who are viewed as the naturally subordinate sex, but also potentially threatens to the ascendency of men. The lack of additional significant findings regarding sociodemographic traits suggests that the stereotypical image of online hate purveyors is perhaps miscast.

Several online activities were related to producing hate material. While the use of some of the most highly trafficked SNS sites, such as Facebook, Twitter, and Instagram, were not associated with the production of hate, using message boards and Reddit were. Online message boards often offer users more anonymity than other SNS, and that could be one reason why their use is associated with producing hate. The anonymous nature of such forums allows users to openly express extreme views that might be perceived as socially unacceptable in other online forums populated by friends and family. Reddit, rightly or wrongly, has a reputation of being a cesspool for hate (Abbruzzese 2014). Reddit allows community members to share news and engage in discussions, quarantining similar topics in subreddits. Moderators, who often lack the ability or desire to effectively monitor their subreddits, are left in charge of monitoring individual subreddits. This makes effectively policing Reddit challenging. Even Reddit's cofounder, Steve Huffam, has recognized Reddit's problem with hate. When asked about the controversy over a subreddit forum mocking black people, Huffman lamented the tightrope walk mandated by the desire to maintain the site's free speech principles, while also noting the need to banish particularly loathsome content (Auerbach 2015). Conversely, Tumblr users were less likely to produce hate. Typically described as MySpace with blogging capabilities, Tumblr does not have the same reputation for encouraging hateful content as some other SNS, such as Reddit. Even so, hate certainly exists on Tumblr. It is unclear why users of Tumblr are less likely to produce hate, although it might be partially attributable to the particular demographic that most use Tumblr or the effectiveness of Tumblr's central monitoring.

One of our more unexpected findings was that the use of first-person shooters was inversely related to hate production. Given the violent nature of such platforms, we anticipated that individuals partaking in these types of games might be more disposed to violence, including violent speech. This, however, was not the case. Perhaps this indicates that first-person shooter games can serve as outlets for aggression, rather than an indicator of the desire to project one's aggression onto others. This finding is in line with others who have found that violent videogames are unrelated to real-world violence and may, in fact, be inversely related to violent behaviors (e.g., Markey et al. 2015).

Equally unexpected was our finding that spending time online was associated with being less likely to produce hate. We hypothesized that time online would be positively associated with spreading hate because of having greater opportunities to do so. To the contrary, although, it may be the case that those who spend more time online are focused on a particular task. As such, goal-oriented people might be less likely to engage in deviant acts (see Hawdon 1999), including producing online hate.

Our finding that being close to an online community correlates positively with producing hate suggests that flocking and feathering might be at work. It is common for likeminded individuals to converge in online spaces that reaffirm their beliefs. In other words, they flock to one another. Feathering, or learning to adopt the attitudes of others, takes place in these group settings. Hence, if an individual surrounds himself or herself online with others who hold hateful views, it becomes more likely that they will adopt, and perhaps even promote, such views (see Costello et al. 2016a for a similar argument).

Finally, targets of hate are more than eight times as likely to produce hate, compared to those who have not been targeted. Past work demonstrates that there is a link between seeing hate online and being targeted by it (Costello et al. 2017), as those who frequent virtual spaces where hate resides not only run the risk of encountering it but also bear the brunt of it. We suspect the same is true for the production of hate. Hate merchants likely spend more time in online forums where hate abounds and is accepted, upping their chances of inevitably being targeted.

Conclusion

This analysis found that online habits, more so than sociodemographic traits, are important correlates of producing online hate. We believe that this represents a vital addition to the existing literature on this topic. It is important to note, although, that our analysis is exploratory in nature and only an initial step in the process of understanding who disseminates hate online. We therefore encourage future researchers to expand on this work, examining the particular features of various SNS that might be more or less amenable to hate material. In addition, more investigation is needed into other online activities that might correlate with producing hate material online. Finally, while we believe our analysis offers insights into the characteristics of hate producers, we realize that we are likely capturing a small fraction of those involved. It would be beneficial for future researchers to use different approaches that could delve deeper into who these people are. For example, virtual ethnographies of hate-material producers could allow us to interrogate their motives and constructed identities.

Footnotes

Acknowledgments

This project was supported by Award No. 2014-ZA-BX-0014, awarded by the National Institute of Justice, Office of Justice Programs, U.S. Department of Justice. The opinions, findings, and conclusions or recommendations expressed in this publication/program/exhibition are those of the author(s) and do not necessarily reflect those of the Department of Justice.

Author Disclosure Statement

No competing financial interests exist.

References

Abbruzzese

. (2014). Hate speech is drowning Reddit and no one can stop it. Mashable http://mashable.com/2014/10/26/reddit-hate-speech-moderation/#McAZi.gdoGqr (accessed July 24, 2017).

Asano

. (2017). How much time do people spend on social media (Infographic). Social Media Today. www.socialmediatoday.com/marketing/how-much-time-do-people-spend-social-media-infographic (accessed July 24, 2017).

Auerbach

. (2015). How to Detox Reddit: Don't ban hate speech on the site. Put it in super quarantine. www.slate.com/articles/technology/bitwise/2015/07/hate_speech_on_reddit_a_simple_novel_plan_to_quarantine_it.html (accessed July 24, 2017).

Bettencourt

, Dorr

, Charlton

, Hume

. (2001). Status differences and in-group bias: A meta-analytic examination of the effects of status stability, status legitimacy, and group permeability. Psychol Bull. 127, 520–542.

Bossler

, Holt

. (2009). On-line activities, guardianship, and malware infection: An examination of routine activities theory. Int J Cyber Criminol. 3, 400–420.

Bossler

, Holt

, May

. (2012). Predicting online harassment: Victimization among a juvenile population. Youth Soc. 44, 500–523.

Brown

. (1995). Prejudice: Its Social Psychology. (Blackwell, Oxford, UK).

Carter

. (2013). Women and violent extremism. GSDRC Helpdesk research report. www.gsdrc.org/docs/open/hdq898.pdf (accessed July 24, 2017).

Costello

, Hawdon

, Cross

. (2017). Virtually standing up or standing by? Correlates of enacting social control online. Int J Criminol Sociol. 6, 16–28.

10.

Costello

, Hawdon

, Ratliff

, Grantham

. (2016a). Who views online extremism? Individual attributes leading to exposure. Comput Hum Behav. 63, 311–320.

11.

Costello

, Hawdon

, Ratliff

. (2016b). Confronting online extremism: The effect of self-help, collective efficacy, and guardianship on being a target for hate speech. Soc Sci Comput Rev. 35, 587–605.

12.

Cowan

, Mettrick

. (2002). The effects of target variables and setting on perceptions of hate speech1. J Appl Soc Psychol. 32, 277–299.

13.

Daniels

. (2008). Race, civil rights, and hate speech in the digital era. In Learning Race and Ethnicity: Youth and Digital Media. Everett

, ed. The John D. and Catherine T. MacArthur Foundation Series on Digital Media and Learning. (The MIT Press, Cambridge, MA) pp. 129–154. DOI: 10.1162/dmal.9780262550673.129.

14.

Eichenwald

. (2016). Right-wing extremists are a bigger threat to America than ISIS. Newsweek www.newsweek.com/2016/02/12/right-wing-extremists-militants-bigger-threat-america-isis-jihadists-422743.html (accessed July 24, 2017).

15.

Evans

, Mathur

. (2005). The value of online surveys. Internet Res. 15, 195–219.

16.

Federal Bureau of Investigation. (2011). Domestic terrorism: Focus on militia extremism. www.fbi.gov/news/stories/2011/september/militia_092211 (accessed July 24, 2017).

17.

Foxman

, Wolf

. (2013). Viral Hate: Containing Its Spread on the Internet. New York, NY: St. Martin's Press (Macmillan).

18.

Freilich

, Belli

, Chermak

. (2011). United States extremist crime database. Retrieved from www.start.umd.edu/research-projects/united-states-extremist-crime-database-ecdb-1990-2010 (accessed July 24, 2017).

19.

Garland

. (2008). Klan's new message of cyber-hate. The New Zealand Herald, World News. Retrieved from http://www.lexisnexis.com/us/lnacademic/search/homesubmitForm.do (accessed July 24, 2017).

20.

Green

, Abelson

, Garnett

. (1999). The Distinctive Political Views of Hate-Crime Perpetrators and White Supremacists. New York, NY: (Russell Sage Foundation).

21.

Green

, Glaser

, Rich

. (1998a). From lynching to gay bashing: The elusive connection between economic conditions and hate crime. J Pers Soc Psychol. 75, 82.

22.

Green

, Strolovitch

, Wong

. (1998b). Defended neighborhoods, integration, and racially motivated crime. Am J Sociol. 104, 372–403.

23.

Hawdon

. (2012). Applying differential association theory to online hate groups: A theoretical statement. J Res Finn Soc. 5, 39–47.

24.

Hawdon

, Oksanen

, Räsänen

. (2014). Victims of online hate groups: American youth's exposure to online hate speech. In The Causes and Consequences of Group Violence: From Bullies to Terrorists. (Lexington Books, London, United Kingdom) pp. 165–182.

25.

Hawdon

. (1999). Daily routines and crime: Using routine activities as measures of Hirschi's involvement. Youth Soc. 30, 395–415.

26.

Huddy

. (2003). Group membership, ingroup loyalty, and political cohesion. In Oxford Handbook of Political Psychology. Sears

, Huddy

, Jervis

, eds. (Oxford University Press, New York), pp. 511–558.

27.

Keipi

, Kaakinen

, Oksanen

, Räsänen

. (2017). Social tie strength and online victimization: An analysis of young people aged 15–30 years in four nations. Social Media+ Society. 3, 2056305117690013.

28.

Kolb

. (1988). The Weimar Republic. Abingdon, Oxon: (Psychology Press).

29.

Marcum

, Higgins

, Freiburger

, Ricketts

. (2014). Exploration of the cyberbullying victim/offender overlap by sex. Am J Crim Justice. 39, 538–548.

30.

Markey

, Markey

, French

. (2015). Violent video games and real-world violence: Rhetoric versus data. Psychol Pop Media Cult. 4, 277–295.

31.

Näsi

, Räsänen

, Hawdon

, et al. (2015). Exposure to online hate material and social trust among Finnish youth. Inform Technol People. 28, 607–622.

32.

Näsi

, Räsänen

, Oksanen

, et al. (2014). Association between online harassment and exposure to harmful online content: A cross-national comparison between the United States and Finland. Comput Hum Behav. 41, 137–145.

33.

Pariser

. (2011). The Filter Bubble: What the Internet Is Hiding from You. (Penguin, UK).

34.

Perrin

. (2015). Social Media Usage. Washington, DC: Pew Research Center.

35.

Plucinska

. (2015). Study says white extremists have killed more Americans in the U.S. than Jihadists since 9/11. Time http://time.com/3934980/right-wing-extremists-white-terrorism-islamist-jihadi-dangerous (accessed July 24, 2017).

36.

Poloni-Staudinger

, Ortbals

. (2014). Terrorism and Violent Conflict: Women's Agency, Leadership, and Responses, vol. 8. New York, NY: Springer Science & Business Media.

37.

Potok

. (2015). The year in hate and extremism. Retrieved from www.splcenter.org/fighting-hate/intelliegence-report/2015/year-hate-and-extremism (accessed July 24, 2017).

38.

Räsänen

, Hawdon

, Holkeri

, et al. (2016). Targets of online hate: Examining determinants of victimization among young Finnish Facebook users. Violence Vict. 31, 708–726.

39.

Ratliff

, Hawdon

, Middleton

, et al. (2015). Domestic extremism in the U.S. 1960–2015. In Presented at the Annual Academy of Criminal Justice Sciences Meeting, March 4, Orlando: FL. Regular Session.

40.

Schuman

, Krysan

. (1996). A study of far right resentment in America. Int J Public Opin Res. 8, 10–30.

41.

The New America Foundation International Security Program. (2015). Homegrown Extremists. Retrieved from http://securitydata.newamerica.net/extremists/deadly-attacks.html (accessed July 24, 2017).

42.

The Southern Poverty Law Center. (2017). Retrieved from www.splcenter.org/fighting-hate/extremist-files/group/stormfront (accessed July 24, 2017).

43.

Turner

. 1997. Hitler's Thirty Days to Power: January 1933. New York, NY: (Basic Books).

44.

Tynes

. (2006). Children, adolescents, and the culture of hate online. In Handbook of Children, Culture, and Violence. Dowd

, Singer

, Wilson

, eds. (Sage, New York) pp. 267–289.

45.

Tynes

, Reynolds

, Greenfield

. (2004). Adolescence, race, and ethnicity on the Internet: A comparison of discourse in monitored vs. unmonitored chat rooms. J Appl Dev Psychol. 25, 667–684.

46.

Wansink

. (2001). Editorial: The Power of Panels. J Database Mark Cust Strategy Manag. 8, 190–194.

47.

Watts

. (2001). Aggressive youth cultures and hate crime: Skinheads and xenophobic youth in Germany. Am Behav Sci. 45, 600–615.