Exploring a Language-Based Interest Assessment: Predicting Vocational Interests on Social Media Using Natural Language Processing

Abstract

For more than a century, self-report inventories have been the traditional method for assessing vocational interests. Little research has examined the use of machine learning techniques, such as natural language processing (NLP), in interest assessment. This paper explores the extent to which natural language on social media can be used to predict individuals’ self-ratings on eight basic interests representing the SETPOINT model: Agriculture, Engineering, Human Resources, Life Science, Management/Administration, Mechanics/Electronics, Media, and Social Science. Leveraging closed- (Linguistic Inquiry and Word Count; LIWC) and open-vocabulary NLP approaches (Latent Dirichlet allocation (LDA) topic modeling), we analyzed 3.2 million Facebook posts from 2,834 participants who completed a 32-item basic interest measure. We found that the convergent validities of these NLP approaches for predicting vocational interest scores (LIWC: $\bar{r}$ = 0.19; LDA topic modeling: $\bar{r}$ = 0.24) are comparable to prior research on language-based personality assessments. Our study also revealed largely face-valid language markers that characterize different basic interests. Implications for developing language-based interest assessments for applied settings (e.g., career guidance and employee selection) and future research directions are discussed.

Keywords

natural language processing machine learning vocational interests basic interests measurement language-based assessment social media text data

Get full access to this article

View all access options for this article.

References

Abdurahman

Zou

Ungar

Bhatia

(2024). A deep learning approach to personality assessment: Generalizing across items and expanding the reach of survey-based research. Journal of Personality and Social Psychology, 126(2), 312–331. https://doi.org/10.1037/pspp0000480

Azucar

Marengo

Settanni

(2018). Predicting the big 5 personality traits from digital footprints on social media: A meta-analysis. Personality and Individual Differences, 124, 150–159. https://doi.org/10.1016/j.paid.2017.12.018

Bai

Liao

H. Y.

(2019). The relation between interest congruence and college major satisfaction: Evidence from the basic interest measures. Journal of Career Assessment, 27(4), 628-644. https://doi.org/10.1177/1069072718793966, https://doi/10.1177/1069072718793966

Benjamini

Hochberg

(1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x

Blei

D. M.

(2012). Probabilistic topic models. Communications of the ACM, 55(4), 77–84. https://doi.org/10.1145/2133806.2133826

Campion

M. A.

Campion

E. D.

(2023). Machine learning applications to personnel selection: Current illustrations, lessons learned, and future research. Personnel Psychology, 76(4), 993–1009. https://doi.org/10.1111/peps.12621

Campion

E. D.

Campion

M. A.

Johnson

Carretta

T. R.

Romay

Dirr

Deregla

Mouton

(2024). Using natural language processing to increase prediction and reduce subgroup differences in personnel selection decisions. Journal of Applied Psychology, 109(3), 307–338. https://doi.org/10.1037/apl0001144

Chu

Russell

M. T.

Hoff

K. A.

Jonathan Phan

W. M.

Rounds

(2022). What do interest inventories measure? The convergence and content validity of four RIASEC inventories. Journal of Career Assessment, 30(4), 776–801. https://doi.org/10.1177/10690727221081554

Connelly

B. S.

Ones

D. S.

(2010). An other perspective on personality: Meta-analytic integration of observers’ accuracy and predictive validity. Psychological Bulletin, 136(6), 1092–1122. https://doi.org/10.1037/a0021212

10.

Devlin

Chang

M. W.

Lee

Toutanova

(2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv. https://arxiv.org/abs/1810.04805

11.

Eichstaedt

J. C.

Kern

M. L.

Yaden

D. B.

Schwartz

H. A.

Giorgi

Park

Hagan

C. A.

Tobolsky

V. A.

Smith

L. K.

Buffone

Iwry

Seligman

M. E. P.

Ungar

L. H.

(2021). Closed- and open-vocabulary approaches to text analysis: A review, quantitative comparison, and recommendations. Psychological Methods, 26(4), 398–427. https://doi.org/10.1037/met0000349

12.

Fan

Sun

Liu

Zhao

Zhang

Chen

Glorioso

Hack

(2023). How well can an AI chatbot infer personality? Examining psychometric properties of machine-inferred personality scores. Journal of Applied Psychology, 108(8), 1277–1299. https://doi.org/10.1037/apl0001082

13.

Ganesan

A. V.

Matero

Ravula

A. R.

Schwartz

H. A.

(2021). Empirical evaluation of pre-trained transformers for human-level NLP: The role of sample size and dimensionality. arXiv. https://arxiv.org/abs/2105.03484

14.

Garcia

Sikström

(2014). The dark side of Facebook: Semantic representations of status updates predict the Dark Triad of personality. Personality and Individual Differences, 67, 92–96. https://doi.org/10.1016/j.paid.2013.10.001

15.

Hanna

Nye

C. D.

Samo

Chu

Hoff

K. A.

Rounds

Oswald

F. L.

(2024). Interests of the future: An integrative review and research agenda for an automated world of work. Journal of Vocational Behavior, 152, Article 104012. https://doi.org/10.1016/j.jvb.2024.104012

16.

Hanna

Rounds

(2020). How accurate are interest inventories? A quantitative review of career choice hit rates. Psychological Bulletin, 146(9), 765–796. https://doi.org/10.1037/bul0000269

17.

Harmon

L. W.

DeWitt

D. W.

Campbell

D. P.

Hansen

J. I. C.

(1994). Strong interest inventory: Applications and technical guide: Form T317 of the strong vocational interest blanks. Stanford University Press.

18.

Hickman

Herde

C. N.

Lievens

Tay

(2023). Automatic scoring of speeded interpersonal assessment center exercises via machine learning: Initial psychometric evidence and practical guidelines. International Journal of Selection and Assessment, 31(2), 225–239. https://doi.org/10.1111/ijsa.12418

19.

Hickman

Saef

Woo

S. E.

Tay

Bosch

(2024). Developing and evaluating language‐based machine learning algorithms for inferring applicant personality in video interviews. Human Resource Management Journal, 34(2), 255–274. https://doi.org/10.1111/1748-8583.12356

20.

Hickman

Tay

Woo

S. E.

(2019). Validity evidence for off-the-shelf language-based personality assessment using video interviews: Convergent and discriminant relationships with self and observer ratings. Personnel Assessment and Decisions, 5(3). Article 3. https://doi.org/10.25035/pad.2019.03.003

21.

Holgate

Cachola

Preoţiuc-Pietro

J. J.

(2018). Why swear? Analyzing and inferring the intentions of vulgar expressions. In Proceedings of the 2018 conference on empirical methods in natural language processing (pp. 4405–4414). Association for Computational Linguistics.

22.

Holland

J. L.

(1997). Making vocational choices: A theory of vocational personalities and work environments (3rd ed.). Psychological Assessment Resources.

23.

Hou

D. X.

Tay

(2024). Measuring SETPOINT vocational interest dimensions: The development and validation of three short scales. Journal of Vocational Behavior, 149, Article 103959. https://doi.org/10.1016/j.jvb.2023.103959

24.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

25.

Jay

(2009). The utility and ubiquity of taboo words. Perspectives on Psychological Science, 4(2), 153–161. https://doi.org/10.1111/j.1745-6924.2009.01115.x

26.

Jones

K. S.

Newman

D. A.

Rounds

(2022). Vocational interests and adverse impact: How attraction and selection on vocational interests relate to adverse impact potential. Journal of Applied Psychology, 107(4), 604–627. https://doi.org/10.1037/apl0000893

27.

Kern

M. L.

Eichstaedt

J. C.

Schwartz

H. A.

Dziurzynski

Ungar

L. H.

Stillwell

D. J.

Kosinski

Ramones

S. M.

Seligman

M. E.

(2014). The online social self: An open vocabulary approach to personality. Assessment, 21(2), 158–169. https://doi.org/10.1177/1073191113514104

28.

Kern

M. L.

McCarthy

P. X.

Chakrabarty

Rizoiu

M. A.

(2019). Social media-predicted personality traits and values can help match people to their ideal jobs. Proceedings of the National Academy of Sciences, 116(52), 26459–26464. https://doi.org/10.1073/pnas.1917942116

29.

Kirkendall

C. D.

Nye

C. D.

Rounds

Drasgow

Chernyshenko

O. S.

Stark

(2020). Adaptive vocational interest diagnostic: Informing and improving the job assignment process. Military Psychology, 32(1), 91–100. https://doi.org/10.1080/08995605.2019.1652480

30.

Kjell

Giorgi

Schwartz

H. A.

(2023). The text-package: An R-package for analyzing and visualizing human language using natural language processing and transformers. Psychological Methods, 28(6), 1478–1498, Advance online publication. https://doi.org/10.1037/met0000542

31.

Kosinski

Matz

S. C.

Gosling

S. D.

Popov

Stillwell

(2015). Facebook as a research tool for the social sciences: Opportunities, challenges, ethical considerations, and practical guidelines. American Psychologist, 70(6), 543–556. https://doi.org/10.1037/a0039210

32.

Koutsoumpis

Oostrom

J. K.

Holtrop

van Breda

Ghassemi

de Vries

R. E.

(2022). The kernel of truth in text-based personality assessment: A meta-analysis of the relations between the big five and the linguistic Inquiry and word count (LIWC). Psychological Bulletin, 148(11–12), 843–868. https://doi.org/10.1037/bul0000381

33.

Liddy

E. D.

(2001). Natural Language processing. In In encyclopedia of library and information science (2nd ed.). Marcel Decker, Inc.

34.

Liu

Ott

Goyal

Joshi

Chen

Levy

Lewis

Zettlemoyer

Stoyanov

(2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv. https://arxiv.org/pdf/1907.11692.pdf

35.

Lynn

Balasubramanian

Schwartz

H. A.

(2020). Hierarchical modeling for user personality prediction: The role of message-level attention. In Jurafsky

Chai

Schuler

Tetreault

(Eds.), ACL-2020: Proceedings of the association for computational linguistics (pp. 5306–5316). Association for Computational Linguistics.

36.

McCallum

A. K.

(2002). Mallet: A machine learning for languagetoolkit. https://mallet.cs.umass.edu

37.

McIntyre

M. M.

(2021). Judging what others enjoy: Desirability and observability of interests. Journal of Career Assessment, 30(3), 557–572. https://doi.org/10.1177/10690727211055862

38.

Moreno

J. D.

Martinez-Huertas

J. A.

Olmos

Jorge-Botana

Botella

(2021). Can personality traits be measured analyzing written language? A meta-analytic study on computational methods. Personality and Individual Differences, 177, Article 110818. https://doi.org/10.1016/j.paid.2021.110818

39.

Mount

M. K.

Barrick

M. R.

Stewart

G. L.

(1998). Five-factor model of personality and performance in jobs involving interpersonal interactions. Human Performance, 11(2), 145–165. https://doi.org/10.1207/s15327043hup1102&3_3

40.

Müller

A. C.

Guido

(2017). Introduction to machine learning with Python: A guide for data scientists. O’Reilly Media Inc.

41.

Nauta

M. M.

(2010). The development, evolution, and status of Holland’s theory of vocational personalities: Reflections and future directions for counseling psychology. Journal of Counseling Psychology, 57(1), 11–22. https://doi.org/10.1037/a0018213

42.

Nayak

(2019). Understanding searches better than ever before. https://blog.google/products/search/search-language-understanding-bert/

43.

Nye

C. D.

(2022). Assessing interests in the twenty-first-century workforce: Building on a century of interest measurement. Annual Review of Organizational Psychology and Organizational Behavior, 9(1), 415–440. https://doi.org/10.1146/annurev-orgpsych-012420-083120

44.

Nye

C. D.

Hoff

K. A.

(2023). Getting interested in interests for employee selection: Key concerns and areas for future research. International Journal of Selection and Assessment, 31(4), 494–499. https://doi.org/10.1111/ijsa.12441

45.

Nye

C. D.

Rounds

Drasgow

(2017). Interest congruence and performance: Revisiting recent meta-analytic findings. Journal of Vocational Behavior, 98, 138–151. https://doi.org/10.1016/j.jvb.2016.11.002

46.

Oswald

F. L.

Behrend

T. S.

Putka

D. J.

Sinar

(2020). Big data in industrial-organizational psychology and human resource management: Forward progress for organizational research and practice. Annual Review of Organizational Psychology and Organizational Behavior, 7, 505–533. https://doi.org/10.1146/annurev-orgpsych-032117-104553

47.

Pang

Eichstaedt

J. C.

Buffone

Slaff

Ruch

Ungar

L. H.

(2020). The language of character strengths: Predicting morally valued traits on social media. Journal of Personality, 88(2), 287–306. https://doi.org/10.1111/jopy.12491

48.

Park

Schwartz

H. A.

Eichstaedt

J. C.

Kern

M. L.

Kosinski

Stillwell

D. J.

Ungar

L. H.

Seligman

M. E. P.

(2015). Automatic personality assessment through social media language. Journal of Personality and Social Psychology, 108(6), 934–952. https://doi.org/10.1037/pspp0000020

49.

Parsons

(1909). Choosing a vocation. Houghton Mifflin.

50.

Pennebaker

J. W.

Boyd

R. L.

Jordan

Blackburn

(2015). The development and psychometric properties of LIWC2015. University of Texas at Austin. https://doi.org/10.15781/T29G6

51.

Putka

D. J.

Oswald

F. L.

Landers

R. N.

Beatty

A. S.

McCloy

R. A.

M. C.

(2023). Evaluating a natural language processing approach to estimating KSA and interest job analysis ratings. Journal of Business and Psychology, 38(2), 385–410. https://doi.org/10.1007/s10869-022-09824-0

52.

Rai

Stade

E. C.

Giorgi

Francisco

Ungar

L. H.

Curtis

Guntuku

S. C.

(2024). Key language markers of depression on social media depend on race. Proceedings of the National Academy of Sciences, 121(14), Article e2319837121. https://doi.org/10.1073/pnas.2319837121

53.

Roemer

Horstmann

K. T.

Ziegler

(2021). Sometimes hot, sometimes not: The relations between selected situational vocational interests and situation perception. European Journal of Personality, 35(2), 212–233. https://doi.org/10.1002/per.2287

54.

Rounds

(2014). The nature and power of interests. Current Directions in Psychological Science, 23(2), 98–103. https://doi.org/10.1177/0963721414522812

55.

Schwartz

H. A.

Eichstaedt

J. C.

Kern

M. L.

Dziurzynski

Ramones

S. M.

Agrawal

Shah

Kosinski

Stillwell

Seligman

M. E. P.

Ungar

L. H.

(2013). Personality, gender, and age in the language of social media: The open-vocabulary approach. PLoS One, 8(9), Article e73791. https://doi.org/10.1371/journal.pone.0073791

56.

Schwartz

H. A.

Giorgi

Sap

Crutchley

Ungar

Eichstaedt

(2017). Dlatk: Differential language analysis toolkit. In Proceedings of the 2017 conference on empirical methods in natural language processing: System demonstrations (pp. 55–60). Association for Computational Linguistics.

57.

Strong

E. K.

Jr. (1943). Vocational interests of men and women. Stanford University Press.

58.

(2020). The three faces of interests: An integrative review of interest research in vocational, organizational, and educational psychology. Journal of Vocational Behavior, 116(Part B), Article 103240. https://doi.org/10.1016/j.jvb.2018.10.016

59.

Tay

Liao

H.-Y.

Zhang

Rounds

(2019). Toward a dimensional model of vocational interests. Journal of Applied Psychology, 104(5), 690–714. https://doi.org/10.1037/apl0000373

60.

Tausczik

Y. R.

Pennebaker

J. W.

(2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24–54. https://doi.org/10.1177/0261927X09351676

61.

Tay

Woo

S. E.

Hickman

Booth

B. M.

D'Mello

(2022). A conceptual framework for investigating and mitigating machine-learning measurement bias (MLMB) in psychological assessment. Advances in Methods and Practices in Psychological Science, 5(1), Article 251524592110613. https://doi.org/10.1177/25152459211061337

62.

Tay

Woo

S. E.

Hickman

Saef

R. M.

(2020). Psychometric and validity issues in machine learning approaches to personality assessment: A focus on social media text mining. European Journal of Personality, 34(5), 826–844. https://doi.org/10.1002/per.2290

63.

Tracey

T. J.

(2020). We can do that? Technological advances in interest assessment. Journal of Career Assessment, 28(1), 3–13. https://doi.org/10.1177/1069072719879910

64.

Van Iddekinge

C. H.

Roth

P. L.

Putka

D. J.

Lanivich

S. E.

(2011). Are you interested? A meta-analysis of relations between vocational interests and employee performance and turnover. Journal of Applied Psychology, 96(6), 1167–1194. https://doi.org/10.1037/a0024343

65.

Abdurahman

Bhatia

Ungar

(2020). Predicting responses to psychological questionnaires from participants’ social media posts and question text embeddings. In Findings of the association for computational linguistics: Emnlp 2020 (pp. 1512–1524). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.findings-emnlp.137

66.

Wee

Newman

D. A.

(2023). Hiring on vocational interests to simultaneously improve validity and organizational diversity. International Journal of Selection and Assessment, 31(4), 504–508. https://doi.org/10.1111/ijsa.12443

67.

Wegmeyer

L. J.

Speer

A. B.

(2022). Understanding, detecting, and deterring faking on interest inventories. International Journal of Selection and Assessment, 30, 562–578. https://doi.org/10.1111/ijsa.12398

68.

Wille

De Fruyt

(2023). The resurrection of vocational interests in human resources research and practice: Evidence, challenges, and a working model. International Journal of Selection and Assessment, 31, 485–493. https://doi.org/10.1111/ijsa.12415

69.

Wolf

Debut

Sanh

Chaumond

Delangue

Moi

Cistac

Rault

Louf

Rush

A. M.

(2019). HuggingFace’s transformers: State-of-the-art natural language processing. arXiv. https://arxiv.org/abs/1910.03771

70.

Woo

S. E.

Tay

Oswald

(2024). Artificial intelligence, machine learning, and big data: Improvements to the science of people at work and applications to practice. Personnel Psychology. Advance online publication. https://doi.org/10.1111/peps.12643

71.

Woo

S. E.

Tay

Proctor

R. W.

(Eds.). (2020). Big data in psychological research. American Psychological Association. https://doi.org/10.1037/0000193-000

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB