Test Performance of Artificial Intelligence in the Chinese Social Work Certification Examination

Abstract

Purpose: The study checked the validity of the entry-level Chinese Social Work Certification Examination in 2024 and tested the social work knowledge of artificial intelligence (AI) by exploring the performance of two systems—ChatGPT 4.0 and ERNIE Bot 4.0. Method: We applied zero-shot learning to the testing models, determined the correctness of their answers based on the reference books, and analyzed their reasons behind. Result: The results show that both AIs passed the exam with scores around 80/100. The AIs answered most questions correctly with adequate reasoning. The reasons for incorrect answers included wrong knowledge, logical fallacies, misunderstanding of question contents, and neglecting the test instruction, which reflected the hallucinations in large-language models. ChatGPT had better reasoning abilities, while ERNIE Bot was more familiar with local contexts and policies. Discussion: The findings support the validity of the exam and also confirm the ability of AI in using social work knowledge.

Keywords

Social work certification exam validity artificial intelligence ChatGPT hallucinations in large-language model

Get full access to this article

View all access options for this article.

References

Albright

D. L.

Thyer

B. A.

(2010). A test of the validity of the LCSW examination: Quis custodiet ipsos custodes? Social Work Research, 34(4), 229–234. https://doi.org/10.1093/swr/34.4.229

Belisle-Pipon

J. C.

(2024). Why we need to be careful with LLMs in medicine. Frontiers in Medicine, 11, Article 1495582. https://doi.org/10.3389/fmed.2024.1495582

China Personnel Testing Authority . (2023a). 社会工作者职业水平考试 [Professional Qualification Examination for Social Workers]. http://www.cpta.com.cn/testCondition/1448.html

China Personnel Testing Authority . (2023b). 社会工作者职业水平考试 [Professional Qualification Examination for Social Workers]. http://www.cpta.com.cn/test/1450.html

Choi

J. H.

Hickman

K. E.

Monahan

A. B.

Schwarcz

(2022). ChatGPT goes to law school. Journal of Legal Education, 71(3), 387–400. https://doi.org/10.2139/ssrn.4335905

Dwivedi

Y. K.

Kshetri

Hughes

Slade

E. L.

Jeyaraj

Kar

A. K.

Baabdullah

A. M.

Koohang

Raghavan

Ahuja

Albanna

Albashrawi

M. A.

Al-Busaidi

A. S.

Balakrishnan

Barlette

Basu

Bose

Brooks

Buhalis

…Wright

(2023). “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management, 71, Article 102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642

Fregly

Barth

Eigenbrode

(2024). 生成式AI入门与AWS实战 [Generative AI on AWS: Building context-aware multimodal reasoning applications] (Generative AI Technology Interest Group, Trans.). The People’s Posts and Telecommunications Press. (Original work published 2023).

Friederichs

W. J.

Marz

(2023). ChatGPT in medical school: how successful is AI in progress testing? Medical Education Online, 28(1). Article 2220920. https://doi.org/10.1080/10872981.2023.2220920

Gao

J. G.

Yan

M. C.

(2015). Social work in the making: The state and social work development in China. International Journal of Social Welfare, 24(1), 93–101. https://doi.org/10.1111/ijsw.12089

10.

Huang

Zhong

Feng

Wang

Liu

(2025). A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2), 1–55. https://doi.org/10.1145/3703155

11.

Kaplan

Haenlein

(2019). Siri, Siri, in my hand: Who’s the fairest in the land? On the interpretations, illustrations, and implications of artificial intelligence. Business Horizons, 62(1), 15–25. https://doi.org/10.1016/j.bushor.2018.08.004

12.

Qin

Zhao

(2025). Effectiveness of China’s social work professional level examination: Based on evaluating professional competence. Asia Pacific Journal of Social Work and Development, 35(1), 37–58. https://doi.org/10.1080/29949769.2024.2326468

13.

Liu

B. Z.

(2025a). The 2025 editions of Social Work Practice (Entry-Level): Real Exam Questions and Detailed Solutions (2018–2024) (1st ed, Vol. 1). East China University of Science and Technology press.

14.

Liu

B. Z.

(2025b). The 2025 editions of Social Work Practice (Entry-Level): Real Exam Questions and Detailed Solutions (2018–2024) (1st ed., Vol. 2). East China University of Science and Technology press.

15.

Markovic

(2024). Current options and limits of digital technologies and artificial intelligence in social work. SHS Web of Conferences, 184, Article 05003. https://doi.org/10.1051/shsconf/202418405003

16.

Rodriguez

M. Y.

Goldkind

Victor

B. G.

Hiltz

Perron

B. E.

(2024). Introducing generative artificial intelligence into the MSW curriculum: A proposal for the 2029 educational policy and accreditation standards. Journal of Social Work Education, 60(2), 174–182. https://doi.org/10.1080/10437797.2024.2340931

17.

Shen

Liu

(2025a). The 2025 editions of Social Work Comprehensive Competence (Entry-Level): Real Exam Questions and Detailed Solutions (2018–2024) (1st ed., Vol. 1). East China University of Science and Technology press.

18.

Shen

Liu

(2025b). The 2025 editions of Social Work Comprehensive Competence (Entry-Level): Real Exam Questions and Detailed Solutions (2018–2024) (1st ed., Vol. 2). East China University of Science and Technology press.

19.

Tan

Wang

Lian

(2025). Professional education or certificate: The factors influencing child maltreatment recognition in China. Research on Social Work Practice, 35(6), 655–670. https://doi.org/10.1177/10497315241269839

20.

Thyer

B. A.

(2011). LCSW Examination pass rates: Implications for social work education. Clinical Social Work Journal, 39, 296–300. https://doi.org/10.1007/s10615-009-0253-x

21.

Victor

B. G.

Kubiak

Angell

Perron

B. E.

(2023). Time to move beyond the ASWB licensing exams: Can generative artificial intelligence offer a way forward for social work? Research on Social Work Practice, 33(5), 511–517. https://doi.org/10.1177/10497315231166

22.

Wang

Zhang

(2025). A comparative analysis of GPT-4o and ERNIE Bot in a Chinese radiation oncology exam. Journal of Cancer Education. Advance online publication. https://doi.org/10.1007/s13187-025-02652-9

23.

Wei

(2024). The use of large language models for translating Buddhist texts from classical Chinese to modern English: An analysis and evaluation with ChatGPT 4, ERNIE Bot 4, and Gemini Advanced. Religions, 15(12). Article 1559. https://doi.org/10.3390/rel15121559

24.

Weng

T. L.

Wang

Y. M.

Chang

Chen

T. J.

Hwang

S. J.

(2023). ChatGPT failed Taiwan’s family medicine board exam. Journal of the Chinese Medical Association, 86(8), 762–766. https://doi.org/10.1097/JCMA.0000000000000946

25.

Xinhua News Agency . (2024). 2024年度全国社会工作者职业资格考试报名人数达188.9万人 [2024 Social Worker Qualification Exam Draws Record Number of 1.889 Million Applicants]. https://www.gov.cn/yaowen/liebiao/202406/content_6957562.htm

26.

Zeng

S. C.

X. y.

Zhang

H. H.

(2024). 社会工作师职业水平考试的内容效度研究[A preliminary test of the content validity of social worker’s professional level examination]. Social Work and Management, 24(3), 12–21+68.

27.

Zeng

Chen

(2020). China Social work professional level examination pass rates: Chronological change and regional difference. Research on Social Work Practice, 30(3), 344–352. https://doi.org/10.1177/1049731519875792