SOCRATES. Developing and Evaluating a Fine-Tuned ChatGPT Model for Accessible Mental Health Intervention

Introduction

Generative artificial intelligence is reshaping mental health care, offering scalable, personalized, and empathetic support. This is not a new discussion—more than a decade ago, Cristea et al. ¹ showed that users perceived both human and chatbot therapists as equally human-like, highlighting a shift in therapeutic interaction paradigms. Today, advances such as ChatGPT have pushed these boundaries even further, with users increasingly surprised and satisfied with the responses they get from chatbots, which resemble those of professionals. ² Yet, an attribution bias persists: once users know the source is artificial, trust often diminishes, revealing a latent technophobia. ³ This raises a crucial question: do we want to use chatbots instead of human psychotherapists, or do we want to think about chatbots in mental health as them, together with the psychotherapists, could offer great support?

Beyond the obvious limits, AI-driven interventions have at least dual potential: to reduce health care costs and to alleviate the burden on health care providers. The automation of psychological support could optimize human therapists’ workload. In addition, AI can analyze vast datasets to predict mental health trends, improving public health policies and resource allocation.

A recent randomized controlled trial ⁴ tested a generative AI-powered chatbot (Therabot) for mental health treatment. Its results show that participants using Therabot experienced significantly greater symptom reductions for depression, anxiety, and eating disorder risk compared to waitlist controls, with medium to large effect sizes maintained at 8-week follow-up. Moreover, the AI therapy system demonstrated strong user engagement with over 6 hours of average use and therapeutic alliance ratings comparable to human therapists. These outcomes suggest that fine-tuned generative AI chatbots may offer a scalable approach to personalized mental health interventions.

The Socrates project aims to design and evaluate an AI-powered chatbot developed for mental health. Unlike specialized mental health chatbots built from the ground up for therapeutic purposes, Socrates is a fine-tuned version of ChatGPT, adapting a general-purpose large language model for mental health support. This approach offers a scalable solution by delivering personalized support on demand. The project is conducted by the Humane Technology Lab at Catholic University, Milan, Italy, in collaboration with the Italian National Ph.D. Program in AI for Society (https://phd-ai-society.di.unipi.it/). The main goal of the project is to explore how AI-driven chatbots based on fine-tuned general models can transform mental health care, alleviate economic inefficiencies, and address ethical challenges to ensure their responsible implementation.

Socrates: A General-Purpose AI System Specifically Fine-Tuned to Support Psychological Well-Being

Socrates is an innovative personalized chatbot developed on the ChatGPT platform—a general-purpose AI system—specifically fine-tuned to support psychological well-being through open and self-reflective conversations. Unlike specialized mental health applications built from scratch, Socrates leverages the capabilities of an existing general-purpose large language model, adapting it to provide meaningful psychological support.

What distinguishes Socrates from standard ChatGPT interactions is its integration of a prior knowledge system, drawing upon a carefully curated repository of knowledge selected from psychological, psychotherapeutic, and philosophical sources. This foundation enables Socrates to ground its responses in relevant evidence-based approaches and theoretical frameworks, significantly enhancing the depth, accuracy, and specificity of its interactions.

Socrates was subjected to rigorous fine-tuning with painstakingly detailed instructions designed to optimize therapeutic interactions. The model was specifically trained to recognize and appropriately respond to a wide spectrum of emotional states expressed by users, while respecting moments of silence or reluctance to engage in conversation. It was calibrated to avoid overly intrusive questioning that might compromise user comfort and trust and instead detect nuanced linguistic patterns that might indicate various psychological states. Throughout its development, emphasis was placed on maintaining an appropriate balance between providing support and encouraging self-reflection.

The conversational tone of Socrates has been carefully calibrated to be calm, empathetic, and non-directive, creating a safe space for users to explore their thoughts and feelings. Rather than prescribing solutions, Socrates employs Socratic questioning techniques to encourage self-reflection and emotional expression. This approach aligns with established therapeutic methodologies that emphasize client autonomy and insight.

Special attention was devoted to ensuring that Socrates could recognize signs of acute psychological distress, including suicidal ideation, self-harm intentions, or severe emotional crises. When such indicators are detected, Socrates is programmed to prioritize user safety by acknowledging the severity of the situation and discontinuing the standard conversational approach. Instead, it explicitly refers the user to appropriate professional resources and provides immediate access to crisis intervention contacts, ensuring users in distress receive proper care beyond what an AI system can provide.

First Results from a Pilot Usability Study

A pilot study evaluated the usability and user experience of Socrates in providing initial psychological support. Eight young working professionals participated in a single session featuring a 20-minute chat interaction with Socrates. While participants were given freedom in their conversation, they were encouraged to share personal challenges and discuss their experiences. The primary objective was to assess whether these interactions could foster emotional engagement and be perceived as genuinely supportive.

Prior to each Socrates session, researchers collected informed consent, demographic information, and baseline measurements, including current distress levels (using the Subjective Units of Distress Scale, SUDS) and emotional state (through the Positive and Negative Affect Schedule, PANAS). Following the interaction, the project conducted comparative pre-post analyses of SUDS and PANAS results while gathering additional evaluations regarding system usability (System Usability Scale, SUS), user engagement (User Experience Scale, UES), and satisfaction with the psychological service (Client Satisfaction Questionnaire, CSQ). Researchers also incorporated a Likert-scale question specifically designed to assess participants’ perceptions of the chatbot’s ability to understand their emotional states.

Initial findings are promising. Notably, participants reported an increase in positive affect following their conversations with Socrates, suggesting that even brief interactions with this general-purpose AI-based chatbot can promote a more positive emotional tone. However, no significant changes in perceived stress levels were observed, indicating that this single-session intervention may be more effective at enhancing positive feelings than reducing acute distress.

Most participants reported feeling strongly understood by the chatbot, with the majority indicating a high degree of perceived emotional attunement. Participants described the experience as intuitive to navigate, personally meaningful, and contextually relevant to their situations. A general sense of satisfaction was expressed across the participant group, with favorable ratings on the satisfaction measures.

These preliminary results highlight both the strong usability of the Socrates chatbot and its potential to promote positive emotional states through conversation. They provide encouraging evidence for the viability of fine-tuned general-purpose AI models in delivering accessible psychological support while also suggesting areas for further refinement and research.

The initial findings demonstrate that Socrates, a fine-tuned general-purpose AI chatbot, shows promise for democratizing mental health services, improving efficiency, and addressing both economic and behavioral challenges in psychological care. Although Socrates has thus far only been tested with the general population, these encouraging results suggest potential applications across various mental health settings.

The findings also align with emerging evidence from similar interventions in the field. As discussed before, a recent randomized controlled trial by Heinz et al. ⁴ evaluated Therabot, another generative AI-powered chatbot designed for mental health treatment. Their results showed significantly greater symptom reductions for depression, anxiety, and eating disorder risk among Therabot users compared to waitlist controls. Importantly, these benefits demonstrated medium to large effect sizes that were maintained at 8-week follow-up, suggesting durability of treatment effects. The Therabot system also achieved strong user engagement metrics, with participants averaging over 6 hours of system use and therapeutic alliance ratings comparable to those typically seen with human therapists.

The parallels between the Therabot findings and preliminary results with Socrates are noteworthy. Both systems leverage fine-tuned generative AI models and demonstrate the capacity to create meaningful engagement with users while producing measurable psychological benefits. While the pilot study with Socrates showed more modest effects focused primarily on enhancing positive affect rather than reducing distress, the Therabot results suggest that with continued refinement and longer-term implementation, AI chatbots like Socrates might achieve more substantial clinical outcomes.

Conclusions

Many mental health services operate with limited human and financial resources. Integrating Socrates into environments such as hospitals or drug rehabilitation centers could therefore provide substantial benefits. Patients could gain additional time for psychological reflection beyond what current staffing constraints allow. Moreover, some individuals might find it easier to discuss sensitive issues with Socrates rather than in face-to-face interactions, potentially experiencing less perceived judgment and accelerating psychological change processes. Health care providers might also benefit from this technology through more streamlined workflows and reduced burnout risk. ⁵

The implementation of AI in mental health requires vigilant oversight, however, to maintain ethical standards, ensure data security, and preserve a human-centered approach to care. While AI chatbots offer valuable support, questions persist regarding their diagnostic precision and therapeutic efficacy.^6–8 This underscores the importance of conceptualizing chatbots as complementary tools for psychotherapists rather than as replacements.

Privacy, security, and data ethics remain paramount concerns in this field. Managing sensitive mental health information demands robust safeguards. Socrates addresses these challenges by adhering to OpenAI’s data security regulations and implementing measures to prevent storage of sensitive data, conversation histories, or user interactions, thereby maintaining rigorous privacy standards and ethical compliance.^7–8

In the future, several strategic directions could further advance the development and impact of the Socrates project. Building upon the pilot findings, a priority would be conducting larger-scale randomized controlled trials with diverse populations to establish more robust evidence for the chatbot’s effectiveness. These trials should include follow-up periods to assess the durability of any psychological benefits and compare outcomes against both waitlist controls and traditional care models.

Expanding the user base to include specific clinical populations represents another important direction. Testing Socrates with individuals experiencing mild to moderate depression, anxiety, or adjustment disorders could clarify its clinical utility and appropriate implementation contexts. This clinical testing phase should incorporate standardized outcome measures and detailed qualitative feedback to refine the chatbot’s capabilities.

Ethical and regulatory framework development should proceed in parallel with technical advancements. Creating comprehensive guidelines for responsible AI implementation in mental health, data governance protocols, and clear boundaries regarding the scope of AI-delivered care would help establish standards for the field. Ongoing stakeholder engagement with patients, providers, policymakers, and ethicists would ensure that diverse perspectives inform this evolving framework.

Finally, developing sustainable funding and deployment models will be crucial for long-term impact. Exploring various access models—from direct-to-consumer applications to health care system integration to community-based initiatives—could help identify the most effective and equitable pathways for implementing Socrates at scale while ensuring financial sustainability.