Comparative analysis of ChatGPT and Google Gemini in creating patient education materials for lower extremity varicose veins and deep vein thrombosis

Introduction

Patient education plays a crucial role in disease management. By increasing disease awareness, patients can identify symptoms early, follow treatment regimens, and enhance collaboration with healthcare providers, ultimately improving outcomes and reducing complications.^1–3 Artificial Intelligence (AI) tools are increasingly being used for patient education, thanks to their ability to efficiently integrate medical information.^4,5 Unlike traditional search engines, AI can offer personalized and easily accessible health guidance. Although AI tools like ChatGPT and Google Gemini are widely used for medical information generation,^6,7 their reliability and effectiveness in vascular disease education have not been systematically assessed.

Methods

A one-week cross-sectional study was conducted in February 2025. Since no human participants were involved, ethical approval was not required. The study focused on two common vascular diseases: varicose veins and deep vein thrombosis of the lower extremities. Two artificial intelligence tools, ChatGPT 3.5 and Google Gemini, were used to generate patient education materials. Prompts provided to both tools included: “Compile educational materials for patients with varicose veins of the lower extremities” and “Compile educational materials for patients with deep vein thrombosis of the lower extremities.” The generated materials were collected in Microsoft Word documents. The Flesch-Kincaid calculator was used to assess ease of understanding, including word count, sentence count, ease of understanding, and readability scores. ⁸ The originality of the content was evaluated using the Quillbot plagiarism tool, ⁹ and the modified DISCERN score was applied to assess the reliability of the information. Two vascular surgeons independently assessed the texts, with the average of their scores serving as the final result. This tool includes five questions to evaluate content quality, accuracy, and reliability on a five-point scale. ¹⁰ Final data were exported to Microsoft Excel for analysis using R version 4.3.2. Comparisons between the materials generated by ChatGPT and Google Gemini were performed using the unpaired t-test, with a p-value <.05 considered statistically significant. The Pearson correlation coefficient was used to evaluate the relationship between easy and reliability scores.

Results

Patient education materials generated by ChatGPT and Google Gemini showed no significant differences across several readability metrics. These included word count (386 vs 367.5, p = .7524), sentence count (39.66 vs 48.65, p = .2184), average words per sentence (9.86 vs 7.71, p = .2982), and average syllables per word (1.92 vs 1.92, p = 1.001) (see Table 1).

OPEN IN VIEWER

Table 1.

Characteristics of responses generated by ChatGPT and Google Gemini.

Variables	ChatGPT		Google gemini		p-value
	Mean	Standard deviation	Mean	Standard deviation
Words	386	89.26	367.5	61.52	0.7524
Sentences	39.66	5.83	48.65	8.64	0.2184
Average words per sentence	9.86	2.06	7.71	2.24	0.2982
Average syllables per word	1.92	0.13	1.92	0.12	1.001
Similarity %	63.52	10.13	52.14	6.63	0.1913
Reliability score	2.32	0.57	2.32	0.57	1.000
Grade level	12.85	2.42	10.28	1.26	0.1965
Ease score	33.28	8.27	35.42	11.24	0.8025

Additionally, no significant differences were found between the two models in similarity percentage (63.52 vs 52.14, p = .1913) or reliability scores (2.32 vs 2.32, p = 1.0). The materials also showed no significant differences in grade level (12.85 vs 10.28, p = .1965) or ease score (33.28 vs 35.42, p = .8025) (see Table 1).

Correlation analysis revealed a strong positive correlation between the ease of use scores of materials generated by ChatGPT and Google Gemini (r = 0.7788, p = .4316). In contrast, reliability scores showed a negative correlation (r = −0.5, p = .6668). However, neither of these correlation coefficients was statistically significant (p > .05) (see Table 2).

OPEN IN VIEWER

Table 2.

Correlation between patient education materials produced by ChatGPT and google gemini: ease score and reliability score comparison.

Variables	r	p-value
Ease score	0.7788	0.4316
Reliability score	−0.5	0.6668

Discussion

Our findings showed no significant differences in readability metrics—including word count, sentence count, average words per sentence, average syllables per word, grade level, and ease score-between ChatGPT and Google Gemini. This suggests that both mainstream large language models (LLMs) offer a similar baseline for creating patient education brochures, with comparable text structure and readability. Clinicians and health educators can expect similar initial material with essential readability features, regardless of the model used.

Although both models in this study performed similarly in terms of readability, the potential of AI technology for personalized health education is vast. LLMs can adapt vocabulary, sentence structure, and content depth by analyzing patient data, such as age, cultural background, education level, and comorbidities, to create highly tailored educational materials. ¹¹ This ability to customize and adjust readability is vital for enhancing patient health literacy and improving adherence to disease management, ensuring that the information aligns with the patient’s comprehension and needs.^12,13 Future research should explore how to maximize this core capability of LLMs in clinical practice.

When evaluating content quality, both ChatGPT and Google Gemini showed no significant differences in similarity percentage or reliability scores, indicating that both models can serve as reliable information sources. However, it is important to note that LLMs are trained on vast amounts of internet text data, which may include inaccurate or inappropriate information, potentially leading to misleading content. ¹⁴ For example, a previous study comparing Google Gemini and ChatGPT on status epilepticus content found both models received a DISCERN score of only 3. ¹¹ Similarly, in our study, both models achieved a reliability score of just 2.32. This underscores the limitations of even advanced LLMs in generating professional medical information, highlighting the need for caution.

Although the patient education guidelines generated by ChatGPT in this study showed a high average similarity percentage, we do not consider this plagiarism. Rather, it reflects the inherent function of LLMs. These models generate content by learning from large volumes of public information, often resulting in overlap with existing materials, especially for basic medical knowledge in patient education. However, it is essential to minimize similarity with existing literature to maintain the credibility of medical education. This is crucial for ensuring academic ethical compliance and protecting institutional reputation. ¹⁵ Even when high similarity arises from shared general knowledge, it should be minimized whenever possible. Future research should focus on strategies to balance LLM efficiency with the need for originality. This could involve refined prompt engineering or the integration of plagiarism detection tools to identify and address potentially problematic “high similarity” content.

Our research found a strong positive correlation in ease of use scores between patient education materials generated by ChatGPT and Google Gemini. In contrast, a moderate negative correlation was observed for reliability scores. However, neither correlation was statistically significant. The positive correlation in ease of use suggests that both models face similar challenges or use similar strategies to optimize usability in patient education materials. The negative correlation in reliability, though not significant, may indicate that the models occasionally sacrifice completeness or depth to enhance comprehensibility, which could indirectly affect reliability. This underscores the need to balance ease of use with reliability when using LLMs for patient education materials. Over-simplification should be avoided to prevent distortion or omission of key information. Future research should explore this potential trade-off further, using refined evaluation tools or user experience testing.

We recommend the cautious use of LLMs in medical settings. LLMs can serve as a starting point for patient education materials or initial drafts for clinician-authored handbooks. Their efficiency, ease of understanding, and ability to personalize content can help save healthcare professionals valuable time. However, for promotional materials used directly in clinics or information related to critical medical decisions, LLM-generated content must undergo rigorous review and validation by clinical professionals. Despite LLMs’ strong readability performance, their potential limitations in reliability and similarity to existing literature require manual verification prior to publication. This ensures the information is accurate, original, and aligned with the latest medical guidelines, thus preventing misinformation and preserving the professionalism and credibility of healthcare services.

Limitations

This study focused on two conditions: varicose veins and deep vein thrombosis of the lower extremities, which may limit the generalizability of the findings. Future research should explore a broader range of diseases to provide a more comprehensive assessment. Additionally, we analyzed only two AI tools, restricting the scope of our comparison.

Conclusion

This study found no significant differences between ChatGPT and Google Gemini in terms of average usability, reliability, or the quality of patient education materials for varicose veins and deep vein thrombosis of the lower extremities. Moreover, there was no clear correlation between the usability and reliability scores generated by these two tools. This suggests that both AI systems produce similar outcomes when creating patient education materials for specific diseases. Further research is needed to assess additional AI platforms and expand the range of diseases studied, particularly newer ones. Most importantly, the generated content should be compared with the latest medical guidelines to ensure that patients receive accurate, up-to-date information.