Evaluation of the Reliability,Utility,and Quality of the Information in Transoral Endoscopic Thyroidectomy Vestibular Approach Videos Shared on Open Access Video Sharing Platform YouTube

Abstract

Background:

The internet is a broadly preferred source of information both by patients and medical professionals. YouTube™ is a significant information source that may be a useful tool to inform the public, medical students, and residents, and may improve the learning experience if used adequately. In this study, we aimed to estimate the quality and accuracy of videos regarding Transoral Endoscopic Thyroidectomy Vestibular Approach (TOETVA) aired on YouTube, which is the most popular video platform of the online world.

Materials and Methods:

We evaluated the first 50 videos, returned by the YouTube research engine, in reply to the “TOETVA” keyword. The popularity of the videos was assessed using the video power index (VPI). The educational quality, accuracy, and transparency of the content regarding TOETVA were estimated by using the DISCERN questionnaire score (DISCERNqs), Journal of American Medical Association benchmark criteria (JAMABC), and global quality score (GQSc). The technical quality was ranked through TOETVA Scoring System (TOETVA-SS) by a TOETVA practicioning endocrine surgeon.

Results:

The content of the videos were about surgical technique with a rate of 68%. According to sources, videos uploaded by physicians had significantly higher DISCERNqs, JAMABC, GQSc, and TOETVA-SS scores. Unlike, videos uploaded by physicians had a lower VPI than videos uploaded by nonmedical sources. The videos of surgical technique had significantly higher DISCERNqs, JAMABC, GQSc, and TOETVA-SS scores. Surgical technique videos also had higher VPI scores than “information about disease or surgery” videos. Also, negative correlations were found between the VPI (popularity index) and educational value (GQSc), transparency (JAMABC), and technical quality (TOETVA-SS) scores.

Conclusions:

The data acquired from YouTube videos regarding TOETVA is of below expected quality and reliability. Nevertheless, the educative potential of the online video platform, YouTube, cannot be underestimated.

Introduction

Before the revolutionary domination of the internet on medical knowledge, education of being a surgeon was supported by a master–apprentice relation and related to sources as textbooks and studies announced in academic publications. However, the widespread use of the online data flow has evolved surgical education since the early 2000s.^1,2 The online world grants reach to unlimited medical data, including visual content. The most popular web-based visual data platform is YouTube™, which has >2 billion daily views.³ Especially in the past few years, medical professionals have realized the value of YouTube as a resource of medical data and visual content for various surgical methods.⁴ These medical/surgical videos help to improve surgical dexterity by giving optical and audible stimuli to the brain, in contradistinction to only reading journals or textbooks.⁵

Many studies have investigated the accuracy of these videos regarding surgical techniques. Nevertheless, the heterogeneity in the absence of the peer-review process has allowed a generation of incorrect or misdirecting online medical data.^5–9 Also, numerous visual content about surgical therapies focuses essentially on technical images in the perioperative period instead of providing a full description of the case, including preoperative management, surgical preparation, or the educational highlights of the procedure.¹⁰

Transoral endoscopic thyroidectomy with vestibular approach (TOETVA) is a minimally invasive natural orifice transluminal endoscopic surgical method with no visible postoperative scar. Lobectomy or total thyroidectomy can be performed through this technique through the vestibular region. This scarless surgical method received great attention from patients and has been preferred widely among surgeons.¹¹

In the presented research, our goal was to estimate whether the view-rate and like-rate (popularity) of TOETVA videos on YouTube are correlated with the reliability and correctness using the common scoring systems; DISCERN questionnaire score (DISCERNqs), global quality score (GQSc), Journal of American Medical Association benchmark criteria (JAMABC), video power index (VPI), and TOETVA Scoring System (TOETVA-SS), which have been composed for a more accurate evaluation of YouTube's visual content considering TOETVA-specific diagnosis, pre- and postoperative management, and adverse outcomes.

Materials and Methods

Online research was performed on YouTube content by applying the keyword “TOETVA” on April 15, 2020. The first 50 online visual content was analyzed. Any online content about surgical method, commercial/marketing videos, videos recorded by patients were involved in the research protocol. We listed YouTube videos in order of relevance, which is the default of searching engine. A surgeon performing TOETVA in his routine practice rated the visual content. Time since the upload day, the running time, the number of views, comments, and likes/dislikes were determined. The videos were classified under two main topics based on their source and content (Table 1).

Table 1.

Classification of the Visual Content

According to the source

1. Academic (author/s was/were affiliated with a university)

2. Physician (author/s not affiliated with a university)

3. Patient

4. Commercial

5. Unclassified

According to the content

1. Surgical technique

2. Information about disease or surgery

3. Patient experience

4. Advertisement

We evaluated the transparency, educational quality, and accuracy of the YouTube content through DISCERNqs,¹² JAMABC,¹³ and GQSc¹⁴ (Table 2). We additionally applied the VPI to estimate the popularity of the YouTube content. (VPI: like ratio × view ratio/100).¹⁵

Table 2.

Scales Used to Assess Transparency, Educational Quality, and Accuracy of the Online Content

DISCERNqs: The DISCERN questionnaire was developed to assess the quality of the information and advised treatment modalities by professionals at Oxford University in the UK. Its scoring ranges between 0 and 80 points. The questionnaire has three sections including 16 questions.

GQSc: It is a 5-point scale described by Bernard et al. that was used to assess the educational value of each video (1—poor quality, very unlikely to be of any use to patients; 2—poor quality but some information present, of very limited use to patients; 3—suboptimal flow, some information covered, but important topics are missing, somewhat useful to patients; 4—good quality and flow, most important topics covered, useful to patients; 5—excellent quality and flow, highly useful to patients).

JAMABC: The transparency and publication information of each video were evaluated according to JAMABC which range from 0 to 4 and was suggested by Silberg et al. (authorship: authors and contributors, their affiliations, and relevant credentials should be provided; attribution: references and sources for all content should be listed clearly, and all relevant copyright information noted; disclosure: website ownership should be prominently and fully disclosed, as should any sponsorship, advertising, underwriting, commercial funding arrangements or support, or potential conflicts of interest; currency: dates that content was posted and updated should be indicated).

DISCERNqs, DISCERN questionnaire score; GQSc, global quality scores; JAMABC, Journal of American Medical Association benchmark criteria.

TOETVA-SS

Considering that DISCERNqs, GQSc, and JAMABC scoring methods do not give a precise evaluation of examined TOETVA-related videos, a surgeon routinely performing TOETVA in his practice utilized a TOETVA-SS for a more comprehensive assessment in terms of TOETVA-specific diagnosis, classification, treatment alternatives, and complications (Table 3). We modified this scoring system from the recent literature focusing on TOETVA surgery.^11,16–19 Based on the TOETVA-SS checklist, each of the 27 criteria was scored as one point if presented verbally or written in the video. K-means clustering analysis was used to classify the quality of videos according to TOETVA-SS scores, and videos are divided into three groups as poor quality (score <1.96), suboptimal quality (score ranging between 1.96 and 11.8) and good quality (score >11.8).

Table 3.

Transoral Endoscopic Thyroidectomy Vestibular Approach Scoring System

Preoperative evaluation

1. Has the complete information been provided on how to select the appropriate patient for the operation?

2. Is it mentioned whether the patient has any comorbid disease?

3. Is the age of the patient specified on video?

4. Is the gender of the patient specified on video?

5. Is information about the patient's previous surgery history stated?

6. Is preoperative radiological imaging findings specified on video?

7. Is preoperative pathological findings (FNAB, etc.) specified on video?

8. Has any information been provided about preoperative dental care and dental health?

9. Has information about preoperative oral care and oral cleaning been given?

10. Has any information been given about the preoperative antibiotic prophylaxis?

During surgery

1. Have the necessary information about mouth anatomy been given?

2. Are essential landmarks (labiomandibular sulcus, mentolabial sulcus, mentum, midpupillary line) specified?

3. Has information about the patient's position in the operation room been given?

4. Is the information given about the position of the tongue in the mouth and the endotracheal intubation method?

5. Is the information given about the position of the surgical team?

6. Has the information been given about the energy devices (Ligasure^®, Harmonic scalpel^®, etc.) to be used?

7. Is it specified what diameter surgical ports will be used for the surgery?

8. Is it indicated where the surgical ports should be located/incision sites at the beginning of the surgery?

9. Has the information been provided on how to make a CO₂ infusion and the optimal pressure of CO₂?

10. Has the information been given about the dissection technique?

11. Has any information been given about the prominent landmarks that must be seen during the surgery?

12. Has the information been given about how the surgical specimen was extracted?

13. Has the information been provided on whether a drain is used?

After surgery

1. Is it stated when oral food intake is initiated?

2. Has the information been provided on how many days to gargle with chlorhexidine?

3. Has the information been provided on the day of the tooth brushing?

4. Has the information been given about postoperative complications?

5. Is the hospitalization period or discharge time specified on video?

CO₂, carbon dioxide; FNAB, fine-needle aspiration biopsy.

Statistical analysis

We applied the GNU PSPP 0.7 computer program for the statistical analysis of the study data. Descriptive statistical methods (mean, standard deviation, median, first quadrant, third quadrant, frequency, percentage, minimum, and maximum) were used. The nonparametric statistics test (Shapiro–Wilk test) and the graphical examinations were utilized to examine the mean distributions of quantitative data. The two-sample t-test was applied in the measurement of two groups of quantitative variables with a normal distribution. Mann–Whitney U-test was used to compare the two groups of quantitative variables without normal distribution. One-way variations of variance test (The Kruskal–Wallis test) and the Dunn's nonparametric comparison were applied for two-way intergroup comparisons of quantitative variables with no normal distribution. Spearman correlation analysis was used to evaluate the relationships between quantitative variables. To find groups in the data, with the number of groups represented, K-means clustering was used. Statistical significance was accepted as P < .05.

Results

Majority of the videos were uploaded by university-affiliated physicians (academic) and were mostly about surgical technique (Figs. 1 and 2).

FIG. 1.

Classification of the videos according to source. Color images are available online.

FIG. 2.

Classification of the videos according to content. Color images are available online.

English is the most commonly used language, followed by Korean. The descriptive data of the videos are demonstrated in Table 4.

Table 4.

Descriptive Data of the Videos

Language	n	%
English	26	52.0
Soundless	4	8.0
Italian	2	4.0
Turkish	6	12.0
Korean	8	16.0
Spanish	4	8.0

	Min–Max	Mean ± SD (median)
Time since upload (days)	11–1875	407.52 ± 681.64 (459)
Running time (seconds)	148–6747	892.57 ± 713.17 (1306)
View	21–164,761	14,761.56 ± 4123.12 (1247)
Like	0–14,375	109.67 ± 2679.32 (17)
Dislike	0–328	14.32 ± 79.18 (1)

The mean VPI, DISCERNqs, JAMABC, GQSc, and TOETVA-SS scores of videos are expressed in Table 5.

Table 5.

Information on Scores

	Min–Max	Mean ± SD (median)
VPI	0–319,069	4524.24 ± 17,447.82 (485.6)
DISCERNqs	16–80	20.42 ± 16.18 (16)
JAMABC	0–4	0.32 ± 0.94 (0)
GQSc	1–5	1.57 ± 1.13 (1)
TOETVA-SS	0–24	3.9 ± 5.29 (2)

DISCERNqs, DISCERN questionnaire score; GQSc, global quality scores; JAMABC, Journal of American Medical Association benchmark criteria; TOETVA-SS, transoral endoscopic thyroidectomy vestibular approach scoring system; VPI, video power index.

DISCERNqs, JAMABC, GQSc, and TOETVA-SS scores of videos uploaded by a university-affiliated physician (academic) were higher than the physician, patient, commercial, and unclassified sourced videos. However, the popularity (VPI score) of videos uploaded by a patient was higher than videos uploaded by a medical doctor (academic-physician sourced videos) (Table 6). Also, visual content focusing on surgical technique had higher DISCERNqs, JAMABC, GQSc, and TOETVA-SS scores than content about the patient experience (P < .001, P < .001, P < .001, P < .001, respectively). VPI scores of visual contents focusing on patient experience were higher than technical (surgical technique) videos (P < .001) (Table 6).

Table 6.

Comparison of Scores According to Descriptive Characteristics

	VPI	DISCERNqs	JAMABC	GQSc	TOETVA-SS
Video source
Academic	90.41 (2.00, 639.86)	48 (32, 64)	2 (1, 3)	3 (2, 4)	6 (4, 11)
Patient	2541.47 (1302.00, 17,950.57)	16 (16, 16)	0 (0, 0)	1 (1, 1)	0 (0, 0)
Commercial	1271.76 (37.3, 80,167.00)	16 (16, 16)	0 (0, 0)	1 (1, 1)	0 (0, 0)
Physician	2.25 (0.58, 17.79)	16 (16, 40)	1 (1, 1)	2 (1, 2)	5 (4, 8)
Unclassified	0 (0, 0.51)	16 (16, 16)	0 (0, 0)	1 (1, 1)	4 (2, 4.5)
Video content
Surgical technique	4.15 (0.28, 46.80)	32 (16, 48)	1 (1, 2)	2 (1, 3)	5 (4, 12)
Information about disease/surgery	1059.84 (324.86, 31,875.46)	16 (16, 40)	0 (0, 1)	1 (1, 2)	0 (0, 0)
Patient experience	46,349.27 (3143.40, 71,686.40)	16 (16, 16)	0 (0, 0)	1 (1, 1)	0 (0, 0)
Advertisement	3060.62 (742.40, 19,082,108)	16 (16, 16)	0 (0, 0)	1 (1, 1)	0 (0, 0)
P	<.001^***	<.001^***	<.001^***	<.001^***	<.001^***

Kruskal–Wallis test, reported as median (first quartile, third quartile).

***

P < .001.

DISCERNqs, DISCERN questionnaire score; GQSc, global quality scores; JAMABC, Journal of American Medical Association benchmark criteria; TOETVA-SS, transoral endoscopic thyroidectomy with vestibular approach scoring system; VPI, video power index.

There was a negative correlation between the running time and VPI scores (r = −0.545, P < .001). We found a positive correlation between the running time and DISCERNqs, JAMABC, GQSc, and TOETVA-SS. Evaluation of like–dislike numbers of YouTube content regarding TOETVA showed that the increasing number of likes had a negative correlation with DISCERNqs, JAMASC, GQSc, TOETVA-SS scores (Table 7). In addition, there were negative correlations VPI and JAMABC, GQSc, and TOETVA-SS (r = −0.430, P < .001; r = −0.402, P = .002; r = −0.367, P < .001, respectively) (Table 8).

Table 7.

Assessment of the Relationship Between Quantitative Variables and Scores

	VPI	DISCERNqs	JAMABC	GQSc	TOETVA-SS
Time since upload
r	0.171	−0.050	0.031	0.021	−0.021
P	.030^*	.260	.804	.819	.809
Run time
r	−0.545	0.345	0.435	0.410	0.796
P	<.001^***	<.001^***	<.001^***	<.001^***	<.001^***
View
r	0.967	−0.195	−0.372	−0.278	−0.446
P	<.001^***	.151	.001^**	.002^**	<.001^***
Like
r	0.967	−0.280	−0.306	−0.321	−0.509
P	<.001^***	.024^*	<.001^***	.003^**	<.001^***
Dislike
r	0.859	−0.280	−0.316	−0.321	−0.412
P	<.001^***	.040^*	<.001^***	.003^**	<.001^***

P < .05, ^**P < .01, ^***P < .001.

DISCERNqs, DISCERN questionnaire score; GQSc, global quality scores; JAMABC, Journal of American Medical Association benchmark criteria; r, Spearman's rho; TOETVA-SS, transoral endoscopic thyroidectomy vestibular approach scoring system; VPI, video power index.

Table 8.

Determining the Relationship Level Between Scores

	VPI	DISCERNqs	JAMABC	GQSc	TOETVA-SS
VPI
r	1.000
P	—
DISCERNqs
r	−0.176	1.000
P	.065	—
JAMABC
r	−0.430	0.726	1.000
P	<.001^***	<.001^***	—
GQSc
r	−0.402	0.875	0.689	1.000
P	.002^**	<.001^***	<.001^***	—
TOETVA-SS
r	−0.367	0.463	0.587	0.492	1.000
P	<.001^***	<.001^***	<.001^***	<.001^***	—

P < .01, ^***P < .001.

Discussion

The presented research showed that the most famous content among YouTube viewers, focusing on TOETVA, had the lowest academic/educational quality. A medical student or resident gains the surgical abilities by following the clinician/surgeon, and then practicing on live cases.²⁰ The online world's visual content, focusing on specific surgical procedures, potentially provides a better perception. It reduces the training curves of medical students, residents, and even specialists compared with current classical medical programs.^21,22 However, the presentation of inaccurate or fallacious data may be detrimental for medical students or residents. YouTube, the world's most popular visual communication center,²³ has reached a tremendous reputation among the medical professionals,²⁴ as a source of virtual training/education site due to its wide range of medical visual content. Also, individuals, whether being or not being a health care professional, have the chance to share their own experiences, ideas, and practice on this unlimited visual communication site without any objective/systematic review process and standardization.²⁵ The first research assessing the reliability of the medical content on YouTube was performed by Keelan et al.²⁶ Since the study of Keelan et al., many other investigations have reported that the content regarding medical issues presented by YouTube were insufficient, ineffectual, and unverified.^27,28 The mean DISCERNqs, JAMABC, GQSc, and TOETVA-SS scores of the YouTube content on TOETVA were 20.42/64, 0.32/3, 1.57/4, and 4.9/28, respectively. These low scores imply that the accuracy, transparency, and quality of YouTube content on TOETVA is far from being educative and the presented outcomes are coherent with the previous literature.^5,15,28

We found that videos having less educational value were more popular among YouTube users. VPI scores of commercial videos and videos uploaded by patients focusing on patient experience were higher. However, these popular videos had less quality (DISCERNqs), educational value (GQSc), and transparency (JAMABC) scores than videos uploaded by the medical professionals. These results suggest that limited quality is equivalent to more views/viewers on YouTube. Considering that the medical professionals do not have an opportunity to restore the misleading data on the online world, it is essential to understand how these websites adversely affect the public. Similar to the presented results, it was showed in the literature that YouTube content on the sleeve gastrectomy procedure was not accurate.⁵ In the study of Erdem and Sislik, the quality of visual content on bariatric surgery aired on YouTube was assessed. Unlike our outcomes, Erdem and Sislik obtained the result that 53.7% of YouTube content on bariatric surgery was useful.²⁹ However, Erdem and Sislik applied a simple usefulness scoring system arranged for online videos, which was initially described by Lee et al.⁴ We assume the application of just one scoring system and not to design or apply a topic-specific evaluation system is inadequate to judge the quality or educative value of YouTube content.

In the presented study, although content transferred by a university-affiliated health professional had higher quality (DISCERNqs), educative value (GQSc), transparency (JAMABC), and technical quality (TOETVA-SS) scores; these videos got the second-lowest popularity score. However, a surprising outcome was that, notwithstanding having the highest TOEVTA-SS score of videos uploaded by a university-affiliated health professional, the technical quality score was just six out of 28. The low TOETVA-SS score of academic content was an indirect proof of low-quality data, even if the content has been uploaded by a university-affiliated medical professional. Madathil et al. reviewed 18 publications focusing on health care data aired on YouTube; and they revealed that YouTube included many anecdotal low-quality ambiguous data.²⁸ Our findings are compatible with the outcomes of the study by Madathil et al.

We noticed a negative correlation between the video running time and VPI (popularity) scores, and a positive correlation between DISCERNqs, JAMABC, GQSc, TOETV-SS, and video running time. And, videos having more likes have lower DISCERNqs, JAMABC, GQSc, TOETVA-SS scores. The negative correlation between VPI and JAMABC, GQSc, TOETVA-SS is an interesting outcome of the presented study. These results imply that YouTube viewers have a positive reply to less optimal content and show less inclination to see long-running time visual content, which potentially has more accurate and qualified data. The presented outcomes are compatible with the literature.^{5,15,24,30,31} Thus, high educational-academic class content has less YouTube user engagement. It looks like neither informative quality nor accuracy of the presented data on YouTube attracts the public attention.

The main shortcoming of the presented investigation was examining the first 50 visual content of YouTube upon researching the keyword “TOETVA.” However, a well-known reality that most YouTube visitors do not click over two web pages of searching engine returns. So, we chose to assess only the first fifty returns of YouTube's searching engine. The second obstacle of this study was that only YouTube content was examined and other video websites were not included in the study protocol. Despite the limitations of this study, we settled the study protocol on an objective methodology to assess the YouTube content for quality, transparency, and accuracy. No study in the literature has accurately reviewed web-based visual content on TOETVA, a promising scarless thyroid surgery technique.

Conclusion

The presented results reveal that YouTube visitors prefer to view less accurate and reliable visual content regarding TOETVA surgery. Furthermore, the open-access characteristic of YouTube™ makes it impossible for expert reviewers to apply a professional data filtration/elimination process. Although the information about TOETVA technique acquired from the YouTube videos is of below expected quality and reliability, the educative potential of YouTube's data archive cannot be neglected. Videos of medical content filtered by a qualified review procedure performed by the medical doctors may improve the perception of the public on health issues, thus the video-based websites may be a valuable instrument for addressing right messages to the public.

Footnotes

Ethics Approval

For this type of study formal consent is not required.

Authors' Contributions

All authors have made substantial contributions to this article being submitted for publications. All authors critically reviewed the article and approved the final form.

Disclosure Statement

The authors declared no conflict of interest.

Funding Information

No funding was received for this article.

References

Schmidt

, Shi

, Sethna

. Use of streaming media (YouTube) as an educational tool for surgeons—A survey of AAFPRS members. JAMA Facial Plast Surg, 2016; 18:230–231.

Gutmann

, Kühbeck

, Berberat

, Fischer

, Engelhardt

, Sarikas

. Use of learning media by undergraduate medical students in pharmacology: A prospective cohort study. PLoS One, 2015; 10:0122624.

Snelson

. YouTube across the disciplines: A review of the literature. MERLOT J Online Learn Teach, 2011; 7:159–169.

Lee

, Seo

, Hong

. YouTube as a source of patient information on gallstone disease. World Gastroenterol, 2014; 20:4066–4070.

Ferhatoglu

, Kartal

, Ekici

, Gurkan

. Evaluation of the reliability, utility, and quality of the information in sleeve gastrectomy videos shared on open access video-sharing platform YouTube. Obes Surg, 2019; 29:1477–1484.

Steinberg

, Wason

, Stern

, Deters

, Kowal

, Seigne

. YouTube as source of prostate cancer information. Urology, 2010; 75:619–622.

Strychowsky

, Nayan

, Farrokhyar

, MacLean

. YouTube: A good source of information on pediatric tonsillectomy?. Int J Pediatr Otorhinolaryngol, 2013; 77:972–975.

Shires

, Wilson

, Sebelik

. Thyroid surgery YouTube videos: Estimating quality by surgeon characteristics and view rate. Gland Surg, 2019; 8:207–211.

Kaya

, Usta

, Baghaki

, Oral

. Relation between educational reliability and viewer interest in YouTube™ videos depicting endometrioma cystectomy surgical techniques. J Gynecol Obstet Hum Reprod, 2020;101808. DOI: 10.1016/j.jogoh.2020.101808

10.

Chauvet

, Botchorishvili

, Curinier

, et al. What is a good teaching video? Results of an online international survey. J Minim Invasive Gynecol, 2019; 22:30286–30289.

11.

Uludag

, İsgor

. Scarless thyroidectomy: Transoral endoscopic thyroidectomy by vestibular approach. Med Bull Sisli Etfal Hosp, 2017; 51:169–183.

12.

DISCERN. Quality criteria for consumer health information. www.discern.org.uk/ (accessed October 30, 2012).

13.

Bernard

, Langille

, Hughes

, Rose

, Leddin

, Veldhuyzen van Zanten

. A systematic review of patient inflammatory bowel disease information resources on the World Wide Web. Am J Gastroenterol, 2007; 102:2070–2079.

14.

Silberg

, Lundberg

, Musacchio

. Assessing, controling and assuring the quality of medical information on the Internet: Caveant lector et viewor—let the reader and viewer beware. JAMA, 1997; 277:1244–1245.

15.

Erdem

, Karaca

. Evaluating the accuracy and quality of the information in kyphosis videos shared on YouTube™. Spine (Phila Pa 1976), 2018; 43:E1334–E1339.

16.

Razavi

, Tufano

, Russell

. Completion thyroidectomy via the transoral endoscopic vestibular approach. Gland Surg, 2018; 7:77–79.

17.

Anuwong

, Kim

, Dionigi

. Transoral endoscopic thyroidectomy using vestibular approach: Updates and evidences. Gland Surg, 2017; 6:277–284.

18.

Wang

, Zhai

, Liu

, et al. Thyroidectomy: A novel endoscopic oral vestibular approach. Surgery, 2014; 155:33–38.

19.

Jitpratoom

, Ketwong

, Sasanakietkul

, Anuwong

. Transoral endoscopic thyroidectomy vestibular approach (TOETVA) for Graves' disease: A comparison of surgical results with open thyroidectomy. Gland Surg, 2016; 5:546–552.

20.

Azer

, Algrain

, Al-Khelaif

, AlEshaiwi

. Evaluation of the educational value of YouTube videos about physical examination of the cardiovascular and respiratory systems. J Med Internet Res, 2013; 15:e241.

21.

Ocak

. Evaluation of the content, quality, reliability and accuracy of YouTube videos regarding endotracheal intubation techniques. Niger J Clin Pract, 2018; 21:1651–1655.

22.

Romanov

, Nevgi

. Do medical students watch video clips in eLearning and do these facilitate learning?. Med Teach, 2007; 29:484–488.

23.

Ferhatoglu

, Kartal

, Filiz

Aİ

, Kebudi

. Comparison of New Era's education platforms, YouTube™ and WebSurg^®, in sleeve gastrectomy. Obes Surg, 2019; 29:3472–3477.

24.

Desai

, Shariff

, Dhingra

, Minhas

, Eure

, Kats

. Is content really king? An objective analysis of the public's response to medical videos on YouTube. PLoS One, 2013; 8:82469.

25.

Sahin

, Sahin

, Türkcü FM. YouTube as a source of information in retinopathy of prematurity. Ir J Med Sci, 2019; 188:613–617.

26.

Keelan

, Pavri-Garcia

, Tomlinson

, Wilson

. YouTube as a source of information on immunization: A content analysis. JAMA, 2007; 298:2482–2484.

27.

Brooks

, Lawrence

, Jones

, McCarthy

. YouTube™ as a source of patient information for lumbar discectomy. Ann R Coll Surg Engl, 2014; 96:144–146.

28.

Madathil

, Rivera-Rodriguez

, Greenstein

, Gramopadhye

. Healthcare information on YouTube: A systematic review. Health Informatics J, 2015; 21:173–194.

29.

Erdem

, Sislik

. The reliability of bariatric surgery videos in YouTube platform. Obes Surg, 2018; 28:712–716.

30.

Pandey

, Patni

, Singh

, Sood

, Singh

. YouTube as a source of information on the H1N1 influenza pandemic. Am J Prev Med, 2010; 38:1–3.

31.

Murugiah

, Vallakati

, Rajput

, Sood

, Challa

. YouTube as a source of information on cardiopulmonary resuscitation. Resuscitation, 2011; 82:332–334.