Evaluation of the Effectiveness and Success of the YLSY Scholarship Program With Content Analysis and Machine Learning Methods

Data Collection

Qualitative Data Analysis

Content analysis was employed to systematically examine the qualitative data obtained from semi-structured interviews. Content analysis enables the identification of patterns, meanings, and relationships within textual data through a structured and rule-based process (Prasad, 2008). In line with qualitative research principles, a human-coded, inductive approach was adopted, allowing themes to emerge directly from participants’ responses rather than being imposed a priori (Neuendorf & Kumar, 2016). The analysis followed a multi-stage coding procedure. First, all interview recordings were transcribed verbatim. During the initial coding phase, six members of the research team independently reviewed the transcripts to become familiar with the data. Open coding was then conducted, whereby meaningful units of text were labeled to capture key ideas, experiences, and challenges expressed by the participants. To enhance coding consistency, the research team jointly discussed sample responses prior to full-scale coding, and trial coding was conducted on selected experts as part of coder training.

In the second stage, the first two interview questions were coded independently by all six coders. The resulting codes were compared during consensus meetings, where overlapping codes were merged, ambiguous labels were refined, and a shared coding language was established. Through this iterative comparison process, preliminary categories were generated. In the final stage, related categories were systematically clustered into higher-order themes by examining conceptual similarities and relationships among codes. This iterative and collaborative process ensured that themes were firmly grounded in the data and reflected participants’ shared perspectives rather than individual interpretations.

Validity, Reliability, and Trustworthiness

To ensure the trustworthiness of the qualitative findings, the study adopted established criteria for qualitative rigor, namely credibility, transferability, dependability, and confirmability (Yıldırım & Şimşek, 2013). Credibility was strengthened through investigator triangulation, as multiple researchers participated independently in the coding and analysis process. In addition, participant diversity was intentionally ensured to capture a wide range of experiences related to the scholarship program. Transferability was supported by providing thick descriptions of the research context and participants, and by incorporating direct quotations from the interview data to allow readers to assess the applicability of findings to similar contexts. Dependability was addressed by maintaining consistency in the data collection process; all participants were asked the same semi-structured interview questions in the same order, and the analytical procedures were applied systematically across the dataset. Confirmability was ensured through inter-coder agreement procedures. The coding of the first interview question was independently conducted by multiple coders, and inter-coder reliability was calculated using Miles and Huberman’s (1994) formula, resulting in an agreement rate of 81.3%, which exceeds the recommended threshold of 80% for qualitative reliability. For the remaining questions, coding was performed by two researchers, followed by consensus discussions until full agreement was reached on codes and themes. Additionally, all interviews were recorded via video conferencing to prevent data loss and to allow for repeated review during analysis. After completing the qualitative analysis, some of the resulting themes indicated which dimensions of academic achievement would be more suitable for measurement in the quantitative phase. In this context, quantitative variables related to academic achievement were determined by considering the themes that directly contributed to the preference for quantitative measurement. This relationship between the qualitative findings and the academic achievement variables used in the quantitative phase is summarized in Table 1.

OPEN IN VIEWER

Table 1.

Dimensions of Academic Achievement Determined Based on Qualitative Findings.

Qualitative theme	Contribution of the theme to the quantitative stage	Quantitative dimensions of academic achievement measured
Threats	This theme is related to risks that could negatively impact publication production and the impact level of publications.	Indexed_Article_Success, Citation_Success
Improvement suggestions	This theme suggests that it is more appropriate to evaluate academic achievement based on different academic outcomes rather than a single outcome.	Indexed_Article_Success, Other_Article_Success, Project_Success, Citation_Success
Additional mechanism proposals	This theme demonstrates that academic production depends on considering the processes that influence it together.	Indexed_Article_Success, Other_Article_Success, Project_Success, Citation Success

Data Collection

The initial phase of the machine learning workflow involves defining the target study population and designing the instruments for data acquisition.

Development of Data Collection Tool

The questionnaire used in this study was developed through a formal, multi-step scale-development process within the TÜBİTAK Project 122G002. Specifically, a 34-item pool was generated from the literature and qualitative findings, reviewed by eight domain experts, refined to a 26-item, five-factor structure, and piloted with six YLSY scholars. Validity and reliability were then established using independent samples for EFA (n = 200; KMO = 0.833; Bartlett χ² = 3,248.00; variance = 66.69%) and CFA (n = 200; χ²/df = 2.88; RMSEA = 0.079; CFI = 0.93), with satisfactory reliability and convergent/discriminant validity (Cronbach’s α = .85; CR = 0.82–0.89; AVE ≥ 0.50). The full methodological report, including item generation, expert review documents, pilot results, and complete EFA/CFA statistical tables, has been published under the same project and is publicly accessible (see Göl et al., 2023). The scale questions within the scope of quantitative analysis in the developed form and the variables corresponding to the questions are presented in Table 2.

OPEN IN VIEWER

Table 2.

Questionnaire Items and Corresponding Variables Used in the Quantitative Analysis.

No.	Question	Attribute
1	What is your gender?	Gender
2	How old are you?	Age
3	What was your marital status at the time you attended study abroad?	Marital_Status
4	Please indicate the department you graduated from.	Department_of_Graduation
5	What is your undergraduate graduation average?	Undergraduate_GPA
6	What is your ALES score for applying for the 1416 education scholarship?	ALES
7	What is your score in the interview exam when you applied for the 1416 education scholarship?	Interview_Exam _Score
8	What is the type of education for which you are applying for the 1416 education scholarship?	Scholarship_Education
9	Which country did you go to study abroad? (For Master’s Degree)	Country_Master
10	Which country did you go to study abroad? (For PhD)	Country_PhD
11	What is the official language of the country(s) in which you went to study abroad?	Official_Language
12	What is the ranking of your university according to Times Higher Education, QS World, or Shanghai?	University_Ranking
13	What is the number of indexed articles you had after returning from your study abroad?	Number_of_Indexed_Articles
14	What is the number of your other articles after returning from study abroad?	Number_of_Other_Articles
15	What is the number of citations you have after returning from your study abroad?	Number_of_Citations
16	What is the number of your projects after returning from study abroad?	Number_of_Projects

Data Preprocessing

At this stage, the data obtained through structural form was made suitable for the development of models. Data editing, data discretization, handling missing values, feature generation, handling outlier values, and data transformation processes were applied in the data preparation and preprocessing steps to obtain the variables to be used in the development of the models.

Data Editing

In this step, various data editing operations were carried out for Undergraduate_GPA, ALES, Country_Master, Country_PhD, and Official_Language attributes. The answers given by the participants to the question “What is your undergraduate graduation average?” in the survey form varied. Some participants reported their GPA score in the 100-point system, and some participants reported their GPA score in the 4.00-point system. Therefore, in order to transform the attribute values into a single scale, GPA scores in the 4.00-point system were converted into GPA scores in the 100-point system. The formula used for this is given in Equation 1.

New GPA Score = 4 . 00 * GPA Score 100

(1)

Similarly, there were differences in the answers given by the participants to the question, “What is your ALES score for applying for the 1416 education scholarship?” While some participants wrote their ALES score values calculated in the 100-point system, some participants reported their LES exam scores calculated in the 80-point system. For this reason, LES scores in the 80-point system were converted to ALES score type using Equation 2.

ALES Score = 80 * LES Score 100

(2)

Finally, in this step, the data in the Country_Master, Country_PhD, and Official_Language attributes were edited and standardized. All answers were checked, uppercase and lowercase inconsistencies were eliminated, spelling errors were corrected, and thus data inconsistency was eliminated.

Data Discretization

Data discretization, one of the basic preprocessing technique, is a data reduction method that converts numerical attributes into discrete or nominal attributes (Garcia et al., 2012). The answers given by the participants for the “University_Ranking” attribute vary within the scope of THE, SHANGAI, QS, URAP, and ARWU international university ranking indicators. For example, one participant answered this question as THE-226, while another participant answered this question as SHANGAI-901. Some participants reported only the ranking without specifying the type of indicator (N1, N2, and N3). For this reason, the answers to this question were discretized and shown in Table 3.

OPEN IN VIEWER

Table 3.

Discretization Ranges for Model Variables.

Value range	New value	Value range	New value	Value range	New value
THE 0–250	THE1	QS 501–…	QS3	SHANGHAI 251–500	SHA2
THE 251–500	THE2	URAP 0–250	URAP1	SHANGHAI 501–…	SHA3
THE 501–…	THE3	URAP 251–500	URAP2	0–250	N1
QS 0–250	QS1	URAP 501–…	URAP3	251–500	N2
QS 251–500	QS2	SHANGHAI 0–250	SHA1	501–…	N3

Feature Generation

The feature generation process, also known as feature engineering in the field of machine learning, is the addition of a new dimension to the data set by creating new attributes from existing attributes (Kamath & Choppella, 2017; Sarkar et al., 2018). In this step, one input variable and four output variables were generated by using the existing variables. First of all, since English is the academic language at the international level, the “Official_Language_English” input variable was generated depending on the “Official_Language” attribute values. This attribute consists of “Yes” and “No” values. On the other hand, the study aims to predict whether the scholarship program will be beneficial for academic staff participants who will go abroad with the scholarship program in the future. To develop supervised machine learning models, attributes representing the benefits seen by participants were needed as output variables. In this study, it was aimed to estimate four attributes of the participants. For this purpose, the “Number_of_Indexed_Articles,”“Number_of_Other_Articles,”“Number_of_Projects,” and “Number_of_Citations” attributes were used to generate these attributes, respectively. Table 4 lists the transformation rules applied to the raw dataset to generate meaningful features that are used as input to machine learning models.

OPEN IN VIEWER

Table 4.

Feature Generation Rules.

Old attribute	New attribute	Rule
Official_Language	Official_Language_English	IF (Official_Language = “English”) THEN Official_Language_English = “Yes” ELSE Official_Language_English = “No”
Number_of_Indexed_Articles	Indexed_Article_Success	IF (Number_of_Indexed_Articles > 0) THEN (Indexed_Article_Success = 1) ELSE (Indexed_Article_Success = 0)
Number_of_Other_Articles	Other_Article_Success	IF (Number_of_Other_Articles > 0) THEN (Other_Article_Success = 1) ELSE (Other_Article_Success = 0)
Number_of_Projects	Project_Success	IF (Number_of_Projects > 0) THEN (Project_Success = 1) ELSE (Project_Success = 0)
Number_of_Citations	Citation_Success	IF (Number_of_Citations > 0) THEN (Citation_Success = 1) ELSE (Citation_Success = 0)

As a result of this step, Table 5 summarizes the distributional features and statistical properties of the resulting features.

OPEN IN VIEWER

Table 5.

Summary of Variables Used for Model Development.

Attribute	Attribute type	Values
Gender	Categorical	Male, Female
Age	Categorical	20–34, 35–44, 45–55, 55–64
Marital_Status	Categorical	Single, Married
Undergraduate_GPA	Continuous	55–100
ALES	Continuous	60–98.7
Scholarship_Education	Categorical	Master’s, PhD, Master’s, and PhD
Country_Master	Categorical	USA, UK, France, Italy, United Kingdom, Netherlands, UK, China, Germany, Japan, Canada, Not available
Country_PhD	Categorical	USA, UK, France, Italy, United Kingdom, Not available
Official_Language_English	Categorical	Yes, No
University_Ranking	Categorical	THE1, THE2, QS1, QS2, QS3, SHA1, SHA2, URAP2, S1, S2
“Project_ Success”	Categorical	0, 1
“Indexed_Article_Success”	Categorical	0, 1
“Other_Article_Success”	Categorical	0, 1
“Citation_Success”	Categorical	0, 1

In this study, academic performance (articles, citations, projects) was measured using a binary classification system. Academics were coded as “present” if they had at least one academic output, and “absent” if they had no output. The aim of this approach is to make the results easier and more understandable for policymakers.

Data Transformation

In this step, the normalization method, one of the data transformation methods, was applied to the numerical values “ALES” and “Undergraduate_GPA.” The normalization method ensures that the attributes take values within a common scale. In this study, the values of the two attributes were transformed to be in the [0–1] value range through the MinMax normalization process given in Equation 3. This method is based on subtracting the smallest value of the X feature from each value and dividing it by the X range, where X′ is the normalized data and X_i is the input data.

X ' = X i − min ( X ) max ( X ) − min ( X )

(3)

Development of Models

At this stage of the study, within the scope of quantitative analysis, it was aimed to develop four classifier models, namely the Indexed Article Success Model, Other Article Success Model, Citation Success Model, and Project Success Model, which will predict the success of academic staff scholarship candidates who applied to the international scholarship program. For this purpose, nine machine learning algorithms, which are two-class classifiers, were used for each model. Thus, a total of 36 models were developed. The training and testing phases of the models were carried out in the Microsoft Azure Machine Learning Studio environment. In developing the models, 50% of the dataset was used as training and 50% as test dataset. In training the models, model parameters were determined using the “Tune Model Hyperparameter” module. The parameter sweeping mode in the module was set to Random sweep. The maximum number of runs on random sweep was set to 500. During random sweep optimization, model performance was evaluated on the validation set using accuracy, and the hyperparameter configuration yielding the best validation performance was selected. Optimum hyperparameter values for each model are presented in Table 6.

OPEN IN VIEWER

Table 6.

Hyperparameters and Their Values of Models.

Model	Hyperparameter	Indexed article success model	Other article success model	Citation success model	Project success model
AP	Learning rate	0.362716	0.983936	0.669393	0.983936
	Maximum number of iterations	2	1	2	1
BPM	Number of training iterations	30	30	30	30
LR	Optimization tolerance	0.000013	0.000001	0	0.000028
	L1Weight	0.112867	0.94861	0.057811	0.496711
	L2Weight	0.737968	0.192928	0.048561	0.963395
	Memory size	35	39	48	6
SVM	Number of iterations	53	3	92	3
	Lambda	.095805	.00074	.000013	.00074
LSVM	LD-DVM tree depth	1	1	2	1
	Lambda W	.038719	.038719	.045879	.038719
	Lambda theta	.034971	.034971	.03042	.034971
	Lambda theta prime	.095788	.095788	.098866	.095788
	Sigma	.510078	.510078	.64627	.510078
	Number of iterations	13,579	13,579	13,814	13,579
NN	Learning rate	0.030998	0.10911	0.030315	0.035442
	Loss function	Squared Error	SquaredError	CrossEntropy	SquaredError
	Number of iterations	139	158	109	21
BDT	Number of leaves	2	2	2	2
	Minimum leaf instances	3	4	4	4
	Learning rate	0.064733	0.064733	0.064733	0.064733
	Number of trees	436	436	436	436
DJ	Number of optimization steps per decision DAG layer	4,663	7,130	7,130	114
	Maximum width of the decision DAGs	11	71	71	39
	Maximum depth of the decision DAGs	2	92	92	5
	Number of decision DAGs	4	10	10	27
DF	Minimum number of samples per leaf node	2	6	2	1
	Number of random splits per node	945	38	945	14
	Maximum depth of the decision trees	14	48	14	4
	Number of decision trees	14	10	14	27

Random sampling was used for dataset partitioning. Within this partitioning process, the hyperparameter tuning process was performed using only the training data, which was partitioned into training and validation subsets by Azure Machine Learning Studio. Model performance was evaluated on the validation subset using an accuracy metric, and the best configuration was selected. The test data was not included in the hyperparameter optimization and model selection processes and was only used to evaluate the final performance.

Although classical statistical significance levels (e.g., p-values) are not directly applicable to machine learning models, SHAP values provide an interpretable alternative by quantifying the relative contribution of each variable to model predictions.

Strengths of the Program

This theme captures the positive aspects of the YLSY scholarship program as experienced and perceived by participating scholars. These strengths demonstrate the program’s potential to make a meaningful contribution to scholars’ development and to the broader goals of higher education and research capacity building in Türkiye. One of the most commonly mentioned aspects was “Code1.” Scholars emphasized how studying abroad broadened their academic horizons, improved their research skills, and immersed them in a global academic culture. Another frequently cited advantage was the “Code2” and “Code3,” which offered participants the chance of “Code4” in academic positions upon their return. One participant commented: “Position which promises permanency… In general, I think it is a good thing that our position is certain.” These aspects were seen as valuable in fostering inclusivity and ensuring career continuity. “Code5” are also appreciated by participants. Participants also mentioned “Code6” as a strength of the program and the potential for “Code7,” which they believed enriched both their professional lives. These findings suggest that the YLSY program plays a significant role in equipping scholars with the competencies and experiences needed for academic careers while also fostering a sense of belonging to a global academic community.

Weaknesses of the Program

This theme focuses on scholars’ critical reflections regarding the structural and procedural shortcomings of the YLSY scholarship program. A frequently cited concern was the “Code8,” which refers to scholars’ perceptions that scholarship amounts are standardized and do not sufficiently account for differences in country-, city-, and cost-of-living conditions, leading to financial strain in high-cost destinations. As one participant stated, “The current scholarship amount may be sufficient at a basic level, but it does not provide comfortable living conditions … in countries with high living and health insurance costs.” (Participant 1). Overall scholars consistently expressed that the current system does not adequately account for variations in the cost of living, leading to financial hardship for those studying in more expensive locations. Another major weakness was the “Code9” such as ministry of national education, Higher education council, and attaché offices. This gap was seen as a root cause of inefficiencies throughout the program process, from pre-departure to post-return. The “Code10” also drew criticism. Participants questioned the fairness and transparency of the evaluation, with some describing the interviews as subjective or inconsistent. As one scholar commented, “It felt more like a formality than a real assessment of our academic potential.” Post-return challenges were also significant. Many participants reported “Code11” including delays in placement or temporary assignments unrelated to their expertise. “Code12” was a source of frustration, particularly for scholars trained in specialized disciplines. These mismatches often led to underutilization of skills and reduced motivation. One participant mentioned “… What should he do at …. University? There is no lecturer or department with a doctorate on drones at …. University ….” Finally, scholars described “Code13.” One participant reflected, “That is, because it is already very difficult to get accepted from departments and from a good university. In other words, it is difficult to find a teacher in that subject, in the subject they want.”

Points Suggested to Be Changed

One of the most emphasized suggestions (Code14) concerned the lack of coordination between institutions involved in the scholarship process. Participants indicated that insufficient coordination created uncertainties and inefficiencies throughout different stages of the program. As one participant stated, “I think you have a lack of coordination. In other words, there is a lack of coordination between institutions. If I had the chance, I would want to change it.” Overall, scholars emphasized that improving institutional coordination was a critical area for change in order to ensure a more coherent and effective implementation of the scholarship program. A second recurring suggestion (Code15) concerned the need for more systematic communication and guidance mechanisms both before and after scholars’ time abroad. Participants emphasized the importance of regular updates, mentorship, and structured channels for receiving guidance and feedback in order to reduce uncertainty and improve preparedness throughout the scholarship process. One participant remarked, “The university needs to check its students and those who go abroad from time to time. I think the university should hold meetings. This problem will be eliminated if the places where students will go are determined in advance.” Revisions to “Code16” were also strongly recommended. Participants advocated for clearer guidelines regarding academic appointments, improved transparency in placement processes, and ensuring that placements align with scholars’ academic backgrounds. “Reorganizing the employment methods of returnees ….” Closely related to this was the call for that “Code17,” including mechanisms to request reassignment if initial placements do not match the scholar’s qualifications or research interests. One participant mentioned “If the university doesn’t accept us when we return, it would be nice if we were at least given a second choice. So, it was scary for us to be dragged into uncertainty there.”“Code18” and the “Code19,” as these were perceived as sources of inconsistency or inefficiency. Furthermore, several scholars suggested that the program include more personalized planning for placement so “Code20.” One said: “When we return, if the university’s appointment criteria are met, we will be appointed as faculty members without any equivalence requirement.”

Adequacy of Existing Mechanisms

This theme captures scholars’ perceptions regarding the effectiveness and sufficiency of the current mechanisms embedded in the YLSY scholarship program. The most frequently mentioned issue was “Code21.” Several scholars reported that counseling services were either insufficient or ineffective, particularly after departure. One participant stated, “Counseling doesn’t actually work very well. As none. There is a communication gap after going abroad.” However, not all views were negative. Another participant noted, “This is about counseling … This is changing. Definitely. This must be it.” Developing a predictive support model tailored to scholars’ evolving needs abroad may contribute to a more proactive and functional counseling framework. The second most frequently mentioned issue was related to “Code22.” Participants noted significant limitations, particularly in terms of staffing and capacity. One participant recalled, “I remember that the number of people working in the education consultancy was quite small, and our number was close to 200. As such, it was not possible for one person to reach 200 people. This reflects a structural gap in human resources ….” Another significant concern was the “Code23.” Participants highlighted the lack of systematic follow-up during their time abroad. A participant emphasized this gap by stating, “Secondly, let me say that the follow-up of the student is carried out. The student should not be abandoned.” This underscores the need for an active and responsive monitoring mechanism that maintains regular contact with scholars and addresses emerging issues promptly. Further concerns were raised regarding the “Code24.” In addition, participants pointed to weaknesses in “Code25,” which can result in misaligned expectations and planning gaps. Lastly, a few participants highlighted the “Code26,” suggesting that existing digital tools for managing the scholarship process are underutilized or poorly integrated.

Suggestions for Additional Mechanisms

This theme reflects scholars’ forward-looking recommendations for mechanisms that do not currently exist within the structure of the YLSY scholarship program but are considered essential for improving its long-term effectiveness. Similar to the previous theme on the adequacy of existing mechanisms, the most frequently emphasized need was for a more robust “Code27.” One participant stated, “There is no structure where you can receive any kind of academic or psychological support ….” This highlights the pressing need for institutionalized systems that go beyond logistics and address holistic support for scholars throughout the scholarship lifecycle. Another area identified for improvement was the “Code28.” Participants argued that better monitoring would help institutions remain informed of scholars’ progress and challenges while abroad, enabling more timely support and accountability. A further significant area of suggestion centered on the “Code29.” The comments reflected a broader expectation that the scholarship program align more closely with national priorities and institutional needs. One participant emphasized this by stating, “In general, determining what the country needs, in which areas of technology the universities need progress, and sending scholars abroad to fill those gaps in advance.” Participants also proposed “Code30,” noting that a proactive and consistent exchange of information is essential for aligning expectations, fostering a sense of inclusion, and ensuring smooth academic reintegration. In a similar vein, the “Code31” was also highlighted as a crucial step to reduce institutional fragmentation and ensure alignment between individual scholar training plans and broader academic strategies. Finally, some scholars stressed the need for “Code32,” as current procedures related to diploma recognition and academic standing were viewed as ambiguous or inconsistent.

Threats

This theme reflects participants’ perceptions of the potential risks that could undermine the long-term success and sustainability of the YLSY scholarship program. Participants expressed concern that “Code33,” especially those in technical fields, may lead to resignation and pursuit of international opportunities after completing their studies. One participant noted, “Here’s a problem. For example, those who find their way, especially those in technical sciences, can resign and leave.” This was perceived as a loss of national investment and talent. The issue of “Code34” was also identified as a major threat. Factors such as poor adaptation, lack of support, or academic challenges may contribute to dropout or delay. As a participant stated, “These factors include not completing the training on time. The biggest threat is on these students.” In some cases, “Code35” was observed upon their return. One participant described this by saying, “So, after all, the student cannot be happy with teaching after coming here. He cannot do his work properly.” Another critical concern was “Code36.” Stress, lack of academic guidance, or poor topic selection were cited as factors that hinder scholarly quality. A participant remarked, “Because of this stress and lack of control, students can study very absurd subjects. Actually, it’s not very up to date.”“Code37” where they are employed was seen as another barrier to long-term academic contribution. One scholar noted, “Maybe it could be something in terms of feeling like you belong to the institution or contributing something to that institution.” Lastly, “Code38” was mentioned, particularly during the initial transition abroad. One participant commented, “Here’s what I said at the beginning. When you first go there, this is a ridiculous security problem due to the lack of coordination. This is an important thing so ….”

Suggestion for Improvement

This theme includes participants’ proposals for improving the overall implementation, sustainability, and impact of the YLSY scholarship program. The most frequently emphasized areas for improvement were “Code39” and “Code40.” One participant stressed the importance of the former by stating, “… people who will go there should be informed in advance, … perhaps the experiences of those who return can be shared with them.” This comment highlights the perceived need for pre-departure orientation to enhance scholars’ academic preparedness and psychological readiness. Concerns about scholars’ reintegration upon return were also prominent. “Code41” and “Code42” were both noted as critical to ensuring continuity in scholars’ career paths and optimal use of their expertise. These suggestions point to the need for proactive institutional engagement throughout the scholarship cycle. At the selection stage, “Code43” was proposed to ensure that highly qualified and motivated individuals are selected. As one participant commented, “An effort could be made to send more qualified people.” This indicates a concern for maintaining the program’s merit-based integrity. Further, “Code44” was mentioned as a strategy to ensure that scholars have clearly defined goals and institutional alignment before beginning their studies. Finally, “Code45” reflects a broader call for strategic foresight in national academic planning. One participant noted, “… institutions need to work in coordination and clearly state what is needed according to long-term plans.” This underscores the relevance of predictive models in shaping the scholarship program’s future direction and aligning it with national development priorities.

Opportunities

This theme highlights the broad range of benefits that scholars perceive as outcomes of their participation in the YLSY scholarship program. The most frequently cited benefit was the “Code46.” One participant reflected, “As I said, I think the most important thing is that it provides that academic culture for us as scholars.” A closely related benefit was the “Code47,” which provided not only financial relief but also allowed scholars to focus entirely on their academic goals without the stress of tuition fees and living expenses. As one participant explained, “So you don’t think about tuition fees. Your salary is deposited …” Participants also highlighted the “Code48” they were able to build during their studies abroad, noting that collaborations with international researchers significantly enriched their academic perspectives and opened new avenues for future cooperation. In terms of personal growth, scholars emphasized “Code49” as a key outcome of their time abroad. Other positive outcomes mentioned include the “Code50,” such as conferences and workshops, which helped scholars stay current in their fields and gain visibility within the global academic community. Finally, the “Code51” itself was regarded as valuable, not only for academic reasons but also for the life experience it provided.

Building on the qualitative findings of Research Question 1, the second phase of the study aimed to explore whether predictive models could be developed to evaluate the effectiveness of the YLSY scholarship program. In line with the exploratory sequential mixed method design, the qualitative phase served as the foundation for shaping the structure and content of the subsequent quantitative analysis. Themes and codes identified through in-depth interviews were systematically examined to inform the selection of input and output variables for predictive modeling. In the following section, the results of the quantitative analysis addressing Research Question 2 are presented.

Predictive Modeling Results

This section presents the results of the machine learning–based predictive models developed to analyze multiple academic success outcomes of YLSY scholars. Separate classification models were constructed for four success dimensions as indexed article success, citation success, project success, and other article success.

Four performance criteria (formulations are given in Equations 4–7) were used to evaluate the performances of 36 machine learning models developed within the scope of quantitative analysis. For each predictive model, the target variable represents a binary academic success outcome with two classes: “0” (successful) and “1” (unsuccessful). In this study, the “successful” class is treated as the positive class, while the “unsuccessful” class is treated as the negative class. In this situation, TP is the number of “successful” samples predicted as “successful,” and TN is the number of “unsuccessful” samples predicted as “unsuccessful” by the model. FP is the number of samples predicted as “successful” when the actual value is “unsuccessful,” and FN is the number of samples predicted as “unsuccessful” when the actual value is “successful.”

Accuracy = TP + TN TP + TN + FP + FN

Precision = TP TP + FP

Recall = TP TP + FN

F 1 − Score = 2 * Precision * Recall Precision + Recall

TP, FP, FN, and TN values, as well as accuracy, precision, recall, F1-Score, and AUC values of the nine classifier models developed to predict the indexed article success of scholarship candidates are given in Figure 3. When the figure is examined, it was concluded that the highest accuracy and precision values with values of 0.774 and 0.826, respectively, were obtained with the BDT model, the highest recall value was obtained with the DJ model with 1.000, and the highest F1-Score value was obtained with the BDT model and DJ model with a value of 0.853, and the highest AUC value was obtained with LR model and BDT model with a value of 0.715.

OPEN IN VIEWER

Figure 3.

Performance criteria values of indexed article success models: (a) TP, (b) FP, (c) FN, (d) TN, (e) Accuracy, (f) Precision, (g) Recall, (h) F1-Score, (i) AUC.

These findings indicate that the indexed article performance within the YLSY program follows a systematic and predictable pattern, rather than a random one. The successful performance of the model suggests that the variables included in the analysis capture significant determinants of academic productivity in terms of indexed outputs.

As shown in Figure 4, the DJ model provided the best prediction performance in terms of accuracy and precision performance measures, with values of 0.711 and 0.735, respectively. The best recall value was achieved by AP with a value of 0.970, and the highest F1-Score value was achieved by the DF model with a value of 0.819, and the highest AUC value was obtained with LDSVM model and DJ model with a value of 0.575.

OPEN IN VIEWER

Figure 4.

Performance criteria values of other article success models: (a) TP, (b) FP, (c) FN, (d) TN, (e) Accuracy, (f) Precision, (g) Recall, (h) F1-Score, (i) AUC.

These findings indicate that the other article outcomes are structured and can be successfully predicted. This suggests that academic output across broader article categories is also influenced by similar systematic factors.

As shown in Figure 5, the BDT model showed the best prediction performance, excluding recall, and the LDSVM model showed the lowest prediction performance, excluding precision value. While the accuracy, precision, and F1-Score values obtained with the BDT model were 0.816, 0.831, and 0.832, the highest recall value was reached with the DJ and DF models, with a value of 0.965, and the highest F1-Score value was achieved by the BDT model with a value of 0.891, and the highest AUC value was obtained with BDT model with a value of 0.711.

OPEN IN VIEWER

Figure 5.

Performance criteria values of citation success models: (a) TP, (b) FP, (c) FN, (d) TN, (e) Accuracy, (f) Precision, (g) Recall, (h) F1-Score, (i) AUC.

These findings indicate that the citation success model, the performance regarding citation outcomes demonstrates that academic achievement improves in a consistent and explainable manner. This finding supports the view that the program contributes to measurable academic impact.

As shown in Figure 6, participation in research projects can be meaningfully predicted, indicating that research engagement follows a systematic and predictable pattern rather than occurring randomly. When the model performances are examined, the NN model achieved the highest accuracy (0.846), while the LDSVM model demonstrated the highest precision (1.000). The highest recall value was obtained with the DJ model (1.000), and the highest F1-Score was achieved by the SVM model (0.879), and the highest AUC value was obtained with SVM model with a value of 0.566.

OPEN IN VIEWER

Figure 6.

Performance criteria values of project success models: (a) TP, (b) FP, (c) FN, (d) TN, (e) Accuracy, (f) Precision, (g) Recall, (h) F1-Score, (i) AUC.

Figure 7 presents the confusion matrix plots of the best-performing models for each prediction task. The results indicate a strong ability of the selected models to correctly identify successful cases across different success criteria. In Figure 7a, corresponding to the Indexed Article Success task, BDT model demonstrates a high number of true positives, indicating effective identification of scholars with indexed article success, while maintaining a moderate level of false positives. Figure 7b shows the DJ model for Other Article Success, where a relatively higher number of false positives is observed. However, the model successfully captures a large proportion of true positive cases, reflecting its high recall-oriented behavior. In Figure 7c, the BDT model applied to Citation Success exhibits a strong balance between true positives and true negatives, suggesting robust discrimination capability with limited misclassification. Finally, Figure 7d illustrates the SVM model for Project Success, which achieves a high number of correctly classified successful cases, although the presence of false positives indicates a tendency to favor sensitivity over specificity.

OPEN IN VIEWER

Figure 7.

Confusion matrix plots for the best-performing models (a) Indexed Article Success, (b) Other Article Success, (c) Citation Success, (d) Project Success.

Figure 8 presents the Receiver Operating Characteristic (ROC) curves of the best-performing models for each prediction task. The BDT models in Figure 8a and c demonstrate strong discriminatory performance, achieving AUC values of 0.715 and 0.711, respectively, and remain well above the diagonal reference line. In Figure 8b and d, the DJ and SVM models exhibit moderate discriminatory ability with AUC values of 0.575 and 0.566.

OPEN IN VIEWER

Figure 8.

ROC curves for the best-performing models (a) Indexed Article Success, (b) Other Article Success, (c) Citation Success, (d) Project Success.