A comprehensive interpretation for medical VQA: Datasets,techniques,and challenges

Abstract

Recent technological developments and improvement in the medical domain demands advancement, to address the issue of early disease detection. Also, the current pandemic has resulted in considerable progress of improvement in the medical domain, through online consultation by physicians for different diseases using clinical reports and medical images. A similar process is adopted in developing a Visual Question Answering (VQA) system in the medical field. In this paper, existing medical VQA datasets, appropriate techniques, suitable quantitative metrics, real time challenges and, the implementation of one VQA approach with algorithms and performance evaluation are discussed. The medical VQA datasets collected from multiple sources are represented in different perspectives (organwise, planewise, modality-type and abnormality-type) for a better understanding and visualization. Then the techniques used in VQA are subsequently grouped and explained, based on evolution, complexity in the dataset and the need for semantics in understanding the questions. In addition, the implementation of a VQA approach using VGGNet and LSTM is carried out for existing and improved datasets, and analyzed with accuracy and BLEU score metrics. The improved datasets, created through dataset reduction and augmentation approaches, resulted in better performance than the existing datasets. Finally, the challenges of the medical VQA domain are examined in terms of datasets, combining techniques, and modifying the parameters of existing performance metrics for future research.

Keywords

Visual question answering medical VQA ImageCLEF VQA-MED dataset VQA-RAD dataset VGGNet LSTM challenges of VQA

1 Introduction

Visual Question Answering (VQA) is a vision-based textual approach in the fields of computer vision and natural language processing. It takes an image, and one or more natural language questions about the image as input, and generates an answer in natural language as output. The amount of information required from an image in textual format is rapidly increasing. Therefore, Natural Language Processing (NLP) and computer vision are considered important to facilitate the retrieval and analysis of the image based on the text. VQA is used to (i). Assist partially sighted people read restaurant menus, instructions on how to cook a packaged meal, or determine if their clothes match, (ii). Help in medical decision-making and assist medical students in evaluating their domain-specific knowledge, and (iii). Support the general public answer questions about road access and travel using satellite images during natural disasters. VQA datasets are broadly classified into three –natural (indoor and outdoor images), abstract (cartoon and gaming images), and medical (images of different organs and modalities) [1 –3]. This research paper focuses on medical VQA datasets, techniques used in implementation and its real time challenges.

The motivation of this paper are: (i). The comprehensive interpretation based on medical VQA datasets, techniques and challenges helps the researchers to know the current status of the medical VQA domain, and further development based on the discussed research gap. (ii). The evolution of dataset from VQA-MED 2018 to 2021 and VQA-RAD datasets are discussed along with the interpretation of the datasets based on the multiple factors. This inference helps to find out the characteristics of the datasets and improve the dataset which influences the performance of the model. (iii). The recent and efficient techniques for the particular dataset can be identified from the categorization of the approaches. It helps in choosing appropriate technique depending on the type of the dataset. (iv). Some of the datasets have deviating number of samples and give poor performance. Improvising such data may promote more research works.

A medical VQA system has become a necessity in recent times because it supports clinical decision-making and improves patient engagement. For instance, patients in Africa can access their medical health-related data (both structured and unstructured) details via a patient portal. However, the difficulty in getting domain-specific medical experts to answer questions resulted in the development of the medical VQA system. In the African nation of Cameroon, for instance, not all hospitals have specialists, radiologists, gynecologists, and cardiologists. Patients are, consequently, forced to get the opinion of a general physician who has minimum domain-specific knowledge, resulting a possibly incorrect diagnosis in certain cases. To address this issue, a medical VQA system has been developed to answer the patient’s queries with correct information when it is essential to them. Further, the clinician’s confidence is increased by considering the automated system-generated answer as a second opinion. Overall, advances in medical VQA have greatly reduced disparities in a country like Cameroon, with its minimum number of medical experts. To achieve this, the reasonable research in the medical VQA domain need to be developed. Hence, this paper focuses on the following objectives: (i). To comprehensively interpret from medical VQA datasets and techniques (ii). To improve the dataset based on the inference from the interpretation of dataset by data reduction and augmentation approaches to address data imbalance problem. (iii). To develop an medical VQA model for original dataset, reduced dataset, augmented dataset and augmented reduced dataset and validate the resulting model using performance metrics. (iv). To summarize the challenges in terms of dataset acquisition, performance metrics, model generation, computing efficiency. The novelty of this work lies in addressing the second and fourth objectives.

In addition to this, few researchers are also working worldwide and publishing their works in different conferences and journal. In Fig. 1, research papers presented at conferences and published in journals, related to medical VQA datasets, techniques, and performance metrics, are depicted as a graph in chronological order as a sunburst representation.

Fig. 1

Sunburst chart –A count of medical VQA research papers.

The medical VQA dataset used in those papers, cannot be created and need to be sourced from hospitals or clinics. Hence, dataset collection acts as a major challenge. For medical VQA datasets, an external forum named as ImageCLEF is giving extended supports in providing realtime medical datasets and conducting tasks to encourage researchers from 2018 onwards. The ImageCLEF VQA-MED 2018, 2019, 2020 and 2021 datasets have 2866, 4200, 5000 and 5500 radiology images and 6413, 15292, 5000 and 5500 question-answer pairs, respectively. As a result, a total of 13,066 radiology images and 27,705 question-answer pairs are currently available which is described along with VQA-RAD dataset in Section 2.

The datasets were used to design and develop medical VQA systems with appropriate deep neural network techniques. These techniques enable the model to progressively learn features from the data, especially unstructured data, at multiple levels to learn complex patterns. The deep neural learning techniques for image feature extraction include the Convolutional Neural Network (CNN) [4], and pre-trained models such as VGGNet [5], ResNet [6, 46] and CoTNet152 [27]. Different deep neural learning techniques for text feature extraction are, Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM) [7], Gated Recurrent Unit (GRU) [8], Gated Linear Unit (GLU) [9], Bidirectional Encoder Representation from Transformers (BERT) [10], and Bio-BERT [11]. Along with deep neural learning techniques for image feature extraction and text feature extraction, Multimodal Compact Bilinear Pooling (MCB) [12], Multimodal Factorized Bilinear Pooling Network (MFB) [6], Multimodal Factorized High-order Pooling (MFB) [11], Support Vector Machine (SVM) [14] or Feed Forward Neural Network (FFNN) [26] are used for concatenation. Other compositional VQA techniques include the Stacked Attention Network (SAN) [14], and Bilateral Branch Network (BBN) [15], and Bilinear Attention Network (BAN) [16].

The VQA model developed for different medical datasets, using the appropriate techniques is validated with performance metrics. Section 3 explains those metrics and discusses their benefits and computation. In addition, the Ground Truth Answer (GTA) of each images are computed by human annotators as given in Equation 1, to be used in analyzing VQA models. The ‘ACC_GTA’ attains 100% accurate answer if at least three human annotators are given exact answer for an image. ${ACC}_{GTA} = \min {\frac{# answer obtained from^{'} n^{'} annotator}{3}, 1}$ (1)

The rest of the paper is structured as follows. Section 2 discusses the medical VQA datasets and their inferences. Section 3 describes the performance metrics used to analyse the model quantitatively, while Section 4 explains different approaches of medical VQA techniques, with their merits and demerits. Section 5 outlines a medical VQA system with its design. Section 6 explains the experiments and results of medical VQA model using quantitative metrics. Section 7 discusses the challenges of existing medical VQA systems and suggests possible solutions to overcome them. Finally, conclusion and future work is summarized in Section 8.

2 Medical VQA datasets

This section discusses the four medical VQA-MED datasets from ImageCLEF, as well as another dataset named as VQA-RAD. The sample images of different categories of VQA-MED and VQA-RAD datasets are shown in Figs. 2 3 respectively. Moreover, the description about these medical VQA datasets are given in Table 1.

Fig. 2

Samples from the ImageCLEF VQA-MED datasets.

Fig. 3

Samples from the VQA-RAD dataset.

Table 1

Dataset description

Datasets	Training set		Validation set		Test set		Total samples
	No. of images	QA pairs	No. of images	QA pairs	No. of images	QA pairs	No. of images	QA pairs
VQA-MED 2018 [17]	2278	5413	324	500	264	500	2866	6413
VQA-RAD [12]	275	3115	–	–	40	400	315	3515
VQA-MED 2019 [18]	3200	12792	500	2000	500	500	4200	15292
VQA-MED 2020 [19]	4000	4000	500	500	500	500	5000	5000
VQA-MED 2021 [20]	4500	4500	500	500	500	500	5500	5500

2.1 VQA-MED 2018 dataset

The medical VQA-MED dataset developed by Hasan et al. in 2018 has radiology images along with their captions, which is extracted from PubMed Central articles and part of the ImageCLEF 2017 caption prediction task [21]. The two expert human annotators were assigned to check and validate the question-answer pairs associated with the radiology images in two passes. The syntactic and semantic correctness were checked by a human annotator in the first pass. In the second pass, the validation and test sets were evaluated for clinical relevance by a medical expert, the second annotator. As a result, the corpus consists of 2,866 medical images associated with 6,413 question-answer pairs that have been finalized.

2.2 VQA-RAD dataset

The VQA-RAD dataset overcomes two drawbacks of the VQA-MED 2018 dataset. The two drawbacks include, firstly, a number of irrelevant questions and, secondly, image sequences and 3D reconstructions, both of which are rarely used in direct patient care.

In [12], Lau et al. constructed this dataset which focuses on CT/MRI images of head, chest, and abdomen. In this, the question types are related to planes, modalities, organ systems, abnormalities, colors, and sizes as shown in Fig. 3. The answer types are open-ended and closed-ended. This dataset consists of 315 images associated with 3515 Question-Answer (QA) pairs, of which 1515 are free-form and 733 are rephrased questions. The remaining 1267 questions are framed and have corresponding free-form and rephrased questions.

2.3 VQA-MED 2019 dataset

The VQA-MED 2019 dataset shows an increase in the number of samples and their diversity, and it is more organized than the 2018 dataset. This is achieved by selecting relevant radiology images from the MedPix database with filters corresponding to captions, planes, categories, modalities, localities, and diagnostic methods. The VQA-MED 2019 dataset consists of samples related to sixteen planes, ten organs, and thirty-six modalities. As a result, the overall dataset size has increased to 4,200 images, each associated with 3 to 4 questions, along with 12,792 QA pairs [18]. Further, a medical doctor and a radiologist performed a manual double validation of the test answers and corrected ten questions, which is 3% of the total test set size.

2.4 VQA-MED 2020 dataset

The VQA-MED 2020 dataset focuses on abnormality-type samples of different organs, planes, and modalities. The ImageCLEF forum created this dataset automatically through the process of (i). Applying filters to select relevant images and associated annotations (ii). Creating patterns to generate questions and answers (iii) Selecting relevant medical images from the Med-Pix database based on their captions, localities, and diagnostic methods. The final list has 330 medical problems, where each problem occurs at least 10 times in the created VQA data. This dataset consists of 5000 radiology images and 5000 QA pairs that are divided among training, validation, and test sets [19]. The most common medical problems and their frequencies in this dataset are pulmonary embolism (114), acute appendicitis (109), angiomyolipoma (68), osteochondral (63), adenocarcinoma of the lung (60), and sarcoidosis (58).

2.5 VQA-MED 2021 dataset

The VQA-MED 2021 dataset [20] is primarily on abnormality-type questions for different categories like organs, planes, and modalities, like the previous year. This dataset consists of 5,500 samples, of which 4,500 are from the VQA-MED 2020 dataset, and the icomplete dataset is divided into training, validation, and test sets. Compared to 2020, the quality of the 2021 data has shown a marked improvement by having the reference answers of the test set validated by a medical doctor.

2.6 Dataset interpretation and analysis

The interpretation and analysis from the medical VQA datasets are given in Tables 2 , 4 and 5 and, Fig. 4 for better understanding. The utilization of these datasets are analyzed in terms of advantages and disadvantages as given in Table 2.

Table 2
Advantages and disadvantages of medical VQA datasets

Datasets Advantages Disadvantages (Drawbacks)

VQA-MED 2018 [17] 1) The VQA-MED 2018 dataset (released by ImageCLEF) is the first open source medical VQA dataset, making of availability of such data.2) Samples are related to different organs, planes, modalities and abnormalities. Number of samples under each modalities is not uniformly distributed (e.g., MRI (74 samples), CT (70 samples), Abdominal CT (36 samples), No (28 samples), Coronal CT (26 samples)).

VQA-RAD [12] 1) The VQA-RAD dataset consists of questions related to different planes, modalities, abnormalities, organs, colours and, sizes, numbers, positions, presence and attributes of objects in the image.2) Reliability of the dataset is evaluated by fourteen annotators. CT and MRI scan images are limited to head, chest and abdomen only.

VQA-MED 2019 [18] 1) As compared to VQA-MED 2018 dataset, the number of samples in the training set are increased from 5413 to 12792 in VQA-MED 2019 dataset.2) The VQA-MED 2019 dataset is organised by equally distributed number of samples among organs, planes, modalities and abnormalities categories (ie., 3198 samples in each category). 1) The number of samples are equally distributed however the number of questions under close-ended type (11498 samples) and open-ended type (1294 samples) in training set is imbalanced.2) The abnormality category samples degrades the overall performance of the model generated from this dataset [18].

VQA-MED 2020 [19] The samples in VQA-MED 2020 dataset is primarily on abnormality category in order to reduce the drawbacks in VQA-MED 2019 dataset. For this, the number of labels are normalized from 828 to 330 labels by eliminating least contributing samples. Some of the labels have less number of samples and others labels have relatively more number of samples and it leads to dataset imbalance problem. For example, “Double aortic arch” has 28 samples and “Pulmonary embolism” label has 80 samples (refer Fig. 4(c)).

VQA-MED 2021 [20] 500 samples are included to the VQA-MED 2020 and hence the samples in the training set is increased from 4000 to 4500 samples. Dataset imbalance problem in abnormality category is reduced to some extent, but still an issue.

Datasets	Advantages	Disadvantages (Drawbacks)
VQA-MED 2018 [17]	1) The VQA-MED 2018 dataset (released by ImageCLEF) is the first open source medical VQA dataset, making of availability of such data.2) Samples are related to different organs, planes, modalities and abnormalities.	Number of samples under each modalities is not uniformly distributed (e.g., MRI (74 samples), CT (70 samples), Abdominal CT (36 samples), No (28 samples), Coronal CT (26 samples)).
VQA-RAD [12]	1) The VQA-RAD dataset consists of questions related to different planes, modalities, abnormalities, organs, colours and, sizes, numbers, positions, presence and attributes of objects in the image.2) Reliability of the dataset is evaluated by fourteen annotators.	CT and MRI scan images are limited to head, chest and abdomen only.
VQA-MED 2019 [18]	1) As compared to VQA-MED 2018 dataset, the number of samples in the training set are increased from 5413 to 12792 in VQA-MED 2019 dataset.2) The VQA-MED 2019 dataset is organised by equally distributed number of samples among organs, planes, modalities and abnormalities categories (ie., 3198 samples in each category).	1) The number of samples are equally distributed however the number of questions under close-ended type (11498 samples) and open-ended type (1294 samples) in training set is imbalanced.2) The abnormality category samples degrades the overall performance of the model generated from this dataset [18].
VQA-MED 2020 [19]	The samples in VQA-MED 2020 dataset is primarily on abnormality category in order to reduce the drawbacks in VQA-MED 2019 dataset. For this, the number of labels are normalized from 828 to 330 labels by eliminating least contributing samples.	Some of the labels have less number of samples and others labels have relatively more number of samples and it leads to dataset imbalance problem. For example, “Double aortic arch” has 28 samples and “Pulmonary embolism” label has 80 samples (refer Fig. 4(c)).
VQA-MED 2021 [20]	500 samples are included to the VQA-MED 2020 and hence the samples in the training set is increased from 4000 to 4500 samples.	Dataset imbalance problem in abnormality category is reduced to some extent, but still an issue.

Table 3

Brief inferences from the medical VQA datasets in terms of questions

Datasets	Longest questions	Length of the longest question	Shortest questions	Length of the shortest question
VQA-MED 2018	What reveals prominent bilateral enhancing parietal occipital lesions on the FLAIR and t2 sequences and small areas of hyperintensity in the left periventricular white matter on diffusion weighted images?	28	What is normal?	3
VQA-RAD	The tissue within the hyperattenuating ovoid mass adjacent to the ascending colon is similar in density to what tissue normally found?	21	Is this a CT?	4
VQA-MED 2019	What organ systems can be evaluated with this gastrointestinal image?	10	Is this a CT scan?	5
VQA-MED 2020/2021	What is most alarming about this gastrointestinal image?	8	Is the MRI normal?	4

Table 4

Brief inferences from the medical VQA datasets in terms of answers

Datasets	Longest answers	Length of the longest answer	Shortest answers	Length of the shortest answer	Answer type
					No. of open-ended answers	No. of closed- ended answers
VQA-MED 2018	An abnormal dilated branch of the middle cerebral artery in the left hemisphere in a patient with morning glory syndrome associated with persistent hyperplastic primary vitreous	26	No	1	31	5382
VQA-RAD	Fat stranding around the appendix, thickened appendiceal walls, dilated appendix, and an appendicolith is seen as well	17	4	1	949	1298
VQA-MED 2019	Benign and malignant cystic ovarian tumors. ⊤ápatient a: ⊤áserous cystadenoma ⊤ápatient b: ⊤ámucinous cystadenoma ⊤ápatient c: ⊤ápapillary serous cystadenocarcinoma	19	No	1	1294	11499
VQA-MED 2020/2021	Common carotid occlusion; collateral reconstitution of ipsilateral ICA from vertebral artery muscular branches	13	No	1	60	3940
4440

Table 5

Maximum-occurring questions and answers with their frequencies

Datasets	Maximum-occurring questions	Frequency	Maximum-occurring answers	Frequency
VQA-MED 2018	What does the CT scan show?	53	MRI	74
VQA-RAD	Is the heart enlarged?	14	No	486
VQA-MED 2019	What abnormality is seen in the image?	776	Axial	1558
VQA-MED 2020	What abnormality is seen in the image?	1001	Pulmonary embolism	80
VQA-MED 2021	What abnormality is seen in the image?	1126	Pulmonary embolism	88

Fig. 4

Radar graph for answers of VQA-MED datasets.

The questions with the maximum and minimum length are tabulated, along with their vocabulary size in Table 3. In this, the questions are of two types, open-ended and closed-ended. The former includes questions that begin with ‘is’, ‘are’ and ‘does’, and the latter with ‘what’, ‘which’, and ‘how’. Of the two question types, those beginning with ‘what’ and ‘is’ comprise the maximum-length and minimum-length questions, respectively.

The maximum and minimum lengthened answer, along with their vocabulary size, are computed and tabulated in Table 4. It is inferred that most of the longest answers offer information related to abnormalities, while the vocabulary size for all the shortest answers is one. Most answers in the datasets are closed-ended, and the number of samples with respect to both open-ended and closed-ended answers is depicted.

The Table 5 shows the maximum occurrence of questions and answers, along with their respective frequencies. In this, the “what” question type (open-ended) and “pulmonary embolism” (answer) occurs frequently in most of the medical VQA datasets.

The consecutively occurring answer and its frequency are illustrated in Fig. 4 for four medical VQA-MED datasets using a radar graph. In the graph, the answer with the maximum frequency takes the outer circle and, depending on the decreasing level of frequency level, the respective answer will take the inner circles. From the Fig. 4(b) of VQA-MED 2019 dataset, it is inferred that “t2” (least frequently occurring answer) takes the smallest elevation in the inner most circle and “axial” (most frequently occurring answer) takes the largest elevation and it touches the outer circle.

3 Performance metrics of medical VQA system

The performance of the VQA model is validated by means of quantitative metrics discussed in this section. The system-generated answer is evaluated with ground-truth answers using performance metrics like the Bilingual Evaluation Understudy (BLEU) score, Word-Based Semantic Similarity (WBSS), and accuracy. All three performance metrics range from 0.0 (completely different results) to 1.0 (the ideal result). The BLEU score [24] is calculated by counting the number of n-grams in the system-generated answer to the number of n-grams in the ground truth answer as given in Equation 2. The WBSS Score is computed based on the Wu-Palmer Similarity (WUPS) (Wu and Palmer, 1994), with WordNet ontology in the back end, as computed in Equation 3. In 2015, Antol et al. [1] stated that accuracy measures the exact match between model-generated and ground truth answers and it is computed based on Equation 4. $p_{n} = \frac{\sum_{S \in C} \sum_{ngram \in S} {Count}_{matched} (ngram)}{\sum_{S \in C} \sum_{ngram \in S} Count (ngram)}$ (2) ${Sim}_{WUPD} (C_{1}, C_{2}) = \frac{2 * depth (LCS)}{depth (C_{1}) + depth (C_{2})}$ (3) $Accuracy = \frac{TP + TN}{TP + FP + FN + TN}$ (4)

In Equation 2, C represents the corpus, S the sentences in the corpus, n-gram the continuous sequence of n-items in a sentence, and p_n the BLEU n-gram precision (i.e., it computes n-gram matches for every hypothesis sentence ‘S’ in the corpus ‘C’). The C₁ and C₂ in Equation 3 represents the two concept nodes in the WordNet taxonomy. The Lowest Common Subsumer (LCS) in this equation represents the common parent of C₁ and C₂, with a minimum node distance. Depth C₁ indicates the number of nodes from C₁ to the LCS node, depth C₂ represents the number of nodes from C₂ to the LCS node, and depth LCS indicates the number of nodes from the LCS node to the root node. In Equation 4, TP, TN, FP, and FN represent true positive, true negative, false positive, and false negative, respectively. The numerator in this equation represents the correctly classified number of samples and the denominator denotes the total number of samples.

4 Deep neural network techniques for medical VQA

This section explains the deep neural network techniques used in developing medical VQA models. Existing VQA approaches are categorized into joint embedding, hybrid and compositional, depends on the uniqueness of the techniques. In the joint embedding approach, individual techniques are used for both image and text feature extraction, while feature concatenation is undertaken by a single architecture. In the hybrid VQA approach, the attention mechanism which is used alongside with the joint embedding. The third category is the compositional approach, in this the feature extraction and concatenation are executed by a single architecture. These approaches are listed in Tables 6 , 7 and 8 in regard to the datasets used and the performance metrics applied under different categories. The merits, demerits and the usage of different VQA approaches are given in Table 9.

Table 6
A comparison and analysis of different medical VQA techniques in the joint embedding VQA approach

Datasets Author and Year Techniques Performance metrics

Accuracy BLEU Score WBSS Score

VQA-MED 2018 Allaouzi et al., [32] VGG16 + LSTM+DT – 0.054 0.101

Talafha et al., [33] VGGNet+Seq2Seq+LSTM – 0.060 0.120

Peng et al., [6] ResNet+LSTM+MFB – 0.162 0.184

VQA-RAD Liu et al., [26] VGGNet+Recurrent Neural Network+FFNN (image retrieval) 0.727 – 0.753

Zhang et al., [27] CoTNet152 + BioM-Transformer 0.765 – –

VQA-MED 2019 Bounaama et al., [34] CNN+RNN+LSTM 0.462 0.486 –

Turner et al., [28] CNN+LSTM+MLP 0.530 0.550 –

Allaouzi et al., [35] DenseNet+LSTM+GSA 0.556 0.583 –

Al-Sadi et al., [29] VGG16 (last layer equal to the no. of classes in each category) 0.570 0.591 –

VQA-MED 2019 (modality category) Lubna et al., [25] CNN (classifies the image to the modality). Generates the answer based on the CNN output 0.838 – –

VQA-MED 2020 Mohamed et al., [7] VGGNet+LSTM+LSTM 0.282 0.330 –

Umada et al., [9] VGG16 + BERT+GLU 0.314 0.350 –

Sarrouti. [4] CNN+LSTM+MLP 0.400 0.441 –

Al-Sadi et al., [30] VGG16 + Data augmentation techniques 0.480 0.339 –

VQA-MED 2021 Sitara et al., [31] VGG16 + LSTM+LSTM (+DA) 0.196 0.227 –

Xiao et al., [11] VGG16 + BioBERT+MFH 0.362 0.402 –

Datasets	Author and Year	Techniques	Performance metrics
VQA-MED 2018	Allaouzi et al., [32]	VGG16 + LSTM+DT	–	0.054	0.101
	Talafha et al., [33]	VGGNet+Seq2Seq+LSTM	–	0.060	0.120
	Peng et al., [6]	ResNet+LSTM+MFB	–	0.162	0.184
VQA-RAD	Liu et al., [26]	VGGNet+Recurrent Neural Network+FFNN (image retrieval)	0.727	–	0.753
	Zhang et al., [27]	CoTNet152 + BioM-Transformer	0.765	–	–
VQA-MED 2019	Bounaama et al., [34]	CNN+RNN+LSTM	0.462	0.486	–
	Turner et al., [28]	CNN+LSTM+MLP	0.530	0.550	–
	Allaouzi et al., [35]	DenseNet+LSTM+GSA	0.556	0.583	–
	Al-Sadi et al., [29]	VGG16 (last layer equal to the no. of classes in each category)	0.570	0.591	–
VQA-MED 2019 (modality category)	Lubna et al., [25]	CNN (classifies the image to the modality). Generates the answer based on the CNN output	0.838	–	–
VQA-MED 2020	Mohamed et al., [7]	VGGNet+LSTM+LSTM	0.282	0.330	–
	Umada et al., [9]	VGG16 + BERT+GLU	0.314	0.350	–
	Sarrouti. [4]	CNN+LSTM+MLP	0.400	0.441	–
	Al-Sadi et al., [30]	VGG16 + Data augmentation techniques	0.480	0.339	–
VQA-MED 2021	Sitara et al., [31]	VGG16 + LSTM+LSTM (+DA)	0.196	0.227	–
	Xiao et al., [11]	VGG16 + BioBERT+MFH	0.362	0.402	–

Table 7

A comparison and analysis of different medical VQA techniques in the hybrid VQA approach

Datasets	Author and Year	Techniques	Performance metrics
			Accuracy	BLEU Score	WBSS Score
VQA-MED 2018	Zhou et al., [42]	Inception ResNet V2 + Bi-LSTM+Attention mechanism	–	0.135	0.174
VQA-MED 2019	Liu et al., [8]	Xception+GRU+Attention mechanism	0.210	0.393	–
	Bansal et al., [43]	ResNet+Attention mechanism (Image)+USE+Word2Vec(text) +Seq2Seq (concatenate)	0.480	0.530	–
	Shi et al., [13]	ResNet152 + Co-attention (Image)+SVM (Ques. category)+LSTM (Text)+MFH (concatenate)	0.566	0.593	–
VQA-MED 2020	Liu et al., [38]	VGG16 + GRU+Seq2Seq+ Attention mechanism	0.376	0.412	–
	Verma and Ramachandran, [37]	VGG16 + BERT+MFB+Bahdanau attention	0.378	0.439	–
	Jung et al., [36]	VGG16 + Bio-BERT+MFH pooling with co-attention	0.466	0.502	–
VQA-RAD	Zhu et al., [41]	ResNet-34 + LSTM+BAN+Semantic attention	0.725	–	–
	Wang et al., [39]	ResNet-34 + Bio-WordVec+LSTM+Cross-model self-attention	0.741	–	–
IDRiD and e-Ophta dataset	Tascon-Morales et al., [40]	Visual encoder+Textual encoder +Attention mechanism+ Multimodal fusion	0.835	–	–

Table 8

A comparison and analysis of different medical VQA techniques in the compositional VQA approach

Datasets	Author and Year	Techniques	Performance metrics
			Accuracy	BLEU Score	WBSS Score
VQA-MED 2018 dataset	Abacha et al., [14]	SAN	–	0.121	0.174
VQA-RAD dataset	Lau et al., [12]	MCB	0.542	0.468	–
VQA-MED 2020 dataset	Chen et al., [45]	BBN	0.426	0.462	–
	Liao et al., [44]	SSM	0.496	0.542	–
Path VQA dataset	Do et al., [16]	SAN+MMQ process+BAN	0.840	–	–
VQA-RAD dataset			0.670
VQA-MED 2021 dataset	Eslami et al., [15]	BBN	0.348	0.391	–

Table 9

Merits, demerits and the usage of medical VQA approaches

Approaches	Merits	Demerits	When can be used/reason for using
Joint embedding	Works well for VQA-MED 2018, 2019, 2021 and VQA-RAD datasets (refer Tables 6 , 7 and 8).	Based on the different characteristics of the dataset (image category, single/multiple region contributes to the answer prediction, count and type of samples), the appropriate for feature extraction techniques need to be selected	This approach can be chosen when there is an clear understanding about the dataset (dataset formation, type of questions and answers, image category) and knowledge about the possible techniques.
Hybrid	Predict the more suitable answer from the significant region of the image.	Usually medical VQA dataset requires more time and memory for execution, attention mechanism still increases the training time.	This approach can be used when the answer to the question requires the features from more than one region of the image.
Compositional	Performs significantly better for VQA-MED 2020 and Path-VQA datasets (refer Tables 6 , 7 and 8).	These approaches are not initially developed for medical VQA dataset so lacks when there is an more number of words in the questions related to medical terms.	This approach is used when the characteristics of the dataset are not completely understandable but the performance of the dataset need to be evaluated.

4.1 Joint embedding VQA approach

The different joint embedding VQA methods are used by various researchers worldwide for four medical VQA datasets. Among these, some of the significant methods are discussed in this section.

Allaouzi et al. developed a VQA model using a combination of VGG16, LSTM and decision tree, resulting in a BLEU score of 0.054 because the decision tree-based classification approach ignored word order information. However, Peng et al. achieved an improved BLEU score of 0.108 using ResNet, LSTM and MFB, since the model captured and integrated fine-grained contextual information effectively for the VQA-MED 2018 dataset.

For the VQA-MED 2019 dataset, Allaouzi et al. used the Greedy Search Algorithm (GSA) to predict the next word based on the previously generated words, and the final answer is built by recursively calling the model resulted a BLEU score of 0.624. Al-Sadi et al. created the VQA model using five sub-models (organs, planes, abnormalities, modalities, and answer prediction). Of these, the first four are image classification models pre-trained on the VGG16 network, while the last is developed by the LSTM, based on the text, with repetitive questions for all categories. The results show that accuracy for planes, organs, and modalities ranges from 65% to 72% with improvisation, while the abnormality model accuracy is as low as 18%, which reduces the overall performance and therefore requires sufficient attention. Lubna et al. [25] constructed a VQA classification model to answer modality-based questions alone for different radiology images using the CNN and obtained the highest accuracy of 83.8%.

The method namely GLU, Multi-Layer Perceptron (MLP), are applied for VQA-MED 2020 dataset and Bio-BERT for VQA-MED 2021 dataset. The GLU technique narrow down the predicted answer to some extent but it was not adequate enough to handle questions related to abnormality types and hence the resulted an BLEU score of 0.350. The MLP classify the abnormality category effectively as a multi-class image classification problem with an attention score for each region, as a result it achieved an BLEU score of 0.441. The BioBERT extract semantic information from the dataset and, in this way, enhance the overall performance of the model with an BLEU score of 0.402. Liu et al., [26] concluded that image retrieval (retrieving the answer for the sample in the test set based on the maximum cosine similarity value with the samples in the training set) lacks for other than VQA-RAD, VQA-MED 2018 and 2019 VQA datasets. Zhang et al., [27] represented medical image as a type point and combined with QA-pairs because it improved the reasoning ability of different modalities.

The overall inferences from the Table 6 are, (i). The role of LSTM is not limited to text feature extraction but also to the feature concatenation (ii). Both Turner et al. [28] and Sarrouti. [4] used the same combination of techniques and attained an reasonable BLEU score of 0.550 and 0.441 for two different datasets namely, VQA-MED 2019 and 2020 datasets. (iii). Al-Sadi et al. [29, 30] used VGG16 for VQA-MED 2019 and 2020 datasets, and they incorporated data augmentation techniques in the later research.

4.2 Hybrid VQA approach

In Hybrid VQA approach, an attention mechanism like co-attention or the bahdanau attention is used with the joint embedding approach and it is shown in Table 7. These attention mechanisms are used alongside image feature extraction or after feature concatenation because they are capable of accessing every feature and selects specific features from a sequence to produce an answer. The co-attention mechanism generated the attentional features for both image and text, based on the attention weight of different grid locations [13]. The MFH with co-attention mechanism learned the relatively important information from each part of the visual and textual features and ignored the irrelevant information [37]. The bahdanau attention maps the relationship between local image features when the mapping between the global image features and the text was insufficient [38]. In 2022, Wang et al. [39] stated that Cross-model self-attention captures the long-range dependencies between image and QA-pairs which helps in better feature representation to focus on significant information. Tascon-Morales et al., [40] focuses on answer consistency instead of improving VQA architecture or reducing data-related limitations. Because the improvement in the answer consistency (answer to sub-questions implies to main questions) increases the performance of the model. The Zhu et al., [41] stated that introduction of semantic attention showed the certain degree of improvement in the answer prediction.

The overall inference from the Table 7 are, Liu et al. [8, 38] used pre-trained along with GRU and attention mechanism for VQA-MED 2019 and 2020 datasets. In the 2020 research, they upgraded the pre-trained model based on the characteristics of the datasets, and applied seq2seq techniques for fusion mechanism.

4.3 Compositional VQA approach

The compositional VQA approach uses advanced and unique techniques like the Multimodal Compact Bilinear Pooling (MCB), Stacked Attention Network (SAN), Bilateral Branch Network (BBN), Skeleton-based Sentence Matching (SSM) and Bilinear Attention Network (BAN). These techniques are arranged in terms of datasets and performance metrics in Table 8, and the inferences are discussed in this section.

In [12], Lau et al. observed that the MCB is able to differentiate the variance in the predicted answers from that of similar questions in the VQA-RAD dataset and it attained an BLEU score of 0.468. Abacha et al. [14] incorporated the MCB for the VQA-MED 2018 dataset, owing to its multimodal representation capability, and compared it with the SAN. The MCB reduced the number of parameters but the higher dimensionality computation increases its complexity. Moreover, SAN outperformed MCB because it queries an image multiple times and infers the answer progressively. In addition to this, the fine-tuning of the model prevented overfitting and improved performance slightly. Going one step further, Do et al. [16] integrated the SAN with the Multiple Meta-model Quantifying (MMQ) process, and the resultant model learned essential information from images and meta-models to answer challenging questions in the VQA-RAD and PathVQA (Pathology) datasets.

Liao et al. [44] incorporated the SSM where questions with a similar structure are summarized into a unified backbone to simplify the question-comprehension segment. According to Chen et al. [45], long-tailed distributions of candidate answers in the VQA-MED 2020 dataset can be addressed by the BBN with a cumulative learning strategy. Their approach constructed a retrieval-based answer section by computing the cosine similarity between the input sample and all the training samples of the top five categories. Taking a step ahead, Eslami et al. [16] used a BBN ensemble to reduce overfitting in the training and validation data of the VQA-MED 2021 dataset by effectively modeling the imbalanced long-tailed distribution.

5 The proposed VQA system

The proposed VQA system comprises of four stages namely, dataset improvisation, medical VQA model generation, answer prediction and performance analysis as shown in Fig. 5.

Fig. 5

Proposed medical VQA system.

The VQA-MED 2020 and 2021 datasets are used in this proposed work. ImageCLEF, focused on abnormality-type questions in VQA-MED 2020 and 2021 datasets, however the overall model performance was minimal due to data imbalance problems. To deal with this difficulty, data reduction and augmentation approaches are used with different combinations in the proposed work for these datasets. Then the dataset is divided into training and test for VQA model generation and prediction.

The dataset reduction ways are, (i). Eliminate the data pertaining to least contributing labels (ii). Identify and reduce the samples from the labels where the count of the samples in the particular label is excess than the average count required. The resulted dataset is named as Reduced Dataset (RD). The dataset augmentation ways are, (i). Identify the similar type samples from other medical VQA datasets and augment. (ii). Incorporate augmentation techniques namely, mixup and label smoothing, and generate data. The resulted dataset is named as Augmented Dataset (AD). The dataset augmentation after dataset reduction gives better results because the reduced dataset consists of effective samples and it is named as Augmented Reduced Dataset (ARD).

The medical VQA model generated using joint embedding approach (VGGNet and LSTM) is explained in Algorithms 1 and 2. The Algorithm 1, explains the generation of VQA model for the training set and it comprises encoder and decoder phases. The Algorithm 2, a sub-part of Algorithm 1, explains word vector generation using embedding method. The answer prediction for the test set based on the generated VQA model is detailed in Algorithm 3. The predicted answer are evaluated using two performance metrics namely, accuracy and BLEU score.

Algorithm 1 Medical VQA model

Algorithm 2 Generate word vectors by embedding

Algorithm 3 Answer prediction form medical VQA model

6 Experiments and results

The performance analysis of medical VQA model for VQA-MED 2020 and 2021 datasets are given in Table 10. In this table, OD, RD, AD and ARD represents the Original Dataset, Reduced Dataset, Augmented Dataset and Augmented Reduced Dataset and ATD represents the augmentation of VQA-MED 2020’s test set (VQA-MED 2020) with VQA-MED 2021 dataset.

Table 10
VQA-MED datasets Vs Performance metrics for the proposed model

VQA-MED datasets Original and improved datasets Training set No. of classes Accuracy BLEU score

Images QA pairs

2020 OD 4500 4500 330 0.276 0.323

AD 4778 4778 316 0.279 0.326

RD 4348 4348 305 0.248 0.292

ARD 4626 4626 316 0.282 0.330

2021 OD 5000 5000 330 0.172 0.213

AD 5626 5626 366 0.172 0.211

RD 4722 4722 316 0.168 0.205

ARD 5348 5348 316 0.170 0.208

ATD 5500 5500 366 0.196 0.227

VQA-MED datasets	Original and improved datasets	Training set	No. of classes	Accuracy	BLEU score
2020	OD	4500	4500	330	0.276	0.323
	AD	4778	4778	316	0.279	0.326
	RD	4348	4348	305	0.248	0.292
	ARD	4626	4626	316	0.282	0.330
2021	OD	5000	5000	330	0.172	0.213
	AD	5626	5626	366	0.172	0.211
	RD	4722	4722	316	0.168	0.205
	ARD	5348	5348	316	0.170	0.208
	ATD	5500	5500	366	0.196	0.227

From Table 10, it has been inferred that ARD of VQA-MED 2020 and ATD of VQA-MED 2021 datasets resulted a better BLEU Score of 0.330 and 0.227 than original datasets. Also, the accuracy of the improved dataset is increased by 0.6% and 2.4% for VQA-MED 2020 (ARD) and 2021 (ATD) datasets than the original datasets. When compared with Mohamed et al., [7], the proposed model achieved 0.3% and 0.4% improved accuracy and BLEU score for VQA-MED 2020 dataset. Besides, the accuracy and BLEU Score for Reduced Dataset (RD) and Augmented Dataset (AD) are also computed but it is comparatively less than the ARD and ATD.

In Table 11, the number of correctly classified samples for the medical VQA dataset are tabulated along with the improved percentage of samples. From the table, it has been inferred that the number of correctly classified samples is increased by 0.6% and 1.4% for improved datasets of VQA-MED 2020 (ARD) and 2021 (ATD), when compared with original datasets. The increase in percentage is calculated from the number of correctly classified samples (Table 11) to the total number of samples (Table 1), for both datasets.

Table 11

Number of correctly classified samples Vs datasets

VQA-MED datasets	Training set		Validation set		Test set		Increase in %
	OD	ARD/ATD	OD	ARD/ATD	OD	ARD/ATD
2020	1310	1341	178	182	162	165	0.6
2021	1001	1069	121	130	107	114	1.4

7 Discussion

From the conducted investigations, the following inferences are made. (i). The VQA-MED datasets from 2018 to 2021 are improved each year based on the knowledge gained from the previous year dataset. (ii). Among the datasets, the length of the question ranges from 3 to 28 words and the length of the answer ranges from 1 to 26 words. (iii). The range of maximum frequency of the question with respect to a class is 14 to 1126 and, for answer is 74 to 1558. (iv). The three approaches are used in different scenarios depending on the characteristics of the dataset as, joint embedding is chosen when there is an clear understanding about the dataset, hybrid embedding is when the answer to the question requires the features from more than one region of the image and compositional approach can be selected when the characteristics of the dataset are not completely understandable but the performance of the dataset need to be evaluated. (v). The dataset reduction and augmentation improves the performance of the model. For instance, the performance of the model is increased by 0.6% and 1.4% for VQA-MED 2020 and 2021 dataset respectively.

Moreover, different challenges in medical VQA datasets, techniques, performance metrics and computational efficiency are discussed in Table 12 with possible solutions, as applicable. Finally, the proposed system has been compared with other medical VQA works.

Table 12
Challenges and solutions in medical VQA domain

Category Challenges Possible solutions

Dataset Collection (i) Datasets collected from clinics, hospitals, medical institutions, or existing medical mediums (say pathology books) need due acknowledgements from medical authorities and patients.(ii) Medical images acquired from different modalities require expert knowledge and suggestions from radiologists, gynecologists, gastroenterologists, and physicians to create ground truth values.(iii) Number of open-sourced medical VQA datasets is insufficient. (i) Augmenting the dataset in terms of flipping, rotating, shifting widths/heights, and zooming, as recommended by Al-Sadi et al., [30].(ii) Reduced the pathology image data insufficiency problem to some extent by capturing images and associated question-answer pairs from pathology textbooks and online digital libraries, using a semi-automated pipeline [46].

Dataset (Images) (i) Different modalities such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Ultrasound, mammography and PET (Positron Emission Tomography) generate medical images in different formats, ranging from grayscale to RGB.(ii) The medical image acquisition medium captures the different radiology images from the same region depending on the location and the severity of the disease (which differs by angle, magnification concerning the requirement and plane). Reduced by segmenting the Region of Interest (ROI), based on the modality type.

Dataset (QA-pairs) (i) QA-pairs need to be created and validated in terms of context by linguistic and medical experts, owing to the occurrence of both medical and non-medical terms.(ii) Medical QA-pairs are complex because questions are related to multiple regions or a very small region in the image.(iii) Certain questions required external expert knowledge base to answer the questions(iv) QA-pairs concerning on all abnormality types are unavailable. (i) Use attention based techniques when questions require information from multiple regions of the image.(ii) Develop and use the corpus to answer the questions which requires expert knowledge.

Quantitative metrics The performance is affected due to wrong interpretation in the following cases.(i) When a predicted answer is semantically similar to the ground truth value (e.g., Ollier disease; enchondromatosis)(ii) When the words in the predicted answer are misordered (eg., CT scan with IV; IV in CT)(iii) When the context of the word is the same but the answer is represented using different phrases (e.g., face, sinuses, and neck; skull and contents)(iv) When an abbreviation in the answer is used instead of the original word (e.g., MRI; magnetic resonance imaging)(v) When a more detailed answer is generated instead of a general terminology (e.g., Paget’s disease; Paget’s disease of bone)(vi) When part of an answer or a partially correct answer is generated (e.g., femoral neck stress fracture; fracture). Some other quantitative metrics which measures the semantically similar answers are also considered to evaluate the performance of the model.

Techniques (i) The main challenge involved in developing an efficient VQA model, is imbalanced class label distribution for samples in the dataset (Al-Sadi et al., [29]).(ii) Similar region of interest extraction techniques may not be used for different medical images that vary in terms of modalities, planes, and organs [35]. (i) Choosing suitable medical VQA approaches based on radiology images and question-answer pair categories, simply because one algorithm good at computing tumour intensity may not be good at counting the number of ribs.(ii) The process of selecting hyperparameters and defining their respective values is to be determined ahead of time for effective VQA model generation

Computing efficiency Challenges in terms of memory and time constraints are common when large volumes of information are extracted. (i) Can be reduced by utilizing high performance computing system.(ii) Problems with memory constraints may be reduced by selecting suitable batch sizes.

Category	Challenges	Possible solutions
Dataset Collection	(i) Datasets collected from clinics, hospitals, medical institutions, or existing medical mediums (say pathology books) need due acknowledgements from medical authorities and patients.(ii) Medical images acquired from different modalities require expert knowledge and suggestions from radiologists, gynecologists, gastroenterologists, and physicians to create ground truth values.(iii) Number of open-sourced medical VQA datasets is insufficient.	(i) Augmenting the dataset in terms of flipping, rotating, shifting widths/heights, and zooming, as recommended by Al-Sadi et al., [30].(ii) Reduced the pathology image data insufficiency problem to some extent by capturing images and associated question-answer pairs from pathology textbooks and online digital libraries, using a semi-automated pipeline [46].
Dataset (Images)	(i) Different modalities such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Ultrasound, mammography and PET (Positron Emission Tomography) generate medical images in different formats, ranging from grayscale to RGB.(ii) The medical image acquisition medium captures the different radiology images from the same region depending on the location and the severity of the disease (which differs by angle, magnification concerning the requirement and plane).	Reduced by segmenting the Region of Interest (ROI), based on the modality type.
Dataset (QA-pairs)	(i) QA-pairs need to be created and validated in terms of context by linguistic and medical experts, owing to the occurrence of both medical and non-medical terms.(ii) Medical QA-pairs are complex because questions are related to multiple regions or a very small region in the image.(iii) Certain questions required external expert knowledge base to answer the questions(iv) QA-pairs concerning on all abnormality types are unavailable.	(i) Use attention based techniques when questions require information from multiple regions of the image.(ii) Develop and use the corpus to answer the questions which requires expert knowledge.
Quantitative metrics	The performance is affected due to wrong interpretation in the following cases.(i) When a predicted answer is semantically similar to the ground truth value (e.g., Ollier disease; enchondromatosis)(ii) When the words in the predicted answer are misordered (eg., CT scan with IV; IV in CT)(iii) When the context of the word is the same but the answer is represented using different phrases (e.g., face, sinuses, and neck; skull and contents)(iv) When an abbreviation in the answer is used instead of the original word (e.g., MRI; magnetic resonance imaging)(v) When a more detailed answer is generated instead of a general terminology (e.g., Paget’s disease; Paget’s disease of bone)(vi) When part of an answer or a partially correct answer is generated (e.g., femoral neck stress fracture; fracture).	Some other quantitative metrics which measures the semantically similar answers are also considered to evaluate the performance of the model.
Techniques	(i) The main challenge involved in developing an efficient VQA model, is imbalanced class label distribution for samples in the dataset (Al-Sadi et al., [29]).(ii) Similar region of interest extraction techniques may not be used for different medical images that vary in terms of modalities, planes, and organs [35].	(i) Choosing suitable medical VQA approaches based on radiology images and question-answer pair categories, simply because one algorithm good at computing tumour intensity may not be good at counting the number of ribs.(ii) The process of selecting hyperparameters and defining their respective values is to be determined ahead of time for effective VQA model generation
Computing efficiency	Challenges in terms of memory and time constraints are common when large volumes of information are extracted.	(i) Can be reduced by utilizing high performance computing system.(ii) Problems with memory constraints may be reduced by selecting suitable batch sizes.

Besides, the proposed work is compared to that of Lin et al. [47] and Al-Sadi et al. [48]. As compared with Lin et al., [47], in this paper the medical VQA datasets are analyzed and significant information are interpreted which is considered for the VQA model creation. Additionally, limitations in terms of datasets, performance metrics, and techniques are also included in each section. Though Al-Sadi et al. [48] worked with the VQA-MED 2019 dataset alone in 2021, our work has dealt with the VQA-MED 2020 and 2021 datasets and used suitable augmentation techniques to maximize the performance of the model.

8 Conclusion and future work

This paper focuses on four different perspectives namely, (i). Comprehensive interpretation of medical VQA datasets and techniques, (ii). Improvisation of dataset based on the inference from the interpretation of dataset using data reduction and augmentation approaches (iii). Generation of medical VQA model for original and improvised dataset and comparison of the results using quantitative metrics. (iv). Summarizing the challenges in the medical VQA dataset acquisition, techniques, performance metrics and memory requirement which pave way for further development in medical VQA domain. The five medical VQA datasets are studied and the significant information from the samples are inferred, which helps to understand the dataset and it also pave way for the dataset improvisation. The dataset is improvised by data reduction and augmentation approaches to enhance the performance. The medical VQA model developed for these datasets are compared and analysed using quantitative metrics. From the results, it has been inferred that Augmented Reduced Dataset (ARD) and Augmentation of VQA-MED 2020 Test set with VQA-MED 2021 Dataset (ATD) gives an improved accuracy of 0.6% and 2.4% compared to VQA-MED 2020 and 2021 datasets respectively. The challenges in medical VQA dataset are discussed along with their possible solutions. This interpretation from datasets, categorization of approaches, development of medical VQA model for five datasets, analyse the research gap in terms of challenges made this paper distinct with other medical VQA review and survey paper.

In future work, we plan to address the challenges in the medical VQA domain and incorporate the advanced medical VQA techniques to improve the performance the model.

References

Antol

, Agrawal

, Mitchell

, Batra

, Zitnick

C.L.

and Parikh

, VQA: Visual Question Answering, Proceeding of the IEEE International Conference on Computer Vision, Santiago, China, 2015, 2425–2433.

Uehara

, Duan

and Harada

, Learning To Ask Informative Sub-Questions for Visual Question Answering, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, 4681–4690.

Gasmi

, Ltaifa

I.B.

, Lejeune

, Alshammari

, Ammar

L.B.

and Mahmood

M.A.

, Optimal deep neural network-based model for answering visual medical question, Cybernetics and Systems 53(5) (2022), 403–424.

Sarrouti

, NLM at VQA-MED 2020: Visual Question Answering and Generation in the Medical Domain, CLEF 2020 Working Notes, CEUR Workshop Proceedings, Greece, 2020, Paper 98, 1–10.

Al-sadi

, Al-Theiabat

and Al-Ayyoub

, The Inception Team at VQA-Med 2020: Pretrained VGG with Data Augmentation for Medical VQA and VQG, CLEF 2020 Working Notes, CEUR Workshop Proceedings, Thessaloniki, Greece, 2020, Paper 69, 1–12.

Peng

, Liu

and Rosen

M.P.

, Umass at Imageclef Medical Visual Question Answering (MED-VQA) 2018 task, Working Notes of CLEF 2018 –Conference and Labs of the Evaluation Forum, Avignon, France, 2018, Paper 163, 1–9.

Mohamed

S.S.N.

and Srinivasan

, ImageCLEF 2020: An approach for Visual Question Answering using VGGLSTM for Different Datasets, CLEF 2020 Working Notes, CEURWorkshop Proceedings, Thessaloniki, Greece, 2020, Paper 94, 1–10.

Liu

, Ou

and Che

, VQA-med: An Xception-GRU model, CLEF 2019 Working Notes, CEUR Workshop Proceedings, Switzerland, 2019, Paper 127, 1–9.

Umada

and Aono

, Kdevqa at VQA-MED 2020: Focusing on GLU-based Classification, CLEF 2020 Working Notes, CEUR Workshop Proceedings, Greece, 2020, Paper 81, 1–9.

10.

Yan

, Li

, Xie

, Xiao

and Gu

, Zhejiang University at Imageclef 2019 Visual Question Answering in the Medical Domain, Working Notes of CLEF 2019 Conference and Labs of the Evaluation Forum, Lugano, Switzerland, 2019, Paper 85, 1–9.

11.

Xiao

, Zhou

, Xiao

and Zhao

, Yunnan University at VQA-Med 2021: Pretrained BioBERT for Medical Domain Visual Question Answering, CLEF 2021 Working Notes, CEUR Workshop Proceedings, Bucharest, Romania, 2021, Paper 120, 1–7.

12.

Lau

J.J.

, Gayen

, Abacha

A.B.

and Dener-Fushman

, A Dataset of Clinically Generated Visual Questions and Answers about Radiology Images, Scientific Data 5 (2018), 1–10.

13.

Shi

, Liu

and Rosen

, Deep Multimodal Learning for Medical Visual Question Answering, CLEF 2019 Working Notes, CEUR Workshop Proceedings, Switzerland, 2019, Paper 123, 1–8.

14.

Abacha

A.B.

, Gayan

, Liu

, Rajaraman

, Demner-Fushman

and Muller

, NLM at ImageCLEF 2018 Visual Question Answering in the Medical Domain, CLEF 2018 Working Notes, CEURWorkshop Proceedings, Switzerland, 2018, Paper 98, 1–10.

15.

Eslami

, Melo

G.D.

and Meinel

, TeamS at VQAMed 2021: BBN-Orchestra for Long-tailed Medical Visual Question Answering, CLEF 2021 Working Notes, CEUR Workshop Proceedings, Bucharest, Romania, 2021, Paper 98, 1–7.

16.

, Nguyen

B.X.

, Tjiputra

, Tran

Q.D.

and Nguyen

, Multiple Meta-model Quantifying for Medical Visual Question Answering, 2021. arXiv preprint arXiv:2105.08913.

17.

Hasan

S.A.

, Ling

, Farri

, Liu

, Lungren

and Muller

, Overview of the Image CLEF 2018 Medical DomainVisual Question Answering task,CLEF2018 Working Notes, CEUR Workshop Proceedings, Avignon, France, 2018, 1–8.

18.

Abacha

A.B.

, Hasan

S.A.

, Datla

V.V.

, Liu

, Demner-Fushman

and Muller

, VQA-MED: Overview of the Medical Visual Question Answering task at ImageCLEF 2019, CLEF 2019 Working Notes, CEUR Workshop Proceedings, Lugano, Switzerland, 2019, Paper 272, 1–11.

19.

Abacha

A.B.

, Datla

V.V.

, Hasan

S.A.

, Demner-Fushman

and Muller

, Overview of the VQA-MED task at Imageclef 2020:Visual Question Answering and Generation in the Medical Eleventhomain, CLEF 2020Working Notes, CEUR Workshop Proceedings, Thessaloniki, Greece, 2020, Paper 106, 1–9.

20.

Abacha

A.B.

, Sarrouti

, Demner-Fushman

, Hasan

S.A.

and Muller

, Overview of the VQA-Med task at Image-CLEF 2021: Visual Question Answering and Generation in the Medical Domain, CLEF 2021 Working Notes, CEUR Workshop Proceedings, Bucharest, Romania, 2021, Paper 87, 1–8.

21.

Eickho

, Schwall

, GarcaSeco de Herrera

and Muller

, Overview of ImageCLEF Caption 2017 –Image Caption Prediction and Concept Detection for Biomedical Images, CLEF 2017 Labs Working Notes, CEUR Workshop Proceedings, Dublin, Ireland, 2017, 1–10.

22.

Ionescu

, Muller

, Peteri

, Abacha

A.B.

, Datla

, Hasan

S.A.

, Demner-Fushman

, Kozlovski

, Liauchuk

, Cid

Y.D.

, Kovalev

, Pelka

, Friedrich

C.M.

, De Herrera

A.G.S.

, Ninh

V.T.

, Le

T.K.

, Zhou

, Piras

, Riegler

, Halvorsen

, Tran

M.T.

, Lux

, Gurrin

, Dang-Nguyen

D.T.

, Chamberlain

, Clark

, Campello

, Fichou

, Berari

, Brie

, Dogariu

, Stefan

L.D.

and Constantin

M.G.

, Overview of the Image-CLEF 2020: Multimediaretrieval in Medical, Lifelogging, Nature and Internet Applications. in: ExperimentalIR Meets Multilinguality, Multimodality and Interaction, Proceedings of the Eleventh International Conference of the CLEF Association (CLEF 2020), LNCS Lecture Notes in Computer Science, Springer, Thessaloniki, Greece, 2020, 533–542.

23.

Ionescu

, Muller

, Peteri

, Ben Abacha

, Sarrouti

, Demner-Fushman

, Hasan

S.A.

, Kovalev

, Kozlovski

, Liauchuk

, Dicente

, Pelka

, de Herrera

A.G.S.

, Jacutprakart

, Friedrich

C.M.

, Berari

, Tauteanu

, Fichou

, Brie

, Dogariu

, Stefan

L.D.

, Constantin

M.G.

, Chamberlain

, Campello

, Clark

, Oliver

T.A.

, Moustahfid

, Popescu

and Deshayes-Chossart

, Overview of the ImageCLEF 2021: Multimedia retrieval in medical, nature, internet and social media applications, in: Experimental IR Meets Multilinguality, Multimodality and Interaction, Proceedings of the Twelfth International Conference of the CLEF Association (CLEF 2021), LNCS Lecture Notes in Computer Science, Springer, Bucharest, Romania, 2021, 345–360.

24.

Papineni

, Roukos

, Ward

and Zhu

W.J.

, BLEU: A Method for Automatic Evaluation of Machine Translation, ACL-2002: 40th Annual meeting of the Association for Computational Linguistics, Pennsylvania, United States, 2002, 311–318.

25.

Lubna

, Kalady

and Lijiya

, MoBVQA: A Modality based Medical Image Visual Question Answering System, TENCON 2019–2019 IEEE Region 10 Conference, Kochi, India, 2019, 727–732.

26.

Liu

, Zhang

, Zhou

and Yang

, BPI-MVQA: a bi-branch model for medical visual question answering, BMC Medical Imaging 22(1) (2022), 1–19.

27.

Zhang

, Tao

, Li

, Wang

and Zhang

, Type-Aware Medical Visual Question Answering, IEEE International Conference on Acoustics, Speech and Signal Processing, 2022, 4838–4842.

28.

Turner

and Spanier

A.B.

, LSTM in VQA-Med, Is it Really Needed? JCE Study on ImageCLEF 2019 Dataset, CLEF 2019 Working Notes, CEUR Workshop Proceedings, Switzerland, 2019, Paper 116, 1–9.

29.

Al-Sadi

, Talafha

, Al-Ayyoub

, Jararweh

and Costen

, JUST at Image CLEF 2019 Visual Question Answering in the Medical Domain, CLEF 2019 Working Notes, CEUR Workshop Proceedings, Switzerland, 2019, Paper 125, 1–13.

30.

Al-Sadi

, Hana’Al-Theiabat

A.A.M.

and Al-Ayyoub

, The Inception Team at VQA-Med 2020: Pretrained VGG with Data Augmentation for MedicalVQAandVQG,CLEF 2020 Working Notes, CEUR Workshop Proceedings, Thessaloniki, Greece, 2020, Paper 69, 1–12.

31.

Sitara

N.M.S.

and Srinivasan

, SSNMLRGatVQA-MED 2021: An Approach for VQA to Solve Abnormality Related Queries using Improved Datasets, CLEF 2021 Working Notes, CEUR Workshop Proceedings, Bucharest, Romania, 2021, 1329–1335.

32.

Allaouzi , Ahmed

M.B.

and Benamrou

, Deep Neural Networks and Decision Tree Classifier for Visual Question Answering in the Medical Domain, CLEF 2018 Working Notes, CEUR Workshop Proceedings, Avignon, France, 2018, Paper 159, 1–7.

33.

Talafha

and Al-Ayyoub

, JUST at VQA-Med:AVGGSeq2Seq Model,Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum, Avignon, France, 2018, Paper 171, 1–8.

34.

Bounaama

and Abderrahim

M.E.A.

, Tlemcen University at ImageCLEF 2019 Visual Question Answering Task, CLEF 2019 Working Notes, CEUR Workshop Proceedings, Switzerland, 2019, Paper 117, 1–6.

35.

Allaouzi , Ahmed

M.B.

and Benamrou

, An Encoder-Decoder Model for Visual Question Answering in the Medical Domain, CLEF 2019 Working Notes, CEUR Workshop Proceedings, Lugano, Switzerland, 2019, Paper 124, 1–8.

36.

Jung

, Gu

and Harada

, Bumjun jung at VQA-Med 2020: VQA Model based on Feature Extraction and Multimodal Feature Fusion, CLEF 2020 Working Notes, CEUR Workshop Proceedings, Thessaloniki, Greece, 2020, Paper 87, 1–9.

37.

Verma

H.K.

and Ramachandran

, HARENDRAKV at VQA-Med 2020: Sequential VQA with Attention for MedicalVisual Question Answering, CLEF 2020Working Notes, CEURWorkshop Proceedings, Greece, 2020, Paper 62, 1–7.

38.

Liu

, Ding

and Zhou

, Shengyan at vqa-med 2020:An encoder-decoder model for medical domain visual question answering task, CLEF 2020 Working Notes. CEUR Workshop Proceedings, Thessaloniki, Greece, 2020, Paper 78, 1–11.

39.

Wang

, He

, Liu

, Qing

, Chen

, Liu

and Ren

, Medical visual question answering based on question-type reasoning and semantic space constraint, Artificial Intelligence in Medicine 131 (2022), 1–11.

40.

Tascon-Morales

, Márquez-Neila

and Sznitman

, Consistency-Preserving Visual Question Answering in Medical Imaging, International Conference on Medical Image Computing and Computer-Assisted Intervention 2022 (2022), 386–395.

41.

Zhu

, He

, Wang

, Zhang

and Qing

, Medical visual question answering via corresponding feature fusion combined with semantic attention, Mathematical Biosciences and Engineering 19(10) (2022), 10192–10212.

42.

Zhou

, Kang

and Ren

, Employing Inception-Resnet-v2 and Bi-LSTM for Medical Domain Visual Question Answering,Working Notes of CLEF 2018 –Conference and Labs of the EvaluationForum,Avignon, France, 2018, Paper 107, 1–11.

43.

Bansal

, Gadgil

, Shah

and Verma

, Medical Visual Question Answering at Image CLEF 2019-VQA Med, CLEF 2019 Working Notes, CEUR Workshop Proceedings, Switzerland, 2019, Paper 147, 1–11.

44.

Liao

, Wu

, Shen

, Hengel

A.V.D.

and Verjans

, AIMS at VQA-MED 2020: Knowledge Inference via a Skeletonbased Sentence Mapping Approach for medical Domain Visual Question Answering, CLEF 2020 Working Notes,CEURWorkshop Proceedings, Thessaloniki, Greece, 2020, Paper 78, 1–14.

45.

Chen

, Gong

and Li

, HCP-MIC at VQA-MED 2020: Effective Visual Representation for Medical Visual Question Answering, CLEF 2020 Working Notes, CEUR Workshop Proceedings, Greece, 2020, Paper 74, 1–8.

46.

, Zhang

, Mou

, Xing

and Xie

, Pathvqa: 30000+questions for medical visual question answering, 2021, arXiv preprint arXiv:2003.10286.

47.

Lin

, Zhang

, Tac

, Shi

, Haffari

, Wu

and Ge

, Medical Visual Question Answering: A Survey, 2021, arXiv preprint arXiv:2111.10056.

48.

Al-Sadi

, Al-Ayyoub

, Jararweh

and Costen

, Visual question answering in the medical domain based on deep learning approaches: A comprehensive study, Pattern Recognition Letters 150 (2021), 57–64.

49.

https://medpix.nlm.nih.gov/ (MedPix database).