Diagnostic accuracy and interobserver variability of Dynamic Vascular Pattern (DVP) in primary liver malignancies – A simple semiquantitative tool for the analysis of contrast enhancement patterns

Abstract

BACKGROUND: Contrast-enhanced ultrasound (CEUS) is a valuable tool in the diagnostic approach of focal liver lesions, but occasionally subjective and observer-dependent. Semiquantitative evaluation of dynamic CEUS (DCEUS) with standardised software programmes such as Dynamic Vascular Pattern (DVP) could help to improve diagnostic accuracy and objectivity in liver tumour assessment.

OBJECTIVES: The present study aimed at evaluation of diagnostic accuracy of DVP in a clinical setting.

MATERIALS AND METHODS: DVP images of 52 focal liver lesions [30 hepatocellular carcinomas (HCCs), 15 intrahepatic cholangiocellular carcinomas (ICCs), 7 focal nodular hyperplasias (FNHs)] were analysed by four blinded observers with different levels of CEUS-experience. Diagnostic accuracies for the assessment of dignity and entity were evaluated.

RESULTS: Mean sensitivity, specificity, positive and negative predictive value for detection of malignancy with DVP were 48.4% /67.8% /92.7% and 29.3%, respectively. Total diagnostic accuracies for dignity/entity were 63.9% /38.5% (HCC: 58.3% /25.8%; ICC: 73.3% /50%; FNH: 67.9% /67.9%). Interreader-agreement was moderate (κ= 0.42–0.58). Differential diagnosis between ICC and HCC was most challenging.

CONCLUSION: Although developed to improve diagnostic accuracy and objectivity in the assessment of focal liver lesions, DVP alone seems insufficient for differential diagnosis of HCC, ICC and FNH and cannot replace the skills of an experienced observer.

Keywords

Liver malignancy hepatocellular carcinoma intrahepatic cholangiocellular carcinoma contrast-enhancedultrasound (CEUS)dynamic contrast-enhanced ultrasound (DCEUS)Dynamic Vascular Pattern (DVP)interobserver-agreement vascularisation pattern semiquantitative analysis diagnostic accuracy

1 Introduction

Characterisation of focal liver lesions in cirrhotic and non-cirrhotic liver is a common clinical concern. Many studies have shown that contrast-enhanced ultrasound (CEUS) leads to a marked improvement in sensitivity and specificity in the assessment of focal liver lesions [1 –9]. For focal nodular hyperplasia (FNH), findings on conventional B-mode are often not diagnostic; thus, CEUS is helpful for clarification [2 –8]. The characteristic enhancement pattern of FNHs is thought to be an initial radial centrifugal arterial hyperenhancement (“wheel spoke pattern”), followed by rapid complete enhancement of the lesion and sustained homogeneous hyper- or isoenhancement of the lesion during the portal venous and late phase [3].

HCCs can be diagnosed non-invasively due to characteristic features upon contrast-enhanced imaging in high-risk patients. The typical enhancement pattern of HCCs is defined as arterial hyperenhancement, followed by washout in the portal venous or late phase. However, well-differentiated HCCs often lack arterial hyperenhancement, appearing iso- or even hypoenhanced in the arterial phase [17–18]. As for intrahepatic cholangiocellular carcinomas (ICCs), their typical contrast enhancement pattern is thought to be a strong, peripheral rim-like arterial hyperenhancement, followed by a rapid and pronounced washout of the contrast agent in the early portal venous phase [16]. Difficulties in the differential diagnosis of ICC versus HCC present a long-disputed and controversial issue and are often quoted as a reason for the exclusion of CEUS as a primary imaging modality equivalent to contrast-enhanced (CE-) CT and CE-MRI from HCC- guidelines of the AASLD and EASL [1 , 19–21].

In conclusion, although improving diagnostic sensitivity and specificity as compared to conventional B-mode ultrasound, CEUS is not devoid of diagnostic difficulties and pitfalls. One major issue is subjectivity in the assessment of enhancement patterns [22]. Studies have shown that the assessment of the contrast enhancement pattern of a given lesion is essentially subjective and depends on a reader’s experience; interreader-agreement has been shown to improve with a reader’s experience [23].

Therefore, attempts have been made to improve objectivity as well as interreader-agreement in the assessment of contrast enhancement behaviour. An advancement of CEUS offering additional semi-quantitative assessment of contrast enhancement is dynamic CEUS (DCEUS), providing quantification software for standardised analysis of enhancement kinetics. Several recent studies have shown good diagnostic accuracy, sensitivity and specificity of DCEUS in the assessment of focal liver lesions [9–13 , 24–30].

A specific tool in the context of DCEUS in focal liver lesion is a programme called “dynamic vascular pattern” (DVP). With DVP, quantitative differences in contrast enhancement between given regions of interest (e.g. a focal liver lesion and the surrounding healthy parenchyma, or different spots within a focal liver lesion) are visualised by means of coloured maps. Thus, hyperenhancement versus a reference region is depicted as “warm” colours (red, orange, yellow), whereas hypoenhancement is visualised through “cold” colours (blue, green). Thus, assessment of hyper- or hypoenhancement of a given lesion is facilitated, potentially enabling less experienced observers to make a more objective assessment of contrast enhancement behaviour. For interpretation of DCEUS, there are several quantification parameters reflecting arterial contrast enhancement (time-to-peak = TTP, rise time = RT), absolute contrast enhancement (maximum intensity), and venous contrast washout (mean Transit Time = mTT, Fall Time = FT). With DVP, the user is provided with time intensity curves of a given region of interest (ROI). Also, quantification parameters can be displayed. Several studies have described a high maximum intensity and short time-to-peak as characteristic of HCC; also, slower washout was related with a higher grade of tumour differentiation [32]. In a study evaluating DCEUS in HCCs, Schacherer et al. [32] conclude that high peak values of contrast enhancement reflect a well-established microvascular network. For intrahepatic cholangiocellular carcinoma (ICC), rapid and pronounced washout are considered characteristic [15]. Whereas parameters reflecting arterial contrast enhancement behaviour such as rise time and time-to-peak have been shown to be similar in HCC and ICC, significant differences can be observed concerning parameters of contrast agent washout such as mTTI and FT [21]. However, as interpretation of time-intensity-curves and single quantification parameters can be difficult and time-consuming, DVP also summarises and illustrates this information by means of coloured maps. However, although a useful tool in the objectification of enhancement behaviour in focal liver lesions, only few studies have been conducted on DVP[24 , 32].

Aim of the present study was to explore DVP as a method of more objective assessment of focal liver lesions by observers with different levels of experience. Therefore, four observers retrospectively evaluated DVP maps of 52 focal liver lesions (30 HCCs, 15 ICCs, 7 FNHs) for their dignity and final diagnosis. Histological findings and findings from contrast-enhanced imaging (CT, MRI) served as reference standards. Sensitivity, specificity, positive and negative predictive values as well as diagnostic accuracies were assessed for every reader, and the interreader-agreement was determined.

2 Materials and methods

2.1 Patients

52 patients with focal liver lesions visible on conventional ultrasound who presented to our ultrasound department between 2012 and 2016 were selected for CEUS examination. Inclusion criteria were a focal liver lesion (HCC, ICC or FNH) visible on conventional ultrasound, and either histology or another contrast-enhanced imaging modality apart from CEUS (CE-CT or CE-MRI) available as a reference standard [4 , 32]. These lesion entities were chosen because as shown by the DEGUM-multi-centre study, they present a diagnostic challenge. FNHs were included as these hypervascular lesions might mimic HCC [3]. The study was approved by the local ethics committee. All patients provided written informed consent for acquisition and pseudo-anonymised evaluation of data.

2.2 Dynamic Contrast Enhanced Ultrasound (DCEUS)

For all patients, conventional ultrasound was performed prior to CEUS. Examinations were performed in a supine or left lateral position using a Siemens Acuson Sequoia 512 ultrasound system (Siemens Medical Solutions, Erlangen, Germany, 4C1 probe). Lesions were first identified on conventional B-mode. In case of more than one lesion in a single patient, only the lesion best accessible for ultrasound examination was chosen for further analysis. Afterwards, CEUS examination was performed according to EFSUMB guidelines [1] following a standardised protocol using a low mechanical index and the specific cadence contrast pulse sequence (CPS) preset. An intravenous injection of 1.5 mL SonoVue^® via an antecubital vein was followed by a 10 mL saline flush. The linear raw data of the examinations were continuously recorded over a time period of 3–5 minutes starting before the first arrival of detectable microbubbles until the beginning of clearance of the microbubbles from parenchymal tissue in the late phase. In order to avoid motion artefacts, patients were asked to respect a relaxed breathing during video clip acquisition. To record valid clips for quantification purpose, continuous scanning had to be carried out during the time of examination.

The vascular phases were defined according to EFSUMB guidelines and recommendations for DCEUS acquisition [1, 31].

2.3 Dynamic Vascular Pattern (DVP)

Quantification was performed using dedicated quantification software (VueBox^®, Bracco Suisse Software Applications, Genève, Suisse). The software programme contains a standardised module called “Liver DVP” for quantitative analysis of focal liver lesions using Dynamic Vascular Pattern. Tools are included for calibration of video clip sequences and correction of in-plane-motion. In a first step, both a “lesion” and “reference parenchyma” are defined manually by drawing an elliptically shaped “region of interest” (ROI). For every lesion, two additional ROIs were drawn within the lesion designating a central and peripheral ROI, respectively, thus enabling the observer to analyse the lesion centre and periphery separately. With the “Liver DVP” software, video clip data is then linearized in an automated process. Both on the level of individual pixels and of ROIs, a parametric perfusion model is used for fitting a curve of echo signal intensities over time, with echo signal intensities being directly proportional to the actual concentration of contrast agent at a given time.

With the DVP programme, a DVP signal is created by subtracting a reference signal from a pixel signal at any given point of time. Pixels with hyperenhancement versus the reference parenchyma at a given time are indicated with warm colours (red, orange, yellow), whereas those with hypoenhancement versus the reference parenchyma are displayed in cold colours (blue, green). Thus, a colour map is created illustrating the relative contrast enhancement of a lesion (ROI) versus the reference parenchyma (Fig. 1).

Fig.1

Principle of DVP. a) Perfusion kinetics in healthy parenchyma (black line) as compared to a lesion with slow contrast uptake and persisting hyperenhancement (green line, lesion 1; typical “benign” lesion) versus a lesion with arterial hyperenhancement and early washout (red line, lesion 2; typical “malignant” lesion). b) DVP processing: signal intensities as differences calculated by subtracting the signal intensity of the lesion from that of the reference parenchyma. The resulting amplitude is coded in warm colours for hyperenhancement or cold colours for hypoenhancement. Copyright© 2015 VueBox user manual, Bracco Suisse SA ALL RIGHTS RESERVED. Printed with permission from Bracco Suisse.

2.4 Lesion assessment

DVP images of all 52 focal liver lesions (30 HCCs, 15 ICCs, 7 FNHs) were created and then independently analysed by four observers with theoretical knowledge of contrast enhancement patterns in focal liver lesions, but different levels of clinical experience in CEUS (observer 1: no practical experience; observer 2 : 5 years of experience; observer 3 : 7 years of experience; observer 4: more than 10 years of experience). Observers were blinded to all additional clinical and imaging data apart from prepared DVP images. Before assessment of lesions, all observers received a brief theoretical training. They were given images of “ideal cases” for each lesion entity for reference (Fig. 2). Observers were instructed to assess for every case first the lesion dignity (benign, malignant or unsure) and then the lesion entity (HCC, ICC, FNH or unsure). Although they were informed which entities were included into the study, observers did not know the number of lesions included for each entity.

Fig.2

Typical DVP images of focal liver lesions.

Results were evaluated separately for every observer. Afterwards, interobserver-agreement was analysed.

2.5 Statistical analysis

Statistical Analysis was performed using Microsoft Excel 2010 and IBM SPSS Statistics 2013. Values are expressed as mean and range, or absolute sum and proportion in percent as appropriate. Student’s t-test for unpaired samples was used for comparison of means. Cohen κ statistics were used for evaluation of interreader-agreement. Differences were considered statistically significant for p < 0.05.

3 Results

3.1 Patient characteristics and focal liver lesions

Characteristics of patients and lesions are summarised in Table 1. In total, 52 focal liver lesions were analysed (30 HCCs, 15 ICCs and 7 FNHs); thus, the majority of lesions was malignant (45/52 = 85.5%). All FNH patients were female. In the HCC and ICC groups, there was a marked male predominance (90% vs. 66.6% males). None of the FNH patients had liver cirrhosis as opposed to 83.3% in the HCC group and 26.7% in the ICC group.

Table 1
Patient and lesion characteristics

HCC ICC FNH Total

Total (N /%) 30 (57.7%) 15 (28.8%) 7 (13.5%) 52 (100%)

Size [mm] (Mean; Range) 48.3 (15–200) 51.3 (10–130) 53.4 (39–76) 51 (10–200)

Patient Age [years] (Mean; Range) 68.1 (48–86) 64.9 (48–77) 34.4 (22–67) 55.8 (22–86)

Patient Sex 27/3 10/5 0/7 37/15

male/female (n/%) 90% /10% 66.6% /33.3% 0% /100% 71.2% /28.8%

Cirrhosis (n/%) 25 (83.3%) 4 (26.7%) 0 29 (55.8%)

	HCC	ICC	FNH	Total
Total (N /%)	30 (57.7%)	15 (28.8%)	7 (13.5%)	52 (100%)
Size [mm] (Mean; Range)	48.3 (15–200)	51.3 (10–130)	53.4 (39–76)	51 (10–200)
Patient Age [years] (Mean; Range)	68.1 (48–86)	64.9 (48–77)	34.4 (22–67)	55.8 (22–86)
Patient Sex	27/3	10/5	0/7	37/15
male/female (n/%)	90% /10%	66.6% /33.3%	0% /100%	71.2% /28.8%
Cirrhosis (n/%)	25 (83.3%)	4 (26.7%)	0	29 (55.8%)

Patients in the FNH group were significantly younger than those in the two other groups (p < 0.01). Lesion size did not differ significantly between tumour entities (p = 0.45).

3.2 Differentiation between benign and malignant lesions

Results from dignity assessment for all four observers and entities are summarised in Tables 2 and Tables 3.

Table 2
Results from dignity assessment of single observers

Total HCC ICC FNH

Observer 1

Correct diagnosis of dignity 32 = 61.5% 17 = 56.7% 11 = 73.3% 4 = 57.1%

Wrong diagnosis of dignity 11 = 21.1% 7 = 23.3% 1 = 6.7% 3 = 42.9%

Dignity uncertain 9 = 17.3% 6 = 20% 3 = 20% 0

Observer 2

Correct diagnosis of dignity 28 = 53.8% 14 = 46.7% 8 = 53.3% 6 = 85.7%

Wrong diagnosis of dignity 15 = 28.8% 11 = 36.7% 3 = 20% 1 = 14.3%

Dignity uncertain 9 = 17.3% 5 = 16.7% 4 = 26.6% 0

Observer 3

Correct diagnosis of dignity 38 = 73.1% 21 = 70% 13 = 86.7% 4 = 57.1%

Wrong diagnosis of dignity 10 = 19.2% 6 = 20% 2 = 13.3% 2 = 28.6%

Dignity uncertain 4 = 7.7% 3 = 10% 0 = 0% 1 = 14.3%

Observer 4

Correct diagnosis of dignity 35 = 67.3% 18 = 60% 12 = 80% 5 = 71.4%

Wrong diagnosis of dignity 10 = 19.2% 6 = 20% 2 = 13.3% 2 = 28.6%

Dignity uncertain 7 = 13.5% 6 = 20% 1 = 6.7% 0 = 0%

Total (Mean)

Correct diagnosis of dignity 133 = 63.9% 70 = 58.3% 44 = 73.3% 19 = 67.9%

Wrong diagnosis of dignity 46 = 22.1% 30 = 25% 8 = 13.3% 8 = 28.6%

Dignity uncertain 29 = 13.9% 20 = 16.7% 8 = 13.3% 1 = 3.6%

	Total	HCC	ICC	FNH
Observer 1
Correct diagnosis of dignity	32 = 61.5%	17 = 56.7%	11 = 73.3%	4 = 57.1%
Wrong diagnosis of dignity	11 = 21.1%	7 = 23.3%	1 = 6.7%	3 = 42.9%
Dignity uncertain	9 = 17.3%	6 = 20%	3 = 20%	0
Observer 2
Correct diagnosis of dignity	28 = 53.8%	14 = 46.7%	8 = 53.3%	6 = 85.7%
Wrong diagnosis of dignity	15 = 28.8%	11 = 36.7%	3 = 20%	1 = 14.3%
Dignity uncertain	9 = 17.3%	5 = 16.7%	4 = 26.6%	0
Observer 3
Correct diagnosis of dignity	38 = 73.1%	21 = 70%	13 = 86.7%	4 = 57.1%
Wrong diagnosis of dignity	10 = 19.2%	6 = 20%	2 = 13.3%	2 = 28.6%
Dignity uncertain	4 = 7.7%	3 = 10%	0 = 0%	1 = 14.3%
Observer 4
Correct diagnosis of dignity	35 = 67.3%	18 = 60%	12 = 80%	5 = 71.4%
Wrong diagnosis of dignity	10 = 19.2%	6 = 20%	2 = 13.3%	2 = 28.6%
Dignity uncertain	7 = 13.5%	6 = 20%	1 = 6.7%	0 = 0%
Total (Mean)
Correct diagnosis of dignity	133 = 63.9%	70 = 58.3%	44 = 73.3%	19 = 67.9%
Wrong diagnosis of dignity	46 = 22.1%	30 = 25%	8 = 13.3%	8 = 28.6%
Dignity uncertain	29 = 13.9%	20 = 16.7%	8 = 13.3%	1 = 3.6%

Table 3

Quality of dignity assessment and diagnostic accuracies

	Sensitivity	Specificity	PPV for	NPV for	Diagnostic	Diagnostic
	for malignancy	for malignancy	malignancy	malignancy	Accuracy (dignity)	Accuracy (entity)
Observer 1	28/45 = 62.2%	4/7 = 57.1%	28/30 = 93.3%	4/12 = 33.3%	61.5%	38.5%
Observer 2	22/45 = 48.9%	6/7 = 85.7%	22/24 = 91.7%	4/19 = 21.1%	53.8%	34.6%
Observer 3	34/45 = 75.6%	5/7 = 71.4%	34/37 = 91.9%	4/12 = 33.3%	73.1%	42.3%
Observer 4	30/45 = 66.7%	4/7 = 57.1%	30/32 = 93.8%	5/17 = 29.4%	67.3%	38.5%
Total (Mean)	48.4%	67.8%	92.7%	29.3%	63.9%	38.5%

PPV, positive predictive value; NPV, negative predictive value.

Table 2 summarises mean values from assessments of all four observers. In total, correct differentiation of benign versus malignant lesions was achieved in 63.9% of cases. Correct dignity assessment differed between lesion entities, with best results for ICC (73.3%) followed by FNH (67.9%). Correct dignity assessment was achieved in 58.3% of HCCs. Uncertainty in dignity assessment was 13.9% on average; it was lowest for the FNH group with 3.6%, but, however, the FNH group also showed the highest rate of misdiagnosis of dignity (28.6% versus 13.3% for ICC and 25% for HCC).

There were marked differences between observers with 61.5% /53.8% /73.1% /67.3% of correct dignity classifications. The more experienced observers (observer 3 and 4) showed the lowest rate of diagnostic uncertainty (7.7% and 13.5% versus 17.3% /17.3%) and also the lowest rate of misclassifications (19.2% /19.2% versus 21.1% /28.8%). Also, they had the highest proportion of correct dignity classifications (73.1% /67.3% versus 61.5% /53.8%).

In total, mean sensitivity for the detection of malignancy was 48.4% with statistically significant differences between observers, who achieved sensitivities of 62.2% /48.9% /75.6% and 66.7%, respectively (Table 3). Specificity for the detection of malignancy was 67.8% (57.1% /85.7% /71.4% and 57.1%). Thus, the highest specificity but lowest sensitivity and diagnostic accuracy (for both dignity and entity) were seen for observer 2. Positive predictive value (PPV) was 92.7% with high values for all four observers (93.3% /91.7% /91.9% /93.8%), with a negative predictive value of 29.3% (33.3% /21.1% /33.3% /29.4%).

3.3 Differentiation between lesion entities

Results for the differentiation between lesion entities for all observers as well as mean diagnostic accuracies are shown in Tables 3 and Tables 4.

Table 4
Results from entity assessment of single observers

Total HCC ICC FNH

Observer 1

Correct diagnosis of entity 20 = 38.5% 8 = 26.7% 8 = 53.3% 4 = 57.1%

Wrong diagnosis of entitiy 21 = 40.4% 16 = 53.3% 2 = 13.3% 3 = 42.9%

Entity uncertain 11 = 21.1% 6 = 20% 5 = 33.3% 0

Observer 2

Correct diagnosis of entity 18 = 34.6% 6 = 20% 6 = 40% 6 = 85.7%

Wrong diagnosis of entitiy 22 = 42.3% 18 = 60% 3 = 20% 1 = 14.3%

Entity uncertain 12 = 23.1% 6 = 20% 6 = 40% 0

Observer 3

Correct diagnosis of entity 22 = 42.3% 8 = 26.7% 10 = 66.7% 4 = 57.1%

Wrong diagnosis of entitiy 22 = 42.3% 17 = 56.7% 3 = 20% 2 = 28.6%

Entity uncertain 8 = 15.4% 5 = 16.7% 2 = 13.3% 1 = 14.3%

Observer 4

Correct diagnosis of entity 20 = 38.5% 9 = 30% 6 = 40% 5 = 71.4%

Wrong diagnosis of entitiy 25 = 48.1% 15 = 50% 8 = 53.3% 2 = 26.6%

Entity uncertain 7 = 13.5% 6 = 20% 1 = 6.7% 0

Total (Mean)

Correct diagnosis of entity 80 = 38.5% 31 = 25.8% 30 = 50% 19 = 67.9%

Wrong diagnosis of entitiy 90 = 43.3% 66 = 55% 16 = 26.7% 8 = 28.6%

Entity uncertain 38 = 17.3% 23 = 19.2% 14 = 23.3% 1 = 3.6%

	Total	HCC	ICC	FNH
Observer 1
Correct diagnosis of entity	20 = 38.5%	8 = 26.7%	8 = 53.3%	4 = 57.1%
Wrong diagnosis of entitiy	21 = 40.4%	16 = 53.3%	2 = 13.3%	3 = 42.9%
Entity uncertain	11 = 21.1%	6 = 20%	5 = 33.3%	0
Observer 2
Correct diagnosis of entity	18 = 34.6%	6 = 20%	6 = 40%	6 = 85.7%
Wrong diagnosis of entitiy	22 = 42.3%	18 = 60%	3 = 20%	1 = 14.3%
Entity uncertain	12 = 23.1%	6 = 20%	6 = 40%	0
Observer 3
Correct diagnosis of entity	22 = 42.3%	8 = 26.7%	10 = 66.7%	4 = 57.1%
Wrong diagnosis of entitiy	22 = 42.3%	17 = 56.7%	3 = 20%	2 = 28.6%
Entity uncertain	8 = 15.4%	5 = 16.7%	2 = 13.3%	1 = 14.3%
Observer 4
Correct diagnosis of entity	20 = 38.5%	9 = 30%	6 = 40%	5 = 71.4%
Wrong diagnosis of entitiy	25 = 48.1%	15 = 50%	8 = 53.3%	2 = 26.6%
Entity uncertain	7 = 13.5%	6 = 20%	1 = 6.7%	0
Total (Mean)
Correct diagnosis of entity	80 = 38.5%	31 = 25.8%	30 = 50%	19 = 67.9%
Wrong diagnosis of entitiy	90 = 43.3%	66 = 55%	16 = 26.7%	8 = 28.6%
Entity uncertain	38 = 17.3%	23 = 19.2%	14 = 23.3%	1 = 3.6%

In total, mean diagnostic accuracy for the correct classification of lesion entities was 38.5% with only minor differences between observers (38.5% /34.6% /42.3% /38.5%) (Table 3). As shown in Table 4, mean diagnostic accuracy differed between entities and was highest for FNHs with 67.9% versus 50% for ICCs and 25.8% for HCCs. Also, HCCs were the entity with the highest rate of misdiagnoses (55% versus 26.7% for ICCs and 28.6% for FNHs). Diagnostic uncertainty as to the differentiation of entities was highest for ICCs (23.3%).

3.4 Interobserver agreement

As shown in Table 5, there were a total of 26 cases (50%) with agreement of all four observers upon dignity of a lesion. In 22/52 lesions (42.3%), all observers correctly identified the dignity of the lesion (12/30 HCCs, 6/15 ICCs, 4/7 FNHs). In 3/52 lesions (5.8%; all HCCs), all four observers misclassified the lesion’s dignity.

Table 5
Interobserver agreement: Agreement/disagreement of all four observers

Total (n/%) HCC (n/%) ICC (n/%) FNH (n/%)

Differentiation between benign and malignant lesions – agreement of all four observers

Correct assessment of dignity 22 12 6 4

Misdiagnosis of dignity 3 3 0 0

Dignity uncertain 1 1 0 0

Total 26 17 6 3

Differentiation between lesion entities – agreement of all four observers

Correct assessment of entity 11 4 3 4

Misdiagnosis of entity 8 7 1 0

Entitiy uncertain 1 1 0 0

Total 20 12 4 4

No agreement between observers^*

Dignity 12 10 2 0

Entity 18 9 9 0

	Total (n/%)	HCC (n/%)	ICC (n/%)	FNH (n/%)
Differentiation between benign and malignant lesions – agreement of all four observers
Correct assessment of dignity	22	12	6	4
Misdiagnosis of dignity	3	3	0	0
Dignity uncertain	1	1	0	0
Total	26	17	6	3
Differentiation between lesion entities – agreement of all four observers
Correct assessment of entity	11	4	3	4
Misdiagnosis of entity	8	7	1	0
Entitiy uncertain	1	1	0	0
Total	20	12	4	4
No agreement between observers^*
Dignity	12	10	2	0
Entity	18	9	9	0

^*no agreement: defined as no more than two observers agreeing upon any option.

As FNHs were the only benign lesion included into the study, results for dignity and entity assessment are the same in this group. For the malignant lesions, correct classification of entity was seen less often than correct classification of dignity. Agreement between all four observers as to the differentiation of a lesion’s entity was seen in 11 correctly classified lesions (4 HCCs, 3 ICCs, 4 FNHs), and in 8 misclassified lesions (7 HCCs, 1 ICC).

For 12 lesions (10 HCCs, 2 ICCs), no agreement was seen between observers concerning dignity, where “no agreement” was defined as no potential options voted for by more than two observers. No agreement concerning entity was seen for 18 lesions (9 HCCs, 9 ICCs).

3.5 Influence of cirrhosis

None of the FNH patients had liver cirrhosis. Out of the ICC patients, 4/15 patients had cirrhosis (26.7%), whereas in the HCC group, there were only 5/30 patients (16.7%) without cirrhosis. It became apparent that for ICCs in cirrhotic liver, assessment of dignity proved slightly more difficult with only 68.8% of correct dignity assessments (versus 73.3% in the whole ICC group); diagnostic uncertainty was slightly higher with 18.8% versus 13.3%. Only 25% of entity classifications for ICC were correct in cirrhotic liver versus 50% in the whole group. Interestingly, for HCC, the “atypical” subgroup of HCCs in non-cirrhotic liver showed slightly superior results with 65% of correct dignity classifications (versus 58.3% in the whole group). Strikingly, none of the HCCs in non-cirrhotic liver was attributed the correct entity, but there were 70% of misclassifications (as opposed to 50% in the whole group).

Interobserver-agreement for differentiation between malignant and benign lesions was moderate with Cohen’s κ= 0.468. Interobserver-agreement for diagnosis of lesion entity was moderate with Cohen’s κ= 0.42. Interreader-agreement was moderate for any pair of readers with, however, best agreement between observer 3 and 4 (κ= 0.576).

4 Discussion

Focal liver lesions are a common finding in both cirrhotic and non-cirrhotic liver [3, 8]. In clinical routine, it is vital to differentiate between benign findings, which mostly require no further intervention unless in rare cases of symptomatic patients, and malignant lesions necessitating further diagnostic workup and immediate therapy. Due to its excellent sensitivity and specificity in the detection and differentiation of focal liver lesions as well as additional advantages such as ready availability and almost complete lack of contraindications and adverse effects, contrast-enhanced ultrasound (CEUS) has been widely accepted as imaging modality for further diagnostic workup of focal liver lesions detected upon conventional B-mode ultrasound [1–5 , 34–43]. Providing a real-time visualisation of contrast enhancement kinetics, CEUS allows for a very sensitive detection of arterial phase hypervascularisation and possibility to detect tumorous as well as infectious lesions at very early stages[47, 48].

More recently, (semi-)quantitative analysis using dynamic CEUS (DCEUS) has attracted scientific attention as a method of more objective and possibly more sensitive assessment of perfusion kinetics in focal liver lesions, allowing for a more accurate assessment of dignity and in some cases also further classification of lesion entity [9–12 , 50]. However, a certain subjectivity inherent in the method as well as great dependency from the reader’s experience led to the development of quantification software programmes aiming at a more standardised and objective evaluation [22–25 , 44]. One such method for facilitating interpretation of contrast enhancement patterns as well as regional differences in perfusion kinetics in focal liver lesions is the “Dynamic Vascular Pattern” (DVP) module presented in this study. Typical examples of DVP and MRI images of benign and malignant focal liver lesions (FNH, HCC and ICC) are shown in Fig. 3 and Fig. 4. Although DVP has been implemented into customised DCEUS quantification programmes for several years now, little research has been conducted to evaluate its performance. Thus, the present work aimed at assessment of the diagnostic accuracy of DVP in the assessment of dignity and classification of entity of benign and malignant focal liver lesions (FNH, HCC, ICC) in cirrhotic and non-cirrhotic liver. Also, results were compared for observers with different levels of clinical experience with CEUS (no practical experience / 5 years / 7 years / >10years).

Fig.3

Typical examples for DVP images of FNH, HCC and ICC.

Fig.4

Typical examples of benign and malignant liver lesions on MRI. MRI images of three different liver lesions:T2-weighted and dynamic T1 weighted dynamic images (pre-contrast, arterial and portal venous phase, from left to right). 4a, 23-year-old woman with a focal nodular hyperplasia (FNH). The lesion is best visible in the arterial phase (white arrow). In T2 w, it is slightly hypointense, and in the pre-contrast and portal venous phase, it is nearly isointense to the normal liver parenchyma. 4b, 45-year-old man with cirrhosis, recanalization of the umbilical vein (open arrow) and a hepatocellular carcinoma (HCC) in segment 6/7. The lesion is isointense in T2w and hyperintense in the arterial phase with washout in the portal venous phase (white arrows). The rim enhancement in the portal venous phase (arrowhead) represents a capsule, which is considered relatively specific for HCC. 4c, 62-year-old woman with a large intrahepatic cholangiocellular carcinoma (ICC) (white arrows) and a suspicious adrenal lesion (white arrowhead). The ICC shows the typical homogeneous low attenuation on the non-contrast scan, and demonstrates heterogeneous contrast enhancement.

Concerning the differentiation between benign and malignant findings, our results showed a low sensitivity for the detection of malignant findings with DVP of 48.4%, with a moderate specificity of 67.8% and a very high positive predictive value (PPV) of 92.7%. Total diagnostic accuracy for the assessment of a lesion’s dignity amounted to 63.9%. However, there were differences between lesion entities with the highest percentage of correct dignity assessment in the ICC group (73.3%), followed by the FNH group with 67.9% and the HCC group with 58.3%.

These results differ from a recent study by Anaye et al. [24] comparing diagnostic accuracy for dignity assessment in 146 focal liver lesions between direct visual assessment of CEUS clips versus interpretation of parametric imaging. The authors found superior sensitivities of 96.5–97.3% and specificities of 90.9% for all three readers with interpretation of parametric imaging as opposed to visual assessment of CEUS video clips. Thus, sensitivities and specificities were markedly higher than in our study. However, the technique used for parametric imaging in the study by Anaye et al. slightly differed from the DVP as used in our study with coloured representation of pixel polarisation and also different interpretation of “malignant appearance”. Also, although there was a time interval of at least 2 months between interpretation of original video clips and parametric images, both were evaluated by the same readers, which might have introduced a certain bias. Moreover, in the study by Anaye, all readers had at least 2 years of experience with CEUS, which was not the case in our study. Furthermore, the benign and malignant entities of focal liver lesions included into the study differed from our work. In the study by Anaye et al., 65/113 (57.5%) malignant lesions were metastases and only 36/113 (31.9%) were HCCs and 10/113 (8.8%) were ICCs, whereas in our own study, 30/52 lesions (57.7%) were HCCs and 15/52 (28.8%) were ICCs. As shown by the analysis of tumour-specific vascularisation patterns from the DEGUM-multi-centre study [3], diagnostic accuracy of CEUS is far better for metastases than for HCC (correct diagnosis with CEUS in 350/383 metastases = 91.4% versus 247/279 HCCs = 84.9%). Also, in a very recent study by Yue et al. [26], parametric imaging with CEUS was shown useful for the differentiation between HCC and liver metastases. Thus, the superior sensitivities and specificities reported by Anaye et al. are probably at least partly due to the choice of lesions. Anaye et al. found moderate interreader-agreement (κ= 0.54) with CEUS clip interpretation and almost perfect interreader-agreement (κ= 0.99) for interpretation of parametric images. Sensitivity and specificity of dignity assessment between individual readers showed marked differences for interpretation of CEUS clips (sensitivities 85% /77.9% /87.6%; specificities 51.5% /78.8% /75.8%) reflecting different experience levels. Accordingly, in our study, marked differences could be seen between results from individual readers.

Unfortunately, there seems to be no work in the literature so far assessing the diagnostic value of DVP in the differentiating of entities of focal liver lesions, so no direct comparison of our results is possible. Nonetheless, we found an insufficient diagnostic accuracy of 38.5% for the correct classification of lesion entities. FNHs seemed to pose the least diagnostic difficulties with 67.9% of correct diagnoses. On the contrary, entity was misclassified in 55% of HCCs and was voted as uncertain in 23.3% of ICCs.

In a study by Vilana et al. [19] evaluating CEUS for ICCs in cirrhotic patients, 10 out of 21 ICCs showed homogeneous arterial hyperenhancement followed by washout, and 7 out of 10 ICCs >20 mm in size were misclassified as HCCs on the basis of CEUS findings. Similarly, Li et al. [15] assessed the influence of cirrhosis and chronic hepatitis B infection versus healthy surrounding liver parenchyma on enhancement patterns of ICCs in CEUS. The authors concluded that in cirrhotic patients or those with chronic hepatitis B, contrast enhancement patterns of ICCs are closely similar to those considered typical of HCC, making differential diagnosis almost impossible in these patients. However, these studies used subjective qualitative evaluation of contrast enhancement patterns and no quantitative assessment. Yet, the presence of cirrhosis or other comorbidities affecting the liver might have an influence on DVP images, too. Although cautious interpretation is required due to the small cohort size, our results seem to indicate greater diagnostic difficulties in “atypical” settings, i.e., both for ICCs in cirrhotic liver and for HCCs in non-cirrhotic liver. These findings might be due to perfusion differences between cirrhotic and non-cirrhotic liver. Accordingly, in a very recent study by Haimerl et al. 11/2016 [49] comparing the diagnostic performance of DCEUS versus MRI-based T1 elastometry, the authors found some quantification parameters of contrast agent wash-in to be reduced in individuals with impaired liver function as compared to healthy individuals. However, the authors concluded that CEUS-based perfusion parameters did not allow for assessing the severity of liver diseases. This emphasises the fact that results for DCEUS examinations might vary between cirrhotic and non-cirrhotic liver.

Moreover, it would be interesting to assess possible correlations of grade of histological differentiation and enhancement patterns in DVP. Several studies have shown that especially well-differentiated HCCs can show very late washout or no washout at all [18]. A study by Schacherer at al. [32] analysing DCEUS in HCCs found a higher peak enhancement in tumour centres as compared to peripheral regions and concluded that this finding might be related to a higher density of microvessels in the central region of tumours. However, histological findings were not available to support this hypothesis.

Another very recent study by Giorgio et al. [41] evaluated the diagnostic value of CEUS in 229 focal liver lesions ≤20 mm in cirrhotic patients. The authors found an enhancement pattern of hyperenhancement followed by isoenhancement in 105/199 HCCs; 85/199 HCCs showed arterial hyperenhancement followed by late washout after more than 3 minutes. Thus, only 4.5% of HCCs lacked arterial hyperenhancement upon CEUS examinations [41]. However, in discordance with these findings, a study by Forner et al. reported a lack of arterial hyperenhancement in 15.1% of 119 HCCs <20 mm in cirrhotic liver [46]. Thus, it becomes apparent that HCCs represent a very heterogeneous tumour entity with sometimes atypical enhancement behaviour upon CEUS –resulting, consequently, in unexpected DVP images.

For CEUS, interobserver-agreement has been shown to be modest with κ-values between 0.47–0.63 and better agreement in more experienced observers [23]. Our results for interpretation of DVP images show similar results with moderate interreader-agreement and κ-values between 0.42–0.58, thus comparable to interpretation of conventional CEUS video clips. The only study evaluating interreader-agreement for parametric imaging in FLLs found good interobserver-agreement (κ= 0.99) with individual experience being far less influencing than with visual interpretation of CEUS clips [24]. However, in our study, results differed markedly for individual readers. Yet, diagnostic accuracies found for DVP were insufficient for all observers in our study. Thus, it might be concluded that although developed as a simple tool to facilitate dignity assessment of focal liver lesions especially for less experienced readers, DVP cannot replace an observer’s expertise. Our findings suggest that diagnostic accuracy of DVP is not superior to accuracies reported for visual interpretation of conventional CEUS clips. Also, DVP does not seem appropriate for differentiation between lesion entities.

Our study has some limitations. First of all, number of lesions was limited and different for lesion entities. Also, readers were given DVP images only without further information about patients’ clinical findings and without access to original CEUS clips. This procedure was chosen to evaluate the pure diagnostic value of DVP, avoiding any bias resulting from information derived from original CEUS clips. Furthermore, the method itself has some limitations. For DCEUS analysis, continuous scanning is mandatory to record clips over a time period of at least 3 minutes. Thus, out-of-plane-motions are a common problem. Also, ROIs have to be placed manually, resulting in some form of bias. As in our study, all ROIs were determined by the same person, this bias might be thought to be minimised. Furthermore, patient factors such as obesity, ascites, meteorism, inability to hold one’s breath and also location of the liver lesion influence the quality of DCEUS clips and thus further analysis of parametric imaging.

5 Conclusion

Dynamic Vascular Pattern (DVP) represents a simple module for quantitative analysis of DCEUS clips, which was developed to improve diagnostic accuracy and reduce subjectivity in the assessment of contrast enhancement patterns. However, our findings suggest that diagnostic accuracy of DVP is insufficient in the differential diagnosis of hepatocellular carcinoma. The differentiation between lesion entities, especially between ICCs and HCCs, remains challenging. Although intended to facilitate differential diagnosis of focal liver lesions especially for observers with little experience in CEUS, DVP alone seems unable to replace a training curve. However, we chose particularly difficult tumour entities. Further studies are needed to evaluate DVP in larger patient cohorts for the differential diagnosis of benign versus metastatic lesions in non-cirrhotic liver.

Conflicts of interest and source of funding

The authors have no conflict to report.

References

Claudon

, Dietrich

C.F.

, Choi

B.I.

, et al., Guidelines and good clinical practice recommendations for contrast enhanced ultrasound (CEUS) in the liver–update 2012: A WFUMB-EFSUMB initiative in cooperation with representatives of AFSUMB, AIUM, ASUM, FLAUS and ICUS, Ultraschall Med 34(1) (2013), 11–29.

Kim

T.K.

and Jang

H.J.

, Contrast-enhanced ultrasound in the diagnosis of nodules in liver cirrhosis, World J Gastroenterol 20(13) (2014), 3590–3596.

Strobel

, Seitz

, Blank

, et al., Tumor-specific vascularization pattern of liver metastasis, hepatocellular carcinoma, hemangioma and focal nodular hyperplasia in the differential diagnosis of 1,349 liver lesions in contrast-enhanced ultrasound (CEUS), Ultraschall Med 30(4) (2009), 376–382.

Seitz

, Bernatik

, Strobel

, et al., Contrast-enhanced ultrasound (CEUS) for the characterization of focal liver lesions in clinical practice (DEGUM Multicenter Trial): CEUS vs, MRI–a prospective comparison in 269 patients. Ultraschall Med 31(5) (2010), 492–499.

Seitz

, Strobel

, Bernatik

, et al., Contrast-Enhanced Ultrasound (CEUS) for the characterization of focal liver lesions - prospective comparison in clinical practice: CEUS vs, CT (DEGUM multicenter trial). Ultraschall Med 30(4) (2009), 383–389.

Strobel

, Bernatik

, Blank

, et al., Diagnostic accuracy of CEUS in the differential diagnosis of small (≤20mm) and subcentimetric (≤10mm) focal liver lesions in comparison with histology, Results of the DEGUM multicenter trial. Ultraschall Med 32(6) (2011), 593–597.

Friedrich-Rust

, Klopffleisch

, Nierhoff

, et al., Contrast-Enhanced Ultrasound for the differentiation of benign and malignant focal liver lesions: A meta-analysis, Liver Int 33(5) (2013), 739–755.

Seitz

, Greis

, Schuler

, et al., Frequency of tumor entities among liver tumors of unclear etiology initially detected by sonography in the noncirrhotic or cirrhotic livers of 1349 patients, Results of the DEGUM multicenter study. Ultraschall Med 32(6) (2011), 598–603.

Jung

, Clevert

, Schreyer

, et al., Evaluation of quantitative contrast harmonic imaging to assess malignancy of liver tumors: A prospective controlled two-center study, World J Gastroenterol 13(47) (2007), 6356–6364.

10.

Beyer

L.P.

, Pregler

, Wiesinger

, et al., Continuous dynamic registration of microvascularization of liver tumors with contrast-enhanced ultrasound, Radiol Res Pract 2014 (2014), 347416.

11.

Choi

J.Y.

, Lee

H.C.

, Yim

J.H.

, et al., Focal nodular hyperplasia or focal nodular hyperplasia-like lesions of the liver:A special emphasis on diagnosis, J Gastroenterol Hepatol 26(6) (2011), 1004–1009.

12.

Choi

J.C.

, Baek

Y.H.

, Jeong

J.S.

et al., Discrete hypoechoic ring in hepatic cavernous hemangioma resembling a malignant tumor: Correlation with histologic features, Gut Liver 3(3) (2009), 226–230.

13.

D’Onofrio

, Crosara

, De Robertis

, et al., Contrast-enhanced ultrasound of focal liver lesions, AJR Am J Roentgenol 205(1) (2015), W56–W66.

14.

, Yuan

M.X.

, Ma

K.S.

, et al., Detailed analysis of temporal features on contrast enhanced ultrasound may help differentiate intrahepatic cholangiocarcinoma from hepatocellular carcinoma in cirrhosis, PLoS One 9(5) (2014), 98612.

15.

, Zhang

, Ma

K.S.

, et al., Dynamic enhancing vascular pattern of intrahepatic peripheral cholangiocarcinoma on contrast-enhanced ultrasound: The influence of chronic hepatitis and cirrhosis, Abdom Imaging 38(1) (2013), 112–119.

16.

Liu

G.J.

, Wang

, Lu

M.D.

, et al., Contrast-Enhanced Ultrasound for the Characterization of Hepatocellular Carcinoma and Intrahepatic Cholangiocarcinoma, Liver Cancer 4(4) (2015), 241–252.

17.

Choi

B.I.

, Lee

J.M.

, Kim

T.K.

, et al., Diagnosing Borderline Hepatic Nodules in Hepatocarcinogenesis: Imaging Performance, AJR Am J Roentgenol 205(1) (2015), 10–21.

18.

Shah

, Shukla

and Paunipagar

, Radiological features of hepatocellular carcinoma, J Clin Exp Hepatol 4(Suppl 3) (2014), S63–S66.

19.

Vilana

, Forner

, Bianchi

, et al., Intrahepatic peripheral cholangiocarcinoma in cirrhosis patients may display a vascular pattern similar to hepatocellular carcinoma on contrast-enhanced ultrasound, Hepatology 51(6) (2010), 2020–2029.

20.

Wildner

, Bernatik

, Greis

, et al., CEUS in Hepatocellular Carcinoma and Intrahepatic Cholangiocellular Carcinoma in 320 Patients –Early or Late Washout Matters: A Subanalysis of the DEGUM Multicenter Trial, Ultraschall Med 36(2) (2015), 132–139.

21.

Wildner

, Pfeifer

, Goertz

R.S.

, et al., Dynamic contrast-enhanced ultrasound (DCE-US) for the characterization of hepatocellular carcinoma and cholangiocellular carcinoma, Ultraschall Med 35(6) (2014), 522–527.

22.

D’Onofrio

, Faccioli

, Malagò

, et al., Standardize and compare contrast-enhanced ultrasonographic digital images obtained with different technologies: How to overcome the subjectivity, J Digit Imaging 20(3) (2007), 256–262.

23.

Quaia

, Alaimo

, Baratella

, et al., Effect of observer experience in the differentiation between benign and malignant liver tumors after ultrasound contrast agent injection, J Ultrasound Med 29(1) (2010), 25–36.

24.

Anaye

, Perrenoud

, Rognin

, et al., Differentiation of focal liver lesions: Usefulness of parametric imaging with contrast-enhanced US, Radiology 261(1) (2011), 300–310.

25.

Rognin

N.G.

, Arditi

, Mercier

, et al., Parametric imaging for characterizing focal liver lesions in contrast-enhanced ultrasound, IEEE Trans Ultrason Ferroelectr Freq Control 57(11) (2010), 2503–2511.

26.

Yue

W.W.

, Wang

, Xu

H.X.

, et al., Parametric imaging with contrast-enhanced ultrasound for differentiating hepatocellular carcinoma from metastatic liver cancer, Clin Hemorheol Microcirc 64(2) (2016), 177–188.

27.

Pei

X.Q.

, Liu

L.Z.

, Xiong

Y.H.

, et al., Quantitative analysis of contrast-enhanced ultrasonography: Differentiating focal nodular hyperplasia from hepatocellular carcinoma, Br J Radiol 86(1023) (2013), 20120536.

28.

Sugimoto

, Shiraishi

, Tanaka

, et al., Computer-aided diagnosis for estimating the malignancy grade of hepatocellular carcinoma using contrast-enhanced ultrasound: An ROC observer study, Liver Int 36(7) (2016), 1026–1032.

29.

Dai

, Chen

M.H.

, Yin

S.S.

, et al., Focal liver lesions: Can SonoVue-enhanced ultrasound be used to differentiate malignant from benign lesions? Invest Radiol 42(8) (2007), 596–603.

30.

Ohno

, Miyati

, Yamashita

, et al., Quantitative assessment of tissue perfusion in hepatocellular carcinoma using perflubutane dynamic contrast-enhanced ultrasonography: A preliminary study, Diagnostics (Basel) 5(2) (2015), 210–218.

31.

Cui

X.W.

, Ignee

, Jedrzejczyk

, et al., Dynamic Vascular Pattern (DVP), a quantification tool for contrast enhanced ultrasound, Z Gastroenterol 51(5) (2013), 427–431.

32.

Schacherer

, Girlich

, Wiest

, Büttner

, et al., Semiquantitative characterization of hepatocellular carcinoma (HCC)–perfusion with contrast-enhanced ultrasound and perfusion analysis, Clin Hemorheol Microcirc 44(2) (2010), 97–105.

33.

S 3- Leitlinie Deutschland: AWMF-Leitlinie “Hepatozelluläres Karzinom“, Registernummer 032 - 053OL, Stand: 01.05.2013, gültig bis 30.04.2018; http://www.awmf.org/uploads/tx_szleitlinien/032-053OLl_S3_Hepatozellul%C3%A4res_Karzinom_Diagnostik_Therapie_2013-verlaengert.pdf

34.

Dietrich

C.F.

, Averkiou

M.A.

, Correas

J.M.

, et al., An EFSUMB introduction into Dynamic Contrast-Enhanced Ultrasound (DCE-US) for quantification of tumour perfusion, Ultraschall Med 33(4) (2012), 344–351.

35.

Barreiros

A.P.

, Piscaglia

and Dietrich

C.F.

, Contrast enhanced ultrasound for the diagnosis of hepatocellular carcinoma (HCC): Comments on AASLD guidelines, J Hepatol 57(4) (2012), 930–932.

36.

Bellissimo

, Pinzone

M.R.

, Cacopardo

, et al., Diagnostic and therapeutic management of hepatocellular carcinoma, World J Gastroenterol 21(42) (2015), 12003–12021.

37.

Bota

, Piscaglia

, Marinelli

, et al., Comparison of international guidelines for noninvasive diagnosis of hepatocellular carcinoma, Liver Cancer 1(3-4) (2012), 190–200.

38.

Bruix

and Sherman

, American Association for the Study of Liver Diseases, Management of hepatocellular carcinoma: An update. Hepatology 53(3) (2011), 1020–1022.

39.

Boozari

, Soudah

, Rifai

, et al., Grading of hypervascular hepatocellular carcinoma using late phase of contrast enhanced sonography - a prospective study, Dig Liver Dis 43(6) (2011), 484–490.

40.

Leoni

, Piscaglia

, Granito

, et al., Characterization of primary and recurrent nodules in liver cirrhosis using contrast-enhanced ultrasound: Which vascular criteria should be adopted? Ultraschall Med 34(3) (2013), 280–287.

41.

Giorgio

, Montesarchio

, Gatti

, et al., Contrast-Enhanced Ultrasound: A Simple and Effective Tool in Defining a Rapid Diagnostic Work-up for Small Nodules Detected in Cirrhotic Patients during Surveillance, J Gastrointestin Liver Dis 25(2) (2016), 205–211.

42.

McEvoy

S.H.

, McCarthy

C.J.

, Lavelle

L.P.

, et al., Hepatocellular carcinoma: Illustrated guide to systematic radiologic diagnosis and staging according to guidelines of the American Association for the Study of Liver Diseases, Radiographics 33(6) (2013), 1653–1668.

43.

Westwood

, Joore

, Grutters

, et al., Contrast-enhanced ultrasound using SonoVue^® (sulphur hexafluoride microbubbles) compared with contrast-enhanced computed tomography and contrast-enhanced magnetic resonance imaging for the characterisation of focal liver lesions and detection of liver metastases: A systematic review and cost-effectiveness analysis, Health Technol Assess 17(16) (2013), 1–243.

44.

Galassi

, Iavarone

, Rossi

, et al., Patterns of appearance and risk of misdiagnosis of intrahepatic cholangiocarcinoma in cirrhosis at contrast enhanced ultrasound, Liver Int 33(5) (2013), 771–779.

45.

Arditi

, Frinking

P.J.

, Zhou

, et al., A new formalism for the quantification of tissue perfusion by the destruction-replenishment method in contrast ultrasound imaging, IEEE Trans Ultrason Ferroelectr Freq Control 53(6) (2006), 1118–1129.

46.

Forner

, Vilana

, Bianchi

, et al., Lack of arterial hypervascularity at contrast-enhanced ultrasound should not define the priority for diagnostic work-up of nodules <2 cm, J Hepatol 62(1) (2015), 150–155.

47.

Hackl

, Schacherer

, Anders

, et al., Improved Detection of preclinical Colorectal Liver Metastases by High Resolution Ultrasound including Molecular Ultrasound Imaging using the targeted Contrast Agent BR55, Ultraschall Med 37(3) (2016), 290–296.

48.

Echtenacher

, Wege

A.K.

, Schardt

, et al., High-Resolution Ultrasound Including Contrast-Enhanced Ultrasound (CEUS) for the Detection of Gas Formation during Aspergillus Fumigatus Infection in Mice, Ultraschall Med 37(3) (2016), 277–282.

49.

Haimerl

, Poelsterl

, Beyer

L.P.

, et al., Chronic liver disease: Quantitative MRI vs CEUS-based microperfusion, Clin Hemorheol Microcirc 64(3) (2016), 435–446.

50.

Yue

W.W.

, Wang

, Xu

H.X.

, et al., Parametric imaging with contrast-enhanced ultrasound for differentiating hepatocellular carcinoma from metastatic liver cancer, Clin Hemorheol Microcirc 64(2) (2016), 177–188.