Fully automated measurement of noise,signal-to-noise ratio,and contrast-to-noise ratio on chest CT images: feasibility and efficiency

Abstract

Background

Rapid and accurate measurement of computed tomography (CT) image noise, signal-to-noise ratio (SNR), and contrast-to-noise ratio (CNR) is a clinical challenge.

Purpose

To explore the feasibility of intelligent measurement of chest CT image noise, SNR, and CNR.

Material and Methods

A total of 300 chest CT scans were included in the study, which was divided into research dataset, internal test dataset, and external test dataset. Based on the research dataset, automatically segment and measure the average CT values and standard deviation (SD) of CT values for background air and lung field under different thresholds to obtain noise, SNR, and CNR results. Using the results of manual measurements as the reference standard, we determine the optimal threshold with the highest consistency. Using internal and external test datasets, validate the consistency of automated measurements of noise, SNR, and CNR at the optimal CT threshold with reference standards.

Results

With background air set at −900 HU and lung field at −800 HU as thresholds, the automated measurements of noise, SNR, and CNR demonstrate the highest consistency with the reference standards. At the optimal threshold, the noise, SNR, and CNR measured automatically on both the internal (intraclass correlation coefficient [ICC] = 0.85–0.96) and external (ICC = 0.75–0.85) test datasets exhibit high consistency with their respective reference standards.

Conclusion

The method we explored can intelligently measure the noise, SNR, and CNR of chest CT images, exhibits high consistency with radiologists, and offers a novel tool for image quality evaluation and analysis.

Keywords

Chest computed tomography noise automatic measurement quality control

Introduction

As technology advances, computed tomography (CT) has become widely used in clinical physical examinations (1). Since the outbreak of COVID-19, in particular, chest CT has been extensively employed as a diagnostic aid in the detection of the virus (2). The image quality (IQ) is crucial for accurate diagnosis by doctors (3). There are many parameters that can be used to indicate IQ, including image noise level, low-contrast detection capability, spatial resolution, and so on (4). The magnitude of noise may be the most important parameter in determining IQ (5,6), and signal-to-noise ratio (SNR) and contrast signal-to-noise ratio (CNR) are also commonly used as evaluation metrics for IQ assessment. Accurate and efficient measurement and evaluation of noise, SNR, and CNR in chest CT images are essential to enhance IQ, optimize diagnostic outcomes, and inform decision-making (3). This lays the foundation for further technological development and optimization of patient scanning parameters (6).

Usually, the measurement of noise level in clinical images is achieved by calculating the standard deviation (SD) of CT values within the region of interest (ROI), which is one of the main indicators for evaluating CT IQ and a core feature for evaluating other IQ indicators such as SNR and CNR (6 –9). In clinical settings, the measurement and calculation of noise, SNR, and CNR still rely on manual intervention. This is a time-consuming and laborious process. Furthermore, the selection of ROIs during manual measurements requires subjective perception from the evaluator, inevitably leading to differences among evaluators (10). Therefore, we hope to have a fully automated tool that can measure the noise, SNR, and CNR of CT images.

In recent years, some researchers have offered insights on the intelligent noise measurement process. Tian et al. (11) employed a method that involves subtracting adjacent image slices to assess the noise level in clinical CT images. Christianson et al. (12) introduced an innovative method utilizing a global noise index. This method segments CT images into soft tissue types based on a threshold, then uses a sliding window to generate SD maps, develops SD histograms from the CT image's SD maps, and identifies the peak pattern of the histogram corresponding to uniform tissue to determine the global noise level. Chun et al. (6) proposed a novel structural consistency feature to automatically measure CT noise in patient images. The method comprises four steps: segmentation of subcutaneous adipose tissue; measurement of structural consistency features; determination of uniform ROI; and estimation of noise magnitude (6,13). Due to the complex structure of the chest, few people have incorporated SNR and CNR into the automated measurement process.

Consequently, the aim of this study was to develop a fully automated measurement method. This method will automatically measure the noise, SNR, and CNR of chest CT images, thereby achieving an objective and quantitative evaluation of chest CT IQ. Furthermore, we evaluated the feasibility of the method by comparing the measurement results of two radiologists with those obtained through the automatic measurement.

Material and Methods

This retrospective study was approved by the ethics committee of this research institution. Informed consent was waived due to the use of retrospective image data.

CT datasets

This study collected the chest CT scan images of 300 patients from the PACS system of two medical centers between October 2018 and August 2019. A total of 200 patient chest CT images were selected from Medical Center A, with 100 cases allocated to the research dataset and 100 cases to the internal test dataset. In total, 100 patient chest CT images were chosen from Medical Center B to form the external test dataset. The inclusion criteria were patients who underwent routine dose, low dose, and high-resolution chest CT scans. The exclusion criteria were patients with serious illnesses that affect CT image quality. The detailed case collection process, disease details, CT acquisition parameters, and image preprocessing process can be found in the Supplementary material.

Selection of representative slice

Each chest CT scan typically comprises hundreds of images, and manually measuring each slice of all datasets is a massive task. Thus, it is necessary to select a specific slice for measurement and analysis. According to the research of Eberhard et al. (14), chest CT scans are divided into three regions—upper, middle, and lower—with representative slices connecting adjacent regions. The first representative slice is where the tracheal bifurcation appears. At the slice of tracheal bifurcation, the distinct bifurcation of the trachea and main bronchus offers clear anatomical markers for the accurate localization and orientation of the image, making it easily identifiable by deep learning algorithms (15). In the study by Do et al. (16), objective indicators of CT images were also measured at the slice of tracheal bifurcation. Consequently, this study designates the tracheal bifurcation slice as the representative slice for the measurement of evaluation indicators in chest CT images.

Manual measurement and calculation of noise, SNR, and CNR

Methods for manually assessing and calculating CT scan image noise, SNR, and CNR have been described in the literature (17). The radiologist selected the ROI in the uniform region, and the change in ROI pixel value was used to evaluate and calculate the noise, SNR, and CNR. All datasets in this study underwent manual measurements, with the results serving as the reference standard. All measurements were performed under a 5-mm lung window on chest CT images. Radiologist 1, with 6 years of experience in radiology, and radiologist 2, with 5 years of experience, used the RadiAnt DICOM Viewer software (as shown in Fig. 1) to measure the ROI in chest CT images, including the mean CT value and the standard deviation of the CT value. Noise is defined as the standard deviation of the average CT value within the same ROI, followed by the calculation of SNR and CNR. The settings for the ROI are as follows: the manually placed ROI is circular with an area of approximately 1 cm² (9,13), placing the ROI in a uniform region, and avoiding uneven regions such as bronchial tubes and lung markings when setting the ROI position. The formulas for calculating SNR and CNR are as follows (15,18):

S N R = \frac{H U_{l u n g}}{S D_{a i r}}

C N R = \frac{2 {(H U_{l u n g} - H U_{a i r})}^{2}}{S D_{l u n g}^{2} + S D_{a i r}^{2}}

Fig. 1.

Example diagram of manual measurement of ROI. (a) Example image of manual measurement of background air ROI; (b) example diagram of manual measurement of lung field ROI. ROI, region of interest.

In the formula, HU_air and HU_lung are the average CT values of the background air and lung field, respectively, and SD_air and SD_lung are the standard deviations of the CT values of the background air and lung field, respectively.

Radiologist 1 conducted two measurements on the three datasets, and the second measurement was conducted on 20 patient images randomly selected from each dataset. Radiologist 2 conducted one measurement. The average of the first measurement results of radiologists 1 and 2 was used as the reference standard level. The reliability between the measurers was evaluated by calculating the intraclass correlation coefficient (ICC) and Pearson’s correlation coefficient (r) between the first measurement results of radiologists 1 and 2, and the internal reliability of the measurer was evaluated by calculating the ICC and r between the repeated measurements of radiologist 1.

Automatic measurement program

Overall process

The automatic measurement program includes three main steps. First, the background air and lung field measurement area of the chest CT image were automatically segmented. Second, according to the predefined threshold, the average CT value and standard deviation within these measurement areas were automatically measured. Finally, the image noise, SNR, and CNR index were calculated.

Segmentation of background air and lung field

Multiple studies have confirmed that even the simplest U-Net network can achieve excellent lung segmentation results (19 –21). In this study, a standard 3D U-Net (https://github.com/MIC-DKFZ/nnUNet/releases/tag/v1.7.1) framework was employed to construct a segmentation model for the task of segmenting chest CT images (22). Our network input is chest CT images interpolated to a fixed resolution (3 × 3 × 3 mm), and the output is images of three different regions (left lung, right lung, and background air). The output activation function employs softmax, the loss function utilizes dice loss, and optimization is performed using the Adam optimizer. The segmented background air and lung field regions are illustrated in Fig. 2. The specific process of segmentation is as follows.

Fig. 2.

Example diagram of CT image segmentation. (a) The blue part is the segmented background air; (b) the blue and yellow parts are the segmented left and right lung fields.

The segmentation process of the lung field: First, utilizing the lung segmentation model, segment the complete left and right lung field regions. Second, based on the 3D trachea segmentation model, search for the layer where the tracheal bifurcation point is located to determine the 2D representative slice, and extract the lung field region from the 3D lung segmentation result at the representative slice. Finally, retract the left and right lung field regions inward by 10 mm, respectively, to obtain the final lung field region.

The segmentation process of background air: First, based on the 3D trachea segmentation model, search for the slice where the tracheal bifurcation point is located to determine the 2D representative slice. Second, identify pixels within the background air region with HU values in the range of −1200 to −900 as air candidate regions. Then, leveraging the segmentation results of the lung field, determine the largest circumscribed rectangular box encompassing the lung field and exclude all air candidate regions below the upper edge of this box. Finally, remove noise points to obtain the air segmentation region.

Two radiologists review all automatically segmented images, making manual adjustments when segmentation failures or inaccuracies are identified.

Selection and determination of threshold

The selection of threshold is crucial for the accuracy of measurement results, so this study conducted several experiments to determine the optimal threshold. The search for the optimal threshold was conducted in the research dataset, using −800, −850, −900, −950, and −1000 HU as the threshold for background air, and measuring the average CT value and SD of the background air. The SD of the CT value was used as an image noise indicator. Thresholds for the long field region were −450, −500, −550, −600, −650, −700, −750, −800, −850, and −900 HU, measuring the average CT value and SD of the lung field, and then calculating the SNR and CNR of the image. The background air threshold and lung field threshold were arranged and combined, comparing the consistency between the parameter results automatically measured under different thresholds and the reference standard results, to determine the optimal threshold combination of background air and lung field. To ensure overall consistency of measurement indicators, threshold combinations with correlation coefficients >0.7 for each indicator are selected for the final comparison.

Compare with results from manual measurements

The noise, SNR, and CNR of chest CT images on both internal and external test datasets were automatically measured under optimal threshold conditions. Then, the automatic measurement results were compared with the reference standard results obtained through manual measurement, and the consistency of the measurement results were assessed using ICC and r. Two radiologists manually measured the noise, SNR, and CNR of 50 randomly selected images in the CT datasets, and the average calculation time for manual measurement was obtained and compared with the average calculation time for automatic measurement.

Statistical analysis

All statistical analyses were conducted using SPSS software version 25.0 (IBM Corp., Armonk, NY, USA) and MedCalc software version 11.2 (MedCalc Software Ltd., Ostend, Belgium). A P value <0.05 was considered statistically significant. The ICC and r were used to evaluate the agreement between the reference standard and automated measurement results, as well as the agreement between manual measurement results among raters. Quantitative data are presented as mean ± SD, and categorical data are presented as proportions or percentages. In addition, a Bland–Altman plot was created to visually demonstrate the consistency between the measurement results and their reference standards.

Results

Demography

The general data distributions are summarized in Table 1. There was no significant difference between the three datasets in terms of gender composition and average age (P > 0.05).

Table 1.

Demographic data of three datasets.

	Research dataset	Internal test dataset	External test dataset	P value
Images, n	100	100	100
Age (years)	51.15 ± 16.41	52.22 ± 14.31	55.83 ± 20.41	>0.05
Sex				>0.05
Male	51 (51%)	61 (61%)	53 (53%)
Female	49 (49%)	39 (39%)	47 (47%)

Values are given as n (%) or mean ± SD.

Consistency of manual measurement

The noise, SNR, and CNR measured by the two radiologists were generally consistent. Radiologist 1’s repeated measurements of noise, SNR, and CNR were also overall consistent. The specific values can be found in Table 2.

Table 2.

Consistency between manual measurements.

Group	Indicators	Reader A₁ vs. reader B₁		Reader A₁ vs. reader A₂
Group	Indicators	ICC	r	ICC	r
Research dataset
	Noise	0.92	0.93	0.99	0.99
	SNR	0.89	0.89	0.98	0.99
	CNR	0.87	0.92	0.98	0.99
Internal test dataset
	Noise	0.94	0.97	0.99	0.99
	SNR	0.88	0.91	0.98	0.98
	CNR	0.93	0.96	0.97	0.97
External test dataset
	Noise	0.94	0.94	0.98	0.98
	SNR	0.95	0.95	0.97	0.97
	CNR	0.85	0.85	0.97	0.98

CNR, contrast-to-noise ratio; ICC, intraclass correlation coefficient; SNR, signal-to-noise ratio.

Determination of optimal threshold

Under all threshold combinations of the research dataset, the ICC and r values of noise indicators are 0.10 to 0.88, respectively; the ICC and r values of SNR indicators are 0.04 to 0.93, respectively; and the ICC and r values of CNR indicators are 0.36 to 0.95. respectively. Detailed data are provided in Table S1. After the screening of correlation coefficients, four threshold combinations were ultimately compared (Table 3). At the threshold combination of background air (−900 HU) and lung field (−800 HU), the consistency between the automatic measurement results and the reference standard results is the highest. Meanwhile, a Bland–Altman plot was drawn to illustrate the consistency between the automatic measurement results and the reference standards (Fig. 3).

Fig. 3.

The Bland–Altman plot shows the consistency between automatic and manual measurements in noise, SNR, and CNR. (a–c) Consistency plots for the automatic and manual measurements of noise, SNR, and CNR in the research dataset; (d–f) consistency plots for the automatic and manual measurements of noise, SNR, and CNR in the internal test dataset; (g–i) consistency plots for the automatic and manual measurements of noise, SNR, and CNR in the external test dataset. CNR, contrast-to-noise ratio; SNR, signal-to-noise ratio.

Table 3.

The consistency results of manual and automatic measurement results under different threshold combinations after screening.

Background air threshold (HU)	Lung field threshold (HU)	ICC			r
Background air threshold (HU)	Lung field threshold (HU)	Noise	SNR	CNR	Noise	SNR	CNR
−900	−850	0.73	0.81	0.90	0.82	0.89	0.94
−900	−800	0.73	0.81	0.95	0.82	0.90	0.95
−900	−750	0.73	0.80	0.86	0.82	0.90	0.89
−900	−700	0.73	0.80	0.76	0.82	0.90	0.85

CNR, contrast-to-noise ratio; ICC, intraclass correlation coefficient; SNR, signal-to-noise ratio.

Compare with results from manual measurements

In both internal and external test datasets, the consistency of automatically measured noise, SNR, and CNR results with the reference standard is visible in Table 4. Meanwhile, a Bland–Altman plot was also used to demonstrate the consistency between automatic and manual measurements of the three indicators (Fig. 3). The average time for manually measuring the three indicators of 50 images is 2902.65 s, and the average calculation time for each case is 58.053 s. The average measurement time per case for automatic measurement is 19.02 s, which is 3.05 times faster than manual measurement.

Table 4.

The consistency evaluation results of manual and automatic measurements on internal and external test datasets.

Group	Indicators	Manual	Automatic	ICC	r
Internal test dataset
	Noise	27.04 ± 9.46	28.03 ± 8.40	0.96	0.97
	SNR	36.39 ± 10.57	34.02 ± 7.56	0.85	0.93
	CNR	6.97 ± 6.53	6.95 ± 5.45	0.90	0.92
External test dataset
	Noise	26.48 ± 9.01	28.59 ± 6.18	0.84	0.94
	SNR	36.57 ± 12.04	32.50 ± 7.04	0.75	0.93
	CNR	11.09 ± 7.00	8.68 ± 6.04	0.85	0.92

Values are given as mean ± SD unless otherwise indicated.

CNR, contrast-to-noise ratio; ICC, intraclass correlation coefficient; SNR, signal-to-noise ratio.

Discussion

This study introduces a fully automated measurement method, leveraging deep learning segmentation models, for assessing noise levels, SNR, and CNR in clinical chest CT images. Leveraging segmentation models, the background air and lung field regions in chest CT images can be automatically segmented. The average CT value and SD of the segmented regions under a specified threshold combination are automatically measured to evaluate the noise level and calculate the SNR and CNR.

The fully automated measurement method proposed in this study, which starts from clinical images of patients, eliminates the need for repetitive CT scans, significantly reducing radiation exposure. Compared to manual reference standards, the automatic measurement results from both the internal and external test datasets demonstrate superior consistency. This suggests that the fully automatic measurement method proposed in this study is sufficiently accurate and reliable for measuring chest CT image noise, SNR, and CNR. During the measurement process, this method remains unaffected by individual subjective factors, thereby reducing subjective errors and significantly enhancing the objectivity and repeatability of the measurement. Moreover, fully automated measurement methods are typically capable of processing a large volume of clinical image data efficiently, eliminating the need for substantial manual intervention.

In their study, Reeves et al. (17) conducted noise measurements using a full CT scan, leading to extensive calculations. In our study, we have significantly streamlined the computational process by opting for the tracheal bifurcation layer as the representative slice for parameter index measurements. During the process of fully automatic measurement, it is crucial to select the appropriate measurement region. In the manual measurement process, radiologists can select uniform regions for measurement based on subjective judgment. Therefore, in the automatic measurement process, we have defined the measurement scope to ensure consistency. For the background air measurement region, due to the complex structure below the chest CT image,this study uses the rectangular region above the upper edge of the lung field as the background air measurement region. For the lung field measurement region, due to the unclear boundary between the segmented whole lung field and the soft tissue of the chest wall, measuring the complete lung field inevitably includes some chest wall components, resulting in differences in measurement results. Therefore, we shrink the complete lung field region inward by 10 mm to obtain the corrected lung field region as the lung field measurement region. In addition, this study established a threshold for the CT value in the automatically measured region to eliminate non-air components in the background and uneven anatomical structures within the lung field. To ensure the rationality of the threshold setting, this study conducted a large number of experiments, using 50 HU as the interval, and measured the results of noise, SNR, and CNR within a wide range of threshold intervals. Finally, the optimal threshold interval was obtained.

The previously published study by Christianson et al. (12) involved automated techniques for measuring noise in clinical CT images. They designated the noise parameter as global noise and verified the precision of the automated assessment by contrasting it with reference noise. In our study, we improved the measurement accuracy to a certain extent by using the segmentation model to measure the local noise of clinical CT images, rather than relying solely on global noise. Recently, Ketola et al. developed a CNN model for the quantification of noise in chest CT images, enabling the prediction of noise SD maps without the need for repeated CT scans (23). In contrast to their approach, we not only measured the local noise level of chest CT images, but also incorporated SNR and CNR, for a more comprehensive assessment of image quality.

The present study has some limitations. First, the sample size selected is relatively small. While images of three different types of chest CT scans from two medical centers were collected, the overall number is limited. Second, the inward indentation for defining the lung field measurement range is quite subjective. Third, only two radiologists were used as expert evaluators to establish the reference standard level in this study.

In conclusion, this study introduces a fully automated method for measuring noise, SNR, and CNR in chest CT images. This method can quickly and accurately measure these parameters from clinical images. The measurement results from this method align closely with the reference standards established by two radiologists, allowing it to replace radiologists in tedious measurement tasks, providing a new method for big data IQ assessment and analysis.

Supplemental Material

sj-docx-1-acr-10.1177_02841851241287315 - Supplemental material for Fully automated measurement of noise, signal-to-noise ratio, and contrast-to-noise ratio on chest CT images: feasibility and efficiency

Supplemental material, sj-docx-1-acr-10.1177_02841851241287315 for Fully automated measurement of noise, signal-to-noise ratio, and contrast-to-noise ratio on chest CT images: feasibility and efficiency by Bozhe Mei, Zhangman Ma, Wanyun Fu, Linyang He, Zhicheng Ma and Xiangyang Gong in Acta Radiologica

Footnotes

Declaration of conflicting interests

The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: LH is an employee of Hangzhou Jianpei Technology Company Ltd.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Key Research and Development Project of Zhejiang Province of China (2020C01058).

ORCID iDs

Bozhe Mei

Xiangyang Gong

Supplementary material

Supplementary material for this article is available online.

References

McCollough

Guimarães

Fletcher

. In defense of body CT. AJR Am J Roentgenol 2009;193:28–39.

Aljondi

Alghamdi

. Diagnostic value of imaging modalities for COVID-19: scoping review. J Med Internet Res 2020;22:e19673.

Samei

Järvinen

Kortesniemi

, et al. Medical imaging dose optimisation from ground up: expert opinion of an international summit. J Radiol Prot 2018;38:967–989.

Goldman

. Principles of CT: radiation dose and image quality. J Nucl Med Technol 2007;35:213–225; quiz 226–218.

Schuhbaeck

Schaefer

Marwan

, et al. Patient-specific predictors of image noise in coronary CT angiography. J Cardiovasc Comput Tomogr 2013;7:39–45.

Chun

Choi

Kim

. Automated measurement of CT noise in patient images with a novel structure coherence feature. Phys Med Biol 2015;60:9107–9122.

Dodge

Tamm

Cody

, et al. Performance evaluation of iterative reconstruction algorithms for achieving CT radiation dose reduction – a phantom study. J Appl Clin Med Phys 2016;17:511–531.

Hussain

Mail

Shamy

, et al. A qualitative and quantitative analysis of radiation dose and image quality of computed tomography images using adaptive statistical iterative reconstruction. J Appl Clin Med Phys 2016;17:419–432.

Anam

Budi

Adi

, et al. Assessment of patient dose and noise level of clinical CT images: automated measurements. J Radiol Prot 2019;39:783–793.

10.

Anam

Triadyaksa

Naufal

, et al. Impact of ROI size on the accuracy of noise measurement in CT on computational and ACR phantoms. J Biomed Phys Eng 2022;12:359–368.

11.

Tian

Samei

. Accurate assessment and prediction of noise in clinical CT images. Med Phys 2016;43:475.

12.

Christianson

Winslow

Frush

, et al. Automated technique to measure noise in clinical CT examinations. AJR Am J Roentgenol 2015;205:W93–W99.

13.

Anam

Arif

Haryanto

, et al. An improved method of automated noise measurement system in CT images. J Biomed Phys Eng 2021;11:163–174.

14.

Eberhard

Blüthgen

Barth

, et al. Vertical off-centering in reduced dose chest-CT: impact on effective dose and image noise values. Acad Radiol 2020;27:508–517.

15.

Schimmöller

Lanzman

Dietrich

, et al. Evaluation of automated attenuation-based tube potential selection in combination with organ-specific dose reduction for contrast-enhanced chest CT examinations. Clin Radiol 2014;69:721–726.

16.

Rheinheimer

Kauczor

, et al. Image quality evaluation of dual-layer spectral CT in comparison to single-layer CT in a reduced-dose setting. Eur Radiol 2020;30:5709–5719.

17.

Reeves

Xie

Liu

. Automated image quality assessment for chest CT scans. Med Phys 2018;45:561–578.

18.

Jiang

Miao

Liang

, et al. The value of deep learning image reconstruction in improving the quality of low-dose chest CT images. Diagnostics (Basel) 2022;12:2560.

19.

Sousa

Pereira

Silva

, et al. Lung segmentation in CT images: a residual U-net approach on a cross-cohort dataset. Appl Sci 2022;12:1959.

20.

Maiello

Ball

Micali

, et al. Automatic lung segmentation and quantification of aeration in computed tomography of the chest using 3D transfer learning. Front Physiol 2021;12:725865.

21.

Wasserthal

Breit

Meyer

, et al. Totalsegmentator: robust segmentation of 104 anatomic structures in CT images. Radiol Artif Intell 2023;5:e230024.

22.

Isensee

Jaeger

Kohl

SAA

, et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods 2021;18:203–211.

23.

Ketola

JHJ

Inkinen

Mäkelä

, et al. Automatic chest computed tomography image noise quantification using deep learning. Phys Med 2024;117:103186.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB