Evaluation Methodology Research

Several Radiology faculty are working to develop new methodology for the evaluation of diagnostic performance. Much of this research focuses on Receiver Operating Characteristic (ROC) analysis, which describes the accuracy of each diagnostic modality in terms of the trade-offs that are available between its sensitivity and specificity in a particular clinical task [1-7]. Major themes of the work include observer-study design [8-10], ROC curve fitting [11-13], formulation of clinically relevant summary indices [14], tests for the statistical significance of differences between ROC estimates [15-17], analysis of sources of statistical variation in ROC estimates [18,19], methods for combining multiple observations [20-23], generalization of conventional ROC analysis to evaluate diagnostic tasks that involve more than two decision alternatives [24,25], and development and free distribution of software for analyses of ROC data Metz ROC Software (/page/metzroc-software) . New efforts now underway include investigation of Bayesian approaches to ROC curve fitting, development of quantitative methods that relate test-result scales to each other and to particular operating points on ROC curves, and a review of the medical-imaging literature to determine the spectrum of ROC methods used in published observer performance studies.

The Department of Radiology has a long history also in developing methodology for technical assessments of image quality. Important contributions to the characterization of x-ray imaging systems using Fourier-based measures such as the modulation transfer function and the Wiener (noise) power spectrum were developed here. Currently, these types of measures are being used to characterize and/or model the performance of new imaging systems such as breast tomosynthesis [26], image-based densitometry systems [27], and new stimulable phosphor (computed radiography) technology [28]. This work is being extended to characterize the whole imaging chain by incorporating model observer [29] and statistical or machine observers [30] to account for presence of anatomical structure noise in the image.

 

Research Faculty