Development of methodology for evaluation of diagnostic performance has been an essential part of our department’s research for many decades. The primary focus of this work has been on Receiver Operating Characteristic (ROC) analysis, which describes the accuracy of a diagnostic modality, for a particular clinical task, in terms of its trade-off between sensitivity and specificity [2-4]. Three early insights of our group [5-9] were that novel ROC methodology was needed to meet the needs of medical image evaluation; that newly proposed statistical methods are of little value without careful validation of their behavior using both simulated and real datasets; and that development and free distribution of convenient, reliable software for ROC analysis could substantially encourage acceptance of the methodology.
Currently, software developed in accord with this theme (ROCFIT, LABROC4, PROPROC, CORROC, INDROC, ROCKIT and LABMRMC) or in collaboration with others (DBM-MRMC, with the University of Iowa) is used by thousands of registered users. These software packages and a Java-based, platform-independent interactive software package for ROC analysis that includes many alternative ROC analysis approaches are available for free download.
Our recent progress has included the addition of many non-parametric methods for analysis of ROC data [10] and a review of the medical-imaging literature to determine the spectrum of ROC methods used in published observer performance studies [11]. Current research efforts that eventually will be incorporated into our released software include development of “proper” ROC models for analysis of both partially- and fully-paired datasets (to be released soon), investigation of Bayesian approaches to ROC curve fitting, and development of quantitative methods that relate different test-result scales to each other and to individual operating points on ROC curves.
References
- Goodenough DJ, Rossmann K, Lusted LB. Radiographic applications of signal detection theory. Radiology 105: 199-200, 1972.
- Metz CE. Basic principles of ROC analysis. Seminars in Nuclear Medicine 8: 283298, 1978.
- Metz CE. ROC methodology in radiologic imaging. Investigative Radiology 21: 720-733, 1986.
- Wagner, RF, Metz, CE, Campbell, G. Assessment of medial imaging systems and computer aids: a tutorial review. Academic Radiology 14: 7238, 2007.
- Metz CE, Kronman HB. Statistical significance tests for binormal ROC curves. Journal of Mathematical Psychology 22: 218-243, 1980.
- Metz CE, Herman BA, Shen J-H. Maximum-likelihood estimation of ROC curves from continuously-distributed data. Statistics in Medicine 17: 1033-1053, 1998.
- Metz CE, Herman BA, Roe CA. Statistical comparison of two ROC-curve estimates obtained from partially-paired datasets. Medical Decision Making 18: 110-121, 1998.
- Metz CE, Pan X. Proper binormal ROC curves: theory and maximum-likelihood estimation. Journal of Mathematical Psychology 43: 1-33, 1999.
- Pesce LL, Metz CE. Reliable and computationally efficient maximum-likelihood estimation of “proper” binormal ROC curves. Academic Radiology 14: 814-ﾁ829, 2007.
- Gallas B, Pesce LL. Comparison of ROC methods for partially-paired data. Proc. SPIE, Vol. 7263, 72630V (2009)
- Shiraishi J, Pesce LL, Metz CE, Doi K. On Experimental Design and Data Analysis in Receiver Operating Characteristic (ROC) Studies: Lessons Learned from Papers Published in RADIOLOGY from 1997 to 2006. Radiology; 253: 822830, 2009
