Evaluation Methods for Machine Learning II
Papers from the 2007 AAAI Workshop
Chris Drummond, William Elazmeh, Nathalie Japkowicz, and Sofus A. Macskassy Program Cochairs
Technical Report WS-07-05 published by The AAAI Press, Menlo Park, California
This technical report is also available in book and CD format.
Contents
Organizing Committee / iii
Chris Drummond, William Elazmeh, Nathalie Japkowicz, and Sofus A. Macskassy
Preface / vii
Chris Drummond, William Elazmeh, Nathalie Japkowicz, and Sofus A. Macskassy
A Review of Performance Evaluation Measures for Hierarchical Classifiers / 1
Eduardo P. Costa, Ana C. Lorena, André C. P. L. F. Carvalho, and Alex A. Freitas
Making Evaluation Robust but Robust to What? / 7
Chris Drummond
Insights from Predicting Pediatric Asthma Exacerbations from Retrospective Clinical Data / 10
William Elazmeh, Stan Matwin, Dympna O'Sullivan, Wojtek Michalowski, and Ken Farion
Classifier Utility Visualization by Distance-Preserving Projection of High Dimensional Performance Data / 16
Nathalie Japkowicz, Pritika Sanghi, and Peter Tischer
A Framework for Analyzing Skew in Evaluation Metrics / 22
Alexander Liu, Joydeep Ghosh, and Cheryl Martin
Obtaining Calibrated Probabilities from Boosting / 28
Alexandru Niculescu-Mizil and Rich Caruana
On Comparison of Feature Selection Algorithms / 34
Payam Refaeilzadeh, Lei Tang, and Huan Liu
Scoring Hypotheses from Threat Detection Technologies: Analogies to Machine Learning Evaluation / 40
Robert C. Schrag and Masami Takikawa
Classifier Loss under Metric Uncertainty / 46
David B. Skalak, Alexandru Niculescu-Mizil, and Rich Caruana