AAAI Publications, The Twenty-Ninth International Flairs Conference

Font Size: 
Ranking Summaries for Informativeness and Coherence without Reference Summaries
Abhishek Singh, Wei Jin

Last modified: 2016-03-30


There are numerous applications of automatic summarization systems currently and evaluating the quality of the summary is an important task. Current summary evaluation methods are limited in their scope since they rely on a reference summary, i.e., a human written summary. In this paper, we present a new summary evaluation technique without the use of reference summaries. The framework consists of two sequential steps: feature extraction and rank learning and generation. The former extracts effective features reflecting generic aspects, coherence, topical relevance, and informativeness of summaries and the latter uses features to train a learning model that provides the capability of generating a pair wise ranking for input summaries automatically. Our proposed framework is evaluated on the DUC multi-document summarization dataset and results indicate that this is a promising direction for automatic evaluation of the summaries without the use of a reference summary.

Full Text: PDF