CiteSeerX: AI in a Digital Library Search Engine

  • Jian Wu Pennsylvania State University
  • Kyle Mark Williams Pennsylvania State University
  • Hung-Hsuan Chen Industrial Technology Research Institute
  • Madian Khabsa Pennsylvania State University
  • Cornelia Caragea University of North Texas
  • Suppawong Tuarob Pennsylvania State University
  • Alexander G. Ororbia Pennsylvania State University
  • Douglas Jordan Pennsylvania State University
  • Prasenjit Mitra Pennsylvania State University
  • C. Lee Giles Pennsylvania State University

Abstract

CiteSeerX is a digital library search engine providing access to more than five million scholarly documents with nearly a million users and millions of hits per day. We present key AI technologies used in the following components: document classification and de-duplication, document and citation clustering, automatic metadata extraction and indexing, and author disambiguation. These AI technologies have been developed by CiteSeerX group members over the past 5–6 years. We show the usage status, payoff, development challenges, main design concepts, and deployment and maintenance requirements. We also present AI technologies implemented in table and algorithm search, which are special search modes in CiteSeerX. While it is challenging to rebuild a system like CiteSeerX from scratch, many of these AI technologies are transferable to other digital libraries and/or search engines.
Published
2015-09-28
Section
Articles