Efficient Identification of Approximate Best Configuration of Training in Large Datasets

Authors

  • Silu Huang University of Illinois, Urbana-Champaign
  • Chi Wang Microsoft Research
  • Bolin Ding Alibaba Group
  • Surajit Chaudhuri Microsoft Research

DOI:

https://doi.org/10.1609/aaai.v33i01.33013862

Abstract

A configuration of training refers to the combinations of feature engineering, learner, and its associated hyperparameters. Given a set of configurations and a large dataset randomly split into training and testing set, we study how to efficiently identify the best configuration with approximately the highest testing accuracy when trained from the training set. To guarantee small accuracy loss, we develop a solution using confidence interval (CI)-based progressive sampling and pruning strategy. Compared to using full data to find the exact best configuration, our solution achieves more than two orders of magnitude speedup, while the returned top configuration has identical or close test accuracy.

Downloads

Published

2019-07-17

How to Cite

Huang, S., Wang, C., Ding, B., & Chaudhuri, S. (2019). Efficient Identification of Approximate Best Configuration of Training in Large Datasets. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 3862-3869. https://doi.org/10.1609/aaai.v33i01.33013862

Issue

Section

AAAI Technical Track: Machine Learning