The Naive Mix is a new supervised learning algorithm based on sequential model selection. The usual objective of model selection is to find a single probabilistic model that adequately characterizes, i.e. fits, the data in a training sample. The Naive Mix combines models discarded during the selection process with the best-fitting model to form an averaged probabilistic model. This is shown to improve classification accuracy when applied to the problem of determining the meaning of an ambiguous word in a sentence.