Automatic subject heading assignment for online government publications using a semi-supervised machine learning approach

Abstract

As the dramatic expansion of online publications continues, state libraries urgently need effective tools to organize and archive the huge number of government documents published online. Automatic text categorization techniques can be applied to classify documents approximately, given a sufficient number of labeled training examples. However, obtaining training labels is very expensive, requiring a lot of manual labor. We present a semi-supervised machine learning approach, an Expectation-Maximization (EM) algorithm text classifier, which makes use of easily obtained unlabeled documents and thus reduces the demand for labeled training examples. This paper describes the whole procedure of applying this approach to a real world online information preservation project where a collection is harvested from the websites of Illinois State Government agencies and a subject heading taxonomy is adapted from the State GILS topic tree. A formal evaluation has been performed based on the intended use of the assigned headings. The results demonstrate the semi-supervised approach improves subject heading assignment compared to the supervised approach, and is more efficient in using labeled documents.