Semi-supervised learning for Natural Language

See link.
"In the spirit of (Miller et al., 2004), our basic strategy for taking advantage of
unlabeled data is to first derive features from unlabeled data|in our case, word
clustering or mutual information features|and then use these features in a supervised
learning algorithm. (Miller et al., 2004) achieved signicant performance gains in
named-entity recognition by using word clustering features and active learning. In
this thesis, we show that another type of unlabeled data feature based on mutual
information can also signicantly improve performance."
"(Shi and Sarkar, 2005) takes a similar approach for the problem of extracting
course names from web pages. They rst solve the easier problem of identifying
course numbers on web pages and then use features based on course numbers to solve
the original problem of identifying course names. Using EM, they show that adding
those features leads to signicant improvements."
The results were not that great.

Intimate Learning: A Novel Approach for Combining Labelled and Unlabelled Data

This paper describes a bootstrapping method
closely related to co-training and scoped-learning and is used for Web information extraction task -learning course names from web pages in which we use very few labelled items as seed data (10 web pages) and combine with an unlabelled set (174 web pages). The overall performance improved the precision/recall from 3.11%/0.31% for a baseline EM-based method to 44.7%/44.1% for intimate learning. They used the WebKB dataset.

In co-training there are two views of the same data but one class - but in their approach - there's one view but labeled into two classes (target and intimate classes)

Link: Researcher SSL NLP

Link to NLP Lab Simon Fraiser University.
They extended Abney's analysis of Yarowsky's algorithm.
In another work, we used the Yarowsky algortihm's idea and did semi-supervised learning to boost the quality of the best existing Machine Translation systems.
Currently, we are investigating new semi-supervised learning techniques for hidden Markov models and probabilistic context free grammars, which are probabilistic models used extensively to model and solve many tasks in many fields.

Semisupervised Learning for Computational Linguistics

Semisupervised Learning for Computational Linguistics book looks interesting. The Math has been tamed down - to help build intuition.
See the review here.

What is parser adaptation?

As per this paper: Leverage labeled data from one domain and create a parser capable of parsing a different domain. Their approach is simple: Train the parser on WSJ and then parse another domain. The parser generates n-best parses which are ranked using a re-ranker. The top ranked parse is used for self-training. Re-ranking and adaptive learning are open issues.

What is "co-training"?

As per this paper : Co-training like self-training learns on its own predictions. However, co-training learns two separate models (which are assumed to be independent typically by training on disjoint feature sets). These models are applied to unlabeled data. Examples on which these two models agree are treated as labeled data for a new round of training. One could incorporate model confidence to include only examples both models are confident of.
The definitive paper is Blum and Mitchel
Also this paper by Collins and Singer.

What is "self-training"?

As per this paper : One learns a model by training on a small amount of labeled data. The model is then evaluated on a large amount of unlabeled data. Its predictions are assumed to be correct and it is retrained on the unlabeled data according to its own predictions.

What is "constraint-based" learning?

I first heard of this in the issues for the NLP workshop. I found this paper.

Incorporate domain knowledge into semi-supervised learning - incorporate task specific constraints. For example: if you are performing an IE task of extracting fields from a citation record, then one can build constraints such as fields that start with 19xx or 20xx are dates. The generated solutions can be penalized for not meeting these constraints.

Workshop on Semi-supervised Learning for Natural Language Processing at NAACL HLT 2009

Interesting write-up of the problems in semi-supervised learning for NLP.
Parts that interest me:
1. "What are the different classes of NLP problem structures (e.g. sequences, trees, N-best lists) and what algorithms are best suited for each class? For instance, can graph-based algorithms be successfully applied to sequence-to-sequence problems like machine translation, or are self-training and feature-based methods the only reasonable choices for these problems? "
2. What kinds of NLP-specific background knowledge can we exploit to aid semi-supervised learning? Recent learning paradigms such as constraint-driven learning and prototype learning take advantage of our domain knowledge about particular NLP tasks?
3. Many semi-supervised learning methods (e.g. transductive SVM, graph-based methods) have been originally developed for binary classification problems. NLP problems often pose new challenges to these techniques, involving more complex structure that can violate many of the underlying assumptions.