Today, I still have been thinking a lot about that question. To remind you all, I was thinking yesterday about how my original intention was to use lexical simplification to create a simplification system that’s adaptive to the target reading audience. But then I wondered: is lexical simplification meaningful enough by itself for this task? Wouldn’t I want to incorporate syntactic simplifications as well?
I decided to address this by researching how different systems measure the readability of a paraphrase or piece of text (i.e., are lexical features important?). And I found three papers that seem to corroborate the theory that lexical features are important:
- Petersen and Ostendorf produced a paper called “A machine learning approach to reading level assessment.” This paper relates the most to what I am doing, because the authors are trying to train a system that predicts readability for different reading audiences. It turns out they they find that lexical features (e.g., average words in sentence, number of OOV words for each grade level) were significant in their machine learning system, at least when compared to syntactic features (such as sentence parsing).
- Xu et. al produced a paper called “Optimizing Machine Translation for Text Simplification,” where they adapt machine translation to the task of text simplification (through lexical means). To do that, they come up with simplification-specific features for paraphrase rules, so that the machine translator knows how to rank paraphrases. It turns out that they had success with mostly lexical features: “length in characters, length in words, number of syllables, language model scores, and fraction of common English words in each rule.”
I take these results a little less seriously than those of Petersen and Ostendorf, because Xu et. al were not trying to assess the quality of a simplification for a specific audience, just the quality in general. Not only that, but Xu et. al did not incorporate any features that quantified anything about the context that the paraphrase in.
- Francois et. al produced a paper “Do NLP and machine learning improve traditional readability formulas?” that explores how indicative “classic” readability metrics (which are more or less lexical) are of reading difficulty when compared to “non-classic” metrics (which are somewhat lexical, as well as syntactic). They found that exclusively “classic” metrics performed better than exclusively “non-classic” metrics, which implies that “classic” metrics (i.e., lexical features) better quantify readability.
I’m not sure what to make of this. I am doubting my theory about yesterday, now that I have seen this results. Maybe lexical simplification will be enough? Hopefully, I can bounce my ideas off of Professor Medero tomorrow morning.