Generating Natural Language Questions to Support Learning On-Line

David Lindberg, Fred Popowich, John Nesbit, Phil Winne

Proceedings of the 14th European Workshop on Natural Language Generation

pages: 105–114

Read Online

Summary: This article describes an approach to automatic question generation that makes use of semantic information contained in the text. Their approach is proposed with a pedagogical goal in mind, to support online learning. They also use templates, but based on semantic roles and syntactic context. They evaluate the generated questions in terms of: grammaticality, semantic validity, vagueness, answerability and learning value. Their initial experiments showed that although most of the questions they generate are grammatical, very few of them had pedagogical value.

My comments: I think the article is well-written. Although I had a few questions left towards the end, I attribute it largely to the page limit of the conference, not to the authors themselves. The rules part was explained clearly and there were good examples of what this system can do.

• I really need to understand how to write rules for extracting specific syntactic and semantic structures from text.

• Although the evaluation was given a less preferential treatment in this paper (in my view), I found it very interesting. Typical QG evaluations range from asking human evaluators to evaluate the grammaticality and appropriateness of generated questions to using automated measures such as BLEU to compare with some gold standard answers. However, here, they also look at two additional aspects:

  • answerability: whether the question can be answered from the given context (usually a sentence)
  • learning value: is this a useful question from a pedagogical point of view? (you can ask all sorts of questions even from a single sentence - should we ask? is the question)

I did not see “learning value” being considered for evaluation in AQG systems before. I wish there was more on this - hopefully, I will find this in some other publications of the authors.

Main take away: I need to gain more insights into the development of rule based approaches, and perhaps work on some small project following such an approach.

One sentence: This article is refreshingly good!