AWE - automated writing evaluators

I'm on the hunt for some technology and some consulting help to implement a grammar checker for the student writing materials on VocabularySpellingCity.com and for broad use with Time4Writing.com (web-based writing tutorials).

Right now, I'm following up on an HP-sponsored competition on automated writing evaluators over at Kaggle.com.  The question ansd answer there quickly led me to the ETS Section on Automated Scoring of Writing Quality.

My RFP is for technology to assess and provide specific feedback on the quality of student writing such as:

1. grammar mechanics: proper construction of sentences (subject, verb, final punctuation)    
2. Basic mechanics such as capitalization,  subject-verb agreement (tense, plural, etc), consistency on plural and singular
3. Advanced mechanics such as prepositional phrases, sentences of appropriate length, nuances of word choice, avoiding awkward, ineffective, or unclear writing
4. Writing quality: vocabulary usage, organization, development

ETC says (and I quote from http://www.ets.org/research/topics/as_nlp/writing_quality,  they have "an active research agenda to develop linguistic features suitable for modeling aspects of meaning structure in essay-length responses, such as:

• metrics of text coherence
• the use of supporting facts from external sources
• the writer's stance toward the material presented
• the identification of particular topics addressed in the response

Featured Publications

Below are some recent or significant publications that our researchers have authored on the subject of automated scoring of writing quality.

2012

• A Fast and Flexible Architecture for Very Large Word n-gram Datasets
  M. Flor
  Natural Language Engineering, FirstView online publication
  This paper presents a versatile architecture that uses a novel trie-based architecture, features lossless compression, and optimizes both speed and memory use. Learn more.

2010

• Using Parse Features for Preposition Selection and Error Detection (PDF)
  J. Tetreault, J. Foster, & M. Chodorow
  Proceedings of the 2010 Association for Computational Linguistics (ACL 2010)
  Association for Computational Linguistics
  This paper evaluates the effect of adding features that aim to improve the detection of preposition errors in writing from speakers of English as a second language. Download the full report (PDF).
• Rethinking Grammatical Error Annotation and Evaluation with the Amazon Mechanical Turk (PDF)
  J. Tetreault, E. Filatova, & M. Chodorow
  NAACL-HLT: 2010 Proceedings of the 5th Workshop on Building Educational Applications (BEA-5)
  Association for Computational Linguistics
  This paper presents the results of two pilot studies that show that using the Amazon Mechanical Turk for preposition error annotation is as effective as using trained raters, but at a fraction of the time and cost. Download the full report (PDF).
• Progress and New Directions in Technology for Automated Essay Evaluation
  J. Burstein & M. Chodorow
  The Oxford Handbook of Applied Linguistics, 2nd Edition, pp. 487–497
  Editor: R. Kaplan
  Oxford University Press
  This ETS-authored work is part of a 39-chapter volume that covers topics in applied linguistics with the goal of providing a survey of the field, showing the many connections among its sub-disciplines, and exploring likely directions of its future development. Learn more about this work.
• Unsupervised Prompt Expansion for Off-Topic Essay Detection (PDF)
  A. Louis & D. Higgins
  Proceedings of the Workshop on Building Educational Applications, HLT-NAACL 2010
  Association for Computational Linguistics
  This paper addresses the problem of getting software based on natural language processing technology to predict, without having previously analyzed essays as training data, whether an essay is "off-topic" — that is, irrelevant to the given prompt or question. Download the full report (PDF).
• Using Entity-Based Features to Model Coherence in Student Essays (PDF)
  J. Burstein, J. Tetreault, & S. Andreyev
  Human language technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pp. 681–684
  Association for Computational Linguistics
  This paper describes a study in which researchers combined an algorithm for observing what computational linguists refer to as entities — nouns and pronouns — with natural language processing features related to grammar errors and word usage with the aim of creating applications that can evaluate evidence of coherence in essays. Download the full report (PDF).

2009

• Examining the Use of Region Web Counts for ESL Error Detection (PDF)
  J. Tetreault & M. Chodorow
  SEPLN 2009: Proceedings of the Web as Corpus Workshop (WAC-5)
  Association for Computational Linguistics
  In this paper, researchers describe an approach to improving the amount of data available to train natural language processing systems to detect writing errors produced by nonnative English speakers. Download the full report (PDF).
• Human Evaluation of Article and Noun Number Usage: Influences of Context and Construction Variability (PDF)
  J. Lee, J. Tetreault, & M. Chodorow
  ACL 2009 Proceedings of the Linguistic Annotation Workshop III (LAW3)
  Association for Computational Linguistics
  In this study, researchers investigated a possible way of improving evaluating systems that correct errors in nonnative writing. Download the full report (PDF).

2008

• Native Judgments of Non-Native Usage: Experiments in Preposition Error Detection(PDF)
  J. Tetreault & M. Chodorow
  COLING 2008: Proceedings of the Workshop on Human Judgments in Computational Linguistics, pp. 24–32
  Association for Computational Linguistics
  In this paper, researchers consider ways to improve the data gathered from human raters used as part of the effort to create natural language processing applications as instructional or diagnostic tools. Download the full report (PDF).
• A Developmental Writing Scale
  Y. Attali & D. Powers
  ETS Research Report No. RR-08-19
  This report describes the development of grade norms for timed-writing performance in two modes of writing: persuasive and descriptive. View the full abstract or download this report.

2006

• Automated Essay Scoring With e-rater v.2.0
  Y. Attali & J. Burstein
  Journal of Technology, Learning, and Assessment, Vol. 4, No. 3
  This article describes Version 2 of ETS's e-rater essay scoring engine. The authors present evidence on the validity and reliability of the scores that the system generates. View the full abstract or download this article.

2005

• Online Assessment in Writing
  N. Horkay, R. E. Bennett, N. Allen, & B. Kaplan
  Online Assessment in Mathematics and Writing: Reports from the NAEP Technology-Based Assessment Project
  NCES Report No. 2005-457
  U.S. Department of Education, National Center for Education Statistics
  The 2002 Writing Online (WOL) study is the second of three field investigations in the Technology-Based Assessment project, which explores the use of new technology in administering the National Assessment of Educational Progress (NAEP). Learn more or download the full report.

2003

• Automated Evaluation of Discourse Structure in Student Essays
  J. Burstein & D. Marcu
  Automated Essay Scoring: A Cross-disciplinary Perspective, pp. 200–219
  Editors: M. D. Shermis & J. Burstein
  Routledge
  This ETS-authored chapter appears in the first edited volume to focus entirely on automated essay scoring and evaluation. Learn more about this work.
• Criterion: Online Essay Evaluation: An Application for Automated Evaluation of Student Essays (PDF)
  J. Burstein, M. Chodorow, & C. Leacock
  Proceedings of the Fifteenth Annual Conference on Innovative Applications of Artificial Intelligence
  Association for the Advancement of Artificial Intelligence
  This paper describes the Criterion Online Writing Evaluation Service, a web-based system that provides automated scoring and evaluation of student essays. Download the full report(PDF).
• Finding the WRITE Stuff: Automatic Identification of Discourse Structure in Student Essays
  J. Burstein, D. Marcu, & K. Knight
  IEEE Intelligent Systems: Special Issue on Advances in Natural Language Processing, Vol. 18, No. 1, pp. 32–39
  In this article, the authors discuss the use of automated essay-scoring applications in the elementary through university levels for large-scale assessment and classroom instruction.View the full abstract or order this article.