NMA Campaign Issue: Machine Scoring of Writing
A Note About NMA Message Frameworks
NMA message frameworks are intended as starting points for NMA members to create their own messages. These might include (but are not limited to) letters to the editor, op-ed pieces, comments for local radio or television stations, and so on. Remember: the more locally grounded your message, the more likely it is to be included in local media. Feel free to add information based on your experience, your class, and/or your writing program.
NMA Message Framework - Machine Scoring
Computers cannot evaluate writing in the same way or as well as humans can. Computers score writing based on algorithms. These algorithms are of two basic types: 1) formulas that evaluate grammar and style like a typical word processor checks grammar and style, and 2) sets of word-matching and spatial relationship formulas. Current software cannot understand the contextualized meaning of a piece of writing. Machine-scoring emphasizes how effectively a piece of writing corresponds to a template; it does not allow us to judge how an effective writer composes his or her ideas about a topic for a real audience.
Computers make it easier for students to research, compose, and communicate with others. Software that connects student writers to one another and encourages multimodal (multimedia) composing activities is valuable for these purposes.
However, some testing and software companies are attempting to use computers for other purposes that will not help students develop as writers. These companies claim that they have developed software to make the teaching of writing easier by helping teachers find and correct errors in student writing. These programs are little more than grammar and style checkers that are part of almost every word processing program. A few of these companies claim that their scoring machines can understand the meaning of student writing.
Unfortunately, these programs can only understand a limited kind of "meaning" in brief responses (of between 350-500 words) about very specific questions with pre-determined answers. They cannot evaluate the kind of analytic, exploratory writing associated with higher order, critical thinking skills that are required for success in school and beyond. Additionally, these programs are not yet able to understand the style, tone, or context for a piece of writing; and they do not help writers develop their analyses of audiences or make conscious decisions about their writing. These are limited tools, not complete answers to the difficulties of developing effective composing skills.
Effective writers analyze the audience for a written text and the purpose(s) for composing it, synthesize initial ideas on the topic and those garnered from research, and organize intentions and content in a meaningful way. Student writers learn the complexities of writing through a social process of receiving feedback from their teacher, their peers, and other communities of learners.
Effective writers also understand that they must make conscious decisions about the form, style, and content of their writing for different audiences. Machines do not yet come close to recognizing how these sophisticated decisions are made and thus cannot effectively assess or rate most writing. Machine scoring software can only understand one audience: a generic 'grader' encoded into the software. When writing to this generic 'grader,' writers do not need to consider the complexities of communicating to different audiences for differing purposes.
The value of computer technologies for developing student writers is their potential as media through which students communicate. When students are communicating multiple complex ideas by composing with computers, they may benefit from using software as a tool to help with revision and editing. However, the use of software as a tool during revision and editing, does not mean that software can replace a human reader. When software is seen both as a medium to communicate among students and teachers and as a tool to help with revision and editing, it strengthens writing instruction. When software is treated as the only, or most important, reader of student writing, it degrades the importance of effective communication skills.
Writing is about conveying ideas, not filling in a template.
Resources for Learning About Machine Scoring
- Carl Whitthaus, "Green Squiggly Lines: Evaluating Student Writing in Computer-Mediated Environments"(from Academic Writing)
- Links to machine scoring programs: Accuplacer, Intellimetric, Criterion, and Intelligent Essay Assessor (part of a "Teaching Composition" module by Patricia Ericsson)
Tips for Attracting Attention of Local Media
- In development
This position statement developed by Patricia Ericsson and Carl Whithaus