Council of Writing Program Administrators

This assessment model is part of the WPA Assessment Gallery and Resources and is intended to demonstrate how the principles articulated in the NCTE-WPA White Paper on Writing Assessment in Colleges and Universities are reflected in different assessments. Together, the White Paper and assessment models illustrate that good assessment reflect research-based principles rooted in the discipline, is locally determined, and is used to improve teaching and learning.

Untitled Document

Assessment Narrative - Carleton College

Institution: Carleton College Type of Writing Program: Sophomore writing portfolio Contact Information: Carol Rutz, Ph.D., crutz@carleton.edu

Assessment Background and Research Question

Carleton College, a small liberal arts school in Minnesota, has emphasized writing across the curriculum since the mid-1970s. At that time, the college moved to a distributed model of writing instruction (integrating it throughout the curriculum), which meant that assessment of writing could not be directly linked to seat time.

As a result of these curricular changes, from 1975 to 2000, writing assessment was conducted by the professor teaching one of dozens of “writing requirement” courses designed to provide instruction and practice, particularly to first- and second-year students. If a student did not meet the instructor’s expectations for writing quality, the student would not be certified for the requirement, although she or he could earn a good grade in the course for other kinds of performance, such as exams and participation.

Growing dissatisfaction with the one-course, one-instructor method of assessing a graduation requirement led to an internal assessment of the writing requirement through interviews, focus groups, and surveys; findings revealed dissatisfaction on the part of both faculty and students, although there was little primary evidence to support the discontent. The task force assessing the writing requirement noted that no student writing was directly examined and evaluated for their report and perhaps some sort of study should be done—for example, the college could collect portfolios that students would carry with them to trace their development as writers.

This situation led to the development of our research question: how can the faculty determine that students can write well enough to succeed in advanced courses in the major?

In 1999, with the one-course assessment still in place, an associate dean applied to a regional foundation for a planning grant for faculty development (already a staple of WAC) tied to writing assessment. With funds in hand, Carleton invited nationally recognized experts on writing instruction and assessment to campus to introduce faculty and staff to models of good assessment and the supporting research.

Assessment Methods

As a result of our ongoing conversations about writing assessment, Carleton College developed a required sophomore writing portfolio. Students must submit between three to five essays from a variety of courses across the curriculum, along with a reflective essay, that demonstrate an acceptable mastery of essential aspects of college writing: observation, analysis of complex information, interpretation, identification and use of appropriate sources, writing thesis-driven arguments, and controlling Standard American English. According to Carleton faculty, these are the writing skills that augur success in the major.

Portfolios are read each summer by faculty volunteers, who receive modest stipends. Reading sessions are prefaced with training in reading—as opposed to grading—portfolios according to a holistic rubric. Three scores are possible: pass, exemplary, and needs work. Training stresses the need to find the boundaries between categories, recognizing that “pass” will be the largest category by far. As portfolio assessment literature predicts (e.g., Hamp-Lyons and Condon; Harrison), percentages tend to be around 80 percent “pass” and 10 percent each for “exemplary” and “needs work.” All files scored “exemplary” or “needs work” are read by at least two readers, and a percentage of “passes” are reread as well. In addition, a percentage of the previous year’s portfolios are reread as a reliability check. Once portfolios for an entire class are read—a three-day process—students are notified of results by email immediately; original portfolios, with reader comments, are returned to students in the fall.

Through the portfolio assessment, Carleton faculty across the curriculum can understand how and whether students are achieving the writing goals that have been established for them and provide additional intervention and support for students whose work falls short of those goals. Currently, these portfolios are submitted in hard copy; however, the process will likely migrate to an e-portfolio. More details on the uses of the portfolio are available at the URLs listed in the appendix.

Assessment Principles

The following principles proved to be important in developing and maintaining Carleton College’s portfolio assessment:

1. Whatever assessment we decided to employ, it had to be locally designed and performed. Because of the college’s long experience with WAC, Carleton faculty were unwilling to rely on a standardized exam or adopt an assessment instrument used elsewhere. Faculty were truly invested in teaching students how to write well, and they wanted to know the results of their efforts. Therefore, faculty agreed that they must be portfolio readers.

2. Before the planning grant, knowledge on campus about assessment in general and writing assessment in particular was largely absent. An understanding of the research and the potential for improvement of student writing went a long way toward convincing faculty that it was worth their trouble to conduct reliable, valid assessment of student work.

3. A research approach to assessment appealed to institutional values and offered data that would be valuable to departments, programs, and individual faculty. The previous assessment provided no information to faculty that would affect teaching or the curriculum in any way. Portfolio assessment combined with a faculty development program is beginning to change the teaching of writing at Carleton (Rutz and Lauer-Glebov).

4. Portfolio assessment provides context for student writing of all kinds, including writing from ESL students. One result of portfolio assessment has been a general relaxation of anxiety about ESL/EFL and similar issues. Those with lower tolerance for surface error or “writing accents” are more troubled than others, but everyone has a better sense of how to recognize the linguistic mechanisms at work in ESL/EFL writing. Furthermore, we can now document that ESL/EFL students generally perform at a consistently acceptable level.

5. Because the portfolio includes work that has already received passing grades in courses, the assessment itself is low stakes. The goal is to identify the segment of a given class that will need additional writing support to succeed in the major. As the literature predicts, that segment turns out to be 8–10 percent annually. Students whose portfolios do not pass meet individually with the Writing Program director or another writing professional to identify the problems and agree on remedies. Students then have one term to resubmit successfully.

6. The faculty development dimension has been essential. An integrated, iterative curriculum of faculty development activities has supported WAC and also provided a model for other curricular initiatives on campus. As a key part of that curriculum, the portfolio reading sessions are collegial, informative, and respectful of student achievement. Junior faculty in particular are able to norm their expectations through reading student work across disciplines in the company of more senior colleagues.

7. Portfolio assessment is open to investigation by faculty conducting other initiatives, notably a FIPSE-funded project on quantitative reasoning (QR), which uses student work collected in writing portfolios for program assessment. Carleton has benefited from having an archive of student work with IRB permission to use that work in research.

8. The portfolio scheme is constantly under review, although no substantive changes have been adopted yet. That may change when a curriculum review currently underway is completed. For example, what has been a paper process is likely to eventually migrate to the World Wide Web. Local infrastructure and management of the reading experience will be the most important factors to consider as that move takes place.

Assessment Results and Follow-Up Activities

Since Carleton College’s portfolio assessment is ongoing, it is continuously generating results and follow-up activities. Initially, when the portfolio requirements were being developed, the portfolio facilitated a process whereby faculty learned methods of assessing student writing and provided a structure through which they could discuss writing skills across the curriculum. As faculty participated in these initial activities, they developed a structure for a portfolio that would speak to rhetorical tasks necessary to succeed in all majors.

As faculty have continued to read portfolios, they have learned that their own teaching is affected by familiarity with student work across the curriculum. Faculty seldom get a chance to read work they have not assigned, and the portfolio allows for an efficient means of appreciating Carleton students’ experience as college writers. Through this rich experience of reading student work, faculty calibrate their expectations in their own courses, often revising their assignments to reflect what they have learned from assignments written by colleagues. Findings from the portfolio assessment have also served as the basis for ongoing faculty development. For example, workshops in December 2005 and 2007 focused on making arguments with numbers, combining the goals of both WAC and QR initiatives by helping faculty design assignments that require students to use data rhetorically. (See the link in the appendix for additional information.)

Students also benefit from the portfolio. They have learned, for example, how to develop a persuasive argument from documentary evidence—their own writing for courses. A community sense of “good writing” at the sophomore level also has clarified expectations of student work in advanced courses for students as well as faculty.

Assessment Resources

To get this assessment project off the ground, external funding was essential. We were fortunate to secure funds for visiting speakers, stipends for faculty workshops and portfolio readers, summer curriculum development grants, conference expenses, and other related activities. Sustainability, however, has been a concern from the beginning. Having funding that lasted six years helped change the local culture so that the portfolio is now an accepted and valued feature of Carleton’s curriculum. Regardless, the assessment requires resources. Recently, the college received a bequest that partially endows faculty development programming for WAC, including the portfolio reading sessions.

Other campus programs that benefited from Writing Program support in the past now include the Writing Program as they plan faculty development. WAC has become a platform for curricular development on campus.

Staffing for the portfolio has been limited to one full-time professional, a part-time administrative assistant, and student workers, who also work on WAC initiatives throughout the year. This particular portfolio assessment, with similar support, could be adapted at other institutions. In fact, a number of small liberal arts colleges have inquired about it, and some of them have launched pilot projects.

Appendix

Additional information about the Carleton College portfolio is available at http://apps.carleton.edu/campus/writingprogram/carletonwritingprogram/po....

Sample course assignments that would be appropriate for the portfolio and also speak to Carleton’s QR initiative are available at: http://serc.carleton.edu/sp/carl_ltc/quantitative_writing/index.html.

Some of the portfolio results, written for a student audience, are available at http://apps.carleton.edu/campus/writingprogram/carletonwritingprogram/.

References

Hamp-Lyons, Liz, and William Condon. Assessing the Portfolio. Cresskill, NJ: Hampton Press, 1999. Harrison, Suzan. “Portfolios Across the Curriculum.” WPA Journal 19.1-2 (1995): 38–49. Rutz, Carol, and Jacqulyn Lauer-Glebov. “Assessment and Innovation: One Darn Thing After Another.” Assessing Writing 10.2 (2005): 80–99.