Assessment Gallery and Resources - Assessment Models and Communication Strategies

Please Note: Some sections of these narratives are available only in the PDF versions. PDF attachments are available only for registered users. For more information see Obtaining a WPA username and password.

Introduction

In the last three years, writing instructors and program administrators have heard more than ever before about the need for assessment. Sometimes calls for this work are generated by instructors and program administrators themselves who are interested in learning about how instructional practices are affecting student learning in writing courses. Increasingly, though, these calls are joined by others from outside—from campus administrators, accrediting bodies, or other external stakeholders. These external calls can be couched in language that has increasingly become part of the assessment discussion, such as “accountability,” “transparency,” or “comparability.”

In response to these increasingly vocal calls for discussion, the Council of Writing Program Administrators (WPA) and the National Council of Teachers of English (NCTE) have partnered to bring writing instructors and program administrators the NCTE-WPA White Paper on Writing Assessment in Colleges and Universities, which draws from existing resources to summarize best practice principles for post-secondary writing assessment.

The Assessment Gallery and Resources includes two components intended to support and illustrate the principles delineated in the White Paper. These include:

Model Assessments. The Assessment Gallery provides models of specific assessments from a range of institutions, from 2-year colleges to R1 universities, that enact the strategies in the NCTE-WPA White Paper. Together, these models illustrate that valid, reliable, and fair assessment reflects consistent principles, and that these principles can be enacted through questions and methods that are appropriate for the institution, the department, and the program.

Communication Strategies. Communicating with interested parties, from department colleagues to campus administrators and external constituencies, is a crucial part of developing a successful assessment. This document offers a framework for communicating and strategies for WPAs to share information and develop alliances with others.

Together, the White Paper and Assessment Gallery and Resources are intended to help writing instructors and program administrators communicate a message made clear in research-based best practices in the English language arts: Valid and reliable assessment is consistent at the level of principle and conceptualization: it is discipline-based, locally determined, and used to inform teaching and learning at the local level.

We hope that these NCTE-WPA materials provide valuable resources for instructors and program administrators as they go about the important work of assessing student writing, writing courses, and writing programs.

NCTE-WPA Task Force Members

Assessment Gallery and Resources - Communication Strategies

Communicating with interested parties, from department colleagues to campus administrators and external constituencies, is a crucial part of developing a successful assessment. This document offers a framework for communicating and strategies for WPAs to share information and develop alliances with others.

The Communication Strategies linked below are intended to help WPAs and writing instructors communicate with interested parties about their assessment work and build these alliances.

NCTE-WPA White Paper on Writing Assessment in Colleges and Universities

The National Council of Teachers of English and the Council of Writing Program Administrators offer this statement, a white paper, on writing assessment in postsecondary education. This white paper is meant to help teachers, administrators, and other stakeholders articulate the general positions, values, and assumptions on writing assessment that both the National Council of Teachers of English and the Council of Writing Program Administrators jointly endorse. What follows is an articulation of common understandings and general agreements in the membership of both organizations on the following:

The connections among language, literacy, and writing assessment

The principles of effective writing assessment

The appropriate, fair, and valid use of writing assessment

The role and importance of reliability in writing assessment

Connections: Language, Literacy, and Writing Assessment

Writing instruction and literacy education at all levels are formal ways in which societies build citizens, and in which citizens develop reading and communication behaviors and competencies in order to participate in various communities. Learning to write better involves engaging in the processes of drafting, reading, and revising; in dialogue, reflections, and formative feedback with peers and teachers; and in formal instruction and imitative activities. A preponderance of research argues that literacy and its teaching are socially contextualized and socially constructed dynamics, evolving as people, exigency, context, and other factors change. The varied language competencies and experiences with which students come to the classroom can sometimes conflict with what they are taught or told to value in school. The assessment of writing, therefore, must account for these contextual and social elements of writing pedagogy and literacy.

Principles of Effective Writing Assessment

The principles of effective writing assessment that can take the form of classroom tests and grades or extracurricular exams measuring student writing ability are highly contextual, and should be adapted or modified in accordance with local needs, issues, purposes, and concerns of stakeholders. These assessments function across large-scale and classroom contexts and are used to make important decisions about students, curriculum, and teachers. Generally, there is agreement about the following principles that tend to be a part of effective, meaningful, and responsible writing assessment:

Writing assessment should place priority on the improvement of teaching and learning.
Writing assessment responds to student, teacher, institutional, and other stakeholder needs. It should be used to foster environments for student learning. In placement testing, this principle might demand that administrators consider the local classroom conditions students will be entering after they have been placed into a writing course, or the places in the local communities from which students come.

Writing assessment should demonstrate that students communicate effectively.
The effectiveness of student performance should be connected to criteria relevant to the educational decisions the assessment is designed to facilitate. For example, in placement testing, student performance should indicate a readiness for the curriculum of the course in which the student is placed. In exit testing, student performance should indicate the completion of course goals and objectives and a readiness to write for the next course or courses in the curriculum. We acknowledge that writing assessment must communicate to a variety of stakeholders the essence of what we want students to learn and the evidence of such learning.

Writing assessment should provide the foundation for data-driven, or evidence-based, decision making.
In some cases, assessment is designed to improve student performance, and in others to improve teaching and curricula. The purposes for assessment differ depending on the desired results of the assessment project. Programs may assess end products of a student’s semester-long work to consider how and whether that work demonstrates the outcomes for the course. Depending on the purpose of the assessment, results can be used to improve instruction at multiple points in the curriculum.

Writing assessment should be informed by current scholarship and research in assessment.
While writing assessment should be locally grown and implemented, those designing, implementing, and validating writing assessments should also stay informed of current developments in the fields of writing assessment, composition theory, and literacy studies. This means that those involved in writing assessment should be supported (financially and otherwise) to share and disseminate their own assessment and validation findings and work.

Writing assessment should recognize diversity in language.
The methods and language that teachers and administrators use to make decisions and engage students in writing, reading, responding, and revising activities should incorporate meaningfully the multiple values and ways of expressing knowledge by students present in the classroom and local communities. Assessments and the decisions made from them should account for students’ rights to their own languages (see the Guideline approved by the Conference on College Composition and Communication in 1974 and reaffirmed in 2003).

Writing assessment should positively impact pedagogy and curriculum.
Curriculum designers and teachers should attempt to understand and incorporate into instruction the ways in which the assessments can improve the curriculum and instruction in classrooms. Positive writing assessment takes into account the nature of writing as a social process and product, situated within particular contexts (e.g., classrooms or timed environments), and limited or shaped by these factors.

Writing assessment should use multiple measures and engage multiple perspectives to make decisions that improve teaching and learning.
These multiple measures and perspectives can include the use of several readers and the perspectives they bring to student texts. A single off-the-shelf or standardized test should never be used to make important decisions about students, teachers, or curriculum.

Writing assessment should include appropriate input from and information and feedback for students.
Students should have access to the goals, purposes, and scoring criteria for required assessments. Students should also receive appropriate feedback for any important decisions made about them.

Writing assessment should be based on continuous conversations with as many stakeholders as possible.
Developing, researching, and validating a writing assessment is a constant process, and one should expect the assessment, its results, and its products to change over time. Thus, it is important to have conversations about the assessment (e.g., dialogue about the features particular teachers notice in student portfolios in various courses).

Writing assessment should encourage and expect teachers to be trusted, knowledgeable, and communicative.
Teachers should be the primary agents in writing assessment, and therefore need to be continually educated in writing assessment, to engage in dialogue with one another locally, and to find ways to gain the trust of the other stakeholders. Additionally, other stakeholders should support teachers in their efforts to become more knowledgeable about writing assessment and to communicate to all stakeholders involved.

Writing assessment should articulate and communicate clearly its values and expectations to all stakeholders, especially students and, if applicable, parents.
Assessment should not be invisible, mysterious, or elusive to any stakeholders. There should be a variety of ways stakeholders can understand and be informed about the local writing assessment and its methods, findings, and products.

Appropriate, Fair, and Valid Use of Writing Assessment

The Appropriate use of writing assessment, whether in a classroom or large-scale context, means that it fits the context and decisions that will be made based on it. Appropriateness can also be understood as a measure of the decisions made. For example, when placing students into courses based on portfolio readings, one might ask—and measure in some way—how appropriate the decisions are (do students and teachers later find that the placements put students in the right places?). Appropriateness might also be considered regarding the kinds of evaluation/feedback provided, based on their purpose or use (e.g., grades, summative feedback, formative feedback, recorded audio responses, no responses, detailed annotations/marginalia, responses offered to the entire class and not individual students, etc.).

The Fair use of writing assessment is crucial, since it can be used to make important decisions about individuals. A concern for fairness should guard against any disproportionate social effects on any language minority group. Writing assessments that are used to make important decisions about individuals and the material and educational conditions that affect these individuals should provide an equal opportunity for students to understand the expectations, roles, and purposes of the assessment. For instance, if students have no recourse, or opportunities to respond to evaluations or judgments of their writing, or if they do not have any access to the criteria used to evaluate their writing or to the uses of the assessments of their writing, then those assessments may be unfair. Considering the fair use of power does not mean giving equal power to decide to all stakeholders in an assessment. It means all stakeholders should have as much power over the assessment as their particular roles and positions dictate they can have, considering the ethical and expedient administration of the assessment, and the purposes of judgments.

The Valid use of writing assessment decisions and evaluations is a complex and technical activity. “Validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests” (American 9). Every use of an assessment requires a validation inquiry in which an argument is made that the theoretical understanding of the assessment and the evidence the assessment generates support the decisions being made on behalf of the assessment. For example, if we use any method to place students into first-year writing courses, we must provide evidence that students are being correctly placed and profit from the educational experience. Questions such as how well students learn in each course of the curriculum must be answered in order to validate placement decisions. This inquiry-driven, researched-based activity is a required part of the appropriate, fair, and valid use of writing assessment.

Reliable Assessment

A reliable assessment provides consistent results, no matter who conducts the assessment. Because writing assessment often involves more than one rater scoring student performances, it can also involve interrater reliability, a measure of the degree of consistency from one rater judgment to another. A student’s score thus might depend upon the bias of the reader rather than upon the document or product being assessed. Attention to reliability is an integral part of any responsible validity argument.

Works Cited

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association, 1999.

Conference on College Composition and Communication. “Students’ Right to Their Own Language.” College Composition and Communication 25 (1974).

Additional References:

Writing Assessment: A Position Statement prepared by the CCCC Assessment Committee

NCTE Framing Statements on Assessment

WPA Assessment Gallery: Assessment Models

The Assessment Gallery provides models of specific assessments from a range of institutions, from 2-year colleges to R1 universities, that enact the strategies in the NCTE-WPA White Paper. Together, these models illustrate that valid, reliable, and fair assessment reflects consistent principles, and that these principles can be enacted through questions and methods that are appropriate for the institution, the department, and the program.

Attached are assessment narratives from the first group of featured institutions. Each narrative includes contact information for the author/contact and addresses assessment questions, principles, methods, findings, application, and adaptability to other institutions. The WPA-NCTE Ad Hoc Task Force is grateful to the authors of these assessments, who invested many hours of time to concisely describe their model projects.

Current featured projects include assessments from:

Carleton College (Carol Rutz)

Frederick Community College (Kenneth Kerr)

George Mason University (Terry Myers Zawacki)

Saint Joseph's College (Judy Arzt and Kristine Barnett)

Salt Lake Community College Community Writing Center (Tiffany Rousculp)

Seattle University (John Bean)

Tidewater Community College (Chris Dixon)

University of Kentucky (Connie Kendall, Darci Thoune and Deborah Kirkman)

Assessment Narrative - Carleton College

This assessment model is part of the WPA Assessment Gallery and Resources and is intended to demonstrate how the principles articulated in the NCTE-WPA White Paper on Writing Assessment in Colleges and Universities are reflected in different assessments. Together, the White Paper and assessment models illustrate that good assessment reflect research-based principles rooted in the discipline, is locally determined, and is used to improve teaching and learning.

Untitled Document

Assessment Narrative - Carleton College

Institution: Carleton College Type of Writing Program: Sophomore writing portfolio Contact Information: Carol Rutz, Ph.D., crutz@carleton.edu

Assessment Background and Research Question

Carleton College, a small liberal arts school in Minnesota, has emphasized writing across the curriculum since the mid-1970s. At that time, the college moved to a distributed model of writing instruction (integrating it throughout the curriculum), which meant that assessment of writing could not be directly linked to seat time.

As a result of these curricular changes, from 1975 to 2000, writing assessment was conducted by the professor teaching one of dozens of “writing requirement” courses designed to provide instruction and practice, particularly to first- and second-year students. If a student did not meet the instructor’s expectations for writing quality, the student would not be certified for the requirement, although she or he could earn a good grade in the course for other kinds of performance, such as exams and participation.

Growing dissatisfaction with the one-course, one-instructor method of assessing a graduation requirement led to an internal assessment of the writing requirement through interviews, focus groups, and surveys; findings revealed dissatisfaction on the part of both faculty and students, although there was little primary evidence to support the discontent. The task force assessing the writing requirement noted that no student writing was directly examined and evaluated for their report and perhaps some sort of study should be done—for example, the college could collect portfolios that students would carry with them to trace their development as writers.

This situation led to the development of our research question: how can the faculty determine that students can write well enough to succeed in advanced courses in the major?

In 1999, with the one-course assessment still in place, an associate dean applied to a regional foundation for a planning grant for faculty development (already a staple of WAC) tied to writing assessment. With funds in hand, Carleton invited nationally recognized experts on writing instruction and assessment to campus to introduce faculty and staff to models of good assessment and the supporting research.

Assessment Methods

As a result of our ongoing conversations about writing assessment, Carleton College developed a required sophomore writing portfolio. Students must submit between three to five essays from a variety of courses across the curriculum, along with a reflective essay, that demonstrate an acceptable mastery of essential aspects of college writing: observation, analysis of complex information, interpretation, identification and use of appropriate sources, writing thesis-driven arguments, and controlling Standard American English. According to Carleton faculty, these are the writing skills that augur success in the major.

Portfolios are read each summer by faculty volunteers, who receive modest stipends. Reading sessions are prefaced with training in reading—as opposed to grading—portfolios according to a holistic rubric. Three scores are possible: pass, exemplary, and needs work. Training stresses the need to find the boundaries between categories, recognizing that “pass” will be the largest category by far. As portfolio assessment literature predicts (e.g., Hamp-Lyons and Condon; Harrison), percentages tend to be around 80 percent “pass” and 10 percent each for “exemplary” and “needs work.” All files scored “exemplary” or “needs work” are read by at least two readers, and a percentage of “passes” are reread as well. In addition, a percentage of the previous year’s portfolios are reread as a reliability check. Once portfolios for an entire class are read—a three-day process—students are notified of results by email immediately; original portfolios, with reader comments, are returned to students in the fall.

Through the portfolio assessment, Carleton faculty across the curriculum can understand how and whether students are achieving the writing goals that have been established for them and provide additional intervention and support for students whose work falls short of those goals. Currently, these portfolios are submitted in hard copy; however, the process will likely migrate to an e-portfolio. More details on the uses of the portfolio are available at the URLs listed in the appendix.

Assessment Principles

The following principles proved to be important in developing and maintaining Carleton College’s portfolio assessment:

Whatever assessment we decided to employ, it had to be locally designed and performed. Because of the college’s long experience with WAC, Carleton faculty were unwilling to rely on a standardized exam or adopt an assessment instrument used elsewhere. Faculty were truly invested in teaching students how to write well, and they wanted to know the results of their efforts. Therefore, faculty agreed that they must be portfolio readers.

Before the planning grant, knowledge on campus about assessment in general and writing assessment in particular was largely absent. An understanding of the research and the potential for improvement of student writing went a long way toward convincing faculty that it was worth their trouble to conduct reliable, valid assessment of student work.

A research approach to assessment appealed to institutional values and offered data that would be valuable to departments, programs, and individual faculty. The previous assessment provided no information to faculty that would affect teaching or the curriculum in any way. Portfolio assessment combined with a faculty development program is beginning to change the teaching of writing at Carleton (Rutz and Lauer-Glebov).

Portfolio assessment provides context for student writing of all kinds, including writing from ESL students. One result of portfolio assessment has been a general relaxation of anxiety about ESL/EFL and similar issues. Those with lower tolerance for surface error or “writing accents” are more troubled than others, but everyone has a better sense of how to recognize the linguistic mechanisms at work in ESL/EFL writing. Furthermore, we can now document that ESL/EFL students generally perform at a consistently acceptable level.

Because the portfolio includes work that has already received passing grades in courses, the assessment itself is low stakes. The goal is to identify the segment of a given class that will need additional writing support to succeed in the major. As the literature predicts, that segment turns out to be 8–10 percent annually. Students whose portfolios do not pass meet individually with the Writing Program director or another writing professional to identify the problems and agree on remedies. Students then have one term to resubmit successfully.

The faculty development dimension has been essential. An integrated, iterative curriculum of faculty development activities has supported WAC and also provided a model for other curricular initiatives on campus. As a key part of that curriculum, the portfolio reading sessions are collegial, informative, and respectful of student achievement. Junior faculty in particular are able to norm their expectations through reading student work across disciplines in the company of more senior colleagues.

Portfolio assessment is open to investigation by faculty conducting other initiatives, notably a FIPSE-funded project on quantitative reasoning (QR), which uses student work collected in writing portfolios for program assessment. Carleton has benefited from having an archive of student work with IRB permission to use that work in research.

The portfolio scheme is constantly under review, although no substantive changes have been adopted yet. That may change when a curriculum review currently underway is completed. For example, what has been a paper process is likely to eventually migrate to the World Wide Web. Local infrastructure and management of the reading experience will be the most important factors to consider as that move takes place.

Assessment Results and Follow-Up Activities

Since Carleton College’s portfolio assessment is ongoing, it is continuously generating results and follow-up activities. Initially, when the portfolio requirements were being developed, the portfolio facilitated a process whereby faculty learned methods of assessing student writing and provided a structure through which they could discuss writing skills across the curriculum. As faculty participated in these initial activities, they developed a structure for a portfolio that would speak to rhetorical tasks necessary to succeed in all majors.

As faculty have continued to read portfolios, they have learned that their own teaching is affected by familiarity with student work across the curriculum. Faculty seldom get a chance to read work they have not assigned, and the portfolio allows for an efficient means of appreciating Carleton students’ experience as college writers. Through this rich experience of reading student work, faculty calibrate their expectations in their own courses, often revising their assignments to reflect what they have learned from assignments written by colleagues. Findings from the portfolio assessment have also served as the basis for ongoing faculty development. For example, workshops in December 2005 and 2007 focused on making arguments with numbers, combining the goals of both WAC and QR initiatives by helping faculty design assignments that require students to use data rhetorically. (See the link in the appendix for additional information.)

Students also benefit from the portfolio. They have learned, for example, how to develop a persuasive argument from documentary evidence—their own writing for courses. A community sense of “good writing” at the sophomore level also has clarified expectations of student work in advanced courses for students as well as faculty.

Assessment Resources

To get this assessment project off the ground, external funding was essential. We were fortunate to secure funds for visiting speakers, stipends for faculty workshops and portfolio readers, summer curriculum development grants, conference expenses, and other related activities. Sustainability, however, has been a concern from the beginning. Having funding that lasted six years helped change the local culture so that the portfolio is now an accepted and valued feature of Carleton’s curriculum. Regardless, the assessment requires resources. Recently, the college received a bequest that partially endows faculty development programming for WAC, including the portfolio reading sessions.

Other campus programs that benefited from Writing Program support in the past now include the Writing Program as they plan faculty development. WAC has become a platform for curricular development on campus.

Staffing for the portfolio has been limited to one full-time professional, a part-time administrative assistant, and student workers, who also work on WAC initiatives throughout the year. This particular portfolio assessment, with similar support, could be adapted at other institutions. In fact, a number of small liberal arts colleges have inquired about it, and some of them have launched pilot projects.

Appendix

Additional information about the Carleton College portfolio is available at http://apps.carleton.edu/campus/writingprogram/carletonwritingprogram/po....

Sample course assignments that would be appropriate for the portfolio and also speak to Carleton’s QR initiative are available at: http://serc.carleton.edu/sp/carl_ltc/quantitative_writing/index.html.

Some of the portfolio results, written for a student audience, are available at http://apps.carleton.edu/campus/writingprogram/carletonwritingprogram/.

References

Hamp-Lyons, Liz, and William Condon. Assessing the Portfolio. Cresskill, NJ: Hampton Press, 1999. Harrison, Suzan. “Portfolios Across the Curriculum.” WPA Journal 19.1-2 (1995): 38–49. Rutz, Carol, and Jacqulyn Lauer-Glebov. “Assessment and Innovation: One Darn Thing After Another.” Assessing Writing 10.2 (2005): 80–99.

Assessment Narrative - Frederick Community College

Assessment Narrative - Frederick Community College

Institution: Frederick Community College Type of Writing Program: Freshman Composition Contact Information: Kenneth Kerr, EdD Professor of English Frederick Community College 7932 Opossumtown Pike Frederick, MD 21702 301-846-2646

Assessment Background and Research Questions

Frederick Community College has embraced Terry O’Banion’s concept of the “learning college.” One of the principles of that philosophy is, “[T]he learning college and its facilitators succeed only when improved and expanded learning can be documented for its learners” (47). Aside from that, as an institution accredited by the Middle State Commission, we have an obligation under standard 14 to assess student learning in a meaningful and rigorous way and to use the results of assessment activities to improve teaching and learning.

FCC has created an ongoing assessment model stemming from these two points in which questions are raised by and stem from teachers’ classroom work, and the curriculum is continuously revised based on responses to those questions. These questions always focus on how well students are achieving the learning outcomes for courses that are at the center of the assessment. Instructors volunteer to participate in the assessment on a three-year cycle.

As part of their annual self-evaluations, each full-time faculty member is required to report what he or she has done over the previous year to improve teaching and learning in his or her courses. In this self-evaluation, faculty identify an area of concern in one or more course or core learning outcome. They discuss what was done to improve teaching and learning in that area and report how learning improved as a result of the change.

On a regular three-year cycle, faculty then self-select to participate in a formal, rigorous assessment project that focuses on issues of learning improvement. Often these issues come from the self-evaluations. These self-selected faculty volunteers shoulder the responsibility for their departments and oversee the three-year assessment project. They are responsible for designing, implementing, collecting, and analyzing data, and for submitting the final course-level assessment report. Over time, all members of the department are expected to take a turn leading the assessment efforts. This is considered service to the college equivalent to a committee assignment. To ensure that the assessment projects are sufficiently rigorous and likely to yield meaningful, reliable, and valid data about teaching and learning, each department submits an assessment project plan to the Outcome Assessment Council, which then considers and approves the plan. Once approval is given, the department proceeds to collect and analyze data, implement change, and reassess for effectiveness. The cycle works like this:

Year 1: Departments submit a plan for assessing student learning. This plan will assess all four aspects of the Maryland Higher Education Commission (MHEC) general education requirements.

Year 2: Departments collect and analyze data and recommend changes to teaching and learning.

Year 3: Departments reassess to determine the effectiveness of the changes and submit a final report.

The cycle then begins again with a new team investigating a new area of learning in a different high-enrollment course.

The Assessment

In the fall of 2004, the English department developed an assessment project to determine how well students are learning college-level communication skills in our English Composition (EN101) classes. Initially, we conducted a general assessment to gather some baseline data about how well our students were meeting our expectations at the end of the course. We developed a rubric to assess what we thought were the most important skills we wanted students to gain from the course: Content, Organization, Grammar/Punctuation/Mechanics.

In 2005, we decided to focus on the same general question, “How well are students learning college-level communication skills in our English Composition (EN101) classes?” We took as our criteria the outcomes for our course:

Write effective, organized, clear, concise, grammatically correct compositions using appropriate stylistic options (tone, word choice, and sentence patterns) for a specific subject, audience, and purpose (informing, arguing, or persuading).
Demonstrate the ability to understand and interpret both written texts and oral presentations in English.
Understand the critical role of listening in communication.
Demonstrate an ability to organize ideas effectively by selecting and limiting a topic.
Develop and support a thesis with relevant and well-reasoned material.
Employ a logical plan of development and use effective transitions.
Demonstrate an understanding of the conventions of the English language writing essays that are substantially free of errors in grammar, spelling, punctuation, and mechanics.

Develop critical thinking skills by:
Evaluating evidence by differentiating among facts, opinions, and inferences;
Generating and evaluating alternative solutions to problems; and
Researching, analyzing, comparing, synthesizing, and drawing inferences from readings and other research materials in order to make valid judgments and rational decisions.

Assessment Methods

All twelve full-time English department faculty collected final research papers (with all identifying information removed) from their regular sections during the spring of 2005. Using the outcomes-based criteria, we examined a sample of the student papers. The results indicated that students were strongest in the area of grammar and mechanics and also demonstrated competency in organizing written communication. However, we found that students did not effectively use information and material to support assertions in their writing.

We then developed a follow-up study focusing on two new research questions: Is the problem that our students do not know how to find information and materials to support the assertion of their theses? Or, once students have found information, do they not know how to use it effectively? In the fall of 2006, we repeated the assessment project, specifically addressing these two questions. We provided students with specific, college-level reading material (articles from a scholarly journal and a general periodical) to be studied outside the classroom. Faculty explained the assignment, distributed materials, and set aside a full class period during the final week of class to have students write an essay responding to the readings. Instructors were cautioned to refrain from dissecting and discussing the materials with students because we were trying to determine their ability to read critically. Students were expected to prepare materials beforehand and could bring in materials and notes to class for drafting. (Although part of our study, the assignment was also a regular class assignment graded by the teacher.)

In total, 281 student papers were in the sample representing 24 sections taught by the full-time faculty and three adjuncts. From this pool, we randomly selected 60 papers (20 percent) for evaluation. The papers were assessed using a four-point rubric, which we developed based on our outcomes and earlier study, that ranged from “accomplished” to “not evident.” EN101 In-Class Writing Assessment Rubric

Learning Objective

4 Accomplished

3 Competent

2 Developing

1 Not Evident

The writing demonstrates that the student can differentiate facts, opinions, and inferences.

Student has clearly indicated when he or she is stating opinion as opposed to fact.

Student does not specifically state whether the evidence is based on fact or opinion.

Student uses data incorrectly to support a position on the issue.

The response is entirely based on personal opinion.

The student analyzes information from various sources.

Student uses information from both sources provided and clearly identifies the specific source of the material.

Student uses information from both sources.

Student uses only one source to support the position.

Student uses no information from the sources provided.

The student recognizes and develops alternative perspectives and solutions.

The response indicates that an original alternative solution to the prompt has been developed.

Student accepts a solution proposed by one or the other of the provided sources.

Information from the material is summarized, but no solution is offered.

Student misreads the data and/or misinterprets what he or she has been asked to write about.

The student evaluates alternatives to make a sound judgment.

The student presents and considers multiple alternatives before proposing one as preferred.

The student mentions the existence of opposing positions.

The student presents a single position as the only possible solution.

No alternatives are presented. No judgment is present.

Three readers read each paper. When a reader determined that a paper was “competent,” or fell above or below “competent,” he or she was asked to provide some explanation for that determination. The purpose of the second assessment project was to determine if students could effectively use college-level materials provided to them in response to a prompt they were given in advance. We found that they had trouble handling this information in a competent manner. Consequently, we concluded that we needed to concentrate our efforts on helping students become better critical readers.

Assessment Principles

Since FCC’s assessment is ongoing, the methods for assessing questions depend on the questions being asked. Both departments and individual faculty are expected to participate in ongoing assessments. In the English department, we used the following principles to help develop our assessment program and projects:

The use of writing assessment should be appropriate, fair, valid, reliable, and equitable. We made sure that every paper had an equal chance of being included in the sample. To show that the emphasis was on student work, we removed any indication of the instructor and the student. All papers were read by all three evaluators, who were calibrated prior to evaluating papers.

Writing assessment places priority on improvement of teaching and learning. To reinforce the idea that the purpose of assessment is improvement, results were not presented in such a way that any individual student or instructor could be identified as either weak or exceptional. The idea is to improve the course as a whole. Assessment must be seen as ameliorative, not punitive.

Student learning outcomes assessment at Frederick Community College is a comprehensive effort focused on measuring student academic achievement. In writing courses, this means assessment results can be used to improve student writing in other classes and other disciplines. As we introduce learning innovations that are effective in improving student learning, these are communicated to other instructors through professional development opportunities throughout the year.

Writing assessment provides evidence of student learning. By assessing random samples of students over many semesters, we are able to determine whether the learning innovations we have introduced as a result of assessment data are effective.

Students also are active partners in the learning process. (Could this extend to a principle of involving students in assessment, or using student work as a central focus of assessment?) Our students are made aware of what is being assessed in the work they produce. We specify what evidence of learning will be looked at closely and remind them.

Writing assessment is ongoing, providing the foundation for data-driven, or evidence-based, decision making. Because the assessment project takes place over a period of years and provides data that can be used to measure effectiveness, we can develop changes to our courses based on evidence and then measure the effectiveness of those changes.

Writing assessment articulates and communicates clearly its values and expectations. Our students are told what is being assessed and how it will be assessed. They are provided with the assessment rubric. Faculty are aware that they have nothing to fear from assessing their students’ learning. The values and expectations are that everyone is committed to improvement and no one need fear reprisal because of disappointing results.

Writing assessment is site-specific. We wanted to develop a program of assessment that would fit within our current practices. We didn’t want to “invent” a way to assess. We wanted to “find” existing opportunities to assess student learning. We wanted to incorporate assessment into what we were already doing so as to cause the least disruption of teaching practices. Our program was also developed with consideration of our size, our resources, and our mission.

Assessment Results

The results of the 2005 assessment project indicated that students were strongest in the area of grammar and mechanics. Students also demonstrated competency in organizing written communication. As previously stated, the area of greatest weakness was content. Specifically, we learned that students did not effectively use information and material to support assertions in their writing. This led us to the two new research questions described earlier: Is the problem that our students do not know how to find information and materials to support the assertion of their theses? Or, once students have found information, do they not know how to use it effectively? We believed that the answer to improving student learning in the area of content lay in the answer to these questions.

The purpose of the second assessment project was to determine whether students could effectively use college-level materials provided to them in response to a prompt they were given in advance. If we found that they could not handle this information in a competent manner, we could then concentrate our efforts on helping students become better critical readers. Analysis of the data showed that, under the best possible interpretation, 67 percent were determined to be fully competent or better in all three learning objectives by at least two of the three evaluators. Only 27 percent of the sample papers were determined to be fully competent or better by all three evaluators. Additionally, fewer than 50 percent of the papers were competent or better on any individual learning objective.

Assessment Follow-Up Activities

Our second research project—spring 2006—showed us that even though we provided the students with appropriate college-level material, they were not consistently able to use it to support a thesis. Our assumption is that we must spend more time teaching critical reading and integration of that reading into essay writing to support a thesis. This assumption is bolstered by the placement data that show the number of students requiring developmental writing decreasing while the number requiring developmental reading increasing at a greater-than-inverse relationship.

Based on these findings, we developed a version of EN101 that will focus more directly on teaching critical reading. We have also designed a new assessment project. We proposed a true experimental design project in which six sections of EN101 Freshman Composition will be given intensive critical reading instruction and practice. Six other sections will be designated as control sections, in which no teaching/learning innovation will be introduced. These students will get the same class they normally would. We will then repeat the in-class essay using provided material for all twelve sections and compare student performance. If the study group out-performs the control group, we will know that we need to rework the EN101 course to include extensive critical reading. We can begin developing professional development materials to teach EN101 instructors how to improve learning in this area.

Assessment Resources

While there was no additional funding for these assessments, FCC invests significantly in the assessment process. FCC provides institutional support in the Office of Assessment, where we have two full-time researchers available to assist with design and analysis. There is professional development money available for anyone involved in assessment who wants additional training. Requests for equipment are made from existing initiative money; there is no supply budget for assessment. Finally, faculty involved in assessment are recognized for their work at a status equivalent with serving on an important college committee. Assessment work is considered the type of service we would expect to see from someone applying for promotion.

Portability and Sustainability of the Design

The course-level assessment used by FCC is sustainable because of its ongoing, three-year project design. This design encourages that assessment data be used to improve teaching and learning. Furthermore, the effectiveness of the instructional innovations, developed as a result of assessment data analysis, is itself assessed for efficacy. Once an opportunity to improve teaching and learning is identified and ameliorated, the methods by which student learning is improved are presented to all instructors through professional development. After that, a new area of student learning is identified and the cycle continues.

This design is portable and adaptable to other two-year colleges. We all have course outcomes that have been identified as important skills, knowledge, or abilities we want our learners to demonstrate at the end of a course. We all have embedded opportunities within these courses that provide students a chance to demonstrate their mastery of these skills, knowledge, or abilities. This design identifies existing assessment opportunities rather than creating new tasks for instructors and students. In this way, we think more deeply about the types of work we are asking students to do and why we are asking them to do them. In doing so, we give more meaningful assignments to our students and receive better data on how well they are learning what it is we want them to learn.

This design requires only a little bit more work by a few people for a limited period. Three people from each department work about 25 to 40 additional hours over the course of a year for three years. Much of this work is done over the summer months when they are not teaching and can be done asynchronously when it best suits their schedules and preferences. These three are then excused from departmental assessment projects until everyone else in the department has served in that capacity.

Additionally, we all complete annual self-evaluations. It is proper that we should self-evaluate the effectiveness of the courses we teach. Reflecting each year on one aspect of one of our courses that we think can be improved, on what we did to improve it, and on how well the change worked is not only an appropriate professional self-reflection but also valuable data for our colleagues and our college.

This design is minimally intrusive, involves minimal additional resources, and requires minimal additional effort on the part of a very few faculty above what they are already doing.

Reference

O’Banion, Terry. A Learning College for the 21st Century. Phoenix: American Council on Education and Oryx Press, 1997.

Assessment Narrative - George Mason University

Assessment Narrative - George Mason University

Institution: George Mason University Type of Writing Program: Writing across the Curriculum; required upper-division writing-intensive courses in the major Contact Information: Terry Myers Zawacki Director, WAC and University Writing Center tzawacki@gmu.edu

Assessment Background and Research Question

George Mason University, a large Virginia state institution located outside of Washington, D.C., has had a well-established Writing across the Curriculum (WAC) program dating from 1977. The components of the program include an upper-division required composition course in a disciplinary field relevant to the student’s major (e.g., Advanced Composition in the Social Sciences) and an upper-division designated writing-intensive course(s) in the major. In 2001, our State Council of Higher Education in Virginia (SCHEV) required all institutions to develop definitions of six specific learning competencies, one of which was writing, and plans for assessing them, with reporting to begin two years later. Each institution was allowed to develop its own assessment plan. The director of the Office of Institutional Assessment (OIA) consulted with me about how we might respond to this mandate so that we would be able to use the results of the assessment to improve the way writing is taught across the disciplines, not just to prove something about our students’ writing competence to an external audience.

The year before we received the 2001 mandate to assess writing, the OIA director and the WAC director had already begun to set in place a process for determining the effectiveness of our writing-intensive (WI) requirement in the major. As a first step, we asked the provost to convene the Writing Assessment Group (WAG), comprising representatives from each of the colleges, many of whom had served or were currently also serving on the senate-elected WAC committee. Our first WAG task was to design a survey, described in detail under Assessment Methods, which we circulated to all faculty to determine the number and kinds of writing tasks they assigned and their level of satisfaction with students’ performance on these tasks. Based on the results of this assessment and in response to the state mandate, we developed a second set of research questions related to students’ competence as writers in their majors.

To fulfill the state’s mandate, all institutions had to (1) submit a plan for assessing students’ writing competence, (2) include a definition of standards for writing competence, along with methods to be used to measure competence, and (3) report results to stakeholders, as well as actions that would be taken based on the results. Mason’s plan focused on the writing of upper-division students in the majors with assessment to be conducted by departmental faculty who would assess representative samples of student writing in the major according to a discipline-specific rubric they had developed. In addition to these departmental results, the proposal also noted that we would include data from the results of the faculty survey on student writing and responses to questions about writing from graduating senior and alumni surveys. Based on all of these findings, we would determine what changes and/or enhancements might need to be made in the WI course(s), to its role in the sequence of major courses, and/or in the faculty development workshops that are targeted to faculty teaching WI courses.

For purposes of reporting to the state higher education council, our writing assessment group decided to aggregate the results from all of the departments that had conducted assessment, so that individual departments would not be singled out for producing unsatisfactory numbers of less-than-competent writers. However, we asked departmental liaisons to write longer, more detailed reports on their assessment findings to be kept in the Office of Institutional Assessment and to be circulated to department members. In a concluding section of the longer reports, departments are asked to describe the actions they will take, as a result of their findings, to improve the way writing instruction is delivered in the major. The report to SCHEV can be found at http://research.schev.edu/corecompetencies/GMU/comp_writing.asp. Departmental reports are not publicly available; however, scoring rubrics are posted at http://wac.gmu.edu/program/assessing/phase4.html.

Assessment Methods

Faculty Survey on Student Writing For the first assessment measure in fall 2000, the Faculty Survey on Student Writing was distributed to all faculty, who were asked about student writing at different points along a continuum, such as the writing preparedness of first-year students and transfers and their level of satisfaction with the ability of seniors on 17 writing criteria. Faculty also noted the number and kinds of writing assignments they use in their undergraduate classes, as well as their perception of and interest in overall departmental support and resources for teaching with writing. While, as could be predicted, response to the survey was disappointingly low, a number of units (Biology, College of Nursing and Health Sciences, Computer Science, Electrical and Computer Engineering, English, New Century College, Public and International Affairs, School of Management) had initial response rates of 40 percent or higher. Some units subsequently readministered the survey and achieved higher response rates. A detailed description of the survey results can be found on page 3 of the InFocus newsletter at http://assessment.gmu.edu/Results/InFocus/2002/WritingAssessment.pdf.

Questions on Writing on Graduating Senior Survey Supplementing the information from the faculty survey are results on the writing questions asked each year on the Graduating Senior Survey. The 2006 senior survey included questions about students’ opportunities for revision and feedback in 300-level courses and above, and the effect of feedback on improving their writing, their confidence, and their understanding of their field. The results can be seen at http://assessment.gmu.edu/Results/GraduatingSenior/2006/index.cfm by selecting “Writing Experiences.”

Course-Embedded Holistic Assessment by Faculty in Majors Our current and ongoing assessment is embedded in required upper-division WI courses in the major. Every department offering undergraduate degrees is asked to appoint a liaison who organizes the assessment effort. The liaisons attend a cross-disciplinary workshop, which is designed to teach them methods for developing criteria and assessing papers holistically. The liaison then goes back to his or her department to lead a similar workshop using papers collected from writing-intensive or writing-infused courses. The following paragraphs give a fuller description of these workshops.

Cross-Disciplinary Training Workshops. For the cross-disciplinary training workshop, departmental liaisons read, discuss, and rank sample student papers written in sections of English 302, an advanced writing-in-the-disciplines course required of all students; the papers were written in response to a standardized assignment prompt for a literature review. After the sample papers have been ranked, the faculty go through the process of developing a scoring rubric based on criteria derived from their discussion of traits they valued in the papers. While the purpose of the cross-disciplinary workshop is to teach the liaisons the process to be used in the departmental workshops, the participants always leave with an awareness of how much their expectations may differ from those in other disciplines and even from members of their own disciplines; they also acquire a greater understanding of the challenges student writers face in meeting the expectations of teachers across disciplines. The WAC director leads these “training-the-liaison” workshops with the assistance of other composition faculty as available. She also leads or co-leads (with the designated liaison or another assessment group member) the half-day departmental workshops.

Departmental Assessment Workshops. Before the departmental scoring session, liaisons determine what assignment will be used to evaluate students’ competence. They are asked to select an assignment that requires students to demonstrate the skills and abilities most characteristic of those that writers should possess in the major. Papers written in response to the assignment or set of assignments are collected from all students with their names removed. Then papers are selected at random to provide a representative sample for scoring (the number of papers scored is based on a reliable percentage of the number of majors). Participants in the workshops are typically those faculty who most often teach the WI course(s) or teach with writing in most of their courses. As in the training workshop, they read and discuss three or four sample papers as a group, articulate traits they value in each of the papers, rank the papers, and, finally, develop a rubric with criteria that reflect the traits they’ve listed. Thus the criteria and the scoring rubric are not only discipline-specific but also specific to courses and assignments.

Using this rubric, faculty score the papers. Each of the papers being assessed gets two readings and a third if the first two overall scores do not agree. Because overall scores can be difficult to determine if there is a spread of scores over individual criteria, faculty, as a group, must decide how they will determine overall competence when some criteria may be assessed as “less-than-satisfactory.” Some groups have decided that any paper receiving a “less-than-satisfactory” on the top one or two criteria must receive an overall “less-than-satisfactory” score. The School of Management decided, for example, that papers assessed as “not competent” in the category of “Formatting and Sentence-Level Concerns” must receive an overall score of “not competent.” Biology faculty agreed that any paper receiving an “unacceptable” rating on “Demonstrates Understanding of Scientific Reasoning” must be judged as “unacceptable” overall. (Note: Departments decide on the language they will use to describe the level of competence, e.g., “less than satisfactory,” “not competent,” “unacceptable.”)

Once the scoring has been completed, the departmental liaison is responsible for analyzing the distribution of scores overall and on each criteria and for writing a report on the results to be circulated to the department and sent to OIA. While an analysis of the overall scores on the rubrics gives departments a general picture of students’ writing competence in the major, it is the analysis of the scores for each of the criteria that is most instructive for the purposes of faculty development, i.e., developing teaching strategies and assignments targeted to those areas in which papers were judged to be weak. As explained below, the assessment results also help departments make decisions about where writing is best placed in the curriculum. A more detailed explanation of our assessment process is available on our WAC site at http://wac.gmu.edu/program/assessing/phase4.html/, as are a number of rubrics developed by the departments that have conducted assessment.

Assessment Principles

We view assessment as part of an overall philosophy about education that states that good assessment—its methods, practices, and results—can be used to correct, change, and enhance the learning experience for our students. Central to our assessment process is the belief that faculty own the curriculum and, further, that program faculty must share a sense of direction and purpose to establish a coherent learning experience for students—in this case, a coherent writing experience in the major. When writing assessment is embedded in writing-intensive courses in the major and when faculty buy into the process, both the process and the results contribute to the development of teachers, to their greater understanding of student writers, and to the effectiveness of the writing instruction in their classes.

Our assessment principles and decisions are also guided by composition and writing-in-the-disciplines research and theory, including Cooper and Odell’s 1977 collection Evaluating Writing: Describing, Measuring, Judging, which describes and provides a rationale for holistic scoring,and Huot’s 2002 (Re)Articulating Writing Assessment for Teaching and Learning, which argues that assessment should be site-based and locally controlled, that writing professionals should lead these efforts, and that our practices should be theoretically grounded, practical, and politically aware. Our process is also informed by genre and activity theory, which accounts for the fact that there are significant disagreements among faculty across and in the same disciplines about what constitutes competent writing. A fuller listing of sources is included at the end of this document.

Assessment Results

It would be difficult to sum up in a brief statement all that we have learned from our assessment efforts. The rubrics that departmental faculty develop as a result of our holistic reading and scoring process reveal widely varied expectations for student writing, based in the discipline but also on faculty members’ sense of the writing that is appropriate for undergraduates in their disciplines. Some results can be found on the website pages listed above. I and coauthor Chris Thaiss also discuss assessment results in Engaged Writers and Dynamic Disciplines: Research on the Academic Writing Life.

One of the most significant things faculty discover as part of the workshop scoring process is that they may not agree with one another on what “good” writing or what “serious” error looks like. While they may start from a position that surface errors are the strongest indicator that students “can’t write,” they see, as a result of collaboratively constructing a scoring rubric, that students’ performance on other higher-order criteria (clear argument, focused thesis, logical evidence, etc.) might be better indicators of students’ ability to write well in the discipline. Faculty can also see how flaws in their assignments might contribute to students’ less-than-successful performance. The subsequent analysis of the scoring results is also useful in helping faculty create more effective assignments, make decisions about appropriate assignments, decide on the best sequence for assignments, and/or improve their teaching-with-writing practices in the areas indicated by the assessment. Further, the reports are also useful for departments in determining appropriate course sequences and whether the current designated WI course is the most appropriate for the major. A more specific discussion of how the assessment results are being used by departments can be found in the InFocus newsletters at http://assessment.gmu.edu/Results/InFocus/2007/CompetenciesSummary_FINAL.pdf .

Assessment Follow-Up Activities

In addition, the Southern Association of Colleges and Schools (SACS) requires the assessment of learning outcomes for every academic program, including general education, for accreditation purposes. The writing assessment we have been doing contributes to this report, with each individual unit discussing the results of its assessment of writing in the major and follow-up actions it will take. The university will also include the assessment of writing as part of our larger assessment of general education for the SACS’s review.

State Council of Higher Education in Virginia has recently mandated that Virginia institutions include a “value-added” component to our assessment plan. We will build on our current plan by adding a preassessment of students’ writing competence at the completion of first-year composition (FYC), using a random and representative sample of research-based essays. Faculty who teach the course will participate in a scoring workshop, in which they first develop a rubric to specify standards and then blind-rate the papers. In addition to providing comparison data for the postassessment that occurs in the WI courses, the results should also allow us to begin assessing our required English 302 advanced writing-in-the-disciplines course.

Assessment Resources

Departmental liaisons are given a very small stipend and a free lunch for participating in the cross-disciplinary training workshops. In some departmental workshops, faculty are given lunch and, if funding is available, a small stipend. In 2004 the provost funded a university-wide reception to recognize faculty for their assessment efforts. Posters describing each department’s assessment procedures, rubrics, and results were created for the reception and subsequently displayed, at the request of our university president, at a meeting of the Board of Visitors. Some posters were also displayed in the bookstore and in departments. Some of the posters can be viewed online at http://wac.gmu.edu/program/assessing/powerpoint.html. Other than this recognition and some small compensation for term and adjunct faculty who participate in scoring, there are no incentives; we must rely on the goodwill of full-time faculty and their commitment to student learning.

The WAC director co-chairs the assessment initiative with the OIA director as part of her responsibilities, not because this is part of the job description but because of what the WAC program gains from participating in the process. The assessment workshop is a valuable faculty development opportunity, and both the process and the resulting data provide the director with a valuable perspective on writing in the disciplines across the university, which, in turn, informs ongoing WAC program and faculty development efforts.

Assessment Design Sustainability and Adaptability

Our assessment efforts are sustainable up to a certain point. A joint WAC-OIA position has been approved for the next fiscal year for an assistant to help with both writing assessment and the WAC program. However, we still need more resources to enable us to recognize the efforts of those faculty who have participated and to provide incentives to encourage more faculty to participate.

Our process is adaptable, as proven by departments using the methods for their own ends, e.g., the Department of Communication using a holistic method to develop a rubric for faculty, mostly adjunct, to use in grading papers from lower-division general education and majors courses; departments also find the process useful for calibrating teachers’ reading and evaluation practices. Our School of Management is using the process to develop writing outcomes for their majors and also to measure growth in writing from the gateway to the capstone course.

The frequent queries we receive from program leaders across the country about our assessment process is evidence of the adaptability of our assessment design to other programs. Indeed, our program has been referred to as “the Mason Model” by some of the WAC and assessment people who frequently contact our program.

Useful References

Bazerman, Charles, and David R. Russell. Writing Selves/Writing Societies: Research from Activity Perspectives. Perspectives on Writing. Fort Collins, CO: The WAC Clearinghouse and Mind, Culture, and Activity. 2002. 11 June 2008 <http://wac.colostate.edu/books/selves_societies/>. Cooper, Charles R. “Holistic Evaluation of Writing.” Evaluating Writing: Describing, Measuring, Judging. Urbana, IL: National Council of Teachers of English, 1977. 3–32. Cooper, Charles R., and Lee Odell, eds. (Eds). Evaluating Writing: Describing, Measuring, Judging. Urbana, IL: National Council of Teachers of English, 1977 Haswell, Richard, and Susan McLeod. “WAC Assessment and Internal Audiences: A Dialogue.” Assessing Writing Across the Curriculum: Diverse Approaches and Practices. Ed. Kathleen Blake Yancey and Brian Huot. Greenwich, CT: Ablex, 1997. Huot, Brian. (Re)Articulating Writing Assessment for Teaching and Learning. Logan: Utah State UP, 2002. Miller, Carolyn R. “Genre as Social Action.” Quarterly Journal of Speech 70.2 (1984): 151–67. Russell, David R. “Rethinking Genre in School and Society: An Activity Theory Analysis.” Written Communication 14.4 (1997): 504–54. Accessed online. Thaiss, Christopher, and Terry Myers Zawacki. Engaged Writers and Dynamic Disciplines: Research on the Academic Writing Life. Portsmouth, NH: Boynton/Cook, 2006. Walvoord, Barbara E. Assessment Clear and Simple: A Practical Guide for Institutions, Departments, and General Education. San Francisco: Jossey-Bass, 2004. White, Edward M. Teaching and Assessing Writing: Recent Advances in Understanding, Evaluating, and Improving Student Performance. San Francisco: Jossey-Bass, 1994. Yancey, Kathleen Blake, and Brian Huot, eds. Assessing Writing Across the Curriculum: Diverse Approaches and Practices. Greenwich, CT: Ablex, 1997.

Assessment Narrative - Saint Joseph College

Assessment Narrative - Saint Joseph College

Institution: Saint Joseph College, Connecticut Type of Writing Program: Across the Curriculum Cumulative Portfolio (required) Contact Information: Dr. Judy Arzt (Director) jarzt@sjc.edu (860) 231-5353

Dr. Kristine Barnett (Writing Portfolio Coordinator) kebarnett@sjc.edu (860) 231-5472

Assessment Background and Research Question

In 1988, Saint Joseph College undertook a study of college-wide writing assessment practices used nationally and internationally with the goal of implementing an assessment complementary to its culture. Although for over 50 years our college catalog indicated that students must achieve writing competency to graduate, the faculty questioned whether measures used in the past followed best practices. For instance, as late as the 1980s, a junior-rising exam was used, but this method was contrary to theories set forth by popular writing theorists such as Peter Elbow and Donald Murray. Our faculty members were disenchanted with a one-shot essay writing sample, and although one was administered to all juniors in the spring of 1987, members of the English department soon agreed the method was not suitable and stored the essays in a basement, unread. With the hiring of a director to oversee writing programs, a full-scale search ensued for an effective assessment tool.

The director, along with an assessment committee, commenced a yearlong study to compile assessment information, and resources were gathered from the Conference on College Composition and Communication (CCCC), American Association for Higher Education (AAHE), and National Testing Network. New publications in the field of assessment were further consulted. In the end, we favored a longitudinal portfolio.

The questions we sought to answer through a portfolio assessment focused on how students develop as writers over time and how well our curriculum helps shape students as writers. As we sought to define the portfolio program for compatibility with the college mission, faculty were a guiding force in the decision-making process. Faculty attended a series of workshops to become better acquainted with the principles of Writing across the Curriculum (WAC). During the 1989 spring term at a one-day WAC workshop, faculty examined model assignment sheets and reviewed individual students’ writing assignments and due dates for a single semester. These activities helped faculty see writing (amounts and expectations) from a student perspective, as well as understand that writing skills develop over time and vary with genre and discipline. Following the workshop, at the end of the spring term, a WAC consultant was invited to spearhead a two-day, all-faculty retreat focused on encouraging faculty commitment to writing-to-learn activities, fostering the perception of the teaching of writing as a joint faculty partnership, and promoting recognition of the cumulative developmental process of writers. In summary, both the retreat and the workshop created a climate receptive to a longitudinal, cross-disciplinary portfolio program as a means to assess student writing. The following fall, a faculty assessment committee advanced a proposal for a portfolio program to the faculty for a vote, and the proposal received unanimous approval.

We envisioned the portfolio program as a means to assess student writing as well as the college’s new core curriculum. Annually, students were required to submit one paper from a core course and an additional paper from another course. All core courses were writing intensive, and faculty designed common syllabi and assignments. However, portfolios revealed that students’ strongest writing did not derive from the core classes, but from disciplinary courses that held the students’ interest, often courses in their major or closely allied disciplines. Not surprisingly, whereas the portfolio program survived the test of time, the core curriculum did not. In 1995 we undertook an assessment of the core program and surveyed seniors who had been through a four-year cycle of the courses. The survey results led faculty to abandon the core courses but not the portfolio, for which there was strong support.

In summary, the research questions guiding the formulation of our portfolio included:

What are the characteristics of student writing on our college campus?
How do students develop as writers over time?
How well do our students succeed in meeting a writing competency requirement for graduation?
How can portfolios inform and support curricular and instructional decisions?
What does a cross-disciplinary portfolio tell us about writers and their rhetorical strategies?

Assessment Methods

Portfolios are evaluated using three techniques. The written commentary section is the most labor intensive but the most helpful to students’ writing development. A criteria checklist and a holistic score complete the evaluative process. (A sample score sheet can be found in the appendix.) The criteria for evaluating portfolios are (1) fluidity and clarity of expression, (2) effective organizational skills, (3) effective use of details and elaboration, (4) critical thinking skills, (5) effective research writing skills, (6) effective use of language and diction, and (7) mechanics and usage. A student’s portfolio is rated in each of these areas using a plus, check, or minus.

Papers are evaluated holistically and more weight is given to recent work to reflect the student’s progress. A 5-point scoring system is used, with 5 as the high score. A minimum score of 3 is required to complete the process. The sophomore-year evaluation serves as formative assessment, and a lengthy commentary educates students about the criteria and the fit of their writing with each of the seven criteria. Usually a paragraph or two explains each rating, and evaluations conclude with a bulleted list of tangible suggestions for writing improvement. The junior-year final evaluation follows the same process, except summative comments are kept to two paragraphs, as students are completing the process and receive acknowledgment for accomplishments to date, though recommendations for the future are included.

The process of completing preliminary evaluations, as noted, is complex and time consuming for evaluators, but students report that the feedback is valuable. In the late 1990s, we conducted exit interviews with graduating seniors to assess their response to the program. Most highlighted the advantages of the comprehensive preliminary evaluation, which has remained a distinctive feature of our program.

The actual scoring process is not particularly difficult in terms of reaching a consensus between two readers. At the preliminary stage, one member of the writing center staff writes a comprehensive evaluation. The second reader responds to this evaluation and offers suggestions for additions and changes. Ultimately, the two readers must agree on all aspects of the commentary as well as the ratings and score. At the final stage, two faculty members evaluate a portfolio independently. The two then meet to synthesize findings. Ordinarily, there is strong congruence, and in the two decades in which the program has been in place, only on the rarest of occasions has there been a discrepancy. In such cases, a third reader resolves the difference. In all cases where two independent readers score a portfolio “below satisfactory” or “poor,” two additional readers must read the portfolio independently. Thus, for a portfolio to receive an unsatisfactory score, four faculty members must agree on that score. On the other hand, students who receive the top score of “5” are commended through awards and the transcript notation of “Writing Portfolio Completed with Distinction,” as opposed to the regular designation of “Writing Portfolio Completed.”

The portfolio process has remained fairly consistent since its inception and is well articulated for students, faculty, and academic advisors. Our “Writing Portfolio Booklet” explains the process, and annual portfolio reports, enumerating results and recommendations, keep the college community abreast of the program and student progress. Class visits and presentations at orientations and other events further serve as communication channels.

Assessment Principles

The portfolio program is based on assessment practices that mirror National Council of Teachers of English (NCTE) and CCCC position statements on writing assessment. Principles that have guided our practices include:

Writing skills develop over time and a writer’s rhetorical strategies shift to accommodate context and audience. To understand a writer’s full range of skills, a collection of samples written in different genres and for different purposes is essential. This principle reflects the CCCC Position Statement: “Best assessment practice uses multiple measures.”

Evaluation of student writing skills needs to reflect the curriculum as well as inform curricular practice. To that end, students submit a sample of papers from a variety of courses. The results of student outcomes on portfolios are annually communicated to the college community and used to make curriculum decisions and shape classroom teaching practices. We believe that timely communication of results is a critical part of the process and keeps the college community informed of the link between the curriculum and student performance.

For assessment to truly benefit students, they need timely review and valued response to their work. Our system provides students with detailed feedback on their growth as writers, and students are invited to confer with evaluators, advisors, faculty, and writing center staff regarding writing progress, portfolio submissions, and evaluations. We see the portfolio process as fostering a community of learners and as a means for promoting conversations about writing on campus.

Writing assessment needs to value teachers’ classroom work. Faculty members are an integral component of the process, and they guide the program. As such, the program is sensitive to faculty response, and all faculty members are invited to be portfolio readers. Training is provided to new evaluators, and faculty reading days are considered a faculty development activity, rewarded in the tenure and promotion process, as well as through monetary compensation. Faculty members have found that reading days provide them with an opportunity to confer with colleagues on assignment design, methods of response to student writing, and the overall college curriculum.

In addition to the specific guidelines set forth by NCTE and CCCC, scholarship in the field of writing assessment, including work done by Edward White, Kathleen Blake Yancey, Brian Huot, Chris Anson, Richard Haswell, and Nancy Sommers, has informed our work.

Assessment Results and Follow-Up Activities

Assessment results are reported directly to students via a written evaluation. In addition to the score sheet and narrative, students are encouraged to meet with evaluators and writing center staff, as well as advisors, to discuss portfolio results and writing skills in general. In the past, students have been surveyed to capture their perceptions regarding the portfolio process. Findings revealed overwhelming support for the continuance of the program and the evaluative techniques, particularly the commentaries.

Faculty advisors are considered essential to the success of the program. In fact, all advisors receive a duplicate set of students’ papers and evaluations. Advisors meet with students to discuss progress and set goals. In addition, faculty are attuned to using the program to evaluate the curriculum, and, of late, plans are underway to use portfolios to assess the college’s new general education curriculum. A representative sample of portfolios will be used to assess how well the new curriculum addresses critical thinking skills.

Collective program results are communicated through an annual portfolio report disseminated to members of the college community. These reports not only give results of student performance but also include recommendations for curricular reform and areas of focus for classroom instruction. For instance, annual reports have led to increased attention to teaching research skills, as portfolio results evidenced a significant number of students struggled in this area.

In terms of informing the greater community, we have made presentations on our program at national and regional conferences, including the American Association for Higher Education, CCCC, Northeast Writing Centers Association, International Writing Centers Association Conference, and New England Assessment Network. The program is listed in AAHE assessment publications. Routinely, we submit proposals for conferences and make presentations at other campuses interested in implementing a writing assessment program. In addition, faculty from other colleges have attended our portfolio faculty reading days to observe the evaluative process.

Assessment Resources and Transferability

The writing portfolio is integrated into the services provided through the Center for Academic Excellence (CAE). Thus, the only additional funding required is compensation for faculty who read portfolios, and the per diem rate for a full reading day is $300. We also compensate faculty who occasionally read portfolios outside of reading days to expedite the evaluations, and these faculty are paid $30 per portfolio. Faculty readers who periodically evaluate portfolios throughout the year are still expected to confer with a second reader, either in person or online.

The program is easily transferable to other institutions, and Springfield College, in Springfield, Massachusetts, has already adopted some aspects of our program to assess student writing in specific programs. The evaluative criteria we use reflect research and scholarship on writing assessment in composition and rhetoric, follow common standards for assessing writing based on that work, and have also been adapted from other assessment processes. An examination of portfolio programs at other institutions revealed use of similar criteria, developed after those institutions established that these criteria were also appropriate for their contexts. The training program is also easy to implement, and the reading days, which are an excellent faculty development activity, are comparable to similar portfolio reading days or faculty workshops held at other institutions.

The program has thrived since 1989 and has continued to be supported by students, faculty, and administration. Despite the program’s solid foundation and successful track record, the college continues to streamline the program to ensure sustainability.

Appendix

SAINT JOSEPH COLLEGE WEST HARTFORD, CONNECTICUT

WRITING PORTFOLIO SCORE SHEET

STUDENT’S NAME:

SCORING SCALE:

Writing portfolios are scored holistically on a scale of 5 to 1. A score of 3 or higher is needed to fulfill graduation requirements. The score scale is as follows:

5 = excellent 4 = good 3 = satisfactory 2 = below satisfactory 1 = poor Inc = Incomplete (work is missing)

SCORE:

CRITERIA LIST:

A plus mark in front of an area indicates strength; a check mark indicates an area is satisfactory; a minus mark indicates an area in need of improvement.

_________ 1. fluidity and clarity of expression _________ 2. use of appropriate organizational structure _________ 3. sufficient use of details and elaboration _________ 4. critical thinking skills _________ 5. effective and correct use of research techniques _________ 6. effective use of language and diction _________ 7. correct mechanics and usage

COMMENTS:

Readers: Date:

Resources

Arzt, Judy. “Electronic Portfolios’ Transformative Effects on Assessment.” Paper presented at Conference on College Composition and Communication Convention. New York City. 21 Mar. 2003. <http://www.sjc.edu/jarzt/C4draft.htm>. Arzt, Judy. “Writing Portfolio as an Exit Requirement.” Reviews and Descriptions of Assessment Instruments. Knoxville, TN: Clearinghouse for Higher Education Assessment Instruments, 1994. “Portfolio Project.” Center for Academic Excellence, Saint Joseph College, CT. <http://www.sjc.edu/content.cfm/pageid/4277>.

Assessment Narrative - Salt Lake Community College Community Writing Center

Assessment Narrative - Salt Lake Community College Community Writing Center

Institution: Salt Lake Community College Community Writing Center Type of Writing Program: Community Writing Center Contact Information: Tiffany Rousculp, Director SLCC Community Writing Center 210 E. 400 South, Suite 8, Salt Lake City, UT 84111 t.rousculp@slcc.edu (801) 957-4992 www.slcc.edu/cwc

Assessment Background and Research Question

The Community Writing Center (CWC) is an outreach site of Salt Lake Community College (SLCC), a drop-in writing center for all members of the Salt Lake community. The mission of the CWC is to support, motivate, and educate people of all abilities and educational backgrounds who use writing for practical needs, civic engagement, and personal expression. The CWC combines pedagogical strategies of academic writing centers with the responsive nature common to nonprofit organizations. Participants in CWC programs are self-motivated to develop their literacy abilities and can do so through one or more of our four programs: Writing Coaching, Writing Workshops, Writing Partners, and the DiverseCity Writing Series.

Writing Coaching provides free, one-on-one individual assistance on any writing task—résumés to poetry to letters—in a supportive mentoring environment. Writing Workshops introduce community members to a variety of writing genres in low-cost, low-stakes, short-term, small-group workshops. The CWC collaborates with nonprofit educational and/or governmental organizations through the Writing Partners program. These partnerships include workshops for staff/clients, mentoring for writing group development, and shared promotion of literacy/writing resources throughout the Salt Lake valley. Finally, the DiverseCity Writing Series is a multigroup, year-round writing group and publishing program that engages community members in personal expression and shared celebration of the written word.

The CWC is directed by a faculty member from the SLCC English department and is staffed by five part-time writing assistants (students from the college and the local university). Recently, the CWC added an assistant director (an English department faculty member with 50 percent reassigned time) to its staffing. The CWC opened in October 2001 and is located on the plaza of the Salt Lake City Main Library in downtown Salt Lake City. To date, the CWC has served over 2,000 community members and has partnered with nearly 80 community organizations.

The CWC does not generate income through number of students served and thus must continually justify the budget expenditures for its programming to SLCC administration and trustees. Although the CWC receives excellent support from SLCC administrators, our budget is tenuous when compared with programs at the college that deal only with registered students. Therefore, it is imperative for the CWC to anticipate—and implement without prompting—the type of assessment that administrators and trustees value. To address this need, and to shape the direction and work of the SLCC Community Writing Center, center staff has engaged in two main types of assessment since our inception:

User satisfaction surveys on each of our four programs that try to address the question, “How satisfied are community writers with the services that the CWC provides?”
Work Plans (reestablished every two years) to assess the use and achievements of the CWC, specifically addressing the question, “Who uses the CWC and what resources are they using?”

These ongoing assessment practices generate mostly quantitative data targeted toward our funding sources (SLCC administration and Boards of Trustees and Regents).

In 2006, after five years of conducting assessment through our Work Plan process, the CWC was well supported by SLCC administrators, yet we believed an external review would enhance our credibility as a well-theorized and responsive program with a new president who arrived at the college that year. As a site of learning, our primary concern was how to establish an assessment process that moved beyond the quantitative findings of our Work Plan and into a deeper examination of the learning (or not learning) that our writers experienced. We had not been able to track individual literacy development in our Writing Coaching programs, and the responses to our public workshop evaluations—though always positive—did not adequately assess acquisition of new or developing writing abilities. As a first step in that process, we invited Dr. Eli Goldblatt from Temple University to conduct an External Review of our programs in the spring of 2007. In that review, we asked Dr. Goldblatt to examine whether the CWC appeared to be fulfilling its mission and to make recommendations on what assessment strategies we might pursue to more thoroughly evaluate our programs.

Assessment Methods

The user satisfaction surveys have not—at this point—provided sufficiently useful data on our programs to merit inclusion in this narrative. (Two sample surveys are provided in the appendix for informational purposes.) We focus instead on the Work Plans and the External Review, which have provided useful information.

Work Plans Every two years, the CWC director, assistant director, and writing assistants come together in a retreat to establish objectives for the next two years. Before proposing objectives, the CWC staff analyzes the “successes and failures” from the current work plan according to the criteria set forth for each objective. For example, we examine whether we have maintained adequate diversity of demographics in our writer population, or we analyze whether the partnerships we have maintained prioritize underserved populations. During this retreat, we collaboratively prioritize center-wide goals, such as “increase diversity of writers,” “establish connections with K–12 institutions,” and/or “broaden reach of community publications.” While the director and assistant director guide this process, all staff members contribute valuable and insightful ideas to the goal-setting process.

After the retreat, the director and assistant director group the center-wide goals into program areas (e.g., Writing Coaching, Workshops) and priority areas (e.g., Diversity, Fundraising). These goals are revised into measurable objectives with deadlines and aligned with SLCC goals and objectives, which group into four main areas: “Provide Quality Education,” “Provide Lifelong Learning,” “Serve People of Diverse Cultures, Abilities, and Ages,” and “Serve the Needs of Community and Government Agencies, Business, and Industry.” The draft Work Plan is returned to the CWC staff members for feedback, revision, and approval. The Work Plan is then distributed to the CWC’s Advisory Committee for review. Any suggested changes are brought back to the CWC staff for consideration, revision, and group consensus.

Upon completion, the Work Plan is distributed to the SLCC administration for final approval. (Since our opening, no recommendations for change to previous Work Plans have been made by upper administration. Should such recommendations be made, the CWC director would negotiate with administration to balance their requests with staff preferences in order to maintain the stability of the CWC in both financial and staffing terms.)

After final approval, the Work Plan is used by the CWC director and staff to evaluate progress toward the goals on a monthly, quarterly, and annual basis. The Work Plan is revisited each month in one-on-one meetings between the director and staff members through assessment of individual staff performance. Staff members set “semester work goals” based on the Work Plan at the beginning of each semester. Significant portions of the Work Plan (numbers of new writers, workshops offered, partnerships) are assessed quarterly for the SLCC English department and for the immediate administrative supervisor overseeing the CWC, the dean of the School of Humanities and Social Sciences. The entire Work Plan is assessed annually for SLCC upper administration and the Boards of Trustees and Regents, as well as for public dissemination on the CWC website (www.slcc.edu/cwc). (See selections from the 2006–2008 Work Plan in the appendix.)

External Review In the spring of 2006, the English department chair approached the CWC director with a proposal to conduct an external review of the Community Writing Center’s programs. The chair, Stephen Ruffus, had spoken with Dr. Goldblatt at the 2006 Conference on College Composition and Communication Convention after a panel that Dr. Goldblatt shared with the CWC director and other community literacy scholars and activists. Over the summer of 2006, we shared emails with Dr. Goldblatt to narrow down our research questions and flesh out the scope of what could be accomplished in a brief external review.

In February 2007, the CWC director compiled a portfolio of CWC documents to present a broad picture of CWC programming successes and challenges. Dr. Goldblatt received the portfolio approximately two weeks prior to his visit. The portfolio included (among other items):

Quantitative Data

Previous Work Plans and Their Assessments
Lists of Community Partners
Demographics of Writers Who Utilize CWC Services
Budget Analyses

Narrative/Program Data

Evolution (including revisions) of CWC Mission Statement
Brief History of the CWC
Selected Reports from Staff Meetings
Selections from Writing Assistant Training Manual
Procedures for Workshop Management

Assessment Samples

Workshop User Satisfaction Surveys
Writing Coaching Comment Cards
Writing Partner Satisfaction Surveys
“Success Stories” (letters, emails, comments from partners and individuals)

Artifacts

Publications from the DiverseCity Writing Series
Sample CWC Newsletters
Bibliography of Publications/Presentations from CWC Staff Members

In early March 2007, Dr. Goldblatt arrived in Salt Lake City for a two-day visit. On the first day, he met with the CWC director, who gave him a brief bird’s-eye view tour of the Salt Lake valley (from the foothills overlooking the city) to establish a sense of place for his review. After talking with the director for the morning, Dr. Goldblatt met with current and previous writing assistants for lunch and discussion of how the CWC impacts students who work there. Following lunch, he met with representatives from a few community partners that the CWC had collaborated with during the past year. In each of these meetings, the director was not present to influence conversation. During dinner, Dr. Goldblatt met with the current DiverseCity Writing Series coordinator (a student writing assistant) and the two previous DWS coordinators. We made this a priority due to Dr. Goldblatt’s expertise in community publishing projects. Next, Dr. Goldblatt was scheduled to attend a DiverseCity Writing Series writing group meeting, but due to miscommunication, did not meet up with the group. Instead, he continued his discussion of community publishing with the DWS coordinator.

In order to get a sense of the college’s relationship with the Community Writing Center, the next morning Dr. Goldblatt met with the SLCC English department chair and the SLCC student writing center coordinator. Lunch followed, with a meeting including the aforementioned, as well as the academic vice president, associate academic vice president, and CWC director. Then, on a request from Dr. Goldblatt, the CWC director arranged a brief meeting with the SLCC president to congratulate her on the center (thus bolstering the CWC in her mind). Next, Dr. Goldblatt met with a few members of the CWC Advisory Committee, followed by dinner with SLCC English department faculty and attendance at a writing workshop in the evening.

Approximately two months after the visit, Dr. Goldblatt sent a letter of review to SLCC administration with his findings.

Assessment Results and Follow-Up Activities

Work Plans Since the CWC Work Plans are a continual assessment process, findings from each Work Plan inform CWC programming throughout the year as described earlier. The Work Plan is discussed at each staff meeting as a way to consider and prioritize efforts and resources for programming, at quarterly English department meetings, and in one-on-one meetings with the dean of the School of Humanities (the new institutional home of the CWC). The findings are also distributed to the CWC advisory committees and SLCC upper administration on a yearly basis.

External Review The External Review produced challenging and useful findings for the CWC and SLCC administrators. The primary findings, which were delineated in a letter to SLCC administrators (the entire letter can be found on the CWC’s assessment website; see address below in the appendix), included successes and suggestions.

The successes included:

Establishing an environment in which a wide range of writers unaffiliated with SLCC or other nonprofit academic institutions could pursue their own projects and receive criticism, support, and access to publication they would have no other means to obtain.

Training, mentoring, and launching student writing assistants on career paths connected to literacy and community engagement.

Focusing attention and material resources on the literacy needs of low-income, marginalized, or otherwise underserved learners in many of the neighborhoods of the city.

Modeling a democratic educational approach, one that is not motivated primarily by competition, not measured by grades or standardized tests, and not limited to groups segregated by age, race, ethnicity, religion, sexual orientation, or socioeconomic status.

Offering literacy instruction in a uniquely interdisciplinary and multidimensional context that truly integrates writing, reading, listening, and speaking abilities whose purposes are practical, civic, and social.

Indicating the possibilities of collaboration between and among large and small local institutions such as youth services, the public library, refugee aid groups, agencies preventing domestic violence, the University of Utah, and theater companies for the enrichment of literacy-related activities in the civic arena.

This was followed by suggestions, which can be summarized as:

Enhanced effort for financial development.

Increased regional connectivity and circulation.

Strengthened internal college alliances.

Focused research as a theme.

The CWC staff, while buoyed by the successes, focused on the suggestions as we made some immediate changes over the summer of 2007. Specifically, follow-up activities based on the External Review have been:

Dismantling the existing Advisory Committee, which was made up of community members and SLCC faculty and staff, and creating a new one more representative of our constituency and better suited to the CWC’s needs. The new advisory structure consists of two advisory committees, the Community Advisory Committee (CAC) and the Academic Advisory Committee (AAC). The CAC focuses on fundraising, community exposure, and volunteer recruitment. The AAC will be responsible for increasing student involvement in CWC programming, research, and scholarship opportunities and, primarily, implementing useful strategies for assessing the impact of our programs on the writing skills/abilities of our community users.

The CWC moved its institutional home from the associate academic vice president’s office to that of the dean of the School of Humanities and Social Sciences. This has put the CWC director in close contact with department and division chairs in that school, which should improve opportunities for student involvement. Also, this move has provided additional support for the CWC director to engage in research and publication work regarding the CWC.

The SLCC president has committed to providing more support for the CWC in our fundraising endeavors. While resources are limited at the college, contributions have been made when possible. For example, a local citywide newspaper donated 12 pages of space to the college to use for promotional purposes. The college’s president has given half of those pages to the CWC for a feature article on our programs and writers.

In addition, the External Review has informed our Work Plan process as we develop our newest Work Plan, which will span 2008–2010. As we have begun work on it, we are prioritizing the need to develop valid assessment instruments of the learning that we assume takes place through the CWC programs. However, as the NCTE-WPA white paper on writing assessment points out in its principles, writing assessment characterizes writing as complex, inherently social, and context driven. In first-year composition programs, this principle challenges assessment practices; in a community writing center context, with multiple persons, purposes, and genres, all outside of an institutional setting, such assessment can appear quixotic at best. However, this will be a priority for the new Academic Advisory Committee as we move into the next two years of the Community Writing Center.

Assessment Principles

Because we anticipated a need for external funding, in 1999—prior to the establishment of the CWC—the future CWC director worked with the SLCC Development Office on documents found in grant-writing and fundraising genres. These documents, which spoke to SLCC administrators in their favored discourses, became effective proposals for funding by the administration. The rhetorical strategies used in these proposals were then utilized in the CWC Work Plan assessment process, as the primary audiences remained SLCC administration and other external funding sources.

The Work Plan documents and the External Review are—and have been—based on the following principles of writing assessment, though perhaps not fully articulated as such at the time:

Writing assessment is based on continuous conversations with as many stakeholders as possible. The CWC is founded on principles of collaboration between the college and the community and challenges the assumption “that higher education can know what a community needs or wants without entering into full and mutually-beneficial partnership with that community” (“Founding Principles”). The Work Plan is a collaborative document, with voices from all among the CWC staff, community members, advisory committees, and SLCC administration. The External Review provided Dr. Goldblatt with open access to each stakeholder in the CWC’s spheres of contact: community, staff, and academy.

Writing assessment articulates and communicates clearly its values and expectations to all stakeholders. Both the Work Plans and the results of the External Review have been disseminated widely through various means. CWC staff and SLCC faculty/administrators are continually provided with reports on the Work Plan, and each was provided with copies of the External Review report. Findings from the Work Plan are incorporated into CWC public documents, such as our newsletter and features in local newspapers. Both assessments are also available on the CWC’s website: www.slcc.edu/cwc.

Writing assessment is site-specific. Each assessment tool, including the Work Plans and the External Review, that the CWC has utilized in its six years of operation has been locally developed by CWC staff members. Input is gathered from multiple local stakeholders and is reported within the context of the Salt Lake community. For example, comparisons of income, ethnicity, and education among CWC writers are done with Salt Lake City, county, and SLCC populations.

Writing assessment uses multiple measures and engages multiple perspectives to make decisions and provide formative feedback. One of the main reasons the CWC undertook the External Review was to get a new perspective on our programming. The Work Plans demonstrated “success” to many of our audiences, but their limited perspectives necessitated an outsider’s eye on the center.

Writing assessment is ongoing, providing the foundation for data-driven, or evidence-based, decision making. The CWC Work Plan is continual, always providing a rubric both for hard data and for critical reflection on our programs. The External Review is not an ongoing assessment practice, but we plan to conduct another external review in four years’ time.

Writing assessment places priority on improvement of teaching and learning. The Work Plan assesses programs, not the writing abilities of the CWC’s community users. This type of assessment, along with the aforementioned user satisfaction surveys, both point to improving our services rather than improving competence or writing skills. It is terribly difficult to measure the writing skills/abilities of community members in such a wide variety of contexts. As such, CWC assessment practices have solely engaged with asking how we can improve our offerings of literacy instruction to the Salt Lake community.

Assessment Resources

Work Plan The Work Plan requires time to create and assess on an ongoing basis. Initially, there is a significant time commitment (approximately 10 hours) for retreats with staff and meetings with stakeholders. Year-round, it takes time to meet with CWC staff members and to collect information on accomplishments and challenges that lie within the Work Plan. However, those conversations are essential to the collaborative principles of the CWC. In addition, the Work Plan provides credibility to the CWC in the eyes of the SLCC administrators who fund us.

External Review The total cost for the CWC’s external review was just under $1,500. That included the flight and hotel for Dr. Goldblatt, meals for Dr. Goldblatt and the stakeholders he met with, and a stipend. Personnel commitments included meetings with CWC staff members, college faculty, and administrators on the two days of his visit. Planning was the purview of the CWC director, who compiled the evaluation portfolio and organized the schedule of meetings/observations with the various stakeholders and communicated with Dr. Goldblatt prior to his arrival. The English department administrative assistant helped in purchasing the flight ticket and reserving the hotel room.

Sustainability/Adaptability

The Work Plan design is certainly sustainable; in fact, it is necessary to the CWC’s own sustainability. The External Review is, by nature, not a sustainable assessment process. However, we do plan to conduct another external review similar to this one in our tenth year of operation.

Appendix

SLCC Community Writing Center Assessment Website This website includes the External Review findings, the full Work Plan assessment, and other assessment materials: http://www.slcc.edu/cwc/Assessment.asp

_______

Sample User Satisfaction Surveys

Writing Workshop Evaluation

Please respond to the following so we know how effective our writing workshops are and how to best improve our programs.

Workshop: Workshop Title

Agree

Somewhat Agree

Neither Agree Nor Disagree

Somewhat Disagree

Disagree

Workshop Leader: Your Name

Date: Workshop Date

1. I felt encouraged to participate in the workshop discussions and writing activities.

2. The workshop was interactive and collaborative.

3. I received helpful feedback on my writing from other participants.

4. I received helpful feedback on my writing from the workshop facilitator.

5. This workshop provided me with writing tools and strategies that I will continue to use.

6. The workshop facilitator was well-prepared and knowledgeable.

Comments:

What worked well for you in this workshop?

What could have been better?

What types of writing workshops would you be interested in attending in the future?

How did you hear about this workshop?