Assessment Narrative - Saint Joseph College

This assessment model is part of the WPA Assessment Gallery and Resources and is intended to demonstrate how the principles articulated in the NCTE-WPA White Paper on Writing Assessment in Colleges and Universities are reflected in different assessments. Together, the White Paper and assessment models illustrate that good assessment reflect research-based principles rooted in the discipline, is locally determined, and is used to improve teaching and learning.

 

Assessment Narrative - Saint Joseph College

Institution: Saint Joseph College, Connecticut Type of Writing Program: Across the Curriculum Cumulative Portfolio (required) Contact Information: Dr. Judy Arzt (Director)  jarzt@sjc.edu  (860) 231-5353

 Dr. Kristine Barnett (Writing Portfolio Coordinator)  kebarnett@sjc.edu  (860) 231-5472

Assessment Background and Research Question

In 1988, Saint Joseph College undertook a study of college-wide writing assessment practices used nationally and internationally with the goal of implementing an assessment complementary to its culture. Although for over 50 years our college catalog indicated that students must achieve writing competency to graduate, the faculty questioned whether measures used in the past followed best practices. For instance, as late as the 1980s, a junior-rising exam was used, but this method was contrary to theories set forth by popular writing theorists such as Peter Elbow and Donald Murray. Our faculty members were disenchanted with a one-shot essay writing sample, and although one was administered to all juniors in the spring of 1987, members of the English department soon agreed the method was not suitable and stored the essays in a basement, unread. With the hiring of a director to oversee writing programs, a full-scale search ensued for an effective assessment tool.

The director, along with an assessment committee, commenced a yearlong study to compile assessment information, and resources were gathered from the Conference on College Composition and Communication (CCCC), American Association for Higher Education (AAHE), and National Testing Network. New publications in the field of assessment were further consulted. In the end, we favored a longitudinal portfolio.

The questions we sought to answer through a portfolio assessment focused on how students develop as writers over time and how well our curriculum helps shape students as writers. As we sought to define the portfolio program for compatibility with the college mission, faculty were a guiding force in the decision-making process. Faculty attended a series of workshops to become better acquainted with the principles of Writing across the Curriculum (WAC). During the 1989 spring term at a one-day WAC workshop, faculty examined model assignment sheets and reviewed individual students’ writing assignments and due dates for a single semester. These activities helped faculty see writing (amounts and expectations) from a student perspective, as well as understand that writing skills develop over time and vary with genre and discipline. Following the workshop, at the end of the spring term, a WAC consultant was invited to spearhead a two-day, all-faculty retreat focused on encouraging faculty commitment to writing-to-learn activities, fostering the perception of the teaching of writing as a joint faculty partnership, and promoting recognition of the cumulative developmental process of writers. In summary, both the retreat and the workshop created a climate receptive to a longitudinal, cross-disciplinary portfolio program as a means to assess student writing. The following fall, a faculty assessment committee advanced a proposal for a portfolio program to the faculty for a vote, and the proposal received unanimous approval.

We envisioned the portfolio program as a means to assess student writing as well as the college’s new core curriculum. Annually, students were required to submit one paper from a core course and an additional paper from another course. All core courses were writing intensive, and faculty designed common syllabi and assignments. However, portfolios revealed that students’ strongest writing did not derive from the core classes, but from disciplinary courses that held the students’ interest, often courses in their major or closely allied disciplines. Not surprisingly, whereas the portfolio program survived the test of time, the core curriculum did not. In 1995 we undertook an assessment of the core program and surveyed seniors who had been through a four-year cycle of the courses. The survey results led faculty to abandon the core courses but not the portfolio, for which there was strong support.

In summary, the research questions guiding the formulation of our portfolio included:

  • What are the characteristics of student writing on our college campus?
  • How do students develop as writers over time?
  • How well do our students succeed in meeting a writing competency requirement for graduation?
  • How can portfolios inform and support curricular and instructional decisions?
  • What does a cross-disciplinary portfolio tell us about writers and their rhetorical strategies?

Assessment Methods

Portfolios are evaluated using three techniques. The written commentary section is the most labor intensive but the most helpful to students’ writing development. A criteria checklist and a holistic score complete the evaluative process. (A sample score sheet can be found in the appendix.) The criteria for evaluating portfolios are (1) fluidity and clarity of expression, (2) effective organizational skills, (3) effective use of details and elaboration, (4) critical thinking skills, (5) effective research writing skills, (6) effective use of language and diction, and (7) mechanics and usage. A student’s portfolio is rated in each of these areas using a plus, check, or minus.

Papers are evaluated holistically and more weight is given to recent work to reflect the student’s progress. A 5-point scoring system is used, with 5 as the high score. A minimum score of 3 is required to complete the process. The sophomore-year evaluation serves as formative assessment, and a lengthy commentary educates students about the criteria and the fit of their writing with each of the seven criteria. Usually a paragraph or two explains each rating, and evaluations conclude with a bulleted list of tangible suggestions for writing improvement. The junior-year final evaluation follows the same process, except summative comments are kept to two paragraphs, as students are completing the process and receive acknowledgment for accomplishments to date, though recommendations for the future are included.

The process of completing preliminary evaluations, as noted, is complex and time consuming for evaluators, but students report that the feedback is valuable. In the late 1990s, we conducted exit interviews with graduating seniors to assess their response to the program. Most highlighted the advantages of the comprehensive preliminary evaluation, which has remained a distinctive feature of our program.

The actual scoring process is not particularly difficult in terms of reaching a consensus between two readers. At the preliminary stage, one member of the writing center staff writes a comprehensive evaluation. The second reader responds to this evaluation and offers suggestions for additions and changes. Ultimately, the two readers must agree on all aspects of the commentary as well as the ratings and score. At the final stage, two faculty members evaluate a portfolio independently. The two then meet to synthesize findings. Ordinarily, there is strong congruence, and in the two decades in which the program has been in place, only on the rarest of occasions has there been a discrepancy. In such cases, a third reader resolves the difference. In all cases where two independent readers score a portfolio “below satisfactory” or “poor,” two additional readers must read the portfolio independently. Thus, for a portfolio to receive an unsatisfactory score, four faculty members must agree on that score. On the other hand, students who receive the top score of “5” are commended through awards and the transcript notation of “Writing Portfolio Completed with Distinction,” as opposed to the regular designation of “Writing Portfolio Completed.”

The portfolio process has remained fairly consistent since its inception and is well articulated for students, faculty, and academic advisors. Our “Writing Portfolio Booklet” explains the process, and annual portfolio reports, enumerating results and recommendations, keep the college community abreast of the program and student progress. Class visits and presentations at orientations and other events further serve as communication channels.

Assessment Principles

The portfolio program is based on assessment practices that mirror National Council of Teachers of English (NCTE) and CCCC position statements on writing assessment. Principles that have guided our practices include:

  • Writing skills develop over time and a writer’s rhetorical strategies shift to accommodate context and audience. To understand a writer’s full range of skills, a collection of samples written in different genres and for different purposes is essential. This principle reflects the CCCC Position Statement: “Best assessment practice uses multiple measures.”
  • Evaluation of student writing skills needs to reflect the curriculum as well as inform curricular practice. To that end, students submit a sample of papers from a variety of courses. The results of student outcomes on portfolios are annually communicated to the college community and used to make curriculum decisions and shape classroom teaching practices. We believe that timely communication of results is a critical part of the process and keeps the college community informed of the link between the curriculum and student performance.
  • For assessment to truly benefit students, they need timely review and valued response to their work. Our system provides students with detailed feedback on their growth as writers, and students are invited to confer with evaluators, advisors, faculty, and writing center staff regarding writing progress, portfolio submissions, and evaluations. We see the portfolio process as fostering a community of learners and as a means for promoting conversations about writing on campus.
  • Writing assessment needs to value teachers’ classroom work. Faculty members are an integral component of the process, and they guide the program. As such, the program is sensitive to faculty response, and all faculty members are invited to be portfolio readers. Training is provided to new evaluators, and faculty reading days are considered a faculty development activity, rewarded in the tenure and promotion process, as well as through monetary compensation. Faculty members have found that reading days provide them with an opportunity to confer with colleagues on assignment design, methods of response to student writing, and the overall college curriculum.

In addition to the specific guidelines set forth by NCTE and CCCC, scholarship in the field of writing assessment, including work done by Edward White, Kathleen Blake Yancey, Brian Huot, Chris Anson, Richard Haswell, and Nancy Sommers, has informed our work.

Assessment Results and Follow-Up Activities

Assessment results are reported directly to students via a written evaluation. In addition to the score sheet and narrative, students are encouraged to meet with evaluators and writing center staff, as well as advisors, to discuss portfolio results and writing skills in general. In the past, students have been surveyed to capture their perceptions regarding the portfolio process. Findings revealed overwhelming support for the continuance of the program and the evaluative techniques, particularly the commentaries.

Faculty advisors are considered essential to the success of the program. In fact, all advisors receive a duplicate set of students’ papers and evaluations. Advisors meet with students to discuss progress and set goals. In addition, faculty are attuned to using the program to evaluate the curriculum, and, of late, plans are underway to use portfolios to assess the college’s new general education curriculum. A representative sample of portfolios will be used to assess how well the new curriculum addresses critical thinking skills.

Collective program results are communicated through an annual portfolio report disseminated to members of the college community. These reports not only give results of student performance but also include recommendations for curricular reform and areas of focus for classroom instruction. For instance, annual reports have led to increased attention to teaching research skills, as portfolio results evidenced a significant number of students struggled in this area.

In terms of informing the greater community, we have made presentations on our program at national and regional conferences, including the American Association for Higher Education, CCCC, Northeast Writing Centers Association, International Writing Centers Association Conference, and New England Assessment Network. The program is listed in AAHE assessment publications. Routinely, we submit proposals for conferences and make presentations at other campuses interested in implementing a writing assessment program. In addition, faculty from other colleges have attended our portfolio faculty reading days to observe the evaluative process.

Assessment Resources and Transferability

The writing portfolio is integrated into the services provided through the Center for Academic Excellence (CAE). Thus, the only additional funding required is compensation for faculty who read portfolios, and the per diem rate for a full reading day is $300. We also compensate faculty who occasionally read portfolios outside of reading days to expedite the evaluations, and these faculty are paid $30 per portfolio. Faculty readers who periodically evaluate portfolios throughout the year are still expected to confer with a second reader, either in person or online.

The program is easily transferable to other institutions, and Springfield College, in Springfield, Massachusetts, has already adopted some aspects of our program to assess student writing in specific programs. The evaluative criteria we use reflect research and scholarship on writing assessment in composition and rhetoric, follow common standards for assessing writing based on that work, and have also been adapted from other assessment processes. An examination of portfolio programs at other institutions revealed use of similar criteria, developed after those institutions established that these criteria were also appropriate for their contexts. The training program is also easy to implement, and the reading days, which are an excellent faculty development activity, are comparable to similar portfolio reading days or faculty workshops held at other institutions.

The program has thrived since 1989 and has continued to be supported by students, faculty, and administration. Despite the program’s solid foundation and successful track record, the college continues to streamline the program to ensure sustainability.

Appendix

 

SAINT JOSEPH COLLEGE WEST HARTFORD, CONNECTICUT

WRITING PORTFOLIO SCORE SHEET

 

STUDENT’S NAME:          

 

SCORING SCALE:

Writing portfolios are scored holistically on a scale of 5 to 1. A score of 3 or higher is needed to fulfill graduation requirements. The score scale is as follows:

            5 = excellent             4 = good             3 = satisfactory             2 = below satisfactory             1 = poor      Inc = Incomplete (work is missing)

SCORE:

CRITERIA LIST:

A plus mark in front of an area indicates strength; a check mark indicates an area is satisfactory; a minus mark indicates an area in need of improvement.

            _________ 1. fluidity and clarity of expression _________ 2. use of appropriate organizational structure             _________ 3. sufficient use of details and elaboration             _________ 4. critical thinking skills             _________ 5. effective and correct use of research techniques             _________ 6. effective use of language and diction             _________ 7. correct mechanics and usage

COMMENTS:

 

Readers:                                                                                                         Date:

 

 

Resources

Arzt, Judy. “Electronic Portfolios’ Transformative Effects on Assessment.” Paper presented at Conference on College Composition and Communication Convention. New York City. 21 Mar. 2003. <http://www.sjc.edu/jarzt/C4draft.htm>. Arzt, Judy. “Writing Portfolio as an Exit Requirement.” Reviews and Descriptions of Assessment Instruments. Knoxville, TN: Clearinghouse for Higher Education Assessment Instruments, 1994. “Portfolio Project.” Center for Academic Excellence, Saint Joseph College, CT. <http://www.sjc.edu/content.cfm/pageid/4277>.