Writing items on NSSE -- preparing for meeting at WPA conference to collaborate with NSSE to enhance its coverage of writing

This post has a two parts: "Background and problem/opportunity" and "Please reply to this question...."

Background and problem/opportunity. The National Survey of Student Engagement (NSSE) has been used by over 1100 higher-ed institutions to help them "identify aspects of the undergraduate experience inside and outside the classroom that can be improved through changes in policies and practices more consistent with good practices in undergraduate education" (http://nsse.iub.edu/html/quick_facts.cfm; a good place to start learning about NSSE). It's becoming an increasingly important instrument used by administrators to make decisions about undergraduate education. In recent postings to the WPA and WAC listservs, many WPAs indicated that their campuses used NSSE and that their upper administrators considered its evidence important.

Here's the problem & opportunity. The current edition of NSSE includes just six items on writing, and most of these focus on the amount of writing. (See the next part of this post for a list of the items.) The good news is, Bob Gonyea is interested in improving the way writing is measured on future versions of NSSE, and is eager to work with WPAs at the upcoming WPA Conference in Tempe (see http://wpacouncil.org/conference2007) and afterward to come up a list of 20 - 30 questions to test in Spring 2008, a subset of which may be included in the next edition of NSSE.

Bob Gonyea (Associate Director, Research & Analysis, at NSSE), believes more and better writing items will improve the NSSE; the idea that the amount and quality of undergraduates' writing would be associated with engagement has, he thinks, good face validity.

To this end Bob will be in Tempe and will spend about 3 hours of conference time with us. First, just after lunch on Saturday (the last full day of the conference), he will give a fairly brief talk on NSSE, how it works, how questions are made and tested, and how NSSE might be improved if it more effectively surveyed students' writing experiences. It will be followed by a 105-minute working group, where we can pound out some details. The session will be open to all. There will be other concurrent sessions scheduled but probably none with a WAC focus.

This opportunity may lead to a NSSE instrument that provides strong evidence that shows (for the first time at a national level) how writing can make a difference in student engagement and learning.

Shirley Rose informs us that the NSSE session is tentatively scheduled for Saturday afternoon, July 14, from 1:45-4:30. It will cross two concurrent session times, 1:45-3:00 and 3:00-4:30. Refreshments will be available 2:45-3:30. so the group can take a break sometime during that window. there might be some slight adjustment to the time, but it's certain that the workshop will be Saturday afternoon.

Please respond to these two questions: 1) What in the current set of NSSE writing questions seems effective? 2) What would you like to see more of? Let's try to keep this conversation (at least for now) on this WPA blog, so please respond to this blog post. Later, we will probably want to throw it open to the WPA listserv. And perhaps we can re-evaluate whether this blog is the best way to communicate.

Here are the six items (go to http://nsse.iub.edu/html/survey_instruments_2007.cfm to see the entire instrument):

In your experience at your institution during the current school year, about how often have you done each of the following? (Very often, Often, Sometimes, Never)

  • (rewropap) Prepared two or more drafts of a paper or assignment before turning it in
  • (integrat) Worked on a paper or project that required integrating ideas or information from various sources

During the current school year, about how much reading and writing have you done? (None, 1-4, 5-10, 11-20, More than 20)

  • ((writemor) During current school year …. number of written papers or reports of 20 pages or more
  • (writemid) During current school year …. number of written papers or reports between 5 and 19 pages
  • (writesml) During the current school year … number of written papers or reports of fewer than 5 pages

To what extent has your experiences at this institution contributed to your knowledge, skills, and personal development in the following areas? (Very much, Quite a bit, Some, Very little)

  • (gnwrite) Writing clearly and effectively

Comments

1. When I read NSSE results, I always feel that the questions and responses are good as far as they go, but I want them to go deeper. I would not eliminate anything current. I think it is very helpful to know something about frequency.

Some issues we might address:

--writing as a social activity: collaborative, team-based writing

--learning to repurpose or repackage existing writing for new purposes or audiences

--learning to integrate data or research results into arguments

--writing for real audiences, for clients, as part of service learning or coops or internships

--learning to write within a disciplinary context, with a growing sense of disciplinary authority

--writing for publication

--gaining a sense of genre or variation based on situation, purpose, audience, medium

--writing in variously mediated genres, as web content or blogs or wikis, as multimodal, designed, visual communications

--writing for learning: response papers, journals, learning logs, reactions, informal postings, discussion or forum postings

--writing for self-sponosored purposes

--sense of confidence in oneself as a writer, able to size up a situation and develp an appropriate response

Steve Bernhardt

Everyone has made some great suggestions here! The direction of them in terms of distinquishing kinds or genres of writing and purposes for writing is the right one, IMO. I don't know how messy a revised NSSE might get, but I'd like to see something in the way of a question or questions that gets more at the quality of students' revision of their writing, not just the quantity. For instance,

If you revised a paper before turning it in, did you do that on your own or was it mandated by your instructor? In either case, from whom did you get feedback on an early draft? -- your instructor, a classmate, a writing center tutor, someone else?

How do you understand revision? As editing your errors and spelling? As reorganizing or refocusing your ideas? As adjusting voice and tone? As looking again at your thoughts? (These aren't asked here in a very easy-to-survey kind of way, I realize, but I think it would be nice if the survey could glean some "deeper' information about student revision of writing.)

JW

I'd like to suggest once again that we try to tie the NSSE questions into the WPA Outcomes Statement, insofar as possible, since it was developed by a large number of active WPAs and adapted by a considerable number of programs. I believe that we need to repeat the same message over and over again to make it take, rather than re-inventing our wheels.

Asking students whether they have used what they learned in FYC or other writing programs is a dangerous query, and one I think we should avoid. My research (to be published in the Fall 2007 WPA Journal) suggests that students can be very negative about what they learned--and I'm not sure how valid their responses will be at the time the survey will be given.

Linda S. Bergmann
Associate Professor of English
Director, Purdue Writing Lab
Purdue University
Department of English
500 Oval Drive
West Lafayette, IN 47907-2038

765-496-2814

I would like to see questions about frequency of use of the writing center, and the kinds of issues that students address while they are there. For example, we could ask how frequently WC visits discuss development of ideas, finding and supporting a thesis with appropriate evidence, organization, and other higher order concerns, vs. finding and correcting sentence level errors, help with documentation, etc. Since writing center work is designed to be collaborative, questions that get at the collaborative nature of the WC visit would also be helpful.

Marcy Trianosky

I echo what Marcy says about writing center and other support services, such as the library. We also could ask about the use of software (at Texas A&M we use Turnitin.com, for example).

At Drew we incorporated 12 slightly modified NSSE questions into our new evaluation of teaching form (for individual faculty and program assessment use *only* -- the data will not go to the committee on Faculty). We just replaced "During the current school year" with "In this course." The NSSE writing questions were not so helpful to us though, so we used the question below. I'd love to see something like this on NSSE. It is a more complex question because it makes the students think about *why* they were assigned writing, but I think it tells us a lot more than number of pages.

Sandra Jamieson

------

[In this course] During the current school year, I used writing in the following ways [choose as many as apply]

• To help me understand the course material (such as notes, exercises, problem sets, annotations, summaries, definitions, or drafts and revisions)

• To share my response to course materials (such as response papers; journal entries; or posts to Blackboard, an online discussions group, or a listserv) Extracurricular events and activities

• To develop my analytical and critical skills or to explore issues in more depth (such as critiques, annotations, abstracts, or short essays spread through the semester

• To practice the writing of the discipline (such as proofs, abstracts, synthesis, annotations, laboratory notebooks or reports, case studies, field reports, ethnographies, research proposals, or reports or research papers using discipline-specific resources and style sheets)

• To demonstrate my learning in the course (such as take-home or in-class essay examinations, quizzes, or a single term paper at the end of the semester)

• To expand my expressive, creative, or professional writing skills (such as creative writing, playwriting, journalism, business writing, or speech writing)

• Other [please use the comment box to describe]

• I did not do any writing for this course.

I agree with the suggestions already offered in the 3 comments below. I especially like Sandra's list of questions because they ask for the "deeper" information that Steve seeks.

I'd like to find a way for NSSE results to show how much / what kind of writing they do in courses that aren't English. Sandra's questions get at that issue in a way, but those were class specific surveys, and the regular NSSE is not. Steve's suggestions point to non-English-class kinds of writing, but do students recognize that? Here's why I suggest that we be specific about writing other than English classes.

Several weeks ago I conducted two days of focus group interviews with science majors (juniors / seniors). Though the interviews were focusing on curricular changes that added writing and research to a lab course, we did ask several questions about writing experiences before this course. A vast majority of the students connected "writing experiences" only with English courses (writing or literature), and we had to ask follow up questions to learn about any writing in other courses (sadly, not much, by the way). I fear that too often students associate questions about "writing" only with English courses (and perhaps some few humanities courses), and as a result don't report or consider the writing they may do in other courses.

So I'd like to see some NSSE items that explicitly attempt to gather information about writing other than English courses.

Then, perhaps a follow up for those who report writing other than English with the kinds of specific questions that Sandra and Steve are suggesting.

(In an ideal world, too, we could ask some of those more specific questions about students' experiences in English / Writing courses. But I'm not sure how many items we might be able to add.

I'm grateful at NSSE's interest in revisiting the writing-related questions (and to Chuck for helping this happen). The focus of the NSSE questions on quantity of writing say so little about quality of writing. My school performs poorly on NSSE unless the question about writing 20 plus pages is figured out of data aggregation because length isn't an emphasis or requirement in our WAC-like courses or our first year composition course which is portfolio-based. I like so much more what's emerging here... I'd like to know what students find themselves learning/doing with writing as others have suggested.

Another distinction that might be helpful (and which is reflected to some extent in the additional questions people have suggested) is how often writing is used to "test" something (accumulated knowledge, skills, or abilities) as opposed to learn something (as in wrestle with a concept, explore an idea or reading, reflect on some experience, and so on). Typically, learning-based writing will be lower-stakes, brief, and less formal; but it would be great to get a sense of how or whether writing to learn contributes to engagement. The Harvard Seminar data from the early 1990's showed a really powerful relationship between engagement and amount of writing, but it didn't really delve into the genres, uses, and levels of formality of that writing. I'm not sure how all that gets manifested in specific questions, but it might be worth thinking about if items on writing will be added. --Chris Anson

The current questions are helpful in showing the extent to which students at our institution are writing for classes. However, it does not tell us much about activites that are not formal writing--students often don't count some writing-to-learn activities, such as microthemes, as writing. Also, it does not address journal writing, posters, oral presentations which include Power Point or handouts, and other forms of writing. It does not address questions about the writing process well--it begins to get at them with the draft question, but much more could be explored there. It does not address attitudes about writing. We are also interested in extra curricular writing.

I really like Sandra's list of questions, the idea behind several of Steve's (can we find out what modes/media they're writing in, and what their confidence levels are), and one that Joel was getting at, which is especially related to "engagement" with university resources: (how often) did you revise after receiving specific feedback from ____ (a professor, writing consultant/tutor, peer)?

Another kind of engagement-level question might be something like this: when you were completing a writing assignment, which if any of the following did you do -- conference with a professor, email a professor, use a campus writing center (f2f or online), consult a writing textbook/handbook (paper or online), share your paper with a peer, ask a peer a question, consult a family member, other, none.

It might be interesting to ask a question about transfer: to what degree did/do you use writing skills (learned in English/Composition) (learned in your major field courses) in other school writing? in writing outside of school?

I wonder if there's a way to get at a perception of value-added: to what degree do you believe writing instruction at this school (in English/Composition) (in courses in your major field) improved your writing (in other courses) (outside the university)?

shelley

I concur with the excellent responses here, and many of them would generate useful data for fyc, WAC (or any!) college class. Kate said “I'd like to know what students find themselves learning/doing with writing as others have suggested” and Glenn would “like to find a way for NSSE results to show how much / what kind of writing they do in courses that aren't English.” But is this the aim of the NSSE, and can we actually change that aim? The questions we are suggesting here seem to address a different purpose than those of a student satisfaction survey (taken by currently enrolled students). Chuck says that Bob Gonyea believes “the idea that the amount and quality of undergraduates' writing would be associated with engagement has, he thinks, good face validity.” Are our questions focused on measuring student engagement (and, the logic goes, their satisfaction as a result of that engagement)?

While we might work to change Bob’s focus on “amount” and even “quality” (and what does that mean to a student?) we can’t change the NSSE’s focus on satisfaction resulting from engagement. I am wondering how we might use that focus to our advantage (resource allocation) and for our own research. For example, students are satisfied and engaged in our first year writing classes because of the one on one attention, because we learn who they are, and we work with/honor what they bring to the table, because they know our names: they vote about engagement with their feet when they ask us their senior year for letters of recommendation because they believe we know them …because our pedagogy with writing has engaged them.

In our alumni survey here, students are most satisfied with the writing in courses where they had useful feedback. As others pointed out, the writing center is mentioned as a place where students engaged in writing satisfactorily. Also, students are not satisfied when the purpose of the writing they is not clear to them: a twenty page research paper teaches them research (they are satisfied with that), but they now (as alumni) have to write paragraphs summarizing their research (they didn’t get enough direction with this and are less satisfied).

How NSSE is framed by the word “satisfaction” should be our guidepost and make us careful about what we ask: “How much writing do you do in other classes” may turn into an indictment of a WAC program if students don’t consider informal writing as writing and answer “not much”; or if the questions we propose address the usefulness of fyc or WAC writing in other classes, students may reply to NSSE (as they do in class evaluations)that they are not satisfied with the kinds of writing they are doing (informal, revised, developmental stages).The response may indicate a problem with writing instruction or, just as likely that students can’t yet see the writing they are doing as useful. Questions that ask “Do smaller classes help you learn to write more effectively?” might well be more advantageous to us and tell others what we know about instruction in writing: what questions/results about satisfaction would be useful to us and to our institution?
Joan

I think it would be useful to know if the students took one or more writing courses on the campus they are at, or if they were given transfer credit, AP credit, dual credit for a high school course, or credit for an exam they passed. We don't have good national data about this, and with so many alternative methods that "count" now for writing instruction, it's hard to say what students are learning about writing and where.

Since the NSSE is about student engagement, we might look at specific ways writing connects with the research on engagement. George D. Kuh’s article in the June 15 Chronicle of Higher Ed lists several research-based steps that improve engagement, some of which connect to points already made in the blog. For example, “Teach first-year students as early as possible how to use college resources effectively.” So a question about whether students know about the writing center could be followed by the “Have you used the writing center for help with your writing?” Another of Kuh’s steps is “Make the classroom the locus of community,” with an explanation that many students have no opportunity to form community except in class. A question about peer review or writing groups in a class could help tie writing to community building.

We are interested in information literacy and how students are learning to locate, evaluate, and use sources. The NSSE question about writing a paper that incorporates ideas and information from other sources is good, but it does not necessarily mean students had to locate and evaluate the sources. Could there also be a question about how often the student has had to locate and evaluate information in sources outside the course texts? Another: When doing research for a paper or project, how well prepared are you to find and evaluate information through the library or internet? Very well prepared, somewhat prepared, I’m clueless. How confident are you that you can correctly document information from other sources? Very confident. Fairly confident. Not at all confident.

I agree with adding a question about whether the student has taken a writing course at the school where he or she is currently enrolled. It would also be great to get some data on where students are taking their writing courses. Because of the time of year the NSSE is sent to students, there are issues for schools on quarters. Due to alpha sectioning, by late winter, a third of first year students have not yet had a writing course.

One thing that has always bothered me about the NSSE length-of-paper questions is an implication that the more 20+ page papers, the better. And so much writing is left out of this forced-choice. As Joan said, these long papers are not necessarily preparing students for writing after college.

I want to second the calls to include information about Writing Center visits. Even just a question about the number of WC visits would be helpful.

I also like the questions about the amount of feedback students have received. Some possible ways to pose these questions:
"I have received substantial feedback on the CONTENT of my writing assignments."
"I have received substantial feedback on the WRITING (i.e., organization, mechanics, grammar, "flow") of my writing assignments."
"I have received advice that would help me improve the quality of my writing."

All of these could be answered on the Very Often - Never scale.

Joanna

I've posted a report on the WPA/NSSE work session at the WPA conference on my blog at this URL: http://wpacouncil.org/NSSE

Shirley K Rose,
Purdue University,
Immediate Past President,
Council of Writing Program Administrators