Tuesday, October 04, 2005

Day Two of Rosh Hashanah Break

So yesterday I did two laps on the Shore Parkway, from the 68th St Pier that looks across to Staten Island, the Statue of Liberty, and Manhattan out to Bay Parkway, the other end of the park that looks out to the ocean and Coney Island. All in all, sore thighs and a sun burn on my thighs—at least I remembered to cover my head. I took a pit-stop at Starbucks to do some writing—I’ve been working on this project about contemporary rhetoric about civil rights. More specifically, I’m interested in the clash between the New Right (typically understood as a coalition of Evangelical Christians and neoconservatives) and the gay community over a series of issues. The larger book will map out their debates and fights over language, but I’m trying to argue that as the queer community follows the precedents of earlier New Left extensions of civil rights, the right simultaneously works to beat back that movement and to change the foundational concept of what civil rights mean. I hope to focus on campaign rhetoric from both movements, media representations (or framing) of the issues and movements, and governmental response. At any rate, I know what I doing in the body of the text, but I’ve been stumped on the introductory chapters and keep spinning my wheels. One day, I want to play off of current events, another day I want to open with a personal narrative, and still another day I think vignettes might be another way in.

After I got frustrated with that, I came home to work on another on-going project at Stony Brook. As some of you may know, assessment has consumed more and more of my energies as my program tries to come to terms with all the data we produced from our SUNY-Central mandated assessment. Our campus-based assessment protocol involved collecting four writing samples (placement, thesis-driven essay from WRT 101 [if placed in that course], thesis-driven essay from WRT 102 [most students, except honors students, cycle through this course], and exit essay) and scoring them for four primary traits (critical thinking, genre knowledge, rhetorical knowledge and mechanics & usage). The placement and exit essays are produced through MIT’s iMoat software; students go on to the system at a pre-determined date, read a prompt, and then have 72 hours to produce an essay. The system is nice because it doesn’t penalize L2 writers and enables students to produce more coherent, cogent responses without as much time pressure, but we don’t have a sense of performance differences between students who wrote in an impromptu manner (before iMoat) and after until this “standardized” form but loose timing. The data we’ve produced is compelling, and our students’ performances generally confirm those that can be seen in national tests like the CAAP or Academic Profile. Aside from the percentages that the state asks from us (how many students don’t meet, approach, meet and exceed standards/expectations), we also needed to show what value was added by the writing program’s instruction.

Cynical readers like me will have their ears prick up on that note: How do you prove value-added? Isn’t that just a euphemism for change and causality? Well, yes and no. So in an earlier cranking of the numbers through my lovely SPSS software, I did discover significant change between placement and exit essays and to a lesser a degree for the subpopulation who took 101 and 102 (thereby having two time-separated essays). More importantly, the 2-semester students came in generally weaker than the 1-semester students, but the 101/102 left showing far more improvement or growth than the 102-only. Nevertheless, the one-semester students still out performed the 101/102 students, even though the former showed marginal value added to their experiences. That’s one of the things I’m trying to get my head around—what’s the significance of stronger students, at least by our measure, not getting value added? Is that the case? Is there value that these students gain that our protocol doesn’t capture? If so, what is it? At any rate, these insights on data representing group performance or changes in statistics that capture the whole cohort, and it doesn’t really look to individual change. On another run on the data, we did that, and then calculated the percentage of students who have value-added (or lack or lose value). Looking at the data this way, 2/3 of our students either stay the same or lose ground—that is, their scores stay the same or drop. To my chagrin, when a binder was compiled for our recent external review, my WPA took this data up with the dean, advisory board, and review team to suggest our full-time instructors weren’t having an impact on student learning. Mind you, the dean and advisory board were suspicious of our numbers—what did it mean to lose value in instruction? Was that possible? Going by our number, at least, it was… and so goes the old adage about being skeptical of stats…

So yesterday, I went back and looked at our raw data after meeting last week with a mathematics expert in our institutional research office. She helped me understand that our rubric and representations were biased against the students who come in and leave performing well. In other words, the scale is not set up to measure changes for students at the ceiling of our rubric, so she looks at the data and said that we’d probably get more interesting information looking at shifts for particular scores. I spent a good part of the day tracking students who came in with X score and left with Y score. In our previous take on the data, 33% lost ground, 33% stayed the same, and 33 showed improvement (that’s a rough generalization across semesters). Looking at the data another way, students who came in (or on the placement essay) scored as:
• not meeting our standards showed 100% improvement (50% moving up to the approaching standards category, and 50% moving up 2 levels to meeting standards)
• approaching standards—63% showed improvement (52% moving up 1 level to meeting standards, and 11% up to exceeding), but 41% showed now change or didn’t move out of that category
• meeting standards—80% showed no change, but 4% moved up to exceeding standards (and 4% moved down one level to approaching standards, and 4% moved two level to not meeting standards)
Now begins the analysis—is this representation of how students performed better or just more complicated and accurate? To what degree is it encouraging? What does it say to whether we ought to have a two-semester or one-semester sequence? Are 14-28 weeks really enough to certify a student’s college-long experience with writing?

Most important to me, how can I make this information useful for staff education in the writing center. We’ve used other profile information from the assessment, particularly from the primary traits, to support our focus on HOCs over LOCs, but this recent data has left me perplexed…

Still all this number talk makes me want to re-read Neal’s bean-counting article.

Stay tuned… harry

5 Comments:

At 9:38 AM, Blogger /WCJ/ said...

Interesting stats stuff, Harry. I can understand why your "show improvement" outcome would penalize those who come in performing higher than others, and I think the approach of separating students based on that those three levels makes a great deal of sense. I suppose one thing you might conclude from the large numbers of students who showed little change is that the "standards" themselves might need some re-examination. Where did they come from? I'd also wonder how the standards map to the actual curriculum in 101/102.

Of course, student growth or the value added by your courses might not only be in terms of improvement in this specific writing task, but in other variables: sense of belonging to the institution, retention, critical thinking. Does Stony Brook give other kinds of large scale assessments at the start and at the end of the first year to which you might map some of these scores?

I also think you have a great opportunity to focus in very specifically on a small number of students who did or did not change and get some rich qualitative data as to what went on.

Cool stuff!

Neal

 
At 2:35 AM, Anonymous Anonymous said...

Hi, Harry--

Don't take this the wrong way, but my favorite thing about these posts of yours have been the window into your life: the fitness regimen, lovely tuna salad, trips on the N train. It's making me think again, as Melissa is thinking and trying to get us to think, about blogs and the purpose of them. I guess we don't need blogs to capture those little details--we could do it as well in an email message, though we might be less likely to--but there's something about blogging that seems particularly well-suited to those kinds of observations/ruminations, in the best essayistic style and tradition (Montaigne, for example). The tutors convinced me to set up a facebook account this week, and I've been surprised by how many of my former tutors have already found me and "written on my wall." I'm sure there's a novelty element to it for them, but it also says something, I think, about genre and technology--that they wouldn't necessarily send me an email message, even though they have my address, but they would write the same thing on facebook. There's a constancy of presence there, as on a blog, and I'm not sure we've really considered that aspect of it.

--Beth

 
At 4:19 AM, Anonymous Anonymous said...

Thanks, Beth. I like those aspects better too. Maybe our personality and voice come out a bit more. We become a bit more real and tangible. How long will it be before we can post MP3 files of us? At that point, I'll get caught on my commute singing to the Rent soundtrack or Joan Osbourne ("Oh St. Teresa..." Who the heck is she anyway? Spoken like a true Catholic...).

Alright, I thought I was tech savy, but what's a facebook? We've started a blog this year, but the tutors are resisting it, wanting instead to open for a journaling system like Meg's at RIC. Ever since that darn NEWCA presentation and reading your book, they've thought that was the best idea since sliced bread.

harry

 
At 7:42 PM, Anonymous Anonymous said...

runescape money runescape gold runescape money runescape gold wow power leveling wow powerleveling Warcraft Power Leveling Warcraft PowerLeveling buy runescape gold buy runescape money runescape items runescape gold runescape money runescape accounts runescape gp dofus kamas buy dofus kamas Guild Wars Gold buy Guild Wars Gold lotro gold buy lotro gold lotro gold buy lotro gold lotro gold buy lotro gold runescape money runescape power leveling runescape money runescape gold dofus kamas cheap runescape money cheap runescape gold Hellgate Palladium Hellgate London Palladium Hellgate money Tabula Rasa gold tabula rasa money lotro gold buy lotro gold Tabula Rasa Credit Tabula Rasa Credits Hellgate gold Hellgate London gold dofus kamas buy dofus kamas wow power leveling wow powerleveling Warcraft PowerLeveling Warcraft Power Leveling World of Warcraft PowerLeveling World of Warcraft Power Leveling runescape power leveling runescape powerleveling eve isk eve online isk eve isk eve online isk 血管瘤 肝血管瘤 音乐剧 北京富码电视 富码电视 富码电视台 7天酒店 7天连锁酒店 7天连锁 自清洗过滤器 过滤器 压力开关 压力传感器 流量开关 流量计 液位计 液位开关 温湿度记录仪 风速仪 可燃气体检测仪 宿县最帅的胖子

 
At 6:38 PM, Anonymous Anonymous said...

We really love your blog, i haven't seen you keeping the posts in in some time now. Is everything ok. You can reach me here = web traffic@simply pet products.com. Thanks Again Rob Dog Crates, Dog Toys ( Dog Supplies & Cat Supplies Pet Store, Dog Collars) Send Traffic Now

 

Post a Comment

<< Home