Now look at this: a grading system based on Slashdot and karma (scroll down to read about karma, it’s under Assignments). Alex Halavais’s Media in the Information Age course, taught last autumn, had students write 500-1000 word responses each week. Students were given karma points to award to the responses they thought were best. A multiple choice concept test set a base level grade for each student (less than 40% was an F no matter what else the student did, 40%-70% was a D no matter what else you did, above 70% got you to C and you could get higher through karma points based on your weekly responses. Here’s the description of the karma system:
Points not used by Monday cannot be carried over to the next week.
At the end of the semester, the average number of karma points for those in the top 15% of the class will be fixed as an A. For each person who passes the competency exam with a score of 70% or more, the final grade will be determined based upon the percentage of this maximum each student achieved.
And, of course, there’s an online system for awarding karma.
Mind you, it didn’t quite work. At the end of the semester, Alex wrote about how students cheated, and how some withheld karma from others because it would lower their own grade a tiny bit. Also, a good many students decided they weren’t going to score well on the karma anyway, so they just stopped going to class and decided C was good enough.
The next semester, Alex didn’t use karma. Instead he used a point system where you were awarded points for things like unannounced quizzes in class, responding to a survey, participation in online discussions, participating in departmental research, astute comments in class as well as conventional things like exams. There’s a really clear list of how many points you need for a particular grade (scroll down the page linked to above to read all the details) and the points required to get an A are well below the maximum possible number of points – so there are many ways to a good grade.
In Norway university teaching is undergoing huge changes, from being a practically medieval system of lectures with no participation and a huge final exam at the end of the year determining your entire grade, to a much more modern, participatory system that’s a lot more like the American or Australian systems. Now we can grade students in creative ways like those Alex has tried out, but in practice this is going to take a while. Most courses still have either a final exam, a term paper, an oral exam or (and this is a radical new concept here) a portfolio assessment. I think we need to see lots of examples of alternatives to move assessment to being more in line with the process-based, socio-constructivist [INSERT more buzzwords here] teaching we’re beginning to implement.
Thomas
I really like the idea of karma points. Up until now the way things worked was probably more based on your karma than on real ability anyway. But why not just introduce a random factor. Say, at least one person will get an A no matter what he/she has done during the semester. And then see what happens:-)
jon
A lot of the marking I do is based on Creative Writing portfolios / Journals. The only trouble with this kind of assessment is that it is SO time consuming. Of course you can’t really give an exam in CW, but institutions need to be aware how to allocate more hours for “creative” assessment methods.
Jill
Are you serious, Thomas?
Thomas
Well, it is pretty close to what is acutally happening, so it would be interesting to see what effect it would have if it was a known factor in the process students go trough when they decide how much work they want to do on a given course.
torill
I have occasionally, when doing oral assessment, worked until I have seen the auras of the students rather than the student. At that point I have normally decided that I am long overdue for a break. Perhaps I should just go on and focus on their aura, grading them according to that rather than what they said? Would be a LOT quicker! Aural assessment! Sounds like a reform to me…
Alex
I really like the idea of aural assessment, but I think I should get tenure first :). If I get a chance, I’ll write it up: I used a karma system that looked a lot more like Slashdot’s last semester, and it worked fairly well, but students still tend to be down on it.
Ironically, I think the students would prefer that I just give three multiple-choice tests. In terms of workload (I’m at a research university), that’s what my chair would like to see me do, as well. Unfortunately, I get to see some of our “better” students in the graduate program and they don’t know how to write a research paper.
This fall: electronic portfolios ( = blogs) and 360 reviews. We will be organizing the class as a web start-up, and each student will be evaluated by the members of her work-team, and those she is leading and led by. We’ll see how that goes.
Jill
Ooh, the web start up sounds fascinating. Do you read about this stuff somewhere, Alex, or do you just invent new ideas every semester on your own? I’m going to subscribe to your teaching syllabi, I reckon 🙂
And do you mean that you used karma in class a second time?
Rorschach
Some of the advantages in the Slashdot moderation system is that it’s anonymous and that it’s spread over the entire Slashdot user base. And then on top of that is the meta-moderation (moderating the moderations). Since the user base is so large this is effective.
With a small number of users (classroom size), I don’t see how this can work very well. Some students will apply their points judiciously but the opportunity for cheating or just laziness is huge.
— With “n” points I can give those (one point per) to my friends. This can’t work on in a Slashdot environment since the posts are all anonymous.
— I don’t want to bother reading all of the postings so I’ll just give the points to the first three I read. This doesn’t work in Slashdot as the meta-moderations would catch it.
— I’ll withhold my points. Doesn’t work ’cause then you don’t get any more moderation points/privledges.
But all of the reasons it *does* work on /. is because there is such a huge amount of people to share the moderation duties and there is no real competition between the users.
To make this work on a small scale would be very difficult, especially if everyone has a vested interest in “winning” (which is what a curve or straight grading system is, isn’t it?). If there is more than one class involved and graded separately, they could (anonymously) moderate each other. This would lower the chances for assigning points only to your friends but not the lazy problem.
Or, when the person sits down to moderate they are presented with three (or however many) anonymous posts. These are rated ala a modified /. system. Instead of Interesting/Informative/Troll/Funny/Flamebait a simple +1 & -1 modifier could be used. The teacher would act as the final meta-moderator for the posts. This would assure that not all a person’s points would be used for friends. Since the posts could be selected randomly (with discretion so that a person never gets to rate their own), each person in the class gets rated the same number of times.
Add to this a modding memory that after a certain number of +1 or -1 mods (i.e., the person mods everything -1 every time) they lose the option for that mod for a certain number of mods. Heh, force ’em to be nice (or mean)!
This could work in pretty much any class where anonymity could be assured. Yeah, writing styles are pretty telling but in a large enough class there should be enough diversity. Make the moderation part of their grade as well, one point per moderation; an easy 12 points if one per week!
I’m not sure how this could be done with a class on blogging though. Each blog is definitely going to be very unique and have the owners signature all over it. It *could* work in an un-anonymous setting but that would take … maturity. Provide a URL for viewing (heck, put it in a frame!) and then have the person moderate the blog? Hmmm…
george.h.williams
conversation as game
There’s an Ask Slashdot discussion taking place in response to a question regarding computer classrooms and freshman composition. It’s the usual Slashdot mix of helpful and not so helpful comments. Many of the responses address the use of blogs in…