You are currently browsing the monthly archive for December 2008.

I’m getting into the specifics of designing my assessments.  Last night I was thinking about how to structure my blogging assessment.  In my research methods class, I want to use blog posts in the early days to assess their development of knowledge and skills of research.  To do this, I want them to make four posts

  1. Describe the research process (Week 9)
  2. Describe how information from different sources may vary in quality and how to differentiate good quality information from poor quality (Week 10)
  3. (Given the choice of several topics)  Describe your search process including the creation of your search query, databases accessed, sources found and information quality (Week 12)
  4. (Given several research articles of different types)  Assess the quality of the research findings in each case (Week 13)

I think these four posts will help to scaffold them into the task of performing first a joint literature review, then an individual literature review (more on the joint literature review later).

So that’s all fine, but when considering our assessment policies I realised that for every assessment, our students have the opportunity to resit the assessment if they’re marked as not competent on the first attempt.  At first glance, I thought that this was going to create a monster, however with a bit of thinking I’ve come up with a solution which I think might work.

The plan is to give the students two submission dates, one week apart.  To meet competency, the students will need to make a post on the topic, and have that post graded at a minimum of 2 on the blogging rubric.  The marker will need to review the post of everyone in the class briefly, record key points of misunderstanding, and provide individual feedback on the blogs of students who have not met the competency requirement.  They will then create generalised feedback for the class as a whole which clarifies the main areas of understanding.

The students will then have a week before their final assessment to read the posts of other students, to develop their understanding, and update their original post if they like.  My hope is that this period of reflection will help to stimulate cross-fertilisation of ideas.  At the end of this week the blog post will be graded using the complete rubric.  This rubric has been updated based on the feedback of Whitney & Leigh – thanks guys.  Here is the updated version.

This process will be reasonably time-intensive, but I think it should be managable.  It strikes me as a teaching model much more along the lines of George Siemen’s curator.

A curatorial teacher acknowledges the autonomy of learners, yet understands the frustration of exploring unknown territories without a map. A curator is an expert learner. Instead of dispensing knowledge, he creates spaces in which knowledge can be created, explored, and connected. While curators understand their field very well, they don’t adhere to traditional in-class teacher-centric power structures. A curator balances the freedom of individual learners with the thoughtful interpretation of the subject being explored. (Siemens, 2007)

Siemens talks about the role of the curator being to locate and structure an “exhibition” of learning objects or resources which the students are then free to explore.  The teacher as a guide rather than the font of all knowledge.

Carrying on from my last post, I’ve developed  a rubric for assessing the blog posts of my students.

Initially I intended that the rubric should motivate the students to

  1. Develop understanding of key subject areas
  2. Act in ways which will support the development of a learning community

However as I got into the process of nutting-out how this was going to work, I realised that it’s also important that it motivates the students to write well, reflect on their process, and develop good scholarly habits (i.e. referencing and referring to sources outside of the ones provided in class).

The rubric is a work-in-process rather than a finished product.  It contains 5 categories with a total of 20 marks.

One of the problems I’ve identified is that understanding of the subject of the blog post is perhaps not weighted heavily enough.  I think it should probably have a weighting of 2 or so, but I do like that nice round number 20 as a total, so I’d need to either merge two of the other categories or weight two of them with ½ weights.

Perhaps the second option is the best.  This would provide me with a certain degree of flexibility.  If I was using this rubric in a  course where reflection was particularly important, I could weight writing quality and scholarship with ½ weights.  If in another course scholarship and writing quality were particularly important, I could weight community involvement and reflection with ½ weights.

What do you think?

In response to my recent post on my online assessment strategy for 2009, my colleague Leigh has commented “could you describe your ideas for assessment 1 and 3 more? I’m pretty familiar with blogging for assessment in the way you describe, but need a better picture of how you plan to do the other two. Hopefully with a clearer idea, I might be able to suggest something.”

Funnily enough, I’ve got a pretty clear idea about how I could use automated formative assessment (Phase 1) & final summative assessments (Phase 3).  We’ve been using similar assessment strategies in our face-to-face classes for years.  The thing I’m not really familiar with is assessing reflective blogging (apart from a couple of experiences as a student in recent years).

After a fairly superficial exploration of this topic using the net, it seems that most assessment of student blogging is based on the normative assessment model rather than a competency based model.  Here are two rubrics for assessment of student blogging that I think have potential for our course, although neither of them provide direct motivation for reading & commenting on the blogs of other students.

  1. Blog reflection rubric courtesy of San Diego State University Edweb
  2. Designing for flexible learning practice – courtesy of Otago Polytechnic’s Educational Development Centre.

Our course has a mixture of competency and normative-like assessment.  Do you know of any examples of competency-based assessment for reflective blogging?

Another other suggestions for the design of assessment of reflective blogging?

I’m getting into writing assessments for next year, and it’s clear that some aspects of our assessment model need to change. The main drivers for me are the need to increase engagement in online learning activities, workload reduction, and improving feedback.

Assess them and they will come

In my review of how things have gone this year, one of the things that’s really stood out for me is the fact that the level of participation in the learning activities that I set for my students this year was not even close to a level that I would be satisfied with. It’s clear to me that their learning has been impaired as a result (or at least their learning of the material that I wanted them to learn), and I’m pretty sure that the one thing that would have led to more participation would be more assessment.

Taming the workload beast

But we already spend too much time marking assessment! In a recent staff meeting, we talked at length about workload reduction. One thing that takes up a considerable amount of our time is marking assessment. I’m sure that I can design assessments to involve less workload for the assessor.

Anderson describes a range of methods that may act to reduce assessment-related workload for teachers (2008)

  • Automated assessment processes – ranging from formative tests (simple) to virtual labs and simulation exercises (complex)
  • Online automated tutors
  • Use of neural networks & other artificial intelligence methods
  • Peer review (of either students within a specific course, or students within a network of similar courses)
  • Student creation of open educational resources which are then assessed by lifelong learners who are using the resources (Farmer, 2005 as cited by Anderson, 2008)

Formative tests are fairly straightforward to implement. They take some time to set up, but then they’re there to use year on year. I have thought about creating a simulated clinical environment in second life, but at this point, the creation of automatically marked simulations is well out of my financial ballpark, so I’ll move on.

The next two are also a bit too high tech, and high budget.

The last two options are possible if the students of the course are a part of a learning network. (Anderson, 2008). One of my goals for the future is to develop this network, but I think it’ll take at least a couple of years of students moving through the programme before this happens to any particular degree.

Feedback

Feedback is crucial to the learning process, and this is something that we can definitely improve on. Formative tests that provide feedback directly following the student’s performance provide a wonderful development opportunity for students, and I believe that this is one of the real strengths of online education. According to Shepard (2000 as cited in Caplan & Graham, 2008), and Wiggins (2004), providing detailed feedback as close as possible to the performance of the assessed behaviour enhances student learning.

We should strive to “create assessments that provide better feedback by design” (Wiggins, 2004). I was inspired last year by the way in which Montessori school activities are based on this principle. Learning activities can be designed to provide feedback to students in the absence of the teacher. This can be facilitated through instructional design (Wiggins, 2004), or through social networks (Anderson, 2008). In my experience when courses I’ve been engaged with have required blogging, a community of learners has developed, where the learners have begun to support each other in their learning.

3 phase assessment process

After considering all of this, I’ve come up with a three phase assessment process that I think would be fairly ideal for most of our online courses. Phase 1 and 2 here test different grades of knowledge (simple/moderate complexity) & overlap in temporal space.

  1. Automated formative testing to test knowledge of discrete chunks of knowledge.
    Facilitator’s role: establish test, monitor results
  2. Reflective blogging on key concepts in the first ½ of the course. Students required to post on each topic, rewarded for commenting, updating the work they’ve done based on future learning, and referencing.
    Facilitator’s role: Monitor class activity, encourage engagement, Provide generalised feedback
  3. Final theoretical assessment which integrates learning.
    Facilitator’s role: Mark assessment, provide feedback & opportunity for resubmission

Students are therefore rewarded for acting as good community members, are given feedback on their developing understanding & are assessed for their integration of knowledge.

The one slight issue with this model is that if anything, I can see myself doing more assessing in this than I was doing previously. However the formative assessment that I’ll do in the early stages of the courses will be integrated with my teaching, so in effect I believe I could save time with this approach.

What do you think? Can you see any big holes in my thinking here?

References

Anderson, T. (2008). Towards a theory of online learning. In T. Anderson (Eds.). The theory and practice of online learning (2nd ed., p. 45-74). Canada: AU Press, Athabasca University.

Caplan, D., Graham, R. (2008). The development of online courses. In T. Anderson (Eds.), The theory and practice of online learning (2nd ed., p. 245-264). Canada: AU Press, Athabasca University.

Wiggins, G. (2004). Assessment as feedback. Retrieved December 11, 2008 from http://www.newhorizons.org/strategies/assess/wiggins.htm.

The first year of the blended delivery programme is up.  What have I learnt from the experience?  What worked, and what didn’t work?

Teaching online requires different skills to teaching in the classroom

It has been a huge learning curve.  I feel that I’m only now just starting to become competent as a teacher in this environment.  I’m not sure there’s any way around this.  Teaching online is significantly different to teaching the classroom, and perhaps you can only develop the skills for this while doing it.  What do you think?

Don’t take knowledge for granted

I didn’t realise how much I knew until I got another teacher to cover a couple of modules for me early in the year, and this didn’t go very well.  It was a lesson in taking things for granted.  Really I’ve now had 2 years of engaging with online education fairly intensely, and it’s probably unrealistic to assume that others will be easily able to pick up the threads of my teaching without a fair amount of oversight (at least initially).  Next year I’ll be working with another teacher on the facilitation of online learning, and am anticipating an apprenticeship type of arrangement, where we share teaching across the course.  I think this’ll work quite well, although working in this way will mean yet another learning curve.

Structure is very important online

One of the comments that I’ve had is that students have found it difficult to find material at times.  Links to all of the course material that was covered was on the course blogs, and I assumed that these would be easy enough to navigate & find material, but I’m now thinking that this was not a good assumption.  The plan for 2009 is to do all of the primary development through a Google docs structure where there is a main course page that links to all of the learning units.  This should provide a fairly easy to navigate structure for any students who are running behind, or want to refer back in their notes (Here’s an example).  Week by week directions will still be facilitated through the blogs.

(I’ve decided to go with Google docs – I’ve found them much easier to use as a development platform than my experiences with wiki-based development, and anything that saves me time gets a big plus in my book)

Set clear expectations

This is a lesson that I learn every year to some degree.  Setting expectations early on is so important.  Some things that we didn’t define expectations clearly enough for this year were

  • the number of hours committed to course-work (even though I tried to make this clear on more than one occasion)
  • clinic work and the penalties for not meeting our expectations
  • what it means to be a student at the level 4, 5 & 6 stages of our programme
  • participation in online activities (e.g. checking emails every 2 days at the least, participating in all elluminate sessions either at the time or at a later date)

Assessing online activities

Many of the online learning activities were not engaged with by the majority of the students.  I found this hard to understand, and fairly disappointing.  Much as I’d like to think that my students will be motivated by learning in reality, I’ve generally found that at some stage in the year even the best students seem to become mainly motivated by assessment.   So it seems that if I want students to engage with the online learning activities that I’ve set for them, I will need to assess them.  Unfortunately, I can’t see any way to do this apart from defining online learning activities before the start of each programme, which doesn’t leave me with a lot of flexibility.

Making use of personal learning environments (PLEs) is not as straightforward as it might seem

Earlier this year the pageflakes course hub that we used to aggregate all of the course RSS feeds stopped working properly.  After a week or two it became clear that the Pageflakes management had no interest in actually fixing the issue, so I hacked up a solution using iGoogle.  I essentially replicated the page on one of my iGoogle tabs, then shared this tab with all of my students.  It seemed simple enough, and it appeared to work fine once I’d sorted out a few teething problems with a couple of students.  The problem was that I then had no way of telling what my students were seeing.  A couple of weeks before the end of the course, one of my students told me that ever since the change over, they had not been able to view some of the feeds.  Quite soon after this I heard the same thing from another student.  I also found out that several students were unaware that this solution had even been organised for them.  This is despite me sending several emails out through the course email group (our central communication channel) asking people to contact me if they were not able to get the iGoogle page working on their computer!!!  The lesson that I’ve learnt here is that if I want to move away from providing a centralised hub toward getting the students to create their own PLE, I will need to regularly monitor the students to ensure that they are getting the information that they need.

Studying a 20 credit course (1/3 of full-time) over a single semester while setting up and running a new full-time blended delivery programme is completely nutty

Don’t do it!