I’ve mentioned wanting to develop an open assessment bank to complement my other projects and that it was recently approved. I’m happy to report that development is well underway. We are calling this project FullMarks.

The logo and website are still pending and, of course, we’re crowdsourcing their design. Feel free to take a look at 99Designs where we’ve just opened the competition.

We are extending the work done by the Human Sciences Research Council (HSRC) on a closed bank (but open source software). We have the blessing of the HSRC to proceed but, in the interests of full disclosure, with a strong word of caution that an assessment bank can be a blunt intstrument. A bank will save teachers time and increase the average quality of items but their research has shown that, unless teachers act on the identification of learners’ strengths and weaknesses, little real impact is felt when considering learners’ understanding.

I’m happy to charge on ahead with development of this particular blunt instrument, not because I’m irresponsible (trust me, if I wanted to be irresponsible I wouldn’t give myself another project to worry about) but because I feel that their concerns provide the very justification for the open bank. I’ll have a crack at a brief explanation of why I think this is so.

Saving Teachers’ Time

Have you spoken to a teacher in South Africa lately? Time is their biggest concern. Now I expect to have some people disagree with that, they’ll cite resources, lack of support etc. but the very first thing teachers tell me is that they don’t have enough time. Sure, I hear about resources, poor support, curriculum obfuscation, and all sorts of other things, but time is always first.

This isn’t a trick to get teachers’ attention, there is no question that time is an issue so we should work to save them time if we can. The time saving decreases stress and leads to focus that can be deployed elsewhere.

Improved Quality of Assessment Items

We know that the sharing of assessment items can lead to an increase in the average quality of items used in the classroom. Again, there is no question that this is a good thing. As the number quality items grows, teachers will need to develop fewer questions, allowing them to spend more time being creative and focusing on quality rather than quantity.

Open Assessment Bank

Just saving time and improving quality can still lead to little impact on the learner as the HSRC point out. In their analysis, however, they’ve been looking at a closed bank where teachers are passive consumers. My bank will be an open bank allowing questions to be viewed, added, duplicated, and adapted. We will also allow for feedback in a few ways. Users will be able to vote as to whether or not a question’s solution is correct, they will be able to rate questions and they will be able to comment on questions and answers.

Feedback is the key to making sure that the assessment bank is an effective tool (as opposed to a blunt instrument). A lot can be learnt from receiving feedback from your peers as to whether or not the solutions to your questions are correct. Even more can be learnt from a detailed discussion around whether or not a question tests the assessment standards that it is tagged with and how to respond if learners struggle with the question. This will also help teachers further their understanding of the outcomes and appropriate assessment of them. Teachers just need to participate in the community, sharing items, ideas and feedback.

The final possibility of improved understanding of the outcomes and how to respond to learners’ responses, correct and incorrect, is what gives FullMarks so much more potential impact. It is the open aspect that is crucial. My job is to create an environment where teachers feel there is real benefit to contributing. At first this will be to save time, but over time it will evolve to active participation in discussions around how to assess according to the standards and respond when learners struggle.

Today I received the good news that the assessment bank I mentioned previously has been approved and we will begin building it as soon as possible, hopefully on Monday.

We are not going to re-invent the wheel and will be extending (NOT forking) an already existing assessment bank that has been developed by the Human Sciences Research Council (HSRC) which they are calling TARMii. The HSRC has indicated that they have no objection to us creating an open bank as they’ve released their source code under the Gnu General Public Licence (GPL). The HSRC implementation locks down the authoring side of the bank to a few specific users at the HSRC. Their bank is populated with Grade 4-6 (Intermediate phase) items for English and Mathematics and will allow teachers to:

• create class-lists,
• generate a test extremely rapidly, providing a question paper and memorandum,
• capture learners results for the tests, and
• generate reports on a learner-by-assessment standard basis.

We will keep all this functionality, which they have already tested, as it is really useful to teachers but we want to allow them to benefit from such a tool for all subjects and grades as quickly as possible. The HSRC will spend some time trialing the site with teachers and the items they have created before extending to other learning areas/subjects and grades.

To make this available to everyone we need to open up the authoring side and provide some means for community vetting. Our extension to the assessment bank software will cover:

• community feedback:
  • ratings
  • comments
  • verify as correct
• authoring:
  • simplify UI
  • allow question cloning for minor modifications
  • add version control
• test generation:
  • allow test to be tweaked (not currently available)
  • browsing for additional/replacement questions.

This compliments Siyavula as it will bring more people into a the OER space but channel time and energy that Siyavula is not trying to channel as teachers spend a lot of time sharing, adapting and creating assessment items. The possibility of integrating the two exists as TARMii has made use of a number of tools/standards used or developed by Connexions, for example the WYSIWYG MathML editor, and the same development team, Upfront Systems, that has helped us extend the Rhaptos platform, on which Connexions is built, built TARMii and will do the extensions for us.

The community of teachers in SA is chomping at the bit for an assessment bank. We have had multiple requests for bank software that communities are already prepared to populate themselves. Providing the different groups with a single tool will allow them to feed off each others energy and allow us to begin with a bang. Working with existing communities also makes the tool much more sustainable.

It is great to be building a tool in response to a real demand as a lot less energy needs to go into advocacy and messaging.

All teachers in South Africa need to have the time, focus, support and resources necessary to deliver the curriculum effectively. I would like to ensure that they have access to an online assessment bank tool plus toolkit to generate tests, capture and analyse learners’ results and provide detailed reporting on a learner-by-learner, class or national basis as to learning outcomes achieved. The assessment bank would be built according to Open Education philosophy encapsulated in the Cape Town Open Education Declaration.

If we support communities of teachers sharing, adapting and enhancing assessment items in iterative, open and transparent ways it will save individual teachers’ time, provide assessment items of the highest quality and enhance the sharing of innovative ideas. The project would need to provide the simple, intuitive tools, support and advocacy necessary to allow teachers to achieve a critical mass of items. Furthermore, assessment items would need to be tagged with meta-data so that learners’ results can be captured and analysed allowing teachers to access reports that pin-point individual learner’s weaknesses and strengths, allowing for targeted interventions to support individual learners’ needs.

I have already been on the receiving end of many requests for an assessment bank, more so now that we are running teachers’ workshops as part of Siyavula. The original software development plan for Siyavula called for the integration of a new resource type into Connexions that would allow the automatic generation of tests. In retrospect it was too ambitious at that time but circumstances have changed and we are now in a position to deploy an assessment bank relatively easily that meets the criteria outlined above.

In embarking on this little exercise I think that we could achieve differing levels of success, all useful:

1. The basic success criterion is that teachers  regularly use assessment items from the bank. For sustainability, it is important that we see items being added by teachers. If we see daily access by teachers and contribution statistics comparable to 1% of visits then we would deem the basic assessment bank as a reference for teachers a success and likely to be sustainable.
2. Should regular test generation take place on the site, it would indicate that we have sufficient items in a number of subject areas to be a useful tool for a new teacher and could be certain that teachers were saving time by using it and that we are making an impact in the classroom.
3. If the tools for capturing results and generating reports are being used it would indicate that the tool  is useful in helping to guide day-to-day classroom practice and that teachers are now more aware of individual learners’ needs.

Teachers contributing items would also have joined the family of OER authors and it would significantly lower the barrier to getting them to contribute modules to Connexions by illustrating the benefits of sharing, acclimatising them to working online and ensuring that they feel like they are part of a much larger, virtual community. I would expect to see teachers who are comfortable authoring items on the assessment bank quickly stepping up to working on modules on Connexions.