The idea of testing crowdsourcing against a more traditional design process came up last year (you can read about it here) and we decided to do an experiment as we needed new covers for the two sets of Free High School Science Texts (FHSST) textbooks.

The crowdsourced covers from 99Designs came in first (featured in this blog post):

Here are the full sets of covers for you to compare (you can leave your opinions about which win in the comments if you like):

Grade 10

Grade 11

Grade 12

Results

To be honest, I prefer the crowdsourced Physical Science covers but, in the interests of full disclosure, I managed that process. I am quite happy with the Mathematics covers. If we had had different designers as part of either process the results could have been different so this is certainly not a definite test with a control group but a little subjectivity makes for better coffee-time debate.

Obviously, the key thing to an end product you can live with is how you manage the process no matter which route you go. I have discovered I am very bad at working with an individual designer on a single project. I find it difficult to articulate what I want and what it is that I like and don’t like. These things make me a difficult client, although I feel this is mitigated slightly by the fact that I’m aware of it.

99Designs competitions are more tolerant of my choices and I feel that some designers seem to be able to figure wear I’m going based on what I rate designs and what designs I eliminate. I’ve started to standardise my own process for running competitions on 99Designs.

Competition

• The competition benefits from designers competing and feeding off each other – if you run a blind contest (where designers cannot see each others entries) you’ll lose out on the complex feedback between different design ideas and will basically be running about 30 individual design processes, requiring a lot more feedback and time. If you’re a bad client (like myself) that struggles to give good feedback this will really be difficult.
• For the first 3 days don’t give a rating higher than 3 stars – no matter how much you like the design. Immediately after launching the competition make a comment to this effect otherwise designers will get unhappy. They’ll grumble but live with it if they know it is your policy. If you don’t comment and do it they’ll just think you’re being difficult. You may well find something you really think you like in the first 48 hours but those strong, early contenders have always faded out as the feedback starts to take place.
• Provide feedback often – everyday – the competition only lasts 7 days but will take a couple of hours a day. If you don’t put effort into getting a design you like why should the designers – they don’t have to live with it.
• Prepay the contest – then 99Designs holds the money – you can still get it back but many designers won’t even consider participating if the competition isn’t prepaid. If you are serious about getting a design and have a budget then prove it.
• Once there are a couple of days left AND some designs you could live with – guarantee the contest – this way you can’t get the money back and must choose a winner – but its an added incentive to the designers to get involved. Many of the best seem to wait until the last 48 hours before entering.
• If designs conflict with the brief – eliminate them – and comment to that effect. Don’t let them hang around. If you hate it – eliminate! The competition feeds on what doesn’t get eliminated and on what gets rated highly.
• Give designers you rate individual feedback on their design – not in the general comments section. Don’t pick an individual designers idea and ask everyone to work on something like that.
• Put general feed back or changes to brief in general comments section.
• Don’t be put out if they moan a bit about your poor design choices, it is your design content after all. If they mail you privately just say that you are following the rules and they should go through the competition page – do NOT argue with them about design, especially not privately. Keep it on the competition page.

Brief (this is for a logo design competition)

Give yourself reasons to eliminate the rubbish easily by including things like:

• no drop shadows
• no gradients
• no 3D effects
• no stock images
• no proprietary fonts (otherwise you have to purchase)
• at most 2 colours (maybe 3
• logo must scale down
• logo must work in B&W
• should be balanced
• colour logo must be presented on white background (if they start
  using textured backgrounds to make the logo look good it is hard to
  compare etc.)

Thats all before you even say anything about what you are looking for. You can easily find much more comprehensive lists for rules for logo design competitions with a quick search online.

Then tell them what you want and include:

• project elevator pitch and link to site or descriptive material if
  you have.

• icon and/or word together
• ideas of colours if you have any
• whether icon should stand on its own
• ideas of fonts

Make sure you ask for:

• Colour version
• B&W version
• Vector files (I’d ask for .ai and .svg files)

I think that crowdsourcing is also a lot more fun than dealing with a single designer and you really do see a lot of different concepts in a very short space of time. I’m sure there is an argument to be made that a longer process might lead a more mature result but I’m sure a counter-argument exists built on intensity and focus for the person running the competition. Ultimately, as with everything in life, you have to make your own decisions and live with them. Luckily taking a chance on a logo for $500 isn’t very risky and worth a shot. The more you pay the more attention you get but for $500 for a logo design I am barely able to keep up with my day job and the competition.

I have finally upgraded the FHSST website to Drupal 6.15. This took me a lot longer than expected because the year got off to a completely rampant start and “spare” time just didn’t exist. I also had fallen very far behind on Drupal upgrades so I had to start with an upgrade to Drupal 5.

Now we can revisit the new functionality I mentioned in a previous blog. The idea was to make it much easier for people to submit errata and manage the corrections, submit contributions and capture testimonials. This will require significant theme changes which is why the site is currently using the default Drupal theme.

I’ve learnt not to even attempt to suggest deadlines for the new functionality to be rolled out and the new theme designed and implemented but it’ll be there as soon as I can get it up. The functionality was being looked into by Telementa and I hope to resurrect that as it seemed like the functionality would be quite quick and simple for them to implement. The bottleneck was myself upgrading the site to Drupal 6.

We’ve chosen the covers for the science textbooks as part of our experiment of using crowdsourcing versus traditional design. We’re still waiting for the traditional design covers to come in for the maths books but I thought I’d share the science ones so long.

We had many submissions and there were some great ones but these were the most popular amongst members of the FHSST and OpenPress teams. I’ll write up a more detailed follow-up post when we’ve got the maths ones in.

We need much better management of the feedback we receive for the Free High School Science Texts project. It needs to be simple and fast and allow us to manage it easily. To help with this we’re looking to have a small feedback module written for the site. We’ll use it once we’ve upgraded the site to Drupal 6.x.

I’m looking for some development quotes and thought I might get some additional exposure if I post the draft specification on my blog. If you’re a Drupal person and would like to have a crack at this feel free to contact me with any clarifying questions or a quote for the work. I intend to make a decision by the middle of next week (18th November) so you’ve got a week to get the information to me.

FHSST Feedback Module

The FHSST site allows authors to build books in LaTeX. The original goal was to have many LaTeX-competent authors working on many sections. We have learnt that very few people are comfortable with LaTeX and PSTricks. To better accommodate people who still want to make contributions but not get their hands dirty we must provide a simple way to give the necessary input.

We would like a feedback module to be written for Drupal to help with this. This module should allow three types of feedback:

1. errata

2. suggested contributions

3. testimonials

Errata

Each erratum needs to contain:

• erratum title (mandatory)

• detailed erratum information (mandatory)

• book for which it is submitted (mandatory)

• book version for which erratum is applicable (mandatory)

• book section in which error occurs (mandatory)

• book page number (optional)

• submitter’s:

  • name and surname (mandatory)

  • email address (mandatory)

  • role, one of:

    • learner

    • educator

    • parent

    • other

• timestamp of submission (recorded automatically)

• status (only changeable by coordinator – starts in pending):

  • pending

  • corrected

  • disputed

• timestamp of resolution (i.e. corrected or disputed state change – recorded automatically)

• person changing state (recorded automatically)

When submitted, an erratum must go into a queue and the coordinators of the relevant book should receive an email notification that it has been submitted. They can then login and review the erratum. Once they have processed it they must be able to set it to either corrected or disputed in the case where the erratum is actually incorrect.

A page should be provided where users can see all errata submitted, the submitters’ name, the relevant book and version, and the status of the errata. The title should link to the full erratum view.

A block should be provided where users can see the 5 most recent errata submitted (title, book and status).

Suggested Contributions

This is to allow general users to submit a contribution to be included in the book by a LaTeX-enabled author. Each contribution should have:

• contribution title (mandatory)

• detailed contribution (mandatory)

• attached files (optional)

• book for which it is submitted (mandatory)

• book section (optional)

• submitter’s:

  • name and surname (mandatory)

  • email address (mandatory)

  • role, one of:

    • learner

    • educator

    • parent

    • other

• timestamp of submission (recorded automatically)

• status (only changeable by coordinator – starts in pending):

  • pending

  • incorporated

  • discarded

• timestamp of resolution (i.e. incorporated or discarded state change to be recorded automatically)

• person changing state (recorded automatically)

When submitted a contribution must go into a queue and the coordinators of the relevant book should receive an email notification that it has been submitted. They can then login and review the contribution. Once they have processed it they set it to either incorporated or discarded state in the case where the contribution cannot be used.

A page should be provided where users can see all contributions submitted, the submitters’ name, the relevant book and version, and the status of the contribution. The title should link to the full contribution view.

A block should be provided where users can see the 5 most recent contributions submitted (title, book and status).

Testimonials

This is simply to collect information about how the content has been used.

• testimonial title (mandatory)

• detailed testimonial (mandatory)

• book for which it is submitted (optional)

• book section (optional)

• submitter’s:

  • name and surname (mandatory)

  • email address (mandatory)

  • role, one of: (mandatory)

    • learner

    • educator

    • parent

    • other

• timestamp of submission (recorded automatically)

Administration

For the feedback module the books for which feedback is being received need to be configured. This should happen from the module administration page and not depend on the book module of Drupal. The administrator will add the books for the module and then add, on a book-by-book basis:

• versions

• chapters and sections

• coordinators

When a book is added at least one version number is required. For each version number provided, a list of chapters and sections is required. A book will have multiple versions, each with their own unique chapter and section listing. This listing can be a simple textarea where each line is a chapter or section heading.

Each book must also have a list of coordinators who are users on the system. Coordinators should be able to access a list of all errata and contributions for their books. They should be able to view them and change the status if necessary. The time when the status is changed should be recorded.

Special Remarks

All the feedback types should be node types and should integrate into the Drupal CMS properly so that they can be integrated with other modules designed to manipulate nodes. The theming information should also be separated so that they can be re-themed.

A special page should be provided that can be used as the homepage. This page should have a customisable body of full html and then have forms for all three feedback types. This will be presented to non-authenticated users as the homepage allowing them to provide the feedback directly from the front page. We would like jQuery/jQuery UI integration in this page so that a button can be shown for each type of feedback and if clicked the form must render as a modal window. Submission using these forms should be via AJAX and should trigger an update on the blocks for the content types if they are shown.

To simplify the forms when a user selects a book the relevant versions should be loaded. Once the version is selected only the relevant sections should be loaded. Validation for all form fields should be implemented.

The module should be written in such a way to conform to the Drupal 6.X API and should only have dependencies on well established modules (modules that are actively maintained and have demonstrable support and traction within the Drupal community).

We would release all the code under the Gnu General Public Licence.

As part of the launch of OpenPress, the FHSST Grade 10 Mathematics book is getting printed. We felt that we needed a fresh set of covers for the books. They will be printed with a colour cover and we felt that the all the Maths covers should have a theme; similarly for the Physical Science books.

The covers will need to be available in the next 10 days or so and we were discussing the best way to get them done. The obvious way to go is to commission the work from a designer and go with that. We also considered the option of crowdsourcing the covers and have had a lot of debate about the merits of crowdsourcing.

To resolve the issue, once and for all (maybe), we’ve decided to run our own internal competition. The prize will be bragging rights as the optimal process for future covers for OpenPress / FHSST texts. We are going to allocate the same budget to the Maths and Physical Science covers, but we are going to have the Maths set commissioned by a regular design firm and we’ll get the Physical Science ones created on 99Designs.

Although it isn’t really a competition it will be fun to see the results. The only way to keep the results comparable is to give both processes the same:

• budget,
• brief,
• quantity and quality of feedback, and
• the same timeline.

I’ll post the brief a little later and the results as well so you can decide which you think is the best set and whether we’ve chosen the best process based on this little experiment. If nothing else this should be fun.

I think that it is fitting that I start my new site off with an update on the project responsible for the fork in my life-trajectory which leads me to be writing this blog. Since the release of version 0 nothing visible has happened on the FHSST website. It sounds like I might be leading up to an announcement of great behind-the-scenes work but I’m not (but one day I will!), at least not on the FHSST front. There has been much activity on the broader open-educational-material front and I’ll blog about that shortly.

There has been some FHSST activity behind the scenes though, we’ve received emails with a number of errata for which we are really grateful. One of the primary benefits of Open Educational Resources (OERs) is that they can be updated rapidly. The errata will be corrected in the next version of the books.

At Science Festival Africa 2009 we also learned of a number of people who had been using the books for tutoring and self-learning. It was always the intention of the project that a resource that could support other initiatives be created. We have always felt that if the redundancy of producing material could be overcome many outreach organisations could be more effective. One notable inclusion on this list are the education outreach activities of the Students’ Health and Welfare Centres Organisation (SHAWCO) at the University of Cape Town who have been using FHSST content.

This afternoon I had a very interesting conversation with Jacky Hood and Kathi Fletcher about the state of the FHSST books. Jacky, and the Community College Open Textbook Project (CCOTP), have been re-purposing FHSST content to support their work to provide openly licenced textbooks for community colleges. As part of the process they’ve reviewed the FHSST content and then, as a first step, reworked one of the Mathematics books to suit their needs. They are looking for funding to continue this work, covering the full set of FHSST books.

Jacky also told me about a Community College lecturer (I don’t have a name) in New York who has been using the Statistics chapters of the FHSST Mathematics books to support her lectures.

Again, we have provided a resource that the CCOTP can use and the best part, in my opinion, is that the work that they put in, over and above what FHSST did, will be released on the Connexions site so that we can build further on that for use in South African Further Education and Training (FET) Colleges (our equivalent of the community colleges). Derivative works are being fed back into the community, instead of being lost, which will really help accelerate OER development. It often (always) becomes a debate heated enough to rival your pick of the religious wars but I think that derivative works which are voluntarily made openly available make our sharing community (commons if you must) stronger than derivative works which are mandated to be shared by certain copyright licences.

We are also in the process of uploading all the FHSST books onto the Connexions site so that it will be much easier for other organisations to adapt them to their specific needs as well as manage derivative works. Rory Adams has uploaded all of the books but a second pass is still required to ensure that our environments have been converted properly. Getting this done is high on my priority list, unfortunately so are many other things right now, and I hope to get it done in a week or two.

So what’s next? We need to:

• be more efficient at getting feedback from users so we can improve the books,
• advertise and link organisations using the books so that learnings/ideas/strategies can be exchanged,
• improve general awareness of the books existence,
• correct the errata,
• update the FHSST website, when that big FHSST announcement arrives it will includes a system for submitting, tracking, managing and publishing errata so that the status of the individual books is clear and readily available,
• finish uploading the FHSST books to Connexions (they’ve got a better authoring platform than we do and they maintain it), and
• organise that printing happens (expect MANY more blog posts on this topic soon!).

The good news is that it turns out that the FHSST content has already proven quite useful to a number of people, now we just need to harness that to add even more momentum to the OER-movement. It is easier to find the time to work on the project knowing that we’ve already been successful in helping other organisations, projects and people.

FHSST (Free High School Science Texts) is a project that aims to provide free science and mathematics textbooks for Grades 10 to 12 science learners in South Africa.