The idea of testing crowdsourcing against a more traditional design process came up last year (you can read about it here) and we decided to do an experiment as we needed new covers for the two sets of Free High School Science Texts (FHSST) textbooks.

The crowdsourced covers from 99Designs came in first (featured in this blog post):

Here are the full sets of covers for you to compare (you can leave your opinions about which win in the comments if you like):

Grade 10

Grade 11

Grade 12

Results

To be honest, I prefer the crowdsourced Physical Science covers but, in the interests of full disclosure, I managed that process. I am quite happy with the Mathematics covers. If we had had different designers as part of either process the results could have been different so this is certainly not a definite test with a control group but a little subjectivity makes for better coffee-time debate.

Obviously, the key thing to an end product you can live with is how you manage the process no matter which route you go. I have discovered I am very bad at working with an individual designer on a single project. I find it difficult to articulate what I want and what it is that I like and don’t like. These things make me a difficult client, although I feel this is mitigated slightly by the fact that I’m aware of it.

99Designs competitions are more tolerant of my choices and I feel that some designers seem to be able to figure wear I’m going based on what I rate designs and what designs I eliminate. I’ve started to standardise my own process for running competitions on 99Designs.

Competition

• The competition benefits from designers competing and feeding off each other – if you run a blind contest (where designers cannot see each others entries) you’ll lose out on the complex feedback between different design ideas and will basically be running about 30 individual design processes, requiring a lot more feedback and time. If you’re a bad client (like myself) that struggles to give good feedback this will really be difficult.
• For the first 3 days don’t give a rating higher than 3 stars – no matter how much you like the design. Immediately after launching the competition make a comment to this effect otherwise designers will get unhappy. They’ll grumble but live with it if they know it is your policy. If you don’t comment and do it they’ll just think you’re being difficult. You may well find something you really think you like in the first 48 hours but those strong, early contenders have always faded out as the feedback starts to take place.
• Provide feedback often – everyday – the competition only lasts 7 days but will take a couple of hours a day. If you don’t put effort into getting a design you like why should the designers – they don’t have to live with it.
• Prepay the contest – then 99Designs holds the money – you can still get it back but many designers won’t even consider participating if the competition isn’t prepaid. If you are serious about getting a design and have a budget then prove it.
• Once there are a couple of days left AND some designs you could live with – guarantee the contest – this way you can’t get the money back and must choose a winner – but its an added incentive to the designers to get involved. Many of the best seem to wait until the last 48 hours before entering.
• If designs conflict with the brief – eliminate them – and comment to that effect. Don’t let them hang around. If you hate it – eliminate! The competition feeds on what doesn’t get eliminated and on what gets rated highly.
• Give designers you rate individual feedback on their design – not in the general comments section. Don’t pick an individual designers idea and ask everyone to work on something like that.
• Put general feed back or changes to brief in general comments section.
• Don’t be put out if they moan a bit about your poor design choices, it is your design content after all. If they mail you privately just say that you are following the rules and they should go through the competition page – do NOT argue with them about design, especially not privately. Keep it on the competition page.

Brief (this is for a logo design competition)

Give yourself reasons to eliminate the rubbish easily by including things like:

• no drop shadows
• no gradients
• no 3D effects
• no stock images
• no proprietary fonts (otherwise you have to purchase)
• at most 2 colours (maybe 3 ;)
• logo must scale down
• logo must work in B&W
• should be balanced
• colour logo must be presented on white background (if they start
  using textured backgrounds to make the logo look good it is hard to
  compare etc.)

Thats all before you even say anything about what you are looking for. You can easily find much more comprehensive lists for rules for logo design competitions with a quick search online.

Then tell them what you want and include:

• project elevator pitch and link to site or descriptive material if
  you have.

• icon and/or word together
• ideas of colours if you have any
• whether icon should stand on its own
• ideas of fonts

Make sure you ask for:

• Colour version
• B&W version
• Vector files (I’d ask for .ai and .svg files)

I think that crowdsourcing is also a lot more fun than dealing with a single designer and you really do see a lot of different concepts in a very short space of time. I’m sure there is an argument to be made that a longer process might lead a more mature result but I’m sure a counter-argument exists built on intensity and focus for the person running the competition. Ultimately, as with everything in life, you have to make your own decisions and live with them. Luckily taking a chance on a logo for $500 isn’t very risky and worth a shot. The more you pay the more attention you get but for $500 for a logo design I am barely able to keep up with my day job and the competition.

The development of the FullMarks open assessment bank is well underway. We’re on track for a launch early next year. Without a beta version of the site up it is hard to show you the progress at this time but we’re about 20 man-days away from the website being ready to show you.

We have got the look of the front-page tied down though and here is a screenshot of what www.fullmarks.org.za will look like early in 2010:

Once the site is up we’ll be organising some content uploading sprints to make sure we get the breadth of subjects covered as quickly as possible.

We’ve chosen the covers for the science textbooks as part of our experiment of using crowdsourcing versus traditional design. We’re still waiting for the traditional design covers to come in for the maths books but I thought I’d share the science ones so long.

We had many submissions and there were some great ones but these were the most popular amongst members of the FHSST and OpenPress teams. I’ll write up a more detailed follow-up post when we’ve got the maths ones in.

I’ve just launched another competition on 99Designs to develop a cartoon character that I’ll use to brand this blog. I just can’t bring myself to post a photo in the heading and I felt that a cartoon would be reasonable given that I’m already mixing work and personal stories in one place. The distinction is rather vague anyway.

Here is the design brief (see the site for the full details and attached images if you are keen on having a crack):

brief overview

I’m a Fellow at the Shuttleworth Foundation where I work on a bunch of projects all related to using technology to drive open education in South Africa. Projects like supporting communities of practice that share content online, developing open assessment banks of items to save teachers time and working with volunteers to write textbooks that are free and open.

I keep a blog related to all my projects and, contrary to popular wisdom, personal stories. I use categories to separate the two.

I can’t face using a “thoughtful” photo of myself as the headline image to my blog and a cartoon depiction would be more fitting, especially seeing as I’m mixing my personal and work blogs. I’m quite taken with this crowdsourcing idea so I’m here to try to get the cartoon made.

brand name

Mark Horner

target audience

The audience is the general public, a mixture of work related visitors, information seekers and those seeking personal updates. Some of the cartoon features will resonate with work related visitors and some will resonate with those interested in personal stories.

requirements

I want a cartoon that will ultimately be part of the header on this blog: www.markhorner.net

Cartoon content all clearly visible:

1) My cartoon self on a camping chair with a laptop (Lenovo T500 if it mattered ;)
2) A bottle and glass of red wine
3) A potjie on a fire in the background, this image captures the idea quite well:

http://www.south-africa-tours-and-t…africa.jpg

Cartoon styles I like:

http://media.smashingmagazine.com/i…/final.jpg

I’m not tied to the style, but I’d like something that will scale down reasonably well, not to an icon size but possibly to fit on a business card.

I’ve attached some pics and sketch to help out. I’m not a designer so please use your own judgement as to the best way to include the elements and don’t feel you have to stick to the attached sketch – its just an idea.

The final files need to be in a vector format suitable for printing purposes as I may need to include it on printed media as well as png files for web-usage.

We need much better management of the feedback we receive for the Free High School Science Texts project. It needs to be simple and fast and allow us to manage it easily. To help with this we’re looking to have a small feedback module written for the site. We’ll use it once we’ve upgraded the site to Drupal 6.x.

I’m looking for some development quotes and thought I might get some additional exposure if I post the draft specification on my blog. If you’re a Drupal person and would like to have a crack at this feel free to contact me with any clarifying questions or a quote for the work. I intend to make a decision by the middle of next week (18th November) so you’ve got a week to get the information to me.

FHSST Feedback Module

The FHSST site allows authors to build books in LaTeX. The original goal was to have many LaTeX-competent authors working on many sections. We have learnt that very few people are comfortable with LaTeX and PSTricks. To better accommodate people who still want to make contributions but not get their hands dirty we must provide a simple way to give the necessary input.

We would like a feedback module to be written for Drupal to help with this. This module should allow three types of feedback:

1. errata

2. suggested contributions

3. testimonials

Errata

Each erratum needs to contain:

• erratum title (mandatory)

• detailed erratum information (mandatory)

• book for which it is submitted (mandatory)

• book version for which erratum is applicable (mandatory)

• book section in which error occurs (mandatory)

• book page number (optional)

• submitter’s:

  • name and surname (mandatory)

  • email address (mandatory)

  • role, one of:

    • learner

    • educator

    • parent

    • other

• timestamp of submission (recorded automatically)

• status (only changeable by coordinator – starts in pending):

  • pending

  • corrected

  • disputed

• timestamp of resolution (i.e. corrected or disputed state change – recorded automatically)

• person changing state (recorded automatically)

When submitted, an erratum must go into a queue and the coordinators of the relevant book should receive an email notification that it has been submitted. They can then login and review the erratum. Once they have processed it they must be able to set it to either corrected or disputed in the case where the erratum is actually incorrect.

A page should be provided where users can see all errata submitted, the submitters’ name, the relevant book and version, and the status of the errata. The title should link to the full erratum view.

A block should be provided where users can see the 5 most recent errata submitted (title, book and status).

Suggested Contributions

This is to allow general users to submit a contribution to be included in the book by a LaTeX-enabled author. Each contribution should have:

• contribution title (mandatory)

• detailed contribution (mandatory)

• attached files (optional)

• book for which it is submitted (mandatory)

• book section (optional)

• submitter’s:

  • name and surname (mandatory)

  • email address (mandatory)

  • role, one of:

    • learner

    • educator

    • parent

    • other

• timestamp of submission (recorded automatically)

• status (only changeable by coordinator – starts in pending):

  • pending

  • incorporated

  • discarded

• timestamp of resolution (i.e. incorporated or discarded state change to be recorded automatically)

• person changing state (recorded automatically)

When submitted a contribution must go into a queue and the coordinators of the relevant book should receive an email notification that it has been submitted. They can then login and review the contribution. Once they have processed it they set it to either incorporated or discarded state in the case where the contribution cannot be used.

A page should be provided where users can see all contributions submitted, the submitters’ name, the relevant book and version, and the status of the contribution. The title should link to the full contribution view.

A block should be provided where users can see the 5 most recent contributions submitted (title, book and status).

Testimonials

This is simply to collect information about how the content has been used.

• testimonial title (mandatory)

• detailed testimonial (mandatory)

• book for which it is submitted (optional)

• book section (optional)

• submitter’s:

  • name and surname (mandatory)

  • email address (mandatory)

  • role, one of: (mandatory)

    • learner

    • educator

    • parent

    • other

• timestamp of submission (recorded automatically)

Administration

For the feedback module the books for which feedback is being received need to be configured. This should happen from the module administration page and not depend on the book module of Drupal. The administrator will add the books for the module and then add, on a book-by-book basis:

• versions

• chapters and sections

• coordinators

When a book is added at least one version number is required. For each version number provided, a list of chapters and sections is required. A book will have multiple versions, each with their own unique chapter and section listing. This listing can be a simple textarea where each line is a chapter or section heading.

Each book must also have a list of coordinators who are users on the system. Coordinators should be able to access a list of all errata and contributions for their books. They should be able to view them and change the status if necessary. The time when the status is changed should be recorded.

Special Remarks

All the feedback types should be node types and should integrate into the Drupal CMS properly so that they can be integrated with other modules designed to manipulate nodes. The theming information should also be separated so that they can be re-themed.

A special page should be provided that can be used as the homepage. This page should have a customisable body of full html and then have forms for all three feedback types. This will be presented to non-authenticated users as the homepage allowing them to provide the feedback directly from the front page. We would like jQuery/jQuery UI integration in this page so that a button can be shown for each type of feedback and if clicked the form must render as a modal window. Submission using these forms should be via AJAX and should trigger an update on the blocks for the content types if they are shown.

To simplify the forms when a user selects a book the relevant versions should be loaded. Once the version is selected only the relevant sections should be loaded. Validation for all form fields should be implemented.

The module should be written in such a way to conform to the Drupal 6.X API and should only have dependencies on well established modules (modules that are actively maintained and have demonstrable support and traction within the Drupal community).

We would release all the code under the Gnu General Public Licence.

As part of the launch of OpenPress, the FHSST Grade 10 Mathematics book is getting printed. We felt that we needed a fresh set of covers for the books. They will be printed with a colour cover and we felt that the all the Maths covers should have a theme; similarly for the Physical Science books.

The covers will need to be available in the next 10 days or so and we were discussing the best way to get them done. The obvious way to go is to commission the work from a designer and go with that. We also considered the option of crowdsourcing the covers and have had a lot of debate about the merits of crowdsourcing.

To resolve the issue, once and for all (maybe), we’ve decided to run our own internal competition. The prize will be bragging rights as the optimal process for future covers for OpenPress / FHSST texts. We are going to allocate the same budget to the Maths and Physical Science covers, but we are going to have the Maths set commissioned by a regular design firm and we’ll get the Physical Science ones created on 99Designs.

Although it isn’t really a competition it will be fun to see the results. The only way to keep the results comparable is to give both processes the same:

• budget,
• brief,
• quantity and quality of feedback, and
• the same timeline.

I’ll post the brief a little later and the results as well so you can decide which you think is the best set and whether we’ve chosen the best process based on this little experiment. If nothing else this should be fun.

I wrote a blog post a little while ago about crowdsourcing the OpenPress logo. That was before we’d actually finished the process and I just wanted to take a quick moment to reflect. This post is long overdue so I’ll keep it short and just hit the highlights. For the record, you’ll find a ton of people for and a ton of people against crowdsourcing, if you want a logo, its an option and the better you manage the process the better your result will be (paying more also helps).

The experience of crowdsourcing the logo on 99Designs was interesting, fun and ultimately quite exhausting. For the record, we are happy with the logo that we have and, as we’re moving forward with the project, here is the logo for your viewing pleasure:

As with anything design-related there are always going to be opinions so it will be interesting to hear what other people think of the logo. We had a total of 140 designers submit 840 different designs. Some extremely good, some incredibly bad. We prepayed a prize of $500 and, as soon as we had a number of logos we felt we could live with we, guaranteed the prize.

Prepaying and guaranteeing a prize are incentives for designers to participate in your competition. We felt that guaranteeing the prize about half-way through the competition would be a strong, positive indication that the designs we had not yet eliminated were serious candidates.

The fact that design is very subjective is a key thing to bear in mind when using a crowdsourcing tool like 99Designs. That is not to say there are aren’t some well established guidelines for design. We found this list of 45 Rules of Great Logo Design to contain a number of good basic checks even though we broke a few. I’d recommend looking through that list (or any list for that matter as there are many) and then making as specific a specification for your logo as possible before launching your competition.

To get the most out of 99Designs do:

• realise that not every designer who will submit a design is really a designer (an amateur with MSPaint can still sign up)
• specify your constraints very clearly including things you’re not sure about (they need to know what parameters they have to play with)
• provide regular feedback (to guarantee convergence)
• be consistent in your treatment of logos
• update your specification early on if you want to stop trends developing
• eliminate designs that you don’t like in a timely manner
• make a checklist of basic things and stick to them, things like:
  • must work in B&W
  • must scale well
  • etc.
• prepay your prize

and definitely don’t:

• don’t spend the first day refreshing your browser, we were really excited by the first 20 designs discussing them all in detail but when you’re going to get 840 its not worth getting too bogged down early
• get involved in any offline discussions with designers about their designs
• consider designs not submitted through the 99Designs site
• try to provide a detailed response to everyone
• get too excited in the first day

If you provide regular feedback that is consistent and you make it available to all the designers by keeping it in the competition you are more likely to get something you like as well as keep the designers interested and keep frustration levels down. If you oscillate and are inconsistent designers will get irritated and move on. The designers definitely feed off each other and the whole contest becomes a dynamic system you get to prod and with many designers tweaking/riffing off each other tempers can flare-up. As long as you’re consistent and following the rules it should converge.

If you do embark on a crowdsourcing experience brace yourself for a busy week and have a look at this post on bad logos to keep your humour up.

For over a year now a new project has been brewing, OpenPress. It isn’t a secret but nobody took the bait and built it so we are going to do it, something that I am over the moon about. The ‘we’ that I am talking about is Roché Compaan, Steve Song, and myself. We’ve just started out on this journey and there will be a lot more written about this in coming months.

We need to establish a bit of a brand identity and we’re currently sourcing a logo. Rather than tying ourselves to one designer, we want to access the largest possible creative pool we can find to try to come up with something inspired. Given that we want to run this project as cost-effectively and efficiently as possible and an army of designers doesn’t come cheap, we’ve agreed to try the crowdsourcing (a definition I like) approach. In this approach one essentially makes a bounty available for the final product and makes an open call so that any and all can have a crack at the problem.

There are web-services that facilitate the process by managing the bounty, providing a central location where a community of designers convenes and provides a loose framework for managing the process, keeping things roughly fair. We decided to use 99 designs for our process but also considered crowdSPRING. I was dragging my heels a little bit about getting the competition started but Roché jumped straight in and ran a contest for a new web-site design for Upfront Systems. The results of which you can see here.

Yesterday at around noon we launched a competition for the OpenPress logo. In the web-site design contest Roché launched it took a few days before designs really came in. Logo contests are a little simpler and in the first 24 hours of the competition we received 72 entries, which I thought was a phenomenal response. They are definitely not all in the running, it is clear that some people didn’t even read the brief. However, there are already some that I could probably live with and we’ve yet to give any feedback.

I am now a complete convert to crowdsourcing. Given my experiences with the Siyavula logo and web-design processes I think that a site like 99 designs, that has a critical mass of designers, really provides a light-weight, cost-effective and rapid solution for most of my design needs. I will launch a contest, in the near future, to start improving the look of this site I just need to bolster the content so the purpose and scope are clearer :)