Why Bother? Examining the Motivations of Users in Large-Scale Crowd-Powered Online Initiatives

As this is not the final copy, please do not distribute it. Rather, link back to this page, so that I can provide the final when available.

Everyone has their own stories, no? The time that Wikipedia made you understand something a textbook couldn’t,

Stack Overflow is a programming help site. The site is heavily controlled by its community; as users gain points and achievements for contributing to the site, they gain more administrative power.

I’ll admit, I went through a period where I watched the questions page daily, looking for something I could answer. However, I found  that every single time, somebody gave a better answer much quicker. I’m sure it is eventually discouraging, but in my experience it made me competitive. If you don’t get the first answer, at least get a higher rated response. Programming is an endless process of puzzles and problem-solving, so it’s good to know that there’s a place where one in need can get help within minutes. As Jeff Atwood notes in the tweet above, it’s quality help too.

Clay Shirky has a wonderful talk about this phenomenon, wherein he argues that passion (or ‘love’) is a resource that is unfairly underestimated. He uses the example of support for the open-source programming language perl: in an absence of commercial support, there are more than enough passionate perl users online to fill the need. Watch it below.

Kickstarter is a site that lets projects collect funding pledges from users. One can add a project with a funding goal and set tiered rewards for patrons. You’re not expected to pay your pledge unless the project goal is met. Hunch is a crowdsourced decision tree site. Users creating content add decision-making questions to a decision, ones that affect the outcome of the final recommendation.

Had either of these been released elsewhere, the project would not provide any particularly extraordinary impact. Indeed, the paradox of crowdsourcing is that the idea alone will not sell the project, because the difficult barrier of critical mass. Critical mass, however, is itself dependent on having sold the idea already. In other words, new crowdsourcing project can have a great idea in principle, but potential users are aware that it will be difficult to achieve in practice. Such doubt is self-propogating, because knowing that you have doubts means admitting that others may too, further removing confidence in the objective.

A site like Fake’s former creation Flickr was able to grow a community on the back of a valuable individual experience. The community was not vital to the experience of Flickr. Kickstarter and Hunch, however, have no individual experience. If there was only one user, that user would have little to do (admitting that Hunch’s employee-created content can only go so for).

With strong talent behind both of the sites, there is no shortage of cleverness in their mechanics. However, the key to their first few months lies in the trusted celebrity behind them. Like doubt, confidence is be self-propagated. Baio and Fake have an audience already in place, and seeing that audience can swing cynics from “great idea but wouldn’t work” to thinking “just maybe.”

“The prolonged, indiscriminate reviewing of books is a quite exceptionally thankless, irritating and exhausting job. It not only involves praising trash but constantly inventing reactions towards books about which one has no spontaneous feelings whatever. The reviewer, jaded though he may be, is professionally interested in books, and out of the thousands that appear annually, there are probably fifty or a hundred that he would enjoy writing about.

… Seeing the results, people sometimes suggest that the solution lies in getting book reviewing out of the hands of hacks. Books on specialized subjects out to be dealt with by experts, and on the other hand a good deal of reviewing, especially of novels, might well be done by amateurs. Nearly every book is capable or arousing passionate feeling, if it is only a passionate dislike, in some or other reader, whose ideas about it would surely be worth more than that of a bored professional. But, unfortunately, as every editor knows, that kind of thing is very difficult to organize.”

George Orwell, “Confessions of a Book Reviewer.” Tribune (1946)

Orwell, one would presume, would view aggregate amateur review as a boon, not a threat.

In my work on crowdsourcing, my advisors warn me to be careful of how I speak about Wikipedia around academics, because scholars are still divided on it. Clay Shirky’s quote perfectly encapsulates the situation: if it is clear that it works and that it works well, the question shouldn’t be “does it work?” Rather, we should be asking why it works. Kevin Kelly suggests that Wikipedia is “impossible in theory, but possible in practice“: shouldn’t we be tweaking our theories then? Perhaps then, the issue is that if an expert were to praise Wikipedia as reliable, they undermine society’s need for experts. Larry Sanger, creator/co-founder or Wikipedia, says no, but it’s certainly food for thought.

Genius is the playlist generation tool in Apple’s iTunes music software. You choose a song that you’re in the mood for, and it creates an entire playlist of similar songs. Essentially, its a recommender system; if you like x you’ll like y. The problem is that you get a very narrow point of view, with very little genre skipping. and no pleasantly clever surprises.

What sets Genius apart from other song recommender systems is that its essentially powered by the crowds. Apple has the luxury of a rich data set of habits and rating, and it appears to factor heavily into the recommendations. Indeed, algorithmic playlist generators were creating better results years before Genius came on the scene. So, what does this mean for the crowd?

The fact that computers can be better than humans in understanding art is off-putting. I’m still working through this problem, but here are some thoughts toward untangling it.

Ratings data is emotionless. When you rate a song 1 or 5, you’re giving it a universal ‘like’/'dislike’. This data doesn’t factor the mood of the song or the emotion of the listener. This is all very removed for circumstance. As I suggested to Bill Turkel, perhaps such simple crowd-based recommendations are better for high-level suggestions, like artists you may like, but useless at the micro-level (unless that data crowds are contributing is more specific to the topic of recommendations). In contrast, technology can quite effective interpreting the types and patterns of sound which represent an emotion. Certainly it can’t easily understand whether a song is good, but if you want a slow, jazzy rock song, that’s fairly achievable. This is something in which music recommendation is fairly unique, as it is easy to interpret than it would be to interpret thousands of movie plots or millions of book themes.

Despite this, perhaps the most-cited example of a good music recommender is Pandora, which is an internet radio based on the Music Genome Project (MGP). The MGP does use humans to categorize songs, having professionals tag each song with over 400 tags and using an algorithm to weigh the values. Pandora’s success shows that humans are indeed effective at understanding music, given that they’re looking at it in the right way.

There’s also the effect of popular media that makes human-based recommendations unbalanced. If a lot of people like Coldplay, the range of music that it will be recommended for will be broad. This additionally creates an echo loop where popular music simply grows in popularity. Inversely, it is very difficult for new music to enter the loop. If everybody that likes The Strokes like Yeah Yeah Yeahs, the recommender will reinforce this, brushing aside any similar new bands.

However, such problems are limited to the balance of the algorithm. Last.fm, which tracks all of its users’ listened music, is fairly effective in recommending similar music. Also, because of their detailed information on what a user has listened to, they can suggest less listened to songs. Though they don’t offer playlist generation, I wouldn’t put this beyond their abilities.

So where do crowds factor in here? If anything, Pandora suggests that this is best left to professionals. Certainly, you can’t get that sort of exhaustivity with crowds. The answer may lie in reliability. Large groups would be able to make much simpler connections, but on a larger and more verified scale. When I make a playlist with Lou Reed’s Take a Walk on the Wild Side, I always follow it with Urge Overkill’s Girl, You’ll Be A Women Soon.  The songs are linked very little, but there’s something in me that recognizes the similarly cool feeling that I feel. If you could somehow capture millions of these sorts of links, that could lead somewhere.

In collecting a crowd, one should consider the incentives that motivate that crowd. However, collecting a crowd in not the only way to gain one. What I’m talking about is repurposing a crowd.

While obviously not an option for everybody, repurposing a crowd offers, in many cases, the best chances for crowdsourced success. This may entail piggybacking functions onto your widely-used product (especially useful if you’re Google or Yahoo—which most of us aren’t) but it can also mean borrowing from somebody else’s (like with Facebook platform). The point comes down to this: it’s hard to provide incentive for users to frequent a new site in their online routine, but it is much easier to utilize the sites that they’re already visiting for other purposes.

Here are three directions to consider.

1. Making an audience into participants

Sometimes you have to make do with what you have. When you’re a local radio station competing with a television network, “what you have” can seem frustratingly limited. When it comes to traffic reporting, the difference may be that the big guys have helicopters in the sky and cameras on busy road. How can you compete with that?

For radio stations, the answer lies in what they do have: an audience actively experiencing the traffic. What has emerged from is traffic reporting based on mobile phone audience tips. This crowdsourced model helps narrow the competitive gap that expensive technologies create.

The magic lies in the clever mobilization of readers. There is already a large, dormant audience armed with the information that a traffic watch needs. Giving that audience a voice is all that is needed to get the information. A similar example is Are You Being Gauged from WNYC.

2. Crowdsource as a feature, not as the main event

Some of the most useful examples of crowdsourcing in the wild formed on the backs of other products. The golden standard here is the tagging feature of Flickr. Users have no rules forced upon them on how to encde their information. They just want to put up their photos, and tagging is something that is simply offered for them to stay organized. However, on the larger scale, all the users that do end up using tags helps create an extremely semantically relevant corpus of images (and, as I’ve mentioned before, the first place you should go when looking for images).

Returning to the earlier example of traffic information, such “incidental crowdsourcing” is being tested in using cellphone tower information to determine how fast traffic is moving. Simply by having one’s cellphone in their cupholder, they’re contributing data. Similar to this are e-commerce recommendation engines, where simply by surfing a site, a user contributes to an algorithm for predicting what similar users would want.

3. Reimagine Popular Actions

In this form of repurposing crowds, the question come down to how one can squeeze extra juice out of something already being used. his is the approach that Luis von Ahn projects take, especially well epitomized in reCaptcha and the ESP Game. If people are already using captchas, why not have them also digitize scanned texts? If people like to relax with an online game, why not also have them encode image metadata?

Before starting a crowd-assisted project, don’t bet on people finding their way to it. Think about how existing groups and communities can be used and you’re much more likely to succeed.

Idea: ‘YouShould’, a suggestion site where people write open letter suggestions of ideas for companies, authors, and services

There had been two things on my mind. The first was the potential benefit to consumers that such feedback could allow.  I was inspired by Gmail’s suggestion page, where one can suggest what they would like to see implemented in Gmail next. Google appears to take it seriously, too, listing past suggestions that have already been implemented. The other reason for my idea was that I had been brainstorming for my senior thesis, which was beginning in September. However, once September rolled around, “YouShould” was crushed by the release of the similarly named Should Do This.  While perhaps no exactly what I had imagined, it was pretty darn close.  I don’t believe in reinventing the wheel, so I dropped the project.

Turns out, dropping the project was probably a good idea.  After Should Do This, there came IdeaScale, and CrowdSound, and Suggestion Box, and UserVoice, FeatureList, Fevote and CollabAndRate.com.  All of these had different approach to the same concept: getting feedback from customers.  Turns out I wasn’t alone in the concept.

Unfortunately, as tends to be the rule, none of these services seem to have gained any traction. Interesting on paper, there was not enough return to attract critical mass and make the idea suceed. One reason is that, with unsolicited advice, users do not gain a sense of contribution. One thinks, what are the odds that a company cared enough to seek out these websites?  Users want to offers their thought and suggestions, but they also want to be heard. It’s like that wonderful game my aunt always played with the kids: “who can stay quiet for the longest”. Sneaky, yes, but we certainly stayed quiet for longer than we would have simply for its own sake. This is why general suggestion boards have been failing, and crowd-suggestion businesses has been moving into infrastructure, offering tools that enable business to ask their customers themselves.

How many times have you liked a television show, and found yourself lamented the fact that —unless you’re directly being asked by Nielson or BBM— your patronage does not actually register? The broadcast system that television uses is by definition clunky: it transmits only one way, from one to many, without a direct capacity for information feedback. This simple concept was outlined in the Shannon-Weaver model of communication back in the 40s. However, while the flow of source > encoder > message > channel > decoder > receiver is adequate for describing technology, attempts to apply it to human communication have been notably shortchanged. It simply is not natural to our nature, not reflective of how humans negotiate meaning. The transmission model is not simply limited to delivery of television and radio signals. In a way, our entire consumer culture attempts this few-to-many transmission. Business online, however, exists within a system constructed to be (though not always realized as) many-to-many. Feedback is the nature of the internet. If you’d like to see organic cotton shirts at the Gap, the time investment in doing so would discourage casual contributions. More likely, your feedback would be much more crude, by shopping elsewhere, in which case the Gap is left trying to figure out why you did so. In contrast, a Gmail user conscious of the idea solicitation page can quickly send in a thought when they have it.

“It’s not the cost we’re looking at, it’s how we are making the application better for the consumer” —Jari Pasanen, Nokia VP for innovation acceleration (BusinessWeek.com)

In “How Nokia Users Drive Innovation“, Business Week outlines Nokia’s solicitation of its users for ideas, and the sucess that that have been having. Other companies that do so are Starbucks (My Starbucks Idea), Salesforce (SalesForce IdeaExchange), and Dell (Dell IdeaStorm). In these example, communities have formed around supporting and expanding on ideas. A cynical observer would suggest that these companies are looking for free business advice. The reality, however, is that it is in the best interest of customers to help build better products for themselves. Companies are constantly looking for feedback and those that respond as the people for whom the company adapts to. This idea is nothing new; what has emerged is the persistance and tenacity of users in doing so when given the proper tools.

With the book out now, I’m digging those back up.  Here’s the excerpt on Chapter 5, where things start getting interest, and some of my thoughts.

Chapter 5: The Rise and Fall of the Firm: Turning Community Into Commerce

Chapter 5 touches upon an oft understated quality of crowds: their natural connection to community.  Crowds are rarely groups of disparate human being.  Rather, they form around common connections in varying degree of community.

Howe explains to us that communities are changing: not for the worse, but toward the more efficient.  A grievance we often hear about modern society is the erosion of neighbourhood communities.  However, geographically-defined communities, in their pre-World War II hey-day, were popular because they were the most accessible common-interest groups (the common-interest being the location).  As new tools became available, humans have found ways of being community members with broader groups, and bound by interests beyond geography.  Thus, the slide of our culture’s individuals into new depths of isolation is not the case.  Rather, our communities are simply moving into less visible ground.  In a great observation that I had not considered, Howe notes that now, with the ease of social connection provided by digital tools, “new types of communities have materialized that are both local and wired at the same time”.

Chapter 5 also looks at the successful online efforts of the Cincinatti Enquirer, particularly through the CincyMoms community blogs.  It is a good look at do-or-die changes in publishing.  There’s also a gem of information I wanted to highlight lest you miss it.  In regards to a reader-submission feature on the Enquirer’s website:

The words “GetPublished” feature prominently on every Enquirer Web page. The results land in Parker’s queue, and they almost never resemble anything commonly considered journalism. “It used to read, “Be a Citizen Journalist,” Parker says. “And no one ever clicked on it. Then we said, ‘Tell Us Your Story,’ and still nothing. For some reason, ‘GetPublished’ were the magic words.” The Enquirer considers the feature to be an unequivocal success.

There’s a reason for this: they rarely interest me.

There’s a time and a place for crowdsourcing, and what I love is when it is used to the achieve something that cannot otherwise be created.  There’s also a soft spot for cleverness in concept.  However, we increasingly see social for social’s sake.

Now, bad ideas are only a fraction of sites. Many others are simply not thought out in a way that the can be successful and, unfortunately for sites built on a foundation of crowds, success and usefulness are invariably linked.

I want crowdsourcing as business to suceed—I really do—but thus far it has been most successful by accident or by incident.  THAT is where the story is: in understanding this phenomenon.  Clearly we have the tools, but are still working on the trade.

ReCaptcha is a tool out of Carnagie Mellon, headed by Luis von Ahn (mentioned previously here).  To understand reCaptcha, one needs to understand captchas.  A captcha is a human verification tool that displays an image with a string of warped characters.  The task is to write those characters into the input box.  Because of the complexity of image recognition, this task more or less confirms that you are a human and not a bot. Of course, spammers can hire low-wage captcha crackers, but captchas nonetheless introduce an enormous hurdle to online spam and other automated cons.

ReCaptcha is an improvement on the original concept.  Amongst other accessibility improvements, reCaptcha’s primary innovation is that it helps digitize old books.  That right, digitize old books.  Rather than offering randomly warped words, reCaptcha instead offers the user words from scanned books that the computer recognition is having trouble with.  This assists in the various efforts to digitize (and in the process preserve and recover) libraries of aging books.

The brilliance of this cannot be understated.  The tool takes an action that millions of people already need to do, and appropriates that manpower into something useful.  Perhaps the best parallel is to solar energy.  The sun is an energy source that is completely wasted in urban areas.  It is everywhere, constantly beaming this energy onto the earth, and the cleverness of solar cells allow is for people to capture this potential (constant and wasted) and convert it to something greatly useful.  Anybody who has ever been awed by solar energy can understand the exciting potential that reCaptcha represents in technology.

I’ve started a page listing crowdsourcing sites and initiative.  Find it here.

Until then, I wrote a short page about staying up-to-date.

The first is what I’d call the “million monkeys” strategy.  Quite simply, this is the appeal to the crowd for the one or few with a commodity —be it information or material– that you need. “With a group this large, I’m bound to find what I need!“  Greater size and diversity offers a bigger box to sift through, but ultimately it is a few individuals that matter, rather than the crowd itself. 

The million monkey strategy is common online today.  Skill auction sites, such as 99Designs, iStockPhoto, and Yahoo! Answers, reward the best provider of solution to a problem while others watch from the sidelines.  Even though the Internet does not provide anything newly achievable, in that the right connection at the right time does not necessitate a gradiose group, the amount of minds online have greatly increased the potential of achieving that ideal connection.

Yet consider the example of 99 Designs, where people offer bounties for their design needs, and then choose the best submission as the winner of the challenge and recipient of the payment.  A few hundred dollars for a job may seem common for an entry-level designer, but that seems much smaller when one considers the discarded man-hours of the unpaid submitters.  Spec (‘speculative’) work is frowned upon by professionals because of its devaluative nature, and this concern can be seen to parallel much million-monkey crowdsourcing: it skins the cow for the leather but leaves the meat to rot.

More exciting is truly collaboratively crowdsourcing, because it represents possibilities for collective creation and problem-solving that have never been seen before.  In the purest form, such crowdsourcing allows thousands to each contribute a small part towards a bigger picture (recall the metaphor of Ten Thousands Cents).  Oftentimes, such crowdsourcing overcomes traditional organization dilemmas, such as costs and management.  For example, to categorize images en masse, as is done both actively with the ESP Game and incidentally with Flickr tagging, could not possibly be done at any reasonable rate of return prior to the arrival of the Internet.

As modern communication technology encourages crowds to grow larger while more streamlined, what new problems will they come to solve?  My communications education has pinned groups as collectively blunt, but now it seems that this is a result of primitive communication techniques; online, the “individual” is much more a part of the whole.  If we continue to repurpose individual minds in new combinations, the results will be something not often characteristic of society: fresh.

As both the million monkey and truly collaborative approaches require the same source—a crowd— projects need not be bound to simply one approach, nor are they.  For example, the ever-popular Threadless has a million monkeys system for t-shirt submissions, while an generally democratic system of voting (mixed with some managerial liberties).  If there is a community with one goal it can be re-purposed for another.  And thus we arrive at a topic for another day: the reworking of existing communities.

If you worried that Howe was simply falling into writing the book by obligation, worry not— the book, as seem through the excerpts, appears to be wonderfully written and insightful.  It’s a highly recommended read, and I hope that the rest keeps up the same calibre.

Here’s links and summaries to the parts that have been thus far posted (Chapters 2-4, more to come later).  Remember that there’s an open call to contribute insight, so if you have an issue to raise, let him have it.

Chapter Two: The Rise of the Amateur excerpt #1

More setting the pace than offering insight, this excerpt introduces us to iStockPhoto, the low-cost stock photo site run by amateurs.  The site is profitable—making more more for parent Getty Images than any other property— but the secret’s not in the commerce.  Rather, it ‘s the community.  The site does not exploit naive photographers by underpaying them, but rather offers them a community for improving their skills, incidentally offering the promise of a bit of money.  “iStock doesn’t offer a chance to get rich. It offers the chance to make friends and become a better photographer.”

Chapter Two: Rise of the Amateur excerpt #2

The second excerpt from Chapter 2 outlines the importance of amateurs in the early stages of the Scientific Revolution.  However, industrialization led to increased specialization, and society began to breed experts over polymaths.  The division of labour, which reached its pinnacle with Fordism, began to cross into academia, creating segmented areas of human thinking.

This excerpt provides the setting for story from Chapter 4 onwards, where we start to see the trend begin to reverse.

Chapter 3: From So Simple a Beginning excerpt #1

Howe introduces us to the remarkably capable open-source movement, and the equally important GNU/General Public License.  From such a seemingly chaotic and unstructured model as open-source comes some truly remarkable software, oftentimes better than commercial alternatives.  Anybody who has used a popular Linux distro will can relate to this fact.

In 1976, Bill Gates and Paul Allen wrote an “open letter to hobbyists.” It did not mince words: ‘As the majority of hobbyists must be aware, most of you steal your software.’ …The hobbyists needed professional programmers because, after all, ‘What hobbyist can put 3-man years into programming, finding all bugs, documenting his product and distribute for free?’ Gates could have never anticipated the answer to his question, which was that no single hobbyist could put 3-man years into such a daunting task, but 3,000 hobbyists easily could, and soon would.

Chapter 3: From So Simple a Beginning excerpt #2

Another great excerpt, here we look at the flaws of the patent system and are introduced to a concept that we will see in later sections: when the mixed masses do a job better than experts.

The patent system was broken. The debate now revolves around how to fix it. Over 90 percent of patent applications are successful, giving rise to a rat’s nest of vague, overlapping patents. “We wind up in these fights over patents where we can’t tell what they mean, and the courts can’t tell what they mean, and even the patentees can’t tell you what they mean,” Kappos says.

Chapter 4—Faster, Cheaper, Smarter, Easier: Democratizing the Means of Production excerpt #1

Chapter 4: Faster, Cheaper, Smarter, Easier excerpt #2

As technology has grown, the tools of experts have come into the reach of the amateur.  Increasingly, the amateur is enabled to do things that had been previously guarded by experts.  Along the way, economic casualties are had, be it stock photographers threatened by iStockPhoto or typesetters by home printers and desktop publishing software.

Chapter 4: Faster, Cheaper, Smarter, Easier excerpt #3

Hawthorne Heights isn’t signed to a major record label.  They don’t have a giant marketing budget, nor a luxurious production and distribution model.  Yet, owing greatly to the tools afforded to them through the internet, namely Myspace, their most recent album debuted at number 3.

This is where things get interesting.  Why?  Because the entire music business is being overhauled, and once again, the big guys are losing to the ones at the bottom of the pyramid.  Though the excerpt does not delve into it, what is apparent is that the magic is in the long tail: the long tail of fans, of musicians, and record labels.  The classic model was based on one big band, supported by many fans, being nurtured and enabled by the horizontal and vertical resources of the big record label.  The big bang successes would usually make up for the big-budget fizzles.

In the past few years, there a been a remarkable change to this model.  All the other bands, those not chosen to be made into ‘the next big thing’, have found themselves a voice on the Internet.  They need neither the budget nor the promotional services which previously allowed major labels to control the market.  “Most up-and-coming bands don’t regard illegal peer-to-peer file sharing as piracy; they view it as a promotional and distribution channel”.  At the same time, the long tail of casual music fans have found music to be much more accessible as a hobby, and there’s something for everyone.  Sales records are no longer being broken, but more bands are getting a cut, more concerts are being attended, and more music is being listened to.  Yet, despite the exciting state of affairs, it comes as no surprise then that the RIAA is up in arms, when the tools that they’ve been using to get a chokehold on the business (promotion, production and distribution) are no longer necessary.

It’s interesting to note the seriousness with which the researchers approached Galaxy Zoo, in that the content was the primary purpose and rarely is the word “experiment” tossed around in regards to the method.

By all accounts, the study was a success.  Here are some notes culled from the first two papers stemming from the project.

Statistics

As of November 28th 2007, GZ had over 36 million classifications (called ’votes’ herein) for 893,212 galaxies from 85,276 users. (Land et al. 2)

“… we are able to produce catalogues of galaxy morphology which agree with those compiled by professional astronomers to an accuracy of better than 10%.” (Lintott et al. 9).

User Reliability

For greater reliability, two methods were employed.  First of all, each galaxy was classified numerous times, and the researchers would “only use objects where one class-type receives a significant majority of the votes.” This technique of independent confirmation is used in most such undertakings, as it limits individual impact by unreliable or malicious users.

There was also a weighted ranking, where “users who tended to agree with the majority have their votes up-weighted, while users who consistently disagreed with the majority have their votes down-weighted” (Land et al. 2).  However, reserachers did not see much difference, and chose to concentrate on the unweighted results.

“More than 99.9% of the galaxies classified as MOSES ellipticals which are in the Galaxy Zoo clean sample are found to be ellipticals by Galaxy Zoo.” (Lintott et al. 2)

Bias and Validity

Numerous types of bias were recorded.  Notably, colour-bias (where more experienced users can classify based on the prior tendencies of the specific colour) and spiral-bias were noted.  The second one, as noted in Galaxy zoo finds people are screwed up, not the Universe, appears to people a product of human psychology, where users would tend to classify a galaxy as rotating counterclockwise, when in theory CW and CCW should be about equal.

To investigate this, the creators ran a bias sample, with occasional monochromatic, vertically-mirrored, and diagonally-mirrored images.  We see this done in Luis von Ahn’s projects, with decoys being used in Recaptcha and ESP Game to help determine reliability.  The GZ researchers note the Hawthorne Effect, in that “users may be more cautious with their classifications if they think that they are being tested for bias” (Lintott et al. 9).  However, considering the example of Recaptcha—which offers one real word and one decoy—perhaps such an effect can be utilized fully.

Participation

To get as many users as possible, simplicity and a low barrier to entry were extremely important considerations in the design.  “Visitors to the site were asked to read a brief tutorial giving examples of each class of galaxy, and then to correctly identify a set of ‘standard’ galaxies. …Those who correctly classified more than 11 of the 15 standards were allowed to proceed to the main part of the site. The bar to entry was kept deliberately low in order to attract as many classifiers to the site as possible” (Lintott et al. 3).

The majority of users classified around 30 galaxies.  As the following chart shows, however, some went up to the tens of thousands.  Even though the use of crowds to dissipate individual time obligations is the core purpose of such a system, it is very beneficial to accommodate the “super-users”, who do hundreds of times as much work as the casual user.

Links

Galaxy Zoo

Bad Astronomy: AAS #14: Galaxy zoo finds people are screwed up, not the Universe

Betsy Devine: Ox, Docs Shocks!

I was wrong.  The results of that project have surfaced, in the form of “Ten Thousand Cents“.  TUrns out we were drawing a one hundred dollar bill.

The total labor cost to create the bill, the artwork being created, and the reproductions available for purchase are all $100. The work is presented as a video piece with all 10,000 parts being drawn simultaneously. The project explores the circumstances we live in, a new and uncharted combination of digital labor markets, “crowdsourcing,” “virtual economies,” and digital reproduction.

This project serves as a brilliant metaphor of the normalizing power of crowds.  When you open up a project to the masses, governance becomes extremely difficult.  Anybody is given the ability to contribute erroneous information.  However, as you gain a larger community of contributors, things balance out despite the fouls.  Consider opinion-based efforts, such as Digg and Travelocity: eventually, the best items shine through.  That is why Wikipedia is so reliable considering the circumstances: because thousands of editors are better than one.  So how is Ten Thousand Cents relevant?

Still Ben, right?

The first is a classification of Sports Illustrated covers over the past thirty years.  Covers were classified by race of the athletes featured and the sport featured.  Having recently led coding for a school study—involving a 2-week census of Digg.com front page stories— I can certainly appreciate how appropriate the Turk is for coding with such straightforward, reliable variables.

The second example is even more fascinating.  Providing Turkers with thousands of random colours, Dolores simply asked each colour to be named by the worker.  What resulted is a fascinating dataset of human-interpreted colour descriptions.  You see the common colour names pop up, but more interesting are how the workers utilized language to describe those words that were more difficult to classify.

Essentially, Dolores Labs is a crowdsourcing consulting company.  Even though they provide deeper services than simply advice, their main commodity is the knowledge of how crowdsourcing works.  There are good ways to mobilize crowds and incorrect or useless ways to do so, and as we come to realize that, crowdsourcing moves beyond simply a trend and into a bona-fida tool.  The existence of a group that specializes in understanding the process shows a maturing of crowdsourcing within culture as a viable method for abstract analysis.

Dolores Labs aren’t even the first ones selling their expertise on crowdsourcing. Amsterdam-based CreativeCrowds have been doing a similar thing for a while.  Like Delores Labs, they also give back to the public, not in the form of test data but in their phenomenal blog, CrowdSourcingDirectory.  Both companies are approaching this the right way, and I hope to see more from both in the future.

I’ve been accepted to grad school on a platform of crowdsourcing research.  I’m still trying to decide which one best serves my needs, but needless to say, I’ll soon be posting more regularly and with much more insight.

What sort of motivation does one have to participate in a crowdsourced system? To do so means virtual anonymity, becoming a single name within a sea of many. It means a compromise of control, a sharing a managerial responsibility and credit that some may find disconcerting. Numerous crowdsourced projects have failed, because they did not present a compelling reason to participate. Yet, there have been a great number that I succeeded; what did they offer that the others did not? Crowdsourcing needs crowds, so I considered how you can get them.

If you build it..they won’t particularly come…

So, what’s the magic answer? Of varying levels of importance, there are basically eight hooks that make a crowdsourced idea work: academia, charity, money, fun, community participation, forced participation, self-benefit from the product, and interest in the content.

The first two motivations to consider are academic interest and charity: the goodwill factor. Some potential users will participate in a system not for any returns from the system, but simply for the sake of its success. Wikipedia is the great benefactor from academic interests: while it has developed into a mature system where time invested by a user is time returned in some way, its initial growth was different. With a wiki, any knowledge you enter will itself be useless encyclopedic knowledge to you, given that it is something you already know. Wikipedia grew because users looked forward and imaged how delightful the system would be if it suceeded. A volunteer encyclopedia is not a new concept—the OED was started by hundred of volunteers, attempting to improve upon the lackluster documentation of the English language. Academic motivation, then, is motivation based on an ideological foresight. A motivation of charity also is centered on the actual system, rather than particularly what it provides. An example of charity is participating in a well-polished product, out of appreciation for the quality put into it. Consider three social community sites: Virb, a design-centric re-imagination of Myspace, or Tumblr and Dopplr, two sites with a small purpose but a solid implementation. Working against goodwill is the fact that few users actually do feel inclined to support something on those grounds. Also, such motivation is limited in time, and without more direct returns, people will eventually give up on supporting a system.

The other reasons for crowds to support crowdsourcing systems are much more self-motivated. Money is the most direct of these, as it is for most of our society. However, there is a problem with paid crowdsourcing: the amount of people there is to pay. This invariably leads to low wages, simply because anything higher would not be viable for most projects . Money is a great motivator, but the extent varies with the amount. It is a straightforward concept; people are more likely to pick up a quarter from the street than a penny. If the pay is low, the impact of the monetary incentive will be low. Tying the project into a commercial system also introduces problems of valuating to the work. I can spend hours sharing ideas in a free forum, but if you offer me 5 cents for it then I’ll feel like I anything more than a few seconds would be too generous. Last is the most concerning: the idea of buying community and the problematic connotations that come with it. The Mechanical Turk is the most prominent platform for paid crowdsourcing. On the Turk, we have seen that it’s implementation has fallen short of it’s ideological concept of an “artificial artificial intelligence”. What it has overwhelmingly become is a mix of people trying to buy crowds for questionable reasons, such as Internet traffic and reviews, and people trying to take advantage of cheap labour. Where money makes a good incentive is when you have a lot of it, and as a last resort, where there are no other incentives for the work that needs to be done (e.g. transcribing audio/video or editing texts).

On quite the opposite end of the spectrum is using fun and boredom to attract support. The concept of fun is basic: make it fun! ‘Games with a Purpose’, headed by Luis von Ahn at Carnegie Mellon, is an exploration of this method. It includes a series of games, most notably the ESP Game and Peekaboom, that make semantic analysis into a game. ESP game, already covered by me, has two strangers try to guess what the other it thinking. The game has become so popular that, according to von Ahn’s talk at Google, they had to introduce limits to how many hours a single user can spend logged in. While on the user-end it’s an addictive game, on the other side, the system is left with a database of independently confirmed semantic tags for digital images. if the 9 billion annual human hours spent on gaming (suggested by von Ahn) are true, then there is definitely potential here. However, ‘fun’ is a hard concept to put in practice, and with so much competition online, it is likely that such games would very much fall back on the goodwill factor. Furthermore, to produce ‘fun’ within the constraints of your crowdsourced purpose may very well be the most difficult task of all of these. It requires cleverness in competing mindsets: those who have large-scale human-computation tasks are likely the ones without a finely-tuned sense of what is fun for the masses, just like those who understand low-entry gaming probably won’t be looking to solve blue-sky problems.

Appealing to the bored masses is not much different from using fun. Fun is certainly one of the ways to do so. However, I’d describe boredom-motivated crowdsourcing not just to be games, but also ephemeral toys. Even though they have a purpose, they inherit a status as a time-waster. Little thought is asked of the user, and few rules are imposed. A bored Internet user does not generally want to mentally exert themselves; rather, they want some pretty, something “cool”, or something new. Consider this click survey, where users simply click on the page, and their click results in relation to others are presented in a heat-map type graph. Very ephemeral, but interesting, producing some fascinating psychological results at the same time. Another example is Human Brain Cloud, which takes an impressively free-form, anything-goes approach and succeeds in it. The marvellous visual brain cloud, however, is as cool as it is useful, not to mention funny (e.g. I typed in ‘racist’ and was given ‘bigot’, ‘xenophobe’, ‘intolerant’ and ‘Mel Gibson’). Human Brain Cloud also has scoreboards and lets you define a simple identity. This strategy of adding goals and achievements is used often, and keeps many users coming back. Arbitrary achievements and badges aren’t even limited to games (e.g. GasBuddy’s status icons), but they introduce a game-like level of competition.

In the next incentive, participation, the crowd is the focus. For one, there is an inclination towards human interaction, even if it is with strangers. Nowhere is this more evident than in the popularity of massively-multiplayer video games, when you would assume (well, I would) that digitally beating up a stranger from Finland would be comparable to destroying an AI player. In the realm of crowdsourcing, communities working on the same thing create emotional bonds, whether of respect or hate. In a community such as Wikipedia’s, the propensity for respect drives the content and quality of the most active users (who power a considerable slice of the site). Returning to the ESP Game, von Ahn claims that one of its addicting qualities is that users feel a strong bond to strangers they agree with, while having a human to blame when the two don’t do too well. Social recommendation is also powerful online incentive: when “everybody’s doing it”. With blogging (and even more powerfully, the Facebook Newsfeed) revolutionizing how users share information, a very likely foil for other incentives is when a familiar participant pulls you into it.

In some cases, the motivation for a crowdsourced action can be attached to an unrelated action. In such a way, it effectively becomes forced, unto any user that wants the other action done. The premier example of this is reCaptcha, which reworks the concept of the Captcha, a skewed character image where the user writes the characters to prove that they are human. ReCaptcha appropriates this common process of proving one’s abstract cognition and applies it to the digitization of OCR-problematic words in scanned texts. To access the action that the user wants, such as online commenting or a service sign-up, they have to join the distributed crowd and do a few seconds of of work.

The last main incentive for users in their own self-benefit from the content created by the crowdsourced system. There are two ways that this can embody itself: direct and indirect. Direct benefit is when the content created by the effort is of immediate use and value to the individual participating. This will be usually related to initiatives that require sharing of knowledge or experiences. By adding your own facet of information to the project, it becomes framed on a greater scale. For example, you can submit the local price of milk, lettuce, or beer to WNYC’s Are You Being Gouged, knowing that by supporting the map, the ultimate product (a map showing the prices of these items throughout Manhattan) will be of use to you. In-direct self-benefit, meanwhile, can be best described as karma. A wiki is an example of this because, as mentioned earlier, what you contribute to a wiki is generally something that you don’t need returned to you. Rather, you may contribute knowing that, by supporting the system for others, the system will likely return the favour in a different area.

Those are my preliminary thought on motivation in crowdsourcing. Are there any I missed? Since beginning this, I’ve identified at least two which I hope to put more thought into: the sense of achievement in the action being done by the individual and of course, and interest in the content. Any others?