To a machine, an image is nothing but a set of pixels in a random pattern.

For most uses, this is alright.  Machines don’t need to know what is in the photo, since the receiving audience is meant to see the photo.  Photo sets are also generally of manageable sizes.  Those 4 rolls of film from your vacation need only a few words and a photo album.

However, as much as we explore their shortcomings here, there’s no way around it: machines do some things great.  When your database of images grows huge, you need a computer to manage it.  Now, assume that the database has uncountable millions of items, and those items are computer-stumping images.  How do you organize it?

The answer seems to be in crowds.  We’ll look at two examples by big online players: Google Image Labeller, and the native Flickr tagging system.

Google Image Labeller

Google Images has been, until recently, fully reliant on machine-decoding of image meaning.  What Google computers did, it seems, was consider the environment in which the image existed: what was featured on the page, how nearby text labeled the photo, and especially the filename and the name given in the ‘alt’ tag of the image.  However, this solution meant that searching for an image had a high ratio of clutter.  HTML is clunky, and unfortunate that clunkiness permeates the majority of the internet.  Without standard image semantic coding, a computer’s best guess often does not come close to the reality.  If Google wanted its borderline-useless tool to regain relevance, they were going to have to come up with a way to encode images with meaning on their own servers.  And they would need people to do it.

Cue the ESP Game.  Coming from Carnegie Mellon University, the ESP Game is simple: two human users log into it, and label images.  As a team, they are presented with the same image, and label it with words to describe it.  As soon as the teammates mutually use a tag, the image is given that tag, and the teammates get a new image.  They race to get through as many images as they can within a time limit, gaining points for speed.

Google has adopted the ESP Game for improving their Google Images database, under the name Google Image Labeller.  The idea behind the game is that if two humans independently arrive at the same word to describe an image, it is probably an appropriate word.  Tags that are applied to the image repeatedly are added to a “taboo list”, which lists words the computer is become sure about and encourages new ones.

By presenting this activity as a game, the problems of motivation and manpower for such an immense task are solved.  Users have a reason (beyond academic interest) to label this images for free, and to do it as fast as they can.  They aren’t bothered with the details— they’re just told to write what the other person is writing.


Flickr emerged at that wonderful time when all the Internet was discovering the organizational wonders of tagging.  The setting for Flickr is different: as an image hosting service, it deals with organizing images within its own walls, with its own rules.  While Google Images is trying to index a wild west Internet with no real standards, Flickr shows how well having a built-in tagging mechanism works.

At first, Flickr only gave the user the opportunity to attach descriptive words to their uploaded content.  Their database allowed one to browse a single user’s tags or the entire site’s tags.  There was also one extra bit of meaning that Flickr images receive: user-defined copyright.  Users can set how lenient or tough they are willing to be with the use of their images. They can even donate their images to the public domain.  The last bit of information, which is not really used for indexing, is the ability to label areas on the actual photo.

As it grew, Flickr made two notable changes, the first being that they added additional encoding possibility through “geotagging” the photo on a map.  This opens ups up a new dynamic database in Flickr: the first being Image search, and the other is plotted mapping, where one can explore a map and see numerous (sometimes thousands) of photos from that location.

The other growth in Flickr is the ability to add tags to other people’s photos.  While many companies would have been prudish in their concern about misuse, this move has allowed the search to grow.  In fact, paired with a search algorithm that organizes results by “Interestingness” and the ability to search by copyright licence, Flickr has become the best online search tool for images.

Where’s do we go from here?

So here we have two tools.  The one is trying to introduce order to a lawless Internet, while the other is working on furthering the basic system to create something useful.

The strengths of Flickr is both the number of venues for encoding that it offers and how it manages all of them without confusing the user.  They offer the tools, but ease users into them.  As users become familiar with the system, they begin to discover the tools at hand.  However, on the flipside, the main problem is motivation.  Since tagging is a choice, and that leaves many unlabelled images from uninterested users.  Certainly, users trying to increase their exposure do it, but beyond the community, there’s little reason for the average user to maintain their images as such.  It’s on this key issue of motivation that lies the essence of the Google Image Labeller / ESP Game.

Google Image Labeller simply adds a fun way to tag images.  What if Flickr used the game?  How about any other of the countless image hosting that use tagging?  Maybe other games are yet to be created, one where bored netizens hunt for untagged image orphans, or try to identify photos that a computer’s distorted.  Flickr may be the best image database online, but I’ll still save some betting chips for the ESP game and its theoretical groundwork.