Learn Piano through AR

I like this:

The Projected Instrument Augmentation system (PIANO) was developed by pianists Katja Rogers and Amrei Röhlig and their colleagues at the University of Ulm in Germany. A screen attached to an electric piano has colourful blocks projected onto it that represent the notes. As the blocks of colour stream down the screen they meet the correct keyboard key at the exact moment that each one should be played.

Florian Schaub, who presented the system last month at the UbiComp conference in Zurich, Switzerland, said that users were impressed by how quickly they could play relatively well, which is hardly surprising given how easily we adapt to most screen interfaces these days.

But while there is real potential for PIANO as a self-guided teaching aid, in my view it’s the potential for a really tight feedback loop that makes this most interesting, and potentially more widely applicable.

When a piano teacher corrects a student’s mistake, they will perhaps specify one or two things that need improving, but this approach would sense each incorrect note and could provide an immediate visual response, flashing red for instance, conditioning the student to success more quickly.

via New Scientist.

Are You Ready For Your Close Up, Miss Colada?

BevShots have discovered what you’d call a niche: they take your favourite alcoholic drink, crystallise a single droplet of it in an airtight container, photograph it at 1000x under a microscope, and then sell the resulting image on a printed canvas.

And man, are these things selling! Since August last year BevShots estimate sales of over 20,000 prints ($24.99-$549). The product is aimed at the ‘hedonist with a mind for science’ segment: those who appreciate good photography, laboratory conditions and a damn-tasty cocktail now and then.

Here’s my favourite image, the classic Vodka and Tonic:

The shots are taken in Florida State University’s chemistry department, where founder Lester Hutt developed the approach, which can take up to three months to produce an image.

Lester says:

“What you can see in the magnified pictures are the crystalised carbohydrates that have become sugars and glucose. With my background in chemistry, I saw the potential in these kind of pictures and am so glad to be able to offer them up as art works. It is a pleasure to show people what makes up their favourite drinks and how beautiful it can look.”

Most alcohols are blends, with varying levels of carbohydrates, sugars, acids and glucose, so each shot taken is entirely different from the last. Some favourite drinks are so pure that when they crystallise  into their component parts, they fall apart or don’t dry out properly. So, not unlike the perfect Margarita, they’re pretty hard to get ‘just right’, sometimes taking up to 200 attempts.

Here’s some more of their work – click through for the full images or visit BevShots.

I’m thirsty! Who’s for a drink?


So that’s it, my series is over. All that’s left to do now is credit the academic sources that influenced and aided in the construction of my argument. Thanks to everyone below, and thanks to you, dear reader, for coming along for the ride.


Baudrillard, Jean (1983). Simulations. New York: Semiotext(e).

Baudrillard, Jean (1988). Selected Writings, ed. Mark Poster. Cambridge: Polity Press.

Baumann, Jim (date unknown). ‘Military applications of virtual reality’ on the World Wide Web. Accessed 20th March 2007. Available at http://www.hitl.washington.edu/scivw/EVE/II.G.Military.html

Benjamin, Walter (1968). ‘The Work of Art in the Age of Mechanical Reproduction’, in Walter Benjamin Illuminations (trans. Harry Zohn), pp. 217–51. New York: Schocken Books.

Bolter, J. D., B. Mcintyre, M. Gandy, Schweitzer, P. (2006). ‘New Media and the Permanent Crisis of Aura’ in Convergence: The International Journal of Research into New Media Technologies, Vol. 12 (1): 21-39.

Botella, Cristina.M, & M.C. Juan, R.M. Banos, M. Alcaniz, V. Guillen, B. Rey (2005). ‘Mixing Realities? An Application of Augmented Reality for the Treatment of Cockroach Phobia’ in CyberPsychology & Behaviour, Vol. 8 (2): 162-171.

Clark, N. ‘The Recursive Generation of the Cyberbody’ in Featherstone, M. & Burrows, R. (1995) Cyberspace/Cyberbodies/Cyberpunk, London: Sage.

Featherstone, Mike. & Burrows, Roger eds. (1995). Cyberspace/ Cyberbodies/ Cyberpunk: Cultures of Technological Embodiment. London: Sage.

Future Image (author unknown) (2006). ‘The 6Sight® Mobile Imaging Report’ on the World Wide Web. Accessed 22nd March 2007. Available at http://www.wirelessimaging.info/

Genosko, Gary (1999). McLuhan and Baudrillard: The Masters of Implosion. London: Routledge.

Kline, Stephen, DePeuter, Grieg, & Dyer-Witheforde, Nick (2003). Digital Play: The Interaction of Technology, Culture, and Marketing. Kingston & Montreal: McGill-Queen’s University Press.

Levinson, Paul (1999). Digital McLuhan: a guide to the information millennium. London: Routledge.

Liarokapis, Fotis (2006). ‘An Exploration from Virtual to Augmented Reality Gaming’ in Simulation Gaming, Vol. 37 (4): 507-533.

Manovich, Lev (2006). ‘The Poetics of Augmented Space’ in Visual Communication, Vol. 5 (2): 219-240.

McLuhan, Marshall (1962). The Gutenberg galaxy: The Making of Typographic Man. Toronto, Canada: University of Toronto Press.

McLuhan, Marshall (1964). Understanding Media: The Extensions of Man. New York: McGraw-Hill.

McLuhan, Marshall and Powers, Bruce R. (1989). The Global Village: Transformations in World Life in the 21st Century. Oxford University Press: New York.

Milgram, Paul & Kishino, Fumio (1994). ‘A Taxonomy of Mixed Reality Visual Displays’ in IEICE Transactions on Information Systems, Vol. E77-D, No.12 December 1994.

Reitmayr, Gerhard & Schmalstieg, Dieter (2001). Mobile Collaborative Augmented Reality. Proceedings of the IEEE 2001 International Symposium on Augmented Reality, 114–123.

Roberts, G., A. Evans, A. Dodson, B. Denby, S. Cooper, R. Hollands (2002) ‘Application Challenge: Look Beneath the Surface with Augmented Reality’ in GPS World, (UK, Feb. 2002): 14-20.

Stokes, Jon (2003). ‘Understanding Moore’s Law’ on the World Wide Web. Accessed 21st March 2007. Available at http://arstechnica.com/articles/paedia/cpu/moore.ars

Straubhaar, Joseph D. & LaRose, Robert (2005). Media Now: Understanding Media, Culture, and Technology. Belmont, CA: Wadsworth.

Thomas, B., Close. B., Donoghue, J., Squires, J., De Bondi, I’,. Morris, M., and Piekarski, W. ‘ARQuake: An outdoor/indoor augmented reality first-person application’ in Proceedings of the Fourth International Symposium on Wearable Computers, (Atlanta, GA, Oct. 2000), 139-141.

Wagner, D., Pintaric, T., Ledermann, F., & Schmalstieg, D. (2005). ‘Towards massively multi-user augmented reality on handheld devices’. In Proc. 3rd Int’l Conference on Pervasive Computing, Munich, Germany.

Weiser, M. (1991) ‘The Computer for the Twenty-First Century’ in Scientific American 265(3), September: 94–104.

Williams, Raymond (1992). Television: Technology and Cultural Form. Hanover and London: University Press of New England and Wesleyan University Press

Further Reading:

Bolter, Jay D. & Grusin, Richard (1999). Remediation: Understanding New Media. Cambridge, MA: MIT Press.

Cavell, Richard (2002). McLuhan in Space: a Cultural Geography. Toronto: University of Toronto Press.

Galloway, Alexander R. (2006). Gaming: Essays on Algorithmic Culture. Minneapolis: University of Minnesota Press.

Horrocks, Christopher (2000). Marshall McLuhan & Virtuality. Cambridge: Icon Books.

Jennings, Pamela (2001). ‘The Poetics of Engagement’ in Convergence: The International Journal of Research into New Media Technologies, Vol. 7 (2): 103-111.

Lauria, Rita (2001). ‘In Love with our Technology: Virtual Reality A Brief Intellectual History of the Idea of Virtuality and the Emergence of a Media Environment’ in Convergence: The International Journal of Research into New Media Technologies, Vol. 7 (4): 30-51.

Lonsway, Brian (2002). ‘Testing the Space of the Virtual’ in Convergence: The International Journal of Research into New Media Technologies, Vol. 8 (3): 61-77.

Moos, Michel A. (1997). Marshall McLuhan Essays: Media Research, technology, art, communication. London: Overseas Publishers Association.

Pacey, Arnold (1983). The Culture of Technology. Oxford: Basil Blackwell.

Salen, Katie & Zimmerman, Eric. (2004) Rules of Play: Game Design Fundamentals. Cambridge, MA: MIT.

Sassower, Raphael (1995). Cultural Collisions: Postmodern Technoscience. London: Routledge.

Wood, John ed. (1998). The Virtual Embodied: Presence/Practice/Technology. London: Routledge.

Five New Interfaces from SIGGRAPH 2009

Just saw these over at MIT’s Technology Review, and thought I’d share…

Touchable Holography:

Researchers at the University of Tokyo have developed a display that lets users “touch” holograms. Virtual objects appear to float in mid-air thanks to an LCD and a concave mirror. The sensation of touching the objects is created using an ultrasound device positioned below the LCD and mirror, creating an area of condensed air.

Augmented Reality Toys:

One student of France’s L’École de Design has developed a way to ‘hack’ toys using AR. His Scope display automatically recognizes ordinary toys that have been mounted onto platforms covered with hexagonal patterns, as seen below. With AR, these hexagons become interactive buttons that are used to make virtual modifications to the toy.

Augmented Reality Toys.v2 (Work in progress) from Frantz Lasorne on Vimeo.

Hyper-Real Virtual Reality:

Another French project, this time from INRIA and Grenoble Universities, could revive the dying science of Virtual Reality. Their new VR system, Virtualization Gate, tracks users’ movements very accurately using multiple cameras, allowing them to interact with virtual objects with never-before-reached realism. This interface demonstrates true physics, as well as crispy graphics, so a cluster of PCs is needed to perform the necessary image capture and 3D modeling.

3D Teleconferencing:

Researchers at the University of Southern California will demo Headspin, a 3D teleconferencing system that maintains eye contact between a three-dimensional head and several participants on the other end of a connection.

To capture an image, a polarized beam-splitter “places” the camera virtually near the eyes of the speaker. The 3D display works by projecting high-speed video onto a rapidly spinning aluminum disk to generate an accurate image for each viewer.

HeadSPIN: A One-to-Many 3D Video Teleconferencing System from MxR on Vimeo.

Scratchable Input:

One researcher from Carnegie Mellon University will demonstrate his new scratch input technology. The system turns any surface into an instant input device by sensing the unique sound produced when a fingernail is dragged across it. The interface is small enough to fit into a mobile device (though I have concerns about calibration) and could thereby turn any surface the device is placed upon into an interface.

Gameware: A Case-Study in AR Development

I have been aided in this series by a connection with Gameware Development Limited, a Cambridge-based commercial enterprise working in the entertainment industry. Gameware was formed in May 2003 from Creature Labs Ltd, developing for the PC games market which produced the market leading game in Artificial Intelligence (AI), Creatures. When Gameware was formed, a strategic decision was made to move away from retail products and into the provision of technical services. They now work within the Broadcasting and Mobile Telephony space in addition to the traditional PC market. I use this business as a platform to launch into a discussion of the developments current and past that could see AR become a part of contemporary life, and just why AR is such a promising technology.

Gameware’s first explorations into AR came when they were commissioned by the BBC to develop an AR engine and software toolkit for a television show to be aired on the CBBC channel. The toolkit lets children build virtual creatures or zooks at home on their PCs which are uploaded back to the BBC and assessed:


A typical Zook, screenshot taken from Gameware's Zook Kit which lets children build virtual creatures
A typical Zook, screenshot taken from Gameware's Zook Kit which lets children build virtual creatures


The children with the best designs are then invited to the BAMZOOKi studio to have their virtual creatures compete against each other in a purpose-built arena comprised of real and digital elements. The zooks themselves are not real, of course, but the children can see silhouettes of digital action projected onto the arena in front of them. Each camera has an auxiliary camera pointed at AR markers on the studio ceiling, meaning each camera’s exact location in relation to the simulated events can be processed in real time. The digital creatures are stitched into the footage, and are then navigable and zoomable as if they were real studio elements. No post-production is necessary. BAMZOOKi is currently in its fourth series, with repeats aired daily:


BAMZOOKi, BBC's AR game show where children’s zooks compete in a studio environment
BAMZOOKi, BBC's AR game show where children’s zooks compete in a studio environment


BAMZOOKi has earned Childrens BBC some of its highest viewing figures (up to 1.2 million for the Monday shows on BBC1 and around 100,000 for each of the 20 episodes shown on digital Children’s BBC), which represents a massive milestone for AR and its emergence as a mainstream media technology. The evidence shows that there is a willing audience already receptive to contemporary AR applications. Further to the viewing figures the commercial arm of the BBC, BBC Worldwide, is in talks to distribute the BAMZOOKi format across the world, with its AR engine as its biggest USP. Gameware hold the rights required to further develop their BAMZOOKi intellectual property (IP), and are currently working on a stripped down version of their complex AR engine for the mobile telephony market.

I argue, however, that Broadcast AR is not the central application of AR technologies, merely an enabler for its wider applicability in other, more potent forms of media. Mobile AR offers a new channel of distribution for a variety of media forms, and it is its flexibility as a platform that could see it become a mainstream medium. Its successful deployment and reception is reliant on a number of cooperating factors; the innovation of its developers and the quality of the actual product being just part of the overall success the imminent release.

As well as their AR research, Gameware creates innovative digital games based on their Creatures AI engine. They recently produced Creebies; a digital game for Nokia Corp. Creebies is one of the first 3D games which incorporates AI for mobile phones. Gameware’s relationship with Nokia was strengthened when Nokia named them Pro-Developers. This is a title that grants Gameware a certain advantage: access to prototype mobile devices, hardware specifications, programming tools and their own Symbian operating system (Symbian OS) for mobile platforms. It was this development in combination with their experiences with BAMZOOKi and a long-standing collaboration with Cambridge University which has led to the idea for their HARVEE project. HARVEE stands for Handheld Augmented Reality Virtual Entertainment Engine.

Their product allows full 3D virtual objects to co-exist with real objects in physical space, viewed through the AR Device, which are animated, interactive and navigable, meaning the software can make changes to the objects as required, providing much space for interesting digital content. The applications of such a tool range from simple toy products; advertising outlets; tourist information or multiplayer game applications; to complex visualisations of weather movements; collaborating on engineering or architectural problems; or even implementing massive city-wide databases of knowledge where users might ‘tag’ buildings with their own graphical labels that might be useful to other AR users. There is rich potential here.

In HARVEE, Gameware attempt to surmount the limitations of current AR hardware in order to deliver the latest in interactive reality imaging to a new and potentially huge user base. Indeed, Nokia’s own market research suggests that AR-capable Smartphones will be owned by 25% of all consumers by 2009 (Nokia Research Centre Cambridge, non-public document). Mobile AR of the type HARVEE hopes to achieve represents not only a significant technical challenge, but also a potentially revolutionary step in mobile telephony technologies and the entertainment industry.

Gameware’s HARVEE project is essentially the creation of an SDK (Software Development Kit) which will allow developers to create content deliverable via their own Mobile AR applications. The SDK is written with the developer in mind, and does the difficult work of augmenting images and information related to the content. This simple yet flexible approach opens up a space for various types of AR content created at low cost for developers and end-users. I see Mobile AR’s visibility on the open market the only impediment to its success, and I believe that its simplicity of concept could see it become a participatory mass-medium of user-generated and mainstream commercial content.

What is AR and What is it Capable Of?

Presently, most AR research is concerned with live video imagery and it’s processing, which allows the addition of live-rendered 3D digital images. This new augmented reality is viewable through a suitably equipped device, which incorporates a camera, a screen and a CPU capable of running specially developed software. This software is written by specialist software programmers, with knowledge of optics, 3D-image rendering, screen design and human interfaces. The work is time consuming and difficult, but since there is little competition in this field, the rare breakthroughs that do occur are as a result of capital investment: something not willingly given to developers of such a nascent technology.

What is exciting about AR research is that once the work is done, its potential is immediately seen, since in essence it is a very simple concept. All that is required from the user is their AR device and a real world target. The target is an object in the real world environment that the software is trained to identify. Typically, these are specially designed black and white cards known as markers:

An AR marker, this one relates to a 3D model of Doctor Who's Tardis in Gameware's HARVEE kit
An AR marker, this one relates to a 3D model of Doctor Who's Tardis in Gameware's HARVEE kit

These assist the recognition software in judging viewing altitude, distance and angle. Upon identification of a marker, the software will project or superimpose a virtual object or graphical overlay above the target, which becomes viewable on the screen of the AR device. As the device moves, the digital object orients in relation to the target in real-time:

Augmented Reality in action, multiple markers in use on the HARVEE system on a Nokia N73

The goal of some AR research is to free devices from markers, to teach AR devices to make judgements about spatial movements without fixed reference points. This is the cutting edge of AR research: markerless tracking. Most contemporary research, however, uses either marker-based or GPS information to process an environment.

Marker-based tracking is suited to local AR on a small scale, such as the Invisible Train Project (Wagner et al., 2005) in which players collaboratively keep virtual trains from colliding on a real world toy train track, making changes using their touch-screen handheld computers:

The Invisible Train Project (Wagner et al., 2005)

GPS tracking is best applied to large scale AR projects, such as ARQuake (Thomas et al, 2000), which exploits a scale virtual model of the University of Adelaide and a modified Quake engine to place on-campus players into a ‘first-person-shooter’. This application employs use of a headset, wearable computer, and a digital compass, which offer the effect that enemies appear to walk the corridors and ‘hide’ around corners. Players shoot with a motion-sensing arcade gun, but the overall effect is quite crude:

ARQuake (Thomas et al, 2000)

More data input would make the game run smoother and would provide a more immersive player experience. The best applications of AR will exploit multiple data inputs, so that large-scale applications might have the precision of marker-based applications whilst remaining location-aware.

Readers of this blog will be aware that AR’s flexibility as a platform lends applicability to a huge range of fields:

  • Current academic work uses AR to treat neurological conditions: AR-enabled projections have successfully cured cockroach phobia in some patients (Botella et al., 2005);
  • There are a wide range of civic and architectural uses: Roberts et al. (2002) have developed AR software that enables engineers to observe the locations of underground pipes and wires in situ, without the need schematics
  • AR offers a potentially rich resource to the tourism industry: the Virtuoso project (Wagner et al., 2005) is a handheld computer program that guides visitors around an AR enabled gallery, providing additional aural and visual information suited to each artefact;

The first commercial work in the AR space was far more playful, however: AR development in media presentations for television has led to such primetime projects as Time Commanders (Lion TV for BBC2, 2003-2005) in which contestants oversee an AR-enabled battlefield, and strategise to defeat the opposing army, and FightBox (Bomb Productions for BBC2, 2003) in which players build avatars to compete in an AR ‘beat-em-up’ that is filmed in front of a live audience; T-Immersion (2003- ) produce interactive visual installations for theme parks and trade expositions; other work is much more simple, in one case the BBC commissioned an AR remote-control virtual Dalek meant for mobile phones, due for free download from BBC Online:

A Dalek, screenshot taken from HARVEE's development platform (work in progress)
A Dalek, screenshot taken from HARVEE's development platform (work in progress)

The next entry in this series is a case study in AR development. If you haven’t already done so, please follow me on Twitter or grab an RSS feed to be alerted when my series continues.