Learn Piano through AR

I like this:

The Projected Instrument Augmentation system (PIANO) was developed by pianists Katja Rogers and Amrei Röhlig and their colleagues at the University of Ulm in Germany. A screen attached to an electric piano has colourful blocks projected onto it that represent the notes. As the blocks of colour stream down the screen they meet the correct keyboard key at the exact moment that each one should be played.

Florian Schaub, who presented the system last month at the UbiComp conference in Zurich, Switzerland, said that users were impressed by how quickly they could play relatively well, which is hardly surprising given how easily we adapt to most screen interfaces these days.

But while there is real potential for PIANO as a self-guided teaching aid, in my view it’s the potential for a really tight feedback loop that makes this most interesting, and potentially more widely applicable.

When a piano teacher corrects a student’s mistake, they will perhaps specify one or two things that need improving, but this approach would sense each incorrect note and could provide an immediate visual response, flashing red for instance, conditioning the student to success more quickly.

via New Scientist.

Screens

[box]This post originally appeared on the FTMF.info planning blog.[/box]

Within the pages of Watchmen, Adrian Veidt, the so-called “smartest man in the world”, esteemed business leader and founding member of the Crimebusters is shown at a wall of televisions, each tuned to a different channel. He uses this clatter of imagery, sound and motion to make sense of the current geopolitical and social climate and to act upon it:

Watchmen 10 - 08

Reads a bit like Social Media Monitoring, doesn’t it? But Adrian Veidt, AKA Ozymandias, was multi-screening before it was even a thing. Nowadays, we do it by default, up to 60% of the time, and in the age of 4.6 connected devices per household it just comes naturally.

Multi-screening can be simultaneous (same journey across devices, as in the above case), sequential (different journeys across devices simultaneously), or separate (different journeys across devices simultaneously) – but it’s an emergent behaviour that needs much further inquiry. There are few real thought leaders, except for SecondSync perhaps, or Microsoft, who so succincinctly define the terms I’ve used here.

One other thought leader is Kevin Kelly, co-founder of Wired, whose view is that as screens proliferate further into each aspect of our lives, their role becomes not just to display but also to help filter information – we’re literally ‘screening out’ the stuff we don’t want to see.

Watch his talk on ‘screening’ and five other ‘Verbs for the New Web’ below – it’s great:

And finally, screens can also be mirrors, lenses or even windows. Clever, aren’t they?!

Nucleus Medical Media

Nucleus Medical Media’s 2011 3D medical animation demo reel shows surgery, anatomy, mechanism of action (MOA), and physiology produced for medical devices, pharmaceutical companies, biotechnology, marketing agencies, lawyers, and more.

What Nucleus don’t include in their showreel’s YouTube description, but will become apparent, is that they are probably among the finest computer animators working today.

In my view they depict very complicated biomechanical processes so very clearly, and quite beautifully too. Here’s the aforementioned showreel:

My question is, how is it these guys are nailing it so hard?! Are they scientists trained in CAD, or the reverse?

Missing: Bella the Dog

This isn’t the sort of thing I’d usually write about here, so for the hardcore who’ve come to expect a blend of media and tech thoughtfulness, an alternative reading is that this is a a live case-study in how digital can play troubleshooter to certain real-world issues…

Those who follow me on Twitter will know the news already:

There’s been a search out for her ever since, with all five Saunters and some amazing neighbours clomping about in our wellies, flashlights in hand.

I’m pretty sure we’ve thought of everything: we’ve scouring the entire surrounding area (see map); spoken to people all over town; put up flyers and posters in areas of high traffic; we’re leaving a scent-trail back to the house from where she went missing; notified all the necessary authorities and secured some media coverage (Star Radio, GumtreeDogLost).

As you’ll see, what we need now are some more eyes and ears…


View Larger Map

Four hours ago, something incredible happened. A lady I’ve never met created the Facebook group ‘Help find BELLA‘ and invited some friends.

Since then it’s grown, and now a whopping 128 people have joined. Tomorrow, there’s a search planned for 10.30am in the Littleport area. Truly awesome. Thank you all. We can’t do this without you!

[box]Update: We found her! I’ll update this blog post when we find her. In the meantime, please tweet, Facebook and G+ this blog post to anybody in your network that lives in the Cambridgeshire/Norfolk area.[/box]

Many thanks indeed, Tom.

Applying Benjamin

AR technology grants virtual objects presence in physicality. This is a concept ripe with potential for academic study. Baudrillardian thought states that we would seek to assign these objects similar values to other, real world objects. The School of Economy offers the truth that scarcity creates value. Our physical, tangible world is finite, but The Virtual is infinite. Now, any object in space denies the opportunity for another object to exist there, however we know this not to be true in an Augmented Reality. I proffer that virtual objects taking up space in a finite world hold economic value, but Benjamin might argue against this view.

In his piece The Work of Art in the Age of Mechanical Reproduction (1968) which was written in the thirties, Walter Benjamin argues that film and other ‘reproductive’ media diminished or destroyed the aura that had belonged to earlier art. As Bolter et al. (2006: 24) put it, “Aura belongs to works of art that are unique, as most art was before technologies of mechanical reproduction. Aura is the sense of the ‘here and now’ that each such work possesses because of its history of production and transmission. This uniqueness lends to each painting or sculpture a special quality, which can in turn evoke an attitude of reverence on the part of the viewer”. Benjamin (1968: 222-223) said that:

We define the aura as the unique phenomenon of a distance, however close it may be. If, while resting on a summer afternoon, you follow with your eyes a mountain range on the horizon or a branch which casts its shadow over you, you experience the aura of those mountains, of that branch.

Benjamin (1968: 222-223)

In this we see that Benjamin equates Aura with immediacy and the uniqueness of the moment of an artefact’s viewing. Since each moment is unique for each of us, it is economically valuable, auratic even. The information age of perfect digital reproduction does not detract from the personal uniqueness of experiential reality, only diminishes the aura offered by certain mass-produced elements within it. In Benjamin’s time, art was sanctified and extremely auratic. Now, most view classical art as a digital copy or encyclopaedia reference, which embodies little of the aura of the original piece. Travelling to the Louvre offers only the added auratic experience not gained from a home computer, where one’s experience is more like anybody else’s experience of the subject, but visitors must get very close to a works in order to experience what Aura remains there.

Benjamin might say that in a world of perfect digital reproduction, virtual objects can have no Aura. The Magic Lens is not spiritually void, however, and my Reality 2.0 is not a stagnant place. It buzzes with an aura of its own, borne from the uniqueness of each interaction, live and in context. A virtual object can be duplicated infinitely, but its location in space and the immediacy of its presence in that moment that helps shape its unique meaning for every AR denizen. Physicality offers the backdrop to the unique experiences offered by virtuality, indeed, experiential reality is improved through the wider opportunity for unique moments.

Returning to the economics of virtual space, virtual objects cannot hold currency in and of themselves, but the context surrounding them has value. Advertisers could exploit this if they act fast in commercialising AR space. More interesting to me is the potential for unique virtual art objects and sculptures that could hold purely aesthetic value. In this case, interested parties would seek out the object and view its intricacies and movements in the space it was designed to fill. There is much room for spatial enquiry in this field, some of which I will come onto in my next section.

Applying Benjamin

Applying Baudrillard

For Jean Baudrillard (1983), “at any moment in the course of our modernity, a particular arrangement of signifying objects and images conditions the way we see the world” (Clark, 1995). “Each major transformation is accompanied by a feeling of disorientation and discomfort over the loss of the previous ‘reality’. This effects a recourse into the imagined certainties of the receding order to ground or stabilise that which is new. In this way, “reality loops around itself”, as “each phase of value integrates into its own apparatus the anterior apparatus as a phantom reference, a puppet or simulation reference”” (Baudrillard, 1988: 145, 121; cited in Clark, 1995). In these words, we see Baudrillard’s perspective can apply neatly to my analysis of Mobile AR. Taking up where McLuhan left us- a view of the Magic Lens constrained by its deterministic overtones- Baudrillard injects the much-needed element of an actively social construction of Mixed Reality, whilst grounding my work in his Postmodern thought on Virtuality.

I am interested in the view that iterations of reality, whilst overlapping and viewable through the Magic Lens, support and influence each other’s existence within a wider structure. I could live wholly in The Virtual, and bring to it conceptions of the reality from whence I came. We see a similar behaviour in Alternate Reality games such as Second Life (Linden Lab: 2003) or The Sims (Maxis: 2000) whereby developers program known physical world causalities, behaviours and actions despite the near-limitless formal opportunities offered by the medium. Users, when given freedom, will likely bring their own conceits and personal experiences to these alternate realities, thereby foregoing what else might be possible in favour of their own culturally-inherited drives and ambitions. The Magic Lens presents a wholly new canvas for the social construction of reality. The collaborative and democratic Mobile 2.0 ethos that Nokia hope to breathe into Mobile AR could falter if users bring too much of our present iteration of reality to it. The Magic Lens offers an opportunity to reshape The Real, not solely through tagging buildings or leaving messages floating in mid-aid, but through the lessons we might learn through engaging with each other in a new way.

Baudrillard focused his work on how we interface with information, and how we build it into our view of reality. He posited that The Media had hijacked reality, becoming a powerful force in the construction of hyper-reality, a social reality that has become more powerful than we exert control over. Through the Magic Lens, we might give form to some aspects of hyper-reality. The medium allows for virtual elements to co-exist with real objects occupying space in the user’s own hyper-reality. In this way, each user can choose which hyper-reality they want to exist in, whether it is one in which 3D AR avatars walk the streets and go about their virtual lives; or one where arrows and directions graphically point out where to go to fulfil a shopping list’s requirements. The Magic Lens makes a shift from mass-media control to personalised, user-focused context-based reality: Reality 2.0 if you will.

Assuming AR does present a new layer to reality, there are certain Baudrillardian imperatives that we will bring to this landscape. One such imperative links the physical properties of real-world space- gravity, mass, optics- to our new environment. To make sense of virtual elements in their context we will employ what we already know about the environment we are in. This means that the most prized virtual objects will exhibit expected behaviour, intuitive interactivity and will be visually suited to its surroundings. Similarly, an object’s location in space alters its perceived importance. I would argue that should a common Mixed Reality exist, governing bodies would write entire protocol for the positioning and size of virtual objects so that one contributor could not take up more than his worth. Important to consider is that even writing hypothetically I am bringing Baudrillardian imperatives to task, applying democracy to a non-existent world! Baudrillard’s “reality loops around itself” has a troublesome effect on my analysis. Let me instead take a fresh perspective, in my next section written from the perspective of Walter Benjamin…