Towards a Cyberpunk Sky

When plans for The Shard were unveiled my first thoughts were that its piercing aesthetic would be too harsh for the London skyline. But now it’s been ‘topped out’ and soon to open to the public, it’s time to get used to it.

I heard somewhere that Irvine Sellar, the property magnate behind the tower, intends for his creation to stand for more than 200 years. Well, of course it does, but this concept did the most to shift my perspective.

Imagine how this city will feel in two centuries: totally transformed, utterly modern and yet (hopefully) still uniquely London. The Shard is another step towards an exciting Future London.

To illustrate my point, here’s a video flyover showing the tower in context. Watch closely and you’ll see  London’s slow climb towards a cyberpunk sky:

Synesthesia

syn·es·the·sia syn·aes·the·sia (sĭn’ĭs-thē’zhə):

  • A condition in which one type of stimulation evokes the sensation of another, as when the hearing of a sound produces the visualization of a color.
  • A sensation felt in one part of the body as a result of stimulus applied to another, as in referred pain.
  • The description of one kind of sense impression by using words that normally describe another.

It is widely considered that synesthesia arises in brains where from a young age, neurological paths that service our sensory perception do not become entirely defined, resulting in cross-chatter between the senses.

Famous synesthetes reportedly include painter David Hockney, who perceives music as color, shape, and configuration; composer Wassily Kandinsky, who had a four senses combined: color, hearing, touch, and smell; and physicist Richard Feynman who percieves elements of equations as different colours.

What a cool way to see the world – a totally unique perception of reality. And how hard must it be to express this reality to others?! Synesthetes must feel in some part driven to demonstrate their unique perspective, perhaps driving them to create great cultural artefacts that we can all enjoy.

One filmmaker has produced their interpretation of a reality touched by synesthesia. I urge you to watch this great short film for a taste of the condition:



Synesthesia
from Terri Timely on Vimeo.

Can anyone out there think of a way us non-synesthetes might be able to experience the world like this, via the medium of Augmented Reality?
Would love to explore some ideas.

Conclusion

I set out to assess the implications of a wholly new medium, one which had received little academic attention written from a media theoretical perspective. I made clear use of an industry connection to gain inside knowledge of the developments occurring to bring this medium to the mainstream. Building a methodology that could sustain the level of analysis that I hoped to achieve, I observed the interactions between technology and industry, market forces and cultural influences. Having positioned my subject at the crest of a curling wave, I employed critical media theory to explore the potential implications of my subject in its wider context of social reality. This ambitious task has granted me insight into how the complex interactions of various fields give rise to social change. Along the way I have revealed seams rich in potential for further analysis.

McLuhan is proven to apply to yet another medium, the perspective he offers served my analysis quite well. A further exploration might make use of his Acoustic and Visual Space probe, Cavell’s basis for McLuhanistic spatial enquiry in his book McLuhan in Space (2002) would be a good starting point for such work, since it applies McLuhanism to the media of time and space, thus a good start for work on the presence of virtual objects. Media analysts occupied with screen design might wish to extend Bolter and Grusin’s (1999) work on remediation to the emergent Mobile AR technology, perhaps from an explicit digital gaming perspective. Those with interest in advertising or business as applied to Augmented Reality would do well to continue Benjaminian thought to its logical end: manipulating a virtual object to hold added-value for commercial enterprise. Those with a more creative bent might enjoy a study of the public perception of AR artworks using Benjamin also. There is scope for research into AR-based social interactions; gaming styles; immersion and identity formation, but this sort of work necessitates that first Mobile AR spends at least some time in public consciousness.

Finally, I believe that I have convincingly laid out an argument showing that AR is currently being developed and packaged as an entertainment technology, but its potential for community-driven, self-proliferating excitement of user-created content makes AR a significant and culturally-transformative technology. Convergence between media types will enable and drive the creation of innovative content which if successful will itself rely on new ways of accessing and viewing content and ultimately new forms of content and user experience entirely. We are at the crest of a wave. Will it wither and let a larger wave pass above it, or will it grow to reach tidal proportions? Despite my predictions, only time will tell.

Applying Baudrillard

For Jean Baudrillard (1983), “at any moment in the course of our modernity, a particular arrangement of signifying objects and images conditions the way we see the world” (Clark, 1995). “Each major transformation is accompanied by a feeling of disorientation and discomfort over the loss of the previous ‘reality’. This effects a recourse into the imagined certainties of the receding order to ground or stabilise that which is new. In this way, “reality loops around itself”, as “each phase of value integrates into its own apparatus the anterior apparatus as a phantom reference, a puppet or simulation reference”” (Baudrillard, 1988: 145, 121; cited in Clark, 1995). In these words, we see Baudrillard’s perspective can apply neatly to my analysis of Mobile AR. Taking up where McLuhan left us- a view of the Magic Lens constrained by its deterministic overtones- Baudrillard injects the much-needed element of an actively social construction of Mixed Reality, whilst grounding my work in his Postmodern thought on Virtuality.

I am interested in the view that iterations of reality, whilst overlapping and viewable through the Magic Lens, support and influence each other’s existence within a wider structure. I could live wholly in The Virtual, and bring to it conceptions of the reality from whence I came. We see a similar behaviour in Alternate Reality games such as Second Life (Linden Lab: 2003) or The Sims (Maxis: 2000) whereby developers program known physical world causalities, behaviours and actions despite the near-limitless formal opportunities offered by the medium. Users, when given freedom, will likely bring their own conceits and personal experiences to these alternate realities, thereby foregoing what else might be possible in favour of their own culturally-inherited drives and ambitions. The Magic Lens presents a wholly new canvas for the social construction of reality. The collaborative and democratic Mobile 2.0 ethos that Nokia hope to breathe into Mobile AR could falter if users bring too much of our present iteration of reality to it. The Magic Lens offers an opportunity to reshape The Real, not solely through tagging buildings or leaving messages floating in mid-aid, but through the lessons we might learn through engaging with each other in a new way.

Baudrillard focused his work on how we interface with information, and how we build it into our view of reality. He posited that The Media had hijacked reality, becoming a powerful force in the construction of hyper-reality, a social reality that has become more powerful than we exert control over. Through the Magic Lens, we might give form to some aspects of hyper-reality. The medium allows for virtual elements to co-exist with real objects occupying space in the user’s own hyper-reality. In this way, each user can choose which hyper-reality they want to exist in, whether it is one in which 3D AR avatars walk the streets and go about their virtual lives; or one where arrows and directions graphically point out where to go to fulfil a shopping list’s requirements. The Magic Lens makes a shift from mass-media control to personalised, user-focused context-based reality: Reality 2.0 if you will.

Assuming AR does present a new layer to reality, there are certain Baudrillardian imperatives that we will bring to this landscape. One such imperative links the physical properties of real-world space- gravity, mass, optics- to our new environment. To make sense of virtual elements in their context we will employ what we already know about the environment we are in. This means that the most prized virtual objects will exhibit expected behaviour, intuitive interactivity and will be visually suited to its surroundings. Similarly, an object’s location in space alters its perceived importance. I would argue that should a common Mixed Reality exist, governing bodies would write entire protocol for the positioning and size of virtual objects so that one contributor could not take up more than his worth. Important to consider is that even writing hypothetically I am bringing Baudrillardian imperatives to task, applying democracy to a non-existent world! Baudrillard’s “reality loops around itself” has a troublesome effect on my analysis. Let me instead take a fresh perspective, in my next section written from the perspective of Walter Benjamin…

Deconstructing AR

The next four entries in my series on AR will assess it’s potential socio-cultural impact, which has now started to begin.

We have now reached a depth of analysis that has taken in the historical, economic and technologic paths leading to Mobile AR, but there are greater depths to plunge. From this vantage point, the question arises: how will the user experience this new medium? This section assesses the implications of Mobile AR from a critical and media theoretical perspective, and offers the other half of my close analysis.

There are very few published articles on the subject of AR, let alone Mobile AR, and fewer still that offer a media theory perspective on user-interactions in augmented environments. There are a clutch of articles that refer to such an area, but like my own analysis, these academics have resorted to applying old frameworks to a new medium, with McLuhan, Baudrillard and Benjamin seeming the most applicable theorists. My own thoughts on Mobile AR resonate with these thinkers yet no such analysis has involved all three epistemologies, so I must tread carefully…

Virtual Reality

AR is considered by some to be a logical progression of VR technologies (Liarokapis, 2006; Botella, 2005; Reitmayr & Schmalstieg, 2001), a more appropriate way to interact with information in real-time that has been granted only by recent innovations. Thus, one could consider that a full historical appraisal would pertain to VR’s own history, plus the last few years of AR developments. Though this method would certainly work for much of Wearable AR- which uses a similar device array- the same could not be said for Mobile AR, since by its nature it offers a set of properties from a wholly different paradigm: portability, connectivity and many years of mobile development exclusive of AR research come together in enhancing Mobile AR’s formal capabilities. Despite the obvious mass-market potential of this technology, most AR research continues to explore the Wearable AR paradigm. Where Mobile AR is cousin to VR, Wearable AR is sister. Most published works favour the Wearable AR approach, so if my assessment of Mobile AR is to be fair I cannot ignore its grounding in VR research.

As aforementioned, VR is the realm at the far right of my Mixed Reality Scale. To explore a Virtual Reality, users must wear a screen array on their heads that cloak the user’s vision with a wholly virtual world. These head-mounted-displays (HMD’s) serve to transpose the user into this virtual space whilst cutting them off from their physical environment:

A Virtual Reality HMD, two LCD screens occupy the wearer's field of vision
A Virtual Reality HMD, two LCD screens occupy the wearer's field of vision

The HMD’s must be connected to a wearable computer, a Ghostbusters-style device attached to the wearer’s back or waist that holds a CPU and graphics renderer. To interact with virtual objects, users must hold a joypad. Aside from being a lot to carry, this equipment is restrictive on the senses and is often expensive:

A Wearable Computer array, this particular array uses a CPU, GPS, HMD, graphics renderer, and human-interface-device
A Wearable Computer array, this particular array uses a CPU, GPS, HMD, graphics renderer, and human-interface-device

It is useful at this point to reference some thinkers in VR research, with the view to better understanding The Virtual realm and its implications for Mobile AR’s Mixed Reality approach. Writing on the different selves offered by various media, Lonsway (2002) states that:

“With the special case of the immersive VR experience, the user is (in actual fact) located in physical space within the apparatus of the technology. The computer-mediated environment suggests (in effect) a trans-location outside of this domain, but only through the construction of a subject centred on the self (I), controlling an abstract position in a graphic database of spatial coordinates. The individual, of which this newly positioned subject is but one component, is participant in a virtuality: a spatio-temporal moment of immersion, virtualised travel, physical fixity, and perhaps, depending on the technologies employed, electro-magnetic frequency exposure, lag-induced nausea, etc.”

Lonsway (2002: 65)

Despite its flaws, media representations of VR technologies throughout the eighties and early nineties such as Tron (Lisberger, 1982), Lawnmower Man (Leonard, 1992) and Johnny Mnemonic (Longo, 1995) generated plenty of audience interest and consequent industrial investment. VR hardware was produced in bulk for much of the early nineties, but it failed to become a mainstream technology largely due to a lack of capital investment in VR content, a function of the stagnant demand for expensive VR hardware (Mike Dicks of Bomb Productions: personal communication). The market for VR content collapsed, but the field remains an active contributor in certain key areas, with notable success as a commonplace training aid for military pilots (Baumann, date unknown) and as an academic tool for the study of player immersion and virtual identity (Lonsway, 2002).

Most AR development uses VR’s same array of devices: a wearable computer, input device and an HMD. The HMD is slightly different in these cases; it is transparent and contains an internal half-silvered mirror, which combines images from an LCD display with the user’s vision of the world:

An AR HMD, this model has a half-mirrored screen at 45 degrees. Above are two LCDs that reflect into the wearer's eyes whilst they can see what lies in front of them
An AR HMD, this model has a half-mirrored screen at 45 degrees. Above are two LCDs that reflect into the wearer's eyes whilst they can see what lies in front of them

 

What Wearable AR looks like, notice the very bright figure ahead. If he was darker he would not be visible
What Wearable AR looks like, notice the very bright figure ahead. If he was darker he would not be visible

There are still many limitations placed on the experience, however: first, the digital graphics must be very bright in order to stand out against natural light; second, they require the use of a cumbersome wearable computer array; third, this array is at a price-point too high for it to reach mainstream use. Much of the hardware used in Wearable AR research is bought wholesale from liquidized VR companies (Dave Mee of Gameware: personal communication), a fact representative of the backward thinking of much AR research.

In their work New Media and the Permanent Crisis of Aura Bolter et al. (2006) apply Benjamin’s work on the Aura to Mixed Reality technologies, and attempt to forge a link between VR and the Internet. This passage offers a perspective on the virtuality of the desktop computer and the World Wide Web:

“What we might call the paradigm of mixed reality is now competing successfully with what we might call ‘pure virtuality’ – the earlier paradigm that dominated interface design for decades.
In purely virtual applications, the computer defines the entire informational or perceptual environment for the user … The goal of VR is to immerse the user in a world of computer generated images and (often) computer-controlled sound. Although practical applications for VR are relatively limited, this technology still represents the next (and final?) logical step in the quest for pure virtuality. If VR were perfected and could replace the desktop GUI as the interface to an expanded World Wide Web, the result would be cyberspace.”

Bolter et al. (2006: 22)

This account offers a new platform for discussion useful for the analysis of the Internet as a component in Mobile AR: the idea that the Internet could exploit the spatial capabilities of a Virtual Reality to enhance its message. Bolter posits that this could be the logical end of a supposed “quest for pure virtuality”. I would argue that the reason VR did not succeed is the same reason that there is no “quest” to join: VR technologies lack the real-world applicability that we can easily find in reality-grounded media such as the Internet or mobile telephone.

Constructing A Methodology

Mobile AR is still highly prototypical, and has not received much previous academic attention thus far. A deep analysis of this technology and its implications requires a specially developed methodology, a methodology which acknowledges the pre-release status of the technology, and recognises that Mobile AR represents a fusion of a number of different media technologies. Given that there is no fixed point of entry for analysis, I look to academics writing on the subject of other radical and emergent technologies that (at the time of publishing) were yet to reach the mainstream.

At this early stage of the product cycle there is an interesting interplay between fields. This interplay is assessed in Digital Play: The Interaction of Technology, Culture, and Marketing by Kline et al. (2003), and it raises some arguments useful to the further analysis of Mobile AR as a cultural artefact. One of its lessons is that assessing new technologies is fraught with a long-standing academic and cultural issue: the problem of technological determinism. The theory is that new technologies drive social, political and cultural changes, and that the perceived linearity of technical progression is somehow representative of humanity’s own progression, parallel trajectories dependent on the other’s existence. The weaknesses arise when one assigns these same values to their own assessments, which isolate the subject technology from its wider context. Indeed, it is often forgotten that in order to achieve these innovations, social, economic, political and cultural forces have all worked in collusion. Digital Play (Kline et al., 2003), quotes Leiss (1990) in an especially provoking summarisation:

“Strictly speaking, there are no imperatives in technology. The chief mistake … is to isolate one aspect (technology) of a dense network of social interactions, to consider it in abstraction from all the rest, and then relate it back to that network as an allegedly independent actor.”

Leiss (1990: 2) in Kline et al. (2003:8)

Leiss’ point is that academic enquiry should seek to observe its subjects in the light of their true context. He highlights the importance of the “network” as the source of each technology, denying the idea that modern culture is ‘Under Technology’s Thumb’ (Leiss, 1990). A personal observation is that within the “network” also lie the forerunning technologies that gave rise to the newest developments, and the means to develop them further. This idea recalls McLuhan, who to the detriment of Leiss’ argument, was sometimes known as “the most famous media technological determinist” (Straubhaar & LaRose, 2005: 51) who, in reference to man’s “perpetually … modifying his technology” McLuhan (1964: 46) states that “man becomes, as it were, the sex organs of the machine world, as the bee of the plant world, enabling it to fecundate and to evolve ever new forms”. He hereby suggests a hidden complexity to human-technologic interactions, a complexity I return to later in this work. For now though, these thinkers’ opposing perspectives make a further analysis rather difficult. I recognise McLuhan’s view that there are forces at work within the “network” that need to be addressed, but accept Leiss’ view that I should view the web of interactions as a whole, in order that technological determinism cannot skew my findings. I must reconcile these perspectives in my own approach. Seeking to refrain from any dangerously deterministic hyperbole, I continue the assessment of Mobile AR as an emergent and potentially “network” enhancing new medium, but from which determinist-proof methodology?

Digital Play, though referring mainly to the digital games industry, looks into the complex dynamics between developer, distributor, market and economy. Its critical evaluation of this medium adopts a methodology that suits my own AR enquiries:

“The story of the emergence of interactive play and of its uncertain crisis-filled transformation into one of the premier industries of digital globalized capital is both exciting and revelatory. Historical perspective is vital to critical understanding. We strongly agree with Williams that it is impossible to diagnose the cultural impact of a new medium until the specific institutional circumstances of its development are understood. Moreover, critical media analysis requires historical perspective in order to argue against the deterministic view that technology “is a self-acting force which creates new ways of life [Williams, 1992: 8]”.

Kline et al. (2003:79)

Now, since Mobile AR is such a new and radical technology, it is best considered in terms of other, previously radical technologies and their own timely impact. This approach recognises the emergent nature of Mobile AR technology and the lack of current research in the field, but also allows the opportunity to reflect on the implications of this technology in a relatively safe way: that is, through the lessons learned from full-fledged media. I propose that a useful lens through which to view Mobile AR is offered by an historical analysis of Mobile AR’s better established component media, with a view to producing an understanding of the implications AR holds for society. This approach allows AR to be considered as a product of a convergence of paths: technologic, academic, social and economic, providing the basis for deeper analysis as a consequence.