Tutorial on Intersecting Art and Science in Augmented Reality

Augmented Reality applications in arts. From the conference Mensch und Computer, http://www.vielmehr.org/ Presenters: Anaisa Franco, Wendy Ann Mansilla

We as ARlab are happy to see that one of our first AR-art projects is mentioned in this tutorial!!
This tutorial will introduce participants to artistic AR creations applied to entertainment and new media art. It will give an insight on the current digital art scene and then be given concrete examples of AR usage in various creative activities, including digital entertainment and new media art.

1/ Background on digital art
2/ Game-oriented AR applications
3/ Links between new media arts and AR
4/ Review of AR in digital art

Since artists as known as artists they are altering the reality, concretizing ideas, concepts, which was only existing in a virtual space inside their minds. The artist’s creativity has a fundamental function on the development and presentations of new perceptions of the world. Through out their vision, they make innovations, always looking forward to open spaces and broke barriers, altering the infected normal vision of reality.

In this tutorial, we are presenting some works examples that deals somehow with Augmented, mixed or hybrid realities; works that are looking forward new ways of perceiving the world. What is perception? It can be a mental impression, or the ability to become aware of something through the senses. On Augmented, mixed or hybrid realities the perception of reality occurs on simultaneous experiences between the ‘real’ and the ‘virtual’. The perception of these different realities has become unified. Thus, mixed reality triggers a new extended vision; a polymorphous way of looking that includes a multiple perspective on realities. Augmented reality adds a layer, modifying, enhancing our notion of perception of the physical world. The works showed by the artists are trying to interfere on that believe of reality, mixing it up, expanding it, confusing our minds and spectators.

We usually believe in what we see, we do not question our eyes, nor our ears. If our senses tell us something, it is part of our reality; the reality is inside our mind being processed by our visual mechanisms, processing the way we perceive life. AR projects are dealing, questioning and investigating the expansion of the space, the reality of the world, mixing up physical space with digital layers of imagination. The description of Imagination is an “action of forming new ideas, or images, or concepts not presented to the senses”, by mixing reality with imagination we are able to augment the reality of the space and the senses simultaneously creating new spectrums to be perceived by the human senses.

The works presented are dealing with the interchange of realities, where artists augments the spaces, adds layers of creativity into physical spaces mixing up physical and digital data. The work Expanded Eye, by Anaisa Franco, 2008, developed during Interactivos08 at Medialab Prado, Madrid is an example of augmented interactive light sculpture. It is composed by a big transparent eye sculpture suspended from the ceiling; the sculpture structure and functionality is the same as an human eye, where the eye pupil is substituted by a camera and the projection occurs in the back part of the globe, working the same as the eye retina. The eye sculpture is looking to the user, but it is in fact the user´s eyes project inside the scultpture. The user´s eyes are captured by the camera and projected in the back part of it, augmenting the user´s reality of vision, expanding it.
The eye sculpture recognizes the blink and generates an interactive animation based on it. Each blink multiplies the number of eyes in the projection in a fragmented, hexagonal and dislocated way. The project is inviting people to look inside and to play with their eyes movements in a magical context where perception links with sensitive self-reflection and observation. A game where people are observed by their own eyes and plays with it through a transparent big eye interface that multiplies visitor’s point of view. The work Augmented Sculpture, developedby Pablo Valbuena, 2007, is focused on the temporary quality of space, investigating space-time not only as a three dimensional environment, but as space in transformation.
For this purpose two layers are produced that explore different aspects of the space-time reality. On the one hand the physical layer, which controls the real space and shapes the volumetric base that serves as support for the next level. The second level is a virtual projected layer that allows controlling the transformation and sequentially of space-time.
The blending of both levels gives the impression of physical geometry suitable of being transformed. The overlapping produces a three-dimensional space augmented by a transformable layer suitable to be controlled, resulting in the capacity through the installation of altering multiple dimensions of space-time.
These ideas come to life in an abstract and geometric envelope, enhanced with synesthetic audio elements and establishing a dialogue with the observer. Diego Diaz and Clara Boj created in 2007, the AR Magic System, which is an Interactive system inspired in the magic tricks of Deli magician that were later reproduced by the famous David Copperfield. This tricks consisted in cutting the head of a duch and a chicken, interchange them and later place them again in their correspondent bodies. In this installation, two or more persons are able to interchange their faces in real time by looking to themselves in a mirrror like videoprojection.
In this work people are able to switch faces, they have the possibility to see themselves in a body of a second person. The work Pileus, by Sho Hashimoto, is an umbrella connected to the Internet to make walking in rainy days fun. Pileus has a large screen on the top surface, a built-in camera, a motion sensor, GPS, and a digital compass, and it provides two main functions; A Social Photo-sharing and A 3D Map Navigation.
Pileus augments the reality of experiencing raining moments adding to it the possibility to be connected in real time with the internet, making the walk unique. Augmented Fish Reality, by Kenneth Rinaldo, 2004 is an interactive installation of 5 rolling robotic fish-bowl sculptures designed to explore interspecies and transpecies communication. These sculptures allow Siamese Fighting fish to use intelligent hardware and software to move their robotic bowls - under their control. Siamese fighting fish has excellent eyes, which allow them to see for great distances outside the water. They have color vision and seem to like the color yellow.
The artists is superposing the fishes movements adding to it a layer of reactions, the fishes controls the navigation and movements of a machine, the work expands normal parameters of movements in fish reality, building a cyborg extension for them. The artist and designer James Auger did lots of experiments mixing up physical spaces reflecting on traditional concepts of communication, interactions, extroverting medias. In the work The Interstitial Space Helmet, he provides an altavista on our camera/screen based interactions, exploring our use of electronic and digital technology in the context of communication and representation.
Traditionally employed in long distance broadcast interactions, cameras and screens may be considered as an extrovert media. The interstitial space helmet is conceived as a tool for exploring the consequences of applying this media in a more introverted or local experience, providing an alta-vista on our camera/screen-mediated existence.
It is becoming increasingly possible that the need for physical presence is diminishing as our interactions and relationships are being provided for by screen and camera based media. With anything up to 8 hours a day spent at our computer terminals and another three or four spent gazing at our televisions not being considered unusual.
Whilst the screen and camera provide an adequate conduit for many forms of interaction, their capacity for altering or even cheating reality has to be acknowledged hence their success in suspending our disbelief in film, advertising and propaganda broadcasts. Our screen-based interactions are not necessarily a seamless conduit and as such are open to a multitude of tweaks, filters and varying degrees of adjustment. The Isophone is essentially a telecommunications device providing a service that can be described simply as a meeting of the telephone and the floatation tank.
The user wears a helmet that blocks out all peripheral sensory distraction whilst keeping the head above the surface of the water. The water is heated to body temperature blurring the physical boundaries of the users body. Floating in this manner frees up to 90% of the brains workload normally engaged with calculating the lie of gravity. In combination, a space is created for providing a pure, distraction free environment for making a telephone call. The project Living-room2, by Jan Torpus, intends to explore different realities that have emerged as a result of digital technologies. Living-room2 takes the concept of augmented reality, or augmented space as a starting point. Living-room2´s specific approach towards Augmented Reality lies within its immersive application. The room is approached as an illusionary space, a simulation of a possible future experience of daily life instead of a tool for content development.
In ‘living-room2’ the space itself becomes the object of transformation. In the virtual layer, the room can be visually transformed, reconstructed, extended, etc. Thus, the user becomes part of an immersive environment.
By giving the visitor the possibility to “change the space”, living-room2 offers new opportunities for applications in the fields of Architecture, Scenography, Tourism, Museology and Education. AR+RFID Lab artist in residence Marina de Haas together with Wim van Eck, Jan Willem Brandenburg, Jurjen Caarls and Alwin van Rooij created a unique audio-visual AR environment called Out of the blue, it was made of round objects leading back to the basics of all organic shapes on earth. The ellipse shapes come out of the walls and find their way into the exhibition space. When you wear the Augmented Reality headset you are bound to stay in a circle of 1.50 meter from the table, which is set in the middle of the space near the wall. This means you will see and hear spheres flying through the space. The whole background setting is also guided by sound, moving, breathing, with sphere’s coming out of the walls and move back into the walls. Out of the blue, by Marina de Haas together with Wim van Eck, Jan Willem Brandenburg, Jurjen Caarls and Alwin van Rooij

Drops of white , by Marina de Haas is an augmented reality installation which means order, symmetry, rhythm, movement. Light is light, transparent, changing, intangible. The work shows an orchestra of delicate sounds and 3D realtime models dancing according to the music. You have to wear a headset of glasses and enjoy the time or the work. Audio Space, by Theo Watson is a 3D augmented aural space. A user wearing a headset can leave messages at any point within the room and hear all the sounds left by everyone before them spatialised as if the people were really still there. It becomes a room full of memories and remembering. LevelHead, by Julian Oliver is a spatial memory game inspired by the ‘memory loci’ systems used by the Ancient Greeks and later in Mediaeval Europe. The game uses a hand-held solid (wood or plastic) cube as its only interface. On-screen it appears that each face of the cube contains a little room, each of which are logically connected by doors. In one of these rooms is a character. By tilting the cube the player directs this character from room to room in an effort to find the exit. Some doors lead nowhere and will send the character back to the room they started in, a trick designed to challenge the player’s spatial memory. There are three cubes (levels) in total, which is connected by a single door. Players have the goal of moving the character from room to room, cube to cube in an attempt to find the final exit door of all three cubes. If this door is found the character will appear to leave the cube, walk across the table surface and vanish. The game then begins again. Julian said:
“Rather than chasing the grail of ‘total immersion’ I prefer to think of AR as a technology of enchantment, a game with belief and believing itself (practical applications aside). I think it’s more interesting to consider this technology not as Augmented Reality so much as Augmented Experience. I say this because AR already admits that experience is our primary interface with the world, that what we call reality is an idea derived from this interaction. As such, AR expresses something quite beautiful about humanhood, a primary desire to find new ways of being in the world through changing our experience of it. I consider experiencing an AR work to be in the same vein as engaging with Trompe l’Oeil, Perspectival Anamorphosis or Optical Illusion Art. They all seek to adapt experience to produce new ways of being somewhere, new ways of being. We know we’re being fooled but this is the point; it’s a game we’re knowingly playing with perception itself, enriching our immediate experience. Human beings have been doing this stuff for centuries and AR is no exception to this preoccupation.

DEVMAP developed by Workspace Unlimited is an interactive immersive virtual world conceived to be sensitive both to external media sources and human presence. The project, commissioned by V2_organisation, was presented for the first time during the DEAF04 festival and will be included in the Virtual World of Art network. The concept is based around the interactive use of live media feeds from the outside world and changing media manipulations triggered by visitors. Webcam footage, video output and live streams captured from the festival, are transformed, manipulated and mapped onto a virtual landscape that reflects and refracts the data of the festival and the interests of its users. Representing both the informative content of the festival and the mood or feeling it evokes, DEVMAP creates an augmented poetic experience connected to a temporary physical event.