May 25, 2018

Interacting in urban environments

Rod McCall
Mixed and augmented reality can enhance our experience of cities, allowing people to enter different time periods and take part in the urban design process.

Metropolitan settings provide us with a rich and diverse range of opportunities, which can be enhanced or brought to life with mixed and augmented reality technologies. One effort focusing on blending realities is IPCity.1 As part of this project consortium, we are developing technologies for urban planning, storytelling, location-based games and large-scale events. These systems incorporate new approaches, including tangible interfaces, augmented reality, multi-touch displays and mobile devices. While each system uses many different elements, they share many goals, such as altering how people interact with their city environment, as well as altering the user's sense of where they feel present and with whom. However, as all these experiences take place in complex uncontrollable environments, traditional presence research—which often makes use of studies under laboratory conditions—becomes impossible or of questionable value. Therefore, our project developed a richer, more complex approach to conceptualizing and evaluating presence and interaction in urban spaces.

One of IPCity's showcase projects is called TimeWarp (see Figure 1). It is a two-player, augmented reality time travel game that takes place in Cologne, Germany. In it, players must rescue the Heinzelmänchen (small elves) from different time periods. The current version uses a small ultra-mobile personal computer (UMPC), which acts as a magic lens into the gaming world. For example, as players move the UMPC around, the webcam captures images of the real environment. TimeWarp then superimposes graphics, such as characters, time portals, space ships and historical elements of the city, onto the webcam images. As similar games and technologies become more widespread, it becomes important to examine the effect they have on participants, non-participants and the wider environment. For example, what are the relationships between the real and virtual elements? What aspects make people feel as if they are more present within each of these elements? In order to address these issues, IPCity developed an overall conceptual model of presence and interaction in urban environments which reflect many themes, ranging from the underlying technology to ambience and collaboration. Moreover, the team also generated a set of design guidelines to provide game developers with insight.2

Players can select time periods from a menu (left), then travel through the time portal (right).

CityWall (see Figure 2), another IPCity project, is a large multi-touch display which lets people interact with user-generated content in real-world settings. For example, one or more people can interact with different elements simply by touching the screen. CityWall has been successfully installed in the Helsinki city centre, and to date thousands have used it. The display encourages novel behaviours from people who are often curious to find out about the new technology, even if they are not always interested in the content. For instance, some have used it to play simple games such as Pong, by throwing pictures across the screen to one another. Others used it as a tool to explore content, where one person assumes the role of the teacher. CityWall encourages new forms of interaction3 within public spaces by allowing participants to play with content in almost any way they choose, in some cases encouraging people to perform for their friends or passers-by.

Two people use CityWall in the centre of Helsinki.

Another project, the mixed-reality tent, introduces novel interface techniques into the urban planning process (see Figures 3 and 4). The tent enables anyone—from city dwellers to professionals—to participate in the planning process, and is placed at or near a location where redevelopment is taking place. It uses an interactive table top that contains objects representing real-world elements (e.g., buildings or benches). It also uses a large display of either a live video stream or a panorama of the area being developed. As people move the objects around the table, their ‘real-world’ equivalents move within the display. This approach allows groups of diverse stakeholders to collaborate in an expressive and natural way.

The mixed-reality tent.

The colour table inside the mixed-reality tent. A live video stream captures the area outside. The coloured blocks on the table represent objects within the real environment.

City life is also shaped by the stories which people create, share and modify. These stories can range from the very personal to major historical events. With IPCity the focus is on allowing normal citizens to share their stories through an application known as City Tales. It takes place in the famous Naschmarkt area of Vienna. Using location-based technologies, users can create stories of their experiences, which other users can subsequently experience as they walk around the area.

While each of the examples spans different aspects of experience and presence, all of them place at their core the underlying nature of city life. For example, all explore how sense of place is altered, either through story sharing, urban redesign or experiencing Cologne in different time periods. They also make use of location in different ways. For example, TimeWarp occurs in a specific set of places and uses both their ambience and history, whereas CityWall is as much about viewing pictures of other parts of Helsinki as it is about altering the ambience in its physical location. City Tales uses a rich location to encourage storytelling, while the mixed-reality tent exploits its physical proximity to the development site to assist the redevelopment process. Another key element across most of these projects is the social aspect, in particular the use of collaboration and negotiation to enhance game play within TimeWarp, or as a method of encouraging diverse interactions with CityWall. Collaboration also plays a key role in the mixed-reality tent, as all those present take part in design decisions. When designing urban mixed reality, it is often too easy to focus on the purely technical aspects. While these are very important, the ultimate success of the system depends just as much on understanding aspects such as collaboration, negotiation, ambience and how users will appropriate the technologies. In future, IPCity will explore how sense of presence changes during a mixed-reality experience: for example, why and when people feel more present within virtual rather than real aspects.


Rod McCall
Fraunhofer Institute for Applied Information Technology (FIT)

Rod McCall is the deputy head of the Collaborative Virtual and Augmented Environments Department. He was previously a European Research Consortium for Informatics and Mathematics research fellow at Fraunhofer FIT and the Centre for Public Research Gabriel Lippmann (Luxembourg).

  1. IPCity Website. Accessed 10 July, 2009.

  2. A. Braun, I. Herbst, W. Broll and R. McCall, TimeWarp: interactive time travel with a mobile mixed reality game, Proc. 10th Int'l Conf. Human Comput. Interact. Mobile Dev. Serv., pp. 235-244, 2008. doi:10.1145/1409240.1409266

  3. P. Peltonen, E. Kurvinen, A. Salovaara, G. Jacucci, T. Ilmonen, J. Evans, A. Salovaara and A. Oulasvirta, It's mine, don't touch!: interactions at a large multi-touch display in a city center, Proc. 26th SIGCHI Conf. Human Fact. Comput. Syst., pp. 1285-1294, 2008.

DOI:  10.2417/2200907.1731