April 29, 2017

Reflective assistant: an ideal friend

Nikola Serbedzija
The ultimate goal of the 'reflective approach' in a multidisplinary computing environment is to design genuinely friendly future control systems suited to the needs of individual users.

The basic operating principle in computing has always been the input–process–output process. From the early days, computers have been cycling endlessly between these three simple steps, to the extent that they have now become an unavoidable part of everyday life. However, we are starting to experience an interesting shift associated with the emergence of new computer systems functioning in sense–process–react cycles. This innovative approach allows for the development of genuinely user-friendly systems. It equips computers with a means to construct a psychophysiological image of both the user and the operating environment, and to adapt the system's functionality accordingly. These essential features of pervasive-adaptive systems1 are currently being developed on the basis of a ‘reflective’ approach.2 A generic software framework is under construction providing a set of practical tools for building context-aware self-organized control systems that enable seamless user interaction.

Creating friendly assistance requires understanding the patterns of friendliness and contrasting these with experiences of unfriendly behavior. Imagine being suddenly left alone. Betrayed and abandoned, one will be in pain, both emotionally and physically. The reflective see-saw (see Figure 1) should prevent this from happening. It functions discretely, remaining neutral as long as both parties are present. If one party suddenly leaves, it senses the resulting imbalance and reacts by injecting benevolent compensation.

Reflective see-saw keeping the balance even if one party suddenly leaves.

A reflective vehicle, equipped with an adaptive-control system, ensures a balance in driving in terms of safety, comfort, and enjoyment.

The basic concept of a helping hand maintaining physical and emotional balance can be applied to a number of complex situations, such as in the context of a driving experience. Generally speaking, it is better to drive in company. A front-seat passenger usually observes the driver carefully, keeps an eye out for potentially dangerous situations the driver cannot see, creates a lively atmosphere on longer trips, and hence generally assists the driver significantly. However, group rides are not very common. The reflective vehicle (see Figure 2) should overcome the possible shortcomings associated with a solitary drive by adopting the role of a friendly passenger.3 Its task is to observe and take into account the driver's emotional, cognitive, and physical state, as well as vehicular, driving, and traffic conditions, optimize the vehicle configuration, and actively participate in the complex driving process. The reflective-vehicle concept is aimed at implementing adaptive control in vehicles, hence contributing to more secure, pleasant, and effective driving behavior.

Additional areas of possible applications requiring friendly behavior include reflective mobile phones which offer hints (whom to call, whose call to ignore, where to go, what to do) depending on the owner's emotional, mental, or physical state and the situation at hand, reflective music players which select music according to the user's mood, reflective advertising panels which adapt their content and presentation according to the number and kind of viewers and their interests, and reflective homes for the elderly which recognize any needs and weaknesses of their inhabitants and assist or call for assistance if and when required.

The driving force behind the sense–process–react paradigm is affective computing,4 a rapidly evolving discipline which investigates how to capture and interpret affects (emotions and their accompanying movements), postures (ways in which a person carries their body), and gestures (expressive, meaningful bodily movements). It relies on so-called psychophysiological variables, which can be measured using modern microelectronic sensors. Through careful data analysis one can precisely determine a user's emotional, mental, or physical state, particularly in situations where predictable behavior prevails. Once the user's state has been evaluated, the system can be fine-tuned to provide a supportive reaction.

Developing software to control pervasive and adaptive systems involves tasks including real-time sensor/actuator control, user-profile and scenario analyses, affective computing, self-organization, and adaptation. To accomplish these requirements, modular middleware architecture is being designed which promises dynamic and reactive adaptive behavior. It consists of a tangible, a reflective, and an application tier (see Figure 3).

The tangible tier is a low-level layer required for sensor and actuator control. It offers its atomic services to the rest of the system. The reflective tier is the central layer combining the services offered by the lower tier with user-profile and scenario descriptions. This allows for more complex services which evaluate the user's emotional, cognitive, and physical state, as well as the operating environment. If needed, system (re-)action is induced. Finally, the application tier forms a high-level layer which runs and controls the entire system.

The core of the service-oriented architecture is the reflective tier, equipped with a specially developed reflective service bus that manages the various services. It contains components that perform transparent messaging, service registering, discovery and deployment, event propagation, user-profile and scenario analyses, situation reasoning, and complex-service composition, among others. Its operation is based on closed-loop control, a major behavioral pattern of reflective systems.

Reflective middleware with closed-loop control. The control loop (initialized through the user-profile and scenario settings) starts by sampling psychophysiological measurements. It then continues with their analysis and finishes with an adaptive-system reaction. In a subsequent iteration the system influence can also be sensed and fine-tuned as appropriate.

The ‘ideal friend’ metaphor underscores the ultimate goal of the reflective approach, aimed at designing genuinely friendly future control systems suited to the needs of individual users. In spite of the inherent complexity, reflective software strives for simplicity in offering a generic solution for a wide range of problems. Such a user-centered approach operates on the basis of non-explicit man–machine interactions. This is needed in many application domains, ranging from toys to embedded real-time systems. As technology and science advance, the potential spectrum of further work in this area is wide. In a multidisciplinary endeavor, psychologists need to shed more light on affective analyses, visionary practitioners will need to create more application scenarios, philosophers and sociologists are asked to consider what is right and wrong, and computer scientists must ensure that the ‘ideal’ friend is genuinely personal and not just a friend of one's ‘big brother.’ Once we have achieved this, we will be getting close to the ideal of creating a loyal, reliable, and omnipresent friend.


Nikola Serbedzija
Fraunhofer FIRST

Nikola Serbedzija is a senior scientist. His major research areas include ubiquitous computing, parallel and distributed systems, and middleware architectures. He is the project coordinator of the REFLECT (responsive flexible collaborating ambient) project, which focuses on adaptive control in pervasive applications and is funded by the European Union's Seventh Framework Project.

  1. C. A. de Costa, Toward a general software infrastructure for ubiquitous computing, Pervas. Comput. 7, pp. 64-74, 2008.

  2. http://reflect.first.fraunhofer.de The REFLECT project: responsive flexible collaborating ambient. Accessed 17 September 2008.

  3. N. S. Serbedzija, A. M. Calvosa and A. Ragnoni, Vehicle as a co-driver, Proc. Ann. Int'l Symp. Vehic. Comput. Syst., ISVCS 1, 2008.

  4. R. Picard, Affective Computing, MIT Press, Cambridge, 1997.

DOI:  10.2417/2200809.1263