April 29, 2017

Event recognition for intelligent resource management

Alexander Artikis and George Paliouras
The PRONTO research programme aims to develop a system for real-time, accurate recognition of complex events given multiple sources of information.

The European Commission's Seventh Framework Programme PRONTO project emphasizes the role of event recognition in intelligent resource management (IRM) and proposes a methodology for fusing data from various sources, analysing it to extract useful information—in the form of events—and delivering the resulting knowledge for decision making. To achieve this, PRONTO uses methods and expertise from the fields of data fusion, information extraction, temporal representation and reasoning, machine learning and knowledge-management systems.

Today's organizations can collect data in various structured and unstructured digital formats, but they do not have the capability to fully use this data to support and improve their resource-management process. Therefore, the analysis and interpretation of the collected data must evidently be automated and transformed into operational knowledge. ‘Events’ are particularly important pieces of knowledge, as they represent the temporal nature of the processes taking place in an organization. PRONTO proposes a synergy of methods to facilitate event recognition from the raw data, which remains underused with current technologies. The extracted knowledge will inform the decision-making process.

The need for IRM covers numerous application domains. In emergency-rescue operations, for example, there is a pressing need for real-time decision support facilitating the fastest possible completion of the operation with the minimum possible casualties. An operation manager needs to decide almost instantaneously how to deploy and manage a rescue team to successfully complete an operation.

Event recognition for intelligent resource management (IRM) in emergency-rescue operations.

The PRONTO approach to IRM includes a number of steps. First, data is continuously acquired, synchronized and aggregated from various sensor types that are installed in the infrastructure of an organization (e.g., a fire brigade), and from various modes of interaction among the relevant actors (e.g., fire-brigade personnel). The aggregated data is analysed to extract current, ‘low-level’ information. Event-recognition techniques, i.e., approaches from the field of temporal representation and reasoning, are subsequently applied to this low-level data to detect—in real time—‘high-level’ events. Given a sequence of low-level information on the interactions of rescue workers and climate-sensor data, for instance, the current status and urgency of a rescue operation will be detected automatically, to the benefit of the operation manager responsible for resource management. A user-friendly IRM tool will support the decision-making process at that level. In parallel, machine-learning techniques will be applied for dynamical fine tuning of the knowledge base, thus constantly improving the accuracy of high-level event recognition.

The highly successful (European Commission) Sixth Framework Programme-funded SHARE1 project developed an advanced mobile service that provided critical multimodal communication support to emergency teams during rescue operations. It was shown that such operations benefited enormously from the multimodal interaction and online, on-site access to data services providing basic operation-status information and updates on various aspects of the emergency, such as location and environment. PRONTO will build upon—and extend—the SHARE approaches and technologies to support IRM for the emergency-operations domain. Knowledge that could not be extracted by the SHARE demonstrator will be retrieved by the PRONTO-developed tool on the basis of event recognition (see Figure 1). Consequently, considerably more valuable information will be available to rescue-operation managers, thus increasing support for online decision making and offline training and debriefing sessions.

Similarly to emergency operations, a city's public-transport system requires real-time intelligent management of the available resources. In a separate case study, the PRONTO technology will use an advanced real-time public-transport information system, currently employed in Helsinki. Buses are equipped with units that send global-positioning-system coordinates to a central server offering information about the current status of the transport system (e.g., the location of buses on the city map). In the context of PRONTO, buses will be equipped with additional types of sensors so that data—such as in-vehicle temperature, acceleration and fuel consumption—can be acquired. Given the low-level events that will be extracted from the types of data available, high-level events will be detected, related to passenger comfort, driver and passenger safety, traffic status and compliance with the timetable. Consequently, PRONTO will offer increased support for the decision-making processes of bus operators, road or rail administrators and city officials/traffic planners (see Figure 2). For instance, prioritization of traffic-light sequences may be adapted to improve fuel economy, passenger comfort and planned timetable accuracy, thus substantially increasing the competitiveness of public transport.

Event recognition for city-transport management.

PRONTO will contribute to the state of the art by providing the technology for information extraction from sensor data. A generic workbench will be developed for real-time multimodal measurement and control, optimized for information extraction with customizable rules and input and output channels. In addition, a generic solution will be devised for object tracking, track analysis and information extraction (independent of the type of sensor used), modelling the spatial context, acquisition of location data and real-time detection of spatial events.

The project also aims at increasing audio-speech-recognition rates under very noisy conditions by researching robust algorithms for speech recognition, language-model adaptation, speech/nonspeech discrimination, acoustic-landmark detection and improved noise-filtering techniques, and selecting possible hardware settings that yield the best recognition results. We will innovatively redevelop state-of-the-art dialogue-management technology—typically used in human-computer interaction systems—to track human-human speech interaction. This will be used to focus speech recognition on the most informative keywords, improving recognition results without compromising the system's flexibility.

PRONTO is further designed to develop novel methods for temporal representation and reasoning for the real-time detection of events. More expressive temporal formalisms will be developed than currently employed for event recognition, thus representing and recognizing more complex events than may be detected at present.

To improve the accuracy of event recognition, developing novel machine-learning techniques to dynamically adapt a temporal knowledge base representing ‘event structures’ is also envisaged. Such techniques will cover the temporal knowledge-representation formalism developed for PRONTO, and be able to refine (partial) hierarchies of event structures.


The PRONTO system will detect real-time environmental changes for the benefit of the decision maker, thus enabling adaptation of resource management. The expected outcomes include a unifying framework for aggregation and processing of sensor and geographical data, dialogue-tracking technology, a methodology for event recognition and an IRM demonstrator.

Signals acquired from various sensor types (e.g., global-positioning system, temperature, wind) will be filtered, efficiently aggregated and analysed to extract information that will be used in event recognition. Moreover, georeferencing of the extracted information will be performed.

Dialogue-management technology will track human-human verbal communication for the purposes of predicting potentially interesting keywords. In addition, a method will be developed to take advantage of, e.g., operation structure and recognized or predicted events into dialogue-management state-transition rules.

A knowledge base will be developed to express complex event structures, emphasizing the temporal constraints of such structures. Reasoning algorithms will be devised for real-time recognition of ‘high-level’ events given a stream of ‘low-level’ information, such as extracted from sensor data, audio and text-based communications. To achieve accurate event recognition, machine-learning algorithms will be devised that will be capable of fine tuning temporal knowledge of event structures.

An IRM demonstrator will be validated by end users in realistic scenarios. The demonstrator will encapsulate all newly developed methodologies and integrate the corresponding software modules. It will operate on top of a communication platform based on existing user-specific networks. The end users will use the demonstrator for off-line training and debriefing, and for runtime decision support.


Alexander Artikis
Department of Electrical and Electronic Engineering, Imperial College London

George Paliouras
Institute of Informatics and Telecommunications, National Centre for Scientific Research ?Demokritos?

  1. http://www.ist-share.org European Commission's Sixth Framework Programme SHARE project. Accessed 15 February 2009.

DOI:  10.2417/2200902.1535