What's OI
OpenInterface originally started in the SIMILAR Network of Excellence as an open source software platform that allows integration of software components dedicated to multimodal interaction and multimodal data fusion. Now it is continued and improved by several projects.
The main contributor is the OpenInterface Project which focuses on the design and development of the platform for rapid development of multimodal interactive systems as a central tool for an iterative user-centered design process. The OpenInterface Interaction Development Environment (OIDE) includes development tools for assembling, combining and testing modalities at multiple levels of abstraction (from the raw data to the semantic of users’ events). The project focuses on two interaction contexts, namely a mobile and an ubiquitous setting, and two testbeds, namely multimodal interaction for large information spaces, and multimodal interaction for games. The two testbeds are used to explore the functionality of the platform.
OpenInterface provides a clear path for transferring research results to industry by adopting an incremental approach to extending current multimodal standards.
The main goal of OpenInterface in SIMILAR was to connect the results from two research communities:
The Signal Processing community who may provide:
- algorithms for multimodal data fusion: for instance, image/video segmentation
- algorithms and fusion mechanisms as being building blocks for defining interaction modalities: for instance vision trackers, speech recognition, etc...
The Human Computer Interaction community who may contribute with
- interaction modalities: for instance, direct manipulation or orientation/location trackers
- software mechanisms for multimodal interaction and in particular fusion of modalities
Four steps to understand OpenInterface !
Step 1: Heterogeneous components
OpenInterface can be seen as a large jigsaw. In the platform, the heterogeneous components are like pieces of a jigsaw that can be registered as reusable and inter-connectable components.
Step 2: Componentization
Each heterogeneous component is described in XML according to the CIDL – Components Interface Description Language. Each heterogeneous component is encapsulated within a C++ proxy and registered as a plugin.
Step 3: Modalities selection
Components are retrieved through the Graphic Editor by the user who can then edit the components properties and compose the execution pipeline of a multimodal application. This execution pipeline is sent interpreted by the OpenInterface Kernel (C/C++) to run the application.
Step 4: Fusion with ICARE – Interaction – Care – Complementarity – Assignment – Redudancy – Equivalence
While designing multimodal applications, the designer needs to specify the multimodal interaction dedicated to a given task of the interactive system under development. To handle multimodal inputs/outputs issues OpenInterface integrates the concepts proposed in ICARE platform. Two kinds of components are considered: (1) elementary components that enable the designer to define “pure interaction modality” and (2) generic composition components that enable the designer to specify combined usage of modalities.
The main objective of the OpenInterface project is to design, implement and test an open source platform for developing multimodal interaction:
-
that handles a rich and extensible set of modalities,
-
that enables quick replication,
-
that enables a focus on innovation (new modalities or forms of multimodality),
-
that supports dynamic selection and combination of modalities to fit the ongoing context of use,
-
that enables iterative user-centred design.
More specifically, the platform will embed a set of pure and combined modalities as reusable components and generic mechanisms for combining modalities.
-
At design-time, for specifying the multimodal interaction of a particular interactive system under design, the designer can reuse and assemble OI components. From this assembling, the code of the interaction part of the system is generated. The open source platform therefore allows rapid development of multimodal interaction by assembling components. It is a central tool for supporting a truly iterative User-Centred Design (UCD) process by enabling the rapid development of early prototypes for exploration, prototypes of different design options, and testbeds for experimental evaluation. Moreover by supporting the exploration of new forms of multimodality, the platform can be an instrument for defining extensions to existing standards.
-
At run-time, the platform includes mechanisms for dynamic reconfiguration of the components in order to support dynamic adaptation of the interaction, such as changing the run-time parameter of a component (e.g. the sampling rate of an audio input device), replacing one component for a functionally equivalent one (e.g. functional equivalence of modalities).