Multimodal Maps: An Agent-based Approach

Abstract

The SRI Multimedia Interfaces Group is currently building a series of prototype map-based applications that accept handwritten, verbal and gestural requests issued in a synergistic fashion. These applications are distinguished by comparatively rich natural language capabilities, access to existing data sources including the World Wide Web, and a mobile handheld interface. To implement the described application, a hierarchical distributed network of heterogeneous software agents was augmented by appropriate functionality for developing synergistic multimodal applications.

Features

Multimodal Input: A user interacts with the system through handwritten, gestural, vocal, and direct manipulation commands.
Multimedial Output: Responses are provided by the system through textual, graphical, audio and video media.
Agent Interaction: User requests are processed by a distributed community of software agents, which automate tasks such as natural language processing, interaction with planners or simulators, and accessing data from remote datasources (Prolog DBs, X.500 DBs, WWW, etc.)

Demos

To see a demonstration (including video) of the multimodal map interface, click here.
To see a picture representing the agents used in the multimodal map demonstration, click here.

Papers

Multimodal Maps: An Agent-based Approach. With Luc Julia. International Conference on Cooperative Multimodal Communication (CMC/95), 24-26 May 1995, Eindhoven, The Netherlands. PostScript: 171K
A Multimodal Computer-augmented Interface for Distributed Applications. With Luc Julia. HCII'95, July, 1995, Tokyo.

Related Work

The Open Agent Architecture (tm) is a general purpose framework for developing multi-agent systems.
A set of Agent Development Tools is being developed to simplify the task of creating new multimodal agent-based applications.

Created by Adam Cheyer, 19 February 1995