Interactieve Multimodale Informatie Extractie (IMIX)

The Demonstrator

The IMIX demonstrator intends to show the status, progress, and results of the research that is carried out in the programme. All research groups participate and collaborate in building the demonstrator. The IMIX demonstrator is meant as a vehicle to prove that the modules under development in the individual projects can operate in the context of an end-to-end Information Extraction system. 

The IMIX demonstrator is an interactive multimodal Q/A system for information about data typical for a medical information system (cf. the Spectrum medical encyclopaedia). The system will be able to deal with more complex questions than simple factual ones, and it will be able to engage in a simple dialogue (1 to 3 turns) w ith the user aiming at obtaining a better understanding of the question or at a more equal level of 'communication' between the user and the system. The answers may consist of  noun phrases, sentences, paragraphs in either text or speech format, tables or graphical displays, depending on the (type of) question, the contents of the answer and the needs/profile of the user.

Four versions of the demonstrator are foreseen. The first version (planned for the end of 2004) will only contain a rudimentary dialogue, i.e. one question and one return answer. The main aim of this version is to integrate a number of modules into a multimodal architecture to form an operational end-to-end system. The next versions will contain interactivity and multimodal input.
The first version of the demonstrator will consist of two parts:

  • A multimodal part focusing on the RSI domain. Here, speech and keyboard/ mouse input will be combined with multimodal output (speech, text, tables, and graphics). For the first version only a limited number of questions will be considered.

  • A text based part focusing on the complete (Spectrum, Merck, and RSI) medical domain. Input will be only in the form of typed text, output will be text that can be converted into speech by means of straightforward TTS. This part of the demonstrator focuses on restricted domain Q/A.