Design Architecture For A Multimodal System Based On...

In this chapter, we will show step by step how to design architecture for a multimodal system based on complex event processing , we take an example of put that there system because its reference the base of multimodal system. 4.1 Define the problem – domain Put That There† the bolt system is a voice and gesture interactive system implemented at the Architecture Machine Group at MIT. It allows a user to build and modify a graphical database on a large Format video Display. The Goal of the research is a simple, conversational interface for sophisticated computer interaction. Natural languages and gestures are used, while speech output allows the system to query the user on ambiguous input. Bolt’s System is primarily a voice-interactive†¦show more content†¦This is particularly evident with connected speech recognizers, which are nonetheless more attractive for many applica ¬tions, as they allow more natural spoken input. We use speech recognizers to transform speech to text using grammar below. Grammar: public = ()| (there) | (move) (that) | (delete)| (this); = create | put; = [a] (yellow | red | blue | black | green); = square | triangle | circle | rectangle; The equivalence of the grammar using graphs: Figure 9 Graphs describe the grammar Commands: â€Å"Create or put†: â€Å"Create a blue square.† Effect of complete utterance is a â€Å"call† to the create routine which needs object to be created (with attributes) as well as x, y pointing input from mouse or eye Tribe. â€Å"Move†: â€Å"Move that there† effect a displacement from old position the new one pointing by mouse or eye Tribe too. â€Å"Delete†: â€Å"Delete that† this command used to deleting an object by pointing input. â€Å"There or this†: those command help to extract the pointing input from mouse. The grammars above help us to extract only what we want as speech and ignore the rest. After using this grammar we extract sentence or words like: †¢ Create a yellow square. †¢ Put a red circle. †¢ There †¢ Delete †¢ This †¢ Move that b) Keyboard and Mouse (input): After the speech the user can use the mouse with a simple left click, to point the position where he wants to create or select or