Multimodal Interaction at VRlab

The research on Virtual Reality being carried out at VRLAB is around the topic of multimodal human-computer interaction in virtual environments. We focus mainly on developing new interaction metaphors integrating 3D sound, stereo visualization, and force feedback, towards more intuitive and immersive systems for several applications.

Multimodal Interactive VR Systems

One of the investigated topics concerns the use of intelligent avatars (or autonomous virtual assistants) to efficiently perform tasks in cooperation with the user. In this context, natural ways of communication between the user and its virtual assistant are being investigated within a multimodal interactive application.

Another topic of research is to define VR systems able to dynamically change the interaction mode, adapting themselves to each situation and to the preferences and motor skills capabilities of each user, offering personalized interaction metaphors in order to optimize the comfort and sense of presence in interactive VR systems.

Take as example the simple problem of grabbing an object with a dataglove: the used interaction mode may dynamically vary according to the different motor skills and manipulation precision requirements: (1) the user must close its virtual hand around objects to grab it, (2) the user simply places the virtual hand near to an object an the virtual hand automatically grabs it, (3) only by pointing to a desired virtual object will trigger an automatic animation of the virtual hand to grab it, or (4) the user asks his “intelligent avatar” (vocal or gesture instructions) to grab the object.

Interaction Metaphors for Specific Applications

Depending on the target VR application, different interaction solutions have been developed for specific applications:

• Interaction with virtual vehicle controls in an immersive vehicle simulation system for training and ergonomics in the scope of the European project VIRTUAL.
• Navigation and interaction in virtual environments for the purpose of test and training of volunteers for emergency situations in the scope of the EU project JUST.
• 3D immersion for therapy of patients suffering of PHOBIA diseases.
• Precise interaction with a virtual piano keyboard.
• Higher level modes of interaction with “smart objects”, allowing concurrent operation with autonomous virtual humans.

Source : Ecole Polytechnique Federale de Lausanne