Collaborative Assistant for Live Semantic Interactions with Tangible Objects.

Calisto is a new multimodal semantic based dialogue system. Natural language and gesture interaction with a touch terminal or smartphone offer simplified, exciting and unexpected possibilities to explore knowledge databases. Comprehensive information linked to a specific domain can be made available very intuitively. Through an intelligent sharing system, it is also possible to share information between two mobile phones.

In our current scenario, a user synchronizes his data (pictures or texts) stored in a shared folder on his mobile device with the multitouch kiosk system. We are using an eKiosk Terminal for our multitouch kiosk system which would perfectly suit as an infotainment terminal in a museum. Over a Frisbee interaction with his mobile phone, a user can really throw the selected medium, like a picture, onto the Calisto terminal; the picture directly appears on the touch-screen.

Due to the fact, that all objects are backed by semantics through annotations, the user can initiate an extended multimedia search through externally connected semantic Web services.

Calisto was developed under the terms of the THESEUS research program. Currently exposed at THESEUS Innovation Center in Berlin, the Calisto terminal shows its functionality by using multimedia content of the fascinating life of the German computer pioneer Konrad Zuse. An extensive amount of pictures and videos for Konrad Zuse are made available through the Calisto System. A semantic search helps the visitor in its search by even allowing him to get new tweets over the Twitter Web service about the domain or by letting him by natural speech directly ask the systems questions such as “Show me pictures of Konrad Zuse”.

The gesture control feature, in particular, allows new forms of interaction for users of smartphones: e.g., a photograph of the historic Z3 computer can be “darted” onto the console with a “frisbee gesture” from a smartphone to be used there as the starting point for a search. Calisto uses several Web services to find additional photos, texts, and videos. Documents in the results that have been provided with location data can immediately be displayed on a map.