The Handbook of Multimodal-Multisensor Interfaces, Volume 2

Description

The Handbook of Multimodal-Multisensor Interfaces provides the first authoritative resource on what has become the dominant paradigm for new computer interfaces: user input involving new media (speech, multi-touch, hand and body gestures, facial expressions, writing) embedded in multimodal-multisensor interfaces that often include biosignals. This edited collection is written by international experts and pioneers in the field. It provides a textbook, reference, and technology roadmap for professionals working in this and related areas. This second volume of the handbook begins with multimodal signal processing, architectures, and machine learning. It includes recent deep learning approaches for processing multisensorial and multimodal user data and interaction, as well as context-sensitivity. A further highlight is processing of information about users' states and traits, an exciting emerging capability in next-generation user interfaces. These chapters discuss real-time multimodal analysis of emotion and social signals from various modalities, and perception of affective expression by users. Further chapters discuss multimodal processing of cognitive state using behavioral and physiological signals to detect cognitive load, domain expertise, deception, and depression. This collection of chapters provides walk-through examples of system design and processing, information on tools and practical resources for developing and evaluating new systems, and terminology and tutorial support for mastering this rapidly expanding field. In the final section of this volume, experts exchange views on the timely and controversial challenge topic of multimodal deep learning. The discussion focuses on how multimodal-multisensor interfaces are most likely to advance human performance during the next decade.
Table of Contents: 1. Multimodal Machine Learning / 2. Classifying Multimodal Data / 3. Learning for Multimodal and Context-sensitive Interfaces / 4. Deep Learning for Multisensorial and Multimodal Interaction / 5. Multimodal User State and Trait Recognition / 6. Multimodal-Multisensor Affect Detection / 7. Multimodal Analysis of Social Signals / 8. Real-time Sensing of Affect and Social Signals in a Multimodal Framwork / 9. How do Users Perceive Multimodal Expressions of Affects? / 10. Multimodal Behavior and Physiological Signals as Indicators of Cognitive Load / 11. Multimodal Learning Analytics / 12. Multimodal Assessment of Depression and Related Disorders Based on Behavioral Signals / 13. Multimodal Deception Detection / 14. Perspectives on Strategic Fusion / 15. Perspectives on Predictive Power of Multimodal Deep Learning

Sujets

• User Interfaces
• Computer Science
• Computers
• Informatique
• Human?computer interaction
• Signal processing

Informations

• Publié par: Association for Computing Machinery and Morgan & Claypool Publishers
• Date de parution: 08 octobre 2018
• Nombre de lectures: 0
• EAN13: 9781970001709
• Langue: English
• Poids de l'ouvrage: 2 Mo

Extrait

The Handbook of Multimodal-Multisensor Interfaces, Volume 2
ACM Books
Editor in Chief
M. Tamer zsu, University of Waterloo
ACM Books is a new series of high-quality books for the computer science community, published by ACM in collaboration with Morgan Claypool Publishers. ACM Books publications are widely distributed in both print and digital formats through booksellers and to libraries (and library consortia) and individual ACM members via the ACM Digital Library platform.
The Handbook of Multimodal-Multisensor Interfaces, Volume 2: Signal Processing, Architectures, and Detection of Emotion and Cognition
Editors: Sharon Oviatt, Monash University
Bj rn Schuller, University of Augsburg and Imperial College London
Philip R. Cohen, Monash University
Daniel Sonntag, German Research Center for Artificial Intelligence (DFKI)
Gerasimos Potamianos, University of Thessaly
Antonio Kr ger, Saarland University and German Research Center for Artificial Intelligence (DFKI)
2018
Declarative Logic Programming: Theory, Systems, and Applications
Editors: Michael Kifer, Stony Brook University
Yanhong Annie Liu, Stony Brook University
2018
The Sparse Fourier Transform: Theory and Practice
Haitham Hassanieh, University of Illinois at Urbana-Champaign
2018
The Continuing Arms Race: Code-Reuse Attacks and Defenses
Editors: Per Larsen, Immunant, Inc .
Ahmad-Reza Sadeghi, Technische Universit t Darmstadt
2018
Frontiers of Multimedia Research
Editor: Shih-Fu Chang, Columbia University
2018
Shared-Memory Parallelism Can Be Simple, Fast, and Scalable
Julian Shun, University of California, Berkeley
2017
Computational Prediction of Protein Complexes from Protein Interaction Networks
Sriganesh Srihari, The University of Queensland Institute for Molecular Bioscience
Chern Han Yong, Duke-National University of Singapore Medical School
Limsoon Wong, National University of Singapore
2017
The Handbook of Multimodal-Multisensor Interfaces, Volume 1: Foundations, User Modeling, and Common Modality Combinations
Editors: Sharon Oviatt, Incaa Designs
Bj rn Schuller, University of Passau and Imperial College London
Philip R. Cohen, Voicebox Technologies
Daniel Sonntag, German Research Center for Artificial Intelligence (DFKI)
Gerasimos Potamianos, University of Thessaly
Antonio Kr ger, Saarland University and German Research Center for Artificial Intelligence (DFKI)
2017
Communities of Computing: Computer Science and Society in the ACM
Thomas J. Misa, Editor, University of Minnesota
2017
Text Data Management and Analysis: A Practical Introduction to Information Retrieval and Text Mining
ChengXiang Zhai, University of Illinois at Urbana-Champaign
Sean Massung, University of Illinois at Urbana-Champaign
2016
An Architecture for Fast and General Data Processing on Large Clusters
Matei Zaharia, Stanford University
2016
Reactive Internet Programming: State Chart XML in Action
Franck Barbier, University of Pau, France
2016
Verified Functional Programming in Agda
Aaron Stump, The University of Iowa
2016
The VR Book: Human-Centered Design for Virtual Reality
Jason Jerald, NextGen Interactions
2016
Ada s Legacy: Cultures of Computing from the Victorian to the Digital Age
Robin Hammerman, Stevens Institute of Technology
Andrew L. Russell, Stevens Institute of Technology
2016
Edmund Berkeley and the Social Responsibility of Computer Professionals
Bernadette Longo, New Jersey Institute of Technology
2015
Candidate Multilinear Maps
Sanjam Garg, University of California, Berkeley
2015
Smarter Than Their Machines: Oral Histories of Pioneers in Interactive Computing
John Cullinane, Northeastern University; Mossavar-Rahmani Center for Business and Government, John F. Kennedy School of Government, Harvard University
2015
A Framework for Scientific Discovery through Video Games
Seth Cooper, University of Washington
2014
Trust Extension as a Mechanism for Secure Code Execution on Commodity Computers
Bryan Jeffrey Parno, Microsoft Research
2014
Embracing Interference in Wireless Systems
Shyamnath Gollakota, University of Washington
2014
The Handbook of Multimodal-Multisensor Interfaces, Volume 2
Signal Processing, Architectures, and Detection of Emotion and Cognition
Sharon Oviatt
Monash University
Bj rn Schuller
University of Augsburg and Imperial College London
Philip R. Cohen
Monash University
Daniel Sonntag
German Research Center for Artificial Intelligence (DFKI)
Gerasimos Potamianos
University of Thessaly
Antonio Kr ger
Saarland University and German Research Center for Artificial Intelligence (DFKI)
ACM Books #21
Copyright 2019 by the Association for Computing Machinery and Morgan Claypool Publishers
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means-electronic, mechanical, photocopy, recording, or any other except for brief quotations in printed reviews-without the prior permission of the publisher.
Designations used by companies to distinguish their products are often claimed as trademarks or registered trademarks. In all instances in which Morgan Claypool is aware of a claim, the product names appear in initial capital or all capital letters. Readers, however, should contact the appropriate companies for more complete information regarding trademarks and registration.
The Handbook of Multimodal-Multisensor Interfaces, Volume 2
Sharon Oviatt, Bj rn Schuller, Philip R. Cohen, Daniel Sonntag, Gerasimos Potamianos, Antonio Kr ger, editors
books.acm.org
www.morganclaypoolpublishers.com
ISBN: 978-1-97000-171-6 hardcover
ISBN: 978-1-97000-168-6 paperback
ISBN: 978-1-97000-169-3 eBook
ISBN: 978-1-97000-170-9 ePub
Series ISSN: 2374-6769 print 2374-6777 electronic
DOIs:
10.1145/3107990 Book
10.1145/3107990.3107991 Preface
10.1145/3107990.3107992 Introduction
10.1145/3107990.3107993 Chapter 1
10.1145/3107990.3107994 Chapter 2
10.1145/3107990.3107995 Chapter 3
10.1145/3107990.3107996 Chapter 4
10.1145/3107990.3107997 Chapter 5
10.1145/3107990.3107998 Chapter 6
10.1145/3107990.3107999 Chapter 7
10.1145/3107990.3108000 Chapter 8
10.1145/3107990.3108001 Chapter 9
10.1145/3107990.3108002 Chapter 10
10.1145/3107990.3108003 Chapter 11
10.1145/3107990.3108004 Chapter 12
10.1145/3107990.3108005 Chapter 13
10.1145/3107990.3108006 Chapter 14
10.1145/3107990.3108007 Index / Bios / Glossary
A publication in the ACM Books series, #21
Editor in Chief: M. Tamer Özsu, University of Waterloo
Area Editor: Michel Beaudouin-Lafon, Universit Paris-Sud
This book was typeset in Arnhem Pro 10/14 and Flama using ZzT E X.
First Edition
10 9 8 7 6 5 4 3 2 1
This book is dedicated to our families, whose patience and support sustained the year-long effort required to organize, write, and manage different stages of this multi-volume project .
Contents
Preface
Figure Credits
Introduction: Trends in Intelligent Multimodal-Multisensorial Interfaces: Cognition, Emotion, Social Signals, Deep Learning, and More
A Very Brief History of HCI and AI-and Their Relationship in Time
Increasingly Robust AI as a Game-Changer for HCI
Multimodal Signal Processing, Architectures and Deep Learning
The Advent of Artificial Emotional and Social Intelligence
Insights in the Chapters Ahead
References
PART I MULTIMODAL SIGNAL PROCESSING AND ARCHITECTURES
Chapter 1 Challenges and Applications in Multimodal Machine Learning
Tadas Baltru aitis, Chaitanya Ahuja, Louis-Philippe Morency
1.1 Introduction
1.2 Multimodal Applications
1.3 Multimodal Representations
1.4 Co-learning
1.5 Conclusion
Focus Questions
References
Chapter 2 Classifying Multimodal Data
Ethem Alpaydin
2.1 Introduction
2.2 Classifying Multimodal Data
2.3 Early, Late, and Intermediate Integration
2.4 Multiple Kernel Learning
2.5 Multimodal Deep Learning
2.6 Conclusions and Future Work
Acknowledgments
Focus Questions
References
Chapter 3 Learning for Multimodal and Affect-Sensitive Interfaces
Yannis Panagakis, Ognjen Rudovic, Maja Pantic
3.1 Introduction
3.2 Correlation Analysis Methods
3.3 Temporal Modeling of Facial Expressions
3.4 Context Dependency
3.5 Model Adaptation
3.6 Conclusion
Focus Questions
References
Chapter 4 Deep Learning for Multisensorial and Multimodal Interaction
Gil Keren, Amr El-Desoky Mousa, Olivier Pietquin, Stefanos Zafeiriou, Bj rn Schuller
4.1 Introduction
4.2 Fusion Models
4.3 Encoder-Decoder Models
4.4 Multimodal Embedding Models
4.5 Perspectives
Focus Questions
References
PART II MULTIMODAL PROCESSING OF SOCIAL AND EMOTIONAL STATES
Chapter 5 Multimodal User State and Trait Recognition: An Overview
Bj rn Schuller
5.1 Introduction
5.2 Modeling
5.3 An Overview on Attempted Multimodal Stait and Trait Recognition
5.4 Architectures
5.5 A Modern Architecture Perspective
5.6 Modalities
5.7 Walk-through of an Example State
5.8 Emerging Trends and Future Directions
Focus Questions
References
Chapter 6 Multimodal-Multisensor Affect Detection
Sidney K. D Mello, Nigel Bosch, Huili Chen
6.1 Introduction
6.2 Background from Affective Sciences
6.3 Modality Fusion for Multimodal-Multisensor Affect Detection
6.4 Walk-throughs of Sample Multisensor-Multimodal Affect Detection Systems
6.5 General Trends and State of the Art in Multisensor-Multimodal Affect Detection
6.6 Discussion
Acknowledgments
Focus Questions
References
Chapter 7 Multimodal Analysis of Social Signals
Alessandro Vinciarelli, Anna Esposito
7.1 Introduction
7.2 Multimodal Communication in Life and Human Sciences
7.3 Multimodal Analysis of Social Signals
7.4 Next Steps
7.5 Conclusions
Focus Questions
References
Chapter 8 Real-Time Sensing of Affect and Social Signals in a Multimodal Framework: A Practical Approach
Johannes Wagner, Elisabeth Andr
8.1 Introduction
8.2 Database Collectio