Silent Paralinguistics

DFG (German Research Foundation) Project

Runtime: 36 Months

Partner: University of Bremen

We propose to combine Silent Speech Interfaces with Computational Paralinguistics to form Silent Paralinguistics (SP). To reach the envisioned project goal of inferring paralinguistic information from silently produced speech for natural spoken communication, we will investigate three major questions: (1) How well can speaker states and traits be predicted from EMG signals of silently produced speech, using the direct and indirect silent paralinguistics approach? (2) How to integrate the paralinguistic predictions into the Silent Speech Interface to generate appropriate acoustic speech from EMG signals (EMG-to-speech)? and (3) Does the resulting paralinguistically enriched acoustic speech signal improve the usability of spoken communication with regards to naturalness and user acceptance?

HearTheSpecies: Using computer audition to understand the drivers of soundscape composition, and to predict parasitation rates based on vocalisations of bird species (#SCHU2508/14-1)

(“Einsatz von Computer-Audition zur Erforschung der Auswirkungen von Landnutzung auf Klanglandschaften, sowie der Parasitierung anhand von Vogelstimmen“)

DFG (German Research Foundation) Project, Schwerpunktprogramm „Biodiversitäts-Exploratorien“ 

Runtime: 36 Months

Partner: University of Freiburg

The ongoing biodiversity crisis has endangered thousands of species around the world and its urgency is being increasingly acknowledged by several institutions – as signified, for example, by the upcoming UN Biodiversity Conference. Recently, biodiversity monitoring has also attracted the attention of the computer science community due to the potential of disciplines like machine learning (ML) to revolutionise biodiversity research by providing monitoring capabilities of unprecedented scale and detail. To that end, HearTheSpecies aims to exploit the potential of a heretofore underexplored data stream: audio. As land use is one of the main drivers of current biodiversity loss, understanding and monitoring the impact of land use on biodiversity is crucial to mitigate and halt the ongoing trend. This project aspires to bridge the gap between existing data and infrastructure in the Exploratories framework and state-of-the-art computer audition algorithms. The developed tools for coarse and fine scale sound source separation and species identification can be used to analyse the interaction among environmental variables, local and regional land-use, vegetation cover and the different soundscape components: biophony (biotic sounds), geophony (abiotic sounds) and anthropophony (human-related sounds).

Agent-based Unsupervised Deep Interactive 0-shot-learning Networks Optimising Machines’ Ontological Understanding of Sound
DFG (German Research Foundation) Reinhart Koselleck-Projekt
# 442218748
 

Soundscapes are a component of our everyday acoustic environment; we are always surrounded by sounds, we react to them, as well as creating them. While computer audition, the understanding of audio by machines, has primarily been driven through the analysis of speech, the understanding of soundscapes has received comparatively little attention.

AUDI0NOMOUS, a long-term project based on artificial intelligent systems, aims to achieve a major breakthroughs in analysis, categorisation, and understanding of real-life soundscapes. A novel approach, based around the development of four highly cooperative and interactive intelligent agents, is proposed herein to achieve this highly ambitious goal. Each agent will autonomously infer a deep and holistic comprehension of sound.  A Curious Agent will collect unique data from web sources and social media; an Audio Decomposition Agent will decompose overlapped sounds; a Learning Agent will recognise an unlimited number of unlabelled sound; and, an Ontology Agent will translate the soundscapes into verbal ontologies.

AUDI0NOMOUS will open up an entirely new dimension of comprehensive audio understanding; such knowledge will have a high and broad impact in disciplines of both the sciences and humanities, promoting advancements in health care, robotics, and smart devices and cities, amongst many others.

Start date: 01.01.2021

Duration: 5 years

Machine Learning für Kameradaten mit unvollständiger Annotation

Industry Cooperation with BMW AG

Runtime: 01.01.2022 – 31.12.2023

Partner: BMW AG

The project aims at self-supervised and reinforced learning for analysis of camera data with incomplete annotation.

A Multimodal Approach to Humor Recognition and an Analysis of the Influence of Leader Humor on Team Performance in Major European Soccer Leagues

DFG (German Research Foundation) Project

Runtime: 36 Months

Partners: University of Passau, University of Augsburg

In this project, scholars active in the fields of management and computerized psychometry take the unique opportunity to join their respective perspectives and complementary capabilities to address the overarching question of “How, why, and under which circumstances does leader humor affect team processes and team performance, and how can (leader) humor be measured on a large scale by applying automatic multimodal recognition approaches?”. Trait humor, which is one of the most fundamental and complex phenomena in social psychology, has garnered increasing attention in management research. However, scholarly understanding of humor in organizations is still substantially limited, largely because research in this domain has primarily been qualitative, survey-based, and small scale. Notably, recent advances in computerized psychometry promise to provide unique tools to deliver unobtrusive, multi-faceted, ad hoc measures of humor that are free from the substantial limitations associated with traditional humor measures. Computerized psychometry scholars have long noted that a computerized understanding of humor is essential for the humanization of artificial intelligence. Yet, they have struggled to automatically identify, categorize, and reproduce humor. In particular, computerized approaches have suffered not only from a lack of theoretical foundations but also from a lack of complex, annotated, real-life data sets and multimodal measures that consider the multi- faceted, contextual nature of humor. We combine our areas of expertise to address these research gaps and complementary needs in our fields. Specifically, we substantially advance computerized measures of humor and provide a unique view into the contextualized implications of leader humor, drawing on the empirical context of professional soccer. Despite initial attempts to join computerized psychometry and management research, these two fields have not yet been successfully combined to address our overall research question. We aspire to fill this void as equal partners, united by our keen interest in humor, computerized psychometry, leader rhetoric, social evaluations, and team performance. 

Bayerischer Forschungsverbund zum gesunden Umgang mit digitalen Technologien und Medien
BayFOR (Bayerisches Staatsministerium für Wissenschaft und Kunst) Project
 

Partners: University of Augsburg, Otto-Friedrichs-University Bamberg, FAU Erlangen-Nuremberg, LMU Munich, JMU Würzburg

Runtime 2019-2023 (48 Months)   

Die Digitalisierung führt zu grundlegenden Veränderungen unserer Gesellschaft und unseres individuellen Lebens. Dies birgt Chancen und Risiken für unsere Gesundheit. Zum Teil führt unser Umgang mit digitalen Technologien und Medien zu negativem Stress (Distress), Burnout, Depression und weiteren gesundheitlichen Beeinträchtigungen. Demgegenüber kann Stress auch eine positive, anregende Wirkung haben (Eustress), die es zu fördern gilt. Die Technikgestaltung ist weit fortgeschritten, sodass digitale Technologien und Medien dank zunehmender künstlicher Intelligenz, Adaptivität und Interaktivität die Gesundheit ihrer menschlichen Nutzerinnen und Nutzer bewahren und fördern können. Ziel des Forschungsverbunds ForDigitHealth ist es, die Gesundheitseffekte der zunehmenden Präsenz und intensivierten Nutzung digitaler Technologien und Medien – speziell in Hinblick auf die Entstehung von digitalem Distress und Eustress und deren Folgen – in ihrer Vielgestaltigkeit wissenschaftlich zu durchdringen sowie Präventions- und Interventionsoptionen zu erarbeiten und zu evaluieren. Dadurch soll der Forschungsverbund zu einem angemessenen, bewussten und gesundheitsförderlichen individuellen wie kollektiven Umgang mit digitalen Technologien und Medien beitragen.

Start date: 01.01.2020

Duration: 36 Months

Funding body: Deutsche Forschungsgemeinschaft (DFG)

Homepage:  http://www.psych1.phil.uni-erlangen.de/forschung/weitere-projekte/paralinguistische-stimmcharakteristika-bei-depression.shtml

Die Erklärung, Diagnostik, Vorhersage und Behandlung der Major Depression stellen nach wie vor zentrale Herausforderungen der Psychotherapieforschung dar.

Als neuer und innovativer Ansatz in der Diagnostik und Therapie der Depression erforscht die Paralinguistik Intonationsmerkmale wie Sprechpausen, Sprachrhythmus, Intonation, Tonhöhe und Lautstärke. In diesem interdisziplinären Projekt arbeiten die klinische Psychologie und Informatik zusammen, um über optimierte Algorithmen Depressionen anhand paralinguistischer Stimmcharakteristika (PSCs) möglichst gut zu erkennen, vorherzusagen und zu klären, inwieweit ein bestimmter Intonationsstil dazu beiträgt, die Depression aufrecht zu erhalten.

Darüber hinaus wollen wir die PSCs perspektivisch auch als Therapie einsetzen. Das bedeutet, dass Therapeuten nicht nur, wie gewohnt in der Depressionsbewältigung, mit ihren Patienten erarbeiten, was sie sich sagen, sondern auch wie. Ein Du schaffst das schon! mit leiser, monotoner und kraftloser Stimme wird nichts bewirken, da es nicht emotional überzeugend klingt. Wenn der Satz dagegen mit kraftvoller, deutlicher und dynamischer Stimme ausgesprochen wird, sind die Chance deutlich größer, dass sich damit auch ein Gefühl von Hoffnung und Optimismus auslösen lässt.

Das von der DFG geförderte Forschungsprojekt will die wissenschaftliche Grundlage hierfür schaffen. Dazu sollen Sprachproben mit Hilfe von maschinellem Lernen untersucht werden, um Intonationsunterschiede zwischen klinisch-depressiven und nicht-depressiven Personen zu erkennen. Wir werden Algorithmen entwickeln, mit deren Hilfe depressionsrelevante Intonations-Muster identifiziert werden können. Die gewonnenen Erkenntnisse sollen dann wiederum helfen, ein Intonations-fokussiertes Feedback-Training zu entwickeln, das Menschen mit Depressionen helfen soll, depressive Phasen zu bewältigen.

causAI: AI Interaktionsoptimierung bei Videoanrufen im Vertrieb (#03EGSBY853)

BMWi (Federal Ministry for Economic Affairs and Energy) EXIST Business Start-up Grant

Runtime: 01.03.2022 - 28.02.2023

causAI analysiert die Sprache, Gestik und Mimik von vertrieblichen Videoanrufen mithilfe von künstlicher Intelligenz, um die digitale Vertriebskompetenz zu verbessern. Ziel ist es, causAI als innovatives Softwareprodukt für Vertriebsgesprächsunterstützung und -schulung im Vertrieb zu etablieren.

The Huawei-University of Augsburg Joint Lab aims to bring together Affective Computing & Human-Centered Intelligence for Human-centred empathic interaction.

The Lehrstuhl for Embedded Intelligence in Health Care and Wellbeing is one of two Lehrstuhls in the collaboration.

Start date: 01.01.2020

Duration: 3 years

Affective loop in Socially Assistive Robotics as an Intervention Tool for Children with Autism

Start date: 01.09.2019

Duration: 36 Months

Funding body: EU, Erasmus Plus Strategic Partnership

Homepage:  emboa.eu

Partners: Politechnika Gdanska, University of Hertfordshire, Istanbul Teknik Universitesi, Yeditepe University Vakif, Macedonian association for applied psychology & University of Augsburg

Description: The EMBOA project aims are the development of guidelines and practical evaluation of applying emotion recognition technologies in robot-supported intervention in children with autism.

Children with autism spectrum disorder (ASD) suffer from multiple deficits and limited social and emotional skills are among those that influence their ability to involve in interaction and communication. Limited communication occurs in human-human interactions and affects relations with family members, peers and therapists.

There are promising results in the use of robots in supporting the social and emotional development of children with autism. We do not know why children with autism are eager to interact with human-like looking robots and not with humans. Regardless of the reason, social robots proved to be a way to get through the social obstacles of a child and make him/her involved in the interaction. Once the interaction happens, we have a unique opportunity to engage a child in gradually building and practicing social and emotional skills.

In the project, we combine social robots that are already used in therapy for children with autism with algorithms for automatic emotion recognition. The EMBOA project goal is to confirm the possibility of the application (feasibility study), and, in particular, we aim to identify the best practices and obstacles in using the combination of the technologies. What we hope to obtain is a novel approach for creating an affective loop in child-robot interaction that would enhance interventions regarding emotional intelligence building in children with autism.

The lessons learned, summarized in the form of guidelines, might be used in higher education in all involved countries in robotics, computer science, and special pedagogy fields of study. The results will be disseminated in the form of training, multiple events, and to the general public by scientific papers and published reports. The project consortium is multidisciplinary and combines partners with competence in interventions in autism, robotics, and automatic emotion recognition from Poland, UK, Germany, North Macedonia, and Turkey.

The methodological approach includes systematic literature reviews and meta-analysis, data analysis based on statistical and machine learning approaches, as well as observational studies. We have planned a double-loop of observational studies. The first round is to analyse the application of emotion recognition methods in robot-based interaction in autism, and to compare diverse channels for observation of emotion symptoms in particular.

The lessons learned will be formulated in the form of guidelines. The guidelines will be evaluated with the AGREE (Appraisal of Guidelines, Research, and Evaluation) instrument and confirmed with the second round of observational studies. The objectives of our project match the Social Inclusion horizontal priority with regards to supporting the actions for improvement of learning performance of disadvantaged learners (testing of a novel approach for improvement of learning performances of children with autism).

MAikI: Mobiler Alltagstherapieassistent mit interaktionsfokussierter künstlicher Intelligenz bei Depression

BMBF Projekt

Runtime: 01.10.2021 – 31.12.2021

Partners: FlyingHealth Incubator GmbH, GET.ON Institut für Online Gesundheitstrainings GmbH, University of Augsburg

Dieses Vorhaben zielt auf einen mobilen digitalen Assistenten ab, der den Patienten interaktiv, intelligent und individualisiert darin unterstützt, seine Therapie im Alltag effektiver umzusetzen. Hierzu werden Methoden künstlicher Intelligenz (Interaktionsanalyse mit Stimmanalyse und Natural Language Processing, Artificial Empathy, Maschinelles Lernen) mit dem Ziel erforscht und entwickelt, die Patienten-Assistenten-Interaktion zu optimieren, therapeutische Interventionen fallspezifisch zu optimieren und deren Umsetzung interaktiv auf intelligente und zugleich personalisierte Art zu unterstützen. Dieser digitale mobile Therapiebegleiter geht über den derzeitigen Stand der Technik hinaus, da er a) eine dauerhafte behandlungsrelevante Kommunikation mit dem Betroffenen aufrechterhält (was bisher die face-to-face Psychotherapie nicht vermag) und b) seine Empfehlungen fortlaufend an aktuelles Erleben, Verhalten und bisherigem Therapieverlauf der Betroffenen anpasst. Auf Basis dieser digitalen Therapieindividualisierung soll der Gesundungsprozess beschleunigt und Rückfallquoten verringert werden.

Smart Environments for Person-centered Sustainable Work and Well-being

Start date: 01.01.2019

End date: 30.06.2022

Funding body: EU Horizon 2020 Research & Innovation Action (RIA)

Homepage: www.sustage.eu  

sustAGE will provide a paradigm shift in human machine interaction, building upon seven strategic technology trends, IoT, Machine learning, micro-moments, temporal reasoning, recommender systems, data analytics and gamification to deliver a composite system integrated with the daily activities at work and outside, to support employers and ageing employees to jointly increase well-being, wellness at work and productivity. The manifold contribution focuses on the support of the employment and later retirement of older adults from work and the optimization of the workforce management.

The sustAGE platform guides workers on work-related tasks, recommends personalized cognitive and physical training activities with emphasis on game and social aspects, delivers warnings regarding occupational risks and cares for their proper positioning in work tasks that will maximize team performance.

By combining a broad range of the innovation chain activities namely, technology R&D, demonstration, prototyping, pilots, and extensive validation, the project aims to explore how health and safety at work, continuous training and proper workforce management can prolongue older workers’ competitiveness at work. The deployment of the proposed technologies in two critical industrial sectors and their extensive evaluation will lead to a ground-breaking contribution that will improve the performance and quality of life at work and beyond for many ageing adult workers.

Anfangsdatum: 01.07.2020

Laufzeit: 14 Monaten

Drittmittelgeber: ZIM, Bundesministerium für Wirtschaft und Energie (BMWi) 

Zielsetzung des Projektes „KIRun“ ist die Entwicklung intelligenter Algorithmen, die Messdaten aus Training (GPS), Biomechanik (Inertialsensorik), Physiologie (Atem-/Herzfrequenz), Umgebungsgeräuschen (Laufuntergrund), Tonalität und Qualität der Sprache und Atmung, und gezielter Spracherfassung während des Laufens aufnehmen, verarbeiten und auswerten.

Hierzu müssen grundlegend neuartige Funktionalitäten erarbeitet beziehungsweise aus anderen Anwendungsfeldern in den Laufsport transferiert werden. Im Rahmen des Projektes sollen bestehende Technologien aus dem IoT-, Sensorik- und dem Audio-Bereich in den sportwissenschaftlichen Bereich transferiert werden und das Know-how der beteiligten Partner durch eine integrierte KI-Plattform kombiniert werden.

Remote Assessment of Disease and Relapse – Central Nervous System

Start date: 01.04.2016

End date: 31.03.2022

Funding body: EU (Europäische Union)

Homepage: www.radar-cns.org 

RADAR-CNS is a major research programme that is developing new ways of monitoring major depressive disorder, epilepsy, and multiple sclerosis using wearable devices and smartphone technology. RADAR-CNS aims to improve patients’ quality of life, and potentially to change how these and other chronic disorders are treated.

Holistic Deep Modelling for User Recognition and Affective Social Behaviour Sensing

Start date: 01.10.2018

Duration: 30 Months

Project paused: 01.05.2020 to 30.04.2021

Funding body: EU Horizon 2020 Marie Skłodowska-Curie action Individual Fellowship

The “Holistic Deep Modelling for User Recognition and Affective Social Behaviour Sensing” (HOL-DEEP-SENSE) project aims at augmenting affective machines such as virtual assistants and social robots with human-like acumen based on holistic perception and understanding abilities.

Social competencies comprising context awareress, salience detection and affective sensitivity present a central aspect of human communication, and thus are indispensable for enabling natural and spontaneous human-machine interaction. Therefore, with the aim to advance affective computing and social signal processing, we envision a “Social Intelligent Multi-modal Ontological Net” (SIMON) that builds on technologies at the leading edge of deep learning for pattern recognition. In particular, our approach is driven by multi-modal information fusion using end-to-end deep neural networks trained on large datasets, allowing SIMON to exploit combined auditory, visual and physiological analysis.

In contrast to standard machine learning systems, SIMON makes use of task relatedness to adapt its topology within a novel construct of subdivided neural networks. Through deep affective feature transformation, SIMON is able to perform associative domain adaptation via transfer and multi-task learning, and thus can infer user characteristics and social cues in a holistic context.

This new unified sensing architecture will enable affective computers to assimilate ontological human phenomena, leading to a step change in machine perception. This will offer a wide range of applications for health and wellbeing in future IoT-inspired environments, connected to dedicated sensors and consumer electronics.

By verifying the gains through holistic sensing, the project will show the true potential of the much sought-after emotionally and socially intelligent AI, and herald a new generation of machines with hitherto unseen skills to interact with humans via universal communication channels.

Start date: 01.05.2018

End date: 31.05.2021

Funding body: BMW AG

The project aims at real-time internet-scale sentiment analysis in unstructured multimodal data in the wild.

Training network on Automatic Processing of PAthological Speech

Start date: 01.11.2017

End date: 31.10.2021

Funding body: EU (Europäische Union)

Homepage: www.tapas-etn-eu.org 

There are an increasing number of people across Europe with debilitating speech pathologies (e.g., due to stroke, Parkinson’s, etc). These groups face communication problems that can lead to social exclusion. They are now being further marginalised by a new wave of speech technology that is increasingly woven into everyday life but which is not robust to atypical speech. TAPAS is proposing a programme of pathological speech research, that aims to transform the well-being of these people.

The TAPAS work programme targets three key research problems:

(a) Detection: We will develop speech processing techniques for early detection of conditions that impact on speech production. The outcomes will be cheap and non-invasive diagnostic tools that provide early warning of the onset of progressive conditions such as Alzheimer’s and Parkinson’s.

(b) Therapy: We will use newly-emerging speech processing techniques to produce automated speech therapy tools. These tools will make therapy more accessible and more individually targeted. Better therapy can increase the chances of recovering intelligible speech after traumatic events such a stroke or oral surgery.

(c) Assisted Living: We will re-design current speech technology so that it works well for people with speech impairments and also helps in making informed clinical choices. People with speech impairments often have other co-occurring conditions making them reliant on carers. Speech-driven tools for assisted-living are a way to allow such people to live more independently.

TAPAS adopts an inter-disciplinary and multi-sectorial approach. The consortium includes clinical practitioners, academic researchers and industrial partners, with expertise spanning speech engineering, linguistics and clinical science. All members have expertise in some element of pathological speech. This rich network will train a new generation of 15 researchers, equipping them with the skills and resources necessary for lasting success.

An Embedded Soundscape System for Personalised Wellness via Multimodal Bio-Signal and Speech Monitoring

Start date: 01.01.2018

End date: 31.12.2020

Funding body: The Bavarian State Ministry of Education, Science and the Arts in the framework of the Centre Digitisation.Bavaria (ZD.B)

Homepage: zentrum-digitalisierung.bayern/initiativen-fuer-die-wissenschaft/graduate-program/graduate-fellowships

The soundscape (the audible components of a given environment), is an omnipresence in daily-life. Yet research has shown, that elements of our acoustic soundscapes can negatively affect mental wellbeing.

Taking a dual analysis-synthesis approach this project, through multimodal feedback analysis, will explore the benefits of synthesised soundscape design and develop a ‘deep-listening’ personalised embedded system to improve human wellness. The project will explore questions pertaining to audible perception and develop novel methods for soundscape generation, informed by intelligent signal state monitoring.

Start date: 01.02.2016

End date: 30.11.2019

Fundeding body: Horizon 2020, EU

Homepage:  de-enigma.eu

DE-ENIGMA is developing artificial intelligence for a commercial robot (Robokind’s Zeno). The robot will be used for an emotion-recognition and emotion-expression teaching programme to school-aged autistic children. This approach combines the most common interests of children of school age: technology, cartoon characters (that Zeno resembles) and socializing with peers.

During the project, Zeno will go through several design phases, getting ‘smarter’ every time. It will be able to process children’s motions, vocalizations, and facial expressions in order to adaptively and autonomously present emotion activities, and engage in feedback, support, and play. 

Assistenzsystem zur Erkennung des emotionalen Zustandes von Werkstatt­mitarbeiterinnen und -mitarbeitern

Start date: 01.06.2015

End date: 30.09.2019

Funding body: EU

Engaging children with ASC (Autism Spectrum Conditions) in communication centred activities during educational therapy is one of the cardinal challenges by ASC and contributes to its poor outcome. To this end, therapists recently started using humanoid robots (e.g., NAO) as assistive tools. However, this technology lacks the ability to autonomously engage with children, which is the key for improving the therapy and, thus, learning opportunities. Existing approaches typically use machine learning algorithms to estimate the engagement of children with ASC from their head-pose or eye-gaze inferred from face-videos. These approaches are rather limited for modeling atypical behavioral displays of engagement of children with ASC, which can vary considerably across the children.

The first objective of EngageME is to bring novel machine learning models that can for the first time effectively leverage multi-modal behavioural cues, including facial expressions, head pose, vocal and physiological cues, to realize fully automated context-sensitive estimation of engagement levels of children with ASC. These models build upon dynamic graph models for multi-modal ordinal data, based on state-of-the-art machine learning approaches to sequence classification and domain adaptation, which can adapt to each child, while still being able to generalize across children and cultures. To realize this, the second objective of EngageME is to provide the candidate with the cutting-edge training aimed at expanding his current expertise in visual processing with expertise in wearable/physiological, and audio technologies, from leading experts in these fields.

EngageME is expected to bring novel technology/models for endowing assistive robots with ability to accurately ‘sense’ engagement levels of children with ASC during robot-assisted therapy, while providing the candidate with a set of skills needed to become one of the frontiers in the emerging field of affect-sensitive assistive technology.

Intelligent systems’ Holistic Evolving Analysis of Real-life Universal speaker characteristics

Start date: 01.01.2014

End date: 31.12.2018

Funding body: EU

Homepage: www.ihearu.eu 

Recently, automatic speech and speaker recognition has matured to the degree that it entered the daily lives of thousands of Europe’s citizens, e.g., on their smart phones or in call services. During the next years, speech processing technology will move to a new level of social awareness to make interaction more intuitive, speech retrieval more efficient, and lend additional competence to computer-mediated communication and speech-analysis services in the commercial, health, security, and further sectors. To reach this goal, rich speaker traits and states such as age, height, personality and physical and mental state as carried by the tone of the voice and the spoken words must be reliably identified by machines.

In the iHEARu project, ground-breaking methodology including novel techniques for multi-task and semi-supervised learning will deliver for the first time intelligent holistic and evolving analysis in real-life condition of universal speaker characteristics which have been considered only in isolation so far. Today’s sparseness of annotated realistic speech data will be overcome by large-scale speech and meta-data mining from public sources such as social media, crowd-sourcing for labelling and quality control, and shared semi-automatic annotation.

All stages from pre-processing and feature extraction, to the statistical modelling will evolve in “life-long learning” according to new data, by utilising feedback, deep, and evolutionary learning methods. Human-in-the-loop system validation and novel perception studies will analyse the self-organising systems and the relation of automatic signal processing to human interpretation in a previously unseen variety of speaker classification tasks.

The project’s work plan gives the unique opportunity to transfer current world-leading expertise in this field into a new de-facto standard of speaker characterisation methods and open-source tools ready for tomorrow’s challenge of socially aware speech analysis.

Assistenzsystem zur Erkennung des emotionalen Zustandes von von Werkstatt­mitarbeiterinnen und -mitarbeitern

Start date: 01.06.2015

End date: 31.05.2018

Funding body: EU

Project Homepage: www.emotass.de 

Im Projekt soll ein emotionssensitives, sprachgesteuertes Assistenzsystem entwickelt werden, das den emotionalen Zustand von Werkstattmitarbeiterinnen und -mitarbeitern zuverlässig aus der Interaktion mit dem Sprachassistenten erkennt. Zusätzlich zu den dafür erforderlichen Arbeiten zur Sprach- und Emotionserkennung wird ein psychologisch fundiertes Nutzerprofil erstellt, welches individuelle Eigenschaften abbildet. Damit zusammenhängende Anforderungen an das Persönlichkeitsrecht und den Datenschutz werden vom Konsortium berücksichtigt. Dieses halbautomatische System soll den individuellen Unterstützungsbedarf zuverlässig ableiten. Auf diese Weise wird eine optimale Anpassung der Arbeitsabläufe, z. B. durch Erläuterung und Anpassung einzelner Arbeitsschritte oder Motivation zur Pause, möglich.

Automatic Sentiment Estimation in the Wild

Start date: 01.02.2015

End date: 31.07.2018

Funding body: EU

Project Homepage: www.sewaproject.eu 

The main aim of SEWA is to deploy and capitalise on existing state-of-the-art methodologies, models and algorithms for machine analysis of facial, vocal and verbal behaviour, and then adjust and combine them to realise naturalistic human-centric human-computer interaction (HCI) and computer-mediated face-to-face interaction (FF-HCI). This will involve development of computer vision, speech processing and machine learning tools for automated understanding of human interactive behaviour in naturalistic contexts. The envisioned technology will be based on findings in cognitive sciences and it will represent a set of audio and visual spatiotemporal methods for automatic analysis of human spontaneous (as opposed to posed and exaggerated) patterns of behavioural cues including continuous and discrete analysis of sentiment, liking and empathy.