PhD position Deep learning for speech perception

Application deadline: 25 April 2022

Start date: before 1 September 2022

Apply here:

Supervisor: Ricard Marxer <[email protected]>


Recent deep learning (DL) developments have been key to breakthroughs in many artificial intelligence (AI) tasks such as automatic speech recognition (ASR) [1] and speech enhancement [2]. In the past decade the performance of such systems on reference corpora has consistently increased driven by improvements in data-modeling and representation learning techniques. However our understanding of human speech perception has not benefited from such advancements. This internship sets the ground for a project that proposes to gain knowledge about our perception of speech by means of large-scale data-driven modeling and statistical methods. By leveraging modern deep learning techniques and exploiting large corpora of data we aim to build models capable of predicting human comprehension of speech at a higher level of detail than any other existing approaches [3].

This PhD position is funded by the ANR JCJC project MIM (Microscopic Intelligibility Modeling). It aims at exploiting AI methods for predicting and describing speech perception at the stimuli, listener and sub-word level.

The project also comprises a 2-year post-doctoral fellow who will work closely with the doctoral student and the principal investigator.

In the context of the project we will also collaborate with the following panel of experts: Jon P. Barker, Martin Cooke, Bernd T. Meyer and Dorothea Kolossa. Furthermore we also foresee the possibility of research stays in their research laboratories at the University of Sheffield (UK), University of the Basque Country (ES) or University of Oldenburg (DE).


The main role of the PhD student is to investigate and propose models that predict listeners’ responses to the noisy speech stimuli. We will target predictions at different levels of granularity such as predicting the type of confusion, which phones are misperceived or how a particular phone is confused. Existing corpora of such data are available and will be used. The student will therefore be able to focus on the development and analysis of the models.

In the MIM project, we focus on a corpora of consistent confusions: speech-in-noise stimuli that evoke the same misrecognition among multiple listeners.  In order to simplify this first approach to microscopic  intelligibility  prediction,  we  will  restrict  to  single-word  data.   This  should  reduce  the  lexical factors to aspects such as usage frequency and neighborhood density, significantly limiting the complexity of the required language model. Consistent confusions are valuable experimental data about the human speech perception process. They provide targets for how intelligibility models should dif-ferentiate from automatic speech recognition (ASR) systems.  While ASR models are optimised to recognise what has been uttered, the proposed models should output what has been perceived by a set of listeners.

Several models regularly used in speech recognition tasks will be trained and evaluated in predicting the misperceptions of the consistent confusion corpora. We will first focus on well established models such as GMM-HMM and/or simple deep learning architectures. Advanced neural topologies such as TDNNs, CTC-based or attention-based models will also be explored, even though the relatively small amount of training data in the corpora is likely to be a limiting factor. As a starting point we envisage solving the 3 tasks described in [3] consisting of 1) predicting the probability of occurrence of misrecognitions at each position of the word, 2) given the position, predicting a distribution of particular phone misperceptions, and 3) predicting the words and the number of times they have been perceived among a set of listeners.  Predictions will be evaluated using the metrics also defined in [3] and random and oracle predictions will be used as references. These baseline models will be trained using only in-domain data and optimized on word recognition tasks.

The work of the selected candidate will focus on using deep learning approaches and more specifically in self-supervised or semi-supervised techniques. Several research directions will be explored, including but not limited to:

  • perceptual-based loss functions
  • advanced speech representation learning pipelines
  • self-supervised and low-resource learning

This topic is recently attracting a growing interest with last year’s launch of the first Clarity challenge. This challenge tackles the difficult task of performing speech enhancement for optimising intelligibility of a speech signal in noisy conditions. The challenge is the first of its kind with the objective of advancing hearing-aid signal processing and the modelling of speech-in-noise perception. Participation in the challenge will allow the researcher to broaden the impact of their research work and compare the proposed methods to others in the field.


The candidate shall have the following profile:

  • Master 2 level or equivalent in one of the following fields: machine learning, computer science, applied mathematics, statistics, signal processing
  • Good English written and spoken language skills
  • Programming skills, preferably in Python

        Furthermore the ideal candidate would have:

  • Experience in one of the main DL frameworks (e.g. PyTorch, Tensorflow)
  • Notions in speech or audio processing

Application procedure

In order to apply please complete the following application form:

The selected candidate will start at any time between April 2022 and August 2022. Applications will be processed on a rolling basis

For any questions regarding the position, please contact [email protected]

Monthly Salary

Around € 1,500 after taxes (note that this salary is standardized across all public universities in France). The salary is compatible with the costs of living in Toulon (e.g., rent prices are 50% lower in Toulon than in Paris) (More info).


DYNI is a team of the LIS laboratory (UMR 7020 CNRS) at the University of Toulon. The team is composed of 5 faculty members including a chair in AI and 2 AI ANR JCJC laureats. The team also comprises 5 post-docs, 6 PhD students.

Ricard Marxer (supervisor of this position) heads the team ( and is the director of a Erasmus Mundus Joint Master’s degree MIR in AI and robotics (