Blind Audiovisual Source Separation Based on Sparse Redundant Representations - Université de Rennes Accéder directement au contenu
Article Dans Une Revue IEEE Transactions on Multimedia Année : 2010

Blind Audiovisual Source Separation Based on Sparse Redundant Representations

Anna Llagostera Casanovas
  • Fonction : Auteur
  • PersonId : 884023
Monaci Gianluca
  • Fonction : Auteur
  • PersonId : 884024
Pierre Vandergheynst
  • Fonction : Auteur
  • PersonId : 839985

Résumé

In this paper, we propose a novel method which is able to detect and separate audiovisual sources present in a scene. Our method exploits the correlation between the video signal captured with a camera and a synchronously recorded one-microphone audio track. In a first stage, audio and video modalities are decomposed into relevant basic structures using redundant representations. Next, synchrony between relevant events in audio and video modalities is quantified. Based on this co-occurrence measure, audiovisual sources are counted and located in the image using a robust clustering algorithm that groups video structures exhibiting strong correlations with the audio. Next periods where each source is active alone are determined and used to build spectral Gaussian mixture models (GMMs) characterizing the sources acoustic behavior. Finally, these models are used to separate the audio signal in periods during which several sources are mixed. The proposed approach has been extensively tested on synthetic and natural sequences composed of speakers and music instruments. Results show that the proposed method is able to successfully detect, localize, separate, and reconstruct present audiovisual sources.
Fichier principal
Vignette du fichier
2010_IEEE_TMM_LlagosteraEtAl_BAVSS.pdf (5.03 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

inria-00541412 , version 1 (27-01-2011)

Identifiants

Citer

Anna Llagostera Casanovas, Monaci Gianluca, Pierre Vandergheynst, Rémi Gribonval. Blind Audiovisual Source Separation Based on Sparse Redundant Representations. IEEE Transactions on Multimedia, 2010, 12 (5), pp.358 -- 371. ⟨10.1109/TMM.2010.2050650⟩. ⟨inria-00541412⟩
302 Consultations
372 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More