Authors
- Hao Shi*
- Kazuki Shimada
- Masato Hirano*
- Takashi Shibuya
- Yuichiro Koyama*
- Zhi Zhong*
- Shusuke Takahashi*
- Tatsuya Kawahara*
- Yuki Mitsufuji
* External authors
Venue
- ICASSP 2024
Date
- 2023
Share
Diffusion-Based Speech Enhancement with Joint Generative and Predictive Decoders
Hao Shi*
Kazuki Shimada
Masato Hirano*
Yuichiro Koyama*
Zhi Zhong*
Shusuke Takahashi*
Tatsuya Kawahara*
* External authors
ICASSP 2024
2023
Abstract
Diffusion-based speech enhancement (SE) has been investigated recently, but its decoding is very time-consuming. One solution is to initialize the decoding process with the enhanced feature estimated by a predictive SE system. However, this two-stage method ignores the complementarity between predictive and diffusion SE. In this paper, we propose a unified system that integrates these two SE modules. The system encodes both generative and predictive information, and then applies both generative and predictive decoders, whose outputs are fused. Specifically, the two SE modules are fused in the first and final diffusion steps: the first step fusion initializes the diffusion process with the predictive SE for improving the convergence, and the final step fusion combines the two complementary SE outputs to improve the SE performance. Experiments on the Voice-Bank dataset show that the diffusion score estimation can benefit from the predictive information and speed up the decoding.
Related Publications
Deep Generative Models (DGMs), including Energy-Based Models (EBMs) and Score-based Generative Models (SGMs), have advanced high-fidelity data generation and complex continuous distribution approximation. However, their application in Markov Decision Processes (MDPs), partic…
Consistency Training (CT) has recently emerged as a promising alternative to diffusion models, achieving competitive performance in image generation tasks. However, non-distillation consistency training often suffers from high variance and instability, and analyzing and impr…
Diffusion models are prone to exactly reproduce images from the training data. This exact reproduction of the training data is concerning as it can lead to copyright infringement and/or leakage of privacy-sensitive information. In this paper, we present a novel way to unders…
JOIN US
Shape the Future of AI with Sony AI
We want to hear from those of you who have a strong desire
to shape the future of AI.
