Authors
- Hao Shi*
- Kazuki Shimada
- Masato Hirano*
- Takashi Shibuya
- Yuichiro Koyama*
- Zhi Zhong*
- Shusuke Takahashi*
- Tatsuya Kawahara*
- Yuki Mitsufuji
* External authors
Venue
- ICASSP 2024
Date
- 2023
Diffusion-Based Speech Enhancement with Joint Generative and Predictive Decoders
Hao Shi*
Kazuki Shimada
Masato Hirano*
Yuichiro Koyama*
Zhi Zhong*
Shusuke Takahashi*
Tatsuya Kawahara*
* External authors
ICASSP 2024
2023
Abstract
Diffusion-based speech enhancement (SE) has been investigated recently, but its decoding is very time-consuming. One solution is to initialize the decoding process with the enhanced feature estimated by a predictive SE system. However, this two-stage method ignores the complementarity between predictive and diffusion SE. In this paper, we propose a unified system that integrates these two SE modules. The system encodes both generative and predictive information, and then applies both generative and predictive decoders, whose outputs are fused. Specifically, the two SE modules are fused in the first and final diffusion steps: the first step fusion initializes the diffusion process with the predictive SE for improving the convergence, and the final step fusion combines the two complementary SE outputs to improve the SE performance. Experiments on the Voice-Bank dataset show that the diffusion score estimation can benefit from the predictive information and speed up the decoding.
Related Publications
Consistency Training (CT) has recently emerged as a promising alternative to diffusion models, achieving competitive performance in image generation tasks. However, non-distillation consistency training often suffers from high variance and instability, and analyzing and impr…
Detecting musical versions (different renditions of the same piece) is a challenging task with important applications. Because of the ground truth nature, existing approaches match musical versions at the track level (e.g., whole song). However, most applications require to …
Diffusion models have demonstrated exceptional performances in various fields of generative modeling, but suffer from slow sampling speed due to their iterative nature. While this issue is being addressed in continuous domains, discrete diffusion models face unique challenge…
JOIN US
Shape the Future of AI with Sony AI
We want to hear from those of you who have a strong desire
to shape the future of AI.