Authors

Venue

Date

Share

Enhancing Whisper's Accuracy and Speed for Indian Languages through Prompt-Tuning and Tokenization

Pankaj Wasnik

Kumud Tripathi

Raj Gothi

ICASSP-25

2025

Abstract

Automatic speech recognition has recently seen a significant advancement with large foundational models such as Whisper. However, these models often struggle to perform well in low-resource languages, such as Indian languages. This paper explores two novel approaches to enhance Whisper's multilingual speech recognition performance in Indian languages. First, we propose prompt-tuning with language family information, which enhances Whisper's accuracy in linguistically similar languages. Second, we introduce a novel tokenizer that reduces the number of generated tokens, thereby accelerating Whisper's inference speed. Our extensive experiments demonstrate that the tokenizer significantly reduces inference time, while prompt-tuning enhances accuracy across various Whisper model sizes, including Small, Medium, and Large. Together, these techniques achieve a balance between optimal WER and inference speed.

Related Publications

Precise Event Spotting in Sports Videos: Solving Long-Range Dependency and Class Imbalance

CVPR, 2025
Sanchayan Santra, Vishal Chudasama, Pankaj Wasnik, Vineeth N Balasubramanian

Precise Event Spotting (PES) aims to identify events and their class from long, untrimmed videos, particularly in sports. The main objective of PES is to detect the event at the exact moment it occurs. Existing methods mainly rely on features from a large pre-trained network…

Faster Machine Translation Ensembling with Reinforcement Learning and Competitive Correction

NAACL, 2025
Kritarth Prasad, Mohammadi Zaki, Pratik Singh, Pankaj Wasnik

Ensembling neural machine translation (NMT) models to produce higher-quality translations than the $L$ individual models has been extensively studied. Recent methods typically employ a candidate selection block (CSB) and an encoder-decoder fusion block (FB), requiring infere…

Cross-Modal Fusion and Attention Mechanism for Weakly Supervised Video Anomaly Detection

CVPRW, 2025
Ayush Ghadiya, Purbayan Kar, Vishal Chudasama, Pankaj Wasnik

Recently, weakly supervised video anomaly detection (WS-VAD) has emerged as a contemporary research direction to identify anomaly events like violence and nudity in videos using only video-level labels. However, this task has substantial challenges, including addressing imba…

  • HOME
  • Publications
  • Enhancing Whisper's Accuracy and Speed for Indian Languages through Prompt-Tuning and Tokenization

JOIN US

Shape the Future of AI with Sony AI

We want to hear from those of you who have a strong desire
to shape the future of AI.