IMAGEShow HN: The First Open Source Diffusion ASR Audio Model 15x Faster Than WhisperWe trained diffusion-gemma-asr, an open-source speech recognition model that is 15x faster than Whisper, based on DiffusionGemma and Whisper Small.Instead of generating text one token at a time like Whisper, in diffusion-gemma-asr we transcribe by denoising an entire transcript in parallel.To enable the model to learn, we don’t touch DiffusionGemma in training, rather we use a frozen whisper-small’s encoder for acoustic features, and train only a 42M parameter adapter, about 0.16% of the entire model.We keep both the Whisper-small encoder and DiffusionGemma’s original weights frozen. And train a convolutional projector that compresses the Whisper’s 1,500 audio frames into 188 audio tokens, along with small LoRA adapters on DiffusionGemma’s attention layers so the model can learn to route those new tokens.DiffusionGemma is different from most diffusion language models as it does not begin with a sequence of mask tokens. It begins with a fixed canvas filled with random vocabulary tokens.
