ISCA Archive Interspeech 2024
ISCA Archive Interspeech 2024

AdaRA: Adaptive Rank Allocation of Residual Adapters for Speech Foundation Model

Zhouyuan Huo, Dongseong Hwang, Gan Song, Khe Chai Sim, Weiran Wang

A recent paradigm shift in artificial intelligence has witnessed the emergence of foundation models. These foundation models, possessing billions of parameters and trained on extensive datasets, are anticipated to demonstrate superior generalization across diverse downstream tasks. Residual Adapters represent a broadly employed methodology for efficient adaptation, achieved by updating a limited set of additive parameters while maintaining a fixed bottleneck dimension. However, when the parameter budget is constrained, allocating additive parameters uniformly across layers proves sub-optimal. In this paper, we propose a novel adaptive efficient adaptation method that automatically determines the optimal number of bottleneck dimensions for Residual Adapters at different layers. Experimental results confirm that the proposed method effectively learns an optimal additive parameter allocation, surpassing the performance of comparable methods in speech recognition domain adaptation.