Speech biomarkers have shown promise for the remote assessment of ALS. However, to demonstrate clinical utility at tracking longitudinal progress of the disease, one needs to understand how well these biomarkers capture changes that are ‘clinically meaningful’, a concept that is not always clearly defined. Therefore, this paper defines and explores multiple methods of computing minimal clinically important difference (MCID) using ratings of speech impairment severity and listener effort as clinical anchors. We analyze how these methods impact the estimated responsiveness of various metrics collected from 125 ALS patients via a multimodal dialog based remote assessment platform. We find that select biomarkers are more responsive than the clinical standard ALSFRS-R across the board at tracking clinically meaningful changes related to speech severity. We further discuss advantages and disadvantages of different MCID computation methods for assessing ALS disease progression.