For individuals who have lost the ability to speak, silent speech interfaces provide a way to communicate naturally, bypassing traditional text-based assistive devices. Although recent efforts have achieved unprecedented results in building speaker-independent models, the fact that every person articulates speech differently often means that speaker adaptation is still necessary to achieve optimal performance. We propose an adaptive phone-aware weighted loss to help speech synthesis models capture individual articulation patterns. This method assigns higher weights to phones with larger errors, guiding the model to focus on its most challenging predictions. Experimental results show its effectiveness across EMG and lipreading modalities, with particular gains when pretraining is unavailable. Subsequent analyses across articulatory phonetic categories revealed speaker-dependent effects.