Brain-to-text (BTT) systems that decode attempted speech from neural activity have achieved 4.2% word error rate (WER). These systems demonstrate potential for daily use similar to automatic speech recognition (ASR) systems, enabling communication for individuals with profound speech loss. To examine fine-grained error patterns in both BTT and ASR, we introduce a refined alignment algorithm to detect word edits, along with four word-level metrics to assess exact correctness and semantic distance of these edits. Analyzing errors by word frequency reveals a significant performance disparity among frequent, infrequent, and rare words across all models. Although transformer-based architectures and self-supervised pre-training achieve lower error rates, the gap between frequent and infrequent words remains substantial. Our analysis indicates that misclassifying infrequent words incurs higher semantic costs, suggesting that addressing this word-level performance gap could enhance overall system usability across ASR and BTT. Our implementation is available.