Contact center conversations often consist of silent segments, where neither the customer nor the agent is speaking. These silences if continued beyond an acceptable level can negatively impact contact center KPIs. Thus, understanding silences and defining measures to handle them better via appropriate coach- ing and alerting for agents is one of the key focus areas for contact centers. In this paper, we demonstrate how dialogue turns around silences could be used to understand the characteristics of silences (expected vs unexpected and agent vs cusomer-caused silences) via two text classification tasks. We propose a methodology to pre-train a silence-aware language model in contact center domain, called Silence-RoBERTa and demonstrate its ability to better capture the conversational characteristics around silences. Finally, we discuss the application of the above methodology in real-time and post-call settings and demonstrate its usability to reduce silences via a reallife case study.