Speaker recognition and audio analysis systems rely on time-frequency features computed from cleartext audio, creating a privacy vulnerability during inference. Fully homomorphic encryption (FHE) can address this by enabling computation on encrypted data, but prior approaches lack secure time-frequency computation. We introduce the first fully secure pipeline that computes STFT, Mel filterbanks, MFCCs, and gammatone filters directly on encrypted raw audio using FHE and quantized neural networks. We propose approximate STFT algorithms that optimize bit usage under quantization constraints, with theoretical error bounds. Experiments on VocalSet and OxVoc datasets show that our approximations reduce error rates in private statistical analysis of audio descriptors and improve classification accuracy. Our pipeline secures the exact frontend underpinning modern speaker verification and paralinguistic systems, with a practical heuristic for parameter selection.