Building Spoken Language Understanding (SLU) systems that do not rely on language specific Automatic Speech Recognition (ASR) is an important yet less explored problem in language processing. In this paper, we present a comparative study aimed at employing a pre-trained language agnostic acoustic model to perform SLU in low resource scenarios. Specifically, we use three different embedding settings extracted using Allosaurus, a pre-trained universal phone decoder: (1) Phone-labels (2) Panphone, and (3) Allo embeddings (proposed by us). These embeddings are then used in identifying the spoken intent. We perform experiments across three different languages: English, Sinhala, and Tamil each with different data sizes to simulate high, medium, and low resource scenarios. Our system improves on the state-of-the-art (SOTA) intent classification accuracy by absolute 2.11% for Sinhala and 7.00% for Tamil and achieves competitive results in English. Furthermore, we also present a quantitative analysis to show how the performance scales with the number of training examples.