This paper presents a new baseline for Malay-English code switched speech corpus; which is constructed using a factored form of time delay neural networks (TDNN-F), which reflected a significant relative percentage reduction of 28.07% in the word-error rate (WER), as compared to the Gaussian Mixture Hidden Markov Models (GMM-HMMs). The presented results also confirm the effectiveness of time delay neural networks (TDNNs) for code-switched speech.