In this study, the problem of identifying in-set versus out-of-set speakers is addressed. Here the emphasis is on low enrolment and test data durations, in a text-independent mode. In order to compensate for the limited enrolment data (5 sec), a method is proposed that utilizes data from speakers that are acoustically close to a particular in-set speaker. A speaker specific model is obtained by adaptation of a base model that is built using data from such speakers. The performance of the proposed algorithm is evaluated using the TIMIT database with an adapted GMM classifier (GMM-UBM) employed as the baseline system. Experimental results show a consistent increase in system performance, with a relative improvement ranging from 10.57-58.33% depending on inset speaker size and test data duration.