This paper presents a speech operated smart-home control system with a focus on users with special needs - elderly people, people suffering from blindness or low vision and people with mobility impairments. The proposed system simplifies the use of various home appliances by providing a unified speech-controlled interface and thus improves the quality of their everyday life. The paper presents the implementation scheme and decisions taken in order to have a real-time operational system as well as real users' experimental results.