Despite the introduction of Universal Primary Education (UPE) program in Uganda in 1997, children with disabilities (CWDs) have continued to fail to attain primary education. Therefore, this study looked at the right of children with disabilities (CWDs) to primary education in Uganda; it investigated the challenges concerning access to primary education of children with disabilities (CWDs) and how can the situation be improved. It was qualitative study that purposely involved 32 children, 8 parents, 8 teachers from 4 primary schools, and 8 district officials from Kabale district in Uganda. Data was collected through interviews and focus group discussions (FGD). Results of the study revealed the following challenges that hinder CWDs from accessing primary education, indirect costs, child labor, lack of individual education plans, lack of teaching and learning materials, lack of teachers trained in special needs education, lack of community health personnel and educational psychologists. Lack of sanitary facilities in schools, negative attitudes by parents towards education of CWDs, lack of cooperation between schools and local communities especially local councils, large class sizes, wider accessibility issues, funding of special units and special schools at the expense of mainstream schools, lack of assistive devices and inadequate funds allocated to CWDs‟ education. In the study, it is recommended that if CWDs are to access primary education, both the central government and local governments should make it mandatory for all new buildings to have ramps, put in place a sustainable community transport system, support CWDs‟ house holds with income generating projects. Provide lunch at school for pupils, school uniform and sanitary pads, put in place affirmative action for CWDs in all education institutions, amend the Education Act and PWD Act to allow compulsory recruitment of social workers and a provision for sanctions to parents who keep CWDs out of school, modify the training program for teachers and other personnel in the education system, improve the conditions of service for teachers, introduce child to child clubs in schools to reduce stigmatization of CWDs, and increase the education budget for special needs education.