Recent advancements in deep learning have significantly contributed to improving the accuracy of Autism Spectrum Disorder (ASD) diagnosis using functional MRI (fMRI) data. However, existing methods explore all the connections from fMRI or often face challenges in selecting the most relevant connections for classification. In this study, we propose a structured and explainable framework that integrates connectivity selection and attention-based modeling for ASD diagnosis. The Transformer-Encoder model, known for its ability to capture long-range dependencies in complex data, is used here to analyze the fMRI connectivity data. The Random Forest algorithm is employed for automatic brain connections selection, ranking and refining the set of connections according to their feature importance scores to retain the most discriminative ones. Experimental results show that our approach achieves an accuracy of 76.29\%, surpassing leading methods in the literature and highlighting its effectiveness in providing a reliable and robust ASD diagnosis.