Fish is the most important food that fulfills the demand for protein. It is very necessary for human growth. As a result, fish farming has a huge impact on a country. This sector provides job opportunities for many people. Thus, it helps to reduce poverty. But the disease of fish is very concerning. Different types of fish diseases affect fish, resulting in a huge economic loss. Therefore, there is a need for advanced technology that can identify the exact disease and classify the disease that affects a fish. However, we have proposed a deep learning-based technology that will help identify and precisely categorize fish diseases. For this novel purpose, we have gathered almost 1500 datasets for different fish diseases: fungal diseases, gill disease, red spot disease, and white tail disease. In this paper, we have utilized pre-trained convolutional neural network architectures: EfficientNetB7, InceptionV3, and Xception. Finally, We have proposed a Vision transformer-based model. We have compared the results of the pre-trained convolutional neural network model with our proposed Vision Transformer model. The experiment results have shown that the vision transformer model has reached a maximum accuracy of 97.92\%, suppressing the results of existing pre-trained convolutional neural network architectures.