The effective monitoring of PM2.5, a major indicator of air pollution, is crucial to human activities. Compared to traditional physiochemical techniques, image-based methods train PM2.5 estimators by using datasets containing pairs of images and PM2.5 levels, which are efficient, economical, and convenient to deploy. However, existing methods either employ handcrafted features, which can be influenced by the image content, or require additional weather data acquired probably by laborious processes. To estimate the PM2.5 concentration from a single image without requiring extra data, we herein proposed a learning-based prior-enhanced (PE) network—comprising a main branch, an auxiliary branch, and a feature fusion attention module—to learn from an input image and its corresponding dark channel (DC) and inverted saturation (IS) maps. In addition, we proposed an histogram smoothing (HS) algorithm to solve the problem of imbalanced data distribution, thereby improving the estimation accuracy in cases of heavy air pollution. To the best of our knowledge, this study is the first to address the phenomenon of a data imbalance in imaged-based PM2.5 estimation. Finally, we constructed a new dataset containing multi-angle images and more than 30 types of air data. Extensive experiments on image-based PM2.5 monitoring datasets verified the superior performance of our proposed neural networks and the HS strategy. The new dataset and codes are available at https://github.com/xxx (open after publication).