With the popularity of drones with vision sensors and the advancement of image processing technology, machine vision tasks based on image matching have received widespread attention. However, due to the complexity of aerial images, traditional matching methods based on handcrafted features unavoidably suffer from the low robustness because of lacking the ability to extract high-level semantics. On the other hand, deep learning shows a great potential in improving matching accuracy, but at the cost of a large amount of specific samples and computing resources, making it infeasible in many scenarios. To fully leverage the strengths of both approaches, we propose DeFusion, a novel image matching solution with a fine-grained decision-level fusion algorithm that effectively combines handcrafted features and deep features. We train generic features on public datasets, enabling us to handle unseen scenes. We use RootSIFT as prior knowledge to guide the extraction of deep features, significantly reducing the computational overhead. We also carefully design preprocessing steps by incorporating the attitude information of the drone. Eventually, as illustrated in our experimental results, the proposed scheme achieves an overall 2.5-6x more correct matches with improved robustness when compared to the existing methods.