Exploiting contextual information is important for road segmentation tasks. In contrast to existing work primarily focusing on expanding the receptive field and stacking network layers, we propose a novel Graph-representation-based Multi-scale Feature Fusion U-net model (GMFFUnet) to address this problem. Our method enhances global context extraction by establishing connections between features of varying types and scales. Specifically, our model projects the entire feature pyramid into multiple interaction spaces and models the complex relationships among them to construct a graph and facilitate information exchange. The enhanced features are then fused back into the original feature pyramid. Our method provides substantial benefits on road segmentation tasks with complex backgrounds. Comparative experiments were carried out to evaluate the performance of the proposed method, and the results demonstrated that our proposed method outperforms all SOTA methods on the Massachusetts road dataset.