Existing unsupervised domain adaptation (UDA) methods have shown success in semantic segmentation of high-resolution remote sensing (HRS) images. However, these methods assume that the source and target domains share the same set of labeled categories, which becomes problematic when new classes appear in the target domain. This assumption complicates the accurate prediction of boundaries and shapes of unknown classes. Additionally, self-supervised easy-hard adaptation strategies may result in the model learning erroneous knowledge from noisy pseudo-labels. To address these challenges, we propose a novel Open Set Domain Adaptation method, OpenRS-Net, specifically designed for remote sensing images. Our framework introduces a boundary-aware loss module, MorphoCon, based on morphological operations (dilation and erosion) to improve the representation of object boundaries. Additionally, we introduce a prototype-based pseudo-label denoising module (PPD) to reduce pseudo-label noise by calculating prototype distances. We conduct experiments on benchmark datasets, including Potsdam, Vaihingen, and the custom MultiLandRS dataset, demonstrating superior performance on both known and unknown class datasets, verifying the generalizability of our method.