Few-shot colorization aims to learn a model to colorize grayscale images with little training data. Yet, existing models often fail to keep color consistency due to ignored patch correlations of the images. In this paper, we propose PCCNet, a novel Patch-wise Contrastive Colorization Network to learn color synthesis by measuring the similarities and variations of image patches in two different aspects: inter-image and intra-image. Specifically, for inter-image, we investigate a patch-wise contrastive learning mechanism with positive and negative samples constraint to distinguish color features between patches across images. For intra-image, we explore a new intra-image correlation loss function to measure the similarity distribution which reveals structural relations between patches within an image. Furthermore, we augment our network with a color memory module to remember the correct color for specific kinds of structures and textures. Experiments show that our method allows the correct color to spread naturally over objects and also achieves higher scores in quantitative comparisons with related methods.