Image segmentation algorithms often depend on appearance models that characterize the distribution of pixel values in different image regions. We describe a novel approach for estimating appearance models directly from an image, without explicit consideration of the pixels that make up each region. Our approach is based on algebraic expressions that relate local image statistics to the appearance models of spatially coherent regions. The approach leads to two different algorithms for estimating appearance models. We present experimental results that demonstrate the proposed methods work well in practice and lead to effective image segmentation algorithms.