This paper addresses the problem of 3D hand pose annotation using a single depth camera. While hand pose annotations are critically important for training deep neural networks, creating such reliable training data is challenging and manual labor intensive. We propose a semi-automatic pipeline for efficiently and accurately labeling the 3D hand poses in a depth video containing single hand images. The process starts by selecting a subset of frames that are representative of all the frames in the dataset and the user only provides an estimate of the 2D hand key-points in these selected frames. We use this information to infer the 3D location of the hand joints for all the frames by enforcing appearance, temporal and distance constraints. Finally, we demonstrate that our method can generate 3D hand pose training data more accurately using less manual intervention and offering more flexibility in comparison to other state-of-the-art methods.