Medical imaging is an essential tool for diagnosing and monitoring various health conditions. Robotic remoti- zation of such diagnostic medical procedures increases the safety of medical personnel and accessibility to the rural populace. However, ultrasound exams can be challenging and require skilled operators to obtain high- quality images. Automating such procedures requires programming robots to perform these dexterous medical skills. The programming constraint can be eliminated by leveraging human tutelage paradigms, enabling the robot to learn from observation and expert feedback. But, robots require massive libraries of demonstrations to learn effective policies using machine learning algo- rithms [?]. While such datasets are achievable for simple tasks, providing many demonstrations for contact-rich procedures such as ultrasound is not practical. This paper presents a novel method to learn complex contact-rich procedures by combining self-supervised practice with sparse expert feedback through coaching. The robotic ultrasound system (RUS) uses reinforcement learning (RL) to learn a policy for autonomous imaging of a urinary bladder phantom. Specifically, we use an off-policy soft actor-critic with a reward based on image quality assessed using a supervised neural network to learn the policy for ultrasound through practice. In addition to practice, experts provide online corrective feedback (coaching), which drives the robot to learn successful policies for ultrasound imaging. We show that leveraging expert feedback results in significantly increased performance than using a state-of-the-art RL policy.