Students’ actions in the classroom play a key role in studying student performance in class. With the development of computer vision technology, automatic recognition of student action has become possible. This study focuses on the automatic recognition of students’ spontaneous actions in the real classroom environment. Considering the lack of data about students’ spontaneous actions, this study first establishes a database of students’ spontaneous actions in the real classroom environment. Our database consists of 4,917 images of students’ spontaneous actions, including the actions of students in 10 learning states, such as raising hand, standing up, taking notes, clapping, taking photo, looking up, holding cheek, playing mobile phone, stretching and laying on the desk. Based on the characteristics of students’ actions in the classroom, we proposed a new 11-layer convolutional neural network algorithm based on EDSR. In this algorithm, in view of the problem that convolutional neural network is easy to overfit small sample data, a data augmentation method is introduced for data processing. The experiment result shows that the proposed algorithm improved the accuracy of action recognition effectively on the database established in this paper. And such a spontaneous database proposed in this paper provides a good data support for the study of students’ actions in the classroom in the field of education.