Nowadays the qualitative characteristics of a scientific document are becoming more and more relevant because a large number of conferences, seminars, and journals generate a huge amount of scientific papers. A scientific paper consists of such elements as title, information about the authors, abstract, keywords, body and a list of references. References are one of the important factors that affect paper quality. Furthermore, the paper list is often used as a tool for manipulating citation indicators. The authors of this work do not explore the complete scientific article as an object of research, but only a list of references. In this paper, the authors analyze the qualitative characteristics of the cited sources and highlight the formal features that characterize the quality of the references. Ukrainian scientific papers were used as data for the experiments. Results showed that the quality question is а topical matter for Ukrainian science. In this paper, the authors have developed an application that allows the reviewer to analyze the list of paper references and an approach to the analysis of the bibliography, which allows identifying those sources that may not be relevant to the research topic. This approach allows determining the artificial increase of irrelevant references.