Score-Informed MIDI Velocity Estimation for Piano Performance by FiLM Conditioning

EasyChair Preprint no. 10251

Abstract

Piano is one of the most popular instruments among people that learn to play music. When playing the piano, the level of loudness is crucial for expressing emotions as well as manipulating tempo. These elements convey the expressiveness of music performance. Detecting the loudness of each note could provide more valuable feedback for music students, helping to improve their performance dynamics. This can be achieved by visualizing the loudness levels not only for self-learning purposes but also for effective communication between teachers and students. Also, given the polyphonic nature of piano music, which often involves parallel melodic streams, determining the loudness of each note is more informative than analyzing the cumulative loudness of a specific time frame.

This research proposes a method using Deep Neural Network (DNN) with score information to estimate note-level MIDI velocity of piano performances from audio input. In addition, when score information is available, we condition the DNN with score information using a Feature-wise Linear Modulation (FiLM) layer. To the best of our knowledge, this is the first attempt to estimate the MIDI velocity using a neural network in an end to end fashion. The model proposed in this study achieved improved accuracy in both MIDI velocity estimation and estimation error deviation, as well as higher recall accuracy for note classification when compared to the DNN model that did not use score information.

Keyphrases: Conditioned Deep Neural Network, FiLM Conditioning, MIDI Velocity Estimation, Note-level Loudness Estimation, Piano Performance Visualization

@Booklet{EasyChair:10251,
  author = {Hyon Kim and Marius Miron and Xavier Serra},
  title = {Score-Informed MIDI Velocity Estimation for Piano Performance by FiLM Conditioning},
  howpublished = {EasyChair Preprint no. 10251},

  year = {EasyChair, 2023}}