Guitar Chord Recognition Using AI
Advisor: Naor Movshovitz
Internship Location: Stanford University
Onsite Mentor: Ivan Rentaria
My goal is to create a chord recognition program that will be able to take the input of an audio file of a played guitar and report back the chords being played. As AI is a rapidly advancing field, I will need to conduct research into the current state of chord recognition programs and get up to date on the field of AI and Music. / Then, I will need to find a comprehensive dataset with annotated audio files and preprocess that data to implement it in a model. I will use the CQT transformation to create spectrograms from the audio files and extract the time-aligned string-fret combinations from the same audio files and train my data using a CNN model. The accuracy of my program will be represented in a learning curve using a Google lab notebook. And after tuning my parameters and optimizing my model, I will present a finalized program that can detect the chord identities of inputted guitar audio files.
Project Blog
Post 1: Senior Project Blog Post #1
Hello, and welcome to the first blog post of my new senior project! In this I will give a brief overview of my project, my goals, and my schedule for the next few weeks.
The objective of my project will be to create a guitar chord recognition program using AI. The program’s functionality will be so that a user can input an audio file of them playing guitar into my program, and it will output the chord identities of the guitar in the audio file. In order to accomplish this, I will need to find a comprehensive dataset with annotated audio files and preprocess that data to implement it in a model. I will use the CQT transformation to create spectrograms from the audio files and extract the time-aligned string-fret combinations from the same audio files and train my data using a CNN model. The accuracy of my program will be represented in a learning curve using a Google lab notebook. And after tuning my parameters and optimizing my model, I will present a finalized program that can detect the chord identities of inputted guitar audio files.
My pursuit of this project stems from my deep passion for the fusion of data science and music. This interest was ignited during my junior year when I developed a music recommender system using AI, merging these two fields. Since then, I've been captivated by the endless possibilities at the intersection of data science and music. With this project, I aim to further explore and contribute to this dynamic space, driven by a fervent curiosity and a desire to uncover new insights within the realm of music through the lens of data science. Next week, I will be working on research and finalizing my literature review. Stay tuned!
Post 2: Blog Post #2
Hello, and welcome to the second blog post of my senior project! In this post, we will go over the foundational research for my project and my literature review.
First, I started my research on Google Scholar and Hindawi, both great sources for finding quality research papers in your field. In my case, I was looking into various pre-existing chord recognition programs and the methodologies used in those programs. One paper I came across, titled "Transfer Learning for Audio Waveform to Guitar Chord Spectrograms Using the Convolution Neural Network," presented an interesting methodology. In this paper, they created a method for detecting chord sequences and notes from audio recordings of solo guitar performances using a Constant-Q-Transform (CQT) over the commonly used Short-time Fourier transform (STFT). The paper incorporates multiple datasets to make the model generalize better and also detects the beat of the audio. However, it only works for audios with just a guitar and no other instruments, meaning it does not work if any other instruments are playing simultaneously.
I plan to adopt the paper’s usage of SQT over STFT in my program in order to achieve a higher accuracy of chord detection. Additionally, I could potentially create a way to overcome the paper’s need for an isolated guitar audio to be inputted into the program by finding a way to isolate and separate the audio of the guitar from the rest of the audio file. So if someone were inputting a file of an entire song, my program could first separate the guitar from the rest of the song and then input the audio into my model.
My research has expanded my view on chord recognition and will greatly help my final project. Next week, I will start planning out my project, starting with my dataset and preprocessing. Stay tuned!
Post 3: Blog Post #3
Hello, and welcome to the third blog post of my senior project! In this post, I will go over my project’s dataset and plan out the preprocessing steps to make it useful in my program.
First, I will need to choose a dataset. After ample research, I chose the GuitarSet dataset developed at the Music and Audio Research Lab and NYU. This dataset contains recordings of musical excerpts played on an acoustic guitar, along with time-aligned annotations of pitch contours, string and fret positions, chords, beats, downbeats, and playing style (for more information about the dataset, see https://tomxi.weebly.com/uploads/1/2/1/6/121620128/xi_ismir_2018.pdf). In order to make this dataset useful for chord recognition, I will first have to preprocess the data. After loading the dataset, visualizing it, and researching its functionality, my advisor and I started pseudo-coding the data preprocessing steps. We plan to extract the relevant spectrogram data from the dataset, which is a visual representation of the audio, and also the time-aligned annotations of the played string-fret combinations. We then will have an array containing the chords played along with a visual representation of the respective audio. This will lay the foundation for creating my model and the rest of my program.
Next week, I will start implementing these steps in my data preprocessing. Stay tuned!
Post 4: Blog Post #4
Hello, and welcome to the fourth blog post of my senior project! In this post, I will go over my progress on preprocessing the GuitarSet dataset.
This week, I focused on continuing the work we started last week with GuitarSet. In a Google Collab notebook, I loaded in the dataset and the necessary libraries to work with it. Then, I created a Transform class with methods to load an audio file from the dataset, extract the string-fret combinations, and compute the audio data into spectrograms. The most important method, computeSpectogram, works by taking the audio signal and sample rate of the musical data and uses a Constant-Q-Transform to convert the audio into a spectrogram or the visual representation of the frequency of the sound. Then, the method extracts the MIDI values from the audio, which represent the string-fret combinations of the chords being played. Together, this method takes an audio file and outputs a visual representation of the data aligned with the chord that the audio file is identified with. The next method, processAudio, implements the computeSpecotgram method and returns an numpy array with the spectrogram data and a dictionary with the chord annotations.
Next week, I will focus on using these preprocessing functions to process the full GutiarSet dataset and turn all of the preprocessed data into a single new dataset ready for model training. Stay tuned!
Post 5: Blog Post #5
Hello, and welcome to the fifth blog post of my senior project! In this post, I will finalize my preprocessing for the GuitarSet dataset.
Last week, I focused on creating the Transfrom class with the necessary preprocessing methods to turn each individual audio file into spectogram data with string-fret annotations. This week, I focused on continuing the work we started last week with GuitarSet. The processAudio method I created last week implements the computeSpecotgram method and returns an numpy array with the spectogram data and a dictionary with the chord annotations, which can be called on any individual audio file. Now, in a new class I created a run method to take an audio file and call the processAudio function in order to get a numpy array of spectogram data and a dictionary of chord annotations. Then I created a combineResults function which takes the array of spectrogram data and dictionary of string-fret annotations and combines them in a tuple. The Gutiarset dataset has around 360 audio files and I wanted to take .2 seconds of audio from each audio file to preprocess and then add to a final dataset. My Main class loops through the audio files in Guitarset and calls the driver methods on each single audio file and adds it to a tuple. With this, my dataset is fully preprocessed
Next week, we move on to planning my next steps and modeling. Stay tuned!
Post 6: Blog Post #6
Hello, and welcome to the sixth blog post of my senior project! In this post, I will debug some issues that came up with my pre-processing and start building my model architecture.
Last week, I finished preprocessing GuitarSet and was ready to move on to building my model. However, after running some tests this week, I noticed that my preprocessing methods only worked for certain tracks and not others. For some audio files, it would simply repeat the same note positionings for each string on the guitar. After debugging this, we were ready to move on to the modeling stage of the project. Because the debugging took a substantial amount of time, we took the rest of the week to plan out the modeling steps for the following week. Since our model will need to take in visual inputs through our spectrograms, we opted to use a convolutional neural network (CNN). Our model's output will be a string-fret combination represented by a 6 × 19 matrix, where 6 stands for a conventional guitar's six strings and 19 for the 18 frets plus an additional space when a string is not being played. We decide to use a DNN architecture known as the multitask learning architecture to get this kind of output from a deep learning model. We plan to make our architecture branch out into 6 endpoints with each endpoint producing a vector size of 19.
Next week, we move on to implementing these steps and creating our model. Stay tuned!
Post 7: Blog Post #7
Hello, and welcome to the seventh blog post of my senior project! In this post, I continue working on the model architecture that we started last week.
Last week, we finished debugging our data set and started planning our model architecture. In order for our CNN model to accurately identify chords, every spectrogram needs to have a solution. This is why the inputted data set of string fret combinations needs to be in the form of a 6 by 18 matrix of MIDI values. The inputted audio file can then be associated with the matrix of string fret combination will act as the “solution” to the spectrogram of the audio file. Each of the endpoints in our model outputs a vector of size 19. To do this, we used a softmax function, where our vector sums to 1 and the highest value is given the value 1 while the rest are given the value 0. Once we combine each vector from each of the 6 branches, we have the string-fret combination of the entire guitar board to associate with the spectrogram image.
Next week, we move on to continue working on building our model. Stay tuned!