MGPHot: A Dataset of Musicological Annotations for Popular Music (1958–2022)

2.1 On the Music Genome Project^®

The Music Genome Project^® is arguably the most comprehensive musicological analysis of music. Since the year 2000, a team of trained music analysts has carefully annotated millions of individual songs across all major genres through a detailed, structured, and quality‑controlled process.

The skills required to be part of this team of music analysts include the ability to transcribe harmonic progressions and identify melodic intervals by ear, and a mastery of Music Theory equivalent to a four‑year college degree. Music analysts also possess broad genre knowledge and a detailed understanding of musical cultural movements.

A typical music analysis involves an average of 15 minutes of careful listening while the music analyst annotates hundreds of predefined attributes per song. These attributes include details of harmony, melody, rhythm, instrumentation, vocal performance, genre and stylistic influence, lyrical subjects and sentiments, compositional focus, overall sonority, and production techniques.

Music analysts use custom software to make annotations efficient. Of the hundreds of attributes, not all need to be considered for every song. For example, instrumental music will exclude many dozens of attributes related to vocal qualities and lyrics. When instruments are absent, there is no need to consider multiple attributes used to describe the activity and tonal qualities of instruments. Due to this variability, the annotation time varies considerably. A New Age solo piano piece might take only a few minutes, while a heavily orchestrated and structurally complex Bollywood song might take 30 minutes.

Many annotations describing the presence of attributes make use of the ‘incidental to dominant’ scaling concept. For example, in the case of an instrument, if the score is zero, the instrument is absent. Low scores, on the ‘incidental’ side of the scale, indicate a minor role for that instrument. For example, the instrument is only present in part of the song or is barely noticeable in the mix. High scores, on the ‘dominant’ side of the scale, indicate a prominent or lead role which dominates the experience of the song. In most popular music, it is common for instruments to receive a middling score, indicating they are performing an expected supporting role for a dominant lead vocal.

Several quality control measures are used to ensure accuracy and alignment among analysts. Although the labor‑intensive annotation process makes it impractical to redundantly analyze every song, each analyst is subject to expert review of a percentage of their annotated songs by dedicated Quality Assurance analysts, and feedback is delivered on a monthly cadence. In addition, analysts meet regularly in genre‑aligned teams to discuss evolutions in music and how to maintain alignment as new popular music trends emerge.

2.2 The MGPHot dataset

MGPHot is derived from the Music Genome Project.^® The song selection for MGPHot is based on a dataset collected from the Billboard Hot 100 chart data ranging from August of 1958 until September of 2022.² The purpose of the Billboard Hot 100 is to provide the music industry a measure of a song’s popularity in the United States. As music consumption has evolved, so has the methodology of the Hot 100 chart. For example, in 2005, Billboard started incorporating digital song sales and in 2013, digital streams were considered, while in 2017 Billboard stopped considering the sales of physical singles. Users of this dataset should be aware that the methodology for compiling the Billboard Hot 100 has shifted over time, but its core objective of reflecting music popularity has remained consistent (Trust, 2023). In addition, the chart may reflect certain biases inherent to the structure and evolution of the music industry.

The artist names and track titles of the Billboard charts were mapped to an internal music catalog using an internal search API and some filtering heuristics. A common issue in this process is the presence of variations in how collaborators are represented, either in the track title or the artist name. To address this, we compared the results provided by the search API with the names listed in the charts, pre‑processing the strings and utilizing n‑grams and string similarity measures to evaluate the accuracy of the mappings. After applying these filtering techniques, we could successfully map a total of 21,320 tracks.

To evaluate the quality of the mapping, we conducted a manual verification of 300 randomly selected track mappings. The process revealed an accuracy rate of 94%. Among the 18 identified non‑exact mappings, only one represented an actual error, where the track was incorrectly mapped to the wrong artist. The remaining 17 were variations of the original song, including three live versions, nine remixes, one karaoke version, and four versions featuring additional artists. Considering only a single actual error, the effective accuracy rate increases to 99%.

The year assigned to each track in the dataset is its first year of appearance in the charts. In Figure 1, we show the number of tracks per year in the original Billboard Hot 100 dataset and those successfully mapped that make up the final dataset. Note that the coverage of mapped tracks increases with the years. Although the coverage is partial, especially for the early years of the charts, it is a large enough proportion (always higher than 50%).

Figure 1

Song attributes were selected for full coverage of the dataset. The final list of attributes is grouped into seven different categories: rhythm, compositional focus, harmony, instrumentation, sonority, vocals, and lyrics. The complete list of attributes and their descriptions is reported in Table 2. Attributes are manually annotated on a 0‑to‑5 scale, which we translated to a 0‑to‑1 scale. Therefore, an attribute with a score towards 1 will imply high dominance of that attribute in the song, and a score towards 0 its absence. However, there is an exception in the case of the Major/Minor Key Tonality gene, where a score towards 0 indicates a predominance of a minor harmony and towards 1, a predominance of a major harmony.

The 58 attributes fall in a spectrum from subjective to objective. For example, measuring the beats per minute, or the presence of a certain rhythmic meter can be done with objective precision, while the level of ‘danceability,’ or the compositional importance of melody, will always be colored by each individual’s taste and experience with music. Although it is impossible to reduce a work of art into a set of purely objective measures, we have worked hard to codify each attribute through extensive training, rigorous QA processes, and detailed documentation including many musical examples, ensuring the best possible alignment between annotators. For instance, when considering the ‘explicitness’ of lyrics, we have developed and systematically applied an internal standard that evaluates the presence of certain explicit words and thematic content, grounded in a contemporary Western context.

3.1 Auto‑Tagging

One of the most immediate applications of the MGPHot dataset is the development of automated tagging systems. Traditional auto‑tagging models are typically trained to classify songs into broad categories such as genre or mood, utilizing datasets like the Ballroom dataset (Gouyon et al., 2006), The Million Song Dataset (Bertin‑Mahieux et al., 2011), the FMA dataset (Defferrard et al., 2017b), or the MTG‑Jamendo dataset (Bogdanov et al., 2019). However, MGPHot offers a more comprehensive set of musical attributes, including rhythm, instrumentation, harmony, vocals, compositional focus, and lyrics, some of which are unique and not found in other datasets. Additionally, MGPHot provides detailed scores that quantify the strength of each attribute. These characteristics make MGPHot a valuable resource for developing and evaluating more sophisticated auto‑tagging models, enabling the automatic generation of tags that more accurately reflect human‑curated annotations.

3.2 Music recommendation

MGPHot provides an opportunity to improve music recommendation algorithms by incorporating multidimensional representations of songs. Many existing recommendation systems rely on collaborative filtering and/or audio features, which may not capture the full complexity of a track’s musical attributes. By integrating the 58 musicological attributes from MGPHot, recommendation systems can offer more personalized and contextually relevant suggestions. For instance, a user’s listening preferences can be matched with specific rhythmic patterns, compositional structures, or lyrical themes, leading to more accurate and satisfying recommendations.

3.3 Chart prediction

Given that the MGPHot dataset focuses on songs that appeared in the Billboard Hot 100 between 1958 and 2022, it offers a unique resource for chart prediction models. These models aim to predict whether a song will become a hit based on its musical features. With MGPHot’s extensive annotations, chart prediction can move beyond basic audio features and consider deeper attributes such as rhythmic complexity, lyrical themes, and instrumental choices. This allows for a more refined analysis of what makes a song popular, providing researchers with better tools for understanding and forecasting musical success.

3.4 Musicological and cultural studies

The detailed time‑based coverage of MGPHot makes it a useful resource for studying the temporal evolution of popular music and conducting broader musicological and cultural analyses. Researchers can use the dataset to explore how specific musical attributes, such as rhythm or harmony, have shifted over the years and correlate these changes with cultural and societal trends. In addition to temporal analysis, MGPHot rich annotations may be useful for musicological and cultural studies. Researchers can investigate how specific musical genres or attributes have driven the development of new styles, while also exploring the cultural impact of different musical trends. The dataset allows for the examination of how shifts in musical preferences reflect broader societal changes, such as through the prominence of certain lyrical themes or compositional styles in relation to historical events. Overall, we hope this dataset serves as a valuable resource for musicological studies and aids in exploring the evolution and cultural significance of popular music.

4.1 Visualizing trend curves

In our quest to comprehend the evolution of popular music, we aim to compare the songs of each year with those of all other years. Each song within our dataset is represented as a 58‑dimensional vector of musical attributes, ranging from 0 to 1, forming what we refer to as the MGPHot track embedding. By leveraging these embeddings, we compute the centroid of all tracks from a given year, thereby generating an embedding that captures the essence of that particular year, referred to as the MGPHot year embedding.

To analyze the evolution of each of the musical attributes present in the MGPHot year embedding, we calculate a smoothed version of the values for each dimension across the years by applying 1‑D convolutions with a Hanning window of 10, following the methodology in (Oramas et al., 2018), and then visualize the trend curves of these musical attributes grouped by category (see Figure 2). Vertical lines in the plots indicate the years of faster change in the dataset (or revolutions), as explained in Section 4.2.

Figure 2

This visualization provides a comprehensive overview of the attributes in the dataset and highlights the evolution of musical characteristics over time. Several insights are immediately apparent from these plots. Notable trends include a shift towards minor harmonies, a transition in sonority from acoustic to electric and later to synthetic sounds, a reduction in the compositional emphasis on riffs and form, an increase in breathiness and nasality in vocals, and a growing prevalence of explicit lyrics. Additionally, we can observe sudden changes at specific points in time, such as the rise of synthetic sonorities, synthesizers, and synthetic drums in the early 1980s, the surge in explicit lyrics in the mid‑2010s, and the end of the prominence of swing feel in the mid‑1960s. While we highlight only a few insights, a deeper analysis of these plots can yield many more detailed conclusions. To facilitate this, we provide an Appendix as supplementary material with all the plots of individual attributes, including the pre‑smoothed mean values and the standard deviation of each year.

4.2 Identifying music revolutions

As observed in the previous plots, some musical changes happen gradually, while others occur more abruptly. These rapid shifts contribute to the fragmented perception of music history, dividing it into distinct eras. To pinpoint the years where these discontinuities occur, we applied the methodology used in Mauch et al. (2015), which analyzed music evolution using descriptors extracted from audio. Our goal was to determine if using musicological annotations would lead us to similar conclusions, while also allowing for a deeper exploration of the specific attributes driving these changes.

Following the aforementioned methodology, we first measure the similarity of the music within years, calculating the pairwise cosine distance among all MGPHot year embeddings, and visualizing the results through a heatmap (see Figure 3). This visualization enables us to visually identify distinct regions—or eras—of popular music. Then, to better identify the inflection points, we employ the Foote novelty metric (Foote, 2000) with a kernel size of 10 years. We selected this kernel size to be consistent with the window size used to compute the 1‑D convolutions of the trend curves. This metric aims to measure local changes by correlating a small checkerboard‑like kernel along the main diagonal of the similarity matrix. This results in a novelty function that unveils peaks at specific years where the kernel meets a transition between two contrasting blocks (see Figure 4).

Figure 3

Figure 4

Applying this function, we identify three major peaks in years 1964, 1983, and 2016, and two minor peaks in 1991 and 2007. This result agrees with the prior study on the evolution of music between 1960 and 2010 based on audio descriptors (Mauch et al., 2015), which identified three music revolutions in 1964, 1983, and 1991. Our results not only confirm these three revolutions but also reveal two additional ones in 2007 and 2016.

4.3 Breakdown by category

To understand the factors contributing to the identified peaks, we conduct a similar analysis on various subsets of dimensions within the MGPHot year embeddings. Since each dimension corresponds to a distinct musical attribute, and these attributes are categorized accordingly, we generate a year embedding for each category. This approach yields distinct year embeddings representing rhythm, instrumentation, sonority, compositional focus, harmony, vocals, and lyrics attributes. Subsequently, we iterate the process of computing pairwise cosine distance between all year embeddings within each category, followed by plotting the Foote novelty metric (refer to Figure 5).

Figure 5

These plots reveal when revolutions occurred for different musical characteristics, allowing us to identify the key drivers behind each revolution. For instance, rapid changes in rhythm occurred in 1964, while instrumentation and sonority shifted significantly in 1983. In contrast, compositional focus, harmony, vocals, and lyrics saw their most substantial changes in 2016. The peaks observed in these isolated categories align with the revolutionary years identified in the previous section, enabling us to associate specific musical changes with those major turning points.

To further explore the relationships between musical attributes across different categories, we complement the novelty metric with a Mantel test (Mantel and Valand, 1970). This test, commonly used in ecology to measure the linear correlation and statistical significance between two proximity matrices, is applied here to compare the distance matrices across different musical categories. Instead of comparing species, we compare the distances between years, using the distance matrices calculated for each category. A heatmap is then generated to illustrate the Pearson correlations among these categorical distance matrices (see Figure 6). The statistical significance of the results is determined via a two‑tailed test with 10,000 permutations, yielding a p‑value < 0.05 for all comparisons.

Figure 6

The analysis reveals strong correlations between lyrics, composition, and vocals, as well as between instrumentation and sonority, which is consistent with the way these attributes changed together during the identified musical revolutions. Rhythm, on the other hand, shows the weakest correlations, with only moderate associations to sonority and instrumentation.

4.4 A closer look into revolutions

Through our analysis, we identified key moments in history when rapid changes occurred in music. In this section, we propose several hypotheses to explain these musical revolutions, drawing on the information available in the dataset. Note, however, that these hypotheses would require further investigation, e.g., through interdisciplinary studies and causal modeling to validate the underlying causes of these transformative periods.