On Evaluation of Inter- and Intra-Rater Agreement in Music Recommendation
By: Arthur Flexer, Taric Lallai and Katja Rašl
References
- Aizenberg, N., Koren, Y., and Somekh, O. (2012). Build your own music recommender by modeling Internet radio streams. In Proceedings of the 21st International Conference on World Wide Web, pages 1–10. DOI: 10.1145/2187836.2187838
- Aucouturier, J.-J. (2009). Sounds like teen spirit: Computational insights into the grounding of everyday musical terms. Language, Evolution and the Brain, pages 35–64.
- Balke, S., Driedger, J., Abeßer, J., Dittmar, C., and Müller, M. (2016). Towards evaluating multiple predominant melody annotations in jazz recordings. In Proceedings of the 17th International Society for Music Information Retrieval Conference, pages 246–252.
- Bosch, J., and Gómez, E. (2014). Melody extraction in symphonic classical music: A comparative study of mutual agreement between humans and algorithms. In Proceedings of the 9th Conference on Interdisciplinary Musicology.
- Cantor, J. R., and Zillmann, D. (1973). The effect of affective state and emotional arousal on music appreciation. The Journal of General Psychology, 89(1): 97–108. PMID: 4715319. DOI: 10.1080/00221309.1973.9710822
- Cleverdon, C. W. (1991). The significance of the Cranfield tests on index languages. In Proceedings of the 14th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3–12. DOI: 10.1145/122860.122861
- Cuadra, C. A., and Katter, R. V. (1967). Opening the black box of ‘relevance’. Journal of Documentation. DOI: 10.1108/eb026436
- Downie, J. S. (2006). The Music Information Retrieval Evaluation eXchange (MIREX). D-Lib Magazine, 12(12). DOI: 10.1045/december2006-downie
- Flexer, A., and Grill, T. (2016). The problem of limited inter-rater agreement in modelling music similarity. Journal of New Music Research, 45(3): 239–251. DOI: 10.1080/09298215.2016.1200631
- Flexer, A., and Lallai, T. (2019). Can we increase interand intra-rater agreement in modeling general music similarity? In Proceedings of the 20th International Society for Music Information Retrieval Conference, pages 494–500.
- Gómez-Cañón, J. S., Cano, E., Herrera Boyer, P., and Gómez Gutiérrez, E. (2020). Joyful for you and tender for us: The influence of individual characteristics and language on emotion labeling and classification. In Proceedings of the 21st International Society for Music Information Retrieval Conference.
- Hu, X., and Kando, N. (2012). User-centered measures vs. system effectiveness in finding similar songs. In Proceedings of the 13th International Society for Music Information Retrieval Conference, pages 331–336.
- Hu, X., Lee, J. H., Bainbridge, D., Choi, K., Organisciak, P., and Downie, J. S. (2017). The MIREX Grand Challenge: A framework of holistic userexperience evaluation in music information retrieval. Journal of the Association for Information Science and Technology, 68(1): 97–112. DOI: 10.1002/asi.23618
- Hu, X., and Liu, J. (2010). Evaluation of music information retrieval: Towards a user-centered approach. In Proceedings of the 4th Workshop on Human-Computer Interaction and Information Retrieval.
- Jones, M. C., Downie, J. S., and Ehmann, A. F. (2007). Human similarity judgments: Implications for the design of formal evaluations. In Proceedings of the 8th International Conference on Music Information Retrieval, pages 539–542.
- Ju, Y., Margot, S., McKay, C., Dahn, L., and Fujinaga, I. (2020). Automatic figured bass annotation using the new Bach Chorales Figured Bass Dataset. In Proceedings of the 21st International Society for Music Information Retrieval Conference.
- Juslin, P. N., and Sloboda, J. A. (2013).
Music and emotion . In Deutsch, D., editor, The Psychology of Music, pages 583–645. Academic Press, third edition. DOI: 10.1016/B978-0-12-381460-9.00015-8 - Klien, V., Grill, T., and Flexer, A. (2012). On automated annotation of acousmatic music. Journal of New Music Research, 41(2): 153–173. DOI: 10.1080/09298215.2011.618226
- Konečni, V. J. (2010).
The influence of affect on music choice . In Juslin, P. N. and Sloboda, J. A., editors, Handbook of Music and Emotion: Theory, Research, Applications, pages 698–723. Oxford University Press. - Koops, H. V., de Haas, W. B., Bransen, J., and Volk, A. (2020). Automatic chord label personalization through deep learning of shared harmonic interval profiles. Neural Computing and Applications, 32(4): 929–939. DOI: 10.1007/s00521-018-3703-y
- Koops, H. V., de Haas, W. B., Burgoyne, J. A., Bransen, J., Kent-Muller, A., and Volk, A. (2019). Annotator subjectivity in harmony annotations of popular music. Journal of New Music Research, 48(3): 232–252. DOI: 10.1080/09298215.2019.1613436
- Lee, J. H., and Cunningham, S. J. (2013). Toward an understanding of the history and impact of user studies in music information retrieval. Journal of Intelligent Information Systems, 41(3): 499–521. DOI: 10.1007/s10844-013-0259-2
- Lee, J. H., Hu, X., Choi, K., and Downie, J. S. (2015). MIREX Grand Challenge 2014 user experience: Qualitative analysis of user feedback. In Proceedings of the 16th International Society for Music Information Retrieval Conference, pages 779–785.
- Lex, E., Kowald, D., and Schedl, M. (2020). Modeling popularity and temporal drift of music genre preferences. Transactions of the International Society for Music Information Retrieval, 3(1): 17–30. DOI: 10.5334/tismir.39
- Mayer, J. D., and Gaschke, Y. N. (1988). The experience and meta-experience of mood. Journal of Personality and Social Psychology, 55(1): 102. DOI: 10.1037/0022-3514.55.1.102
- Moore, J. L., Chen, S., Turnbull, D., and Joachims, T. (2013). Taste over time: The temporal dynamics of user preferences. In Proceedings of the 14th International Society for Music Information Retrieval Conference, pages 401–406.
- Ni, Y., McVicar, M., Santos-Rodriguez, R., and De Bie, T. (2013). Understanding effects of subjectivity in measuring chord estimation accuracy. IEEE Transactions on Audio, Speech, and Language Processing, 21(12): 2607–2615. DOI: 10.1109/TASL.2013.2280218
- Nieto, O., Farbood, M. M., Jehan, T., and Bello, J. P. (2014). Perceptual analysis of the f-measure for evaluating section boundaries in music. In Proceedings of the 15th International Society for Music Information Retrieval Conference, pages 265–270.
- Panteli, M., Rocha, B., Bogaards, N., and Honingh, A. (2017). A model for rhythm and timbre similarity in electronic dance music. Musicae Scientiae, 21(3): 338–361. DOI: 10.1177/1029864916655596
- Porcaro, L., Gómez, E., and Castillo, C. (2021). Perceptions of diversity in electronic music: The impact of listener, artist, and track characteristics. arXiv preprint arXiv:2101.11916.
- Quinton, E., Harte, C., and Sandler, M. (2015). Extraction of metrical structure from music recordings. In Proceedings of the 18th International Conference on Digital Audio Effects.
- Salamon, J., Gómez, E., Ellis, D. P., and Richard, G. (2014). Melody extraction from polyphonic music signals: Approaches, applications, and challenges. IEEE Signal Processing Magazine, 31(2): 118–134. DOI: 10.1109/MSP.2013.2271648
- Schamber, L. (1994). Relevance and information behavior. Annual Review of Information Science and Technology, 29: 3–48.
- Schedl, M., Flexer, A., and Urbano, J. (2013). The neglected user in music information retrieval research. Journal of Intelligent Information Systems, 41(3): 523–539. DOI: 10.1007/s10844-013-0247-6
- Schedl, M., Zamani, H., Chen, C.-W., Deldjoo, Y., and Elahi, M. (2018). Current challenges and visions in music recommender systems research. International Journal of Multimedia Information Retrieval, 7(2): 95–116. DOI: 10.1007/s13735-018-0154-2
- Selvi, C., and Sivasankar, E. (2019).
An efficient context-aware music recommendation based on emotion and time context . In Data Science and Big Data Analytics, pages 215–228. Springer. DOI: 10.1007/978-981-10-7641-1_18 - Selway, A., Koops, H. V., Volk, A., Bretherton, D., Gibbins, N., and Polfreman, R. (2020). Explaining harmonic inter-annotator disagreement using Hugo Riemann’s theory of ‘harmonic function’. Journal of New Music Research, 49(2): 136–150. DOI: 10.1080/09298215.2020.1716811
- Serra, J., Müller, M., Grosche, P., and Arcos, J. L. (2014). Unsupervised music structure annotation by time series structure features and segment similarity. IEEE Transactions on Multimedia, 16(5): 1229–1240. DOI: 10.1109/TMM.2014.2310701
- Serra, X., Magas, M., Benetos, E., Chudy, M., Dixon, S., Flexer, A., Gómez Gutiérrez, E., Gouyon, F., Herrera Boyer, P., Jordà Puig, S., Paytuvi, O., Peeters, G., Schlüter, J., Vinet, H., and Widmer, G. (2013). Roadmap for music information research.
http://mires.eecs.qmul.ac.uk/files/MIRES_Roadmap_ver_1.0.0.pdf . - Seyerlehner, K., Widmer, G., and Knees, P. (2010).
A comparison of human, automatic and collaborative music genre classification and user centric evaluation of genre classification systems . In International Workshop on Adaptive Multimedia Retrieval, pages 118–131. Springer. DOI: 10.1007/978-3-642-27169-4_9 - Smith, J. B., and Chew, E. (2013). A meta-analysis of the MIREX structure segmentation task. In Proceedings of the 14th International Society for Music Information Retrieval Conference, pages 45–47.
- Sturm, B. L. (2014). The state of the art ten years after a state of the art: Future research in music information retrieval. Journal of New Music Research, 43(2): 147–172. DOI: 10.1080/09298215.2014.894533
- Trochim, W. (2000). The Research Methods Knowledge Base. Atomic Dog Publishing, Cincinnati, OH, 2nd edition.
- Urbano, J., Schedl, M., and Serra, X. (2013). Evaluation in music information retrieval. Journal of Intelligent Information Systems, 41(3): 345–369. DOI: 10.1007/s10844-013-0249-4
- Weiß, C., Schreiber, H., and Müller, M. (2020). Local key estimation in music recordings: A case study across songs, versions, and annotators. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28: 2919–2932. DOI: 10.1109/TASLP.2020.3030485
- Wiggins, G. A. (2009). Semantic gap?? Schemantic schmap!! Methodological considerations in the scientific study of music. In Proceedings of the 11th IEEE International Symposium on Multimedia, pages 477–482.
IEEE . DOI: 10.1109/ISM.2009.36
DOI: https://doi.org/10.5334/tismir.107 | Journal eISSN: 2514-3298
Language: English
Submitted on: Mar 26, 2021
Accepted on: Oct 12, 2021
Published on: Nov 24, 2021
Published by: Ubiquity Press
In partnership with: Paradigm Publishing Services
Publication frequency: 1 issue per year
© 2021 Arthur Flexer, Taric Lallai, Katja Rašl, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.