"Music is the effort we make to explain to ourselves how our brains work..." - Lewis Thomas (from The Medusa & the Snail)

Introductory Readings

Play It Again, Vladimir (via Computer). By Anne Midgette. The New York Times (June 5, 2005; subscription req'd). "This is the new world of computer music. In its infancy, way back in the 1960's, the goal was to use digital technology to create new sounds and new musical forms. Today scientists around the world are turning computers on human performance, seeking to quantify an element once thought to be intangible: the expressivity of a human artist. ... The reactions demonstrate a basic difficulty with mechanical reproduction of music: there is a subjective element involved in determining if it works. The final criterion for any such reproduction is the rather imprecise 'Turing test' of artificial intelligence: that is, whether it can make the listener think he or she is hearing a person rather than a machine. At the Austrian Research Institute for Artificial Intelligence, a group of leading researchers known as the Machine Learning, Data Mining and Intelligent Music Processing Group are trying to pinpoint just what it is that fools the ear. Led by Gerhard Widmer, they are looking at everything from improving the way computers 'hear' music to isolating the elements of individual performance style, as well as creating graphs and animations to illustrate different pianists' interpretations of the same passage of music. In a 2003 paper, 'In Search of the Horowitz Factor,' Dr. Widmer and his team described giving the computer 13 recordings of Mozart piano sonatas, played into a Bösendorfer Disklavier by the pianist Roland Batik, to see if they could use the computer to determine rules that described the pianist's interpretive choices. ... [T]here's still the thorny matter of how to get data from an audio recording into the computer. It's a question not just of having the computer play back a CD, but of translating the music into a language the computer can understand. A computer, by itself, can't recognize the difference between a note of music and a cough."

• See below for more information about the Machine Learning, Data Mining and Intelligent Music Processing Group.

Robo-music gives musicians the jitters - Realtime has never played Broadway, but touring shows and 'Les Miz' in London use it. By Gregory M. Lamb. The Christian Science Monitor (December 14, 2006). "The Venice (Fla.) Little Theater has a tiny orchestra pit, with room for only a handful of players, and a modest budget. So when it mounts a big musical like "Beauty and the Beast," it brings in an electronic ringer. A laptop computer, loaded with a program called OrchEXTRA, serves as a 'virtual orchestra,' from strings to woodwinds, drums to horns, giving the music such a rich sound that audience members may wonder how a full Broadway orchestra fits into the tiny pit. ... Virtual orchestras - computer programs that can vary dynamics and tempo and follow the singers on stage and the music director's baton - are changing the music world. ... Music students benefit, too. ... [Christopher Raphael's] Music Plus One system, under development for 13 years, begins with a recording of an orchestra playing the piece, minus the solo. Computer programming allows the orchestral accompaniment to 'listen' to the soloist and follow. The program also uses predictive programming, based on the player's previous playing style and past rehearsals, to anticipate what to do next. ... Virtual orchestras have yet to pass the musical version of the 'Turing test' - Alan Turing's 70-year-old test for how to tell when a computer's artificial intelligence has become indistinguishable from that of human intelligence. ... Concern about virtual orchestras, [Vicky Smolik] says, is 'a big thing' with her union members, 'from New York to Los Angeles ... and everywhere in between.' ... Raphael, a former member of the musicians' union, says he's 'totally opposed to the musicians' union position' against the virtual enhancement of orchestras. 'Ultimately, what they do ends up giving the world less music, not more music,' he says."

The Mozart Code. By Kate Taylor. The New York Sun (July 25, 2006). "By the time 'Enlightenment' is unveiled on Thursday evening at 10 p.m., in the colonnade outside Avery Fisher Hall, it will be able to run by itself in 35-minute sequences, 24 hours a day. ... In the most basic terms, 'Enlightenment' is an algorithm that allows 10 computers, working by trial and error, to reconstruct the composition of the 30-second coda to Mozart's Jupiter symphony, starting from scratch. Each of the 10 computers respresents one section of the orchestra. It's like waiting for monkeys to type Hamlet -- only they're specially trained monkeys that go back every time they make a mistake. ... The work represents the marriage of art and artificial intelligence; as such, it is a thoroughly contemporary work that reflects the various, unconventional ways in which artists today approach their chosen field. ... The three have been together since 2001, when the MIT Media Lab, where Mr. [Marc] Downie was working, asked Mr. [Paul] Kaiser and Mr. [Shelley] Eshkar to collaborate with him on a digital 'portrait' of the choreographer Merce Cunningham."

• "Musical analysis & performance: Behind Enlightenment is artificial intelligence code that analyzes Mozart’s music by examining 2 kinds of input: ..."

• Also see:
  • Musical robot composes, performs and teaches. By Matthew Abshire. CNN.com (October 3, 2006).
  • Drummers tune in to robot rhythm. New Scientist (May 13, 2006; Issue 255: page 27).
  • Music made modern with robotic marvels. Kirsten Tagami. The Atlanta Journal-Constitution (May 7, 2006).

Music and Artificial Intelligence. By Chris Dobrian (1993), Associate Professor in the Music Department of the Claire Trevor School of the Arts at the University of California, Irvine.

Understanding Music with AI: Perspectives on Music Cognition. Edited by Mira Balaban, Kemal Ebcioglu, and Otto Laske. AAAI Press. "This book addresses itself to four different communities: the professional musician, the professional music technologist and designer of music systems, the professional AI researcher, and the professional cognitive scientist and cognitive psychologist. ... It goes without saying that the book also addresses itself to teachers of music. The book could serve as a textbook for a course introducing questions of musical problem solving and performance, as well as the nature of musical knowledge and of theories of music. ... The book should [also] be of benefit to readers generally interested in the relationship of music and technology."

The Creative Processor - With a souped-up reproducing piano and some ingenious learning machines, AI maestro Gerhard Widmer is discovering how performers unlock the art in Mozart. By Pat Blashill. Wired Magazine (September 2001; Issue 9.09). "Widmer is a machine-learning specialist.... He's a classically trained pianist who took a left turn on the way to the conservatory and ended up in artificial intelligence. Now, backed by a prestigious grant from the Austrian government, 40-year-old Gerhard Widmer is heading up a six-year investigation into the expressive aspects of live musical performance. His goal, simply put, is to quantify the elusive, often rapturously mythologized, sound of music." And when you've finished reading the article, you can visit his lab!

Requiem for the soul. If creating sublime music is the highest of human achievements, how come a pile of computer code writes better music than most people? By Bob Holmes. New Scientist Magazine. (August 9, 1997). "How could Mozart write a symphony more than 200 years after his death? Meet a computer program called EMI (pronounced Emmy) and its creator, a living, human composer named David Cope. Under Cope's tutelage, EMI created the 42nd symphony by analysing some of Mozart's other 41 and extracting 'essence of Mozart'."

The Artist's Angst is All in Your Head. By George Johnson. Week in Review, New York Times on the Web, Nov. 16, 1997. A marvelous article about Aaron and EMI, computer programs that produce paintings and write music.

Understanding Musical Activities. A 1991 interview with Marvin Minsky, edited by Otto Laske. "When you hear a piece of music, different parts of your brain do different things with it, but we know too little about those different processes."

Novices and Pros Use High Technology to Compose. By Courtney Macavinta. Staff Writer, CNET News.com. (February 4, 2000)

General Readings

The Guitarist Is Metal. No, Not Heavy Metal. By Michael Beckerman. The New York Times (November 30, 2004; subscription req'd). "'We weren't interested in making robots that played musical instruments,' said Mr. Singer, of Lemur (League of Electronic Musical Urban Robots), in the subsequent conversation. 'We wanted robots that were musical instruments.'"

FLAIRS. See the Proceedings of the International Florida Artificial Intelligence Research Society Conference (AAAI Press) for papers collected for the AI and Music Special Track (2004, 2005).

Semantic descriptors to help the hunt for music. IST Results (January 4, 2006). "Currently under development by the IST programme-funded project SIMAC, the system represents a major leap forward in the application of semantics to audio content, allowing songs to be described not just by artist, title and genre but by their actual musical properties such as rhythm, timbre, harmony, structure and instrumentation. This in turn allows comparisons between songs to be made and listeners to find little-known tracks that suit their tastes but may otherwise go unnoticed."

Composer harnesses artificial intelligence to create music. By R. Colin Johnson. EE Times (December 30, 2002). "Just as IBM's Deep Blue showed the world a computer can play chess as well as a human master, Eduardo Reck Miranda, a researcher for the Sony Computer Science Laboratories Inc., aims to demonstrate a computer program able to compose original music. So far, neural networks have succeeded in imitating distinct musical styles, but truly original compositions have remained elusive. Miranda is tackling that problem with an orchestra of virtual musicians — called agents — that interact to compose original music. ... In his latest book, Composing Music with Computers (Focal Press), Miranda summarizes his AI research, which began with cellular automata and evolved into an 'adaptive games' strategy based on artificial-life models. ... For a computer to create truly novel compositions, Miranda has turned to artificial life (AL) models — the fodder for what he calls evolutionary musicology."

The Science of Art. By Raymond Kurzweil. Chapter 9 of his book, The Age of Intelligent Machines (1990). "The computer can be a powerful partner in exploring our thoughts and emotions and finding new ways of expressing them."

• Just How Old Can He Go? By Steve Lohr. The New York Times (registration req'd). "In 1965, as a teenager, [Ray Kurzweil] appeared on the television program, 'I've Got a Secret,' hosted by Steve Allen, for having written a computer program that composed piano music."

In Search of a Lost Melody - Computer assisted music: identification and retrieval. By Kjell Lemstrom. Finnish Music Quarterly Magazine 3-4/2000.

Pitch-perfect PC - Software that turns a computer into a smart, sensitive practice partner for music students. By Alex Markels. U.S. News & World Report (March 17, 2003). "From outside her bedroom, it sounds as if 16-year-old Carolina DePaulis is practicing trombone as an accompanist plays piano. They begin Guilmant's 'Morceau Symphonique' together, then DePaulis launches into a trombone solo. When she slows down, the pianist does too. But open the door and you'll find the junior from Minnesota's Mound Westonka High School all alone. DePaulis's mentor is a computer with a microphone and speakers, running a program called SmartMusic. Computer-aided music instruction isn't new; programs like Band in a Box and Music Minus One also provide accompaniment. But SmartMusic compares students' playing with a digital template, which lets it detect mistakes and mark them on a score. It also simulates the rapport between musicians by sensing and reacting to tempo changes. 'It makes me want to play more,' says DePaulis.'"

• Visit the SmartMusic site.

Play in a top orchestra, virtually. By Sonali Paul. Reuters UK (June 13, 2004). "Ever dreamed of playing in an orchestra? Well now you can and from the comfort of your own home or school. ... Australia's Adelaide Symphony Orchestra and a local software designer have created 'In The Chair', a cross between a karaoke machine and flight simulator, which allows you to play your favourite symphony via a computer, with a conductor on screen and tuition while you play. ... Using artificial intelligence, the software converts the sound into data about pitch, volume, timing and quality and compares it with an ideal performance. It then responds instantly, flagging you when you're playing sharp or flat, not in time, too loudly or not blending with the rest of the ensemble. ... As a short demonstration on www.inthechair.com shows, the feedback comes as text on the screen, arrows on the sheet music or recorded comments from members of the Adelaide Symphony. ... With dwindling funding for orchestras worldwide, getting a place in an orchestra will be increasingly tough for young musicians, so the software could give them a unique opportunity. 'It'll be very useful as a tool for students to have an opportunity to play with an orchestra, without actually having an orchestra,' said [Neal] Holmes."

High Tech Bots Play Ancient Tune. Here and Now radio program hosted by Robin Young. WBUR (January 25, 2006). "Last week, Japanese scientists announced the creation of a robot that can do sign language. That brings the science of robotics another step away from the assembly line, and closer to human contact. But robotic music? Won't it sound like that cheesy 'Theme from the Entertainer' on your telephone hold button? No, says 'Ensemble Robot' director Christine Southworth. This week the ensemble is premiering 'Heavy Metal,' a new piece for Balinese gamelan, robots and strings. The piece debuts at 'Music and the Invasion of Technology,' part of the 'When Science Meets Art' series underway at the Boston Museum of Science here in Boston." Use the Listen button at the top of their page to access the broadcast.

Related Resources

The Computer Music Project at CMU is developing computer music and interactive performance technology to enhance human musical experience and creativity. This interdisciplinary effort draws on Music Theory, Cognitive Science, Artificial Intelligence and Machine Learning, Human Computer Interaction, Real-Time Systems, Computer Graphics and Animation, Multimedia, Programming Languages, and Signal Processing. ."

• Be sure to check out the video demos: "Video example of Computer Accompaniment ... The computer listens to the singer (using the microphone you see in the video), follows her performance in the score, and synchronizes its accompaniment to her vocal performance. ... Video excerpt from 'In Transit' ... The computer listens to the soloist and classifies the performance according to style. The styles are "lyrical," "syncopated," "pointilistic," and "frantic." The absolute meanings of these terms is not important. What is important is that the player and computer agree so that the computer can understand what the player intends. The computer actually learns about style from examples ...."

Conferences and programmes related to AI and Music. Maintained by The Music Informatics Research Group.

EvoMUSART 2006, the 4th European Workshop on Evolutionary Music and Art, at EvoWorkshops2006. "The application of Evolutionary Computation (EC) techniques for the development of creative systems is a new, exciting and significant area of research. There is a growing interest in the application of these techniques in fields such as: art and music generation, analysis and interpretation; architecture; and design." Be sure to scroll down the page to see the abstracts of accepted papers.

• Also see the EvoMUSART 2005, EvoMUSART 2004 and EvoMUSART 2003 pages for abstracts of accepted papers.

GenJam. From Al Biles. "GenJam (short for Genetic Jammer) is an interactive genetic algorithm that learns to play jazz solos. It may well be the only evolutionary algorithm that is a 'working musician.'" Be sure to follow his links to the magazine articles about this cool program.

i-Maestro - "supported by the European Commission under the IST Sixth Framework Programme to develop interactive multimedia environment for technology enhanced music education. The project aims to explore novel solutions for music training in both theory and performance, building on recent innovations resulting from the development of computer and information technologies, by exploiting new pedagogical paradigms with cooperative and interactive self-learning environments, gestural interfaces, and augmented instruments, with computer-assisted tuition in classrooms to offer technology-enhanced environments for ear- and practical-training, creativity-, analysis-, and theory-training, ensemble playing, composition, etc."

Intelligent Music Processing and Machine Learning Group, Austrian Research Institute for Artificial Intelligence (OFAI). One of their many projects is:

• Computer-Based Music Research: Artificial Intelligence Models of Musical Expression. "The goal of this project is to use Artificial Intelligence methods to study the phenomenon of expressive music performance. The focus of the project is on developing and using machine learning and data mining methods for the analysis of expressive performance data. The goal is to gain a deeper understanding of this complex domain of human competence and to contribute new methods to the (relatively new) branch of musicology that tries to develop quantitative models and theories of musical expression."
  • Also see:
    • In Search of the Horowitz Factor. By Gerhard Widmer, Simon Dixon, Werner Goebl, Elias Pampalk, and Asmir Tobudic. AI Magazine 24(3): Fall 2003, 111-130. "The article introduces the reader to a large interdisciplinary research project whose goal is to use AI to gain new insight into a complex artistic phenomenon. We study fundamental principles of expressive music performance by measuring performance aspects in large numbers of recordings by highly skilled musicians (concert pianists) and analyzing the data with state-of-the-art methods from areas such as machine learning, data mining, and data visualization. The article first introduces the general research questions that guide the project and then summarizes some of the most important results achieved to date, with an emphasis on the most recent and still rather speculative work. A broad view of the discovery process is given, from data acquisition through data visualization to inductive model building and pattern discovery, and it turns out that AI plays an important role in all stages of such an ambitious enterprise. Our current results show that it is possible for machines to make novel and interesting discoveries even in a domain such as music and that even if we might never find the 'Horowitz Factor,' AI can give us completely new insights into complex artistic behavior."
    • Music Analysis. Tomorrow Today, the science magazine on Deutsche Welle's DW-TV (February 28, 2006). "What makes a concert a concert? When do notes become music? And what distinguishes one interpretation of Mozart from another? That's what computer expert Gerhard Widmer decided to find out with the help of a Bösendorfer grand piano equipped with chips and measuring tools designed to record the volume and length of every chord via a laptop. The computer also isolated individual styles of playing, allowing scientists at the Vienna Institute for Artificial Intelligence to identify the stylistic trademarks of players such as Horowitz, Barenboim and Rubinstein."

Interdisciplinary Centre for Scientific Research in Music (ICSRiM) at the University of Leeds. Among their many projects are:

• Optical Manuscript Recognition
• Scorebot: Theory and Design of an Automated Film Scoring Application.

Ircam, the Institut de Recherche et Coordination Acoustique/Musique (Institute for music/acoustic research and coordination). Research activities include Music Representation.

MUSIC-AI 2007: The International Workshop on Artificial Intelligence and Music. Held in conjunction with IJCAI2007, The Twentieth International Joint Conference on Artificial Intelligence Hyderabad, India January 6-12, 2007.

The Music Informatics Research Group: "The Music Informatics Research Group, formerly known as the Artificial Intelligence and Music Research Group, is part of the Division of Informatics at the University of Edinburgh, although it has strong links with other research groups and institutes within and outwith Edinburgh. Most members of the Music Informatics Research Group take part in one of the Division of Informatics' Research Institutes, mainly in the Institute for Communicating and Collaborative Systems, the Institute of Perception, Action and Behaviour and the Institute for Representation and Reasoning. The group's research interest is the application of Artificial Intelligence techniques and methods to modelling human musical behaviour and communication, and so to support music analysis, performance, education and composition research."

"The Music, Mind and Machine Group at the MIT Media Laboratory is developing new audio technologies for future interactive media applications. This ranges from automatic sensing of features in existing audio content to extremely compact representations of sound for efficient transmission and control in a networked future." [The group evolved from The Machine Listening Group].

MusicStrands.^TM "Powered by the MusicStrands Recommender^TM, the initial offering of MusicStrands^TM website provides music lovers with recommendations which are independant of label, artist and genre." "Our technology is powered by patent-pending innovations in search, artificial intelligence and collaborative filtering. The MusicStrands^TM team members are world-renowned experts in the field of Artificial Intelligence, including: statistical learning, Bayesian forecasting, probabilistic reasoning, recommender systems, data visualization techniques, and constraint-based reasoning." [Also see this article from AI in the news.]

Music Technology Group (MTG), part of the Departament of Technology and the Audiovisual Institute of the Pompeu Fabra University of Barcelona.

Saxex. From Josep Lluís Arcos of The Institut d'Investigació en Intel.ligència Artificial (IIIA), a center devoted to research in Artificial Intelligence (AI) belonging to the Spanish Scientific Research Council (CSIC). "We have developed Saxex, a case-based reasoning system for generating expressive performances of melodies based on examples of human performances. ... We have started to study the issue of musical expression in the context of tenor saxophone interpretations." And be sure to follow the link to other music projects that appears at the bottom of their page.

• Related article: AI and Music - From Composition to Expressive Performance. By Ramon Lopez de Mantaras and Josep Lluís Arcos. AI Magazine 23(3): Fall, 43-58. "In this article, we first survey the three major types of computer music systems based on AI techniques: (1) compositional, (2) improvisational, and (3) performance systems. Representative examples of each type are briefly described. Then, we look in more detail at the problem of endowing the resulting performances with the expressiveness that characterizes human-generated music. This is one of the most challenging aspects of computer music that has been addressed just recently. The main problem in modeling expressiveness is to grasp the performer's 'touch,' that is, the knowledge applied when performing a score. Humans acquire it through a long process of observation and imitation. For this reason, previous approaches, based on following musical rules trying to capture interpretation knowledge, had serious limitations. An alternative approach, much closer to the observation-imitation process observed in humans, is that of directly using the interpretation knowledge implicit in examples extracted from recordings of human performers instead of trying to make explicit such knowledge. In the last part of the article, we report on a performance system, SAXEX, based on this alternative approach, that is capable of generating high-quality expressive solo performances of jazz ballads based on examples of human performers within a case-based reasoning (CBR) system."

Song Tapper - Song Search and Retrieval by Tapping. As explained on the About page: "The SongTapper is the brainchild of Geoff Peters, Caroline Anthony, and Michael Schwartz, who were/are students at Simon Fraser University in BC, Canada. ... Our original proof of concept system was first presented at the Intelligent Systems Demonstrations at the AAAI-05 Conference in Pittsburgh PA, in July 2005, to a large audience of Artificial Intelligence experts and industry representatives."

• Also see these related articles:
  • Naming that tune just got easier. By Marianne Meadahl. SFU News (February 23, 2006; Volume 35, Number 4). "The simple, rhythmic bopping of a finger has led a trio of SFU [Simon Fraser University] computing science students to solve a musical dilemma -- how to name that unknown tune. They call it Song Tapper, and it can be found at www.songtapper.com/. The website has been designed to enable users to identify songs by tapping the melody on their spacebars. ... Geoff Peters, a jazz pianist who graduated from SFU in the fall, says the group came up with the idea last spring while brainstorming for a project for an artificial intelligence course. ... 'We have lots of ideas on where the tapping project could go, including commercial and educational applications,' says [Caroline] Anthony, who is majoring in cognitive science. 'It could be used on internet music sites for music search and discovery, on mobile devices for ring tone search, or in schools to teach rhythm. It could also possibly be used as a way for a computer to identify a person -- like a password.'"
  • February 25, 2006: Tuneful site taps into song rhythms. By Maurice Bridge. The Vancouver Sun & canada.com. "What does O Canada have in common with the William Tell Overture by Gioachino Rossini, Eight Days a Week by The Beatles, I Feel Good by James Brown and No Rain by Blind Melon? They're the answers you get when you tap out the first few bars of our national anthem on a website called The Song Tapper, which identifies tunes by the rhythm of their lyrics. It's a catchy little concept which has started to spread around the world, much to the surprise of its creators. The site (www.songtap per.com) is the brainchild of Simon Fraser University computing-science student Geoff Peters and two other students who developed it last year as a project for an artificial-intelligence course. ... The site has been up and running since September, but the Internet community really jumped on it it early this year. 'It really took off in the beginning of January,' Peters said. 'It learned songs by people teaching it, and it grew to 500 songs, and then a few thousand songs, and then it couldn't handle it, so we had to upgrade the server.' The site now knows about 14,000 songs, although it can only search about 3,500 at a time."

World Anthem Project. "Imagine creating the first, original World Anthem, from the common notes and tendencies of 193 recognized national anthems of the world and doing it on a home computer? Denver Music Producer John Guillot, University of California Music Professor David Cope and Associate Composer Stephen Bigger have done just that. ... The Anthem and lyrics were composed using Experiments in Musical Intelligence - a computer system that scientifically analyzes the common notes and tendencies of a musical work." - from their Idea page.

Other References Offline

Ames, Charles. 1992. Artificial Intelligence and Musical Composition. In The Age of Intelligent Machines, 2nd edition, ed. Kurzweil, Raymond, 351-379. Cambridge, MA:The MIT Press.

Balaban, M., K. Ebcioglu, and O. Laske, editors. 1992. Understanding Music with AI: Perspectives on Music Cognition. Menlo Park, CA: AAAI Press.

Chang, Yahlin. 1996. Roll Over, Beethoven. Newsweek 128 (July 29, 1996): 71.

Cook, J. 1998. Mentoring, metacognition and music: interaction analyses and implications for intelligent learning environments. International Journal of Artificial Intelligence in Education 9: 45-87.

Cope, David. 1999. One Approach to Muscial Intelligence. IEEE Intelligent Systems. 14(3): pp. 21 - 25.

Johnson, George. 1997. Undiscovered Bach? No, A Computer Wrote It. New York Times, Late NY Edition/November 11, 1997: Section F; pp. 1-2.

Kurzweil, Raymond. 1992. The Musical Arts/The Visual Arts/The Literary Arts and A Kind of Turing Test. In The Age of Intelligent Machines, 2nd edition, ed. Kurzweil, Raymond, 351-379. Cambridge, MA: The MIT Press.

Riecken., R. Douglas 1989. Wolfgang: Musical Composition by Emotional Computation. In Innovative Applications of Artificial Intelligence, ed. Schorr, Herbert and Alain Rappaport, 251-269. Menlo Park, CA: AAAI.

Smith, Matt, Alan Smaill, and Geraint A. Wiggins, editors. 1993. Proceedings of a Workshop held as part of AI-ED 93, World Conference on Artificial Intelligence in Education on Music Education: An Artificial Intelligence Approach. London, UK: Springer-Verlag. [Link is to the ACM Portal.]

Sowa, John F. 2000. Knowledge Representation. Pacific Grove,CA; Brooks/Cole. "Representing Music" at pages 15 -18.