Human or AI
This weekend I trained an AWD-LSTM on music by many different composers from Classical Archives. Here are some generations paired with some random selections from the human composers. Following the great internet tradition, I’ve made a quiz (without the ads, but you can still share on Facebook or Twitter at the end if you like!)
For each pair, select the piece that was composed by a human.
[…] has even created a human or computer quiz that you can try for yourself; see if you can figure which pieces were generated by her […]
[…] new samples. I was quite excited about the early results — I have some sample clips here: http://christinemcleavey.com/human-or-ai/ . I’m now joining up with the language and safety teams here at OpenAI to see if we can push […]Leave a Comment
You must be logged in to post a comment.
Wow, that very interesting. Having some musical education and experience, you still had to ‘Go with your gut feeling’. Very hard to tell the difference in such complex pieces of music. I got 3 of 4 correct. Keep up the great work!
Thanks, glad you found it interesting! I see only the running totals of how many people get each score (and I obviously haven’t set this up as an actual scientific test), but it’s been fun to watch. So far I see an approximately equal number of 1’s, 2’s, and 3’s out of 4. No one has scored a 0, but I’ve only seen 2 people get 4’s (one of them I know personally & he’s a professional classical musician).
I guessed 3 out of 4 correctly. What I find interesting is how difficult it felt – a lot of what I conditioned on was trying to find areas of the piece that didn’t “intuitively make sense” then using that to update my probability of the sample being human. Now, defining the “intuitive” part is really tricky. Overall, I found this quite remarkable for how *close* the samples sound to the real thing – I wasn’t hugely confident in 50% of my answers to this as a consequence. Great stuff!
I got 4 out 4 correct. Not a professional musician or have any association to music. However, the machine generated music has always some odd notes which are recognizable just be years of listening to music. Very interesting. I don’t know how exactly, but is there a way to give negative training examples to the model so that it learns never to generate odd notes?
Thanks, that’s an interesting comment & idea. I suspect two things are going on with the odd notes: first, the training data itself is sometimes messy (I didn’t include any bad human clips here, but since the midi files are contributed by lots of different pianists, some of them do have frequent mistakes). Second, and probably more important is the way I sample the output generated by the model. I have a parameter that sets the randomness of the piece. If it’s low, then I’m almost always taking the model’s top guess for what note should come next. If it’s higher, I’m more often taking a 2nd or 3rd best guess. Higher randomness seems to yield more interesting sounding pieces, but it does sometimes produce a note that just sounds flat out wrong. I’ll have to think more about the idea of negative training examples. I do have a “critic” – a separate neural net which learned to distinguish if a piece is AI or human (exactly like this quiz), but right now I don’t use that to train the generator network, I just use it to help pick the best pieces out of a batch of generations. Thanks for the suggestion!
Yes indeed we can distinguish the human from the machine noticing the erratic notes most off the time in the machine piece of music.. could be interesting to see yet how your “critic” reacts between 2nd or 3rd choice !!
This was fun to do. It was interesting to see how well tghe machine adapted to different styles. What tipped the balance for me in favor of the one or the other was the form. A piece of music often has a journey that it takes the listener on. In traditional classical music that is aided by the form of the piece (ABA, Rondo, etc…). With more modern music that form isn’t quite as clearly defined but there still is a journey. When I previously listened a sample of ensemble music it was this journey that I missed. So in the quiz I went looking for this journey. I picked the piece that seemed more likely to take me on its journey. It seems as though that was a strategy that enabled me to correctly identify all 4 human pieces. But it’s a small sample size. I could have been lucky.
@Martin: This is what I tried to do as well, but with rather less success. An interesting quiz, to be sure! Sort of a musical Turing test.
I had 4 out of 4 correct and for me it actually wasn’t that hard.
What i feel is the biggest difference, is that the AI gets it’s inspiration from all types of melodies, but puts it into one type of “musical melody”.
The humans compose with different types of melodic music in one piece.
Repetitive bundles of notes hereby show which one the AI made.
Nonetheless is the AI music really good and the “bundles of notes” that it made couldv just as well been written by humans!
Super cool and keep up the good work!
Can you provide us with the names of the human-composed pieces in the quiz?
I got all incorrect, though I have some experience in both DL and music….
Can you share the github link for the code?
This is really cool. I got 3 out of 4, and missed the 1st one, which is a shame since I went back and listened again and it was quite obvious. As I moved along, I noticed how the AI composed music that lacks “structure”. Maybe due to the fact that RNN’s are used, we are seeing a lot of good transitions from chords to chords, but the music seems somehow lacking an overall “flow” and “structure. Music written by human have obvious pauses between phrases, but the AI generated ones seem to lack it. Hopefully, as deep learning understands more about semantics rather than just syntax, we would be able to tackle this in music generation.