Neural Karaoke

Generating lyrics based on image

Researchers from University of Toronto developed an AI system that creates and then sings songs by analyzing the visual components of an uploaded image.

The researchers trained their neural network on 100 hours of online music. Once trained, the program can take a musical scale and melodic profile and produce a simple 120-beats-per-minute melody, adding chords and drums to the final result.

The next step was to train their ‘Neural karaoke’ program on a collection of pictures and their captions to learn how specific works can be linked to visual patterns and objects – once fed an image, the program can compile relevant lyrics and sing them.