In this video, I utilized artificial intelligence to generate an animated music video for the song Canvas by Resonate. This tool allows anyone to generate beautiful images using only text as the input. My question was, what if I used song lyrics as input to the AI, can I make perfect music synchronized videos automatically with the push of a button? Let me know how you think the AI did in this visual interpretation of the song.
After getting caught up in the excitement around DALL·E2 (latest and greatest AI system, it's INSANE), I searched for any way I could use similar image generation for music synchronization. Since DALL·E2 is not available to the public yet, my search led me to VQGAN + CLIP (Vector Quantized Generative Adversarial Network and Contrastive Language–Image Pre-training), before settling more specifically on Disco Diffusion V5.2 Turbo. If you don't know what any of these words or acronyms mean, don't worry, I was just as confused when I first started learning about this technology. I believe we're reaching a turning point where entire industries are about to shift in reaction to this new process (which is essentially magic!).
Important note:
While this AI is impressive, it still required additional input beyond just the song lyrics to achieve the music video I was looking for. For example, I added keyframes for camera motion throughout the generated world. These keyframes were manually synchronized to the beat by me. I also specified changes to the art style at different moments of the song. Since many of the lyrics are quite non-specific, even a human illustrator would have a hard time making visual representations. To make the lyrics more digestible by the AI, I sometimes modified the phrase to be more coherent, such as specifying a setting or atmosphere.
This was my first time working with DDV5, and I'm very happy with the results! There were many times where my jaw dropped upon seeing what the AI came up with. I haven't felt this sense of wonder from technology since I first experienced a HD videogame as a child.
Music - Rezonate - Canvas
3 Comments
newtboyLyrics-
Tuning out of star-light, the innocent lifetime flickers and flashes onto a canvas. But no, nobody seems worth it
And while the feeling's sophisticated, the view is so restrained. The past is nearing closer, and closer I fear.
So let's just cut the bullshit, and tell me who's independent. Thumbs twiddling' and your life's escaping your mind
Keep In mind, well the air ain't as free as it used to be but that's quite alright while you're still eighteen
It seems, to be, and i'll tell you who's over-rated. What's a beautiful person with some hideous dreams?
And while the lining isn't silver, we're verging on the side of the road, that'll take us home.
The edge of us is getting weighed, read about your life in your local papers.
Realize, the end of us is getting near, you're just a droplet of paint trickled off your canvas.
Delirious, illusions flowing backwards through the river of colour while we analyze to try to see it clearer. Secretly appealing to a lonely prisoner. Now what's your vision?
Initiate your shading, blend the universe together in time it only gets a little easier. Suddenly you piece yourself back together. Now what's your vision?
And I don't wanna have to give a damn about you or your inhibitions.
After all, you're the one who taught me love was not for every body.
However, I'm starting to see this world in the right light, yet still gets darker.
You see, it's just a simple symphony, or a painting? Whatever
This unnatural path, it spans infinitely, but my world is crumbling.
Let it collapse into the abyss, in the midst of summer it won't exist.
This emptiness, it's just a blank slate, waiting for a painter to interfere.
It's quite complex, but what do I know?
BSRGreat. AI on acid now.
noimsI read that South Park style:
Durr takin rrr ac'd!!!!!
Great. AI on acid now.
Discuss...
Enable JavaScript to submit a comment.