499 views

5:53

Watch
- Search Videos
  
  Advanced search »
- Submit a New Video
Discuss
About
- Try clicking on the various channels to see videos posted to that channel - if you're into music you can click the Music channel to get more music sifts - go on and explore! Use the "hotness" when you have opened the channel link to get videos that have gotten a lot of votes, or use "newness" to have a look at the new sifts.
  When browsing sifts you like, have a look at the Related Posts list to see other related videos, click the tags under the video title to see more sifts tagged with the same tags or have a look at other sifts from the same user in the More Posts By list. If the sifted video is in a play list, have a look at the play list to see other interesting videos the Sift users have linked together.
  Enter the Sift Lounge to chat with online sifters about anything and everything, or read interesting posts in the Sift Talk forum.
  The Top New Videos list is always full of interesting, current videos. It is always recommended that you have a look through the Top New Videos, but also have a look at these:
  - Upcoming Unsifted Videos - active video submissions that neither are published to the front page nor archived to the submitter's personal queue
- Have you seen a really funny, surprising, scary, interesting, cute or in any other way extra good quality video?
  Post and share it here - VideoSift is the best video aggregator on the Web, where we share, discuss, promote, and interact with the best videos out there on the web. Post it here, and share it with the large community of video-interested users. And - you are guaranteed to find more very good videos here already!
  Mind you, we only share videos you discover that someone else has made; if you "self link," by posting a video you have made, have posted or have interests in, you risk the ban hammer - read the posting guidelines before posting. The sift is NOT for promoting your own videos - it is for sharing videos others have made.
  Submit a Video Now »
Login

I Asked AI To Make A Music Video - This Is The Result

submitted by newtboy

2 years 5 months 3 weeks ago

In this video, I utilized artificial intelligence to generate an animated music video for the song Canvas by Resonate. This tool allows anyone to generate beautiful images using only text as the input. My question was, what if I used song lyrics as input to the AI, can I make perfect music synchronized videos automatically with the push of a button? Let me know how you think the AI did in this visual interpretation of the song.

After getting caught up in the excitement around DALL·E2 (latest and greatest AI system, it's INSANE), I searched for any way I could use similar image generation for music synchronization. Since DALL·E2 is not available to the public yet, my search led me to VQGAN + CLIP (Vector Quantized Generative Adversarial Network and Contrastive Language–Image Pre-training), before settling more specifically on Disco Diffusion V5.2 Turbo. If you don't know what any of these words or acronyms mean, don't worry, I was just as confused when I first started learning about this technology. I believe we're reaching a turning point where entire industries are about to shift in reaction to this new process (which is essentially magic!).

Important note:
While this AI is impressive, it still required additional input beyond just the song lyrics to achieve the music video I was looking for. For example, I added keyframes for camera motion throughout the generated world. These keyframes were manually synchronized to the beat by me. I also specified changes to the art style at different moments of the song. Since many of the lyrics are quite non-specific, even a human illustrator would have a hard time making visual representations. To make the lyrics more digestible by the AI, I sometimes modified the phrase to be more coherent, such as specifying a setting or atmosphere.

This was my first time working with DDV5, and I'm very happy with the results! There were many times where my jaw dropped upon seeing what the AI came up with. I haven't felt this sense of wonder from technology since I first experienced a HD videogame as a child.

Music - Rezonate - Canvas