In this video, I utilized artificial intelligence to generate an animated music video for the song Canvas by Resonate. This tool allows anyone to generate beautiful images using only text as the input. My question was, what if I used song lyrics as input to the AI, can I make perfect music synchronized videos automatically with the push of a button? Let me know how you think the AI did in this visual interpretation of the song.
After getting caught up in the excitement around DALL·E2 (latest and greatest AI system, it's INSANE), I searched for any way I could use similar image generation for music synchronization. Since DALL·E2 is not available to the public yet, my search led me to VQGAN + CLIP (Vector Quantized Generative Adversarial Network and Contrastive Language–Image Pre-training), before settling more specifically on Disco Diffusion V5.2 Turbo. If you don't know what any of these words or acronyms mean, don't worry, I was just as confused when I first started learning about this technology. I believe we're reaching a turning point where entire industries are about to shift in reaction to this new process (which is essentially magic!).
Important note:
While this AI is impressive, it still required additional input beyond just the song lyrics to achieve the music video I was looking for. For example, I added keyframes for camera motion throughout the generated world. These keyframes were manually synchronized to the beat by me. I also specified changes to the art style at different moments of the song. Since many of the lyrics are quite non-specific, even a human illustrator would have a hard time making visual representations. To make the lyrics more digestible by the AI, I sometimes modified the phrase to be more coherent, such as specifying a setting or atmosphere.
This was my first time working with DDV5, and I'm very happy with the results! There were many times where my jaw dropped upon seeing what the AI came up with. I haven't felt this sense of wonder from technology since I first experienced a HD videogame as a child.
If you would like to learn more about how this video was made, try this yourself, or ask me any questions, I'll post a more detailed explanation of how to get started on Patreon (link below). The post is free to the public, no need to pay. If you do want to support me and become a member that would be much appreciated, you'll also automatically be entered into the end screen minigames where you earn points on each video and move up the leaderboard!
Join on Patreon to automatically have your name included in the next video: https://www.patreon.com/doodlechaos
Want to add lyrics and color beat blocks to your Disco Diffusion project like I did in my video? Here is my code: https://www.patreon.com/posts/67249569
My social media:
Twitter: https://twitter.com/doodlechaos
Discord: https://discord.com/invite/7FCrWAzDY7
Tiktok: https://www.tiktok.com/@doodlechaos
Shorts Channel: https://www.youtube.com/channel/UCMqgJk1o2eWE7WeNtRIfnpg
Instagram Shorts: https://www.instagram.com/doodlechaos_shorts/
Email: contact@doodlechaos.com
While Disco Diffusion is based on the contributions of many, show some love to the two most prominent contributors:
https://twitter.com/somnai_dreams
https://twitter.com/gandamu_ml
Music:
[Indie Dance] - Rezonate - Canvas [Monstercat EP Release] : https://www.youtube.com/watch?v=i0Ew3cl1gyc
Tuning out of starlight The innocent lifetime flickers and flashes Onto a canvas No, nobody seems worth it And while the feeling's sophisticated The view is so restrained The past is nearing closer and closer I fear So let's just cut the bullshit And tell me who's independent Thumbs twiddling and your life's escaping your mind Keep in mind Well the air ain't free as it used to be But that's quite alright when you're still eighteen It seems to be And I'll tell you who's overrated What's a beautif
ul person with some hideous dreams? And while the lining isn't silver We're verging on the side of the road that'll take us home The edge of us is getting weighed Read about your life in your local papers Realize the end of us is getting near You're just a droplet of paint trickled off your canvas Delirious illusions flowing backwards through the river of color While we analyze it to try see it clearer Secretly appealing to a lonely prisoner Now what's your vision? Initiate your shading, blend t
he universe together in time It only gets a little easier Suddenly piece yourself back together now What's your vision? And I don't wanna have to give a damn about you Or your inhibitions After all, you're the one who taught me love Was not for everybody However, I'm starting to see this world in the right light Yet it still gets darker You see, it's just a simple symphony Or a painting Whatever This unnatural path It spans infinitely But my world is crumbling Let it collapse into the abyss In t
he midst of summer it won't exist This emptiness It's just a blank slate Waiting for a painter to interfere It's quite complex But what do I know?
Comments