MetaSound Visualizations and UX | GameSoundCon 2023 | Unreal Engine

STEFAN EVRARD: I'm Stefan. I'm an audio technical artist for Epic Games. And together, we're going to see different ways to visualize MetaSounds and audio in different ways using user experience, using UI, and Niagara. But before I start, let me make sure that-- what is an audio technical artist? And that is an excellent question that I wrote myself. It's pretty much something we made up, because I actually am a technical artist. So normally, I work with automotive brands where I learn to push U

nreal to the maximum limits burning GPUs after GPU, but it looks good. So I learned how to do very complex shaders to match the expectations of clients. After that, I had a brief stint in virtual production. So I kind of learned how to tone things down because everything had to run at 120 frames per second. I learned some Blueprinting on the side. So that's my background. Now, why is this really related to audio? I've always had a phase of-- when I was a teenager, I had a drum and bass phase. Th

e phase is still alive today. So I really love music where synthesizers really push to the limit or sound design. So that led me into the sound design environment and backgrounds, but I never did anything professional. It was just kind of a hobby on the side. So when MetaSounds was created, it was the perfect tool for me, because I could combine my visual knowledge of Unreal, like my basic Blueprinting, and then MetaSounds, which is pretty much a cool synthesizer in my head to create interesting

visual components. So I'm pretty much the bridge between the rendering team and the audio team trying to take what the guys are making at MetaSounds, the crazy tools, and make maybe interesting little projects and demos. And then hopefully they'll convince some maybe not-so-audio-inspired people to maybe start getting into audio work. So what we're going to see here is I'm going to-- section one is pretty much the narcissistic phase where I'm just showing, look what I can do. But more seriously

, hopefully this will inspire people to see what you can do alone with just Unreal without going outside of Unreal, how you can create little projects. There's also some R&D that I did for Epic Games, so it's kind of a mix of everything. After that, we're going to see some UI designs that I made. I already presented some last time, but this time I'll be a bit more in depth. After that, we're going to see Niagara and MetaSounds. So for those who don't know, Niagara is the particle system in Unrea

l. It's insanely powerful. I'm by far not an expert because it's just so deep, but the fact that it has a built-in audio frequency analyzer makes it insane because it means that you can connect pretty much any value you want from your audio to any particle value. But we'll go into details later on. And then the oscilloscope experiments is pretty much still Niagara, but a bit more in-depth. And then finally, I'm going to show how I combine everything I do together with a little UI experiment and

interface, and then some Q&A if there are any. So let me just show you some demos I made in the past year. And again, these are kind of prototypes, work-in-progress. So yeah, enjoy. [MUSIC PLAYING] All right. Cool, cool, cool. Thank you. [APPLAUSE] Thank you, appreciate it. And as you can see, there's really no limit to creativity. So it's really fun to create an interface that's completely unusual, and then just throw it away to someone and see what they make with it. But to get in a more serio

us interface, obviously, we still need to have some reasonable interface from time to time, so that's what I did. I did a bunch of prototypes for Epic Games, which is infinitely scalable. So if I was lazy or-- well, not lazy. The traditional way to create UI would maybe use Photoshop or Illustrator, create some sliders, some knobs and whatnots, and use it as a texture, but that would be too easy and too simple, right? So why not do everything using only maths, because that way you can infinitely

scale them up or down. You'll never see texture compression and GPU doesn't use anything. So some things were pretty straightforward. I definitely had to go way back in my mind and remember some basic trigonometry from high school-- because it's been a while since I used that-- for the curves and so on. So I made a little compilation of all the knobs and things I made. The Blueprinting is still proof of concept, so it's still a prototype. It's not built-in, but we would like to build in Unreal

so you can create your own presets or your own little things. Kind of like what Aaron said, if you saw his talk-- sometimes you have two VSTs that it's the same sound, but one has better UI. Maybe you'll be more inclined to move around, play around with it. Maybe it will even sound better in your head, even though it's the same sound, it's just better UI. So that's why UI is probably quite important. So again, all of these 100% mathematics. There's no texture involved. [MUSIC PLAYING] There we g

o, so. Thanks a lot. Thank you. [APPLAUSE] One of the things that is the most difficult to me is to stay within the bounds, so UI work was actually quite hard for me mentally to keep things regular. So thankfully, the next part is a bit more insane. Niagara, as I said, is a very powerful particle system in Unreal. So yeah, I never used it before. And actually, MetaSounds kind of allowed me to investigate Niagara a lot more. And it's a bit scary at first, so I'll go in detail how I created some o

f those Niagara representations from scratch. That way hopefully you guys can see it's not that crazy and hopefully maybe it'll inspire you to go in that direction, maybe create your own little Winamp simulation or something like that. So in the next video, the first video is pretty much me the first time I used Niagara. I'm just applying audio effects to random particle attributes, so it's a bit messy. And then after that, I made a 3D cascade frequency analyzer, which again, sounds very complic

ated to do, but I was surprised how easy it is because the great guys, the programmers at Epic, they made like an insane tool. And then the third video will be kind of a real-time reactive Niagara system that reacts to actually audio coming from your computer. So you could actually use audio from Spotify or VLC or whatever you want and Niagara would react to. So that's a bit more-- it's pretty much the same principle, because you actually use the mic system in Unreal Engine. The only difference

is you have to set up your own computer to reroute the output to the mic. So that's kind of the annoying part, but otherwise, it works pretty well. [MUSIC PLAYING] All right, which is-- thank you again. [APPLAUSE] I have a lot of videos, but I appreciate all the support. Yeah, it's also great for these demos because I have so many unreleased music. I just basically use them for demos, which is pretty cool. So this is the more complicated part. I'm going to explain how I pretty much made this par

t of the cascade. It won't be as advanced as this version, but it will show you how I can set it up from scratch. So first of all, let's create a new Niagara system. There's an empty system right here, or you can go New System from Selected Emitters, which is all the presets. And then you have a preset called Empty, which doesn't make that much sense, but actually, this version of Empty has a lot of setup done for you, because otherwise, this talk is going to be way too long. So I'm really just

going to call this Demo, very original, and open this up. So this, unfortunately, I can't zoom in more, so hopefully, you guys can see at least a tiny bit. So how it works is that you have an emitter, which is this, which is constantly-- right now, it's not spawning anything because it's empty, but it's kind of on the loop. So the first thing that I'm going to set up is to make sure that it's using the GPU. The main difference is that the GPU can handle many, many particles. And once again, I co

me from rendering cars and not caring about burning GPUs, so I'm spawning 100,000 particles with gravity applied to them and everything. So GPUs are a lot better. And then it's just going to ask you to stay in fixed bounds, so that's like this. So it's kind of limited and it's not a crazy particle system that's around the world. Let me see. You have Emitter Updates. So this is how many times the emitter will refresh itself. Right now, it's set to System, so let's set it to Self. That way we can

define ourselves how it works. So what's happening right now is that every second, the loop duration, the emitter starts again. You're probably going to see it right here kind of looping. So what we're going to do is initialize particle. We're going to make sure the particle is one-second long. It can be anything. It can be five and five if you want. It's really just to keep a nice normalized number. So in Emitter Updates, let's go Spawn Particles and Grids. I'll explain why we have to use the g

rid system in a while. But first of all, why when you select that, there's an error? It says da-da-da-da. A lot of text, boring, boring, boring. We have a Fix Issue button, which should be standard in every piece of software I think. It's insane. I love that thing. So sometimes, it's actually faster to create errors, and then just fix, fix, fix instead of sending it out from scratch. But yeah, if you play around, you'll figure that one out. So what I'm going to do here is I just need the grid to

create a line of particles. So that's going to be the base of our analyzer software. I'm using 1,024 samples-- sorry, not samples, particles, which might sound like a lot, but it's actually pretty reasonable, for me anyways. And there we go. We have a beautiful line. Aaron's mouse is super sensitive. I like more of a less sensitive mouse, so sorry for the sudden brutal movements. So we're going to go to initialize particle, and that's where you're going to decide, like we saw before, the lifeti

me of a particle. But let's choose also the size to a uniform size. Let's just put one. Yeah, maybe that's not visible enough. Let's put four. There we go. That's a bit better. So the reason I use the grid is that Unreal knows how large the line of particles is. Because you kind of need to make sure that things communicate with other things. So the audio system needs to know which particle is assigned to which frequency. So one thing I'm going to use instead of padding per cell in the grid locat

ion system is I'm just going to use bounding box size. So that way, you can define the size of your analyzer pretty easily, because if it's padding per cell, it's going to be the distance between each particle, which is a bit annoying to calculate, so this is much easier. And when you modify this value, Unreal updates it itself. But imagine there's an invisible box wrapping this system right here and it knows that the particles to the left is technically particles 0 and the particle to the right

is 1, and then it interpolates between each other because it's normalized. And that's why I use this system. But again, it's so powerful, you could literally use anything else. This is just how I like to work. So in here, I'm going to create a new existing parameter. So what this does is it allows you to get information from the Niagara system, and then store it somewhere so you can reuse it. So this is the boring part. It's not super entertaining. But unfortunately, we have to go through it to

get to the good part. So yeah, let's create a new float, a new value. And the important part is to rename that float. I'm going to call it uv_x. It can be anything you want. uv, if you don't know what it, is pretty much the same as x and y in a graph. And unfortunately, in 3D space for some reason, z is upwards whereas in mathematics it's like-- yeah. So don't ask me. I'm not responsible for that. But yeah, let's go float from vector, if I write it down correctly. Float from vector, where is it

? Probably need to write vector. I can't be too lazy either. So again, that's why I created the grid is I just want to make sure that I'm extracting the x value of the grid to store it into our uv_x. So right here, I'm going to search for grids, and that's why I use the grid system is that Niagara will create the grid uvw, which is xyz pretty much. So now, you're saying, OK, now we have a new value. It's stored there and it's defined by the size of the grid we're going to use. Now, here's the fu

n part. It's called the Scratchpad Module. I don't know why it's called that, but this is pretty much where you'll get all the information from all the particles, apply anything you want to them, and then store them back. So that's where you're going to have the audio-related stuff. So finally, some audio-related content. You have an audio oscilloscope, a player, and a spectrum. Now, we're not going to look at the player that much, but pretty much imagine that you can play an audio file when the

particle does something. So let's say when a particle finishes its animation, it plays a sound for fireworks or something like that. We're going to use the spectrum though. I'm not going to rename it because I'm lazy, and then just go to Audio Spectrum. And that's why we created the grids before is that we have normalized position in spectrum. So we need to tell Unreal which particle applies to which frequency range. So when you go to the parameters, you can go to all the parameters we created.

And if I search for uv_x, It's right there, the thing you created. It's just an easy shortcut to say this value, I need to reuse it later on. So create it like I did it before, and then now you can just plug it in easily. Oops. Like I said, the mouse is very, very sensitive. So we're going to take the particle position because what I want is to move each particle upwards depending on the volume of each frequency. So let's write position correctly. That's probably better. It's still not-- so we

have particle position. What we want to do is we want to break it. Again, sorry for the terrible writing. The reason is that we want to make sure that wherever we spawn our particles, we don't go completely out of range. So all I'm doing here is I'm going to plug the position back into position. And we need to make sure something that when you start with Niagara is that you might use this position. And it's actually local position, it's not particle position. So that's something you need to bear

in mind, otherwise it won't work. So if you put particle position, that way you know it's the same thing. So you have break position to split them up, and then you have make position. So what I'm going to do here is plug the x back into the x, the y back into the y-- because again, if you use maths, it's going to be a bit confusing, but z is upwards, as you can see right here. So we could plug the amplitude right into the z directly, or another part of the scratch cards is to create your own in

puts. So if I put float and write intensity, and then I can multiply this with intensity, plug this in here, and that should be working right away. So when I go back to this setup, when you see the scratch module, right now, you have new parameters. And this is the insanely cool thing is that you have a built-in frequency range. So you could isolate just the bass, and then assign that to a particle system, then assign the mid-range to another particle system-- maybe one like the bass will change

the color, maybe the mid-range will change the size. It's really up to you. You can go absolutely nuts with that. And the resolution is pretty much as it says. So if the analyzer is set to 16, you're going to have super low res and a range, and that's why I did 1,024. Again, I push everything to the maximum, so the slider was maximum there. So when I spawn the particles in the early stages, I spawned 1,024 particles simply because each particle will now be assigned to a right frequency range. A

nd in the intensity-- that's right, not 1,000, 100. And this can react to any submix you send it to, which is quite useful if you want to isolate a certain part of your audio. When it's off by default, it reacts to any audio within Unreal. So it's quite cool, because you can just go into your Content Browser and play a sound wave. [MUSIC PLAYING] There you go. [MUSIC PLAYING] So this is cool and all, but it's a bit dull. So what we're going to do is we're going to copy and paste this one. Most i

mportantly is to rename an emitter, something I keep forgetting to do, but I'm going to call this Free-- just Freak, whatever. That's fine. And then here, we're going to delete most of the stuff we did because what I want to do is something that's also very, very cool-- let's delete that, let's delete that, let's delete that-- is you can spawn particles from another emitter. So what I'm going to do is I'm going to spawn the particles. Again, you have Fix Issue, which is awesome, just Fix Issue.

And then there's another issue here, Fix Issue. It's set up correctly. It's awesome. All we have to do is write the emitter name that you want to spawn it from. So let's write this. And let's say we want the particles to last a bit longer. So let's say we want the particles to be five seconds long. Let's go back to initialize particle. Like I said, it's a bit complex at first, but you realize it's the same steps over and over again. Now, the cool thing I'm going to add here is velocity, because

I want the particles to spawn at the right positions-- and right now, it's spawning in a weird way-- to create the fake histogram. So here, let's say I'm going to put 100 and this one to 0. Actually, I think I want it the other way around, 200. And here, we have a multiplier, which is quite nifty. There we go. One thing I don't like is by default, it kind of tries to understand the interpolation, which is not what I need. So thankfully, you have different options. Here, you have Sequential. I'm

going to use Random. Much better. And so now, if I play audio-- [MUSIC PLAYING] You have a histogram. [MUSIC PLAYING] And I mean, once you have that, you can just go wild. Obviously, now it's y, so it's maybe not very interesting. So you can actually colorize. You can scale the color depending on the volume of frequencies. You can scale the sizes, like I said. So let's go to Scale Color. This is just stuff that you probably see on most Niagara tutorials. But what you can do is you can scale the

color-- So right now its white. So you pick a color, and then you pick another color, and you will see that it will kind of interpolate it over time. So it's going to start like super white, and then this color. And then when it fades out here, it's kind of a bit darker. I think you can actually change the preview scene setting, which is quite useful, because I don't want to see the environment. That's much better. Yeah, there we go. Much, much better. So let me play some audio again, and I'll a

dd some different things and you'll see how I can kind of interpolate. That's a weird bug. Very interesting. [MUSIC PLAYING] All right. So sorry for starting and stopping. One thing you'll probably see in any single Niagara tutorial is the Curl Noise Force. It just looks cool. This is kind of the stupid thing to do because you want the analyzer to be very rigid, and we don't care about that. It just has to look cool, so let's add that one. There we go. Sorry about the flashing thing. I don't kno

w why it's happening. Yeah, sorry. We're going to have to keep that. There must be a new bug. All right, so the Curl Noise Force, you don't see it right now, but it's kind of going to dissipate the particles depending on the velocity. So if I put 1,000 here, you're going to see it kind of dissolves, which looks pretty cool. You can change the size of the noise. And again, these values, you can change them depending on the audio. So you could change the amounts of dissipation that happens on the

general audio volume or whichever you want. You can just play around with settings and it looks pretty cool. Maybe let's put 100 here. And I'm going to play it one last time just to show what I did for the video you saw just before. [MUSIC PLAYING] Right, you get the idea. You could technically spawn particles from this emitter if you want for some reason if you're crazy. Already here it's pretty reasonable. It could look better, though. I mean, we only have 50,000 particles. That's not enough,

right? Let's go to 100 more particles. There we go. 100,000 particles, 200,000 300,000, 400,000. OK. The computer is fine, so yeah. OK, this is pretty much the "This is Fine" meme with the dog burning in a house But yeah, I mean, it still runs fine. So this is a pretty beefy CPU, but that's one thing I like about Niagara is you can just play around with it till it works. So you could just literally-- actually, since we have a bit more time, I'm just going to show you how to set up an oscilloscop

e from the same principle. Wow, that's wild. OK, cool. So if you want to turn this-- like, let's say your client says, OK, but can you do an oscilloscope? Like, all smug, you can just say, yes, I can, oscilloscope. And it's exactly the same setup. Just take the oscilloscope. You just plug it to the sample audio buffer, plug the uv_x into here, plug this back into the multiplication, apply scratch, compile, go back here, play audio. [MUSIC PLAYING] That's an oscilloscope. Let me scale it a bit mo

re. [MUSIC PLAYING] And there we go. So pretty easy. And that's pretty much like everything I did in the demos. You'll see that I kind of like to have this little dust effect that goes around. It's the same thing. It's spawning emitters from another emitter. So one interesting thing that maybe some oscilloscope nerds know about is if you plug the left channel as the x-axis and the right channel as the y-axis, you can create shapes. So let me show you a video that I muted, because it's just two s

ine waves. Here, you have a sine wave on the left channel, one on the right. And you can see how the frequencies interact where you can create some basic shapes and basic curves. This is just mathematical stuff, but I'm not smart enough to explain why it does those shapes, but it just looks cool. And now, you can kind of visualize what I do. It's pretty much the same thing all the time. The Curl Noise Force, spawn emitter from another emitter, and there we go. So I could have just called it a da

y and say, OK, that works. That's nice. But then I realized that there are some people who actually make music with oscilloscope shapes. And there are plug-ins where you can convert images or 3D objects into audio. So I was like, OK, why limit myself to just basic shapes? As cool as they look, why not actually create a sound to represent the Unreal logo? So I didn't even stop there. I went a bit further, and then I started making music. And I kind of figured out it's probably the nichest music i

n the world because you need an oscilloscope to see what's happening, but yeah. So you'll see a couple of tests I did with this principle. [MUSIC PLAYING] Let me just save your ears a tiny bit. [DISCORDANT NOTES] [MUSIC PLAYING] SPEAKER: Woo! STEFAN EVRARD: So-- [APPLAUSE] Thank you. I truly don't know when to stop, so I actually have to limit myself. But I appreciate it. Yeah, once you get it to work, it's pretty fun. It's probably, like I said, the nichest way to make music. Because you have t

his audio file that you absolutely cannot change. It is interesting to see how audio effects changes the logo. Like, ring modulations are a pretty interesting one. A bitcrusher kind of turns it into this low-poly version of the logo. But any drum you add will kind of destroy the logo itself. So you kind of have to figure out what music you're going to make. But I don't know many people who make oscilloscope art music. I know there's one guy who's super famous, but I haven't seen really many othe

r people. But the fact that I could figure out how to do this without being much of a programmer just showcases how fun it is to use Unreal, because you have all the tools readily available to you. And so this is kind of the final part where I bring everything together. Unfortunately, right now, there's no way to put a Niagara system in the widgets, which is pretty much the UI system. So I'm using a very stupid method where the particle system is actually in 3D space. And I have a camera taking

a picture of the particle system every frame, and then saving that into a material. So it's not the most optimized thing, but like I said, I'm not an optimization guy. So yeah, cool. So actually, let me full screen this because this is much better. There we go. I think it's Alt P. There we go. So this is the particle system that now you recognize, no mystery behind. So I created all this interface using the UI that I made using maths, and then using the widget system in Unreal. And again, the Bl

ueprint thing is kind of just to showcase what you can do. I'm pretty sure if programmers saw what I did with the Blueprints, they would probably hate me, but if it works, it works. So what I did here is I took my xy slider, and using MetaSounds-- so that's really everything together-- using MetaSounds, I kind of added a filter that you can kind of change. I'm going to play music just in a while. And yeah, totally I'm going to play with it, and you'll kind of figure out what everything does. [MU

SIC PLAYING] And there we go. You get the main idea. Thank you so much. [APPLAUSE]