Coding Adventure: Ray Tracing

[Music] hello everyone welcome to another episode of coding Adventures today I'd like to experiment with retracing now I have actually done some super simple retracing before but it wasn't the main focus of the video so I rarely didn't even scratch the surface later on I also played around a bunch with the related idea of Ray matching so I have at least a vague idea of what's in store for us today but I'd still like to start out with a bit of research so I've been reading through this thrilling

Ray tracing in one weekend Trilogy accompanied by a great series of Vlog posts titled casual Shader toy path tracing and now with a bunch of new ideas floating around my head I'm excited to get started I'm going to be building this inside of a game engine just for the convenience of having this little scene view set up for us where we can place objects and move them around and so on what's also nice about this is that we'll be able to easily compare how things look out of the box with this built

-in renderer versus our fancy retrace renders [Music] for the comparison to work though we'll need to make sure that we render things with the same perspective as this game camera here so I've written some code quickly to make sure I understand how the camera is set up and that just draws this grid of points on the projection plane so if we change the field of view for example we can see our points updating correctly along with it and of course if we move the camera around or rotate it [Music] [

Applause] [Music] or rotate it we can see that it works perfectly as well so here's the code for this we start by calculating the height of the projection plane using the field of view and some trigonometry and the width follows from the aspect ratio of the display using that information we then calculate a grid of points and each of those points is transformed to respective position and orientation of the camera what we can now do is calculate the direction towards each of these points from the

camera and let's just draw some little arrows to visualize that [Music] so these represent the Rays that we're going to be shooting out into the world in order to figure out what we should be able to see this might seem a little Vector front since of course we know that in reality light is emitted from sources like the sun then bounces around of various objects and whatever makes it into the camera or our eyes is what's visible to us problematically though the vast majority of that light will m

iss our eyes leaking this approach wildly inefficient so that's why we're going to do things ancient Greek style and shoot the light beams out of our eyes instead [Music] only the Rays that actually end up at a light source can contribute to the image but it's still a lot less wasteful than what reality is doing all right now we're going to want to send out millions of these arrays which sounds like a job for the GPU so I've written this Ray tracing manager script and all it really does is send

that camera information we figured out to a Shader that we're going to be writing in a moment and then it runs that Shader and draws the results to the screen so all that's left to do is write the Shader whatever we put in here is going to be drawn to every pixel on the screen so for example if we return the color red we get a red screen one vital piece of information we have though is the coordinate of the current pixel so if you return the x coordinate we get a gradient from zero on the left E

dge to one on the right Edge and if we return the y-coordinate instead then obviously the gradient will be vertical okay I've added some code now that uses those coordinates along with the camera info we supplied to just calculate the position of the current pixel on the camera's projection plane and I'm doing it with a matrix multiplication this time but it's essentially the same thing we did earlier then I've also added a ray structure up here which just holds An Origin point and a Direction s

o the origin will be the camera position which is given to the Shader by default and then the direction is just like we calculated before let's then quickly visualize this direction as a color just to make sure that it looks correct so I'll enable the Shader in the scene View and now we're expecting to see XYZ correspond to RGB so if we look along the positive X direction we should see things getting more red which they do then if we look up along the y-axis we should see things becoming more gr

een and we already saw that looking forward along the z-axis we have a lot of blue we can also look behind us and we should see this one octane here where all three axes are negative is just a black void everything checks out so let's take our next baby step and try drawing these spheres like most 3D Graphics these are made up of triangles which will definitely be tackling later on but for such a simple shape we can be a lot more accurate and efficient with appear mathematical representation so

the equation for a sphere centered at zero is x squared plus y squared plus Z squared equals the radius squared to figure out if array intersects the sphere then we can say that we started the ray origin and move along the ray direction for some distance when the squared length of that Vector equals the squared radius of the sphere we've hit the sphere solving for distance we end up with this equation and if the part under the square root is negative then the equation has no real solution meanin

g the Ray has missed the sphere if it's zero however that means the ray touches the sphere at just one point and if it's positive that means that there are two points with array intersects the sphere so in the code I've created this hit info structure to tell us first of all whether the ray did actually hit something and if so the distance position and normal Vector of the hit for anyone unfamiliar the normal Vector just means if you were standing on the surface what direction would be up which

isn't to be helpful when we try to figure out how the Rays should bounce off the surface anyway here is the sphere intersection code which computes all of that and by the way to handle seers with different positions we can conceptually keep the sphere centered at zero and just move the ray away instead okay let's quickly edit our Shader to check if the array hits a sphere at the center of the scene with a radius of one and return white or black depending on that then we can go see what that look

s like and it seems pretty spherical so for our next step I'd like to handle multiple spheres and have them automatically match up with the ones we've placed in the scene View so I've created two new structures there's a ray tracing material which currently just describes the color of an object but I'm sure we'll find more things to put in there later and there's also a sphere structure which just holds a position radius and material this information about the Spheres in the scene will get sent

over to the Shader and end up in this buffer over here which is then used by this little re-collision function to figure out which sphere the ray will actually hit all this does is Loop over every sphere and perform our intersection test then if it hits the sphere it checks to see if this is the closest thing that's been hit so far and keeps track of the hit information if it is along with now the material of the object as well so we can then replace that single crochs tweeted here with our new

array Collision function and return the color of whatever it hits okay let's head back to the scene and we should now be able to see all three spheres in their correct places I'm also going to try setting some colors quickly just to make sure that that works and let's also look around a bit because I want to make sure that nearby things are appearing properly in front of distant things and that does seem to be behaving so our next goal is to take these Bland balls and make them more interesting

looking with Sumatra lighting to begin with let's think about what we want to happen when a ray of light hits a surface obviously that's going to depend on the type of material if it's really smooth for example the ray might bounce off it like this with the same angle going out as it had coming in which is called a specular reflection but if the material is more rough the incoming light could potentially scatter out in any direction which is known as a diffuse reflection let's focus on that kind

for now so when already hits a surface we wanted to just bounce off completely randomly I've started making this little test Shader to just figure out how we're going to handle random numbers and so far I've just calculated this pixel index which is a value from 0 up to the total number of pixels in the display so if we visualize that on this tiny display we can see that each pixel has a unique value which we can use essentially as a seed to generate pseudo-random numbers for that pixel so let'

s make a function called random value which takes in this initial seed and transforms it in some way I'll need to look up a good way to do this but let's just play around with it first for fun so how about we multiply the seed by some large number and then maybe Square the result I really haven't thought this through at all but I'm thinking that these huge values will overflow the 32-bit integer and hopefully the remainders will be kind of all over the place and appear random to try it out let's

divide the result by around 4.2 billion which is the maximum unsigned integer value so it'll scale the result down between 0 and 1 and then we can just output our pseudo-random numbers let's see how it looks okay well that's not very convincing we got a funky looking pattern at least though so I'll quickly try multiplying by some different values because I'm curious to see what are the kinds of patterns we can make all right I've been randomly trying some different things behind the scenes and

my latest attempt was to multiply the seed by itself a couple of times but with some random offsets added and the output from that actually does look reasonably random I'd say so now to generate multiple values per pixel we can just add this in outer argument type here which just means that whatever change we make to the seed will be applied to the original value over here so each time we call it we should get a different result and we can visualize that with a random color okay I've had fun pla

ying around with this but let's get back on track and I'll replace a little experimental function with some code I found online which has had a lot more thought and testing put into it all right with random numbers out of the way we need to figure out random directions which is the same thing as generating random points on the surface of a sphere the approach I'm using at the moment is super simple it just generates three random values scaled between negative one and positive one so it's a point

inside for Cube and it just keeps generating these cue points until we get one that is inside of a sphere which should be around half the time on average and then we can just normalize that to get our random Direction unfortunately whenever there's a wild true Loop like this I seem to find a way to get stuck inside of it usually this just causes some crashes and random black rectangles on my screen but today I guess it just stopped bothering to clear the screen between frames which was an inter

esting effect anyway after forcing the computer to restart things were back to normal and I tracked down the bug and also added a limit to the number of Loops just to be safe I was curious then though to see just how bad it would look if we actually removed the loops and just use the first random point we generated so I tested that with 50 000 random points and as you can see they do Clump up quite a lot near where the corners and edges of the cube would be so let's not do that I then wanted to

test another approach I've come across though which is instead of using uniform random numbers where every value has an equal chance of appearing we use random numbers in a normal distribution which looks like this here's some code I found for doing that and now if we use the values from there to generate our random Point instead then for maths reasons I haven't yet grasp done account of not being a good probabilist but this year is apparently the reason our points will be distributed uniformly

randomly over the surface of the sphere on my computer at least this approach was quite a bit faster for the GPU to confuse and I'm sure there are even faster methods out there but let's go with this for now so with random directions working we need to consider now that when array hits a surface we're actually going to want to pick a random Direction in this hemisphere oriented around the normal Vector of the surface so imagine we generate a random Direction and it's rudely pointing outside of t

he desired hemisphere a simple way we can detect and fix this is to just take the dot product between the normal vector and the random Direction and that will be negative if there are more than 90 degrees apart so if it's negative we can invert the random Direction and just like that we're back in the right hemisphere so here's a tiny function for doing that and here's a quick little test I made just to verify that this really does do what we're expecting it to do okay I think we're finally read

y to integrate all the stuff we've been working on into our Ray Tracer so I've started by creating this little function which takes in our array coming from the camera and traces its path as it bounces randomly through the scene and hopefully reaches the light source how this works is super simple we just look to see what the ray will collide with using that function from earlier and if it hits something then we move the ray to be positioned at the point of intersection and pick a random directi

on for the ray to bounce off in that then keeps repeating until either the maximum bounce limit is reached or the ray misses everything and shoots off into empty space in which case we can break out of the loop early now we still need to figure out how much light will actually be received by the camera from all this so let's create a variable called the Ray color which starts out as pure white then whenever the ray hits something we can recover its material and simply multiply the ray color by t

he object color so as an example in the real world if a particular object absorbs all visible wavelengths of light then it will appear black and similarly if we've defined that the object should appear black in its material then the red color will be multiplied by zero which is like all the light has been absorbed okay so the red color keeps track of how the incoming light will be affected by the objects it interacts with but we don't know from the outset what light source is if any actually lie

along the path of this Ray so let's create another variable to keep track of the light that we encounter and that just starts out at zero then when we hit an object now let's first calculate how much light that object is emitting using these emission color and strength properties that I've added to the material most objects don't emit light so this will typically be zero but if the object is a light source we can add that emitted light to the total incoming light and tinted based on the color o

f all the objects the Ray has encountered so far at the end of all of this we can just return whatever light we've accumulated along the path this would be more intuitive if we weren't doing everything backwards but hopefully that will make sense anyway the core of our Shader now looks like this we calculate a seed for generating random numbers then we create array looking out from the camera and finally we trace the path of that Ray through the scene to figure out the color of the pixel okay so

to test this I've set up a super simple scene here with just a bunch of little balls sitting on top of one very big ball all right let me grab our settings window here quickly so we can enable retracing and look at that a photorealistic rendering of a room without any light but okay let me try select one of these balls in the dark and I'm going to duplicate it and then open up its material properties because I want to turn it into a light source so let's set the color to Black because we don't

need it reflecting any light only emitting so let's then raise the emission strength here and we can start to make out some pixels of our scene at least I'll move this growing old far away and scale it up and maybe a bit more so that it's kind of like a sun Alright good night everyone and back to work at the moment over here I have the bounce limit set to one so the ray from the camera has to hit an object and then bounce directly into a light in order for that pixel to be illuminated but let's

try raising the limiter 2 and we can see the back sides of these spheres are now able to receive some indirect illumination each additional bounce is going to have less and less effect though so for example I'll crank this up all the way to 30 and we can see the difference is pretty subtle the main problem we have now is that our image is extremely noisy which is something I'd like to address but unfortunately my seat has been stolen the issue though is that we're only sending out a single Ray p

er pixel but really we need to be sending out as many as possible letting them all bounce around on their own random parts and then averaging their contributions to get a better estimate of how much light is really reaching the camera so I've modified our Shader to just Loop a bunch of times and add up all the incoming light calculated by the trace function then at the end it just outputs the average so let's take our single rate per pixel and dial that all the way up to 100. that's suddenly loo

king a lot better than just a single array but the image is still quite noisy of course we could set the number of rays even higher but if we take a quick look at the performance of this it's already struggling to reach even 60 frames per second which is a bit pitiful for just a handful of spheres for today at least though I am not aiming to have this all running in real time I just don't want to slow down the editor too much because that's not very pleasant to work with so let's keep the number

of rays per pixel low but to average the result over multiple frames so that the quality gets better with time that's as simple as taking whatever was displayed on the previous frame and the new render and combining them like this obviously we want the Rays to bounce differently on each frame then so we'll also need to modify our random number seed so that it changes with each frame and I don't know a particularly good way of doing this so I'm just going to add the current frame number to the p

ixel index which multiply it by some big number and hope for the best [Music] let's try it out and okay clearly we can't have objects moving every frame now or it turns into a smudgy mess but conceptually it seems to be working at least I'll test it properly in a second but I just quickly want to modify our Trace function so that when array misses everything instead of just going out into the void it gathers some kind of background environment lighting so I've made this simple function that just

Blends between a ground color and two sky colors based on the vertical axis of the ray Direction and it also adds in a little circle for the sun based on a given light Direction here's what it looks like in the scene so really nothing fancy but it's a bit nicer than the nothingness we had before I think we can change the sun Direction like this by the way and at some point I'd like to integrate the atmosphere rendering from one of my previous videos so that we could get sunsets and so on but th

is will have to do for now okay let's try rendering this little scene and we can see how over time the image becomes less noisy I'm not quite happy with how it looks yet though so let's think a bit more about what happens when our Ray hits the surface currently the rays are bouncing off in random directions to see if they reach a light source or if we consider this from reality's perspective rather light is coming in potentially from all directions and we're trying to estimate how much of that i

s going to randomly bounce towards the camera but here's something we haven't considered yet imagine a beam of light coming straight down onto the surface from above we can see that that light covers this little portion of the surface over here now imagine the same beam of light Birch coming in at an angle to the surface clearly that team is going to be more spread out meaning the overall light received by any one point on the surface will be weaker and so it should contribute less to the lighti

ng of that pixel so let's go back into our Trace function and we can calculate the strength of the light as the cosine of the angle between the Surface normal and the direction towards the light then let's multiply the recolor By the Light strength and overall this is going to have the amount of light that's received so just as a quick hack to keep the brightness the same for comparison's sake I'll multiply this by two alright so here's a render of the balls before this cosine rating and here's

what it looks like now which I think is a lot nicer let's also quickly see how it looks with the built-in renderer which obviously is not a fair comparison because not only was this rendered in a fraction of the time but also there's a bunch of stuff we could do to improve it the point though is to just help us appreciate some of the things our raytracer is doing which we might otherwise take for granted for example I really like this bit of extra Darkness we're getting beneath the Spheres which

really makes them look like they're physically touching the ground or another nice detail is this area of red light that we can faintly see bouncing off the white ball here anyway I quickly want to go back to our Trace function because through some maths that I don't understand it turns out that we can remove this light strength calculation we just added if we instead make a small tweak to how we calculate the random bounce directions like so this change means that instead of the bounce directi

ons being uniformly distributed over the hemisphere we have what's called a cosine weighted distribution this will send few arrays in directions that contribute less light and consequently more rays in directions that contribute more light so in the end this gives us the same result as when we're applying that light strength calculation we just get there a bit faster alright I reckon we've spent enough time looking at these balls so let's move on to triangles code for calculating the intersectio

n of array with a triangle and now prepare to Feast your eyes on this beauteous creation [Music] okay maybe it doesn't look all that exciting but of course from The Humble triangle we can build pretty much whatever we want so what should we make well I love the shape of this little chess horsey so I've been doing my best to model a low poly version of it in blender this is made from just 456 triangles but even that's probably going to be quite taxing on our little Ray Tracer we'll have to see fi

rst though I've added this buffer to the Shader for holding all the triangles in the scene and then there's also a buffer to store information about each mesh which mainly just tells us when the big triangle list a particular mesh begins and how many triangles it contains finally I've added this bit of code to the recollision function which is essentially identical to how we handled the Spheres simply testing if the ray hits any of the triangles and storing the information about the closest one

if it does before looking at all the triangles though it does quickly test if the ray intersects the object's bounding box just to optimize things a tiny bit at least all right I'm going to set up a simple test scene here quickly something along the lines of the classic Cornell box so basically just a little room with walls of different colors then let's bring in the night and I'll give it a plain white color so we can see how it interacts with the colors of the room unfortunately I can already

tell that the editor is lagging quite a lot from my inefficient retracing code so let's just drop the number of rays all the way down to one pathixel I'll need to try optimize this at some point which let's just put up with the noise for now to finish this off I'm going to add in a front wall so the room is completely enclosed and I'll use a quad mesh for this instead of a cube just so that it's invisible from the back side allowing us to still see into the room of course the room is now complet

ely dark so that's not much use and I guess we might as well turn off the sun now but then let's duplicate the ceiling scale it down a little bit and use that as our light source instead okay let's let this render for a bit and we can see our Majestic night gradually emerging from the noise you can also see some nice bounce lighting with this red tint along the one side here and the blue is shading on the other side and even some green under here and along the base coming from the floor just so

interesting here's what it looks like if we only allow the light to bounce once then here's with two bounces and here's back to our original image with 10. to try this out with another model quickly so I'm going to bring in Suzanne the blender monkey to pose for us this is excruciatingly laggy though so I'll disable our retracer while we set up the scene here and then let's see how it looks that's looking pretty good but I know what you're thinking what if Suzanne were to enter the Avatar stage

and light the room with her glowing eyeballs well there you go behind the scenes I've been trying to optimize the rendering a bit and I actually did manage to speed things up dramatically the only downside is that Suzanne now looks like this I'm a bit tone because it really is a lot faster but I think it means after revert my changes anyway let's move on to specular Reflections which remember is when we have a very smooth surface like a mirror so light bounces off it perfectly with the same angl

e going out as it had coming in this perfect reflection can be calculated based on the normal vector and the incoming Direction like this so in our Trace function we're calculating the diffuse Direction over here and instead of always using that for the new redirection let's now also calculate the specular direction using this built-in reflect function then the raised New Direction can blend from the diffuse direction to the specular Direction based on the smoothness property that I've added to

the material so let's try it out I faced a few spheres in our little test chamber here and let's just see how that looks at the moment all right now let's make these balls shiny so I'll set the smoothness on the first one to around 0.4 then the next one may be 0.6 then 0.8 and the last one can go all the way up to one so here's what we had before and now with our new specular reflections as we'd expect this goes from very blurry Reflections on the left since it has a low smoothness value to a ni

ce crisp mirror on the right I'd like to play with this a little bit more so I've set up a similar scene here this time with just two big balls though and I'm going to make them both fully reflective since they're opposite one another we can see the other ball reflected in this ball and that reflection of the other ball of course contains a reflection of this ball and so on so let's render that and I've animated the balls to move forwards until they're just touching one another giving us some ni

ce infinite reflections [Music] I want to try one last reflection test with our little horse here quickly here's what the scene looks like at the moment which is pretty dull so let's spice it up by making this wall here be a mirror all right let's then make the wall opposite it reflective as well and I'll give it a smoothness of just less than one so that things get slightly blurrier with each reflection which I think looks nice but why stop there let's make the red wool reflective and the blue

wall too next the ceiling and the floor finally we might as well make the horse reflective too truly one of the artworks of all time anyway what's going on with this tomato the tomato is red but these specular highlights we see on edge are white or rather they're the color of the light that's hitting it and bouncing specularly into the camera but in our Ray Tracer if we take a red object and make it shiny we can see that the specular highlights are tinted red which seems intuitive so what's goin

g on with the tomato well it turns out that fruits have a transparent protective coating and when light hits that some of it will go through and be scattered diffusely but some amount of light will also bounce off the smooth coating giving us those untinted Reflections and the same idea applies to many other materials such as varnished wood for example so in our Trace function let's figure out if the current Ray should be bouncing off some imaginary coating by taking this specular probability th

at I've added to the material and testing if it's greater than a random value between 0 and 1. then let's just make sure that the diffuse Direction gets used if this is not a specular bounce and we can also use this information to select between either the materials regular color or its specular color which we'll probably just leave white most of the time to see what this looks like I'm going to take these four bolts from earlier and make them all totally smooth but we're now able to say that th

at smoothness should only apply some of the time so I've set up some probabilities and let's see how that turns out I'd say that's looking pretty good but for the real test let's try it out on our tomato ball at the moment it looks like some strange kind of metal I guess but now using our new settings we are able to make something like this which definitely feels a lot more tomatoey okay now there are still so many things I want to experiment with but let's end off today with just one last effec

t which is depth of field the basic idea here is that we're able to choose a distance at which things appear in focus and outside of that distance things will be blurry so let's go all the way back to the beginning where we were calculating the direction of the Rays for each pixel and just focus in for a moment on a single Pixel we are actually now sending out multiple Rays per pixel but they're all starting from the same position and going in the same direction so it just looks like a single ar

ray to make things blurry then all we need to do is make the Rays diverge a little bit so they're all heading out in slightly different directions so over in the code where we're sending out all the Rays for the current pixel let's grab the ray setup code and move it inside the loop so that we can make each one slightly different which we can do simply by randomly nudging this viewpoint to do that I want to generate a random Point inside of a circle so here's a little function that begins by cal

culating a random angle and then converts that to a point on the unit circle using sine and cosine finally that gets scaled down by the square root of a random value to be somewhere inside the circle the square root might seem a little mysterious at first but without that we can see that the random points cluster towards the center of the circle so the square root just ensures that they spread out nicely anyway let's now take that random vector and multiply it by the strength setting I've added

then using that we can make a new Viewpoint that lies somewhere inside a circle around the original Viewpoint and let's plug that into our Direction calculation here okay that is not working at all apparently I'm just making up variables that don't exist at this point so after Consulting The Matrix documentation because I always forget the Syntax for accessing the different rows and columns I've extracted the missing camera directions that we used for offsetting the Viewpoint so the code should

actually compile now alright I have set up this little test scene and let's try increasing the size of that random circle using this string setting here that does seem to be making things blurry but it's also extremely noisy as usual so let's render it quickly of course this does make everything blurry and we can't focus on anything so we'll need to make some adjustments but it's a stat at least and actually if we turn this off quickly something that it could already help us with is these jagged

y edges our image has which is simply due to the limited number of pixels in the screen so if we just set the blur strength extremely low that'll help to smooth out these harsh edges giving us some nice anti-aliasing here's a little render I made to test this and I'd say the edges are looking pretty smooth [Music] okay so we can make things blurry if we want but we still need to figure out how to focus at a particular distance so what if instead of randomly offsetting this Viewpoint we randomly

offset the ray origin let it give us something like this where there is now all Converge on the viewpoint so if we have some model and we bring that over here the Rays will be hitting it at a bunch of different spots giving a blurry result for the current pixel but if the model moves a bit further back then all the Rays will be hitting the same point and so it'll be in focus of course the Rays don't just stop at this plane they extend out potentially forever and so if the model were to move even

further back past this focal plane I guess we could call it now then it will start getting blurry again so to control what's in Focus here we can simply adjust the distance of the plane back in the code let's keep our old approach since it's helpful for anti-aliasing and we can just calculate another random Vector using our point in circle function again and then use that to offset the ray origin that's actually all we need to do I think because changing the focus distance is already handled by

the Z component of these view parameters so to test this I've set up a slightly larger scene here and let's just play around a bit [Music] to begin with I'll try setting the defocus strength to 2 which doesn't seem to have done much so let's try 20 or maybe even 200 okay that's made everything super blurry so let's play with the focus distance now I'll turn on this visualization I added behind the scenes and let's just try shifting This Plane all the way back to the same depth as the Knight as

we'd expect the Knight seems to be pretty sharp whereas these Pawns in the foreground here are still very out of focus I'd like to see this with a bit less noise so I'm going to temporarily crank the number of rays up from 3 to 30 and my computer is dying but I think it looks nice okay I'll turn that back down again and let's try bringing the foreground into Focus now and let's also see how that looks with some more rays this seems to be working quite well so I'll just do one last render to test

it here we have the king and queen in focus and now I'll animate the focus plane shifting out to highlight the Knight all right there's still so many things I want to do for example rendering objects made from glass adding volumetric effects and of course improving the performance so we can experiment with slightly more complex scenes but we've covered a lot of ground already so let's leave it here for today and perhaps tackle those topics in the future okay I hope you enjoyed the video and unt

il next time cheers [Music]