Top 5 Wireshark tricks to troubleshoot SLOW networks

There was a bit of a delay. There was a bit of a lag that was experienced by a user just going out to the internet. Now that user complains. Of course, it's the network, the network slow, all the things. This problem, it wasn't super huge. It's not like the person was sitting there waiting with a spinny wheel for minutes and minutes and minutes. But this example is a snapshot of something that you may see out there in the real world. And I'm going to show you my workflow of how to find it. That'

s brilliant. You got to be able. You have to do this for yourself. Yeah. No amount of watching these videos out there, everybody. If we can just have a real sidebar. You've got to take Wireshark, open up these captures and dig through them on your own. If you're just watching this video, you're getting something from it. But those of you that downloaded that PCAP and are examining it, if you hit up my channel and do this, if you if you check out Wireshark and how to configure the screen and thes

e delta times, if you're doing that work, it's going to help you get the most out of it. And this is going to become a part of your thought process. OK, so you need Wi-Fi. Are you going to live dangerously and connect to a free Wi-Fi access point? You need to be really careful connecting to free Wi-Fi because you don't know if that's actually a legitimate Wi-Fi access point or a rogue Wi-Fi access point. I personally would be very careful just connecting to free Wi-Fi or public Wi-Fi out there.

If you do need to connect to public Wi-Fi, I personally would only do that. If I'm running a VPN and my VPN of choice is ProtonVPN. I've been using ProtonVPN for a long time. I started using it based on the recommendations that are found in books such as these and the discussions and interviews I've had with both hackers as well as cybersecurity experts. ProtonVPN, ProtonMail have been highly recommended by people that I've spoken to in the cybersecurity space. The moral of the story is, do you

trust the Wi-Fi that you're connecting to? Because hackers can very easily set up a rogue access points. So in my case, I would never use a public Wi-Fi connection unless I really had to. And if I do need to use it, I would make sure that I have my VPN enabled and running. And the VPN that I would use once again is ProtonVPN. Big shout out to Proton for sponsoring this video. Hey everyone is David Bombal back with the amazing Chris Greer. Chris, welcome. It's great to be back, David. Thanks for

having me again. You're the guy I always talk to about packets and you have the saying and I don't want to take it from you, but something about packets that don't lie or packets don't lie, something like that. Yeah, you're going to see that around the packet world. The packet sphere is people say packets don't lie and there's a good reason for that. And that's what I want to talk to you about. Looking forward to this. So for everyone who hasn't watched our previous videos, I've linked a whole b

unch below. Chris is the person I always talk to when I really want to see what's going on on the wire, if you like, or in the air, if you're using Wi-Fi. Chris, just for people who haven't seen our previous videos, tell us a bit about yourself. You do this for a living, right? You help companies troubleshoot their networks, help with threat hunting, that kind of thing, right? Yes, that's exactly what I do. So I'm a packet head. And that's why you see this shirt on me, packet head, because I enj

oy looking at packets and trying to find the answer to problems. Now, one thing that got me into this space actually goes into that phrase, packets don't lie. I used to be a network engineer. I was running around the network trying to find the answers to mysterious problems. And especially when I was getting blamed about the ever so common phrase, what is it, David? It's always the network's fault, right? That's it. The network's slow, the network this, the network that, the network's dropping,

the network, all these things. So as a network engineer, I found myself trying to troubleshoot these problems and I would go to the end of my experience, end of my skill level, and still wouldn't be able to find the root cause to some of these mysterious issues. Now, especially the ones that were intermittent. Sometimes it was the network, we were able to resolve it, but there was always these pesky, weird problems. So what I found is that there was these ones that I would work with some of my c

olleagues that developed the art of packet analysis. They were able to come in with Wireshark or Ethereal at the time, or another type of packet analysis tool, capture some traffic, get down to the wire, find out what was really happening on the wire. And then they were able to find root cause. I was always blown away. Like, wow, that is amazing that they could capture this traffic and it always looks so crazy. It looks like you're walking into like an airline cockpit. There's so much there. You

're like, what's important? What what should I be paying attention to? What does that button do? What does that button do? That's how I felt when I got started. But then over time, as I started to understand traffic and how to leverage it as a very important data source, I too was able to see how to use it to resolve network problems or to better understand cybersecurity incidents. So again, for everyone who's watching, we can only cover so much on my channel. But Chris, you have over 100,000 su

bs on YouTube now and you go into crazy amount of depth on your channel. So for everyone who's watching, please go and subscribe to Chris's channel. Show the love. But Chris, you also have courses and a whole bunch of other things, right? Yes, I do. In fact, getting back to how I got here and we've covered this on your channel and also I talk a bit about it on mine is in addition to learning the art of packet analysis, something I really wanted to do was help others learn it too. Yeah. I don't f

orget what it felt like when I first got started with all of this. And I hopefully communicate that in my content that you're going to see on my YouTube channel. If you go check that out or any of my Udemy courses or any of the material that we've collabed on David. But it's really important for me to try to help others get up to speed and how to use packet analysis. And that's because I think it's such an important data set. Back to our original thing that we said, packets don't lie. So I know

a lot of you out there, the viewers, you're out there trying to troubleshoot mysterious things, or you're trying to better understand how this whole networking thing works. Or if you're on the security side, what does an attack really look like on the wire? Yeah. What does an IDS, IPS system look for? Or, Hey, you're studying for a cybersecurity certification or a networking certification. And these questions keep coming up about TCP flows or what does this little value do within a wire shark tr

ace? Or if you see a packet trace come up, you have to be able to read and understand it. So that's really been a goal of mine to help as many people as possible learn as much as possible about packet analysis and doing that with wire shark. I love it. But Chris, you've got a demo, right? Because I want to actually see you do it. And for everyone who's watching, I hope you enjoy the demo. But again, Chris has a whole bunch of these kind of demos on his channel. So go and sub. But Chris, take it

away. Show us, you know, your, I'm hoping it's an example of, you know, why packets don't lie. Yeah, exactly. And that's, let's get to it. Let's get to an actual demo. And for me, I don't just like showing you what I see in this packet trace. I want you to be able to follow along with me. So I'm going to go ahead and show you where you can download this PCAP. You can get the link down in the description down below. You can download it off of my GitHub and you can follow right along. So let's do

that. So the link in the description down below is going to take you to my GitHub and you can go ahead and download this lab one GreerBombal ItsNotTheNetwork.pcapng. So all I got to do is just come over here and we're going to hit the little three dots and we can just download that content or pull it down any way you like. While you're there, you can peruse through, you can see some other different files that I use on my channel. You'll notice if you go and then you check out some of the content

of my channel, I'm often coming back here and showing you where to download the example. So let's have everybody do that. Glad you created a GitHub that makes it much easier. Oh man, so much easier. So David, as I walk through this PCAP, I want to show you the top five things that I look for to pinpoint problems in a PCAP. Wow. Say that five times fast. We followed Fong to the defunct futon factory on fifth. Say that five times fast. But there's five things that I usually look for as a part of

my workflow. And I want to break them down to you one at a time. So the first, I examine the TCP handshake. So let's go ahead and start off with that. So, okay. So download Wireshark, open up that file that we're sharing with you. And now let's get to it. Now, why are we talking about this specific file? First, there was a bit of a delay. There was a bit of a lag that was experienced by a user just going out to the internet. Now that user complains. Of course, it's the network, the network slow,

all the things. And I just want to pick out a few things that I look for when I'm troubleshooting a problem like this. David, there's a lot of different things that can go wrong where an end user can experience performance problems. There's not any one thing that I can tell you right now. Oh, it's always this. Cause that's not true. There's a lot of things that it could be. But what I want to show you is my workflow and how to get to the biggest causer of delay. So does that sound interesting?

That's great. And I just want to confirm this is a real world example, right? Yes, it is. Now this problem, it wasn't super huge. It's not like the person was sitting there waiting with a spinny wheel for minutes and minutes and minutes. But this example is a snapshot of something that you may see out there in the real world. And I'm going to show you my workflow of how to find it. That's brilliant. Okay. So first thing that I do, and you've seen this, if you ever stopped by my channel and absol

utely, if you've seen me here with David, my friend, David, as we've dug through different PCAPs and that is we got to capture the TCP handshake and understand a bit more about this flow from that. So let's start there. So first packet, client, clients initiate, sends, okay. Clients start connections usually. So I can see 10.0.2.15. In fact, I don't really want to look at that as an IP and this server over here as an IP either. I want to show you a little trick that I use sometimes. And I've bee

n using it a lot more lately, so I'm kind of in the habit now. I'm going to come up here to Wireshark preferences. And if you're following along on Windows or Linux, then you're going to find this, uh, this interface. If you go over to edits and down here at the bottom, that's where you're going to find preferences in Wireshark. And there's just one little preference that I'm going to change. If I come down here to name resolution and by default, resolve network IP addresses is unchecked. I'm go

ing to go ahead and check that. Okay. So let's go ahead and activate that. And for those of you that are watching, if you have any of these other guys checked down here, let's go ahead and uncheck those just for now. Okay. Just for the purposes of this walkthrough. This is where we start using DNS and this is where Wireshark can actively go and resolve those names using active DNS or resolve them for DNS traffic that's contained within the trace file. Don't worry about that right now. I just wan

t to click resolve network IP addresses. If you look like mine, then we're good. Going to say, okay, I'm going to come back here to my PCAP and I'm going to come to this first one, first packet, going to right click it. And you see edit resolved name. Okay. I'm going to click that and I'm just going to type in, there's this little toolbar that appears name. I was going to call it client and hit. Okay. That's nice. So you see now instead of that IP address, now I see the word client going to do t

he same thing on the other side, destination, edit resolved name. Now I just right click that, that IP up there and I'm just going to type in server. Okay. So now I've got client server, client server, client server, a little nicer when I'm dealing with IPs that aren't on my environment or new things that I'm seeing and just to get my head around it, especially useful data when I'm troubleshooting with IPv6 and I have these huge massive IP addresses and I don't want to see all that. I just want

to do the client servers. So that's, that's how you can do that. Now this is going to be sticky for this PCAP until I either come back up here. One way that I just like to remove this and this is just my workflow. I can go to edit resolved name and I can just remove the name client and hit enter and my IPs come back, or I can go uncheck that checkbox that we activated in the name preference and then that will let me get my IP addresses back. So anyway, a couple of ways we can do that beyond what

we'll talk about here. So client goes to server 25 milliseconds later. This is my Delta time. Now I'm going to go ahead and link right here. Maybe we're going to have a card show up that links you to a Wireshark setup video that will help you orient yourself and get the same kind of information that on your screen that I see on mine. This is a Delta time column. It shows you the amount of time in between packets. Also, you'll notice that my SYNs are bright green and you can see my TLS handshake

packets or that Aqua color. So these are some settings that you can adjust with Wireshark. We've definitely done that here on this channel before. And there's a link quickly to my channel where I show you how to set that up real, real quickly. All right. So 25 milliseconds. SYN goes out. SYN comes back 25 milliseconds later. That right there, David, is something that I store in my brain. That's my network round trip time. Basically that's latency, latency as perceived when this actual user was

interacting with this application. Some, if I'm a network engineers perspective, I might say, Ooh, you know, I can just bring up a terminal and I can ping the the server and I can measure the network latency. Yeah. But if I do that right now, that tells me the latency now. Yeah. Where this tells me the latency that was then. And it's different traffic. And it's different traffic. This is a TCP port 443. This now, it could alter the amount of delay that I see along the way, depending on packet in

spection and what's going on. And is that good or bad? That's the question, right? Is it the client was complaining that it was slow, the network. That's a good question. So is that good or bad? Right now, let me ask you this. Do you think as a human being, do I worry too much about 25 milliseconds? Right. So right now that's not bad. I'm not worried about that. That's not what they called me. That's not why my customer is, you know, punching their screen because things are so slow and buffering

. And it's also, it's like, um, what's the baseline, right? Cause if you've got fiber, your expectations are much different to someone who's on like a dodgy cell phone network in the middle of nowhere or satellite or something. Right. It's all expectation expectations, but I mean, I can't imagine someone complaining. All that. Yeah. Yeah, exactly. So, uh, that that's the better question right there. Rather than is it bad? Is it normal? If this is an intermittent problem and the problem comes and

goes, do you see 25 milliseconds when there's no problem? Because if you do right there, I can remove that from my troubleshooting. That's not root cause yet. All right, David, we're going to go into the next thing. This is the second thing that I look for in a PCAP when I'm looking for performance problems. First of all, I'm looking at the handshake, but specifically within the handshake, I'm also going to be interested in a couple of things. And that is my TCP options. Okay. So options are th

ings that are only exchanged in the TCP handshake. Once they're exchanged, it's done. They don't happen again. So let's take a look in this handshake. Let's go ahead and go to client to server and everybody can join me. If you come down here to the lower left, this is where we've got our display going here. And if I come down here to options, this is only exchanged in the TCP handshake. So right here, my maximum segment size, my SACK permitted and timestamps and window scale. So there's four opt

ions that the client is basically saying, Hey server, these are the parameters that I would like to work with in this TCP conversation. So now, Hey server, what can you do on your side? Now the reason why we do this, David, just to go into TCP, a little bit of history. So TCP is pretty old protocol, or at least the original TCP when it came out. I don't know. Can you guess when the original TCP RFC came out? I can't remember. It's eighties or something, isn't it? Well, seventies, I can't remembe

r. It was a really long time ago. So would you believe that RFC 793 came out in 1981? Now I'm not going to tell you how old I was. I don't know how old you were in 1981. I'm sure some of the audience weren't around just yet, but so, so TCP has been around a while. So over time, what's happened is one of the things that's been adjusted is these options. It's allowed us to extend the lifetime of TCP to get more out of it, to make it more efficient for today's networks. Now right there, I think tha

t's pretty amazing that it's been around for 42 years or longer, depending on when you're seeing this, but TCP really has staying power in a reliable connection oriented protocol that does the job well. That's why it's still around. However, it's got some weaknesses that these options help us to overcome. So when two clients connect, that's one of the first things that they do is they establish the nature of the TCP relationship. What are we going to do here? So the client is giving these option

s to the server. So do this with me, David, check out these options. Maximum segment size. This is the client saying, Hey, don't send me anything larger than 1460 as a payload. That's my maximum segment size. That does not count the TCP header, the IP header, the ethernet frame, just the payload. Now, next selective acknowledgement is permitted. Thumbs up. Great timestamps. Don't worry about it. Not going to talk about it, but window scale. Now what window scale does is that allows me to take th

is window size here and greatly increase it because this window size value right here is just a two byte value. You can see over here on the right. When I click on window over here on the right, these two bytes that are highlighted, that feel is only two bytes long. The maximum number that it can be in decimal is 65,535. What that means is basically, Hey David, if you and I are sending data to each other, you can only have a maximum of 65,535 at once in flight on its way to me. Which was fine in

the eighties, but not now. A hundred percent. Back in 1981, no big deal. But now you're in the UK, I'm in the United States. Let's just say we had a gig link across the Atlantic Ocean with 150 milliseconds of latency. 65K is nothing. Yeah. We would be, you would have to send me data so many times and experience that network latency that things would take forever to transfer. So for that reason, we saw TCP evolve, if you will, to this window scale option. So now what I'm doing is I'm saying, Hey

David, when you see my window size, multiply it by 128. So that results in the calculated window size, which Wireshark does for us. This number here is the real calculated window size. That's an internal value. Wireshark is just doing me a solid here. It's just helping me to know what that number truly is. Okay. So I send those in flight, right? Server comes back. Let's notice what the server has. Now, what do you notice that's different, everybody? If you go to that second packet, server comes

back and I see some options. Or rather, I see one option. Server's saying, okay, great. 1460. I can also accept maximum segment sizes of 1460, but what do we not see here? Oh, that multiply by 128, right? That's it. It's not using a window scale. It's not using selective acknowledgement. Timestamps, I don't really care about right now, but those other very important options. In fact, Wireshark's kind of complaining about it. The SYN packet does not contain a selective acknowledgement permitted

option. Ooh, no good. So this means that I'm talking to a device. This is one thing that's strange here. I'm talking to a device or a middleware box or something is coming back with this. This SYN/ACK doesn't have a window scale or selective acknowledgement. So that's kind of interesting to me for one. So this client, what that means is this client and the server for that matter, when both sides don't support an option like window scaling, then neither side gets to use it. So right now the clien

t is saying, oh man, David, I gave you a multiplier of 128. You're stuck in the 80s, David. You're stuck in the 80s, David. Fine. We'll use 80s language here. Now that might mess me up because I'm not optimized for 80s TCP. So you're making me go back to the 80s. Fine. I'll do that. But we can't use this window scale. We can't use selective acknowledgement. Now selective acknowledgement, that's a whole nother conversation. We can link to another video that talks about selective acknowledgement o

r maybe a fuel for a future video, David. That's a whole big topic. That'd be good. In fact, if you'd like to learn more about TCP selective acknowledgement, please comment down below. I like that. You've got me a real YouTuber, Chris. Trying, trying. All right. Hang around with you long enough. That might happen. So SYN goes out, options. SYN/ACK comes back. Only one option. So right there, think about this, David, how, how many packets am I into this product, this PCAP? Yeah, two, one each way

. I'm two, I'm two in and I've already found something that looks weird. Now, how do I know that it looks weird? Because done this a lot. I've seen a lot of handshakes. That's why packet analysis, really just real talk. If we can have a sidebar for a second. Sure. You gotta be able, you have to do this for yourself. Yeah. No, no amount of watching these videos out there, everybody. If we can just have a real sidebar, you've got to take Wireshark, open up these captures and dig through them on yo

ur own. If you're just watching this video, you're getting something from it. But those of you that downloaded that PCAP and are examining it, if you hit up my channel and do this, if you, if you check out Wireshark and how to configure the screen and these Delta times, if you're doing that work, it's going to help you get the most out of it. And this is going to become a part of your thought process. Right. Okay. So now that we've looked at the handshake, I'm going to take you to the third thin

g that I look for when I'm examining slow things, when I'm looking for slowness. Now by its nature of even saying slow, I'm looking for where time is being spent. Where is time going? Now, right now I'm just on a single thread conversation, but if we're not looking at a single conversation, that's where I just want to say right click conversation filter. If this was a real world PCAP, I'd want to do conversation filter TCP to just focus in on only this conversation. All right. Now this is just a

single 64 packets on a single conversation. So that filter that I just said did nothing. But for those of you out in the real world, when you're looking at a conversation, you got to set a conversation filter for what I'm about to show you. All right. So the next thing that I like to do, where is time being spent? That's why a Delta time column is so important. I'm going to come up here and literally, I'm just going to sort the Delta time. What that does is it's going to show me where's the max

imum amount of time between any two packets in this whole thread. Where is the weight coming from? So if I click it once, it's going to put the fastest times at the top of the list. If I click it again, now I'm going to, I'm going to hit these green buttons up here. You see these green left, right, jump to specific packet, jump to top, jump to bottom, let's jump up to the top. This is going to take me to packet number 51. And here on packet 51, I've got 808 milliseconds between this packet and t

he one previous to it. And this was a window update. Interesting. So, Ooh, I also see another indicator here. TCP ZeroWindow. All right. Let's go ahead and take a closer look. So I sorted Delta time to just find the slowest packet here in the, the peak cap or rather the, the delay, the biggest delay in the peak cap. So what I'm going to do now is I'm just going to put this right back in order. And here I can see packet 51 is where the biggest cause of delay is for the whole trace file compared t

o 25 milliseconds. David, let me ask you this. 25 milliseconds. Does a standard human worry about that much time? No. How about 808 milliseconds? Is that closer to what a human might feel? Yeah. You're going to notice that for sure. Yeah. And you'll especially notice it if you suffer it many times. Yeah, that's a good point. Yeah. One packet, perhaps not so much, but if it happens a lot, right. Yeah. And so, and a lot of times, you know, when, when data is moving, if I'm downloading a file, I'm

uploading a file, I'm interacting with an application. There's a lot of transactions that are happening. So it's not uncommon for an end user to experience that type of delay many, many, many times. Yeah. That's where delays can turn into many seconds or even minutes. Okay. So let's figure out what's up here. So packet 51 client is going to server with a window update. Notice the packet previous though. Client is going to server with a TCP ZeroWindow. Now these are the ones that I just love Davi

d, because ZeroWindow, 100% they're blaming the network. They're totally blaming the network and being slow. This problem has absolutely nothing to do with network latency. You could scrap your entire network, replace it with full fiber, 10 gig, end point to end point, and it would do absolutely nothing to resolve this issue. You've had that. I think you told me in a previous video, some client wanted to spend lots and lots of money, or you saved them spending lots and lots of money or something

. Cause they were moaning it was slow and then they wanted to rip it out or something. Right? Yes. And a lot of times they'll first spend that lots and lots of money, and then it's still a problem. And then that's when they'll call me and I'll say, Hey, can I have some of that other lots and lots of money? Usually that's already spent in the budget, but anyway, just buy training. Let's buy training instead of a lot of equipment. But what does it mean? Like ZeroWindow? I'm hoping you can explain

how you got to that conclusion. How would us normal people know that that means it's not a network problem? Yeah, let's dig into that. That's going into the fourth thing that I look for. So the delay was the third thing, but let's go to the number four. And number four is these types of TCP indicators. So where is Wireshark if Wireshark is giving me these kinds of clues. So ZeroWindow, window full. So then I can start honing in on some of those alerts that Wireshark's actually trying to help me

out with. It's like black and red, right? So that gives you a good indication there's a problem. Is that what you mean? Absolutely. Like look here. In fact, over here in my intelligent scroll bar, this is like this micro scroll bar over here, you can see the coloring rules. And this is one reason why I like to change the colors, the default colors to something a bit more bright as I can see up on top, that's the beginning of my conversation. There's my bright green up there. I see a couple of bl

ue lines. That's my TCP or I'm sorry, my TLS handshake beginning. Then stuff happens. And then I run into trouble down here with these black lines and the very end I can see, and then there were some resets at the very end. Okay. Now with a glance, you can get used to even doing troubleshooting, just, just looking at that intelligent scroll bar. Okay. So what is going on here? Window full, ZeroWindow. So first of all, let me back up a couple of packets. Now, one other thing that I'd like to talk

to you about is that, Dave, you know, at the beginning, we were talking about how packets don't lie. Yeah. Well, I hate to tell you that sometimes they do. You're an immigrant. And we see that. I know, I know, right? Hey, it's not my fault. I'm just the messenger. All right. So it's not that I just want to get ahead of right here. You can see these larger packets. Now, let me ask you, David, as a network guide, does this look normal? Can you have these huge packet? It looks way too big, right?

Can you have packets that are this size? Yeah. In fact, what's usually the largest one we can have typically? Or what range? So, I mean, is it typically Ethernet you're referring to like 1500? Yeah, that's exactly it. You know, that's, that's often what you'll see as a, as a maximum around 15, 15, 18, technically, if you have the Ethernet frame, or if you have a VLAN tag or something like that, that'll extend it. Or if you're in a data center environment where you have jumbos, not talking about

any of those things, clearly 20,000 bytes in the single packet breaks rules. Yeah. Just can't have packets that size. So what we're actually seeing is what the packet driver thought it saw when things were either being transmitted or upon receipt. This is either transmit reassembly. Like stitching it together or something. Sorry, Don. That's exactly it. It's either, so sometimes what happens when we're capturing on the device that is involved in the conversation, what can happen is our packets g

et captured before the NIC card chops them up. Okay. That's what happens. So it's TCP segmentation offloading and we capture the traffic before that happens. Or we can even receive a block of data and it gets reassembled and that's when it hits the packet driver. So if you ever look like you're capturing these packets that are just really large, it's very likely this is just a symptom of segment, segmentation offloading or receive reassembly. So the point is if we captured on the wire itself, if

we install a tap in the network or captured off of a span port, we would actually see a bunch of small packets. Okay. Okay. So getting back to where we were supposed to be with our windows, I don't want, sorry, I keep going on these detailed tangents, David. No, it's good. It's good. Cause I mean, the problem is when, you know, Chris, this is real world versus a course, right? Or we measure the traffic. So I'd rather you just give it to us like the real world stuff. So go for it. Yeah. And mayb

e I'll ask that of the audience. If you don't mind, if I can, sometimes I go on those detailed tangents and sometimes people complain, get to the point, get to the point, Chris. And sometimes people really like it. So you can just let me know what you think down in the comments. Okay. So let's go ahead and take a look at the data coming in from the server right now. So here's a block of data coming in. Client turns around and you notice on packet 43, if you click it packet 42 is being act a litt

le visual check mark there. So client is ACKing packet 42 more stuff comes in from the server. More stuff comes in from the server client acts, then more stuff comes in from the server client acts. We start to run into trouble. So let's go ahead and do this. TCP window full. So what this means is that this packet is filling the receive window on the opposite side. So everybody follow me here. Let's go to packet number 46. I'm going to come down to my details and I'm going to come down to calcula

ted window size or regular window. It doesn't matter. There's no window scale. So we can go to the regular window, right click, apply as column. What that does is it shows me the TCP window size, the receive buffer on the client. So now check this out. Let's go up a little bit. I'm going to come up here to packet 28. This is on the client. All right. So server sends data or clients sending something server acts clients has something server acts. Okay. Now server starts to send data. Okay. Now ch

eck out the, the window size 61, 320. This is the client to server. So the window size went down a little bit. It used to be 65, 535. Now it's 61, 320. What that means is data is pooling in that receive window. It means that this is all the, this is the amount of space I have left to receive data. Yeah. Server says, great, here's more data. Thank you. But check out our window size. It just dropped. Yeah. Server still sending stuff. Client window size dropping server still sending stuff, client d

ropping, sending stuff, client dropping. You see 39 K 26 K. And then finally we get down here. Server's still sending stuff because it still has some room. Come down to packet 46. Everybody I'm telling the server. I've only got 11 6 80 left in my TCP receive buffer. The server says fine. Here's 11 6 80. It exactly fills the window. That's why, and by the way, everybody, here's one of my little tangents. You probably are looking at a length field in your copy of Wireshark. That length includes al

l the headers. What you see on mine is the TCP segment length. So to get that, all you got to do is just come up here to, where are you? TCP segment length. And what you can do is just take that value, drag it is just, just above the sequence number, drag it and drop it upstairs. And you'll get the same length that you see, or you'll see the same amount of data, you see the same data point. Now I've got it twice now, so I'm just going to remove this column, but you can see why I like that instea

d of the full frame length, because that's the amount of payload that is in a packet rather than all the extra header information. So client says, okay, 11 6 80. Here's 11 6 80. Now that fills the receive window of the receiver client. 29 20. So it actually digested a couple of packets because we're dealing with 14 60s. So to two 14 60s would be 29 20. So the client digested a couple of packets. Hey server, you can send me 29 20. Server sends 29 20. Here's a window full. Now client goes, oh, I'm

full. I'm congested. Stop sending. TCP ZeroWindow is, Hey server, stop. I'm dealing with something on my side. I can't, I can't bring in this data as fast as you're sending it right now. The server has to wait. Server can't do anything. It'd be like, if I have a glass of water that's completely full and I say, Hey David, go ahead and send me some water. David, you say your glass is full. I can't send you anything. It's going to spill out. It's interesting though, because you said, you said orig

inally the, the problem looked like in packet two that it was on the server side, but here it's actually the client that's got a buffer that's full. Yes. And this is where it's a bit of a combo problem, right? So the client's getting congested, but the client is also restricted to this non-scaled window because of the server, because of the server. Yeah. So there's a few things in play here. So first I'll finish this part. So a ZeroWindow. So we go down to ZeroWindow. That's where we wait. We wa

it almost that full second. Yeah. And that's where we see our TCP window update happen. The, the, the client comes back and says, all right, server, I'm back up to 62 780. Keep resuming your data transfer. You can keep sending servers like about time. Here's more stuff. And then the connection shuts. So of this, I mean, up to this point right here, this is the amount of, um, total packets since the beginning, uh, of the traffic that has occurred up to this point. We're only 1.333 seconds into th

is PCAP. Yeah. And if I'm going to do a little something, if I go just before these TCP window folds, and if I take a look at packet 48, in fact, I'm going to come up to 46. If I right click 46, I just say set time reference. So from 46 to the end, when we actually get a window update and we start moving data again, 894, so nine, almost 900 milliseconds. So, uh, if I just remove that time reference. So of this 1.3 seconds, the majority of it is waiting for the client to clear its window and to c

ontinue to resume data. Now it would be really easy to beat up on the client here. So now let's move to the fifth thing that I do when I see things like this is I like getting to root cause I like figuring out, Hey, wait a second. How do I actually fix this? It's one thing to capture it. One thing to capture. It's another thing to understand. And then it's another to come to move forward and start to address. That's why you paid the big bucks. Chris. No, just why did you have anyone wants to pay

me big bucks? I'm here for it. So there's two things in play, right? One, the client windows filling. That means the client is becoming congested. But here's the thing. If I look at these other indicators up here, the beginning, the client has a pretty resilient TCP window stack. Uh, it wants to do, or it has a pretty resilient TCP stack. It wants to use larger packets, SACK permitted, window scale. So this server is basically clipping our wings, right? It's, uh, not giving us the ability to us

e more normal, more modern TCP implementations. So back to what you said, this server is forcing us back to the eighties. The client is not good at eighties TCP. Oh, that's a good, I like that. Yeah. Yeah. That's all. So I would say the client, although it's, it's easy to try to blame it because it's, it's suffering under these circumstances. Let's go ahead and blame the one that's putting it there. So now I'm going to focus more on this server. So this is where the boomers keep blaming the Gen

Z, but it's actually the boomers that are the problem in this example. Yes, exactly. That's what we'll say. Oh, it's yeah. I'll let you say that. So the server here would be where I would focus my attention. And this is where I got a few things that I also, some other clues that I can look at here, David, first of all, you see this client, hello, look who I'm going out to this server name indicator connect.facebook.net. It's a Facebook problem, right? Right. So this is just a, just a web crawl,

but it's interesting that here, this, uh, so-called robust, very, very, this should be a very powerful system is coming back with this strange TCP behavior. So something that I might look at here is I'd really want to be interested in. Is there anything in the middle that is proxying me? Is there something that's intercepting this? Is there something that's changing in the middle? So this is where I got one other clue that I haven't gotten to yet. And this is where things can get pretty deep in

the weeds. If I just come up, up here to IP and I just like to peek at the time to live. So right here, I can see time to live is 64. What that means is that this IP part, either there's two things in play, either the time to live is wrong or being reset along the way, or it could be that I'm hitting a proxy of some kind. This packet was basically it's advertising as if it was generated on my network. The reason, because time to lives usually start at 64, 128 or 255. That number, as we cross rou

ters, that number goes down. So this number hasn't been decremented yet. So what I might want to do is just look at my gateway. Am I doing any kind of proxying? Is there something, can I get a capture on the outbound side on the other side of the router? How is this SYN/ACK coming in? Is it coming in with a bunch of options and it's really coming from Facebook and then it's being proxied by some setting here locally on my gateway. Then things are coming in and being adjusted there. So that helps

me get to a bit more weeds in figuring it out. So to just take this away or to put just an absolute stamp on it, if possible, I'd love to have a server-side packet capture, because that would tell me how things are actually leaving the server. Yeah. And if the server is truly letting this go with no options, I'd be like, okay, man, let's get into 2024 here. So let's get in the now. I doubt it for Facebook though. So exactly. Right. So it looks like something along the way is reshaping this SYN/

ACK. So that's what I want to look at. Am I doing anything on my network that would be reshaping this or proxying this? Or am I being unknowingly proxied in my service provider? So David, this was just an example of the packets don't lie. The purpose in showing everybody this one was to talk about how these TCP settings can really change how the application can behave. In this example, these are types of things that I absolutely have seen in the wild. Sometimes clients will come to me and they'l

l have a problem like this. In this exact one though, as I recall, this is a lab environment. This is something I was able to generate using a virtual machine that was going through VirtualBox and connecting out to the internet. So this was a natted device. And let me show you how I know that. If I just come to packet number one, if you take a look at the client IP address, so that's source IP 10.0.2.15. If you Google that, you're going to see that that's a default address that can be assigned t

o a virtual NIC by VirtualBox. Now, so that tells me on the client side, just as a clue, if I start to see something strange like this on the client side, running a virtual machine. Okay. Now I understand some of the resource limitations, right? Maybe this virtual machine was just under resourced. As far as the TCP settings coming in from the server, what I would do now is I would want to go past the point of NAT. I'd want to go external to that virtual environment and figure out, okay, were tho

se settings actually coming from the server or was this SYN/ACK coming back from that server and then that web proxy or whatever was proxying that virtually proxying that information as it came in. Were those TCP settings being stripped? Were they being adjusted before being sent to that client? So I showed you this to be able to see how these little settings can impact application performance. And in this case, it just wasn't the network. This was a virtual setting on a host. So I shared this e

xample just so everybody can see just a clean example of how these settings can impact performance. In this case, yes, it was this client that was running out of resource in that virtual environment, and I'd need to capture further to determine what those server settings were in that ingress SYN/ACK. Yeah, I love it because I mean, it depends how the company set up, right? Because it could be the security team that have mandated a proxy or management have mandated a proxy so that they can stop p

eople going to bad websites or whatever the reason is. That's not necessarily a network engineer issue. It could be a security issue. But I like that, you know, just stop blaming people and just try and fix the problem. That's it. But that's why as network engineers, we need to level up and be able to capture the traffic and be able to understand even things that are outside our domain. So it's not just looking for ARPs or ICMP redirects or routing protocols that are doing some strange update, b

ut capturing and leveraging TCP data and above up to the application layer and being able to find what symptoms and clues reside there to help us to to point to root cause. I think the days of network engineers doing peer networking are long gone. We have to go wider. Yeah, absolutely. And that's something that that's another reason why I'm so passionate about packet analysis. And I want you to be able to do this as well. So I definitely invite the good viewers to check out my YouTube channel. I

have a lot more content like this stuff where you can download the packet capture and follow along. Also, David, you and I have a Udemy course that we collaborated on and that's getting started with Wireshark. So if this was interesting to you, please go check that out. But overall, is just do it for yourself. You got to capture the traffic and work through the exercise of understanding what's happening and then you can begin to develop that artifact analysis to. Chris, thanks so much for shari

ng, you know, you could keep all this knowledge yourself. But thanks for educating people and changing lives. Always David, I love this stuff. Looking forward to coming back again if you'd have me. Of course. So for everyone who's watching, put comments below things that you want Chris and I to cover. Otherwise, go and sub to Chris's channel. Fantastic content there. As always, Chris, thanks. Thanks, David.