WEBVTT 1 00:00.000 --> 00:04.920 Of course, there's more specific issues we need to keep an eye on as well. 2 00:04.920 --> 00:10.460 So for example, we have some common DTLS issues where perhaps the next hop is 3 00:10.460 --> 00:11.520 not reachable. 4 00:11.520 --> 00:13.740 That's of course always going to be a problem. 5 00:13.740 --> 00:17.660 If we can't get to the next hop, then having a route doesn't do us a whole lot 6 00:17.660 --> 00:18.540 of good. 7 00:18.540 --> 00:21.180 But maybe we don't have the route at all. 8 00:21.180 --> 00:26.040 So maybe BGP is down with our service provider or it's not configured properly. 9 00:26.040 --> 00:31.610 We are static default is set incorrectly or maybe we know how to get there, but WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 9 00:26.040 --> 00:31.610 We are static default is set incorrectly or maybe we know how to get there, but 10 00:31.610 --> 00:32.040 the correct 11 00:32.040 --> 00:37.360 DTLS port is not open either on the end device because it's not listening 12 00:37.360 --> 00:39.320 properly or maybe 13 00:39.320 --> 00:43.650 some device in the middle such as a firewall or an IPS system, something like 14 00:43.650 --> 00:44.840 that is intercepting 15 00:44.840 --> 00:48.000 the traffic and not letting it get through. 16 00:48.000 --> 00:51.180 Some good commands we have for verification here. 17 00:51.180 --> 00:53.360 Just good old show IP route. 18 00:53.360 --> 00:58.580 Of course, to make sure that we do in fact have the necessary routes show IP AR 19 00:58.580 --> 00:59.080 P so that 20 00:59.080 --> 01:02.710 we know that we have layer two reachability and that our gateway tracking is WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 20 00:59.080 --> 01:02.710 we know that we have layer two reachability and that our gateway tracking is 21 01:02.710 --> 01:04.600 working properly. 22 01:04.600 --> 01:06.800 And of course, you have the good old traditionals. 23 01:06.800 --> 01:09.480 We can paying a destination on the internet. 24 01:09.480 --> 01:13.720 Of course, eight dot eight dot eight dot eight is one of Google's DNS servers. 25 01:13.720 --> 01:16.640 So that's a rather popular target for testing. 26 01:16.640 --> 01:18.840 We of course also have trace route. 27 01:18.840 --> 01:21.440 We could trace route to an address on the internet. 28 01:21.440 --> 01:26.830 And then for good troubleshooting tools here, we have show SD WAN control 29 01:26.830 --> 01:28.720 connection history. 30 01:28.720 --> 01:32.560 So let's go ahead and take a look at some of these commands as well. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 30 01:28.720 --> 01:32.560 So let's go ahead and take a look at some of these commands as well. 31 01:32.560 --> 01:34.960 So we'll start here with our show IP route. 32 01:34.960 --> 01:40.630 In this case, you can see I'm using a static default pointing to the next top 33 01:40.630 --> 01:41.280 exit. 34 01:41.280 --> 01:46.220 If we do a show IP ARP, we see that we do in fact have that next hop listed 35 01:46.220 --> 01:46.960 here in our 36 01:46.960 --> 01:49.680 ARP table. 37 01:49.680 --> 01:51.920 We can ping eight dot eight dot eight dot eight. 38 01:51.920 --> 01:55.560 So our internet connectivity is working. 39 01:55.560 --> 02:01.240 And of course, we can do a trace to verify the path to that destination. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 39 01:55.560 --> 02:01.240 And of course, we can do a trace to verify the path to that destination. 40 02:01.240 --> 02:04.720 So those are of course, just testing general connectivity. 41 02:04.720 --> 02:10.070 One that's specific to SD WAN here is the show SD WAN control connection 42 02:10.070 --> 02:11.080 history. 43 02:11.080 --> 02:16.880 And if we hit enter here, this gives us a ton of legend to tell us about all 44 02:16.880 --> 02:18.160 the different 45 02:18.160 --> 02:22.720 errors that this can display and what they mean. 46 02:22.720 --> 02:26.820 When we get to the bottom here, of course, we can see that once again, this is 47 02:26.820 --> 02:27.400 a rather 48 02:27.400 --> 02:29.120 wide output. 49 02:29.120 --> 02:31.680 So let's zoom back out again. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 49 02:29.120 --> 02:31.680 So let's zoom back out again. 50 02:31.680 --> 02:35.220 So once again, this is unfortunately going to be a little bit difficult to read 51 02:35.220 --> 02:35.280 . 52 02:35.280 --> 02:39.560 But what we're really looking for is what are the errors. 53 02:39.560 --> 02:44.750 So if we look towards the end of this line, we see things like state local 54 02:44.750 --> 02:46.040 error and remote 55 02:46.040 --> 02:50.160 error and notice that these are listing different errors here. 56 02:50.160 --> 02:55.200 This again is a log of the different errors that have occurred on this device. 57 02:55.200 --> 02:59.540 The errors that I'm showing here in particular are because I had to change the 58 02:59.540 --> 03:00.280 address on WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 58 02:59.540 --> 03:00.280 address on 59 03:00.280 --> 03:02.160 my V bond server. 60 03:02.160 --> 03:06.680 And while that was changed, this device was trying to contact it. 61 03:06.680 --> 03:08.680 And of course, it wasn't working. 62 03:08.680 --> 03:12.250 But overall, this is a good place to look at your errors and then use the 63 03:12.250 --> 03:12.800 legend up 64 03:12.800 --> 03:17.160 at the top to find out what those errors actually are. 65 03:17.160 --> 03:20.550 One of the more common errors that we'll see on this output are the 66 03:20.550 --> 03:21.960 disconnected T-lock 67 03:21.960 --> 03:23.420 errors. 68 03:23.420 --> 03:28.680 Your best troubleshooting for this, you can do a show SD-WAN control 69 03:28.680 --> 03:30.040 connections, which WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 69 03:28.680 --> 03:30.040 connections, which 70 03:30.040 --> 03:33.180 we'll take a look at here in just a moment. 71 03:33.180 --> 03:36.780 If there are issues there, the next step in troubleshooting is probably going 72 03:36.780 --> 03:37.320 to be just 73 03:37.320 --> 03:41.450 to clear the connections to make sure there's not some form of a temporary 74 03:41.450 --> 03:42.120 issue. 75 03:42.120 --> 03:47.560 So it's almost the same command, but it's clear SD-WAN control connections. 76 03:47.560 --> 03:52.080 And if those don't work, two very, very common things to have incorrect. 77 03:52.080 --> 03:56.720 First, try changing the T-lock color because that can often be put in 78 03:56.720 --> 03:58.120 incorrectly and not 79 03:58.120 --> 04:00.160 match up with the other side. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 79 03:58.120 --> 04:00.160 match up with the other side. 80 04:00.160 --> 04:03.120 And the other thing is to change the system IP. 81 04:03.120 --> 04:07.250 It's quite common to make an error on this and have two devices with the same 82 04:07.250 --> 04:07.720 system 83 04:07.720 --> 04:08.720 IP. 84 04:08.720 --> 04:12.160 But let's take a look at the show SD-WAN control connections, what that's 85 04:12.160 --> 04:12.960 supposed to look 86 04:12.960 --> 04:14.280 like. 87 04:14.280 --> 04:17.680 So the command again is show SD-WAN control connections. 88 04:17.680 --> 04:22.540 We're not going to zoom back out because as you can see, this is a rather wide 89 04:22.540 --> 04:23.240 command 90 04:23.240 --> 04:24.240 as well. 91 04:24.240 --> 04:28.920 In fact, to really see it properly, we have to zoom out even one more level. 92 04:28.920 --> 04:32.000 But here we see the different peer type we're talking about. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 92 04:28.920 --> 04:32.000 But here we see the different peer type we're talking about. 93 04:32.000 --> 04:36.640 So in this case, we have a connection to VSmart, VBond, and VManage. 94 04:36.640 --> 04:39.000 All of these are running over DTLS. 95 04:39.000 --> 04:44.330 We see the peer system IP, what site they're in, what domain they're in, their 96 04:44.330 --> 04:44.880 private 97 04:44.880 --> 04:51.910 IP address, their peer private port number, the peer's public IP address, the 98 04:51.910 --> 04:52.920 peer public 99 04:52.920 --> 04:57.280 port number, the organization name, the local color. 100 04:57.280 --> 05:01.400 We can see I have all of mine configured for Biz internet. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 100 04:57.280 --> 05:01.400 We can see I have all of mine configured for Biz internet. 101 05:01.400 --> 05:06.170 And again, we see that the state is currently up and we see how long it's been 102 05:06.170 --> 05:06.880 up for our 103 05:06.880 --> 05:08.480 various connections. 104 05:08.480 --> 05:12.600 Again, this is what it should look like when it's operating properly. 105 05:12.600 --> 05:16.460 So if we see that any of these are down or they're missing, these are things to 106 05:16.460 --> 05:16.760 keep 107 05:16.760 --> 05:18.360 an eye on. 108 05:18.360 --> 05:22.910 Now don't forget that there could be issues with our controllers or the data 109 05:22.910 --> 05:24.000 plane itself. 110 05:24.000 --> 05:26.400 So let's take a look at some of those issues. 111 05:26.400 --> 05:30.040 First, let's talk about the controller issues. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 111 05:26.400 --> 05:30.040 First, let's talk about the controller issues. 112 05:30.040 --> 05:34.640 Probably one of the most common issues there is to have incorrect serial 113 05:34.640 --> 05:35.480 numbers. 114 05:35.480 --> 05:39.980 So we have some commands we can use here to verify that we have the correct 115 05:39.980 --> 05:41.080 serial numbers 116 05:41.080 --> 05:42.600 on our devices. 117 05:42.600 --> 05:46.780 So we have one command here we can use on the validator and then we have two 118 05:46.780 --> 05:47.360 commands 119 05:47.360 --> 05:51.520 we can use here on the managers and controllers. 120 05:51.520 --> 05:54.240 Let's jump over and take a look at these real quick. 121 05:54.240 --> 05:59.060 So starting off here on the validator, we can do a show orchestrator valid v 122 05:59.060 --> 05:59.800 smarts and 123 05:59.800 --> 06:05.040 we can see here the serial numbers of the two v smart controllers. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 123 05:59.800 --> 06:05.040 we can see here the serial numbers of the two v smart controllers. 124 06:05.040 --> 06:09.350 For the manager and the controllers, we had listed here the show control 125 06:09.350 --> 06:11.040 connections history 126 06:11.040 --> 06:16.520 and the show control local properties, both of which we've seen on our edge. 127 06:16.520 --> 06:21.440 We can just use these commands again on manager and the controllers. 128 06:21.440 --> 06:26.390 Let's talk about some data plane issues and one of the problems that can occur 129 06:26.390 --> 06:26.920 here is 130 06:26.920 --> 06:29.760 having discontinuous networks. 131 06:29.760 --> 06:33.920 These are networks that don't actually physically connect to each other. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 131 06:29.760 --> 06:33.920 These are networks that don't actually physically connect to each other. 132 06:33.920 --> 06:38.210 So in other words, we can't have one side of our tunnel being connected to the 133 06:38.210 --> 06:38.880 internet 134 06:38.880 --> 06:43.290 having the other side of the tunnel or what's supposed to be the other side of 135 06:43.290 --> 06:43.960 the tunnel 136 06:43.960 --> 06:49.590 being connected to MPLS because these two services generally don't have reach 137 06:49.590 --> 06:50.240 ability 138 06:50.240 --> 06:51.500 to each other. 139 06:51.500 --> 06:56.650 So there's no way that I can build a tunnel between two different T locks that 140 06:56.650 --> 06:57.020 are in 141 06:57.020 --> 07:01.120 different networks that don't actually connect to each other. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 141 06:57.020 --> 07:01.120 different networks that don't actually connect to each other. 142 07:01.120 --> 07:07.040 We also need to make sure that the necessary IPsec ports are not blocked. 143 07:07.040 --> 07:11.480 Again usually talking here about something in the middle on our path. 144 07:11.480 --> 07:15.560 So again, maybe a firewall or an intrusion prevention system. 145 07:15.560 --> 07:19.830 So we need to make sure that the T lock colors don't mismatch so that it will 146 07:19.830 --> 07:20.640 actually try 147 07:20.640 --> 07:26.480 to form that tunnel assuming that we have strict color conformity enabled. 148 07:26.480 --> 07:32.720 And a great way to verify all of this is your show SD WAN BFD sessions. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 148 07:26.480 --> 07:32.720 And a great way to verify all of this is your show SD WAN BFD sessions. 149 07:32.720 --> 07:36.760 So let's hop over and take a look at this command. 150 07:36.760 --> 07:41.870 So here, although I highly doubt you can actually read it, we've done the show 151 07:41.870 --> 07:44.480 SD WAN BFD sessions. 152 07:44.480 --> 07:48.050 But notice this command is even wider than all of the other ones we've looked 153 07:48.050 --> 07:48.440 at. 154 07:48.440 --> 07:51.520 So we have to zoom out even further. 155 07:51.520 --> 07:58.340 But what we can see are the BFD sessions we have up to our other SD WAN edge 156 07:58.340 --> 07:59.580 devices. 157 07:59.580 --> 08:04.170 In this case, we can see here looking at the destination public IP that these WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 157 07:59.580 --> 08:04.170 In this case, we can see here looking at the destination public IP that these 158 08:04.170 --> 08:05.120 are in fact 159 08:05.120 --> 08:10.480 the tunnels that are going out to our AWS cloud on ramp environment. 160 08:10.480 --> 08:15.840 But we see here what their system IP is, their site ID, their current state is 161 08:15.840 --> 08:16.120 up. 162 08:16.120 --> 08:20.000 We see that all of our T lock colors are set to Biz internet. 163 08:20.000 --> 08:25.110 We see our local source IP and of course that destination IP going to AWS as 164 08:25.110 --> 08:25.680 well as the 165 08:25.680 --> 08:27.360 ports that are being used. 166 08:27.360 --> 08:31.760 The encapsulation for this data plane tunnel is IP sec. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 166 08:27.360 --> 08:31.760 The encapsulation for this data plane tunnel is IP sec. 167 08:31.760 --> 08:35.720 And again, we can see things like the current uptime. 168 08:35.720 --> 08:40.380 So we do have a lot of really good commands here for keeping an eye on the SD W 169 08:40.380 --> 08:42.200 AN underlay. 170 08:42.200 --> 08:47.500 So in this video, we took a look at some of the attributes and tools that we 171 08:47.500 --> 08:48.160 can use 172 08:48.160 --> 08:52.320 when troubleshooting our SD WAN underlay network. 173 08:52.320 --> 08:56.930 Now as we saw and we can still see on the screen here, some of these commands 174 08:56.930 --> 08:57.600 get very, 175 08:57.600 --> 09:01.080 very wide in order to be able to read them properly. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 175 08:57.600 --> 09:01.080 very wide in order to be able to read them properly. 176 09:01.080 --> 09:05.160 Of course, that makes them very difficult to read here in the video, but again, 177 09:05.160 --> 09:05.560 on your 178 09:05.560 --> 09:09.230 live equipment, you should have no trouble reading the output of these commands 179 09:09.230 --> 09:10.440 whatsoever. 180 09:10.440 --> 09:14.000 Next, we need to talk about the overlay. 181 09:14.000 --> 09:19.020 That's of course why we care about the underlay is to build our overlay tunnels 182 09:19.020 --> 09:19.040 . 183 09:19.040 --> 09:23.650 So let's jump in in the next video and take a look at how to troubleshoot the 184 09:23.650 --> 09:24.320 overlay. 185 09:24.320 --> 09:34.320 [BLANK_AUDIO]