WEBVTT 1 00:00.300 --> 00:01.530 All right, so for the next flow, 2 00:01.530 --> 00:06.390 from VM-1 to VM-3 is from here going over here. 3 00:06.390 --> 00:08.400 So from a routing perspective, 4 00:08.400 --> 00:10.680 it's as if these two VMs 5 00:10.680 --> 00:13.770 live on the same logical layer three network which they do 6 00:13.770 --> 00:14.603 and they're also connected 7 00:14.603 --> 00:16.800 to the same layer two broadcast domain, 8 00:16.800 --> 00:19.170 this logical overlay segment, segment 10. 9 00:19.170 --> 00:20.040 And as a result, 10 00:20.040 --> 00:23.640 there shouldn't be any routing decision whatsoever. 11 00:23.640 --> 00:25.920 However what will happen is that when this client 12 00:25.920 --> 00:27.780 tries to communicate with this device, 13 00:27.780 --> 00:31.050 if it doesn't have the layer two address or VM-3 it, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 13 00:27.780 --> 00:31.050 if it doesn't have the layer two address or VM-3 it, 14 00:31.050 --> 00:32.370 it'll do an ARP request, 15 00:32.370 --> 00:37.370 and this host can actually respond on behalf of this VM 16 00:37.620 --> 00:38.670 and not bother sending 17 00:38.670 --> 00:42.780 the actual ARP broadcast request over the network 18 00:42.780 --> 00:44.340 'cause if it did, it'd have to go ahead 19 00:44.340 --> 00:46.680 and use the transport network, and send it across, 20 00:46.680 --> 00:48.270 and that'd be received here, 21 00:48.270 --> 00:50.190 and then deencapsulated just for the ARP request. 22 00:50.190 --> 00:52.290 So if the ARP information is already known, 23 00:52.290 --> 00:54.750 we can save that step by having ESXi-A 24 00:54.750 --> 00:57.060 just respond back to the client with the ARP response 25 00:57.060 --> 00:57.893 and never have to send 26 00:57.893 --> 00:59.520 the initial request across the network. 27 00:59.520 --> 01:00.570 And then after VM-1 WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 27 00:59.520 --> 01:00.570 And then after VM-1 28 01:00.570 --> 01:03.180 has the destination layer two address of VM-3 29 01:03.180 --> 01:06.030 and tries to send traffic to that layer two address, 30 01:06.030 --> 01:07.860 ESXi-A is gonna take a look 31 01:07.860 --> 01:09.240 at that destination layer two address 32 01:09.240 --> 01:10.620 and say okay, where does that live? 33 01:10.620 --> 01:13.080 And then based on the tables that NSX provides, 34 01:13.080 --> 01:15.690 ESXi-A would know that oh, that destination Mac address 35 01:15.690 --> 01:19.200 of VM-3 is reachable through ESXi-B. 36 01:19.200 --> 01:22.470 And fortunately we have a tunnel endpoint at both ends 37 01:22.470 --> 01:23.940 and so at that point, 38 01:23.940 --> 01:27.600 ESXi-A would just encapsulate the data in the Geneve tunnel, 39 01:27.600 --> 01:30.210 ship it over to ESXi-B who would deencapsulate it, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 39 01:27.600 --> 01:30.210 ship it over to ESXi-B who would deencapsulate it, 40 01:30.210 --> 01:32.070 and then forward it onto its destination. 41 01:32.070 --> 01:34.020 So for VM-1 to VM-3, 42 01:34.020 --> 01:37.800 what we're predicting here is no L3 routing. 43 01:37.800 --> 01:38.970 So if we do a Traceflow, 44 01:38.970 --> 01:41.490 it should not even involve this gateway at all. 45 01:41.490 --> 01:44.310 But what we should see is we should see some tunnels 46 01:44.310 --> 01:47.220 being used for the Geneve tunnel as we encapsulate 47 01:47.220 --> 01:50.700 and send the traffic from ESXi-A where VM-1 lives 48 01:50.700 --> 01:53.370 over to ESXi-B where VM-3 lives. 49 01:53.370 --> 01:54.900 So let's verify that next. 50 01:54.900 --> 01:56.700 So back to Plan and Troubleshoot, 51 01:56.700 --> 01:58.920 once again back to Traffic Analysis, 52 01:58.920 --> 01:59.940 and I'm gonna go ahead 53 01:59.940 --> 02:01.470 and collapse this to the left a little bit WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 53 01:59.940 --> 02:01.470 and collapse this to the left a little bit 54 02:01.470 --> 02:03.060 and we'll say a New Trace. 55 02:03.060 --> 02:04.260 And click on Proceed 56 02:04.260 --> 02:05.250 and for this new trace, 57 02:05.250 --> 02:07.080 we wanna trace from VM-1 again. 58 02:07.080 --> 02:09.870 But this time we wanna trace over to VM-3. 59 02:09.870 --> 02:11.550 So we'll select VM-3 60 02:11.550 --> 02:12.630 and there are the details 61 02:12.630 --> 02:14.820 including the layer three addresses 62 02:14.820 --> 02:17.280 and the MAC addresses associated with those two VMs, 63 02:17.280 --> 02:18.690 and we'll click on Trace. 64 02:18.690 --> 02:21.600 All right, so here it's showing us the logical flow 65 02:21.600 --> 02:25.950 from this VM going through segment 10 over to this VM 66 02:25.950 --> 02:27.150 and if we click on this little icon 67 02:27.150 --> 02:28.500 for the host over here on the left, 68 02:28.500 --> 02:30.780 it's showing us that this is ESXi-A. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 68 02:28.500 --> 02:30.780 it's showing us that this is ESXi-A. 69 02:30.780 --> 02:32.910 And then over here on the right, 70 02:32.910 --> 02:35.070 this one should be ESXi-B, 71 02:35.070 --> 02:36.240 the other end of the tunnel. 72 02:36.240 --> 02:37.770 And that traffic logically just went through 73 02:37.770 --> 02:39.930 this layer two broadcast domain 74 02:39.930 --> 02:42.960 that we know and love as the overlay segment 10. 75 02:42.960 --> 02:44.250 So to see the play-by-play, 76 02:44.250 --> 02:46.470 we can go ahead and scroll up here 77 02:46.470 --> 02:47.820 and take a look at the details. 78 02:47.820 --> 02:50.100 So that traffic was injected into the network 79 02:50.100 --> 02:53.340 from the position where VM-1's network adapter is. 80 02:53.340 --> 02:55.770 It went logically through a distributed firewall 81 02:55.770 --> 02:58.560 past those rules and then it went to the physical network, 82 02:58.560 --> 03:00.420 and check this out, this is fantastic. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 82 02:58.560 --> 03:00.420 and check this out, this is fantastic. 83 03:00.420 --> 03:02.310 ESXi-A, as that traffic 84 03:02.310 --> 03:04.050 was going over the transport network, 85 03:04.050 --> 03:06.000 it was sent from the source IP address 86 03:06.000 --> 03:10.290 of the tunnel endpoint for ESXi-A to the remote endpoint, 87 03:10.290 --> 03:12.960 and that's the tunnel endpoint for ESXi-B. 88 03:12.960 --> 03:16.260 Then ESXi-B received that on its local 22, et cetera, 89 03:16.260 --> 03:18.330 and there's the remote side that it came from. 90 03:18.330 --> 03:19.470 If we scroll down, 91 03:19.470 --> 03:22.080 then it also went through a logical distributed firewall 92 03:22.080 --> 03:23.310 and one of the rules that was there 93 03:23.310 --> 03:25.080 that's permitting the traffic by default 94 03:25.080 --> 03:26.730 before it was delivered to that client. 95 03:26.730 --> 03:29.070 So if we looked at this MAC address for a moment, 96 03:29.070 --> 03:30.780 this Mac address right there, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 96 03:29.070 --> 03:30.780 this Mac address right there, 97 03:30.780 --> 03:32.820 that's the Mac address of VM-3. 98 03:32.820 --> 03:36.270 One of the things that ESXi-A had to do was to identify 99 03:36.270 --> 03:39.210 okay, I know the traffic needs to go to this same segment, 100 03:39.210 --> 03:40.830 but where in the world is that MAC address? 101 03:40.830 --> 03:43.440 How did the host transport node ESXi-A 102 03:43.440 --> 03:47.610 know that the MAC address ending in da:1d 103 03:47.610 --> 03:49.380 was a MAC address on that same segment, 104 03:49.380 --> 03:53.160 but reachable over through ESXi-B as in boy? 105 03:53.160 --> 03:56.370 And the answer is NSX provided that information. 106 03:56.370 --> 03:58.230 So what I also recommend we do real quick 107 03:58.230 --> 04:00.480 is let's just validate that as well. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 107 03:58.230 --> 04:00.480 is let's just validate that as well. 108 04:00.480 --> 04:02.010 So I just made a little note that 109 04:02.010 --> 04:05.190 that is the MAC address ending in da:1d 110 04:05.190 --> 04:07.230 that's associated with VM-3 111 04:07.230 --> 04:08.700 and in our topology, 112 04:08.700 --> 04:10.710 that's this little VM right here. 113 04:10.710 --> 04:13.830 Again, that MAC address ends in da:1d. 114 04:13.830 --> 04:16.620 So let's head over to ESXi-A 115 04:16.620 --> 04:18.990 and then in the NSX CLI interface, 116 04:18.990 --> 04:23.070 let's verify that ESXi learned via NSX 117 04:23.070 --> 04:24.960 that that layer two address was reachable 118 04:24.960 --> 04:27.090 via the tunnel over to ESXi-B. 119 04:27.090 --> 04:30.090 So here at the NSX CLI for ESXi-A, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 119 04:27.090 --> 04:30.090 So here at the NSX CLI for ESXi-A, 120 04:30.090 --> 04:31.290 let's do a get, 121 04:31.290 --> 04:33.060 and we can either type in logical switches 122 04:33.060 --> 04:35.310 or we can type in segments just to get a listing 123 04:35.310 --> 04:38.190 of our layer two broadcast domains that are set up. 124 04:38.190 --> 04:40.860 So I'll use the more current option of get segments. 125 04:40.860 --> 04:43.680 So if we're looking for the segment 10 yellow, 126 04:43.680 --> 04:46.590 that's this UUID or we could also just use 127 04:46.590 --> 04:47.670 the VNI number for that, 128 04:47.670 --> 04:49.230 the virtual network identifier. 129 04:49.230 --> 04:51.960 So let's do a get segment space. 130 04:51.960 --> 04:53.730 I'll plug in that VNI number. 131 04:53.730 --> 04:56.820 I'll copy and paste that space and a question mark 132 04:56.820 --> 04:57.900 and let's ask for 133 04:57.900 --> 05:01.290 the MAC address table associated with that. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 133 04:57.900 --> 05:01.290 the MAC address table associated with that. 134 05:01.290 --> 05:03.060 So if we type in a MAC table and tab it out, 135 05:03.060 --> 05:06.150 this is the MAC address table associated with that segment. 136 05:06.150 --> 05:08.430 So notice right here that we have 137 05:08.430 --> 05:10.934 the MAC address ending in da:1d 138 05:10.934 --> 05:14.040 and that's the MAC address associated with VM-3. 139 05:14.040 --> 05:15.000 And here's showing effectively 140 05:15.000 --> 05:18.000 that to reach that layer two address, it's not local, 141 05:18.000 --> 05:19.920 but we'd have to send it over the tunnel, 142 05:19.920 --> 05:21.870 to this tunnel endpoint of 22 et cetera 143 05:21.870 --> 05:25.609 which is the tunnel endpoint for ESXi-B. 144 05:25.609 --> 05:26.550 And this is the layer two address 145 05:26.550 --> 05:28.560 associated with the other end of that tunnel. 146 05:28.560 --> 05:32.010 So if we looked at the VM kernel adapter on ESXi-B WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 146 05:28.560 --> 05:32.010 So if we looked at the VM kernel adapter on ESXi-B 147 05:32.010 --> 05:35.550 that has this IP address assigned to it, 22.22.22.22, 148 05:35.550 --> 05:38.670 its layer two address associated with that tunnel endpoint 149 05:38.670 --> 05:40.830 would be this one ending in 90. 150 05:40.830 --> 05:44.010 And we can verify that right here in the vSphere client 151 05:44.010 --> 05:45.840 simply by going to Host and Clusters, 152 05:45.840 --> 05:48.390 going to ESXi-B, going to Configure, 153 05:48.390 --> 05:50.310 and then going down to VMkernel adapters. 154 05:50.310 --> 05:52.830 And here's that IP address, 22.22.22, 155 05:52.830 --> 05:54.480 associated with VMkernel adapter. 156 05:54.480 --> 05:55.620 If we expand that, 157 05:55.620 --> 05:58.050 there's the MAC address ending in 70:90. 158 05:58.050 --> 06:02.070 So again, ESXi-A has that all cached up, ready to go. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 158 05:58.050 --> 06:02.070 So again, ESXi-A has that all cached up, ready to go. 159 06:02.070 --> 06:04.680 So if it ever needs to forward a frame 160 06:04.680 --> 06:06.180 to the destination address, 161 06:06.180 --> 06:08.430 it knows exactly who the other end point of the tunnel is 162 06:08.430 --> 06:10.320 and it also knows the MAC address 163 06:10.320 --> 06:11.760 associated with that endpoint. 164 06:11.760 --> 06:15.570 It encapsulates, it puts on the right VNI, and ships it. 165 06:15.570 --> 06:16.830 And for that same segment, 166 06:16.830 --> 06:17.970 we could also go ahead 167 06:17.970 --> 06:20.280 and take a look at all the tunnel endpoints as well. 168 06:20.280 --> 06:22.680 So get segment and this is the segment ID 169 06:22.680 --> 06:23.790 for our segment 10. 170 06:23.790 --> 06:25.860 Then we can followed up with vtep-table. 171 06:25.860 --> 06:27.720 That's the virtual tunnel endpoint table 172 06:27.720 --> 06:30.090 and effectively that'll be a list of tunnel endpoints WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 172 06:27.720 --> 06:30.090 and effectively that'll be a list of tunnel endpoints 173 06:30.090 --> 06:32.010 that'll have some association, or connection, 174 06:32.010 --> 06:35.580 or responsibility as far as that layer two segment goes. 175 06:35.580 --> 06:36.690 So we'll type in vtep-table 176 06:36.690 --> 06:38.160 as the option there and press Enter. 177 06:38.160 --> 06:40.110 So here's our local control plane. 178 06:40.110 --> 06:43.020 Our local VTEP is 11.11.11, et cetera. 179 06:43.020 --> 06:46.260 There's our MAC address associated with our tunnel endpoint 180 06:46.260 --> 06:47.520 and then these three right here 181 06:47.520 --> 06:49.860 represent the tunnel endpoints for other nodes. 182 06:49.860 --> 06:52.890 So 22 et cetera is ESXi too. 183 06:52.890 --> 06:56.100 Should we need to forward traffic to one of our edge nodes, 184 06:56.100 --> 06:58.260 we have edge one at 33, et cetera. 185 06:58.260 --> 06:59.577 Edge two is at 44 186 06:59.577 --> 07:03.270 and as we start to integrate a tier zero gateway WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 186 06:59.577 --> 07:03.270 and as we start to integrate a tier zero gateway 187 07:03.270 --> 07:06.870 and add gateway services such as NAT and VPNs and so forth, 188 07:06.870 --> 07:09.840 we are gonna need to leverage the services of those VMs. 189 07:09.840 --> 07:11.670 We'll start using those teps as well. 190 07:11.670 --> 07:13.860 So going back to this one right here, 191 07:13.860 --> 07:16.260 which is the tunnel endpoint, ESXi-B. 192 07:16.260 --> 07:17.910 Once again, there's the MAC address 193 07:17.910 --> 07:20.220 associated with that tunnel endpoint. 194 07:20.220 --> 07:23.070 And in the event VM-1 was doing an initial ARP request 195 07:23.070 --> 07:25.590 for VM-3's MAC address on the same segment, 196 07:25.590 --> 07:27.030 if we hit the up arrow key 197 07:27.030 --> 07:29.730 and we said instead of showing us the vtep-table, 198 07:29.730 --> 07:31.860 let's take a look at the ARP table. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 198 07:29.730 --> 07:31.860 let's take a look at the ARP table. 199 07:31.860 --> 07:33.960 We'll type in ARP table, press Enter, 200 07:33.960 --> 07:35.550 and this is showing us the ARP table 201 07:35.550 --> 07:38.010 that ESXi-A knows about. 202 07:38.010 --> 07:40.920 So if VM-1 which is at 10.10.11, 203 07:40.920 --> 07:42.120 if it didn't ARP request 204 07:42.120 --> 07:45.480 for the layer two address associated with 10.10.10, 205 07:45.480 --> 07:48.177 ESXi-A locally can just respond back to it 206 07:48.177 --> 07:49.320 and say here it is, 207 07:49.320 --> 07:51.510 and the client wouldn't even know the difference. 208 07:51.510 --> 07:52.710 It simply made an ARP request, 209 07:52.710 --> 07:53.670 it got the response 210 07:53.670 --> 07:54.630 and at the same time, 211 07:54.630 --> 07:58.260 ESXi-A did not have to ship that broadcast, 212 07:58.260 --> 08:00.570 that ARP request over the tunnel WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 212 07:58.260 --> 08:00.570 that ARP request over the tunnel 213 08:00.570 --> 08:03.720 and flood it effectively to any other transport node. 214 08:03.720 --> 08:05.220 Because they had it cached locally, 215 08:05.220 --> 08:08.040 it could feed the response directly back to the client. 216 08:08.040 --> 08:09.030 So for the next flow, 217 08:09.030 --> 08:12.420 what if VM-1 right here at 10.10.11 218 08:12.420 --> 08:15.210 was gonna send traffic over here to VM-2 219 08:15.210 --> 08:16.830 which is on a different host 220 08:16.830 --> 08:18.690 and it's on a different subnet? 221 08:18.690 --> 08:21.600 So the rule about routing in the world of NSX 222 08:21.600 --> 08:24.600 is we're gonna route as close to the source as possible 223 08:24.600 --> 08:27.270 and what that means is that ESXi-A 224 08:27.270 --> 08:28.920 is gonna do all the logical routing. 225 08:28.920 --> 08:31.170 So as the traffic is sourced from VM-1 WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 225 08:28.920 --> 08:31.170 So as the traffic is sourced from VM-1 226 08:31.170 --> 08:32.490 and it's going to VM-4, 227 08:32.490 --> 08:33.840 I'll circle those for reference, 228 08:33.840 --> 08:36.990 logically the traffic is gonna go over the segment. 229 08:36.990 --> 08:39.330 It's gonna come into the router on this interface, 230 08:39.330 --> 08:41.160 be routed out, go over this segment, 231 08:41.160 --> 08:43.290 and then forwarded down to this VM. 232 08:43.290 --> 08:45.090 It's gonna be ESXi-A 233 08:45.090 --> 08:47.940 that logically is making all the routing decisions. 234 08:47.940 --> 08:51.840 So as the traffic comes into ESXi-A, 235 08:51.840 --> 08:54.000 it's gonna look at the destination address, 236 08:54.000 --> 08:56.130 consider its distributed router. 237 08:56.130 --> 08:57.510 It'll make a routing decision 238 08:57.510 --> 08:58.830 saying oh, this needs to be forwarded 239 08:58.830 --> 09:00.120 to the 10.20.0 network. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 239 08:58.830 --> 09:00.120 to the 10.20.0 network. 240 09:00.120 --> 09:03.060 It'll choose the appropriate egress or outbound interface 241 09:03.060 --> 09:03.893 and then at that point, 242 09:03.893 --> 09:06.480 it'll consider the destination layer two address 243 09:06.480 --> 09:08.190 which is on segment 20, 244 09:08.190 --> 09:10.950 but it'll also realize that oh, that layer two address 245 09:10.950 --> 09:13.500 that I'm trying to forward to is reachable 246 09:13.500 --> 09:15.600 over the transport network 247 09:15.600 --> 09:17.280 and it would know exactly who the other end point 248 09:17.280 --> 09:18.113 of the tunnel is. 249 09:18.113 --> 09:20.400 So it'd go ahead and use the Geneve tunnel, 250 09:20.400 --> 09:23.340 encapsulate that traffic, forward it over to ESXi-B 251 09:23.340 --> 09:25.380 who then deencapsulate it and forward it. 252 09:25.380 --> 09:27.570 So when the traffic comes in to ESXi-B, 253 09:27.570 --> 09:30.060 the routing decision has already been done. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 253 09:27.570 --> 09:30.060 the routing decision has already been done. 254 09:30.060 --> 09:32.550 So in the Geneve header when it specifies the VNI, 255 09:32.550 --> 09:35.007 it's gonna have the VNI information for segment 20. 256 09:35.007 --> 09:36.810 And then all ESXi-B has to do 257 09:36.810 --> 09:39.000 is simply when it receives it, deencapsulate it, 258 09:39.000 --> 09:41.310 and forward it to the VM that has that MAC address. 259 09:41.310 --> 09:42.600 So to help reinforce that, 260 09:42.600 --> 09:45.240 we're gonna once again use Traceflow to do that. 261 09:45.240 --> 09:48.570 But I'd also like to capture that traffic on the network 262 09:48.570 --> 09:51.090 so you can actually see over the transport network 263 09:51.090 --> 09:54.360 right here being encapsulated as part of the Geneve tunnel. 264 09:54.360 --> 09:57.660 I'd like to demonstrate that before ESXi-A forwards it, 265 09:57.660 --> 10:00.960 it already put the segment 20 VNI number on there WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 265 09:57.660 --> 10:00.960 it already put the segment 20 VNI number on there 266 10:00.960 --> 10:03.420 because it already made that logical routing decision. 267 10:03.420 --> 10:04.620 So let's do all of that right now 268 10:04.620 --> 10:07.620 with the trace from VM-1 over to VM-4. 269 10:07.620 --> 10:09.060 So back in the NSX manager, 270 10:09.060 --> 10:12.180 we'll say we wanna do a New Trace, click on Proceed, 271 10:12.180 --> 10:14.610 and then we'll specify that our source VM 272 10:14.610 --> 10:16.470 is going to be VM-1. 273 10:16.470 --> 10:18.810 And there's its IP address and MAC address 274 10:18.810 --> 10:20.760 and then the destination is gonna be 275 10:20.760 --> 10:23.190 VM number four at 10.20.0.21. 276 10:23.190 --> 10:25.560 So there's its IP address, there's its MAC address, 277 10:25.560 --> 10:26.880 and let's click on Trace. 278 10:26.880 --> 10:29.070 Now meanwhile while this is going, 279 10:29.070 --> 10:31.590 I'm actually capturing the traffic in the background WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 279 10:29.070 --> 10:31.590 I'm actually capturing the traffic in the background 280 10:31.590 --> 10:32.880 over the transport network. 281 10:32.880 --> 10:34.350 So when that traffic goes over 282 10:34.350 --> 10:36.510 whatever the VLAN is they're using for that, 283 10:36.510 --> 10:38.760 that's gonna be copied to my Wireshark 284 10:38.760 --> 10:41.190 and then we can take a look at the actual packets involved. 285 10:41.190 --> 10:43.830 So here in the output, let me bring this full screen, 286 10:43.830 --> 10:47.010 here is the logical representation of what it just did. 287 10:47.010 --> 10:50.040 So the traffic was sourced from VM-1, 288 10:50.040 --> 10:52.590 went through this logical segment, segment 10, 289 10:52.590 --> 10:53.760 went to a router interface 290 10:53.760 --> 10:54.930 that was connected to that segment. 291 10:54.930 --> 10:56.340 It made a routing decision, 292 10:56.340 --> 10:59.550 forwarded it down through segment 20 down to the final VM. 293 10:59.550 --> 11:00.540 So on the left-hand side, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 293 10:59.550 --> 11:00.540 So on the left-hand side, 294 11:00.540 --> 11:04.200 if we look on the host icon here, that was ESXi-A. 295 11:04.200 --> 11:07.350 And up here, if we click on the icon here for the host, 296 11:07.350 --> 11:09.090 that's also ESXi-A. 297 11:09.090 --> 11:10.500 And what may be a little surprising 298 11:10.500 --> 11:13.950 is that because ESXi-A was the closest to the source 299 11:13.950 --> 11:16.380 and it had a distributed routing table, 300 11:16.380 --> 11:19.110 if we look over here at segment 20 and click here, 301 11:19.110 --> 11:22.110 check it out, it's also ESXi-A 302 11:22.110 --> 11:23.760 because all that logic between the routing 303 11:23.760 --> 11:28.410 between subnet 10.10 and subnet 10.20 was done by ESXi-A. 304 11:28.410 --> 11:30.000 But then when that traffic WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 305 11:30.000 --> 11:31.830 needed to go to the final destination, 306 11:31.830 --> 11:33.690 at this point right here in our journey, 307 11:33.690 --> 11:35.520 it was forwarded over the tunnel 308 11:35.520 --> 11:37.410 and that's where ESXi-B received it 309 11:37.410 --> 11:39.870 and was able to finally forward it to its client 310 11:39.870 --> 11:42.240 or in this case to its VM, VM number four. 311 11:42.240 --> 11:45.210 So I go ahead and close that full screen 312 11:45.210 --> 11:47.280 and we'll look at the play-by-play to verify that. 313 11:47.280 --> 11:48.660 So here the traffic was injected 314 11:48.660 --> 11:51.030 in the network on behalf of VM-1. 315 11:51.030 --> 11:53.310 It went logically through a distributed firewall 316 11:53.310 --> 11:54.690 which allowed it to pass. 317 11:54.690 --> 11:56.253 Logically it went through segment 10, 318 11:56.253 --> 11:59.460 then it went to the router on the inbound interface, 319 11:59.460 --> 12:00.630 and then it was routed out. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 319 11:59.460 --> 12:00.630 and then it was routed out. 320 12:00.630 --> 12:03.900 The outbound interface again, still being done by ESXi-A. 321 12:03.900 --> 12:06.210 Forward logically over the pink segment. 322 12:06.210 --> 12:07.170 Then at this point right here, 323 12:07.170 --> 12:08.760 it forwarded over the Geneve tunnel 324 12:08.760 --> 12:10.260 over the transport network 325 12:10.260 --> 12:12.540 and that once again, is because of the tables 326 12:12.540 --> 12:15.330 that are being maintained that let ESXi know 327 12:15.330 --> 12:17.820 that hey, based on this destination layer two address, 328 12:17.820 --> 12:20.490 you need to forward it over to this endpoint 329 12:20.490 --> 12:22.680 so it can go ahead and continue forwarding. 330 12:22.680 --> 12:26.310 And then the next one is ESXi-B receiving it over the tunnel 331 12:26.310 --> 12:27.360 and then processing it 332 12:27.360 --> 12:29.430 logically through the distributed firewall. 333 12:29.430 --> 12:32.370 It passes by default because the default rules permit it WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 333 12:29.430 --> 12:32.370 It passes by default because the default rules permit it 334 12:32.370 --> 12:34.440 and then finally being delivered to the client. 335 12:34.440 --> 12:36.720 So if we were to capture the traffic right here 336 12:36.720 --> 12:38.190 on the transport network 337 12:38.190 --> 12:40.500 and from host A's perspective, 338 12:40.500 --> 12:43.380 it's using Uplink two and VLAN 101 339 12:43.380 --> 12:44.850 and on my transport network 340 12:44.850 --> 12:46.050 I have routing and everything set up. 341 12:46.050 --> 12:48.480 So it's being delivered over here to ESXi-B 342 12:48.480 --> 12:51.210 at its tep of 22.22.22, et cetera, 343 12:51.210 --> 12:52.920 and it's using a different VLAN 344 12:52.920 --> 12:55.290 for the support of that subnet where the tep is. 345 12:55.290 --> 12:56.460 And again, my transport network 346 12:56.460 --> 12:58.500 is forwarding on routing all that traffic. 347 12:58.500 --> 12:59.880 So the traffic from the client 348 12:59.880 --> 13:02.760 would've been sourced in segment 10. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 348 12:59.880 --> 13:02.760 would've been sourced in segment 10. 349 13:02.760 --> 13:04.740 So ESXi would make your routing decision. 350 13:04.740 --> 13:06.420 Just logically like this router would that say 351 13:06.420 --> 13:08.190 oh, it needs to be routed out this interface. 352 13:08.190 --> 13:10.500 So that decision is made by ESXi-A 353 13:10.500 --> 13:12.270 and then as a result of needing to forward it 354 13:12.270 --> 13:13.860 to that final layer two address 355 13:13.860 --> 13:14.880 based on that layer two address 356 13:14.880 --> 13:16.350 being reachable over the tunnel 357 13:16.350 --> 13:18.900 and then that traffic would be encapsulated 358 13:18.900 --> 13:20.850 and forwarded over to the other end point. 359 13:20.850 --> 13:22.770 And as far as the actual VNI 360 13:22.770 --> 13:24.990 that was included in the Geneve header, 361 13:24.990 --> 13:28.770 it would be the VNI that segment 20 is using 362 13:28.770 --> 13:30.540 because once again, the routing was already done WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 362 13:28.770 --> 13:30.540 because once again, the routing was already done 363 13:30.540 --> 13:32.340 in the mind of ESXi-A 364 13:32.340 --> 13:33.420 and now we're just delivering it 365 13:33.420 --> 13:35.700 by forwarding it through the tunnel to ESXi-B 366 13:35.700 --> 13:36.810 where it can deencapsulate 367 13:36.810 --> 13:38.460 and forward it to the final VM. 368 13:38.460 --> 13:40.170 Also before we jump into the capture, 369 13:40.170 --> 13:43.860 let's just verify what the segment 20 VNI is. 370 13:43.860 --> 13:45.210 So here on the vSphere client, 371 13:45.210 --> 13:46.650 let's go to the Network tab. 372 13:46.650 --> 13:48.090 It's gonna be super easy to see right here. 373 13:48.090 --> 13:50.610 So there is segment 20 374 13:50.610 --> 13:52.440 and if we go to Summary, 375 13:52.440 --> 13:56.310 right here it's showing us the segment number of 73728. 376 13:56.310 --> 13:59.190 So I'm gonna make a note that the Segment 20 Pink 377 13:59.190 --> 14:02.490 has the VNI number 73728. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 377 13:59.190 --> 14:02.490 has the VNI number 73728. 378 14:02.490 --> 14:04.260 All right, fantastic, with that in mind, 379 14:04.260 --> 14:05.730 let's take a look at our capture. 380 14:05.730 --> 14:07.470 All right, so here is our capture. 381 14:07.470 --> 14:08.940 So I had this being delivered 382 14:08.940 --> 14:10.200 over to my management computer. 383 14:10.200 --> 14:12.900 So the first four rows 384 14:12.900 --> 14:16.050 are all about just taking the traffic that was captured 385 14:16.050 --> 14:19.410 and shipping it over with GRE over to my workstation. 386 14:19.410 --> 14:22.200 So the real story as far as the captured traffic 387 14:22.200 --> 14:24.990 off the transport network that we care about is right here. 388 14:24.990 --> 14:26.670 So as far as the IPS involved 389 14:26.670 --> 14:28.271 for the transport network it was coming from, 390 14:28.271 --> 14:31.290 this packet was coming from 11.11.11 WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 390 14:28.271 --> 14:31.290 this packet was coming from 11.11.11 391 14:31.290 --> 14:32.790 which is the IP address 392 14:32.790 --> 14:34.770 for the tunnel endpoint on ESXi-A 393 14:34.770 --> 14:36.927 going to the tunnel endpoint on ESXi-B, 394 14:36.927 --> 14:39.810 and the ethernet header looks interesting to me. 395 14:39.810 --> 14:40.920 Let's take a look at that for a moment. 396 14:40.920 --> 14:44.310 So the destination is this MAC address, 70:90. 397 14:44.310 --> 14:48.000 I recognize that as the literal layer two address 398 14:48.000 --> 14:49.740 associated with the VM criminal adapter. 399 14:49.740 --> 14:52.230 But what I'm actually a little surprised about 400 14:52.230 --> 14:54.510 is this source Mac address right here. 401 14:54.510 --> 14:56.970 The source MAC address based on Wireshark 402 14:56.970 --> 14:59.820 and what it believes the identifier here, 403 14:59.820 --> 15:02.070 it thinks it's an address owned by Cisco. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 403 14:59.820 --> 15:02.070 it thinks it's an address owned by Cisco. 404 15:02.070 --> 15:03.960 And what just occurred to me 405 15:03.960 --> 15:07.200 is that that's because that traffic was being forwarded 406 15:07.200 --> 15:11.220 by my transport network layer three routed infrastructure. 407 15:11.220 --> 15:12.540 So on my transport network, 408 15:12.540 --> 15:14.490 I've got a router function here 409 15:14.490 --> 15:17.670 and I captured this traffic right here on the network. 410 15:17.670 --> 15:20.430 So this would be from one of my router interfaces 411 15:20.430 --> 15:23.760 which is a Cisco interface on VLAN 10 412 15:23.760 --> 15:26.040 that's going to the tep, tunnel endpoint address, 413 15:26.040 --> 15:29.700 of 22, et cetera that ESXi-B is using. 414 15:29.700 --> 15:32.550 And that would explain why the source layer two address WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 414 15:29.700 --> 15:32.550 And that would explain why the source layer two address 415 15:32.550 --> 15:35.550 represents a layer two address on my router interface 416 15:35.550 --> 15:36.750 that was making that forwarding 417 15:36.750 --> 15:38.310 on behalf of the transport network 418 15:38.310 --> 15:39.950 and the destination layer two address, 419 15:39.950 --> 15:41.700 it was the actual destination address 420 15:41.700 --> 15:46.230 of our VM kernel adapter 22.22.22.22, 421 15:46.230 --> 15:48.210 which is acting as our tunnel endpoint. 422 15:48.210 --> 15:50.070 So it's always fun to look at the packet captures 423 15:50.070 --> 15:52.560 to help reinforce what we think is happening 424 15:52.560 --> 15:54.030 and verify that it literally is. 425 15:54.030 --> 15:56.670 So once that traffic was then delivered 426 15:56.670 --> 15:59.790 to this destination IP address of 22.22.22. 427 15:59.790 --> 16:01.860 At layer four it's using UDP, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 427 15:59.790 --> 16:01.860 At layer four it's using UDP, 428 16:01.860 --> 16:03.390 so here it has a VNI number. 429 16:03.390 --> 16:05.040 Now this is in hexadecimal, 430 16:05.040 --> 16:07.140 so thank goodness it's doing the conversion 431 16:07.140 --> 16:08.370 for us right here 432 16:08.370 --> 16:11.460 and confirming for us that the VNI number 433 16:11.460 --> 16:13.050 in decimal is 73728 434 16:14.220 --> 16:18.210 which is what we verified is the VNI for segment 20. 435 16:18.210 --> 16:19.530 So this header is just representing 436 16:19.530 --> 16:23.430 that this is for VNI 73728 437 16:23.430 --> 16:26.340 and then once ESXi-B deencapsulates that, 438 16:26.340 --> 16:28.230 here is the original content. 439 16:28.230 --> 16:30.090 So this destination MAC address right here WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 439 16:28.230 --> 16:30.090 So this destination MAC address right here 440 16:30.090 --> 16:33.060 is the destination MAC address of VM-4. 441 16:33.060 --> 16:33.893 And we can see that right here 442 16:33.893 --> 16:35.400 in the results of our Traceflow. 443 16:35.400 --> 16:37.770 So there's the MAC address, c9:d3. 444 16:37.770 --> 16:39.090 And then for the source Mac address, 445 16:39.090 --> 16:40.440 the source MAC address of VM-1 446 16:40.440 --> 16:42.990 which ends in d9:97, 447 16:42.990 --> 16:45.360 that isn't the MAC address that we have showing up 448 16:45.360 --> 16:47.340 inside of our capture. 449 16:47.340 --> 16:48.747 We have some kind of a VMware address 450 16:48.747 --> 16:52.080 and the reason for that is that this logical router 451 16:52.080 --> 16:53.880 made a logical routing decision. 452 16:53.880 --> 16:56.370 So ESXi-A, when it made that routing decision 453 16:56.370 --> 16:57.870 just as a real layer of three router, 454 16:57.870 --> 17:01.170 would it used its own layer two MAC address WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 454 16:57.870 --> 17:01.170 would it used its own layer two MAC address 455 17:01.170 --> 17:03.480 associated with this egress outbound interface 456 17:03.480 --> 17:05.400 before making the forwarding decision 457 17:05.400 --> 17:06.930 down towards the destination. 458 17:06.930 --> 17:08.520 So for that virtual source MAC address 459 17:08.520 --> 17:09.660 that we saw in the capture, 460 17:09.660 --> 17:11.760 that'd be the source virtual MAC address 461 17:11.760 --> 17:14.100 that this logical gateway is using 462 17:14.100 --> 17:18.060 for its interface that's connected down to our segment 20. 463 17:18.060 --> 17:19.980 And just for a moment 'cause I'm curious, 464 17:19.980 --> 17:21.750 I wanna verify what that Mac address is. 465 17:21.750 --> 17:25.170 So back at ESXi-A who made that routing decision 466 17:25.170 --> 17:28.140 and has a copy of the distributed router running on it, 467 17:28.140 --> 17:29.760 then we'll do a get gateway 468 17:29.760 --> 17:33.870 and we'll plug in that virtual distributed router UUID, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 468 17:29.760 --> 17:33.870 and we'll plug in that virtual distributed router UUID, 469 17:33.870 --> 17:35.340 a space, a question mark, 470 17:35.340 --> 17:38.040 and then we'll type in interfaces and press Enter 471 17:38.040 --> 17:40.020 and there are interfaces. 472 17:40.020 --> 17:41.880 And so the egress or outbound interface 473 17:41.880 --> 17:43.770 would've been this one right here 474 17:43.770 --> 17:46.950 and this is the MAC address that would've been seen 475 17:46.950 --> 17:50.760 for the source on that frame as it was being sent 476 17:50.760 --> 17:53.250 and being routed out of this logical router interface. 477 17:53.250 --> 17:55.500 So again, that's one ending in 4452. 478 17:55.500 --> 17:57.090 Let's take a look at the packet capture 479 17:57.090 --> 17:58.710 and verify that matches up. 480 17:58.710 --> 18:01.410 And sure enough, right there for the source Mac address, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 480 17:58.710 --> 18:01.410 And sure enough, right there for the source Mac address, 481 18:01.410 --> 18:03.480 the one ending in 4452 482 18:03.480 --> 18:06.750 which is the logical interface on the logical router 483 18:06.750 --> 18:08.520 that was making the routing decision 484 18:08.520 --> 18:11.790 for forwarding that packet over down to the segment 20. 485 18:11.790 --> 18:13.800 Then we have the layer three header for the original packet 486 18:13.800 --> 18:16.740 which is sourced from the VM-1's IP address 487 18:16.740 --> 18:18.510 going to VM four's IP address 488 18:18.510 --> 18:19.620 and then the actual payload 489 18:19.620 --> 18:22.170 which was an ICMP Echo request, 490 18:22.170 --> 18:24.780 and that's what they used for the actual traffic 491 18:24.780 --> 18:26.370 by default in Traceflow. 492 18:26.370 --> 18:27.810 So let's do a quick recap. 493 18:27.810 --> 18:31.080 We created this logical distributed router. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 493 18:27.810 --> 18:31.080 We created this logical distributed router. 494 18:31.080 --> 18:32.730 We connected it to segments. 495 18:32.730 --> 18:34.810 We gave it the IP address we wanted to use 496 18:34.810 --> 18:36.570 of .1 on each of the segments. 497 18:36.570 --> 18:39.570 We then had our VMs on those respective segments 498 18:39.570 --> 18:41.490 use that IP address as their default gateway. 499 18:41.490 --> 18:43.320 I provided that via DHCP 500 18:43.320 --> 18:44.153 or we could have done it 501 18:44.153 --> 18:46.140 manually as well on the VMs, either way. 502 18:46.140 --> 18:47.460 And then for the forwarding, 503 18:47.460 --> 18:49.560 if both VMs are on the same host, 504 18:49.560 --> 18:51.570 each host with the distributed router 505 18:51.570 --> 18:53.190 can make those routing decisions 506 18:53.190 --> 18:55.080 and forward the traffic without an ounce of that 507 18:55.080 --> 18:57.120 having to go over the transport network. 508 18:57.120 --> 18:58.860 Or when traffic does need to go, 509 18:58.860 --> 19:00.780 for example, from a VM over here WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 509 18:58.860 --> 19:00.780 for example, from a VM over here 510 19:00.780 --> 19:03.030 on ESXi-A on that transport node 511 19:03.030 --> 19:05.040 and be sent over here to one of these VMs 512 19:05.040 --> 19:07.440 over here on a second transport node, 513 19:07.440 --> 19:09.570 the transport node that is closest to the source 514 19:09.570 --> 19:10.950 will make any routing decisions, 515 19:10.950 --> 19:12.990 layer three routing decisions that are needed. 516 19:12.990 --> 19:15.000 And then once those routing decisions are made, 517 19:15.000 --> 19:17.340 if it needs to forward the traffic over the tunnel 518 19:17.340 --> 19:19.020 to another transport node, 519 19:19.020 --> 19:20.280 it'll do exactly that. 520 19:20.280 --> 19:21.113 And then it's up 521 19:21.113 --> 19:23.070 to the receiving transport node to deencapsulate 522 19:23.070 --> 19:25.563 and forward to the correct and final destination.