WEBVTT 00:00:00.000 --> 00:00:06.300 Welcome back to vSphere 8 in the real world, and now we're going to discuss SR-IOV. 00:00:06.300 --> 00:00:12.800 So SR-IOV is Single Root I-O Virtualization, and it's actually a pretty cool feature. 00:00:12.800 --> 00:00:16.500 It's something that does require compatible hardware to do. 00:00:16.500 --> 00:00:21.100 You need compatible network adapters specifically for this to work. 00:00:21.100 --> 00:00:26.500 So the way virtual networking normally will work is you have your physical adapter 00:00:26.700 --> 00:00:30.200 and a packet comes in to the adapter. 00:00:30.200 --> 00:00:38.000 Now, the adapter might be able to read the MAC address, but in order to process that particular packet, 00:00:38.000 --> 00:00:48.799 there's going to be an interrupt sent from the network card to a core on the CPU to get processing power 00:00:48.799 --> 00:00:52.700 to then go through, process that packet, and then what's going to happen, 00:00:52.700 --> 00:00:56.299 it's going to go through the entire VM kernel networking stack there. 00:00:56.299 --> 00:01:02.700 It's going to go from the adapter, through the PCI bus, to the kernel, through the virtual switch, 00:01:02.700 --> 00:01:08.300 through the port group, reprocess all the policies for that port group, and then to the VM. 00:01:08.300 --> 00:01:14.400 And the VM is going to need a core, an interrupt from the core, to accept that packet, 00:01:14.400 --> 00:01:19.300 and the network card needs a core or an interrupt to process that packet. 00:01:19.300 --> 00:01:26.099 And so going through that system is going to actually use quite a bit of CPU time. 00:01:26.099 --> 00:01:34.400 What the single-root IO virtualization does is it's a network card that has the ability to create these queues, 00:01:34.400 --> 00:01:38.699 and the queues can be associated with certain virtual machines, 00:01:38.699 --> 00:01:45.400 and then the virtual machines are able to get the packet directly from the adapter. 00:01:45.400 --> 00:01:50.900 The adapter can write into the VM's RAM, and now the VM has that packet, 00:01:50.900 --> 00:01:55.500 without having to do a bunch of additional processing by the physical processor, 00:01:55.500 --> 00:01:58.500 by our Intel Xeon whatever processor. 00:01:58.500 --> 00:02:01.699 We're not having to send as many interrupts through the system. 00:02:01.699 --> 00:02:06.300 The actual network card handles receiving the packet, processing the packet, 00:02:06.300 --> 00:02:10.300 and then putting it into RAM on that virtual machine. 00:02:10.300 --> 00:02:14.699 So we need to have host to support it, meaning we need to have a version of ESXi that supports it. 00:02:14.699 --> 00:02:17.300 We need to have physical NICs that support it. 00:02:17.300 --> 00:02:24.899 Then we need to install a driver, a specific driver, an SR-IOV or single-root IO virtualization, 00:02:24.899 --> 00:02:31.699 SR-IOV network adapter on the VM, and that requires VMware Tools, 00:02:31.699 --> 00:02:34.199 and then, of course, the right version of VMware Tools. 00:02:34.199 --> 00:02:41.800 So if we have a VM that we migrated over from 7, and when we got it to 8, we didn't upgrade VMware Tools, this may not work. 00:02:41.800 --> 00:02:44.699 We want to make sure we have the newest version of VMware Tools. 00:02:44.699 --> 00:02:56.600 If we go through and we update our version 8 ESXi to 8.0 update 1, update 2, update 3, whatever it may be, 00:02:56.600 --> 00:03:04.800 and that comes with new versions of VMware Tools, and we don't update the VMware Tools in the VMs that are running this SR-IOV feature, 00:03:04.800 --> 00:03:09.000 it may break that virtual machine until we upgrade VMware Tools over there. 00:03:09.000 --> 00:03:13.500 So now, all of a sudden, we're making our patching a lot more critical, 00:03:13.500 --> 00:03:21.500 and our network way more reliant on making sure when we patch the hosts, we update VMware Tools in the VMs. 00:03:21.500 --> 00:03:24.899 I've seen it many times where I say, okay, we're going to be patching hosts this weekend, 00:03:24.899 --> 00:03:31.300 and we're going to schedule it for later to patch the VMs because I can patch my hosts and not take anything down, 00:03:31.300 --> 00:03:36.100 but to patch the VMware Tools or update the VMware Tools in the VMs requires a reboot, 00:03:36.100 --> 00:03:40.699 and reboots of the VMs require a maintenance window, and that means scheduling. 00:03:40.699 --> 00:03:45.300 That means a bunch of people have to be notified that these certain services this time go down. 00:03:45.300 --> 00:03:52.300 Certain VMs have to go down in a certain order and then come back in a certain order in certain environments or certain applications. 00:03:52.300 --> 00:04:00.000 Because of the complexity of patching certain VMs, a lot of VMware admins don't patch VMware Tools very often. 00:04:00.100 --> 00:04:01.899 That is not a good practice. 00:04:01.899 --> 00:04:06.300 You should patch your VMware Tools as often and as frequently as you patch your hosts, 00:04:06.300 --> 00:04:09.399 but in the real world, that's not exactly how it works. 00:04:09.399 --> 00:04:11.000 In the perfect world, it would be. 00:04:11.000 --> 00:04:13.300 It's not always how it works. 00:04:13.300 --> 00:04:16.399 So because of that, now this can be dependent. 00:04:16.399 --> 00:04:24.000 Your network operation can be dependent upon having the correct version of VMware Tools in your VMs to match your hosts 00:04:24.000 --> 00:04:28.299 so you can get the right version of the driver so that all of this works, okay? 00:04:28.299 --> 00:04:32.100 So again, this is going to be dependent upon a lot of different pieces. 00:04:32.100 --> 00:04:37.000 But if you have this working correctly, you can make your network performance be a lot faster, 00:04:37.000 --> 00:04:40.299 take away a lot of overhead from your hosts, okay? 00:04:40.299 --> 00:04:44.299 So if you want to know if you have the right host and the right network cards and all that kind of stuff, 00:04:44.299 --> 00:04:48.000 check your VMware Compatibility Guide. 00:04:48.000 --> 00:04:50.600 So here's the path. 00:04:50.600 --> 00:04:57.500 Your guest OS requests a change for a virtual function, okay? 00:04:57.600 --> 00:05:02.600 So the virtual function forwards the request to your physical function, the actual physical network card itself, right? 00:05:02.600 --> 00:05:05.500 The physical network card driver checks for the request. 00:05:05.500 --> 00:05:08.200 The virtual switch verifies, right? 00:05:08.200 --> 00:05:13.000 So example request against whatever policies you have configured on the networking there, right? 00:05:13.000 --> 00:05:21.000 And then your physical function, your actual physical device driver configures this virtual function, settings are compliant. 00:05:21.000 --> 00:05:26.600 So there are policies matched with the virtual machines trying to do, all right? 00:05:26.600 --> 00:05:28.600 How do we actually enable this? 00:05:28.600 --> 00:05:30.299 Here's the important part. 00:05:30.299 --> 00:05:32.200 So we need to go to our vSphere client. 00:05:32.200 --> 00:05:33.899 We're going to go to our host. 00:05:33.899 --> 00:05:38.600 On our host, we're going to go to configuration and networking and then go to the physical adapter. 00:05:38.600 --> 00:05:44.799 We're going to select the adapter, and then we're going to go to the settings, and we're going to enable SRIOV. 00:05:44.799 --> 00:05:51.500 If you right-click on your physical adapter and go to settings and you don't have any option there other than maybe like, you know, 00:05:51.500 --> 00:05:56.500 it's set to 10 gig or 1 gig or whatever the case may be set to, that's the only option you have. 00:05:56.500 --> 00:05:58.299 Then your SRIOV is not on that card. 00:05:58.299 --> 00:06:02.200 You don't have an SRIOV compliant compatible card. 00:06:02.200 --> 00:06:08.100 Or if you do, you don't have a driver for ESXi or ESXi call it a VIB. 00:06:08.100 --> 00:06:12.299 You don't have the right VIB for that card to use that function. 00:06:12.299 --> 00:06:26.399 So maybe, you know, the card's newer and the VIB's newer and you need to flip the VIB into the, you know, the ESXi host in order for it to see that card and see all the functions of that card and be able to use the SRIOV. 00:06:26.399 --> 00:06:36.000 So if you right-click on, if you go into your, you know, your environment at work and you right-click on your adapter and go to settings and you're like, it's not there, your card just isn't supported for it. 00:06:36.000 --> 00:06:44.500 Either this doesn't have it, which is probably the case, or if your card does support it, we wouldn't got brand new, the latest and greatest cards. 00:06:44.500 --> 00:06:46.500 Are they on the hardware compatibility list? 00:06:46.500 --> 00:06:51.399 Does VMware, you know, supply a driver with the operating system that match that? 00:06:51.399 --> 00:06:54.700 Or do you need to slip the VIB in for that? 00:06:54.700 --> 00:06:58.899 And in which case, is it certified by VMware to work with your host? 00:06:58.899 --> 00:07:06.000 Okay, so some things to consider there when you go to edit that setting if you do or don't see this option. 00:07:06.000 --> 00:07:12.399 So once we've enabled the SRIOV on the adapter, then we're going to set the number of virtual functions. 00:07:12.399 --> 00:07:14.299 So we're going to set the number of queues. 00:07:14.299 --> 00:07:19.000 So we can have multiple queues and different queues can have different options set up. 00:07:19.000 --> 00:07:21.399 Maybe we want to enable certain features. 00:07:21.399 --> 00:07:22.100 We'll talk about them. 00:07:22.100 --> 00:07:29.500 Basically, all the features that you're going to look at on a port group, are we going to allow, you know, MAC spoofing? 00:07:29.500 --> 00:07:31.600 Are we going to allow promiscuous mode? 00:07:31.600 --> 00:07:33.000 Things like that. 00:07:33.000 --> 00:07:36.799 For every group of settings you need for that, you're going to need a different virtual function, okay? 00:07:36.799 --> 00:07:39.399 You're going to set those policies on the virtual function. 00:07:39.399 --> 00:07:44.200 Are we going to allow, again, promiscuous mode, MAC address spoofing? 00:07:44.200 --> 00:07:45.600 You know, things like that on there. 00:07:45.600 --> 00:07:49.500 Say, okay, we're going to restart that host. 00:07:49.500 --> 00:07:56.899 On the virtual machine, we're going to then go to the VM and we're going to want to add another NIC to the virtual machine. 00:07:56.899 --> 00:08:09.500 And for the NIC, we're going to say, you know, instead of being on E1000 or VMXNet3, we're going to say we want an SRIOV network adapter for this particular virtual machine, right? 00:08:09.500 --> 00:08:13.399 Then we can, you know, select the VM and we're going to specify what adapter we want to put on there, okay? 00:08:16.200 --> 00:08:20.600 So, then you're going to assign a virtual function to that adapter. 00:08:20.600 --> 00:08:21.700 We're going to add the adapter. 00:08:21.700 --> 00:08:30.500 We're going to add the adapter for the, you know, SRIOV and then you'll have an option on the adapter for selecting the virtual function that we want to use. 00:08:30.500 --> 00:08:33.799 And so then, again, we have in there what are all these different options we want to do. 00:08:33.799 --> 00:08:35.400 Do we want to change the MTU size? 00:08:35.400 --> 00:08:38.200 Do we want to enable jumbo frames on this one for some reason? 00:08:38.200 --> 00:08:40.200 Are we going to allow promiscuous mode? 00:08:40.200 --> 00:08:45.900 So, some of the other options that I mentioned, you're going to say okay to that and then you're going to power on the virtual machine. 00:08:45.900 --> 00:08:47.799 So, we power on the virtual machine. 00:08:47.799 --> 00:09:05.000 You're going to have this new network card and this new network card is going to be associated with this new virtual function that's going to be able to receive packets directly loaded into its memory from SRIOV-compliant physical adapters. 00:09:05.000 --> 00:09:12.099 So, some of these options that we mentioned, MTU size, are we going to allow promiscuous mode on this adapter or not? 00:09:12.099 --> 00:09:17.400 You know, basically any kind of traffic shaping you want to configure on there, so your security policies, that kind of stuff, right? 00:09:17.400 --> 00:09:18.599 Also, your DLAN tagging. 00:09:18.599 --> 00:09:31.200 So, keep in mind that when we get information from the network in this SRIOV NIT, we circumvented the virtual switch and the port group. 00:09:31.200 --> 00:09:48.700 All of that information that the virtual switch and the port group had, like my DLAN tag, my MTU size, my security settings for MAC address spoofing and all that kind of stuff, promiscuous mode, all of that was circumvented when I circumvented the port group or the virtual switch and the virtual switch. 00:09:48.700 --> 00:10:01.799 So, now I need to get those settings for all that stuff from the virtual function that I created on the physical NIT when I enabled SRIOV on it. 00:10:01.799 --> 00:10:04.700 So, some different modes that we can use for these virtual functions. 00:10:04.700 --> 00:10:17.099 Mixed means that I can support both SRIOV adapters and I can support ones that don't support SRIOV, right, or VMs that are supporting both, right? 00:10:17.099 --> 00:10:24.500 SRIOV only means I'm only using, you know, physical adapters to support it and only communicating with VMs that support SRIOV. 00:10:24.500 --> 00:10:33.000 Non-SRIOV mode means I am not using SRIOV on the adapter and I'm not sending any SRIOV traffic to any VMs that are using SRIOV. 00:10:33.000 --> 00:10:44.599 So, that means that it's all going through the normal channel, from the adapter, through the kernel, through the port group, to the virtual NIC on the VM. 00:10:44.599 --> 00:10:48.900 So, some common known issues, again, VMware tools. 00:10:48.900 --> 00:10:54.099 If your VMware tools aren't installed, you're not going to be able to use the SRIOV adapter on that virtual machine. 00:10:54.099 --> 00:11:00.500 If you don't have SRIOV enabled on a host, you won't even have that adapter as an option to install on a virtual machine. 00:11:00.500 --> 00:11:06.400 So, make sure you have VMware tools installed and make sure that the current version for the version of EFXI you're running. 00:11:06.400 --> 00:11:08.700 Check the compatibility list, right? 00:11:08.700 --> 00:11:12.799 You don't want to use a device that's not on the hardware compatibility list. 00:11:12.799 --> 00:11:14.900 You don't know how the kernel is going to respond to it. 00:11:14.900 --> 00:11:17.799 You don't know if there's going to be bugs or issues. 00:11:17.799 --> 00:11:29.599 VMware, when they release updates for the kernel, for the host, aren't going to worry about this non-signed driver working or not working from their update. 00:11:29.599 --> 00:11:38.000 If they're releasing updates, they're going to try and write them in such a way that they're compatible and compliant with the rules they set for their vendors, 00:11:38.000 --> 00:11:43.299 the ones that are using the correct APIs and the ones that are on their hardware compatibility list. 00:11:43.299 --> 00:11:49.799 If some vendor is trying to sneak in and not pay the money to be on the hardware compatibility list to not be tested by VMware, 00:11:49.799 --> 00:11:54.500 they're also not being tested to know if they wrote to VMware specifications. 00:11:54.500 --> 00:12:00.200 And so, then when VMware runs updates, they're not worried about people who didn't write their specifications. 00:12:00.200 --> 00:12:03.099 And so, their update might break your adapter. 00:12:03.099 --> 00:12:11.700 This function that used to work but wasn't advertised, it wasn't a documented function that people knew was there, they can get rid of it. 00:12:11.700 --> 00:12:15.500 They can change it, and it'll start breaking things because they didn't say, 00:12:15.500 --> 00:12:18.400 this is a supported function that you can write to, things like that. 00:12:18.400 --> 00:12:21.799 So, make sure your hardware is on the hardware compatibility list. 00:12:21.799 --> 00:12:23.400 Resource management and limits. 00:12:23.400 --> 00:12:30.000 So, you can run these systems, and you might want to enable logging. 00:12:30.000 --> 00:12:32.400 You might want to enable monitoring of the health. 00:12:32.400 --> 00:12:36.200 You want to make sure that you are keeping track of what's happening. 00:12:36.200 --> 00:12:38.200 What is the utilization on the card? 00:12:38.200 --> 00:12:40.500 Because, again, this card is doing a lot of work on its own. 00:12:40.500 --> 00:12:46.099 We're not going to the CPU anymore because this card has, you know, it's a smart adapter. 00:12:46.099 --> 00:12:50.700 It has a chip on there to do all this work and verify all these settings for me. 00:12:50.700 --> 00:12:57.500 But it's not as big and huge a chip as the CPU that, you know, from Intel that's running the whole server. 00:12:57.500 --> 00:13:01.500 So, it should be, you know, sufficient for running your workloads. 00:13:01.500 --> 00:13:06.599 But if your workloads become heavy, and you have a lot of different tasks you've assigned it to do, 00:13:06.599 --> 00:13:13.400 you've given it, you know, a larger number of VRFs, a lot of custom options, and all these different VFs. 00:13:13.400 --> 00:13:16.299 Then, all of a sudden, that is going to increase the load on the card. 00:13:16.299 --> 00:13:20.400 You might start seeing the card becoming overloaded and getting slower performance on there. 00:13:20.400 --> 00:13:22.599 And then, of course, logging limitations. 00:13:22.599 --> 00:13:28.299 So, logging is a resource-intensive operation, whether it be on the kernel, whether it be from a card. 00:13:28.299 --> 00:13:31.200 Logging is a resource-intensive operation. 00:13:31.200 --> 00:13:38.299 So, the more you log, the more detail you log, the more overhead logging is going to create. 00:13:38.299 --> 00:13:44.000 Now, sometimes that overhead is absolutely necessary and worth it because you've got a problem you need to fix. 00:13:44.000 --> 00:13:46.599 And you need that logging to fix that problem. 00:13:46.599 --> 00:13:50.000 But once you fix the problem, once the logging is no longer needed, 00:13:50.000 --> 00:13:56.299 lower the logging level to whatever is acceptable in your organization from being that max debugging. 00:13:56.400 --> 00:13:57.900 So, maybe I just want information now. 00:13:57.900 --> 00:14:01.200 I just want, if there's a problem, there's a warning, let me know, right? 00:14:01.200 --> 00:14:03.700 Don't give me every little detail, though, for every event. 00:14:03.700 --> 00:14:07.000 Just give me, you know, warnings, give me events, give me critical problems. 00:14:07.000 --> 00:14:09.700 So, don't give me all that little detailed information. 00:14:09.700 --> 00:14:15.599 Now, when there's a problem and we're, you know, trying to troubleshoot it, go to debug mode and give me everything. 00:14:15.599 --> 00:14:21.200 And then, again, once I fix that problem, I'm going to lower it back down to only giving me warnings 00:14:21.200 --> 00:14:24.000 and only giving me, like, critical events and things like that. 00:14:24.000 --> 00:14:26.200 So, know what to log and when to log it. 00:14:26.200 --> 00:14:31.799 Also, it's a good idea to send your logs off the server because sometimes it'll come up in the server, it'll crash, 00:14:31.799 --> 00:14:33.500 and you're like, what caused this crash? 00:14:33.500 --> 00:14:35.500 Well, the logs are on your crash server. 00:14:35.500 --> 00:14:37.299 Go figure that one out. 00:14:37.299 --> 00:14:45.200 But if we're dumping logs to a syslog server, maybe that log, that error got logged to the syslog server before the system crashed. 00:14:45.200 --> 00:14:49.299 We can see what's happening and led up to the crash to help us understand what happened there. 00:14:49.299 --> 00:14:51.500 So, logging limitations are great. 00:14:51.500 --> 00:14:56.200 So, that is our single-route IO virtualization. 00:14:56.200 --> 00:14:57.900 Very cool feature. 00:14:57.900 --> 00:15:04.599 If you have a network card, it's going to be a more expensive, higher-end network card that supports the function, 00:15:04.599 --> 00:15:07.799 and we have it on the hardware compatibility list from VMware, 00:15:07.799 --> 00:15:11.299 it's actually a pretty cool thing that we can enable on our ESXi host now.