WEBVTT 00:00:00.000 --> 00:00:04.760 Welcome back to vSphere 8 in the real world, and now we're going to discuss upgrading hosts. 00:00:04.760 --> 00:00:11.840 So in a previous lesson, we discussed upgrading vCenter, and we discussed that when you're upgrading your vSphere environment as a whole, 00:00:11.840 --> 00:00:18.600 the first thing you need to do is update your vCenter server. Once your vCenter server has been updated, then you update your ESXi hosts. 00:00:18.600 --> 00:00:21.900 Then you update your virtual machine hardware, and then you update your VMware tools. 00:00:21.900 --> 00:00:24.799 So here we're going to discuss upgrading your ESXi hosts. 00:00:24.799 --> 00:00:30.959 So we're going to go under the assumption that we've already updated vCenter to the current version that we're moving to, 00:00:30.959 --> 00:00:36.599 and now we're going to upgrade the hosts. But you want to do it in this order because, one, you don't want any kind of data loss 00:00:36.599 --> 00:00:42.000 due to incompatibilities between the host and the vCenter server, and you don't want access loss. 00:00:42.000 --> 00:00:49.599 Meaning you don't want the host to no longer be able to access your vCenter server, and they won't be able to communicate with the server. 00:00:49.599 --> 00:00:57.799 They won't be able to participate in your cluster, right? So you want to make sure that you don't upgrade your host before you upgrade vCenter 00:00:57.799 --> 00:00:59.500 because then you could run into some issues. 00:01:00.900 --> 00:01:07.300 So methods for doing this upgrade, so an external disk, we can do scripted upgrades, we can do it from a CLI, 00:01:07.300 --> 00:01:11.599 and we can also use the lifecycle manager to upgrade our ESXi hosts. 00:01:11.599 --> 00:01:15.599 So external disk means that, like a USB drive, right, of some sort. 00:01:15.599 --> 00:01:23.500 So I might, I'm definitely going to download the ISO, so I have the installation, you know, tool, the files to do the upgrade. 00:01:23.500 --> 00:01:32.300 And then I can choose to either extract it and put the files onto a, like, FTP server or something like that and pull it over the network. 00:01:32.300 --> 00:01:36.599 So if I'm doing some kind of a scripted upgrade, then I might do that. 00:01:37.900 --> 00:01:43.699 Or an external drive means I basically made a bootable USB drive, bootable thumb drive of some sort, 00:01:43.699 --> 00:01:47.199 and I'm going to boot up from that, and I'm going to, you know, then go through and do my installation. 00:01:47.199 --> 00:01:52.099 I can boot from a thumb drive and still do a scripted installation. 00:01:52.099 --> 00:01:58.599 Just when the setup starts, I'm going to interrupt the startup, and I'm going to point it to where my answer file is on the network 00:01:58.599 --> 00:02:04.300 and the answer file that it could then have more information about where the rest of the files are and things like that for the actual installation. 00:02:04.300 --> 00:02:13.199 So I could boot from a thumb drive, point it to an answer file, and have it pull the installation files over the network if I wanted to. 00:02:13.399 --> 00:02:14.000 All right. 00:02:15.699 --> 00:02:21.100 I can also use Lifecycle Manager, and we'll look at how to do an upgrade like that with Lifecycle Manager in just a moment. 00:02:23.600 --> 00:02:25.699 So obviously, hardware compatibility. 00:02:26.199 --> 00:02:37.199 It is becoming more and more common that hardware that is compatible with an older version of vSphere is not compatible with newer versions, especially processors. 00:02:37.199 --> 00:02:42.699 So you'll see a lot of situations where I was running 6.7, and now I'm going to upgrade to 8. 00:02:42.699 --> 00:02:46.199 I'm going to do a big jump like that, so I'll go from 6.7 to 7 to 8. 00:02:46.199 --> 00:02:48.500 The first thing you want to do is check your hardware. 00:02:48.500 --> 00:02:56.699 Do I have a compatible, you know, version 8 driver for my HBAs, for my NICs, for all the various devices that are on my system? 00:02:56.699 --> 00:02:58.399 The big thing is processors. 00:02:58.399 --> 00:02:59.800 My processors support it. 00:02:59.800 --> 00:03:07.399 If I have an older proc in my system, it might run fantastic with 6.7, and it's not going to be compatible with 8, for example. 00:03:07.399 --> 00:03:09.699 You might have issues like TPM. 00:03:09.699 --> 00:03:10.199 You might have that. 00:03:10.199 --> 00:03:13.899 It's just the model in general is not supported, all right? 00:03:13.899 --> 00:03:22.199 So make sure that you do check the hardware compatibility list before we start doing our upgrade, before you even really start planning your upgrade. 00:03:22.199 --> 00:03:25.199 The first step to a plan is, is my hardware compatible? 00:03:25.199 --> 00:03:34.000 Because if you need to purchase new hardware, I have seen many scenarios where everything is perfect except my processors. 00:03:34.000 --> 00:03:44.000 And I can buy new processors, and my motherboards support a newer processor that is compatible with the version of ESXi that I want to go to. 00:03:44.000 --> 00:03:46.199 So I may not have to throw out my entire host. 00:03:46.199 --> 00:03:48.399 I might need to buy a new processor. 00:03:48.399 --> 00:03:49.699 I've seen things like that. 00:03:49.699 --> 00:03:50.899 So just keep that in mind. 00:03:50.899 --> 00:03:51.699 Check your procs. 00:03:51.699 --> 00:03:58.699 Make sure you're, you know, having things that are, you know, going to be compatible with the version you're trying to go to. 00:03:58.699 --> 00:04:06.699 There are ways you'll see online where I can say, oh, my proc isn't compatible with ESXi 8, but I can do this and ignore that. 00:04:06.699 --> 00:04:09.199 You don't want to do that in a production environment. 00:04:09.199 --> 00:04:14.399 If you're running it in a lab environment, maybe you feel comfortable doing that. 00:04:14.399 --> 00:04:18.200 But I would like to know that what's in my lab is what I'm running in production. 00:04:18.200 --> 00:04:28.200 And so if I'm running an old proc in production and something new, sorry, an old proc in my lab and something new in production, that isn't necessarily a great thing. 00:04:28.200 --> 00:04:32.700 You want the two to be very, very similar, especially things like processors. 00:04:32.700 --> 00:04:37.700 So going through and ignoring, you know, the incompatibility isn't recommended. 00:04:37.700 --> 00:04:43.700 And if you end up doing that and you call VMware for support, first thing they're going to say is your proc's not supported and they're going to be out. 00:04:43.700 --> 00:04:49.200 Because they're going to ask you for their text file, which has all the logs and things like that, and they'll know right away that it's not supported. 00:04:49.200 --> 00:04:53.700 So I strongly encourage you to make sure that all your hardware is supported. 00:04:53.700 --> 00:04:56.200 Then make sure you have enough of it. 00:04:56.200 --> 00:04:58.200 So minimum two virtual CPUs. 00:04:58.200 --> 00:05:01.700 You want to have the NXXD bits turned on. 00:05:01.700 --> 00:05:04.700 You want to have, you know, virtualization support turned on. 00:05:04.700 --> 00:05:06.700 At least four gigs of RAM. 00:05:06.700 --> 00:05:08.200 Really better to have at least eight. 00:05:08.200 --> 00:05:09.700 And that's just to install the operating system. 00:05:09.700 --> 00:05:17.200 If you have four gigs of RAM and you're trying to run virtual machines on there, you're probably going to end up kicking yourself into not being happy with yourself. 00:05:17.200 --> 00:05:19.700 Even in a lab environment, you probably don't want to do that. 00:05:19.700 --> 00:05:21.200 All right. 00:05:21.200 --> 00:05:24.700 You want, again, hardware virtualization support enabled. 00:05:24.700 --> 00:05:26.200 Guzzy disks are non-RAID-less. 00:05:26.200 --> 00:05:32.200 You want to install the operating system on something that's not RAID. 00:05:32.200 --> 00:05:34.200 You want to install, like, on a disk. 00:05:34.200 --> 00:05:43.200 It used to be not that long ago that people were installing ESXi on little SD drives that were, like, you know, installed on the motherboard. 00:05:43.200 --> 00:05:46.200 There'd be a slot on the motherboard for these SD drives. 00:05:46.200 --> 00:05:47.700 No longer supported. 00:05:47.700 --> 00:05:52.200 They kind of went out for a little bit, and then VMware kind of brought them back a little bit. 00:05:52.700 --> 00:05:56.700 Because the OS was just chewing through them because it was logging quite a bit. 00:05:56.700 --> 00:06:02.700 So, VMware actually then went and tweaked the OS to log a little less and write its logs differently. 00:06:02.700 --> 00:06:05.200 So, it didn't eat up those SD drives so quickly. 00:06:05.200 --> 00:06:08.700 But now VMware is, again, saying no SD drives. 00:06:08.700 --> 00:06:11.200 And what they recommend is that you use, like, an NVMe drive. 00:06:11.200 --> 00:06:16.200 They recommend that you get some kind of a, you know, solid state disk, NVMe, and install on that. 00:06:16.200 --> 00:06:25.700 And manufacturers are now making PCI cards that will have two NVMe drives that are just automatically – slide the card in. 00:06:25.700 --> 00:06:29.200 It automatically has two NVMe drives. 00:06:29.200 --> 00:06:33.700 They're automatically mirrored in a RAID 0 and you just install onto that. 00:06:33.700 --> 00:06:36.700 And they're kind of designed just for installing ESXi on. 00:06:36.700 --> 00:06:41.700 And they are obviously compatible with ESXi on the ESXi hard compatibility list. 00:06:41.700 --> 00:06:43.200 So, that's something to consider. 00:06:43.200 --> 00:06:55.200 If you are using SD drives currently, you can add some 512-gig SD drives – or, I'm sorry, NVMe drives that you can install your OS on instead pretty easily. 00:06:57.200 --> 00:06:58.700 So, storage issues. 00:06:58.700 --> 00:07:00.200 Make sure you have enough storage. 00:07:00.200 --> 00:07:01.200 Make sure that it's big enough. 00:07:01.200 --> 00:07:08.200 Again, if you're, you know, before using SD drives that were just small, 8 is bigger and you don't want to install the SD drives anymore. 00:07:08.200 --> 00:07:10.200 You want to have enough room for all your logins. 00:07:10.200 --> 00:07:17.200 Maybe you want to put your logs onto, like, a NAS storage device or NFS or, better yet, a syslog server. 00:07:18.200 --> 00:07:21.200 So, remove the fiber channel systems before the upgrade. 00:07:21.200 --> 00:07:22.200 That's a recommendation. 00:07:22.200 --> 00:07:24.200 It's not a requirement. 00:07:24.200 --> 00:07:39.200 What happens – and it happens, unfortunately, probably more than people want to, you know, talk about – is when they go to install, if you have a fiber channel HBA that the installer can see, it's going to show you all the fiber channel LUNs also. 00:07:40.200 --> 00:07:47.200 And I've heard about, more than once, people choosing the wrong disk to install to, and they're not choosing a local disk on the system. 00:07:47.200 --> 00:07:50.200 They choose that fiber channel LUN, and it wipes the LUN out. 00:07:50.200 --> 00:07:58.200 And so, now, all of a sudden, you've lost a bunch of data, and you have to rebuild the LUN, and you have to restore VMs from backups, and systems are down, and things like that. 00:07:58.200 --> 00:08:05.200 So, removing the HBA from the system, disabling it before you go in, it's just a good practice. 00:08:05.200 --> 00:08:08.200 You don't accidentally install over it and format it and things like that. 00:08:09.200 --> 00:08:25.200 If you're extremely confident in yourself, and you know you're going to choose the right internal disk, then – and then, also, also, during the installer, it'll ask you if you want to format VMFS data stores, and you have to say no to that. 00:08:25.200 --> 00:08:30.200 Otherwise, again, it can go off to your LUN and format your LUNs for your proper channel adapters. 00:08:30.200 --> 00:08:33.200 So, you can see there's a lot of different gotchas in there. 00:08:33.200 --> 00:08:38.200 So, it's better off just not choosing – or not having the LUN even available. 00:08:38.200 --> 00:08:43.200 You can just go into the SAN and stop presenting all the LUNs to that host. 00:08:43.200 --> 00:08:57.200 And when you go and you boot up to do the installation, and it shows you the disk you can install to, if you're seeing your fiber channel LUNs, stop the – you know, stop the installation, go back to the SAN, unplug the cables from the HBA. 00:08:57.200 --> 00:09:10.200 You can get – you know, if you're physically near it, whatever you need to do, just make sure you're not installing onto VMFS data stores that are on your fiber channel SAN or formatting them and things like that. 00:09:10.200 --> 00:09:12.200 That's why that idea of removing that's there. 00:09:12.200 --> 00:09:21.200 Not that it will cause a problem with the installation, but you might end up losing data accidentally. 00:09:21.200 --> 00:09:23.200 So, some pro tips here. 00:09:23.200 --> 00:09:29.200 Before you start a backup or before you start the installation, backup everything you can. 00:09:29.200 --> 00:09:37.200 If you want to retain logs and things like that, backup the log files, backup data stores, backup VMs, backup anything on local storage. 00:09:37.200 --> 00:09:40.200 Okay, backup anything on local storage. 00:09:40.200 --> 00:09:44.200 Make sure you have a backup of any ISOs that are in local storage. 00:09:44.200 --> 00:09:50.200 If you have any VMs, if you have any templates on local storage, things like that, back all that stuff up. 00:09:50.200 --> 00:09:53.200 You may plan on not saying to format anything. 00:09:53.200 --> 00:09:59.200 You're just going to go in and install and leave your VMFS data stores, you know, un-reformatted. 00:09:59.200 --> 00:10:00.200 Don't take the chance. 00:10:00.200 --> 00:10:03.200 Just back everything up, okay? 00:10:03.200 --> 00:10:08.200 Any VMs that are on that host, if you can't migrate them off, you'll need to shut them down. 00:10:08.200 --> 00:10:15.200 If they are on there, before you start the installation, you should be able to put that host in a maintenance mode and evacuate all the VMs off of it that way. 00:10:15.200 --> 00:10:19.200 But if you're using local storage, you might have to shut them down. 00:10:19.200 --> 00:10:23.200 And then be very, very careful not to format that data store. 00:10:23.200 --> 00:10:27.200 Back things up first. 00:10:27.200 --> 00:10:34.200 So, for remote management, here they're showing some, you know, ILOs, DRACs, IBM RSA tools. 00:10:34.200 --> 00:10:38.200 Basically, those give you access to the host remotely. 00:10:38.200 --> 00:10:40.200 So, it's like you're sitting at the host at a console. 00:10:41.200 --> 00:10:47.200 You can mount the ISO into the ILO or into the DRAC, and it will look like a CD-ROM drive to the device, 00:10:47.200 --> 00:10:51.200 and it tells you what version of Java these different versions of ILOs and DRACs use. 00:10:51.200 --> 00:10:52.200 All right? 00:10:52.200 --> 00:10:58.200 So, those are pretty common in, you know, any kind of enterprise-level server. 00:10:58.200 --> 00:11:02.200 The ISO, so you're going to download the ISO from VMware's website. 00:11:02.200 --> 00:11:07.200 You will want to view the MD5 checksum and make sure that it's, you know, integrity is good. 00:11:07.200 --> 00:11:14.200 And then you can either use a – burn it to a CD or DVD and put it into the drive of the host, 00:11:14.200 --> 00:11:17.200 if your host has a CD and DVD drive still. 00:11:17.200 --> 00:11:19.200 Make a USB bootable drive. 00:11:19.200 --> 00:11:21.200 That is a very common way. 00:11:21.200 --> 00:11:24.200 You can do a network boot and PXE it. 00:11:24.200 --> 00:11:26.200 You can use a fiber channel controller. 00:11:26.200 --> 00:11:31.200 And you can also, as we mentioned, mount the ISO into your ILO or your DRAC. 00:11:31.200 --> 00:11:36.200 So, once you have that mounted and it's visible, then you want to configure the boot process. 00:11:36.200 --> 00:11:43.200 You want to go into your BIOS and you want to configure the BIOS to boot from whatever device it is that you have the ISO on, 00:11:43.200 --> 00:11:47.200 either it be a USB drive or it be you're going to do a PXE boot 00:11:47.200 --> 00:11:52.200 or if you're going to be booting from your remote management card, your ILO, your DRAC. 00:11:52.200 --> 00:11:53.200 All right? 00:11:53.200 --> 00:11:59.200 You're going to select that device to be your – this is after you disconnected the LUNs from the fiber channel HBAs, 00:11:59.200 --> 00:12:01.200 things like that. 00:12:01.200 --> 00:12:02.200 You're going to reboot the host. 00:12:02.200 --> 00:12:05.200 You're going to go into your BIOS, make your changes. 00:12:05.200 --> 00:12:08.200 Select where you want it to upgrade, right? 00:12:08.200 --> 00:12:11.200 So, where are you going to install the upgrade to? 00:12:11.200 --> 00:12:14.200 Press F11 to start the installation. 00:12:14.200 --> 00:12:16.200 Eject the disk when it's done. 00:12:16.200 --> 00:12:17.200 Reboot the host. 00:12:17.200 --> 00:12:20.200 And then it will come back up again when your upgrade is completed. 00:12:20.200 --> 00:12:23.200 And then you want to do things like check your logs and make sure that there's no errors 00:12:23.200 --> 00:12:29.200 and that all the drivers got installed correctly without error and things like that. 00:12:29.200 --> 00:12:33.200 So, here's an example of a startup script, you know, how you would do that. 00:12:33.200 --> 00:12:35.200 You're going to start the installation, right? 00:12:35.200 --> 00:12:36.200 So, you could boot to the ISO. 00:12:36.200 --> 00:12:38.200 You could boot to a thumb drive, something like that. 00:12:38.200 --> 00:12:40.200 So, you're going to start your installation. 00:12:40.200 --> 00:12:45.200 In the installer, you're going to do a Shift-O, right, to edit your boot options. 00:12:45.200 --> 00:12:49.200 And then you're going to point it to where's your kickstart file. 00:12:49.200 --> 00:12:51.200 So, here's an example. 00:12:51.200 --> 00:12:58.200 You can say KS, which is your kickstart file, equals – and then here's an HTTPS path or HTTP path to that. 00:12:58.200 --> 00:13:05.200 You can also point it to, like, a TFTP server or something like that as well. 00:13:05.200 --> 00:13:10.200 So, after your installation is done, so you've booted up to the ISO and you've gone through the installation, 00:13:10.200 --> 00:13:16.200 so a few quick little clicks, now it's rebooted and you're back in your ESXi host, 00:13:16.200 --> 00:13:18.200 go ahead and view your log files. 00:13:18.200 --> 00:13:21.200 Make sure that, again, there's no errors trying to install any devices. 00:13:21.200 --> 00:13:27.200 There's no errors, you know, getting anything up and running and that you are completely on the new version 00:13:27.200 --> 00:13:30.200 that you want to be on and all your device drivers are loaded successfully, 00:13:30.200 --> 00:13:32.200 all your devices are working successfully. 00:13:32.200 --> 00:13:40.200 You don't want to try and do your upgrade and find that your HPA controller doesn't have a driver associated with it 00:13:40.200 --> 00:13:44.200 or, you know, it's not compatible on the compatibility list for ESXi 8, 00:13:44.200 --> 00:13:48.200 and so therefore doesn't load a driver for it, so that device isn't going to work, 00:13:48.200 --> 00:13:53.200 or, you know, a 10 gig NIC or something like that, right? 00:13:53.200 --> 00:13:56.200 Make sure that the server comes back up in vCenter. 00:13:56.200 --> 00:13:59.200 You might need to go in and tell it to connect back into vCenter. 00:13:59.200 --> 00:14:05.200 Now, when you first do your installation, your host is going to be in an evaluation mode for up to 60 days. 00:14:05.200 --> 00:14:09.200 I recommend you install your licenses right away. 00:14:09.200 --> 00:14:13.200 People will sometimes get complacent when they do these upgrades and then, okay, 00:14:13.200 --> 00:14:15.200 they go through and upgrade all their hosts and they get, you know, vCenter updated 00:14:15.200 --> 00:14:19.200 and all their hosts are upgraded and everything's upgraded, and they feel very confident, 00:14:19.200 --> 00:14:25.200 okay, I have three or two months, I have 60 days, two months to get in there and add the licenses, 00:14:25.200 --> 00:14:29.200 and they're making configurations and they're all excited to see the new version. 00:14:29.200 --> 00:14:34.200 Then later, maybe at 30 days or 45 days or whatever, they add their license, 00:14:34.200 --> 00:14:41.200 not realizing that they've enabled features that they don't pay for, they don't have paid for, 00:14:41.200 --> 00:14:45.200 and when they add their license right away, those features stop working. 00:14:45.200 --> 00:14:49.200 So maybe they did something like, like, oh, look, I can create distributed virtual switches now. 00:14:49.200 --> 00:14:53.200 They can do all their NIC switching, you know, to distribute virtual switches, 00:14:53.200 --> 00:14:56.200 and they're all excited about that, and then they add their license, 00:14:56.200 --> 00:14:59.200 and then their distributed virtual switches stop working and go away, 00:14:59.200 --> 00:15:02.200 because they don't have an Enterprise Plus license to pay for that. 00:15:02.200 --> 00:15:04.200 So that's a problem. 00:15:04.200 --> 00:15:06.200 So make sure that you add your licenses right away, 00:15:06.200 --> 00:15:10.200 so that way you don't enable features that you haven't paid for. 00:15:10.200 --> 00:15:11.200 Then upgrade your VMs. 00:15:11.200 --> 00:15:12.200 So first upgrade vCenter. 00:15:12.200 --> 00:15:14.200 We discussed that. 00:15:14.200 --> 00:15:18.200 Then I upgrade my hosts, and I got all my hosts configured and got my licenses installed. 00:15:18.200 --> 00:15:20.200 Now it's time to upgrade my hosts, my VMs. 00:15:20.200 --> 00:15:23.200 I'm going to upgrade the virtual hardware on my VMs, 00:15:23.200 --> 00:15:26.200 and then I'm going to upgrade the VMware tools on my VMs. 00:15:26.200 --> 00:15:31.200 That's the last step after your installation. 00:15:31.200 --> 00:15:33.200 So some data store concerns. 00:15:33.200 --> 00:15:38.200 So with VMFS, new versions of ESXi can bring along new versions of VMFS, 00:15:38.200 --> 00:15:41.200 which is the file system used, you know, native for ESXi. 00:15:41.200 --> 00:15:43.200 So you upgrade your ESXi. 00:15:43.200 --> 00:15:47.200 It may require an update to VMFS. 00:15:47.200 --> 00:15:49.200 What updates are required for your environment? 00:15:49.200 --> 00:15:51.200 So a mandatory upgrade. 00:15:51.200 --> 00:15:54.200 If you have VMFS 3, you're going to have to upgrade your VMFS, 00:15:54.200 --> 00:15:58.200 and it's going to automatically do that upgrade for you when you boot into the 00:15:58.200 --> 00:16:00.200 new version, like version 7. 00:16:00.200 --> 00:16:07.200 Now, if you have VMFS 5 and you want to go to VMFS 6, not going to be as easy. 00:16:07.200 --> 00:16:10.200 You're going to need to have a data store that you can move all the VMs to 00:16:10.200 --> 00:16:12.200 to evacuate a data store. 00:16:12.200 --> 00:16:17.200 Then you can go ahead and basically reformat it as version VMFS 6. 00:16:18.200 --> 00:16:20.200 Then you can move VMs to it and go to the next. 00:16:20.200 --> 00:16:24.200 If you don't have a data store big enough to get all your VMs off of one of 00:16:24.200 --> 00:16:28.200 your other data stores, then upgrading to 6 is going to be a challenge for you. 00:16:28.200 --> 00:16:32.200 You need a place to put the VMs because it's going to basically wipe the 00:16:32.200 --> 00:16:34.200 drives and do that upgrade. 00:16:34.200 --> 00:16:36.200 So make sure you have a place to put your VMs. 00:16:36.200 --> 00:16:38.200 That's why it's good to not, you know, 00:16:38.200 --> 00:16:43.200 overallocate the space on your storage arrays because a lot of different 00:16:43.200 --> 00:16:48.200 things are going to require you to do some upgrades there. 00:16:48.200 --> 00:16:50.200 So we discussed host upgrade process. 00:16:50.200 --> 00:16:52.200 We discussed compatibility, 00:16:52.200 --> 00:16:56.200 making sure that our hardware is compatible with the version of ESX 5 you 00:16:56.200 --> 00:16:57.200 want to go to. 00:16:57.200 --> 00:17:00.200 We discussed storage, the different kinds of storage we can use, 00:17:00.200 --> 00:17:05.199 the different kinds of media that we can boot up to from our ISO image. 00:17:05.199 --> 00:17:07.199 We discussed doing a scripted installation. 00:17:07.199 --> 00:17:10.199 And we discussed upgrading data stores.