1 00:00:00,000 --> 00:00:08,640 Sometimes while booting, it goes wrong. 2 00:00:08,640 --> 00:00:10,360 Now, what can you do? 3 00:00:10,360 --> 00:00:12,600 Well, first, you can boot into one 4 00:00:12,600 --> 00:00:16,360 of the troubleshooting targets, like systemd.unit 5 00:00:16,360 --> 00:00:19,160 is emergency.target or systemd.unit 6 00:00:19,160 --> 00:00:22,760 is rescue.target, just the way we have just seen. 7 00:00:22,760 --> 00:00:27,559 It's also possible to use init is bin bash as an alternative. 8 00:00:27,559 --> 00:00:31,920 That dives in before you even get to your systemd. 9 00:00:31,920 --> 00:00:34,000 But if a bootloader no longer shows, 10 00:00:34,000 --> 00:00:36,360 a rescue disk can be used. 11 00:00:36,360 --> 00:00:38,400 Now, the good news is that normally, 12 00:00:38,400 --> 00:00:42,520 any Linux installation disk can be used as a rescue disk. 13 00:00:42,520 --> 00:00:45,119 Before showing you, let me draw an overview 14 00:00:45,119 --> 00:00:47,400 of the different options of what you can 15 00:00:47,400 --> 00:00:50,279 use in which specific case. 16 00:00:50,279 --> 00:00:52,439 So before we explore the boot procedure, 17 00:00:52,439 --> 00:00:54,439 what can you do if things go wrong? 18 00:00:54,520 --> 00:00:58,720 Well, if you don't see grub at all, there's only one option, 19 00:00:58,720 --> 00:01:02,360 and that is going to be the rescue disk. 20 00:01:02,360 --> 00:01:04,519 So you need to boot from an external media 21 00:01:04,519 --> 00:01:06,040 to get access to your system. 22 00:01:06,040 --> 00:01:08,680 If you do see grub, but your kernel doesn't load, 23 00:01:08,680 --> 00:01:14,040 there's another option, and that is the rescue kernel. 24 00:01:14,040 --> 00:01:16,800 The rescue kernel is offered in the grub boot menu, 25 00:01:16,800 --> 00:01:19,199 and it offers an alternative. 26 00:01:19,199 --> 00:01:22,919 Then your kernel and your init ramfs are loading, 27 00:01:22,919 --> 00:01:25,480 and you can jump into the system 28 00:01:25,480 --> 00:01:28,360 directly after loading the init ramfs. 29 00:01:28,360 --> 00:01:32,040 On Red Hat, there is an option rd.break, 30 00:01:32,040 --> 00:01:35,199 but that option is Red Hat only, and for that reason, 31 00:01:35,199 --> 00:01:37,080 not so very powerful. 32 00:01:37,080 --> 00:01:40,800 On generic Linux, you can also use the boot argument, 33 00:01:40,800 --> 00:01:45,919 init is slash bin slash bash. 34 00:01:45,919 --> 00:01:49,080 So both of these options are added as kernel arguments, 35 00:01:49,120 --> 00:01:53,040 and the most generic kernel argument is init is bin bash. 36 00:01:53,040 --> 00:01:56,440 It skips your systemd, it throws you a bash shell, 37 00:01:56,440 --> 00:01:59,680 and you don't even have to provide a root user password. 38 00:01:59,680 --> 00:02:03,000 Once systemd is loaded, things are getting more complicated. 39 00:02:03,000 --> 00:02:11,639 So you would use systemd.unit equals emergency.target, 40 00:02:11,639 --> 00:02:14,600 and this emergency target will get you in right 41 00:02:14,600 --> 00:02:15,759 before early boot. 42 00:02:15,759 --> 00:02:18,639 So you really have a minimum of services 43 00:02:18,639 --> 00:02:21,199 which maximizes the chance of you getting back 44 00:02:21,199 --> 00:02:23,559 in a normal operational state. 45 00:02:23,559 --> 00:02:26,479 Now, in troubleshooting, it's always good to get into 46 00:02:26,479 --> 00:02:28,800 as much of your system as you can, 47 00:02:28,800 --> 00:02:34,240 and that is why there is also the rescue target. 48 00:02:34,240 --> 00:02:36,919 And a rescue target is bringing you all the way 49 00:02:36,919 --> 00:02:38,160 up to the early boot, 50 00:02:38,160 --> 00:02:40,800 you just don't load the services here. 51 00:02:40,800 --> 00:02:43,520 So these are your different options for troubleshooting, 52 00:02:43,520 --> 00:02:46,119 and in most cases, using one of them 53 00:02:46,119 --> 00:02:49,119 will get you back into your system. 54 00:02:49,119 --> 00:02:51,600 All right, let me demonstrate, 55 00:02:51,600 --> 00:02:56,000 and I'm going to demonstrate how you can use a rescue disk. 56 00:02:56,000 --> 00:02:59,960 In order to do so, I need to connect my disk, 57 00:02:59,960 --> 00:03:02,279 and I'm doing that in VMware. 58 00:03:02,279 --> 00:03:05,479 Your virtualization software is where you present your disk, 59 00:03:05,479 --> 00:03:08,279 and normally, it will find the disk 60 00:03:08,279 --> 00:03:10,240 that you have used for installation, 61 00:03:10,240 --> 00:03:12,399 as you can see right here, grayed out, 62 00:03:12,399 --> 00:03:15,600 and when you use connect CD, DVD, 63 00:03:15,600 --> 00:03:17,679 it should be connected. 64 00:03:17,679 --> 00:03:20,960 And then you need to make sure that you boot from it. 65 00:03:20,960 --> 00:03:23,320 Now, there are different ways to boot 66 00:03:23,320 --> 00:03:26,600 into your specific environment. 67 00:03:26,600 --> 00:03:29,520 I am going to use sudo minus I, 68 00:03:29,520 --> 00:03:33,720 and I'm going to show you the EFI boot manager. 69 00:03:33,720 --> 00:03:37,399 So EFI boot manager is showing me what? 70 00:03:37,399 --> 00:03:39,880 Showing me all these different options, 71 00:03:39,880 --> 00:03:43,960 and I want to boot from option number one. 72 00:03:43,960 --> 00:03:51,679 So EFI boot MGR minus N0001, 73 00:03:51,679 --> 00:03:54,240 and then I'm going to reboot, 74 00:03:54,240 --> 00:03:58,000 and as you can see, I'm booting from my rescue disk. 75 00:03:58,000 --> 00:04:01,759 So the alternative of using EFI boot manager 76 00:04:01,759 --> 00:04:03,880 is to press escape at the right moment, 77 00:04:03,880 --> 00:04:07,559 and let's face it, the reason you are going to a rescue disk 78 00:04:07,559 --> 00:04:09,279 is probably because you don't get 79 00:04:09,279 --> 00:04:12,520 into a fully operational environment anymore, 80 00:04:12,520 --> 00:04:14,160 and then you need to press escape 81 00:04:14,160 --> 00:04:15,839 at the right moment while booting, 82 00:04:15,839 --> 00:04:18,480 or F2, F10, or F12. 83 00:04:18,480 --> 00:04:20,440 The key that you are using depends 84 00:04:20,440 --> 00:04:23,239 on the hardware that you are using. 85 00:04:24,200 --> 00:04:25,200 Now, for rescue disk, 86 00:04:25,200 --> 00:04:27,600 the option that we need is troubleshooting. 87 00:04:27,600 --> 00:04:29,399 So in troubleshooting, 88 00:04:29,399 --> 00:04:32,480 you select a rescue CentOS stream system, 89 00:04:32,480 --> 00:04:34,519 in case it is CentOS, of course, 90 00:04:34,519 --> 00:04:37,359 with the other rescue disks, Ubuntu and Red Hat, 91 00:04:37,359 --> 00:04:41,519 and other distributions are offering similar solutions. 92 00:04:41,519 --> 00:04:43,799 So the idea is that you are going to load 93 00:04:43,799 --> 00:04:47,040 your operating system from the installation disk, 94 00:04:47,040 --> 00:04:50,000 and then while booting from the installation disk, 95 00:04:50,000 --> 00:04:53,959 you are going to try to address your hard disk. 96 00:04:53,959 --> 00:04:55,839 Some distributions require you 97 00:04:55,839 --> 00:04:58,079 to do a completely manual procedure 98 00:04:58,079 --> 00:05:00,679 where you need to find your information. 99 00:05:00,679 --> 00:05:05,040 Some distributions are making it a lot easier for you. 100 00:05:05,040 --> 00:05:08,320 Now, I want to show you in a generic way 101 00:05:08,320 --> 00:05:10,640 how you can access your hard disk 102 00:05:10,640 --> 00:05:12,720 from the rescue disk. 103 00:05:12,720 --> 00:05:14,519 And there, well, 104 00:05:14,519 --> 00:05:17,160 the rest of the troubleshooting would follow. 105 00:05:17,160 --> 00:05:18,679 Let's not dive into that. 106 00:05:18,679 --> 00:05:22,239 Let's just see if we can access the hard disk again. 107 00:05:22,239 --> 00:05:24,959 So here, CentOS is telling me what it is doing. 108 00:05:24,959 --> 00:05:28,559 It's attempting to find the installation 109 00:05:28,559 --> 00:05:31,880 and mount it on the directory, mnt-sysroot. 110 00:05:31,880 --> 00:05:33,160 I don't want to do that. 111 00:05:33,160 --> 00:05:35,720 I want to do option number three, 112 00:05:35,720 --> 00:05:38,399 because not all distributors are giving 113 00:05:38,399 --> 00:05:40,440 a nice option like this. 114 00:05:40,440 --> 00:05:43,320 Option number three, that is where you go to a shell, 115 00:05:43,320 --> 00:05:46,119 and that is where you can try to find your hardware. 116 00:05:46,119 --> 00:05:49,760 So LSBLK is what I'm using, and what do I see? 117 00:05:49,760 --> 00:05:52,839 I see my block devices. 118 00:05:52,839 --> 00:05:55,440 So that's good news already to start with. 119 00:05:55,440 --> 00:05:58,239 So I have my SCA1, 2, and 3. 120 00:05:58,239 --> 00:06:01,920 Now, I'm going to use VGScan. 121 00:06:01,920 --> 00:06:05,920 VGScan is LVM Volume Group Scan. 122 00:06:05,920 --> 00:06:09,720 This is going to try to find LVM information 123 00:06:09,720 --> 00:06:11,320 on your system. 124 00:06:11,320 --> 00:06:15,200 Now, when I use LSBLK again, I still see the same thing. 125 00:06:15,200 --> 00:06:18,440 But when I use VGS, I see my volume group. 126 00:06:18,440 --> 00:06:23,000 And when I use LVS, I see my logical volumes. 127 00:06:23,000 --> 00:06:27,399 And with that, there is my root logical volume. 128 00:06:27,399 --> 00:06:34,679 So let's try to give it a mount dev cs slash root on slash mnt. 129 00:06:34,679 --> 00:06:36,760 That's a temporary mount. 130 00:06:36,799 --> 00:06:41,839 And, uh-oh, we have a cannot lookup block device. 131 00:06:41,839 --> 00:06:45,119 DMessage may have more information about it. 132 00:06:45,119 --> 00:06:46,200 What is going on? 133 00:06:46,200 --> 00:06:51,000 It doesn't know about dev cs root yet. 134 00:06:51,000 --> 00:06:53,519 Now, if we get back to LSBLK, that's 135 00:06:53,519 --> 00:06:57,440 understandable because LSBLK can't show it as well. 136 00:06:57,440 --> 00:07:01,079 And if we use LS on slash dev slash cs, 137 00:07:01,079 --> 00:07:03,959 we see no such root or directory. 138 00:07:03,959 --> 00:07:09,760 And also, LS on dev mapper is showing no LVM information. 139 00:07:09,760 --> 00:07:11,640 Now, what do we need to do next? 140 00:07:11,640 --> 00:07:13,760 We need LVChange. 141 00:07:13,760 --> 00:07:17,040 Now, it's maybe wise to use LVChange minus minus help 142 00:07:17,040 --> 00:07:19,119 to find out how to use it. 143 00:07:19,119 --> 00:07:22,799 And here in the help, we can see that, oh my goodness, 144 00:07:22,799 --> 00:07:25,519 there are so many options. 145 00:07:25,519 --> 00:07:32,359 And in this so many options, I'm looking for the minus A. 146 00:07:32,359 --> 00:07:34,079 Minus A is activate. 147 00:07:34,079 --> 00:07:36,600 And that has a Y. And we need the volume group name. 148 00:07:36,600 --> 00:07:37,559 Aha. 149 00:07:37,559 --> 00:07:43,160 So LVChange cs minus A Y. 150 00:07:43,160 --> 00:07:44,920 That's what I'm looking at. 151 00:07:44,920 --> 00:07:47,160 And now I'm using LSBLK again. 152 00:07:47,160 --> 00:07:48,200 And what do we see? 153 00:07:48,200 --> 00:07:50,679 We see the LVM logical volumes. 154 00:07:50,679 --> 00:07:56,239 Because of the VG scan first and the LVChange volume group 155 00:07:56,239 --> 00:07:59,760 name minus A Y, I now have my volume group available. 156 00:07:59,760 --> 00:08:06,200 And I should be able to mount dev cs root on MNT. 157 00:08:06,200 --> 00:08:09,160 And now I'm in my MNT directory. 158 00:08:09,160 --> 00:08:12,320 And from there, I can continue working 159 00:08:12,320 --> 00:08:15,799 and try to recover my system. 160 00:08:15,799 --> 00:08:19,119 Now that we are here, let's do the next step as well. 161 00:08:19,119 --> 00:08:23,880 And the next step would be that we make the other file systems 162 00:08:23,880 --> 00:08:25,359 available as well. 163 00:08:25,399 --> 00:08:29,760 My problem is that the proc file system in MNT is empty. 164 00:08:29,760 --> 00:08:31,880 The sys file system is empty. 165 00:08:31,880 --> 00:08:34,359 And the dev file system is empty. 166 00:08:34,359 --> 00:08:37,280 But we do have these guys on the rescue system. 167 00:08:37,280 --> 00:08:38,919 Now the thing with these file systems 168 00:08:38,919 --> 00:08:42,359 is that they are automatically created while booting. 169 00:08:42,359 --> 00:08:45,239 And I need to make sure that the file systems on the rescue 170 00:08:45,239 --> 00:08:49,200 system are present in my recovered system. 171 00:08:49,200 --> 00:08:53,799 And in order to do so, I'm using mount minus O bind. 172 00:08:53,840 --> 00:08:55,359 That's a bind mount. 173 00:08:55,359 --> 00:08:58,239 And in a bind mount, you can connect a directory 174 00:08:58,239 --> 00:09:00,080 to another directory. 175 00:09:00,080 --> 00:09:02,239 I do it for proc on MNT proc. 176 00:09:02,239 --> 00:09:05,320 I do that for dev on MNT dev. 177 00:09:05,320 --> 00:09:11,280 And I do that for sys on MNT sys. 178 00:09:11,280 --> 00:09:14,559 And there we are a bit further. 179 00:09:14,559 --> 00:09:18,119 Now I'm going to use chroot on slash MNT. 180 00:09:18,119 --> 00:09:21,919 And that brings me into the root of my normal operational 181 00:09:21,919 --> 00:09:23,039 system. 182 00:09:23,080 --> 00:09:24,400 I can even write. 183 00:09:24,400 --> 00:09:27,359 And that means that I can do my troubleshooting commands. 184 00:09:27,359 --> 00:09:30,440 Let's just try one common troubleshooting command 185 00:09:30,440 --> 00:09:33,760 to reset the root user password. 186 00:09:33,760 --> 00:09:34,679 Does that work? 187 00:09:34,679 --> 00:09:36,400 That's the only thing I care about. 188 00:09:36,400 --> 00:09:37,760 And yes, it works. 189 00:09:37,760 --> 00:09:41,640 So this concludes my demo about rescuing. 190 00:09:41,640 --> 00:09:46,400 Final step, use exit to exit the chroot environment. 191 00:09:46,400 --> 00:09:50,960 And use reboot to reboot in your normal operational system.