1 00:00:00,000 --> 00:00:11,560 In this video, you'll learn what you can do if you need to troubleshoot Linux memory. 2 00:00:11,560 --> 00:00:16,320 To start with, use 3-M. That will give you a generic overview of current memory usage. 3 00:00:16,320 --> 00:00:22,240 And you should check if a minimum of 20% of RAM is available as free or cached. If it's 4 00:00:22,240 --> 00:00:28,400 not, you need to further investigate what is going on. Also, consider the use of swap 5 00:00:28,400 --> 00:00:35,040 space. Are you currently using any swap? Do you even have swap? And if swap is used, use 6 00:00:35,040 --> 00:00:41,360 vmstat to get usage information. Also, check logs to find out if the own killer is being 7 00:00:41,360 --> 00:00:42,360 issued. 8 00:00:42,360 --> 00:00:47,480 Let me show you. In order to investigate memory problems, it would be cool if you have memory 9 00:00:47,480 --> 00:00:53,279 problems. And I'm going to reboot and limit the amount of available memory while rebooting 10 00:00:53,279 --> 00:00:57,720 so that we can see that actually something is happening. Because right now, on this four 11 00:00:57,720 --> 00:01:05,040 gig system, nothing really is happening. So once you are in a group menu, at the end 12 00:01:05,040 --> 00:01:11,639 of the line that loads the Linux kernel, temporarily remove rhdb and quiet and then use mem equals 13 00:01:11,639 --> 00:01:17,760 1g. That will make that you have one gigabyte of memory. And that makes it all a little 14 00:01:17,760 --> 00:01:24,800 bit more interesting. Next, control X to start. So to the terminal to investigate, and a good 15 00:01:24,800 --> 00:01:35,160 first step is to use 3 minus m. And now we see interesting stuff. So one gigabyte of 16 00:01:35,160 --> 00:01:40,000 memory is what I passed while starting up. Unfortunately, for my system, part of that 17 00:01:40,000 --> 00:01:47,839 memory is used for the graphics card. So we only have 708 remaining. And that means that 18 00:01:47,839 --> 00:01:57,680 we fail all sanity tasks. So 529 used, only 56 free. And you can also see that 770 mebibytes 19 00:01:57,680 --> 00:02:04,239 of memory is currently in swap. What does that mean? Well, we have more memory in swap. 20 00:02:04,239 --> 00:02:08,960 Then we have a total memory available. And that is perfect for troubleshooting. So I 21 00:02:08,960 --> 00:02:17,119 would like to use a VM stat to five, two second interval, five polling loops in total. And what 22 00:02:17,119 --> 00:02:22,880 I'm observing is swap activity. And here you can see the swap activity, which is pretty close to 23 00:02:22,880 --> 00:02:29,520 block activity. Now the thing that you are not supposed to like here is that swap in and swap 24 00:02:29,520 --> 00:02:35,360 out, it's continuously happening. And I wasn't even doing something. And that is really an indicator 25 00:02:35,360 --> 00:02:42,479 of a poor state of your memory. Of course, we have 10 times the amount of swap as the amount 26 00:02:42,479 --> 00:02:48,559 of RAM. So it's going to take a while before you are running out of swap. But you are seriously 27 00:02:48,559 --> 00:02:56,639 running out of memory. I would also advise you grab on active memories in PROC MEMINFO. 28 00:02:57,279 --> 00:03:01,600 I'm skipping the A, by the way, so that I don't have to worry about is that an uppercase or 29 00:03:01,600 --> 00:03:09,520 lowercase. And this is showing what I want to see. So what I see is that I have this amount of 30 00:03:09,520 --> 00:03:17,839 active memory. And here I have my inactive memory. And this is application memory. And the thing is 31 00:03:18,399 --> 00:03:27,360 104244 kilobytes. That means we have 104 megabytes of inactive application memory. The amount of 32 00:03:27,360 --> 00:03:32,479 memory that we have in swap is way higher. And that means that active memory is being swapped 33 00:03:32,479 --> 00:03:38,080 out. As it doesn't show as active and inactive anonymous memory, which is your typical application 34 00:03:38,080 --> 00:03:45,119 memory, we are most likely talking about kernel procedures and routines. So beyond any doubt, 35 00:03:45,119 --> 00:03:53,600 this system is way out of memory. Then I'm going to do something fun to trigger an out of memory. 36 00:03:53,600 --> 00:03:59,520 I want you to be able to recognize an out of memory condition. And in order to do so, first, 37 00:03:59,520 --> 00:04:08,639 I'm echoing an H to the file PROC SYSRQ trigger. The SYSRQ trigger is giving access to some magical 38 00:04:08,639 --> 00:04:14,880 controls on your Linux. Now, if you echo an H, you get help. And this help is only visible 39 00:04:14,880 --> 00:04:21,679 in the D message output. And the help I'm looking for is the memory full ohm kill. You can see that 40 00:04:21,679 --> 00:04:27,679 on the bottom, it's the letter F that we need to echo. So this time, I'm going to echo an F 41 00:04:27,679 --> 00:04:36,559 to PROC SYSRQ trigger. And oh, boy, I get, oh, do you see that? Device memory is almost full. 42 00:04:36,559 --> 00:04:43,119 Terminal was using a lot of memory. And I'm going to start my terminal again. At least I'm going to 43 00:04:43,119 --> 00:04:50,480 try, because I would like to show you what you can find in the logs if this happens. 44 00:04:51,200 --> 00:05:02,799 So journal CTL, uppercase G to go all the way to the end. And then I'm scrolling up a little bit. 45 00:05:03,440 --> 00:05:10,000 And here, I can see out of memory, out of memory, killed process, whatever. But you should be 46 00:05:10,000 --> 00:05:16,399 looking for this ohm kill. Now, by echoing an F into this PROC SYSRQ trigger, I manually force 47 00:05:16,399 --> 00:05:22,720 an ohm kill. What you should be worried about is if you can see it in any other circumstances. 48 00:05:23,359 --> 00:05:29,359 Now, this is the same ohm kill than the one that I just triggered. And if there is only one of them, 49 00:05:29,359 --> 00:05:36,239 then it's not an issue. But if you have multiple out of memory killers that occurred, then really, 50 00:05:36,239 --> 00:05:42,480 you have a problem. So that is how you can identify poorly behaving memory. Let's boot 51 00:05:42,480 --> 00:05:49,119 back into a normal system. So one solution, if you are running out of memory, is to allocate some 52 00:05:49,119 --> 00:05:55,279 swap. So swap space can be created on a block device or on a swap file. And if you are using 53 00:05:55,279 --> 00:06:01,440 a partition, you must set the partition type to swap. And next, you use mkswap to format it 54 00:06:01,440 --> 00:06:08,320 as a swap partition. And you swap on to activate the swap space. Normally, swap is mounted through 55 00:06:08,320 --> 00:06:15,119 etcfsstep. And from there, systemd will pick it up and mount it as a swap unit type. Let me show 56 00:06:15,119 --> 00:06:21,519 you how this works. So let's talk about swap. We are back in a normal operational state where 57 00:06:21,519 --> 00:06:26,559 we have 8 gigs of swap, which is not used at all. But I want to show you how to use swap. 58 00:06:27,119 --> 00:06:35,600 So what am I going to do? Well, I'm going to use dd. If is dev0. Of is slash swap file. We're 59 00:06:35,600 --> 00:06:45,440 going to make a swap file this time. Bs is 1M. Oops. Bs is 1M. And count is 1024 to make for a 60 00:06:45,440 --> 00:06:52,079 one gigabyte swap file. Good. Then what are we going to do? Well, I don't have to create a partition 61 00:06:52,079 --> 00:07:00,079 this time. But I do need mkswap on slash swap file to format the swap file as a swap device. 62 00:07:00,880 --> 00:07:05,279 And there you can see that the mkswap command is kind enough to tell me about the insecure 63 00:07:05,279 --> 00:07:14,799 permissions. So I need to fix it using chmod 600 on slash swap file. The reason is that only the 64 00:07:14,799 --> 00:07:20,799 root user should have read and write access to the swap file, nobody else. You can even see that 65 00:07:20,799 --> 00:07:27,200 the swap file has created a label. Now, as this is a file, and files typically don't change their 66 00:07:27,200 --> 00:07:36,239 name spontaneously, I'm just going to put it in etcfsstep as such. So slash swap file. And I'm 67 00:07:36,239 --> 00:07:42,079 going to mount it the way you should mount swap. And for swap, we have seen it before. There's a 68 00:07:42,079 --> 00:07:48,959 couple of examples here. So how do I mount it? Well, I mount it on none. The type is swap. The 69 00:07:48,959 --> 00:07:56,799 mount options are defaults, and then the zero and zero. And next, I can use swap on minus A. 70 00:07:56,799 --> 00:08:03,359 That's comparable to mount minus A. And it will activate all swap that is in etcfsstep and not 71 00:08:03,359 --> 00:08:10,239 yet initialized. So when I use free minus M, I can see that I have another gigabyte of swap 72 00:08:10,880 --> 00:08:16,320 available. Swap on minus S, by the way, is showing all the different swap devices. 73 00:08:16,320 --> 00:08:22,079 So here we can see that we really got it all. We have a swap logical volume, a swap partition, 74 00:08:22,079 --> 00:08:29,119 and a swap file. Also interesting to see is the priority. Minus two, minus three, minus four. 75 00:08:29,119 --> 00:08:34,719 The minus two priority is used first. And that's how you can add swap to give your system 76 00:08:34,719 --> 00:08:38,320 the opportunity to calm down if it is running out of memory.