1 00:00:00,000 --> 00:00:10,280 Next area to investigate is CPU performance. 2 00:00:10,280 --> 00:00:13,320 So what can you do to monitor CPU performance? 3 00:00:13,320 --> 00:00:18,480 Well, of course, apart from the obvious thing like top command, which is revealing a lot 4 00:00:18,480 --> 00:00:24,719 of information about CPUs, there is proc CPU info, which gives detailed information about 5 00:00:24,719 --> 00:00:28,600 the CPUs, like flags that are supported. 6 00:00:28,600 --> 00:00:34,720 If, for instance, you want to use hardware virtualization, you need to make sure that 7 00:00:34,720 --> 00:00:42,279 either the SVM or the VMX flag is available, and proc CPU info will tell you about it. 8 00:00:42,279 --> 00:00:47,200 Uptime is showing how long the system has been up, and it also shows load averages. 9 00:00:47,200 --> 00:00:52,720 All the uptime output is also integrated in the top command output. 10 00:00:52,720 --> 00:00:58,720 And SAR is a very convenient command that can be used to gather a long-time usage statistics. 11 00:00:58,720 --> 00:01:04,879 The nice thing about SAR is that it gathers performance data every x minutes, and once 12 00:01:04,879 --> 00:01:10,040 performance data has been gathered, you can analyze it using the SAR command, but you 13 00:01:10,040 --> 00:01:14,400 need to enable it because by default it won't do anything. 14 00:01:14,400 --> 00:01:20,559 And finally, there are CCTL interface parameters in slash proc to tune different CPU-related 15 00:01:20,559 --> 00:01:21,559 settings. 16 00:01:21,559 --> 00:01:23,120 Let me show you. 17 00:01:23,120 --> 00:01:26,559 So to start it, I'm going to show you systemd timers. 18 00:01:26,559 --> 00:01:34,720 Let me use systemctl list unit files minus t timer. 19 00:01:34,720 --> 00:01:39,680 And what I'm looking for is the sysstat collect timer. 20 00:01:39,680 --> 00:01:42,080 So what is in the sysstat collect timer? 21 00:01:42,120 --> 00:01:54,199 Well, let's use systemctl get on sysstat collect dot timer. 22 00:01:54,199 --> 00:02:00,360 And there we can see that this is a job that is running every 10 minutes to collect information 23 00:02:00,360 --> 00:02:02,959 that the SAR utility is using. 24 00:02:02,959 --> 00:02:06,599 Now, the critical question here is what time is it? 25 00:02:06,599 --> 00:02:08,520 It's 1.33. 26 00:02:08,520 --> 00:02:09,520 Good. 27 00:02:09,520 --> 00:02:15,320 Now, I installed the SAR utility earlier when I've shown you the iostat command output. 28 00:02:15,320 --> 00:02:22,320 And that means that I'm expecting this sysstat utility to have collected some data. 29 00:02:22,320 --> 00:02:28,160 If you go in the var log directory, then you can find an SA subdirectory. 30 00:02:28,160 --> 00:02:33,279 And this SA subdirectory is a directory that is normally filled by the sysstat collect 31 00:02:33,279 --> 00:02:34,360 utility. 32 00:02:34,360 --> 00:02:38,000 And oh, boy, I can see no files in there. 33 00:02:38,000 --> 00:02:39,000 What does that mean? 34 00:02:39,479 --> 00:02:43,199 It means that SAR minus uppercase A, which is a good start. 35 00:02:43,199 --> 00:02:50,720 Well, as you can see, it's telling me it cannot open var log SA SA11 no switch file or directory. 36 00:02:50,720 --> 00:02:54,479 It's looking for a file, but the file does not exist. 37 00:02:54,479 --> 00:02:58,880 So apparently, data collection has not been happening yet. 38 00:02:58,880 --> 00:03:05,520 If I want to know more about it, then I can use systemctl status on sysstat collect dot 39 00:03:05,520 --> 00:03:07,520 timer. 40 00:03:07,520 --> 00:03:12,119 And there we can see that it is, oh, it is enabled, enabled, and active. 41 00:03:12,119 --> 00:03:15,240 It is inactive and triggers sysstat collect service. 42 00:03:15,240 --> 00:03:18,279 Well, apparently, it just hasn't started yet. 43 00:03:18,279 --> 00:03:19,720 But you know what? 44 00:03:19,720 --> 00:03:23,199 We can also start sysstat collect service manually. 45 00:03:23,199 --> 00:03:31,679 So systemctl start on sysstat collect dot service is going to run the process for me 46 00:03:31,679 --> 00:03:33,800 manually. 47 00:03:33,800 --> 00:03:38,559 So now that the sysstat information is starting to gather, we need to give it a little bit 48 00:03:38,559 --> 00:03:39,600 of time. 49 00:03:39,600 --> 00:03:43,119 So let's have a look at the proc parameters. 50 00:03:43,119 --> 00:03:48,759 In slash proc slash sys, you find information about tunables. 51 00:03:48,759 --> 00:03:55,600 And these kernel tunables, well, there's a couple of tunables for CPU as well. 52 00:03:55,600 --> 00:04:01,360 You find them in proc sys kernel, because in the end, the Linux kernel determines what 53 00:04:01,360 --> 00:04:03,919 the CPU is doing. 54 00:04:03,919 --> 00:04:09,559 And here you can find a couple of these options, which are the scheduler options. 55 00:04:09,559 --> 00:04:15,240 The scheduler determines how much time every process gets that wants to do something on 56 00:04:15,240 --> 00:04:16,440 the CPU. 57 00:04:16,440 --> 00:04:17,799 So how does it work? 58 00:04:17,799 --> 00:04:25,959 Well, let's say that you want to change the amount of time for your SCAD RT runtime microseconds. 59 00:04:25,959 --> 00:04:28,559 Now first, let's investigate what is in there. 60 00:04:28,559 --> 00:04:32,320 So we have SCAD RT, which is for real-time processes. 61 00:04:32,320 --> 00:04:39,079 And every real-time process is getting this amount of microseconds as its runtime, meaning 62 00:04:39,079 --> 00:04:46,480 that the remaining amount of microseconds on a second is for all the other processes. 63 00:04:46,480 --> 00:04:53,279 And if you think this is too much or not enough, you echo the new value into the file. 64 00:04:53,279 --> 00:05:01,079 So echo 900,000 greater than SCAD RT runtime US. 65 00:05:01,079 --> 00:05:04,160 And that will set the setting right now. 66 00:05:04,160 --> 00:05:09,720 And if you're happy with it, you can make it persistent using sysctl, as we have discussed 67 00:05:09,720 --> 00:05:11,100 before. 68 00:05:11,100 --> 00:05:12,679 So how does that work? 69 00:05:12,679 --> 00:05:17,079 Well, I would suggest you do the following. 70 00:05:17,079 --> 00:05:23,880 You copy the SCAD RT runtime US, because that's a difficult parameter name, isn't it? 71 00:05:23,880 --> 00:05:25,839 Well, here we have it copied. 72 00:05:25,839 --> 00:05:30,399 Now I'm going into etc.sysctl.d. 73 00:05:30,399 --> 00:05:34,519 And there we have the 99.sysctl.conf. 74 00:05:34,519 --> 00:05:38,239 And you can just add it to this. 75 00:05:38,239 --> 00:05:40,000 So what am I going to edit? 76 00:05:40,000 --> 00:05:49,559 Well, I am going to edit kernel.parameter that I want to change equals the value that 77 00:05:49,559 --> 00:05:52,040 I want to give to the parameter. 78 00:05:52,040 --> 00:05:57,880 And that makes sure that next time your system is started, the sysctl service is picking 79 00:05:57,880 --> 00:05:59,440 up the value. 80 00:05:59,440 --> 00:06:04,880 And in case you forgot the exact naming of this parameter, well, this here is a relative 81 00:06:04,880 --> 00:06:05,880 file name. 82 00:06:05,880 --> 00:06:09,519 It's a relative file name where the dot is used as a separator. 83 00:06:09,519 --> 00:06:14,440 And it is relative to the slash proc slash sys directory. 84 00:06:14,440 --> 00:06:17,600 So this will make it persistent. 85 00:06:17,600 --> 00:06:19,760 Now let's get back to SAR. 86 00:06:19,760 --> 00:06:25,160 SAR minus minus help is showing all the different SAR utilities. 87 00:06:25,160 --> 00:06:31,880 And as you can see, many, many command line options are available. 88 00:06:31,880 --> 00:06:33,799 Here we can see the minus U option. 89 00:06:33,799 --> 00:06:37,440 After all, we're talking about CPU utilization. 90 00:06:37,440 --> 00:06:46,160 So SAR minus U all should show information that has been gathered about all of my CPUs. 91 00:06:46,160 --> 00:06:50,239 So a little bit of troubleshooting about this service collection. 92 00:06:50,239 --> 00:06:59,239 If you do a sysctl cat on sysstat collect.service, you can see what it is actually doing. 93 00:06:59,239 --> 00:07:01,320 And here you can see that this is one shot. 94 00:07:01,320 --> 00:07:03,119 So it will run it once. 95 00:07:03,440 --> 00:07:08,320 And what it runs is uselib64sa1, et cetera. 96 00:07:08,320 --> 00:07:13,720 So if you just copy this command and you run it manually, then the result is that it is 97 00:07:13,720 --> 00:07:17,519 going to gather some system information for you. 98 00:07:17,519 --> 00:07:20,119 Now the timer is doing this every 10 minutes. 99 00:07:20,119 --> 00:07:25,720 But if you're impatient and if you don't want to work, this allows you to get started. 100 00:07:25,720 --> 00:07:28,079 And next, you can use your SAR command. 101 00:07:28,079 --> 00:07:33,079 So SAR minus minus help is giving all these different options. 102 00:07:33,079 --> 00:07:39,480 Maybe a good way to get started is by using CPU information only. 103 00:07:39,480 --> 00:07:41,480 That will be minus U all. 104 00:07:41,480 --> 00:07:46,239 So try SAR minus U all. 105 00:07:46,239 --> 00:07:53,200 And here you see the statistics over time that have been gathered for your CPU utilization. 106 00:07:53,200 --> 00:07:55,200 So what do we see? 107 00:07:55,200 --> 00:08:00,160 Well, we see the user and the nice and the system space. 108 00:08:00,160 --> 00:08:03,600 And it is capturing this information every 10 minutes. 109 00:08:03,600 --> 00:08:09,480 And that is useful because if every day you want to know which are your peak moments and 110 00:08:09,480 --> 00:08:15,239 your low moments, well, you can clearly figure out if there are moments when your CPUs are 111 00:08:15,239 --> 00:08:17,679 hit harder than at other moments. 112 00:08:17,679 --> 00:08:21,239 And that is how SAR can be an absolutely very useful utility.