WEBVTT 1 00:00.000 --> 00:04.100 So now what I want to do is I want to briefly show you how we can validate the 2 00:04.100 --> 00:04.960 configuration 3 00:04.960 --> 00:09.400 of our RAID deployment. So what I have done in the meantime is I've added two 4 00:09.400 --> 00:10.800 more disks to my 5 00:10.800 --> 00:14.760 yellow box device and I'm going to also create a RAID array with them. So let's 6 00:14.760 --> 00:16.000 go to that device. 7 00:16.000 --> 00:23.600 If I do an LSBOK-F we can see these two devices here 0N2 and 0N3 and I've also 8 00:23.600 --> 00:24.240 made them both 9 00:24.240 --> 00:28.130 one gig in size pretty much the exact same thing that we saw on the blue box in 10 00:28.130 --> 00:29.680 the previous video. 11 00:29.680 --> 00:34.190 So what I'm going to do here is I'm going to go into my playbook here play 1.y WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 11 00:29.680 --> 00:34.190 So what I'm going to do here is I'm going to go into my playbook here play 1.y 12 00:34.190 --> 00:34.960 ml and I'm just going 13 00:34.960 --> 00:38.680 to go into insert mode and I'm going to deploy the RAID array onto the yellow 14 00:38.680 --> 00:39.680 box too. Let's just 15 00:39.680 --> 00:44.060 change this here and save here. So I'm now going to say Ansible Playbook play 1 16 00:44.060 --> 00:45.520 .yml and to enter 17 00:45.520 --> 00:49.890 and as we can see here we've now made the changes on the yellow box. So arrow 18 00:49.890 --> 00:51.440 up LSBOK-F now we can 19 00:51.440 --> 00:56.300 see we have this RAID array mounted in the RAID dev directory. So what I can do 20 00:56.300 --> 00:57.200 is if I go to my 21 00:57.200 --> 01:04.940 devices and I say sudo mda m - - detail and I target my dev mda 0 device and to WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 21 00:57.200 --> 01:04.940 devices and I say sudo mda m - - detail and I target my dev mda 0 device and to 22 01:04.940 --> 01:05.840 enter let's type in the 23 01:05.840 --> 01:09.310 password here. So we can see the information relating to this particular RAID 24 01:09.310 --> 01:10.640 array we can see it's RAID 1 25 01:10.640 --> 01:14.980 we can see the RAID devices but interestingly we have some diagnostic 26 01:14.980 --> 01:16.720 information relating to the 27 01:16.720 --> 01:21.050 state of the device here we can see here that the state happens to be clean 28 01:21.050 --> 01:22.480 which is a good sign 29 01:22.480 --> 01:27.190 and we can see we have zero failed devices. Now sometimes what you might see in 30 01:27.190 --> 01:28.160 this particular 31 01:28.160 --> 01:32.290 output of one of the devices happens to be broken is that you might see the WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 31 01:28.160 --> 01:32.290 output of one of the devices happens to be broken is that you might see the 32 01:32.290 --> 01:33.520 option or rather the 33 01:33.520 --> 01:37.780 description degraded. Similarly whilst things are fully operational the failed 34 01:37.780 --> 01:38.960 devices happen to be 35 01:38.960 --> 01:42.690 zero but let's say a device does happen to fail this number will be increment 36 01:42.690 --> 01:43.680 by one and perhaps 37 01:43.680 --> 01:48.390 if both devices fail then this number would increment to pretty much if this 38 01:48.390 --> 01:50.800 number is not zero then we 39 01:50.800 --> 01:55.540 have a problem on our hands. So how about we go and write a little playbook to 40 01:55.540 --> 01:56.960 test the status 41 01:56.960 --> 02:01.840 of both of our devices RAID arrays. So let's go back to the control node and WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 41 01:56.960 --> 02:01.840 of both of our devices RAID arrays. So let's go back to the control node and 42 02:01.840 --> 02:02.480 let's just create a 43 02:02.480 --> 02:07.480 playbook called RAID checker.yml. So going to insert mode have our free hyphens 44 02:07.480 --> 02:09.120 at the top give it a name 45 02:09.120 --> 02:13.490 and I'll just say playbook for checking RAID arrays. Now who am I going to 46 02:13.490 --> 02:14.640 target in this case here 47 02:14.640 --> 02:17.950 I will target the all groups of both devices since they both have RAID arrays I 48 02:17.950 --> 02:18.960 want to see them both 49 02:18.960 --> 02:23.280 and I'll set my gather facts to be the value false. Now what I'm going to do is 50 02:23.280 --> 02:24.960 define my tasks. 51 02:24.960 --> 02:30.400 So let's give this task a name and I'll just say check for any failed devices WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 51 02:24.960 --> 02:30.400 So let's give this task a name and I'll just say check for any failed devices 52 02:30.400 --> 02:31.760 in the RAID array. 53 02:31.760 --> 02:35.210 Now what am I going to do here I'm going to use the Ansible built-in command 54 02:35.210 --> 02:36.400 module and the command 55 02:36.400 --> 02:45.110 I'm going to send is going to be MDADM - detail /dev/md0 like I say we're just 56 02:45.110 --> 02:46.240 issuing this particular 57 02:46.240 --> 02:50.150 command via the playbook on both of our devices. Now here is the thing when we 58 02:50.150 --> 02:51.200 happen to issue this 59 02:51.200 --> 02:55.480 particular command we want to be able to capture the output and effectively 60 02:55.480 --> 02:56.880 assess what we find in 61 02:56.880 --> 03:00.770 that output. So recall how we can actually capture the output what we want to WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 61 02:56.880 --> 03:00.770 that output. So recall how we can actually capture the output what we want to 62 03:00.770 --> 03:02.240 do is we want to register 63 03:02.240 --> 03:07.570 it as a variable value. So let's go and do this then. So let's register this as 64 03:07.570 --> 03:08.560 we'll say RAID 65 03:08.560 --> 03:12.570 output and now let's create a task that is going to assess the contents of that 66 03:12.570 --> 03:13.280 output. So let's 67 03:13.280 --> 03:16.830 just say this task is going to be examine the outputs. So I will use the Ans 68 03:16.830 --> 03:17.920 ible built-in 69 03:17.920 --> 03:21.970 debug module here and I'm going to write an alert message I will say alerts and 70 03:21.970 --> 03:23.120 I'll say there 71 03:23.120 --> 03:30.640 are failed devices on MD0 for and now I want to specify the name of the device WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 71 03:23.120 --> 03:30.640 are failed devices on MD0 for and now I want to specify the name of the device 72 03:30.640 --> 03:31.920 that happens to fail. 73 03:31.920 --> 03:36.840 Now I don't want to hard code this as blue box or perhaps yellow box because it 74 03:36.840 --> 03:37.600 depends on the 75 03:37.600 --> 03:40.950 device that happens to fail. We want to populate this with a variable value and 76 03:40.950 --> 03:41.760 the way we can do 77 03:41.760 --> 03:45.360 this is by having our double curly braces. I'm just going to reference this 78 03:45.360 --> 03:46.320 magic variable that's 79 03:46.320 --> 03:52.130 well known as a variable called inventory_hostname that will populate whatever 80 03:52.130 --> 03:53.440 device happens to fail 81 03:53.440 --> 03:56.770 here. Now how are we going to ensure that a device fails or rather how are we 82 03:56.770 --> 03:57.600 going to ensure that 83 03:57.600 --> 04:01.520 we alert ourselves when a device fails? Well we know we're going to use a WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 83 03:57.600 --> 04:01.520 we alert ourselves when a device fails? Well we know we're going to use a 84 04:01.520 --> 04:02.880 conditional check. So I'm 85 04:02.880 --> 04:06.890 going to specify the when keyword to my double quotation mark and then side of 86 04:06.890 --> 04:07.920 this I'm going to 87 04:07.920 --> 04:12.600 open my single quotations and I'm going to check for the output that I want to 88 04:12.600 --> 04:13.920 see. So let's go here. 89 04:13.920 --> 04:18.540 Now the output that I want to see is exactly this. I want to ensure that failed 90 04:18.540 --> 04:19.920 devices is equal to 91 04:19.920 --> 04:23.960 the value zero. So let's copy this here and let me just paste this in and then 92 04:23.960 --> 04:25.120 I will close my 93 04:25.120 --> 04:29.170 single quotation marks. Now think about it like this. If I happen to see failed 94 04:29.170 --> 04:30.480 devices zero, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 94 04:29.170 --> 04:30.480 devices zero, 95 04:30.480 --> 04:34.010 that is actually good for me. That means I have no failures on the radar array. 96 04:34.010 --> 04:35.120 So I would not want 97 04:35.120 --> 04:39.440 to print this particular message if that condition was true. So what I actually 98 04:39.440 --> 04:40.320 want to do here is 99 04:40.320 --> 04:44.930 I want to use some inverted logic by making sure that this message is not in 100 04:44.930 --> 04:46.160 the actual output. 101 04:46.160 --> 04:49.450 Now how do we test the actual output? Well we just take the name of the 102 04:49.450 --> 04:50.640 variable that we use to 103 04:50.640 --> 04:55.440 capture that output which is read_output and to read the actual data we're 104 04:55.440 --> 04:56.720 going to have to grab 105 04:56.720 --> 05:01.230 the STD out property. So check this out. If I run this right now, given the WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 105 04:56.720 --> 05:01.230 the STD out property. So check this out. If I run this right now, given the 106 05:01.230 --> 05:02.160 fact that both are 107 05:02.160 --> 05:05.910 healthy, this particular condition will not be triggered and we're not going to 108 05:05.910 --> 05:07.280 print it a warning. 109 05:07.280 --> 05:10.800 Okay. So let's see this in action. So it will say Ansible Playbook, 110 05:10.800 --> 05:16.530 Read Checker, dot yml and to enter the Playbook runs. We run the checks and 111 05:16.530 --> 05:17.280 because we do not 112 05:17.280 --> 05:21.950 have any failures we don't print any alerts. Now let's go and simulate a 113 05:21.950 --> 05:23.120 failure then. So what 114 05:23.120 --> 05:27.040 I will do here is let's maybe go to the yellow box here and let me manually 115 05:27.040 --> 05:28.240 induce this failure 116 05:28.240 --> 05:34.640 source is sudo atm - - manage and now I'll target the dev MD0 device and I'm WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 116 05:28.240 --> 05:34.640 source is sudo atm - - manage and now I'll target the dev MD0 device and I'm 117 05:34.640 --> 05:35.360 going to say - 118 05:35.360 --> 05:40.090 fail and I'm going to fail a particular disk. So let's let me grab this disk 119 05:40.090 --> 05:40.880 right here, 120 05:40.880 --> 05:47.280 the dev NVM E0 N3 disk. So paste this in here and if I hit enter and I type in 121 05:47.280 --> 05:48.080 my super user 122 05:48.080 --> 05:51.630 password, look at this right here. We've now got this faulty disk. So let's 123 05:51.630 --> 05:52.480 clear the screen. 124 05:52.480 --> 05:56.400 If I happen to run my same check, my detailed command and I enter, look at what 125 05:56.400 --> 05:56.960 we actually see 126 05:56.960 --> 06:02.180 here. We see degraded in the states and more crucially failed devices no longer WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 126 05:56.960 --> 06:02.180 here. We see degraded in the states and more crucially failed devices no longer 127 06:02.180 --> 06:03.680 equals the value zero. 128 06:03.680 --> 06:08.680 It's now incremented to the value one. So pretty much we should actually 129 06:08.680 --> 06:09.200 trigger 130 06:09.200 --> 06:12.620 our conditional check with thinner playbook imprint the alert message for that 131 06:12.620 --> 06:13.200 particular 132 06:13.200 --> 06:16.960 device. So let's go and try this again. Let's clear the screen. Let's add a 133 06:16.960 --> 06:18.160 walk and to enter, 134 06:18.160 --> 06:21.910 look at this right here for the yellow box we get an alert. There are failed 135 06:21.910 --> 06:23.760 devices on MD0 136 06:23.760 --> 06:27.840 for yellow box. And again, we can see we have this variable value populated. 137 06:27.840 --> 06:28.400 Now similarly, 138 06:28.400 --> 06:33.340 if we also go to a blue box, we'll say pseudo adm - - detail and we check for WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 138 06:28.400 --> 06:33.340 if we also go to a blue box, we'll say pseudo adm - - detail and we check for 139 06:33.340 --> 06:35.280 dev MD0. We can see 140 06:35.280 --> 06:39.610 here, type in the password. We can see here we have no failed devices right now 141 06:39.610 --> 06:40.480 . However, 142 06:40.480 --> 06:45.360 if I run the same command on this particular device, I'll say pseudo MD adm - 143 06:45.360 --> 06:46.640 manage forward slash 144 06:46.640 --> 06:50.710 dev forward slash MD0 and I'll specify we want to fail and we'll target the 145 06:50.710 --> 06:51.760 disk. Let's go with 146 06:51.760 --> 06:58.110 dev NVMe0 and to enter. Now we have a faulty disk. Let's check the status once 147 06:58.110 --> 06:59.680 again. Now the value 148 06:59.680 --> 07:04.170 is no longer zero on this particular device also. So therefore if we run our WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 148 06:59.680 --> 07:04.170 is no longer zero on this particular device also. So therefore if we run our 149 07:04.170 --> 07:05.440 playbook once again, 150 07:05.440 --> 07:09.600 let's clear the screen, add a walk and to enter, give it a moment. Now we can 151 07:09.600 --> 07:10.400 actually see we have 152 07:10.400 --> 07:15.030 an alert for a failed device on the yellow box as well as a failed device on 153 07:15.030 --> 07:16.400 the blue box. So you 154 07:16.400 --> 07:19.890 can imagine if at scale you were managing thousands of devices just how handy 155 07:19.890 --> 07:21.520 this would be able to be 156 07:21.520 --> 07:25.520 to be able to quickly check the status of your radar rays across all those 157 07:25.520 --> 07:26.160 devices 158 07:26.160 --> 07:29.520 instantaneously using automation with Ansible.