WEBVTT 1 00:00.000 --> 00:03.040 So friends, to kick us off, the very first thing that I want to talk to you 2 00:03.040 --> 00:08.000 about is the concept of compression as well as archiving files on Linux. 3 00:08.000 --> 00:12.490 Now quite honestly, very often people do confuse these different concepts, so 4 00:12.490 --> 00:16.510 it is important that we have some clarity before we begin automating these 5 00:16.510 --> 00:17.500 particular tasks. 6 00:17.500 --> 00:21.280 Now ultimately, when we happen to invoke compression and archiving, we can 7 00:21.280 --> 00:25.000 basically use them independently or we can combine them together, 8 00:25.000 --> 00:28.500 but really at its core when we're talking about compression, we are talking 9 00:28.500 --> 00:30.000 about making files smaller. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 10 00:30.000 --> 00:33.750 We are, after all, squishing them and compressing them into a smaller footprint 11 00:33.750 --> 00:34.000 . 12 00:34.000 --> 00:36.000 This is really what we're talking about. 13 00:36.000 --> 00:40.150 Whereas with respect to our archiving of files, what we're doing is essentially 14 00:40.150 --> 00:44.000 taking multiple files, or perhaps maybe many directories and subdirectories, 15 00:44.000 --> 00:48.000 and essentially grouping them together to be one particular file. 16 00:48.000 --> 00:51.360 So effectively, when we happen to combine the archiving of files with 17 00:51.360 --> 00:52.000 compression, 18 00:52.000 --> 00:56.560 we can take many different files and many different directories and combine 19 00:56.560 --> 00:58.000 them into one particular file, 20 00:58.000 --> 01:02.340 and then we can further compress that individual file as we see fit and then WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 20 00:58.000 --> 01:02.340 and then we can further compress that individual file as we see fit and then 21 01:02.340 --> 01:04.000 move it around as we want. 22 01:04.000 --> 01:07.810 Moving around, simply meaning that the file can be more portable as opposed to 23 01:07.810 --> 01:11.000 having many different large directories and files, 24 01:11.000 --> 01:15.000 you could have one small compressed file that you could perhaps, for example, 25 01:15.000 --> 01:20.210 just email over to someone or upload to a particular file sharing site, that 26 01:20.210 --> 01:21.000 type of thing. 27 01:21.000 --> 01:25.000 So with respect to compression, there are three main utilities that we have 28 01:25.000 --> 01:27.000 available to us on Linux systems. 29 01:27.000 --> 01:32.460 We have a compression utility known as Gzip, another known as Bzip2, and WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 29 01:27.000 --> 01:32.460 We have a compression utility known as Gzip, another known as Bzip2, and 30 01:32.460 --> 01:34.000 another known as XZ. 31 01:34.000 --> 01:37.990 Now, like I said, each of these tools allow us to compress files, and they also 32 01:37.990 --> 01:42.550 have corresponding tools to decompress or uncompress the files back to the 33 01:42.550 --> 01:43.000 normal state, 34 01:43.000 --> 01:48.730 so Gzip can be uncompressed with something known as Gunzip, and Bzip2 can be 35 01:48.730 --> 01:55.000 uncompressed with Bonsip2, and XZ can be uncompressed with OnXZ. 36 01:55.000 --> 01:58.610 Now, you might be thinking, why on earth do we have all these different 37 01:58.610 --> 02:01.900 utilities to effectively do the same thing, all of these tools on the left here WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 37 01:58.610 --> 02:01.900 utilities to effectively do the same thing, all of these tools on the left here 38 02:01.900 --> 02:02.000 , 39 02:02.000 --> 02:06.600 well, they all effectively make files smaller after all, and the ones on the 40 02:06.600 --> 02:09.000 right ultimately return the files back to their original state. 41 02:09.000 --> 02:12.300 Now, quite honestly, we don't have to get two into the weeds about the 42 02:12.300 --> 02:16.000 different types of compression and the encoding used, that type of thing, 43 02:16.000 --> 02:19.560 but ultimately at its core, different compression ratios can produce smaller 44 02:19.560 --> 02:22.000 files, for example, have better compression ratios, 45 02:22.000 --> 02:27.000 and others, whilst not maybe compressing quite as much, can do the job faster, 46 02:27.000 --> 02:32.000 whereas others yet again, maybe just provide a happy medium between WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 46 02:27.000 --> 02:32.000 whereas others yet again, maybe just provide a happy medium between 47 02:32.000 --> 02:36.000 aggressively compressing the file and performing the task very, very quickly. 48 02:36.000 --> 02:39.890 So Gzip is a very, very common implementation, what it tends to do is it 49 02:39.890 --> 02:43.000 provides fairly good compression, but it does so very, very quickly. 50 02:43.000 --> 02:46.840 So if you just want to maybe compress files and you're not too concerned about 51 02:46.840 --> 02:49.000 the actual size or the resultant output, 52 02:49.000 --> 02:52.450 you just want to get the job done quick and easy, Gzip2 can be a good solution 53 02:52.450 --> 02:56.880 for you, whereas Bzip2 happens to produce smaller files, particularly when you 54 02:56.880 --> 02:58.000 're dealing with text files. 55 02:58.000 --> 03:02.220 Ultimately it requires more CPU than Gzip and it's going to take more time to WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 55 02:58.000 --> 03:02.220 Ultimately it requires more CPU than Gzip and it's going to take more time to 56 03:02.220 --> 03:04.000 perform a particular task. 57 03:04.000 --> 03:07.430 Now, if you just happen to be compressing just a few files that are not so 58 03:07.430 --> 03:11.000 large, the difference in speed is going to be absolutely negligible, 59 03:11.000 --> 03:15.840 we're talking about compressing at scale here, and then the latest one, XZ 60 03:15.840 --> 03:17.000 compression. 61 03:17.000 --> 03:20.560 This typically produces the best compression ratio of the three, but it does 62 03:20.560 --> 03:24.000 seem to depend on the actual data that you're trying to compress. 63 03:24.000 --> 03:27.850 If for example you're just trying to compress a small text file, then you may 64 03:27.850 --> 03:31.000 notice that Bzip2 actually produces a smaller footprint, WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 64 03:27.850 --> 03:31.000 notice that Bzip2 actually produces a smaller footprint, 65 03:31.000 --> 03:35.800 whereas dealing with very, very large datasets, you may notice that XZ is in 66 03:35.800 --> 03:38.000 fact the best option for you. 67 03:38.000 --> 03:41.840 Now like I say, we don't want to go into too much detail about these different 68 03:41.840 --> 03:45.270 compression ratios, we just want to have a general understanding of the 69 03:45.270 --> 03:48.430 different options that we have available to us before we begin to automate 70 03:48.430 --> 03:49.000 things. 71 03:49.000 --> 03:52.440 With that said, let's dive in and use these tools to get a simple example of 72 03:52.440 --> 03:54.000 what we're trying to do here. 73 03:54.000 --> 03:57.790 So what I will do is I will create a little text file here, and I'll just call 74 03:57.790 --> 04:02.000 this "File1.txt" because as you know, I am super duper original. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 74 03:57.790 --> 04:02.000 this "File1.txt" because as you know, I am super duper original. 75 04:02.000 --> 04:06.310 Now just to get some text, I've just googled for Ansible on Wikipedia and I'm 76 04:06.310 --> 04:09.000 just going to grab all of this text here within this page. 77 04:09.000 --> 04:12.000 It's not a huge amount of information, but it will do for the demonstration. 78 04:12.000 --> 04:16.000 So I'll just paste all this in and I'll just press WQ to write and quit. 79 04:16.000 --> 04:21.080 Now if I do an ls-l on the file1.txt, we can actually see the size of the file, 80 04:21.080 --> 04:25.300 in fact, if I just happen to do a -h, we can see it as human readable, we can 81 04:25.300 --> 04:27.000 see this as 13 kilobytes. 82 04:27.000 --> 04:31.460 So let's try to compress this particular file. If I happen to say gzip and I WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 82 04:27.000 --> 04:31.460 So let's try to compress this particular file. If I happen to say gzip and I 83 04:31.460 --> 04:37.520 target file1 and just hit enter, notice the file has actually been converted to 84 04:37.520 --> 04:39.000 have a .gz extension. 85 04:39.000 --> 04:45.010 This tells us that it has been compressed using gzip. Now crucially, if I do an 86 04:45.010 --> 04:51.560 ls-l on the file.txt.gz now and hit enter, notice the file is only 5.2k in size 87 04:51.560 --> 04:54.000 as compared to before when it was 13. 88 04:54.000 --> 04:58.480 So clearly we can see here this particular file is less than half the original 89 04:58.480 --> 04:59.000 size. 90 04:59.000 --> 05:02.020 Now in order to return this file back to its original state, we can say gunsip WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 90 04:59.000 --> 05:02.020 Now in order to return this file back to its original state, we can say gunsip 91 05:02.020 --> 05:04.000 and then target the actual file right here. 92 05:04.000 --> 05:08.920 If I now do an ls, the file is back to its original state. And as we can see 93 05:08.920 --> 05:13.660 here, if I just do an ls-l on the file, it goes back to its original size of 13 94 05:13.660 --> 05:14.000 .K. 95 05:14.000 --> 05:18.770 Now if I happen to say bzip2 on the file and to enter and I now do an ls, 96 05:18.770 --> 05:23.450 notice the file now has a .bz extension. This tells us we're using bzip2 97 05:23.450 --> 05:28.000 compression and if we want to uncompress this file, we would use gunsip2. 98 05:28.000 --> 05:31.840 But before we actually do this, let's assess the new size of this particular WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 98 05:28.000 --> 05:31.840 But before we actually do this, let's assess the new size of this particular 99 05:31.840 --> 05:35.530 file if I hit enter. Notice the file is actually a little bit smaller than we 100 05:35.530 --> 05:37.000 got with gzip compression. 101 05:37.000 --> 05:41.410 So like I say bzip2 gives us a smaller footprint but generally speaking it's 102 05:41.410 --> 05:46.000 going to take more CPU and a little bit longer to perform this compression. 103 05:46.000 --> 05:49.980 In the case here where we have one small file, the difference is absolutely 104 05:49.980 --> 05:52.000 negligible in terms of performance. 105 05:52.000 --> 05:56.340 So if I want to uncompress this, I just say bzip2 and then go into the file and 106 05:56.340 --> 06:01.000 to enter, if I do an ls, the file has returned and I get an ls-lh. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 106 05:56.340 --> 06:01.000 to enter, if I do an ls, the file has returned and I get an ls-lh. 107 06:01.000 --> 06:04.930 On the file we can see it goes back to its original state and then at the very 108 06:04.930 --> 06:09.400 end if I just say xz on file1.txt, do an ls, we get the file once again and it 109 06:09.400 --> 06:12.000 has this time the .xz compression. 110 06:12.000 --> 06:16.200 Now like I say with respect to xz on small text files, you're not going to 111 06:16.200 --> 06:21.000 notice any better performance versus bzip2 or rather bzip2 should I say. 112 06:21.000 --> 06:25.050 Notice it's actually a little bit bigger but like I say with very very large 113 06:25.050 --> 06:29.140 datasets of different data types, xz tends to actually outperform bzip2 in 114 06:29.140 --> 06:30.000 these scenarios. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 115 06:30.000 --> 06:35.070 So again how do we uncompress in this case here, we just say on xz on file1.txt 116 06:35.070 --> 06:40.000 , do an ls on the file and the file has returned back to its original state. 117 06:40.000 --> 06:43.000 Now check this out. Let's go and get some more text here. 118 06:43.000 --> 06:46.870 Okay so I've just googled for red hat on Wikipedia. Let me just grab all of 119 06:46.870 --> 06:50.000 this stuff here. Let's create a new file called file2.txt. 120 06:50.000 --> 06:54.000 So let's paste this in. There we go. Let's wq to write and quits. 121 06:54.000 --> 07:00.740 So now I have file1 and file2.txt. Now if I happen to say gzip on file1 as well WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 121 06:54.000 --> 07:00.740 So now I have file1 and file2.txt. Now if I happen to say gzip on file1 as well 122 07:00.740 --> 07:06.200 as file2.txt, watch what happens and if I do an ls-lh, the files have actually 123 07:06.200 --> 07:10.700 been compressed but notice they happen to be compressed separately than not one 124 07:10.700 --> 07:11.000 file. 125 07:11.000 --> 07:14.550 So if I wanted to compress a whole directory of files and perhaps maybe share 126 07:14.550 --> 07:19.030 them, now we'd have to effectively upload multiple different compressed files 127 07:19.030 --> 07:21.000 as opposed to a one singular file. 128 07:21.000 --> 07:24.110 Similarly, let me just show you an example here. Let's create a directory 129 07:24.110 --> 07:27.000 called example. And I'll just guns up these two files. 130 07:27.000 --> 07:30.970 So now they're back to the original state. If I move these files into the WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 130 07:27.000 --> 07:30.970 So now they're back to the original state. If I move these files into the 131 07:30.970 --> 07:37.020 example there and I now try to say gzip on the example directory and to enter, 132 07:37.020 --> 07:40.000 notice here that says we can't actually compress a directory itself. 133 07:40.000 --> 07:43.300 So if we go inside the directory and do an ls, the files are still in their 134 07:43.300 --> 07:45.000 original uncompressed format. 135 07:45.000 --> 07:49.270 So this is now when we begin to talk about archiving particular files. Like I 136 07:49.270 --> 07:52.600 said earlier, this is going to allow us to group the files together as a single 137 07:52.600 --> 07:53.000 file. 138 07:53.000 --> 07:57.280 Now in this case here, the utility that we want to use is one called tar. This 139 07:57.280 --> 07:59.000 is known as a tape archive. 140 07:59.000 --> 08:03.710 Now like I say, this particular utility by default is not going to actually WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 140 07:59.000 --> 08:03.710 Now like I say, this particular utility by default is not going to actually 141 08:03.710 --> 08:09.000 compress the files but it can actually do so with particular options. 142 08:09.000 --> 08:12.110 So let me go and show you how this operates now. So let me just move these 143 08:12.110 --> 08:14.000 particular files back in directory. 144 08:14.000 --> 08:17.020 So if I do an ls, I'm no longer here. Go back in directory and try that again. 145 08:17.020 --> 08:19.000 Go back in directory. There we go. 146 08:19.000 --> 08:23.340 Now we can see they're back in the main home door. Check this out now. Here's 147 08:23.340 --> 08:24.000 the syntax. 148 08:24.000 --> 08:27.750 If you happen to look at the man page for the tar commands, if you press space 149 08:27.750 --> 08:32.620 bar, you're going to see all these different switches such as -c to create a new WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 149 08:27.750 --> 08:32.620 bar, you're going to see all these different switches such as -c to create a new 150 08:32.620 --> 08:37.000 archive, -x to extract the archive back to its original form, 151 08:37.000 --> 08:43.000 and -f allows us to specify the file that we want to archive or unarchive. 152 08:43.000 --> 08:46.660 Now just one point, when we happen to create an archive, this is known as a tar 153 08:46.660 --> 08:50.230 ball, which is the new singular file that we have created, which is comprising 154 08:50.230 --> 08:52.000 of the multiple files within. 155 08:52.000 --> 08:55.780 And like I say, this is not going to be compressed, but if we want to add the - 156 08:55.780 --> 09:00.800 j flag to our tarball, then we will effectively filter the archive through the b WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 156 08:55.780 --> 09:00.800 j flag to our tarball, then we will effectively filter the archive through the b 157 09:00.800 --> 09:02.000 zip to compression. 158 09:02.000 --> 09:08.570 Whereas if we use -g we will filter the archive through xz compression, and if 159 09:08.570 --> 09:13.080 we use lowercase z, we will filter the archive through the gzip compression. 160 09:13.080 --> 09:14.000 You see that? 161 09:14.000 --> 09:19.020 So let's actually do this then. So what I will do here is I'll say tar, -cf to 162 09:19.020 --> 09:24.000 create a file, and let's just call this my archive.tar, okay? 163 09:24.000 --> 09:28.150 And now we'll specify all the files that I want to belong within this archive, 164 09:28.150 --> 09:32.000 so I'll say file 1 should be part of it, as well as file number 2. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 164 09:28.150 --> 09:32.000 so I'll say file 1 should be part of it, as well as file number 2. 165 09:32.000 --> 09:36.200 So if I enter to an LS, notice the original files actually stay here, they've 166 09:36.200 --> 09:42.000 not been converted directly, but we also have this copy known as myarchive.tar. 167 09:42.000 --> 09:46.000 Now check this out, if I happen to remove these individual files that we use to 168 09:46.000 --> 09:51.150 create the archive, so all that we have left is the myarchive.tar, and if I do 169 09:51.150 --> 09:53.000 an LS-LH on the actual archive itself, 170 09:53.000 --> 09:59.860 we can see the size is 60k. Now if I happen to uncompress this, I can say tar - 171 09:59.860 --> 10:04.620 cf to extract the file on my archive.tar, notice the files have now been WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 171 09:59.860 --> 10:04.620 cf to extract the file on my archive.tar, notice the files have now been 172 10:04.620 --> 10:08.760 returned to the directory, and the archive still actually remains, you see that 173 10:08.760 --> 10:09.000 ? 174 10:09.000 --> 10:14.580 Now if I happen to remove my archive.tar, and I once again say tar, and I'll 175 10:14.580 --> 10:20.240 say -c to create -lhkj to run through bzip2 compression, and then if, and I 176 10:20.240 --> 10:23.000 will create an archive called my compress archive, okay? 177 10:23.000 --> 10:29.010 And I'll just say -tar, and I'll also add on the bz to extension, just to let 178 10:29.010 --> 10:30.370 people know that the file has been put through the bzip2 compression, and now I WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 178 10:29.010 --> 10:30.370 people know that the file has been put through the bzip2 compression, and now I 179 10:30.370 --> 10:34.570 specify the files that I want to belong to this archive, so again, file 1 and 180 10:34.570 --> 10:35.000 file 2. 181 10:35.000 --> 10:40.230 If I hit enter, do an LS, we've now created this new archive right here, if I 182 10:40.230 --> 10:46.440 do an LS-LH on my compressed archive, notice before it was 60k in size, whereas 183 10:46.440 --> 10:51.270 the compress version is only 18k in size, much much smaller, and again, if we 184 10:51.270 --> 10:57.150 want to uncompress, we can just say tar -x, j, f, so extract the file with bzip 185 10:57.150 --> 11:00.000 2 compression, and then we specify the name of the file again. WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 186 11:00.000 --> 11:03.900 In fact, before I do that, let me just remove the individual files that we use 187 11:03.900 --> 11:07.000 to make it, so that all we have left is the actual archive. 188 11:07.000 --> 11:11.390 So then I say tar, x, j, f, on my compressed archive, and hit enter, do an LS, 189 11:11.390 --> 11:14.000 now the files are back again, you see that? 190 11:14.000 --> 11:16.000 Cat file 1.txt. 191 11:16.000 --> 11:19.490 So that is the main operations behind our archiving and compression on Linux, 192 11:19.490 --> 11:22.930 we have different options that we can use to compress the files with different 193 11:22.930 --> 11:26.520 compression ratios, with different rates of performance, and when we want to 194 11:26.520 --> 11:30.700 group together multiple files, as one which can either be compressed or uncomp WEBVTT X-TIMESTAMP-MAP=LOCAL:00:00:00.000,MPEGTS:900000 194 11:26.520 --> 11:30.700 group together multiple files, as one which can either be compressed or uncomp 195 11:30.700 --> 11:34.000 ressed, we want to use the tar utility. 196 11:34.000 --> 11:38.040 So to understand the differences between these concepts, let's look at how we 197 11:38.040 --> 11:41.000 can use Ansible to automate these particular tasks.