WEBVTT 0:00:03.360000 --> 0:00:07.340000 Log collection, shipping and aggregation. 0:00:07.340000 --> 0:00:14.780000 So now that we have an understanding of the logging process, and we've 0:00:14.780000 --> 0:00:21.020000 taken a look at the Windows logging and Linux logging, I now want to turn 0:00:21.020000 --> 0:00:30.100000 our attention to a very important aspect of the detection side of things 0:00:30.100000 --> 0:00:33.140000 when it comes to security operations. 0:00:33.140000 --> 0:00:37.700000 And of course, you may be asking yourself, why is this important to understand 0:00:37.700000 --> 0:00:39.080000 as an instant responder? 0:00:39.080000 --> 0:00:42.460000 Or you may already have an understanding of this process. 0:00:42.460000 --> 0:00:46.860000 So, you know, an operating system that you're responsible for monitoring 0:00:46.860000 --> 0:00:50.620000 generates logs, now what? 0:00:50.620000 --> 0:00:53.660000 And that's essentially what I seek to address in this video. 0:00:53.660000 --> 0:00:58.380000 So again, if you're familiar with what happens next in terms of the log 0:00:58.380000 --> 0:01:05.540000 pipeline, so how you get those logs from a particular system onto, you 0:01:05.540000 --> 0:01:10.320000 know, how you get those logs from one point to another. 0:01:10.320000 --> 0:01:15.180000 The other point could be a different system, but it's typically going 0:01:15.180000 --> 0:01:20.180000 to be an aggregation point, or let's say a scene that you want to ingest 0:01:20.180000 --> 0:01:22.580000 the logs into, right? 0:01:22.580000 --> 0:01:26.380000 And the reason why you need to understand this process will become apparent 0:01:26.380000 --> 0:01:32.500000 by the end of this video and more importantly, why it is important for 0:01:32.500000 --> 0:01:35.960000 you to understand this process as an incident responder. 0:01:35.960000 --> 0:01:42.140000 So to kick things off, again, you may already have an understanding of 0:01:42.140000 --> 0:01:47.820000 centralized logging as a system or architecture if you're, you know, or 0:01:47.820000 --> 0:01:52.040000 if you're already a SOC analyst or an instant responder, but it's very 0:01:52.040000 --> 0:01:53.720000 important to contextualize this. 0:01:53.720000 --> 0:01:58.400000 So a centralized logging pipeline is the architecture and process through 0:01:58.400000 --> 0:02:03.300000 which logs from, you know, diverse systems or multiple systems are collected, 0:02:03.300000 --> 0:02:08.360000 transported, processed, stored, and analyzed in a central location. 0:02:08.360000 --> 0:02:16.960000 So it's one model whereby, you know, all the logs from all the systems 0:02:16.960000 --> 0:02:20.960000 that you want to monitor. 0:02:20.960000 --> 0:02:25.300000 It's a system or architecture that defines, you know, the process of collecting 0:02:25.300000 --> 0:02:30.060000 those logs, transporting them, processing them, you know, how they're 0:02:30.060000 --> 0:02:31.520000 stored, analyzed, et cetera. 0:02:31.520000 --> 0:02:36.260000 So the key point is that the central location you'll be transferring these 0:02:36.260000 --> 0:02:41.620000 logs to is, you know, typically going to be a seem like Splunk or a log 0:02:41.620000 --> 0:02:44.160000 analytics platform like ELK, right? 0:02:44.160000 --> 0:02:49.020000 And it is essential for incident detection, threat hunting, auditing, 0:02:49.020000 --> 0:02:50.860000 and forensic investigation. 0:02:50.860000 --> 0:02:54.500000 So pretty much what I'm highlighting here, what I'm trying to put across 0:02:54.500000 --> 0:03:00.280000 is, you know, you have systems like just take your own computer and let's 0:03:00.280000 --> 0:03:06.420000 say you wanted to connect it or to ship the logs from your computer to 0:03:06.420000 --> 0:03:08.820000 a scene that you've just set up. 0:03:08.820000 --> 0:03:16.360000 The implication here is that the scene is the centralized location where 0:03:16.360000 --> 0:03:24.020000 you want to, you know, collect ship and aggregate and, you know, analyze, 0:03:24.020000 --> 0:03:28.560000 et cetera, the logs from, or that's where you want the logs to be for 0:03:28.560000 --> 0:03:29.640000 analysis, right? 0:03:29.640000 --> 0:03:34.180000 And so that means you're utilizing the centralized logging architecture 0:03:34.180000 --> 0:03:38.420000 in that there's a centralized place where these logs are going. 0:03:38.420000 --> 0:03:41.700000 Okay, so I just wanted to explain that. 0:03:41.700000 --> 0:03:45.020000 Now, I've sort of outlined in a diagram here. 0:03:45.020000 --> 0:03:48.240000 It's not really a flow chart, but it's sort of a sequential diagram that 0:03:48.240000 --> 0:03:56.360000 outlines the core components or phases or steps of the centralized logging 0:03:56.360000 --> 0:03:59.740000 pipeline. So you first have your log generation. 0:03:59.740000 --> 0:04:01.040000 So these are your sources. 0:04:01.040000 --> 0:04:05.960000 So, you know, the actual devices, the logs are being generated on. 0:04:05.960000 --> 0:04:11.020000 And then you have log collection, which is in the form of agents or forders, 0:04:11.020000 --> 0:04:12.940000 which will get to shortly. 0:04:12.940000 --> 0:04:14.960000 And then you have log shipping. 0:04:14.960000 --> 0:04:18.920000 So this is the transport layer that refers to the process of, you know, 0:04:18.920000 --> 0:04:24.700000 transferring or transporting these logs from the source or the host, right, 0:04:24.700000 --> 0:04:30.460000 operating system, whether logs have been generated to a specific destination. 0:04:30.460000 --> 0:04:34.500000 And then at that point, you have log aggregation and ingestion. 0:04:34.500000 --> 0:04:39.220000 You then have log passing and normalization, storage and indexing, and 0:04:39.220000 --> 0:04:42.360000 then finally, analysis and visualization. 0:04:42.360000 --> 0:04:44.960000 And then from that, you have alerting and response. 0:04:44.960000 --> 0:04:54.840000 So this sort of outlines how you go from the logs on, let's say, using 0:04:54.840000 --> 0:05:00.440000 a CIM or, and consequently, you know, configuring alerts and then using 0:05:00.440000 --> 0:05:06.200000 those alerts to, you know, detect and consequently respond to an incident 0:05:06.200000 --> 0:05:11.560000 as it were. So we're going to go through the three main phases that are 0:05:11.560000 --> 0:05:15.100000 important to understand, right? 0:05:15.100000 --> 0:05:19.580000 And as I said, this will become the reason why this is important will 0:05:19.580000 --> 0:05:22.580000 become apparent by the end of this video. 0:05:22.580000 --> 0:05:25.200000 So let's start off with log collection, right? 0:05:25.200000 --> 0:05:27.380000 So log collection, what is this about? 0:05:27.380000 --> 0:05:30.380000 Well, log collection is the process of gathering log data from various 0:05:30.380000 --> 0:05:34.340000 system services and devices within an IT environment. 0:05:34.340000 --> 0:05:37.880000 And, you know, these logs are typically generated by endpoints or Windows, 0:05:37.880000 --> 0:05:42.220000 Linux systems, network devices like firewalls or routers, applications 0:05:42.220000 --> 0:05:47.020000 like web servers or databases or even security tools like antivirus or 0:05:47.020000 --> 0:05:52.220000 IDS, IPS, etc. And the primary goal of log collection is to centralize 0:05:52.220000 --> 0:05:56.920000 visibility, support, monitoring and prepare for analysis or alerting. 0:05:56.920000 --> 0:05:59.580000 Now, how does log collection work? 0:05:59.580000 --> 0:06:01.100000 And this is very, very important. 0:06:01.100000 --> 0:06:05.920000 You may already be familiar with these systems or, you know, methods of 0:06:05.920000 --> 0:06:10.340000 log collection, but, you know, the key point is that log collection is 0:06:10.340000 --> 0:06:14.200000 the first step in a centralized logging pipeline, which is usually, you 0:06:14.200000 --> 0:06:19.920000 know, the most frequently deployed type of logging architecture. 0:06:19.920000 --> 0:06:26.180000 And regardless of that, log collection is crucial for, as I mentioned, 0:06:26.180000 --> 0:06:29.980000 monitoring detection, alerting and, you know, forensic investigation or 0:06:29.980000 --> 0:06:33.240000 analysis, as it were, in the context of instant response. 0:06:33.240000 --> 0:06:38.160000 Now, key thing here is that logs are either pulled, which means, you know, 0:06:38.160000 --> 0:06:44.320000 queried on demand or pushed, sent automatically, right, to their destination. 0:06:44.320000 --> 0:06:48.420000 In addition to that, collection can also be agent based. 0:06:48.420000 --> 0:06:51.880000 So think of file beat or win log beat if you're not familiar with what 0:06:51.880000 --> 0:06:56.060000 those are in terms of, you know, those agents don't worry. 0:06:56.060000 --> 0:06:57.800000 Or it can be agentless. 0:06:57.800000 --> 0:07:00.740000 An example of that, a good example of that is sys log, right? 0:07:00.740000 --> 0:07:06.000000 Now, logs may be collected in their raw format, or they can be passed 0:07:06.000000 --> 0:07:08.400000 or normalized during ingestion. 0:07:08.400000 --> 0:07:13.820000 And that's entirely up to you or how an organization wants to do that, 0:07:13.820000 --> 0:07:17.120000 right? And, you know, don't worry if you're not familiar with some of 0:07:17.120000 --> 0:07:19.180000 these terms, it will become clearer. 0:07:19.180000 --> 0:07:25.020000 So let's understand push based collection, pull based collection, agent 0:07:25.020000 --> 0:07:29.800000 based collection, and agentless collection, because this is very important. 0:07:29.800000 --> 0:07:32.900000 And again, it will become apparent shortly. 0:07:32.900000 --> 0:07:37.300000 The reason it's important to understand this will become apparent shortly. 0:07:37.300000 --> 0:07:42.260000 So in the, in a push model, the source system, let's say a Linux server 0:07:42.260000 --> 0:07:48.160000 actively sends or pushes, if you will, it's logs to a central destination, 0:07:48.160000 --> 0:07:52.280000 such as a log server or a seam, right? 0:07:52.280000 --> 0:07:53.560000 And how does this work? 0:07:53.560000 --> 0:07:58.040000 Well, a log shipper or agent on the endpoint forwards the logs. 0:07:58.040000 --> 0:08:01.560000 That's why you hear of things like a folder, a log folder. 0:08:01.560000 --> 0:08:06.260000 So in that case, you're dealing with a push based collection, right? 0:08:06.260000 --> 0:08:10.620000 And this is very important to contextualize because each of these systems 0:08:10.620000 --> 0:08:16.440000 have each of these systems or methods of collection have their own advantages 0:08:16.440000 --> 0:08:17.780000 and disadvantages. 0:08:17.780000 --> 0:08:21.580000 And this is where you'll start to see, you know, the importance of understanding 0:08:21.580000 --> 0:08:27.200000 them. That, you know, typically send over TCP or UDP or beats. 0:08:27.200000 --> 0:08:32.260000 Don't worry, we'll explain that in the next video or HTTP protocols. 0:08:32.260000 --> 0:08:35.020000 And the receiving system is passive. 0:08:35.020000 --> 0:08:38.400000 So the seam or the log serve in this case is passive. 0:08:38.400000 --> 0:08:40.940000 What that means is it's waiting for the logs, right? 0:08:40.940000 --> 0:08:45.800000 So it's not proactively reaching out to the source system, right? 0:08:45.800000 --> 0:08:50.040000 To get the logs, hopefully you're now understanding or you probably know 0:08:50.040000 --> 0:08:52.280000 what pull based collection is all about. 0:08:52.280000 --> 0:08:56.880000 In any case, an example of push based collection is our syslog on Linux 0:08:56.880000 --> 0:09:01.440000 pushing logs to a remote server or win log beat pushing windows event 0:09:01.440000 --> 0:09:03.000000 logs to elk, right? 0:09:03.000000 --> 0:09:08.960000 And another example is a firewall pushing logs to via syslog to a seam. 0:09:08.960000 --> 0:09:14.740000 So what are the advantages or benefits and challenges of push based collection? 0:09:14.740000 --> 0:09:20.340000 In terms of the benefits, you know, near time or near real time delivery, 0:09:20.340000 --> 0:09:23.200000 minimal configuration on the central system. 0:09:23.200000 --> 0:09:27.860000 So the receiving system, the system receiving the logs. 0:09:27.860000 --> 0:09:31.300000 What are some of the challenges of push based collection? 0:09:31.300000 --> 0:09:33.760000 The source must be trusted to ship logs. 0:09:33.760000 --> 0:09:37.720000 What that means is you're giving the responsibility of shipping the logs 0:09:37.720000 --> 0:09:43.260000 to the actual system on which the logs are being generated. 0:09:43.260000 --> 0:09:48.420000 So the source. This is another, you know, important point in terms of 0:09:48.420000 --> 0:09:53.120000 the challenge of this system is that delivery can fail silently if network 0:09:53.120000 --> 0:09:57.840000 issues occur. So if there's a network issue or any kind of issue that 0:09:57.840000 --> 0:10:03.080000 affects, you know, the source system or where the logs are coming from, 0:10:03.080000 --> 0:10:05.680000 there's no real indicator. 0:10:05.680000 --> 0:10:10.740000 There's no real way of, you know, getting that notification or being made 0:10:10.740000 --> 0:10:16.620000 aware of that. Because again, you're relying on the, you're relying on 0:10:16.620000 --> 0:10:20.860000 the source system to send the logs to the seam, for example. 0:10:20.860000 --> 0:10:25.320000 Now, in the inverse with pull based collection, the seam, because the 0:10:25.320000 --> 0:10:30.300000 seam tries to reach out to the source, if it is unable to reach the source, 0:10:30.300000 --> 0:10:33.620000 you'll get notified of that via, you know, the seam, the seam telling 0:10:33.620000 --> 0:10:34.680000 you, I should say. 0:10:34.680000 --> 0:10:37.160000 So that brings us to pull based collection. 0:10:37.160000 --> 0:10:42.020000 So in the pull model, the central system, let's say an example being the 0:10:42.020000 --> 0:10:43.000000 seam or collector. 0:10:43.000000 --> 0:10:46.920000 So whenever you hear collector, just know you're dealing with a pull based 0:10:46.920000 --> 0:10:53.200000 collection. So the central system requests or indeed pulls logs from endpoints 0:10:53.200000 --> 0:10:57.600000 on a schedule or trigger, which you can define, right? 0:10:57.600000 --> 0:11:00.940000 Typically, every couple of minutes or something like this. 0:11:00.940000 --> 0:11:02.840000 And how does this work exactly? 0:11:02.840000 --> 0:11:05.960000 So the collector authenticates with the source system. 0:11:05.960000 --> 0:11:10.200000 That's very important, because, you know, you need to have authentication 0:11:10.200000 --> 0:11:17.960000 in place. And then it utilizes SSH, WMI on Windows, an API or shared folders 0:11:17.960000 --> 0:11:19.700000 to retrieve the logs. 0:11:19.700000 --> 0:11:24.180000 And it's suitable for systems that cannot run agents and will get into 0:11:24.180000 --> 0:11:28.620000 agent based collection and agent list collection shortly. 0:11:28.620000 --> 0:11:34.560000 But some examples of the pull based model are, you know, a seam pulling 0:11:34.560000 --> 0:11:40.200000 logs via WMI from Windows machines or Windows endpoints, pulling Apache 0:11:40.200000 --> 0:11:46.660000 logs. This is quite common from a web server over SSH or SAP using SNMP 0:11:46.660000 --> 0:11:52.600000 or APIs to collect logs from devices, you know, different variety of devices 0:11:52.600000 --> 0:11:56.700000 where, you know, an agent may not be available, or you can't install an 0:11:56.700000 --> 0:11:59.020000 agent on that particular endpoints. 0:11:59.020000 --> 0:12:02.880000 So think of, you know, IoT devices, which in most cases where you'll see 0:12:02.880000 --> 0:12:04.780000 SNMP being used. 0:12:04.780000 --> 0:12:12.520000 So pull definitely is quite robust in terms of the fact that the requirements 0:12:12.520000 --> 0:12:18.780000 on the on the end point are fairly limited or minimal. 0:12:18.780000 --> 0:12:21.660000 But let's take a look at the benefits and challenges. 0:12:21.660000 --> 0:12:25.420000 So the benefits of pull based collection are, you know, there's a centralized 0:12:25.420000 --> 0:12:29.940000 control of collection, you know, for obvious reasons. 0:12:29.940000 --> 0:12:31.700000 You probably understand by now. 0:12:31.700000 --> 0:12:35.720000 And as I mentioned, I was trying to allude to a few seconds ago, it's 0:12:35.720000 --> 0:12:40.140000 useful for legacy systems or those where agents are impractical. 0:12:40.140000 --> 0:12:43.620000 So you can really, you know, install an agent because you need an agent 0:12:43.620000 --> 0:12:47.520000 to facilitate, you know, the previous or the other model, which is the 0:12:47.520000 --> 0:12:49.500000 push based model. 0:12:49.500000 --> 0:12:53.840000 In any case, the challenges are there's a higher network load. 0:12:53.840000 --> 0:12:59.020000 The pull intervals can create latency or latency issues and it requires 0:12:59.020000 --> 0:13:01.640000 secure and reliable access to source systems. 0:13:01.640000 --> 0:13:05.600000 So because you're giving the responsibility to the collector or indeed 0:13:05.600000 --> 0:13:10.660000 the, the seem, let's say, to reach out to these endpoints to collect the 0:13:10.660000 --> 0:13:14.480000 logs. You know, firstly, they require authentication, but you're, it's 0:13:14.480000 --> 0:13:16.920000 sort of a single point of failure. 0:13:16.920000 --> 0:13:21.400000 But yeah, that's pull based collection. 0:13:21.400000 --> 0:13:25.180000 So let's talk a little bit about agent based collection and agent less 0:13:25.180000 --> 0:13:30.060000 collection, which, you know, sort of a subset of the earlier two models. 0:13:30.060000 --> 0:13:33.600000 But it's very important to understand these concepts when you have an 0:13:33.600000 --> 0:13:35.260000 agent. What does that mean? 0:13:35.260000 --> 0:13:40.600000 So in terms of the agent based collection model, a lightweight agent is 0:13:40.600000 --> 0:13:42.280000 installed on the host system. 0:13:42.280000 --> 0:13:46.500000 So if you've ever used was a or was who and you've installed the agent 0:13:46.500000 --> 0:13:50.040000 on your, on an endpoint, this year, you're using agent based collection, 0:13:50.040000 --> 0:13:54.820000 right? And the way agent based collection works is it monitors logs in 0:13:54.820000 --> 0:13:59.260000 real time and sends them to a log processor or directly to the seem now 0:13:59.260000 --> 0:14:04.720000 seem will obviously most seems comprised of a log processor as part of 0:14:04.720000 --> 0:14:06.820000 the overall seem package. 0:14:06.820000 --> 0:14:12.140000 So it's not, it's not that important to, you know, get tangled with, you 0:14:12.140000 --> 0:14:14.800000 know, the difference between a process, etc. 0:14:14.800000 --> 0:14:16.500000 So how does this work? 0:14:16.500000 --> 0:14:22.220000 The agent reads log files or system APIs formats and forwards the logs, 0:14:22.220000 --> 0:14:26.680000 typically in JSON format and we'll talk about formatting in the next video. 0:14:26.680000 --> 0:14:34.580000 And another thing is that the, the agent, this model allows for, you know, 0:14:34.580000 --> 0:14:39.620000 the addition of metadata or tags before shipping. 0:14:39.620000 --> 0:14:45.660000 Therefore, doing a lot of the normalization for you from the, you know, 0:14:45.660000 --> 0:14:47.420000 perspective of the seem. 0:14:47.420000 --> 0:14:50.260000 So it's not done at point of ingestion. 0:14:50.260000 --> 0:14:53.440000 Some examples of this are, you know, using file beat. 0:14:53.440000 --> 0:14:56.900000 Don't worry if you don't know what that is using file beat for Linux log 0:14:56.900000 --> 0:15:01.820000 files or win log beat for Windows event logs or, you know, an ex log for 0:15:01.820000 --> 0:15:04.100000 Sysmon or advanced Windows telemetry. 0:15:04.100000 --> 0:15:07.260000 So hopefully you're understanding that agent based collection, you typically 0:15:07.260000 --> 0:15:14.460000 see this in the, you typically see this in the push based, you know, push 0:15:14.460000 --> 0:15:15.820000 collection model. 0:15:15.820000 --> 0:15:19.720000 You know, I should have put this under, but I just wanted to explain the 0:15:19.720000 --> 0:15:22.600000 difference between push and pull first and then, you know, I mentioned 0:15:22.600000 --> 0:15:27.820000 agents there. So now I'm going to explain agent versus agent less collection 0:15:27.820000 --> 0:15:35.220000 in terms of them being the, the means through which push and pull based 0:15:35.220000 --> 0:15:37.960000 models of collection are facilitated. 0:15:37.960000 --> 0:15:44.440000 So agent based collection is obviously most likely associated, not most 0:15:44.440000 --> 0:15:50.780000 likely is associated with the push model. 0:15:50.780000 --> 0:15:54.360000 So, if we take a look at the benefits of agent based collection, it's, 0:15:54.360000 --> 0:15:57.020000 you know, efficient and real time resilient. 0:15:57.020000 --> 0:16:01.080000 So local buffering retry mechanisms, because there's an agent in place 0:16:01.080000 --> 0:16:05.720000 and an agent can be as custom as sophisticated as you want it to be. 0:16:05.720000 --> 0:16:11.560000 And as I mentioned in the previous slide, agents usually have the ability 0:16:11.560000 --> 0:16:14.960000 to enrich or filter logs before shipping. 0:16:14.960000 --> 0:16:19.540000 So you can actually customize what you want shipped quite in a quite fine, 0:16:19.540000 --> 0:16:27.300000 in a quite fine grained manner or granular manner in a granular way, I 0:16:27.300000 --> 0:16:31.560000 should say. Some of the challenges are that it requires installation and 0:16:31.560000 --> 0:16:32.960000 maintenance on every endpoint. 0:16:32.960000 --> 0:16:34.800000 So you need to ensure this is usually the case. 0:16:34.800000 --> 0:16:39.160000 Trust me, I've done this, I've dealt with this headache before. 0:16:39.160000 --> 0:16:42.480000 If the, the, the SEAM gets a major update, you typically need to update 0:16:42.480000 --> 0:16:44.760000 the agents typically, not always the case. 0:16:44.760000 --> 0:16:48.040000 There's usually, you know, some backwards compatibility, but at a certain 0:16:48.040000 --> 0:16:52.400000 point in time, you will need to update the agents on each of the endpoints. 0:16:52.400000 --> 0:16:56.760000 Now, of course, that can be automated, you know, via administration, but 0:16:56.760000 --> 0:16:58.580000 we'll not get into that. 0:16:58.580000 --> 0:17:01.900000 It's still something that adds to the overhead. 0:17:01.900000 --> 0:17:05.480000 And obviously, it may raise concerns in tightly controlled environments. 0:17:05.480000 --> 0:17:10.980000 So you typically not see agent based collection, or generally speaking, 0:17:10.980000 --> 0:17:17.300000 the, the push model being used in, you know, ICS environments, you know, 0:17:17.300000 --> 0:17:20.820000 industrial control systems, SCADA environments, etc. 0:17:20.820000 --> 0:17:23.700000 And then you have agentless collection, right? 0:17:23.700000 --> 0:17:27.160000 So agentless methods don't require installing anything on the source. 0:17:27.160000 --> 0:17:33.180000 Instead, they rely on native protocols and open ports to collect logs. 0:17:33.180000 --> 0:17:34.300000 So how does it work? 0:17:34.300000 --> 0:17:38.440000 You know, users, WMI, SSH, SysLog or API polling. 0:17:38.440000 --> 0:17:41.840000 And typically involves periodic access from a collector. 0:17:41.840000 --> 0:17:48.420000 So this obviously is associated with the pull collection method. 0:17:48.420000 --> 0:17:52.580000 So examples here are collecting logs from windows using WMI, gathering 0:17:52.580000 --> 0:17:57.480000 logs from network devices via SysLog, querying audit logs from a web app 0:17:57.480000 --> 0:17:59.520000 via, you know, REST API. 0:17:59.520000 --> 0:18:04.420000 In terms of agentless collection, the benefits and challenges are as follows. 0:18:04.420000 --> 0:18:07.940000 So the benefits here, no agent deployment overhead. 0:18:07.940000 --> 0:18:10.040000 So you don't have to deal with that issue. 0:18:10.040000 --> 0:18:13.440000 And it's much easier to manage, you know, deploy, manage, etc. 0:18:13.440000 --> 0:18:18.700000 in environments with strict software policies in direct juxtaposition 0:18:18.700000 --> 0:18:23.040000 to the, you know, challenges of agent based collection. 0:18:23.040000 --> 0:18:27.440000 So the benefits here are pretty much the disadvantages of agent based 0:18:27.440000 --> 0:18:32.100000 collection. So the challenges of agentless collection are, you know, less 0:18:32.100000 --> 0:18:35.820000 reliable or slower in comparison to agent based collection. 0:18:35.820000 --> 0:18:39.220000 And, you know, this is something that's quite common. 0:18:39.220000 --> 0:18:43.600000 You typically may miss transient logs due to a polling delay. 0:18:43.600000 --> 0:18:47.520000 This is something that you'll typically see when you're using a collector. 0:18:47.520000 --> 0:18:52.640000 You know, there's typically a queue or a schedule and, you know, depending 0:18:52.640000 --> 0:18:56.120000 on the number of endpoints you're monitoring, you need to really factor 0:18:56.120000 --> 0:19:03.280000 in how this is all orchestrated to avoid, you know, a delay, a polling 0:19:03.280000 --> 0:19:07.840000 delay. And the other challenge is, you know, it's complex to secure and 0:19:07.840000 --> 0:19:11.920000 to scale. So now that you understand that, we can talk about the next 0:19:11.920000 --> 0:19:20.820000 phase of the log, the logging pipeline, as it were, which is log shipping. 0:19:20.820000 --> 0:19:24.320000 So we've talked about collection and how this is facilitated. 0:19:24.320000 --> 0:19:28.500000 But now let's take a look at the process of log shipping. 0:19:28.500000 --> 0:19:31.580000 So what is log shipping now? 0:19:31.580000 --> 0:19:35.200000 Log shipping is a term you don't see anymore. 0:19:35.200000 --> 0:19:40.140000 But what it's referring to really simply is the process of how the collected 0:19:40.140000 --> 0:19:46.780000 logs are transferred, right, from the source system to a centralized destination, 0:19:46.780000 --> 0:19:59.780000 typically a log analysis platform or a C, right, as I've just stated here. 0:19:59.780000 --> 0:20:06.280000 Log analysis platform or CIM, right, and examples here would be elk, or 0:20:06.280000 --> 0:20:09.480000 Splunk, etc. So what's the importance of log shipping? 0:20:09.480000 --> 0:20:14.460000 You know, the importance is fairly obvious, but, you know, it enables 0:20:14.460000 --> 0:20:18.020000 centralized analysis and correlation prevents logs from being tampered 0:20:18.020000 --> 0:20:19.220000 with on compromise systems. 0:20:19.220000 --> 0:20:23.400000 So because you're also shipping the logs or you're transferring them to 0:20:23.400000 --> 0:20:30.340000 this centralized location, log analysis platform or CIM, you know, you 0:20:30.340000 --> 0:20:34.040000 essentially have a form of redundancy. 0:20:34.040000 --> 0:20:38.240000 Because if on the end point the logs are deleted, you still have them 0:20:38.240000 --> 0:20:45.580000 on the centralized endpoint, whether it's a log analysis platform or CIM, 0:20:45.580000 --> 0:20:50.400000 which is why a lot of organizations, so you'll typically see, let's, you 0:20:50.400000 --> 0:20:52.460000 know, an example being Linux servers. 0:20:52.460000 --> 0:21:00.200000 They need not necessarily be having log shipped to a CIM for security 0:21:00.200000 --> 0:21:07.280000 monitoring. They may just be logs may just be shipped to a syslog server 0:21:07.280000 --> 0:21:11.400000 for as a form of redundancy or backup. 0:21:11.400000 --> 0:21:15.920000 And obviously it supports real time alerting and automated detection and, 0:21:15.920000 --> 0:21:18.960000 you know, ensures long term storage and compliance. 0:21:18.960000 --> 0:21:21.860000 So log collection versus log shipping. 0:21:21.860000 --> 0:21:23.740000 So there's a comparison. 0:21:23.740000 --> 0:21:27.420000 The comparison points are the purpose, examples, destination, transport 0:21:27.420000 --> 0:21:32.000000 layer, right? Which only applies to shipping, but in the case of the purpose, 0:21:32.000000 --> 0:21:37.000000 log collection as I've already mentioned is the, yeah, the log collection 0:21:37.000000 --> 0:21:41.980000 deals with gathering logs from local system, local or system sources. 0:21:41.980000 --> 0:21:45.540000 Log shipping deals with sending these logs to a centralized platform, 0:21:45.540000 --> 0:21:47.040000 whatever that may be. 0:21:47.040000 --> 0:21:51.560000 Some examples of log collection tools would be, you know, or the D, general 0:21:51.560000 --> 0:21:53.720000 D, then via these are, you know, very basic. 0:21:53.720000 --> 0:21:58.700000 I've not talked about the ones that facilitate the collection and shipping, 0:21:58.700000 --> 0:22:02.820000 which is, in this case, you'd find things like file beat, win log beat, 0:22:02.820000 --> 0:22:05.760000 our syslog, syslog, ng, etc. 0:22:05.760000 --> 0:22:10.480000 In terms of the destination, log collection is typically the local store 0:22:10.480000 --> 0:22:12.420000 or a temporary buffer. 0:22:12.420000 --> 0:22:16.320000 But in the case of log shipping, the destination is think of things like 0:22:16.320000 --> 0:22:21.700000 log stash, elastic search or just the actual, you know, think of Splunk 0:22:21.700000 --> 0:22:24.900000 or the actual CIM for all intents and purposes. 0:22:24.900000 --> 0:22:28.840000 As for the transport layer, in the case of log shipping, you have, you 0:22:28.840000 --> 0:22:34.420000 know, things like syslog, beats, Kafka, HTTP API or, you know, standard 0:22:34.420000 --> 0:22:37.160000 TCP UDP transmission. 0:22:37.160000 --> 0:22:40.060000 So what are some common log shipping tools? 0:22:40.060000 --> 0:22:44.920000 I mentioned them in my descriptions of the, or my explanations of pull 0:22:44.920000 --> 0:22:48.900000 and push as well as agent and agentless collection. 0:22:48.900000 --> 0:22:53.160000 So we have, you know, I sort of have the tool, the operating system and 0:22:53.160000 --> 0:22:56.980000 the use case. So let's say, let's take file beat, which is specific to 0:22:56.980000 --> 0:23:00.900000 Linux. It's the use case is to ship flat log files. 0:23:00.900000 --> 0:23:05.140000 So think of the log files stored under var log, which I already explained. 0:23:05.140000 --> 0:23:09.200000 And then win log beat, which, you know, self explanatory, it's specific 0:23:09.200000 --> 0:23:13.560000 to Windows. Its use case or purpose is typically to ship Windows event 0:23:13.560000 --> 0:23:18.260000 logs. Our syslog or syslog next generation, that specific to Linux, its 0:23:18.260000 --> 0:23:21.220000 use cases to ship logs using the syslog format. 0:23:21.220000 --> 0:23:24.000000 Very important. You then have log stash. 0:23:24.000000 --> 0:23:25.700000 This is cross platform. 0:23:25.700000 --> 0:23:31.520000 Its use case is not really log shipping per se, but it passes and reaches 0:23:31.520000 --> 0:23:33.660000 and then forwards the logs. 0:23:33.660000 --> 0:23:39.260000 You also have an X log, which is an alternative to win log beat and, you 0:23:39.260000 --> 0:23:40.500000 know, allows for custom logs. 0:23:40.500000 --> 0:23:42.040000 That's specific to Windows. 0:23:42.040000 --> 0:23:45.200000 You then have fluent D, which is cross platform. 0:23:45.200000 --> 0:23:48.380000 And, you know, use cases, flexible plugin based shipping. 0:23:48.380000 --> 0:23:50.140000 Not really important at this point. 0:23:50.140000 --> 0:23:54.420000 The key ones you'll typically be encountering are file beat, win log beat, 0:23:54.420000 --> 0:23:57.240000 or syslog for obvious reasons, log stash. 0:23:57.240000 --> 0:23:58.880000 Those are the main ones, right? 0:23:58.880000 --> 0:24:00.540000 And you may already be familiar with them. 0:24:00.540000 --> 0:24:04.500000 So that brings us to the security best practices when it comes to log 0:24:04.500000 --> 0:24:08.500000 shipping. Because, you know, you're transferring very important information 0:24:08.500000 --> 0:24:12.160000 from this source to a centralized destination. 0:24:12.160000 --> 0:24:17.220000 The actual data in transit is, you know, very important. 0:24:17.220000 --> 0:24:19.360000 Or the data that you're transferring is very important. 0:24:19.360000 --> 0:24:23.620000 So what are the security best practices that need to be applied here? 0:24:23.620000 --> 0:24:28.420000 Well, you need to obviously use TLS encryption to secure logs in transit. 0:24:28.420000 --> 0:24:31.720000 Buffer logs locally to avoid data loss during downtime. 0:24:31.720000 --> 0:24:37.800000 So you create your utilize a buffer to avoid data loss during any potential 0:24:37.800000 --> 0:24:40.920000 downtime. And then you enable authentication and signature verification 0:24:40.920000 --> 0:24:47.140000 if it is supported and separate log collection infrastructure from production 0:24:47.140000 --> 0:24:49.400000 systems. Very, very important. 0:24:49.400000 --> 0:24:53.320000 So finally, that brings us to log aggregation, which by this point should 0:24:53.320000 --> 0:24:56.820000 be, you know, you should already understand it to a certain extent. 0:24:56.820000 --> 0:25:00.940000 But log aggregation is the process of collecting and combining log data 0:25:00.940000 --> 0:25:05.260000 from multiple sources into a single centralized system for easier search, 0:25:05.260000 --> 0:25:07.300000 correlation, analysis, and monitoring. 0:25:07.300000 --> 0:25:11.840000 So log aggregation typically happens on the centralized destination. 0:25:11.840000 --> 0:25:14.020000 Think of your log analysis. 0:25:14.020000 --> 0:25:17.560000 Sorry. Log just think of your seem. 0:25:17.560000 --> 0:25:21.340000 So your seem performs the aggregation as well as other things, which I 0:25:21.340000 --> 0:25:24.420000 mentioned in the centralized logging pipeline. 0:25:24.420000 --> 0:25:28.900000 But log aggregation acts as the consolidation phase in the log pipeline 0:25:28.900000 --> 0:25:33.660000 and is essential for building situational awareness and enabling effective 0:25:33.660000 --> 0:25:36.260000 incident detection and response. 0:25:36.260000 --> 0:25:41.240000 So modern environments generate logs from a wide range of systems and 0:25:41.240000 --> 0:25:43.420000 these logs are often scattered. 0:25:43.420000 --> 0:25:46.240000 They use different formats and are stored in different location. 0:25:46.240000 --> 0:25:51.320000 Log aggregation solves these problems by unifying logs, therefore allowing 0:25:51.320000 --> 0:25:57.980000 you or responders, generally speaking to a view and analyze logs from 0:25:57.980000 --> 0:26:02.300000 one place or one, you know, centralized location. 0:26:02.300000 --> 0:26:05.800000 One centralized platform, I should say, perform the correlation across 0:26:05.800000 --> 0:26:10.940000 systems. So an example would be login from X followed by alert on Y. 0:26:10.940000 --> 0:26:13.980000 That's what correlation is all about. 0:26:13.980000 --> 0:26:17.740000 You know, C detect patterns and anomalies across hosts networks. 0:26:17.740000 --> 0:26:23.440000 Pretty much any system or endpoint that you're monitoring and de accelerate 0:26:23.440000 --> 0:26:28.460000 investigations and reduce detection time. 0:26:28.460000 --> 0:26:32.760000 So that's the importance of, you know, that's log aggregation and the 0:26:32.760000 --> 0:26:37.880000 importance of log aggregation specific to you as an instant responder. 0:26:37.880000 --> 0:26:43.600000 So now I want to discuss the, you know, I mentioned it earlier on in this 0:26:43.600000 --> 0:26:49.360000 video that why is understanding Collection shipping and aggregation important 0:26:49.360000 --> 0:26:51.700000 for you as an instant responder. 0:26:51.700000 --> 0:26:55.140000 And of course, I'm taking into account the fact that you may already know 0:26:55.140000 --> 0:26:59.340000 about this, because again, you may be a SOC analyst or you may be an instant 0:26:59.340000 --> 0:27:00.380000 responder already. 0:27:00.380000 --> 0:27:04.140000 But I think it's very important to, you know, sort of contextualize why 0:27:04.140000 --> 0:27:06.460000 we're going through these topics. 0:27:06.460000 --> 0:27:10.420000 So over here, you know, I sort of have the components here. 0:27:10.420000 --> 0:27:14.180000 So log collection, shipping, aggregation, the definitions and why it matters 0:27:14.180000 --> 0:27:17.560000 to instant response or to an instant responder. 0:27:17.560000 --> 0:27:21.240000 In the case of log collection, this, you know, as I mentioned is the process 0:27:21.240000 --> 0:27:25.820000 of the process of gathering logs from endpoint servers, apps and devices. 0:27:25.820000 --> 0:27:29.980000 Why it matters to our, you know, it ensures all relevant security events 0:27:29.980000 --> 0:27:34.540000 are captured for monitoring and analysis in terms of log shipping. 0:27:34.540000 --> 0:27:40.380000 The definition being, you know, the, that this is the process of transmitting 0:27:40.380000 --> 0:27:43.180000 logs from source to central location. 0:27:43.180000 --> 0:27:45.600000 So think of, you know, C, L, et cetera. 0:27:45.600000 --> 0:27:47.420000 Why is this important? 0:27:47.420000 --> 0:27:53.600000 It enables reliable timely delivery of logs for detection and investigation. 0:27:53.600000 --> 0:27:59.820000 So hopefully understanding the implicit role that log shipping plays or 0:27:59.820000 --> 0:28:07.780000 let's say, well thought out log shipping plays to you as an instant responder. 0:28:07.780000 --> 0:28:13.280000 Because if your log shipping is mod by latency and delays and you're not 0:28:13.280000 --> 0:28:19.520000 getting, you know, logs, let's say, in almost real time, you're not going 0:28:19.520000 --> 0:28:21.780000 to be able to detect incidents. 0:28:21.780000 --> 0:28:25.920000 You know, as they happen, you're going to find out after the fact and 0:28:25.920000 --> 0:28:29.920000 as a result, the response is going to be affected. 0:28:29.920000 --> 0:28:33.920000 So hopefully you're understanding why I'm sort of covering this. 0:28:33.920000 --> 0:28:36.220000 And then log aggregation. 0:28:36.220000 --> 0:28:39.500000 So, you know, this is centralizing logs from multiple sources for unified 0:28:39.500000 --> 0:28:43.060000 analysis. Why is this important in incident response? 0:28:43.060000 --> 0:28:45.560000 Or why is it important to you as an instant responder? 0:28:45.560000 --> 0:28:49.440000 You know, is that, you know, it supports correlation, visibility and timeline 0:28:49.440000 --> 0:28:53.120000 reconstruction across the environment, which, you know, I've already sort 0:28:53.120000 --> 0:28:56.740000 of alluded to. But that's the importance in incident response. 0:28:56.740000 --> 0:29:00.620000 Now, I sort of outlined some key terminology here that, again, I'll not 0:29:00.620000 --> 0:29:04.820000 be going through, you know, in its entirety because there's quite a bit 0:29:04.820000 --> 0:29:08.260000 that I've outlined here. 0:29:08.260000 --> 0:29:12.580000 You know, and a lot you already know, but these are the terms that you'll 0:29:12.580000 --> 0:29:15.960000 typically be dealing with as an incident responder, not directly, not 0:29:15.960000 --> 0:29:18.320000 every day, but you need to be aware of it. 0:29:18.320000 --> 0:29:22.080000 If you are again working in the blue team or your SOC analyst or instant 0:29:22.080000 --> 0:29:25.420000 responder, then this should already be familiar to you. 0:29:25.420000 --> 0:29:37.500000 So log collector or agent, you know, I already explained that. 0:29:37.500000 --> 0:29:39.480000 So let's say ED of visibility. 0:29:39.480000 --> 0:29:40.660000 What is an aggregator? 0:29:40.660000 --> 0:29:45.040000 This is a system that receives and consolidates logs from multiple sources. 0:29:45.040000 --> 0:29:47.720000 So log stash is an example there. 0:29:47.720000 --> 0:29:51.700000 And it centralizes data for easy analysis and correlation. 0:29:51.700000 --> 0:29:52.940000 You then have the shipper. 0:29:52.940000 --> 0:29:55.220000 This is the component or agent. 0:29:55.220000 --> 0:29:58.640000 So hopefully understanding now what agents do as well. 0:29:58.640000 --> 0:30:02.100000 So it's the component or agent that forwards collected logs to a centralized 0:30:02.100000 --> 0:30:03.380000 platform or SEAM. 0:30:03.380000 --> 0:30:07.180000 And it enables secure and reliable delivery of logs for detection and 0:30:07.180000 --> 0:30:12.560000 analysis. The beats protocol, I mentioned this, you know, not directly, 0:30:12.560000 --> 0:30:17.520000 but this is a lightweight protocol utilized by elastic beats, which is 0:30:17.520000 --> 0:30:22.080000 part of the elastic stack or elk stack as it were. 0:30:22.080000 --> 0:30:28.860000 So the example here being, you know, it's the protocol used by the elastic 0:30:28.860000 --> 0:30:31.780000 beats, examples being file beat, win log beat, etc. 0:30:31.780000 --> 0:30:35.880000 to send logs. So whenever you hear beats protocol, just know it's referring 0:30:35.880000 --> 0:30:43.020000 to the protocol utilized by elastic beats, which again comprises of file 0:30:43.020000 --> 0:30:47.480000 beat, win log beat, etc, which is specific to the elk stack. 0:30:47.480000 --> 0:30:51.860000 So, you know, you can see, I pretty much explained it in the next column. 0:30:51.860000 --> 0:30:55.460000 So the relevant string student response is that it's preferred for structured 0:30:55.460000 --> 0:30:59.800000 reliable log transmission in elk based environments or environments, you 0:30:59.800000 --> 0:31:02.400000 know, using elk. 0:31:02.400000 --> 0:31:06.180000 Then the log format, which we'll cover in the next video, but it's very 0:31:06.180000 --> 0:31:07.860000 important to understand now. 0:31:07.860000 --> 0:31:11.460000 This is the structure in which a log message is recorded. 0:31:11.460000 --> 0:31:16.000000 So think of JSON, sys log or SEF. 0:31:16.000000 --> 0:31:19.060000 And the reason this is important is because it affects how easily logs 0:31:19.060000 --> 0:31:21.300000 can be passed index. 0:31:21.300000 --> 0:31:25.580000 Generally speaking understood, but more importantly correlated. 0:31:25.580000 --> 0:31:27.360000 And then log ingestion. 0:31:27.360000 --> 0:31:31.120000 This is the process of receiving and passing logs into a storage or analysis 0:31:31.120000 --> 0:31:34.820000 system. So this is typically done by the SEAM. 0:31:34.820000 --> 0:31:37.860000 So, you know, it's crucial for ensuring that logs are stored searchable 0:31:37.860000 --> 0:31:41.800000 and usable. You then have normalization, which I mentioned. 0:31:41.800000 --> 0:31:45.440000 So this is converting different log formats into a standard structure. 0:31:45.440000 --> 0:31:47.740000 So, you know, fields, labels, etc. 0:31:47.740000 --> 0:31:51.640000 And the reason this is important is because it enables cross source correlation 0:31:51.640000 --> 0:31:57.520000 and detection. You then have passing, which is extracting structured fields 0:31:57.520000 --> 0:31:59.260000 of raw log data. 0:31:59.260000 --> 0:32:03.400000 So think of timestamp, user action, etc. 0:32:03.400000 --> 0:32:08.400000 You know, to essentially, you know, get the information that is relevant 0:32:08.400000 --> 0:32:12.960000 to you. So it's needed to create detection rules, dashboards and alerts. 0:32:12.960000 --> 0:32:15.240000 And then you have buffering, which I mentioned. 0:32:15.240000 --> 0:32:19.440000 This is the temporary local storage of logs during network interruptions 0:32:19.440000 --> 0:32:24.380000 or load. So if, for example, in the push model, the push collection model 0:32:24.380000 --> 0:32:28.040000 where the agent is responsible or sorry, the source or the end point is 0:32:28.040000 --> 0:32:36.000000 responsible for pushing or sending the logs to a centralized location. 0:32:36.000000 --> 0:32:41.760000 SEAM, for example, if it cannot, for whatever reason, you know, perform 0:32:41.760000 --> 0:32:45.680000 the push, let's say, then there needs to be a buffer that actually holds 0:32:45.680000 --> 0:32:52.340000 the logs that have been generated during this disruption or interruption 0:32:52.340000 --> 0:33:01.400000 as it were. Once connection is reestablished, the buffer takes precedent. 0:33:01.400000 --> 0:33:05.020000 It's been maintaining these logs, you know, so it then sends everything 0:33:05.020000 --> 0:33:08.980000 that was missed or that was supposed to be sent, but wasn't able to be 0:33:08.980000 --> 0:33:19.260000 sent, sends it and then resumes, you know, operations resume normally. 0:33:19.260000 --> 0:33:22.480000 So the relevance to incident response is fairly obvious. 0:33:22.480000 --> 0:33:25.820000 It prevents data loss during outages or SEAM downtime. 0:33:25.820000 --> 0:33:31.580000 The inverse is also true in terms of the model, but you also have the 0:33:31.580000 --> 0:33:32.700000 folder which I explained. 0:33:32.700000 --> 0:33:37.400000 This is another term for a shipper or lightweight agent that sends logs. 0:33:37.400000 --> 0:33:39.960000 So push model, right? 0:33:39.960000 --> 0:33:43.760000 And the relevance to incident response is that it ensures continuous log 0:33:43.760000 --> 0:33:47.780000 flow to central analysis tools or centralized location, right? 0:33:47.780000 --> 0:33:49.400000 You then have indexing. 0:33:49.400000 --> 0:33:52.660000 So this is the process of organizing and storing logs in a searchable 0:33:52.660000 --> 0:33:56.420000 structure. So think of the elastic search index in the elk stack. 0:33:56.420000 --> 0:34:00.880000 This makes it possible the query and correlate logs quickly. 0:34:00.880000 --> 0:34:02.320000 And then log enrichment. 0:34:02.320000 --> 0:34:03.460000 That's just one here. 0:34:03.460000 --> 0:34:03.980000 That's quite important. 0:34:03.980000 --> 0:34:07.060000 This is the process of adding context to logs. 0:34:07.060000 --> 0:34:12.460000 So think of things like geolocation, host name, user details. 0:34:12.460000 --> 0:34:16.160000 And this is extremely relevant or important to instant response or to 0:34:16.160000 --> 0:34:20.580000 you as an incident responder because it improves incident triage, detection, 0:34:20.580000 --> 0:34:22.860000 accuracy and investigation speed. 0:34:22.860000 --> 0:34:26.280000 Now, you may be saying to yourself, well, how is this important to me 0:34:26.280000 --> 0:34:27.600000 as an incident responder? 0:34:27.600000 --> 0:34:34.720000 Well, the reason this is important to you is because it forms the reason 0:34:34.720000 --> 0:34:40.600000 it's important to use because it's important to the SOC team or let's 0:34:40.600000 --> 0:34:47.500000 say SOC tier one and list that is going to perform the initial analysis 0:34:47.500000 --> 0:34:51.680000 or initial triage. 0:34:51.680000 --> 0:34:57.240000 And if in the case of log enrichment or let's say, you know, collection, 0:34:57.240000 --> 0:35:01.980000 shipping and aggregation, if there's any weaknesses in that process, then, 0:35:01.980000 --> 0:35:06.220000 you know, their decision making process as to whether or not to escalate, 0:35:06.220000 --> 0:35:11.120000 you know, an incident to you, the instant responder will be affected either 0:35:11.120000 --> 0:35:15.700000 in terms of delays, either in terms of the information contained as part 0:35:15.700000 --> 0:35:18.880000 of, you know, what is escalated to you. 0:35:18.880000 --> 0:35:22.320000 So if they only tell you what they know, which, you know, is not good 0:35:22.320000 --> 0:35:26.960000 enough or could be improved in terms of log enrichment, then you are starting 0:35:26.960000 --> 0:35:28.320000 at a disadvantage. 0:35:28.320000 --> 0:35:30.960000 So that's why this is important. 0:35:30.960000 --> 0:35:34.140000 But again, you know, that's just one example. 0:35:34.140000 --> 0:35:36.900000 So with that being said, that's going to be it for this video. 0:35:36.900000 --> 0:35:38.880000 And I will be seeing you in the next video.