WEBVTT

0:00:03.700000 --> 0:00:07.780000
 Log shipping protocols and log formats.

0:00:07.780000 --> 0:00:13.700000
 So in this video, we're going to be
 getting an understanding as to the

0:00:13.700000 --> 0:00:19.600000
 various different log formats out there,
 or what you're likely to encounter,

0:00:19.600000 --> 0:00:21.800000
 as well as the log shipping protocols.

0:00:21.800000 --> 0:00:29.680000
 So the network protocols that facilitate
 the shipping of logs from source

0:00:29.680000 --> 0:00:34.080000
 to the centralized to a centralized
 location, regardless of what that

0:00:34.080000 --> 0:00:37.860000
 is, most cases are going to be a seam.

0:00:37.860000 --> 0:00:45.060000
 So let's understand, let's sort of get
 a contextual understanding of what

0:00:45.060000 --> 0:00:52.440000
 this is, you know, log shipping
 protocols and formats.

0:00:52.440000 --> 0:00:58.680000
 And this way, you know, it'll sort
 of give you an, this will give you

0:00:58.680000 --> 0:01:02.560000
 an understanding of why this
 is important to you.

0:01:02.560000 --> 0:01:09.520000
 But before we get into any of them, let's
 go through this contextual explanation

0:01:09.520000 --> 0:01:14.600000
 or breakdown. So in the context of centralized
 logging, log shipping protocols

0:01:14.600000 --> 0:01:19.420000
 define how logs are transmitted from
 a source, an example being a Windows

0:01:19.420000 --> 0:01:23.680000
 or Linux system or any other endpoint
 for that matter to a destination

0:01:23.680000 --> 0:01:28.740000
 that, you know, could be a
 seam or elk or whatever.

0:01:28.740000 --> 0:01:32.320000
 The key point is that
 log for log formats.

0:01:32.320000 --> 0:01:36.880000
 Now we're talking about log formats,
 log formats define how the log data

0:01:36.880000 --> 0:01:42.940000
 is structured during transport or shipping,
 as it were, and or as well

0:01:42.940000 --> 0:01:44.760000
 as ingestion, right?

0:01:44.760000 --> 0:01:50.120000
 And these two aspects ensure that logs
 are reliably transferred, securely

0:01:50.120000 --> 0:01:55.660000
 delivered and properly passed
 for detection and analysis.

0:01:55.660000 --> 0:01:59.660000
 So what are some common
 log shipping protocols?

0:01:59.660000 --> 0:02:00.440000
 Let's start there.

0:02:00.440000 --> 0:02:04.660000
 So in this table, you have the protocol
 transport layer, the use cases

0:02:04.660000 --> 0:02:09.020000
 and some notes, right, that I've added
 there based on my experience.

0:02:09.020000 --> 0:02:15.080000
 So in the case of syslog, this operates
 on either UDP, TCP, TLS sort of

0:02:15.080000 --> 0:02:17.440000
 underpinning that we're applicable.

0:02:17.440000 --> 0:02:21.460000
 The use case is, you know, routers, switches,
 firewalls or Linux servers.

0:02:21.460000 --> 0:02:25.480000
 That's where you typically see syslog
 being used for log shipping.

0:02:25.480000 --> 0:02:29.860000
 And some notes here is that, you
 know, syslog is widely supported.

0:02:29.860000 --> 0:02:34.260000
 And it's typically the standard for
 many of, for many appliances.

0:02:34.260000 --> 0:02:38.820000
 So when I see appliances, think of routers,
 switches, firewalls, et cetera.

0:02:38.820000 --> 0:02:43.120000
 So you have the hardware firewalls
 here, hardware firewalls.

0:02:43.120000 --> 0:02:46.980000
 You then have the Beats protocol, which
 I explained in the previous video,

0:02:46.980000 --> 0:02:51.860000
 regardless of that fact, you have this
 operates on TCP with TLS being

0:02:51.860000 --> 0:02:53.760000
 optional. Make of that what you will.

0:02:53.760000 --> 0:02:59.840000
 In any case, the use case is typically,
 you know, for the elk stack where

0:02:59.840000 --> 0:03:04.360000
 you have, you know, a file beat or
 way you're using file beat, win log

0:03:04.360000 --> 0:03:11.260000
 beat, you know, directly correlating
 to log stash, elastic search, et

0:03:11.260000 --> 0:03:17.960000
 cetera. Some key notes here are the
 fact that, or are, you know, it's

0:03:17.960000 --> 0:03:23.640000
 quite lightweight, reliable, and,
 you know, event streaming format.

0:03:23.640000 --> 0:03:29.160000
 In the case of HTTPS, which you also
 see, you know, this operates on TCP.

0:03:29.160000 --> 0:03:34.160000
 The use case here is typically API
 based shipping to cloud or software

0:03:34.160000 --> 0:03:38.680000
 as a service platforms, you know, typically
 used in flu and D or in custom

0:03:38.680000 --> 0:03:45.220000
 collectors. So if you think of something
 like Fortissim, the Fortissim

0:03:45.220000 --> 0:03:49.320000
 collectors, you know, utilize, you
 know, it is a custom collector.

0:03:49.320000 --> 0:03:54.720000
 So it utilizes just standard
 HTTP, HTTPS as it were.

0:03:54.720000 --> 0:03:59.320000
 And that's how, you know, that's a protocol
 that you typically see used

0:03:59.320000 --> 0:04:05.480000
 there. You then have Kafka, which
 operates on, you know, TCP.

0:04:05.480000 --> 0:04:09.460000
 The use case here is, you know, high volume
 in high volume scalable environments.

0:04:09.460000 --> 0:04:13.680000
 And it's typically or often used
 between log stash nodes.

0:04:13.680000 --> 0:04:18.940000
 You also have GRPC or proto buff,
 you know, that operates on TCP.

0:04:18.940000 --> 0:04:24.760000
 The use case is, you know, typically used
 or it's typically used in efficient

0:04:24.760000 --> 0:04:26.060000
 binary shipping.

0:04:26.060000 --> 0:04:29.780000
 So think of tools like Velociraptor
 or EDRs, right?

0:04:29.780000 --> 0:04:34.880000
 And as my notes suggest, it's used
 in modern EDRs or forensic tools.

0:04:34.880000 --> 0:04:40.460000
 And some security considerations here
 with regards to log shipping, which

0:04:40.460000 --> 0:04:44.880000
 I also alluded to in the previous video
 are you really must use in a production

0:04:44.880000 --> 0:04:50.620000
 environment TLS or SSL encryption, especially
 with syslog over TCP beats

0:04:50.620000 --> 0:04:56.140000
 or HTTP, right? You need to authenticate
 agents or clients with keys or

0:04:56.140000 --> 0:04:58.260000
 tokens. So you need to
 have authentication.

0:04:58.260000 --> 0:05:00.360000
 That's sort of the bottom line here.

0:05:00.360000 --> 0:05:04.480000
 And you should implement rate limiting
 and buffering to prevent data loss,

0:05:04.480000 --> 0:05:09.540000
 which again, I alluded to in the
 previous, in the previous video.

0:05:09.540000 --> 0:05:13.040000
 So that brings us now to the
 actual log formats, right?

0:05:13.040000 --> 0:05:17.800000
 And of course, here you have the format
 structure, some example tools

0:05:17.800000 --> 0:05:21.020000
 where this is applicable and the purpose.


0:05:21.020000 --> 0:05:28.440000
 So syslog, that is, you know,
 specific to RFC 5424.

0:05:28.440000 --> 0:05:32.400000
 The structure is typically, you know,
 your PRI and then they had a message

0:05:32.400000 --> 0:05:36.340000
 and then some example tools that apply
 here obviously going to be our

0:05:36.340000 --> 0:05:38.220000
 syslog or syslog and g.

0:05:38.220000 --> 0:05:42.680000
 The purpose here is, you know, purpose
 of this format is, you know, fact

0:05:42.680000 --> 0:05:45.160000
 that it's compact and it's text based.

0:05:45.160000 --> 0:05:48.580000
 It's a compact text based format
 for network devices.

0:05:48.580000 --> 0:05:51.220000
 You also have JSON quite common.

0:05:51.220000 --> 0:05:55.080000
 The structure of this of JSON, as you
 probably know, is, you know, key

0:05:55.080000 --> 0:05:56.980000
 value structured text.

0:05:56.980000 --> 0:06:01.260000
 The tools that utilize this format
 are file beat, flu and D log stash,

0:06:01.260000 --> 0:06:07.200000
 etc. Why is this, you know, what are
 the potential use cases of the JSON

0:06:07.200000 --> 0:06:13.020000
 log format? Oh, it's fairly obvious,
 you know, just native or in respect

0:06:13.020000 --> 0:06:17.620000
 to JSON and how it is typically used
 today and that or why it is typically

0:06:17.620000 --> 0:06:22.200000
 used and that is, you know, for easy
 passing and it's quite common in

0:06:22.200000 --> 0:06:24.400000
 modern log pipelines.

0:06:24.400000 --> 0:06:31.740000
 You also have plain text,
 believe it or not.

0:06:31.740000 --> 0:06:36.060000
 The tools that utilize this would be
 things like Apache or, you know,

0:06:36.060000 --> 0:06:41.780000
 additional or other custom apps and
 reason why this would be used, you

0:06:41.780000 --> 0:06:47.780000
 know, would be not the reason why it
 would be used, but, you know, how

0:06:47.780000 --> 0:06:54.620000
 it can sort of be analyzed as it were,
 you know, in this particular case,

0:06:54.620000 --> 0:06:56.720000
 it requires rejects passing.

0:06:56.720000 --> 0:06:59.440000
 Another advantage is that
 it's human readable.

0:06:59.440000 --> 0:07:05.080000
 You also have CEF or CEF, which is an
 abbreviation for common event format.

0:07:05.080000 --> 0:07:09.580000
 This is quite popular with a lot of
 the, you know, big boys out there.

0:07:09.580000 --> 0:07:12.920000
 The big boys, when I'm referring to
 the big boys, I'm referring to the

0:07:12.920000 --> 0:07:18.360000
 seams. So this, the structure of the
 CEF format is key value pairs.

0:07:18.360000 --> 0:07:25.540000
 So example, or some tools that use
 this format are oxide, Fortinet or

0:07:25.540000 --> 0:07:30.540000
 Fortissim, Macafee, and as alluded to in
 the purpose column, it's standardized

0:07:30.540000 --> 0:07:32.860000
 for seam ingestion.

0:07:32.860000 --> 0:07:38.020000
 You also have leaf or L-E-E-F, which
 is an abbreviation for the log event

0:07:38.020000 --> 0:07:39.940000
 extended format.

0:07:39.940000 --> 0:07:45.380000
 It's quite similar to CEF example of
 a tool that utilizes this is IBM

0:07:45.380000 --> 0:07:51.120000
 Qradar. As stated in the purpose column,
 it's optimized for Qradar.

0:07:51.120000 --> 0:07:52.980000
 You then have Protobuf.

0:07:52.980000 --> 0:07:55.120000
 This is a structures binary.

0:07:55.120000 --> 0:07:59.360000
 Some tools that utilize this,
 obviously going to be GRPC.

0:07:59.360000 --> 0:08:05.140000
 Velocirapt is a good example of a tool
 that utilizes this particular format.

0:08:05.140000 --> 0:08:13.460000
 And the purpose is, you know, compact and
 efficient for EDR or DFIR applications.

0:08:13.460000 --> 0:08:18.580000
 So now in this table, I have, you know,
 I've essentially performed, you

0:08:18.580000 --> 0:08:23.340000
 know, protocol and format pairings
 in the form of example.

0:08:23.340000 --> 0:08:28.100000
 So in, let's say, you know, the first
 column, you have the source and

0:08:28.100000 --> 0:08:32.300000
 then the protocol format and where
 it would be shipped to or what the

0:08:32.300000 --> 0:08:34.840000
 ideal destination is.

0:08:34.840000 --> 0:08:40.620000
 So let's say in the case of the source
 being Linux syslog, you know, the

0:08:40.620000 --> 0:08:42.140000
 protocol is going to be TCP.

0:08:42.140000 --> 0:08:46.680000
 The format is the syslog format and
 it's shipped to our syslog and, you

0:08:46.680000 --> 0:08:49.240000
 know, eventually into log stash.

0:08:49.240000 --> 0:08:52.720000
 Filebeat Linux. Again, this
 utilizes the beats protocol.

0:08:52.720000 --> 0:08:54.600000
 The format here is going to be JSON.

0:08:54.600000 --> 0:08:57.280000
 So log stash into Elasticsearch.

0:08:57.280000 --> 0:09:00.200000
 Win log beats. This is obviously
 going to be Windows.

0:09:00.200000 --> 0:09:03.880000
 So the protocol is going to be the
 source is going to be Windows.

0:09:03.880000 --> 0:09:07.880000
 The protocol is beats format
 is JSON ship to elk stack.

0:09:07.880000 --> 0:09:11.720000
 So again, these are just examples of
 what you're likely to see when, you

0:09:11.720000 --> 0:09:16.800000
 know, and this can sort of, this can
 be used as sort of a cheat sheet

0:09:16.800000 --> 0:09:25.120000
 to sort of allude to or to sort of derive
 or deduce a particular destination

0:09:25.120000 --> 0:09:29.680000
 based on, let's say, the
 protocol being used.

0:09:29.680000 --> 0:09:34.140000
 So if you see, let's say the source
 being Win log beat, then it's most

0:09:34.140000 --> 0:09:39.520000
 likely that, you know, the source is
 Windows and the destination is going

0:09:39.520000 --> 0:09:46.200000
 to be the elk stack because again,
 Win log beat is specific to the elk

0:09:46.200000 --> 0:09:50.180000
 stack, right? Another source could
 be a firewall and the protocol here

0:09:50.180000 --> 0:09:55.380000
 is UDP. The format is syslog and it's
 shipped to the seem fluently.

0:09:55.380000 --> 0:10:01.800000
 The protocol here is HTTP, which operates
 under or on top of, sorry, TCP.

0:10:01.800000 --> 0:10:05.820000
 The format is going to be JSON and
 some example of where this could be

0:10:05.820000 --> 0:10:11.560000
 shipped to or Splunk, Splunk
 AGC, gray log, etc.

0:10:11.560000 --> 0:10:17.400000
 So finally, why is understanding log
 shipping protocols and log formats

0:10:17.400000 --> 0:10:23.240000
 important or relevant to you as an
 instant responder or why does this

0:10:23.240000 --> 0:10:29.680000
 matter, you know, for instant response,
 a, choosing the right protocol

0:10:29.680000 --> 0:10:33.020000
 ensures reliability and security
 in log delivery.

0:10:33.020000 --> 0:10:38.020000
 Fairly obvious by this point, be choosing
 the right format simplifies

0:10:38.020000 --> 0:10:42.200000
 passing correlation and consequently
 analysis, which is important to us

0:10:42.200000 --> 0:10:48.260000
 and see misconfigured protocols or formats
 can result in this is the worst

0:10:48.260000 --> 0:10:53.960000
 nightmare dropped logs, passing failures
 or even alerting delays, which

0:10:53.960000 --> 0:10:59.140000
 is actually more common than you would
 like to, than you would believe.

0:10:59.140000 --> 0:11:03.400000
 But with that being said, that brings
 us to the end of this video and

0:11:03.400000 --> 0:11:05.640000
 I will be seeing you in the next video.