WEBVTT

00:00:00.000 --> 00:00:04.000
Welcome back to vSphere 8 in the real world, and now we're going to discuss security logging.

00:00:07.840 --> 00:00:13.440
So, logging is obviously a pretty important thing. Oftentimes, if you're having some kind

00:00:13.440 --> 00:00:19.120
of issues, you might see something is failing, and messages that you get from the vSphere

00:00:19.120 --> 00:00:23.760
graphical environment might not be very detailed. They might not really give you a good insight to

00:00:23.760 --> 00:00:28.160
what's happening. You'll see something fail, but you don't really necessarily know why.

00:00:28.160 --> 00:00:34.720
So, your log files are going to be a great resource. Sometimes, the logging will have to be

00:00:35.439 --> 00:00:42.160
enhanced to be more robust than what you get by default. You don't want to always log in a very

00:00:42.160 --> 00:00:49.200
verbose mode because that can start to create a lot of data, a lot of log files, a lot of space

00:00:49.200 --> 00:00:55.360
gets used, and it can be a lot of noise. So, when you get too much logging information on a regular

00:00:55.360 --> 00:01:01.200
basis, it may be hard to find the items that are actually relative to what you're trying to do.

00:01:01.200 --> 00:01:06.320
And again, it's just a lot of space and things like that. Now, if you're trying to troubleshoot

00:01:06.320 --> 00:01:14.480
a specific problem, then it might be very relevant to go in and increase the logging level to a more

00:01:14.480 --> 00:01:20.160
robust level, maybe even like the most robust level, while you're trying to troubleshoot this

00:01:20.160 --> 00:01:24.720
problem to get that information. And you also probably have an idea what you're searching for

00:01:24.720 --> 00:01:28.800
to kind of make it not such a needle in a haystack, right? If you're just kind of doing

00:01:28.800 --> 00:01:33.760
general observation of the logs, not really trying to troubleshoot any specific issue,

00:01:33.760 --> 00:01:38.320
then a huge amount of information is just overwhelming. But if you know I'm looking

00:01:38.320 --> 00:01:43.599
for information about a certain virtual machine or about a certain VM kernel port or something like

00:01:43.599 --> 00:01:49.279
that, about the HBA not functioning properly or something, then you can start narrowing down your

00:01:49.279 --> 00:01:53.919
search and start going through that huge amount of data. Once you've got your information,

00:01:53.919 --> 00:01:59.760
you've resolved your issue, you're past the troubleshooting and implementing your solution,

00:02:01.120 --> 00:02:06.480
and you're back to a status quo, then you can lower your logging level back to a more normal

00:02:06.480 --> 00:02:10.639
type of logging level. So, logging is super important. If you want to troubleshoot things,

00:02:10.639 --> 00:02:15.919
if you're looking for any kind of security warnings about unusual traffic behaviors,

00:02:15.919 --> 00:02:21.919
things like that, and you might have even legal records maybe. I don't have a lot of experience

00:02:22.479 --> 00:02:27.440
legal records unless it's like for the insurance company during a ransomware remediation or

00:02:27.440 --> 00:02:32.320
something like that, that will come in for the insurance company where they want logs and things

00:02:32.320 --> 00:02:37.440
like that from what's happening. If you have no logging enabled, you can turn logging off.

00:02:37.440 --> 00:02:43.279
Obviously, not a great idea, but you can. Then all of a sudden your insurance company might

00:02:43.279 --> 00:02:47.360
start to get suspicious and things like that if you ever need to go to them, and that is becoming

00:02:47.440 --> 00:02:52.479
something more and more common. So, where your log files are stored,

00:02:52.479 --> 00:02:58.800
if you're searching for log file information from vCenter, then it will be stored on that

00:02:58.800 --> 00:03:04.479
vCenter virtual appliance. Log files for virtual machines are going to be stored on the data store

00:03:04.479 --> 00:03:10.639
with the virtual machine, and your ESXi host will have its own log files. Not that long ago,

00:03:10.639 --> 00:03:15.679
I had a client contact me, and their vCenter was having all kinds of problems and issues,

00:03:15.679 --> 00:03:18.080
and the hard drive was full, and they couldn't figure out what was going on.

00:03:18.080 --> 00:03:23.199
Come to find out their log files were filling up their vCenter server, and I said, well,

00:03:23.199 --> 00:03:27.600
why don't we look at sending your log files to a syslog server? And they, oh, we have a syslog

00:03:27.600 --> 00:03:33.039
server. Come to find out their syslog server had inadvertently been shut down. And so,

00:03:33.039 --> 00:03:38.160
the vCenter server was just sitting on all these log files it was creating, and they had a pretty

00:03:38.160 --> 00:03:44.720
robust logging level set. Not the most, but pretty robust. And once we powered on the syslog server,

00:03:44.720 --> 00:03:47.759
and we got it, you know, to reestablish communication with it, and started sending

00:03:47.759 --> 00:03:53.199
those log files to the syslog server, all of a sudden, all their instability issues with vCenter

00:03:53.199 --> 00:03:57.679
went away. And they were logging, you know, fully again and things like that, and they didn't even

00:03:57.679 --> 00:04:02.479
realize that their syslog server wasn't up and running. So, that's another conversation,

00:04:02.479 --> 00:04:09.039
another kind of scary issue there. But syslog servers are a great resource. Not only will

00:04:09.039 --> 00:04:14.399
allow you to send all your vCenter, you know, files there for your logging files there,

00:04:14.399 --> 00:04:17.760
you can send your vCenter log files there, you can send your user client host log files there,

00:04:17.760 --> 00:04:22.559
you can send all your, you know, systems log files to this one place. And if you're looking

00:04:22.559 --> 00:04:27.519
for a problem, you can say, I want to find all the information, you know, what's happened from

00:04:27.519 --> 00:04:32.480
vCenter and on these hosts during this time, during this 30-minute window. If you know,

00:04:32.480 --> 00:04:36.079
you know, something happened that time, and you may not know exactly where the problem

00:04:36.079 --> 00:04:39.920
happened at, but if you can say, okay, if I know what happened during this window here,

00:04:39.920 --> 00:04:43.920
I'm going to search for 15 minutes before and 15 minutes after, and try and find all this,

00:04:43.920 --> 00:04:47.920
maybe you want to do 20 minutes before, 20 minutes after, whatever, then you're getting,

00:04:47.920 --> 00:04:51.760
oh, you know, vCenter did this, and the host did that, and this other host did this,

00:04:51.760 --> 00:04:57.200
and vCenter did that. And so, you can kind of sift through all that. If you don't have a syslog

00:04:57.200 --> 00:05:02.640
server kind of, you know, aggregating all this information together, then what ends up happening

00:05:02.640 --> 00:05:06.239
is you have to go to vCenter and look at that timeframe, then go to the host and kind of look

00:05:06.239 --> 00:05:11.119
at the timeframe, and go to another host and try and do that manually. And it can be very,

00:05:11.119 --> 00:05:16.880
very difficult to do. It's great when you can kind of have all that information aggregated.

00:05:17.440 --> 00:05:22.720
So, logging is super important. Definitely recommend you do, you know, you do enable logging.

00:05:22.720 --> 00:05:26.799
Don't get too carried away with your logging. Don't, you know, enable the logging level to be

00:05:26.799 --> 00:05:33.440
too high unless you're troubleshooting this specific problem that that logging level calls

00:05:33.440 --> 00:05:39.279
for. And then when you're done troubleshooting that problem and you have your issue resolved,

00:05:39.279 --> 00:05:46.559
put your logging level back to a more appropriate level for your environment. All right. So,

00:05:46.559 --> 00:05:50.799
how can we enable logging? We can enable logging on a virtual machine level. We can also enable

00:05:50.799 --> 00:05:56.720
logging for our ESXi host. We can also enable logging for our vCenter. The two places that

00:05:56.720 --> 00:06:01.040
we're going to look at primarily will be our ESXi host, you know, checking our logging level for

00:06:01.040 --> 00:06:05.359
that, and then saying, you know, how can we tune that logging a little bit for our VMs.

00:06:05.359 --> 00:06:12.559
And then we can also enable logging or, you know, tune logging for vCenter. All right.

00:06:16.000 --> 00:06:20.720
So, some logging options. Now, these levels here we're looking at, this is vCenter specific. So,

00:06:21.040 --> 00:06:24.959
this is not the levels you'll have for a virtual machine. But when we're talking about logging for

00:06:24.959 --> 00:06:31.040
vCenter, and vCenter is where a lot of information is gathered, right? Communication with the

00:06:31.040 --> 00:06:36.559
different VMs. If you try to do a vMotion, the host will have information about that in their

00:06:36.559 --> 00:06:42.000
logs, but vCenter will have information about, you know, initiated a vMotion from host one to host

00:06:42.000 --> 00:06:46.399
two, and this is what happened, and this is what the success and failure was, and things like that,

00:06:46.399 --> 00:06:50.559
right? So, the logging information from vCenter is going to be really, really important.

00:06:51.040 --> 00:06:57.040
So, you can say you want no logging at all, or you can say only errors. So, very obvious, like,

00:06:57.040 --> 00:07:03.600
this failed, this stopped type errors. Warnings are a little more detailed, right? And as you can

00:07:03.600 --> 00:07:07.359
see, as they go down, it becomes more and more detailed, right? So, warnings are more detailed

00:07:08.399 --> 00:07:13.359
rather than be at the hard critical, like, you know, a red stop, like more of a yellow level

00:07:13.359 --> 00:07:22.160
type of situation where we're seeing, like, slow response times. CPU is not pegged at 100%,

00:07:22.160 --> 00:07:28.000
but we're creeping up at 85%. You know, that would be a warning. For information, that would

00:07:28.000 --> 00:07:36.239
be errors, okay? Again, a little more information. So, not necessarily, like, a hard failure, but,

00:07:36.239 --> 00:07:41.600
like, something you might want to, like, watch out for, that kind of stuff. Verbose is going to be,

00:07:42.160 --> 00:07:48.799
again, more detailed, and that's going to give you a lot of information. And trivia means, like,

00:07:48.799 --> 00:07:54.880
even trivial things are being logged. So, trivia means every single thing. That's going to be some

00:07:54.880 --> 00:08:00.239
very, very detailed logging. You might kind of consider it more like, or another term you'll

00:08:00.239 --> 00:08:04.880
see in other vendors is a debug, right, where they just give you every single thing,

00:08:04.880 --> 00:08:11.359
every little detail. And a lot of the information you see may not be an error. It may not be,

00:08:11.359 --> 00:08:15.119
you know, something failed. It's something happened successfully, even, you know,

00:08:15.119 --> 00:08:20.000
and here's all this information, every little thing that happens. And you might want to say,

00:08:20.000 --> 00:08:24.640
well, this succeeded, and this succeeded. Not just, well, this failed and this failed,

00:08:24.640 --> 00:08:28.640
but what happened between these two things failing? Well, this failed, these five things

00:08:28.640 --> 00:08:33.280
happened successfully, then that failed. And that kind of, oh, I see why this failed,

00:08:33.280 --> 00:08:39.200
because this and this. So, that trivia will give you a huge amount of information, which means

00:08:39.200 --> 00:08:45.840
huge log files, huge information being sent to your syslog server. Some syslog solutions will

00:08:45.840 --> 00:08:51.359
charge you based upon, like, the gigabyte, like how much is being, you know, ingested every day.

00:08:51.359 --> 00:08:57.440
And so, if you ran this trivia every day, all day, then you'd be paying a ton to your syslog vendor.

00:08:58.479 --> 00:09:02.960
So, you might want to just do errors or maybe warnings at the most, right? But when you're

00:09:02.960 --> 00:09:06.960
troubleshooting, trying to, you know, dig into a problem that you just haven't been able to resolve

00:09:06.960 --> 00:09:11.440
with a warning or information, you might go, you know, as far as verbose or trivia.

00:09:11.440 --> 00:09:15.039
All right. So, this is, you know, the further down you go, the more details you get.

00:09:15.039 --> 00:09:22.000
Most people are pretty comfortable for their status quo being errors and or warnings only.

00:09:22.000 --> 00:09:24.080
All right. Those are some of your logging options there.

00:09:27.440 --> 00:09:32.080
All right. So, we discussed how to enable logging and some logging options. Let's go ahead and go

00:09:32.080 --> 00:09:36.080
into our demo environment. I'll show you where you can enable logging for a VM, where you can

00:09:36.080 --> 00:09:40.960
enable logging for vCenter, and a couple of examples of what you might want to configure

00:09:40.960 --> 00:09:47.200
for some advanced options for your VM logging. So, if you want to configure logging for your

00:09:47.200 --> 00:09:51.679
vCenter, we are here in our vCenter server. I'm going to click on this vCenter,

00:09:53.440 --> 00:09:55.520
and I will want to go to my configuration.

00:09:59.119 --> 00:10:03.840
And right under general, you see I have my logging options here, right? So, I'm going to go

00:10:03.840 --> 00:10:07.599
ahead and I see that my logging levels are set to info. I'm going to click on edit here

00:10:09.280 --> 00:10:19.440
and go to logging settings. And right now, I'm set to info. I have, you know, none, error,

00:10:19.440 --> 00:10:26.400
warning, info. I can go to verbose or to trivia. So, I can say, you know, which of these. Info is

00:10:26.400 --> 00:10:31.840
probably fine. A lot of people are going to be okay with errors or warnings, right? Info me at

00:10:31.840 --> 00:10:36.239
the most. You wouldn't want to go as far as verbose on an ongoing regular basis just because

00:10:36.239 --> 00:10:42.000
that, again, can become an overwhelming amount of information. But, you know, one of these three

00:10:42.559 --> 00:10:47.760
is usually where most people are going to land. Again, you're not going to do none. Error is the

00:10:47.760 --> 00:10:52.400
least amount and still doing some logging, and that might not be enough. You might not want

00:10:52.400 --> 00:10:56.400
only errors. You might want some warnings. So, before something becomes a problem,

00:10:56.400 --> 00:11:00.880
when you see, like, utilization on something creeping up, you might want that warning,

00:11:00.960 --> 00:11:05.440
right? You might not want necessarily just to get blindsided by a failure. You want a warning

00:11:05.440 --> 00:11:10.400
something's coming, right? That kind of thing. So, you can go in and change your logging level

00:11:10.400 --> 00:11:14.880
for your – this is for vCenter specifically, right? This is how we can enable and change our

00:11:14.880 --> 00:11:23.599
vCenter logging is from here. We can also, for example, enable vCenter logging or, you know,

00:11:23.599 --> 00:11:33.039
modify logging for a VM. I can go to this VM, and I can go to my edit settings. I'll go to VM

00:11:33.039 --> 00:11:41.679
options, go to advanced here, right? And I can say enable logging, and I have a couple of different

00:11:41.679 --> 00:11:48.159
options, right? For a VM, I have – I want, you know, run normally record debugging information

00:11:48.159 --> 00:11:52.559
or record statistics. So, just some basic, you know, what I want to record here. I have some

00:11:52.559 --> 00:12:00.400
options for that. Now, some things we can do is we can also go in and specify, like, how long do I

00:12:00.400 --> 00:12:04.719
want to retain log files for? Basically, what I can say is how many log files do I want to retain,

00:12:04.719 --> 00:12:10.159
how big can my log files get, that kind of thing. To configure that, I'm going to go to my advanced

00:12:10.159 --> 00:12:15.840
parameters. And I could look to see do I have anything here. Now, one thing you'll notice real

00:12:15.840 --> 00:12:18.960
quick, though, before I go and start looking at that, I can't go in there, and I can't change

00:12:18.960 --> 00:12:23.840
this. The reason why I can't change anything here is this VM is running. So, if I hit cancel,

00:12:24.719 --> 00:12:31.359
this VM is running. I'm just going to quickly just power this off. I probably should do a

00:12:31.359 --> 00:12:34.159
shutdown, but this is a lab. I just want to be quick. I'm going to say power it off.

00:12:38.000 --> 00:12:44.960
Now that it's off, if I go in and say edit settings and go to my advanced parameters,

00:12:44.960 --> 00:12:50.000
now you see I can click in here. I can make changes. So, one, like, one thing you might

00:12:50.000 --> 00:12:57.679
look for is do vmx.log and see if we have, do we have anything in here. So, nothing there.

00:12:57.679 --> 00:13:06.880
So, I made some notes over here. Oops. Here. So, here are some options here for

00:13:06.880 --> 00:13:17.679
this vmx.log.keepold. This is going to say how many log files do I want to keep. The default,

00:13:17.679 --> 00:13:22.479
if there's no parameter set here, it's going to be 10. I can go in there and say make it 10. So,

00:13:22.479 --> 00:13:26.799
now it's, like, you know, hard-coded. Or I can say, you know what, I have a lot of storage space,

00:13:26.799 --> 00:13:32.400
and logs are really important to my organization. So, I want to keep 20, and I can add that,

00:13:33.039 --> 00:13:46.000
right. So, now if I do a search for vmx, I have my vmx.keepold and a value of 20, right.

00:13:48.479 --> 00:13:56.000
I can also, another option is rotate size. By default, if we don't configure this, like,

00:13:56.000 --> 00:14:02.000
right now, that is not configured. What will happen is the system is going to create a new

00:14:02.000 --> 00:14:08.960
log file every time a VM is powered on. It's removed from being suspended. It gets vMotion

00:14:08.960 --> 00:14:14.719
to a new host. All these different things can cause a new log file to be created.

00:14:15.359 --> 00:14:22.080
If I want to, I can say I want it to reach a certain size. Rotate logs when it reaches a

00:14:22.080 --> 00:14:27.440
certain size. So, this is my rotate log file size, and I believe it's measured in bits. So,

00:14:27.440 --> 00:14:32.159
the numbers are going to be some really huge number. I think I actually have a note of it

00:14:32.960 --> 00:14:40.159
right here. Like, this right here would be about 2 megs, right. So, you can see it can get to be

00:14:42.239 --> 00:14:47.359
a little bit cumbersome to try, because this is just in bits. So, if we wanted it to be,

00:14:48.320 --> 00:14:55.359
you know, instead of 2 megs, we want it to be 20 megs. We can add a zero onto there,

00:14:55.359 --> 00:14:59.440
and say add that, and then all of a sudden our log files can start to get bigger and bigger.

00:14:59.440 --> 00:15:05.280
By default, it's going to just rotate, though, when we have a new event happen on the VM,

00:15:05.280 --> 00:15:10.719
like a power on or a migration or being unsuspended from, you know, a suspended state.

00:15:11.280 --> 00:15:15.760
So, under your advanced parameters, when the VM's off, we can go in and we can customize

00:15:15.760 --> 00:15:20.880
these advanced options. For example, how many log files. The default being 10. We can make it

00:15:20.880 --> 00:15:25.039
whatever we need to, as long as we don't get too carried away and start filling up our hard disk

00:15:25.039 --> 00:15:28.239
space. We don't want to crash VMs because we filled up our SAM of log files.