ESXi Syslog Redirection and log file locations

The reduced footprint of and ESXi install makes it easier to manage from a file system perspective, but there are a number of best practices when it comes to managing this file structure. One being the location of the syslog’s and the other is just knowing where to find some of the critical log file locations. There seems to be multiple sources to derive this information and I thought it would be a great blog post to get this information on one page.

With the move from ESX to ESXi in all future releases of vSphere, there are few daemons that you should redirect on your cluster nodes especially if you are going to use SD or USB to boot ESXi. This will prevent ESXi from writing to these solid state devices thus increase the longevity of the drives.

One of the most chatty daemons on ESX is the syslogd service which is for logging VMkernel messages as well as other system level components. There are a number of ways to redirect the logging to another location and VMware does recommend that you ship these logs to another location.

1 Use the vSphere Client to either connect to directly to the node or through vCenter to the configuration tab > advanced and set these parameters:

2 Use PowerCLI or better yet – the vCLI in the vMA virtual appliance (see my blog post on vMA for install tips) via a perl script called “vicfg-syslog” that just so happens to be installed in it.

This can be done by running the following command:

#vicfg-syslog.pl –server <ESXi Host> –username root –password <password> –setserver x.x.x.x –setport 514

To verify the redirection is there:

#vicfg-syslog.pl –server <ESXi Host> –username root –password <password> –show

Gotcha: As of VMware vCenter Server 4.1, host profiles are not suitable for redirecting syslog paths (or scratch locations for that matter) since Host Profiles will filter out these advanced settings since they could be host specific. Such as the Syslog.Local.DatastorePath and ScratchConfig.ConfiguredScratchLocation or ScratchConfig.CurrentScratchLocation. I will cover Host Profiles in a separate post and this information will definitely be in there!

A few things to remember about syslog redirection and log files:

  • The syslog daemon will only redirect the logs to one remote address.
  • You cannot use the esxcfg-advcfg command to redirect the syslog.
  • You can remove the redirection on the host by re-issuing the vicfg-syslog.pl command and setting the remote server to a null string.
  • The sysboot.log file was first introduced in ESXi 4.0.
  • Log rotation or removal happens only during a restart of the node.

Scratch partition for log files

As of ESXi 4.0 Update 1, the file system now has a scratch partition (which is recommended, but not required) for the log files. This location is used for storing system level temporary logs, the system swap and other diagnostic data. The benefit of having this partition is to diagnose core dumps (PSOD’s) or other useful system logs to help diagnose problems (many of these being memory issues).

Few things to remember with the scratch partition:

  • If the partition is gone or was deleted, you can simply recreate the directory
  • It may be provisioned on VMFS or even FAT16 partitions
  • If there is no scratch partition, ESXi will store this data in RAMDISK which has significant space limitations
  • A scratch location is automatically created during installation
  • Do NOT share a scratch partition between ESXi hosts.

Log File Locations:

Logging Type Location
VMkernel / vmkwarning / hostd /var/log/messages
System Boot Log /var/log/sysboot.log (ESXi 4.x)
vCenter Agent /var/log/vmware/vpx/vpxa.log
Automatic Availability Manager (aam) /var/log/vmware/aam/vmware_<host>-xxx.log
Host Management Service (hostd) /var/log/vmware/hostd.log

 

I’ve also heard many people talk about turning the vMA into a log host and Simon Long has a great write-up about it. But without the ability to parse out the massive amount of data that is shipped there, it makes it really hard to locate node and time frame specific data. I would suggest a product such as Splunk that has a really nice interface on it to locate data in a timescale format.

-Rick

 

Leave a Reply