There are literally hundreds of possible types of log sources around your environment and choosing which bubble to the top of your IT consciousness can be difficult. In a job where everything seems to be a top priority, understanding all the log types and sources available for selection can be daunting. In your environment, some logs may be more valuable than others, but having general guidance about logging and what types of logs may be available to monitor can help make you a better technologist.
There’s no way we could ever think to cover every possible source of logs, but let’s start with some of the classics and go from there.
1 – Infrastructure Devices
These are those devices that are the “information superhighway” of your infrastructure. Switches, routers, wireless controllers, and access points can be teased to provide logging information about the health and state of your environment. The logs can provide insights ranging from wireless AP hopping to hardware failures. Probably most impactful to your environment are notifications of configuration changes. Knowing who changed what and when can help you diagnose and recover from any misconfigurations.
2 – Security Devices
As organizations push towards a cloud-first methodology, the edge devices in your environment can become even more vital to your business. Your firewalls and other security devices are handling more and more traffic as loads are shifted to cloud infrastructures. The logs on these security devices can provide a plethora of interesting information—not least is blocked traffic, health of the VPN, intrusion detection and prevention systems, and unusual user activity. These Security Information and Event Management (SIEM) logs may be your first defense in understanding an attack or isolating an anomaly in your user experience. Tools like SolarWinds Security Event Manager are designed to monitor event logs for any suspicious activity, which can allow you to respond in real time to potential threats.
3 – Server Logs
It may go without saying, but I’m going to say it anyway: server logs can offer abundant information about the state of your environment. Windows and Linux servers are constantly pumping out logs that give you an understanding of how and why systems are behaving the way they are. There are literally hundreds of thousands of events that can trigger within an operating system and its associated applications. Knowing which log events are frivolous and which require immediate action is a skill honed on the battlefield. Regardless, you shouldn’t overlook server logs as a viable source of information.
4 – Web Servers
Yes, I’m aware that capturing web server logs can be construed as a tedious process, but it is one of the best ways, if not the best way, to understand how end users interact with your web properties. IIS, Apache, Tomcat, Web Sphere, NGINX, and every other web engine out there can provide some measure of web server logging. Depending on your needs, sometimes just understanding when people are going to your site and from where can prove invaluable to understanding the needs of your customers. Unfortunately, a web server log is a common log type that can sometimes be overlooked when organizations are developing their logging strategy.
5 – Authentication Servers
Whether you use Active Directory, an implementation of OpenLDAP, or another alternative, knowing who and what is poking around your infrastructure can be key to a maintaining a good security posture. Each of your authentication servers will provide some measure of logging, but what’s key for you to is understanding what to look for. Most commonly, you should be looking for token requests, authorization revocation, and authentication failures. These types of logs can aid in determining failing logins due to account expiration, isolate the source of a potential attack, and pinpoint problem areas that need to be addressed.
6 – Hypervisors
Hypervisors can let us IT professionals do our jobs better by balancing workloads and utilizing resources more efficiently. Clusters can now run hundreds, if not thousands, of simultaneous workloads. However, much of the work associated with hypervisors is behind the curtain, and you never get to see the wizard. Your hypervisors are juggling all the time—allocate resources from this virtual machine to this one, move the storage from this cluster node to this other one, shift this entire virtual machine to another node—and it’s a precarious balance. Capturing and monitoring hypervisor logs can be one of the best ways to understand what your hypervisors are doing when you aren’t watching.
7 – Containers
Although relatively new compared to most other log types on this list, containers are becoming more and more business critical. Extrapolating to a higher-level would be container management services like Kubernetes, Docker Swarm, and Apache Mesos. These services are like hypervisors in many ways, but just different enough to warrant a separate category. Understanding why the host felt it was necessary to drop back your scaled-out deployment from eight endpoints to only four would prove useful in diagnosing and tuning. Most of this information is located only in the container logs, so make sure that you get them.
8 – SAN Infrastructure
This may seem an odd addition to this list of the best log types to monitor because of the IT trend to move towards a more hyper-converged infrastructure or moving everything to the cloud, but it’s something that’s frequently overlooked. If your fibre switch loses connectivity to a server-side transceiver, then that data is no longer available to that server. In today’s world, there are normally redundant pathways so that connectivity is not truly lost, but the scenario still applies in a multi-path environment. Say you have four connections from your server to your SAN infrastructure, but after a series of unfortunate events over several months, three of them have failed. This means that you have restricted data movement by 75%. You’ve not encountered a failure in the traditional sense, because the connectivity still exists, and data is moving, but with performance hampered this badly, is it any wonder end users are complaining? In my opinion, this is one of the top overlooked log sources.
9 – Applications
This applies to pretty much any application log. Although some software applications will leverage the operating system’s existing logging functionality for log management, these are becoming fewer and fewer. Most critical logs for applications are stored in flat files on your disks somewhere. Often, these logs are used by your application support people for troubleshooting, but what about multitier applications? If you have a front-end, middleware, and back-end deployment, each may collect logs slightly differently. Make sure you aren’t sleeping on collecting and monitoring these logs—from each tier—and getting them into a system so that you can compare transactions by lining up the timestamps.
10 – Client Machines
Yes, really. In IT, a common trope is to blame the end user, but sometimes it’s not their fault. Sometimes it’s the fault of the endpoint itself. I’m not saying that every log on every machine needs to be collected all the time—in fact, I’m saying that you should probably not do that, but selective log collection from endpoints can be critical in gaining a larger grasp of the scope of the problem. This is probably the most overlooked log type needed for actively troubleshooting issues.
Everything Else
There are additional log sources that I’ve neglected, like proxy servers, load balancers, and cloud management systems, to just name a few, but this isn’t meant to be an exhaustive list. Hopefully, after reviewing these ten log types, you gain a little perspective into what would be relevant for your situation. It’s also something to keep in mind as new hardware and software enters your infrastructure.
Whether you choose one, all, or none of these as potential log sources to monitor is dependent on your exact needs. Simply thinking about what types of monitoring or log analysis tool you need moving forward could help you choose those relevant to your situation. Every bit of information can help you gain a deeper understanding of your infrastructure and how to best handle its care and feeding. Remember, it’s not if something will go sideways, it’s when. Having the best log types to back up your decision-making can be a welcome tool in your IT arsenal.