With the three pillars of observability considered more or less canon for IT monitoring philosophy these days, and logging being one of those pillars, it’s important to understand the tools, systems, and solutions that help IT practitioners do a good job of aggregating, normalizing, and leveraging logs in all their forms to provide improved insight into application and system health.
You Keep Using That Word…
Before diving into the best log tools, it’s important to clarify what is meant by “log monitoring” for two reasons. First, logs are present in several different forms on a variety of different systems around the enterprise. Second, those logs can be a rich source of insight for everything from security events to pure hardware status to application health. While there’s a lot of overlap in terms of how a source of log data can be used, there’s an equal amount of overlap about how a particular log message may be valued around the IT organization. This is more than a case of “one person’s trash is another’s treasure.” User connection data is valuable to multiple teams in IT, but that same dataset represents one thing to the security team and another to the appdev group.
Therefore, let’s clarify a few things.
What Is This “Log” of Which You Speak?
When IT folks talk about logs, they could mean:
- Syslog messages generated by the operating system and sent into a central listener
- Trap messages, a close cousin to syslog, that operate similarly
- The Windows Event Log
- Individual files (usually text files) on a single system that contain information about application performance, errors, and events
- The aggregation of log data (again, usually text files) from multiple systems that means very little individually, but when pulled into a single data set, can show trends or patterns across the overall system
“Unlimited” Licensing Still Doesn’t Break the Law of Physics
One of the biggest mistakes I see in log monitoring implementations is the belief that a single system can scale infinitely as long as the license says it’s “unlimited.” Let’s be clear: your license level does not mean you can stuff a five-pound sack with 10 pounds of cra…ckers. So, something I advise everyone to consider is a “log file filtration layer”—especially important when it comes to trap and syslog. Without going too deep into it, you want to make sure that whatever solution you choose will let you put multiple processing servers behind a single IP address, and balance the incoming load. Just acquired a new company and your logging doubled? No problem! Throw a few more servers behind that load balancing solution and you’re off to the races. Without some means of doing this, you are going to end up maxing out any system.
Some of the log management tools I feature here are either free or have a free tier of service. While it’s fantastic to find a bargain, that doesn’t always make it the best choice for a given use case. IT budgets aren’t infinite, I get that. But don’t let budget considerations alone keep you trapped in log management solutions that don’t ultimately serve your needs or accomplish your goals.
Also keep in mind that free tools are still going to require an investment in time and resources to learn, install, configure, and use. Missing features for these free tools may have to be built with the help of community support or an in-house IT team. In some cases, you won’t get the polling frequency or data retention you need without moving to a paid tier of service. In other cases, paid log monitoring software has features you don’t think you need today, but down the road you might realize you could have used those features, if only they’d been available.
Best Log Management Tools
SolarWinds Log & Event Manager (LEM) straddles the line between a “simple” log file aggregation and management platform and a full Security Information and Event Management (SIEM) solution. LEM combines the ability to receive messages from a variety of sources and normalize and aggregate them together with a powerful analytics engine that helps identify potentially system-impacting events. In addition, you can use Log & Event Manger to validate compliance, thanks to reporting purpose-built for HIPAA, PCI DSS, SOX, DISA STIG, and more.
Kiwi Syslog Server acts as a syslog and trap receiver, using rules to filter those messages based on source, keywords, and other patterns, then processing them in a variety of ways. You can receive messages from an unlimited number of sources and have a dozen processing options at your disposal, including transparent forwarding, storing in a database, running an external program or API, and more. Remember when I mentioned a “filtration layer” that can scale out to handle a greater number of messages? This is my top choice in log management tools for the job. With the ability to handle up to 2 million messages per hour, one installation will be more than enough for many environments. But if not, you can always add another.
A relative newcomer on the scene, Log Manager for Orion® was released mid-2018 and acts either as an add-on to the existing suite of SolarWinds monitoring tools or as a standalone logging solution. Focusing on syslog and trap, the strength of this tool lies in its visualization capabilities and powerful searching engine, which can filter and search past events or perform those same actions on the logs as they are received in real-time.
ManageEngine is another trusted name for monitoring professionals. With the ability to collect, manage, analyze, correlate, and search through the 700 sources of log data and handle up to 25,000 messages per second, it’s worth a look. With the ability to do forensic analysis of past events as well as leverage real-time pattern matching, it has the potential to minimize security breaches. It comes preconfigured with over 30 rules to identify brute force attacks, account lockouts, data theft, web server attacks, and more. Finally, the log parser is highly customizable.
For many IT practitioners, IPSwitch’s WhatsUp Gold is their first experience with a log monitoring tool. WhatsUp Log Management Suite is an automated tool that collects, stores, archives, and saves system logs, Windows events, and W3C/IIC logs. On top of that, it performs ongoing pattern analysis, so it can trigger alerts based on abnormal activity. The types of events tracked include access rights and file, folder, and object privileges. It can also use collected data for compliance reports for HIPAA, SOX, FISMA, PCI, MiFID, or Basel II. In actuality, WhatsUp Log Management Suite is really a set of four integrated applications:
- Event Archiver
- Event Alarm
- Event Analyst
- Event Rover
LogDNA is available in either a cloud-based or a self-hosted version, depending on your preference. It scales to “hundreds of thousands of logs per second,” generating terabytes of data per day, all the while offering complete security of that data as well as real-time log analysis. Both the company and the LogDNA product itself are SOC2, PCI, and HIPAA compliant as well as Privacy Shield certified.
Best “Freemium” Log Management Tool Options
Between free and paid options, there lies a category of solutions that offer a subset of features for free, but you have to move to the paid tier to enjoy all of the benefits (and usually support, and sometimes even upgrades). But for some IT pros, what comes in at the free level is all they need, and if they need the extra features, the upgrade doesn’t require a rip-and-replace installation.
Graylog is a free, open-source log management platform that can parse, normalize, and enrich logs and event data. Its processing rules allow you to set multiple options for routing messages, black- or white-listing, and even modifying (“enriching”) log messages before moving them to the next step of processing. Graylog also has a robust dashboarding capability that lets you filter out metrics from log messages and then display them in multiple ways, including charts and graphs. Of course, alerting and notifications are possible as well. The only difference between the open-source (free) version and the paid is the addition of offline archiving, user audit logs, support, and an “implementation jumpstart” to get you up and running faster.
XpoLog aggregates log files from selected sources and will monitor those locations/files included in its scope. Once data is centralized, the data is merged into the XpoLog database for processing. Those records can be searched and filtered for analysis, and results can be written out to files, parsing by date or other criteria. XpoLog analyzes data from a wide variety of sources, including Apache server logs, AWS, Windows, and Linux event logs, and Microsoft IIS. It can be installed on systems running Mac OS X 10.11 through 10.13; Windows 8 through 10; Windows Server 2008 R2 through 2016; and any Linux distros running Kernel 2.6 or later. There is also a cloud-based option. The free version allows you to process up to 1GB of data per day, and the system will retain that data for five days. From there, paid tiers add to either the volume of log data that can be processed, the retention period, or both.
Through its use of additional sensors, PRTG can extend its Network Monitor solution to monitor a wide variety of other targets. For log monitoring and management, two different sensors are available. The Event Log Windows API sensor is, as the name implies, built to capture Windows Event Log messages. However, rather than triggering on a specific message type or keyword pattern, this sensor monitors the rate of log messages and generates an alarm if the rate reaches a critical threshold. The other log-related sensor is for syslog. This sensor aggregates messages and then alerts either when a particular message type is received, or when the rate of a type of message crosses a threshold.
Splunk is well-known within the system administration and monitoring communities. Logfile sources (whether that is text file data shipped from a remote system, syslog, trap, or some other stream) are aggregated on the server running Splunk, indexed, and stored. A data sorting and filtering utility is built-in, as is the ability to alert, write out to files, and more. The free version of Splunk is limited to receiving 500MB of data per day.
Best Free Log Management Tools
One of the three-dozen-plus free tools from SolarWinds®, Event Log Consolidator does just what the name implies—it takes the Windows Event Log from multiple systems (up to five) across your network and pulls them into a single repository, then highlights patterns and trends across all systems to help you spot persistent but systemically dispersed issues.
Another from the SolarWinds free-tool stable, this will receive trap or syslog from up to five systems and then act on those messages by forwarding, alerting, or storing the data through the use of filter rules.
ManageEngine is another well-known maker of network administration tools among IT professionals. This utility collects, manages, analyzes, correlates, and searches through the log data of over 700 sources using a combination of agentless and agent-based log collection as well as allowing you to directly import logs if you want. Clocking in at 25,000 messages/second, with real-time attack detection, it can also quickly perform forensic analysis and reduce the potential impact of a breach. Note that the free version is limited to five log sources.
The Mostly Un-Necessary Summary
Like the other “top monitoring software” articles, this list is by no means complete. Log monitoring is a vast and varied sub-specialty within the monitoring discipline, and there are solutions out there to fit almost any use case. If you are just beginning your search for the right tool for the job, I hope this has given you a head start. If you already have a log management tool and are either considering a change or addition, I recommend trying a log tool like SolarWinds Log & Event Manager primarily due to its focus on helping IT departments easily manage security and compliance with an easy-to-use event log monitoring solution.