Logging refers to the practice of creating and maintaining the logs an application or system generates. As logging can take varied forms depending on factors like severity level, destination and format, having a log management tool becomes quintessential. A comprehensive log management tool allows IT teams to analyze a large number of log entries and gain useful insights out of them. It also offers capabilities like unlimited log retention, intuitive UI, automated log parsing, health metrics, and audit trail amongst others.
As there are dozens of log management tools, each offering an array of features, it becomes crucial for organizations and users to pick the right tool. Let’s deep dive into the two key tools: SolarWinds® Loggly® and Scalyr.
Loggly
Loggly is a cloud-based log management tool. It’s a scalable service enabling users to collect and aggregate logs from the systems across the network and store them at one accessible location. As it makes log sharing easy, teams can collaborate. It provides faster troubleshooting and lowers operational costs, while working agentless and supporting text-based logs from any source. It offers automated parsing capability and provides a near-instant search response over colossal amounts of log data. It also saves the work pages through its persistent workspaces feature and users can scan full events or see the relevant fields in a grid view.
Scalyr
Scalyr provides fast data ingestion and searching capabilities. It gives real-time visibility into the logs and generates instant alerts. It works well even on large amounts of data sets. It leverages NoSQL columnar databases along with having a cloud compute infrastructure to visualize and analyze the log data. Scalyr is also preferred by users having limited expertise or knowledge of search query language. It comes with built-in parsers and enables users to create custom dashboards. Scalyr seamlessly integrates with several container types, databases and interfaces, and collaboration tools.
Kubernetes Logging
Despite the increase in Kubernetes adoption, it does have a few limitations concerning log collection and analysis in Kubernetes environment. A major problem users may experience in Kubernetes logging is the difficulty to monitor cluster activity and understand system interdependencies. Another difficulty arises with the dynamic nature of Kubernetes, which requires fast log aggregation. It results in a high volume of log data causing an increase in cost and performance degradation. Scalyr is designed for the Kubernetes environment, and it provides built-in support for metadata and namespaces and also supports controller types like Deployment, DaemonSet, StatefulSet, Job, and CronJob among various others. It also includes log summaries based on deployment in contrast to individual containers enabling IT teams to get a clear view of the complex nature of Kubernetes logs. Loggly, in comparison, offers a simple way to Kubernetes logging. It collects logs at a centralized location. Users can access the logs via a web interface and control the access. It expedites the entire process of Kubernetes logging by managing ELK scaling for ingestion and searching and provides everything from dashboards to visualization. The Loggly Dynamic Field Explorer lets users drill down further on the Kubernetes logs. Users can also use Fluentd plugins for viewing relevant Kubernetes tags to get an in-depth view of Kubernetes logs. It also offers an easy visual analysis of Kubernetes logs via interactive charts and dashboards easily shared within the team.
API Support
Scalyr API is capable of sending or retrieving log data directly from the server. It supports some crucial APIs, including the upload logs API, add events API, and numeric query API. Upload logs API allows users to upload raw data to its servers. It’s best suited for conditions involving lightweight integrations or individual batches of data from a stateless environment such as AWS Lambda. The addEvents API allows users to insert structured log events. Its query API lets users retrieve log events as per needs and can also be invoked by GET. The numericQuery API lets users retrieve numeric data. Users also can count the rate of events matching some criterion or retrieve a numeric field. The Scalyr API format requires users to send an HTTPS POST to the URL where the request should have content-type “application/JSON,” and the body should be a JSON-formatted, UTF-8 encoded string for most of the APIs except for the uploadlogs API, which uses simple text bodies. Loggly works with RESTful APIs, and its flexible API lets users send log events to Loggly via GET or POST methods. To provide advanced security for interaction with Loggly API, it implements a token-based authentication mechanism.
Amazon S3 Log Ingestion
Scalyr requires setting up S3 buckets or creating SQS queues along with the Amazon Identity and Access Management (IAM) tools to create an IAM role with limited permission. This IAM role is then only used to read S3 bucket and queue. Loggly automatically retrieves new log files added to S3 bucket and supports logs from Elastic Load Balancer (ELB), Application Load Balancer (ALB), Cloudfront, or any uncompressed line-separated text files. It provides users with the choice of configuring AWS accounts either automatically or manually. Loggly uses the Amazon Simple Queue Service (SQS), to help ensure reliability in the process of sending events and saving the event until it’s retrieved. It supports many file formats including .txt, .gz, .json.gz, and .log in addition to any plain text or zipped file, provided the S3File Metadata on AWS is of the format of text/plain, text/rtf, application/x-gzip, or application/zip.
Conclusion
Both of these tools offer extensive log monitoring features. While Scalyr integrates seamlessly with organizational workflows, Loggly provides much more. It provides proactive monitoring, and its common trace ID capability lets users have a common view of impacted systems and event data. Users can also share dashboards via shareable links. Loggly also provides Atlassian, Jira, and GitHub ticketing integration enabling users to be more productive. Check out the free versions of Loggly and Scalyr and get a hands-on experience of their features and capabilities today.
*As of June 2020