Log management stopped being a very simple operation quite some time ago. Long gone are the “good old days” when you could log into the machine, check the logs, and grep for the interesting parts. Right now things are better. With the observability tools that are now a part of our everyday lives, we can easily troubleshoot without the need to connect to servers at all. With the right tools, we can even predict potential issues and be alerted at the same time an incident happens.
This is where log analysis tools come into play – especially cloud logging services. Cloud logging services aim to provide a service that lets you ship your logs and store them securely, aggregate, and analyze them so that you can take actions based on the current and historical data and correlate it with other information that is part of the whole observability solution – like metrics and traces.
If you’re interested in such a solution, look no further. In this post, we will compare some of the best cloud logging services available to help you get started.
Why Do You Need Cloud Log Management Services
In the modern software era, it is close to impossible to keep an eye on everything, especially in medium and large-scale systems. The number of systems, servers, and IoT devices that are a part of such systems makes it impossible to manually manage, monitor, and analyze their logs. Add to those different business requirements, different compliance requirements and we quickly run into a situation where a well configured and maintained log centralization solution is a necessity.
Keeping the data in-house with one of the available log management tools may seem like an easier and cheaper choice at first, but as the amount of data grows managing the data store used for logs becomes a job on its own. It is not unusual to have a dedicated team managing the clusters responsible for the whole observability pipeline – which logs are a part of.
The cost of housekeeping and managing the logs yourself can be mitigated to one of the commercial vendors that provide SaaS products that allow us to send, store, manage and analyze logs.
Best Cloud Logging Services
With quite a large number of logging services on the market let’s look at the best ones available.
1. Sematext
Sematext Logs is a cloud logging service that allows you to centralize the management of your logs coming from various sources like applications, microservices, operating systems, and various devices. The platform enables you to structure, visualize and analyze all collected data passively and actively. You can create informative dashboards connecting every piece of information and observe how your systems are behaving in real-time or set up alerts to be notified when a critical event happens.
You ship your logs securely with the use of TLS/SSL channel via HTTPS or syslog and use per-user access restrictions to fully control who can access which data. With the possibility to store the data in your own S3-compatible storage you can keep your logs indefinitely without any additional cost.
Sematext Logs is a part of Sematext Cloud, an observability platform that provides a single pane of glass for log management and log monitoring, infrastructure monitoring, real user monitoring, and synthetic monitoring enabling you to combine all the information together for full system visibility.
Features:
- Out-of-the-box integration with popular tools like Syslog, Logstash, Beats, and many more.
- Ease of correlation between various log sources and metrics, frontend and backend.
- Simple slicing and dicing of data allowing quick and easy root cause analysis.
- Powerful alerting including threshold and anomaly detection-based alerts with the possibility of sending the alerts to different destinations, such as Slack, PagerDuty, OpsGenie, and more.
- Log security including TLS/SSL channels via HTTPS, API access control, and per-user access restrictions.
- Saved search with powerful language allowing you to get back exactly the logs that you were looking for.
- Possibility of archiving logs to S3 compatible destinations.
- Flexible pricing allows for overage and overage capping, ensuring that all your data will be accepted and ready for access when needed, with controls that let you limit your costs.
Pricing:
Sematext Logs offers various pricing tiers depending on the features, daily ingestion, and data retention with the option to start with a 14-days free trial that allows you to test all the features.
After the Sematext Logs trial ends, you can start with the Standard plan of as low as $50/month for 1GB worth of data daily and up to 7 days of retention time. The Pro plan with all the platform features starts at only $60/month for 1GB of daily data ingestion with up to 7 days of retention. The platform comes with a free, Basic plan allowing you to store up to 500MB worth of data daily and 7 days of data retention, which is often sufficient for small organizations.
2. SolarWinds Papertrail
Papertrail is a cloud log management service that allows users to aggregate log data from various sources starting from Syslog sources and ending up with applications like MySQL, Apache, Heroku, and various routers and firewalls. With the simple installation and DevOps-friendly UI, it is very easy to get used to and work regularly. You can get notified about the interesting events with the out-of-the-box alerting that can be integrated with numerous external services such as Slack, PagerDuty, or custom webhook.
Features:
- Easy Syslog integration allowing you to start sending logs in no time.
- Grep-like UI for easy to use and friendly everyday logs searching.
- Log events-based alerting support with external ChatOps services integrations.
Pricing:
Papertrail pricing is a factor of daily ingestion, search period, and data archive retention. It starts with $7 with up to 1GB of data monthly (note: not daily!), search on one week worth of data and 1 year data archive, and goes up to $230 with 25GB of monthly ingestion, search up to two weeks of data and 1 year of data archive. You can also choose to build your own plan with up data ingestion up to 1500GB/month and up to 4 weeks of searchable data that will end up with $5,470 on your invoice.
SolarWinds Papertrail provides a free plan that allows sending up to 50MB of data monthly with 48 hours of searchable data and also gives a bonus of 16GB of data ingestion during the first month.
Want to see how Sematext stacks up? Check out our page on Sematext vs Papertrail.
3. SolarWinds Loggly
SolarWinds Loggly is another cloud logging service providing log centralization, analysis, and visualizations along with APM integration for full visibility into your environment. The built-in alerting and support for anomaly detection allows you to stay on top of the issues no matter if you were prepared for them or not. Providing the data in the raw text format is not a problem with the Derived Fields functionality enabling you to define how fields should be extracted and used if your data requires it.
Features:
- Automated data indexing and parsing with the promise of supporting any text log format.
- Easy to use query language with the support of the search context for easier troubleshooting.
- Live tail with regex support for efficient, live monitoring of logs coming from different sources.
- Automated logs grouping and linking for faster root cause analysis.
Pricing:
SolarWinds Loggly pricing depends on the features and the amount of data that you need to send and store. The free tier called Light allows for a single user, 200 MB of daily volume and up to 7 days worth of data retention with no support for alerting, customized dashboards, etc.
The first paid plan called Standard starts at $79 per month billed annually and includes up to 3 users, 1GB of daily ingestion volume and up to 15 days worth of data retention with e-mail based alerting. The most expensive Enterprise plan starts at $279 monthly billed annually with all the features, unlimited number of users, custom data volume and up to 90 days of data retention.
Want to see how Sematext stacks up? Check out our page on Sematext vs Loggly.
4. Sumo Logic
Sumo Logic platform combines logs, metrics, traces, SIEM, and business intelligence in a single product with a focus on logs. It allows powerful log analytics with the LogReduce algorithm reducing the number of similar logs helping with repeating log messages. LogCompare allows comparing different periods of time to find the differences for efficient root cause analysis. Together with its other functionalities, Sumo Logic gives you an observability platform that goes beyond logs.
Features:
- Log analysis with the LogReduce algorithm allows clustering of similar messages.
- LogCompare lets you compare data from two time periods.
- Field extraction enables rule-based data extraction from unstructured data.
- Out-of-the-box support for popular cloud vendors and orchestrated environments.
Pricing:
Sumo Logic pricing is organized around the volume of data that you send and features and is aimed at larger Enterprises. The Essentials plan starts with an estimated cost of $277 monthly billed annually that includes around 3GB of log data a day and 30 days of data retention, metrics, and traces. The Enterprise Suite plan starts with an estimated price of $2,189 monthly billed annually and includes around 15GB of daily ingest and all the features of the platform such as metrics, traces, and SIEM.
The pricing was based on the North America location and will differ in different Sumo Logic locations – for example, at the time of this writing, the Ireland location was more expensive compared to North America. Sumo Logic offers a free tier as well which includes up to 500MB of daily ingest and 7 days of data retention.
5. Mezmo (Formerly known as LogDNA)
LogDNA is a scalable, SaaS cloud logging service offering log indexing, aggregation, and analysis. It supports various log event sources from applications, orchestration platforms, and common cloud vendors. The solution provides a rich UI including live tail for real-time log analysis, searching, graphing, and alerting on your log data. When it comes to the log data, LogDNA provides numerous integrations and log format parsing.
Features:
- Easy to use UI for searching and visualizing the log data in the form of graphs.
- Live tail for real-time log analytics.
- Alerting with the support for external services such as Slack, PagerDuty, or custom defined webhooks.
- Numerous sources integrations including common cloud vendors and orchestrated environments.
Pricing:
LogDNA pricing is organized around the amount of data you send and features. It starts with the Birch that includes up to 5 users, 7 days of data retention with the cost of $1.5 per GB of data. The most expensive plan costs $4 per GB of data, 30 days of retention, up to 30 users with a minimum charge of $200/month.
The features of the platform can be tested during a free, 14-days trial with no credit card requirements. It is worth mentioning that LogDNA offers a free plan that offers unlimited live tail functionality for real-time logging without any kind of historical data access.
Want to see how Sematext stacks up? Check out our page on Sematext vs Mezmo.
6. Datadog
Datadog platform combines log management with metrics and traces in a single, cloud-based SaaS solution giving you rich context to analyze and quick root cause analysis. The platform allows you to search, filter and analyze the data with the use of various visualizations and generate metrics derived from logs. Send data from numerous sources with the provided out-of-the-box integrations and enrich the ingested logs to give your data more context for easier troubleshooting.
Features:
- Live-tail for real-time log analysis.
- Easy data exploration with Log Patterns helping you in detecting patterns and trends in your data.
- Processing and data enrichment defined from the platform using the Pipeline Library.
- Log access management for limiting access to only those who should see given data.
Pricing:
DataDog log management pricing starts with $0.10 per GB of ingested or scanned data and includes enrichment of data, parsing on ingestion, log-based metrics, and left-hosted archives. If you wish to retain the data you can do it starting with 3 days retention that will cost you $1.06 per million events when billed annually or $1.59 when billed on demand.
When using Amazon Web Services, Google Cloud Platform, or Microsoft Azure you can rely on the log analysis capabilities offered as part of their platforms or solutions described so far. The cloud provider dedicated solutions may not be as powerful as the solutions that we discussed above and tend to have more basic log monitoring capabilities. If you are using a cloud service solution from Amazon, Microsoft, or Google and you would like to use monitoring provided by those companies have a look at what they offer.
Want to see how Sematext stacks up? Check out our page on Sematext vs Datadog.
7. Amazon CloudWatch Logs
Amazon CloudWatch Logs is a cloud logging service that lets you monitor, store, and access your log files coming from various Amazon services, like Amazon Elastic Compute Cloud instances, AWS CloudTrail, Route 53, and others.
CloudWatch Logs enables you to centralize the log events from all of your systems, applications, and AWS services that you use, in a single, scalable service. You can see all of your logs, regardless of their source, as a single and consistent flow of events ordered by time, order them by the time of the event and visualize them in the form of dashboards. This allows you to easily view them, search and filter them based on specific fields, or archive them for future analysis.
Features:
- Monitor logs from Amazon EC2 instances, Route 53 DNS queries, CloudTrail, and more.
- Easily archive the logs with flexible retention from one day to infinity.
- Use a powerful query language to interactively search and analyze the log data coming from your systems.
Pricing:
Amazon CloudWatch Logs pricing is organized around the volume of data that you generate. The free tier includes up to 5GB logs per month with up to 3 dashboards. After that, the first 10TB of data is charged $0.50/GB, the next 20TB is charged $0.25GB, and so on with up to $0.05/GB over 50TB of data sent.
8. Microsoft Azure Logs
The Azure Monitor Logs is a logging feature of the Azure Monitor that collects and organizes performance and log data coming from different monitored resources, such as Application and Container Insights, Azure network monitor, and certain other Microsoft Azure services. Gathered data can be analyzed, visualized, and alerted on giving a deep insight into the monitored Azure services.
Features:
- Analyze log data gathered from various Microsoft Azure services.
- Present the data in the form of tables or charts generated on the basis of the log data.
- Alert on the data and be notified when an event of interest occurred.
Pricing:
The logs part of the Azure monitor has the option to pay as you go which gives you up to 5GB of logs per billing account per month free and then $2.76 per GB of data. You can also choose to go for the reserved data option – for example, 100GB of data per day will cost you $219.52 daily.
9. Google Operations (formerly Stackdriver)
Formerly Stackdriver, Google Operations suite is designed to monitor Google Cloud Platform infrastructure resources usage and application performance, but it also supports custom metrics and monitoring of other cloud service providers like AWS. The platform provides metrics, logs, and trace support along with the visibility into Google Cloud platform audit logs giving you the full visibility of what is happening inside your GCP account.
Features:
- Rich and powerful query language for navigating through log events.
- Audit logs for visibility info security-related events in your Google Cloud account.
- Support for logs and logs routing with error reporting and alerting.
Pricing:
Similar to Amazon CloudWatch Logs and Microsoft Azure Monitor Logs the Google Operations logs pricing is based on the amount of data your services and applications are generating and sending to the platform. The logs free tier includes up to 50GB of logs per project, and everything above that falls into the paid tier for the price of $0.50 for each additional GB of log data.
What Cloud Logging Service Will You Use?
Choosing the right cloud logging service is not a simple task. You need to look at the features that you need at the moment and that you may potentially need in the future. For example, your use case may require you to have the live-tail with filtering support, so that you can look for certain log events live. You may need to have alerting with anomaly detection so that unusual patterns can be caught and reported to the destination of your choice.
Yes, available ChatOps integrations are also important, because getting the alert to the destination that is close to the on-call person is crucial when issues happen. Those of us that are legally bound to keep data for auditing will require long-term storage, such as S3 compatible storage that can be configured and automatically used. The ability to connect logs with metrics is a crucial enabler for faster troubleshooting, shorter downtimes, and faster MTTR.
Finally, the price may be an important factor – the more logs you store, the higher the invoice will be and different vendors have different prices – for a similar amount of data.
Look for cloud log management solutions that:
- Let you jump between logs and metrics easily and quickly. That’s one of the keys for faster troubleshooting and making your job and life easier.
- Offer powerful log search syntax so you can more quickly and more easily find data you need.
- Offer easy setup, save you time with out-of-the-box dashboards and alert rules.
- Have attractive and flexible pricing. Avoid vendors whose pricing involves per-user and per-usage pricing because that will force you to start thinking about who, and how many people have access to your logs, which is a complication you do not need.
- Can collect logs not only from various backend infrastructure, but also collect mobile application logs, for example.
Remember that migrating to another vendor may not be easy and may end up losing the possibility of searching through your data archive. Such migration will require reconfiguration of your log shipping software or even applications if you have chosen to ship the data directly from them. For large, highly distributed environments such operation can take a vast amount of time and is not negligible.
If you’re looking for a cloud logging service give Sematext Logs a try. Our log management service allows you to stay on top of your application, system, and platform logs. There’s a 14-day free trial for you to test all its functionalities. Give it a try!