How Sumo Logic Turns the Event Data Tsunami into Continuous Intelligence
The ongoing explosion of event data can easily overwhelm your ability to make sense of it. Modern applications — many containerized and running on the cloud — generate huge streams of log data describing everything that goes on. You could build your own system to trap and analyze this data. Many organizations have tried. So why have 2,000 companies signed up for Sumo Logic’s service instead?
Here at Illuminate 2019, Sumo Logic’s third annual user conference, a possible answer has been discovered: Building distributed log analysis systems is really hard. Ambitious development shops have picked up various open source NoSQL and Hadoop tools in an attempt to build their own, and many have failed.
In fact, Sumo Logic also failed in its attempt to use various open source technologies before it found the right recipe for building the back-end distributed system that’s responsible for analyzing 100PB of customer data per day. A mix of open source and proprietary technologies sit behind Sumo’s software as a service (SaaS) offering, according to Bruno Katic, co-founder and vice president of product strategy of the San Francisco company.
“In many cases, we tried to use open source technologies, but it just didn’t scale,” Katic says. “The 500 trillion records scanned in a single day by our customers — we couldn’t get anything to work at that scale.”
The company does avail itself of many open source components, including Kubernetes for managing clusters, NoSQL databases like Casandra for serving data, and React for developing front-end dashboards, among many others.
“But a large majority of things under the hood we did ourselves,” Katic says. “We have a proprietary indexing engine that does distributed sharding of fragments of data. It’s highly, highly parallelized and real time.”
By all accounts, it appears to be working.
Closing the ‘Intelligence Gap’ in DevSecOps
Sumo Logic has filled a niche for providing “continuous intelligence” solutions that help developers, operators, security personnel, and business analysts do their jobs with the modern application stack. You could call it DevSecOps.
In his keynote address, Sumo Logic CEO Ramin Sayer lamented how the “tsunami of data” that threatens to hit 16 zettabytes by 2020 is making it harder for DevSecOps folks to do their jobs.
“These intelligence challenges are just becoming bigger and bigger,” Sayer said. “The continuous data tsunami is impacting the way every team, every individual needs to be able to communicate and more importantly collaborate to make real time intelligence decisions. Every team is dealing with the lack of visibility because they have siloed tools, and siloed tools actually prevent intelligent decisions.”
Many of Sumo’s customers are cloud-native firms, like Airbnb and Pinterest, as well as traditional enterprises, like Marriott International and Alaska Airlines. PagerDuty, the web-based incident as a service firm, relies on Sumo to help it monitor and alert on incidents for over 10,000 customers.
“We love the fact that time-to-value is near instantaneous with Sumo. It’s exactly how we approach our product platform,” Tim Armandour, SVP of engineering for PagerDuty, told the Illuminate 2019 audience yesterday. “Sumo Logic is our 10x engineer. We have so much leverage that we can gain. The fact that we can count on Sumo to be our eyes and ears in live environments really feels like what sets us apart from the rest of the pack.”
Different Faces for Diverse Teams
Depending on your role, you could use Sumo Logic for different tasks.
For developers, the SaaS offering can suck in and track various data relating to the modern development path, including from CI/CD tools like Jira, Jenkins, and GitHub, as well as instrumented data from applications undergoing tests.
The company just released the fourth edition of its “The Continuous Intelligence Report,” which provides an insightful glimpse into state of the cloud applications of its customers. The report has some interesting nuggets, such as the fact that the average Amazon Web Services customer uses just 15 AWS services, and that NoSQL databases are more popular than relational databases.
“Our customers are hungry for benchmarks,” says Kalyan Ramanathan, vice president of marketing and the primary author of the report. “A lot of people are going to the cloud. They’re stumbling through it, a little bit like we were in 2010…They want to know, what do I do? What is acceptable? What is unacceptable? What is out of norm for us?”
For operations folks, Sumo Logic collects, monitors, and analyzes a variety of log and event data that’s relevant to managing applications in today’s modern cloud stack. (The company also grabs data from on-prem systems, but its forte clearly is with cloud and SaaS applications). Yesterday’s launch of a new app blueprint for Kubernetes will help operators deal with the 200,000-odd log messages that can emanate from everybody’s favorite cluster manager.
The combination of the DevOps phenomenon and modern cloud applications is leading to an overhaul of the tools and processes required for the software development lifecycle, Sayar says.
“Those who were born in the cloud are becoming a little bit more centralized,” he tells Datanami. “There are a lot of shared functions versus the traditional software development lifecycle that you see in traditional enterprise — the design team, the development team, the build and release team, the operations team, and hand off to IT. That factory chain is fundamental changing. Effectively that whole lifecycle has been liposuctioned.”
For security professionals, Sumo Logic can behave like a massive cloud-based security information and event management (SIEM) tool. It serves as a data lake for storing security logs collected from a variety of systems, and its pre-canned dashboards and metrics can help people spot possible security issues, hopefully before they turn into problems.
“Think of Sumo as a cloud SIEM that takes in all the data from every source, all the alerts from every source, analyzes it and says ‘Here are the things you should pay attention to with your scarce resources,'” says David Frampton, Sumo’s vice president of security solutions. “Most security groups get to about 10% of what they ideally would wish to. So it’s a really important question of which 10% do I pay attention to.”
Business analysts can also get into the act with Sumo Logic, which can function like a business intelligence tool if you like. While Sumo Logic isn’t going to compete with the likes of established BI vendors Tableau or Qlik, or up-and-comers like SiSense, ThoughtSpot, or Looker, analysts can and do use Sumo Logic to monitor business metrics, which invariably begin life as log data generated by applications.
No matter which persona is using Sumo Logic, the company relies on patented machine learning algorithms to help it detect anomalies with large streams of event data. Customers can also build their own alerts using the Sumo Logic query language, which is described as “SQL-like.”
Since it was founded nine years ago, Sumo has grown into a company with more than $100 million in annual revenue. In May, Sumo closed its latest round of financing, which netted it an additional $110 million at a valuation in excess of $1 billion. The presence of scores of financial analysts in the Hyatt Regency SFO this week has raised speculation that the company could be on the path for a public offering, which is something that Sayer has talked about publicly in the past.
If Sumo were to go public, it would following the footsteps of two of its closest competitors, including Splunk and Elasticsearch. Of the two, Sumo’s business model is arguably closer to that of Splunk, owing to the proprietary nature of Splunk’s offering compared to Elastic’s open source approach.
Whatever becomes of its ownership, the company must deliver value for customers. Feedback from customers indicate that man are satisfied with Sumo’s offerings now, but that they would like more prescriptive guidance on how they should respond to the intelligence surfaced by Sumo, Ramanathan says.
Since Sayar first coined the term Continuous Intelligence in a blog post nearly four years ago, the idea has caught on, with Gartner and others. With a solid log collection and analysis system that works with a variety of data, the company is well-positioned to expand the scope of its solutions and help customers make better use of ever-growing streams of data.
“We’re in a fortunate position where we defined this category and we’ve become the category leader with respect to continuous intelligence,” Sayer says “The fortune news for us is we’re still early in that, despite all the success we’ve had, primarily because more and more the traditiona and enterprises are going through their digital and cloud transformations, hence need a partner and technology like Sumo. So I see continuation a lot of the same.”