Data Platforms Will Eat The SIEM

Every day last week I spoke with enterprise customers about moving SIEM workloads into their Snowflake. Security leaders, from SOC architects to CISOs, are responding to fundamental changes in technological requirements and capabilities. In short, it looks like data platforms will eat the SIEM.

Over 10 years ago, Security Information Event Management (SIEM) emerged as the hub of the Security Operation Center (SOC). SIEMs combined the storage of security data, primarily machine logs, with monitoring dashboards and alerts. This centralization satisfied compliance requirements and enabled writing alert rules that could be applied across logs from firewalls, antivirus, and other security sensors.

Unfortunately, SIEMs have broadly failed in detecting threats to the enterprise. The 2019 Verizon Data Breach Investigations Report describes how “the time from the attacker’s first action in an event chain to the initial compromise of an asset is typically measured in minutes. Conversely, the time to discovery is more likely to be months.”

SIEM innovations such as User and Entity Behavior Analytics (UEBA) haven’t secured traditional Windows environments, never mind spotting new threats like compromised Azure cloud admins. While this was never going to be an easy problem, the effectiveness of SIEM has been declining as the technology fails to keep up with changes to the landscape.

What are the new requirements that SIEMs are failing to meet? And why will this situation result in enterprise data platforms absorbing them?

New Requirements and New Capabilities

Handle exploding data volumes

A SOC architect tasked with setting up a new security stack recently described how log volumes changed when his company moved from on-prem infrastructure to the cloud. From 2 TB a day, the environment is now generating 20 TB.

Stretching a SIEM to deal with 10x data is not just a technology problem. The entire business model of traditional SIEM vendors was not built for the verbosity of new log sources like AWS CloudTrail, where every action can be logged. Meanwhile Kubernetes is taking off and it is chatty.

The only way to keep up with the explosion in log data volume is to separate storage and compute. Even all of your VPC flow logs and object access logs won’t fill up the S3 service or the Snowflake database that runs on top of it.

Correlate across SaaS and IaaS

Enterprise users are now doing sensitive work across dozens of SaaS solutions ranging from Okta and Workday to Salesforce and Github. Security teams can’t assume that an attacker will limit themselves to one environment, so they need to connect the dots across activity in each of these SaaS solutions.

The challenge of normalizing and correlating these datasets is a much better fit for modern data platforms where ETL from cloud sources is par for the course.

Meet new standards like MITRE ATT&CK

One of the advantages that defenders have gained from the move to the cloud has been the establishment of threat detection standards. The people behind MITRE’s ATT&CK framework may not know what the inside of your datacenter looks like but they have a good sense for your AWS or Azure environment.

Security teams are expected to keep up with these standards but building out the necessary analytics is a tall order. It makes sense to have dedicated vendors building the capabilities needed to reliably detect attacker Tactics, Techniques, and Procedures (TTPs). With a cloud data platform that supports secure data sharing between companies, expert shops like Hunters.ai can apply their detection capabilities directly to the aggregated data.

The Secret Ingredient

Unblocking this shift is that platforms like Snowflake and BigQuery have great support for JSON. That wasn’t the case in traditional data warehouses, keeping their natural advantages out of reach for security teams that must deal with weakly-structured log data.

United at Last

The mounting challenges facing SIEM deployments mean that they are increasingly a losing proposition to create and maintain. Ballooning costs and dubious results put implementers at risk and looking for alternatives.

Luckily, this is happening at the same time as modern data platforms like Snowflake are being embraced at enterprises in every industry. One of the strongest trends today is the growth of the data organization. Infosec can now join marketing, finance, and HR to build on the central data solution of the enterprise.

Looking back, the disadvantages of having a separate security database product will seem obvious. Siloed away from important data sets, requiring dedicated maintenance, and isolated from data team support, the SIEM will be slowly deprecated. Workloads from threat detection to vulnerability management will be migrated and when there are none left someone will pull the plug. It might even be a physical plug.

Unifying security analytics with the rest of enterprise data is a natural technological evolution but might also be the revolution that cybersecurity has been waiting for.