Challenges in Cloud Compliance
Like salmon swimming upstream, most security leaders trying to adopt established frameworks won’t make it. Based on conversations with security heads and their compliance teams, I’ve found that the problem is not the threat actors, the overblown “talent shortage” or grizzly bears. It’s a data problem.
There’s no shortage of well-defined standards. NIST, CIS, and ISO each provide a set of best practices that are clear and actionable. For example, the AWS architecture site describes the CIS Security Benchmarks program as providing “well-defined, unbiased, consensus-based industry best practices to help organizations assess and improve their security”. These benchmarks are a great place to start when looking for improvements to the organization’s security posture.
While a number of solutions provide off-the-shelf configuration checks, their canned reports have not been widely successful for their customers. Several limitations of these validation tools were detailed in my previous post but can security organizations generate their own, more impactful, compliance reports?
There are big obstacles facing any team trying to take a data-driven approach to compliance validation. The biggest hurdle to data-driven compliance is cloud vendors making it tough for to centralize raw configuration data.
Configuration validation in AWS
To get a sense for the problem, let’s consider how we can check our AWS environment for compliance with CIS 1.2: “Ensure multi-factor authentication (MFA) is enabled for all IAM users that have a password”. We want to know if everyone with a login to our AWS console also has an MFA device in order to reduce the impact of stolen passwords.
Our first validation approach might be to use the AWS Config service. We need data on which of our users have console login access and which of those users don’t have an MFA device enabled. However, Config does not record some key data points including the MFA device details for an AWS user.
This and other data gaps can be seen in the AWS documentation and are described in the excellent article titled “Should you use AWS Config?” by Summit Route’s Scott Piper. He provides a detailed indictment of Config’s gaps, writing that
My biggest complaint with AWS Config is the lack of coverage it has over AWS services. In almost every AWS account there will be resources that it does not keep track of...
Even AWS’s oldest service, SQS, is not recorded. This isn’t full support for the services, but just any support whatsoever, which works out to about 19% of services…
That 19% figure is for entire services not showing up in Config. Even more challenging is when a service is partially covered. Security engineers are liable to assume that if a service is monitored by Config then all of the data they need for analysis is being recorded, while really Config only records part of the resources for many services. Piper writes that according to his calculations,
AWS Config only supports 20% (67/334) of the resources CloudFormation supports.
If you expected to have some sort of historical record about your account by using AWS Config, you will be sorely disappointed…
Any service and resource in the gray portion of Piper’s chart will not be recorded by AWS Config and cannot be a source for compliance validation.
Configuration validation in Azure
The situation in Azure is even worse. Consider CIS Azure Benchmark 1.15 within the Identity and Access Management section. This control reasonably suggests that organizations prevent non-administrators from accessing the administration portal. In the same section, however, the authors admit that the only way to validate this control is for someone from the security team to manually log into the console. That’s not scalable and not data-driven.
Azure writes in their documentation that “Some controls are grayed out. These controls don’t have any Security Center assessments associated with them. Check the requirements for these and assess them in your environment on your own”.
The screenshot below is taken from a section of Azure’s built-in compliance report. At the top of the page, the description mentions that “not all controls for any particular regulation are covered by Security Center assessments, and therefore this report is only a partial view of your overall compliance status”. The number of grayed-out controls is significant, placing the validation burden on the customer.
As a customer, it’s been frustrating to find that many third-party security vendors have cut corners to get around this problem. Entire sections of compliance standards are grayed out in vendor reports too. In some cases, results marked as “manual” are mixed in between automated findings. The expectation is that the customer will manually validate the requirement. How many will? And at what cost?
Fixing the problem
The change won’t come from the auditors. Many auditors are glancing over compliance reports, not calling out that some controls (and some entire sections) have not been validated by the system. If the auditors don’t raise the issue, it’s easy enough for companies to accept these gaps or plug them with manual, best-effort validation.
As a result of this data scarcity, governance teams are spending days of effort on monthly, quarterly, or annual checks. These are frequent enough to distract and divert limited headcount but not frequent enough to support an effective “checkbox security” strategy. Misconfigurations that are so risky in the cloud may not be found until it’s too late.
Can we as a security community demand from cloud providers easy access to our configuration data? Configuration data availability should be a feature request in every meeting with an IaaS or SaaS vendor.
In the meanwhile, open-source tools are emerging that grab configuration data from the major cloud providers. As the necessary data becomes available, security teams will become equipped to automate compliance validation. These exciting initiatives will free team members from manual validation work and reduce the prevalence of dangerous cloud configurations.
