Is your security team the scarecrow, the tin man or the cowardly lion?
Security teams that want to be data driven need to realize that they’re not in Kansas any more. Auntie Em’s generic vendor dashboard won’t do the hard work of measuring how your team is doing. Instead, you must establish your own team metrics in order to know if you’re making progress towards real assurance for the organization, or if you’re falling behind the tornado speed of your company’s DevOps and software engineers.
To set effective metrics for your team, start by considering which of the following archetypes you are. For each type illustrated below, I’ve highlighted how to tell it’s you and what your focus should be for measuring improvement over time. And the truth is that we’re all a little bit of each character.
The Scarecrow
Security teams never want to think of themselves as a scarecrow. Scarecrows are fake protection, stuck in the mud, and only keep out the most bird-brained of intruders. If your team is advancing its log collection from compliance to insights, track metrics such as:
Server to Alert Ratio: Better analytics will be reflected in having fewer false positive alerts over time.
Percentage of detections powered by statistical methods: Static signatures are easy to bypass and become less effective as attackers evolve. Statistical baselines of “known good” activity, on the other hand, become more reliable over time.
Team data science proficiency: If you want your team to be analytically minded, provide data science training to team members and track their proficiency over time. This metric can be as simple as a self-reported confidence rating or a more objective function tracking data science courses successfully completed in online schools like Udacity or Udemy.
The Tin Man
Your team might be doing the most solid security ever but they’re still just a part of a larger machine. If your partners in compliance, engineering and management don’t feel the love, you need to be seeking a heart. Metrics to track your progress should include:
Report reliability score: As the company’s security team, your reports should guide developers, operations engineers, and management in decisions such as code updates, infrastructure configurations and solution purchases. These teams are your internal customers. But are they satisfied with the reliability of your reports, or are they frustrated with broken dashboards, findings that aren’t actionable, and ambiguous delivery schedules? Find out!
Detection transparency score: For enterprises required to meet cybersecurity standards such as PCI or SOC, compliance teams are frequently required to attest as to what sort of attacks would be discovered by the security team. But do your friends in compliance have a complete picture of what threat detection is in place? Ask for a confidence score between 1 and 10, and use better communication to improve your partner’s understanding of what you’re watching for and what you’re not yet.
End-user satisfaction score: Sometimes security teams are so eager to protect the organization that end-users are made to struggle with slow systems, blocked actions, and ambiguous approval procedures. Regularly polling users will help you to feel their pain and empathetically focus your efforts on achieving a better balance between protection and convenience.
The Cowardly Lion
If you’re responsible for protecting both Linux and Windows servers but you’re more comfortable with Bash than PowerShell, you might be choosing Linux security projects that are within your comfort zone even if the greater risk to the company is in a Domain Controller that you can’t even RDP into. Start tracking metrics that pull your security team out of their comfort zone.
Tabletop exercise count: Running a tabletop exercise is one of the best ways to surface gaps in your threat detection and response strategy. Spending an hour with someone from another team who is both knowledgable about your environment and has the attacker mindset can leave you with a list of a dozen good action items. Expect to find activity that isn’t logged, attacks that don’t have corresponding detections, and investigation dashboards that need to be built. Track how many of these you do each quarter and strive to increase the frequency.
Bug bounty: If your team is responsible for application security, track the volume of confirmed vulnerabilities reported via a bug bounty program. Ideally, group your metrics into categories such as “code injection” or “Vulnerable hosted application”. These metrics will pull you into examining areas that may be neglected in your current strategy.
Credentialed red team exercises: After you’re feeling good about your security posture in a certain area, bring in a professional red team with a defined objective to evade and exploit. Any respectable consultancy will poke enough holes to cause some embarrassment: quantify their success and refocus on those areas where they got past your defenses.
Conclusion
Quantitive metrics that track how your security team needs to improve are like the Yellow Brick Road of security analytics. Follow them!
