Are all of our servers running security agents?
Have you heard the one about the drunk guy that lost his keys one night? A policeman sees the guy searching for something under a streetlight and walks over to help. After they spend ten minutes looking everywhere under the streetlight, the policeman asks him if he’s sure that he lost his keys here. The drunk guy replies no, he dropped his keys somewhere in the park. So the policeman asks why the heck they’re searching here and the drunk replies, “this is where the light is”.
That old joke describes the streetlight effect where people only search for something where it’s easiest to look. Keeping the streetlight effect in mind when building our security analytics helps us to prioritize the essentials, like identifying servers missing security agents. It’s those servers where we don’t have visibility that are most likely to serve as points of entry and bases of operation for our adversaries.
At Snowflake, we track the percentage of servers where we have visibility gaps and closely measure our performance on this metric. While Operations is responsible for installing agents, we at Security hit our targets by building daily lists of servers missing security agents and making these lists accessible to our operations and compliance teams. We’ve found that once the data is made available, this problem of visibility gaps gets quickly addressed. It’s generating the data that’s the key challenge.
Our approach is to use our cloud infrastructure APIs as the source of truth for what servers we have under our responsibility. Note that we don’t trust any agent-based solution for our inventory. That would be like searching under the streetlight.
In a previous post I described how we keep track of our AWS accounts. So that we don’t rely on agents for our inventory, we query the EC2 API for a full list of servers from each AWS account. A code example of using the EC2 API for listing instances is on GitHub.
The remaining piece of analytics is comparing this complete list of servers with recent log data from each agent type. For example, we have a query that joins between the servers list and our recent osquery log data. If a certain instance ID exists in inventory but not within the osquery logs then that indicates a visibility gap that needs to be addressed. An analytics code sample is available within the SnowAlert project, showing how we use a Snowflake SQL query to shine light all across our AWS cloud infrastructure.
