If security professionals are to have any chance of successfully containing advanced threats, they must automate the infection validation processes. The attack vectors are seemingly endless and there are simply too many alerts — many of which are false positives — to investigate and analyze manually. In my last post, I talked about how automation can be achieved via an intelligent decision-making system (IDMS). Let’s look at what it takes.

IDMS for Automated Infection Validation

An IDMS begins with a big data infrastructure such as a Hadoop cluster and massive amounts of raw data. You can subscribe to intelligence feeds and have them load into the system, along with data from your own security and other network tools. Security information and event management (SIEM) tools can help in this regard, as well as tools like Splunk, but they only get you partway there. These tools allow a human to run queries off the data you’re gathering, but the queries don’t make a decision for you as to what threats are present on the network, the risks they pose and how they should be addressed.

The decision-making aspect requires massive algorithms that essentially tell the system, “If this, then that.” But it is never that simple. In reality, the algorithms are more like, “If x exhibits y behavior for more than z time, set n variable to q.” In order to write these algorithms, data scientists must work closely with security threat experts to understand the variables, their relationship and the desired outcomes.

Most importantly, an IDMS requires a software system that sits on top of the big data infrastructure and determines whether a device on the network is infected. This decision-making component must be built by developers, who also put algorithms built by the data scientists into this system’s code infrastructure.

The system is now ready to perform automated infection validation, but it will never be “done.” The algorithms are massive programs that are works in progress, changing to accommodate an evolving threat landscape. Security threat researchers, developers and data scientists must continually evaluate and tweak the algorithms, building additional systems that will automate the continuous learning of the models.

Worth the Effort

The process of building and maintaining an IDMS is not easy. It requires very specialized skill sets and is resource-intensive — and those resources are not easy to come by. Data scientists are in short supply, and few enterprises can afford them or additional security experts. But that doesn’t change the need for automated infection validation and investigation.

I know from talking to enterprises that organizations are attempting to build an IDMS with various levels of success. A recent Ponemon Institute survey seems to concur, with 41 percent of respondents claiming to use some automated tools to capture intelligence and evaluate the true threats in a sea of alerts.

I’m curious to hear from others. Is this something you’re trying to build today? If so, how are you addressing the challenges and what successes have you had?