In today's security analytics, every bit of data matters

There is a change brewing in information security and information management. In the early days, this discipline really came down to event detection. Security information management systems scanned a bunch of data looking for needle-in-the-haystack events that indicated trouble. All other data was considered "noise" and thrown away.

With the onset of regulatory compliance a few years ago, this model went through an initial change. The "noisy" data was now necessary information to demonstrate security controls for compliance audits. Still, event data and compliance data remained separate entities.

Now things are changing yet again. In today's dangerous security landscape, no data is considered "noise" anymore. Rather, security analysts now want access to terabytes of historical data for analysis. Furthermore, this underlying data has become more complex. Beyond just log files, security analytics now encompasses other data types like network flows, directories, physical access, and video surveillance. If there is reason to believe that Joe the IT administrator has been covertly accessing quarterly financial data, a subsequent security investigation will encompass everything and anything including when Joe was in the building, when he logged onto the network, which systems he accessed, and what he did.

This type of investigation requirement changes the security technology model. It means collecting, normalizing, and storing a ton of data. It means sophisticated algorithms and processor-intensive query engines. It means the integration of physical and information security, including video surveillance. Sound like law enforcement or the NSA? Perhaps, but large organizations are already headed down this path.

From an industry perspective, security information management systems will need to re-architected for this type of scale and power. Vendors like ArcSight, eIQ, Nitro Security, RSA, and SenSage have already anticipated this change--as have log management vendors like LogLogic and LogRhythm. This may also introduce the heavyweight security vendors like Comverse, Narus, and NICE into the enterprise space. In either case, I anticipate lots of activity in 2009 regardless of the current economic woes.