Can Big Data solve our Security Challenges?

On a daily basis, you hear about some company getting hacked or losing customer records.

This pattern has become so routine that the shock factor has gone away.

The world has come to the acceptance that we cannot be secure. There will be hacks and customer records will be compromised.

Is that the right thing?

Should we have a callous attitude toward these recurring news stories?

• Do companies have a moral responsibility to keep their customer records safe?
• Do governments have an obligation to keep personal details of its employees and citizens safe?

There can be many such questions whose answers depends on the perspective and the person answering.

You probably have now heard about the Ashley Madison saga.
If you need a refresher or the latest, google is the best source.

http://www.wired.com/2015/08/happened-hackers-posted-stolen-ashley-madison-data/

There has been a lot of focus on using data to make networks and systems safer, in recent times. You may have seen presentations on these topics.

There can be Context Driven Authentication, Context Driven Authorization and Context Driven Audit in play. All these are critical for a secure system in operation.

But without the proper infrastructure and processing in place, it will be hard to deal with the plethora of data.

• Will Hadoop help? Isn't that more batch oriented?
• If Map Reduce Framework is suited for batch processing, will the new Apache Spark processing help in real time processing of security data?
• Do we need Lambda Architecture to tie in Near Real Time and Batch processing?
• How about machine learning? (http://www.technologyreview.com/news/542971/antivirus-that-mimics-the-brain-could-catch-more-malware/)

These questions may pop in mind when you think of using Big Data to solve security problems.

If contextual data is collected for security decision making, you will need petabyte scale storage and processing. The processing has to be near real time (NRT).

It is encouraging to know that Apache Spark is petabyte tested at NetFlix. 

http://blog.hampisoftware.com/index.php/2015/10/03/is-apache-spark-ready-for-petabyte-scale/

Maybe Spark processing with HDFS elastic storage is the way forward.

What are your thoughts? 

Can Big Data and Security Intelligence solve our challenges with customer data compromise?