Security information and event management (SIEM)

SIEM systems perform two main functions, according  to documentation:

Security information management (SIM) : This sub-system (historically), was designed for      compliance purposes(policy) and threat management. SIM was designed to collect, report and prematurely analyze logs (it was not created to provide sophisticated methods of correlation or any). Logs/flows can be taken from any network/security devices but also from servers, databases and applications. Additionally , more often SIEMs are supporting variety of network appliances making the connections more effective and flexible.

Security event management (SEM): SEM is – or was – responsible for aggregating, correlating and analyzing logs for real time needs. It is crucial for incident response, clear and wide visibility into infrastructure and reporting.

During some lectures I was trying to figure out which type of data warehouse SIEM is – if SIEM is a warehouse at all.  Looking for words combination ‘warehouse SIEM’ I found some articles:

…approaches SIEM as a data management issue. …solution is built atop an event data warehouse that leverages a columnar database uniquely designed for time-stamped, unstructured data to be correlated and analyzed,  either in real time or over months or even years. The solution runs on a platform employing massively parallel processing and commodity hardware and storage for  enhanced load and query throughput, very high data compression (…)  and  extremely large data retention (up to petabytes a year). It provides the ability to correlate events across multiple data  sources and systems, either in real-time or on a historical basis.

Taken from : http://www.sensage.com. It seems that SIEM can be some kind of hybrid between OLAP and OLTP warehouses or even a modification of mediatory system – easy access to real time data (structured, parsed and correlated) but also giving information about trends, statistics and other historical patterns (which are capabilities of standard database warehouses). But before moving forward, let look at what ‘standard’ features provide SIEM:

Log management : supports effective correlation (indexing), collection. There are deployed mechanism for fast and efficient searches.  System should also support reporting and capability to attach external data sources – not only those standard (for example syslog or more exclusive opsec).

Compliance/event managment : allows for signatures creation (behavioral anomalies rules also) . What is more, it supports incident handling and documentation.

Visibility and dashboarding : SIEM provides clear dashboards to view system availability and performance (ideal situation) . Alert notificatiosn and trends can be viewed in easy format.

Log source management: system allows for maintenance, tuning, editing and additional extensions (third party software as example).

External data sources/feed: SIEM should be accessible and aggregate also non-standard information taken from outside of infrastructure.

Moving back to warehouses. Basing on this article http://www.networkworld.com  ,we are finding the definition for Security Data Warehouse, which is:

(…)  making security decisions based on mining business intelligence and combining it with security-related event data from security devices.
Today, security analysis more typically relies on what's known as security information and event management (SIEM) tools which can aggregate security and other technical information for a birds-eve view of network activity or detect possible unauthorized actions. (…) but it's now possible to go further through correlation of business activities, based on feeds from other sources too.
 A SIEM may have trouble "dealing with massive amounts of historical data," (…) but by using the Hadoop framework with core components that can handle "terabytes, even petabytes of information," it's possible to achieve better analysis by combining business and security data. " A SIEM becomes one main feed into the Security Data Warehouse. Improved historical analysis is also resulting," (…)

 The conclusion may be: SIEM is the sub-system for bigger mechanism that can provide predictive analytics and possibly track anomalies. Basing on that hypothesis SIEM alone is not typical data warehouse as it provides real time information and trends (rrd), collects information for compliance regulatory but very limited in terms of post-factum response or data mining. To some point it is a hybrid warehouse giving ideas (and even limited capabilities) of big data approach (statistics, historical patterns, rrd dashboards .etc)  but possibly to weak to compete with the typical one. SIEM is a main component in Security data warehouse that is now in developmental and evolutionary status - Big data.

(…) Security Data Warehouse approach is making it more possible to detect phishing attempts by analyzing email and other events, "and that allows you to respond more quickly than in the past."

I would like to stop here, and strongly  encourage you to read following materials.

http://www.sans.org/reading_room/

http://www.sans.org/score/

http://www.hpenterprisesecurity.com/collateral/report/GartnerMQ_SIEM_2011.pdf

http://csrc.nist.gov/publications/nistpubs/800-92/SP800-92.pdf - Guide to Computer Security Log Management (Recommendations of the National Institute of Standards and Technology)