Infrastructure Availability Monitoring

Here, just several words about physical infrastructure monitoring – somehow about one of  the component of CIA triad – availability. It is not new or surprising  that IT guys try to make their jobs easier by applying – to some degree - some automatization. Manual  log revision,  or  checking all servers one by one can be tiring and time consuming task.  Before making any statements let have a look why the monitoring is so important.

Why it is important to monitor physical infrastructure? On in other words, why availability monitoring is so crucial? First – the most prosaic one – because administrators want to know that their servers are working and performing tasks. In fact this is the most important argument – and the most difficult. Secondly, to check servers resiliency. What is the system load, what is the performance and how the servers can be used to increase efficiency. This also refers to the problem of virtualization and costs – which at the same time are additional arguments. One may say, that compliance/ regulations are arguments for availability monitoring – nobody wants their data to be lost due to downtime on machines (software and hardware). Security aspect – checking how the system is behaving, and looking for anomalies may be the indicator of DoS attack or any IT Sabotage going on. The last one, is testing purposes – it is sometimes nice to have  estimation how the particular appliance/software is working, if it is resource-demanding and so on.

Why it is important to monitor physical infrastructure?

-  servers resiliency (load distribution, load balancers)

-  virtualization (still performance issue)

- security (anomalies, downtime)

- regulations (data loss)

- costs

- development

There are of course tons of good software (open-source and commercial) that are designed to monitor different factors of our machines. Those are: load average, CPU, disk capacity, IN/OUT on multiple interfaces and others. Many of tools use round-robin databases (additionally use simple in configuration SNMP)  which is great option when there is no enough time for maintenance – such applications just work. It is of course possible to use more sophisticated scripts (using ssh for communication) which can report what is going on with particular processes on the remote server. And here starts interesting discussion. How far should we push administrators for monitoring? Should we monitor every working process on servers, or only the most important ones? 

The work flow is simply. We ‘add’ remote machines to our monitoring server, and present performance and other factors as simple graphs (aggregated!) , with colors or even sound warnings. The worst idea is to view tables with numbers, but it can also be applied, may be with additional visualizations or meaningful color (red/green?). To such solution it would be beneficial to add  some threshold alerts (moving average, moving average with dynamic deviation) and configure emails for notice/ warning and alert levels.  In my humble opinion, this is still not enough.

I would like to make here, a one step back. Imagine that you have really important process working on your remote machines. This process is responsible for collecting data or anything critical for your service. Due to performance issues, or unexpected bug in application (such bugs tend to reveal only when the system is overloaded, and the ‘catch-up’ behavior can be spotted) the ‘really important’ process stopped and your graphs do not show the problem! The solutions in this case is easy – why the machine was overloaded before, and why nobody solved it? What is the point of ‘monitoring the monitoring tool’ when nobody looks  and care for warnings? Human factor is crucial, and even the best availability monitoring tools cannot replace a man.

Combination of  dashboards, warning system, daily checks and human consciousness is the best option when dealing with infrastructure monitoring. 

Malware Forensics with Cuckoo

When it comes to malware forensics, it appears that live response is the best option. Imagine that you have infection -problem with one family of malware on multiple workstations -  miles away. The only way you can reach those hosts  is remote connections via multiple different technologies agent-less  - it would be great to have some servlet, agent installed -  or have prepared team on-site (this rarely happen).  Furthermore, when contacting user (whose workstation is infected), you should have some policy with you, clearly explain what is going on, and what do you want. This is not always an easy task. Moving step away from such discussion, it seems that the only information needed is memory – dump. It is always better option to get this first, without ‘old-shool’ approach – which tend to be more aggressive and evidence destructive.  Of course, when you have Encase, or combo X-Ways + F-Response, the task is much, much easier.  But here, I do not want to be ‘vendor-driven’ and just simply discuss the problem of memory-dump analysis.

After memory is preserved in a forensically sound manner, employ a strategy and associated methods to extract the maximum amount of information relating to the malware incident.

There are – literally – tons of specialized memory forensic tools for multi-layer interpretation, and multi-vector analysis. What is more very often those tools are specialized in memory carving, debugging, or disassembly. On the other hand there are many information in the dump that can be easily extracted using some old-school approaches. The first one is ‘strings’. This command can reveal variety of evidence, such as : URLs, fragments of files, web pages, commands run in the cli, IPs payloads, prefetch .etc.  This is always the first thing that – I suppose – should be done. It confirms or not, whether there are multiple hypothesis on the begging of analysis. (I am assuming here, that we have NBE stored, with accompaniment from SIEM). As I remember the first step – Encase – is to extract hidden as much information as it is possible from raw dump. This can be done with success with ‘foremost’ or ‘scalpel’.  Those file carving tools can give us executables, .pdf, and other types of files for further analysis. In this stage we have : file with readable text extracted from dump, and files reconstructed from the same RAM file. In my opinion, only reviewing this kind information is a waste of time. Of course, it gives great context of situation, but this is it.. When we have a really hard sample of memory, with hidden malware, I am sure that is would be very difficult just to check manually some hypothesis and be successful. I would like here to point great methodology that can be applied here, before moving with analysis on specialized forensic tools. After file carving procedure we are facing a problem of analysis. Here goes my question, is it better to analyze dump with forensics tools, looking for hypothesis and clues  and then extract some suspicious looking object for further examination,  or firstly use automated system to examine all extracted processes and objects? The first approach will end on static  analysis of extracted suspicious object. I think Farmer/Venema introduced nice categorization:

There are many ways to study a program's behavior. With static analysis, we study a program without actually executing it. Tools of the trade are disassemblers, decompilers, source code analyzers, and even such basic utilities as strings and grep.

With dynamic analysis, we study a program as it executes. Here, tools of the trade are debuggers, function call tracers, machine emulators, logic analyzers, and network sniffers. The advantage of dynamic analysis is that it can be fast and accurate. However, dynamic analysis has the disadvantage that "what you see is all you get".

A special case is "black box" dynamic analysis, where a system is studied without knowledge about its internals. The only observables are the external inputs, outputs, and their timing relationships.

Finally, there is post-mortem analysis, the study of program behavior by looking at the after effects of execution.

Taken from: http://www.porcupine.org/forensics/forensic-discovery/chapter6.html, a really good book by the way.

While static analysis is understood, I would like to discuss the problem of dynamic examination. Here it should be mentioned that there are two basic method to analyze unknown binary. Those are : program confinement with hard virtual machines and program confinement with soft virtual machines. First example of the soft environment is ReVirt :

ReVirt system allows an investigator to replay an "incident", and to rewind, pause or fast-forward the virtual machine at any point in time. This is possible because the ReVirt virtual monitor records all interrupts and external inputs including keystrokes and network packet contents. This information, combined with a complete record of the initial file system state, allows an investigator to replay every machine instruction and to view data before, while and after it is modified.

Taken from : http://static.usenix.org/event/osdi02/tech/full_papers/dunlap/dunlap.pdf , for more about ReVirt please follow : http://www.docstoc.com/docs/91511349/ReVirt.

Another example of system designed for dynamic analysis is cuckoo. Cuckoo is a sandbox and can be explained using words taken form documentation (http://www.cuckoosandbox.org/):

As defined by Wikipedia, “in computer security, a sandbox is a security mechanism for separating running programs. It is often used to execute untested code, or untrusted programs from unverified third-parties, suppliers, untrusted users and untrusted websites.”. This concept applies to malware analysis’ sandboxing too: our goal is to run an unknown and untrusted application or file inside an isolated environment and get information and what it does. Malware sandboxing is a practical application of the dynamical analysis approach: instead of statically analyze the binary file, it gets executed and monitored in real-time.

 Installation can be tricky when attempting for the first time, for help please follow

http://arisri.tistory.com/entry/Cuckoo-Installing-Cuckoo-Sandbox-on-Ubuntu-1204-LTS-for-Malware-Analysis-Leave-a-comment,and  http://www.edwiget.name/2012/07/installing-cuckoo-sandbox-in-backtrack-for-malware-analysis/. Great cuckoo work-flow graph can be found :

from :  http://blog.michaelboman.org/2012/06/mart-malware-analyst-research-toolkit_25.html

Another awesome solution introduced by Lenny Zeltser (http://zeltser.com/remnux/:

REMnux incorporates a number of tools for analyzing malicious software that runs on Microsoft Windows, as well as browser-based malware, such as Flash programs and obfuscated JavaScript. The toolkit includes programs for analyzing malicious documents, such PDF files, and utilities for reverse-engineering malware through memory forensics. 

 
Step back. We have used some tools for memory carving and extracted multiple binaries. Now, this is the moment for automated job.  Cuckoo will be awesome only when prepared with plan. Multiple virtual machines should be prepared (with different version of software) , everything up-to-date and ready for malware object submission. All extracted files can be executed in prepared environment -  during analysis ‘agent’ will be capturing any information and then report what was going on test machine :  traffic, registry changes, files, new jobs and processes, any activity that was spotted during unknown object execution.  Having such results we can move on with static dynamic, or using forensic tool for confirming our hypothesis and look for other clues. On the other hand, the process of verification can be done after extracting suspicious files during ‘standard approach’ (without checking everything in virtual machine). For this purposes, you do not have to prepare environment and hard process of installation, instead go to http://malwr.com/ submit you file and wait for analysis! This web page uses cuckoo engine for analysis (check also https://www.virustotal.com/) .

What is the limit of malware forensics, and how far investigators are pushed to reveal the true hidden in extracted object? Any digital information found during investigation can be evidence, and only with context collected during previous steps (acquisition, ‘old-school’ approach, NBE, and clues found in memory with forensics tools) can be clearly and fully understood.  Simply, there are no limits. Stay focused.