November – 2011 ^December

Detection and containment of unknown malware are challenging tasks. Typically the detection is performed by experts who use anomaly detection systems or Honeypots-based systems. Such a detection process is very slow and it is not suited for detection of rapidly propagating threats such as worms. In this research we propose to automate the detection process by introducing an innovative distributed framework for detection and containment of new malware. The framework consists of distributed agents that are installed in several client computers and a Centralized Decision Maker module (CDM) that interacts with the agents. The new detection process is performed in two phases. In the first phase agents detect potential malware on local machines and send their detection results to the CDM. In the second phase, the CDM builds a propagation graph for every potential malware. These propagation graphs are compared to known malware propagation characteristics in order to determine whether the potential malware is indeed a malware. All the agents are notified of a final decision in order to start the containment process. Another contribution of this study is a method for detecting new malicious executables locally. It is based on monitoring run-time system calls and comprises the following steps: (a) in an offline training phase, finding a set of (not necessary consecutive) system call sequences that are characteristic only to malicious files, when such malicious files are executed, and storing said sequences in a database; (b) in a real time detection phase, for each running executable, continuously monitoring its issued system calls and comparing them with the stored sequences of system calls within the database to determine whether there exists a match between a portion of the sequence of the run-time system calls and one or more of the database sequences, and when such a match is found, declaring said executable as malicious. In addition to the collaborative detection, the method can be used for independent (local) malware detection, replacing (or in addition to) traditional antivirus software.