a,∗a a Kevin W. Hamlen , Vishwath Mohan , Mohammad M. Masud , a a Latifur Khan , Bhavani Thuraisingham
a Computer Science Department, University of Texas at Dallas, 800 W. Campbell Rd., Richardson, Texas 75080, USA
Abstract We propose a technique for defeating signaturebased malware detectors by ex ploiting information disclosed by antivirus interfaces. This information is lever aged to reverse engineer relevant details of the detector’s underlying signature database, revealing binary obfuscations that suffice to conceal malware from the detector. Experiments with real malware and antivirus interfaces on Windows operating systems justifies the effectiveness of our approach. Key words:Security, Signaturebased malware detection, Data mining, Binary obfuscation
1. Introduction
Traditionalsignaturebasedmalware detectors identify malware by scanning untrusted binaries for distinguishing byte sequences orfeaturesunique. Features to malware are maintained in asignature database, which must be continually updated as new malware is discovered and analyzed. Signaturebased malware detection generally enforces a static approximation of some desired dynamic (i.e., behavioral) security policy. For example, access control policies, such as those that prohibit code injections into operating system executables, are statically undecidable and can therefore only be approximated by any purely static decision procedure such as signaturematching. A signature based malwaredetector approximates these policies by identifying syntactic fea tures that tend to appear only in binaries that exhibit policyviolating behavior when executed. This approximation is both unsound and incomplete in that it is susceptible to both false positive and false negative classifications of some binaries. For this reason signature databases are typically kept confidential, since they contain information that an attacker could use to craft malware that
∗ Corresponding author Email addresses:hamlen@utdallas.edu(Kevin W. Hamlen), vishwath.mohan@utdallas.edu(Vishwath Mohan),mehedy@utdallas.edu (Mohammad M. Masud),lkhan@utdallas.edu(Latifur Khan), bhavani.thuraisingham@utdallas.edu(Bhavani Thuraisingham)