The techniques employed by viruses to avoid detection by antivirus scanners are becoming increasingly advanced. One technique commonly used by viruses to evade detection is polymorphism. The level of polymorphism in a virus indicates its ability to create different forms of itself. The use of junk instructions is a common technique to increase the level of polymorphism in a virus. Junk instructions are machine code instructions with no other function than to alter the appearance of a virus. Junk instructions do not contribute to the function of the virus, only the form.
This master thesis focuses on the problem of separating junk instructions from nonjunk instructions in computer viruses. To assail the problem, a junk instruction detection (JID) framework has been developed, capable of detecting junk in viruses created for the Intel IA-32 Architecture R (x86). JID relies on the static instruction information produced by a disassembler. Static instruction information describes the static information found in an instruction: the number of input/output operands and their accesses. Because JID only depends on the static instruction information, JID can possible be ported to other processor architectures. As long as there exist a disassembler for the architecture capable of producing static instruction information, JID is portable.
The results of testing JID on polymorphic viruses are promising. Tests show that JID manages to detect junk instructions found in the polymorphic virus Zmist. It is believed that JID would work as a tool to detect and remove junk instructions from future polymorphic viruses, thus reducing the time spent analysing the virus. Additionally in this thesis, the virus Zmist is analysed as a case study, and a detection algorithm devised. The detection algorithm manages to detect 100% of the polymorphic version of Zmist, although shortcomings in the disassembler algorithm reduces this number to 94%.