Abstract
Machine-learning based Intrusion Detection and Prevention Systems provide significant value to organizations because they can efficiently detect previously unseen variations of known threats, new threats related to known malware or even zero-day malware, unrelated to any other known threats. However, while such systems prove invaluable to security personnel, researchers have observed that data subject to inspection by behavioral analysis can be perturbed in order to evade detection. We investigated the use of adversarial techniques for adapting the communication patterns between botnet malware and control unit in order to evaluate the robustness of an existing Network Behavioral Analysis solution. We implemented a packet parser that let us extract and edit certain properties of network flows and automated an approach for conducting a grey-box testing scheme of Stratosphere Linux IPS (Slips). As part of our implementation, we provided several techniques for providing perturbation to network flow parameters, including a Simultaneous Perturbation Stochastic Approximation (SPSA) method, which was able to produce sufficiently perturbed network flow patterns while adhering to an underlying objective function. Our results showed that network flow parameters could indeed be perturbed, which ultimately enabled evasion of intrusion detection based on the detection models that were available for our IDS. Additionally, we demonstrated that it was possible to combine evading detection with stochastic techniques for optimization problems, effectively enabling adaptive network flow behavior.