IJCNC 08

A Systematic Review of Obfuscated Malware Detection: From Traditional Analysis to Deep Learning

Mohammed bin Shamlan 1, Mohammed Fadhl Abdullah 1, Khaled Hassan Balhaf 1, Ahmed Saleh Khaled 1, Makarem Mohamed Bamatraf 2
1 Faculty of computing and engineering, university of science and technology, Aden, Yemen.
2 Computer Engineering Department, College of Engineering, Hadhramaut University, Yemen

ABSTRACT

Obfuscation has been increasingly difficult in the subject of cybersecurity, since malware developers use it to change code appearance without changing its malicious behavior. As a result, signature-based and basic heuristic detection systems are easily bypassed by these techniques. This article reviews recent and ongoing research in the analysis and detection of obfuscated malware, giving special attention to methods that were recently developed to address this problem. The reviewed methods are divided into five major classes: static analysis, dynamic analysis, hybrid analysis, machine learning, and deep learning. thirty-six recent research papers from 2018 to 2025 are analyzed, with a detailed summary of each, including merits and demerits. The review is intended to generate a broad picture of the research field, point out strengths and weaknesses in each category, and identify the way forward, especially for the area of hybrid and deep learning-oriented memory analysis.

KEYWORDS

Obfuscated Malware, Static and Dynamic Analysis, Malware Detection, Memory Analysis, Cybersecurity, Explainable Artificial Intelligence.

1. INTRODUCTION

Obfuscation means a continuous and growing challenge in the cyberspace spectrum-for malware developers employ it to thoroughly change the appearance of code while still maintaining its malicious essence, thus evading traditional signature-based or simple heuristic detection systems. The technique intends to impede the reverse engineering and analysis of the executables, and

Figure 1: A unified taxonomy of obfuscated malware detection techniques.

AUTHORS