CONDITIONAL VARIATIONAL AUTOENCODER (CVAE) APPROACH FOR EARLY DETECTION OF UNKNOWN MALWARE ATTACKS

Zero day malware attacks can be referred to as malicious software that exploits vulnerabilities unknown to vendors or security products which pose a severe threat to modern computing environments because they bypass traditional signature based antivirus solutions. This paper presents an unsupervised deep learning model for detecting unknown malware at an early stage using a Conditional Variational Autoencoder (CVAE). Unlike standard autoencoders, the proposed CVAE incorporates conditioning variables such as section entropy, import table size, and resource characteristics into both the encoder and decoder, enabling context aware anomaly detection. The model is trained exclusively on benign Windows Portable Executable (PE) files from the publicly available Malware Dataset (blackarcher/malware dataset) with a balanced collection of 100,000 examples (50,000 benign, 50,000 malware). During training and threshold calibration no malicious samples are exposed, adhering to standard unsupervised anomaly detection principles. Detection is based on reconstruction error: benign files are reconstructed accurately, while unknown malware produces a large error. At the optimal operating point (2% false positive rate), the model achieves a recall of 90.5%, precision of 83.8%, an F1 score of 0.870, and an overall AUC PR of 0.928. The separation ratio between benign and malware reconstruction errors improves from 2.85 (unconditioned VAE) to 7.92 (full CVAE). Per malware analysis shows consistently high detection (0.87-0.93) for different malware families, with slightly lower results for packed or obfuscated samples (0.85). False positives (2.1% overall) occur mainly on benign files with unusual structural properties (atypical section counts or high entropy in legitimate packers). Latent space visualisation reveals benign files forming tight, family aware clusters, while unknown malware maps to out of distribution regions. These results demonstrate that a CVAE trained solely on benign file features provides a powerful, unsupervised, context aware approach for detecting unknown malware without requiring labelled attack samples.