Background The detection of malware in network traffic remains a critical cybersecurity challenge. Traditional signature-based intrusion detection demonstrates a high level of familiarity with issues that have been recorded in the database; but show significantly lower effectiveness when it comes to polymorphic or zero-day attacks. Conversely, anomaly-based approaches are also endowed with the ability to detect new incursions, but often have a high false-positive rate. Methods This study proposes a combined malware-detection framework which makes use of RNA encoding network-flow attributes alongside Convolutional Neural Network (CNN) classifiers. The framework has three functionalities: a Signature-CNN, which is trained on RNA-encoded representation of known malicious flows; an Anomaly-CNN, which is developed to distinguish between benign and malicious traffic without any signature prior knowledge; and a Hybrid-CNN, which combines both paradigms in a two-stage detection pipeline. Results The research is carried out on the 10,000 samples that are split into training and testing subsets based on the 70/30 split strategy. The given model is trained in the context of a supervised learning model and assessed in terms of common performance metrics, such as accuracy, precision, recall, and F1-score. The experimental design is written in Python and deep learning libraries, so that the evaluation environment of all experiments is consistent and reproducible. Experiments conducted on the Malicious Network Dataset show that the Signature-CNN achieves 91% accuracy with strong precision on known threats, the Anomaly-CNN achieves 93% detection rate on unknown malware, and the Hybrid-CNN achieves the best overall performance with 95% detection rate and 94.5% F1 score. Conclusions The results demonstrate that RNA encoding combined with CNN classifiers offers a robust and scalable solution for malware detection in networked environments.
