Detection of malicious DoH traffic using autoencoders

Abstract

Domain Name System (DNS) is the phonebook of the Internet, translating domain names to IP addresses of Internet servers hosting information user wants to find online. DNS is unencrypted, revealing user’s browsing habits to Internet Service Providers. To protect users’ privacy, most major browsers replaced DNS with DNS-over- HTTPS (DoH), an encrypted form of DNS. Unfortunately, DoH can be exploited by botnets to communicate with Command-and-Control center. The reason is that they can use DoH to bypass traditional detection techniques that rely on the unencrypted DNS traffic. Therefore, this research aims to design a method to detect botnet activity that uses DoH, while protecting users’ privacy. To detect DoH traffic from botnet malware. we propose to use autoencoders, a form of deep neural network. The main idea of our detection approach is to use the autoencoder recreation error to find the malicious DoH traffic. Specifically, an autoencoder aims to recreate the input network traffic on its output. By training the autoencoder to recreate both benign and malicious network traffic, we can obtain a 3D visualization, called embeddings. . We then use K-means clustering to find the clusters of benign and malicious traffic in the 3D-space. We observe that the classification precision is 91.38 %, accuracy is 89.31 %, recall is 87.10 %, and f-score is 89.19 %. In summary, the autoencoder provides a 3D representation using known labels for packet flows. K-means performs the detection of whether the traffic is legitimate or malicious. This yielded highly accurate and reliable classification results.