Enhancing Network Security

A Review of Machine Learning Techniques for Detecting TCP SYN Flood Attacks

Authors

DOI:

https://doi.org/10.14500/aro.12210

Keywords:

Anomaly detection, Distributed denial of service, Deep learning, Machine learning, Network security, Transmission control protocol SYN flood

Abstract

Distributed denial of service (DDoS) attacks are a significant danger to network security, with SYN flood assaults being particularly known for exploiting the transmission control protocol (TCP) handshake to deplete server resources. This review paper analyzes the current research on classifying DDoS attacks using machine learning (ML) approaches, with a focus on SYN f lood scenarios. Traditional algorithms such as XGBoost, Random Forest, and k-Nearest Neighbors are examined alongside modern deep learning methods such as convolutional neural networks and long short-term memory networks. Deep learning, noted for its capacity to automatically learn complex properties from data, is particularly effective in dynamic contexts like the internet of things. The review analyzes the usefulness of various strategies, obstacles in feature engineering and model training, and their implications for real-time detection. This study presents a comprehensive overview of the accomplishments in employing ML and deep learning for TCP SYN flood attack classification and exposes gaps in the field that indicate options for further research.

Downloads

Download data is not yet available.

References

Aggarwal, S., Behera, B., Singh, M.K., and Sharma, A.K., 2025. Optimizing DDoS Attack Detection Using Machine Learning. In: 2025 2nd International Conference on Computational Intelligence, Communication Technology and Networking (CICTN), pp.245-250.

Aighuraibawi, A.H.B., Manickam, S., Abdullah, R., Alyasseri, Z.A.A., Jasim, H.M., and Sani, N.S., 2023. Modified Flower Pollination Algorithm for ICMPv6-Based DDoS Attacks Anomaly Detection. In: Procedia Computer Science. Elsevier B.V., Netherlands, pp.776-781.

Alasadi, S.A., Manaa, M.E., Hussain, S.M., and Al-Khamees, H.A.A., 2024. DDoS attacks detection based on machine learning algorithms in IoT environments. Inteligencia Artificial Revista Iberoamericana de Inteligencia Artificia, 27, pp.152-165.

Ali, T.E., Chong, Y.W. and Manickam, S., 2023. Machine learning techniques to detect a DDoS attack in SDN: A systematic review. Applied Sciences, 13(5), p.3183.

Bamasag, O., Alsaeedi, A., Munshi, A., Alghazzawi, D., Alshehri, S., and Jamjoom, A., 2022. Real-time DDoS flood attack monitoring and detection (RT-AMD) model for cloud computing. PeerJ Computer Science, 7, p.e814.

Bawany, N.Z., Shamsi, J.A., and Salah, K., 2017. DDoS attack detection and mitigation using SDN: Methods, practices, and solutions. Arabian Journal for Science and Engineering, 42, pp.425-441.

Bensaid, R., Labraoui, N., Abba Ari, A.A., Maglaras, L., Saidi, H., Abdu Lwahhab, A.M., and Benfriha, S., 2024. Toward a real-time TCP SYN

flood DDoS mitigation using adaptive neuro-fuzzy classifier and SDN assistance in fog computing. Security and Communication Networks, 2024(1), p.6651584.

Bhayo, J., Shah, S.A., Hameed, S., Ahmed, A., Nasir, J., and Draheim, D., 2023. Towards a machine learning-based framework for DDOS attack detection in software-defined IoT (SD-IoT) networks. Engineering Applications of Artificial Intelligence, 123, p.106432.

Bhutani, P., and Dash, C.S., 2024. Deep learning approaches for SYN flood detection in internet service providers network. International Journal of Innovative Research in Engineering and Management, 11(4), pp.86-94.

Bovenzi, G., Di Monda, D., Montieri, A., Persico, V., and Pescapè, A., 2024. Classifying attack traffic in IoT environments via few-shot learning. Journal of Information Security and Applications, 83, p.103762.

Cai, T., Jia, T., Adepu, S., Li, Y., and Yang, Z., 2023. ADAM: An adaptive DDoS attack mitigation scheme in software-defined cyber-physical system. IEEE Transactions on Industrial Informatics, 19(6), pp.7802-7813.

Chandana Swathi, G., Kishor Kumar, G., and Siva Kumar, A.P., 2024. Ensemble classification to predict botnet and its impact on IoT networks. Measurement: Sensors, 33, p.101130.

Chicco, D., and Jurman, G., 2020. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21, p.6.

Cui, J., Wang, M., Luo, Y., and Zhong, H., 2019. DDoS detection and defense mechanism based on cognitive-inspired computing in SDN. Future Generation Computer Systems, 97, pp.275-283.

Das, T., Hamdan, O.A., Sengupta, S., and Arslan, E., 2022. Flood control: TCP-SYN Flood Detection for Software-Defined Networks using OpenFlow Port Statistics. In: 2022 IEEE International Conference on Cyber Security and Resilience (CSR). pp.1-8.

Dasari, K.B., and Devarakonda, N., 2022. Detection of DDoS attacks using machine learning classification algorithms. International Journal of Computer Network and Information Security, 6, pp.89-97.

Dasari, S., and Kaluri, R., 2024. An effective classification of DDoS attacks in a distributed network by adopting hierarchical machine learning and hyperparameters optimization techniques. IEEE Access, 12, pp.10834-10845.

Dash, S.K., Dash, S., Mahapatra, S., Mohanty, S.N., Khan, M.I., Medani, M., Abdullaev, S., and Gupta, M., 2024. Enhancing DDoS attack detection in IoT using PCA. Egyptian Informatics Journal, 25, p.100450.

Deb, D., Rodrigo, H., and Kumar, S., 2024. Performance Analysis of Machine Learning Algorithms on Imbalanced DDoS Attack Dataset. In: 2024 IEEE World AI IoT Congress (AIIoT). pp.349-355.

Deshmukh, R.V., and Devadkar, K.K., 2015. Understanding DDoS attack & its effect in cloud environment. Procedia Computer Science, 49, pp.202-210.

Dimolianis, M., Kalogeras, D.K., Kostopoulos, N., and Maglaris, V., 2022. DDoS Attack Detection via Privacy-aware Federated Learning and Collaborative Mitigation in Multi-domain Cyber Infrastructures. In: 2022 IEEE 11th International Conference on Cloud Networking (CloudNet). pp.118-125.

classification and mitigation for DDoS attacks using programmable network data planes. IEEE Access, 9, pp.113061-113076.

Dimolianis, M., Pavlidis, A., and Maglaris, V., 2021b. SYN Flood Attack Detection and Mitigation using Machine Learning Traffic Classification and Programmable Data Plane Filtering. In: 2021 24th Conference on Innovation in Clouds, Internet and Networks and Workshops, ICIN 2021. Institute of Electrical and Electronics Engineers Inc., pp.126-133.

Doshi, R., Apthorpe, N., and Feamster, N., 2018. Machine Learning DDoS Detection for Consumer Internet of Things Devices. In: Proceedings - 2018 IEEE Symposium on Security and Privacy Workshops, SPW 2018. Institute of Electrical and Electronics Engineers Inc. pp.29-35.

Echeverría, A.D., Pinilla, M.A., and Mora, H.R.C., 2024. Securing the IoT: An In-Depth Analysis of Ubuntu Core Hardening Measures Using CIS LTS Guide. In: 2024 4th Interdisciplinary Conference on Electrics and Computer (INTCEC). pp.1-8.

Feng, Y., Zhang, W., Yin, S., Tang, H., Xiang, Y., and Zhang, Y., 2023. A collaborative stealthy DDoS detection method based on reinforcement learning at the edge of internet of things. IEEE Internet of Things Journal, 10(20), pp.17934-17948.

Gaurav, A., Zhou, Z., Tai Chui, K., Colace, F., Chaurasia, P., and Hsu, C.H., 2021. A Novel Approach for DDoS Attack Detection Using Big Data and Machine Learning. In: CEUR Workshop Proceedings.Ghafoor, K.Z., 2022. Social bot detection using machine learning algorithms: A survey and research challenges. Polytechnic Journal, 12(2), pp.219-228.

Haider, S., 2020. A deep CNN ensemble framework for efficient DDoS attack detection in software defined networks. IEEE Access, 8, pp.53972-53983.

Hamad, Z.O., 2022. Review of feature selection methods using optimization algorithm (Review paper for optimization algorithm). Polytechnic Journal, 12(2), pp.203-214.

Hassan, S.K.H., and Daneshwar, M.A., 2022. Anomaly-based network intrusion detection system using deep intelligent technique. Polytechnic Journal, 12(2), pp.100-113.

Hasan, M.K., Habib, A.A., Islam, S., Safie, N., Abdullah, S.N.H.S. and Pandey, B., 2023. DDoS: Distributed denial of service attack in communication standard vulnerabilities in smart grid applications and cyber security with recent developments. Energy Reports, 9, pp.1318-1326.

Hong, K., Kim, Y., Choi, H., and Park, J., 2017. SDN-assisted slow HTTP DDoS attack defense method. IEEE Communications Letters, 22, 688-691.

Hoque, N., Kashyap, H., and Bhattacharyya, D.K., 2017. Real-time DDoS attack detection using FPGA. Computer Communications, 110, pp.48-58.

Hossain, M.A., and Islam, M.S., 2024. Enhancing DDoS attack detection with hybrid feature selection and ensemble-based classifier: A promising solution for robust cybersecurity. Measurement: Sensors, 32, 101037.

Hsu, F.H., Lee, C.H., Wang, C.Y., Hung, R.Y., and Zhuang, Y., 2021. DDoS flood and destination service changing sensor. Sensors (Basel), 21, p.1980.

Hussain, K., Syed Jawad, H., Veena, D., Muhammad, N., and Muhammad Awai, A., 2016. An adaptive SYN flooding attack mitigation in DDOS environment. International Journal of Computer Science and Network Security, 16, pp.27-33.

Hwang, R.H., 2020. An unsupervised deep learning model for early network traffic anomaly detection. IEEE Access, 8, pp.30387-30399.

Ismail, Mohmand, M.I., Hussain, H., Khan, A.A., Ullah, U., Zakarya, M., Ahmed, A., Raza, M., Rahman, I.U., and Haleem, M., 2022. A machine learning-based classification and prediction technique for DDoS attacks. IEEE Access, 10, pp.21443-21454.

Jaafar, G.A., Abdullah, S.M., and Ismail, S., 2019. Review of recent detection methods for HTTP DDoS attack. Journal of Computer Networks and Communications, 2019, p.1283472.

Jaraba, F., Mahajan, G., Jani, J., Ipu, R., and Butakov, S., 2024. Exploring Dimolianis, M., Pavlidis, A., and Maglaris, V., 2021a. Signature-based traffic current solutions against DDoS attacks in SDN environment. Procedia Computer Science, 238, pp.127-134.

Javadpour, A., 2020. Providing a way to create balance between reliability and delays in SDN networks by using the appropriate placement of controllers. Wireless Personal Communications, 110, pp.1057-1071.

Javadpour, A., and Wang, G., 2022. cTMvSDN: Improving resource management using combination of Markov-process and TDMA in software-defined networking. Journal of Supercomputing, 78, pp.3477-3499.

Javadpour, A., Wang, G., and Rezaei, S., 2020. Resource management in a peer-to-peer cloud network for IoT. Wireless Personal Communications, 115, pp.2471-2488.

Javanmardi, S., Ghahramani, M., Shojafar, M., Alazab, M., and Caruso, A.M., 2024. M-RL: A mobility and impersonation-aware IDS for DDoS UDP flooding attacks in IoT-Fog networks. Computers and Security, 140, p.103778.

Jr, E.P.F., Tavares, A.C.J., and Nogueira, M., 2023. A Runtime DDoS Attack Detection Technique Based on Stochastic Mathematical Model. In: 2023 IEEE Latin-American Conference on Communications (LATINCOM). pp.1-6.

Kanimozhi, S., and Radhika, D., 2022. Detection of DDos attack using machine learning algorithms in cloud computing. Turkish Online Journal of Qualitative Inquiry, 13 (1), pp.2079-2088.

Kellerer, W., Schembra, G., Hwang, J., Kamiyama, N., Kang, J.M., Martini, B., Pasquini, R., Pezaros, D., Zhang, H., Zhani, M.F., and Zinner, T., 2021. Guest EDITORS Introduction: Special issue on advanced management of softwarized networks. IEEE Transactions on Network and Service Management, 18(1), pp.20-29.

Kim, Y., Hakak, S., and Ghorbani, A., 2024. Detecting distributed denial-of service (DDoS) attacks that generate false authentications on Electric Vehicle (EV) charging infrastructure. Computers and Security, 144, p.103989.

Kreutz, D., Ramos, F.M., Verissimo, P.E., Rothenberg, C.E., Azodolmolky, S., and Uhlig, S., 2014. Software-defined networking: A comprehensive survey. Proceedings of the IEEE, 103, pp.14-76.

Kumari, P., and Jain, A.K., 2023. A comprehensive study of DDoS attacks over IoT network and their countermeasures. Computers and Security, 127, p.103096.

Liu, Z., Wang, Y., Feng, F., Liu, Y., Li, Z., and Shan, Y., 2023. A DDoS detection method based on feature engineering and machine learning in software-defined networks. Sensors (Basel), 23, p.6176.

Magnani, S., Doriguzzi-Corin, R., and Siracusa, D., 2023. Enhancing Network Intrusion Detection: An Online Methodology for Performance Analysis. In: 2023 IEEE 9th International Conference on Network Softwarization (NetSoft). pp.510-515.

Meng, B., Andi, W., Jian, X., and Fucai, Z., 2017. DDOS Attack Detection System Based on Analysis of Users’ Behaviors for Application Layer. In: Proceedings - 2017 IEEE International Conference on Computational Science and Engineering and IEEE/IFIP International Conference on Embedded and Ubiquitous Computing, CSE and EUC 2017. Institute of Electrical and Electronics Engineers Inc., pp.596-599.

Mirmohseni, S.M., Tang, C., and Javadpour, A., 2020. Using Markov learning utilization model for resource allocation in cloud of thing network. Wireless Personal Communications, 115, pp.1-25.

Mohammadi, R., Javidan, R., and Conti, M., 2017. Slicots: An SDN-based lightweight countermeasure for TCPSYN flooding attacks. IEEE Transactions on Network and Service Management, 14, pp.487-497.

Nadeem, M.W., Goh, H.G., Ponnusamy, V., and Aun, Y., 2022. DDoS detection in SDN using machine learning techniques. Computers, Materials and Continua, 71(1), pp.771-789.

Nath Rimal, A., and Praveen, R., 2020. DDOS attack detection using machine learning. Journal of Emerging Technologies and Innovative Research, 7, pp.1-7.

Naveen, B., and Manu, S., 2019. Detecting DDoS attacks using machine learning techniques and contemporary intrusion detection dataset. Automatic Control and Computer Sciences, 53(5), pp.419-428.

Novaes, M.P., 2020. Long short-term memory and fuzzy logic for anomaly detection and mitigation in software-defined network environment. IEEE Access, 8, pp.83765-83781.

Özçam, B., Kilinc, H.H., and Zaim, A.H., 2021. Detecting TCP Flood DDoS Attack by Anomaly Detection based on Machine Learning Algorithms. In: 2021 6th International Conference on Computer Science and Engineering (UBMK). pp.512-516.

Pai, K., and Bha, A., 2014. Detection and performance evaluation of DoS/DDoS attacks using SYN flooding attacks. International Journal of Computer Applications, 975, pp.1-4.

Pari, S.N., Ritika, E.C., Ragul, B., and Bharath, M., 2023. AI-based Network Flooding Attack Detection in SDN using Multiple Learning Models and Controller. In: 2023 12th International Conference on Advanced Computing (ICoAC). pp.1-7.

Patel, M., Amritha, P.P., Sudheer, V.B., and Sethumadhavan, M., 2024. DDoS Attack detection model using machine learning algorithm in next generation firewall. Procedia Computer Science, 233, pp.175-183. Patel, N.K., Anagha, N., and Santhosh Kumar, J., 2024. Effective Intrusion

Detection and Prevention System of Botnet attack in Blockchain Technology using Recurrent Neural Network. In: 2024 Control Instrumentation System Conference (CISCON). pp.1-6.

Ramadhani, E.H., Enriko, I.K.A., Alamsyah, A.T., Nuha, M.A.U., and Sari, E.L.I.P., 2025. Comparative Analysis of QoS between LEO Satellite and Cellular Internet Networks for IoT Smart Farming. pp.479-489.

Ravi, N., and Shalinie, S.M., 2021. BlackNurse-SC: A novel attack on SDN controller. IEEE Communications Letters, 25(7), pp.2146-2150.

Rawashdeh, A., Alkasassbeh, M., and Al-Hawawreh, M., 2018. An anomaly based approach for DDoS attack detection in cloud environment. International Journal of Computer Applications in Technology, 57, p.312.

Rimal, A.N. and Praveen, R., 2020. DDOS attack detection using machine learning. Journal of Emerging Technologies and Innovative Research, 7(6), pp.185-188.

Sahi, A., Lai, D., Li, Y., and Diykh, M., 2017. An efficient DDoS TCP flood attack detection and prevention system in a cloud environment. IEEE Access, 5, pp.6036-6048.

Sahosh, Z.H., Faheem, A., Tuba, M.B., Tasnim, S.A., Anika, S., and Tasnim, 2024. A Comparative review on DDoS attack detection using machine learning techniques. Malaysian Journal of Science and Advanced Technology, 4, pp.75-83.

Saif, S., Widyawan, W., and Ferdiana, R., 2024. IoT-DH dataset for classification, identification, and detection DDoS attack in IoT. Data in Brief, 54, p.110496.

Saiyed, M.F., and Al-Anbagi, I., 2024. A genetic algorithm- and t-test-based system for DDoS attack detection in IoT networks. IEEE Access, 12, pp.25623-25641.

Sambangi, S., and Gondi, L., 2020a. A machine learning approach for DDoS (distributed denial of service) attack detection using multiple linear regression. Proceedings, 63, p.51.

Sambangi, S., and Gondi, L., 2020b. A machine learning approach for DDoS (distributed denial of service) attack detection using multiple linear regression. Proceedings, 63, p.51.

Shao, Z., Chen, T., Cheng, G., Hu, X., Li, W., and Wu, H., 2023. AF-FDS: An accurate, fast, and fine-grained detection scheme for DDoS attacks in high-speed networks with asymmetric routing. IEEE Transactions on Network and Service Management, 20(4), pp.4964-4981.

Sharma, V.K., and Kumar, M., 2017. Adaptive congestion control scheme in mobile ad-hoc networks. Peer-to-Peer Networking and Applications, 10, pp.633-657.

Sharma, V.K., Verma, L.P., and Kumar, M., 2019. CL-ADSP: Cross-Layer adaptive data scheduling policy in mobile ad-hoc networks. Future Generation Computer Systems, 97, pp.530-563.

Sharma, V.K., Verma, L.P., Kumar, M., Naha, R.K., and Mahanti, A., 2020. A-CAFDSP: An adaptive-congestion aware Fibonacci sequence based data scheduling policy. Computer and Communications, 158, pp.141-165.

Sikos, L.F., 2020. Packet analysis for network forensics: A comprehensive survey. Forensic Science International: Digital Investigation, 32, p.200892.

Singh, A., Çamtepe, S.A., Jang, J.S., Wei, Y., and Sabrina, F., 2023. Classification and Explanation of Distributed Denial-of-Service (DDoS) Attack Detection using Machine Learning and Shapley Additive Explanation (SHAP) Methods. ArXiv, abs/2306.17190.

Singh, S., Jeong, Y.S., and Park, J.H., 2016. A survey on cloud computing security: Issues, threats, and solutions. Journal of Network and Computer Applications, 75, p.200-222.

Sreeram, I., and Vuppala, V.K., 2019. HTTP flood attack detection in application layer using machine learning metrics and bio inspired bat algorithm. Applied Computing and Informatics, 15, pp.59-66.

Srinivasu, P.N., Bhoi, A.K., Nayak, S.R., Bhutta, M.R., and Woźniak, M., 2021. Blockchain technology for secured healthcare data communication among the non-terminal nodes in IoT architecture in 5G network. Electronics, 10, 1437.

Subashini, V., Janaki, R., Mol, M.S., and Kokilavani, G.M., 2022. Implementation of Effective IoT Architecture for Early Flood Detection and Management System. In: 2022 International Conference on Computer, Power and Communications (ICCPC). pp.158-165.

Sudar, K.M., and Deepalakshmi, P., 2020a. A two level security mechanism to detect a DDoS flooding attack in software-defined networks using entropy-based and C4.5 technique. Journal of High Speed Networks, 26, pp.1-22.

Sudar, K.M., and Deepalakshmi, P., 2020b. Comparative study on IDS using machine learning approaches for software defined networks. International Journal of Intelligent Enterprise, 7, pp.15-27.

Swami, R., Dave, M., and Ranga, V., 2021. Detection and analysis of TCP SYN DDoS attack in software-defined networking. Wireless Personal Communications, 84, pp.2295-2317.

Syafiuddin, N.H., Mandala, S., and Cahyani, N.D.W., 2023. Detection Syn Flood and UDP Lag Attacks Based on Machine Learning Using AdaBoost. In: 2023 International Conference on Data Science and Its Applications (ICoDSA). pp.36-41.

Tang, D., Zheng, Z., Wang, X., Xiao, S., and Yang, Q., 2023. PeakSAX: Real-time monitoring and mitigation system for LDoS attack in SDN. IEEE Transactions on Network and Service Management, 20(3), pp.3686-3698.

Tuan, N.N., Hung, P.H., Nghia, N.D., Tho, N.V, Phan, T.V., and Thanh, N.H., 2020. A DDoS attack mitigation scheme in ISP networks using machine learning based on SDN. Electronics, 9, 413.

Tuyen, N.D., Quan, N.S., Linh, V.B., Tuyen, V.V., and Fujita, G., 2022. A comprehensive review of cybersecurity in inverter-based smart power system amid the boom of renewable energy. IEEE Access, 10, pp.35846-35875.

Wang, H., and Li, Y., 2024. Overview of DDoS attack detection in software defined networks. IEEE Access, 12, pp.38351-38381.

Wang, M., Lu, Y., and Qin, J., 2022. Source-based defense against DDoS attacks in SDN based on sFlow and SOM. IEEE Access, 10, pp.2097-2116.

Xiao, M., Cui, Y., Qian, Q., and Shen, G., 2022. KIND: A novel image-mutual information-based decision fusion method for saturation attack detection in SD-IoT. IEEE Internet of Things Journal, 9(23), pp.23750-23771.

Yang, C.H., Wu, J.P., Lee, F.Y., Lin, T.Y., and Tsai, M.H., 2023. Detection and mitigation of SYN flooding attacks through SYN/ACK packets and black/white lists. Sensors (Basel), 23(8), 3817.

Zamrai, M.A.H., Yusof, K.M., and Azizan, M.A., 2024. Random Forest Stratified K-Fold Cross Validation on SYN DoS Attack SD-IoV. In: 2024 7th International Conference on Communication Engineering and Technology (ICCET). pp.7-12.

Zeeshan, M., Riaz, Q., Bilal, M.A., Shahzad, M.K., Jabeen, H., Haider, S.A., and Rahim, A., 2022. Protocol-based deep intrusion detection for DoS and DDoS attacks using UNSW-NB15 and Bot-IoT data-sets. IEEE Access, 10, pp.2269-2283.

Zhou, Q., Li, R., Xu, L., Nallanathan, A., Yang, J., and Fu, A., 2022. Towards Explainable Meta-Learning for DDoS Detection. SN Computer Science, 5 (1), 115.

Zubaydi, H.D., Anbar, M., and Wey, C.Y., 2017. Review on Detection Techniques against DDoS Attacks on a Software-Defined Networking Controller. In: Proceedings - 2017 Palestinian International Conference on Information and Communication Technology, PICICT 2017. Institute of Electrical and Electronics Engineers Inc., pp.10-16.

Published

2026-02-11

How to Cite

Hamad , S. S. and Ghafoor, K. Z. (2026) “Enhancing Network Security: A Review of Machine Learning Techniques for Detecting TCP SYN Flood Attacks”, ARO-THE SCIENTIFIC JOURNAL OF KOYA UNIVERSITY, 14(1), pp. 86–99. doi: 10.14500/aro.12210.

Issue

Section

Review Articles
Received 2025-04-17
Accepted 2025-11-20
Published 2026-02-11

Similar Articles

1 2 3 4 5 6 7 8 9 10 > >> 

You may also start an advanced similarity search for this article.