Automated Machine Learning for IoT Intrusion Detection: A Comparative Evaluation of FLAML and TPOT Under Multiple Validation Strategies

Volume 12 ,Issue 3 ,August 2026 ,Pages 205-211

Authors

Susan M. Al Naqshbandi 1

1 Department of Computer Engineering, College of Engineering, Komar University of Science and Technology, Sulaimani, Kurdistan Region, Iraq

DOI logo 10.17656/sjes.10224

Keywords

Abstract

The vast expansion of IoT devices significantly expands the potential attack surface for networks; therefore, the increased complexity in detecting intrusions from these new sources is caused by the large dimensionality of the data, the severe class imbalance issue that exists when defining a normal versus abnormal behavior model based upon this data and the chaotic nature of network traffic.

This paper proposes a fully automated IoT-based Network Intrusion Detection System (NIDS), utilizing the Gotham Dataset 2025. Two AutoML approaches are used as part of the proposed system: TPOT (Tree-based Pipeline Optimization Tool) and FLAML (cost-aware lightweight AutoML framework). Both systems were evaluated using four different validation methods against approximately 5.8 million network traffic instances. FLAML achieved better results than TPOT in all evaluations, including achieving 99.98% accuracy at one evaluation. TPOT achieved comparable or slightly better performance than FLAML for precision and recall, but was less stable in its performance when dealing with class imbalance.

References

  1. A. Al-Fuqaha, M. Guizani, M. Mohammadi, M. Aledhari, and M. Ayyash, “Internet of Things: A Survey on Enabling Technologies, Protocols, and Applications,” IEEE Commun. Surv. Tutor., vol. 17, no. 4, pp. 2347–2376, 2015, doi: 10.1109/COMST.2015.2444095.
  2. L. Atzori, A. Iera, and G. Morabito, “The Internet of Things: A survey,” Comput. Netw., vol. 54, no. 15, pp. 2787–2805, Oct. 2010, doi: 10.1016/j.comnet.2010.05.010.
  3. M. A. Ferrag, L. Maglaras, S. Moschoyiannis, and H. Janicke, “Deep learning for cyber security intrusion detection: Approaches, datasets, and comparative study,” J. Inf. Secur. Appl., vol. 50, p. 102419, Feb. 2020, doi: 10.1016/j.jisa.2019.102419.
  4. M. Conti, A. Dehghantanha, K. Franke, and S. Watson, “Internet of Things security and forensics: Challenges and opportunities,” Future Gener. Comput. Syst., vol. 78, pp. 544–546, Jan. 2018, doi: 10.1016/j.future.2017.07.060.
  5. H. Hindy et al., “A Taxonomy of Network Threats and the Effect of Current Datasets on Intrusion Detection Systems,” IEEE Access, vol. 8, pp. 104650–104675, 2020, doi: 10.1109/ACCESS.2020.3000179.
  6. R. Sommer and V. Paxson, “Outside the Closed World: On Using Machine Learning For Network Intrusion Detection”.
  7. D. E. Denning, “An Intrusion-Detection Model,” IEEE Transactions on Software Engineering, vol. SE-13, no. 2, pp. 222–232, 1987.
  8. B. Mukherjee, L. T. Heberlein, and K. N. Levitt, “Network intrusion detection,” IEEE Netw., vol. 8, no. 3, pp. 26–41, 1994.
  9. A. Patcha and J.-M. Park, “An overview of anomaly detection techniques: Existing solutions and latest technological trends,” Computer Networks, vol. 51, no. 12, pp. 3448–3470, 2007.
  10. M. Roopak, “Deep learning models for cyber security in IoT networks,” J. Inf. Secur. Appl., vol. 57, 2021.
  11. I. Sharafaldin, A. Habibi Lashkari, and A. A. Ghorbani, “Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization:,” in Proceedings of the 4th International Conference on Information Systems Security and Privacy, Funchal, Madeira, Portugal: SCITEPRESS - Science and Technology Publications, 2018, pp. 108–116. doi: 10.5220/0006639801080116.
  12. J. H. Friedman, “Greedy function approximation: A gradient boosting machine.,” Ann. Stat., vol. 29, no. 5, Oct. 2001, doi: 10.1214/aos/1013203451.
  13. T. Chen and C. Guestrin, “XGBoost: A Scalable Tree Boosting System,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Aug. 2016, pp. 785–794. doi: 10.1145/2939672.2939785.
  14. J. Bergstra, J. Bergstra, Y. Bengio, and Y. Bengio, “Random Search for Hyper-Parameter Optimization”.
  15. A. Verma and V. Ranga, “Statistical analysis of CIDDS-001 dataset for intrusion detection systems,” Procedia Comput. Sci., vol. 125, pp. 736–743, 2018.
  16. M. Tavallaee, E. Bagheri, W. Lu, and A. A. Ghorbani, “A detailed analysis of the KDD CUP 99 data set,” in 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications, Ottawa, ON, Canada: IEEE, Jul. 2009, pp. 1–6. doi: 10.1109/CISDA.2009.5356528.
  17. IoTSiM Consortium, “IoTSiM dataset: A large-scale IoT network traffic dataset,” IoTSiM Consortium, Dataset, 2023. [Online]. Available: https://... (dataset link)
  18. V. W. Samawi, S. A. Yousif, and Nadia M., “Intrusion detection system: An automatic machine learning algorithms using Auto-WEKA,” in Proceedings of the 2022 IEEE 13th Control and System Graduate Research Colloquium (ICSGRC), IEEE, 2022, pp. 42–46.
  19. R. S. Olson and J. H. Moore, “TPOT: A tree-based pipeline optimization tool,” in Proceedings of the Genetic and Evolutionary Computation Conference (GECCO), 2016, pp. 485–492.
  20. M. Feurer, A. Klein, K. Eggensperger, J. T. Springenberg, M. Blum, and F. Hutter, “Efficient and robust automated machine learning,” in Advances in Neural Information Processing Systems, 2015, pp. 2962–2970.
  21. T. T. Le, “Scaling tree-based automated machine learning to biomedical big data,” Bioinformatics, vol. 36, no. 1, pp. 250–256, 2020.
  22. C. Wang, Q. Wu, M. Weimer, and E. Zhu, “FLAML: A fast and lightweight AutoML library,” in Proceedings of Machine Learning and Systems (MLSys), 2021, pp. 1–12.
  23. X. Meng, “FLAML: Efficient and scalable AutoML for large datasets,” IEEE Transactions on Knowledge and Data Engineering, pp. 1–14, 2023.
  24. M. Feurer, K. Eggensperger, S. Falkner, M. Lindauer, and F. Hutter, “Auto-sklearn 2.0: Hands-free AutoML via meta-learning,” in Proceedings of the AutoML Conference, 2020, pp. 1–10.
  25. N. Erickson, J. Mueller, A. Shirkov, H. Zhang, and P. Xu, “AutoGluon-Tabular: Robust and accurate AutoML for structured data,” in Proceedings of the AutoML Conference, 2020, pp. 1–12.
  26. E. LeDell and S. Poirier, “H2O AutoML: Scalable automatic machine learning,” in Proceedings of the AutoML Conference, 2020, pp. 1–10.
  27. P. Gijsbers, “AMLB: An AutoML benchmark,” Journal of Machine Learning Research, vol. 25, pp. 1–10, 2024.
  28. H. Eldeeb, “AutoMLBench: Comprehensive evaluation of AutoML frameworks,” Expert Systems with Applications, vol. 230, pp. 1–15, 2024.
  29. F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
  30. S. Zhang, “Intrusion detection for IoT based on machine learning and deep learning methods,” IEEE Access, vol. 10, pp. 12345–12360, 2022.
  31. Y. Xin, “Machine learning and deep learning methods for cybersecurity,” IEEE Access, vol. 6, pp. 35365–35381, 2018.
  32. A. Alshamrani, “A survey on advanced persistent threats: Techniques, solutions, challenges,” IEEE Communications Surveys & Tutorials, pp. 1–20, 2019.
  33. J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, “Making a science of model search: Hyperparameter optimization,” in Proceedings of the 30th International Conference on Machine Learning (ICML), 2013, pp. 1–9.
  34. T. Dietterich, “Approximate statistical tests for comparing supervised learning algorithms,” Neural Computation, vol. 10, no. 7, pp. 1895–1923, 1998.
  35. G. Forman and M. Scholz, “Apples-to-apples in cross-validation studies,” SIGKDD Explorations, vol. 12, no. 1, pp. 49–57, 2010.
  36. S. Sokolova and G. Lapalme, “A systematic analysis of performance measures for classification tasks,” Information Processing & Management, vol. 45, no. 4, pp. 427–437, 2009.
  37. N. Moustafa and J. Slay, “UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set),” in 2015 Military Communications and Information Systems Conference (MilCIS), Canberra, Australia: IEEE, Nov. 2015, pp. 1–6. doi: 10.1109/MilCIS.2015.7348942.
  38. N. Maher and S. A. Yousif, “An automated machine learning model for diagnosing COVID-19 infection,” IAES International Journal of Artificial Intelligence, vol. 12, no. 3, pp. 1360–1369, 2023.
Statistics
  • Article view20
  • Downloads0
  • First online28 July 2026
  • Published at10 August 2026

  • RIS
  • BibTeX
  • EndNote
  • Mendeley
  • APA (7th edition)
  • MLA (9th edition)
  • Chicago
  • Harvard
  • IEEE
  • Vancouver