An improved hyperparameter optimization framework for AutoML systems using evolutionary algorithms

  • Feurer, M. & Hutter, F. Hyperparameter optimization. In Automated Machine Learning, The Springer Series on Challenges in Machine Learning 3–33. doi.org/10.1007/978-3-030-05318-5 (2018).

  • Belete, D. M. & Huchaiah, M. D. Grid search in hyperparameter optimization of machine learning models for prediction of HIV/AIDS test results. Int. J. Comput. Appl. 44, 875–886. doi.org/10.1080/1206212X.2021.1974663 (2022).

    Article 

    Google Scholar
     

  • Fuadah, Y. N., Pramudito, M. A. & Lim, K. M. An optimal approach for heart sound classification using grid search in hyperparameter optimization of machine learning. Bioengineering 10. doi.org/10.3390/bioengineering10010045 (2023).

  • Bergstra, J. & Bengio, Y. Random search for hyperparameter optimization. J. Mach. Learn. Res. 13, 281–305. doi.org/10.5555/2188385.2188395 (2012).

    Article 
    MathSciNet 
    MATH 

    Google Scholar
     

  • Hutter, F., Lucke, J. & Schmidt-Thieme, L. Beyond manual tuning of hyperparameters. KI Kunstliche Intelligenz 29, 329–337. doi.org/10.1007/s13218-015-0381-0 (2015).

    Article 

    Google Scholar
     

  • Bergstra, J., Bardenet, R., Bengio, Y. & Kégl, B. Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems, 2546–2554. doi.org/10.5555/2986459.2986743 (ACM, 2011).

  • Victoria, A. H. & Maragatham, G. Automatic tuning of hyperparameters using Bayesian optimization. Evol. Syst. 12, 217–223. doi.org/10.1007/s12530-020-09345-2 (2021).

    Article 

    Google Scholar
     

  • Dewancker, I. et al. Bayesian optimization primer. SigOpt. 1–2 (2015).

  • Hazan, E., Klivans, A. & Yuan, Y. Hyperparameter optimization: A spectral approach. Int. Conf. Learn. Represent. 17, 2 (2018).


    Google Scholar
     

  • Eggensperger, K. et al. Towards an empirical foundation for assessing Bayesian optimization of hyperparameters. In NIPS Workshop on Bayesian Optimization in Theory and Practice, Vol. 10, 3 (2013).

  • Shahriari, B., Swersky, K., Wang, Z., Adams, R. P. & Freitas, N. D. Taking the human out of the loop: A review of Bayesian optimization. Proc. IEEE 104(1), 148–175 (2015).

    Article 

    Google Scholar
     

  • Zulfiqar, M., Gamage, K. A. A., Kamran, M. & Rasheed, M. B. Hyperparameter optimization of Bayesian neural network using Bayesian optimization and intelligent feature engineering for load forecasting. Sensors 22 (2022).

  • ALGorain, F. T. & Clark, J. A. Bayesian hyper-parameter optimisation for malware detection. Electronics 11. doi.org/10.3390/electronics11101640 (2022).

  • Snoek, J., Larochelle, H. & Adams, R. P. Practical Bayesian optimization of machine learning algorithms. In Advances on Neural Information Processing Systems, 2–8. doi.org/10.5555/2999325.2999464 (2012).

  • Blum, C. Metaheuristics in combinatorial optimization: Overview and conceptual comparison. ACM Comput. Surv. 35, 268–308. doi.org/10.1145/937503.937505 (2003).

    Article 

    Google Scholar
     

  • Maier, H. R. et al. Introductory overview: Optimization using evolutionary algorithms and other metaheuristics. Environ. Model. Softw. 114, 195–213. doi.org/10.1016/j.envsoft.2018.11.018 (2019).

    Article 

    Google Scholar
     

  • Tani, L., Rand, D., Veelken, C. & Kadastik, M. Evolutionary algorithms for hyperparameter optimization in machine learning for application in high energy physics. Eur. Phys. J. 81, 1–9. doi.org/10.1140/epjc/s10052-021-08950-y (2021).

    Article 
    CAS 

    Google Scholar
     

  • Deb, K. Practical optimization using evolutionary methods Kangal report (2005).

  • Orive, D., Sorrosal, G., Borges, C. E., Martin, C. & Alonso-Vicario, A. Evolutionary algorithms for hyperparameter tuning on neural network models. In Proceedings of the 26th European Modelling and Simulation Symposium, 402–409 (2014).

  • Goldberg, D. E. Genetic Algorithms in Search, Optimization and Machine Learning (Addison-Wesley Longman Publishing Co., 1989).

    MATH 

    Google Scholar
     

  • Ahmadlou, M., Ghajari, Y. E. & Karimi, M. Enhanced classification and regression tree (CART) by genetic algorithm (GA) and grid search (GS) for flood susceptibility mapping and assessment. Geocarto Int., 1–20. doi.org/10.1080/10106049.2022.2082550 (2022).

  • Demir, S. & Åžahin, E. K. Liquefaction prediction with robust machine learning algorithms (SVM, RF, and XGBoost) supported by genetic algorithm-based feature selection and parameter optimization from the perspective of data processing. Environ. Earth Sci. 81, 459. doi.org/10.1007/s12665-022-10578-4 (2022).

    Article 
    ADS 

    Google Scholar
     

  • Kennedy, J. & Eberhart, R. Particle swarm optimization. In Proceedings of ICNN’95—International Conference on Neural Networks, 1942–1948. doi.org/10.1109/ICNN.1995.488968 (IEEE, 1995).

  • Lorenzo, P. R., Nalepa, J., Kawulok, M., Ramos, L. S. & Pastor, J. R. Particle swarm optimization for hyper-parameter selection in deep neural networks. In GECCO ’17: Proceedings of the Genetic and Evolutionary Computation Conference, 481–488. doi.org/10.1145/3071178.3071208 (2017).

  • Dorigo, M., Maniezzo, V. & Colorni, A. Ant system: Optimization by a colony of cooperating agents. IEEE Trans. Syst. Man Cybern. Part B (Cybern.), 29–41. doi.org/10.1109/3477.484436 (1996).

  • Costa, V. O. & Rodrigues, C. R. Hierarchical ant colony for simultaneous classifier selection and hyperparameter optimization. In IEEE Congress on Evolutionary Computation (CEC), 1–8. doi.org/10.1109/CEC.2018.8477834 (IEEE, 2018).

  • Lakra, A. V. & Jena, S. Optimization of random forest hyperparameter using improved PSO for handwritten digits classification. In Computing, Communication and Learning (eds Panda, S. K. et al.) 266–276 (Springer Nature, 2022).

    Chapter 

    Google Scholar
     

  • Xiong, Y. et al. Landslide susceptibility mapping using ant colony optimization strategy and deep belief network in Jiuzhaigou region. IEEE J. Select. Top. Appl. Earth Observ. Remote Sens. 14, 11042–11057. doi.org/10.1109/JSTARS.2021.3122825 (2021).

    Article 
    ADS 

    Google Scholar
     

  • Loshchilov, I. & Hutter, F. CMA-ES for hyperparameter optimization of deep neural networks. doi.org/10.48550/ARXIV.1604.07269 (2016).

  • Domhan, T., Springenberg, J. T. & Hutter, F. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In IJCAI’15: Proceedings of the 24th International Conference on Artificial Intelligence, 3460–3468. doi.org/10.5555/2832581.2832731 (2015).

  • Swersky, K., Snoek, J. & Adams, R. P. Freeze-thaw Bayesian optimization (2014).

  • Jamieson, K. & Talwalkar, A. Non-stochastic best arm identification and hyperparameter optimization. In International Conference on Artificial Intelligence and Statistics (AISTATS), 240–248 (2015).

  • Soper, D. S. Hyperparameter optimization using successive halving with greedy cross validation. Algorithms 16 (2023).

  • Karnin, Z., Koren, T. & Somekh, O. Almost optimal exploration in multi-armed bandits. Dasgupta and McAllester, 1238–1246 (2013).

  • Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A. & Talwalkar, A. Hyperband a novel bandit based approach to hyperparameter optimization. J. Mach. Learn. Res. 18, 1–52 (2018).

    MathSciNet 
    MATH 

    Google Scholar
     

  • Dores, S. C. N. D., Soares, C. & Ruiz, D. Bandit-based automated machine learning. In 7th Brazilian Conference on Intelligent Systems (BRACIS), 121–126. doi.org/10.1109/BRACIS.2018.00029 (2018).

  • Swersky, K., Snoek, J. & Adams, R. P. Multi-task Bayesian optimization. Advances in Neural Information Processing Systems, 121–126 (2013).

  • Falkner, S., Klein, A. & Hutter, F. Bohb: Robust and efficient hyperparameter optimization at scale. In Proceedings of the 35th International Conference on Machine Learning (PMLR 80, 2018).

  • Kandasamy, K., Dasarathy, G., Oliva, J., Schneider, J. & Póczos, B. Gaussian process bandit optimisation with multi-fidelity evaluations. In 30th Conference on Neural Information Processing Systems, 992–1000 (2016).

  • Kandasamy, K., Dasarathy, G., Oliva, J., Schneider, J. & Póczos, B. Multi-fidelity Bayesian optimisation with continuous approximations. In Precup and Teh, 1799–1808 (2015).

  • Cui, H. & Bai, J. A new hyperparameters optimization method for convolutional neural networks. Pattern Recognit. Lett. 125, 828–834 (2019).

    Article 
    ADS 

    Google Scholar
     

  • Baldib, P. & Gillen, L. D. Reproducible hyperparameter optimization. J. Comput. Graph. Stat. 31, 84–99 (2022).

    Article 
    MathSciNet 
    MATH 

    Google Scholar
     

  • Mohan, B. & Badrah, J. A novel automated superlearner using a genetic algorithm-based hyperparameter optimization. Adv. Eng. Softw. 175, 103358 (2022).

    Article 

    Google Scholar
     

  • Osama Ahmed, M., Vaswani, S. & Schmidt, M. Combining Bayesian optimization and Lipschitz optimization. Mach. Learn. 109, 79–102 (2020).

    Article 
    MathSciNet 
    MATH 

    Google Scholar
     

  • Guo, B., Hu, J., Wu, W., Peng, Q. & Wu, F. The Tabu genetic algorithm: A novel method for hyper-parameter optimization of learning algorithms. Electronics 8, 579–598 (2019).

    Article 

    Google Scholar
     

  • Zahedi, L., Mohammadi, F. G. & Amini, M. H. Hyp-abc: A novel automated hyper-parameter tuning algorithm using evolutionary optimization. arXiv:1810.13306 (2021).

  • Amirabadi, M., Kahaei, M. & Nezamalhosseini, S. Novel suboptimal approaches for hyperparameter tuning of deep neural network [under the shelf of optical communication]. Phys. Commun. 41 (2020).

  • Lakhmiri, D., Digabel, S. L. & Tribes, C. Hypernomad: Hyperparameter optimization of deep neural networks using mesh adaptive direct search. ACM Trans. Math. Softw. 47. doi.org/10.1145/3450975 (2021).

  • Lakhmiri, D. & Digabel, S. Use of static surrogates in hyperparameter optimization. SN Oper. Res. Forum 3, 1–18. doi.org/10.1007/s43069-022-00128- (2022).

    Article 
    MathSciNet 
    MATH 

    Google Scholar
     

  • Kotthoff, L., Thornton, C., Hoos, H. H., Hutter, F. & Leyton-Brown, K. Auto-WEKA: Automatic model selection and hyperparameter optimization in WEKA. Automated Machine Learning, 81–95. doi.org/10.1007/978-3-030-05318-5_4 (2018).

  • Feurer, M. et al. Auto-sklearn: Efficient and robust automated machine learning. Automated Machine Learning, 113–134. doi.org/10.1007/978-3-030-05318-5_6 (2018).

  • Olson, R. S. & Moore, J. H. TPOT: A tree-based pipeline optimization tool for automating machine learning. Automated Machine Learning, 151–160. doi.org/10.1007/978-3-030-05318-5_8 (2018).

  • Jin, H., Song, Q. & Hu, X. Auto-keras: An efficient neural architecture search system. In KDD ’19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1946–1956. doi.org/10.1145/3292500.3330648 (2019).

  • Zimmer, L., Lindauer, M. & Hutter, F. AutoPyTorch tabular: Multi-fidelity metalearning for efficient and robust AutoDLL. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020).

  • Erickson, N. et al. AutoGluon-Tabular: Robust and accurate AutoML for structured data. arXiv:2003.06505 (2020).

  • LeDell, E. & Poirier, S. H2o automl: Scalable automatic machine learning. In 7th ICML Workshop on Automated Machine Learning, Vol. 2020 (2020).

  • Hutter, F., Hoos, H. H. & Leyton-Brown, K. Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization, 507–523. doi.org/10.1007/978-3-642-25566-3_40 (Springer, 2011).

  • Lindauer, M. et al. Smac3: A versatile Bayesian optimization package for hyperparameter optimization. J. Mach. Learn. Res. 23, 1–9 (2022).

    MATH 

    Google Scholar
     

  • Komer, B., Bergstra, J. & Eliasmith, C. Hyperopt-sklearn. Automated Machine Learning, 97–111. doi.org/10.1007/978-3-030-05318-5_5 (2018).

  • Awad, N., Mallik, N. & Hutter, F. Dehb: Evolutionary hyperband for scalable, robust and efficient hyperparameter optimization. In Proceedings of IJCIA, 2147–2153 (2021).

  • Fortin, F., De Rainville, F., Gardner, M., Parizeau, M. & Gagné, C. Deap: Evolutionary algorithms made easy. J. Mach. Learn. Res. 13, 2171–2175 (2012).

    MathSciNet 

    Google Scholar
     

  • Akiba, T., Sano, S., Yanase, T., Ohta, T. & Koyama, M. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th International Conference on Knowledge Discovery and Data Mining, 2623–2631 (2019).

  • Srinivasan, D. & Seow, T. Particle swarm inspired evolutionary algorithm (PS-EA) for multiobjective optimization problems. In The 2003 Congress on Evolutionary Computation, 2003. CEC ’03, Vol. 4, 2292–2297. doi.org/10.1109/CEC.2003.1299374 (2003).

  • Pant, M., Thangaraj, R., Grosan, C. & Abraham, A. Hybrid differential evolution—particle swarm optimization algorithm for solving global optimization problems. In 2008 Third International Conference on Digital Information Management, 18–24. doi.org/10.1109/ICDIM.2008.4746766 (2008).

  • Mashwani, W. K. Comprehensive survey of the hybrid evolutionary algorithms. Int. J. Appl. Evol. Comput. (IJAEC) 4, 1–19. doi.org/10.4018/jaec.2013040101 (2008).

    Article 

    Google Scholar
     

  • Biswas, S., Cobb, A. D., Sistrunk, A., Ramakrishnan, N. & Jalaian, B. Better call surrogates: A hybrid evolutionary algorithm for hyperparameter optimization. doi.org/10.48550/ARXIV.2012.06453 (2020).

  • Cho, H., Shin, J. & Rhee, W. B2EA: An evolutionary algorithm assisted by two Bayesian optimization modules for neural architecture search. doi.org/10.48550/ARXIV.2202.03005 (2022).

  • de Pison, F. M., Gonzalez-Sendino, R., Aldama, A., Ferreiro-Cabello, J. & Fraile-Garcia, E. Hybrid methodology based on Bayesian optimization and GA-parsimony to search for parsimony models by combining hyperparameter optimization and feature selection. Neurocomputing 354, 20–26. doi.org/10.1016/j.neucom.2018.05.136 (2019). Recent Advancements in Hybrid Artificial Intelligence Systems.

  • Lan, G., Tomczak, J. M., Roijers, D. M. & Eiben, A. Time efficiency in optimization with a Bayesian-evolutionary algorithm. Swarm Evol. Comput. 69, 100970. doi.org/10.1016/j.swevo.2021.100970 (2022).

    Article 

    Google Scholar
     

  • Storn, R. & Price, K. Differential evolution—a simple and efficient heuristic for global optimization over continuous spaces. J. Glob. Optim. 11, 341–359 (1997).

    Article 
    MathSciNet 
    MATH 

    Google Scholar
     

  • Hansen, N. & Ostermeier, A. Completely derandomized self-adaptation in evolution strategies. Evol. Comput. 9(2), 159–195 (2001).

    Article 
    CAS 
    PubMed 

    Google Scholar
     

  • Yang, L. & Shami, A. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing 415, 295–316. doi.org/10.1016/j.neucom.2020.07.061 (2020).

    Article 

    Google Scholar
     

  • N. Knight, J. & Lunacek, M. Reducing the space-time complexity of the CMA-ES. In Genetic and Evolutionary Computation Conference, GECCO, 658–665 (ACM, 2007).

  • Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems Vol. 25 (eds Pereira, F. et al.) 1097–1105 (Curran Associates Inc, 2012).


    Google Scholar
     

  • Huang, G., Liu, Z., van der Maaten, L. & Weinberger, K. Q. Densely connected convolutional networks. doi.org/10.48550/ARXIV.1608.06993 (2016).

  • Read more here: Source link