ExPSO-DL: An Exponential Particle Swarm Optimization Package for Deep Learning Model Optimization

Introduction

The use of machine learning, including deep learning, algorithms has expanded substantially across numerous domains [1, 2, 3]. These algorithms are used for various tasks, including classification, regression, anomaly detection, and segmentation [4]. Training these algorithms depends on parameters such as the learning rate and the number of epochs. Selecting the right parameters is vital. It accelerates convergence, enhances generalization, prevents overfitting, and improves robustness. This step significantly impacts the model’s ability to effectively address real-world problems [5].

To address the challenge of selecting optimal parameters, researchers propose several optimization methods, including swarm-based selection algorithms inspired by social behavior. The complexity of the selection problem varies significantly based on the attributes and properties of the objective function. Among these methods, Particle Swarm Optimization (PSO) stands out as a highly durable and efficient swarm-based selection algorithm, widely appreciated for its simplicity and versatility in various scenarios and applications [6, 7, 8]. PSO consistently delivers high performance in multi-objective optimization [9], constraint optimization [10], and global optimization [11].

Xia, et al [12] introduced Fitness-based Multi-role PSO (FMPSO), an extension that adds a sub-social learning aspect. This enables the swarm to employ different search strategies by utilizing both local and global information. Another work presented Expanded Particle Swarm Optimization (XPSO) [13], which is based on biological and human society’s approaches to discarding unused data. XPSO incorporates this into standard PSO by utilizing global best and local best as examples and assigning each particle a forgetting ability. This improves the algorithm’s adaptability and efficiency [14]. Phasor Particle Swarm Optimization (PPSO) [15] is a new adaptation. It replaces control parameters with a scalar phasor angle derived from trigonometric functions, offering a new approach to enhance PSO.

However, achieving convergence is not always guaranteed, and there is a risk of encountering premature convergence. To address this issue, the authors in [16] introduce Exponential Particle Swarm Optimization (ExPSO), a novel variation of the standard PSO. ExPSO incorporates a leaping strategy based on dynamic parameters. It divides the swarm population into three sub-populations and employs a search strategy using an exponential function, enabling particles to make significant leaps in the search space. Additionally, it adjusts the control of each particle’s velocity range to balance exploration and exploitation phases. ExPSO is designed to take large jumps at the start of the search, followed by smaller jumps for refining solutions in specific regions of the search space [17, 18].

The selection method in ExPSO lacks comprehensive details for its re-implementation, creating a gap in transferring the developed method into a usable function for other research groups. Additionally, designing a package that excels in multiple criteria, demonstrates fast convergence, thoroughly explores the search space, and accommodates various types of objective functions presents a significant challenge. Meeting these criteria requires careful consideration and innovation. Table 1 shows a comparative summary of key features across ExPSO and four established PSO-based algorithms (XPSO, PPSO, FMPSO, and TAPSO). The table highlights the distinctive capabilities of ExPSO, including its exponential leaping strategy, adaptive subpopulation dynamics, and integration with deep learning frameworks, which collectively enhance its suitability for complex optimization and hyperparameter tuning tasks.

In this paper, we present a Python package that extends the Exponential Particle Swarm Optimization (ExPSO) algorithm with a robust, accessible implementation tailored for deep learning model optimization. This implementation surpasses the original by incorporating enhanced usability features, including a modular API, automatic neural network parameter tuning, and integration with machine learning libraries such as TensorFlow [19] and PyTorch [20]. We further provide deployment-oriented benefits, such as parallelized evaluation for faster convergence and customizable termination criteria for real-world applications. These improvements make ExPSO more practical for researchers and practitioners seeking effective and reliable hyperparameter optimization in deep learning.

Implementation and Architecture

To enhance the balance between exploration and exploitation in swarm-based optimization, we introduce a novel variant of Particle Swarm Optimization (PSO). Our approach is designed to adaptively regulate particle movement using exponential selection pressures and velocity control (see Figure 1), enabling the swarm to effectively escape local optima and converge toward global optima with improved stability.

Figure 1

The complete workflow of our method appears in Algorithm 1. This algorithm outlines the initialization, evaluation, and iterative update steps of the ExPSO optimizer. The population of particles is initialized with random positions and velocities within defined bounds. Each particle maintains a record of its personal best pbest and personal worst pworst positions. The swarm also keeps track of the global best pgbest and global worst pgworst particles.

Algorithm 1

Our design features three distinct subpopulations. Each subgroup employs its own velocity update strategy, utilizing adaptive exponential coefficients. Algorithm 2 describes this process: particles are divided into three subgroups based on their index and remain in these groups throughout the algorithm:

Algorithm 2

• The first subgroup uses a velocity equation that attracts particles to both their personal best and worst positions, as well as their global best and worst positions.

• The second subgroup excludes the global worst components, simplifying the trajectory.

• The third subgroup follows classic PSO, using only personal and global best terms.

Each velocity update employs an exponential decay term:

which increases convergence pressure as particles approach their personal best, thus refining local search. This formulation enables adaptive intensification or diversification based on particle experience.

To further manage convergence behavior and avoid premature stagnation, we incorporate a Velocity Controller described in Algorithm 3. This mechanism adaptively compresses the velocity range after a specified number of iterations by scaling the velocity bounds and updating the inertia weight, w, dynamically using:

Algorithm 3

This damping mechanism ensures smoother convergence by gradually reducing the exploration radius while maintaining directional diversity. The optimization loop continues until a stopping criterion is met, typically based on the maximum number of function evaluations MaxFES. At each iteration, the fitness of each particle is evaluated, and the personal/global best and worst records are updated accordingly.

To construct an effective model with optimal parameters, specific steps are followed. First, data are processed and split into training, validation, and test sets. The training and validation sets are used to develop the model. The test set evaluates the model’s performance and generalization.

During the parameter selection phase, the user employs the ExPSO package to find the most suitable parameters. This package is built around a primary Python file that includes several PyTorch classes, such as ExPSO and Particle. These classes work together to implement the ExPSO algorithm effectively. The ExPSO class is initialized with several parameters, including the objective function, dimensions, number of particles, maximum iterations, bounds, and runs. Below, we provide a detailed description of each parameter:

• Objective function (ObjFunction): The function that the algorithm aims to optimize using a given set of input parameters.

• Dimensions (D): Refers to the shape or size of the input parameters in the objective function.

• The number of particles (nPop): The number of particles used to explore the solution space. A higher number of particles can improve solution quality, but it also increases computational cost.

• Maximum iteration number (MaxIt): Sets the maximum number of times the algorithm runs before it stops. Set this to a reasonable number to balance good solutions and to avoid long runtimes.

• Upper and lower bounds (ub, Ib): These are the limits within which the algorithm searches for solutions for each input parameter. Properly set bounds ensure the algorithm searches within the appropriate range.

• Number of runs (runs): Specifies how many times the algorithm runs with the same parameters to ensure robust and reliable results.

When the user invokes the optimize() method, the ExPSO algorithm starts by creating multiple combinations of parameters, which are randomly initialized. These combinations are then used in multiple iterations of the algorithm. During each iteration, the particles are converted into parameter sets and the algorithm is trained. Through iterative communication, the particles collaborate to identify the optimal solution. The objective is to discover a parameter combination that results in the most effective model. The flowchart of the optimization algorithm is detailed in [16]. Upon termination, the ExPSO algorithm extracts and stores the parameters of the best-performing model within the instantiated ExPSO object. Various factors influence the computation time of ExPSO, including the number of particles, the number of iterations, and the maximum allowed iterations. Additionally, supplementary constraints may impact the computation time by causing the optimization process to stop if no improvements are detected over a series of consecutive iterations.

The ExPSO package is designed to optimize a range of problems by treating them as a black box, making it particularly suitable for parameter selection tasks. This package supports a variable number of parameters and accommodates multiple parameter types across various algorithms. It is especially beneficial for researchers seeking to investigate the impact of various selection methods on the effectiveness of their models. The package enables users to visualize multiple metrics, providing insight into how different parameter settings affect model performance.

• GlobalBestPosition: Represents the optimal or near-optimal solution obtained by the ExPSO algorithm in the search space.

• GlobalBestCost: Represents the optimal or near-optimal value achieved by the ExPSO algorithm.

• WorstSol: Refers to the highest value of the best cost found during the optimization process. It helps evaluate the quality of the obtained solutions and identify the worst performing solution.

• MEAN: Represents the average value of the best cost found across multiple optimization runs.

• BestSol: Refers to the lowest value of the best cost found during the optimization process.

• STD (Standard Deviation): Refers to the standard deviation, which is often used to assess the diversity or convergence of the obtained solutions. A lower standard deviation indicates that the solutions are more closely clustered and concentrated around the optimal solution.

• AvgFES (Average Function Evaluations): Represents the average number of function evaluations performed during the optimization process. It indicates the computational effort required to find the optimal solution.

By leveraging the flexibility of PyTorch and its ability to define new structures, users can customize the optimization process and experiment with different variations of the ExPSO algorithm.

Implementation

PyTorch was selected for this work due to its dynamic computation graph (eager execution), which provides more flexibility and transparency during model development, debugging, and implementation of custom optimization or attack strategies. This is particularly advantageous for research-focused tasks such as counterfactual generation, adversarial attacks, and iterative model updates. Compared to TensorFlow [19] and Keras [22], PyTorch offers more intuitive control flow and better support for low-level operations, which are essential when fine-tuning optimization processes or integrating explainability components [20]. While (Just After eXecution (JAX) [23] is also a strong candidate for high-performance computing, its functional paradigm and limited debugging support make PyTorch more suitable for iterative and experimental work. While JAX provides strong capabilities for high-performance and accelerated computing through its functional programming paradigm and XLA compilation, the limited flexibility in debugging and dynamic execution makes PyTorch a more practical and accessible framework for iterative development and experimental research workflows.

Hyperparameters

In our proposed approach, we adopted a carefully tuned set of parameters to balance exploration, exploitation, and diversity within the swarm. The exponential weight parameter (a = 2) ensures moderate sensitivity to fitness differences, allowing meaningful selection pressure without premature convergence. The cognitive and social acceleration coefficients (b = c = 2) are set to their empirically optimal values, maintaining a stable trade-off between individual learning and collective guidance. To avoid stagnation in local optima, we introduce worst-case coefficients (d = e = –1) that occasionally push particles away from suboptimal regions. The cognitive scaling factor (c1 = –1) promotes individual diversity, while the social scaling factor (c2 = 2) fosters cohesion within the swarm. We maintain a relatively high inertia weight (w = 0.9) to encourage broader exploration, and apply a damping factor (r = 0.9) to stabilize the velocity updates. During the exploration phase, 10 iterationsare executed with a controlled velocity decay (k = 0.2) to gradually narrow the search. Additionally, the swarm is partitioned into three equally sized subpopulations (N1 = N2 = N3 = 10) to ensure parallel and diversified exploration across multiple regions of the solution space.

Quality Control

To ensure the reliability and correctness of the ExPSO library, we developed a comprehensive unit testing suite using the pytest framework. These unit tests validate the core functionalities of the package, including particle initialization, velocity and position updates, fitness evaluations, convergence criteria, and integration with external deep learning frameworks such as PyTorch and TensorFlow. Table 2 summarizes the unit tests that verify core functionalities of the ExPSO algorithm, all of which have successfully passed, ensuring reliable initialization, updates, evaluation, and integration.

Example Usages

We provide two examples designed to help users become acquainted with the ExPSO package. These examples gradually increase in complexity, allowing users to gain familiarity with the package. The full code for the experiments is available on our GitHub site.

Example 01: ExPSO with Rosenbrock function

In this example, we demonstrate the utilization of the ExPSO package for a straightforward parameter optimization task. Our objective is to find optimal solutions for the Rosenbrock function [21] (Figure 2). The code begins by importing necessary libraries (lines 1 and 2) and initializing inputs such as runs, Ib, ub, D, nPop, and MaxIt (lines 3–8). We then define the objective function specifically for the Rosenbrock function and instantiate an ExPSOCIass (lines 9 and 12). The optimization process starts with a call to the optimize() function (line 14), and the resulting best parameters are captured in a variable presented on line 16. Figure 3 illustrates the multiple metric outputs obtained from executing the code.

Figure 2

Figure 3

Example 02: ExPSO with MLP Model

In this example, we illustrate the application of ExPSO for fine-tuning the parameters of a Multilayer Perceptron (MLP) model. Figure 4 outlines the initial steps familiar from the previous example: importing necessary modules, initializing an ExPSO instance, and initiating the optimization process. However, instead of employing the Rosenbrock function, we define an objective function tailored for the MLP model, detailed in Figure 5. The objective function calculates the loss by invoking the MLP method, as illustrated in Figure 6. This method utilizes the “neurons” and “epochs” values obtained in each iteration (lines 2 and 3) to train the model, as shown in lines 7 and 17.

Figure 4

Figure 5

Figure 6

Analysis using benchmark data

To validate the effectiveness of ExPSO across diverse scenarios, we selected a range of optimization algorithms for comparison. For standard benchmark functions (both unimodal and multi-modal), we included ExPSO, Phasor Particle Swarm Optimization (PPSO) [15], Fitness-based Multi-role PSO (FMPSO) [12], and eXpanded PSO (XPSO) [13] to ensure fair assessment across different landscape complexities, highlighting the balance between exploration and exploitation achieved by our proposed method. For real-world engineering problems, algorithms like PPSO, FMPSO, MSPSO, XPSO, TAPSO, EO, DSOS, HHO, IRGA, EBO with CMAR, and IMODE were chosen due to their proven performance in handling constrained, high-dimensional, and non-convex tasks. We conducted tests on 12 benchmark functions and compared its results with three state-of-the-art algorithms. These benchmark functions, extensively described in [16], cover a variety of problem types and dimensions, specifically 12 scalable problems with dimension D = 30. The chosen functions represent different search landscape characteristics; some are unimodal with a single global minimum, which tests the algorithm’s exploitation ability, while others are multimodal with numerous local minima, testing the algorithm’s exploration capabilities. The benchmark functions include:

• Unimodal benchmark functions (f1 to f6)—namely, the Sphere function (f1) [24], Schwefel’s functions v2.22 (f2), v1.2 (f3), and v2.21 (f4) [25], Rosenbrock function (f5) [21], and Step function (f6) [26], respectively—are used to evaluate exploitation capability and algorithm convergence performance.

• Multimodal benchmark functions (f7 to f12)—namely, Schwefel’s function (f7) [25], Rastrigin’s function (f8) [25], Ackley’s function (f9) [27], Griewank function (f10) [28], Generalized Penalized function 1 (f11) [29], and Generalized Penalized function 2 (f12) [30], respectively—are used to evaluate exploration capability and the ability to handle multiple local optima.

Tables 3 and 4 present the comparison results, detailing average values (Avg.) and standard deviations (S.D.) across each experimental setup. For unimodal functions, Table 3 shows that all algorithms successfully identify the global optimum in most cases. ExPSO consistently achieves the global minimum across all 30 runs for functions f1 to f6, with average values of 0.00e+00, indicating not only exact convergence but also high numerical stability. Standard deviations for ExPSO in these cases are either exactly zero or on the order of 1e–10, which is significantly lower than competing methods, where deviations often range from 1e–4 to 1e+01. This highlights ExPSO’s strong exploitation capabilities, attributed to its exponential PSO strategy.

For multimodal functions, Table 4 reveals that ExPSO consistently identifies the global minimum for all functions tested. In contrast, PPSO, XPSO, and FMPSO occasionally fail to achieve the global minimum for specific functions (i.e., PPSO for f7, XPSO for f10, and FMPSO for f8 and f10). The repeated achievement of 0.00e+00 values across multiple runs, along with consistently lower standard deviations compared to other methods, further underscores ExPSO’s robust and repeatable exploration capabilities.

Analysis using engineering data

In this section, ExPSO is tested on four well-known engineering design problems: Pressure Vessel Design (PVD) [31], Compression Spring Design (CSD) [32], Welded Beam Design (WBD) [33], and Speed Reducer Design (SRD) [34]. These problems have different constraints that should not be violated by the optimal solution(s) obtained, and thus, a constraint handling method must be utilized.

Tables 5, 6, 7, 8 present comprehensive performance comparisons of twelve optimization algorithms across four classic engineering design problems. The results demonstrate varying algorithmic effectiveness depending on the problem characteristics. For the Pressure Vessel Design (PVD) problem, the results indicate that ExPSO, EO, and DSOS achieve nearly identical minimum cost values. In the Coil Spring Design (CSD) task, most algorithms, except XPSO, converge to comparable solutions, while ExPSO, DSOS, HHO, EBOwithCMAR, and IMODE attain the same optimal result. For the Welded Beam Design (WBD) problem, ExPSO delivers a solution similar to EO, DSOS, EBOwithCMAR, and IMODE, surpassing the remaining algorithms in performance. Statistical analysis further shows that the mean solution produced by ExPSO is very close to its best outcome, confirming its consistency and robustness across multiple runs. In the Speed Reducer Design (SRD) problem, comparison of the optimal and statistical results reveals that EBOwithCMAR attains the best overall cost (2639.499), whereas most other algorithms yield 2994.424. In contrast, FMPSO, XPSO, and HHO record slightly higher costs of 3048.377, 3060.097, and 2997.639, respectively.

Classification analysis

In Figure 7, we present a detailed comparison of the performance achieved using the proposed ExPSO algorithm against four baseline optimizers across five different model architectures: CNN [35], LSTM [36], XLNet [37], MLP [38], and VGGNet [39]. The experiments were conducted on benchmark classification datasets, including MNIST [40] for image-based models (CNN and VGGNet), and IMDB [41] for text-based models (LSTM and XLNet), while the UCI Breast Cancer dataset was used for the MLP model. Each dataset includes clearly defined class labels—for example, digit categories (0–9) in MNIST, sentiment polarity (positive/negative) in IMDB, news topic categories in AG News, and benign vs. malignant in the breast cancer dataset.

Figure 7

Across all tested models, ExPSO consistently achieves the highest classification accuracy, demonstrating strong robustness and generalization across both deep and traditional neural architectures. On the CNN model, ExPSO reaches an accuracy of 97%, significantly outperforming PSO (91.4%), QPSO (92%), and FastPSO (90%), suggesting superior early-stage exploration capabilities. Similarly, for the LSTM model, ExPSO again achieves 97%, surpassing PSO (92.11%) and QPSO (94%), indicating that its fine-grained control of position updates is particularly effective in sequence-based learning tasks. In more complex transformer-based architectures, such as XLNet, ExPSO outperforms QPSO (81%) and FastPSO (82%), notably by 6 percentage points, which underscores its efficiency in navigating high-dimensional attention-based search spaces. Even in simpler feedforward networks, such as MLP, where optimizer performance tends to converge, ExPSO still maintains a slight edge at 89%, compared to QPSO at 88% and FST-PSO at 87%, reflecting its stable optimization behavior. Finally, on VGGNet, a deep convolutional architecture, ExPSO attains 92%, considerably outperforming PSO (85.19%) and FastPSO (87%), further demonstrating its ability to escape suboptimal minima in deep layered structures. These consistent improvements across a variety of models validate ExPSO’s design as a balanced optimizer capable of adapting to both the structural complexity and training dynamics of modern learning systems.

The performance results reveal several consistent trends that underscore the effectiveness of ExPSO as an optimization framework. Notably, performance gains are more pronounced in complex architectures such as CNN, LSTM, and XLNet, where ExPSO demonstrates its capacity to scale with increasing model depth and parameter complexity. This suggests that ExPSO is particularly well-suited for high-dimensional and non-convex search spaces that often challenge conventional PSO variants. Unlike standard PSO and FastPSO, which exhibit fluctuating performance across tasks, ExPSO displays stable convergence behavior, maintaining a narrow and consistently high accuracy range of 87% to 97%.

Limitations

While the results demonstrate that ExPSO performs competitively across a range of benchmark functions, engineering problems, and classification models, several limitations remain that warrant further investigation. First, although ExPSO achieves high accuracy and repeatability in both unimodal and multimodal optimization tasks, its current performance evaluation is limited to standard benchmark functions (f1–f12) and four engineering design scenarios. A broader evaluation on real-world, noisy, and dynamic optimization problems—such as scheduling, resource allocation, and neural architecture search—would provide a more comprehensive understanding of its robustness under diverse conditions. Second, the exponential search component, which enhances convergence, introduces moderate computational overhead. While this is mitigated through parallelism and adaptive control, the exact trade-off between added complexity and optimization gain remains to be formally quantified. In future work, we plan to benchmark computational time and memory usage across scalable high-dimensional problems to further validate ExPSO’s efficiency in time-constrained or resource-limited environments. Third, while ExPSO supports integration with PyTorch and TensorFlow, current deep learning tests focus primarily on classification tasks. Future iterations will explore regression, generative modeling, and reinforcement learning settings to better understand ExPSO’s applicability across varied ML pipelines. Finally, the optimizer’s parameter sensitivity (particularly with respect to subpopulation size, velocity damping factor, and exponential decay rate—deserves deeper empirical tuning). Automated meta-optimization or self-adaptive control mechanisms could enhance usability by reducing manual tuning and improving generalizationacross tasks.

Operating System

This package can be run on any operating system where Python can be run (GNU/Linux, Mac OSX, Windows).

Programming Language

python 3.6.1+

Additional System Requirements

None.

Dependencies

pytorch, numpy, math, tensorflow, keras, scikit-learn

Software Location

Archive

Code repository

Language

English