Dynamic inertia in ParticleSwarmOptimizer

[MatjazBostic]

Is there a possibility to dynamically change the inertia of the ParticleSwarmOptimizer?
I am following this article for optimizing CNN hyperparameters: https://www.sciencedirect.com/science/article/pii/S2210650221000249
It says that inertia should change in the following way:

Is there already a way to achieve that? Maybe by changing this parameter in the model() function? If not, could you implement some sort of a callback to provide a function which dynamically changes inertia?

[SimonBlanke]

Hello @mbostic,

this is a very interesting idea! Thanks for opening this issue.

I already thought about the possibility to change optimization parameters during the run inside the objective function. Currently it is possible to change some of the parameter, but there are some problems:

1. Changing some parameters would not work: Changing the population size during the optimization run would probably be not trivial to implement.
2. Some parameters are hidden inside subclasses: The Particle-class within the particle swarm optimizer.

I will take a closer look at this within the next days and come back to you.

[MatjazBostic]

Well I understand that having the ability to change any of the parameters would be hard. In the case of the ParticleSwarmOptimizer, I think it makes the most sense to have the ability to change at least the inertia, because that's what I saw mostly for this optimizer (I also saw linearly decreasing inertia in another article). Inertia is usually decreasing because this enables the optimizer to converge better.
Here is how I implemented it with the hyperactive:

optimizer=ParticleSwarmOptimizer(
        inertia=initial_inertia,
        ...
    )

i = 0

def model(opt):
  <define the model>
  
  i += 1
  
  if i % population == 0:
    w = <calculate the new inertia to be considered in the next iteration>
    for particle in optimizer._optimizer.particles:
        particle.inertia = w

But I am not sure if it worked properly, so I rather implemented my own PSO and used that to have more customizability, so you don't have to implement it just because of me :)

[SimonBlanke]

Hello @mbostic,

when creating Hyperactive v3 and v4 I already wrote the backend so that accessing the parameters of the optimizers is possible. But it is not tested or documented, yet.
The argument of the objective-function in Hyperactive is not just a dictionary, but contains the entire optimizer (from gradient-free-optimizer).

The code below shows an example, that sets all parameters to 0. If it works, this should reduce all particle movements to 0 as well.
This code is written in Hyperactive v4:

from hyperactive import Hyperactive
from hyperactive.optimizers import ParticleSwarmOptimizer


def model(para):
    x = para["x"]
    y = para["y"]

    for particle in para.optimizer.particles:
        particle.inertia = 0
        particle.cognitive_weight = 0
        particle.social_weight = 0
        particle.temp_weight = 0

    score = -(x ** 2 + y ** 2)
    return score


search_space = {
    "x": list(range(-100, 100)),
    "y": list(range(-100, 100)),
}


optimizer = ParticleSwarmOptimizer()

hyper = Hyperactive()
hyper.add_search(model, search_space, optimizer=optimizer, n_iter=200)
hyper.run()

search_data = hyper.search_data(model)
print("\n search_data \n", search_data.to_string())  # prints entire dataframe

Let me know if this is helpful to you or if you have more questions :-)

[MatjazBostic]

That makes sense, thanks. I think you can close this one.