Optimization Algorithms

Gradient-Free-Optimizers provides 23 optimization algorithms organized into four categories based on their search strategy. Each algorithm has unique characteristics that make it suitable for different types of problems.

Algorithm Overview

Algorithm	Category	Best For
Hill Climbing	Local	Smooth, unimodal functions with good starting points
Stochastic Hill Climbing	Local	Functions with minor noise or small local optima
Repulsing Hill Climbing	Local	Escaping flat regions or plateaus
Simulated Annealing	Local	Functions with multiple local optima
Downhill Simplex	Local	Low-dimensional continuous optimization
Random Search	Global	Baseline comparison, embarrassingly parallel
Grid Search	Global	Systematic coverage of small search spaces
Random Restart Hill Climbing	Global	Combining local refinement with global exploration
Random Annealing	Global	Broad initial exploration with gradual focusing
Pattern Search	Global	Derivative-free optimization without randomness
Powell’s Method	Global	Separable objective functions
Lipschitz Optimizer	Global	Functions with known smoothness bounds
DIRECT Algorithm	Global	Guaranteed global convergence
Particle Swarm	Population	Continuous optimization with multiple processors
Spiral Optimization	Population	Balanced exploration and exploitation
Parallel Tempering	Population	Multi-modal landscapes with distinct basins
Genetic Algorithm	Population	Discrete/combinatorial optimization
Evolution Strategy	Population	Continuous optimization with noise
Differential Evolution	Population	Continuous non-linear optimization
CMA-ES	Population	Continuous optimization with adaptive covariance
Bayesian Optimization	SMBO	Expensive functions with continuous parameters
TPE	SMBO	Expensive functions, especially with conditionals
Forest Optimizer	SMBO	Large search spaces with discrete parameters

Categories

Local Search

Algorithms that explore the neighborhood of the current best solution. Fast and efficient for smooth functions, but may get stuck in local optima.

Algorithms: Hill Climbing, Stochastic Hill Climbing, Repulsing Hill Climbing, Simulated Annealing, Downhill Simplex

Local Search Algorithms

Global Search

Algorithms designed to explore the entire search space. Better at avoiding local optima but may be slower to converge.

Algorithms: Random Search, Grid Search, Random Restart, Random Annealing, Pattern Search, Powell’s Method, Lipschitz, DIRECT

Global Search Algorithms

Population-Based

Multiple search agents work together, sharing information about good regions. Natural parallelism and diverse exploration.

Algorithms: Particle Swarm, Spiral, Parallel Tempering, Genetic Algorithm, Evolution Strategy, Differential Evolution, CMA-ES

Population-Based Algorithms

Sequential Model-Based

Build a surrogate model of the objective function to predict promising regions. Ideal for expensive evaluations.

Algorithms: Bayesian Optimization, TPE, Forest Optimizer

Sequential Model-Based Optimization

How to Choose

Use this decision tree to narrow down the best algorithm category for your problem. For a more detailed guide, see Optimizer Selection.

Tip

Quick Decision Guide:

1. Is your objective function expensive to evaluate?

   • Yes: Use SMBO algorithms (Bayesian, TPE, Forest)

   • No: Continue to question 2

2. Do you have a good starting point?

   • Yes: Start with local search (Hill Climbing, Simulated Annealing)

   • No: Continue to question 3

3. Can you parallelize evaluations?

   • Yes: Use population-based (Particle Swarm, Genetic Algorithm)

   • No: Use global search (Random Search, Pattern Search)

Computational Cost

Algorithms differ in their computational overhead (time spent in the optimizer vs. evaluating the objective function):

Category	Overhead	Notes
Local Search	Very Low	Simple neighbor generation
Global Search	Very Low	Random/grid sampling
Population-Based	Low	Population management overhead
SMBO	High	Model training and prediction

For expensive objective functions (training ML models, running simulations), the overhead is negligible. For cheap functions (mathematical expressions), consider the simpler algorithms.