Optimizer Selection

GFO provides 23 optimization algorithms organized into four categories. Each category uses a fundamentally different search strategy, making some algorithms better suited to certain problem types than others. Local search algorithms exploit promising regions efficiently but can get stuck in local optima. Global search algorithms explore broadly but may be slow to converge. Population-based methods maintain multiple candidate solutions simultaneously. Sequential model-based optimization (SMBO) builds a surrogate model of the objective function to make informed decisions about where to search next.

Local Search

Hill Climbing, Simulated Annealing, Downhill Simplex. Best for exploiting promising regions.

Global Search

Random Search, Grid Search, Pattern Search, DIRECT. Best for broad exploration.

Population-Based

Particle Swarm, Genetic Algorithm, Evolution Strategy. Best for parallel evaluation.

Sequential Model-Based

Bayesian, TPE, Forest. Best for expensive objective functions.

Decision Tree

Algorithm selection flowchart

Follow the decision tree to find the best algorithm for your problem.

Text version of the decision tree
Is your objective function expensive (> 1 second)?
├── Yes → Use SMBO (Bayesian, TPE, Forest)
│         └── Continuous params? → BayesianOptimizer
│         └── Many categoricals? → TreeStructuredParzenEstimators
│         └── Many iterations? → ForestOptimizer
│
└── No → Continue...

Do you have a good starting point?
├── Yes → Use Local Search
│         └── Smooth function? → HillClimbingOptimizer
│         └── Multiple local optima? → SimulatedAnnealingOptimizer
│
└── No → Continue...

Can you evaluate in parallel?
├── Yes → Use Population-Based
│         └── Continuous? → ParticleSwarmOptimizer
│         └── Discrete? → GeneticAlgorithmOptimizer
│
└── No → Use Global Search
          └── Need baseline? → RandomSearchOptimizer
          └── Small space? → GridSearchOptimizer
          └── Unknown landscape? → DirectAlgorithm

By Problem Type

Hyperparameter Tuning (ML)

  • First choice: BayesianOptimizer (efficient with expensive training)

  • With many categoricals: TreeStructuredParzenEstimators

  • Large search spaces: ForestOptimizer

  • Quick baseline: RandomSearchOptimizer

Continuous Optimization

  • Smooth, unimodal: HillClimbingOptimizer

  • Multi-modal: SimulatedAnnealingOptimizer or ParticleSwarmOptimizer

  • Need global guarantee: DirectAlgorithm

Discrete/Combinatorial

  • Feature selection: GeneticAlgorithmOptimizer

  • Mixed spaces: TreeStructuredParzenEstimators

  • Small discrete space: GridSearchOptimizer

Unknown Landscape

  • Start with RandomSearchOptimizer to explore

  • Then refine with BayesianOptimizer or SimulatedAnnealingOptimizer

By Evaluation Budget

Budget

Recommended Optimizers

< 50 iterations

Bayesian, TPE (learn quickly)

50-200 iterations

Bayesian, Forest, SimulatedAnnealing

200-1000 iterations

Forest, ParticleSwarm, EvolutionStrategy

> 1000 iterations

Any (cheaper algorithms work well)

By Function Cost

Very cheap (< 0.01s per evaluation)

  • Use simple algorithms: Hill Climbing, Random Search

  • SMBO overhead may dominate

Cheap (0.01s - 1s)

  • Most algorithms work well

  • Population-based gives good coverage

Expensive (1s - 60s)

  • SMBO algorithms (Bayesian, TPE, Forest)

  • Enable memory caching

Very expensive (> 60s)

  • Bayesian Optimization is most sample-efficient

  • Consider parallelization with Hyperactive

Algorithm Characteristics

Algorithm

Exploration

Exploitation

Notes

Hill Climbing

Low

High

Fast, may get stuck

Simulated Annealing

Medium

High

Good general-purpose

Random Search

High

Low

Great baseline

Particle Swarm

Medium

Medium

Good for parallelism

Genetic Algorithm

High

Medium

Good for discrete

Bayesian

Adaptive

Adaptive

Best for expensive

TPE

Adaptive

Adaptive

Handles categoricals well

Starting Recommendation

If unsure, this progression often works well:

  1. RandomSearchOptimizer (10% of budget) - establish baseline

  2. BayesianOptimizer or SimulatedAnnealingOptimizer (90% of budget) - optimize

Or for very limited budgets:

  1. BayesianOptimizer with xi=0.1 (higher exploration) for first half

  2. BayesianOptimizer with xi=0.01 (higher exploitation) for second half