Gradient Free Optimizers

Lightweight optimization with local, global, population-based and sequential techniques across mixed search spaces

Gradient-Free-Optimizers provides a unified interface to gradient-free optimization algorithms for discrete, continuous, and mixed search spaces. It covers local, global, population-based, and sequential model-based techniques with minimal dependencies.

Why Gradient-Free?

Not every optimization problem has gradients. Machine learning hyperparameter tuning, simulation optimization, feature selection, and black-box optimization all require algorithms that can navigate parameter spaces without derivative information.

Gradient-Free-Optimizers provides a unified interface to 23 optimization algorithms, from simple hill climbing to sophisticated Bayesian optimization, all designed for discrete, continuous, and mixed search spaces.

Algorithm Categories

Local Search

5 Algorithms for exploiting promising regions

Hill Climbing, Simulated Annealing, Downhill Simplex, and more. These algorithms excel at fine-tuning solutions in the neighborhood of good starting points.

Local Search Algorithms

Global Search

8 Algorithms for broad exploration

Random Search, Grid Search, Pattern Search, Powell’s Method, Lipschitz Optimization, and DIRECT. Designed for thorough coverage of the entire search space.

Global Search Algorithms

Population-Based

7 Algorithms using collective intelligence

Particle Swarm, Genetic Algorithm, Evolution Strategy, Differential Evolution, CMA-ES, and more. Multiple agents work together to find optimal solutions.

Population-Based Algorithms

Sequential Model-Based

3 Algorithms that learn from evaluations

Bayesian Optimization, TPE, and Forest Optimizer. Build surrogate models to predict promising regions and balance exploration vs. exploitation.

Sequential Model-Based Optimization

Key Features

Simple Search Space Definition

Define search spaces with tuples, NumPy arrays, lists, or optional SciPy distributions. Supports discrete, continuous, and mixed spaces.

search_space = {
    "x": np.linspace(-10, 10, 100),
    "y": np.arange(1, 100),
    "method": ["adam", "sgd", "rmsprop"],
}

Constraint Support

Define constraints as simple Python functions. The optimizer automatically respects constraints during search, retrying invalid positions.

constraints = [
    lambda p: p["x"] + p["y"] < 100,
    lambda p: p["x"] > 0,
]
opt = Optimizer(space, constraints=constraints)

Memory & Warm Starts

Cache expensive function evaluations automatically. Continue previous searches using memory_warm_start with a DataFrame of past evaluations.

opt.search(objective, memory=True)

# Later, continue with warm start
opt.search(objective,
          memory_warm_start=previous_data)

Multiple Stopping Conditions

Stop optimization based on iteration count, wall-clock time, target score, or early stopping when no improvement is found. Combine conditions as needed.

opt.search(
    objective,
    n_iter=1000,
    max_time=3600,
    max_score=0.99,
    early_stopping={"n_iter_no_change": 50}
)

Flexible Initialization

Control how the search starts: grid sampling, random points, vertices/corners, or specific warm-start positions you define.

opt = Optimizer(
    search_space,
    initialize={
        "grid": 4,
        "random": 2,
        "vertices": 4,
        "warm_start": [{"x": 0, "y": 50}],
    }
)

Quick Install

pip install gradient-free-optimizers

Quick Start

Basic Example

import numpy as np
from gradient_free_optimizers import HillClimbingOptimizer

def objective(para):
    return -(para["x"] ** 2 + para["y"] ** 2)

search_space = {
    "x": np.linspace(-10, 10, 100),
    "y": np.linspace(-10, 10, 100),
}

opt = HillClimbingOptimizer(search_space)
opt.search(objective, n_iter=1000)

print(opt.best_para)   # Best parameters found
print(opt.best_score)  # Best score achieved

Hyperparameter Tuning

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_iris
from gradient_free_optimizers import BayesianOptimizer

X, y = load_iris(return_X_y=True)

def objective(para):
    clf = RandomForestClassifier(
        n_estimators=para["n_estimators"],
        max_depth=para["max_depth"],
        min_samples_split=para["min_samples_split"],
    )
    return cross_val_score(clf, X, y, cv=5).mean()

search_space = {
    "n_estimators": np.arange(10, 200, 10),
    "max_depth": np.arange(2, 20),
    "min_samples_split": np.arange(2, 20),
}

opt = BayesianOptimizer(search_space)
opt.search(objective, n_iter=50)

print(f"Best accuracy: {opt.best_score:.3f}")
print(f"Best params: {opt.best_para}")

Optimization Ecosystem

Gradient-Free-Optimizers is part of a broader optimization ecosystem for Python.

Gradient-Free-Optimizers

You are here

Pure optimization algorithms with minimal dependencies. Perfect when you need fine-grained control over the optimization process.

23 algorithms
Mixed search spaces
Constraint support

Hyperactive

Higher-level optimization

Built on GFO, adding features for ML hyperparameter tuning: multi-process optimization, experiment tracking, and integration with scikit-learn, PyTorch, and more.