Introduction

This guide will help you install Gradient-Free-Optimizers and run your first optimization in under 5 minutes.

Quick Installation

Install from PyPI using pip:

pip install gradient-free-optimizers

Requirements:

  • Python 3.10+

  • NumPy and pandas

SciPy, tqdm, and scikit-learn are optional extras for distribution-backed dimensions, progress bars, and alternative surrogate models.

Optional dependencies:

For surrogate model-based optimizers with sklearn estimators:

pip install gradient-free-optimizers[sklearn]

Your First Optimization

Let’s optimize a simple 2D function to find its minimum:

import numpy as np
from gradient_free_optimizers import HillClimbingOptimizer

# 1. Define your objective function
# The optimizer will try to MAXIMIZE this score
def objective(para):
    x = para["x"]
    y = para["y"]
    # Negative because we want to minimize x^2 + y^2
    return -(x**2 + y**2)

# 2. Define the search space
# Each dimension is a NumPy array of possible values
search_space = {
    "x": np.linspace(-10, 10, 100),
    "y": np.linspace(-10, 10, 100),
}

# 3. Create an optimizer and run the search
opt = HillClimbingOptimizer(search_space)
opt.search(objective, n_iter=1000)

# 4. Get the results
print(f"Best parameters: {opt.best_para}")
print(f"Best score: {opt.best_score}")

Expected output:

Best parameters: {'x': 0.0, 'y': 0.0}
Best score: -0.0

Tip

By default, GFO maximizes the objective function. To minimize, either:

  • Return the negative of your function (as shown above)

  • Use optimum="minimum" in search()

Understanding Search Spaces

Search spaces in GFO are defined as dictionaries where:

  • Keys are parameter names

  • Values are tuples, NumPy arrays, lists, or optional SciPy distributions

search_space = {
    # Continuous range
    "learning_rate": (0.001, 1.0),

    # Discrete integers: 10, 20, 30, ..., 200
    "n_estimators": np.arange(10, 210, 10),

    # Categorical choices
    "optimizer": ["adam", "sgd", "rmsprop"],

    # Boolean choices
    "use_bias": [True, False],
}

The optimizer samples from these arrays, so the granularity of your array determines how precisely you can tune each parameter.

Choosing an Optimizer

Here’s a quick guide to help you choose:

Scenario

Recommended

Why

Fast baseline

RandomSearchOptimizer

No overhead, establishes baseline performance

Smooth functions

HillClimbingOptimizer

Fast, effective for convex problems

Many local optima

SimulatedAnnealingOptimizer

Can escape local optima via temperature

Expensive evaluations

BayesianOptimizer

Learns from past evaluations

Large populations

ParticleSwarmOptimizer

Parallel exploration of search space

Systematic coverage

GridSearchOptimizer

Guarantees coverage of search space

Real-World Example: Hyperparameter Tuning

Here’s a practical example tuning a Random Forest classifier:

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_iris
from gradient_free_optimizers import BayesianOptimizer

# Load data
X, y = load_iris(return_X_y=True)

# Define objective: return cross-validation accuracy
def objective(para):
    clf = RandomForestClassifier(
        n_estimators=para["n_estimators"],
        max_depth=para["max_depth"],
        min_samples_split=para["min_samples_split"],
        random_state=42,
    )
    return cross_val_score(clf, X, y, cv=5).mean()

# Define search space
search_space = {
    "n_estimators": np.arange(10, 200, 10),
    "max_depth": np.arange(2, 20),
    "min_samples_split": np.arange(2, 20),
}

# Run Bayesian optimization
opt = BayesianOptimizer(search_space)
opt.search(
    objective,
    n_iter=50,
    verbosity=["progress_bar", "print_results"],
)

# Results
print(f"\nBest accuracy: {opt.best_score:.4f}")
print(f"Best parameters: {opt.best_para}")

# Access all evaluations as a DataFrame
print(f"\nAll evaluations:")
print(opt.search_data.head())

Next Steps

User Guide

Deep dive into all features: constraints, memory, stopping conditions, initialization strategies, and more.

User Guide
All Algorithms

Detailed documentation for all 23 optimization algorithms with visualizations and parameter guides.

Optimization Algorithms
Examples

Complete code examples for various optimization scenarios from simple functions to ML hyperparameter tuning.

Examples
API Reference

Full API documentation with all parameters and return types.

API Reference