Mixed Search Spaces

Gradient-Free-Optimizers natively supports continuous, discrete, categorical, and SciPy distribution-backed dimensions in a single search space. There is no need to encode or transform parameter types – the library detects the dimension type from the object you provide and applies appropriate optimization logic for each type internally.

Dimension Types

A search space is a Python dictionary mapping parameter names to dimension definitions. A dimension can be a tuple for a continuous range, a NumPy array for a discrete grid, a Python list for categorical choices, or an optional SciPy distribution object.

Continuous – a tuple (lower, upper):

"learning_rate": (0.001, 1.0)

Discrete – integer or evenly spaced numerical values, typically created with np.arange:

# Integers: 10, 20, 30, ..., 200
"n_estimators": np.arange(10, 210, 10)

Categorical – strings or other non-numeric values, typically provided as a list:

# Categorical choices
"kernel": ["linear", "rbf", "poly"]

Boolean – a special case of categorical with two values:

"use_bias": [True, False]

Distribution – a SciPy stats continuous distribution. SciPy is optional, and this type is available when the user passes a SciPy distribution object:

from scipy import stats

"learning_rate": stats.loguniform(1e-5, 1e-1)

Mixing Types Freely

All dimension types can coexist in a single search space dictionary. The optimizer handles each dimension according to its type, so you can define mixed-type problems naturally:

import numpy as np
from scipy import stats
from gradient_free_optimizers import BayesianOptimizer

# Mixed search space for SVM hyperparameter tuning
search_space = {
    "C": (0.01, 100.0),                         # continuous
    "gamma": stats.loguniform(1e-5, 1e-1),      # distribution
    "degree": np.arange(2, 6),                    # discrete
    "kernel": ["linear", "rbf", "poly"],          # categorical
    "shrinking": [True, False],                    # boolean
}

def objective(para):
    from sklearn.svm import SVC
    from sklearn.model_selection import cross_val_score
    from sklearn.datasets import load_iris

    X, y = load_iris(return_X_y=True)
    clf = SVC(
        C=para["C"],
        degree=para["degree"],
        kernel=para["kernel"],
        shrinking=para["shrinking"],
    )
    return cross_val_score(clf, X, y, cv=3).mean()

opt = BayesianOptimizer(search_space)
opt.search(objective, n_iter=50)
print(opt.best_para)

Granularity

The optimizer samples from the values you provide, so the array length controls the resolution of each dimension. More values mean finer granularity but a larger search space:

# Coarse: 10 values
"x": np.linspace(-10, 10, 10)    # step size = 2.22

# Fine: 1000 values
"x": np.linspace(-10, 10, 1000)  # step size = 0.02

For discrete parameters like n_estimators, the step size in np.arange directly controls the granularity.

Why Native Mixed-Type Support Matters

Many optimization libraries require all dimensions to be the same type, or force you to encode categoricals as integers. GFO uses dimension-type-aware routing internally: the optimization logic that generates new candidate positions adapts its strategy per dimension. Continuous dimensions use perturbation-based moves, while categorical dimensions use swap-based moves. Distribution dimensions are optimized in quantile space and converted to ppf values before objective evaluation. This means:

• No manual encoding or decoding of categorical parameters

• The optimizer uses moves that make sense for each type

• Results are returned using the original values you defined