BayesianOptimizer

class BayesianOptimizer(search_space: dict[str, list], initialize: dict[Literal['grid', 'vertices', 'random', 'warm_start'], int | list[dict]] = None, constraints: list[callable] = None, random_state: int = None, rand_rest_p: float = 0, nth_process: int = None, warm_start_smbo: pd.DataFrame | None = None, max_sample_size: int = 10000000, sampling: dict[Literal['random'], int] = None, replacement: bool = True, gpr: object | None = None, xi: float = 0.03)[source]

Sequential model-based optimizer using Gaussian Process surrogate models.

Bayesian Optimization is a powerful technique for optimizing expensive black-box functions. It builds a probabilistic surrogate model (Gaussian Process) of the objective function and uses an acquisition function to determine the most promising points to evaluate next. This approach is sample-efficient, requiring fewer function evaluations than many other methods to find good solutions.

The algorithm works by: (1) fitting a Gaussian Process to observed data, (2) using the GP to predict mean and uncertainty at unobserved points, (3) selecting the next point to evaluate based on an acquisition function (Expected Improvement), and (4) updating the model with the new observation.

The algorithm is well-suited for:

  • Expensive objective functions (e.g., ML model training, simulations)

  • Low to moderate dimensional problems (typically < 20 dimensions)

  • Problems where sample efficiency is critical

  • Hyperparameter optimization of machine learning models

The xi parameter controls the exploration-exploitation trade-off in the Expected Improvement acquisition function. Higher values encourage more exploration of uncertain regions.

Parameters:
search_spacedict[str, list]

The search space to explore, defined as a dictionary mapping parameter names to arrays of possible values.

Each key is a parameter name (string), and each value is a numpy array or list of discrete values that the parameter can take. The optimizer will only evaluate positions that are on this discrete grid.

Example: A 2D search space with 100 points per dimension:

search_space = {
    "x": np.linspace(-10, 10, 100),
    "y": np.linspace(-10, 10, 100),
}

The resolution of each dimension (number of points in the array) directly affects optimization quality and speed. More points give finer resolution but increase the search space size exponentially.

initializedict[str, int], default={“vertices”: 4, “random”: 2}

Strategy for generating initial positions before the main optimization loop begins. Initialization samples are evaluated first, and the best one becomes the starting point for the optimizer.

Supported keys:

  • "grid": int – Number of positions on a regular grid.

  • "vertices": int – Number of corner/edge positions of the search space.

  • "random": int – Number of uniformly random positions.

  • "warm_start": list[dict] – Specific positions to evaluate, each as a dict mapping parameter names to values.

Multiple strategies can be combined:

initialize = {"vertices": 4, "random": 10}
initialize = {"warm_start": [{"x": 0.5, "y": 1.0}], "random": 5}

More initialization samples improve the starting point but consume iterations from n_iter. For expensive objectives, a few targeted warm-start points are often more efficient than many random samples.

constraintslist[callable], default=[]

A list of constraint functions that restrict the search space. Each constraint is a callable that receives a parameter dictionary and returns True if the position is valid, False if it should be rejected.

Rejected positions are discarded and regenerated: the optimizer resamples a new candidate position (up to 100 retries per step). During initialization, positions that violate constraints are filtered out entirely.

Example: Constrain the search to a circular region:

def circular_constraint(para):
    return para["x"]**2 + para["y"]**2 <= 25

constraints = [circular_constraint]

Multiple constraints are combined with AND logic (all must return True).

random_stateint or None, default=None

Seed for the random number generator to ensure reproducible results.

  • None: Use a new random state each run (non-deterministic).

  • int: Seed the random number generator for reproducibility.

Setting a fixed seed is recommended for debugging and benchmarking. Different seeds may lead to different optimization trajectories, especially for stochastic optimizers.

rand_rest_pfloat, default=0

Probability of performing a random restart instead of the normal algorithm step. At each iteration, a uniform random number is drawn; if it falls below rand_rest_p, the optimizer jumps to a random position instead of following its strategy.

  • 0.0: No random restarts (pure algorithm behavior).

  • 0.01-0.05: Light diversification, helps escape shallow local optima.

  • 0.1-0.3: Aggressive restarts, useful for highly multi-modal landscapes.

  • 1.0: Equivalent to random search.

This is especially useful for local search optimizers (Hill Climbing, Simulated Annealing) that can get trapped. For population-based optimizers, the effect is less pronounced since they already maintain diversity through multiple agents.

warm_start_smboobject or None, default=None

Previous SMBO state for warm-starting the surrogate model. Allows continuing optimization from a previous run by reusing the fitted model state, avoiding the cost of refitting from scratch.

Pass None to start fresh without warm-starting.

max_sample_sizeint, default=10000000

Maximum number of candidate points to consider when optimizing the acquisition function. The surrogate model predicts scores for these candidates, and the best one according to the acquisition function is selected for evaluation.

Larger values improve acquisition optimization quality but increase memory usage and computation time. For most problems, the default is more than sufficient. Reduce this if memory is a concern.

samplingdict, default={“random”: 1000000}

Configuration for how candidate points are generated for acquisition function optimization. The key specifies the sampling strategy and the value specifies the number of samples.

Currently supported: {"random": N} for uniform random sampling of N candidate points from the search space.

Example:

sampling = {"random": 500000}  # Fewer candidates, faster
replacementbool, default=True

Whether to sample candidate points with replacement when generating candidates for acquisition function optimization. When True, the same point can appear multiple times. When False, each candidate is unique, ensuring diversity but potentially slower for very large sample sizes.

gprobject or None, default=None

The Gaussian Process Regressor used as the surrogate model. Accepts three forms:

  • None: Uses the default GPR implementation.

  • Class: A GPR class that will be instantiated automatically.

  • Instance: A pre-configured GPR instance.

Custom GPR implementations should follow the scikit-learn API with fit(X, y) and predict(X, return_std=True) methods. This allows using different kernels, noise models, or entirely different GP libraries.

xifloat, default=0.03

Exploration-exploitation trade-off parameter for the Expected Improvement (EI) acquisition function. Controls how much the optimizer values uncertain regions over predicted-good regions.

  • 0.0: Pure exploitation, always samples where the GP predicts the best score. Tends to converge fast but may miss the global optimum.

  • 0.01-0.05: Mild exploration (default region). Good balance for most problems.

  • 0.1-0.3: Moderate exploration, favors uncertain regions.

  • 1.0+: Strong exploration, heavily prefers unexplored areas.

Higher xi is useful early in optimization or when the search space is large relative to the number of evaluations.

Attributes:
best_para

Return the best parameters found as a dictionary.

best_value

Return the best values found (raw parameter values).

search_data

Lazily construct and return the search results DataFrame.

Methods

search(objective_function, n_iter[, ...])

Run the optimization loop.

eval_time

init_stats

iter_time

See also

ForestOptimizer

SMBO using tree ensembles, scales better.

TreeStructuredParzenEstimators

SMBO using kernel density estimation.

EnsembleOptimizer

Combines multiple surrogate model types for robustness.

LipschitzOptimizer

Global optimization using Lipschitz continuity bounds.

Notes

The algorithm follows a sequential loop:

  1. Fit a Gaussian Process to all observed (position, score) pairs.

  2. For each candidate point, predict mean \(\\mu(x)\) and standard deviation \(\\sigma(x)\).

  3. Compute Expected Improvement:

\[\begin{split}EI(x) = (\\mu(x) - f_{\\text{best}} - \\xi) \\cdot \\Phi(Z) + \\sigma(x) \\cdot \\phi(Z)\end{split}\]

where \(Z = (\\mu(x) - f_{\\text{best}} - \\xi) / \\sigma(x)\), \(\\Phi\) is the standard normal CDF, and \(\\phi\) is the standard normal PDF.

  1. Evaluate the point with the highest EI.

The GP surrogate provides a principled balance between exploration (sampling where \(\\sigma(x)\) is high) and exploitation (sampling where \(\\mu(x)\) is high). Fitting the GP has complexity \(O(n^3)\) where n is the number of observations, which can become a bottleneck for large evaluation budgets.

For visual explanations and tuning guides, see the Bayesian Optimization user guide.

Examples

>>> import numpy as np
>>> from gradient_free_optimizers import BayesianOptimizer

>>> def expensive_function(para):
...     # Simulating an expensive evaluation
...     x, y = para["x"], para["y"]
...     return -((x - 0.5) ** 2 + (y - 0.5) ** 2)

>>> search_space = {
...     "x": np.linspace(0, 1, 100),
...     "y": np.linspace(0, 1, 100),
... }

>>> opt = BayesianOptimizer(search_space, xi=0.01)
>>> opt.search(expensive_function, n_iter=50)
search(objective_function: Callable[[dict[str, Any]], float], n_iter: int, max_time: float | None = None, max_score: float | None = None, early_stopping: dict[str, Any] | None = None, memory: bool = True, memory_warm_start: pd.DataFrame | None = None, verbosity: list[str] | Literal[False] = ['progress_bar', 'print_results', 'print_times'], optimum: Literal['maximum', 'minimum'] = 'maximum', callbacks: list[Callable[[CallbackInfo], bool | None]] | None = None, catch: dict[type[Exception], int | float] | None = None) → None[source]

Run the optimization loop.

Evaluates objective_function up to n_iter times, searching for the parameters that maximize (or minimize) the returned score. The search proceeds in two phases: an initialization phase that evaluates starting positions (controlled by the initialize constructor parameter), followed by an iteration phase where the optimizer’s strategy generates new candidate positions.

After the search finishes, results are available via best_para, best_score, and search_data.

Parameters:
objective_functioncallable

The function to optimize. Must accept a single dictionary mapping parameter names to values and return either:

  • A float score, or

  • A tuple (float, dict) where the second element contains custom metrics (accessible via callbacks and search_data).

Example:

def objective(params):
    return -(params["x"] ** 2 + params["y"] ** 2)

def objective_with_metrics(params):
    loss = params["x"] ** 2
    return -loss, {"loss": loss}
n_iterint

Total number of iterations (including initialization). Each iteration evaluates the objective function once (unless a cached result is found when memory=True).

max_timefloat or None, default=None

Maximum wall-clock time in seconds. The search stops after the current iteration if the elapsed time exceeds this limit. None means no time limit.

max_scorefloat or None, default=None

Target score threshold. The search stops when the best score found so far reaches or exceeds this value. When optimum="minimum", this refers to the original (non-negated) score. None means no score limit.

early_stoppingdict or None, default=None

Configuration for stopping the search when progress stalls. None disables early stopping. When provided, the dictionary supports the following keys:

  • "n_iter_no_change" (int, required): Stop if no improvement is observed for this many consecutive iterations.

  • "tol_abs" (float, optional): Minimum absolute improvement required over the window to count as progress.

  • "tol_rel" (float, optional): Minimum relative improvement (in percent) required over the window to count as progress.

Example:

early_stopping = {"n_iter_no_change": 50}
early_stopping = {"n_iter_no_change": 30, "tol_abs": 0.001}
memorybool, default=True

If True, cache objective function evaluations in an in-memory dictionary keyed by position. When the optimizer revisits a previously evaluated position, the cached score is returned without calling the objective function again. This is especially useful for discrete search spaces where revisits are common.

memory_warm_startpd.DataFrame or None, default=None

A DataFrame from a previous search (typically obtained via search_data) to pre-populate the evaluation cache. The DataFrame must contain columns matching the search space parameter names plus a "score" column. Requires memory=True.

Example:

opt1 = HillClimbingOptimizer(search_space)
opt1.search(objective, n_iter=50)

opt2 = HillClimbingOptimizer(search_space)
opt2.search(objective, n_iter=50,
            memory_warm_start=opt1.search_data)
verbositylist[str] or False, default=[“progress_bar”, “print_results”, “print_times”]

Controls console output during and after the search. Pass False or an empty list for silent operation.

Supported values:

  • "progress_bar": Show a live tqdm progress bar during the search.

  • "print_results": Print best score and best parameters after the search completes.

  • "print_times": Print timing breakdown (evaluation time, optimization overhead, throughput) after the search completes.

  • "print_search_stats": Print search statistics including iteration counts, acceptance rate, number of improvements, and longest plateau.

  • "print_statistics": Print score statistics (min, max, mean, std) after the search completes.

  • "debug_stop": Print detailed stopping condition debug info when the search terminates early.

optimum{“maximum”, “minimum”}, default=”maximum”

Whether to maximize or minimize the objective function. When set to "minimum", the objective function’s return value is negated internally so that the optimizer always maximizes. The reported best_score is in original (non-negated) units.

callbackslist[callable] or None, default=None

A list of callback functions invoked after each iteration. Each callback receives a single argument info with the following attributes:

  • info.iteration (int): Current iteration index (0-based).

  • info.score (float): Score from the current evaluation.

  • info.params (dict): Parameters evaluated in this iteration.

  • info.best_score (float): Best score found so far.

  • info.best_para (dict): Parameters of the best score.

  • info.n_iter (int): Total iterations planned.

  • info.phase (str): "init" or "iter".

  • info.elapsed_time (float): Seconds since search started.

  • info.metrics (dict): Custom metrics from the objective function (empty if the objective returns only a score).

  • info.convergence (list[float]): Best score at each iteration so far.

If any callback returns False, the search stops immediately. Any other return value (including None) is ignored and the search continues.

Example:

def log_progress(info):
    if info.iteration % 10 == 0:
        print(f"Iter {info.iteration}: best={info.best_score:.4f}")

def stop_early(info):
    if info.best_score > 0.99:
        return False  # stops the search

opt.search(objective, n_iter=100,
           callbacks=[log_progress, stop_early])
catchdict[type, float] or None, default=None

Error handling for the objective function. Maps exception types to fallback scores. When the objective function raises a caught exception, the optimizer records the fallback score instead of crashing. Exception subclasses are matched via isinstance, so {Exception: ...} catches all.

The fallback score is in the user’s original units (before any negation from optimum="minimum"). Use float('nan') or float('inf') to mark positions as invalid without inventing an artificial score.

Example:

catch = {ValueError: -1000, RuntimeError: float('nan')}

opt.search(objective, n_iter=100, catch=catch)

Examples

Basic usage with default settings:

>>> import numpy as np
>>> from gradient_free_optimizers import HillClimbingOptimizer
>>> def objective(para):
...     return -(para["x"] ** 2)
>>> search_space = {"x": np.linspace(-10, 10, 100)}
>>> opt = HillClimbingOptimizer(search_space)
>>> opt.search(objective, n_iter=30)

Using multiple stopping conditions:

>>> opt.search(
...     objective,
...     n_iter=1000,
...     max_time=60,
...     max_score=-0.01,
...     early_stopping={"n_iter_no_change": 50},
... )
property best_para[source]

Return the best parameters found as a dictionary.

Uses the Converter to transform the best position into user-friendly parameter names and values.

Returns:
dict or None

Dictionary mapping parameter names to their best values, or None if no evaluation has been performed yet.

property best_value[source]

Return the best values found (raw parameter values).

Returns:
list or None

List of best values in parameter order, or None if no evaluation has been performed yet.

property search_data: pd.DataFrame[source]

Lazily construct and return the search results DataFrame.

The DataFrame is only built when this property is accessed, avoiding a large memory spike at the end of high-dimensional optimizations. The result is cached so subsequent accesses don’t rebuild it.