lunax.ensembles

This module implements a hill climbing algorithm for optimizing ensemble model weights.

class HillClimbingEnsemble

A class that implements hill climbing optimization for ensemble model weights.

Parameters:

• models (List[BaseModel]) – List of trained model instances inheriting from BaseModel

• metric (Union[str, List[str]]) – Evaluation metric(s) to optimize (‘accuracy’, ‘precision’, ‘recall’, ‘f1’, ‘auc’)

• maximize (bool) – Whether to maximize the metric (True) or minimize it (False)

• max_iter (int) – Maximum number of iterations for each random start

• step_size (float) – Step size for weight adjustments

• tolerance (float) – Convergence threshold

• n_random_starts (int) – Number of random initializations

• random_state (Optional[int]) – Random seed for reproducibility

Methods:

fit(X_val: pd.DataFrame, y_val: pd.Series) → np.ndarray

Optimize ensemble weights using hill climbing.

Parameters:

• X_val – Validation features

• y_val – Validation labels

Returns:

Optimal weights array

Return type:

np.ndarray

predict(X: pd.DataFrame) → np.ndarray

Make predictions using the optimized ensemble.

Parameters:

X – Input features

Returns:

Predicted labels

Return type:

np.ndarray

Raises:

ValueError – If model hasn’t been fitted

predict_proba(X: pd.DataFrame) → np.ndarray

Get probability predictions using the optimized ensemble.

Parameters:

X – Input features

Returns:

Predicted probabilities

Return type:

np.ndarray

Raises:

ValueError – If model hasn’t been fitted

Private Methods:

_get_ensemble_predictions(X: pd.DataFrame, weights: np.ndarray) → tuple[np.ndarray, np.ndarray]

Get weighted ensemble predictions.

Returns:

Tuple of (probabilities, labels)

_evaluate_weights(weights: np.ndarray, X: pd.DataFrame, y: pd.Series) → float

Evaluate current weight combination performance.

Returns:

Evaluation score

_get_neighbors(weights: np.ndarray) → List[np.ndarray]

Get neighboring weight combinations.

Returns:

List of neighbor weight arrays

Features:

• Supports multiple evaluation metrics

• Handles both binary and multi-class classification

• Implements weighted averaging of model predictions

• Maintains optimization history

• Ensures weights sum to 1 and are non-negative

Example Usage:

from lunax.ensembles import HillClimbingEnsemble

# Initialize with trained models
ensemble = HillClimbingEnsemble(
    models=[model1, model2, model3],
    metric=['accuracy', 'f1'],
    max_iter=100,
    n_random_starts=5
)

# Optimize weights
best_weights = ensemble.fit(X_val, y_val)

# Make predictions
predictions = ensemble.predict(X_test)
probabilities = ensemble.predict_proba(X_test)

Notes:

• The algorithm performs multiple random starts to avoid local optima

• Convergence is determined by the tolerance parameter

• Supports both single and multiple metric optimization

• Automatically handles probability and non-probability predictions

• Implements weighted model averaging for ensemble predictions