Hypermodels, Search Spaces & Search Strategies

Hyperparameters vs. Learned Weights

Keras Tuner automates hyperparameter search for Keras models. A hyperparameter (learning rate, layer count, units per layer) is a configuration value set before training, controlling how the model is built or trained — unlike weights, which are learned automatically via gradient descent. Manually trying combinations doesn't scale: possibilities grow combinatorially, and each combination needs a full, often expensive training run to evaluate.

Defining a Hypermodel

def build_model(hp):
    model = keras.Sequential()
    units = hp.Int('units', min_value=32, max_value=512, step=32)
    model.add(keras.layers.Dense(units=units, activation='relu'))
    model.add(keras.layers.Dense(10, activation='softmax'))
    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
    return model

The hypermodel-building function defines architecture as a function of hyperparameters — Keras Tuner calls it repeatedly with different suggested values (hp.Int, hp.Choice, hp.Float) to construct and evaluate candidate models.

Search Strategies

Grid search exhaustively tries every combination — full coverage, but scales combinatorially poorly. Random search samples randomly instead, often more efficient than grid search when only a few hyperparameters really matter. Bayesian optimization builds a probabilistic model from prior trial results to intelligently pick the next combination to try, needing fewer total trials than an uninformed search. Hyperband allocates resources adaptively — starting many candidates with a small budget (few epochs), then giving more resources only to the most promising, discarding poor performers early.