Machine learning/Deep Learning, with its ever-evolving algorithms and complex models, has evolved into a powerful tool in the hands of Data Scientists and AI practitioners. The process of training a Deep Learning model is intricate and often unpredictable. It requires careful tuning and monitoring. Sometimes we set some parameters for training the model and end up with the model being overfitted. This is where the callback functions come into play, serving as the secret sauce that empowers us to own our model training game.

In this article, we are going to explore the essential role of callbacks in controlling and optimizing the training of deep learning models, exploring their significance, usage, and how to implement them using Keras. This guide will help you master the art of callbacks and optimize your training process.

Understanding Callbacks

Callbacks are a widely used concept in most programming languages. It is a concept of passing functions/methods or pieces of code as an argument to other functions or methods. These functions are intended to be executed at a later time, often in response to a specific event or condition, such as completing a task or handling user interaction.

Callbacks in Deep Learning work on the same principle enabling us to monitor, adapt, and optimize the training process of our models. Keras provides built-in API to their callback object which has some amazing callback methods used for a variety of tasks. They come to the rescue when we need to implement real-time actions based on the model’s performance. The possibilities with callbacks are endless, and they play a pivotal role in ensuring that your Deep Learning model evolves into a powerful asset.

These Callback methods can be passed to Keras methods including fit, evaluate, and predict as a list in order to hook into the various stages of the model training and inference lifecycle.

model.fit(
    train_data, 
    epochs=2, 
    batch_size=32,
    callbacks=[early_stopping]
)

This was a basic introduction to Keras Callbacks, Now let’s dive into these callbacks and their practical implementations.

Built-in Callbacks in Keras

Keras provides some built-in Callbacks that help to boost the training process. This includes early stopping, Model Checkpoint, and Learning Rate Scheduler. Let's check out all the amazing callbacks from Keras and their usage.

EarlyStopping

One of the most common challenges in training machine learning or deep learning models is overfitting. EarlyStopping is a callback that allows us to address this issue effectively. By monitoring the model’s performance on a validation set, this callback can halt the training process when no improvement is observed, preventing overfitting and saving valuable time and resources.

from keras.callbacks import EarlyStopping

early_stopping = EarlyStopping(
    monitor="val_loss",
    min_delta=0,
    patience=5,
    verbose=0,
    mode="auto",
    baseline=None,
    restore_best_weights=False,
    start_from_epoch=0,
)

Arguments

• monitor: This parameter specifies the quantity to be monitored for improvements. It can be metrics like “val_loss,” “val_accuracy,” etc.

• min_delta: Minimum change required in the monitored quantity to be considered for improvement. If the change is less than min_delta, it won’t count as an improvement.

• patience: The number of epochs with no improvement after which training will be stopped. For instance, if patience=5, training will stop if there is no improvement for 5 consecutive epochs.

• verbose: It controls the verbosity mode. Use 0 for silent mode, which means no messages are displayed. Set it to 1 for mode where messages are displayed when the callback takes action.

• mode:{"auto", "min", "max"}. In min mode, training stops when the monitored quantity stops decreasing; In "max" mode, it stops when the monitored quantity stops increasing; In "auto" mode, the direction is automatically determined based on the name of the monitored metric.

• baseline: Baseline value for the monitored quantity. Training will stop if the model doesn’t show improvement over this baseline.

• restore_best_weights: When set to True, it restores model weights from the epoch with the best value of the monitored quantity. If set to False, the model uses weights obtained at the last step of training, regardless of performance relative to the baseline.

• start_from_epoch: This parameter allows you to specify the number of warm-up epochs before monitoring for improvement. During this period, no improvement is expected, and training won’t be stopped.

ModelCheckpoint

Saving the model’s weights at regular intervals is crucial to ensure that your progress is not lost. ModelCheckpoint comes to the rescue by saving the model’s weights in a structured manner. This allows you to resume training from where you left off, ensuring continuity in your Deep Learning journey.

from keras.callbacks import ModelCheckpoint 

model_checkpoint = ModelCheckpoint(
    filepath,
    monitor: "val_loss",
    verbose: 0,
    save_best_only: False,
    save_weights_only: False,
    mode: "auto",
    save_freq="epoch",
    options=None,
    initial_value_threshold=None,
    **kwargs
)

Arguments

• filepath: file path to save the model weights.

• monitor: This parameter specifies the quantity to be monitored for improvements. It can be metrics like “val_loss,” “val_accuracy,” etc.

• verbose: Controls the verbosity mode. Use 0 for silent, and 1 for displaying messages when the callback takes action.

• save_best_only: For True, the callback saves only the best model (based on the monitored quantity). If set False, it saves the model at every checkpoint according to the specified frequency.

• save_weights_only: For True, it saves only the model’s weights. If set to False, it saves the entire model.

• mode:{"auto", "min", "max"}. In min mode, training stops when the monitored quantity stops decreasing; In "max" mode, it stops when the monitored quantity stops increasing; In "auto" mode, the direction is automatically determined based on the name of the monitored metric.

• save_freq: Determines how often to save the model. It can be set to “epoch” (saves at the end of each epoch) or an integer value (saves every n batches or steps).

• options: Additional options for saving the model, such as compression options.

• initial_value_threshold: To skip some initial epochs based on some initial value for monitored quantity, so as to save the model when the monitored quantity when threshold exceeds

• **kwargs: Additional keyword arguments that are passed to the underlying object for backward compatibility.

TensorBoard

Visualization is key to understanding your model’s behavior. TensorBoardCallback helps you create visual representations of training progress. By monitoring metrics like loss and accuracy, you can spot issues and optimize your model training effectively.

from keras.callbacks import TensorBoard 

tensorboard_callback = TensorBoard(
    log_dir="logs",
    histogram_freq=0,
    write_graph=True,
    write_images=False,
    write_steps_per_second=False,
    update_freq="epoch",
    profile_batch=0,
    embeddings_freq=0,
    embeddings_metadata=None,
    **kwargs
)

Arguments

• log_dir: Path to the directory where the TensorBoard logs will be saved

• histogram_freq: It controls how frequently histograms of your tensors are computed for visualization. Set to 0 for no histograms or a positive integer to define the frequency.

• write_graph: True, saves the computation graph for visualization in TensorBoard.

• write_image: True, save images like model architecture graphs and feature maps.False, these images won't be saved.

• write_steps_per_second: True, records the number of steps per second, giving you insights into training speed. Useful for performance monitoring.

• update_freq: Determines how often the logs are updated. It can be set to "epoch" (updating once per epoch) or a positive integer for a different frequency.

• profile_batch: allows you to profile the performance of your model. Setting 0 will disable profiling, while a positive integer specifies the batch to profile.

• embeddings_freq: Set to a positive integer to determine how frequently the embeddings are computed and visualized.

• embeddings_metadata: A file path to metadata that describes your embeddings, providing context for visualization.

• **kwargs: Additional keyword arguments that are passed to the underlying object for backward compatibility.

ReduceLROnPlateau

Learning rate optimization is a critical aspect of training deep learning models. ReduceLROnPlateau automatically adjusts the learning rate when the validation loss plateaus. This dynamic adaptation ensures your model converges efficiently.

from keras.callbacks import ReduceLROnPlateau 

reduce_lr = ReduceLROnPlateau(
    monitor="val_loss",
    factor=0.1,
    patience=10,
    verbose=0,
    mode="auto",
    min_delta=0.0001,
    cooldown=0,
    min_lr=0,
    **kwargs
)

Arguments

• monitor: This parameter specifies the quantity to be monitored for learning rate reduction. It can be metrics like “val_loss,” “val_accuracy,” etc.

• factor: The factor by which the learning rate will be reduced. For example, if is set to 0.1, the learning rate will be reduced to 10% of its current value when the condition is met.

• patience: The number of epochs with no improvement after which the learning rate will be reduced.

• verbose: Controls the verbosity mode. Use 0 for silent, and 1 for displaying messages when the callback takes action.

• mode:{"auto", "min", "max"}. In min mode, training stops when the monitored quantity stops decreasing; In "max" mode, it stops when the monitored quantity stops increasing; In "auto" mode, the direction is automatically determined based on the name of the monitored metric.

• min_delta: Minimum change required in the monitored quantity to be considered for improvement. If the change is less than min_delta, it won’t count as an improvement.

• cooldown: The No. of epochs to wait after a learning rate reduction before resuming normal monitoring. During this “cooldown” period, learning rate changes won’t occur.

• min_lr: The lower limit on the learning rate. If the learning rate falls below this value, it won’t be reduced further.

• **kwargs: Additional keyword arguments that are passed to the underlying object for backward compatibility.

LearningRateScheduler

Sometimes, standard learning rate schedules may not be suitable for your specific problem. LearningRateScheduler gives you the flexibility to craft custom learning rate schedules tailored to your model’s needs.

from keras.callbacks import LearningRateScheduler

def custom_schedule(epoch):
    if epoch < 10:
        return 0.001
    else:
        return 0.0001

lr_scheduler = LearningRateScheduler(custom_schedule)

Arguments

• schedule: A custom function with the user-defined rule that takes an epoch index (integer) and current learning rate (float) as parameters and returns the new learning rate.

• verbose: Controls the verbosity mode. Use 0 for silent, and 1 for displaying messages when the callback takes action.

TerminateOnNaN

It’s crucial to maintain data integrity during training. TerminateOnNaN is a simple yet effective callback that stops training if NaN values appear in your model’s outputs, acting as a data guardian to keep your training on track.

from keras.callbacks import TerminateOnNaN

terminate_on_nan = TerminateOnNaN()

Creating Custom Callbacks

While pre-built callbacks are incredibly useful, there are situations where you might need a callback tailored to your specific needs. Fortunately, Keras provides a straightforward way to create custom callbacks. Keras has an Abstract base class that can be used to build new callbacks. This base class is parent to each callback method in Keras.

keras.callbacks.Callback()

Let’s dive into how you can do it:

Step 1: Define Your Custom Callback Class

You can create a custom callback by defining a Python class that inherits the Keras base class. This class should include methods for the events you want to monitor and customize.

Here’s a basic structure to get you started:

class CustomCallback(keras.callbacks.Callback):
    def on_train_begin(self, logs=None):
        # Called at the beginning of training
        pass
    def on_epoch_end(self, epoch, logs=None):
        # Called at the end of each epoch
        pass    # You can add more event-specific methods here

Step 2: Implement Your Callback Logic

Within your custom callback class, you can implement your specific logic for each callback event. For example, you might want to save the model’s weights at the end of every epoch. Here’s how you can achieve this:

class CustomCallback(keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs=None):
        model.save_weights(f"model_weights_epoch_{epoch}.h5")

This code will save the model’s weights to a file at the end of each epoch.

Step 3: Incorporate Your Custom Callback

Once you’ve defined your custom callback, you can use it by passing an instance of your class to the callback parameter when fitting your model. For example:

model.fit(
    x_train, y_train,
    batch_size=32, epochs=10,
    callbacks=[CustomCallback()]
)

Your custom callback will now be executed during training, and you can tailor it to perform actions that suit your project’s needs.

Creating custom callbacks gives you the flexibility to track and manipulate various aspects of the training process, making it a powerful tool in your deep learning toolkit.

You have seen several Keras callback methods for different tasks. These callback methods will boost your training process, saving time and resources. So, what are you waiting for? Start implementing these powerful Keras callbacks in your projects and witness the transformation of your deep learning journey. Your AI-powered solutions are just a callback away from greatness!