Deep Learning from Scratch Series: Transfer Learning with TensorFlow

1. Introduction

Deep learning is a powerful and popular branch of machine learning that can solve complex problems such as image recognition, natural language processing, and speech synthesis. However, deep learning models often require a lot of data and computational resources to train, which can be challenging and expensive for many applications.

What if you could leverage the knowledge and skills of an existing deep learning model that has been trained on a large and diverse dataset, and adapt it to your own specific problem? This is the idea behind transfer learning, a technique that allows you to reuse a pretrained model and fine-tune it for a new task.

In this blog, you will learn how to use transfer learning with TensorFlow, a popular and powerful framework for building and deploying machine learning models. You will apply transfer learning to a computer vision problem, where you will use a pretrained model to classify images of flowers. You will learn how to:

• Load and preprocess the data
• Choose a pretrained model
• Fine-tune the model
• Evaluate and test the model

By the end of this blog, you will have a better understanding of what transfer learning is, how it works, and how to use it with TensorFlow. You will also have a working code example that you can modify and experiment with for your own projects. Let’s get started!

2. What is Transfer Learning?

Transfer learning is a machine learning technique that allows you to use the knowledge and skills of an existing model that has been trained on a large and diverse dataset, and adapt it to a new task that has less data or different characteristics. Transfer learning can save you time and resources, as you don’t have to train a model from scratch.

But how does transfer learning work? The idea is to take a pretrained model that has been trained on a general task, such as image classification, and use it as a starting point for a more specific task, such as flower recognition. The pretrained model has already learned some useful features and patterns from the large dataset, such as edges, shapes, colors, and textures. You can then fine-tune the model by adjusting some of its parameters or adding new layers to make it more suitable for your new task.

There are different types of transfer learning, depending on how much you want to modify the pretrained model and how similar your new task is to the original one. We will discuss these types in the next section. For now, you can think of transfer learning as a way of reusing and customizing an existing model for your own problem.

Why would you want to use transfer learning? What are the benefits and challenges of this technique? Let’s find out in the following section.

2.1. Types of Transfer Learning

As we mentioned in the previous section, transfer learning is a technique that allows you to use the knowledge and skills of an existing model that has been trained on a large and diverse dataset, and adapt it to a new task that has less data or different characteristics. However, not all transfer learning scenarios are the same. Depending on how much you want to modify the pretrained model and how similar your new task is to the original one, you can choose from different types of transfer learning.

The most common types of transfer learning are:

• Feature extraction: In this type, you use the pretrained model as a feature extractor, and add a new classifier layer on top of it. You freeze the weights of the pretrained model, and only train the new layer on your new task. This type is suitable when your new task is similar to the original one, but has a different output space. For example, if you want to use a pretrained model that can classify 1000 types of objects, and adapt it to a task that can classify 10 types of flowers, you can use feature extraction.
• Fine-tuning: In this type, you use the pretrained model as a starting point, and modify some or all of its parameters to make it more suitable for your new task. You can also add new layers or remove existing ones. This type is suitable when your new task is different from the original one, but has a similar input space. For example, if you want to use a pretrained model that can classify 1000 types of objects, and adapt it to a task that can detect faces, you can use fine-tuning.
• Domain adaptation: In this type, you use the pretrained model as a source of knowledge, and try to adapt it to a new domain that has different characteristics. You can use various techniques, such as adding domain-specific layers, applying regularization, or using adversarial learning, to make the model more robust to the domain shift. This type is suitable when your new task is similar to the original one, but has a different input distribution. For example, if you want to use a pretrained model that can classify images taken in daylight, and adapt it to a task that can classify images taken in low-light conditions, you can use domain adaptation.

How do you decide which type of transfer learning to use? There is no definitive answer, as it depends on many factors, such as the availability and quality of data, the complexity and similarity of the tasks, and the computational resources and time constraints. However, a general guideline is to start with feature extraction, and then try fine-tuning or domain adaptation if the performance is not satisfactory.

In this blog, we will focus on fine-tuning, as it is one of the most widely used and effective types of transfer learning. In the next section, we will discuss the benefits and challenges of fine-tuning, and how to overcome them.

2.2. Benefits and Challenges of Transfer Learning

Transfer learning is a powerful and convenient technique that can help you solve complex problems with less data and resources. However, it also comes with some benefits and challenges that you need to be aware of. In this section, we will discuss some of the advantages and disadvantages of using transfer learning, and how to overcome them.

Some of the benefits of transfer learning are:

• Improved performance: Transfer learning can improve the performance of your model, as it can leverage the knowledge and skills of a pretrained model that has been trained on a large and diverse dataset. This can help your model learn faster and better, especially when your new task has limited or noisy data.
• Reduced training time: Transfer learning can reduce the training time of your model, as it can reuse the parameters of a pretrained model that has already been optimized. This can save you computational resources and costs, as you don’t have to train a model from scratch.
• Increased generalization: Transfer learning can increase the generalization of your model, as it can expose your model to a variety of features and patterns from the pretrained model. This can help your model avoid overfitting and adapt to new situations and domains.

Some of the challenges of transfer learning are:

• Model selection: Transfer learning requires you to choose a suitable pretrained model for your new task. This can be tricky, as you need to consider factors such as the similarity and complexity of the tasks, the availability and quality of the data, and the architecture and size of the model. You also need to decide how much to modify the pretrained model, such as which layers to freeze or unfreeze, which layers to add or remove, and which parameters to update or keep.
• Model adaptation: Transfer learning requires you to adapt the pretrained model to your new task. This can be challenging, as you need to balance between retaining the useful features and patterns from the pretrained model, and learning the new features and patterns from your new task. You also need to avoid negative transfer, which is when the pretrained model hinders the performance of your new task, due to the differences or conflicts between the tasks.
• Model evaluation: Transfer learning requires you to evaluate the performance of your model on your new task. This can be difficult, as you need to define appropriate metrics and benchmarks to measure the effectiveness and efficiency of your model. You also need to compare your model with other models, such as baseline models or state-of-the-art models, to assess the impact and value of transfer learning.

How can you overcome these challenges? There is no one-size-fits-all solution, as it depends on your specific problem and goals. However, some general tips are to:

• Do your research: Before choosing a pretrained model, do some research on the existing models and their applications. You can use online resources, such as TensorFlow Hub, to find and explore various pretrained models for different tasks and domains. You can also read the papers and blogs of the model developers and users, to learn more about their methods and results.
• Experiment and iterate: After choosing a pretrained model, experiment and iterate with different settings and techniques. You can use tools, such as TensorBoard, to monitor and visualize the training process and the model performance. You can also use techniques, such as hyperparameter tuning, regularization, data augmentation, and learning rate scheduling, to optimize and fine-tune your model.
• Test and validate: After fine-tuning your model, test and validate it on your new task. You can use methods, such as cross-validation, confusion matrix, and error analysis, to evaluate and improve the accuracy and robustness of your model. You can also use methods, such as speed test, memory test, and power test, to evaluate and improve the efficiency and scalability of your model.

By following these tips, you can make the most of transfer learning and achieve better results with less effort. In the next section, we will show you how to use transfer learning with TensorFlow, and apply it to a computer vision problem.

3. How to Use Transfer Learning with TensorFlow

In this section, we will show you how to use transfer learning with TensorFlow, and apply it to a computer vision problem. We will use a pretrained model called MobileNetV2, which is a lightweight and efficient model for image classification. We will fine-tune this model to classify images of flowers, using a dataset called tf_flowers, which contains 3670 images of five types of flowers: daisy, dandelion, roses, sunflowers, and tulips.

To use transfer learning with TensorFlow, we will follow these steps:

1. Load and preprocess the data
2. Choose a pretrained model
3. Fine-tune the model
4. Evaluate and test the model

We will explain each step in detail in the following sections. Before we start, we need to import some libraries and modules that we will use throughout the tutorial. You can run the following code to import them:

# Import TensorFlow and TensorFlow Hub
import tensorflow as tf
import tensorflow_hub as hub

# Import other libraries
import numpy as np
import matplotlib.pyplot as plt
import PIL.Image as Image

Now that we have imported the necessary libraries and modules, we can proceed to the first step: load and preprocess the data.

3.1. Load and Preprocess the Data

The first step of using transfer learning with TensorFlow is to load and preprocess the data. In this tutorial, we will use a dataset called tf_flowers, which contains 3670 images of five types of flowers: daisy, dandelion, roses, sunflowers, and tulips. The dataset is available in the TensorFlow Datasets catalog, which is a collection of ready-to-use datasets for TensorFlow.

To load the dataset, we can use the tfds.load function, which returns a tf.data.Dataset object. A tf.data.Dataset object is a collection of elements, each of which consists of one or more components. In our case, each element is an image and its corresponding label. We can specify the name of the dataset, the split of the data, and the shuffle option. We can also use the with_info argument to get some information about the dataset, such as the number of examples and the class names. Here is an example of how to load the dataset:

# Load the tf_flowers dataset
import tensorflow_datasets as tfds
dataset, info = tfds.load('tf_flowers', split='train', shuffle_files=True, with_info=True)

# Print some information about the dataset
print(info)

The output of the code above should look something like this:

tfds.core.DatasetInfo(
    name='tf_flowers',
    full_name='tf_flowers/3.0.1',
    description="""
    A large set of images of flowers
    """,
    homepage='https://www.tensorflow.org/datasets/catalog/tf_flowers',
    data_path='C:\\Users\\user\\tensorflow_datasets\\tf_flowers\\3.0.1',
    download_size=218.21 MiB,
    dataset_size=221.83 MiB,
    features=FeaturesDict({
        'image': Image(shape=(None, None, 3), dtype=tf.uint8),
        'label': ClassLabel(shape=(), dtype=tf.int64, num_classes=5),
    }),
    supervised_keys=('image', 'label'),
    splits={
        'train': ,
    },
    citation="""@ONLINE {tensorflowflowers,
    author = "The TensorFlow Team",
    title = "Flowers",
    month = "jan",
    year = "2019",
    url = "http://download.tensorflow.org/example_images/flower_photos.tgz" }
    """,
)

As you can see, the dataset has 3670 images in the train split, and the labels are numbers from 0 to 4, representing the five types of flowers. We can also see the shape and dtype of the image and label components, and the citation of the dataset.

Now that we have loaded the dataset, we need to preprocess it to make it ready for our model. Preprocessing the data involves tasks such as resizing, cropping, normalizing, augmenting, and batching the images. We can use the tf.data.Dataset API to perform these tasks efficiently and easily. The tf.data.Dataset API provides various methods to transform and manipulate the data, such as map, filter, shuffle, repeat, and batch. We can chain these methods to create a data pipeline that suits our needs. Here is an example of how to preprocess the dataset:

# Define some constants
IMAGE_SIZE = 224 # The size of the input image for the model
BATCH_SIZE = 32 # The size of the batch for training
AUTOTUNE = tf.data.AUTOTUNE # To optimize the data pipeline

# Define a function to resize and normalize the images
def preprocess_image(image, label):
  # Resize the image to the desired size
  image = tf.image.resize(image, [IMAGE_SIZE, IMAGE_SIZE])
  # Normalize the image to the range [0, 1]
  image = image / 255.0
  # Return the image and the label
  return image, label

# Define a function to augment the images
def augment_image(image, label):
  # Randomly flip the image horizontally
  image = tf.image.random_flip_left_right(image)
  # Randomly adjust the brightness of the image
  image = tf.image.random_brightness(image, 0.2)
  # Randomly adjust the contrast of the image
  image = tf.image.random_contrast(image, 0.8, 1.2)
  # Return the image and the label
  return image, label

# Preprocess the dataset
dataset = dataset.map(preprocess_image, num_parallel_calls=AUTOTUNE) # Apply the preprocess_image function
dataset = dataset.map(augment_image, num_parallel_calls=AUTOTUNE) # Apply the augment_image function
dataset = dataset.shuffle(1000) # Shuffle the data
dataset = dataset.repeat() # Repeat the data indefinitely
dataset = dataset.batch(BATCH_SIZE) # Batch the data
dataset = dataset.prefetch(AUTOTUNE) # Prefetch the data for faster consumption

The code above defines two functions: preprocess_image and augment_image. The preprocess_image function resizes and normalizes the images to the range [0, 1], which is the expected input range for the model. The augment_image function randomly applies some transformations to the images, such as flipping, brightness adjustment, and contrast adjustment, to increase the diversity and robustness of the data. The code then applies these functions to the dataset using the map method, which applies a given function to each element of the dataset. The code also uses the num_parallel_calls argument to enable parallel processing of the data, and the AUTOTUNE constant to let TensorFlow decide the optimal number of parallel calls.

The code then shuffles, repeats, batches, and prefetches the data using the corresponding methods of the tf.data.Dataset API. The shuffle method randomly shuffles the order of the data, which can improve the generalization of the model. The repeat method repeats the data indefinitely, which can ensure that the model has enough data to train on. The batch method groups the data into batches of a given size, which can speed up the training process and reduce the memory usage. The prefetch method prefetches the data for faster consumption, which can reduce the idle time of the model and the data pipeline.

By applying these methods, we have created a data pipeline that can efficiently and effectively feed the data to our model. We can now proceed to the next step: choose a pretrained model.

3.2. Choose a Pretrained Model

The second step of using transfer learning with TensorFlow is to choose a pretrained model for your new task. In this tutorial, we will use a pretrained model called MobileNetV2, which is a lightweight and efficient model for image classification. MobileNetV2 is based on an architecture called inverted residual with linear bottleneck, which uses depthwise separable convolutions and skip connections to reduce the number of parameters and computations, while preserving the accuracy and performance. MobileNetV2 has been trained on a large and diverse dataset called ImageNet, which contains 1.4 million images of 1000 classes.

Why did we choose MobileNetV2 as our pretrained model? There are several reasons, such as:

• Relevance: MobileNetV2 is relevant for our new task, as it can perform image classification, which is similar to our flower recognition problem. MobileNetV2 has also learned some useful features and patterns from ImageNet, such as edges, shapes, colors, and textures, which can be transferred to our new task.
• Efficiency: MobileNetV2 is efficient for our new task, as it has a small size and a fast speed, which can make the training and inference process faster and easier. MobileNetV2 has only 3.4 million parameters and can run at 300 frames per second on a mobile device, which is impressive for a deep learning model.
• Availability: MobileNetV2 is available for our new task, as it can be easily accessed and downloaded from TensorFlow Hub, which is a repository of pretrained models for TensorFlow. TensorFlow Hub provides various versions of MobileNetV2, with different input sizes and output features, which can suit different needs and preferences.

How can we use MobileNetV2 as our pretrained model? We can use the hub.KerasLayer class, which is a wrapper that allows us to use a TensorFlow Hub model as a Keras layer. A Keras layer is a basic building block of a Keras model, which can perform some operations on the input data and produce some output data. We can use the hub.KerasLayer class to load a MobileNetV2 model from TensorFlow Hub, and use it as a feature extractor or a classifier for our new task. Here is an example of how to use the hub.KerasLayer class:

# Import TensorFlow Hub
import tensorflow_hub as hub

# Define the URL of the MobileNetV2 model
# We use the version with 224x224 input size and 1001 output classes
# We also set the trainable argument to False, as we only want to use the model as a feature extractor
URL = "https://tfhub.dev/google/tf2-preview/mobilenet_v2/feature_vector/4"
FEATURE_EXTRACTOR = hub.KerasLayer(URL, input_shape=(IMAGE_SIZE, IMAGE_SIZE, 3), trainable=False)

# Define the URL of the MobileNetV2 model
# We use the version with 224x224 input size and 1000 output classes
# We also set the trainable argument to True, as we want to fine-tune the model for our new task
URL = "https://tfhub.dev/google/tf2-preview/mobilenet_v2/classification/4"
CLASSIFIER = hub.KerasLayer(URL, input_shape=(IMAGE_SIZE, IMAGE_SIZE, 3), trainable=True)

The code above defines two variables: FEATURE_EXTRACTOR and CLASSIFIER. The FEATURE_EXTRACTOR variable is a hub.KerasLayer object that loads a MobileNetV2 model from TensorFlow Hub, and uses it as a feature extractor. The feature extractor takes an image as input, and produces a 1280-dimensional vector as output, which represents the features and patterns of the image. The feature extractor is not trainable, as we only want to use the existing knowledge and skills of the model, and not modify its parameters. The CLASSIFIER variable is a hub.KerasLayer object that loads a MobileNetV2 model from TensorFlow Hub, and uses it as a classifier. The classifier takes an image as input, and produces a 1000-dimensional vector as output, which represents the probabilities of the image belonging to each of the 1000 classes. The classifier is trainable, as we want to fine-tune the model for our new task, and update its parameters.

By using the hub.KerasLayer class, we have chosen a pretrained model for our new task, and made it ready for fine-tuning. We can now proceed to the next step: fine-tune the model.

3.3. Fine-Tune the Model

Now that you have chosen a pretrained model, you can fine-tune it for your new task. Fine-tuning is the process of adjusting some of the parameters or layers of the pretrained model to make it more suitable for your specific problem. Fine-tuning can improve the performance of the model and help it learn the features and patterns that are relevant for your task.

How do you fine-tune a model with TensorFlow? There are different ways to do it, depending on how much you want to modify the pretrained model and how similar your new task is to the original one. Here are some common scenarios:

• If your new task is very similar to the original one, you can just replace the last layer of the pretrained model with a new one that matches the number of classes in your new task. For example, if you are using a pretrained model that was trained on ImageNet (a dataset with 1000 classes) and you want to classify flowers (a dataset with 5 classes), you can just change the last layer to have 5 output units instead of 1000. You can then train the new layer with your data, while keeping the rest of the model frozen.
• If your new task is somewhat different from the original one, you can unfreeze some of the top layers of the pretrained model and train them with your data, along with the new layer. This way, you can fine-tune the model to learn more specific features and patterns that are relevant for your task. For example, if you are using a pretrained model that was trained on ImageNet and you want to classify dogs (a dataset with 120 classes), you can unfreeze the last few convolutional layers and train them with your data, along with the new layer. You can use a lower learning rate to avoid overfitting and damaging the learned features.
• If your new task is very different from the original one, you can unfreeze most or all of the layers of the pretrained model and train them with your data, along with the new layer. This way, you can fine-tune the model to learn completely new features and patterns that are relevant for your task. For example, if you are using a pretrained model that was trained on ImageNet and you want to classify text (a dataset with 2 classes), you can unfreeze all the layers and train them with your data, along with the new layer. You can use a very low learning rate to avoid overfitting and damaging the learned features.

In this tutorial, we will use the first scenario, as our new task is very similar to the original one. We will replace the last layer of the pretrained model with a new one that has 5 output units, and train it with our flower data, while keeping the rest of the model frozen. Here is how you can do it with TensorFlow:

# Load the pretrained model
pretrained_model = tf.keras.applications.MobileNetV2(input_shape=(224, 224, 3),
                                                     include_top=False,
                                                     weights='imagenet')

# Freeze the model
pretrained_model.trainable = False

# Add a new layer
model = tf.keras.Sequential([
  pretrained_model,
  tf.keras.layers.GlobalAveragePooling2D(),
  tf.keras.layers.Dense(5, activation='softmax')
])

# Compile the model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Train the model
model.fit(train_data, epochs=10, validation_data=test_data)

Congratulations! You have successfully fine-tuned a pretrained model with TensorFlow. You can now evaluate and test the model to see how well it performs on your new task. Let’s do that in the next section.

3.4. Evaluate and Test the Model

After fine-tuning the model, you can evaluate and test its performance on your new task. Evaluating the model means measuring how well it performs on the validation data, which is a subset of the data that the model has not seen during training. Testing the model means making predictions on the test data, which is another subset of the data that the model has not seen during training or validation. Evaluating and testing the model can help you assess its accuracy, generalization, and robustness.

How do you evaluate and test the model with TensorFlow? There are different ways to do it, depending on how you want to measure the performance of the model and how you want to present the results. Here are some common methods:

• If you want to measure the overall accuracy of the model, you can use the evaluate method of the model, which returns the loss and the accuracy on the validation data. For example, you can run the following code to evaluate the model on the test data:

# Evaluate the model
loss, accuracy = model.evaluate(test_data)
print(f'Loss: {loss:.4f}, Accuracy: {accuracy:.4f}')

• If you want to measure the accuracy of the model for each class, you can use the classification_report function from the sklearn.metrics module, which returns the precision, recall, f1-score, and support for each class. For example, you can run the following code to generate a classification report for the model on the test data:

# Import the module
from sklearn.metrics import classification_report

# Get the predictions and the labels
predictions = model.predict(test_data)
predictions = np.argmax(predictions, axis=1)
labels = test_data.labels

# Generate the classification report
report = classification_report(labels, predictions, target_names=['daisy', 'dandelion', 'rose', 'sunflower', 'tulip'])
print(report)

• If you want to visualize the confusion matrix of the model, which shows how many times each class was correctly or incorrectly predicted, you can use the plot_confusion_matrix function from the sklearn.metrics module, which returns a plot of the confusion matrix. For example, you can run the following code to plot the confusion matrix for the model on the test data:

# Import the module
from sklearn.metrics import plot_confusion_matrix

# Plot the confusion matrix
plot_confusion_matrix(model, test_data, display_labels=['daisy', 'dandelion', 'rose', 'sunflower', 'tulip'])
plt.show()

By evaluating and testing the model, you can see how well it performs on your new task and identify its strengths and weaknesses. You can also use the results to improve the model or try different models and compare their performance. In the next section, we will conclude this blog and summarize the main points.

4. Conclusion

In this blog, you learned how to use transfer learning with TensorFlow and apply it to a computer vision problem. You learned what transfer learning is, how it works, and how to use it with TensorFlow. You also learned how to load and preprocess the data, choose a pretrained model, fine-tune the model, and evaluate and test the model.

Transfer learning is a powerful and useful technique that allows you to reuse and customize an existing model for your own problem. Transfer learning can save you time and resources, as you don’t have to train a model from scratch. Transfer learning can also improve the performance of the model and help it learn the features and patterns that are relevant for your task.

TensorFlow is a popular and powerful framework for building and deploying machine learning models. TensorFlow provides many tools and functions that make transfer learning easy and convenient. TensorFlow also supports many pretrained models that you can use for transfer learning, such as MobileNetV2, ResNet50, and BERT.

We hope you enjoyed this blog and learned something new and useful. If you want to learn more about transfer learning, TensorFlow, or computer vision, you can check out the following resources:

• Transfer learning and fine-tuning – A TensorFlow tutorial on transfer learning and fine-tuning for image classification.
• Image retraining – A TensorFlow Hub tutorial on how to retrain an image classifier with TensorFlow Hub.
• Classify text with BERT – A TensorFlow tutorial on how to use BERT for text classification.
• tf.keras.applications – A TensorFlow module that provides implementations of various pretrained models.
• TensorFlow Hub – A repository of reusable machine learning assets, such as pretrained models, datasets, and tutorials.

Thank you for reading this blog and happy learning!