Convolutional Denoising Autoencoders for image noise reduction

By | November 2, 2020
2 Comments

Autoencoders are unsupervised Deep Learning techniques that are extensively used for dimensionality reduction, latent feature learning (Learning Representations), and also as generative models (Generative Adversarial Networks: GANs). Denoising Autoencoders are slight modifications to the vanilla autoencoders that can be used for reducing noise from real-world noisy datasets. In this tutorial, we will investigate Convolutional Denoising Autoencoders to reduce noise from the images.

Autoencoders have proved to be very useful in learning complex representations of data and are utilized to solve many complex real-world problems. In my previous article on ‘Autoencoders in Keras and Deep Learning‘, I have given a basic overview of autoencoders using Keras in python.

This tutorial will focus on Convolutional Denoising Autoencoders where we will train a denoising autoencoder from scratch using Keras and TensorFlow. Here is a quick peek into the content-

  1. Convolutional Denoising Autoencoders
  2. Data Preparation
  3. Training Convolutional Denoising Autoencoder
  4. Results
  5. Summary
Convolutional Denoising Autoencoders for image noise reduction
Convolutional Denoising Autoencoders for image noise reduction

1. Convolutional Denoising Autoencoders

Background

Autoencoders are typically used for representing data in lower-dimensional space(latent feature space), usually for dimensionality reduction tasks. They achieve this task by first converting input data into a lower-dimensional encoding (bottle-neck) and then take that encoding to reconstruct the original data as an output. In this way, the model learns the important aspects of input data as latent features(ignoring non-important information) such that it is able to recreate it efficiently.

Denoising Autoencoders are slightly different in working. They learn to capture only useful information from input data (ignoring noise) by changing the reconstruction criterion. Denoising Autoencoders are used for reducing noise from noisy datasets. They take noisy data points as input and reconstruct the clean version as output thus the input data and output data are not the same in the case of the Denoising Autoencoders.

This kind of training (noisy data -> clean data), helps the model to learn only valuable features(ignoring the noise) in latent space, and thus it re-constructs the clean version. This kind of setup is utilized in multiple applications for cleaning real-world noisy datasets.

Denoising Autoencoder will only be able to remove noise from the dataset when the following two conditions are true-

  1. Original Features of data are stable and robust to noise.
  2. The model should be able to learn the useful structure from the distribution of the dataset.

Convolutional Denoising Autoencoders

Denoising autoencoders work well in multiple different domains(or application areas) with slight modifications depending upon the kind of dataset is being fed. In this tutorial, we will learn about one such variant called convolutional denoising autoencoders which are utilized for denoising image data.

These models take a noisy image as input and extract important features using multiple convolutional operations, these latent features are then passed to a set of de-convolutional layers in order to reconstruct the cleaner version of this image with the same height and width. Deconvolution operations can be performed with something like Conv2DTransponse implementations from TensorFlow and Keras.

Note: Latent feature dimensions needn’t be smaller in the case of denoising autoencoders as inputs and outputs are different. Thus there is no risk of learning identity function (An identity function basically copies the input into output without learning important aspects of data).

2. Data Preparation

We will train a denoising autoencoder on MNIST handwritten digits dataset available through Keras. Here is how we can download and load the dataset in our Python notebook-

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline

from tensorflow.keras.datasets import mnist

(trainX, trainy), (testX, testy) = mnist.load_data()

print('Training data shapes: X=%s, y=%s' % (trainX.shape, trainy.shape))
print('Testing data shapes: X=%s, y=%s' % (testX.shape, testy.shape))

for j in range(5):
    i = np.random.randint(0, 10000)
    plt.subplot(550 + 1 + j)
    plt.imshow(trainX[i], cmap='gray')
    plt.title(trainy[i])
plt.show()

MNIST handwritten digits dataset comes with 60K labeled handwritten digit images in the training partition and 10K images in the test partition. Here is a glimpse of 5 images from the dataset. Each image is of 28 * 28 dimensions.

Convolutional Denoising Autoencoders for image noise reduction
Convolutional Denoising Autoencoders for image noise reduction

Creating Noisy Data

As our task is to learn a denoising autoencoder, we will need noisy images (for corresponding clean images) for training the network. We will manually add four different kinds of noise into our training and test images.

Here are the python scripts capable of adding following four different kinds of noise to the input images-

  1. Gaussian Noise
  2. Salt and Pepper Noise
  3. Poisson Noise
  4. Speckle Noise
# Adding Noise to the dataset
def guassian_noise(image):
    r,c= image.shape
    mean = 0
    var = 0.1
    sigma = var**0.5
    gaussian = np.random.normal(mean,sigma,(r,c))
    gaussian = gaussian.reshape(r,c)
    noisy = image + gaussian
    return noisy

def salt_and_pepper_noise(image):
    ratio = 0.9
    amount = 0.1
    noisy = np.copy(image)
    
    salt_count = np.ceil(amount * image.size * ratio)
    coords = [np.random.randint(0, i - 1, int(salt_count)) for i in image.shape]
    noisy[coords] = 1

    pepper_count = np.ceil(amount* image.size * (1. - ratio))
    coords = [np.random.randint(0, i - 1, int(pepper_count)) for i in image.shape]
    noisy[coords] = 0
    return noisy

def poisson_noise(image):
    vals = len(np.unique(image))
    vals = 2 ** np.ceil(np.log2(vals))
    noisy = np.random.poisson(image * vals) / float(vals)
    return noisy

def speckle_noise(image):
    r,c = image.shape
    speckle = np.random.randn(r,c)
    speckle = speckle.reshape(r,c)        
    noisy = image + image * speckle
    return noisy    

def add_noise(image):
    p = np.random.random()
    if p <= 0.25:
        #print("Guassian")
        noisy = guassian_noise(image)
    elif p <= 0.5:
        #print("SnP")
        noisy = salt_and_pepper_noise(image)
    elif p <= 0.75:
        #print("Poison")
        noisy = poisson_noise(image)
    else:
        #print("speckle")
        noisy = speckle_noise(image)
    return noisy

Let’s checkout some corrupted samples with randomly chosen noise as per above python functions-

print ("Corrupted Example Samples")
for j in range(9):
    i = np.random.randint(0, 10000)
    plt.subplot(330 + 1 + j)
    noisy = add_noise(trainX[i]/255)
    plt.imshow(noisy, cmap='gray')
plt.show()

We can see that the images have been fairly corrupted with multiple different types of corruption. This kind of corruption is often observed in real-world images. We will focus on removing this noise and obtaining a cleaner version of each image.

Convolutional Denoising Autoencoders for image noise reduction
Convolutional Denoising Autoencoders for image noise reduction

Data Preparation

We will prepare training and test datasets separately for clean and noisy versions of images. Noisy images will be fed as input and the reconstructed images would be compared with the cleaner versions of the corresponding images. Images are reshaped to add an extra dimension for channels (as per the Convolutional layers input format defined in Keras and TensorFlow).

train_clean = [image/255 for image in trainX]
test_clean = [image/255 for image in testX]

train_noisy = [add_noise(image/255) for image in trainX]
test_noisy = [add_noise(image/255) for image in testX]

train_clean = np.reshape(train_clean, (60000, 28, 28, 1))
test_clean = np.reshape(test_clean, (10000, 28, 28, 1))

train_noisy = np.reshape(train_noisy, (60000, 28, 28, 1))
test_noisy = np.reshape(test_noisy, (10000, 28, 28, 1))

print (train_clean.shape, train_noisy.shape, test_clean.shape, test_noisy.shape)

(60000, 28, 28, 1) (60000, 28, 28, 1) (10000, 28, 28, 1) (10000, 28, 28, 1)

3. Training Convolutional Denoising Autoencoder

In this section, we will define the architecture and train it on the prepared dataset.

Architecture

The model starts with an input layer that takes a batch of images as input with each image having dimensions=(28 * 28 * 1). The first few layers(encoder part) are composed of Convolutional layers followed by MaxPooling layers. The encoder part of the model downsamples the image across width and height and extracts features.

These extracted features from the encoder part are passed into the decoder part as input where the decoder reconstructs the image of the original dimensions. The decoder part is consists of multiple De-convolution layers, each followed by an upsampling layer.

Here is the python implementation of the model-

import tensorflow

input_data = tensorflow.keras.layers.Input(shape=(28, 28, 1))

#Encoder part
encoder = tensorflow.keras.layers.Conv2D(64, (5,5), activation='relu')(input_data)
encoder = tensorflow.keras.layers.MaxPooling2D((2,2))(encoder)

encoder = tensorflow.keras.layers.Conv2D(128, (3,3), activation='relu')(encoder)
encoder = tensorflow.keras.layers.MaxPooling2D((2,2))(encoder)

encoder = tensorflow.keras.layers.Conv2D(256, (3,3), activation='relu')(encoder)
encoder = tensorflow.keras.layers.MaxPooling2D((2,2))(encoder)

#Decoder Part
decoder = tensorflow.keras.layers.Conv2DTranspose(256, (3,3), activation='relu')(encoder)

decoder = tensorflow.keras.layers.Conv2DTranspose(128, (3,3), activation='relu')(decoder)
decoder = tensorflow.keras.layers.UpSampling2D((2,2))(decoder)

decoder = tensorflow.keras.layers.Conv2DTranspose(64, (3,3), activation='relu')(decoder)
decoder = tensorflow.keras.layers.UpSampling2D((2,2))(decoder)

decoded = tensorflow.keras.layers.Conv2DTranspose(1, (5,5), activation='relu')(decoder)

Loss

The final task is to compare the reconstructed image with the cleaner version of the input image. As these images are represented with pixel values, we can use mean squared error(MSE) to quantify the goodness of the autoencoder. We will use the Adam optimizer that will tune the model weights such that MSE is reduced.

autoencoder = tensorflow.keras.models.Model(inputs=input_data, outputs=decoded)
autoencoder.compile(loss='mse', optimizer='adam')
autoencoder.summary()

Here is the model summary-

Model: "functional_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 28, 28, 1)]       0         
_________________________________________________________________
conv2d (Conv2D)              (None, 24, 24, 64)        1664      
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 12, 12, 64)        0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 10, 10, 128)       73856     
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 5, 5, 128)         0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 3, 3, 256)         295168    
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 1, 1, 256)         0         
_________________________________________________________________
conv2d_transpose (Conv2DTran (None, 3, 3, 256)         590080    
_________________________________________________________________
conv2d_transpose_1 (Conv2DTr (None, 5, 5, 128)         295040    
_________________________________________________________________
up_sampling2d (UpSampling2D) (None, 10, 10, 128)       0         
_________________________________________________________________
conv2d_transpose_2 (Conv2DTr (None, 12, 12, 64)        73792     
_________________________________________________________________
up_sampling2d_1 (UpSampling2 (None, 24, 24, 64)        0         
_________________________________________________________________
conv2d_transpose_3 (Conv2DTr (None, 28, 28, 1)         1601      
=================================================================
Total params: 1,331,201
Trainable params: 1,331,201
Non-trainable params: 0

Training

We will train the model on training data where noisy images are passed as input and clean images are passed as the output of the model. The test dataset is kept as validation data in order to check the MSE on test batches. The model is trained for 30 epochs with a batch of 64 images and validated on the complete test dataset. After 30 epochs of training, the validation loss comes down to 0.0069.

autoencoder.fit(train_noisy, train_clean, epochs=30, batch_size=64, validation_data=(test_noisy, test_clean))

Epoch 1/30
938/938 [==============================] - 84s 90ms/step - loss: 0.0244 - val_loss: 0.0135
Epoch 2/30
938/938 [==============================] - 75s 80ms/step - loss: 0.0119 - val_loss: 0.0107
Epoch 3/30
938/938 [==============================] - 78s 84ms/step - loss: 0.0100 - val_loss: 0.0095
Epoch 4/30
938/938 [==============================] - 80s 85ms/step - loss: 0.0090 - val_loss: 0.0088
Epoch 5/30
938/938 [==============================] - 80s 85ms/step - loss: 0.0084 - val_loss: 0.0087
Epoch 6/30
938/938 [==============================] - 80s 85ms/step - loss: 0.0080 - val_loss: 0.0083
Epoch 7/30
938/938 [==============================] - 80s 85ms/step - loss: 0.0076 - val_loss: 0.0078
Epoch 8/30
938/938 [==============================] - 80s 85ms/step - loss: 0.0073 - val_loss: 0.0078
Epoch 9/30
938/938 [==============================] - 78s 84ms/step - loss: 0.0071 - val_loss: 0.0079
Epoch 10/30
938/938 [==============================] - 79s 84ms/step - loss: 0.0069 - val_loss: 0.0076
Epoch 11/30
938/938 [==============================] - 79s 84ms/step - loss: 0.0068 - val_loss: 0.0073
Epoch 12/30
938/938 [==============================] - 79s 84ms/step - loss: 0.0066 - val_loss: 0.0074
Epoch 13/30
938/938 [==============================] - 78s 84ms/step - loss: 0.0065 - val_loss: 0.0074
Epoch 14/30
938/938 [==============================] - 79s 84ms/step - loss: 0.0064 - val_loss: 0.0074
Epoch 15/30
938/938 [==============================] - 79s 84ms/step - loss: 0.0063 - val_loss: 0.0070
Epoch 16/30
938/938 [==============================] - 79s 85ms/step - loss: 0.0062 - val_loss: 0.0072
Epoch 17/30
938/938 [==============================] - 79s 85ms/step - loss: 0.0061 - val_loss: 0.0071
Epoch 18/30
938/938 [==============================] - 79s 84ms/step - loss: 0.0060 - val_loss: 0.0071
Epoch 19/30
938/938 [==============================] - 79s 84ms/step - loss: 0.0059 - val_loss: 0.0071
Epoch 20/30
938/938 [==============================] - 78s 84ms/step - loss: 0.0059 - val_loss: 0.0069
Epoch 21/30
938/938 [==============================] - 75s 80ms/step - loss: 0.0058 - val_loss: 0.0070
Epoch 22/30
938/938 [==============================] - 76s 81ms/step - loss: 0.0057 - val_loss: 0.0070
Epoch 23/30
938/938 [==============================] - 77s 83ms/step - loss: 0.0057 - val_loss: 0.0069
Epoch 24/30
938/938 [==============================] - 84s 90ms/step - loss: 0.0056 - val_loss: 0.0071
Epoch 25/30
938/938 [==============================] - 78s 83ms/step - loss: 0.0056 - val_loss: 0.0069
Epoch 26/30
938/938 [==============================] - 76s 81ms/step - loss: 0.0056 - val_loss: 0.0072
Epoch 27/30
938/938 [==============================] - 75s 80ms/step - loss: 0.0055 - val_loss: 0.0068
Epoch 28/30
938/938 [==============================] - 76s 81ms/step - loss: 0.0055 - val_loss: 0.0070
Epoch 29/30
938/938 [==============================] - 80s 85ms/step - loss: 0.0054 - val_loss: 0.0071
Epoch 30/30
938/938 [==============================] - 80s 86ms/step - loss: 0.0054 - val_loss: 0.0069

4. Results

In the last section, we have trained our denoising autoencoder reasonably. In this section, we will check how it performs. We will pass a set of noisy images from the test dataset for prediction and plot the results (model output) for them.

Here is the python code for inference on test dataset-

offset=92
# print ("Real Test Images")
# # Real Images
# for i in range(9):
#     plt.subplot(330 + 1 + i)
#     plt.imshow(test_clean[i+offset,:,:, -1], cmap='gray')
# plt.show()

print("Noisy test images")
for i in range(9):
    plt.subplot(330 + 1 + i)
    plt.imshow(test_noisy[i+offset,:,:, -1], cmap='gray')
plt.show()

# Reconstructed Images
print ("Cleaned Version(Denoising Autoencoder):) ")
for i in range(9):
    plt.subplot(330 + 1 + i)
    output = autoencoder.predict(np.array([test_noisy[i+offset]]))
    op_image = np.reshape(output[0]*255, (28, 28))
    plt.imshow(op_image, cmap='gray')
plt.show()
Convolutional Denoising Autoencoders for image noise reduction
Convolutional Denoising Autoencoders for image noise reduction

The above output shows that our Convolutional Denoising autoencoder is doing a good job at learning important features and ignoring the noise from the corrupted images. The input images were fairly corrupted and the model was able to reconstruct the cleaner version of them.

5. Summary

This tutorial gives a background of Convolution based denoising autoencoders adopted for noise reduction in image datasets. With a little background on how it works, it explains how to prepare the training dataset for it. It further explains how to write a simple convolution-based denoising autoencoder in Keras and TensorFlow.

The study is finally concluded with model training and displaying results. The results were quite good for our toy dataset. This kind of image denoising autoencoder can be utilized in multiple applications as a denoising utility to boost the overall performance. Some of the relevant application areas are listed below-

  1. Optical character recognition tasks (OCR)
  2. Handwritten text recognition engine(HTR)
  3. Traffic sign classification
  4. Photo editing tools and many more similar areas

Github Link: https://github.com/kartikgill/Autoencoders

With this i would like to end this article here.

Thanks for reading, I hope it was helpful! Do let me know your thoughts and suggestions by commenting below. See you in the next article.

Read Next >>>

  1. Autoencoders in Keras and Deep Learning (Introduction)
  2. Variational AutoEncoders and Image Generation with Keras
  3. Optimizers explained for training Neural Networks
  4. Optimizing TensorFlow models with Quantization Techniques
  5. 1D-CNN based Fully Convolutional Model for Handwriting Recognition
  6. Deep Learning with PyTorch: Introduction
  7. Deep Learning with PyTorch: First Neural Network

Related Papers

  1. Extracting and Composing Robust Features with Denoising
    Autoencoders
  2. Speech Enhancement Based on Deep Denoising Autoencoder
  3. Deep Collaborative Filtering via Marginalized Denoising Auto-encoder
  4. Medical Image Denoising Using Convolutional Denoising Autoencoders
  5. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder

2 thoughts on “Convolutional Denoising Autoencoders for image noise reduction”

  1. Pingback: Autoencoders in Keras and Deep Learning - Drops of AI

  2. Pingback: Variational AutoEncoders and Image Generation with Keras - Drops of AI

Comments are closed.