Pixel Pal (PART 3) : Measuring model quality

Pixel Pal is a project I have started whose purpose is to bring new life to old icon themes. Many old icon themes are in raster format and are not available for HiDPI screens (their resolution are too small). Currently the only way to handle old icon themes is to resample them by the desired factor, but the icons look pixelated.

This is the third article about this project. Here are the other two:

1. Part #1 Explaining the project.
2. Part #2 Building the dataset

In this part I will be talking about losses, metrics and targets. Before going on about training deep learning networks to up sample intelligently icon images, we need ways to measure if our models are doing a good job. Especially we need to know if a model does better than simply pixel doubling.

32x32 pixels version of the icon up sampled (pixel doubling)	64x64 pixels version of the icon

The images above show side by side the same vector icon but the left icon was rasterized as a 32 by 32 pixels image and the right icon was rasterized as a 64 by 64 pixels. Furthermore the 32 by 32 pixels was up sampled (simple pixel doubling)

As you can see the two images are similar but not identical. We need some way to measure how similar the two are. Simple counting the number of identical pixels is problematic. Because two pixels aren't identical does not mean they are not similar. Counting the number of identical pixels as a metric will probably be too critical. So for this project I will rely on two metrics suggested in the following article : An Introduction to Super Resolution using Deep Learning. The metrics in question are the following :

1. Peak Signal to Noise Ratio (PSNR)
2. Structural Similarity (SSIM)

For information the metrics for the pair of images above are :

1. Peak Signal to Noise Ratio: 16.913143157958984
2. Structural Similarity : 0.8310688138008118

Measuring how similar two images are is already a good element. It provides us an idea of how well our models are working. However there are some limits to metrics:

1. Metrics are not always differentiable. However training neural networks require a way to measure the distance between the wanted output and the output the network is producing. This is where losses come into play. 2. You need to have some good baselines. In other words the metric in itself is not useful, you need to have an objectif. Something like do better than simple pixel doubling or bicubic interpolation. 3. Metrics will be sensitive to the data you use. You may land on images that are easy to upsample and the metric may be deceptively good. This is why you should compute the metrics on the entire dataset and do some statistics on it. 4. Metrics are not perfect. In the end we want something to be visually pleasing not something that maximise a mathematical function. This is why you should also visually look at the result of the neural networks.

To train the network will use a loss function. Loss function are ways to measure the distance between the ideal output and the current output. In our case we will use mean squared error as it is simple and fits our purpose. In general always make sure you are using an appropriate loss function.

In terms of baselines my objectif will be to perform better thant straight upsampling via nearest and bilinear interpolation.