In this project, we have hands-on experience with implicit neural representation (INR). With INR, we parameterize some signal (in our case images) with a neural network (feed-forward network). While in practice this might be useful for outpainting, super-resolution, and compression, in this project we will mainly focus on the basics, with some proof-of-concept outpainting at the end. Additionally, we apply Sine activation function [2] and Gaussian Fourier feature mapping [1] to improve the reconstruction quality of the system.
For the data that can be found in the image data folder bird.jpg, we resized the image to 150 * 150 pixels in order to reduce the training time.
pre_process.py: Resize the image to 150 * 150 pixels.
data_reader.py: A data loader that retrieves x & y coordinates and RGB values of each pixel.
model.py: A network of fully connected layers (FFN) that output the RGB values.
train.ipynb: putting all together and train the model, also evaluating the model by PSNR.
train_mapping.ipynb: Same as train.ipynb but train the network with Gaussian Fourier feature mapping.
train_Sine.ipynb: Same as train.ipynb but train the network by using Sine activation function.
The figures below show the visualization of image restoration of three different approaches.
 |
|---|
| Normal Feed-foward Network |
 |
|---|
| FFN with Gaussian Fourier feature mapping |
 |
|---|
| FFN with Sine activation function |
 |
|---|
| Image Outpainting |
We can observe from the figures that FNN with Gaussian Fourier feature mapping seems to outperform the other approaches. We also present the quantitative results are summarized below. The PSNR of three methods.
| Method | PSNR (dB) |
|---|---|
| Normal FCN | 21.928 |
| Position Mapping | 32.796 |
| Sine Activation | 25.556 |
- Tancik, Matthew, et al. "Fourier features let networks learn high-frequency functions in low dimensional domains." Advances in Neural Information Processing Systems 33 (2020): 7537-7547.
- Sitzmann, Vincent, et al. "Implicit neural representations with periodic activation functions." Advances in neural information processing systems 33 (2020): 7462-7473.