/Neural-Photo-Editor

A simple interface for editing natural photos with generative neural networks.

Primary LanguagePythonMIT LicenseMIT

Neural Photo Editor

A simple interface for editing natural photos with generative neural networks.

GUI

This repository contains code for the paper "Neural Photo Editing with Introspective Adversarial Networks," and the Associated Video.

Installation

To run the Neural Photo Editor, you will need:

  • Theano, development version.
  • lasagne, development version.
  • I highly recommend cuDNN as speed is key, but it is not a dependency.
  • numpy, scipy, PIL, Tkinter and tkColorChooser, but it is likely that your python distribution already has those.

Running the NPE

By default, the NPE runs on IAN_simple. This is a slimmed-down version of the IAN without MDC or RGB-Beta blocks, which runs without lag on a laptop GPU with ~1GB of memory (GT730M)

If you're on a Windows machine, you will want to create a .theanorc file and at least set the flag FLOATX=float32.

If you're on a linux machine, you can just insert THEANO_FLAGS=floatX=float32 before the command line call.

If you don't have cuDNN, simply change line 64 of the NPE.py file from dnn=True to dnn=False. Note that I only have the non-cuDNN option working for IAN_simple.

Then, run the command:

python NPE.py

If you wish to use a different model, simply edit the line with "config path" in the NPE.py file.

You can make use of any model with an inference mechanism (VAE or ALI-based GAN).

Commands

  • You can paint the image by picking a color and painting on the image, or paint in the latent space canvas (the red and blue tiles below the image).
  • The long horizontal slider controls the magnitude of the latent brush, and the smaller horizontal slider controls the size of both the latent and the main image brush.
  • You can select different entries from the subset of the celebA validation set (included in this repository as an .npz) by typing in a number from 0-999 in the bottom left box and hitting "infer."
  • Use the reset button to return to the last inferred result.
  • Use the sample button to generate a random latent vector and corresponding image.
  • Use the scroll wheel to lighten or darken an image patch (equivalent to using a pure white or pure black paintbrush). Note that this automatically returns you to sample mode, and may require hitting "infer" rather than "reset" to get back to photo editing.

Training an IAN on celebA

You will need Fuel along with the 64x64 version of celebA. See here for instructions on downloading and preparing it.

If you wish to train a model, the IAN.py file contains the model configuration, and the train_IAN.py file contains the training code, which can be run like this:

python train_IAN.py IAN.py

By default, this code will save (and overwrite!) the weights to a .npz file with the same name as the config.py file (i.e. "IAN.py -> IAN.npz"), and will output a jsonl log of the training with metrics recorded after every chunk.

Use the --resume=True flag when calling to resume training a model--it will automatically pick up from the most recent epoch.

Sampling the IAN

You can generate a sample and reconstruction+interpolation grid with:

python sample_IAN.py IAN.py

Note that you will need matplotlib. to do so.

Known Issues/Bugs

Occasionally the color picker freaks out and returns "None"s rather than values. Simply re-choose a color to fix this.

My MADE layer currently only accepts hidden unit sizes that are equal to the size of the latent vector, which will present itself as a BAD_PARAM error.

Since the MADE really only acts as an autoregressive randomizer I'm not too worried about this, but it does bare looking into.

I messed around with the keywords for get_model, you'll need to deal with these if you wish to run any model other than IAN_simple through the editor.

Notes

More pre-trained models (which will be in the repo rather than a drive folder) and the remainder of the IAN experiments (including SVHN and the full pre-trained IAN used to generate the samples from the paper) coming soon.

I'm considering putting up a TensorFlow implementation, if people would be interested in that, let me know (maybe with an Issue).

Acknowledgments

This code contains lasagne layers and other goodies adopted from a number of places: