[CVPR2023] Neural Transformation Fields for Arbitrary-Styled Font Generation

This is the official Pytorch implementation of Neural Transformation Fields for Arbitrary-Styled Font Generation.

Neural Transformation Fields for Arbitrary-Styled Font Generation

Bin Fu, Junjun He, Jianjun Wang, Yu Qiao

Few-shot font generation (FFG), aiming at generating font images with a few samples, is an emerging topic in recent years due to its academic and commercial values. Typically, the FFG approaches follow the style-content disentanglement paradigm, which transfers the target font styles to characters by combining the content representations of source characters and the style codes of reference samples. Most existing methods attempt to increase font generation ability via exploring powerful style representations, which may be a sub-optimal solution for the FFG task due to the lack of modeling spatial transformation in transferring font styles. In this paper, we model font generation as a continuous transformation process from the source character image to the target font image via the creation and dissipation of font pixels, and embed the corresponding transformations into a neural transformation field. With the estimated transformation path, the neural transformation field generates a set of intermediate transformation results via the sampling process, and a font rendering formula is developed to accumulate them into the target font image. Extensive experiments show that our method achieves state-of-the-art performance on the few-shot font generation task, which demonstrates the effectiveness of our proposed model.

Prerequisites

Since our codes is based on FFG-benchmarks, installing this benchmark and then putting our code into the corresponding positions is the fastest way to implement our model. The code is tested on Python >= 3.6 (we recommend conda) with the following libraries

conda create -n NTF python=3.8.5
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install sconf
pip install scipy
pip install scikit-image
pip install tqdm
pip install jsonlib-python3
pip install fonttools
pip install tensorboard
pip install click
pip install Pillow==9.5.0
git clone https://github.com/clovaai/fewshot-font-generation.git
git clone https://github.com/fubinfb/NTF.git
# merge the files in fewshot-font-generation and NTF

Datasets

In this project, we select the native true-type font (TTF) formats for datasets, which is storage-efficient and easy-to-use. The structure of the dataset is constructed following the instructions from FFG-benchmarks.

You can collect your own fonts from the following web site (for non-commercial purpose):

https://www.foundertype.com/index.php/FindFont/index (acknowledgement: DG-Font and FFG-benchmarks refer this web site)

Training

Modify the configuration file

The configuration file: cfgs/NTF/train.yaml

- resume: # the resume checkpoint (if available)
- work_dir: # the directory to save checkpoints, validation images, and the log.

The configuration file: cfgs/data/train/custom.yaml

dset:   # leave blank
  train:   # leave blank
    data_dir:  # The font folder for training
    chars:  # The json file for training characters
    extension: ttf

Note that the hyper-parameters are provided in the file: cfgs/NTF/default.yaml, you can modify this file to change parameters.

Run training

CUDA_VISIBLE_DEVICES=GPUID python train_NTF.py cfgs/NTF/train.yaml cfgs/data/train/custom.yaml

Testing

Modify the configuration file

The configuration file: cfgs/data/eval/chn_ttf.yaml

dset:
  test:
    extension: ttf # extension of training data
    data_dir:  # path to training data
    source_path: data/chn/source.ttf  # path to the source font or source directory to use for the validation.
    source_ext: ttf   # extension of the source data
    ref_chars: # The json file containing the reference characters
    gen_chars: # The json file containing the characters list to generate

Note that the hyper-parameters are provided in the file: cfgs/NTF/default.yaml, you can modify this file to change parameters.

Run testing

CUDA_VISIBLE_DEVICES=GPUID python inference.py cfgs/data/eval/chn_ttf.yaml --model NTF --weight  --result_dir

Code license

This project is distributed under MIT license.

Acknowledgement

This project is based on FFG-benchmarks.

How to cite

@InProceedings{Fu_2023_CVPR,
    author    = {Fu, Bin and He, Junjun and Wang, Jianjun and Qiao, Yu},
    title     = {Neural Transformation Fields for Arbitrary-Styled Font Generation},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {22438-22447}
}