The implementation of ACM MM 2022 paper "ChartStamp: Robust Chart Embedding for Real-World Applications".
This repository contains the code that implements the ChartStamp described in our paper "ChartStamp: Robust Chart Embedding for Real-World Applications" published at ACM MM 2022. ChartStamp is a chart embedding method that is robust to real-world printing/displaying and photography. Depending on target image manipulations, ChartStamp uses encoder-decoder network to embed 100, 1000, or 10,000 bits into a chart image.
Our code is implemented and tested on TensorFlow with the following packages and versions:
python=3.8.5tensorflow=1.15.1bchlib=0.14.0opencv-python=4.5.1.48pillow=8.1.0numpy=1.18.5
Run detect_decode.py, additional package is needed:
pytorch=1.7.1
The chart dataset we collected contains 3,525 chart images from three sources : (1) ReVision chart image corpus1; (2) ICPR 2020 CHART competition datasets2; and (3) charts extracted from a corpus of Excel files crawled from the Internet.
This dataset can be download from following link:
[1] Manolis Savva, Nicholas Kong, Arti Chhajta, Fei-Fei Li, Maneesh Agrawala, and Jeffrey Heer. 2011. Revision: Automated classification, analysis and redesign of chart images. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 393–402.[2] ICPR. 2020. CHART-Infographics 2020. https://chartinfo.github.io/.
The training code for perceptual model is not included in this repo. The perceptual model is trained using Enet model released here, and is trained with 1,000 chart images randomly selected from the training set and labelled semi-automatically (segmented and labelled automatically and then manually examined and corrected).
The pretrained perceptual model can be download from following link:
Set dataset path in train.py
TRAIN_PATH = DIR_OF_DATASET_IMAGES
Set perceptual model with command:
--perceptual_model DIR_OF_PERCEPTUAL_MODEL
Train model
python train.py
The pretrained model for embedding 100 bit, in this version ChartStamp is robust to real-world printing and photography
Download the pretrained perceptual model from following link, put 'saved_100' in base directory:
The pretrained model for embedding 1000 bit, in this version ChartStamp is robust to real-world displaying and photography
Download the pretrained perceptual model from following link, put 'saved_1000' in base directory:
The detector model is trained using the EAST model released here. The dataset used for training detector is generated by randomly placing warped ChartStamps onto larger images.
Download the pretrained perceptual model from following link, put 'weight' in base directory:
The script encode.py can be used to embed a message into an image or a directory of images. The default model expects a utf-8 encoded secret that is <= 7 characters (100 bit message -> 56 bits after ECC).
Example usage:
python encode.py --model ./save_100 --image test_img --save_dir ./embed_bit_100 --secret chart!
The script decode.py can be used to decode a message from embedded chart image.
Example usage:
python decode.py --model ./save_100 --image test_embeded_img
The script detect_decode.py can be used to detect and decode ChartStamps in an image. This is used for real-world printing/displaying and photography when ChartStamp lies on a small part of the captured picture.
Example usage:
python detect_decode.py --model ./save_100 --image test_embeded_img --detect_model_path ./weight/100bit_checkpoint.pth
If you are using our code for research purpose, please cite our paper.
@inproceedings{chartstamp_acmmm2022,
author = {Jiayun Fu, Bin B. Zhu, Haidong Zhang, Yayi Zou1, Song Ge, Weiwei Cui, Yun Wang,
Dongmei Zhang, Xiaojing Ma and Hai Jin},
title = {ChartStamp: Robust Chart Embedding for Real-World Applications},
booktitle = {In Proceedings of the 30th ACM International Conference on Multimedia (MM ’22), October 10–14, 2022, Lisboa, Portugal. ACM, New York, NY, USA, 9 pages.},
publisher = {{ACM}},
year = {2022},
url = {https://doi.org/10.1145/3503161.3548286}
}