pix2tex - LaTeX OCR

Just a bit modify from Lukas' work. This now read image from the local file by using -n FILENAME and add the resize to let the model worked more stabilized.

The goal of this project is to create a learning based system that takes an image of a math formula and returns corresponding LaTeX code. As a physics student I often find myself writing down Latex code from a reference image. I wanted to streamline my workflow and began looking into solutions, but besides the Freemium Mathpix I could not find anything ready-to-use that runs locally. That's why I decided to create it myself.

Requirements

Evaluation

PyTorch (tested on v1.7.0)
Python 3.7+ & dependencies (requirements.txt)
```
pip install -r requirements.txt
```

Dataset

In order to render the math in many different fonts we use XeLaTeX, generate a PDF and finally convert it to a PNG. For the last step we need to use some third party tools:

XeLaTeX
ImageMagick with Ghostscript. (for converting pdf to png)
Node.js to run KaTeX (for normalizing Latex code)
de-macro >= 1.4 (only for parsing arxiv papers)
Python 3.7+ & dependencies (requirements.txt)

Using the model

Download/Clone this repository
For now you need to install the Python dependencies specified in requirements.txt (look above)
Download the weights.pth (and optionally image_resizer.pth) file from my Google Drive and place it in the checkpoints directory

The pix2tex.py file offers a quick way to get the model prediction of an image. First you need to copy the formula image into the clipboard memory for example by using a snipping tool (on Windows built in Win+Shift+S). Next just call the script with python pix2tex.py. It will print out the predicted Latex code for that image and also copy it into your clipboard.

Note: As of right now it works best with images of smaller resolution. Don't zoom in all the way before taking a picture. Double check the result carefully. You can try to redo the prediction with an other resolution if the answer was wrong.

Update: I have trained an image classifier on randomly scaled images of the training data to predict the original size. This model will automatically resize the custom image to best resemble the training data and thus increase performance of images found in the wild. To use this preprocessing step, all you have to do is download the second weights file mentioned above. You should be able to take bigger (or smaller) images of the formula and still get a satisfying result

Training the model

First we need to combine the images with their ground truth labels. I wrote a dataset class (which needs further improving) that saves the relative paths to the images with the LaTeX code they were rendered with. To generate the dataset pickle file run

python dataset/dataset.py --equations path_to_textfile --images path_to_images --tokenizer path_to_tokenizer --out dataset.pkl

You can find my generated training data on the Google Drive as well (formulae.zip - images, math.txt - labels). Repeat the step for the validation and test data. All use the same label text file.

Edit the data entry in the config file to the newly generated .pkl file. Change other hyperparameters if you want to. See settings/default.yaml for a template.
Now for the actual training run

python train.py --config path_to_config_file

weiwenlan/LaTeX-OCR

pix2tex - LaTeX OCR

Just a bit modify from Lukas' work. This now read image from the local file by using -n FILENAME and add the resize to let the model worked more stabilized.

Requirements

Evaluation

Dataset

Using the model

Training the model

Model

Performance

Data

Fonts

TODO

Contribution

Acknowledgement

References