MathReader: API for handwritten mathematical expressions recognition

Paper: MathReader: API for handwritten mathematical expressions recognition, published in 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI)

Details about implementation are described in the paper.

Interface for validation:

MathReader Neural Network Training

If you want to re-train the neural network model. Feel free to improve it (details about the training process is described in the paper mentioned above).

GitHub Project: MathReader Training

About:

Accepted symbols:
- {"0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "a", "b", "c", "m", "n", "x", "y", "z", "+", "-", "*", "sqrt", "=", "neq"}
Internal modules:
- Image preprocessing
- Image posprocessing
- Symbol recognition
- Structural analysis
- Lexical analysis
- Sintatic analysis
- Grammar correction

Usage:

Example in: main.py

from api import *

hme_recognizer = HME_Recognizer()

image = ['images/numbers/teste10.png']

try:
    hme_recognizer.load_image(image)
    expression, img = hme_recognizer.recognize()
    print("Latex: ", expression)
except Exception as e:
    print('Exception: ', e)

Configuration

Configuration file is in docs/config.json

{
    "application": {
        "debug_mode": "active",
        "debug_mode_image": "active"
    }
}

debug_mode: active activate debug messages in terminal
debug_mode_image: active show the images during the execution for debugging

HME_Recognizer

All attributes and methods avaiable in HME_Recognizer class at api.py

Attributes:

- image
- parsed_expression
- processed_image
- predictions
- configurations
- expression_after_recognition
- expression_after_parser
- expression_after_grammar
- parser_tree
- parser_list
- lex_errors
- yacc_errors
- pure_lex_errors
- pure_yacc_errors

Methods

load_image(image)
- First method you should call.
- It loads the image containing the expression.
- Input: image.
- Returns: None.
recognize()
- Second method you should call.
- It makes the recognition of the image containing the expression.
- Input: None.
- Returns:
  - expression: latex of the expression
  - image: image after processing
reset()
- Reset all attributes from HME_Recognizer()
- Input: None.
- Returns: None
get_predictions()
- Get all predictions from neural network
- Input: None.
- Returns: List of predictions.
get_expression_after_parser()
- Get expression after going to parser module
- Input: None.
- Returns: (list) expression
get_expression_after_grammar()
- Get expression after grammar check and correction
- Input: None.
- Returns: (str) expression
get_expression_after_recognition()
- Get expression after recognition module
- Input: None.
- Returns: (dict) expression
get_configurations()
- Get configuration from configuration file docs/config.json
- Input: None.
- Returns: configurations
get_lex_errors()
- Get lex errors with a structure used by the grammar check
get_yacc_errors()
- Get yacc errors with a structure used by the grammar check
get_pure_lex_errors()
- Get 'pure' lex errors coming from PLY (Python Lex Yacc)
get_pure_yacc_errors()
- Get 'pure' yacc errors coming from PLY (Python Lex Yacc)

bloobglob/mathreader