ocrd_butler

Processing tasks in the ecosystem of the OCR-D project project.

Features

REST API to define workflows of processors and run tasks with it in the OCR-D ecosystem.

Hints

Free software: MIT License
This software is in alpha state yet, so don't expect it to work properly. Support is currently not guarenteed.

Development installation

We rely on the excellent installation repository ocrd_all. Please check it out for installation.

Installation is currently tested on Debian 10 and Ubuntu 18.04. Be aware that on more up-to-date systems with Python >= 3.8.x there is currently a problem installing tensorflow==1.15.x, so you have to use at most Python 3.7.

Installation for development:

Install Redis Server (needed as backend for Celery and Flower)

user@server:/ > sudo apt install redis
user@server:/ > sudo service redis start

Follow the installation for ocrd_all

/home/ocrd > git clone --recurse-submodules https://github.com/OCR-D/ocrd_all.git && cd ocrd_all
/home/ocrd/ocrd_all > make all
... -> download appropriate modules...

Install german language files for Tesseract OCR:

user@server:/ > sudo apt install tesseract-ocr-deu

Install ocrd-butler in the virtual environment created by ocrd_all:

/home/ocrd > git clone https://github.com/StaatsbibliothekBerlin/ocrd_butler.git & cd ocrd-butler
/home/ocrd/ocrd-butler > source ../ocrd_all/venv/bin/activate
(venv) /home/ocrd/ocrd-butler > pip install -e .[dev]

For some modules in ocrd_all there are further files nessesary, e.g. trained models for the OCR itself. The folders on the server can be overwritten it every single task.

sbb_textline_detector (i.e. make textline-detector-model):

> mkdir -p /data && cd /data; \
> ocrd resmgr download ocrd-sbb-textline-detector default -al cwd

ocrd_calamari (i.e. make calamari-model):

> mkdir -p /data && cd /data; \
> ocrd resmgr download ocrd-calamari-recognize qurator-gt4histocr-1.0 -al cwd

ocrd_tesserocr (i.e. make tesseract-model):

> mkdir -p /data/tesseract_models && cd /data/tesseract_models
> wget https://qurator-data.de/tesseract-models/GT4HistOCR/models.tar
> tar xf models.tar
> cp GT4HistOCR_2000000.traineddata /usr/share/tesseract-ocr/4.00/tessdata/

ocrd-sbb-binarize (i.e. make sbb-binarize-model)

> mkdir -p /data && cd /data; \
> ocrd resmgr download ocrd-sbb-binarize default -al cwd

Start celery worker (i.e. make run-celery):

╰─$ TESSDATA_PREFIX=/usr/share/tesseract-ocr/4.00/tessdata celery worker -A ocrd_butler.celery_worker.celery -E -l info

Start flower monitor (i.e. make run-flower):

╰─$ flower --broker redis://localhost:6379 --persistent=True --db=flower [--log=debug --url_prefix=flower]

Flower monitor: http://localhost:5555

Run the app (i.e. make run-flask):

╰─$ FLASK_APP=ocrd_butler/app.py flask run

Flask frontend: http://localhost:5000 Swagger interface: http://localhost:5000/api

Run the tests:

╰─$ make test

Usage

For API documentation, open the Swagger API user interface at /api/. A complete list of all routes mapped by the OCRD Butler application is available under the /api/_util/routes endpoint.

Creating a workflow

A Butler workflow consists of a name and one or more OCRD processor invocations. Use the /api/workflows POST endpoint to create a new workflow (all examples given using HTTPie):

╰─$ http POST :/api/workflows < workflow.json

...where the content of workflow.json looks something like this:

{
  "name": "binarize && segment to regions",
  "processors": [
    {
      "name": "ocrd-olena-binarize",
      "input_file_grp": "DEFAULT",
      "output_file_grp": "OCR-D-IMG-BIN"
    },
    {
      "name": "ocrd-tesserocr-segment-region",
      "input_file_grp": "OCR-D-IMG-BIN",
      "output_file_grp": "OCR-D-SEG-REGION"
    }
  ]
}

The response body will contain the ID of the newly created workflow. Use this ID for retrieval of the newly created workflow:

╰─$ http :/api/workflows/1  # or whatever ID obtained in previous step

Creating a task

A Butler task is an invocation of a workflow with a specific METS file as its input. A task consists of at least such a METS source file location, and a workflow ID. Use the /api/tasks POST endpoint to create a new task using an existing workflow:

╰─$ http POST :/api/tasks src=https://content.staatsbibliothek-berlin.de/dc/PPN718448162.mets.xml workflow_id=1

The response body will contain the ID of the newly created task.

Running a task

In order to execute an existing Butler task, call the /api/tasks/{id}/run endpoint, with the placeholder replaced by the actual task ID obtained in the previous step:

╰─$ http POST :/api/tasks/1/run

Known problems

ModuleNotFoundError: No module named 'tensorflow.contrib'

. venv/activate
pip install --upgrade pip
pip uninstall tensorflow
pip install tensorflow-gpu==1.15.*

TODOs

input and output filegroups are not always from the previous processor - more complicated input/output group scenarios - check the infos we get from ocrd-tools.json
dinglehopper: - If there are Ground Truth data it could be placed in a configured folder on the server with the data as page xml files inside a folder id named with the work id. Then we show a button to start a run against this data. Otherwise we can search for all other tasks with the same work_id and present a UI to run against the choosen one.
Use processor groups to be able to build forms with these presented.
Check if ocrd-olena-binarize fail with another name for a METS file in a workspace then mets.xml.
Refactor ocrd_tool information collection to https://ocr-d.de/en/spec/cli#-j---dump-json

This package was created with Cookiecutter and the elgertam/cookiecutter-pipenv project template, based on audreyr/cookiecutter-pypackage.