/MinerU

MinerU is a one-stop, open-source, high-quality data extraction tool, supports PDF/webpage/e-book extraction.一站式开源高质量数据提取工具，支持PDF/网页/多格式电子书提取。

Primary LanguagePythonGNU Affero General Public License v3.0AGPL-3.0

stars forks open issues issue resolution PyPI version Downloads Downloads

English | 简体中文

MinerU

Introduction

MinerU is a one-stop, open-source, high-quality data extraction tool, includes the following primary features:

Magic-PDF

Introduction

Magic-PDF is a tool designed to convert PDF documents into Markdown format, capable of processing files stored locally or on object storage supporting S3 protocol.

Key features include:

  • Support for multiple front-end model inputs
  • Removal of headers, footers, footnotes, and page numbers
  • Human-readable layout formatting
  • Retains the original document's structure and formatting, including headings, paragraphs, lists, and more
  • Extraction and display of images and tables within markdown
  • Conversion of equations into LaTeX format
  • Automatic detection and conversion of garbled PDFs
  • Compatibility with CPU and GPU environments
  • Available for Windows, Linux, and macOS platforms
pdf_zh_cn.mp4

Project Panorama

Project Panorama

Flowchart

Flowchart

Submodule Repositories

  • PDF-Extract-Kit
    • A Comprehensive Toolkit for High-Quality PDF Content Extraction

Getting Started

Requirements

  • Python >= 3.9

Using a virtual environment is recommended to avoid potential dependency conflicts; both venv and conda are suitable. For example:

conda create -n MinerU python=3.10
conda activate MinerU

Usage Instructions

1. Install Magic-PDF

Install using pip:

pip install magic-pdf

Alternatively, for built-in high-precision model parsing capabilities, use:

pip install magic-pdf[full-cpu]

The high-precision models depend on detectron2, which requires a compiled installation.
If you need to compile it yourself, refer to facebookresearch/detectron2#5114
Or directly use our pre-compiled wheel packages (limited to python 3.10):

pip install detectron2 --extra-index-url https://myhloli.github.io/wheels/

2. Downloading model weights files

For detailed references, please see below how_to_download_models

After downloading the model weights, move the 'models' directory to a directory on a larger disk space, preferably an SSD.

3. Copy the Configuration File and Make Configurations

You can get the magic-pdf.template.json file in the repository root directory.

cp magic-pdf.template.json ~/magic-pdf.json

In magic-pdf.json, configure "models-dir" to point to the directory where the model weights files are located.

{
  "models-dir": "/tmp/models"
}

4. Usage via Command Line

simple
magic-pdf pdf-command --pdf "pdf_path" --inside_model true

After the program has finished, you can find the generated markdown files under the directory "/tmp/magic-pdf".
You can find the corresponding xxx_model.json file in the markdown directory.
If you intend to do secondary development on the post-processing pipeline, you can use the command:

magic-pdf pdf-command --pdf "pdf_path" --model "model_json_path"

In this way, you won't need to re-run the model data, making debugging more convenient.

more
magic-pdf --help

5. Acceleration Using CUDA or MPS

CUDA

You need to install the corresponding PyTorch version according to your CUDA version.
This example installs the CUDA 11.8 version.More information https://pytorch.org/get-started/locally/

pip install --force-reinstall torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu118

Also, you need to modify the value of "device-mode" in the configuration file magic-pdf.json.

{
  "device-mode":"cuda"
}
MPS

For macOS users with M-series chip devices, you can use MPS for inference acceleration.
You also need to modify the value of "device-mode" in the configuration file magic-pdf.json.

{
  "device-mode":"mps"
}

6. Usage via Api

Local
image_writer = DiskReaderWriter(local_image_dir)
image_dir = str(os.path.basename(local_image_dir))
jso_useful_key = {"_pdf_type": "", "model_list": model_json}
pipe = UNIPipe(pdf_bytes, jso_useful_key, image_writer)
pipe.pipe_classify()
pipe.pipe_parse()
md_content = pipe.pipe_mk_markdown(image_dir, drop_mode="none")
Object Storage
s3pdf_cli = S3ReaderWriter(pdf_ak, pdf_sk, pdf_endpoint)
image_dir = "s3://img_bucket/"
s3image_cli = S3ReaderWriter(img_ak, img_sk, img_endpoint, parent_path=image_dir)
pdf_bytes = s3pdf_cli.read(s3_pdf_path, mode=s3pdf_cli.MODE_BIN)
jso_useful_key = {"_pdf_type": "", "model_list": model_json}
pipe = UNIPipe(pdf_bytes, jso_useful_key, s3image_cli)
pipe.pipe_classify()
pipe.pipe_parse()
md_content = pipe.pipe_mk_markdown(image_dir, drop_mode="none")

Demo can be referred to demo.py

Magic-Doc

Introduction

Magic-Doc is a tool designed to convert web pages or multi-format e-books into markdown format.

Key Features Include:

  • Web Page Extraction

    • Cross-modal precise parsing of text, images, tables, and formula information.

  • E-Book Document Extraction

    • Supports various document formats including epub, mobi, with full adaptation for text and images.

  • Language Type Identification

    • Accurate recognition of 176 languages.
extract1.mp4
extract2.mp4
extract3.mp4

Project Repository

  • Magic-Doc Outstanding Webpage and E-book Extraction Tool

All Thanks To Our Contributors

License Information

LICENSE.md

The project currently leverages PyMuPDF to deliver advanced functionalities; however, its adherence to the AGPL license may impose limitations on certain use cases. In upcoming iterations, we intend to explore and transition to a more permissively licensed PDF processing library to enhance user-friendliness and flexibility.

Acknowledgments

Citation

@misc{2024mineru,
    title={MinerU: A One-stop, Open-source, High-quality Data Extraction Tool},
    author={MinerU Contributors},
    howpublished = {\url{https://github.com/opendatalab/MinerU}},
    year={2024}
}

Star History

Star History Chart