Formula recognition#

Formula recognition models read the image of a mathematical formula and predict its LaTeX source. In the OCR pipeline, they read the Formula regions found by the layout model, which are exported as display math (e.g. $$...$$ in Markdown).

Architectures#

formulanet combines a hgnetv2_b4() encoder with a small transformer decoder, which predicts LaTeX tokens from LATEX_TOKENS, e.g. \frac { a } { b ^ { 2 } }. The pretrained model is trained on the printed and handwritten formulas of UniMER-1M.

Benchmark#

Architecture

Input shape

# params

Exact match

Edit similarity

BLEU

formulanet

(256, 768, 3)

26.4 M

44.40

91.30

0.840

The metrics are produced by references/formula/evaluate.py with FormulaMetric on UniMER-Test (23,757 formulas), on the LaTeX tokens: exact match rate, edit similarity (one minus the relative token edit distance) and corpus-level BLEU-4.

Note

Equation numbers are predicted with \tag{...} or as text after the formula. A few formulas may still use TeX commands that KaTeX does not render (e.g. \boldmath): they render in LaTeX, but not in a KaTeX-based Markdown or HTML viewer.

Usage#

from doctr.models import formula_predictor, ocr_predictor

# In the OCR pipeline
predictor = ocr_predictor(pretrained=True, detect_formulas=True)

# On its own
formulas = formula_predictor("formulanet", pretrained=True)

See Formula recognition to work with the recognized formulas, and the reference scripts to train on your own data.