Layout analysis & tables#
Text alone loses the structure of a document. docTR can also tell what each region of a page is (a title, a paragraph, a table, a figure…), rebuild tables as rows and columns and read formulas as LaTeX.
Layout analysis#
A layout model detects and classifies the regions of a page. The pretrained lw_detr_s model recognizes
11 classes:
Title Section-header Text List-item Caption Footnote Formula Table Picture Page-header Page-footer
In the OCR pipeline#
Pass detect_layout=True and the regions are attached to every page as
LayoutElement objects:
from doctr.io import DocumentFile
from doctr.models import ocr_predictor
predictor = ocr_predictor(pretrained=True, detect_layout=True)
result = predictor(DocumentFile.from_pdf("invoice.pdf"))
for region in result.pages[0].layout:
print(f"{region.type:<15} {region.confidence:.2f}")
Page-footer 0.89
Section-header 0.75
Section-header 0.67
Table 0.95
Text 0.78
...
The layout also improves the exports: headings, lists, tables, figures, headers and footers are rendered with the
right structure and in the right place (see Exporting results). It is part of the JSON export under the layout
key, and page.show() draws it.
The layout model shares the input settings of the predictor (assume_straight_pages, preserve_aspect_ratio…)
and the batch size of the detection model (det_bs). Use layout_arch to select another architecture, or to pass
your own fine-tuned model.
Ignore some regions#
Pictures, stamps or tables can produce noisy text. ignore_regions masks the regions of the given layout classes
before text detection and recognition, so their content is simply not read:
predictor = ocr_predictor(pretrained=True, detect_layout=True, ignore_regions=["Picture", "Page-footer"])
Important
ignore_regions relies on the layout model: enable it with detect_layout=True (or detect_tables=True /
detect_formulas=True),
otherwise the regions cannot be located and nothing is masked. The CLI enables it automatically.
Table structure recognition#
With detect_tables=True, each region the layout model labels as a Table is cropped and passed to a table
structure model (tablecenternet), which predicts its cells and the rows and columns they span. Every word inside the
table region is regrouped into a Table: the words outside of the predicted cells (e.g. a missed
header row or a row of totals) complete the grid, and the rows and columns follow the text lines of the cells (the
rows and columns without any text are removed):
from doctr.io import DocumentFile
from doctr.models import ocr_predictor
predictor = ocr_predictor(pretrained=True, detect_tables=True)
result = predictor(DocumentFile.from_pdf("invoice.pdf"))
table = result.pages[0].tables[0]
print(table.num_rows, table.num_cols) # 5 4
print(table.render())
Description Qty Unit price Amount
Recycled paper A4 10 8.50 85.00
Ballpoint pens (box) 5 12.00 60.00
Desk lamp 2 45.00 90.00
Office chair 1 1,015.00 1,015.00
Important
The words of a table are removed from the regular blocks of the page, so the same text is not returned
twice. Read them through page.tables, or use an export: tables are rendered in place by render(), Markdown,
AsciiDoc and HTML, and stored under the tables key of the JSON export.
Load a table into pandas#
to_grid() returns a dense num_rows × num_cols grid of strings. A cell spanning several
rows or columns has its value at its top-left position:
import pandas as pd
grid = table.to_grid()
df = pd.DataFrame(grid[1:], columns=grid[0]) # first row as header
print(df)
Description Qty Unit price Amount
0 Recycled paper A4 10 8.50 85.00
1 Ballpoint pens (box) 5 12.00 60.00
2 Desk lamp 2 45.00 90.00
3 Office chair 1 1,015.00 1,015.00
Cells, spans and geometry#
For full control, iterate over the TableCell objects:
for cell in table.cells:
print(cell.value, (cell.row_start, cell.row_end), (cell.col_start, cell.col_end), cell.geometry)
Attribute |
Description |
|---|---|
|
The text of the words assigned to the cell |
|
First and last row spanned by the cell (0-indexed, inclusive) |
|
First and last column spanned by the cell (0-indexed, inclusive) |
|
Number of rows / columns spanned |
|
Position of the cell, relative to the page |
|
Mean recognition confidence of the words of the cell |
|
Table model score of the cell ( |
Formula recognition#
With detect_formulas=True, each region the layout model labels as a Formula (a display formula, on its own lines)
is masked out of the text detection, cropped and read by a formula recognition model (formulanet), which predicts its
LaTeX source. The formulas are attached to every page as Formula objects:
from doctr.io import DocumentFile
from doctr.models import ocr_predictor
predictor = ocr_predictor(pretrained=True, detect_formulas=True)
result = predictor(DocumentFile.from_pdf("attention_is_all_you_need.pdf")) # arXiv:1706.03762
formula = result.pages[4].formulas[0]
print(formula.value)
# \mathrm { F F N } ( x ) = \operatorname* { m a x } ( 0 , x W _ { 1 } + b _ { 1 } ) W _ { 2 } + b _ { 2 } \tag { $ 2 $ }
print(formula.confidence) # 0.98
The formulas are read in place with the text, and exported as display math:
Export |
Rendering |
|---|---|
|
the LaTeX source |
Markdown |
|
HTML |
|
AsciiDoc |
a |
hOCR |
an |
JSON |
the |
Important
The formula regions are not read by the text recognition model, so a formula is not returned twice. The LaTeX source is never escaped as Markdown or AsciiDoc text, and it is HTML-escaped in HTML exports.
Standalone predictors#
The models can also run on their own, without text recognition:
from doctr.models import layout_predictor
predictor = layout_predictor("lw_detr_s", pretrained=True)
out = predictor(pages)
# One dict per page
out[0]["class_names"] # ['Page-footer', 'Section-header', ..., 'Table', 'Text', ...]
out[0]["boxes"] # (N, 4) array of relative (xmin, ymin, xmax, ymax)
out[0]["scores"] # N confidence scores
Pass images of tables, e.g. crops of the Table regions:
from doctr.models import table_predictor
predictor = table_predictor("tablecenternet", pretrained=True)
out = predictor([table_image])
out[0]["num_rows"], out[0]["num_cols"] # (5, 4)
out[0]["cells"][0]
# {'geometry': [[x1, y1], ..., [x4, y4]], # relative to the table image
# 'score': 0.42, 'row_start': 0, 'row_end': 0, 'col_start': 0, 'col_end': 0}
With the layout regions of whole pages and their words (e.g. from your own OCR), the table regions are cropped and the words assigned to the cells:
from doctr.models import layout_predictor
regions = layout_predictor("lw_detr_s", pretrained=True)(pages)
tables = predictor.predict_regions(pages, regions, word_boxes) # one (N, 4) array of relative boxes per page
tables[0][0]["cells"][0]["words"] # indices of the words of the cell in `word_boxes[0]`, in reading order
Pass images of formulas, e.g. crops of the Formula regions, margins included:
from doctr.models import formula_predictor
predictor = formula_predictor("formulanet", pretrained=True)
out = predictor([formula_image])
# The LaTeX tokens separated by spaces, and the confidence
out[0] # ('J ( u , y ) = \\int _ { 0 } ^ { 1 } \\frac { | u | ^ { 2 } } { 2 } \\, d x + \\Psi ( y )', 0.98)
With the layout regions of whole pages, the formula regions are cropped and read:
from doctr.models import layout_predictor
regions = layout_predictor("lw_detr_s", pretrained=True)(pages)
formulas = predictor.predict_regions(pages, regions)
formulas[0][0] # {'geometry': ..., 'value': '\\frac { a } { b ^ { 2 } }', 'confidence': 0.97, 'score': 0.91}
See Task-specific predictors for all the task-specific predictors.