doctr.utils#

Complementary tools: visualization, page reconstruction and evaluation metrics.

Visualization#

visualize_page(
page,
image,
words_only=True,
display_artefacts=True,
display_layout=True,
scale=10,
interactive=True,
add_labels=True,
**kwargs,
)[source]#

Visualize a full page with predicted blocks, lines and words

>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> from doctr.utils.visualization import visualize_page
>>> from doctr.models import ocr_db_crnn
>>> model = ocr_db_crnn(pretrained=True)
>>> input_page = (255 * np.random.rand(600, 800, 3)).astype(np.uint8)
>>> out = model([[input_page]])
>>> visualize_page(out[0].pages[0].export(), input_page)
>>> plt.show()
Parameters:
  • page (dict[str, Any]) – the exported Page of a Document

  • image (ndarray) – np array of the page, needs to have the same shape than page[‘dimensions’]

  • words_only (bool) – whether only words should be displayed

  • display_artefacts (bool) – whether artefacts should be displayed

  • display_layout (bool) – whether detected layout regions should be displayed

  • scale (float) – figsize of the largest windows side

  • interactive (bool) – whether the plot should be interactive

  • add_labels (bool) – for static plot, adds text labels on top of bounding box

  • **kwargs (Any) – keyword arguments for the polygon patch

Returns:

the matplotlib figure

Return type:

Figure

visualize_kie_page(
page,
image,
words_only=False,
display_artefacts=True,
display_layout=True,
scale=10,
interactive=True,
add_labels=True,
**kwargs,
)[source]#

Visualize a full page with predicted blocks, lines and words

>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> from doctr.utils.visualization import visualize_page
>>> from doctr.models import ocr_db_crnn
>>> model = ocr_db_crnn(pretrained=True)
>>> input_page = (255 * np.random.rand(600, 800, 3)).astype(np.uint8)
>>> out = model([[input_page]])
>>> visualize_kie_page(out[0].pages[0].export(), input_page)
>>> plt.show()
Parameters:
  • page (dict[str, Any]) – the exported Page of a Document

  • image (ndarray) – np array of the page, needs to have the same shape than page[‘dimensions’]

  • words_only (bool) – whether only words should be displayed

  • display_artefacts (bool) – whether artefacts should be displayed

  • display_layout (bool) – whether detected layout regions should be displayed

  • scale (float) – figsize of the largest windows side

  • interactive (bool) – whether the plot should be interactive

  • add_labels (bool) – for static plot, adds text labels on top of bounding box

  • **kwargs (Any) – keyword arguments for the polygon patch

Returns:

the matplotlib figure

Return type:

Figure

draw_boxes(boxes, image, color=None, **kwargs)[source]#

Draw an array of relative straight boxes on an image.

Parameters:
  • boxes (ndarray) – array of relative boxes, of shape (*, 4)

  • image (ndarray) – np array, float32 or uint8

  • color (tuple[int, int, int] | None) – color to use for bounding box edges

  • **kwargs – keyword arguments from matplotlib.pyplot.plot

Reconstitution#

synthesize_page(
page,
draw_proba=False,
font_family=None,
min_font_size=8,
max_font_size=50,
background_color=(255, 255, 255),
text_color=(0, 0, 0),
draw_placeholders=False,
)[source]#

Draw the content of the element page (OCR response) on a blank page.

Parameters:
  • page (dict[str, Any]) – exported Page object to represent

  • draw_proba (bool) – if True, draw words in colors to represent confidence. Blue: p=1, red: p=0

  • font_family (str | None) – family of the font

  • min_font_size (int) – minimum font size

  • max_font_size (int) – maximum font size

  • background_color (tuple[int, int, int]) – RGB color of the page background

  • text_color (tuple[int, int, int]) – RGB color of the rendered text

  • draw_placeholders (bool) – if True, draw the grid of the recognized tables and outline the layout regions holding no text of their own

Returns:

the synthesized page

Return type:

ndarray

synthesize_kie_page(
page,
draw_proba=False,
font_family=None,
min_font_size=8,
max_font_size=50,
background_color=(255, 255, 255),
text_color=(0, 0, 0),
draw_placeholders=False,
)[source]#

Draw the content of the element page (KIE OCR response) on a blank page.

Parameters:
  • page (dict[str, Any]) – exported Page object to represent

  • draw_proba (bool) – if True, draw words in colors to represent confidence. Blue: p=1, red: p=0

  • font_family (str | None) – family of the font

  • min_font_size (int) – minimum font size

  • max_font_size (int) – maximum font size

  • background_color (tuple[int, int, int]) – RGB color of the page background

  • text_color (tuple[int, int, int]) – RGB color of the rendered text

  • draw_placeholders (bool) – if True, draw the grid of the recognized tables and outline the layout regions holding no text of their own

Returns:

the synthesized page

Return type:

ndarray

Task evaluation#

Metrics to evaluate models on each task, used by the evaluation scripts of the references and in the model zoo.

class TextMatch[source]#

Implements text match metric (word-level accuracy) for recognition task.

The raw aggregated metric is computed as follows:

\[\forall X, Y \in \mathcal{W}^N, TextMatch(X, Y) = \frac{1}{N} \sum\limits_{i=1}^N f_{Y_i}(X_i)\]

with the indicator function \(f_{a}\) defined as:

\[\begin{split}\forall a, x \in \mathcal{W}, f_a(x) = \left\{ \begin{array}{ll} 1 & \mbox{if } x = a \\ 0 & \mbox{otherwise.} \end{array} \right.\end{split}\]

where \(\mathcal{W}\) is the set of all possible character sequences, \(N\) is a strictly positive integer.

>>> from doctr.utils import TextMatch
>>> metric = TextMatch()
>>> metric.update(['Hello', 'world'], ['hello', 'world'])
>>> metric.summary()
update(gt, pred)[source]#

Update the state of the metric with new predictions

Parameters:
  • gt (list[str]) – list of groung-truth character sequences

  • pred (list[str]) – list of predicted character sequences

summary()[source]#

Computes the aggregated metrics

Returns:

a dictionary with the exact match score for the raw data, its lower-case counterpart, its anyascii counterpart and its lower-case anyascii counterpart

Return type:

dict[str, float]

class LocalizationConfusion(iou_thresh=0.5, use_polygons=False)[source]#

Implements common confusion metrics and mean IoU for localization evaluation.

The aggregated metrics are computed as follows:

\[\begin{split}\forall Y \in \mathcal{B}^N, \forall X \in \mathcal{B}^M, \\ Recall(X, Y) = \frac{1}{N} \sum\limits_{i=1}^N g_{X}(Y_i) \\ Precision(X, Y) = \frac{1}{M} \sum\limits_{i=1}^M g_{X}(Y_i) \\ meanIoU(X, Y) = \frac{1}{M} \sum\limits_{i=1}^M \max\limits_{j \in [1, N]} IoU(X_i, Y_j)\end{split}\]

with the function \(IoU(x, y)\) being the Intersection over Union between bounding boxes \(x\) and \(y\), and the function \(g_{X}\) defined as:

\[\begin{split}\forall y \in \mathcal{B}, g_X(y) = \left\{ \begin{array}{ll} 1 & \mbox{if } y\mbox{ has been assigned to any }(X_i)_i\mbox{ with an }IoU \geq 0.5 \\ 0 & \mbox{otherwise.} \end{array} \right.\end{split}\]

where \(\mathcal{B}\) is the set of possible bounding boxes, \(N\) (number of ground truths) and \(M\) (number of predictions) are strictly positive integers.

>>> import numpy as np
>>> from doctr.utils import LocalizationConfusion
>>> metric = LocalizationConfusion(iou_thresh=0.5)
>>> metric.update(np.asarray([[0, 0, 100, 100]]), np.asarray([[0, 0, 70, 70], [110, 95, 200, 150]]))
>>> metric.summary()
Parameters:
  • iou_thresh (float) – minimum IoU to consider a pair of prediction and ground truth as a match

  • use_polygons (bool) – if set to True, predictions and targets will be expected to have rotated format

update(gts, preds)[source]#

Updates the metric

Parameters:
  • gts (ndarray) – a set of relative bounding boxes either of shape (N, 4) or (N, 5) if they are rotated ones

  • preds (ndarray) – a set of relative bounding boxes either of shape (M, 4) or (M, 5) if they are rotated ones

summary()[source]#

Computes the aggregated metrics

Returns:

a tuple with the recall, precision and meanIoU scores

Return type:

tuple[float | None, float | None, float | None]

class OCRMetric(iou_thresh=0.5, use_polygons=False)[source]#

Implements an end-to-end OCR metric.

The aggregated metrics are computed as follows:

\[\begin{split}\forall (B, L) \in \mathcal{B}^N \times \mathcal{L}^N, \forall (\hat{B}, \hat{L}) \in \mathcal{B}^M \times \mathcal{L}^M, \\ Recall(B, \hat{B}, L, \hat{L}) = \frac{1}{N} \sum\limits_{i=1}^N h_{B,L}(\hat{B}_i, \hat{L}_i) \\ Precision(B, \hat{B}, L, \hat{L}) = \frac{1}{M} \sum\limits_{i=1}^M h_{B,L}(\hat{B}_i, \hat{L}_i) \\ meanIoU(B, \hat{B}) = \frac{1}{M} \sum\limits_{i=1}^M \max\limits_{j \in [1, N]} IoU(\hat{B}_i, B_j)\end{split}\]

with the function \(IoU(x, y)\) being the Intersection over Union between bounding boxes \(x\) and \(y\), and the function \(h_{B, L}\) defined as:

\[\begin{split}\forall (b, l) \in \mathcal{B} \times \mathcal{L}, h_{B,L}(b, l) = \left\{ \begin{array}{ll} 1 & \mbox{if } b\mbox{ has been assigned to a given }B_j\mbox{ with an } \\ & IoU \geq 0.5 \mbox{ and that for this assignment, } l = L_j\\ 0 & \mbox{otherwise.} \end{array} \right.\end{split}\]

where \(\mathcal{B}\) is the set of possible bounding boxes, \(\mathcal{L}\) is the set of possible character sequences, \(N\) (number of ground truths) and \(M\) (number of predictions) are strictly positive integers.

>>> import numpy as np
>>> from doctr.utils import OCRMetric
>>> metric = OCRMetric(iou_thresh=0.5)
>>> metric.update(np.asarray([[0, 0, 100, 100]]), np.asarray([[0, 0, 70, 70], [110, 95, 200, 150]]),
>>>               ['hello'], ['hello', 'world'])
>>> metric.summary()
Parameters:
  • iou_thresh (float) – minimum IoU to consider a pair of prediction and ground truth as a match

  • use_polygons (bool) – if set to True, predictions and targets will be expected to have rotated format

update(gt_boxes, pred_boxes, gt_labels, pred_labels)[source]#

Updates the metric

Parameters:
  • gt_boxes (ndarray) – a set of relative bounding boxes either of shape (N, 4) or (N, 5) if they are rotated ones

  • pred_boxes (ndarray) – a set of relative bounding boxes either of shape (M, 4) or (M, 5) if they are rotated ones

  • gt_labels (list[str]) – a list of N string labels

  • pred_labels (list[str]) – a list of M string labels

summary()[source]#

Computes the aggregated metrics

Returns:

a tuple with the recall & precision for each string comparison and the mean IoU

Return type:

tuple[dict[str, float | None], dict[str, float | None], float | None]

class DetectionMetric(iou_thresh=0.5, use_polygons=False)[source]#

Implements an object detection metric.

The aggregated metrics are computed as follows:

\[\begin{split}\forall (B, C) \in \mathcal{B}^N \times \mathcal{C}^N, \forall (\hat{B}, \hat{C}) \in \mathcal{B}^M \times \mathcal{C}^M, \\ Recall(B, \hat{B}, C, \hat{C}) = \frac{1}{N} \sum\limits_{i=1}^N h_{B,C}(\hat{B}_i, \hat{C}_i) \\ Precision(B, \hat{B}, C, \hat{C}) = \frac{1}{M} \sum\limits_{i=1}^M h_{B,C}(\hat{B}_i, \hat{C}_i) \\ meanIoU(B, \hat{B}) = \frac{1}{M} \sum\limits_{i=1}^M \max\limits_{j \in [1, N]} IoU(\hat{B}_i, B_j)\end{split}\]

with the function \(IoU(x, y)\) being the Intersection over Union between bounding boxes \(x\) and \(y\), and the function \(h_{B, C}\) defined as:

\[\begin{split}\forall (b, c) \in \mathcal{B} \times \mathcal{C}, h_{B,C}(b, c) = \left\{ \begin{array}{ll} 1 & \mbox{if } b\mbox{ has been assigned to a given }B_j\mbox{ with an } \\ & IoU \geq 0.5 \mbox{ and that for this assignment, } c = C_j\\ 0 & \mbox{otherwise.} \end{array} \right.\end{split}\]

where \(\mathcal{B}\) is the set of possible bounding boxes, \(\mathcal{C}\) is the set of possible class indices, \(N\) (number of ground truths) and \(M\) (number of predictions) are strictly positive integers.

>>> import numpy as np
>>> from doctr.utils import DetectionMetric
>>> metric = DetectionMetric(iou_thresh=0.5)
>>> metric.update(np.asarray([[0, 0, 100, 100]]), np.asarray([[0, 0, 70, 70], [110, 95, 200, 150]]),
>>>               np.zeros(1, dtype=np.int64), np.array([0, 1], dtype=np.int64))
>>> metric.summary()
Parameters:
  • iou_thresh (float) – minimum IoU to consider a pair of prediction and ground truth as a match

  • use_polygons (bool) – if set to True, predictions and targets will be expected to have rotated format

update(gt_boxes, pred_boxes, gt_labels, pred_labels)[source]#

Updates the metric

Parameters:
  • gt_boxes (ndarray) – a set of relative bounding boxes either of shape (N, 4) or (N, 5) if they are rotated ones

  • pred_boxes (ndarray) – a set of relative bounding boxes either of shape (M, 4) or (M, 5) if they are rotated ones

  • gt_labels (ndarray) – an array of class indices of shape (N,)

  • pred_labels (ndarray) – an array of class indices of shape (M,)

summary()[source]#

Computes the aggregated metrics

Returns:

a tuple with the recall & precision for each class prediction and the mean IoU

Return type:

tuple[float | None, float | None, float | None]

class ObjectDetectionMetric(iou_thresholds=None, num_classes=None, use_polygons=False)[source]#

Implements a COCO-style object detection metric (mAP@[.5:.95]) inspired by the COCO evaluation protocol.

The aggregated metrics are computed as follows:

\[ \begin{align}\begin{aligned}\begin{split}\forall (B, C) \in \mathcal{B}^N \times \mathcal{C}^N, \forall (\hat{B}, \hat{C}, S) \in \mathcal{B}^M \times \mathcal{C}^M \times \mathbb{R}^M, \\\end{split}\\\begin{split}AP_t(C) = \frac{1}{101} \sum\limits_{r \in \{0, 0.01, \dots, 1.0\}} \max_{\tilde{r} \geq r} Precision_t(\tilde{r}, C) \\\end{split}\\mAP@[.5:.95] = \frac{1}{|\mathcal{T}|} \sum\limits_{t \in \mathcal{T}} \frac{1}{|\mathcal{C}|} \sum\limits_{c \in \mathcal{C}} AP_t(c)\end{aligned}\end{align} \]
where:
  • \(\mathcal{B}\) is the set of possible bounding boxes,

  • \(\mathcal{C}\) is the set of possible class indices,

  • \(S\) are confidence scores associated to predictions,

  • \(\mathcal{T} = \{0.5, 0.55, \dots, 0.95\}\) is the set of IoU thresholds,

  • \(AP_t(c)\) is the Average Precision for class \(c\) at IoU threshold \(t\).

For a given class and IoU threshold, predictions from all images are aggregated and sorted globally by decreasing confidence score.

Each prediction is greedily matched to the unmatched ground-truth box with the highest IoU, provided that:

  • the IoU is greater than or equal to the threshold,

  • the ground-truth box has not already been matched.

True positives and false positives are accumulated to build a precision-recall curve.

Average Precision is computed using the COCO 101-point interpolated precision-recall curve.

>>> import numpy as np
>>> from doctr.utils import ObjectDetectionMetric
>>> metric = ObjectDetectionMetric()
>>> metric.update(
...     np.asarray([[0, 0, 100, 100]]),
...     np.asarray([[0, 0, 80, 80], [120, 120, 200, 200]]),
...     np.asarray([0]),
...     np.asarray([0, 1]),
...     np.asarray([0.9, 0.3])
... )
>>> metric.summary()
Parameters:
  • iou_thresholds (ndarray | None) – sequence of IoU thresholds used to compute the metric (defaults to np.arange(0.5, 1.0, 0.05))

  • num_classes (int | None) – total number of classes. If None, inferred from data

  • use_polygons (bool) – if set to True, predictions and targets will be expected to have rotated format

summary()[source]#

Computes the aggregated metrics

Returns:

a dictionary with the mAP@[.5 – .95], AP@[.5], AP@[.75] and AP per IoU threshold

Return type:

dict[str, float | dict[float, float]]

class TableCellMetric(iou_thresh=0.5, use_polygons=False)[source]#

Implements a table-structure-recognition metric.

Predicted cells are matched to ground-truth cells by maximising the total IoU (Hungarian assignment); a pair counts as a match when its IoU is at least iou_thresh. From the matches it reports cell-detection recall, precision, F1 and the structure accuracy (the fraction of matched cells whose logical coordinates [start_col, end_col, start_row, end_row] exactly equal the ground-truth ones).

>>> import numpy as np
>>> from doctr.utils import TableCellMetric
>>> metric = TableCellMetric(iou_thresh=0.5)
>>> gt = np.array([[0, 0, 1, 1]], dtype=np.float32)
>>> metric.update(gt, np.array([[0, 0, 0, 0]]), gt, np.array([[0, 0, 0, 0]]))
>>> metric.summary()
Parameters:
  • iou_thresh (float) – minimum IoU for a predicted/ground-truth cell pair to be considered a match

  • use_polygons (bool) – if set to True, predictions and targets will be expected to have polygon format

update(gt_cells, gt_logic, pred_cells, pred_logic)[source]#

Update the metric with one sample.

Parameters:
  • gt_cells (ndarray) – ground-truth cells, shape (N, 4) or (N, 4, 2) when use_polygons=True

  • gt_logic (ndarray) – ground-truth logical coordinates, shape (N, 4)

  • pred_cells (ndarray) – predicted cells, shape (M, 4) or (M, 4, 2) when use_polygons=True

  • pred_logic (ndarray) – predicted logical coordinates, shape (M, 4)

summary()[source]#

Compute the aggregated metrics.

Returns:

a dict with recall, precision, f1 (cell detection) and structure_acc

Return type:

dict[str, float | None]

class FormulaMetric[source]#

Implements formula recognition metrics, computed on LaTeX tokens (cf. doctr.datasets.tokenize_latex()), so they do not depend on the whitespaces of the LaTeX sources.

It reports the exact match rate (the share of formulas whose token sequences are identical), the edit similarity (the mean over formulas of \(1 - \frac{d(X, Y)}{\max(|X|, |Y|)}\) with \(d\) the token-level Levenshtein distance) and the corpus-level BLEU-4 score (geometric mean of the 1- to 4-gram precisions with a brevity penalty, without smoothing).

>>> from doctr.utils import FormulaMetric
>>> metric = FormulaMetric()
>>> metric.update([r"\frac{a}{b}"], [r"\frac { a } { c }"])
>>> metric.summary()
update(gt, pred)[source]#

Update the state of the metric with new predictions

Parameters:
  • gt (list[str]) – list of ground-truth LaTeX strings

  • pred (list[str]) – list of predicted LaTeX strings

summary()[source]#

Computes the aggregated metrics

Returns:

a dictionary with the exact_match rate, the mean edit_similarity and the corpus bleu score

Return type:

dict[str, float]