doctr.utils#
Complementary tools: visualization, page reconstruction and evaluation metrics.
Visualization#
- visualize_page(
- page,
- image,
- words_only=True,
- display_artefacts=True,
- display_layout=True,
- scale=10,
- interactive=True,
- add_labels=True,
- **kwargs,
Visualize a full page with predicted blocks, lines and words
>>> import numpy as np >>> import matplotlib.pyplot as plt >>> from doctr.utils.visualization import visualize_page >>> from doctr.models import ocr_db_crnn >>> model = ocr_db_crnn(pretrained=True) >>> input_page = (255 * np.random.rand(600, 800, 3)).astype(np.uint8) >>> out = model([[input_page]]) >>> visualize_page(out[0].pages[0].export(), input_page) >>> plt.show()
- Parameters:
image (ndarray) – np array of the page, needs to have the same shape than page[‘dimensions’]
words_only (bool) – whether only words should be displayed
display_artefacts (bool) – whether artefacts should be displayed
display_layout (bool) – whether detected layout regions should be displayed
scale (float) – figsize of the largest windows side
interactive (bool) – whether the plot should be interactive
add_labels (bool) – for static plot, adds text labels on top of bounding box
**kwargs (Any) – keyword arguments for the polygon patch
- Returns:
the matplotlib figure
- Return type:
Figure
- visualize_kie_page(
- page,
- image,
- words_only=False,
- display_artefacts=True,
- display_layout=True,
- scale=10,
- interactive=True,
- add_labels=True,
- **kwargs,
Visualize a full page with predicted blocks, lines and words
>>> import numpy as np >>> import matplotlib.pyplot as plt >>> from doctr.utils.visualization import visualize_page >>> from doctr.models import ocr_db_crnn >>> model = ocr_db_crnn(pretrained=True) >>> input_page = (255 * np.random.rand(600, 800, 3)).astype(np.uint8) >>> out = model([[input_page]]) >>> visualize_kie_page(out[0].pages[0].export(), input_page) >>> plt.show()
- Parameters:
image (ndarray) – np array of the page, needs to have the same shape than page[‘dimensions’]
words_only (bool) – whether only words should be displayed
display_artefacts (bool) – whether artefacts should be displayed
display_layout (bool) – whether detected layout regions should be displayed
scale (float) – figsize of the largest windows side
interactive (bool) – whether the plot should be interactive
add_labels (bool) – for static plot, adds text labels on top of bounding box
**kwargs (Any) – keyword arguments for the polygon patch
- Returns:
the matplotlib figure
- Return type:
Figure
Reconstitution#
- synthesize_page(
- page,
- draw_proba=False,
- font_family=None,
- min_font_size=8,
- max_font_size=50,
- background_color=(255, 255, 255),
- text_color=(0, 0, 0),
- draw_placeholders=False,
Draw the content of the element page (OCR response) on a blank page.
- Parameters:
draw_proba (bool) – if True, draw words in colors to represent confidence. Blue: p=1, red: p=0
font_family (str | None) – family of the font
min_font_size (int) – minimum font size
max_font_size (int) – maximum font size
background_color (tuple[int, int, int]) – RGB color of the page background
text_color (tuple[int, int, int]) – RGB color of the rendered text
draw_placeholders (bool) – if True, draw the grid of the recognized tables and outline the layout regions holding no text of their own
- Returns:
the synthesized page
- Return type:
- synthesize_kie_page(
- page,
- draw_proba=False,
- font_family=None,
- min_font_size=8,
- max_font_size=50,
- background_color=(255, 255, 255),
- text_color=(0, 0, 0),
- draw_placeholders=False,
Draw the content of the element page (KIE OCR response) on a blank page.
- Parameters:
draw_proba (bool) – if True, draw words in colors to represent confidence. Blue: p=1, red: p=0
font_family (str | None) – family of the font
min_font_size (int) – minimum font size
max_font_size (int) – maximum font size
background_color (tuple[int, int, int]) – RGB color of the page background
text_color (tuple[int, int, int]) – RGB color of the rendered text
draw_placeholders (bool) – if True, draw the grid of the recognized tables and outline the layout regions holding no text of their own
- Returns:
the synthesized page
- Return type:
Task evaluation#
Metrics to evaluate models on each task, used by the evaluation scripts of the references and in the model zoo.
- class TextMatch[source]#
Implements text match metric (word-level accuracy) for recognition task.
The raw aggregated metric is computed as follows:
\[\forall X, Y \in \mathcal{W}^N, TextMatch(X, Y) = \frac{1}{N} \sum\limits_{i=1}^N f_{Y_i}(X_i)\]with the indicator function \(f_{a}\) defined as:
\[\begin{split}\forall a, x \in \mathcal{W}, f_a(x) = \left\{ \begin{array}{ll} 1 & \mbox{if } x = a \\ 0 & \mbox{otherwise.} \end{array} \right.\end{split}\]where \(\mathcal{W}\) is the set of all possible character sequences, \(N\) is a strictly positive integer.
>>> from doctr.utils import TextMatch >>> metric = TextMatch() >>> metric.update(['Hello', 'world'], ['hello', 'world']) >>> metric.summary()
- class LocalizationConfusion(iou_thresh=0.5, use_polygons=False)[source]#
Implements common confusion metrics and mean IoU for localization evaluation.
The aggregated metrics are computed as follows:
\[\begin{split}\forall Y \in \mathcal{B}^N, \forall X \in \mathcal{B}^M, \\ Recall(X, Y) = \frac{1}{N} \sum\limits_{i=1}^N g_{X}(Y_i) \\ Precision(X, Y) = \frac{1}{M} \sum\limits_{i=1}^M g_{X}(Y_i) \\ meanIoU(X, Y) = \frac{1}{M} \sum\limits_{i=1}^M \max\limits_{j \in [1, N]} IoU(X_i, Y_j)\end{split}\]with the function \(IoU(x, y)\) being the Intersection over Union between bounding boxes \(x\) and \(y\), and the function \(g_{X}\) defined as:
\[\begin{split}\forall y \in \mathcal{B}, g_X(y) = \left\{ \begin{array}{ll} 1 & \mbox{if } y\mbox{ has been assigned to any }(X_i)_i\mbox{ with an }IoU \geq 0.5 \\ 0 & \mbox{otherwise.} \end{array} \right.\end{split}\]where \(\mathcal{B}\) is the set of possible bounding boxes, \(N\) (number of ground truths) and \(M\) (number of predictions) are strictly positive integers.
>>> import numpy as np >>> from doctr.utils import LocalizationConfusion >>> metric = LocalizationConfusion(iou_thresh=0.5) >>> metric.update(np.asarray([[0, 0, 100, 100]]), np.asarray([[0, 0, 70, 70], [110, 95, 200, 150]])) >>> metric.summary()
- Parameters:
- class OCRMetric(iou_thresh=0.5, use_polygons=False)[source]#
Implements an end-to-end OCR metric.
The aggregated metrics are computed as follows:
\[\begin{split}\forall (B, L) \in \mathcal{B}^N \times \mathcal{L}^N, \forall (\hat{B}, \hat{L}) \in \mathcal{B}^M \times \mathcal{L}^M, \\ Recall(B, \hat{B}, L, \hat{L}) = \frac{1}{N} \sum\limits_{i=1}^N h_{B,L}(\hat{B}_i, \hat{L}_i) \\ Precision(B, \hat{B}, L, \hat{L}) = \frac{1}{M} \sum\limits_{i=1}^M h_{B,L}(\hat{B}_i, \hat{L}_i) \\ meanIoU(B, \hat{B}) = \frac{1}{M} \sum\limits_{i=1}^M \max\limits_{j \in [1, N]} IoU(\hat{B}_i, B_j)\end{split}\]with the function \(IoU(x, y)\) being the Intersection over Union between bounding boxes \(x\) and \(y\), and the function \(h_{B, L}\) defined as:
\[\begin{split}\forall (b, l) \in \mathcal{B} \times \mathcal{L}, h_{B,L}(b, l) = \left\{ \begin{array}{ll} 1 & \mbox{if } b\mbox{ has been assigned to a given }B_j\mbox{ with an } \\ & IoU \geq 0.5 \mbox{ and that for this assignment, } l = L_j\\ 0 & \mbox{otherwise.} \end{array} \right.\end{split}\]where \(\mathcal{B}\) is the set of possible bounding boxes, \(\mathcal{L}\) is the set of possible character sequences, \(N\) (number of ground truths) and \(M\) (number of predictions) are strictly positive integers.
>>> import numpy as np >>> from doctr.utils import OCRMetric >>> metric = OCRMetric(iou_thresh=0.5) >>> metric.update(np.asarray([[0, 0, 100, 100]]), np.asarray([[0, 0, 70, 70], [110, 95, 200, 150]]), >>> ['hello'], ['hello', 'world']) >>> metric.summary()
- Parameters:
- update(gt_boxes, pred_boxes, gt_labels, pred_labels)[source]#
Updates the metric
- class DetectionMetric(iou_thresh=0.5, use_polygons=False)[source]#
Implements an object detection metric.
The aggregated metrics are computed as follows:
\[\begin{split}\forall (B, C) \in \mathcal{B}^N \times \mathcal{C}^N, \forall (\hat{B}, \hat{C}) \in \mathcal{B}^M \times \mathcal{C}^M, \\ Recall(B, \hat{B}, C, \hat{C}) = \frac{1}{N} \sum\limits_{i=1}^N h_{B,C}(\hat{B}_i, \hat{C}_i) \\ Precision(B, \hat{B}, C, \hat{C}) = \frac{1}{M} \sum\limits_{i=1}^M h_{B,C}(\hat{B}_i, \hat{C}_i) \\ meanIoU(B, \hat{B}) = \frac{1}{M} \sum\limits_{i=1}^M \max\limits_{j \in [1, N]} IoU(\hat{B}_i, B_j)\end{split}\]with the function \(IoU(x, y)\) being the Intersection over Union between bounding boxes \(x\) and \(y\), and the function \(h_{B, C}\) defined as:
\[\begin{split}\forall (b, c) \in \mathcal{B} \times \mathcal{C}, h_{B,C}(b, c) = \left\{ \begin{array}{ll} 1 & \mbox{if } b\mbox{ has been assigned to a given }B_j\mbox{ with an } \\ & IoU \geq 0.5 \mbox{ and that for this assignment, } c = C_j\\ 0 & \mbox{otherwise.} \end{array} \right.\end{split}\]where \(\mathcal{B}\) is the set of possible bounding boxes, \(\mathcal{C}\) is the set of possible class indices, \(N\) (number of ground truths) and \(M\) (number of predictions) are strictly positive integers.
>>> import numpy as np >>> from doctr.utils import DetectionMetric >>> metric = DetectionMetric(iou_thresh=0.5) >>> metric.update(np.asarray([[0, 0, 100, 100]]), np.asarray([[0, 0, 70, 70], [110, 95, 200, 150]]), >>> np.zeros(1, dtype=np.int64), np.array([0, 1], dtype=np.int64)) >>> metric.summary()
- Parameters:
- update(gt_boxes, pred_boxes, gt_labels, pred_labels)[source]#
Updates the metric
- Parameters:
gt_boxes (ndarray) – a set of relative bounding boxes either of shape (N, 4) or (N, 5) if they are rotated ones
pred_boxes (ndarray) – a set of relative bounding boxes either of shape (M, 4) or (M, 5) if they are rotated ones
gt_labels (ndarray) – an array of class indices of shape (N,)
pred_labels (ndarray) – an array of class indices of shape (M,)
- class ObjectDetectionMetric(iou_thresholds=None, num_classes=None, use_polygons=False)[source]#
Implements a COCO-style object detection metric (mAP@[.5:.95]) inspired by the COCO evaluation protocol.
The aggregated metrics are computed as follows:
\[ \begin{align}\begin{aligned}\begin{split}\forall (B, C) \in \mathcal{B}^N \times \mathcal{C}^N, \forall (\hat{B}, \hat{C}, S) \in \mathcal{B}^M \times \mathcal{C}^M \times \mathbb{R}^M, \\\end{split}\\\begin{split}AP_t(C) = \frac{1}{101} \sum\limits_{r \in \{0, 0.01, \dots, 1.0\}} \max_{\tilde{r} \geq r} Precision_t(\tilde{r}, C) \\\end{split}\\mAP@[.5:.95] = \frac{1}{|\mathcal{T}|} \sum\limits_{t \in \mathcal{T}} \frac{1}{|\mathcal{C}|} \sum\limits_{c \in \mathcal{C}} AP_t(c)\end{aligned}\end{align} \]- where:
\(\mathcal{B}\) is the set of possible bounding boxes,
\(\mathcal{C}\) is the set of possible class indices,
\(S\) are confidence scores associated to predictions,
\(\mathcal{T} = \{0.5, 0.55, \dots, 0.95\}\) is the set of IoU thresholds,
\(AP_t(c)\) is the Average Precision for class \(c\) at IoU threshold \(t\).
For a given class and IoU threshold, predictions from all images are aggregated and sorted globally by decreasing confidence score.
Each prediction is greedily matched to the unmatched ground-truth box with the highest IoU, provided that:
the IoU is greater than or equal to the threshold,
the ground-truth box has not already been matched.
True positives and false positives are accumulated to build a precision-recall curve.
Average Precision is computed using the COCO 101-point interpolated precision-recall curve.
>>> import numpy as np >>> from doctr.utils import ObjectDetectionMetric >>> metric = ObjectDetectionMetric() >>> metric.update( ... np.asarray([[0, 0, 100, 100]]), ... np.asarray([[0, 0, 80, 80], [120, 120, 200, 200]]), ... np.asarray([0]), ... np.asarray([0, 1]), ... np.asarray([0.9, 0.3]) ... ) >>> metric.summary()
- Parameters:
iou_thresholds (ndarray | None) – sequence of IoU thresholds used to compute the metric (defaults to np.arange(0.5, 1.0, 0.05))
num_classes (int | None) – total number of classes. If None, inferred from data
use_polygons (bool) – if set to True, predictions and targets will be expected to have rotated format
- class TableCellMetric(iou_thresh=0.5, use_polygons=False)[source]#
Implements a table-structure-recognition metric.
Predicted cells are matched to ground-truth cells by maximising the total IoU (Hungarian assignment); a pair counts as a match when its IoU is at least iou_thresh. From the matches it reports cell-detection recall, precision, F1 and the structure accuracy (the fraction of matched cells whose logical coordinates [start_col, end_col, start_row, end_row] exactly equal the ground-truth ones).
>>> import numpy as np >>> from doctr.utils import TableCellMetric >>> metric = TableCellMetric(iou_thresh=0.5) >>> gt = np.array([[0, 0, 1, 1]], dtype=np.float32) >>> metric.update(gt, np.array([[0, 0, 0, 0]]), gt, np.array([[0, 0, 0, 0]])) >>> metric.summary()
- Parameters:
- update(gt_cells, gt_logic, pred_cells, pred_logic)[source]#
Update the metric with one sample.
- Parameters:
gt_cells (ndarray) – ground-truth cells, shape (N, 4) or (N, 4, 2) when use_polygons=True
gt_logic (ndarray) – ground-truth logical coordinates, shape (N, 4)
pred_cells (ndarray) – predicted cells, shape (M, 4) or (M, 4, 2) when use_polygons=True
pred_logic (ndarray) – predicted logical coordinates, shape (M, 4)
- class FormulaMetric[source]#
Implements formula recognition metrics, computed on LaTeX tokens (cf.
doctr.datasets.tokenize_latex()), so they do not depend on the whitespaces of the LaTeX sources.It reports the exact match rate (the share of formulas whose token sequences are identical), the edit similarity (the mean over formulas of \(1 - \frac{d(X, Y)}{\max(|X|, |Y|)}\) with \(d\) the token-level Levenshtein distance) and the corpus-level BLEU-4 score (geometric mean of the 1- to 4-gram precisions with a brevity penalty, without smoothing).
>>> from doctr.utils import FormulaMetric >>> metric = FormulaMetric() >>> metric.update([r"\frac{a}{b}"], [r"\frac { a } { c }"]) >>> metric.summary()