doctr.utils#
Complementary tools: visualization, page reconstruction and evaluation metrics.
Visualization#
- visualize_page(
- page,
- image,
- words_only=True,
- display_artefacts=True,
- display_layout=True,
- scale=10,
- interactive=True,
- add_labels=True,
- **kwargs,
Visualize a full page with predicted blocks, lines and words
>>> import numpy as np >>> import matplotlib.pyplot as plt >>> from doctr.utils.visualization import visualize_page >>> from doctr.models import ocr_db_crnn >>> model = ocr_db_crnn(pretrained=True) >>> input_page = (255 * np.random.rand(600, 800, 3)).astype(np.uint8) >>> out = model([[input_page]]) >>> visualize_page(out[0].pages[0].export(), input_page) >>> plt.show()
- Parameters:
image (ndarray) – np array of the page, needs to have the same shape than page[‘dimensions’]
words_only (bool) – whether only words should be displayed
display_artefacts (bool) – whether artefacts should be displayed
display_layout (bool) – whether detected layout regions should be displayed
scale (float) – figsize of the largest windows side
interactive (bool) – whether the plot should be interactive
add_labels (bool) – for static plot, adds text labels on top of bounding box
**kwargs (Any) – keyword arguments for the polygon patch
- Returns:
the matplotlib figure
- Return type:
Figure
- visualize_kie_page(
- page,
- image,
- words_only=False,
- display_artefacts=True,
- display_layout=True,
- scale=10,
- interactive=True,
- add_labels=True,
- **kwargs,
Visualize a full page with predicted blocks, lines and words
>>> import numpy as np >>> import matplotlib.pyplot as plt >>> from doctr.utils.visualization import visualize_page >>> from doctr.models import ocr_db_crnn >>> model = ocr_db_crnn(pretrained=True) >>> input_page = (255 * np.random.rand(600, 800, 3)).astype(np.uint8) >>> out = model([[input_page]]) >>> visualize_kie_page(out[0].pages[0].export(), input_page) >>> plt.show()
- Parameters:
image (ndarray) – np array of the page, needs to have the same shape than page[‘dimensions’]
words_only (bool) – whether only words should be displayed
display_artefacts (bool) – whether artefacts should be displayed
display_layout (bool) – whether detected layout regions should be displayed
scale (float) – figsize of the largest windows side
interactive (bool) – whether the plot should be interactive
add_labels (bool) – for static plot, adds text labels on top of bounding box
**kwargs (Any) – keyword arguments for the polygon patch
- Returns:
the matplotlib figure
- Return type:
Figure
Reconstitution#
- synthesize_page(
- page,
- draw_proba=False,
- font_family=None,
- min_font_size=8,
- max_font_size=50,
- background_color=(255, 255, 255),
- text_color=(0, 0, 0),
- draw_placeholders=False,
Draw the content of the element page (OCR response) on a blank page.
- Parameters:
draw_proba (bool) – if True, draw words in colors to represent confidence. Blue: p=1, red: p=0
font_family (str | None) – family of the font
min_font_size (int) – minimum font size
max_font_size (int) – maximum font size
background_color (tuple[int, int, int]) – RGB color of the page background
text_color (tuple[int, int, int]) – RGB color of the rendered text
draw_placeholders (bool) – if True, draw the grid of the recognized tables and outline the layout regions holding no text of their own
- Returns:
the synthesized page
- Return type:
- synthesize_kie_page(
- page,
- draw_proba=False,
- font_family=None,
- min_font_size=8,
- max_font_size=50,
- background_color=(255, 255, 255),
- text_color=(0, 0, 0),
- draw_placeholders=False,
Draw the content of the element page (KIE OCR response) on a blank page.
- Parameters:
draw_proba (bool) – if True, draw words in colors to represent confidence. Blue: p=1, red: p=0
font_family (str | None) – family of the font
min_font_size (int) – minimum font size
max_font_size (int) – maximum font size
background_color (tuple[int, int, int]) – RGB color of the page background
text_color (tuple[int, int, int]) – RGB color of the rendered text
draw_placeholders (bool) – if True, draw the grid of the recognized tables and outline the layout regions holding no text of their own
- Returns:
the synthesized page
- Return type:
Task evaluation#
Metrics to evaluate models on each task, used by the evaluation scripts of the references and in the model zoo.
- class TextMatch[source]#
Implements text match metric (word-level accuracy) for recognition task.
The raw aggregated metric is computed as follows:
\[\forall X, Y \in \mathcal{W}^N, TextMatch(X, Y) = \frac{1}{N} \sum\limits_{i=1}^N f_{Y_i}(X_i)\]with the indicator function \(f_{a}\) defined as:
\[\begin{split}\forall a, x \in \mathcal{W}, f_a(x) = \left\{ \begin{array}{ll} 1 & \mbox{if } x = a \\ 0 & \mbox{otherwise.} \end{array} \right.\end{split}\]where \(\mathcal{W}\) is the set of all possible character sequences, \(N\) is a strictly positive integer.
>>> from doctr.utils import TextMatch >>> metric = TextMatch() >>> metric.update(['Hello', 'world'], ['hello', 'world']) >>> metric.summary()
- class LocalizationConfusion(iou_thresh=0.5, use_polygons=False)[source]#
Implements common confusion metrics and mean IoU for localization evaluation.
The aggregated metrics are computed as follows:
\[\begin{split}\forall Y \in \mathcal{B}^N, \forall X \in \mathcal{B}^M, \\ Recall(X, Y) = \frac{1}{N} \sum\limits_{i=1}^N g_{X}(Y_i) \\ Precision(X, Y) = \frac{1}{M} \sum\limits_{i=1}^M g_{X}(Y_i) \\ meanIoU(X, Y) = \frac{1}{M} \sum\limits_{i=1}^M \max\limits_{j \in [1, N]} IoU(X_i, Y_j)\end{split}\]with the function \(IoU(x, y)\) being the Intersection over Union between bounding boxes \(x\) and \(y\), and the function \(g_{X}\) defined as:
\[\begin{split}\forall y \in \mathcal{B}, g_X(y) = \left\{ \begin{array}{ll} 1 & \mbox{if } y\mbox{ has been assigned to any }(X_i)_i\mbox{ with an }IoU \geq 0.5 \\ 0 & \mbox{otherwise.} \end{array} \right.\end{split}\]where \(\mathcal{B}\) is the set of possible bounding boxes, \(N\) (number of ground truths) and \(M\) (number of predictions) are strictly positive integers.
>>> import numpy as np >>> from doctr.utils import LocalizationConfusion >>> metric = LocalizationConfusion(iou_thresh=0.5) >>> metric.update(np.asarray([[0, 0, 100, 100]]), np.asarray([[0, 0, 70, 70], [110, 95, 200, 150]])) >>> metric.summary()
- Parameters:
- class OCRMetric(iou_thresh=0.5, use_polygons=False)[source]#
Implements an end-to-end OCR metric.
The aggregated metrics are computed as follows:
\[\begin{split}\forall (B, L) \in \mathcal{B}^N \times \mathcal{L}^N, \forall (\hat{B}, \hat{L}) \in \mathcal{B}^M \times \mathcal{L}^M, \\ Recall(B, \hat{B}, L, \hat{L}) = \frac{1}{N} \sum\limits_{i=1}^N h_{B,L}(\hat{B}_i, \hat{L}_i) \\ Precision(B, \hat{B}, L, \hat{L}) = \frac{1}{M} \sum\limits_{i=1}^M h_{B,L}(\hat{B}_i, \hat{L}_i) \\ meanIoU(B, \hat{B}) = \frac{1}{M} \sum\limits_{i=1}^M \max\limits_{j \in [1, N]} IoU(\hat{B}_i, B_j)\end{split}\]with the function \(IoU(x, y)\) being the Intersection over Union between bounding boxes \(x\) and \(y\), and the function \(h_{B, L}\) defined as:
\[\begin{split}\forall (b, l) \in \mathcal{B} \times \mathcal{L}, h_{B,L}(b, l) = \left\{ \begin{array}{ll} 1 & \mbox{if } b\mbox{ has been assigned to a given }B_j\mbox{ with an } \\ & IoU \geq 0.5 \mbox{ and that for this assignment, } l = L_j\\ 0 & \mbox{otherwise.} \end{array} \right.\end{split}\]where \(\mathcal{B}\) is the set of possible bounding boxes, \(\mathcal{L}\) is the set of possible character sequences, \(N\) (number of ground truths) and \(M\) (number of predictions) are strictly positive integers.
>>> import numpy as np >>> from doctr.utils import OCRMetric >>> metric = OCRMetric(iou_thresh=0.5) >>> metric.update(np.asarray([[0, 0, 100, 100]]), np.asarray([[0, 0, 70, 70], [110, 95, 200, 150]]), >>> ['hello'], ['hello', 'world']) >>> metric.summary()
- Parameters:
- update(gt_boxes, pred_boxes, gt_labels, pred_labels)[source]#
Updates the metric
- class DetectionMetric(iou_thresh=0.5, use_polygons=False)[source]#
Implements an object detection metric.
The aggregated metrics are computed as follows:
\[\begin{split}\forall (B, C) \in \mathcal{B}^N \times \mathcal{C}^N, \forall (\hat{B}, \hat{C}) \in \mathcal{B}^M \times \mathcal{C}^M, \\ Recall(B, \hat{B}, C, \hat{C}) = \frac{1}{N} \sum\limits_{i=1}^N h_{B,C}(\hat{B}_i, \hat{C}_i) \\ Precision(B, \hat{B}, C, \hat{C}) = \frac{1}{M} \sum\limits_{i=1}^M h_{B,C}(\hat{B}_i, \hat{C}_i) \\ meanIoU(B, \hat{B}) = \frac{1}{M} \sum\limits_{i=1}^M \max\limits_{j \in [1, N]} IoU(\hat{B}_i, B_j)\end{split}\]with the function \(IoU(x, y)\) being the Intersection over Union between bounding boxes \(x\) and \(y\), and the function \(h_{B, C}\) defined as:
\[\begin{split}\forall (b, c) \in \mathcal{B} \times \mathcal{C}, h_{B,C}(b, c) = \left\{ \begin{array}{ll} 1 & \mbox{if } b\mbox{ has been assigned to a given }B_j\mbox{ with an } \\ & IoU \geq 0.5 \mbox{ and that for this assignment, } c = C_j\\ 0 & \mbox{otherwise.} \end{array} \right.\end{split}\]where \(\mathcal{B}\) is the set of possible bounding boxes, \(\mathcal{C}\) is the set of possible class indices, \(N\) (number of ground truths) and \(M\) (number of predictions) are strictly positive integers.
>>> import numpy as np >>> from doctr.utils import DetectionMetric >>> metric = DetectionMetric(iou_thresh=0.5) >>> metric.update(np.asarray([[0, 0, 100, 100]]), np.asarray([[0, 0, 70, 70], [110, 95, 200, 150]]), >>> np.zeros(1, dtype=np.int64), np.array([0, 1], dtype=np.int64)) >>> metric.summary()
- Parameters:
- update(gt_boxes, pred_boxes, gt_labels, pred_labels)[source]#
Updates the metric
- Parameters:
gt_boxes (ndarray) – a set of relative bounding boxes either of shape (N, 4) or (N, 5) if they are rotated ones
pred_boxes (ndarray) – a set of relative bounding boxes either of shape (M, 4) or (M, 5) if they are rotated ones
gt_labels (ndarray) – an array of class indices of shape (N,)
pred_labels (ndarray) – an array of class indices of shape (M,)
- class ObjectDetectionMetric(iou_thresholds=None, num_classes=None, use_polygons=False)[source]#
Implements a COCO-style object detection metric (mAP@[.5:.95]) inspired by the COCO evaluation protocol.
The aggregated metrics are computed as follows:
\[ \begin{align}\begin{aligned}\begin{split}\forall (B, C) \in \mathcal{B}^N \times \mathcal{C}^N, \forall (\hat{B}, \hat{C}, S) \in \mathcal{B}^M \times \mathcal{C}^M \times \mathbb{R}^M, \\\end{split}\\\begin{split}AP_t(C) = \frac{1}{101} \sum\limits_{r \in \{0, 0.01, \dots, 1.0\}} \max_{\tilde{r} \geq r} Precision_t(\tilde{r}, C) \\\end{split}\\mAP@[.5:.95] = \frac{1}{|\mathcal{T}|} \sum\limits_{t \in \mathcal{T}} \frac{1}{|\mathcal{C}|} \sum\limits_{c \in \mathcal{C}} AP_t(c)\end{aligned}\end{align} \]- where:
\(\mathcal{B}\) is the set of possible bounding boxes,
\(\mathcal{C}\) is the set of possible class indices,
\(S\) are confidence scores associated to predictions,
\(\mathcal{T} = \{0.5, 0.55, \dots, 0.95\}\) is the set of IoU thresholds,
\(AP_t(c)\) is the Average Precision for class \(c\) at IoU threshold \(t\).
For a given class and IoU threshold, predictions from all images are aggregated and sorted globally by decreasing confidence score.
Each prediction is greedily matched to the unmatched ground-truth box with the highest IoU, provided that:
the IoU is greater than or equal to the threshold,
the ground-truth box has not already been matched.
True positives and false positives are accumulated to build a precision-recall curve.
Average Precision is computed using the COCO 101-point interpolated precision-recall curve.
>>> import numpy as np >>> from doctr.utils import ObjectDetectionMetric >>> metric = ObjectDetectionMetric() >>> metric.update( ... np.asarray([[0, 0, 100, 100]]), ... np.asarray([[0, 0, 80, 80], [120, 120, 200, 200]]), ... np.asarray([0]), ... np.asarray([0, 1]), ... np.asarray([0.9, 0.3]) ... ) >>> metric.summary()
- Parameters:
iou_thresholds (ndarray | None) – sequence of IoU thresholds used to compute the metric (defaults to np.arange(0.5, 1.0, 0.05))
num_classes (int | None) – total number of classes. If None, inferred from data
use_polygons (bool) – if set to True, predictions and targets will be expected to have rotated format
- class TableCellMetric(iou_thresh=0.5, use_polygons=False)[source]#
Implements a table-structure-recognition metric.
Predicted cells are matched to ground-truth cells by maximising the total IoU (Hungarian assignment); a pair counts as a match when its IoU is at least iou_thresh. From the matches it reports cell-detection recall, precision, F1 and the structure accuracy (the fraction of matched cells whose logical coordinates [start_col, end_col, start_row, end_row] exactly equal the ground-truth ones).
>>> import numpy as np >>> from doctr.utils import TableCellMetric >>> metric = TableCellMetric(iou_thresh=0.5) >>> gt = np.array([[0, 0, 1, 1]], dtype=np.float32) >>> metric.update(gt, np.array([[0, 0, 0, 0]]), gt, np.array([[0, 0, 0, 0]])) >>> metric.summary()
- Parameters:
- update(gt_cells, gt_logic, pred_cells, pred_logic)[source]#
Update the metric with one sample.
- Parameters:
gt_cells (ndarray) – ground-truth cells, shape (N, 4) or (N, 4, 2) when use_polygons=True
gt_logic (ndarray) – ground-truth logical coordinates, shape (N, 4)
pred_cells (ndarray) – predicted cells, shape (M, 4) or (M, 4, 2) when use_polygons=True
pred_logic (ndarray) – predicted logical coordinates, shape (M, 4)