Workflows

There are several steps necessary to get the fulltext of a scanned print. The whole OCR process is shown in the following figure:

The following instructions describe all steps of an OCR workflow. Depending on your particular print (or rather images), not all of those steps might be necessary to obtain good results. Whether a step is required or optional is indicated in the description of each step. This guide provides an overview of the available OCR-D processors and their required parameters. For more complex workflows and recommendations see the OCR-D-Website-Wiki. Feel free to add your own experiences and recommendations in the Wiki! We will regularly amend this guide with valuable contributions from the Wiki.

Note: In order to be able to run the workflows described in this guide, you need to have prepared your images in an OCR-D-workspace. We expect that you are familiar with the OCR-D-user guide which explains all preparatory steps, syntax and different solutions for executing whole workflows.

Image Optimization (Page Level)

At first, the image should be prepared for OCR.

Step 0.1: Image Enhancement (Page Level, optional)

Optionally, you can start off your workflow by enhancing your images, which can be vital for the following binarization. In this processing step, the raw image is taken and enhanced by e.g. grayscale conversion, brightness normalization, noise filtering, etc.

Note: ocrd-preprocess-image can be used to run arbitrary shell commands for preprocessing (original or derived) images, and can be seen as a generic OCR-D wrapper for many of the following workflow steps, provided a matching external tool exists. (The only restriction is that the tool must not change image size or the position/coordinates of its content.)

Available processors

Step 0.2: Font detection

Optionally, this processor can determine the font family (e.g. Antiqua, Fraktur, Schwabacher) to help select the right models for text detection.

ocrd-typegroups-classifier annotates font families on page level, including the confidence value (separated by colon). Supported fontFamily values:

• Antiqua
• Bastarda
• Fraktur
• Gotico-Antiqua
• Greek
• Hebrew
• Italic
• Rotunda
• Schwabacher
• Textura
• other_font
• not_a_font

Note: ocrd-typegroups-classifier was trained on a very large and diverse dataset, with both geometric and color-space random augmentation (contrast, brightness, hue, even compression artifacts and 2 different binarization methods), so it works best on the raw, non-binarized RGB image.

Note: ocrd-typegroups-classifier comes with a non-OCR-D CLI that allows for the generation of “heatmaps” on the page to visualize which regions of the page are classified as using a certain font with a certain confidence, see the project’s README for usage instructions.

Available processors

Step 1: Binarization (Page Level)

All the images should be binarized right at the beginning of your workflow. Many of the following processors require binarized images. Some implementations (for deskewing, segmentation or recognition) may produce better results using the original image. But these can always retrieve the raw image instead of the binarized version automatically.

In this processing step, a scanned colored /gray scale document image is taken as input and a black and white binarized image is produced. This step should separate the background from the foreground.

Note: Binarization tools usually provide a threshold parameter which allows you to increase or decrease the weight of the foreground. This is optional and can be especially useful for images which have not been enhanced.

Available processors

Step 2: Cropping (Page Level)

In this processing step, a document image is taken as input and the page is cropped to the content area only (i.e. without noise at the margins or facing pages) by marking the coordinates of the page frame. We strongly recommend to execute this step if your images are not cropped already (i.e. only show the page of a book without a ruler, footer, color scale etc.). Otherwise you might run into severe segmentation problems.

Available processors

Step 3: Binarization (Page Level)

For better results, the cropped images can be binarized again at this point or later on (on region level).

Available processors

Step 4: Denoising (Page Level)

In this processing step, artifacts like little specks (both in foreground or background) are removed from the binarized image. (Not to be confused with raw denoising in step 0.)

This may not be necessary for all prints, and depends heavily on the selected binarization algorithm.

Available processors

Step 5: Deskewing (Page Level)

In this processing step, a document image is taken as input and the skew of that page is corrected by annotating the detected angle (-45° .. 45°) and rotating the image. Optionally, also the orientation is corrected by annotating the detected angle (multiples of 90°) and transposing the image. The input images have to be binarized for this module to work.

Available processors

Step 6: Dewarping (Page Level)

In this processing step, a document image is taken as input and the text lines are straightened or stretched if they are curved. The input image has to be binarized for the module to work.

Available processors

Layout Analysis

By now the image should be well prepared for segmentation.

Step 7: Region segmentation

In this processing step, an (optimized) document image is taken as an input and the image is segmented into the various regions, including columns. Segments are also classified, either coarse (text, separator, image, table, …) or fine-grained (paragraph, marginalia, heading, …).

Note: The ocrd-tesserocr-segment, ocrd-tesserocr-recognize, ocrd-eynollah-segment, ocrd-sbb-textline-detector and ocrd-cis-ocropy-segment processors do not only segment the page, but also the text lines within the detected text regions in one step. Therefore with those (and only with those!) processors you don’t need to segment into lines in an extra step and can continue with step 13 - line-level dewarping.

Note: If you use ocrd-tesserocr-segment-region, which uses only bounding boxes instead of polygon coordinates, then you should post-process via ocrd-segment-repair with plausibilize=True to obtain better results without large overlaps. Alternatively, consider using the all-in-one capabilities of ocrd-tesserocr-segment and ocrd-tesserocr-recognize, which can do region segmentation and line segmentation (and optionally also text recognition) in one step by querying Tesseract’s internal iterator (accessing the more precise polygon outlines instead of just coarse bounding boxes with lots of hard-to-recover overlap). Alternatively, run with shrink_polygons=True (accessing that same iterator to calculate convex hull polygons).

Note: All the ocrd-tesserocr-segment* processors internally delegate to ocrd-tesserocr-recognize, so you can replace calls to these task-specific processors with calls to ocrd-tesserocr-recognize with specific parameters:

Note: The three parameters segmentation_level, textequiv_level and model define the behavior of ocrd-tesserocr-recognize:

• segmentation_level determines the highest level to segment. Use "none" to disable segmentation altogether, i.e. only recognize existing segments.
• textequiv_level determines the lowest level to segment. Use "none" to segment until the lowest level ("glyph") and disable recognition altogether, only analyse layout.
• model determines the model to use for text recognition. Use "" or do not set at all to disable recognition, i.e. only analyse layout.

Examples:

• To segment existing regions into lines (and only lines) only: segmentation_level="line", textequiv_level="line", model=""
• To segment existing regions into lines (and only lines) and recognize text: segmentation_level="line", textequiv_level="line", model="Fraktur"

For detailed descriptions of behaviour and options, see tesserocr’s README and ocrd-tesserocr-recognize/segment/segment-region/segment-table/segment-line/segment-word --help help.

Available processors

Image Optimization (Region Level)

In the following steps, the text regions should be optimized for OCR.

Step 8: Binarization (Region Level)

In this processing step, a scanned colored /gray scale document image is taken as input and a black and white binarized image is produced. This step should separate the background from the foreground.

The binarization should be at least executed once (on page or region level). If you already binarized your image twice on page level, and have no large images, you can probably skip this step.

Available processors

Step 9: Clipping (Region Level)

In this processing step, intrusions of neighbouring non-text (e.g. separator) or text segments (e.g. ascenders/descenders) into text regions of a page (or text lines or a text region) can be removed. A connected component analysis is run on every segment, as well as its overlapping neighbours. Now for each conflicting binary object, a rule based on majority and proper containment determines whether it belongs to the neighbour, and can therefore be clipped to the background.

This basic text-nontext segmentation ensures that for each text region there is a clean image without interference from separators and neighbouring texts. (On the region level, cleaning via coordinates would be impossible in many common cases.) On the line level, this can be seen as an alternative to resegmentation.

Note: Clipping must be applied before any processor that produces derived images for the same hierarchy level (region/line). Annotations on the next higher level (page/region) are fine of course.

Available processors

Step 10: Deskewing (Region Level)

In this processing step, text region images are taken as input and their skew is corrected by annotating the detected angle (-45° .. 45°) and rotating the image. Optionally, also the orientation is corrected by annotating the detected angle (multiples of 90°) and transposing the image.

Available processors

Step 11: Line segmentation

In this processing step, text regions are segmented into text lines. A line detection algorithm is run on every text region of every PAGE in the input file group, and a TextLine element with the resulting polygon outline is added to the annotation of the output PAGE.

Note: If you use ocrd-cis-ocropy-segment, you can directly go on with Step 13.

Note: If you use ocrd-tesserocr-segment-line, which uses only bounding boxes instead of polygon coordinates, then you should post-process with the processors described in Step 12. Alternatively, consider using the all-in-one capabilities of ocrd-tesserocr-recognize, which can do line segmentation and text recognition in one step by querying Tesseract’s internal iterator (accessing the more precise polygon outlines instead of just coarse bounding boxes with lots of hard-to-recover overlap). Alternatively, run with shrink_polygons=True (accessing that same iterator to calculate convex hull polygons)

Note: As described in Step 7, ocrd-eynollah-segment, ocrd-sbb-textline-detector and ocrd-cis-ocropy-segment do not only segment the page, but also the text lines within the detected text regions in one step. Therefore with those (and only with those!) processors you don’t need to segment into lines in an extra step.

Available processors

Step 12: Resegmentation (Line Level)

In this processing step the segmented text lines can be corrected in order to reduce their overlap.

This can be done either via coordinates (polygonalizing the bounding boxes tightly around the glyphs) – which is what ocrd-cis-ocropy-resegment and ocrd-segment-project offer – or via derived images (clipping pixels that do not belong to a text line to the background color) – which is what ocrd-cis-ocropy-clip (on the line level) offers. The former is usually more accurate, but not always possible (for example, when neighbors intersect heavily, creating non-contiguous contours). The latter is only possible if no preceding workflow step has already annotated derived images (AlternativeImage references) on the line level (see also region-level clipping).

Available processors

Step 13: Dewarping (Line Level)

In this processing step, the text line images get vertically aligned if they are curved.

Available processors

Text Recognition

Step 14: Text recognition

This processor recognizes text in segmented lines.

An overview on the existing model repositories and short descriptions on the most important models can be found here.

We strongly recommend to use the OCR-D resource manager to download the models, as this way you don’t have to specify the path to each model.

Available processors

Note: For ocrd-tesserocr the environment variable TESSDATA_PREFIX has to be set to point to the directory where the used models are stored unless the default directory (normally $VIRTUAL_ENV/share/tessdata) is used. The directory should at least contain the following models: deu.traineddata, eng.traineddata, osd.traineddata.

Note: Faster models for tesserocr-recognize are available from https://ub-backup.bib.uni-mannheim.de/~stweil/ocrd-train/data/Fraktur_5000000/tessdata_fast/. A good and currently the fastest model is Fraktur-fast. UB Mannheim provides many more models online which were trained on different GT data sets, for example from Austrian Newspapers.

Note: If you want to go on with the optional post correction, you should also set the textequiv_level to glyph or in the case of ocrd-calamari-recognize at least word (which is already the default for ocrd-tesserocr-recognize).

Step 14.1: Font style annotation

This processor can determine the font style (e.g. italic, bold, underlined) and font family text recognition results.

ocrd-tesserocr-fontshape can either use existing segmentation or segment on-demand. It can detect the following font styles:

• fontSize
• fontFamily
• bold
• italic
• underlined
• monospace
• serif

Note: ocrd-tesserocr-fontshape needs the old, pre-LSTM models to work at all. You can use the pre-installed osd (which is purely rule-based), but there might be better alternatives for your language and script. You can still get the old models from Tesseract’s Github repo at the last revision before the LSTM models replaced them, usually under the same name. (Thus, deu.traineddata used to be a rule-based model but now is an LSTM model. deu-frak.traineddata is still only available as rule-based model and was complemented by the new LSTM models frk.traineddata and script/Fraktur.traineddata.) If you do need one of the models that was replaced completely, then you should at least rename the old one (e.g. to deu3.traineddata).

Available processors

Post Correction (Optional)

Step 15: Text alignment

In this processing step, text results from multiple OCR engines (in different annotations sharing the same line segmentation) are aligned into one annotation.

Available processors

Comparison

Step 16: Post-correction

In this processing step, the recognized text is corrected by statistical error modelling, language modelling, and word modelling (dictionaries, morphology and orthography).

Note: Most tools benefit strongly from input which includes alternative OCR hypotheses. Currently, models for ocrd-cor-asv-ann-process are optimised for input from specific OCR models, whereas ocrd-cis-postcorrect expects input from multi-OCR alignment. For more information, see this presentation at vDHd 2021 (held on 23rd May 2021) (slides / video in German)

Note: There is some overlap with text alignment here, which can also be used (or contribute to) post-correction.

Available processors

#!/bin/bash
cat > /dev/null
echo '{}'

Evaluation (Optional)

If Ground Truth data is available, the OCR and layout recognition can be evaluated.

Step 17: Layout Evaluation

In this processing step, GT annotation and segmentation results are matched and evaluated.

Available processors

Step 18: OCR Evaluation

In this processing step, the text output of the OCR or post-correction can be evaluated by aligning with ground truth text and measuring the error rates.

Available processors

Comparison

Generic Data Management (Optional)

OCR-D produces PAGE XML files which contain the recognized text as well as detailed information on the structure of the processed pages, the coordinates of the recognized elements etc. Optionally, the output can be converted to other formats, or copied verbatim (re-generating PAGE-XML)

Step 19: Adaptation of Coordinates

All OCR-D processors are required to relate coordinates to the original image for each page, and to keep the original image reference (Page/@imageFilename). However, sometimes it may be necessary to deviate from that strict requirement in order to get the overall workflow to function properly.

For example, if you have a page-level dewarping step, it is currently impossible to correctly relate to the original image’s coordinates for any segments annotated after that, because there is no descriptive annotation of the underlying coordinate transform in PAGE-XML. Therefore, it is better to replace the original image of the output PAGE-XML by the dewarped image before proceeding with the workflow. (If the dewarped image has also been cropped or deskewed, then of course all existing coordinates are re-calculated accordingly as well.)

Another use case is exporting PAGE-XML for tools that cannot apply cropping or deskewing, like LAREX or Transkribus.

Conversely, you might want to align two PAGE-XML files for the same page that have different original image references, projecting all segments below the page level from the one to the other (transforming all coordinates according to the page-level annotation, or keeping them unchanged).

Available processors

Step 20: Format Conversion

In this processing step the produced PAGE XML files can be converted to ALTO, PDF, hOCR or text files. Note that ALTO and hOCR can also be converted into different formats whereas the PDF version of PAGE XML OCR results is a widely accessible format that can be used as-is by expert and layman alike.

Available processors

-P from-to "alto2.0 alto3.0"
      # or "alto2.0 alto3.1"
      # or "alto2.0 hocr"
      # or "alto2.1 alto3.0"
      # or "alto2.1 alto3.1"
      # or "alto2.1 hocr"
      # or "alto page"
      # or "alto text"
      # or "gcv hocr"
      # or "hocr alto2.0"
      # or "hocr alto2.1"
      # or "hocr text"
      # or "page alto"
      # or "page hocr"
      # or "page text"
      

cat OCR* > full.txt

--ocr -T FULLTEXT -I OCR-D-IMG

{
  # font file name to use for rendering text
  "font": "AletheiaSans.ttf",
  # fix (invalid) negative coordinates
  "negative2zero": true,
  # concatenate to multi-page PDF (empty for none)
  "multipage": "name_of_pdf",
  # multi-page PDF page labels
  "pagelabel": "pageId",
  # render text on this hierarchy level
  "textequiv_level": "word",
  # draw polygon outlines in the PDF (empty for none)
  "outlines": "line"
}

-P colordict '{
  "#969696": "TableRegion", 
  "#00FF00": "TextRegion:page-number", 
  "#FFFF00": "TextRegion:heading", 
  "#00FFFF": "GraphicRegion:logo", 
  "#0000FF": "TextRegion:subject", 
  "#FF0000": "TextRegion:catch-word", 
  "#FF00FF": "TextRegion:footnote", 
  "#646464": "TextRegion:paragraph" }'

Step 20.1: Generic transformations

Sometimes PAGE-XML annotations need to be processed specially to make a workflow’s processors interoperate properly. For example, a text producing processor might forget to make TextEquiv consistent between hierarchy levels, or it might be necessary to remove specific region types. Also, repairing minor syntactic or semantic deficiencies is usually required for export or visualization, like removing empty ReadingOrder and dead @regionRefs, ensuring each TextEquiv has a Unicode, or fixing negative or floating-point coordinates. While it is always possible to do that ad-hoc via scripts, it might help formulate this as a proper workflow step via processor CLI.

Available processors

Step 21: Archiving

After you have successfully processed your images, the results should be saved and archived. OLA-HD is a longterm archive system which works as a mixture between an archive system and a repository. For further details on OLA-HD see the extensive concept paper. You can also check out the prototype to make sure, OLA-HD meets your needs and requirements. To use the prototype, specify http://141.5.98.232/api as the endpoint parameter in your call.

Available processors

Step 22: Dummy Processing

Sometimes it can be useful to have a dummy processor, which takes the files in an Input fileGrp and copies them the a new Output fileGrp, re-generating the PAGE XML from the current namespace schema/model.

Available processors

Recommendations

In order to facilitate the usage of OCR-D and the configuration of workflows, we provide two workflows which can be used as a start for your OCR-D-tests. They were determined by testing the processors listed above on selected pages of some prints from the 17th and 18th century.

The results vary quite a lot from page to page. In most cases, segmentation is a problem.

Note that for our test pages, not all steps described above werde needed to obtain the best results. Depending on your particular images, you might want to include those processors again for better results.

We are currently working on regression tests with the help of which we will be able to provide more profound workflows soon, which will replace those interim solutions.

Minimal workflow

Since ocrd-tesserocr-recognize can do binarization (Otsu), region segmentation, table recognition, line segmentation and text recognition at once, just like the upstream tesseract command line tool, it’s a good single-step workflow to get a baseline result to compare to granular workflows.

Note: Be aware that you will most likely obtain significantly better results by configuring a more granular workflow like e.g. the workflows below.

Example with ocrd-process

ocrd process "tesserocr-recognize -P segmentation_level region -P textequiv_level word -P find_tables true -P model GT4HistOCR_50000000.997_191951"

Best results for selected pages

The following workflow has produced best results for ‘simple’ pages (e.g. this page) (CER ~1%).

Example with ocrd-process

ocrd process \
  "cis-ocropy-binarize -I OCR-D-IMG -O OCR-D-BIN" \
  "anybaseocr-crop -I OCR-D-BIN -O OCR-D-CROP" \
  "skimage-binarize -I OCR-D-CROP -O OCR-D-BIN2 -P method li" \
  "skimage-denoise -I OCR-D-BIN2 -O OCR-D-BIN-DENOISE -P level-of-operation page" \
  "tesserocr-deskew -I OCR-D-BIN-DENOISE -O OCR-D-BIN-DENOISE-DESKEW -P operation_level page" \
  "cis-ocropy-segment -I OCR-D-BIN-DENOISE-DESKEW -O OCR-D-SEG -P level-of-operation page" \
  "cis-ocropy-dewarp -I OCR-D-SEG -O OCR-D-SEG-LINE-RESEG-DEWARP" \
  "calamari-recognize -I OCR-D-SEG-LINE-RESEG-DEWARP -O OCR-D-OCR -P checkpoint_dir qurator-gt4histocr-1.0"

Note: (1) This workflow expects your images to be stored in a folder called OCR-D-IMG. If your images are saved in a different folder, you need to adjust -I OCR-D-IMG in the second line of the call above with the name of your folder, e.g. -I MAX (2) For the last processor in this workflow, ocrd-calamari-recognize, you need to specify the model which is to be used. If you didn’t download it via the OCR-D resource manager, you have to use the checkpoint parameter and pass your local path to the model on your hard drive as parameter value! In this case, the last line of the ocrd-process call above could e.g. look like this:

  "calamari-recognize -I OCR-D-SEG-LINE-RESEG-DEWARP -O OCR-D-OCR -P checkpoint /test/data/calamari_models/\*.ckpt.json"

All the other lines can just be copied and pasted.

Good results for slower processors

If your computer is not that powerful you may try this workflow. It works fine for simple pages and produces also good results in shorter time.

Example with ocrd-process

ocrd process \
  "cis-ocropy-binarize -I OCR-D-IMG -O OCR-D-BIN" \
  "anybaseocr-crop -I OCR-D-BIN -O OCR-D-CROP" \
  "skimage-denoise -I OCR-D-CROP -O OCR-D-BIN-DENOISE -P level-of-operation page" \
  "tesserocr-deskew -I OCR-D-BIN-DENOISE -O OCR-D-BIN-DENOISE-DESKEW -P operation_level page" \
  "tesserocr-segment -I OCR-D-BIN-DENOISE-DESKEW -O OCR-D-SEG -P shrink_polygons true" \
  "cis-ocropy-dewarp -I OCR-D-SEG -O OCR-D-SEG-DEWARP" \
  "tesserocr-recognize -I OCR-D-SEG-DEWARP -O OCR-D-OCR -P textequiv_level glyph -P overwrite_segments true -P model GT4HistOCR_50000000.997_191951"

Note: (1) This workflow expects your images to be stored in a folder called OCR-D-IMG. If your images are saved in a different folder, you need to adjust -I OCR-D-IMG in the second line of the call above with the name of your folder, e.g. -I my_images (2) For the last processor in this workflow, ocrd-tesserocr-recognize, the environment variable TESSDATA_PREFIX has to be set to point to the directory where the used models are stored if they are not in the default location. If you downloaded your models with the OCR-D resource manager, this is already taken care of.