<>

OCRD-ZIP

This document describes an exchange format to bundle a workspace described by a METS file following OCR-D’s conventions.

Rationale

METS is the exchange format of choice by OCR-D for describing relations of files such as images and metadata about those images such as PAGE or ALTO files. METS is a textual format, not suitable for embedding arbitrary, potentially binary, data. For various use cases (such as transfer via network, long-term preservation, reproducible tests etc.) it is desirable to have a self-contained representation of a workspace.

With such a representation, data producers are not forced to provide dereferenceable HTTP-URL for the files they produce and data consumers are not forced to dereference all HTTP-URL.

While METS does have mechanisms for embedding XML data and even base64-encoded binary data, the tradeoffs in file size, parsing speed and readability are too great to make this a viable solution for a mass digitization scenario.

Instead, we propose an exchange format (“OCRD-ZIP”) based on the BagIt spec used for data ingestion adopted in the web archiving community.

BagIt profile

As a baseline, an OCRD-ZIP must adhere to v0.97+ of the BagIt specs, i.e.

In accordance with the BagIt standard, bagit.txt MUST consist of exactly these two lines:

BagIt-Version: 1.0
Tag-File-Character-Encoding: UTF-8

In addition, OCRD-ZIP adhere to a BagIt profile (see Appendix A for the full definition):

BagIt-Profile-Identifier

The BagIt-Profile-Identifier must be the string https://ocr-d.de/en/spec/bagit-profile.json.

Ocrd-Mets

Ocrd-Mets can be provided to declare that the METS file will not be the standard mets.xml but another path relative to /data/.

Implementations MUST check for the Ocrd-Mets tag: If it has a value, look for the METS file at that location, relative to /data. Otherwise, assume the default mets.xml.

Ocrd-Identifier

A globally unique identifier identifying the work/works/parts of works this bundle of file represents.

This is to be used for repositories to identify new ingestions of existing works.

To ensure global uniqueness, the identifier should be prefixed with an identifier of the organization, e.g. an ISIL or domain name.

Ocrd-Base-Version-Checksum

The SHA512 checksum of the manifest-sha512.txt file of the version this bag was based on, if any.

Invariants

ZIP

An OCRD-ZIP MUST be a serialized as a ZIP file.

manifest-sha512.txt

Checksums for the files in /data must be calculated with the SHA512 algorithm only and provided as manifest-sha512.txt.

Since the checksum of this manifest file can be relevant (see Ocrd-Base-Version-Checksum), in addition to the requirements of the BagIt spec, the entries MUST be sorted.

NOTE: These checksums can be generated with find data -type f | sort -sf |xargs sha512sum > manifest-sha512.txt.

File names must be relative to METS

Within an OCRD-ZIP, all local file resources referenced in the METS (and consequently all those referenced in other files within the workspace – see rule “If in PAGE then in METS” must be relative to the location of the METS file.

Example

/tmp/foo/ws1/data
├── mets.xml
├── foo.tif
└── foo.xml

Valid mets:FLocat/@xlink:href in /tmp/foo/ws1/data/mets.xml:

Invalid mets:FLocat/@xlink:href in /tmp/foo/ws1/data/mets.xml:

When in data then in METS

All files except mets.xml itself that are contained in data directory must be referenced in a mets:file/mets:Flocat in the mets.xml.

When in METS and not in data

All local files (mets:file/mets:FLocat/@xlink:href that represent file paths) must be part of the OCRD-ZIP.

Optional metadata about the payload

In addition to the actual data files in /data, the following metadata files are allowed to be present in the root of the bag:

These files are purely for documentation and should not be used by processors in any way.

Algorithms

Packing a workspace as OCRD-ZIP

To pack a workspace to OCRD-ZIP:

Unpacking OCRD-ZIP to a workspace

Appendix A - BagIt profile definition

BagIt-Profile-Info:
  BagIt-Profile-Identifier: https://ocr-d.de/en/spec/bagit-profile.json
  BagIt-Profile-Version: '1.2.0'
  Source-Organization: OCR-D
  External-Description: BagIt profile for OCR data
  Contact-Name: Konstantin Baierer
  Contact-Email: konstantin.baierer@sbb.spk-berlin.de
  Version: 0.1
Bag-Info:
  Bagging-Date:
    required: false
  Source-Organization:
    required: false
  Ocrd-Mets:
    required: false
    default: 'mets.xml'
  Ocrd-Identifier:
    required: true
  Ocrd-Checksum:
    required: false
    # echo -n | sha512sum
    default: 'cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e'
Manifests-Required: ['sha512']
Tag-Manifests-Required: []
Tag-Files-Required: []
Tag-Files-Allowed:
  - README.md
  - Makefile
  - build.sh
  - sources.csv
  - metadata/*.xml
  - metadata/*.txt
Allow-Fetch.txt: false
Serialization: required
Accept-Serialization: application/zip
Accept-BagIt-Version:
  - '1.0'

Appendix B - IANA considerations

Proposed media type of OCRD-ZIP: application/vnd.ocrd+zip

Proposed extension: .ocrd.zip