bitsgalore.org - Johan van der Knijff https://bitsgalore.org https://bitsgalore.org 2026-02-17T15:29:06+01:00 2026-02-17T15:29:06+01:00 1800 <figure class="image"> <img src="/images/2026/02/y2k-furnishings.jpg" alt="Photograph of store named 'Y2K Furnishings'. The storefront window shows signs that read 'SALE', 'GOING OUT OF BUSINESS' and 'FOR LEASE'." /> <figcaption><a href="https://www.flickr.com/photos/walkingsf/3670113255" title="1339 Mission, 2000">1339 Mission, 2000</a> by <a href="https://www.flickr.com/photos/walkingsf/">Erica Fischer</a>, <a href="https://creativecommons.org/licenses/by/2.0/deed.en" rel="license noopener noreferrer">CC BY 2.0</a>.</figcaption> </figure> <p>A while ago, colleagues from our imaging studio approached me about an old <a href="https://lucidea.com/inmagic-dbtextworks/">Inmagic DB/TextWorks</a> database. The database was originally created around the year 2000, and contained information about photographed items from our collection. From an old machine of a retired former colleague, they had recovered both the database, and installer files of two versions of the DB/TextWorks software. However, they were unable to either read the database with modern software, or to get the original software running on modern hardware.</p> <p>The obvious solution would be to run the software in an emulated environment, and use the emulation to export the database to a more readable format. Oracle VirtualBox used to be my go-to software for such emulation jobs. As I’m trying to move away from (US-based) big tech software, this time I decided to give <a href="https://www.qemu.org/">QEMU</a> a try instead. Although I’ve used QEMU before, it’s been quite a while, and I noticed that a lot of online QEMU resources are quite outdated, which can make things confusing if you don’t know where to look.</p> <p>The main goal of this post is to document the main steps that make up a “migration through emulation” use case like this:</p> <ol> <li>The installation of QEMU.</li> <li>The creation of a virtual machine (VM) in QEMU, and the installation of an operating system (in this case Windows XP).</li> <li>The use of a “virtual thumb drive” to exchange data between the VM and the host machine.</li> </ol> <p>Even though this specifically covers the Inmagic DB/TextWorks case, most of this will be applicable to other “migration though emulation” use cases as well.</p> <p>In the final section I also give an overview of the general file structure of a DB/TextWorks 7.0 database, and I provide two openly-licensed example databases.</p> <!-- more --> <h2 id="software">Software</h2> <p>The data recovered by my colleagues included two versions of the DB/TextWorks software: version 3.0 version from 1998, version 7.0.1 version from 2004. Since the time stamps of the database files indicated they were created in 2000, I went for the 7.0.1 version. Its documentation lists the following requirements:</p> <ul> <li>Windows 2000 + Service Pack 3 or Windows XP.</li> <li>Internet Explorer 5.01 or later, which is needed “for certain features to work properly”.</li> </ul> <p>I happened to have an ISO image of an old installation CD of Windows XP (Home Edition) lying around, which made XP the obvious emulation target. This also comes with a compatible Internet Explorer version preinstalled.</p> <h2 id="host-system">Host system</h2> <p>As a host system I used a desktop PC running Linux Mint 22.3 (MATE 64-bit).</p> <h2 id="install-qemu">Install QEMU</h2> <p>Installation of QEMU is as simple as<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt install qemu-system </code></pre></div></div> <h2 id="create-virtual-machine">Create virtual machine</h2> <p>In an empty directory, create a sufficiently large (here: 10 GB) image file<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">2</a></sup> (here called “winxp.img”):</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>qemu-img create -f qcow2 winxp.img 10G </code></pre></div></div> <p>Then copy the ISO image of the Windows XP installation CD (“winxp.iso”) to the same directory. Now start a virtual machine (VM) using:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>qemu-system-x86_64 -m 1024 -hda winxp.img -cdrom winxp.iso -boot d </code></pre></div></div> <p>This boots up a VM that runs the Windows XP setup program<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>. To test the installation, close down Windows, and start the VM using:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>qemu-system-x86_64 -m 1024 -hda winxp.img </code></pre></div></div> <p>If all goes well, this should boot the VM running Windows XP.</p> <h2 id="data-exchange-between-vm-and-host-machine">Data exchange between VM and host machine</h2> <p>In its current state, there’s no way to exchange data between the VM and host machine. For our use case, we need this to:</p> <ol> <li>Make the Inmagic DB/TextWorks installation files and the database available to the VM.</li> <li>Make the exported database from the VM available to the host machine.</li> </ol> <p>There are a couple of ways to achieve this. In this case I opted for a “virtual thumb drive” approach, which works similarly to how you would use a physical thumb drive to exchange data between two machines.</p> <h2 id="create-virtual-thumb-drive">Create virtual thumb drive</h2> <p>Start by creating a 200 MB disk image (“tw.img”):</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dd bs=512 count=390625 if=/dev/zero of=tw.img </code></pre></div></div> <p>Then format this disk image as a FAT 32 filesystem<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mformat -F -i tw.img </code></pre></div></div> <h2 id="copy-data-from-host-to-virtual-thumb-drive">Copy data from host to virtual thumb drive</h2> <p>I used the <a href="https://linux.die.net/man/1/mcopy">mcopy</a> tool to copy the required data from my host machine to the virtual thumb drive. In my case the data are in a directory tree under <code class="language-plaintext highlighter-rouge">/home/johan/kb/beeldstudio-textworks/Dbtmsg/</code>. The following command recursively copies everything in that directory to the virtual thumb drive:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mcopy -i tw.img -s /home/johan/kb/beeldstudio-textworks/Dbtmsg/ :: </code></pre></div></div> <p>You can verify the result by mounting the image file on your host machine. In Linux Mint, right-click the file in the file manager, and then select “Open with Disk Image Mounter”.</p> <h2 id="start-vm-with-virtual-thumb-drive">Start VM with virtual thumb drive</h2> <p>The following command starts up the Windows XP VM, and mounts the virtual thumb drive as a removable USB device:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>qemu-system-x86_64 -m 1024 -hda winxp.img \ -drive if=none,id=usbstick,format=raw,file=tw.img \ -usb \ -device usb-ehci,id=ehci \ -device usb-storage,bus=ehci.0,drive=usbstick </code></pre></div></div> <p>In my case the virtual thumb drive is available as the “E:” drive in Windows XP. Note that it can take some time before Windows detects the device, so be patient here.</p> <h2 id="inmagic-dbtextworks-installation">Inmagic DB/TextWorks installation</h2> <p>To install DB/TextWorks, open Windows Explorer, navigate to the folder with the installer files on the virtual thumb drive (in my case the “E:” drive on the WinXP VM), and run setup.exe.</p> <h2 id="open-a-database">Open a database</h2> <p>A DB/TextWorks database is made up of various separate files that share a common base name. The primary entry point is a file with a “.TBA” file extension. To open a database, select “Open” from the “File” menu, and navigate to the folder on the virtual thumb drive that contains the database:</p> <figure class="image"> <img src="/images/2026/02/textworks-open.png" /> </figure> <p>After loading, DB/TextWorks shows a window with all fields in the database:</p> <figure class="image"> <img src="/images/2026/02/twDB.png" /> </figure> <p>It’s a good idea to make a screenshot of this, as the CSV exports don’t contain the full field names!</p> <h2 id="export-database-to-comma-delimited-text">Export database to comma-delimited text</h2> <p>From the “File” menu, select “Export …”, navigate to a location on your virtual thumb drive where you want to store your export, and enter an output file name. In the Export Options dialog that appears, select “Delimited ASCII Format” as the output format<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>, and make sure to check the “Store Field Names in First Row” checkbox.</p> <figure class="image"> <img src="/images/2026/02/twExportOptions.png" /> </figure> <p>Click “OK” to start the export. When done, close DB/TextWorks, and shut down the virtual machine.</p> <h2 id="access-the-exported-database-from-the-host-machine">Access the exported database from the host machine</h2> <p>To access the exported database from the host machine, simply mount the virtual thumb drive image file. In Linux Mint, you can do this by right-clicking the file in the file manager, and then selecting “Open with Disk Image Mounter”. Now you can access the file system, and copy the exported file to the host machine’s file system.</p> <h2 id="some-observations-on-the-exported-csv-files">Some observations on the exported CSV files</h2> <p>The column headings in the exported CSV files only shows the first two characters of each field’s name. For instance, in my case the database contains the fields “Signatuur” and “Magazijn”, which show up as “SI” and “MA” in the exported file. I wasn’t able to change this behavior. Because of this, it’s a good idea to make a note (or a screenshot) of the full names from the DB/TextWorks interface.</p> <p>Another thing that caught my attention is that DB/TextWorks variables may contain multiple entries. In that case, each comma-separated item in the exported file is sub-divided in fields that are separated by pipe (“|”) characters.</p> <h2 id="database-structure">Database structure</h2> <p>Each DB/Textbase database is made up of multiple files. These files share a common base name, but each has a different file extension. The User Manual gives the following overview<sup id="fnref:6"><a href="#fn:6" class="footnote" rel="footnote" role="doc-noteref">6</a></sup>:</p> <blockquote> <table> <thead> <tr> <th style="text-align: left">Extension</th> <th style="text-align: left">Description</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">.TBA</td> <td style="text-align: left">Primary textbase definition file, which also contains textbase elements (for example, forms, query screens, sets, record skeletons) stored in the textbase.</td> </tr> <tr> <td style="text-align: left">.ACF</td> <td style="text-align: left">Access control file; controls simultaneous access to the textbase by multiple users or software instances, or applications (for example, DB/Text PowerPack Lite).</td> </tr> <tr> <td style="text-align: left">.DBS</td> <td style="text-align: left">Textbase structure file; contains field definitions and other information about the structure of the textbase.</td> </tr> <tr> <td style="text-align: left">.IXL</td> <td style="text-align: left">Indexed list file; contains the validation and substitution lists, and the leading article and stop word lists.</td> </tr> <tr> <td style="text-align: left">.DBR</td> <td style="text-align: left">Contains the records (including deferred new, deleted, or changed records).</td> </tr> <tr> <td style="text-align: left">.DBO</td> <td style="text-align: left">Contains a directory to the records in the .DBR file.</td> </tr> <tr> <td style="text-align: left">.SDO</td> <td style="text-align: left">Contains a directory to records with deferred updates in the .DBR file.</td> </tr> <tr> <td style="text-align: left">.BTX</td> <td style="text-align: left">Contains the Term and Word indexes.</td> </tr> <tr> <td style="text-align: left">.OCC</td> <td style="text-align: left">Contains the lists of records indexed by the terms and words in the .BTX file.</td> </tr> <tr> <td style="text-align: left">.LOG</td> <td style="text-align: left">Optional textbase log file; lists changes to the textbase structure and records.</td> </tr> <tr> <td style="text-align: left">.TML</td> <td style="text-align: left">Thesaurus maintenance locking file, prevents more than one person at a time from modifying records in that thesaurus textbase. Note that .TML files do not have to be backed up. The software automatically creates them if they do not exist.</td> </tr> <tr> <td style="text-align: left">.HLP</td> <td style="text-align: left">Optional textbase-specific help file.</td> </tr> <tr> <td style="text-align: left">.INI</td> <td style="text-align: left">Optional file used with Copy Special applications, Textbase-Specific Help, the DB/Text ODBC Driver, and the Applications menu.</td> </tr> <tr> <td style="text-align: left">.SLT</td> <td style="text-align: left">Optional file that is created when EnableSlotlog=1 appears in the [Advanced] section of the DBTEXT.INI or textbase .INI file. This option can be set in DBTEXT.INI during Setup using the Track Textbase Access button on the Configuration dialog box. The machine name and login name of each user who has a textbase open is recorded in the .SLT file. The line is cleared when each closes the textbase.</td> </tr> </tbody> </table> </blockquote> <h2 id="example-databases">Example databases</h2> <p>Since DB/TextWorks running already, I though it would be useful to use it to make some sample files. I created two simple databases: one with DB/TextWorks 7.0.1, and another one with the older DB/TextWorks 3.0. I added both <a href="https://github.com/openpreserve/format-corpus/tree/master/DBTextWorks">to the OPF Format Corpus</a>.</p> <h2 id="further-resources">Further resources</h2> <ul> <li><a href="https://github.com/openpreserve/format-corpus/tree/master/DBTextWorks">DB/TextWorks sample files in the OPF Format Corpus</a>.</li> <li><a href="https://www.andornot.com/software/dbtextworks/dbtextworks-versions-and-features/">DB/TextWorks Versions and Features</a>.</li> <li><a href="http://support.inmagic.com/Web/DBTWandWPP900/DBTextWorksv9UsersManual.pdf">DBTextWork User’s Manual</a> (this documents version 9, which is slightly more recent than version 7 covered in this post).</li> <li><a href="http://support.inmagic.com/Web/DBTWandWPP900/Readme.HTM">Inmagic® DB/TextWorks® and DB/Text® WebPublisher PRO Version 9.00 README File</a> (also covers version 9).</li> <li><a href="https://en.wikibooks.org/wiki/QEMU/Windows_XP">QEMU/Windows XP</a>.</li> <li><a href="https://www.wikihow.com/Activate-Windows-XP">How to Activate Windows XP in 2024</a> - method 3 (Disabling Activation) is a Windows registry hack that disables activation altogether, which is useful for VMs with no internet access.</li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:2"> <p>I largely followed the instructions I found <a href="https://en.wikibooks.org/wiki/QEMU/Windows_XP">here</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>The actual file size is much smalller, because qemu uses compression. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>Note that the Windows XP setup process requires you to enter a 25-character product key. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>This requires the <a href="https://www.gnu.org/software/mtools/">mtools</a> package. If I’m not mistaken this comes preinstalled with most Linux distributions, but if needed you can install it using <code class="language-plaintext highlighter-rouge">sudo apt install mtools</code>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:1"> <p>TextWorks 7 also supports XML as an export format. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:6"> <p>Taken from the User Manual of version 7 of the software, which is not available online. A copy of the more recent version 9 User Manual is <a href="http://support.inmagic.com/Web/DBTWandWPP900/DBTextWorksv9UsersManual.pdf">available here</a>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2026/02/16/emulating-inmagic-dbtextworks-databases-with-qemu https://bitsgalore.org/2026/02/16/emulating-inmagic-dbtextworks-databases-with-qemu 2026-02-16T00:00:00+01:00 <figure class="image"> <img src="/images/2025/06/1024px-Tools_66.jpeg" alt="Photo that shows assortment of tools." /> <figcaption>Tools by <a href="https://commons.wikimedia.org/wiki/File:Tools_66.jpg">Wilfredor</a>, CC0, via Wikimedia Commons.</figcaption> </figure> <p>My recent work on the <a href="/2025/05/19/emulating-microsoft-multiplan-spreadsheets-in-dosbox-x">Microsoft Multiplan</a> and <a href="/2025/05/28/quattro-pro-for-dos-revisited-an-obsolete-format-no-more">Quattro Pro for DOS</a> formats made me think how far LibreOffice has come over the past years in its support of legacy file formats. These formats are supported through software libraries that are developed within the <a href="https://www.documentliberation.org/">Document Liberation Project</a> (DLP). This project was set up in 2014 by <a href="https://www.documentfoundation.org/">The Document Foundation</a>, which is also the home of LibreOffice. Aside from their use in LibreOffice, these libraries are also the foundation of a set of stand-alone command-line tools that allow you to convert a wide range of legacy file formats to the <a href="https://en.wikipedia.org/wiki/OpenDocument">OpenDocument</a> formats and EPUB.</p> <p>Much of the available information about these libraries and tools is scattered across different platforms. The command-line tools are also surprisingly hard to find, even though they have been around for a long time. This short post is an attempt at bringing the most important information I could find about them together.</p> <!-- more --> <h2 id="libraries">Libraries</h2> <p><a href="https://www.documentliberation.org/projects/">The projects page</a> on the Document Liberation Project website lists all libraries and tools that are developed as part of the project. The “Import libraries” section is particularly interesting. Many of these libraries support multiple file formats, but this is not always obvious from their names. More detailed information, including the file formats they support, is available <a href="https://wiki.documentfoundation.org/DLP/Libraries">on their Wiki page</a>. As an example, for the <a href="https://sourceforge.net/projects/libwps">libwps</a> library, the Wiki page currently shows 10 different formats, including Quattro Pro and Lotus 123 spreadsheets<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>.</p> <h2 id="command-line-tools">Command-line tools</h2> <p>The DLP import libraries are also used in a set of command-line conversion tools. These tools are quite difficult to find. For a start, their source code is hidden away in the <a href="https://sourceforge.net/projects/libwpd/">libpwd</a> library, as a sub-project called <a href="https://sourceforge.net/p/libwpd/writerperfect/">writerperfect</a>. I haven’t (yet) tried to compile the tools myself, but binaries are available for various Linux-based platforms. For Debian-based systems, each binary that is built from the writerperfect source is provided as a separate Debian package, which results in 27 distinct tools/packages<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>. For convenience this table list all of these tools:</p> <table> <thead> <tr> <th style="text-align: left">Tool</th> <th style="text-align: left">Description</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">abw2epub</td> <td style="text-align: left">AbiWord to EPUB format converter</td> </tr> <tr> <td style="text-align: left">abw2odt</td> <td style="text-align: left">AbiWord to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">cdr2odg</td> <td style="text-align: left">Corel Draw graphics to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">ebook2epub</td> <td style="text-align: left">other E-Book formats to EPUB converter</td> </tr> <tr> <td style="text-align: left">ebook2odt</td> <td style="text-align: left">E-Book formats to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">fh2odg</td> <td style="text-align: left">Freehand to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">key2odp</td> <td style="text-align: left">Keynote to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">mwaw2epub</td> <td style="text-align: left">old Mac formats to EPUB converter</td> </tr> <tr> <td style="text-align: left">mwaw2odf</td> <td style="text-align: left">old Mac formats to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">numbers2ods</td> <td style="text-align: left">Apple Numbers spreadsheet documents to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">pages2epub</td> <td style="text-align: left">Apple Pages to EPUB converter</td> </tr> <tr> <td style="text-align: left">pages2odt</td> <td style="text-align: left">Apple Pages text documents to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">pmd2odg</td> <td style="text-align: left">Apple Pagemaker to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">pub2odg</td> <td style="text-align: left">Publisher documents to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">qxp2epub</td> <td style="text-align: left">QuarkXPress to EPUB converter</td> </tr> <tr> <td style="text-align: left">qxp2odg</td> <td style="text-align: left">QuarkXPress to OpenDocument graphics converter</td> </tr> <tr> <td style="text-align: left">sd2epub</td> <td style="text-align: left">StarOffice to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">sd2odf</td> <td style="text-align: left">StarOffice to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">vsd2odg</td> <td style="text-align: left">Visio to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">wks2ods</td> <td style="text-align: left">Works spreadsheet documents to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">wpd2epub</td> <td style="text-align: left">WordPerfect document to EPUB converter</td> </tr> <tr> <td style="text-align: left">wpd2odt</td> <td style="text-align: left">WordPerfect to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">wpg2odg</td> <td style="text-align: left">WordPerfect Graphics to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">wps2epub</td> <td style="text-align: left">Works text document to EPUB converter</td> </tr> <tr> <td style="text-align: left">wps2odt</td> <td style="text-align: left">Works text documents to OpenDocument converter</td> </tr> <tr> <td style="text-align: left">zmf2epub</td> <td style="text-align: left">Zoner Draw to EPUB converter</td> </tr> <tr> <td style="text-align: left">zmf2odg</td> <td style="text-align: left">Zoner Draw to OpenDocument converter</td> </tr> </tbody> </table> <h2 id="installation-debian">Installation (Debian)</h2> <p>The binary package names are identical to the tool names, so in order to install the “wks2ods” tool from its Debian package, use this:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt install wks2ods </code></pre></div></div> <p>All other tools follow the same pattern. As with the underlying import libraries, the tool names don’t always immediately give away what formats they support. The documentation is also quite minimal. As an example, despite its name, the “wks2ods” tool supports a range of legacy spreadsheet formats that goes far beyond “Works spreadsheet documents”. The <a href="https://wiki.documentfoundation.org/DLP/Libraries">information on the import libraries Wiki page</a> and some experimentation should get you a long way though.</p> <h2 id="using-the-command-line-tools">Using the command-line tools</h2> <p>The basic usage of these tools is very simple. Taking the “wks2ods” tool as an example again:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wks2ods [OPTIONS] INPUT [OUTPUT] </code></pre></div></div> <p>Here, <code class="language-plaintext highlighter-rouge">INPUT</code> is the input file, and <code class="language-plaintext highlighter-rouge">OUTPUT</code> defines the output file. If <code class="language-plaintext highlighter-rouge">OUTPUT</code> is omitted, the tool will print flat ODF to standard output. The tool has the following options:</p> <ul> <li><code class="language-plaintext highlighter-rouge">--encoding ENCODING</code>: this sets the INPUT encoding.</li> <li><code class="language-plaintext highlighter-rouge">--list-encodings</code>: this shows the available encodings and then exits.</li> <li><code class="language-plaintext highlighter-rouge">--password PASSWORD</code>: this sets a password to open the input file.</li> <li><code class="language-plaintext highlighter-rouge">--stdout</code>: this prints the result as flat XML to standard output.</li> </ul> <p>All tools share the same general command-line interface (note that I haven’t tried all of them, so there may be minor differences that I’m not aware of!).</p> <h2 id="examples">Examples</h2> <p>In the simplest case we can just run “wks2ods” with the name of the input and output files as command-line arguments. For example, the command below converts a Quattro Pro for DOS spreadsheet to OpenDocument Spreadsheet format:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wks2ods WHATEVER.WQ2 WHATEVER.ods </code></pre></div></div> <p>You may need to use the <code class="language-plaintext highlighter-rouge">--encoding</code> option to define the encoding of the input file. It’s a good idea to always inspect the output file for any obvious encoding problems.</p> <p>The following command converts a WordPerfect file to OpenDocument Text format with the “wpd2odt” tool:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wpd2odt WHATEVER.WPD WHATEVER.odt </code></pre></div></div> <p>And this converts a WordPerfect file to EPUB using the “wpd2epub” tool:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wpd2epub WHATEVER.WPD WHATEVER.epub </code></pre></div></div> <h2 id="final-thoughts">Final thoughts</h2> <p>Since the Writerperfect tools are based on the same software libraries that are also used by LibreOffice, the conversion results will be identical to a LibreOffice conversion in most cases. The main advantage of these tools is that they can be easily integrated in automated workflows, independent of the LibreOffice applications.</p> <h2 id="further-resources">Further resources</h2> <ul> <li><a href="https://www.documentliberation.org/">Document Liberation Project</a></li> <li><a href="https://wiki.documentfoundation.org/DLP/Libraries">DLP Libraries Wiki page</a></li> <li><a href="https://sourceforge.net/p/libwpd/writerperfect/">writerperfect source code</a></li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>The information on the Wiki doesn’t appear to be entirely up to date, as this library also supports the Microsoft Multiplan formats, which aren’t listed here. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>See for example <a href="https://packages.ubuntu.com/source/oracular/writerperfect">here</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2025/06/10/writerperfect-conversion-tools-for-legacy-file-formats https://bitsgalore.org/2025/06/10/writerperfect-conversion-tools-for-legacy-file-formats 2025-06-10T00:00:00+02:00 <figure class="image"> <img src="/images/2025/05/quattro-floppy.jpg" alt="Photo of 3.5 inch installation floppy of Quattro for DOS, version 5.0." /> <figcaption>Image sourced from <a href="https://archive.org/details/BorlandQuattroPro5.0ForDOSGerman">Internet Archive</a>, license unknown.</figcaption> </figure> <p>in 2014 I wrote <a href="/2014/10/29/quattro-pro-dos-obsolete-format-last">a post</a> on the <a href="http://fileformats.archiveteam.org/wiki/Quattro_Pro">Quattro Pro</a> for DOS spreadsheet formats. This documents my attempts at reading a few old Quattro Pro for DOS spreadsheets from my personal archives with modern (at the time) software. Back then, neither Microsoft Excel nor LibreOffice Calc supported these formats. Only the then-current Quattro Pro X7 was able to read the files, but there were several issues related to formatting, rendering of charts, and the handling of external references. Based on these tests, I argued that Quattro Pro for DOS could be a case of that rarest species in the world of digital preservation: a file format that had truly become obsolete!</p> <p>While working on my <a href="/2025/05/19/emulating-microsoft-multiplan-spreadsheets-in-dosbox-x">previous post about Microsoft Multiplan spreadsheets</a>, I was surprised to see that <a href="https://www.libreoffice.org/">LibreOffice</a> actually supported this ancient format. A quick glance at <a href="https://wiki.documentfoundation.org/Feature_Comparison:_LibreOffice_-_Microsoft_Office">LibreOffice’s feature matrix</a> showed that LibreOffice has added support for many legacy formats over the past years, including Quattro Pro for DOS<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. Under the hood LibreOffice uses the <a href="https://sourceforge.net/p/libwps/wiki/Home/">Microsoft Works format import library (libwps)</a> for this. The same library is also used to read the <a href="https://en.wikipedia.org/wiki/Multiplan">Microsoft Multiplan</a> for DOS and <a href="https://en.wikipedia.org/wiki/Lotus_1-2-3">Lotus 123</a> file formats. This made me wonder if the conclusions of my 2014 post would still hold up. This follow-up post puts this to the test.</p> <!-- more --> <h2 id="quattro-pro">Quattro Pro</h2> <p><a href="https://en.wikipedia.org/wiki/Quattro_Pro">Quattro Pro</a> is a spreadsheet program that was first released in 1988. It’s still around today as part of the <a href="https://www.wordperfect.com/en/product/office-suite/">WordPerfect Office suite</a>. <a href="http://fileformats.archiveteam.org/wiki/Quattro_Pro">A number of file formats</a> are associated with the software. As with my 2014 post, the scope of this follow-up is restricted to the old Quattro Pro for DOS formats:</p> <ul> <li><a href="http://fileformats.archiveteam.org/wiki/WQ1">Quattro Pro for DOS, versions 1-4 (WQ1)</a></li> <li><a href="http://fileformats.archiveteam.org/wiki/WQ2">Quattro Pro for DOS, versions 5.0 and 5.5 (WQ2)</a></li> </ul> <h2 id="test-files">Test files</h2> <p>I used the same test files as in my 2014 post (which are all part of the <a href="https://github.com/openpreserve/format-corpus/">OPF Format Corpus</a>):</p> <ul> <li>One <a href="https://github.com/openpreserve/format-corpus/tree/master/office/spreadsheet/wq1">WQ1 file</a></li> <li>Three <a href="https://github.com/openpreserve/format-corpus/tree/master/office/spreadsheet/wq2">WQ2 files</a></li> </ul> <p>These are all files from my personal archives, and I originally created them in 1996.</p> <h2 id="emulation-of-original-software">Emulation of original software</h2> <p>To evaluate LibreOffice Calc’s handling of my test files, I emulated the original software, and used the emulation as a reference. <a href="https://winworldpc.com/home">WinWorld</a> has <a href="https://winworldpc.com/product/quattro-pro/">installers for most old Quattro Pro versions</a>. From what I recall I created most of my files with Quattro Pro 5.0 for DOS, so I downloaded <a href="https://winworldpc.com/download/5274c3aa-0825-c389-11c3-a6e280947e52">the installers of that version</a>. I then installed the software in <a href="https://dosbox-x.com/">DOSBox-X</a>. My initial attempts at this failed, because the installer wouldn’t recognise the second installation disk. As a workaround, I simply copied the files from both installation disks to a temporary folder, and then ran the installer from there. This resulted in a successful installation.</p> <h2 id="simple-numbers-and-text">Simple numbers and text</h2> <p>I started out with <a href="https://github.com/openpreserve/format-corpus/blob/master/office/spreadsheet/wq1/KSBASE.WQ1">this WQ1 spreadsheet</a>, which only contains simple numerical and text data. The DOSBox-X emulation rendered this file without any apparent problems:</p> <figure class="image"> <img src="/images/2025/05/qp5-wq1.png" /> </figure> <p>You may notice that the first column appears as shaded. Apparently the file was originally saved with the “locked titles” option enabled, which can be used to lock the first row or column during scrolling. This can easily be disabled.</p> <p>Next I opened the same file in LibreOffice Calc<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>:</p> <figure class="image"> <img src="/images/2025/05/lo-wq1.png" /> </figure> <p>Calc reads the file without any problems. There are some minor differences compared to the emulation: Calc doesn’t “lock” the first column, the date format is slightly different, and Calc shows more decimal places for the numbers in the J, K and L columns. All of these are mostly cosmetic differences, and for practical purposes Calc’s rendering looks perfectly usable. The results were largely identical for <a href="https://github.com/openpreserve/format-corpus/blob/master/office/spreadsheet/wq2/KSBASE.WQ2">the WQ2 version of the same spreadsheet</a>.</p> <h2 id="simple-formulas-charts">Simple formulas, charts</h2> <p>Next up was <a href="https://github.com/openpreserve/format-corpus/blob/master/office/spreadsheet/wq2/KS4001.WQ2">KS4001.WQ2</a>, which contains some formulas and a chart. Here are two screenshots of the DOSBox-X emulation of this file:</p> <figure class="image"> <img src="/images/2025/05/qp-wq2-1.png" /> </figure> <figure class="image"> <img src="/images/2025/05/qp-wq2-2.png" /> </figure> <p>And here’s a screenshot that shows what LibreOffice Calc makes of this:</p> <figure class="image"> <img src="/images/2025/05/lo-wq2.png" /> </figure> <p>Compared to the emulated version, the most notable difference is the formatting of the chart, but overall the Calc rendering gives a good representation of the data in the original spreadsheet. Interestingly, in <a href="/2014/10/29/quattro-pro-dos-obsolete-format-last">my 2014 analysis</a> the then-current Quattro Pro X7 version failed to render the chart correctly:</p> <figure class="image"> <img src="/images/2014/10/ks4001_wq2.png" /> </figure> <p>It’s reassuring to see that LibreOffice Calc now does a much better job!</p> <h2 id="external-references">External references</h2> <p>Finally there’s <a href="https://github.com/openpreserve/format-corpus/blob/master/office/spreadsheet/wq2/KS4000.WQ2">this WQ2 spreadsheet</a>, which contains references to another spreadsheet (KSBASE.WQ2) that is located in the same directory. This file exposed some interesting issues, and the following sub-sections are an attempt to demonstrate these as best as I can. Be warned though that what follows is slightly convoluted, which is due to a combination of the nature of the spreadsheet, and the fact that both issues are interrelated.</p> <h3 id="quattro-pro-emulation">Quattro Pro emulation</h3> <p>First I opened this file in the emulator. On loading, Quattro Pro detects that it contains external references, and prompts the user what to do with them:</p> <figure class="image"> <img src="/images/2025/05/qp-wq2-ext-dialog.png" /> </figure> <p>Here I chose “Load Supporting”, after which the file loaded like this:</p> <figure class="image"> <img src="/images/2025/05/qp-wq2-ext.png" /> </figure> <p>The external references, which are in cells E4, G4, B5, E5 and H5, are all resolved correctly. To double-check this, I also tried to open the file after removing the referenced file. This resulted in “NA” values for all cells with references to this file, as well as for all cells that depend on it.</p> <p>I initially assumed that the shown “ERR” values were related to the external references, but on closer inspection this doesn’t seem to be the case at all. Instead, they simply result from the fact that the C column doesn’t contain any data.</p> <h3 id="initial-behaviour-in-libreoffice-calc">Initial behaviour in LibreOffice Calc</h3> <p>Opening the file in LibreOffice Calc also results in a notification message that “Automatic update of external links has been disabled”:</p> <figure class="image"> <img src="/images/2025/05/lo-wq2-ext-dialog.png" /> </figure> <p>Note how the values in cells E4 and G4 (which both contain external references that return a text string in the emulation) are shown as “#REF”. Interestingly though, cells B5, E5 and H5 also contain external references, but the (correct!) numerical values are shown nevertheless. This could mean either of the following:</p> <ol> <li>The numbers shown are “cached” values that were saved alongside the formulas at the time of the file’s creation.</li> <li>The numbers are imported from the linked spreadsheet (even though this shouldn’t really happen).</li> </ol> <p>To test which of these is true, I created a copy of the spreadsheet in an empty directory. When I opened it in Calc, it showed the exact same behaviour. This means the numbers really <em>are</em> cached values. This does raise the question why Calc only shows cached <em>numbers</em>, and not cached <em>text strings</em>. I will return to this in the final sections of this post.</p> <h3 id="restoring-the-external-references">Restoring the external references</h3> <p>First let’s go back to our original spreadsheet. After clicking on “Allow updating”, Calc showed a notification that the external file that is referenced could not be loaded:</p> <figure class="image"> <img src="/images/2025/05/lo-external-file.png" /> </figure> <p>The reason for this is, that all references to the external file are defined as a file name without a file extension (i.e. KSBASE instead of KSBASE.WQ2). A quick look at the <a href="https://help.libreoffice.org/latest/en-US/text/shared/01/02180000.html">LibreOffice documentation</a> shows how to fix this:</p> <ul> <li>In Calc’s “Edit” menu, go to “Links to External Files…”. This brings up a link editor dialog.</li> <li>Select the link you want to change, and click on “Modify …”</li> <li>In the file dialog that appears, select the file you want to link to (in this case KSBASE.WQ2).</li> <li>Click on the “Update” button, and then close the dialog.</li> </ul> <h3 id="behaviour-after-recalculate">Behaviour after recalculate</h3> <p>With the link restored, I opened the “Data” menu, and selected <a href="https://help.libreoffice.org/latest/en-US/text/scalc/01/recalculate_hard.html">“Calculate/Recalculate Hard”</a>. This forces all cells to be re-calculated. The results of this operation were not quite what I expected though:</p> <figure class="image"> <img src="/images/2025/05/lo-wq2-ext-updated.png" /> </figure> <p>Even though the “#REF” values are now replaced by actual values from the external spreadsheet, these values are different from those in the original Quattro Pro rendering. For example:</p> <ul> <li>The value of cell B5 is supposed to be a number (variable DEPTH, 30), but instead it is now a text string (“sal”).</li> <li>The value of cell G4 is supposed to be a text string (variable SOIL, “c”), but instead it is now a numerical value (3133.03).</li> </ul> <p>So what’s going on here?</p> <h3 id="vlookup-function-behaviour-in-quattro-pro-and-calc">VLOOKUP function behaviour in Quattro Pro and Calc</h3> <p>To understand why this happens, it is first important to know that the external references in rows 4 and 5 of the spreadsheet are all table lookup operations. As an example, the Quattro Pro formula in cell B5 (copied here from the emulated version) is:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>@VLOOKUP($B$4,[KSBASE]A:$A$2..$I$83,6) </code></pre></div></div> <p>This function has 3 arguments:</p> <ul> <li>A lookup value.</li> <li>A data block in the external spreadsheet, where the first column is used as an index column.</li> <li>The column number in the data block that contains the requested value.</li> </ul> <p>The equivalent function in LibreOffice Calc is <a href="https://help.libreoffice.org/25.2/en-US/text/scalc/01/04060109.html?&amp;DbPAR=SHARED&amp;System=UNIX#Section9">VLOOKUP</a>. Here’s a portion of the relevant data in the external spreadsheet (which is used as the data block):</p> <figure class="image"> <img src="/images/2025/05/ksbase-lookup.png" /> </figure> <p>Note how column G contains the expected value (30) for the DEPTH variable (highlighted here in green), but this is the 7th column in the data block, not the 6th! The 6th column (variable SOIL) contains the text string “sal” (highlighted here in red), and this is also what is returned by Calc’s VLOOKUP function. Other variables that are based on this function are similarly offset by exactly one column.</p> <p>As it turns out, Quattro Pro’s @VLOOKUP function and Calc’s VLOOKUP function each treat the data block geometry slightly differently: in Quattro Pro, the index of the first column is defined as 0, whereas it is 1 in Calc. This means that when you use Calc to open a Quattro Pro for DOS spreadsheet that uses the @VLOOKUP function, the values that are returned by Calc’s VLOOKUP will be offset by exactly one column<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>.</p> <p>I created a small, self-contained test file that demonstrates this issue, and wrote some accompanying documentation. Both are <a href="https://github.com/openpreserve/format-corpus/tree/master/office/spreadsheet/wq2/vlookup-compat-demo">available here in the OPF Format Corpus</a>.</p> <h3 id="handling-of-cached-formula-values">Handling of cached formula values</h3> <p>As I explained before, my initial attempt at opening the spreadsheet (before recalculation) showed that Calc loads “cached” values for externally referenced cells, but <em>only</em> if these values are numbers. In case of strings, a “#REF” value was shown in my tests. There are two plausible explanations for this. Either:</p> <ol> <li>Quattro Pro stores a cached value if the formula result is a number, but not if it is a text string, or,</li> <li>Quattro Pro stores a cached value irrespective of the formula result’s data type, but Calc doesn’t correctly handle it if it is a string.</li> </ol> <p>Again, this is something we can easily test by isolating a copy of the spreadsheet in an empty directory, and then opening that copy in the Quatro Pro emulation:</p> <figure class="image"> <img src="/images/2025/05/qp-ref-isolated.png" /> </figure> <p>Here we see that Quattro Pro shows the (correct) cached value for each cell with an external reference, irrespective of whether it is a number or a text string. This leads to the conclusion that Quattro Pro <em>does</em> store cached string values, but that Calc doesn’t correctly handle them. I was able to confirm this with some additional tests on another self-contained test file. For brevity I won’t go into details here, but the documentation of these tests and the test file <a href="https://github.com/openpreserve/format-corpus/tree/master/office/spreadsheet/wq2/external-reference-demo">can be found here</a>.</p> <h3 id="combined-effect-of-vlookup-and-cached-values-issues">Combined effect of VLOOKUP and cached values issues</h3> <p>The cached values issue does not only affect cells with external references, but <em>any</em> cell that contains a formula. We can see this in the previously created <a href="https://github.com/openpreserve/format-corpus/tree/master/office/spreadsheet/wq2/vlookup-compat-demo">VLOOKUP demo file</a>, which renders like this in the Quattro Pro emulation:</p> <figure class="image"> <img src="/images/2025/05/qp-vlookup-demo.png" /> </figure> <p>The top 11 rows here contain a 5-column block of data, which is queried with the @VLOOKUP function in row 15. When I opened this file in Calc, this initialy produced:</p> <figure class="image"> <img src="/images/2025/05/lo-init.png" /> </figure> <p>Note how the values in cells D15 and E15 (which both result in a text string in the emulation) are different from the original rendering, while the values in B15 and C15 (both numbers) are correct. After re-calculating (“Data/Calculate/Recalculate Hard”), I got this:</p> <figure class="image"> <img src="/images/2025/05/lo-recalc.png" /> </figure> <p>This is the familiar column shift pattern that results from the @VLOOKUP compatibility issue. However, by itself this doesn’t explain why cells B15 and C15 showed the correct values <em>before</em> the recalculation!</p> <p>My best guess is that on opening a Quattro Pro for DOS spreadsheet, Calc’s intended behaviour is to load the cached cell values, instead of recalculating the underlying formulas. This would be entirely sensible for legacy spreadsheet formats, since it is highly likely that not all functions in their original creation software are completely compatible with Calc. Since the tests in the previous section showed that Calc is unable to read cached values that are text strings, I suspect that in this particular case, Calc recalculates those cells as a fallback. This then results in a rendering that contains a mix of both cached and recalculated values. In most situations this would go unnoticed by the user, but not here, since the recalculated values are affected by the @VLOOKUP compatability issue.</p> <h2 id="conclusions">Conclusions</h2> <p>In the concluding section of <a href="/2014/10/29/quattro-pro-dos-obsolete-format-last">my 2014 post</a> I wrote:</p> <blockquote> <p>[N]o modern-day software is able to correctly handle the Quattro Pro for DOS formats. Add to this that the Quattro Pro for DOS formats are proprietary with (as far as I’m aware) no publicly available specifications, and I think we have a pretty strong candidate for a format that may be (nearly) obsolete.</p> </blockquote> <p>More than ten years onward, things are looking much better now. LibreOffice Calc, which is both free and open-source, was able to read all my old WQ1 and WQ2 files, and it did a better job at this than the (proprietary/closed) Quattro Pro X7 version I used in 2014. The tests with my external reference spreadsheet highlighted two issues though:</p> <ol> <li>An incompatability between a legacy Quattro Pro for DOS function and its modern equivalent in Calc.</li> <li>A lack of support for cached formula values that are text strings, which in turn triggers inconsistent recalculation behaviour.</li> </ol> <p>Realistically, I doubt the compatibility issue itself can (or even should!) be fixed in Calc. Given the large number of legacy spreadsheet formats that Calc supports, and the wide array of functions within each of these formats, there may be many other, similar compatability issues lurking beneath the surface. That being said, the effects of such issues could be mitigated by addressing the second issue, specifically:</p> <ol> <li>By adding read support for cached formula values that are not numbers (text strings, possibly other types as well).</li> <li>By reconsidering the current recalculation behaviour on opening a file. If no cached value can be retrieved, I think Calc shouldn’t automatically recalculate the formula as a fallback, but instead just let the user know that it is not available. (Of course, after this it’s up to the user if they want to manually force a recalculation or not.)</li> </ol> <p>With these changes, Calc’s rendering of a Quattro Pro for DOS file would provide the best approximation of its state when it was last saved.</p> <p>I’ve submitted a <a href="https://bugs.documentfoundation.org/show_bug.cgi?id=166706">bug report</a> about both issues, where I also mentioned the above suggestions.</p> <p>The results of my tests also underline -once again- the risks of migrating or “normalizing” legacy spreadsheets to some modern format. On their own, each of the reported issues can already lead to lost or altered data in a migration action. The combined effect of both issues can result in more complex changes that would be nearly impossible to trace back to the source data. At the very minimum, the original source file should always be kept, as well as a full audit trail of the migration process. Both should also be made available to the user, so they can use it to make an informed judgment of the accuracy of the migrated data.</p> <p>Just like <a href="/2025/05/19/emulating-microsoft-multiplan-spreadsheets-in-dosbox-x">Microsoft Multiplan</a>, the Quattro Pro for DOS case confirms <a href="https://blog.dshr.org/2009/01/are-format-specifications-important-for.html">David Rosenthal’s 2009 assertion</a>:</p> <blockquote> <p>Clearly, formats with open source renderers are, for all practical purposes, immune from format obsolescence.</p> </blockquote> <p>So would I still call the Quattro Pro for DOS format(s) obsolete? Probably not. This would make it the first example I’m aware of, of a file format that was effectively obsolete 10 years ago, but isn’t anymore!</p> <h2 id="further-resources">Further resources</h2> <ul> <li> <p><a href="/2014/10/29/quattro-pro-dos-obsolete-format-last">Quattro Pro for DOS: an obsolete format at last?</a> (original 2014 post)</p> </li> <li> <p><a href="https://openpreservation.org/blogs/opening-johans-quattro-pro-files-quattro-pro-6-win-311/">Opening Johan’s Quattro Pro files in Quattro Pro 6 for Win 3.11</a> by Euan Cochrane</p> </li> <li> <p><a href="https://github.com/openpreserve/format-corpus/tree/master/office/spreadsheet/">Legacy spreadsheet sample files in the OPF Format Corpus</a> (this includes the WQ1 and WQ2 files used here)</p> </li> <li> <p><a href="https://github.com/openpreserve/format-corpus/tree/master/office/spreadsheet/wq2/vlookup-compat-demo">VLOOKUP compatibility demo</a> (showcases VLOOKUP compatibility issue)</p> </li> <li> <p><a href="https://github.com/openpreserve/format-corpus/tree/master/office/spreadsheet/wq2/external-reference-demo">External reference demo</a> (showcases cached values issue)</p> </li> <li> <p><a href="https://bugs.documentfoundation.org/show_bug.cgi?id=166706">LibreOffice bug report</a> on inconsistent handling of cached values in Quattro Pro for DOS (WQ2) spreadsheets</p> </li> <li> <p><a href="https://sourceforge.net/p/libwps/wiki/Home/">Microsoft Works format import library (libwps)</a> (this is the library used by LibreOfice Calc to read the Quattro Pro formats, as well as the Microsoft Multiplan for DOS and Lotus 123 formats)</p> </li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>Here’s <a href="https://web.archive.org/web/20150117070210/https://wiki.documentfoundation.org/Feature_Comparison:_LibreOffice_-_Microsoft_Office">a snapshot of the same feature matrix</a> from January 2015. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>The screenshots in this post are all from the (rather old) 6.4.7.2 version; I later re-did some of my tests in LibreOffice 24.2.7.2, which gave identical results. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>This most likely applies to quattro Pro’s @HLOOKUP function as well. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2025/05/28/quattro-pro-for-dos-revisited-an-obsolete-format-no-more https://bitsgalore.org/2025/05/28/quattro-pro-for-dos-revisited-an-obsolete-format-no-more 2025-05-28T00:00:00+02:00 <figure class="image"> <img src="/images/2025/05/multiplan-box.jpg" alt="Photo of original box in which Multiplan 2.0 was shipped." /> <figcaption>Image sourced from <a href="https://archive.org/details/microsoft-multiplan-2.00-5.25-2.-7z/">Internet Archive</a>, license unknown.</figcaption> </figure> <p>Last week I was contacted by Tristan Zondag from the Netherlands Institute for Sound and Vision for advice on files he had recovered from some old floppy disks as part of an ongoing <a href="https://www.avanet.nl/digitale-archeologie-graven-naar-oude-data/">digital archaeology project</a>. Based on the files’ byte structure, Tristan suspected that these were <a href="https://en.wikipedia.org/wiki/Multiplan">Multiplan</a> files. Multiplan is a spreadsheet application that was developed by Microsoft between 1982 and 1990.</p> <p>However, the files weren’t recognised by <a href="https://digital-preservation.github.io/droid/">DROID</a>, and Tristan also wasn’t able to open them in the original Multiplan software, which is why he asked me to have a look at it. In response to this I did some tests where I ran old MS-DOS versions of the Multiplan software in <a href="https://dosbox-x.com/">DOSBox-X</a>.</p> <p>The main purpose of this post is to document how I made this work. This is probably of interest to others who are working with old Multiplan files. It might also serve as a useful introduction to emulating old MS-DOS software in DOSBox-X.</p> <p>Since the files from Sound and Vision are subject to access restrictions, I’m not able to share them here. So, all examples in this post are based on publicly available Multiplan files. In the final sections I briefly explain how I used the emulation to make the data in those files accessible in modern software, and I also discuss some alternative options.</p> <!-- more --> <h2 id="about-the-multiplan-formats">About the Multiplan formats</h2> <p>First of all it’s important that the version history of Multiplan and its file formats is quite convoluted. In addition to the changes to the format between successive versions of the software, there are also platform-specific differences between the formats used by the DOS, Macintosh and Xenix versions of Multiplan. <a href="https://preservation.tylerthorsted.com/2023/11/10/multiplan/">This 2023 post by Tyler Thorsted</a> gives a good overview, but most likely doesn’t even cover all variants.</p> <h2 id="get-the-old-multiplan-releases">Get the old Multiplan releases</h2> <p>The <a href="https://winworldpc.com/">WinWorld</a> abandonware site hosts <a href="https://winworldpc.com/product/multiplan/">downloads for Multiplan versions 1, 2, 3 and 4</a> for various platforms, as well as some scanned user manuals. A word of warning though: it seems that later versions of the software only offer limited support for earlier versions of the the format.</p> <p>As I was dealing with one of the earlier Multiplan formats here, I used the <a href="https://winworldpc.com/download/c396c2a6-0f5d-3bc3-b611-c3a4c2a83d70">Microsoft Multiplan 2.01 (5.25) (alt)</a> for MS-DOS download, and most of what follows will cover this version and its associated peculiarities. Your mileage may vary for other versions and platforms.</p> <h2 id="emulation-platform-dosbox-x">Emulation platform: DOSBox-X</h2> <p>In order to run old MS-DOS software on a modern PC, we need some sort of emulation or virtualisation platform. For my initial tests I used an old virtual machine that I had set up ages ago in VirtualBox, using disk images of original MS-DOS and Windows 3.11 installation floppies. Since Tristan mentioned he had been using <a href="https://dosbox-x.com/">DOSBox-X</a>, I then gave that a spin as well (I had never even used it before myself!). As it turned out, this worked even better than my VirtualBox setup, and it is also much simpler to set up and operate. So, at this point I decided to abandon my VirtualBox setup, and continue with DOSBox-X.</p> <p>For my tests I used DOSBox-X v2025.05.03, Linux SDL2 64-bit on Linux Mint (Flatpak installer). After launching the software, it shows the following welcome screen:</p> <figure class="image"> <img src="/images/2025/05/dosbox-startup.png" /> </figure> <h2 id="create-work-directory-and-mount-as-drive">Create work directory and mount as drive</h2> <p>At startup DOSBox-X only shows a Z: drive, which is an internal virtual device that contains the DOS environment. In order to install software and work with our own files, we first need to create an empty work directory on our host machine (in my case a Linux desktop machine), and mount that as a new drive in DOSBox-X.</p> <p>To mount this work directory to a drive, go to the “Drive” menu in DOSBox-X, pick a drive letter from the drop-down list (I used “X”), and select “Mount folder as hard drive”. Then navigate to the work directory you just created. In response, DOSBox-X shows this confirmation dialog:</p> <figure class="image"> <img src="/images/2025/05/dosbox-drive-mounted.png" /> </figure> <p>We can set this newly created drive as the work directory in DOSBox-X using:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Z:\&gt;X: </code></pre></div></div> <p>Then generate a directory listing with the “dir” command:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>X:\&gt;dir </code></pre></div></div> <p>This should result in something like this:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> Volume in drive X is X_DRIVE Volume Serial Number is 0000-1234 Directory of X:\ . &lt;DIR&gt; 05/15/2025 3:22p .. &lt;DIR&gt; 05/15/2025 3:23p 0 File(s) 0 Bytes 2 Dir(s) 2,013,265,920 Bytes free </code></pre></div></div> <h2 id="create-multiplan-installation-folder">Create Multiplan installation folder</h2> <p>Now create an installation folder for Multiplan using this command:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>X:\&gt;md mp_2 </code></pre></div></div> <h2 id="mount-multiplan-floppy-image-files">Mount Multiplan floppy image files</h2> <p>Unpack your downloaded 7z file with the Multiplan binaries. After unpacking, you’ll see a directory “Microsoft Multiplan 2.01 (5.25)”, which contains two image files:</p> <ul> <li>Install.img</li> <li>Program.img</li> </ul> <p>These can be mounted as virtual floppy disks in DOSBox-X. To do so, go the the “Drive” menu again, pick the “A” drive, and select “Mount a disk or CD image file”. Then navigate to the folder with the Multiplan disk images, and select the “Install.img” file. This again brings up a confirmation dialog.</p> <p>If all goes well, you can now access the image file like a regular floppy. For example, you can show a directory listing using:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>X:\&gt;dir A: </code></pre></div></div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> Volume in drive A has no label Volume Serial Number is 0000-1234 Directory of A:\ CONVERTD EXE 34,944 02/10/1986 12:00p IBMSETUP COM 19,808 01/05/1987 4:24p INSTALL COM 6,934 02/10/1986 12:00p INSTALL DAT 48,145 02/10/1986 12:00p INSTALL MSG 14,876 02/10/1986 12:00p INSTALL OVL 37,498 02/10/1986 12:00p INSTALL SPC 128 02/10/1986 12:00p PATCH21 EXE 12,040 02/10/1986 12:00p 8 File(s) 174,373 Bytes 0 Dir(s) 183,296 Bytes free </code></pre></div></div> <h2 id="copy-files-from-virtual-floppies">Copy files from virtual floppies</h2> <p>I was unable to install this version of Multiplan directly from the virtual floppies<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. Instead, I managed to make things work by first copying all files from the virtual floppies to the installation folder in my work directory. To do this, first set the work directory to the Multiplan installation folder using:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>X:\&gt;cd mp_2 </code></pre></div></div> <p>Then copy all files from the virtual floppy to this folder:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>X:\MP_2&gt;copy a:\*.* . </code></pre></div></div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> CONVERTD.EXE IBMSETUP.COM INSTALL.COM INSTALL.DAT INSTALL.MSG INSTALL.OVL INSTALL.SPC PATCH21.EXE 8 File(s) copied. </code></pre></div></div> <p>Now unmount the floppy (“Drive”/ “A”/”Unmount drive”), and mount the second image file (“Program.img”). Repeat the above command to copy all files from the second floppy to the “mp_2” folder. This adds another 17 files.</p> <h2 id="install-multiplan">Install Multiplan</h2> <p>Install Multiplan by typing:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>X:\MP_2&gt;install </code></pre></div></div> <p>The installer will prompt you to enter a terminal type. Honestly I don’t know much about the differences between the myriad of choices here, but value “2” (Microsoft Windows) worked fine for me. After hitting the Return key the installation completes.</p> <h2 id="installation-of-later-multiplan-versions">Installation of later MultiPlan versions</h2> <p>The installation procedure is more straighforward for later MultiPlan versions. For example, to install <a href="https://winworldpc.com/download/f3e515e9-39ec-11ee-8470-0200008a0da4">the final 4.20 release</a>, you can simply mount the image file of the first installation floppy to the A: drive, and then run:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>X:\&gt;a:\setup </code></pre></div></div> <p>This launches an interactive installer that will install MultiPlan to a user-defined directory on your virtual hard drive. The installer prompts you to mount the other floppy images along the way.</p> <h2 id="run-multiplan">Run Multiplan</h2> <p>To start Multiplan, simply type “mp” in the terminal<sup id="fnref:6"><a href="#fn:6" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>X:\MP_2&gt;mp </code></pre></div></div> <p>Now Multiplan launches with this gloriously oldschool interface:</p> <figure class="image"> <img src="/images/2025/05/multiplan-startup.png" /> </figure> <p>At the bottom of the screen you see two rows of commands. You can select a command by typing its first letter (e.g. “t” will activate the “Transfer” command), or alternatively you can use the Tab key to move between the commands.</p> <h2 id="load-a-file">Load a file</h2> <p>To load a file, type “t”, which activates the “Transfer” command:</p> <figure class="image"> <img src="/images/2025/05/multiplan-transfer.png" /> </figure> <p>Then type “l” (“Load”), and press one of the arrow keys on your keyboard. This results in a list of all files in the installation directory:</p> <figure class="image"> <img src="/images/2025/05/multiplan-load.png" /> </figure> <p>All files with a “.mod” file extension here are Multiplan spreadsheets, so let’s load the first one (“amor.mod”):</p> <figure class="image"> <img src="/images/2025/05/multiplan-amor.png" /> </figure> <h2 id="load-your-own-files">Load your own files</h2> <p>Confusingly, it seems that this Multiplan version doesn’t allow you to load files that are not in the installation directory<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>. So, to load your own files, you first need to copy them to the Multiplan installation directory. Then instruct DOSBox-X to rescan the drive (“Drive”/”X”/”Rescan drive”). Next time you launch the “Load” command, the new files will appear. As an example, below I loaded one of <a href="https://github.com/thorsted/PRONOM_Research/tree/main/Submissions/Microsoft%20Excel%20v1/Samples/MultiPlan%20v1%20DOS">Tyler’s Multiplan 1 example files</a>:</p> <figure class="image"> <img src="/images/2025/05/multiplan-thorsted.png" /> </figure> <h2 id="using-multiplan-data-in-modern-software">Using Multiplan data in modern software</h2> <p>When I started these tests, I expected that it would be quite difficult to make data in Multiplan files available in modern software. Most online resources I found on this suggested to use the original software to read any old Multiplan files, export the contents to <a href="https://en.wikipedia.org/wiki/Symbolic_Link_(SYLK)">SYLK</a> format, and then import that SYLK file into a modern spreadsheet application. Since neither Microsoft Excel nor LibreOffice Calc were able to read the file Tristan sent me, I initially tried that route. However, as far as I can tell, Multiplan 2 cannot export spreadsheets to SYLK (or any other format for that matter!).</p> <h2 id="libreoffice-can-read-re-saved-file">LibreOffice can read re-saved file</h2> <p>Unexpectedly, re-saving the file (which I suspect was originally created by one of the Multiplan 1.x releases) in Multiplan resulted in a version 2 file that I could open without problems in <a href="https://www.libreoffice.org/discover/calc/">LibreOffice Calc</a>. To my surprise, Calc has <a href="https://wiki.documentfoundation.org/Feature_Comparison:_LibreOffice_-_Microsoft_Office">import support</a> for the v1-3 DOS versions of the Multiplan format. It’s not entirely clear to me why the v1 source file couldn’t be opened, but perhaps this was simply some unsupported variant of the format. For such cases, re-saving the file<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">4</a></sup> (while of course keeping a copy of the original!) in Multiplan provides a useful way to make the data available in LibreOffice Calc. In turn, Calc can then be used to migrate the data to some modern format<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>.</p> <h2 id="export-data-using-the-print-command">Export data using the Print command</h2> <p>Multiplan’s “Print” command (“p”) provides another (but less sophisticated) way to export data. In the simplest case we can use the print to file (“f”) command, which creates a plain text dump:</p> <figure class="image"> <img src="/images/2025/05/multiplan-print-file.png" /> </figure> <p>For best results you might want to experiment with the “Margins” settings (“m”), as the default settings often won’t preserve the column structure.</p> <p>It’s also possible to print to a Postscript file. In order to make this work, you first need to check that your DOSBox-X configuration file contains the following settings<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">6</a></sup>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[parallel] parallel1=printer [printer] printer=true printoutput=ps multipage=true docpath=~/kb/multiplan-dosbox/export timeout=1000 </code></pre></div></div> <p>On my Linux Mint machine (on which I installed the Flatpak version of DOSBox-X) the configuration file is located at<sup id="fnref:7"><a href="#fn:7" class="footnote" rel="footnote" role="doc-noteref">7</a></sup>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~/.var/app/com.dosbox_x.DOSBox-X/config/dosbox-x </code></pre></div></div> <p>You need to close and re-start DOSBox-X for any changes in the configuration file to take effect.</p> <p>Now, if you print to “Printer”, Multiplan creates a Postscript file in the directory that is defined by the “docpath” configuration variable. You can easily convert this to PDF using <a href="https://ghostscript.com/">Ghostscript</a>’s “ps2pdf” tool:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ps2pdf doc1.ps </code></pre></div></div> <p>In my tests I noticed that printing to Postscript resulted in Postscript files where each page is rendered as a bitmapped image, which makes them of limited use. It’s not clear to me if this is a limitation of DOSBox-X, or something that can be resolved with some further configuration tweaks.</p> <h2 id="final-remarks">Final remarks</h2> <p>The Multiplan file format provides another nice example of <a href="https://blog.dshr.org/2009/01/are-format-specifications-important-for.html">David Rosenthal’s 2009 assertion</a>:</p> <blockquote> <p>Clearly, formats with open source renderers are, for all practical purposes, immune from format obsolescence.</p> </blockquote> <p>For the specific case of Tristan’s file, the open source renderer did need a little help from the original software though. It also highlights the importance of open source software such as LibreOffice for keeping data in such legacy formats usable.</p> <h2 id="acknowledgements">Acknowledgements</h2> <p>Thanks are due to Tristan Zondag (the Netherlands Institute for Sound and Vision) for the various useful suggestions in our e-mail exchange about the Multiplan files.</p> <h2 id="further-resources">Further resources</h2> <ul> <li> <p><a href="https://preservation.tylerthorsted.com/2023/11/10/multiplan/">Tyler Thorsted’s blog post on Multiplan</a></p> </li> <li> <p><a href="https://github.com/thorsted/PRONOM_Research/tree/main/Submissions/Microsoft%20Excel%20v1/Samples">Tyler Thorsted’s Multiplan sample files</a></p> </li> <li> <p><a href="https://discmaster.textfiles.com/search?format=multiplanSpreadsheet">Multiplan sample files in DiscMaster</a></p> </li> <li> <p><a href="https://winworldpc.com/product/multiplan/">Multiplan downloads on WinWorld</a></p> </li> <li> <p><a href="https://dosbox-x.com/">DOSBox-X website</a></p> </li> <li> <p><a href="https://www.avanet.nl/digitale-archeologie-graven-naar-oude-data/">Blog post by Tristan Zondag about Sound and Vision’s digital archaeology project</a> (in Dutch)</p> </li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:2"> <p>The software was meant to be run directly from a set of floppies, without an actual harddisk installation. This was not uncommon for early to mid ’80s software. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:6"> <p>For some reason this <em>only</em> seems to work if the command is launched from the installation directory (e.g. typing the full path from some other directory did not work for me). <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:1"> <p>According to the <a href="https://winworldpc.com/download/79c2a9c3-82c3-9a6e-1911-c3a6e280947e">documentation</a> it should be possible to configure a some other data location, but I wasn’t able to make this work. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>To re-save a file, type “t” to activate the “Transfer” command, and then “s” to save. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>LibreOffice also allows you to do this from the command line, which is useful for bulk processing. See: <a href="https://help.libreoffice.org/latest/he/text/shared/guide/convertfilters.html">https://help.libreoffice.org/latest/he/text/shared/guide/convertfilters.html</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>See <a href="https://dosbox-x.com/wiki/Guide:Setting-up-printing-in-DOSBox%E2%80%90X#_example_print_to_postscript">this explanation on the DOSBox-X wiki</a>. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:7"> <p>This folder also contains a “capture” directory, which is where DOSBox-X stores any screenshots made with “Take screenshot” (F12+P) from the “Capture” menu. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2025/05/19/emulating-microsoft-multiplan-spreadsheets-in-dosbox-x https://bitsgalore.org/2025/05/19/emulating-microsoft-multiplan-spreadsheets-in-dosbox-x 2025-05-19T00:00:00+02:00 <figure class="image"> <img src="/images/2025/04/donkey-cart-codeberg.png" alt="Illustration of a stylized, icon-like donkey that is pulling a cart with the words bitsgalore.org written on it, walking from left to right. On its left is the Github Octicon logo, and to its right the Codeberg logo, which depicts a mountain." /> <figcaption>Donkey and cart icons licensed from <a href="https://thenounproject.com/">the Noun Project</a>. Github Octicon icon from <a href="https://en.m.wikipedia.org/wiki/File:Octicons-mark-github.svg">Wikimedia Commons</a>, released under <a href="https://github.com/github/octicons/blob/master/LICENSE">MIT</a> license. Codeberg logo from <a href="https://commons.wikimedia.org/wiki/File:Codeberg_logo.svg">Wikimedia Commons</a>, released under <a href="https://creativecommons.org/publicdomain/zero/1.0/deed.en">CC0 1.0</a>.</figcaption> </figure> <p>Ever since its start in late 2013, this blog has been hosted on Github Pages, using the <a href="https://jekyllrb.com/">Jekyll</a> static site generator. On a technical level this always worked flawlessly, but in the current geopolitical climate I no longer want my site being hosted at a US-based tech giant. After reviewing some options, I decided to migrate the site to <a href="https://codeberg.page/">Codeberg Pages</a>, which is operated by a non-profit organization that is based in Germany. I also implemented a new comments system that is based on ActivityPub. This allows readers to post comments with a Fediverse (e.g. Mastodon) account.</p> <!-- more --> <h2 id="overview">Overview</h2> <p>This post is quite technical in its nature, and is primarily aimed at readers who are also considering to move their website away from Github. I’m assuming some familiarity with building and configuring static websites and Jekyll. First I give an overview of the main differences between Github Pages and Codeberg Pages. Then I explain how I implemented the ActivityPub-based comments system, and how I created an archive of the legacy (Github-based) comments. In the final sections I briefly explain the configuration I used to make the site work on my custom web domain.</p> <h2 id="github-pages-vs-codeberg-pages">Github pages vs Codeberg pages</h2> <p>Unlike Github, Codeberg doesn’t have a built-in Jekyll integration. This means that for a Jekyll-based website (like mine), you explicitly need to build the static HTML locally from the source, and then upload the HTML to a dedicated repo. There are various ways to do this. In my case, I simply created 2 separate repos: <a href="https://codeberg.org/bitsgalore/blog-src">one with the source data</a>, and <a href="https://codeberg.org/bitsgalore/pages">a “pages” repo with the generated HTML</a>. To make this setup work, you need to add a “destination” variable to the source repo’s <a href="https://codeberg.org/bitsgalore/blog-src/src/branch/main/_config.yml">_config.yml</a> file, and set the value to the path of the “pages” repo. In my case this is a sibling directory called “pages”:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>destination: ../pages/ </code></pre></div></div> <p>So, the main difference with the Github-based workflow is that pushing changes to the source repo doesn’t affect the live site, and that in order to update the live site one has to push changes to the “pages” repo<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>.</p> <h2 id="comments-section">Comments section</h2> <p>In 2020 I added a comments feature to this site, which allowed readers to post comments <a href="https://aristath.github.io/blog/static-site-comments-using-github-issues-api">using Github issues</a>. This introduced two challenges to the current migration. First, I had to think of an alternative commenting system that is independent on Github. Since I didn’t want to lose any existing comments, it also meant I had to devise a way to archive those, and incorporate them into the site.</p> <h2 id="activitypub-based-comments">ActivityPub-based comments</h2> <p>I briefly considered a comments system based on Codeberg issues (which has an API that is similar to Github). I ultimately decided against this, mostly because of Codeberg’s much smaller user base. After some thought, I realised that <a href="https://en.wikipedia.org/wiki/ActivityPub">ActivityPub</a>-based comments would be a much better option, as this allows anyone with a Fediverse (e.g. Mastodon) account to post comments. This idea isn’t new, and several others have been doing this already<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>.</p> <h2 id="how-it-works">How it works</h2> <p>The way this works is actually pretty simple. Fediverse services such as Mastodon have a public API, which allows you to get a Toot and all its replies as structured data. As an example, consider this Toot:</p> <p><a href="https://digipres.club/@bitsgalore/114387914694269341">https://digipres.club/@bitsgalore/114387914694269341</a></p> <p>The following API call uses the Toot’s unique identifier, which returns the corresponding data in JSON format:</p> <p><a href="https://digipres.club/api/v1/statuses/114387914694269341/context">https://digipres.club/api/v1/statuses/114387914694269341/context</a></p> <p>With some JavaScript it’s possible to parse these data files, and inject the parsed content back into a page’s HTML. To a visitor this is then visible as a comments thread. See <a href="https://jan.wildeboer.net/2023/02/Jekyll-Mastodon-Comments/">Jan Wildeboer’s post</a> for a more detailed explanation.</p> <h2 id="implementation">Implementation</h2> <p>For readers who are interested in the nitty-gritty of things, my implementation can be found <a href="https://codeberg.org/bitsgalore/blog-src/src/branch/main/_includes/comments.html">here</a>. The JavaScript is actually a straightforward adaptation of <a href="https://codeberg.org/bitsgalore/blog-src/src/commit/d0a4cc139143d497b2dca27c21a484fd35486dd2/_includes/comments.html">the code I previously used for the Github comments</a>. This requires that each page (blog post) for which comments are enabled has an “ap_id” identifier. This is the aforementioned unique Toot identifier, and it is defined in the YAML frontmatter of each post’s Markdown source file. Here’s an example (full file <a href="https://codeberg.org/bitsgalore/blog-src/src/branch/main/_posts/2024/2024-10-30-jpeg-quality-estimation-using-simple-least-squares-matching-of-quantization-tables.md">here</a>):</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>--- ap_id: 114416014116498692 comment_id: 92 description: This post describes a simple method for estimating JPEG compression quality. It is based on a straightforward comparison of a file's quantization tables against the quantization tables from the JPEG standard using least squares matching. It also proposes a measure to characterize the similarity of an image's quantization tables to these standard tables, which is useful for assessing the accuracy of the quality estimate. headImage: /images/2024/10/quality-sign.jpg headImageAltText: Photograph of faded sign on building front showing the word 'Quality'. layout: post tags: - JPEG - ImageMagick - ExifTool title: JPEG quality estimation using simple least squares matching of quantization tables --- </code></pre></div></div> <h2 id="comments-bot">Comments bot</h2> <p>Although I could have used my <a href="https://digipres.club/@bitsgalore">regular Fediverse account on digipres.club</a> for initiating comment threads, people might unknowingly end up in my blog’s comments section by replying to a Toot. To reduce the chances of this from happening, I set up a <a href="https://digipres.club/@bitsgaloreBlogComments">dedicated blog comments account for this instead</a>.</p> <p>From the outset I wanted to enable commenting for <em>all</em> posts on my blog (currently 80). This means that for each post the following actions are needed:</p> <ol> <li>Create a Toot that initiates the comments thread.</li> <li>Add the identifier of this Toot to the post’s YAML frontmatter as the “ap_id” variable.</li> </ol> <p>Doing this manually for 80 posts would be very tedious, so instead I wrote a <a href="https://codeberg.org/bitsgalore/commentsBot">commentsBot</a> application that completely automates this. It uses <a href="https://github.com/halcy/Mastodon.py">Mastodon.py</a> to post to the blog comments Fediverse account, and <a href="https://github.com/eyeseast/python-frontmatter">Python Frontmatter</a> to add the corresponding identifiers to the YAML frontmatter. I primarily wrote the comments bot for this blog, but I imagine the code might be useful to others as well.</p> <h2 id="archiving-the-legacy-github-comments">Archiving the legacy Github comments</h2> <p>To preserve the legacy (Github-based) comments, I first downloaded the JSON files that contain the comments data using <a href="https://codeberg.org/bitsgalore/blogResources/src/branch/main/fetch-github-comments.py">this script</a><sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>. I then added all these files to a <a href="https://codeberg.org/bitsgalore/blog-src/src/branch/main/_data/comments-gh">dedicated subdirectory of the source repo’s <em>_data</em> folder</a>. The base name of each JSON file corresponds to the value of the “comment_id” variable in the YAML frontmatter of each post’s Markdown source file.</p> <p>As an example, for <a href="https://codeberg.org/bitsgalore/blog-src/src/branch/main/_posts/2020/2020-03-11-does-microsoft-onedrive-export-large-ZIP-files-that-are-corrupt.md">this post</a> the “comment_id” is 70. So, the corresponding data file with the comments is:</p> <p><a href="https://codeberg.org/bitsgalore/blog-src/src/branch/main/_data/comments-gh/70.json">https://codeberg.org/bitsgalore/blog-src/src/branch/main/_data/comments-gh/70.json</a></p> <p>I wrote some <a href="https://shopify.github.io/liquid/">Liquid</a> code that <a href="https://codeberg.org/bitsgalore/blog-src/src/commit/e98f61bf65ba046dd132d90ac8db84da8ff72396/_includes/comments.html#L87">parses the JSON into styled HTML</a>. It uses the exact same CSS classes that are also used for the ActivityPub comments, so both are rendered with identical formatting. Scroll down to the comments section of <a href="/2021/09/06/pdf-processing-and-analysis-with-open-source-tools.html">this post</a> to see what this looks like.</p> <h2 id="making-the-domains-file-work">Making the .domains file work</h2> <p>Another thing to watch out for: if (like me) you serve your site on a <a href="https://docs.codeberg.org/codeberg-pages/using-custom-domain/">custom domain</a>, you need to define this domain (and any subdomains) in a “.domains” file (<a href="https://codeberg.org/bitsgalore/blog-src/src/branch/main/.domains">here’s mine</a>). However, by default Jekyll ignores “hidden” files that start with “.”, which means they won’t be included in the generated HTML! This in turn will break the domain configuration. The solution is to add any “hidden” files or directories to the “include” variable in <a href="https://codeberg.org/bitsgalore/blog-src/src/branch/main/_config.yml">_config.yml</a>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># By default Jekyll ignores files and dirs starting with ".", so need to bypass # for ".well-known" dir and ".domains" file include: [".well-known", ".domains"] </code></pre></div></div> <h2 id="making-dnssec-work-with-custom-domains">Making DNSSEC work with custom domains</h2> <p>My bitsgalore.org domain is secured with <a href="https://en.wikipedia.org/wiki/Domain_Name_System_Security_Extensions">DNSSEC</a>. I initially had some difficulty getting it to work on the www subdomain. I eventually made this work (<a href="https://codeberg.org/Codeberg/Community/issues/1881">thanks to a suggestion by one of the Codeberg admins</a>) using the following configuration:</p> <ol> <li>Create A, AAAA and TXT records for both the apex domain and the www subdomain.</li> <li><strong>Don’t</strong> create a CNAME record.</li> <li>Define both the apex domain and the www subdomain in the .domains file.</li> </ol> <p>In my case this looks like this:</p> <table> <thead> <tr> <th style="text-align: left">Domain</th> <th style="text-align: left">Type</th> <th style="text-align: left">Data</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"> </td> <td style="text-align: left">A</td> <td style="text-align: left">217.197.84.141</td> </tr> <tr> <td style="text-align: left">www</td> <td style="text-align: left">A</td> <td style="text-align: left">217.197.84.141</td> </tr> <tr> <td style="text-align: left"> </td> <td style="text-align: left">AAAA</td> <td style="text-align: left">2a0a:4580:103f:c0de::2</td> </tr> <tr> <td style="text-align: left">www</td> <td style="text-align: left">AAAA</td> <td style="text-align: left">2a0a:4580:103f:c0de::2</td> </tr> <tr> <td style="text-align: left"> </td> <td style="text-align: left">TXT</td> <td style="text-align: left">bitsgalore.codeberg.page</td> </tr> <tr> <td style="text-align: left">www</td> <td style="text-align: left">TXT</td> <td style="text-align: left">bitsgalore.codeberg.page</td> </tr> </tbody> </table> <p>And <a href="https://codeberg.org/bitsgalore/blog-src/src/branch/main/.domains">my .domains file</a> contains the following lines:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bitsgalore.org www.bitsgalore.org </code></pre></div></div> <h2 id="final-thoughts">Final thoughts</h2> <p>Unlike Github, Codeberg is run by a non-profit organization that is funded through donations. Many Fediverse/Mastodon instances are run by volunteers as well. If this post inspired you to move your website there as well, consider <a href="https://donate.codeberg.org/">making a donation to Codeberg</a> (or <a href="https://join.codeberg.org/">become a Codeberg member</a>), and check if your Fediverse instance accepts donations.</p> <p>Thanks are due to Codeberg and the admins at <a href="https://digipres.club/">digipres.club</a> for making this possible!</p> <h2 id="further-resources">Further resources</h2> <ul> <li><a href="https://codeberg.org/bitsgalore/blog-src/">Blog source repo</a></li> <li><a href="https://codeberg.org/bitsgalore/commentsBot">commentsBot code</a></li> <li><a href="https://codeberg.org/bitsgalore/blogResources">blogResources</a> - various utility scripts I’ve written for managing this website</li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:2"> <p>I supppose it would also be possible to implement this as two separate branches in one single repo, but I haven’t looked into this. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:1"> <p>See eg. <a href="https://jan.wildeboer.net/2023/02/Jekyll-Mastodon-Comments/">Jan Wildeboer</a> and <a href="https://cassidyjames.com/blog/fediverse-blog-comments-mastodon/">Cassidy James Blaede</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>It’s important to use a Github token for this: although it’s possible to use the Github API without a token, you’re likely to hit the API’s rate limit pretty quickly, with the result that the script won’t download all JSON files. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2025/04/30/changes-to-the-blog-migration-to-codeberg-and-activitypub-based-comments https://bitsgalore.org/2025/04/30/changes-to-the-blog-migration-to-codeberg-and-activitypub-based-comments 2025-04-30T00:00:00+02:00 <figure class="image"> <img src="/images/2024/12/ddac-logo-oscilloscope.jpg" alt="Digital Dark Age logo, which depicts a human figure holding a large floppy disk, with to its left the words Digital Dark Age Crew. The background shows the screen of an oscilloscope." /> <figcaption>Digital Dark Age Crew logo.</figcaption> </figure> <p>One of the most elusive items in the <a href="/2022/11/03/wheel-out-the-digital-dark-age-klaxon">Digital Dark Age Crew</a> back catalogue is “Y2K”, which deals with the <a href="https://en.wikipedia.org/wiki/Year_2000_problem">Year 2000 problem</a>. Originally planned as a December 1999 release, the track was never finished due to a succession of technical problems. Some early demos of “Y2K” have surfaced as bootlegs, and many fans of the group rate these amongst the most sought-after Digital Dark Age Crew tracks.</p> <!-- more --> <h2 id="lost-and-found">Lost and found</h2> <p>The original recordings were thought to have been lost in the fire that occurred while the group were shooting their infamous 2007 “Watch the Digital Dark Age Crew Burn a Million PDFs” video. Earlier this year, the original DAT tapes of the “Y2K” sessions were unexpectedly found in a derelict taxidermy shop. The finder, who prefers to remain anonymous, kindly returned the tapes to sole remaining Digital Dark Age Crew member Marinus Nullbyte.</p> <h2 id="definitive-version">Definitive version</h2> <p>Listening to these early recordings again after 25 years, Nullbyte was struck at how the central message of the track hadn’t lost any of its power. So, Nullbyte embarked on a journey to finally finish “Y2K”, using the rough demos as a guide. This has resulted in the “definitive” version of “Y2K”, which will no doubt delight Digital Dark Age Crew fans all around the globe.</p> <h2 id="video">Video</h2> <p>Nullbyte also produced a brand new video to accompany the release of “Y2K”.</p> <figure class="image"> <img src="/images/2024/12/composite-y2k.jpg" /> <figcaption>Composite of stills from the video of the track "Y2K" by the Digital Dark Age Crew.</figcaption> </figure> <p>Enjoy, and remember: tomorrow, we could all be living in the dark ages!</p> <figure class="image"> <iframe width="750" height="422" src="https://www.youtube-nocookie.com/embed/pJ86meMFqT8" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe> <figcaption>Digital Dark Age Crew - Y2K.</figcaption> </figure> <p>Direct link to video on YouTube, in case the embedded player doesn’t work:</p> <p><a href="https://youtu.be/pJ86meMFqT8">https://youtu.be/pJ86meMFqT8</a></p> https://bitsgalore.org/2024/12/30/y2k https://bitsgalore.org/2024/12/30/y2k 2024-12-30T00:00:00+01:00 <figure class="image"> <img src="/images/2024/12/Control_room_pt_tupper.jpg" alt="Photo of the interior of the control room in a fossil fuel power plant. The walls at the back are completely covered with large control panels. In front of it an operator sits at a desk in his chair." /> <figcaption><a href="https://commons.wikimedia.org/wiki/File:Control_room_pt_tupper.jpg">Control room in Fossil fuel power plant in Point Tupper, Nova Scotia</a>. Achim Hering, Public domain, via Wikimedia Commons.</figcaption> </figure> <p>This post introduces <a href="https://github.com/KBNLresearch/pdfquad/tree/main">Pdfquad</a>, a software tool that for automated quality assessment for large digitisation batches. The software was developed specifically for the Digital Library for Dutch Literature (DBNL), but it might be adaptable to other users and organisations as well.</p> <!-- more --> <h2 id="context-of-this-work">Context of this work</h2> <p>The Digital Library for Dutch Literature (<a href="https://www.dbnl.org/">DBNL</a>) is a collection of literary texts from the Dutch language area. It is a collaboration between the Dutch Language Union, the Flanders Heritage Libraries and the KB. As part of its mission, DBNL digitises books and periodicals, which are then made available on its website.</p> <p>Most of the digitisation work is contracted out to external suppliers. Unlike most of the KB’s digitised collections, the DBNL material is scanned to PDF format. For each publication, two versions are created:</p> <ol> <li>A production master PDF, which serves as the source material for several derived products (PDF and EPUB versions with high quality, manually transcribed text).</li> <li>A (relatively) small access PDF with the scanned pages, which is made available on the DBNL website.</li> </ol> <p><a href="https://www.dbnl.org/tekst/eeml001duis02_01/">Here’s an example publication</a>. Note the “Downloads” section, which contains links to the access PDFs and EPUB.</p> <h2 id="pdf-requirements">PDF requirements</h2> <p>The scanned PDFs need to conform to requirements that are defined in a technical specifications document, which has been updated last year. It covers both aspects related to capture quality, as well as technical characteristics of the PDF. The latter category includes things like:</p> <ul> <li>Scanned pages must be encoded using JPEG compression, using 85% quality for the production master, and 50% quality for the access file.</li> <li>Pages must be scanned in full-colour, with a colour space that is defined by an embedded ICC profile.</li> <li>Encryption and any other access restrictions are not allowed.</li> <li>Digital signatures and watermarks are not allowed.</li> <li>Any content besides the scans is not allowed (no layers or active content).</li> <li>The PDF must not open with thumbnails.</li> </ul> <p>Thus far, conformance to such technical aspects was mostly tested manually on a sample basis, using a variety of software tools. This situation is not ideal, especially given the fact that the PDFs are delivered as part of huge production batches. At the request of our Digitisation department, I looked into a way to do these quality checks in a more systematic and scalable manner. This has resulted in the <a href="https://github.com/KBNLresearch/pdfquad">Pdfquad</a> software. The name is an acronym for “PDF QUality Assessment for Digitisation batches”. It can be used to automatically assess PDF documents in a digitisation batch against one or more sets of user-defined technical requirements.</p> <h2 id="general-concept">General concept</h2> <p>The general idea behind Pdfquad isn’t entirely new. In 2013 I wrote the <a href="https://github.com/KBNLresearch/jprofile">Jprofile</a> tool, which does automated quality checks on batches of JP2 (JPEG 2000 Part 1) images. Jprofile uses <a href="https://jpylyzer.openpreservation.org/">Jpylyzer</a> to extract technical characteristics of a JP2 file, and then evaluates these characteristics against a set of technical requirements that are encoded as <a href="https://en.wikipedia.org/wiki/Schematron">Schematron</a> rules. This software was used for several years operationally by our digitisation department<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>.</p> <p>Even though Jprofile was designed for a different file format, the overall concept is equally usable for batches with PDFs. So, I took the old Jprofile code as a starting point, removed all JPEG 2000 specific code, adapted it to our PDF case, and made it quite a bit more flexible in the process.</p> <h2 id="pdfquad-overview">Pdfquad overview</h2> <p>The following figure gives a general overview of how Pdfquad works:</p> <figure class="image"> <img src="/images/2024/12/pdfquad-flow.png" alt="Flowchart that shows high-level overview of Pdfquad's functional components." /> <figcaption>Overview of Pdfquad's functional components and workflow.</figcaption> </figure> <p>For each PDF in a batch, it extracts the relevant technical characteristics, which are serialized as XML. This XML is then assessed against a set of Schematron rules, which represent the requirements. There may be different schemas (e.g. for production masters and access PDFs). A profile defines which schema must be used for a specific PDF, based on its file name or the name of its direct parent directory. Finally, the results of both the feature extraction and the Schematron assessment are written to a set of output files. In the following sections I will describe these components in more detail.</p> <h2 id="pdf-characteristics-vs-image-characteristics">PDF characteristics vs image characteristics</h2> <p>Some of the DBNL PDF requirements reflect technical characteristics at the PDF document level (e.g. the presence of encryption), whwereas aspects like JPEG quality must be evaluated at the level of the image stream. In most cases, an image in PDF is implemented as an <a href="https://github.com/pdf-association/arlington-pdf-model/blob/master/tsv/latest/XObjectImage.tsv">Image XObject</a>. This is a PDF dictionary with various key-value pairs that describe the image properties, followed by the image stream. In our case, we expect the image stream to contain data in JPEG format. This means we cannot just use any PDF feature extraction tool (e.g. <a href="https://verapdf.org/">VeraPDF</a>), as such tools are unable to provide the necessary information at the image stream level.</p> <h2 id="feature-extraction-with-pymupdf-and-pillow">Feature extraction with PyMuPDF and Pillow</h2> <p>As a solution to this, Pdfquad uses the <a href="https://pymupdf.readthedocs.io/">PyMuPDF</a> library to parse the PDF. PyMuPDF is also used to extract all relevant PDF-level characteristics. For each image, it then passes the image stream data to the <a href="https://pillow.readthedocs.io/">Python Imaging Library</a> (Pillow), which is used to extract the image-level characteristics. With this combination of PyMuPDF and Pillow, I was able to extract all information that is needed to verify all but one of our PDF requirements. The exception here was JPEG quality, which doesn’t immediately follow from Pillow’s output. However, Pillow can report the quantization tables of a JPEG image, and these can be used to estimate JPEG quality. My <a href="/2024/10/30/jpeg-quality-estimation-using-simple-least-squares-matching-of-quantization-tables">earlier post on JPEG quality estimation</a> covers this in detail, and presents a least-squares matching method that is also used in Pdfquad.</p> <h2 id="feature-extraction-walkthrough">Feature extraction walkthrough</h2> <p>It’s worth pointing out that Pdfquad’s feature extraction is currently limited to those characteristics that are needed to verify the DBNL requirements. However, extending the extraction to additional characteristics would be pretty straightforward. Readers who are interested in this (or in feature extraction with PyMuPDF and Pillow in general) might want to check out Pdfquad’s <a href="https://github.com/KBNLresearch/pdfquad/blob/main/pdfquad/properties.py">properties module</a>. Here’s a brief walkthrough:</p> <ul> <li>The <a href="https://github.com/KBNLresearch/pdfquad/blob/7f33e7820e61a07ffb1f5c2b6c3e6e9084a57c82/pdfquad/properties.py#L57C5-L57C18">getProperties</a> function parses a PDF document, extracts some document-level characteristics and then iterates over all pages.</li> <li>The <a href="https://github.com/KBNLresearch/pdfquad/blob/7f33e7820e61a07ffb1f5c2b6c3e6e9084a57c82/pdfquad/properties.py#L177">getPageProperties</a> function extracts all page-level characteristics, and iterates over all images on a page.</li> <li>The <a href="https://github.com/KBNLresearch/pdfquad/blob/7f33e7820e61a07ffb1f5c2b6c3e6e9084a57c82/pdfquad/properties.py#L209">getImageProperties</a> function process an image object, and calls two functions that extract the dictionary-level and stream-level characteristics, respectively.</li> <li>The <a href="https://github.com/KBNLresearch/pdfquad/blob/7f33e7820e61a07ffb1f5c2b6c3e6e9084a57c82/pdfquad/properties.py#L232">getImageDictProperties</a> function extracts all characteristics at the image dictionary level.</li> <li>The <a href="https://github.com/KBNLresearch/pdfquad/blob/7f33e7820e61a07ffb1f5c2b6c3e6e9084a57c82/pdfquad/properties.py#L253">getImageStreamProperties</a> function extracts all characteristics at the image stream level, using Pillow.</li> </ul> <p>The feature extraction code also keeps track of any exceptions that may occur during parsing. These usually indicate a problem with the PDF, and the DBNL Schematron includes rules that verify their absence.</p> <h2 id="profiles-and-schemas">Profiles and schemas</h2> <p>Pdfquad needs to understand the structure of a digitisation batch, and how the PDFs inside it must be evaluated. As an example, let’s assume we have the following batch structure:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>└── 20241105 └── _boe012192401 ├── 300dpi-50 │   └── _boe012192401_01.pdf └── 300dpi-85 └── _boe012192401_01.pdf </code></pre></div></div> <p>Here we have a directory tree where each “300dpi-85” directory contains a high-quality production master PDF, and each “300dpi-50” directory an access PDF. In order to analyse this batch structure, Pdfquad needs to know two things:</p> <ol> <li>A file or directory name pattern that differentiates between production master and access PDFs.</li> <li>A reference to the Schematron file that is used to evaluate both.</li> </ol> <p>Both are defined in an XML-formatted profile, which Pdfquad expects as a command-line argument. Here’s an example:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">&lt;?xml version="1.0"?&gt;</span> <span class="nt">&lt;profile&gt;</span> <span class="nt">&lt;schema</span> <span class="na">type=</span><span class="s">"parentDirName"</span> <span class="na">match=</span><span class="s">"endswith"</span> <span class="na">pattern=</span><span class="s">"pi-85"</span><span class="nt">&gt;</span>pdf-dbnl-85.sch<span class="nt">&lt;/schema&gt;</span> <span class="nt">&lt;schema</span> <span class="na">type=</span><span class="s">"parentDirName"</span> <span class="na">match=</span><span class="s">"endswith"</span> <span class="na">pattern=</span><span class="s">"pi-50"</span><span class="nt">&gt;</span>pdf-dbnl-50.sch<span class="nt">&lt;/schema&gt;</span> <span class="nt">&lt;/profile&gt;</span> </code></pre></div></div> <p>Here, the profile contains two <em>schema</em> elements, which represent the production master and access PDF, respectively. Each element refers to a corresponding Schematron file. In this case, the values of the “type”, “match” and “pattern” “attributes in the first element tell Pdfquad that if the name of a PDF’s direct parent directory ends with “pi-85”, it must use Schematron file “pdf-dbnl-85.sch”.</p> <p>This file, which can be <a href="https://github.com/KBNLresearch/pdfquad/blob/main/pdfquad/schemas/pdf-dbnl-85.sch">viewed here</a>, contains the Schematron rules that are used to evaluate the extracted characteristics.</p> <p>Although the profile and Schemas that are part of the current Pdfquad release are specific to the DBNL case, it is possible to use the software with custom profiles and schemas. This could extend its applicability to other PDF-based digitisation workflows and batch structures.</p> <h2 id="output">Output</h2> <p>For each batch, Pdfquad produces two types of output:</p> <ol> <li>A comma-delimited summary file. This lists, for each PDF, whether the Schematron assessment completed without errors, the outcome of the Schematron assessment, the number of pages, and a reference to the comprehensive output file (see below).</li> <li>One or more comprehensive output files in XML format. These contain all extracted properties, as well a the Schematron report and the assessment status. <a href="https://github.com/KBNLresearch/pdfquad/blob/main/examples/pq_batchtest_001.xml">Here’s an example</a>.</li> </ol> <p>The comprehensive output can get really large. Because of this, Pdfquad splits the results across multiple output files, using the following naming convention (the actual file prefixes can be defined as a command-line option):</p> <ul> <li>pq_mybatch_001.xml</li> <li>pq_mybatch_002.xml</li> <li>etcetera</li> </ul> <p>By default, Pdfquad limits the number of reported PDFs for each output file to 10, after which it creates a new file. There’s a command-line option to change this behaviour.</p> <h2 id="final-thoughts">Final thoughts</h2> <p>Pdfquad is still in early development, and the current feature extraction component in particular is limited to the specific needs of our DBNL digitisation workflow. However, I would expect that the software might be useful for others as well. The profile and schema definitions offer a lot of flexibility for different batch structures, without any changes to the code. This was a deliberate design choice, as we may use the software for PDFs from some other digitisation projects in the future. Meanwhile, the feature extraction module could be extended quite easily to include additional characteristics.</p> <p>On a different note, I was pleasantly surprised by PyMuPDF. I had never worked with it before (or any other Python PDF library for that matter), but using it turned out to be unexpectedly straighforward. I will definitely return to it for future PDF-related work, and other PDF/Python heads should probably check it out.</p> <h2 id="link-to-pdfquad">Link to Pdfquad</h2> <p>Pdfquad and its documentation can be found here:</p> <p><a href="https://github.com/KBNLresearch/pdfquad/tree/main">PDF QUality Assessment for Digitisation batches</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>It has since been repaced by third-party software with the same functionality (which also uses Jpylyzer) <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2024/12/13/pdf-quality-assessment-for-digitisation-batches-with-python-pymupdf-and-pillow https://bitsgalore.org/2024/12/13/pdf-quality-assessment-for-digitisation-batches-with-python-pymupdf-and-pillow 2024-12-13T00:00:00+01:00 <figure class="image"> <img src="/images/2024/11/ghost.jpg" alt="Aquatint showing a graveyard scene in front of a church. At the center is a man in armour with a distressed expression on his face. To his left is a skeleton, and to his right a ghost." /> <figcaption><a href="https://wellcomecollection.org/works/sfqkzeu2">A man in armour is confronted by a ghost and a skeleton. </a>Aquatint. Wellcome Collection, Public Domain.</figcaption> </figure> <p>In a <a href="https://digitalpreservation.fi/en/2024-phantom-pdf-file">recent blog post</a>, colleagues at the National Digital Preservation Services in Finland addressed an issue with PDF files that contain strings with octal escape sequences. These are not parsed correctly by <a href="https://jhove.openpreservation.org/">JHOVE</a>, and the resulting parse errors ultimately lead to (seemingly unrelated) validation errors. The authors argue that octal escape sequences present a preservation risk, as they may confuse other software besides JHOVE. Since this claim is not backed up by any evidence, here I put this to the test using 8 different PDF processing tools and libraries.</p> <!-- more --> <h2 id="the-phantom-of-a-pdf-file">The Phantom of a PDF File</h2> <p>A few weeks ago I came across <a href="https://digitalpreservation.fi/en/2024-phantom-pdf-file">The Phantom of a PDF File</a>, a blog post by Juha Lehtonen &amp; Johan Kylander. In this post, they address an issue with a <a href="https://digitalpreservation.fi/sites/default/files/2024/phantom_of_a_pdf_file_blog_post_2024.pdf">PDF file</a> that gives an unexpected validation error in <a href="https://jhove.openpreservation.org/">JHOVE</a>. Digging into the PDF structure, they were able to track this down to the presence of <a href="https://en.wikipedia.org/wiki/Escape_sequences_in_C">octal escape sequences</a> in the “Producer” field of the file’s Document Information Dictionary:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>9 0 obj &lt;&lt; /Title (Boo) /CreationDate (D:20241029134330Z00'00') /Producer (\376\377\000P\000D\000F\000 \000P\000h\000a\000n\000t\000o\000m\000\000) &gt;&gt; endobj </code></pre></div></div> <p>JHOVE cannot handle octal escape sequences properly, which leads to a parse error that ultimately results in JHOVE reporting a validation error that is completely unrelated to the “Producer” field. In their original post, the authors claimed that the way the octal escape sequences are used in this file is not allowed by the PDF specification. They also advised against the use of octal escape sequences, and to restrict the values in PDF metadata fields to plain ASCII, or otherwise UTF-16BE. Finally they argue that the presence of octal escape sequences<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup> “raises the risk of causing problems in the future”, and that “JHOVE probably is not the only software that will get confused” by this.</p> <h2 id="is-jhove-the-only-software-that-gets-confused-by-octal-escape-sequences">Is JHOVE the only software that gets confused by octal escape sequences?</h2> <p>After reading this post, I posted the link to <a href="https://github.com/openpreserve/jhove/issues/927">this existing ticket on PDF parse issues</a>. This immediately resulted in <a href="https://github.com/openpreserve/jhove/issues/927#issuecomment-2454914910">a response by Peter Wyatt</a> of the <a href="https://pdfa.org/">PDF Association</a>, who pointed out that, contrary to the claims made in the initial version of the post, the use of octal escape sequences in the “problematic” file is actually perfectly valid according to <a href="https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/PDF32000_2008.pdf">ISO 32000-1</a>. In response to this comment, the authors corrected this in an update to their post. However, they seem to stick by their claim that “JHOVE probably is not the only software that will get confused” by octal escape sequences. Out of curiosity, I decided to put this to the test.</p> <h2 id="enter-the-opf-phantom">Enter the OPF Phantom</h2> <p>The <a href="https://digitalpreservation.fi/en/2024-phantom-pdf-file">original file</a> contains two “Producer” fields: one in the Document Information Dictionary (which is the one we’re interested in here), and another one that is part of the XMP metadata. Since both have the same value (“PDF Phantom”), I changed the XMP value to “OPF Phantom” in a Hex editor. This way, we can easily distiguish between both fields. The modified file <a href="https://github.com/user-attachments/files/17685339/phantom_modified_xmp.pdf">is available here</a>.</p> <p>Next, I ran the modified file through 8 different PDF tools and libraries, and inspected the result to see how the “Producer” field is handled. Below are the commands and results.</p> <h2 id="exiftool"><a href="https://exiftool.org/">ExifTool</a></h2> <p>Command:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>exiftool -X phantom_modified_xmp.pdf </code></pre></div></div> <p>Result:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">&lt;?xml version='1.0' encoding='UTF-8'?&gt;</span> <span class="nt">&lt;rdf:RDF</span> <span class="na">xmlns:rdf=</span><span class="s">'http://www.w3.org/1999/02/22-rdf-syntax-ns#'</span><span class="nt">&gt;</span> <span class="nt">&lt;rdf:Description</span> <span class="na">rdf:about=</span><span class="s">'phantom_modified_xmp.pdf'</span> <span class="na">xmlns:et=</span><span class="s">'http://ns.exiftool.org/1.0/'</span> <span class="na">et:toolkit=</span><span class="s">'Image::ExifTool 12.60'</span> <span class="na">xmlns:ExifTool=</span><span class="s">'http://ns.exiftool.org/ExifTool/1.0/'</span> <span class="na">xmlns:System=</span><span class="s">'http://ns.exiftool.org/File/System/1.0/'</span> <span class="na">xmlns:File=</span><span class="s">'http://ns.exiftool.org/File/1.0/'</span> <span class="na">xmlns:PDF=</span><span class="s">'http://ns.exiftool.org/PDF/PDF/1.0/'</span> <span class="na">xmlns:XMP-x=</span><span class="s">'http://ns.exiftool.org/XMP/XMP-x/1.0/'</span> <span class="na">xmlns:XMP-pdf=</span><span class="s">'http://ns.exiftool.org/XMP/XMP-pdf/1.0/'</span><span class="nt">&gt;</span> <span class="nt">&lt;ExifTool:ExifToolVersion&gt;</span>12.60<span class="nt">&lt;/ExifTool:ExifToolVersion&gt;</span> <span class="nt">&lt;System:FileName&gt;</span>phantom_modified_xmp.pdf<span class="nt">&lt;/System:FileName&gt;</span> <span class="nt">&lt;System:Directory&gt;</span>.<span class="nt">&lt;/System:Directory&gt;</span> <span class="nt">&lt;System:FileSize&gt;</span>5.9 kB<span class="nt">&lt;/System:FileSize&gt;</span> <span class="nt">&lt;System:FileModifyDate&gt;</span>2024:11:09 00:22:05+00:00<span class="nt">&lt;/System:FileModifyDate&gt;</span> <span class="nt">&lt;System:FileAccessDate&gt;</span>2024:11:09 00:22:46+00:00<span class="nt">&lt;/System:FileAccessDate&gt;</span> <span class="nt">&lt;System:FileInodeChangeDate&gt;</span>2024:11:09 00:22:05+00:00<span class="nt">&lt;/System:FileInodeChangeDate&gt;</span> <span class="nt">&lt;System:FilePermissions&gt;</span>-rw-rw-r--<span class="nt">&lt;/System:FilePermissions&gt;</span> <span class="nt">&lt;File:FileType&gt;</span>PDF<span class="nt">&lt;/File:FileType&gt;</span> <span class="nt">&lt;File:FileTypeExtension&gt;</span>pdf<span class="nt">&lt;/File:FileTypeExtension&gt;</span> <span class="nt">&lt;File:MIMEType&gt;</span>application/pdf<span class="nt">&lt;/File:MIMEType&gt;</span> <span class="nt">&lt;PDF:PDFVersion&gt;</span>1.4<span class="nt">&lt;/PDF:PDFVersion&gt;</span> <span class="nt">&lt;PDF:Linearized&gt;</span>No<span class="nt">&lt;/PDF:Linearized&gt;</span> <span class="nt">&lt;PDF:PageCount&gt;</span>1<span class="nt">&lt;/PDF:PageCount&gt;</span> <span class="nt">&lt;PDF:Title&gt;</span>Boo<span class="nt">&lt;/PDF:Title&gt;</span> <span class="nt">&lt;PDF:CreateDate&gt;</span>2024:10:29 13:43:30Z<span class="nt">&lt;/PDF:CreateDate&gt;</span> <span class="nt">&lt;PDF:Producer&gt;</span>PDF Phantom<span class="nt">&lt;/PDF:Producer&gt;</span> <span class="nt">&lt;XMP-x:XMPToolkit&gt;</span>Image::ExifTool 12.71<span class="nt">&lt;/XMP-x:XMPToolkit&gt;</span> <span class="nt">&lt;XMP-pdf:Producer&gt;</span>OPF Phantom<span class="nt">&lt;/XMP-pdf:Producer&gt;</span> <span class="nt">&lt;/rdf:Description&gt;</span> <span class="nt">&lt;/rdf:RDF&gt;</span> </code></pre></div></div> <p>ExifTool correctly decodes the octal escape sequences (PDF:Producer), and also extracts the XMP value (XMP-pdf:Producer).</p> <h2 id="pdfcpu"><a href="https://pdfcpu.io/">Pdfcpu</a></h2> <p>Command:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pdfcpu info phantom_modified_xmp.pdf </code></pre></div></div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> PDF version: 1.4 Page count: 1 Page size: 595.28 x 841.89 points ............................................ Title: Boo Author: Subject: PDF Producer: PDF Phantom Content creator: Creation date: D:20241029134330Z00'00' Modification date: ............................................ Tagged: No Hybrid: No Linearized: No Using XRef streams: No Using object streams: No Watermarked: No ............................................ Encrypted: No Permissions: Full access </code></pre></div></div> <p>Pdfcpu correctly decodes the octal escape sequences (PDF Producer).</p> <h2 id="pdfinfo-poppler">pdfinfo (<a href="https://poppler.freedesktop.org/">Poppler</a>)</h2> <p>Command:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pdfinfo phantom_modified_xmp.pdf </code></pre></div></div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Title: Boo Producer: PDF Phantom CreationDate: Tue Oct 29 14:43:30 2024 CET Tagged: no UserProperties: no Suspects: no Form: none JavaScript: no Pages: 1 Encrypted: no Page size: 595.276 x 841.89 pts (A4) Page rot: 0 File size: 5906 bytes Optimized: no PDF version: 1.4 </code></pre></div></div> <p>Poppler correctly decodes the octal escape sequences (Producer).</p> <h2 id="verapdf"><a href="https://verapdf.org/">VeraPDF</a></h2> <p>Command:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>verapdf --off --extract phantom_modified_xmp.pdf </code></pre></div></div> <p>The result includes:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;informationDict&gt;</span> <span class="nt">&lt;entry</span> <span class="na">key=</span><span class="s">"Title"</span><span class="nt">&gt;</span>Boo<span class="nt">&lt;/entry&gt;</span> <span class="nt">&lt;entry</span> <span class="na">key=</span><span class="s">"Producer"</span><span class="nt">&gt;</span>PDF Phantom#x000000<span class="nt">&lt;/entry&gt;</span> <span class="nt">&lt;entry</span> <span class="na">key=</span><span class="s">"CreationDate"</span><span class="nt">&gt;</span>2024-10-29T13:43:30.000Z<span class="nt">&lt;/entry&gt;</span> <span class="nt">&lt;/informationDict&gt;</span> </code></pre></div></div> <p>VeraPDF correctly decodes the octal escape sequences. Note the null character at the end (which is actually part of the object string).</p> <h2 id="apache-tika"><a href="https://tika.apache.org/">Apache Tika</a></h2> <p>Command:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java -jar ~/tika/tika-app-2.9.2.jar phantom_modified_xmp.pdf </code></pre></div></div> <p>Result:</p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">&lt;?xml version="1.0" encoding="UTF-8"?&gt;</span><span class="nt">&lt;html</span> <span class="na">xmlns=</span><span class="s">"http://www.w3.org/1999/xhtml"</span><span class="nt">&gt;</span> <span class="nt">&lt;head&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"pdf:PDFVersion"</span> <span class="na">content=</span><span class="s">"1.4"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"pdf:docinfo:title"</span> <span class="na">content=</span><span class="s">"Boo"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"pdf:hasXFA"</span> <span class="na">content=</span><span class="s">"false"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"access_permission:modify_annotations"</span> <span class="na">content=</span><span class="s">"true"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"access_permission:can_print_degraded"</span> <span class="na">content=</span><span class="s">"true"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"dcterms:created"</span> <span class="na">content=</span><span class="s">"2024-10-29T13:43:30Z"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"dc:format"</span> <span class="na">content=</span><span class="s">"application/pdf; version=1.4"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"access_permission:fill_in_form"</span> <span class="na">content=</span><span class="s">"true"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"pdf:hasCollection"</span> <span class="na">content=</span><span class="s">"false"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"pdf:encrypted"</span> <span class="na">content=</span><span class="s">"false"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"dc:title"</span> <span class="na">content=</span><span class="s">"Boo"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"Content-Length"</span> <span class="na">content=</span><span class="s">"5906"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"pdf:hasMarkedContent"</span> <span class="na">content=</span><span class="s">"false"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"Content-Type"</span> <span class="na">content=</span><span class="s">"application/pdf"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"pdf:producer"</span> <span class="na">content=</span><span class="s">"OPF Phantom"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"access_permission:extract_for_accessibility"</span> <span class="na">content=</span><span class="s">"true"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"access_permission:assemble_document"</span> <span class="na">content=</span><span class="s">"true"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"xmpTPg:NPages"</span> <span class="na">content=</span><span class="s">"1"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"resourceName"</span> <span class="na">content=</span><span class="s">"phantom_modified_xmp.pdf"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"pdf:hasXMP"</span> <span class="na">content=</span><span class="s">"true"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"access_permission:extract_content"</span> <span class="na">content=</span><span class="s">"true"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"access_permission:can_print"</span> <span class="na">content=</span><span class="s">"true"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"X-TIKA:Parsed-By"</span> <span class="na">content=</span><span class="s">"org.apache.tika.parser.DefaultParser"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"X-TIKA:Parsed-By"</span> <span class="na">content=</span><span class="s">"org.apache.tika.parser.pdf.PDFParser"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"access_permission:can_modify"</span> <span class="na">content=</span><span class="s">"true"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"pdf:docinfo:producer"</span> <span class="na">content=</span><span class="s">"PDF Phantom�"</span><span class="nt">/&gt;</span> <span class="nt">&lt;meta</span> <span class="na">name=</span><span class="s">"pdf:docinfo:created"</span> <span class="na">content=</span><span class="s">"2024-10-29T13:43:30Z"</span><span class="nt">/&gt;</span> <span class="nt">&lt;title&gt;</span>Boo<span class="nt">&lt;/title&gt;</span> <span class="nt">&lt;/head&gt;</span> <span class="nt">&lt;body&gt;&lt;div</span> <span class="na">class=</span><span class="s">"page"</span><span class="nt">&gt;&lt;p/&gt;</span> <span class="nt">&lt;/div&gt;</span> <span class="nt">&lt;/body&gt;&lt;/html&gt;</span> </code></pre></div></div> <p>Tika correctly decodes the octal escape sequences (field pdf:docinfo:producer); like VeraPDF it shows the null character at the end of the string. Note that Tika also reports the XMP Producer value (field pdf:producer).</p> <h2 id="qpdf"><a href="https://qpdf.sourceforge.io/">Qpdf</a></h2> <p>Command:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>qpdf --json phantom_modified_xmp.pdf </code></pre></div></div> <p>Output contains:</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nl">"9 0 R"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"/CreationDate"</span><span class="p">:</span><span class="w"> </span><span class="s2">"D:20241029134330Z00'00'"</span><span class="p">,</span><span class="w"> </span><span class="nl">"/Producer"</span><span class="p">:</span><span class="w"> </span><span class="s2">"PDF Phantom</span><span class="se">\u</span><span class="s2">0000"</span><span class="p">,</span><span class="w"> </span><span class="nl">"/Title"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Boo"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <p>Qpdf correctly decodes the octal escape sequences, including the trailing null character.</p> <h2 id="pdftk"><a href="https://www.pdflabs.com/tools/pdftk-server/">Pdftk</a></h2> <p>Command:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pdftk phantom_modified_xmp.pdf dump_data </code></pre></div></div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>InfoBegin InfoKey: CreationDate InfoValue: D:20241029134330Z00&amp;apos;00&amp;apos; InfoBegin InfoKey: Producer InfoValue: PDF Phantom InfoBegin InfoKey: Title InfoValue: Boo PdfID0: 71a810587639eb130aefddee35e3c49d PdfID1: 71a810587639eb130aefddee35e3c49d NumberOfPages: 1 PageMediaBegin PageMediaNumber: 1 PageMediaRotation: 0 PageMediaRect: 0 0 595.276 841.89 PageMediaDimensions: 595.276 841.89 </code></pre></div></div> <p>Pdftk correctly decodes the octal escape sequences.</p> <h2 id="pymupdf"><a href="https://pymupdf.readthedocs.io/">PyMuPDF</a></h2> <p>I tested PyMuPDF with this simple test script:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="n">pprint</span> <span class="kn">import</span> <span class="n">pymupdf</span> <span class="n">myPDF</span> <span class="o">=</span> <span class="sh">"</span><span class="s">phantom_modified_xmp.pdf</span><span class="sh">"</span> <span class="n">doc</span> <span class="o">=</span> <span class="n">pymupdf</span><span class="p">.</span><span class="nf">open</span><span class="p">(</span><span class="n">myPDF</span><span class="p">)</span> <span class="n">metadata</span> <span class="o">=</span> <span class="n">doc</span><span class="p">.</span><span class="n">metadata</span> <span class="n">pprint</span><span class="p">.</span><span class="nf">pp</span><span class="p">(</span><span class="n">metadata</span><span class="p">)</span> </code></pre></div></div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{'format': 'PDF 1.4', 'title': 'Boo', 'author': '', 'subject': '', 'keywords': '', 'creator': '', 'producer': 'PDF Phantom', 'creationDate': "D:20241029134330Z00'00'", 'modDate': '', 'trapped': '', 'encryption': None} </code></pre></div></div> <p>PyMUPDF correctly decodes the octal escape sequences.</p> <h2 id="conclusion">Conclusion</h2> <p>All the above tools and libraries were able to decode the octal escape sequences without any problems. So JHOVE’s behaviour really seems to be the exception here, rather than the rule. Octal escape sequences are both allowed by the standard, and widely supported by PDF processing tools. Concluding, I would argue that they don’t pose a significant (if at all) preservation risk.</p> <h2 id="acknowledgments">Acknowledgments</h2> <p>Thanks are due to Peter Wyatt (PDF Association) for his helpful comments in <a href="https://github.com/openpreserve/jhove/issues/927">this Github thread</a>.</p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>Actually they refer to this as a “dual encoding”, but this term is confusing because the octal escape sequences aren’t really an “encoding” at all, but rather a part of the lexical definition of PDF literal string objects. See <a href="https://github.com/openpreserve/jhove/issues/927#issuecomment-2466229030">this follow-up comment by Peter Wyatt</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2024/11/14/escape-from-the-phantom-of-the-pdf https://bitsgalore.org/2024/11/14/escape-from-the-phantom-of-the-pdf 2024-11-14T00:00:00+01:00 <figure class="image"> <img src="/images/2024/10/quality-sign.jpg" alt="Photograph of faded sign on building front showing the word 'Quality'." /> <figcaption>Adapted from <a href="https://www.flickr.com/photos/120143184@N05/47939868992/">Quality Coal</a> by <a href="https://www.flickr.com/photos/120143184@N05/">Greenville Daily Photo</a>. Used under <a href="https://creativecommons.org/publicdomain/zero/1.0/">CC0 1.0. license</a>.</figcaption> </figure> <p>In my <a href="/2024/10/23/jpeg-quality-estimation-experiments-with-a-modified-imagemagick-heuristic">previous post</a> I addressed several problems I ran into when I tried to estimate the “last saved” quality level of JPEG images. It described some experiments based on <a href="https://imagemagick.org/">ImageMagick</a>’s quality heuristic, which led to a <a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/jpegquality-im-modified.py">Python implementation of a modified version of the heuristic</a> that improves the behaviour for images with a quality of 50% or less.</p> <p>I still wasn’t entirely happy with this solution. This was partially because ImageMagick’s heuristic uses <em>aggregated</em> coefficients of the image’s quantization tables, which makes it potentially vulnerable to collisions. Another concern was, that the reasoning behind certain details of ImageMagick’s heuristic seems rather opaque (at least to me!).</p> <p>In this post I explore a different approach to JPEG quality estimation, which is based on a straightforward comparison with “standard” JPEG quantization tables using least squares matching. I also propose a measure that characterizes how similar an image’s quantization tables are to its closest “standard” tables. This could be useful as a measure of confidence in the quality estimate. I present some tests where I compare the results of the least squares matching method with those of the ImageMagick heuristics. I also discuss the results of a simple sensitivity analysis.</p> <!-- more --> <h2 id="jpeg-quality-and-standard-quantization-tables">JPEG quality and standard quantization tables</h2> <p>ImageMagick’s JPEG quality heuristic is based on the “standard” quantization tables that are defined in Annex K of <a href="http://www.w3.org/Graphics/JPEG/itu-t81.pdf">the JPEG standard</a>. Its overall objective seems to be to match an image’s quantization tables with the most similar “standard” quantization tables. Since the quality level of each of these “standard” tables is known, this then provides the quality estimate.</p> <p>ImageMagick’s heuristic does this in an indirect (and to me somewhat opaque) way, possibly to avoid computational cost. This post explores a more straightforward approach, which simply compares the coefficients in an image’s quantization tables against the corresponding coefficients in the “standard” tables, and then returns the quality level of the best match<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>.</p> <h2 id="scaling-of-standard-tables-to-quality-levels">Scaling of standard tables to quality levels</h2> <p>To understand how this works, it’s first important to know that Annex K in the <a href="http://www.w3.org/Graphics/JPEG/itu-t81.pdf">JPEG standard</a> describes two “standard” “quantization tables for luminance and chrominance. Here’s the luminance table:</p> <table> <tbody> <tr> <td>16</td> <td>11</td> <td>10</td> <td>16</td> <td>24</td> <td>40</td> <td>51</td> <td>61</td> </tr> <tr> <td>12</td> <td>12</td> <td>14</td> <td>19</td> <td>26</td> <td>58</td> <td>60</td> <td>55</td> </tr> <tr> <td>14</td> <td>13</td> <td>16</td> <td>24</td> <td>40</td> <td>57</td> <td>69</td> <td>56</td> </tr> <tr> <td>14</td> <td>17</td> <td>22</td> <td>29</td> <td>51</td> <td>87</td> <td>80</td> <td>62</td> </tr> <tr> <td>18</td> <td>22</td> <td>37</td> <td>56</td> <td>68</td> <td>109</td> <td>103</td> <td>77</td> </tr> <tr> <td>24</td> <td>35</td> <td>55</td> <td>64</td> <td>81</td> <td>104</td> <td>113</td> <td>92</td> </tr> <tr> <td>49</td> <td>64</td> <td>78</td> <td>87</td> <td>103</td> <td>121</td> <td>120</td> <td>101</td> </tr> <tr> <td>72</td> <td>92</td> <td>95</td> <td>98</td> <td>112</td> <td>100</td> <td>103</td> <td>99</td> </tr> </tbody> </table> <p>This is the base table with coefficients that are valid for quality level 50. The tables for all other quality levels can be derived from this base table using Equations 1 and 2 from <a href="https://www.sciencedirect.com/science/article/pii/S1742287608000285">this paper by Kornblum (2008)</a><sup id="fnref:12"><a href="#fn:12" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>. First, for each quality level <em>Q</em> we can calculate a corresponding scaling factor <em>S</em>:</p> <math xmlns="http://www.w3.org/1998/Math/MathML"> <mrow> <mi>S</mi> <mo>=</mo> <mfrac> <mrow> <mn>5000</mn> </mrow> <mrow> <mi>Q</mi> </mrow> </mfrac> <mspace width="6em" /> <mo>(</mo> <mi>Q</mi> <mo>&lt;</mo> <mn>50</mn> <mo>)</mo> </mrow> </math> <math xmlns="http://www.w3.org/1998/Math/MathML"> <mrow> <mi>S</mi> <mo>=</mo> <mn>200</mn> <mo>&middot;</mo> <mi>Q</mi> <mspace width="5em" /> <mo>(</mo> <mi>Q</mi> <mo>&ge;</mo> <mn>50</mn> <mo>)</mo> </mrow> </math> <p><em>S</em> is then used to calculate scaled quantization coefficients <em>Tⁱ_s</em> from the base coefficients <em>Tⁱ_b</em> using the following equation:</p> <math xmlns="http://www.w3.org/1998/Math/MathML"> <mrow> <msubsup> <mi>T</mi> <mi>s</mi> <mi>i</mi> </msubsup> <mo>=</mo> <mo>max</mo> <mo>(</mo> <mo>&#x230A;</mo> <mfrac> <mrow> <mi>S</mi> <mo>&middot;</mo> <msubsup> <mi>T</mi> <mi>b</mi> <mi>i</mi> </msubsup> <mo>+</mo> <mn>50</mn> </mrow> <mrow> <mn>100</mn> </mrow> </mfrac> <mo>&#x230B;</mo> <mo>,</mo> <mn>1</mn> <mo>)</mo> </mrow> </math> <p>Here <em>i</em> is the <em>i</em>th element in the table. Note the <a href="https://en.wikipedia.org/wiki/Floor_and_ceiling_functions">floor brackets</a>, which mean the expression inside them is rounded down to the nearest integer number. For 8-bit quantization tables (which is the most common situation) the scaled coefficients also need to be capped at a maximum of 255.</p> <p>As an example, applying these equations to <em>Q=75</em> results in a scaling factor <em>S</em> of 15000, and the quantization coefficients become:</p> <table> <tbody> <tr> <td>8</td> <td>6</td> <td>5</td> <td>8</td> <td>12</td> <td>20</td> <td>26</td> <td>31</td> </tr> <tr> <td>6</td> <td>6</td> <td>7</td> <td>10</td> <td>13</td> <td>29</td> <td>30</td> <td>28</td> </tr> <tr> <td>7</td> <td>7</td> <td>8</td> <td>12</td> <td>20</td> <td>29</td> <td>35</td> <td>28</td> </tr> <tr> <td>7</td> <td>9</td> <td>11</td> <td>15</td> <td>26</td> <td>44</td> <td>40</td> <td>31</td> </tr> <tr> <td>9</td> <td>11</td> <td>19</td> <td>28</td> <td>34</td> <td>55</td> <td>52</td> <td>39</td> </tr> <tr> <td>12</td> <td>18</td> <td>28</td> <td>32</td> <td>41</td> <td>52</td> <td>57</td> <td>46</td> </tr> <tr> <td>25</td> <td>32</td> <td>39</td> <td>44</td> <td>52</td> <td>61</td> <td>60</td> <td>51</td> </tr> <tr> <td>36</td> <td>46</td> <td>48</td> <td>49</td> <td>56</td> <td>50</td> <td>52</td> <td>50</td> </tr> </tbody> </table> <p>This way it is possible to calculate the quantization tables for all 100 quality levels. For the chrominance tables the same procedure can be used.</p> <h2 id="estimating-quality-from-scaled-tables">Estimating quality from scaled tables</h2> <p>For a given JPEG file, the quality can then be estimated by comparing its quantization tables against each of the scaled tables that are derived from the standard tables. As a basis for this comparison we can calculate, for each quality level, the sum of squared errors between the coefficients in the image’s quantization table and the corresponding coefficients in the scaled standard table. For an image with 2 quantization tables (one for luminance and another one for chrominance) this is given by:</p> <math xmlns="http://www.w3.org/1998/Math/MathML"> <mrow> <mi>SSE</mi> <mo>=</mo> <munderover> <mo>&sum;</mo> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mn>64</mn> </munderover> <msup> <mrow> <mo>(</mo> <msubsup> <mi>T</mi> <mi>lum</mi> <mi>i</mi> </msubsup> <mo>-</mo> <msubsup> <mi>T</mi> <mrow> <mi>s</mi> <mo>,</mo> <mi>lum</mi> </mrow> <mi>i</mi> </msubsup> <mo>)</mo> </mrow> <mn>2</mn> </msup> <mo>+</mo> <msup> <mrow> <mo>(</mo> <msubsup> <mi>T</mi> <mi>chrom</mi> <mi>i</mi> </msubsup> <mo>-</mo> <msubsup> <mi>T</mi> <mrow> <mi>s</mi> <mo>,</mo> <mi>chrom</mi> </mrow> <mi>i</mi> </msubsup> <mo>)</mo> </mrow> <mn>2</mn> </msup> </mrow> </math> <p>Here, <em>Tⁱ_lum</em> and <em>Tⁱ_chrom</em> represent the coefficients for luminance and chrominance from the image’s quantization table, and <em>Tⁱ_s,lum</em> and <em>Tⁱ_{s, chrom}</em> are the corresponding coefficients from the (scaled) standard tables.</p> <p>Repeating this for all quality levels results in 100 <em>SSE</em> values. The quality level with the smallest <em>SSE</em> value is then our best estimate for the quality of the image. An <em>SSE</em> value of exactly 0 means the image uses the standard JPEG quantization tables. In that case we can be confident that our estimate is the exact quality level at which the image was compressed. Larger values indicate the use of non-standard quantization tables, in which case the quality estimate may be less accurate.</p> <h2 id="characterizing-similarity-to-standard-tables">Characterizing similarity to standard tables</h2> <p>For images that <em>don’t</em> use the standard quantization tables, it would be useful to have some measure that expresses <em>how much</em> the quantization tables deviate from the best matching standard table. This could be used as a measure of confidence in the quality estimate. By itself, <em>SSE</em> is not a good measure of this. Firstly, it is influenced by the number of quantization tables in the image. We could remove this influence by transforming the <em>SSE</em> value to a root mean squared error (<em>RMSE</em>):</p> <math xmlns="http://www.w3.org/1998/Math/MathML"> <mrow> <mi>RMSE</mi> <mo>=</mo> <msqrt> <mfrac> <mrow> <mi>SSE</mi> </mrow> <mrow> <mi>tables</mi> <mo>&middot;</mo> <mn>64</mn> </mrow> </mfrac> </msqrt> </mrow> </math> <p>Here, <em>tables</em> is the number of quantization tables (which is either 1 or 2). The interpretation of these <em>RMSE</em> values is still complicated by the fact that the quantization coefficients are significantly larger at lower quality levels. In practice this has the effect that the <em>RMSE</em> values are generally much larger at low quality levels relative to higher quality levels, even for images with a similar overall “fit” to a standard quantization table.</p> <p>One “goodness of fit” measure that does not have this drawback is the <a href="https://en.wikipedia.org/wiki/Nash%E2%80%93Sutcliffe_model_efficiency_coefficient">Nash–Sutcliffe efficiency coefficient (<em>NSE</em>)</a>. It is mostly used in the field of hydrology to characterize how well the output of hydrological simulation models agrees with observations. Here, we will use it to characterize how well the coefficients in the standard table agree with those in the image’s quantization table. Rewritten for our quantization coefficients, <em>NSE</em> is given by:</p> <math xmlns="http://www.w3.org/1998/Math/MathML"> <mrow> <mtext>NSE</mtext> <mo>=</mo> </mrow> <mrow> <mn>1</mn> <mo>−</mo> </mrow> <mrow> <mfrac> <mrow> <msubsup> <mo stretchy="true">∑</mo> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>N</mi> </msubsup> <msup> <mrow> <mo fence="true" form="prefix">(</mo> <msup> <mi>T</mi> <mi>i</mi> </msup> <mo>−</mo> <msubsup> <mi>T</mi> <mi>s</mi> <mi>i</mi> </msubsup> <mo fence="true" form="postfix">)</mo> </mrow> <mn>2</mn> </msup> </mrow> <mrow> <msubsup> <mo stretchy="true">∑</mo> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>N</mi> </msubsup> <msup> <mrow> <mo fence="true" form="prefix">(</mo> <msup> <mi>T</mi> <mi>i</mi> </msup> <mo>−</mo> <menclose notation="top" class="tml-overline"> <mi>T</mi> </menclose> <mo fence="true" form="postfix">)</mo> </mrow> <mn>2</mn> </msup> </mrow> </mfrac> </mrow> </math> <p>Here <em>Tⁱ</em> represents the <em>i</em>th coefficient from the image’s quantization tables, and <em>Tⁱ_s</em> is the corresponding coefficient from the (scaled) standard tables. <em>N</em> is the total number of coefficients in the image’s quantization tables. Note that, unlike in the <em>SSE</em> equation, the luminance and chrominance coefficients are lumped here for simplicity. Finally, <span style="border-top: 1px solid #000000;"><em>T</em></span> is the mean of all coefficients <em>Tⁱ</em> in the image’s quantization tables. The interpretation of <em>NSE</em> is quite straightforward:</p> <ul> <li>A value of 1 indicates a perfect agreement between the image quantization tables and the corresponding standard tables.</li> <li>For a value of 0, the standard tables are as good (or rather, bad) an approximation of the image’s quantization tables as <span style="border-top: 1px solid #000000;"><em>T</em></span>.</li> <li>Negative values indicate an extremely poor agreement.</li> </ul> <p>As an example, the below scatter plot shows the quantization coefficients (<em>T</em>) from <a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/dbnl/mul-master.jpg">one of out dbnl master images</a>, plotted against the corresponding coefficients (<em>T_s</em>) from the best matching standard table. It also shows the line of perfect agreement (green, dashed), the quality estimate, the root mean squared error, and the Nash-Sutcliffe Efficiency:</p> <figure class="image"> <img src="/images/2024/10/mul-master-scatter.png" alt="scatter plot of T against Ts for file mul-master.jpg, with Quality = 84%, RMSE = 1.057 and NSE = 0.997." /> </figure> <p>The plot shows that, aside from one outlier in the chrominance table, the image’s quantization coefficients are closely approximated by those in the standard quantization tables. This is reflected by the <em>NSE</em> value, which is close to 1. Now compare this with the plot I made for <a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/dbnl/mul-access.jpg">the corresponding access image</a>:</p> <figure class="image"> <img src="/images/2024/10/mul-access-scatter.png" alt="scatter plot of T against Ts for file mul-access.jpg, with Quality = 18%, RMSE = 6.244 and NSE = 0.999." /> </figure> <p>Visually, the agreement between the standard quantization table coefficients and those of the image looks very similar to the previous plot. But note how the <em>RMSE</em> value is much larger here, which is caused by the much larger overall coefficients in the quantization tables. However, this doesn’t have any effect on <em>NSE</em>, which is even marginally closer to 1 than for the master image.</p> <p>By contrast, things are very different for <a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/image-177.jpg">this image</a>:</p> <figure class="image"> <img src="/images/2024/10/image-177-scatter.png" alt="scatter plot of T against Ts for file image-177.jpg, with Quality = 81%, RMSE = 20.509 and NSE = 0.732." /> </figure> <p>The plot shows that the standard JPEG tables are a relatively poor approximation here, and this is refected by the low <em>NSE</em> value. A similar case is <a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/sample-jpg-files-sample-4.jpg">this image</a>:</p> <figure class="image"> <img src="/images/2024/10/sample-jpg-files-sample-4-scatter.png" alt="scatter plot of T against Ts for file sample-jpg-files-sample-4.jpg, with Quality = 89%, RMSE = 12.294 and NSE = 0.734." /> </figure> <p>Despite the different quality estimate and <em>RMSE</em> value, this visually looks like it’s in the same ballpark as the previous image, and the similar <em>NSE</em> value confirms this.</p> <p>Based on these examples, <em>NSE</em> appears to give a good indication of the fit between the image and standard quantization coefficients. This makes it useful as a measure to asses the confidence in the method’s quality estimates.</p> <h2 id="python-implementation-of-least-squares-matching-method">Python implementation of least squares matching method</h2> <p>I created <a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/jpegquality-lsm.py">a test script with a Python implementation</a> of the method. Apart from the quality estimate, it also reports the corresponding <em>RMSE</em> and <em>NSE</em> values.</p> <h2 id="tests-with-pillow-and-imagemagick-jpegs">Tests with Pillow and ImageMagick JPEGs</h2> <p>As a first test I ran the script on the <a href="https://github.com/KBNLresearch/jpeg-quality-demo/tree/main/images/im_pil">Pillow and ImageMagick JPEGs</a> I discussed in my previous post. This gave the following result:</p> <table> <thead> <tr> <th style="text-align: left">Q_enc</th> <th style="text-align: left">Q_est(PIL)</th> <th style="text-align: left">RMSE(PIL)</th> <th style="text-align: left">NSE(PIL)</th> <th style="text-align: left">Q_est(IM)</th> <th style="text-align: left">RMSE(IM)</th> <th style="text-align: left">NSE(IM)</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">5</td> <td style="text-align: left">5</td> <td style="text-align: left">0.0</td> <td style="text-align: left">1.0</td> <td style="text-align: left">5</td> <td style="text-align: left">0.0</td> <td style="text-align: left">1.0</td> </tr> <tr> <td style="text-align: left">10</td> <td style="text-align: left">10</td> <td style="text-align: left">0.0</td> <td style="text-align: left">1.0</td> <td style="text-align: left">10</td> <td style="text-align: left">0.0</td> <td style="text-align: left">1.0</td> </tr> <tr> <td style="text-align: left">25</td> <td style="text-align: left">25</td> <td style="text-align: left">0.0</td> <td style="text-align: left">1.0</td> <td style="text-align: left">25</td> <td style="text-align: left">0.0</td> <td style="text-align: left">1.0</td> </tr> <tr> <td style="text-align: left">50</td> <td style="text-align: left">50</td> <td style="text-align: left">0.0</td> <td style="text-align: left">1.0</td> <td style="text-align: left">50</td> <td style="text-align: left">0.0</td> <td style="text-align: left">1.0</td> </tr> <tr> <td style="text-align: left">75</td> <td style="text-align: left">75</td> <td style="text-align: left">0.0</td> <td style="text-align: left">1.0</td> <td style="text-align: left">75</td> <td style="text-align: left">0.0</td> <td style="text-align: left">1.0</td> </tr> <tr> <td style="text-align: left">100</td> <td style="text-align: left">100</td> <td style="text-align: left">0.0</td> <td style="text-align: left">1.0</td> <td style="text-align: left">100</td> <td style="text-align: left">0.0</td> <td style="text-align: left">1.0</td> </tr> </tbody> </table> <p>Here <em>Q_enc</em> is the encoding quality, and <em>Q_est(PIL)</em> and <em>Q_est(IM)</em> are the script’s estimates for the Pillow and ImageMagick images, respectively. The script correctly reproduced the encoding quality for all test images. The values <em>RMSE</em>=0 and <em>NSE</em>=1 also indicate that all images use the standard JPEG quantization tables.</p> <h2 id="comparison-of-quality-estimation-methods">Comparison of quality estimation methods</h2> <p>Far more interesting is the behaviour for images that <em>don’t</em> use the standard tables. The previous section already showed the results for the <a href="https://github.com/KBNLresearch/jpeg-quality-demo/tree/main/images/dbnl">dbnl master and access JPEGs</a> that started this work. To test the method on a more diverse selection of images, I downloaded JPEGs from a variety of sources<sup id="fnref:9"><a href="#fn:9" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>. I then ran them through <a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/jpegquality-compare.py">this test script</a> that estimates the JPEG quality using my Python port of the original ImageMagick heuristic, the modified ImageMagick heuristic from my <a href="/2024/10/23/jpeg-quality-estimation-experiments-with-a-modified-imagemagick-heuristic">previous post</a>, and the least squares matching method. I then identified images for which one or more of these methods came up with different results, and added a selection of them to <a href="https://github.com/KBNLresearch/jpeg-quality-demo/tree/main/images/misc">this dataset</a>. The following table shows the results of the script for this dataset:</p> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">Q<br />(im)</th> <th style="text-align: left">Q<br />(im, mod)</th> <th style="text-align: left">Exact<br />(im, mod)</th> <th style="text-align: left">Q<br />(lsm)</th> <th style="text-align: left">RMSE<br />(lsm)</th> <th style="text-align: left">NSE<br />(lsm)</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/psgradient.jpg">psgradient.jpg</a></td> <td style="text-align: left">92</td> <td style="text-align: left">92</td> <td style="text-align: left">False</td> <td style="text-align: left">93</td> <td style="text-align: left">4.438</td> <td style="text-align: left">0.747</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/hopper_16bit_qtables.jpg">hopper_16bit_qtables.jpg</a></td> <td style="text-align: left">na</td> <td style="text-align: left">1</td> <td style="text-align: left">False</td> <td style="text-align: left">13</td> <td style="text-align: left">0.795</td> <td style="text-align: left">1.0</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/image-177.jpg">image-177.jpg</a></td> <td style="text-align: left">63</td> <td style="text-align: left">63</td> <td style="text-align: left">True</td> <td style="text-align: left">81</td> <td style="text-align: left">20.509</td> <td style="text-align: left">0.732</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/image-98.jpg">image-98.jpg</a></td> <td style="text-align: left">63</td> <td style="text-align: left">63</td> <td style="text-align: left">True</td> <td style="text-align: left">81</td> <td style="text-align: left">20.509</td> <td style="text-align: left">0.732</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/jpeg420exif.jpg">jpeg420exif.jpg</a></td> <td style="text-align: left">90</td> <td style="text-align: left">90</td> <td style="text-align: left">False</td> <td style="text-align: left">89</td> <td style="text-align: left">0.424</td> <td style="text-align: left">0.999</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/jpeg422jfif.jpg">jpeg422jfif.jpg</a></td> <td style="text-align: left">96</td> <td style="text-align: left">96</td> <td style="text-align: left">False</td> <td style="text-align: left">97</td> <td style="text-align: left">0.0</td> <td style="text-align: left">1.0</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/jpeg444.jpg">jpeg444.jpg</a></td> <td style="text-align: left">60</td> <td style="text-align: left">60</td> <td style="text-align: left">False</td> <td style="text-align: left">75</td> <td style="text-align: left">0.0</td> <td style="text-align: left">1.0</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/sample-birch-400x300.jpg">sample-birch-400x300.jpg</a></td> <td style="text-align: left">95</td> <td style="text-align: left">95</td> <td style="text-align: left">False</td> <td style="text-align: left">94</td> <td style="text-align: left">3.188</td> <td style="text-align: left">0.838</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/sample-jpg-files-sample-4.jpg">sample-jpg-files-sample-4.jpg</a></td> <td style="text-align: left">78</td> <td style="text-align: left">78</td> <td style="text-align: left">True</td> <td style="text-align: left">89</td> <td style="text-align: left">12.294</td> <td style="text-align: left">0.734</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/tapedeck1.jpg">tapedeck1.jpg</a></td> <td style="text-align: left">90</td> <td style="text-align: left">90</td> <td style="text-align: left">False</td> <td style="text-align: left">93</td> <td style="text-align: left">0.753</td> <td style="text-align: left">0.992</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/tapedeck2.jpg">tapedeck2.jpg</a></td> <td style="text-align: left">92</td> <td style="text-align: left">92</td> <td style="text-align: left">False</td> <td style="text-align: left">95</td> <td style="text-align: left">0.421</td> <td style="text-align: left">0.996</td> </tr> </tbody> </table> <p>Here <em>Q(im)</em> is the quality estimate from the original ImageMagick heuristic, <em>Q(im), mod</em> is the quality estimate from the modified ImageMagick heuristic, <em>Q(lsm)</em> is the quality estimate from the least squares matching method. In addition <em>Exact(im, mod)</em> is the “exactness” indicator of the modified ImageMagick heuristic, and <em>RMSE(lsm)</em> and <em>NSE(lsm)</em> are the root mean squared error and Nash-Sutcliffe Efficiency values reported by the least squares matching method, respectively.</p> <p>Below I highlight some of the more interesting results.</p> <h3 id="jpeg444jpg"><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/jpeg444.jpg">jpeg444.jpg</a></h3> <p>For this image, both the original and modified ImageMagick heuristics estimate the quality at 60%, with no “exact” match. By contrast, the least squares matching method came up with a quality of 75%, with <em>RMSE</em> and <em>NSE</em> indicating a perfect match with the standard JPEG quantization tables. This is comfirmed by plotting the coefficients from the quantization tables against the standard coefficients:</p> <figure class="image"> <img src="/images/2024/10/jpeg444-scatter.png" alt="scatter plot of T against Ts for file jpeg444.jpg, with Quality = 75%, RMSE = 0.0 and NSE = 1.0." /> </figure> <p>I double-checked the result by uploading the image to FotoForensics, which <a href="https://fotoforensics.com/analysis.php?id=2e8c6fc55fefbdf2e3b96e9c531d3d24b7b5ea16.5667">also came up with 75% quality, and an exact match with the standard tables</a>.</p> <h3 id="image-98jpg"><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/image-98.jpg">image-98.jpg</a></h3> <p>Here, the quality is estimated at 63% by both the original and modified ImageMagick heuristics, with an “exact” match. However, the least squares matching method results in a much higher (81%) quality, but the relatively low <em>NSE</em> value of 0.732 indicates a poor fit to the standard JPEG tables. This is confirmed by the scatter plot of <em>T</em> against <em>T_s</em> :</p> <figure class="image"> <img src="/images/2024/10/image-98-scatter.png" alt="scatter plot of T against Ts for file image-98.jpg, with Quality = 81%, RMSE = 25.509 and NSE = 0.732." /> </figure> <p>The <a href="https://fotoforensics.com/analysis.php?id=3f271d3383ea2984b461620f2d54075dc5ec26da.37769">FotoForensics result</a> also indicates a quality of 81%.</p> <h3 id="hopper_16bit_qtablesjpg"><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/hopper_16bit_qtables.jpg">hopper_16bit_qtables.jpg</a></h3> <p>This image is interesting for a number of reasons. ImageMagick’s original heuristic fails to come up with a quality estimate, while the modified ImageMagick heuristic returns a 1% estimate. Meanwhile, the least squares matching method estimates the quality at 13%. Here’s the corresponding scatter plot:</p> <figure class="image"> <img src="/images/2024/10/hopper_16bit_qtables-scatter.png" alt="scatter plot of T against Ts for file hopper_16bit_qtables., with Quality = 13%, RMSE = 0.795 and NSE = 1.0." /> </figure> <p>The coefficients in the JPEG quantization tables are usually stored as 8-bit unsigned integers, which means the highest possible value is 255. This particular image uses 16-bit values instead, which we can see from the range of <em>T</em> values in the plot, which goes all the way up to 380! ImageMagick’s heuristic is unable to deal with this<sup id="fnref:10"><a href="#fn:10" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>, which results (for the modified version) in an unrealistically low value. The least squares matching method explicitly checks for coefficients outside the 8-bit range, and adjusts its calculations accordingly. Its quality estimate corresponds to <a href="https://fotoforensics.com/analysis.php?id=079bf03e3187859bf2bdf0b28c341d3b75ad8442.2044">the assessment by FotoForensics</a>.</p> <p>One detail that caught my attention is the non-zero <em>RMSE</em> value, which at first sight seems at odds with the reported <em>NSE</em> value of 1.0. On closer inspection, it turned out that <em>NSE</em> is actually marginally smaller than 1 here, but this is obscured by rounding the reported values at 3 decimals<sup id="fnref:11"><a href="#fn:11" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>.</p> <h3 id="sample-jpg-files-sample-4jpg"><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/misc/sample-jpg-files-sample-4.jpg">sample-jpg-files-sample-4.jpg</a></h3> <p>Both ImageMagick heuristics estimate the quality of this image at 78% with an “exact” match, whereas the least squares matching method gives a much higher estimate of 89% (with quite a poor fit with the standard tables). The corresponding <a href="https://fotoforensics.com/analysis.php?id=9271e2a81a4105d7bc326c76bac043c6265e4d8e.63379">FotoForensics estimate</a> is marginally different from this at 88%, but still very close. For completeness here’s its scatter plot (again):</p> <figure class="image"> <img src="/images/2024/10/sample-jpg-files-sample-4-scatter.png" alt="scatter plot of T against Ts for file sample-jpg-files-sample-4.jpg, with Quality = 89%, RMSE = 12.294 and NSE = 0.734." /> </figure> <h2 id="conclusions-from-this-comparison">Conclusions from this comparison</h2> <p>Even though the tests presented here are quite limited, the differences that can occur between the quality estimates by the least squares matching method and the ImageMagick heuristics are quite striking. What surprised me in particular, was that even for JPEGs that use the standard quantization tables, ImageMagick’s heuristic may still provide quality estimates that are quite inaccurate. Of course, a major limitation here is the lack of reliable “ground truth” in the form of known quality settings at the time the test images were created. However, the good agreement between the quality estimates of the least squares matching method and the FotoForensics service does inspire some confidence in the methodology.</p> <p>Another surprise was that ImageMagick’s “exactness” flag isn’t actually indicative of an exact match with the standard JPEG tables. None of the test images for which it returned a “True” value actually contains standard quantization tables, whereas the two images that <em>do</em> contain standard tables resulted in a “False” value!</p> <h2 id="sensitivity-analysis">Sensitivity analysis</h2> <p>To get a better impression of how the method behaves with non-standard quantization tables, I did a simple sensitivity analysis using the cjpeg utility that is part of <a href="https://en.wikipedia.org/wiki/Libjpeg">libjpeg</a>. This tool supports compression with separate quality levels for the luminance and chrominance tables. I used this to <a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/generate-testimages-cjpeg.sh">compress one source image to all possible quality level combinations</a>. This resulted in 10,000 images, which I then ran through the least squares matching method.</p> <p>The plot below shows the distribution of estimated quality values <em>Q_lsm</em> against <em>Q_av</em>, which are the corresponding averages of encoding qualities <em>Q_lum</em> and <em>Q_chrom</em><sup id="fnref:15"><a href="#fn:15" class="footnote" rel="footnote" role="doc-noteref">6</a></sup>:</p> <figure class="image"> <img src="/images/2024/10/qav-qlsm.png" alt="scatter plot of average encoding quality versus estimated encoding quality." /> </figure> <p>At low average encoding qualities, the bandwidth of corresponding estimates by the least squares matching method is very wide, but this narrows considerably towards higher quality levels.</p> <p>Another, probably more useful view on the results appears if we plot the quality estimation errors, here expressed as absolute differences between <em>Q_av</em> and <em>Q_lsm</em>, against the Nash-Sutcliffe Efficiency coefficients:</p> <figure class="image"> <img src="/images/2024/10/deltaq-nse.png" alt="scatter plot of prediction errors (absolute differences between estimated and average encoding), versus Nash-Sutcliffe Efficiency." /> </figure> <p>This shows a strong association between <em>NSE</em> values in the upper range with small prediction errors. At lower <em>NSE</em> values, the range of prediction errors becomes progressively wider. Thus, this demonstrates how <em>NSE</em> can be useful as a measure of confidence in the quality estimate.</p> <h2 id="performance">Performance</h2> <p>One potential concern about the least squares matching method might be that it is computationally not very efficient: for each image, the analysis typically involves 200 comparisons of 64-element tables. Out of interest I did a little performance test using <a href="https://github.com/yavuzceliker/sample-images">this collection of 700 JPEGs</a>. I analyzed these files with my scripts with the Python ports of the original and modified ImageMagick heuristics, and the least squares matching method.</p> <p>For each script run, I first ran this command to empty the cache memory:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo sysctl vm.drop_caches=3 </code></pre></div></div> <p>I then ran each script like this:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(time python3 ./jpeg-quality-demo/jpegquality-lsm.py ./sample-images/images/*.jpg &gt; sample-images.txt) 2&gt; time-lsm.txt </code></pre></div></div> <p>The “time” command results in 3 performance metrics, The most important of which are:</p> <ul> <li>“real” - the actual amount of time passed between starting the script and its termination.</li> <li>“user” - actual CPU time used in executing the process<sup id="fnref:8"><a href="#fn:8" class="footnote" rel="footnote" role="doc-noteref">7</a></sup>.</li> </ul> <p>Below table shows these metrics for the three scripts<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">8</a></sup>:</p> <table> <thead> <tr> <th style="text-align: left">Method</th> <th style="text-align: left">time (real)</th> <th style="text-align: left">time (user)</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">im original</td> <td style="text-align: left">0m7,624s</td> <td style="text-align: left">0m3,321s</td> </tr> <tr> <td style="text-align: left">im modified</td> <td style="text-align: left">0m7,215s</td> <td style="text-align: left">0m2,968s</td> </tr> <tr> <td style="text-align: left">lsm</td> <td style="text-align: left">0m13,134s</td> <td style="text-align: left">0m8,673s</td> </tr> </tbody> </table> <p>This shows the least squares matching method is almost 3 times slower than the ImageMagick heuristics in terms of “user” time, and about 2 times slower in terms of “real” time. This translates to an average processing time of 0.01 to 0.02 s per file. Therefore, the reduced performance shouldn’t be any problem in practical terms.</p> <h2 id="final-thoughts">Final thoughts</h2> <p>I originally wrote the least squares matching method code in an attempt to better understand JPEG quality estimation, and to make a more informed assesment of how ImageMagick’s heuristic works. Based on the tests described here, I think I ended up with something that might actually be quite useful, and preferrable to either the original or modified ImageMagick heuristic.</p> <p>By itself the method isn’t in any way novel, as it’s basically just another implementation of the “Approximate Quantization Tables” quality estimation method as described by Neal Krawetz<sup id="fnref:14"><a href="#fn:14" class="footnote" rel="footnote" role="doc-noteref">9</a></sup>. I expect many other, very similar implementations exist that I’m simply not aware of, particularly in the digital forensics domain. This makes it all the more surprising that these apparently haven’t made it to popular image processing and analysis software like ImageMagick. The use of the Nash-Sutcliffe Efficiency as a measure of confidence in the quality estimate <em>may</em> be somewhat novel, but I wouldn’t be surprised if other (and possibly better) methods for this exist.</p> <p>Finally, it’s important to be aware that the characterization of JPEG quality using the 1 - 100 scale that follows from the “standard” quantization tables is by itself pretty arbitrary<sup id="fnref:13"><a href="#fn:13" class="footnote" rel="footnote" role="doc-noteref">10</a></sup>. Essentially, the corresponding “quality” values are just pointers to sets of quantization tables that only have ordinal significance (i.e. higher values mean better quality), but not much more. Due to its wide use, and the lack of any better alternative, it’s still a useful benchmark. This is also why I think it’s important to provide some information on the similarity of an image’s quantization tables to the “standard” ones, as this helps assessing the confidence in the quality estimate.</p> <p>As always, any feedback and suggestions in response to this post are very welcome!</p> <h2 id="scripts-and-test-data">Scripts and test data</h2> <ul> <li><a href="https://github.com/KBNLresearch/jpeg-quality-demo">jpeg-quality-demo Github repository</a> - Github repo with all scripts and test data that were used in this analysis.</li> <li><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/jpegquality-lsm.py">jpegquality-lsm.py</a> - Python implementation of the least squares matching method.</li> </ul> <h3 id="important-note-on-python-pillow-version">Important note on Python Pillow version</h3> <p>The Python implementation of the least squares matching method (and most of the other scripts as well) requires a recent version of the <a href="https://python-pillow.org/">Pillow Imaging Library</a>. This is because around the release of version 8.3 (I think) Pillow changed the order in which it returns the values inside JPEG quantization tables (<a href="https://github.com/python-pillow/Pillow/pull/4989">details here</a>). All scripts in the repo expect the current/new behaviour, and they will give <em>very</em> wrong results when used with older Pillow versions!</p> <h2 id="revision-history">Revision history</h2> <ul> <li>1 November 2024: added paragraph on significance of JPEG quality scale; added sensitivity analysis.</li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:3"> <p>This corresponds to the “Approximate Quantization Tables” method that is mentioned on (and used by) <a href="https://fotoforensics.com/tutorial.php?tt=estq">Neal Krawetz’s FotoForensics site</a> (but the site doesn’t provide any details about the implementation). <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:12"> <p>See also <a href="https://stackoverflow.com/a/29216609/1209004">this post on StackOverflow</a>. <a href="#fnref:12" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:9"> <p>Most notably <a href="https://github.com/yavuzceliker/sample-images">sample-images</a>, <a href="https://samplelib.com/sample-jpeg.html">samplelib.com</a>, <a href="https://toolsfairy.com/tools/image-test/sample-jpg-files">toolsfairy.com</a>, <a href="https://www.w3.org/MarkUp/Test/xhtml-print/20050519/tests/A_2_1-BF-01.htm">w3.org</a> and <a href="https://github.com/python-pillow/Pillow/tree/main/Tests/images">the Pillow source repository</a>. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:10"> <p>This is because its hard-coded <code class="language-plaintext highlighter-rouge">sums</code> and <code class="language-plaintext highlighter-rouge">hashes</code> lists (see my <a href="/2024/10/23/jpeg-quality-estimation-experiments-with-a-modified-imagemagick-heuristic">previous post</a>) are based on 8-bit values. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:11"> <p>In this case the numerator of the <em>NSE</em> equation (the <em>SSE</em> value) was 81, and the denominator (the variance of the quantization coefficients) 7743557. This results in <em>NSE</em> = 1 - (81/7743557) = 0.99998954, which is reported as 1.0 when rounded to 3 decimals. Meanwhile <em>RMSE</em> = √(81/128) = 0.795. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:15"> <p>The actual significance of <em>Q_av</em> is somewhat questionable, especially for extreme high/low combinations of <em>Q_lum</em> and <em>Q_chrom</em>. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:8"> <p>For a more in-depth explanation of these metrics see: <a href="https://stackoverflow.com/a/556411/1209004">https://stackoverflow.com/a/556411/1209004</a> <a href="#fnref:8" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>I ran the test on a pretty low spec machine with an Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz with 4 cores. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:14"> <p>In response to this post, Krawetz <a href="https://noc.social/@hackerfactor/113397249743521959">let me know</a> that FotoForensics uses a different algorithm that isn’t based on least squares. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:13"> <p>See e.g. <a href="http://www.faqs.org/faqs/jpeg-faq/part1/section-5.html">the JPEG FAQ</a> for an explanation. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2024/10/30/jpeg-quality-estimation-using-simple-least-squares-matching-of-quantization-tables https://bitsgalore.org/2024/10/30/jpeg-quality-estimation-using-simple-least-squares-matching-of-quantization-tables 2024-10-30T00:00:00+01:00 <figure class="image"> <img src="/images/2024/10/bailey-1024.jpg" alt="Photograph of golden retriever dog Bailey sitting at a desk in front of a laptop, bashing her paws away at the laptop's keyboard while wearing a necktie." /> <figcaption><a href="https://imgur.com/a/golden-baileys-story-pictures-XGli7">Bailey AKA the "I have no idea what I'm doing" dog</a>. License unknown.</figcaption> </figure> <p>In this post I explore some of the challenges I ran into while trying to estimate the quality level of JPEG images. By quality level I mean the percentage (1-100) that expresses the <a href="https://en.wikipedia.org/wiki/Lossy_compression">lossiness</a> that was applied by the encoder at the last “save” operation. Here, a value of 1 results in very aggressive compression with a lot of information loss (and thus a very low quality), whereas at 100 almost no information loss occurs at all<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>.</p> <p>More specifically, I focus on problems with <a href="https://imagemagick.org/">ImageMagick</a>’s JPEG quality heuristic, which become particularly apparent when applied to low quality images. I also propose a simple tentative solution, that applies some small changes to ImageMagick’s heuristic.</p> <!-- more --> <h2 id="context-of-this-work">Context of this work</h2> <p>I’m currently working on an automated workflow for quality-checking scanned books and periodicals in PDF format. These PDFs are created by external suppliers for The Digital Library for Dutch Literature (<a href="https://www.dbnl.org/">dbnl</a>), which has been managed by the KB since 2015.</p> <p>Each book or periodical volume is scanned to PDF. For each publication, 2 versions are made:</p> <ol> <li>A “production master” PDF with scans that are encoded at 85% JPEG quality.</li> <li>A (relatively) small access PDF with scans encoded at 50% JPEG quality.</li> </ol> <p>It’s important to have some way of verifying the approximate quality of both versions. The production master serves as input for derived EPUB, XML and unformatted text versions. If the scans are compressed too heavily, this adversely affects these derived products. On the other hand, too little compression on the access PDFs will result in files that are impractically large for access.</p> <h2 id="estimating-jpeg-quality">Estimating JPEG quality</h2> <p>Probably the best explainer on JPEG quality and its estimation is <a href="https://fotoforensics.com/tutorial.php?tt=estq">this tutorial on Neal Krawetz’s Fotoforensics site</a>. The information under the “Estimating Quality” tab is particularly useful. I will return to this on various occasions later in this post.</p> <h2 id="estimating-jpeg-quality-with-imagemagick">Estimating JPEG quality with ImageMagick</h2> <p><a href="https://imagemagick.org/">ImageMagick</a> is able to estimate the quality of a JPEG image. For example, let’s create a test image at 70% quality:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>convert -quality 70 wizard: wizard-70.jpg </code></pre></div></div> <p>We can then get an estimate of the JPEG quality using:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>identify -format '%Q\n' wizard-70.jpg </code></pre></div></div> <p>Which results in:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>70 </code></pre></div></div> <p>However, when I ran this command on some of our access scans, the results were not what I expected. An example is <a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/images/dbnl/260761857-bdf17e14-c697-4f4a-b6d5-e3067e0afc08.jpg">this image</a>, which I extracted from one of our PDFs<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>. According to ImageMagick<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>, this image is compressed at 92% quality. This JPEG was extracted from a small access PDF, for which I would expect a quality of 50% or less. Its file size is also much smaller than I would expect for a 92% quality image.</p> <h2 id="check-with-fotoforensics">Check with Fotoforensics</h2> <p>To verify this unexpected result I uploaded the image to Neal Krawetz’s <a href="https://fotoforensics.com/">Fotoforensics</a> service. The <a href="https://fotoforensics.com/analysis.php?id=2e0a9f3203e35ece9a23c68c9e6dc7c908891372.353235&amp;show=estq">result is available here</a>. Fotoforensics estimates the JPEG quality at a paltry 18%. So why does ImageMagick report a value of 92% here?</p> <h2 id="imagemagick-uses-92-as-a-fallback-value">ImageMagick uses “92” as a fallback value</h2> <p>From a cursory look at its source code, it seems that ImageMagick <a href="https://github.com/ImageMagick/ImageMagick/blob/f5bdfdd62af7109ad105f8af4e28111e353edecd/MagickCore/property.c#L2725">uses 92 as a fallback value</a> if it cannot come up with a quality estimate. This makes the interpretation of its quality output needlessly difficult, since it’s impossible to differentiate between images that have a true 92% quality, and images for which the quality cannot be established. I <a href="https://github.com/ImageMagick/ImageMagick6/issues/260">created a ticket</a> on Github when I first came across this issue over a year ago, but thus far it hasn’t been fixed.</p> <p>Even if it was fixed, this still leaves the question <em>why</em> ImageMagick’s quality heuristic is failing here in the first place. As I’m not proficient in <em>C</em>, I didn’t pursue things any further when I first ran into this issue.</p> <h2 id="imagemagicks-jpeg-quality-heuristic">ImageMagick’s JPEG quality heuristic</h2> <p>This changed when I recently came across <a href="https://stackoverflow.com/a/75204019/1209004">this StackOverflow post</a>, which points to <a href="https://gist.github.com/eddy-geek/c0f01dc5401dc50a49a0a821cdc9b3e8#file-jpg_quality_pil_magick-py">a Python port of ImageMagick’s JPEG quality heuristic</a> by one “Edward O”. It is based <a href="https://github.com/ImageMagick/ImageMagick/blob/c6410959676151a94bb1efc32667571dadadd5df/coders/jpeg.c#L866">on ImageMagicks original code</a>, and estimates the JPEG compression quality from the quantization tables. This immediately grabbed my attention, since my quality-checking workflow is also implemented in Python. The ability to estimate JPEG quality natively in Python would remove the need to wrap any external tools for this.</p> <p>A quick test showed that it produced results that were identical to ImageMagick in most cases, but like ImageMagick, it failed to come up with a quality estimate for my problematic JPEG. To find out why, I incorporated Edwards’s code into <a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/jpegquality-im-original.py">this test script</a>.</p> <h2 id="how-it-works">How it works</h2> <p>The algorithm reads the image’s quantization tables (usually 2), each of which is a list of 64 quantization coefficients. It first adds up all of these numbers, resulting in variable <code class="language-plaintext highlighter-rouge">qsum</code>. It then calculates <code class="language-plaintext highlighter-rouge">qvalue</code>, which is the sum of the quantization coefficients at 2 specific positions in each table (these are then summed for all quantization tables). The main “meat and potatoes” of the heuristic is this loop at the very end of the function:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nf">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="nf">if </span><span class="p">((</span><span class="n">qvalue</span> <span class="o">&lt;</span> <span class="n">hashes</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="ow">and</span> <span class="p">(</span><span class="n">qsum</span> <span class="o">&lt;</span> <span class="n">sums</span><span class="p">[</span><span class="n">i</span><span class="p">])):</span> <span class="k">continue</span> <span class="nf">if </span><span class="p">(((</span><span class="n">qvalue</span> <span class="o">&lt;=</span> <span class="n">hashes</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="ow">and</span> <span class="p">(</span><span class="n">qsum</span> <span class="o">&lt;=</span> <span class="n">sums</span><span class="p">[</span><span class="n">i</span><span class="p">]))</span> <span class="ow">or</span> <span class="p">(</span><span class="n">i</span> <span class="o">&gt;=</span> <span class="mi">50</span><span class="p">)):</span> <span class="k">return</span> <span class="n">i</span><span class="o">+</span><span class="mi">1</span> </code></pre></div></div> <p>For each iteration, <code class="language-plaintext highlighter-rouge">qvalue</code> and <code class="language-plaintext highlighter-rouge">qsum</code> are evaluated against the corresponding values in two hard-coded numerical lists (<code class="language-plaintext highlighter-rouge">hashes[i]</code> and <code class="language-plaintext highlighter-rouge">sums[i]</code>). Although I couldn’t find any documentation, a little digging showed that the values in the <code class="language-plaintext highlighter-rouge">sums</code> and <code class="language-plaintext highlighter-rouge">hashes</code> lists are derived from the “standard” quantization tables defined in Annex K of <a href="http://www.w3.org/Graphics/JPEG/itu-t81.pdf">the JPEG standard</a><sup id="fnref:7"><a href="#fn:7" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>. The first <em>if</em> block makes sure that as long as <code class="language-plaintext highlighter-rouge">qvalue</code> is smaller than <code class="language-plaintext highlighter-rouge">hashes[i]</code> <em>and</em> <code class="language-plaintext highlighter-rouge">qsum</code> is smaller than <code class="language-plaintext highlighter-rouge">sums[i]</code>, the code will immediately jump to the next iteration, skipping the second <em>if</em> block. The second <em>if</em> block (which reports the quality estimate as <code class="language-plaintext highlighter-rouge">i+1</code>) is only evaluated if the test condition in the first block fails.</p> <h2 id="tracing-all-loop-variables">Tracing all loop variables</h2> <p>To get a better impression of why the heuristic cannot come up with a meaningful quality estimate for my problematic JPEG, I added a line of code that prints out all variables at the start of each iteration. This gave the following output:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>i: 0, qvalue: 586, hashes[i]:1020, qsum:24028, sums[i]32640 i: 1, qvalue: 586, hashes[i]:1015, qsum:24028, sums[i]32635 i: 2, qvalue: 586, hashes[i]:932, qsum:24028, sums[i]32266 i: 3, qvalue: 586, hashes[i]:848, qsum:24028, sums[i]31495 i: 4, qvalue: 586, hashes[i]:780, qsum:24028, sums[i]30665 i: 5, qvalue: 586, hashes[i]:735, qsum:24028, sums[i]29804 i: 6, qvalue: 586, hashes[i]:702, qsum:24028, sums[i]29146 i: 7, qvalue: 586, hashes[i]:679, qsum:24028, sums[i]28599 i: 8, qvalue: 586, hashes[i]:660, qsum:24028, sums[i]28104 i: 9, qvalue: 586, hashes[i]:645, qsum:24028, sums[i]27670 i: 10, qvalue: 586, hashes[i]:632, qsum:24028, sums[i]27225 i: 11, qvalue: 586, hashes[i]:623, qsum:24028, sums[i]26725 i: 12, qvalue: 586, hashes[i]:613, qsum:24028, sums[i]26210 i: 13, qvalue: 586, hashes[i]:607, qsum:24028, sums[i]25716 i: 14, qvalue: 586, hashes[i]:600, qsum:24028, sums[i]25240 i: 15, qvalue: 586, hashes[i]:594, qsum:24028, sums[i]24789 i: 16, qvalue: 586, hashes[i]:589, qsum:24028, sums[i]24373 i: 17, qvalue: 586, hashes[i]:585, qsum:24028, sums[i]23946 quality: -1 </code></pre></div></div> <p>Here we see that at <code class="language-plaintext highlighter-rouge">i=17</code>, the value of <code class="language-plaintext highlighter-rouge">sums[i]</code> becomes smaller than <code class="language-plaintext highlighter-rouge">qsum</code>. As a result, we end up in the second <em>if</em> block. This reports the quality factor as <code class="language-plaintext highlighter-rouge">i+1</code>, but <em>only</em> if either of the following conditions is met:</p> <ol> <li>both <code class="language-plaintext highlighter-rouge">qvalue</code> is smaller than or equal to <code class="language-plaintext highlighter-rouge">hashes[i]</code>, <em>and</em> <code class="language-plaintext highlighter-rouge">qsum</code> is smaller than or equal to <code class="language-plaintext highlighter-rouge">sums[i]</code>, <em>or</em>:</li> <li>the value of <code class="language-plaintext highlighter-rouge">i</code> is larger than or equal to 50.</li> </ol> <p>In this case, we see that <code class="language-plaintext highlighter-rouge">qvalue</code> is indeed smaller than <code class="language-plaintext highlighter-rouge">hashes[i]</code>, but <code class="language-plaintext highlighter-rouge">qsum</code> is <em>not</em> smaller than (or equal to) <code class="language-plaintext highlighter-rouge">sums[i]</code>. So the first condition is not met. Since <code class="language-plaintext highlighter-rouge">i</code> equals 17, the second condition is not met either, meaning that the code doesn’t come up with a meaningful quality estimate, and reports the fallback value (-1) instead.</p> <h2 id="effect-of-quality-threshold">Effect of quality threshold</h2> <p>It’s not clear why the heuristic uses the quality threshold value of 50, although <a href="https://fotoforensics.com/tutorial.php?tt=estq">Neal Krawetz points out</a> that “the JPEG Standard changes algorithms at quality values below 50%”<sup id="fnref:6"><a href="#fn:6" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>. As a test, I tried changing the threshold to 0:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">if </span><span class="p">(((</span><span class="n">qvalue</span> <span class="o">&lt;=</span> <span class="n">hashes</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="ow">and</span> <span class="p">(</span><span class="n">qsum</span> <span class="o">&lt;=</span> <span class="n">sums</span><span class="p">[</span><span class="n">i</span><span class="p">]))</span> <span class="ow">or</span> <span class="p">(</span><span class="n">i</span> <span class="o">&gt;=</span> <span class="mi">0</span><span class="p">)):</span> <span class="k">return</span> <span class="n">i</span><span class="o">+</span><span class="mi">1</span> </code></pre></div></div> <p>With this change, the code now reports a quality value of 18. Incidentally this is identical to the <em>Fotoforensics</em> estimate.</p> <h2 id="modified-imagemagick-heuristic">Modified ImageMagick heuristic</h2> <p>Setting the threshold of 0 effectively makes the first condition in the second <em>if</em> block superfluous, which means we could simplify things further to:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nf">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span> <span class="nf">if </span><span class="p">((</span><span class="n">qvalue</span> <span class="o">&lt;</span> <span class="n">hashes</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="ow">and</span> <span class="p">(</span><span class="n">qsum</span> <span class="o">&lt;</span> <span class="n">sums</span><span class="p">[</span><span class="n">i</span><span class="p">])):</span> <span class="k">continue</span> <span class="k">else</span><span class="p">:</span> <span class="k">return</span> <span class="n">i</span><span class="o">+</span><span class="mi">1</span> </code></pre></div></div> <p>I also noticed that the original ImageMagick code includes a variable that indicates whether the quality estimate is “exact” or “approximate” (<a href="https://github.com/ImageMagick/ImageMagick6/blob/bf9bc7fee9f3cea9ab8557ad1573a57258eab95b/coders/jpeg.c#L1030">here</a>). I initially assumed here that an “exact” match implies a perfect agreement with the standard JPEG quantization tables. This would be useful information to assess the accuracy of the quality estimate. I created <a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/jpegquality-im-modified.py">a test script with a modified version of the ImageMagick heuristic</a> that incorporates the following changes to the ImageMagick heuristic:</p> <ol> <li>Removal of the quality thresold.</li> <li>Added reporting of the “exactness” flag, based on the original ImageMagick code.</li> </ol> <h2 id="tests-with-pillow-and-imagemagick-jpegs">Tests with Pillow and ImageMagick JPEGs</h2> <p>As a first test I created a set of small test images with known quality values. I did this using Python’s <a href="https://python-pillow.org/">Pillow library</a> and <a href="https://imagemagick.org/">ImageMagick</a>. In both cases I generated test images with quality values 5, 10, 25, 50, 75 and 100, respectively. The images are available <a href="https://github.com/KBNLresearch/jpeg-quality-demo/tree/main/images/im_pil">here</a>. Running the script with my modified ImageMagick heuristic gave the following result:</p> <table> <thead> <tr> <th style="text-align: left">Q_enc</th> <th style="text-align: left">Q_est(Pillow)</th> <th style="text-align: left">Exact(Pillow)</th> <th style="text-align: left">Q_est(IM)</th> <th style="text-align: left">Exact(IM)</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">5</td> <td style="text-align: left">5</td> <td style="text-align: left">True</td> <td style="text-align: left">5</td> <td style="text-align: left">True</td> </tr> <tr> <td style="text-align: left">10</td> <td style="text-align: left">10</td> <td style="text-align: left">True</td> <td style="text-align: left">10</td> <td style="text-align: left">True</td> </tr> <tr> <td style="text-align: left">25</td> <td style="text-align: left">25</td> <td style="text-align: left">True</td> <td style="text-align: left">25</td> <td style="text-align: left">True</td> </tr> <tr> <td style="text-align: left">50</td> <td style="text-align: left">50</td> <td style="text-align: left">True</td> <td style="text-align: left">50</td> <td style="text-align: left">True</td> </tr> <tr> <td style="text-align: left">75</td> <td style="text-align: left">75</td> <td style="text-align: left">True</td> <td style="text-align: left">75</td> <td style="text-align: left">True</td> </tr> <tr> <td style="text-align: left">100</td> <td style="text-align: left">100</td> <td style="text-align: left">True</td> <td style="text-align: left">100</td> <td style="text-align: left">True</td> </tr> </tbody> </table> <p>Here <em>Q_enc</em> is the encoding quality, and <em>Q_est(Pillow)</em> and <em>Q_est(IM)</em> represent the script’s estimates for the Pillow and ImageMagick images, respectively. <em>Exact(Pillow)</em> and <em>Exact(IM)</em> represent the reported values of the “exactness” flag. The table shows that the script was able to reproduce the encoding quality with an “exact” match for all test images.</p> <p>I also ran the script on some of the problematic JPEGs from our dbnl access PDFs. Here, it estimated the JPEG quality at 18%, but without an “exact” match. The JPEGs from the corresponding master PDFs resulted in a 84% quality estimate (which is slightly less than the expected quality of 85%), but again without an “exact” result.</p> <p>Although the results of these (very limited!) tests look encouraging at first sight, this exercise left me with some doubts and reservations.</p> <h2 id="limitations-of-imagemagicks-heuristic">Limitations of ImageMagick’s heuristic</h2> <p>Most importantly, ImageMagick’s heuristic is based on a comparison of <em>aggregated</em> coefficients of the image’s quantization tables, which makes it potentially vulnerable to collisions. As an example, for any value of <code class="language-plaintext highlighter-rouge">qsum</code> (the sum of all coefficients in the quantization table), many possible combinations of quantization coefficients exist that will add up to the same value. The <code class="language-plaintext highlighter-rouge">qvalue</code> check (which is based on coefficients at specific positions in the quantization table) partially overcomes this, but it does this in a pretty crude way using yet another aggregate measure.</p> <p>Another thing that bothered me, is that the reasoning behind certain aspects of ImageMagick’s heuristic isn’t entirely clear to me. Examples are the 50% threshold, the <code class="language-plaintext highlighter-rouge">qvalue</code> check (again) and the precise meaning of its “exact” vs “approximate” designation.</p> <h2 id="further-down-the-jpeg-rabbit-hole">Further down the JPEG rabbit hole</h2> <p>When I wrote the first draft of this post, I hoped it would provoke some response from people who are better versed than me in the inner workings of JPEG compression. As happens so often with these things, only a few days after publishing it I came across <a href="https://www.sciencedirect.com/science/article/pii/S1742287608000285">this 2008 paper</a> which explains how quantization tables in JPEG work. This inspired me to do some more in-depth testing, and ultimately this resulted in an alternative, more straightforward quality estimation method. This is the subject of a <a href="/2024/10/30/jpeg-quality-estimation-using-simple-least-squares-matching-of-quantization-tables">follow-up blog post</a>, which is out now!</p> <h2 id="acknowledgment">Acknowledgment</h2> <p>Thanks are due to <a href="https://github.com/eddy-geek">Eddy O (AKA “eddygeek”)</a> for creating the <a href="https://gist.github.com/eddy-geek/c0f01dc5401dc50a49a0a821cdc9b3e8">Python port of ImageMagick’s JPEG quality heuristic</a> from which most of the results in this post are derived.</p> <h2 id="annex-other-jpeg-quality-estimation-tools-and-methods">Annex: other JPEG quality estimation tools and methods</h2> <p>While working on this, I came across a few alternative tools and methods for JPEG quality estimation. Here’s a brief overview, which is largely for my own reference (but I imagine others may find it useful as well).</p> <h3 id="exiftool">ExifTool</h3> <p><a href="https://exiftool.org/">ExifTool</a> also reports a JPEG quality estimate if its <code class="language-plaintext highlighter-rouge">-JPEGQualityEstimate</code> option is invoked. For example:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>exiftool -JPEGQualityEstimate test_im_050.jpg </code></pre></div></div> <p>Results in:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>JPEG Quality Estimate : 50 </code></pre></div></div> <p>A peek at the <a href="https://github.com/exiftool/exiftool/blob/4981552ec9bf94a0b5a64a06919b5e4f797c208e/lib/Image/ExifTool/JPEGDigest.pm#L2447">source code</a> shows it uses a ported version of ImageMagick’s heuristic, so it largely has the same limitations. As an example, running it on my problematic JPEG results in:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>JPEG Quality Estimate : &lt;unknown&gt; </code></pre></div></div> <h3 id="approximate-ratios-method">Approximate Ratios method</h3> <p>This method is outlined on <a href="https://fotoforensics.com/tutorial.php?tt=estq">Neal Krawetz’s Fotoforensics site</a>, which also links to a <a href="https://www.hackerfactor.com/src/jpegquality.c">an inplementation in C</a>. A more detailed explanation can be found in Section 3.3.3 of his <a href="https://blackhat.com/presentations/bh-dc-08/Krawetz/Whitepaper/bh-dc-08-krawetz-WP.pdf">Digital Image Analysis and Forensics whitepaper</a>. As Krawetz explains, this method can become unreliable at quality values below 50%, so I didn’t consider it suitable for my use case.</p> <h3 id="cogranne-method">Cogranne method</h3> <p>A <a href="https://arxiv.org/abs/1802.00992">2018 paper by Rémi Cogranne</a> describes an alternative method for estimating JPEG quality. It claims to overcome some of the limitations of established methods, such as the ImageMagick heuristic. However, as it is only valid for a quality factor greater than 49, this also isn’t ideally suited to my use case.</p> <h2 id="scripts-and-test-data">Scripts and test data</h2> <ul> <li><a href="https://github.com/KBNLresearch/jpeg-quality-demo">jpeg-quality-demo Github repository</a> - Github repo with all scripts and test data that were used in this analysis.</li> <li><a href="https://github.com/KBNLresearch/jpeg-quality-demo/blob/main/jpegquality-im-modified.py">jpegquality-im-modified.py</a> - Python implementation of modified ImageMagick heuristic.</li> </ul> <h2 id="revision-history">Revision history</h2> <ul> <li> <p>24 October 2024: re-arranged introductory section, and added an explanation on the difference between quality level and image quality.</p> </li> <li> <p>30 October 2024: revised earlier draft in preparation of follow-up post.</p> </li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:4"> <p>Note that the quality level does not necessarily reflect the <em>image quality</em>! As an example, if an image was first compressed at 20% quality and subsequently re-saved at 90%, the image quality will be very low (relative to the source image), despite the high quality level at the last save operation. So the quality level only says something about the <em>compression process</em> that was used on the last save. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:1"> <p>I used <a href="https://poppler.freedesktop.org/">Poppler</a>’s <em>pdfimages</em> tool to extract the images from the PDF, using the <code class="language-plaintext highlighter-rouge">-all</code> switch which ensures images are kept in their original format. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>I used ImageMagick 6.9.10-23. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:7"> <p>More precisely, each value in the <code class="language-plaintext highlighter-rouge">sums</code> lists represents the sum of all “standard” quantization coefficients for a particular quality level. Similarly, each value in the <code class="language-plaintext highlighter-rouge">hashes</code> lists represents a value of <code class="language-plaintext highlighter-rouge">qvalue</code> for a particular quality level in the “standard” tables. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:6"> <p>He mentions this in the context of the “Approximate Ratios” quality estimation method (which indeed becomes unreliable for low qualities). It’s not clear to me if other methods such as the one used by ImageMagick are also affected by this. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2024/10/23/jpeg-quality-estimation-experiments-with-a-modified-imagemagick-heuristic https://bitsgalore.org/2024/10/23/jpeg-quality-estimation-experiments-with-a-modified-imagemagick-heuristic 2024-10-23T00:00:00+02:00 <figure class="image"> <img src="/images/2024/03/confused-muddled-illogical-957696-1024.jpg" alt="Photograph that shows a hammer that is used to smash a screw into a piece of wood. On the left is a nail that is partially pushed into the same piece of wood, with an adjustable wrench immediately next to it." /> <figcaption><a href="https://jenikirbyhistory.getarchive.net/media/confused-muddled-illogical-957696">"Confused, muddled, illogical"</a>. Used under <a href="https://web.archive.org/web/20170727004823/https://pixabay.com/en/service/license/">Pixabay License</a>.</figcaption> </figure> <p>The KB has been using JP2 (JPEG 2000 Part 1) as the primary file format for its mass-digitisation activities for over 15 years now. Nevertheless, we still use uncompressed TIFF for a few collections. At the moment there’s an ongoing discussion about whether we should migrate those to JP2 as well at some point to save storage costs. Last week I ran a small test on a selection of TIFFs from those collections. I first converted them to JP2, and then verified whether no information got lost during the conversion. This resulted in some unexpected surprises, which turned out to be caused by the presence of thumbnail images in some of the source TIFFs. This post discusses the impact of having multiple images indide a TIFF on preservation workflows, and also provides some suggestions on how to identify such files.</p> <!-- more --> <h2 id="tiff-to-jp2-workflow">TIFF to JP2 workflow</h2> <p>For my tests, I took a selection of 20 test targets in uncompressed TIFF format from five different digitisation batches. For each of these targets, I then:</p> <ol> <li>converted the source TIFF to lossless JP2 with Kakadu;</li> <li>converted the JP2 from step 1 back to uncompressed TIFF;</li> <li>compared the pixel values of the TIFF from step 2 against those of the source TIFF<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup> with <a href="https://imagemagick.org/script/compare.php">ImageMagick’s <em>compare</em> tool</a>.</li> </ol> <p>For the pixel comparison in step 3, I used the following general command (using ImageMagick version 6.9.10-23):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>compare <span class="nt">-quiet</span> <span class="nt">-metric</span> AE source.tif fromjp2.tif null: </code></pre></div></div> <p>This command computes the absolute error count (i.e. the number of different pixels) between both images<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>, which is printed to standard error (<a href="https://en.wikipedia.org/wiki/Standard_streams">stderr</a>). The <code class="language-plaintext highlighter-rouge">-quiet</code> switch is included to suppress warning messages (these are also printed to stderr, and can muddle up the error count output).</p> <h2 id="imagemagick-reports-changed-pixel-values">ImageMagick reports changed pixel values</h2> <p>Since we used lossless compression in our JP2 conversion step, the expected outcome here is that all pixel values are unchanged, and ImageMagick’s AE output metric is exactly zero for all images. For 12 images this was indeed the case. However, for 8 images the AE metric indicated that nearly all pixels had changed. An additional check showed that the computed <a href="https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio">peak signal-to-noise ratio</a> (PSNR) values for these images were also extremely poor, ranging between 3 and 13. This was definitely unexpected!</p> <p>A subsequent visual inspection of the affected images in <a href="https://www.gimp.org/">Gimp</a> did not show any obvious degradation. As an additional check, I also loaded one pair of images in Gimp as separate layers, and then <a href="https://www.reddit.com/r/GIMP/comments/32dgfq/comment/cqa95dl/">subtracted one layer from the other one</a>. This resulted in an image where all RGB values were exactly 0, which is only possible if both images are identical. So what’s going on here?</p> <h2 id="subfiles-and-image-file-directories">Subfiles and image file directories</h2> <p>My first idea was to analyze all source TIFFs with <a href="https://exiftool.org/">ExifTool</a>. For each source TIFF, I used the following general command (using ExifTool version 12.60):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>exiftool <span class="nt">-X</span> source.tif <span class="o">&gt;</span> source.xml </code></pre></div></div> <p>An inspection of the resulting output files showed that all problematic TIFFs contain two separate “image file directories”. Within TIFF, it is possible to bundle multiple images in one single file. This is most commonly done by defining each image as a “subfile”, whose properties are described by a corresponding “image file directory” (<em>IFD</em>). The individual images can be pages in a multi-page document, or different representations of the same image (e.g. a full size image and a low resolution thumbnail). In our particular case, we have this:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nt">&lt;IFD1:SubfileType&gt;</span>Reduced-resolution image<span class="nt">&lt;/IFD1:SubfileType&gt;</span> <span class="nt">&lt;IFD1:ImageWidth&gt;</span>160<span class="nt">&lt;/IFD1:ImageWidth&gt;</span> <span class="nt">&lt;IFD1:ImageHeight&gt;</span>126<span class="nt">&lt;/IFD1:ImageHeight&gt;</span> <span class="nt">&lt;IFD1:BitsPerSample&gt;</span>8 8 8<span class="nt">&lt;/IFD1:BitsPerSample&gt;</span> <span class="nt">&lt;IFD1:Compression&gt;</span>Uncompressed<span class="nt">&lt;/IFD1:Compression&gt;</span> <span class="nt">&lt;IFD1:PhotometricInterpretation&gt;</span>RGB<span class="nt">&lt;/IFD1:PhotometricInterpretation&gt;</span> <span class="nt">&lt;IFD1:StripOffsets&gt;</span>6510<span class="nt">&lt;/IFD1:StripOffsets&gt;</span> <span class="nt">&lt;IFD1:SamplesPerPixel&gt;</span>3<span class="nt">&lt;/IFD1:SamplesPerPixel&gt;</span> <span class="nt">&lt;IFD1:RowsPerStrip&gt;</span>126<span class="nt">&lt;/IFD1:RowsPerStrip&gt;</span> <span class="nt">&lt;IFD1:StripByteCounts&gt;</span>60480<span class="nt">&lt;/IFD1:StripByteCounts&gt;</span> <span class="nt">&lt;IFD1:PlanarConfiguration&gt;</span>Chunky<span class="nt">&lt;/IFD1:PlanarConfiguration&gt;</span> <span class="nt">&lt;IFD1:ThumbnailTIFF&gt;</span>(Binary data 60696 bytes, use -b option to extract)<span class="nt">&lt;/IFD1:ThumbnailTIFF&gt;</span> </code></pre></div></div> <p>Here, all “IFD1” tags describe a subfile that is a reduced resolution thumbnail image (this is indicated by the <em>SubfileType</em> tag and its value). The full-resolution image is described by “IFD0”, which is the first image file directory:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nt">&lt;IFD0:ImageWidth&gt;</span>9458<span class="nt">&lt;/IFD0:ImageWidth&gt;</span> <span class="nt">&lt;IFD0:ImageHeight&gt;</span>7429<span class="nt">&lt;/IFD0:ImageHeight&gt;</span> <span class="nt">&lt;IFD0:BitsPerSample&gt;</span>8 8 8<span class="nt">&lt;/IFD0:BitsPerSample&gt;</span> <span class="nt">&lt;IFD0:Compression&gt;</span>Uncompressed<span class="nt">&lt;/IFD0:Compression&gt;</span> <span class="nt">&lt;IFD0:PhotometricInterpretation&gt;</span>RGB<span class="nt">&lt;/IFD0:PhotometricInterpretation&gt;</span> <span class="nt">&lt;IFD0:Make&gt;</span>Leaf<span class="nt">&lt;/IFD0:Make&gt;</span> <span class="nt">&lt;IFD0:Model&gt;</span>Leaf Aptus-II 12R(LI201033 )/Other<span class="nt">&lt;/IFD0:Model&gt;</span> <span class="nt">&lt;IFD0:StripOffsets&gt;</span>(Binary data 72 bytes, use -b option to extract)<span class="nt">&lt;/IFD0:StripOffsets&gt;</span> <span class="nt">&lt;IFD0:Orientation&gt;</span>Horizontal (normal)<span class="nt">&lt;/IFD0:Orientation&gt;</span> <span class="nt">&lt;IFD0:SamplesPerPixel&gt;</span>3<span class="nt">&lt;/IFD0:SamplesPerPixel&gt;</span> <span class="nt">&lt;IFD0:RowsPerStrip&gt;</span>1024<span class="nt">&lt;/IFD0:RowsPerStrip&gt;</span> <span class="nt">&lt;IFD0:StripByteCounts&gt;</span>(Binary data 70 bytes, use -b option to extract)<span class="nt">&lt;/IFD0:StripByteCounts&gt;</span> <span class="nt">&lt;IFD0:XResolution&gt;</span>300<span class="nt">&lt;/IFD0:XResolution&gt;</span> <span class="nt">&lt;IFD0:YResolution&gt;</span>300<span class="nt">&lt;/IFD0:YResolution&gt;</span> <span class="nt">&lt;IFD0:PlanarConfiguration&gt;</span>Chunky<span class="nt">&lt;/IFD0:PlanarConfiguration&gt;</span> <span class="nt">&lt;IFD0:ResolutionUnit&gt;</span>inches<span class="nt">&lt;/IFD0:ResolutionUnit&gt;</span> <span class="nt">&lt;IFD0:Software&gt;</span>Capture One 8 Macintosh<span class="nt">&lt;/IFD0:Software&gt;</span> </code></pre></div></div> <h2 id="subifds">SubIFDs</h2> <p>For completeness, I should mention here that an alternative way to store multiple images in a TIFF, is by combining them in one single subfile. The corresponding image file directory then contains <em>SubIFD</em> child tags, each of which describes an individual image. The <em>SubIFD</em> tag is defined in the <a href="https://www.awaresystems.be/imaging/tiff/specification/TIFFPM6.pdf">Adobe PageMaker 6.0 TIFF Technical Notes</a>. According to the documentation of the <a href="https://libtiff.gitlab.io/libtiff/">LibTIFF</a> library, <a href="https://libtiff.gitlab.io/libtiff/multi_page.html">“SubIFD chains are rarely supported”</a>. I also couldn’t find any example files that use them.</p> <h2 id="remove-thumbnail-with-exiftool">Remove thumbnail with ExifTool</h2> <p>To test whether the presence of the thumbnail is indeed the cause of my weird pixel check results, I removed it from (a copy of) one of the problematic source TIFFs with the following ExifTool command:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>exiftool <span class="nt">-ifd1</span>:all<span class="o">=</span> <span class="nt">-m</span> source.tif </code></pre></div></div> <p>Note that this removes <em>IFD1</em> (which is the second <em>IFD</em>) and its associated data.</p> <p>When I re-ran ImageMagick’s <em>compare</em> tool with the modified file, the output was as expected: the absolute error count was 0, and the PSNR value “inf”<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>.</p> <h2 id="pixel-compare-with-unmodified-source-tiffs">Pixel compare with unmodified source TIFFs</h2> <p>Even though removing the thumbnail gets the job done, it adds another processing step to our workflow. Fortunately, a <a href="https://github.com/ImageMagick/ImageMagick/discussions/3279#discussioncomment-387226">little digging</a> revealed that it’s possible to explicitly address individual images in a multi-image file in ImageMagick. If we (only) want to use the first image in both our input images, we can call the compare tool like this:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>compare <span class="nt">-quiet</span> <span class="nt">-metric</span> AE source.tif[0] fromjp2.tif[0] null: </code></pre></div></div> <p>Note that the number between square brackets sets the index of the subfile that is used for the comparison. Applying this to our unmodified source TIFFs, this indeed results in an absolute error count of 0 for all image pairs. I implemented the above command in a simple <a href="https://github.com/KBNLresearch/jp2totiff/blob/master/pixelCheck-im.sh">test script</a> that compares two directory trees with TIFF images, and reports the results to a comma-delimited file.</p> <h2 id="imagemagick-reads-last-subfile-by-default">ImageMagick reads last subfile by default</h2> <p>As a final test, I deliberately instructed ImageMagick to use the second subfile (i.e. the thumbnail) of the source TIFF as a basis for the comparison:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>compare <span class="nt">-quiet</span> <span class="nt">-metric</span> AE source.tif[1] fromjp2.tif[0] null: </code></pre></div></div> <p>The resulting (large) AE value is identical to what is reported if the subfiles are not set by the user at all. So, it seems that by default ImageMagick uses the <em>last</em> subfile of each input image, and that this is the root cause of the unexpected behaviour!</p> <h2 id="are-multi-image-tiffs-a-preservation-risk">Are multi-image TIFFs a preservation risk?</h2> <p>ImageMagick’s behaviour made me wonder to what extent the presence of multiple images poses a preservation risk. In our specific example, the second IFD only represents a thumbnail, and losing this in a format migration isn’t such a big deal.</p> <p>Section 7 of the <a href="https://web.archive.org/web/20180810205359/https://www.adobe.io/content/udp/en/open/standards/TIFF/_jcr_content/contentbody/download/file.res/TIFF6.pdf">TIFF 6.0 specification</a> describes how baseline TIFF readers should handle files with multiple images:</p> <blockquote> <p>TIFF readers must be prepared for multiple images (subfiles) per TIFF file, although they are not required to do anything with images after the first one.</p> </blockquote> <p>For the thumbnail case, I can think of two potential problems:</p> <ol> <li>If the writer application mistakenly wrote the thumbnail as the first subfile (and the full resolution images as the last one), a migration tool that is based on a conforming reader would then migrate the thumbnail, and ignore the full resolution image data.</li> <li>If a migration tool mistakenly reads the last subfile instead of the first one, this would also result in the loss of data.</li> </ol> <p>I expect the above problems are largely hypothetical, especially since TIFF has been around for such a long time. But I wouldn’t completely rule them out either. Importantly, such errors could pass unnoticed by image comparison tools like ImageMagick’s <em>compare</em> if they use the wrong subfile as a comparison basis. Fortunately, the extremely small size of the resulting image files would be a pretty obvious clue that something is amiss.</p> <p>TIFFs with multiple images that represent pages in a multi-page document could pose a larger risk. I don’t really expect any of these to show up in our own collections<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>, but the situation might be different for other collections.</p> <p>Finally, TIFFs with subfiles that in turn contain multiple images through the use of <em>SubIFD</em> tags add another layer of complexity.</p> <p>So, depending on your specific situation, it might be prudent to check your TIFFs for the presence of multiple images before doing any preservation actions on them, like a migration to some other format.</p> <h2 id="detection-of-multi-image-tiffs">Detection of multi-image TIFFs</h2> <p>In order to detect the presence of multiple images, we have a couple of options.</p> <h3 id="exiftool">ExifTool</h3> <p>With <a href="https://exiftool.org/">ExifTool</a>, we can use the aforementioned command:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>exiftool <span class="nt">-X</span> source.tif <span class="o">&gt;</span> source.xml </code></pre></div></div> <p>As an example, here’s some output for a 6-page TIFF (<a href="https://www.leadtools.com/support/forum/resource.ashx?a=544&amp;b=1">file available here</a>):</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nt">&lt;IFD0:SubfileType&gt;</span>Full-resolution image<span class="nt">&lt;/IFD0:SubfileType&gt;</span> <span class="nt">&lt;IFD0:ImageWidth&gt;</span>363<span class="nt">&lt;/IFD0:ImageWidth&gt;</span> <span class="nt">&lt;IFD0:ImageHeight&gt;</span>7429<span class="nt">&lt;/IFD0:ImageHeight&gt;</span> :: <span class="nt">&lt;IFD1:SubfileType&gt;</span>Full-resolution image<span class="nt">&lt;/IFD1:SubfileType&gt;</span> <span class="nt">&lt;IFD1:ImageWidth&gt;</span>363<span class="nt">&lt;/IFD1:ImageWidth&gt;</span> <span class="nt">&lt;IFD1:ImageHeight&gt;</span>382<span class="nt">&lt;/IFD1:ImageHeight&gt;</span> :: :: <span class="nt">&lt;IFD5:SubfileType&gt;</span>Full-resolution image<span class="nt">&lt;/IFD5:SubfileType&gt;</span> <span class="nt">&lt;IFD5:ImageWidth&gt;</span>363<span class="nt">&lt;/IFD5:ImageWidth&gt;</span> <span class="nt">&lt;IFD5:ImageHeight&gt;</span>382<span class="nt">&lt;/IFD5:ImageHeight&gt;</span> :: </code></pre></div></div> <p>If the output contains one or more <code class="language-plaintext highlighter-rouge">&lt;IFD#:SubfileType&gt;</code> tags (where <code class="language-plaintext highlighter-rouge">#</code> represents an integer number), this indicates the file contains subfiles. It’s worth noting that page 41 of the <a href="https://web.archive.org/web/20180810205359/https://www.adobe.io/content/udp/en/open/standards/TIFF/_jcr_content/contentbody/download/file.res/TIFF6.pdf">TIFF 6.0 specification</a> mentions that the <em>SubfileType</em> tag is deprecated, and the <em>NewSubfileType</em> tag should be used instead. So why does ExifTool report <em>SubFileType</em>? A quick look at <a href="https://exiftool.org/TagNames/EXIF.html">ExifTool’s documentation</a> reveals that ExifTool’s <em>SubfileType</em> output actually corresponds to the TIFF spec’s <em>NewSubfileType</em> tag. If a TIFF contains the deprecated tag (<em>SubfileType</em> in the TIFF 6.0 spec), ExifTool reports this as <em>OldSubfileType</em> (yes, this is a bit confusing!).</p> <p>I also ran Exiftool on <a href="https://www.leadtools.com/support/forum/resource.ashx?a=545&amp;b=1">this 11-page TIFF</a>, which uses a different compression type for each page. The compression types can be inferred from the <em>Compression</em> fields in ExifTool’s output:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nt">&lt;IFD0:SubfileType&gt;</span>Single page of multi-page image<span class="nt">&lt;/IFD0:SubfileType&gt;</span> :: <span class="nt">&lt;IFD0:Compression&gt;</span>Uncompressed<span class="nt">&lt;/IFD0:Compression&gt;</span> :: <span class="nt">&lt;IFD1:SubfileType&gt;</span>Single page of multi-page image<span class="nt">&lt;/IFD1:SubfileType&gt;</span> :: <span class="nt">&lt;IFD2:SubfileType&gt;</span>Single page of multi-page image<span class="nt">&lt;/IFD2:SubfileType&gt;</span> :: <span class="nt">&lt;IFD2:Compression&gt;</span>JPEG 2000<span class="nt">&lt;/IFD2:Compression&gt;</span> :: <span class="nt">&lt;IFD3:SubfileType&gt;</span>Single page of multi-page image<span class="nt">&lt;/IFD3:SubfileType&gt;</span> :: <span class="nt">&lt;IFD3:Compression&gt;</span>JPEG<span class="nt">&lt;/IFD3:Compression&gt;</span> :: :: </code></pre></div></div> <p>Also, as explained above, a single subfile can in turn contain multiple images, each of which are described by a <em>SubIFD</em> tag. I couldn’t find any example files that use <em>SubIFDs</em>, but ExifTool reports them according to <a href="https://exiftool.org/TagNames/EXIF.html">the documentation</a>.</p> <p>So, if you just want to cover all of the aforementioned cases, you might want to check your TIFFs for the presence of:</p> <ol> <li><em>IFD</em> with <em>SubfileType</em> tag (<code class="language-plaintext highlighter-rouge">&lt;IFD#:SubfileType&gt;</code>)</li> <li><em>IFD</em> with <em>OldSubfileType</em> tag (<code class="language-plaintext highlighter-rouge">&lt;IFD#:OldSubfileType&gt;</code>)</li> <li><em>SubIFD</em> tag.</li> </ol> <h3 id="imagemagick-identify">ImageMagick identify</h3> <p>You can also identify the presence of multiple images with ImageMagick’s <a href="https://imagemagick.org/script/identify.php"><em>identify</em></a> tool (via <a href="https://preservation.tylerthorsted.com/2023/09/15/tiff/">Tyler Thorsted</a>):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>identify <span class="nt">-quiet</span> source.tif </code></pre></div></div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>source.tif[0] TIFF 9458x7429 9458x7429+0+0 8-bit sRGB 201.089MiB 0.000u 0:00.010 source.tif[1] TIFF 160x126 160x126+0+0 8-bit sRGB 0.000u 0:00.010 </code></pre></div></div> <p>However, running this on the 11-page multi-format TIFF gave me the following error (with no meaningful information on the number of images):</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>identify-im6.q16: compression not supported `MultipleFormats.tif' @ error/tiff.c/ReadTIFFImage/1433. </code></pre></div></div> <p>It’s also not clear to me how ImageMagick handles <em>SubIFD</em> tags.</p> <h3 id="jhove">JHOVE</h3> <p>The output of <a href="https://jhove.openpreservation.org/">JHOVE</a>’s TIFF module also provides information on subfiles. You can use the following command:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jhove <span class="nt">-m</span> TIFF-hul <span class="nt">-i</span> source.tif <span class="nt">-h</span> xml <span class="nt">-o</span> source-jhove.xml </code></pre></div></div> <p>JHOVE (version 1.28.0) reports each <em>NewSubfileType</em> tag as:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>NewSubfileType<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"Scalar"</span> <span class="na">type=</span><span class="s">"Long"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>0<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> </code></pre></div></div> <p>Unlike ExifTool, JHOVE also reports a <em>NewSubfileType</em> property for the main image, which means the output always contains at least one <em>NewSubfileType</em> property. So, TIFFs with subfiles can be singled out by the presence of more than one <em>NewSubfileType</em> property in JHOVE’s output.</p> <p>I noticed that JHOVE’s <em>NewSubfileType</em> output doesn’t always describe the role of each subfile. For example, for a TIFF with a subfile that is a thumbnail, JHOVE reports this (which is as expected):</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>NewSubfileType<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"List"</span> <span class="na">type=</span><span class="s">"String"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>reduced-resolution image of another image in this file<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> </code></pre></div></div> <p>But for <a href="https://www.leadtools.com/support/forum/resource.ashx?a=544&amp;b=1">this 6-page TIFF</a>, the <em>NewSubfileType</em> tag of each TIFF is reported as:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>NewSubfileType<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"Scalar"</span> <span class="na">type=</span><span class="s">"Long"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>0<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> </code></pre></div></div> <p>Unlike ExifTool, JHOVE’s output doesn’t provide any direct clue here that each subfile is a full-resolution image. Oddly, for <a href="https://www.leadtools.com/support/forum/resource.ashx?a=545&amp;b=1">this 11-page multi-format TIFF</a>, JHOVE <em>does</em> report that each subfile is a single page:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>NewSubfileType<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"List"</span> <span class="na">type=</span><span class="s">"String"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>single page of multi-page image<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> </code></pre></div></div> <p>Further perusal reveals that JHOVE’s output on compression is also incomplete for this file: for 3 pages, it reports an “unknown” compression scheme:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;mix:Compression&gt;</span> <span class="nt">&lt;mix:compressionScheme&gt;</span>Unknown<span class="nt">&lt;/mix:compressionScheme&gt;</span> <span class="nt">&lt;/mix:Compression&gt;</span> </code></pre></div></div> <p>This happens for pages with JBIG, JPEG and JPEG 2000 compression.</p> <p>Again, if you’re working with very old TIFFs, you might also want to check JHOVE’s output for the presence of <em>SubfileType</em> properties, which represent the deprecated TIFF tag discussed in the ExifTool section.</p> <p>JHOVE’s <a href="https://jhove.openpreservation.org/modules/tiff/">documentation</a> doesn’t mention the <em>SubIFD</em> tag, so my best guess is this isn’t supported.</p> <h2 id="conclusion">Conclusion</h2> <p>The presence of multiple images in TIFF files, either as thumbnails or pages in a multi-page document, can result in unexpected software behaviour. Because of this, it is important to identify them in any TIFF-based preservation workflows. Based on these tests with ExifTool, ImageMagick and JHOVE, it appears that ExifTool is the most reliable, robust and complete tool for identifying multi-image TIFFS. ExifTool was also the only tool that was able to correctly identify the compression scheme of each image in a particularly challenging 11-page TIFF with multiple compression schemes.</p> <h2 id="further-resources">Further resources</h2> <ul> <li><a href="https://preservation.tylerthorsted.com/2023/09/15/tiff/">Blog post on TIFF by Tyler Thorsted</a> (discusses TIFF preservation risks, including multi-image files)</li> <li><a href="https://www.leadtools.com/support/forum/posts/t10960-">Multipage TIFF sample files from Leadtools</a></li> <li><a href="https://web.archive.org/web/20180810205359/https://www.adobe.io/content/udp/en/open/standards/TIFF/_jcr_content/contentbody/download/file.res/TIFF6.pdf">TIFF 6.0 specification</a></li> <li><a href="https://www.awaresystems.be/imaging/tiff/specification/TIFFPM6.pdf">Adobe PageMaker 6.0 TIFF Technical Notes</a> (defines <em>SubIFDs</em>)</li> <li><a href="https://github.com/KBNLresearch/jp2totiff/blob/master/pixelCheck-im.sh">Test script (bash)</a> that performs a pixel comparison on two directory trees with TIFF images.</li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>You might wonder why I didn’t do the comparison directly between the source TIFF and the JP2, which would eliminate step 2. The reason for this is, that the ImageMagick builds don’t include JPEG 2000 support by default. The only way to make this work is to build ImageMagick from its source. This requires that all the necessary development libraries are installed, and ImageMagick’s build configuration is set up to include JPEG 2000 support. Even though I’ve successfully managed to make this work in the past, the procedure is quite time consuming, and I didn’t want to go through it right now. So, for this test I used the round-trip conversion to TIFF as a workaround. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>An overview of all available <code class="language-plaintext highlighter-rouge">-metric</code> values can be found <a href="https://imagemagick.org/script/command-line-options.php#metric">here</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>PSNR approaches infinity if both inputs are identical, which is exactly what you would expect for lossless compression. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>This is mainly because these were created according to very specific guidelines. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2024/03/11/multi-image-tiffs-subfiles-and-image-file-directories https://bitsgalore.org/2024/03/11/multi-image-tiffs-subfiles-and-image-file-directories 2024-03-11T00:00:00+01:00 <figure class="image"> <img src="/images/2023/06/barnum-bailey.jpg" alt="Vintage lithograph circus poster that shows a circus ring. In the front is a woman in a red dress, standing on horseback. Behind her there are more horses, with a variety of circus artists, including acrobats and jugglers, performing on horseback as well. In the background acrobats are walking on a tightrope." /> <figcaption><a href="https://www.flickr.com/photos/boston_public_library/6554392117/in/album-72157629549177588/">"The Barnum &amp; Bailey greatest show on earth"</a>. Used under <a href="https://creativecommons.org/licenses/by/2.0/">CC BY-BY 2.0</a>, via Boston Public Library.</figcaption> </figure> <p>Last month I <a href="/2023/05/25/identification-of-pdf-preservation-risks-with-verapdf-and-jhove">wrote this post</a>, which addresses the use of <a href="https://jhove.openpreservation.org/">JHOVE</a> and <a href="https://verapdf.org/">VeraPDF</a> for identifying preservation risks in PDF files. In the concluding section I suggested that VeraPDF’s parse status might be used as a rough “validity proxy” to identify malformed PDFs. But does VeraPDF’s parse status actually have any predictive value for rendering? And how does this compare to what JHOVE tells us? This post is a first attempt at answering these questions, using data from the <a href="https://www.radar-service.eu/radar/en/dataset/JtlOdwQquZWDqQdq">Synthetic PDF Testset for File Format Validation</a> by Lindlar, Tunnat and Wilson.</p> <!-- more --> <h2 id="goal-and-objectives-of-this-post">Goal and objectives of this post</h2> <p>The main goal of this post is to get more insight into the associations between VeraPDF parse errors and rendering behaviour, and to see if the occurrence of these parse errors can be used as a rough “validity proxy” to identify malformed PDFs. For this I propose a tentative methodology, and apply this to a previously published annotated dataset of synthetic PDF files. I then try to use the results to answer the following questions:</p> <ol> <li>To what extent is VeraPDF’s reported parse status associated with rendering behaviour in Adobe Acrobat?</li> <li>How does this compare against the occurrence of JHOVE validation errors?</li> <li>What are the implications of the answers to 1. and 2. for preservation workflows?</li> </ol> <h2 id="synthetic-pdf-testset-for-file-format-validation">Synthetic PDF Testset for File Format Validation</h2> <p>In 2017, Lindlar, Tunnat and Wilson published the <a href="https://www.radar-service.eu/radar/en/dataset/JtlOdwQquZWDqQdq">Synthetic PDF Testset for File Format Validation</a>. It is a corpus of 88 small, hand-crafted PDFs, that each violate the PDF format specification in different ways. The focus of the dataset is on “basic structure violations at the header, trailer, document catalog, page tree node and cross-reference levels”, and it also includes “violations at the page node, page resource and stream object level”. The dataset comes with a descriptive spreadsheet, which includes a column with rendering behaviour of each file in Adobe Acrobat Professional XI Pro (11.0.15). The dataset was specifically created for testing the validation functionality of JHOVE’s PDF module. A detailed discussion of this dataset can be found in the authors’ <a href="https://phaidra.univie.ac.at/detail/o:931074">2017 iPRES paper</a>.</p> <h2 id="running-jhove-and-verapdf">Running JHOVE and VeraPDF</h2> <p>As a first test, I ran all PDFs in the dataset through JHOVE (v 1.28.0) and VeraPDF (v 1.22.3). For JHOVE I used the following generic command line arguments:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jhove -m PDF-hul -h XML -i whatever.pdf whatever-jhove.xml </code></pre></div></div> <p>And for VeraPDF:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>verapdf --off --addlogs --extract whatever.pdf &gt; whatever-vera.xml </code></pre></div></div> <p>Here, the <code class="language-plaintext highlighter-rouge">--off</code> switch disables (PDF/A or PDF/UA) validation, and the <code class="language-plaintext highlighter-rouge">--addlogs</code> switch ensures that information about any run-time warnings is included in the output. The <code class="language-plaintext highlighter-rouge">--extract</code> switch enables feature extraction<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. The raw JHOVE and VeraPDF output files are available <a href="https://github.com/KBNLresearch/pdf-characterisation/tree/main/output/lindlar-tunnat-wilson">here</a>. I created <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/scripts/jhove-verapdf-validation-run.py">this script</a>, which runs JHOVE and VeraPDF for all files in the dataset, and creates a <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/lindlar-tunnat-wilson/data.csv">comma-delimited file</a> with the following output metrics for each PDF:</p> <ol> <li>The JHOVE validation status</li> <li>A Boolean flag that is <em>True</em> if VeraPDF reported one or more parse errors, and <em>False</em> otherwise</li> <li>A Boolean flag that is <em>True</em> if VeraPDF reported one or more warnings, and <em>False</em> otherwise</li> </ol> <h2 id="jhove-validation-status-vs-verapdf-parse-errors-and-warnings">JHOVE validation status vs VeraPDF parse errors and warnings</h2> <p>First, let’s look at to what degree the JHOVE and VeraPDF results are interrelated. For this, I created two contingency tables that cross-tabulate the JHOVE and VeraPDF results. First, here are the JHOVE validation status outcomes<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup> versus the VeraPDF parse error flags:</p> <table> <thead> <tr> <th style="text-align: left">JHOVE status</th> <th style="text-align: right">VeraPDF parse errors</th> <th style="text-align: right">No VeraPDF parse errors</th> <th style="text-align: right">All</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">Not well-formed</td> <td style="text-align: right">62</td> <td style="text-align: right">7</td> <td style="text-align: right">69</td> </tr> <tr> <td style="text-align: left">Well-Formed, but not valid</td> <td style="text-align: right">2</td> <td style="text-align: right">4</td> <td style="text-align: right">6</td> </tr> <tr> <td style="text-align: left">Well-Formed and valid</td> <td style="text-align: right">4</td> <td style="text-align: right">9</td> <td style="text-align: right">13</td> </tr> <tr> <td style="text-align: left">All</td> <td style="text-align: right">68</td> <td style="text-align: right">20</td> <td style="text-align: right">88</td> </tr> </tbody> </table> <p>Most PDFs that JHOVE considers “Not well-formed” also result in VeraPDF parse errors, which isn’t unexpected. Interestingly, out of the 13 PDFs that JHOVE considers “Well-Formed and valid”, 4 nevertheless result in VeraPDF parse errors. Most of the PDFs that JHOVE considers “Well-Formed, but not valid” don’t result in veraPDF parse errors.</p> <p>Here’s the contingency table of JHOVE validation status outcomes versus the VeraPDF warning flags:</p> <table> <thead> <tr> <th style="text-align: left">JHOVE status</th> <th style="text-align: right">VeraPDF warnings</th> <th style="text-align: right">No VeraPDF warnings</th> <th style="text-align: right">All</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">Not well-formed</td> <td style="text-align: right">62</td> <td style="text-align: right">7</td> <td style="text-align: right">69</td> </tr> <tr> <td style="text-align: left">Well-Formed, but not valid</td> <td style="text-align: right">2</td> <td style="text-align: right">4</td> <td style="text-align: right">6</td> </tr> <tr> <td style="text-align: left">Well-Formed and valid</td> <td style="text-align: right">5</td> <td style="text-align: right">8</td> <td style="text-align: right">13</td> </tr> <tr> <td style="text-align: left">All</td> <td style="text-align: right">69</td> <td style="text-align: right">19</td> <td style="text-align: right">88</td> </tr> </tbody> </table> <p>The results are almost identical to those of the VeraPDF parse errors. The only difference here, is that out of the PDFs that are “Well-Formed and valid” according to JHOVE, 5 resulted in VeraPDF warnings, whereas only 4 resulted in VeraPDF parse errors.</p> <p>We can also use statistical measures to characterise the association between the JHOVE and VeraPDF metrics. A suitable measure for nominal-scale variables is <a href="https://en.wikipedia.org/wiki/Cram%C3%A9r%27s_V">Cramér’s <em>V</em></a>. Cramér’s <em>V</em> can have any value from 0 to 1, where 0 indicates no association between two variables, and 1 a complete association. The following table shows the calculated Cramér’s V<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">3</a></sup> values between JHOVE status and VeraPDF parse errors, and JHOVE status and VeraPDF warnings, respectively:</p> <table> <thead> <tr> <th style="text-align: left">Metrics</th> <th style="text-align: right">Cramér’s V</th> <th style="text-align: right">p</th> <th style="text-align: right">df</th> <th style="text-align: right">N</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">JHOVE status vs VeraPDF parse errors</td> <td style="text-align: right">0.56</td> <td style="text-align: right">&lt;.001</td> <td style="text-align: right">2</td> <td style="text-align: right">88</td> </tr> <tr> <td style="text-align: left">JHOVE status vs VeraPDF warnings</td> <td style="text-align: right">0.51</td> <td style="text-align: right">&lt;.001</td> <td style="text-align: right">2</td> <td style="text-align: right">88</td> </tr> </tbody> </table> <p>The <em>p</em>-values were calculated from a <a href="https://en.wikipedia.org/wiki/Chi-squared_test">Chi-square test</a> of independence between the JHOVE and VeraPDF metrics. Put simply, for each metric pair, they denote the probability of the observed metric combinations occurring for the null hypothesis that the metrics are independent. Since both <em>p</em>-values are very small, this adds credence that the null hypothesis must be rejected, and that both metrics are in fact related. The Cramér’s <em>V</em> values express the strength of this association, confirming the relatively strong association between JHOVE’s validation status and the occurrence of VeraPDF parse errors and warnings. The <em>df</em> column shows the degrees of freedom of the Chi-square distribution.</p> <h2 id="simplifying-the-ground-truth">Simplifying the ground truth</h2> <p>Since the spreadsheet that is part of the PDF Testset includes information on rendering behaviour in Acrobat, we can use this as “ground truth” to test to what degree the JHOVE and VeraPDF results are indicative of rendering problems. The rendering outcomes in the spreadsheet<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">4</a></sup> are not based on any controlled vocabulary. For about half of the files, rendering behaviour is described as a simple “Yes” or “No”. For the remaining files, it is described in more elaborate terms, such as:</p> <ul> <li>“Yes, but missing content”</li> <li>“Yes. Wrong font”</li> <li>“Yes. Adobe tries to change something when opening”.</li> </ul> <p>This multitude of very specific (and often unique) values makes it hard to do any meaningful (quantitative) analysis. Because of this, I simplified the rendering outcomes into 3 discrete classes:</p> <ul> <li>Does not render</li> <li>Renders with issues</li> <li>Renders normally</li> </ul> <p><a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/misc/lindlar-tunnat-wilson/lindlar-tunnat-wilson-jhove-vera-rendering.csv">This file</a> contains, for all PDFs in the dataset, the simplified rendering outcomes, along with the JHOVE and VeraPDF results.</p> <h2 id="jhove-and-verapdf-results-vs-rendering-behaviour">JHOVE and VeraPDF results vs rendering behaviour</h2> <p>As a first step towards exploring any associations between the JHOVE and VeraPDF metrics and rendering behaviour, I used this file to create some more contingency tables. The first one shows the joint frequencies of rendering outcomes against JHOVE validation status:</p> <table> <thead> <tr> <th style="text-align: left">Rendering \ JHOVE status</th> <th style="text-align: right">Not well-formed</th> <th style="text-align: right">Well-Formed, but not valid</th> <th style="text-align: right">Well-Formed and valid</th> <th style="text-align: right">All</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">Does not render</td> <td style="text-align: right">31</td> <td style="text-align: right">1</td> <td style="text-align: right">0</td> <td style="text-align: right">32</td> </tr> <tr> <td style="text-align: left">Renders with issues</td> <td style="text-align: right">27</td> <td style="text-align: right">2</td> <td style="text-align: right">9</td> <td style="text-align: right">38</td> </tr> <tr> <td style="text-align: left">Renders normally</td> <td style="text-align: right">11</td> <td style="text-align: right">3</td> <td style="text-align: right">4</td> <td style="text-align: right">18</td> </tr> <tr> <td style="text-align: left">All</td> <td style="text-align: right">69</td> <td style="text-align: right">6</td> <td style="text-align: right">13</td> <td style="text-align: right">88</td> </tr> </tbody> </table> <p>Out of the 32 files in the “Does not render” category, JHOVE considers 31 as “Not well-formed”, and 1 as “Well-Formed, but not valid”. JHOVE was less successful at identifying files in the “Renders with issues” category. Out the 38 files, 9 (24%) are “Well-Formed and valid”. One striking result is that from the 18 files in the “Renders normally” category, the majority (11, or 61%) are “Not well-formed” according to JHOVE.</p> <p>Here’s the table for the veraPDF parse errors:</p> <table> <thead> <tr> <th style="text-align: left">Rendering \ VeraPDF status</th> <th style="text-align: right">Parse errors</th> <th style="text-align: right">No parse errors</th> <th style="text-align: right">All</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">Does not render</td> <td style="text-align: right">30</td> <td style="text-align: right">2</td> <td style="text-align: right">32</td> </tr> <tr> <td style="text-align: left">Renders with issues</td> <td style="text-align: right">31</td> <td style="text-align: right">7</td> <td style="text-align: right">38</td> </tr> <tr> <td style="text-align: left">Renders normally</td> <td style="text-align: right">7</td> <td style="text-align: right">11</td> <td style="text-align: right">18</td> </tr> <tr> <td style="text-align: left">All</td> <td style="text-align: right">68</td> <td style="text-align: right">20</td> <td style="text-align: right">88</td> </tr> </tbody> </table> <p>VeraPDF’s parse errors output is slightly less effective at identifying PDFs in the “Does not render” category: 30 of these result in parse errors, and 2 in no parse errors. From the 38 files in the “Renders with issues” category, 7 (18%) don’t result in parse errors. Interestingly, 61% of the files in the “Renders normally” category (11 out of 18) does not result in parse errors. This is a marked contrast with the JHOVE, which shows the opposite behaviour by considering 61% of these files “Not well-formed”.</p> <p>And finally here’s the table for VeraPDF’s warnings:</p> <table> <thead> <tr> <th style="text-align: left">Rendering \ VeraPDF status</th> <th style="text-align: right">Warnings</th> <th style="text-align: right">No warnings</th> <th style="text-align: right">All</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">Does not render</td> <td style="text-align: right">30</td> <td style="text-align: right">2</td> <td style="text-align: right">32</td> </tr> <tr> <td style="text-align: left">Renders with issues</td> <td style="text-align: right">31</td> <td style="text-align: right">7</td> <td style="text-align: right">38</td> </tr> <tr> <td style="text-align: left">Renders normally</td> <td style="text-align: right">8</td> <td style="text-align: right">10</td> <td style="text-align: right">18</td> </tr> <tr> <td style="text-align: left">All</td> <td style="text-align: right">69</td> <td style="text-align: right">19</td> <td style="text-align: right">88</td> </tr> </tbody> </table> <p>The pattern here is nearly identical to the parse errors<sup id="fnref:6"><a href="#fn:6" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>.</p> <p>As before, we can use the Cramer’s <em>V</em> statistic to express the strength of the associations between the tool metrics and the rendering results:</p> <table> <thead> <tr> <th style="text-align: left">Tool metric</th> <th style="text-align: right">Cramér’s V</th> <th style="text-align: right">p</th> <th style="text-align: right">df</th> <th style="text-align: right">N</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">JHOVE status</td> <td style="text-align: right">0.23</td> <td style="text-align: right">.011</td> <td style="text-align: right">4</td> <td style="text-align: right">88</td> </tr> <tr> <td style="text-align: left">VeraPDF parse errors</td> <td style="text-align: right">0.46</td> <td style="text-align: right">&lt;.001</td> <td style="text-align: right">2</td> <td style="text-align: right">88</td> </tr> <tr> <td style="text-align: left">VeraPDF warnings</td> <td style="text-align: right">0.41</td> <td style="text-align: right">&lt;.001</td> <td style="text-align: right">2</td> <td style="text-align: right">88</td> </tr> </tbody> </table> <p>The low <em>p</em>-values (again from a Chi-square test) indicate that all tool metrics are related to the rendering results. The strength of these associations is expressed through Cramér’s <em>V</em>. From the table we see that both VeraPDF metrics show a stronger association with rendering behaviour, compared to JHOVE. This pattern persists (although to a slightly lesser degree) if we adjust for the fact that JHOVE’s validation status can have 3 possible values, whereas the VerapDF metrics are simply binary flags<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">6</a></sup>. The large <em>V</em> value for the VeraPDF parse errors metric is mostly caused by the large overlap between the files in the “No parse errors” and “Renders normally” categories.</p> <h2 id="interpretation-and-conclusions">Interpretation and conclusions</h2> <h3 id="verapdf-vs-jhove">VeraPDF vs JHOVE</h3> <p>Given the higher Cramér’s <em>V</em> values, it would be tempting to conclude that VeraPDF is the “better” tool here compared to JHOVE. However, such a conclusion would be overly simplistic. The higher <em>V</em> values are primarily the result of the association between the “Renders normally” rendering class and VeraPDF’s “No parse errors” class<sup id="fnref:7"><a href="#fn:7" class="footnote" rel="footnote" role="doc-noteref">7</a></sup>. By contrast, the majority of these “Renders normally” PDFs fall into JHOVE’s “Not well-formed” class. This does not indicate any shortcoming on JHOVE’s part. Instead, it simply suggests that JHOVE is more sensitive to (minor) deviations from the PDF specification that don’t have a direct impact on rendering. This is exactly what we would expect if the rendering (viewer) application follows <a href="https://blog.dshr.org/2009/01/postels-law.html">Postel’s Law</a> (also known as the robustness principle), which states:</p> <blockquote> <p>be conservative in what you do, be liberal in what you accept from others</p> </blockquote> <p>Unlike most rendering applications (which are designed to be forgiving when it comes to small deviations from the specifications), JHOVE is meant to be picky. The primary purpose of VeraPDF’s parser is to provide the information needed to check conformance against PDF profiles (such as PDF/A and PDF/UA), and to extract technical features. It was <em>not</em> designed with JHOVE-style low-level (PDF 1.x … PDF 2.0) format validation in mind. Therefore, its behaviour may be more similar to the parsers that are used by rendering applications.</p> <h3 id="implications-for-digital-preservation-workflows">Implications for digital preservation workflows</h3> <p>Whether this behaviour is desirable within a digital preservation workflow highly depends on the specific purpose for which we want to “validate”. If the objective is to identify PDFs that are seriously malformed, VeraPDF’s parse status provides a simple and most likely sufficient indicator. JHOVE’s validation output gives more detail, but its interpretation can be difficult. Also, since JHOVE hasn’t yet been made up-to-date to <a href="https://jhove.openpreservation.org/modules/pdf/">neither PDF 1.7</a> nor PDF 2.0, JHOVE’s results for those PDF versions may be misleading.</p> <p>With that said, Postel’s law also implies that more subtle rendering issues may go unnoticed. Since the PDF standard is so feature-rich, PDF rendering software is typically designed to ignore any unknown features in a file. This can result in incomplete rendering, without any errors being reported in the process. Since VeraPDF’s parser appears to behave similarly to the parsers used by rendering applications, we should expect that subtle deviations from the standards may not result in any VeraPDF parse errors.</p> <h2 id="limitations">Limitations</h2> <p>The above analysis is a first attempt at exploring the association between VeraPDF’s parse errors and rendering behaviour. It’s important to keep in mind the following limitations:</p> <ul> <li>It is based exclusively on small, synthetic test files. It’s unclear how these results translate to ordinary PDFs “in the wild”, which are typically much more complex.</li> <li>The ground truth (rendering results) was taken at face value from the 2017 data set, which was based on a version of Adobe Acrobat that is now outdated. Results may be different for more recent versions, or other rendering applications.</li> </ul> <h2 id="jhove-verapdf-and-the-future-of-pdf-validation">JHOVE, VeraPDF and the future of PDF validation</h2> <p>While working on this write-up, the Open Preservation Foundation and the PDF Association <a href="https://openpreservation.org/news/development-preview-pdf-file-checker-based-on-the-arlington-pdf-model/">released</a> a first development preview of a new veraPDF-powered PDF-checker that is based on the <a href="https://github.com/pdf-association/arlington-pdf-model">Arlington PDF model</a>. This software is capable of analyzing PDF files against the full PDF 2.0 (and earlier) specifications. An earlier draft of this post included an overview of the current PDF validation software landscape, with some reflections on where things may be heading. However, I ultimately left this out, as it would make this post too unwieldy. I will most likely address this in another follow-up post.</p> <h2 id="feedback-welcome">Feedback welcome</h2> <p>As always, feedback to this post is highly appreciated. This includes (but is not limited to):</p> <ul> <li>The idea of using VeraPDF parse status a rough proxy for PDF validity.</li> <li>Comments, suggestions or criticism related to the methodology I used for analyzing the Synthetic PDF Testset.</li> </ul> <h2 id="further-resources">Further resources</h2> <ul> <li> <p><a href="https://github.com/KBNLresearch/pdf-characterisation">Github repo with analysis scripts and raw data</a></p> </li> <li> <p><a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/scripts/jhove-verapdf-validation-run.py">Script for running JHOVE and VeraPDF</a></p> </li> <li> <p><a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/scripts/jhove-verapdf-validation-analyze.py">Analysis script (contingency tables + Cramér’s V calculation</a></p> </li> <li> <p><a href="https://www.radar-service.eu/radar/en/dataset/JtlOdwQquZWDqQdq">Synthetic PDF Testset for File Format Validation</a></p> </li> <li> <p><a href="https://phaidra.univie.ac.at/detail/o:931074">A PDF Test-Set for Well-Formedness Validation in JHOVE - The Good, the Bad and the Ugly</a> - 2017 iPRES paper by Lindlar, Tunnat &amp; Wilson</p> </li> <li> <p><a href="https://doi.org/10.2218/ijdc.v12i2.578">How Valid is your Validation? A Closer Look Behind the Curtain of JHOVE</a> - 2017 International Journal of Digital Curation paper by Lindlar &amp; Tunnat.</p> </li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>Actually, feature extraction is not really needed for the purpose of this analysis. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>One PDF caused an exception in JHOVE, which resulted in the “unknown” validation status. For convenience I recoded this to “Not well-formed” for all analyses here. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>Using <a href="https://doi.org/10.1016%2Fj.jkss.2012.10.002">Bergsma’s bias-corrected version</a> (<a href="https://stats.lse.ac.uk/bergsma/pdf/cramerV3.pdf">non-paywalled copy</a>), following the implemention given <a href="https://stackoverflow.com/questions/20892799/using-pandas-calculate-cram%c3%a9rs-coefficient-matrix/39266194#39266194">here</a>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>Column “Adobe Professional XI Pro (11.0.15) - can file be opened?”. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:6"> <p>The only difference is one file in the “Renders normally” category that resulted in a warning, without resulting in any parse errors. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>As a test I re-calculated the statistics after lumping JHOVE’s ‘Well-Formed, but not valid’ and “Not well-formed” classes. This did result in a slightly higher value of <em>V</em> (0.28). Lumping JHOVE’s “Well-Formed, but not valid” and “Well-Formed and valid” classes raised <em>V</em> even further (0.32), but this is still less than the values for the VeraPDF metrics. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:7"> <p>And to a lesser extent VeraPDF’s “No warnings” class. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2023/06/29/verapdf-parse-status-as-a-proxy-for-rendering https://bitsgalore.org/2023/06/29/verapdf-parse-status-as-a-proxy-for-rendering 2023-06-29T00:00:00+02:00 <figure class="image"> <img src="/images/2023/05/Rock_em_Sock_em_Robots_Game.jpg" alt="Photo of a red toy robot and a similar looking blue toy robot in a boxing ring. Both robots face each other in a threatening stance." /> <figcaption><a href="https://commons.wikimedia.org/wiki/File:Rock_%27em_Sock_%27em_Robots_Game.jpg">"Rock 'em Sock 'em Robots Game"</a> by Lorie Shaull, used under <a href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">CC BY-SA 4.0</a>, via Wikimedia Commons.</figcaption> </figure> <p>The PDF format has a number of features that don’t sit well with the aims of long-term preservation and accessibility. This includes encryption and password protection, external dependencies (e.g. fonts that are not embedded in a document), and reliance on external software. In this post I’ll review to what extent such features can be detected using <a href="https://verapdf.org/">VeraPDF</a> and <a href="https://jhove.openpreservation.org/">JHOVE</a>. It further builds on earlier work I did on this subject between 2012 and 2017.</p> <!-- more --> <h2 id="context-of-this-work">Context of this work</h2> <p>In 2019 the KB published its current <a href="https://www.kb.nl/en/file-download/download/public/845">preservation policy</a> for its digital collections. The policy expresses a number of preservation-related goals for the upcoming years. One of these is a (gradual) move from pure <a href="https://blogs.loc.gov/thesignal/2011/09/b-is-for-bit-preservation/">bit preservation</a> towards full functional preservation. To this end, the KB has defined three “knowledge levels”, which are linked to file formats<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>:</p> <ol> <li>For <strong>stored</strong> formats, only bit preservation is done, without any formal format identification (no PRONOM identifier).</li> <li>For <strong>identified</strong> formats, formal format identification has resulted in a PRONOM identifier.</li> <li>For <strong>known</strong> formats, format validation has been performed, and technical metadata have been extracted. This information can then be used to identify preservation-related risks at the level of individual files, and actions to mitigate these risks can be taken if necessary.</li> </ol> <p>The move from bit preservation to functional preservation involves raising the “knowledge level” status of the formats in our digital collections from “stored” (which is the current situation) to “identified” or, ideally, “known”. This development coincides with the ongoing migration of these collections to our new <a href="https://www.kb.nl/en/actueel/nieuws/dutch-national-library-steps-new-future-digital-archiving">Rosetta archiving system</a>.</p> <h2 id="pdf-preservation-risks">PDF preservation risks</h2> <p>Being one of the <a href="/2015/04/29/top-50-file-formats-in-the-kb-e-depot">most prevalent formats</a> in our collections, it comes as no surprise that PDF is one of the first formats for which we want to raise the current knowledge level. This is why my colleagues at our digital preservation department asked me take a new dive into PDF features that are potential preservation risks, and, more importantly, how to detect such features with existing software tools.</p> <p>As long-term readers may remember, risk detection in PDF isn’t an unfamiliar subject to me. Between 2012 and 2014 I experimented with detecting “risky” PDF features using the open-source <a href="https://pdfbox.apache.org/">Apache PDFBox</a> software<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>. This work was done as part of the <a href="https://scape-project.eu/">SCAPE project</a>. In 2017 I <a href="/2017/06/01/policy-based-assessment-with-verapdf-a-first-impression">repeated some of the SCAPE-era experiments</a>, this time using <a href="https://verapdf.org/">VeraPDF</a>. For various reasons that work never saw a proper follow-up, but six years onwards it’s finally on the agenda once again! In the meantime others have produced relevant related work as well. The <a href="https://docs.google.com/spreadsheets/d/1eW7R8yACBciNimr16Z2ptC7fs1FlmZMnzdtG_DHBuD4/edit?usp=sharing">PDF Significant Properties Spreadsheet</a> by Tyler Thorsted (currently at Brigham Young University) deserves a special mention here. It gives an overview of metadata fields and technical properties that are reported by 12 different software tools.</p> <h2 id="pdf-features-and-risks">PDF features and risks</h2> <p>Before going any further, it’s important to be clear about our definition of preservation risks, and how they are related to PDF features. For this analysis I largely followed the earlier work that I mentioned above as a starting point. This means the focus is mostly on the following features and their associated risks:</p> <ul> <li>Encryption and security-related features. The associated risks are restricted functionality (e.g. text access, copying, printing), or files that are inaccessible altogether (because they can only be opened using a password).</li> <li>External dependencies, such as fonts that are not embedded, or dependencies on external documents. The associated risk is that that documents may not render as originally intended if the external resources are not available.</li> <li>Multimedia features, including 3-D content. The associated risk is that the multimedia content may not render without dedicated software.</li> <li>File attachments. The associated risk is that attached files may require external software to render.</li> <li>JavaScript. This presents various risks, which are mostly security-related.</li> </ul> <p>This list is not exhaustive, but it provides a useful starting point. Deviations from the format specification (PDF validity) can also pose potential preservation risks. <a href="https://blog.dshr.org/2009/01/postels-law.html">Opinions are divided</a> about the importance of format validity in actual practice. PDF format validation is a huge subject in its own right<sup id="fnref:7"><a href="#fn:7" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>, which is out of the scope of the current analysis.</p> <h2 id="verapdf-vs-jhove">VeraPDF vs JHOVE</h2> <p>The <a href="/2017/06/01/policy-based-assessment-with-verapdf-a-first-impression">2017 analysis</a> already showed that VeraPDF was able to detect most risk-associated PDF features, which made the inclusion of VeraPDF an obvious choice for this follow-up. VeraPDF is currently not included in Rosetta, and my colleagues wondered to what extent <a href="https://jhove.openpreservation.org/">JHOVE</a> (which is part of Rosetta’s default setup) would be up to the job. So, in the remainder of this blog post I will present a comparison of VeraPDF and JHOVE.</p> <p>It’s important to mention that VeraPDF and JHOVE have different (but partially overlapping) scopes and functionalities. VeraPDF was primarily written to test for conformance against the various <a href="https://en.wikipedia.org/wiki/PDF/A">PDF/A</a> profiles, with recent versions also supporting (partial) conformance testing for <a href="https://en.wikipedia.org/wiki/PDF/UA">PDF/UA</a>. It does not (or at best only to a limited degree) validate the lower-level data structures that make up a PDF file. The latter is the primary domain of JHOVE’s PDF module, which was designed for “full on” PDF validation<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>. The main overlap between VeraPDF and JHOVE lies in their ability to extract metadata and technical features, and for the current analysis I have only looked at this particular functionality of both tools.</p> <p>The following table shows the evaluated VeraPDF and JHOVE versions:</p> <table> <thead> <tr> <th style="text-align: left">Software</th> <th style="text-align: left">Version</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">VeraPDF</td> <td style="text-align: left">1.22.3</td> </tr> <tr> <td style="text-align: left">JHOVE</td> <td style="text-align: left">1.28.0</td> </tr> </tbody> </table> <p>By default, VeraPDF’s feature extraction mode only extracts metadata from a PDF’s document information dictionary. This is not sufficient for the purpose of this analysis, so I updated VeraPDF’s configuration and enabled <em>all</em> feature extraction types<sup id="fnref:8"><a href="#fn:8" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>.</p> <h2 id="methods-and-test-data">Methods and test data</h2> <p>I used 3 different data sets for this analysis. First, used the <a href="https://github.com/openpreserve/format-corpus/tree/master/pdfCabinetOfHorrors">PDF Cabinet of Horrors corpus</a>. This is a small, hand-curated corpus of PDFs that is part of the <a href="https://github.com/openpreserve/format-corpus/tree/master">Open Preservation Foundation’s file format corpus</a>. Since this is an annotated dataset with files that have known features (such as security-related features, non-embedded fonts, and multimedia), it provides excellent ground truth. I ran VeraPDF and JHOVE on all files in this data set, and scrutinised the resulting output files in detail. The main goal of this part of the analysis (which is presented in detail in the next section) was twofold:</p> <ol> <li>Establish to what extent VeraPDF and JHOVE are able to detect the known features in these files.</li> <li>Identify broad patterns in VeraPDF’s and JHOVE’s output that are associated with potential preservation risks.</li> </ol> <p>The results then served as input for a second test, which is based on a subset of files from the (now defunct) <a href="https://web.archive.org/web/20130503115947/http://acroeng.adobe.com/wp/">Adobe Acrobat Engineering website</a>.</p> <p>Finally, I tested how well VeraPDF and JHOVE are able to handle PDF 2.0 documents by running them on the PDF Association’s <a href="https://github.com/pdf-association/pdf20examples">PDF 2.0 examples</a> dataset.</p> <h2 id="advance-warning">Advance warning</h2> <p>The level of detail in the following sections may be too much for some (most?) readers to digest, except perhaps for the most hardcore PDF freaks (you know who you are!). This applies in particular to the “Horror Corpus” section. If this is the case, you may want to skip right to the “Discussion” section from here. Otherwise, now is the time to fasten your seatbelts!</p> <h2 id="analysis-of-horror-corpus">Analysis of Horror Corpus</h2> <p>I will start with a rather detailed analysis of the <a href="https://github.com/openpreserve/format-corpus/tree/master/pdfCabinetOfHorrors">Horror Corpus</a>, as this is an annotated corpus of files with known features. Each of the following sub-sections covers one “risky” feature, which is demonstrated by one or more files in the Horror Corpus. For each feature, I show how it is represented in VeraPDF’s and JHOVE’s output (if it is represented at all). Even though this only covers a limited selection of “risky” features, this analysis is useful to get a first impression of the differences between VerapDF and JHOVE. It also enables us to identify which parts of the output of both tools are of potential interest for further perusal.</p> <h3 id="open-password">Open password</h3> <h4 id="test-files">Test files</h4> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">Description</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/encryption_openpassword.pdf?raw=true">encryption_openpassword.pdf</a></td> <td style="text-align: left">Opening the document requires password.</td> </tr> </tbody> </table> <figure class="image"> <img src="/images/2015/07/openpassword.png" alt="Screenshot of dialog window that asks to enter a Document Open password." /> <figcaption>Adobe Acrobat dialog on opening a PDF with an open password.</figcaption> </figure> <h4 id="verapdf">VeraPDF</h4> <p>For this file, VeraPDF reports an exception (full output <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/encryption_openpassword-vera.xml">here</a>):</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;taskResult</span> <span class="na">type=</span><span class="s">"PARSE"</span> <span class="na">isExecuted=</span><span class="s">"true"</span> <span class="na">isSuccess=</span><span class="s">"false"</span><span class="nt">&gt;</span> <span class="nt">&lt;duration</span> <span class="na">start=</span><span class="s">"1683818755295"</span> <span class="na">finish=</span><span class="s">"1683818755343"</span><span class="nt">&gt;</span>00:00:00.048<span class="nt">&lt;/duration&gt;</span> <span class="nt">&lt;exceptionMessage&gt;</span>Exception: The PDF stream appears to be encrypted. caused by exception: Reader::init(...)encrypted pdf is not supported<span class="nt">&lt;/exceptionMessage&gt;</span> <span class="nt">&lt;/taskResult&gt;</span> </code></pre></div></div> <h4 id="jhove">JHOVE</h4> <p>Unlike VeraPDF, JHOVE appears to be able to parse the file in spite of the open password. Its <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/encryption_openpassword-jhove.xml">output</a> contains an “Encryption” element with several child elements:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>Encryption<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"List"</span> <span class="na">type=</span><span class="s">"Property"</span><span class="nt">&gt;</span> <span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>SecurityHandler<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"Scalar"</span> <span class="na">type=</span><span class="s">"String"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>Standard<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> <span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>EFF<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"Scalar"</span> <span class="na">type=</span><span class="s">"String"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>Standard<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> <span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>Algorithm<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"Scalar"</span> <span class="na">type=</span><span class="s">"String"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>Document-defined<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> <span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>KeyLength<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"Scalar"</span> <span class="na">type=</span><span class="s">"Integer"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>128<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> <span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>StandardSecurityHandler<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"List"</span> <span class="na">type=</span><span class="s">"Property"</span><span class="nt">&gt;</span> <span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>UserAccess<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"List"</span> <span class="na">type=</span><span class="s">"String"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>Print<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;value&gt;</span>Modify<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;value&gt;</span>Extract<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;value&gt;</span>Add/modify annotations/forms<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;value&gt;</span>Fill interactive form fields<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;value&gt;</span>Extract for accessibility<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;value&gt;</span>Print high quality<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> <span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>Revision<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"Scalar"</span> <span class="na">type=</span><span class="s">"Integer"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>4<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> <span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>OwnerString<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"Scalar"</span> <span class="na">type=</span><span class="s">"String"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>0x03a59d10aae3b50f1a30c34fbb1be09ababfc1fb19f0d3491ee84c1752671918<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> <span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>UserString<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"Scalar"</span> <span class="na">type=</span><span class="s">"String"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>0xf240da93aa83bb9f336cf249812c4db700000000000000000000000000000000<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> </code></pre></div></div> <p>Even though it’s helpful that JHOVE shows that this file contains security-related features, the above output doesn’t provide any direct clue of the more specific open password feature<sup id="fnref:9"><a href="#fn:9" class="footnote" rel="footnote" role="doc-noteref">6</a></sup>. This is unfortunate, given that open passwords are one of the most serious PDF preservation risks.</p> <h3 id="copy-printing-and-text-access-passwords">Copy, printing and text access passwords</h3> <h4 id="test-files-1">Test files</h4> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">Description</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/encryption_nocopy.pdf?raw=true">encryption_nocopy.pdf</a></td> <td style="text-align: left">Copying document contents requires password.</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/encryption_noprinting.pdf?raw=true">encryption_noprinting.pdf</a></td> <td style="text-align: left">Printing requires password.</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/encryption_notextaccess.pdf?raw=true">encryption_notextaccess.pdf</a></td> <td style="text-align: left">Text access (e.g. by a screen reader) requires password.</td> </tr> </tbody> </table> <figure class="image"> <img src="/images/2015/07/printpassword.png" alt="Screenshot of Adobe Acrobat showing a 'you cannot print this document' message." /> <figcaption>Adobe Acrobat's rendering of a PDF with that requires a password for printing.</figcaption> </figure> <h4 id="verapdf-1">VeraPDF</h4> <p>Information on usage restrictions can be found in the <em>documentSecurity</em> element of VeraPDF’s output. Below is an example (taken from the <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/encryption_nocopy-vera.xml">output</a> of the copy-protected file):</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;documentSecurity&gt;</span> <span class="nt">&lt;filter&gt;</span>Standard<span class="nt">&lt;/filter&gt;</span> <span class="nt">&lt;version&gt;</span>4<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;length&gt;</span>128<span class="nt">&lt;/length&gt;</span> <span class="nt">&lt;ownerKey&gt;</span>5F7FA03AD5A40C66...<span class="nt">&lt;/ownerKey&gt;</span> <span class="nt">&lt;userKey&gt;</span>A572ACDFF8B3DC74...<span class="nt">&lt;/userKey&gt;</span> <span class="nt">&lt;encryptMetadata&gt;</span>true<span class="nt">&lt;/encryptMetadata&gt;</span> <span class="nt">&lt;printAllowed&gt;</span>true<span class="nt">&lt;/printAllowed&gt;</span> <span class="nt">&lt;printDegradedAllowed&gt;</span>true<span class="nt">&lt;/printDegradedAllowed&gt;</span> <span class="nt">&lt;changesAllowed&gt;</span>true<span class="nt">&lt;/changesAllowed&gt;</span> <span class="nt">&lt;modifyAnnotationsAllowed&gt;</span>true<span class="nt">&lt;/modifyAnnotationsAllowed&gt;</span> <span class="nt">&lt;fillingSigningAllowed&gt;</span>true<span class="nt">&lt;/fillingSigningAllowed&gt;</span> <span class="nt">&lt;documentAssemblyAllowed&gt;</span>false<span class="nt">&lt;/documentAssemblyAllowed&gt;</span> <span class="nt">&lt;extractContentAllowed&gt;</span>false<span class="nt">&lt;/extractContentAllowed&gt;</span> <span class="nt">&lt;extractAccessibilityAllowed&gt;</span>true<span class="nt">&lt;/extractAccessibilityAllowed&gt;</span> <span class="nt">&lt;/documentSecurity&gt;</span> </code></pre></div></div> <p>Here, the “false” value of the <em>extractContentAllowed</em> element indicates this file is copy-protected. The <em>printAllowed</em> and <em>printDegradedAllowed</em> elements indicate whether printing is allowed (example <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/encryption_noprinting-vera.xml">here</a>), and “false” values of <em>extractContentAllowed</em> and <em>extractAccessibilityAllowed</em> indicate a text access password (example <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/encryption_notextaccess-vera.xml">here</a>).</p> <h4 id="jhove-1">JHOVE</h4> <p>For JHOVE, the <em>UserAccess</em> property (which is a child element of the <em>Encryption</em> property) provides information about usage restrictions. Below for the copy-restricted document (full output <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/encryption_nocopy-jhove.xml">here</a>):</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>UserAccess<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"List"</span> <span class="na">type=</span><span class="s">"String"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>Print<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;value&gt;</span>Modify<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;value&gt;</span>Add/modify annotations/forms<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;value&gt;</span>Fill interactive form fields<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;value&gt;</span>Extract for accessibility<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;value&gt;</span>Print high quality<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> </code></pre></div></div> <p>And here for the print-restricted document (full output <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/encryption_noprinting-jhove.xml">here</a>):</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>UserAccess<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"List"</span> <span class="na">type=</span><span class="s">"String"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>Modify<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;value&gt;</span>Extract<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;value&gt;</span>Add/modify annotations/forms<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;value&gt;</span>Fill interactive form fields<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;value&gt;</span>Extract for accessibility<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> </code></pre></div></div> <p>The above examples show that JHOVE only reports user access types that are <em>permitted</em>. This means that if we want to verify if a document is print-restricted, we need to check for the <em>absence</em> of the values <em>Print</em> and <em>Print high quality</em>. This is impractical, because it assumes the user has <em>a priori</em> knowledge of all possible values that JHOVE is capable of reporting (which are also undocumented). VeraPDF’s makes this considerably easier by always reporting <em>all</em> user access types and their respective values.</p> <h3 id="multimedia">Multimedia</h3> <h4 id="test-files-2">Test files</h4> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">Description</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/embedded_video_quicktime.pdf?raw=true">embedded_video_quicktime.pdf</a></td> <td style="text-align: left">Contains embedded Quicktime movie.</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/embedded_video_avi.pdf?raw=true">embedded_video_avi.pdf</a></td> <td style="text-align: left">Contains embedded AVI movie.</td> </tr> </tbody> </table> <figure class="image"> <img src="/images/2015/07/embeddquicktime.png" alt="Screenshot of Adobe Acrobat's rendering of a PDF with an embedded Quicktime movie. As Acrobat cannot play multimedia content natively, it shows a dialog saying 'The media requires an additional player. Please click Get Media Player to download the correct media player'." /> <figcaption>Adobe Acrobat's rendering of a PDF with an embedded Quicktime movie.</figcaption> </figure> <h4 id="verapdf-2">VeraPDF</h4> <p>For both files the presence of the multimedia content is indicated by two elements in <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/embedded_video_quicktime-vera.xml">VeraPDF’s output</a>. First, the <em>actions</em> element contains a reference to a Rendition action:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;action</span> <span class="na">type=</span><span class="s">"Rendition"</span><span class="nt">&gt;</span> <span class="nt">&lt;location&gt;</span>Annotation<span class="nt">&lt;/location&gt;</span> <span class="nt">&lt;/action&gt;</span> </code></pre></div></div> <p>Furthermore, the <em>annotations</em> element contains a reference to a Screen annotation:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;annotation</span> <span class="na">id=</span><span class="s">"annotIndir35"</span><span class="nt">&gt;</span> <span class="nt">&lt;subType&gt;</span>Screen<span class="nt">&lt;/subType&gt;</span> <span class="nt">&lt;rectangle</span> <span class="na">lly=</span><span class="s">"360.647"</span> <span class="na">llx=</span><span class="s">"73.180"</span> <span class="na">urx=</span><span class="s">"393.180"</span> <span class="na">ury=</span><span class="s">"600.647"</span><span class="nt">&gt;&lt;/rectangle&gt;</span> <span class="nt">&lt;width&gt;</span>320.000<span class="nt">&lt;/width&gt;</span> <span class="nt">&lt;height&gt;</span>240.000<span class="nt">&lt;/height&gt;</span> <span class="nt">&lt;resources&gt;</span> <span class="nt">&lt;xobject</span> <span class="na">id=</span><span class="s">"xobjIndir39"</span><span class="nt">&gt;&lt;/xobject&gt;</span> <span class="nt">&lt;/resources&gt;</span> <span class="nt">&lt;invisible&gt;</span>false<span class="nt">&lt;/invisible&gt;</span> <span class="nt">&lt;hidden&gt;</span>false<span class="nt">&lt;/hidden&gt;</span> <span class="nt">&lt;print&gt;</span>true<span class="nt">&lt;/print&gt;</span> <span class="nt">&lt;noZoom&gt;</span>false<span class="nt">&lt;/noZoom&gt;</span> <span class="nt">&lt;noRotate&gt;</span>false<span class="nt">&lt;/noRotate&gt;</span> <span class="nt">&lt;noView&gt;</span>false<span class="nt">&lt;/noView&gt;</span> <span class="nt">&lt;readOnly&gt;</span>false<span class="nt">&lt;/readOnly&gt;</span> <span class="nt">&lt;locked&gt;</span>false<span class="nt">&lt;/locked&gt;</span> <span class="nt">&lt;toggleNoView&gt;</span>false<span class="nt">&lt;/toggleNoView&gt;</span> <span class="nt">&lt;lockedContents&gt;</span>false<span class="nt">&lt;/lockedContents&gt;</span> <span class="nt">&lt;/annotation&gt;</span> </code></pre></div></div> <h4 id="jhove-2">JHOVE</h4> <p>The <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/embedded_video_quicktime-jhove.xml">JHOVE output</a> only refers to the Screen annotation for these files (JHOVE doesn’t report on actions):</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>Annotation<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"List"</span> <span class="na">type=</span><span class="s">"Property"</span><span class="nt">&gt;</span> <span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>Subtype<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"Scalar"</span> <span class="na">type=</span><span class="s">"String"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>Screen<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> :: <span class="nt">&lt;/property&gt;</span> </code></pre></div></div> <p>Besides Rendition actions and Screen annotations, there are several other action and annotation types that indicate multimedia content, and we’ll see some examples later on in the analysis of the Adobe Acrobat Engineering files.</p> <h3 id="javascript">JavaScript</h3> <h4 id="test-files-3">Test files</h4> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">Description</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/javascript.pdf?raw=true">javascript.pdf</a></td> <td style="text-align: left">Contains JavaScript.</td> </tr> </tbody> </table> <h4 id="verapdf-3">VeraPDF</h4> <p>The <em>actions</em> element of <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/javascript-vera.xml">VeraPDF’s output</a> contains the following JavaScript action reference<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">7</a></sup>:</p> <p>Result:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;action</span> <span class="na">type=</span><span class="s">"JavaScript"</span><span class="nt">&gt;</span> <span class="nt">&lt;location&gt;</span>Document<span class="nt">&lt;/location&gt;</span> <span class="nt">&lt;/action&gt;</span> </code></pre></div></div> <h4 id="jhove-3">JHOVE</h4> <p>I couldn’t find any JavaScript reference in the <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/javascript-jhove.xml">JHOVE output</a> for the same file.</p> <h3 id="font-embedding">Font embedding</h3> <h4 id="test-files-4">Test files</h4> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">Description</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/text_only_fontsNotEmbedded.pdf?raw=true">text_only_fontsNotEmbedded.pdf</a></td> <td style="text-align: left">Only uses fonts that are not embedded.</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/text_only_fontsEmbeddedAll.pdf?raw=true">text_only_fontsEmbeddedAll.pdf</a></td> <td style="text-align: left">Only uses fonts that are embedded.</td> </tr> </tbody> </table> <figure class="image"> <img src="/images/2015/07/fontslinux.png" alt="Screenshot of Adobe Acrobat rendering a PDF with a font whose glyphs are placed at odd positions." /> <figcaption>Adobe Acrobat's rendering of a PDF that uses non-embedded font that is not available on the target machine.</figcaption> </figure> <h4 id="verapdf-4">VeraPDF</h4> <p>The <em>fonts</em> element in VeraPDF’s output contains child elements for each font that is used in a document. For the PDF that only uses fonts that are not embedded, this results in the following output (full output file <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/text_only_fontsNotEmbedded-vera.xml">here</a>):</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;font</span> <span class="na">id=</span><span class="s">"fntIndir30"</span><span class="nt">&gt;</span> <span class="nt">&lt;type&gt;</span>TrueType<span class="nt">&lt;/type&gt;</span> <span class="nt">&lt;baseFont&gt;</span>TimesNewRomanPSMT<span class="nt">&lt;/baseFont&gt;</span> <span class="nt">&lt;firstChar&gt;</span>32<span class="nt">&lt;/firstChar&gt;</span> <span class="nt">&lt;lastChar&gt;</span>255<span class="nt">&lt;/lastChar&gt;</span> <span class="nt">&lt;encoding&gt;</span>WinAnsiEncoding<span class="nt">&lt;/encoding&gt;</span> <span class="nt">&lt;fontDescriptor&gt;</span> <span class="nt">&lt;subset&gt;</span>false<span class="nt">&lt;/subset&gt;</span> <span class="nt">&lt;fontName&gt;</span>TimesNewRomanPSMT<span class="nt">&lt;/fontName&gt;</span> <span class="nt">&lt;fontFamily&gt;</span>Times New Roman<span class="nt">&lt;/fontFamily&gt;</span> <span class="nt">&lt;fontStretch&gt;</span>Normal<span class="nt">&lt;/fontStretch&gt;</span> <span class="nt">&lt;fontWeight&gt;</span>400.000<span class="nt">&lt;/fontWeight&gt;</span> <span class="nt">&lt;fixedPitch&gt;</span>false<span class="nt">&lt;/fixedPitch&gt;</span> <span class="nt">&lt;serif&gt;</span>true<span class="nt">&lt;/serif&gt;</span> <span class="nt">&lt;symbolic&gt;</span>false<span class="nt">&lt;/symbolic&gt;</span> <span class="nt">&lt;script&gt;</span>false<span class="nt">&lt;/script&gt;</span> <span class="nt">&lt;nonsymbolic&gt;</span>true<span class="nt">&lt;/nonsymbolic&gt;</span> <span class="nt">&lt;italic&gt;</span>false<span class="nt">&lt;/italic&gt;</span> <span class="nt">&lt;allCap&gt;</span>false<span class="nt">&lt;/allCap&gt;</span> <span class="nt">&lt;smallCap&gt;</span>false<span class="nt">&lt;/smallCap&gt;</span> <span class="nt">&lt;forceBold&gt;</span>false<span class="nt">&lt;/forceBold&gt;</span> <span class="nt">&lt;fontBBox</span> <span class="na">lly=</span><span class="s">"-307.000"</span> <span class="na">llx=</span><span class="s">"-568.000"</span> <span class="na">urx=</span><span class="s">"2000.000"</span> <span class="na">ury=</span><span class="s">"1007.000"</span><span class="nt">&gt;&lt;/fontBBox&gt;</span> <span class="nt">&lt;italicAngle&gt;</span>0.000<span class="nt">&lt;/italicAngle&gt;</span> <span class="nt">&lt;ascent&gt;</span>891.000<span class="nt">&lt;/ascent&gt;</span> <span class="nt">&lt;descent&gt;</span>-216.000<span class="nt">&lt;/descent&gt;</span> <span class="nt">&lt;leading&gt;</span>0.000<span class="nt">&lt;/leading&gt;</span> <span class="nt">&lt;capHeight&gt;</span>1000.000<span class="nt">&lt;/capHeight&gt;</span> <span class="nt">&lt;xHeight&gt;</span>1000.000<span class="nt">&lt;/xHeight&gt;</span> <span class="nt">&lt;stemV&gt;</span>82.000<span class="nt">&lt;/stemV&gt;</span> <span class="nt">&lt;stemH&gt;</span>0.000<span class="nt">&lt;/stemH&gt;</span> <span class="nt">&lt;averageWidth&gt;</span>0.000<span class="nt">&lt;/averageWidth&gt;</span> <span class="nt">&lt;maxWidth&gt;</span>0.000<span class="nt">&lt;/maxWidth&gt;</span> <span class="nt">&lt;missingWidth&gt;</span>0.000<span class="nt">&lt;/missingWidth&gt;</span> <span class="nt">&lt;embedded&gt;</span>false<span class="nt">&lt;/embedded&gt;</span> <span class="nt">&lt;/fontDescriptor&gt;</span> <span class="nt">&lt;/font&gt;</span> </code></pre></div></div> <p>Here, the value of <em>embedded</em> in the <em>fontDescriptor</em> sub-element indicates whether a font is embededded or not (in this example it isn’t). So, to check if a PDF uses fonts that are not embedded, one can simply iterate over all <em>font</em> elements and check the value of <em>/fontDescriptor/embedded</em> for each of these.</p> <h4 id="jhove-4">JHOVE</h4> <p>The <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/text_only_fontsNotEmbedded-jhove.xml">JHOVE output for the same PDF</a> does contain several font-related properties, but none of them are related to embedding. So, I initially assumed that JHOVE simply didn’t report this. However, when I looked at <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/text_only_fontsEmbeddedAll-jhove.xml">JHOVE’s output</a> for the PDF with only embedded fonts, I noticed this property (part of a <em>FontDescriptor</em> element):</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>FontFile2<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"Scalar"</span> <span class="na">type=</span><span class="s">"Boolean"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>true<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> </code></pre></div></div> <p>Since JHOVE doesn’t report the <em>FontFile2</em> property for the PDF without embedded fonts, I wondered if this was a coincidence. As the meaning of this property (or any of JHOVE’s properties for that matter) is not documented, I took a peek at <a href="https://github.com/openpreserve/jhove/blob/94da570caa55759354fa6fcd50e4ea7edbba1e7d/jhove-modules/pdf-hul/src/main/java/edu/harvard/hul/ois/jhove/module/PdfModule.java#L3830">JHOVE’s source code</a>. This revealed that the <em>FontFile2</em> property originates from the “font descriptor” dictionary, which is documented in section 9.8 (Font Descriptors) of the <a href="https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/PDF32000_2008.pdf">ISO 32000-1 PDF specification</a>. Here we can see (Table 122) that the keys <em>FontFile</em>, <em>FontFile2</em> and <em>FontFile3</em> (which also exist as separate JHOVE properties) all indicate embedded fonts.</p> <p>This means we can actually use JHOVE’s output to check for font embedding, but to identify a PDF with one or more fonts that are not embedded, one needs to iterate over all font property groups, and then check for the <em>absence</em> of either of three separate properties (<em>FontFile</em>, <em>FontFile2</em> and <em>FontFile3</em>). None of these properties are documented. This is not ideal to begin with, and made worse by JHOVE’s rather labyrinthine output format.</p> <h3 id="file-attachments">File attachments</h3> <h4 id="test-files-5">Test files</h4> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">Description</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/fileAttachment.pdf?raw=true">fileAttachment.pdf</a></td> <td style="text-align: left">Contains file attachment that uses EmbeddedFiles entry.</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/fileAttachment_fileAttachmentAnnotation.pdf?raw=true">fileAttachment_fileAttachmentAnnotation.pdf</a></td> <td style="text-align: left">Contains file attachment that uses File Attachment Annotation.</td> </tr> </tbody> </table> <p>File attachments in PDF can be implemented in two different ways, using either an <em>EmbeddedFiles</em> entry in the document’s name dictionary, or a File Attachment Annotation.</p> <h4 id="verapdf-5">VeraPDF</h4> <p>For the sample file with the <em>EmbeddedFiles</em> entry, <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/fileAttachment-vera.xml">VeraPDF’s output</a> contains an <em>embeddedFiles</em> element with one or more child elements:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;embeddedFiles&gt;</span> <span class="nt">&lt;embeddedFile</span> <span class="na">id=</span><span class="s">"file1"</span><span class="nt">&gt;</span> <span class="nt">&lt;fileName&gt;</span>KSBASE.WQ2<span class="nt">&lt;/fileName&gt;</span> <span class="nt">&lt;description&gt;&lt;/description&gt;</span> <span class="nt">&lt;filter&gt;</span>FlateDecode<span class="nt">&lt;/filter&gt;</span> <span class="nt">&lt;creationDate&gt;</span>2012-11-23T15:40:38.000+01:00<span class="nt">&lt;/creationDate&gt;</span> <span class="nt">&lt;modDate&gt;</span>2012-11-19T12:35:10.000Z<span class="nt">&lt;/modDate&gt;</span> <span class="nt">&lt;checkSum&gt;</span>)#x000002Afi˛½�flô#x000004‘SèŠ-¡<span class="nt">&lt;/checkSum&gt;</span> <span class="nt">&lt;size&gt;</span>20668<span class="nt">&lt;/size&gt;</span> <span class="nt">&lt;/embeddedFile&gt;</span> <span class="nt">&lt;/embeddedFiles&gt;</span> </code></pre></div></div> <p>For the sample file with the File Attachment Annotation, the <em>annotations</em> element in VeraPDF’s <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/fileAttachment_fileAttachmentAnnotation-vera.xml">output</a> contains an <em>annotation</em> child element that has “FileAttachment” as its subtype:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;annotation</span> <span class="na">id=</span><span class="s">"annotIndir26"</span><span class="nt">&gt;</span> <span class="nt">&lt;subType&gt;</span>FileAttachment<span class="nt">&lt;/subType&gt;</span> <span class="nt">&lt;rectangle</span> <span class="na">lly=</span><span class="s">"562.840"</span> <span class="na">llx=</span><span class="s">"185.281"</span> <span class="na">urx=</span><span class="s">"199.281"</span> <span class="na">ury=</span><span class="s">"582.840"</span><span class="nt">&gt;&lt;/rectangle&gt;</span> <span class="nt">&lt;width&gt;</span>14.000<span class="nt">&lt;/width&gt;</span> <span class="nt">&lt;height&gt;</span>20.000<span class="nt">&lt;/height&gt;</span> <span class="nt">&lt;contents&gt;</span>PF.WK1<span class="nt">&lt;/contents&gt;</span> <span class="nt">&lt;annotationName&gt;</span>9a716f25-da75-4309-b918-483cfa9f6473<span class="nt">&lt;/annotationName&gt;</span> <span class="nt">&lt;modifiedDate&gt;</span>D:20131024143706+02'00'<span class="nt">&lt;/modifiedDate&gt;</span> <span class="nt">&lt;resources&gt;</span> <span class="nt">&lt;xobject</span> <span class="na">id=</span><span class="s">"xobjIndir29"</span><span class="nt">&gt;&lt;/xobject&gt;</span> <span class="nt">&lt;/resources&gt;</span> <span class="nt">&lt;color</span> <span class="na">red=</span><span class="s">"0.250000"</span> <span class="na">green=</span><span class="s">"0.333328"</span> <span class="na">blue=</span><span class="s">"1.000000"</span><span class="nt">&gt;&lt;/color&gt;</span> <span class="nt">&lt;invisible&gt;</span>false<span class="nt">&lt;/invisible&gt;</span> <span class="nt">&lt;hidden&gt;</span>false<span class="nt">&lt;/hidden&gt;</span> <span class="nt">&lt;print&gt;</span>true<span class="nt">&lt;/print&gt;</span> <span class="nt">&lt;noZoom&gt;</span>true<span class="nt">&lt;/noZoom&gt;</span> <span class="nt">&lt;noRotate&gt;</span>true<span class="nt">&lt;/noRotate&gt;</span> <span class="nt">&lt;noView&gt;</span>false<span class="nt">&lt;/noView&gt;</span> <span class="nt">&lt;readOnly&gt;</span>false<span class="nt">&lt;/readOnly&gt;</span> <span class="nt">&lt;locked&gt;</span>false<span class="nt">&lt;/locked&gt;</span> <span class="nt">&lt;toggleNoView&gt;</span>false<span class="nt">&lt;/toggleNoView&gt;</span> <span class="nt">&lt;lockedContents&gt;</span>false<span class="nt">&lt;/lockedContents&gt;</span> <span class="nt">&lt;/annotation&gt;</span> </code></pre></div></div> <h4 id="jhove-5">JHOVE</h4> <p>For the sample file with the <em>EmbeddedFiles</em> entry, <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/fileAttachment-jhove.xml">JHOVE’s output</a> contains a <em>PageMode</em> element with <em>UseAttachments</em> as one of its values:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>PageMode<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"Scalar"</span> <span class="na">type=</span><span class="s">"String"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>UseAttachments<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> </code></pre></div></div> <p>For the sample file with the File Attachment Annotation, JHOVE’s <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/fileAttachment_fileAttachmentAnnotation-jhove.xml">output</a> reports a “FileAttachment” annotation:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>Annotation<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"List"</span> <span class="na">type=</span><span class="s">"Property"</span><span class="nt">&gt;</span> <span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>Subtype<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"Scalar"</span> <span class="na">type=</span><span class="s">"String"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>FileAttachment<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> <span class="nt">&lt;property&gt;</span> </code></pre></div></div> <h3 id="link-to-external-file">Link to external file</h3> <h4 id="test-files-6">Test files</h4> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">Description</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/externalLink.pdf?raw=true">externalLink.pdf</a></td> <td style="text-align: left">Contains link to an external document.</td> </tr> </tbody> </table> <h4 id="verapdf-6">VeraPDF</h4> <p>For the test file, the <em>actions</em> element of <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/externalLink-vera.xml">VeraPDF’s output</a> contains a “Launch” action:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;action</span> <span class="na">type=</span><span class="s">"Launch"</span><span class="nt">&gt;</span> <span class="nt">&lt;location&gt;</span>Annotation<span class="nt">&lt;/location&gt;</span> <span class="nt">&lt;/action&gt;</span> </code></pre></div></div> <p>Further to that, the <em>annotations</em> element contains a child element for a “Link” annotation:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;annotation</span> <span class="na">id=</span><span class="s">"annotIndir35"</span><span class="nt">&gt;</span> <span class="nt">&lt;subType&gt;</span>Link<span class="nt">&lt;/subType&gt;</span> <span class="nt">&lt;rectangle</span> <span class="na">lly=</span><span class="s">"678.816"</span> <span class="na">llx=</span><span class="s">"70.860"</span> <span class="na">urx=</span><span class="s">"200.820"</span> <span class="na">ury=</span><span class="s">"694.584"</span><span class="nt">&gt;&lt;/rectangle&gt;</span> <span class="nt">&lt;width&gt;</span>129.960<span class="nt">&lt;/width&gt;</span> <span class="nt">&lt;height&gt;</span>15.768<span class="nt">&lt;/height&gt;</span> <span class="nt">&lt;invisible&gt;</span>false<span class="nt">&lt;/invisible&gt;</span> <span class="nt">&lt;hidden&gt;</span>false<span class="nt">&lt;/hidden&gt;</span> <span class="nt">&lt;print&gt;</span>true<span class="nt">&lt;/print&gt;</span> <span class="nt">&lt;noZoom&gt;</span>false<span class="nt">&lt;/noZoom&gt;</span> <span class="nt">&lt;noRotate&gt;</span>false<span class="nt">&lt;/noRotate&gt;</span> <span class="nt">&lt;noView&gt;</span>false<span class="nt">&lt;/noView&gt;</span> <span class="nt">&lt;readOnly&gt;</span>false<span class="nt">&lt;/readOnly&gt;</span> <span class="nt">&lt;locked&gt;</span>false<span class="nt">&lt;/locked&gt;</span> <span class="nt">&lt;toggleNoView&gt;</span>false<span class="nt">&lt;/toggleNoView&gt;</span> <span class="nt">&lt;lockedContents&gt;</span>false<span class="nt">&lt;/lockedContents&gt;</span> <span class="nt">&lt;/annotation&gt;</span> </code></pre></div></div> <p>As explained in section 12.6.4.5 (Launch actions) of <a href="https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/PDF32000_2008.pdf">ISO 32000-1</a>:</p> <blockquote> <p>A launch action launches an application or opens or prints a document.</p> </blockquote> <p>So, it is not exclusively associated with external <em>documents</em>, but it might also indicate a dependency on external <em>software</em>, which makes it all the more relevant for preservation. Link annotations can also serve several purposes. Section 12.5.6.5 (Link annotations) of ISO 32000-1:</p> <blockquote> <p>A link annotation represents either a hypertext link to a destination elsewhere in the document (…) or an action to be performed.</p> </blockquote> <p>If I’m reading this correctly, the presence of a link annotation alone not always imply an external dependency.</p> <h4 id="jhove-6">JHOVE</h4> <p><a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/externalLink-jhove.xml">JHOVE’s output</a> only contains this reference to the Link annotation:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>Annotation<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"List"</span> <span class="na">type=</span><span class="s">"Property"</span><span class="nt">&gt;</span> <span class="nt">&lt;property&gt;</span> <span class="nt">&lt;name&gt;</span>Subtype<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;values</span> <span class="na">arity=</span><span class="s">"Scalar"</span> <span class="na">type=</span><span class="s">"String"</span><span class="nt">&gt;</span> <span class="nt">&lt;value&gt;</span>Link<span class="nt">&lt;/value&gt;</span> <span class="nt">&lt;/values&gt;</span> <span class="nt">&lt;/property&gt;</span> :: <span class="nt">&lt;property&gt;</span> </code></pre></div></div> <p>JHOVE doesn’t report any actions (including the Launch action in this file).</p> <h3 id="web-capture-content">Web Capture content</h3> <h4 id="test-files-7">Test files</h4> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">Description</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/webCapture.pdf?raw=true">webCapture.pdf</a></td> <td style="text-align: left">Contains Web Capture content.</td> </tr> </tbody> </table> <p>When I created this test file back in 2012, I was under the impression that all Web Capture content by its nature had a dependency on the live web. Reading section 14.10 (Web Capture) of <a href="https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/PDF32000_2008.pdf">ISO 32000-1</a> once more, it turns out that this is not the case:</p> <blockquote> <p>The information in the Web Capture data structures enables conforming products to perform the following operations:</p> <ul> <li>Save locally and preserve the visual appearance of material from the Web</li> <li>Retrieve additional material from the Web and add it to an existing PDF file</li> <li>Update or modify existing material previously captured from the Web</li> </ul> </blockquote> <p>So, Web Capture content may simply be a local copy of material that was captured from the web at the time of the PDF’s creation. This seems to be the case for this test file. When I open it on my machine (using <a href="https://github.com/linuxmint/xreader/">Xreader</a>), it shows a snapshot of the Open Planets Foundation website from the time of the document’s creation (2012):</p> <figure class="image"> <img src="/images/2023/05/webcapture-xreader.png" alt="Screenshot that shows how the test file with web capture content is rendered in Xreader." /> </figure> <p>It’s not entirely clear to me how and under what circumstances additional material is retrieved from the web, or existing material is updated. This makes it hard to judge how much of a preservation risk Web Capture content poses in actual practice.</p> <h4 id="verapdf-7">VeraPDF</h4> <p>The <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/webCapture-vera.xml">output</a> of VeraPDF contains several interesting bits. First, there are these references to a GoTo action, a URI action and a SubmitForm action:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nt">&lt;action</span> <span class="na">type=</span><span class="s">"GoTo"</span><span class="nt">&gt;</span> <span class="nt">&lt;location&gt;</span>Annotation<span class="nt">&lt;/location&gt;</span> <span class="nt">&lt;/action&gt;</span> <span class="nt">&lt;action</span> <span class="na">type=</span><span class="s">"URI"</span><span class="nt">&gt;</span> <span class="nt">&lt;location&gt;</span>Annotation<span class="nt">&lt;/location&gt;</span> <span class="nt">&lt;/action&gt;</span> <span class="nt">&lt;action</span> <span class="na">type=</span><span class="s">"SubmitForm"</span><span class="nt">&gt;</span> <span class="nt">&lt;location&gt;</span>Annotation<span class="nt">&lt;/location&gt;</span> <span class="nt">&lt;/action&gt;</span> </code></pre></div></div> <p>Moreover, the <em>annotations</em> element contains a large number of Link annotations:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;annotation</span> <span class="na">id=</span><span class="s">"annotIndir163"</span><span class="nt">&gt;</span> <span class="nt">&lt;subType&gt;</span>Link<span class="nt">&lt;/subType&gt;</span> <span class="nt">&lt;rectangle</span> <span class="na">lly=</span><span class="s">"720.000"</span> <span class="na">llx=</span><span class="s">"11.000"</span> <span class="na">urx=</span><span class="s">"88.000"</span> <span class="na">ury=</span><span class="s">"760.000"</span><span class="nt">&gt;&lt;/rectangle&gt;</span> <span class="nt">&lt;width&gt;</span>77.000<span class="nt">&lt;/width&gt;</span> <span class="nt">&lt;height&gt;</span>40.000<span class="nt">&lt;/height&gt;</span> <span class="nt">&lt;invisible&gt;</span>false<span class="nt">&lt;/invisible&gt;</span> <span class="nt">&lt;hidden&gt;</span>false<span class="nt">&lt;/hidden&gt;</span> <span class="nt">&lt;print&gt;</span>false<span class="nt">&lt;/print&gt;</span> <span class="nt">&lt;noZoom&gt;</span>false<span class="nt">&lt;/noZoom&gt;</span> <span class="nt">&lt;noRotate&gt;</span>false<span class="nt">&lt;/noRotate&gt;</span> <span class="nt">&lt;noView&gt;</span>false<span class="nt">&lt;/noView&gt;</span> <span class="nt">&lt;readOnly&gt;</span>false<span class="nt">&lt;/readOnly&gt;</span> <span class="nt">&lt;locked&gt;</span>false<span class="nt">&lt;/locked&gt;</span> <span class="nt">&lt;toggleNoView&gt;</span>false<span class="nt">&lt;/toggleNoView&gt;</span> <span class="nt">&lt;lockedContents&gt;</span>false<span class="nt">&lt;/lockedContents&gt;</span> <span class="nt">&lt;/annotation&gt;</span> </code></pre></div></div> <h4 id="jhove-7">JHOVE</h4> <p>Looking at <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/webCapture-jhove.xml">JHOVE’s output</a>, I couldn’t find any reference to the Web Capture content at all. The missing references to the GoTo action, URI action and SubmitForm action are understandable, since JHOVE doesn’t report any actions. The absence of any reference to the numerous Link annotations is nevertheless surprising.</p> <h3 id="byte-corruption">Byte corruption</h3> <h4 id="test-files-8">Test files</h4> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">Description</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/corruptionOneByteMissing.pdf?raw=true">corruptionOneByteMissing.pdf</a></td> <td style="text-align: left">Has one byte of missing data, immediately following the file header.</td> </tr> </tbody> </table> <h4 id="verapdf-8">VeraPDF</h4> <p>The byte corruption in this file causes an exception in VeraPDF. Details can be found in the <em>taskResult</em> element of the <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/corruptionOneByteMissing-vera.xml">output</a> file:</p> <p>Result:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;taskResult</span> <span class="na">type=</span><span class="s">"PARSE"</span> <span class="na">isExecuted=</span><span class="s">"true"</span> <span class="na">isSuccess=</span><span class="s">"false"</span><span class="nt">&gt;</span> <span class="nt">&lt;duration</span> <span class="na">start=</span><span class="s">"1683818745078"</span> <span class="na">finish=</span><span class="s">"1683818745121"</span><span class="nt">&gt;</span>00:00:00.043<span class="nt">&lt;/duration&gt;</span> <span class="nt">&lt;exceptionMessage&gt;</span>Exception: Couldn't parse stream caused by exception: Pages not found<span class="nt">&lt;/exceptionMessage&gt;</span> <span class="nt">&lt;/taskResult&gt;</span> </code></pre></div></div> <p>It looks like the <em>taskResult</em> element is currently <em>only</em> included if an exception occurs. It would be more consistent to include it for <em>every</em> processsed file. This would make it more straightforward to use the output to identify PDFS that caused any exceptions or parse errors in VeraPDF. These are usually an indication that something is seriously wrong with a file, and the value of the <em>isSuccess</em> attribute could be used as a rough validation proxy. I’ve created <a href="https://github.com/veraPDF/veraPDF-library/issues/1336">an issue</a> for this.</p> <h4 id="jhove-8">JHOVE</h4> <p><a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/horror/corruptionOneByteMissing-jhove.xml">JHOVE’s output</a> contains the following error:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;message</span> <span class="na">offset=</span><span class="s">"0"</span> <span class="na">severity=</span><span class="s">"error"</span> <span class="na">id=</span><span class="s">"PDF-HUL-140"</span><span class="nt">&gt;</span>Document catalog dictionary object number and trailer root ref number are inconsistent.<span class="nt">&lt;/message&gt;</span> </code></pre></div></div> <p>Since this test file only represents one very specific case of byte corruption, I expect that other types might result in quite different JHOVE errors (but this is outside the scope of this investigation).</p> <h2 id="analysis-adobe-acrobat-engineering-dataset">Analysis Adobe Acrobat Engineering dataset</h2> <p>The results of the Horror Corpus analysis highlight the significance of actions and annotations, which are often indicative of features that are associated with preservation risks. The Horror Corpus analysis already showed that for one test file, annotations that were correctly reported by VeraPDF, were subsequently not picked up by JHOVE. I did some further tests to determine whether this was an isolated case, or perhaps an indication of a more structural problem. For these tests I used the files in the <a href="https://web.archive.org/web/20130726144923/http://acroeng.adobe.com/wp/?page_id=61">“Classic Multimedia”</a> category of the Adobe Acrobat Engineering dataset. These files make heavy use of actions and annotations, which makes them well suited for this purpose.</p> <h3 id="results">Results</h3> <p>The full VeraPDF and JHOVE output for these files can be found <a href="https://github.com/KBNLresearch/pdf-characterisation/tree/main/output/ae-multimedia">here</a>. The following table summarizes the actions and annotations that are reported by VeraPDF and JHOVE:</p> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">Actions (VeraPDF)</th> <th style="text-align: left">Annotations (VeraPDF)</th> <th style="text-align: left">Annotations (JHOVE)</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">AdobeChassisDemo-commented_Review.pdf</td> <td style="text-align: left">JavaScript</td> <td style="text-align: left">3D<br />Ink<br />Text<br />Popup<br />Polygon</td> <td style="text-align: left">Popup<br />Ink<br />Text<br />3D<br />Polygon</td> </tr> <tr> <td style="text-align: left">movie.pdf</td> <td style="text-align: left"> </td> <td style="text-align: left">Movie</td> <td style="text-align: left"> </td> </tr> <tr> <td style="text-align: left">MusicalScore.pdf</td> <td style="text-align: left">GoTo<br />JavaScript<br />URI<br />Rendition</td> <td style="text-align: left">Screen<br />Widget<br />Link</td> <td style="text-align: left">Link<br />Widget<br />Screen</td> </tr> <tr> <td style="text-align: left">LabelExample.pdf</td> <td style="text-align: left">GoTo3DView<br />JavaScript</td> <td style="text-align: left">3D<br />Widget</td> <td style="text-align: left">3D<br />Widget</td> </tr> <tr> <td style="text-align: left">MultiMedia_Acro6.pdf</td> <td style="text-align: left">Rendition</td> <td style="text-align: left">Screen</td> <td style="text-align: left">Screen</td> </tr> <tr> <td style="text-align: left">AVI+Transitions Demo.pdf</td> <td style="text-align: left"> </td> <td style="text-align: left"> </td> <td style="text-align: left"> </td> </tr> <tr> <td style="text-align: left">20020402_CALOS.pdf</td> <td style="text-align: left">Movie<br />Hide<br />Named<br />GoTo</td> <td style="text-align: left">Link<br />Movie<br />Widget</td> <td style="text-align: left"> </td> </tr> <tr> <td style="text-align: left">Disney-Flash.pdf</td> <td style="text-align: left">URI<br />Rendition<br />SubmitForm</td> <td style="text-align: left">Widget<br />Link<br />Screen</td> <td style="text-align: left">Widget<br />Link<br />Screen</td> </tr> <tr> <td style="text-align: left">gXsummer2004-stream.pdf</td> <td style="text-align: left"> </td> <td style="text-align: left"> </td> <td style="text-align: left"> </td> </tr> <tr> <td style="text-align: left">SVG-AnnotAnim.pdf</td> <td style="text-align: left"> </td> <td style="text-align: left">SVG</td> <td style="text-align: left"> </td> </tr> <tr> <td style="text-align: left">Service Form_media.pdf</td> <td style="text-align: left">SubmitForm<br />Named<br />ResetForm<br />JavaScript<br />Rendition</td> <td style="text-align: left">Widget<br />Link<br />Screen</td> <td style="text-align: left">Screen<br />Widget<br />Link</td> </tr> <tr> <td style="text-align: left">Binder_6-3DPages.pdf</td> <td style="text-align: left">GoTo</td> <td style="text-align: left">3D</td> <td style="text-align: left">3D</td> </tr> <tr> <td style="text-align: left">us_population.pdf</td> <td style="text-align: left">GoTo</td> <td style="text-align: left">SVG</td> <td style="text-align: left">SVG</td> </tr> <tr> <td style="text-align: left">SVG.pdf</td> <td style="text-align: left">JavaScript<br />URI</td> <td style="text-align: left">Widget</td> <td style="text-align: left">Widget</td> </tr> <tr> <td style="text-align: left">phlmapbeta7.pdf</td> <td style="text-align: left">Rendition</td> <td style="text-align: left">Screen</td> <td style="text-align: left">Screen</td> </tr> <tr> <td style="text-align: left">ScriptEvents.pdf</td> <td style="text-align: left">JavaScript<br />GoTo<br />Rendition</td> <td style="text-align: left">Screen<br />Widget</td> <td style="text-align: left">Widget<br />Screen</td> </tr> <tr> <td style="text-align: left">Trophy.pdf</td> <td style="text-align: left">JavaScript<br />URI<br />GoTo<br />Rendition</td> <td style="text-align: left">Screen<br />Widget<br />Link</td> <td style="text-align: left">Widget<br />Link<br />Screen</td> </tr> <tr> <td style="text-align: left">VolvoS40V50-Full.pdf</td> <td style="text-align: left">Named<br />JavaScript<br />GoTo<br />URI<br />Rendition</td> <td style="text-align: left">Widget<br />Link<br />Screen</td> <td style="text-align: left">Screen<br />Widget<br />Link</td> </tr> <tr> <td style="text-align: left">Jpeg_linked.pdf</td> <td style="text-align: left">Named<br />GoTo<br />Rendition</td> <td style="text-align: left">Link<br />Screen</td> <td style="text-align: left">Link<br />Screen</td> </tr> <tr> <td style="text-align: left">AdobeChassisDemo-commented.pdf</td> <td style="text-align: left"> </td> <td style="text-align: left">3D<br />Popup<br />Polygon<br />Text</td> <td style="text-align: left">Popup<br />Polygon<br />3D<br />Text</td> </tr> <tr> <td style="text-align: left">drape_raster_contour_sample.pdf</td> <td style="text-align: left">URI</td> <td style="text-align: left">Link<br />3D</td> <td style="text-align: left">3D<br />Link</td> </tr> <tr> <td style="text-align: left">remotemovieurl.pdf</td> <td style="text-align: left"> </td> <td style="text-align: left">Movie<br />FreeText</td> <td style="text-align: left"> </td> </tr> <tr> <td style="text-align: left">3-D_PDF.pdf</td> <td style="text-align: left"> </td> <td style="text-align: left">3D</td> <td style="text-align: left">3D</td> </tr> <tr> <td style="text-align: left">movie_down1.pdf</td> <td style="text-align: left"> </td> <td style="text-align: left">Movie</td> <td style="text-align: left"> </td> </tr> </tbody> </table> <p>Note that the table doesn’t have an “Actions” column for JHOVE. This is because, unlike VeraPDF, JHOVE’s output doesn’t include any information on actions at all.</p> <p>Both VeraPDF and JHOVE provide information about annotations, and even though the results are broadly similar for both tools, there are some noteworthy differences.</p> <p>For the files <a href="https://web.archive.org/web/20100714002808/http://acroeng.adobe.com:80/Test_Files/movie/movie.pdf">“movie.pdf”</a>, <a href="https://web.archive.org/web/20140519130059/http://acroeng.adobe.com/Test_Files/classic_multimedia//20020402_CALOS.pdf">“20020402_CALOS.pdf”</a>, <a href="https://web.archive.org/web/20100714002816/http://acroeng.adobe.com:80/Test_Files/movie/remotemovieurl.pdf">“remotemovieurl.pdf”</a> and <a href="https://web.archive.org/web/20100714002811/http://acroeng.adobe.com:80/Test_Files/movie/movie_down1.pdf">“movie_down1.pdf”</a>, JHOVE doesn’t report any annotations at all. In all four cases the <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/ae-multimedia/20020402_CALOS-jhove.xml">output</a> contains the following annotation-related validation error;</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;message</span> <span class="na">offset=</span><span class="s">"17342"</span> <span class="na">severity=</span><span class="s">"error"</span> <span class="na">id=</span><span class="s">"PDF-HUL-120"</span><span class="nt">&gt;</span>Annotation dictionary missing required type (S) entry<span class="nt">&lt;/message&gt;</span> </code></pre></div></div> <p>VeraPDF’s output shows that all of these files contain a Movie annotation. I’m not sure if this is a coincidence.</p> <p>For the file “SVG-AnnotAnim.pdf”, JHOVE doesn’t report the SVG annotation. A look at the <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/ae-multimedia/SVG-AnnotAnim-jhove.xml">output</a> shows that JHOVE wasn’t able to parse this file at all, with the following validation error:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;message</span> <span class="na">offset=</span><span class="s">"0"</span> <span class="na">severity=</span><span class="s">"error"</span> <span class="na">id=</span><span class="s">"PDF-HUL-144"</span><span class="nt">&gt;</span>Pages dictionary has no Type key or it has a null value.<span class="nt">&lt;/message&gt;</span> </code></pre></div></div> <h3 id="behaviour-with-corrupted-files">Behaviour with corrupted files</h3> <p>Two PDFs in this dataset are corrupted: “gXsummer2004-stream.pdf” and “AVI+Transitions Demo.pdf”. For both files, <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/ae-multimedia/gXsummer2004-stream-vera.xml">VeraPDF’s output</a> reports a parse exception:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;taskResult</span> <span class="na">type=</span><span class="s">"PARSE"</span> <span class="na">isExecuted=</span><span class="s">"true"</span> <span class="na">isSuccess=</span><span class="s">"false"</span><span class="nt">&gt;</span> </code></pre></div></div> <p>Meanwhile <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/ae-multimedia/gXsummer2004-stream-jhove.xml">JHOVE</a> throws a “No PDF header” validation error:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;message</span> <span class="na">offset=</span><span class="s">"0"</span> <span class="na">severity=</span><span class="s">"error"</span> <span class="na">id=</span><span class="s">"PDF-HUL-137"</span><span class="nt">&gt;</span>No PDF header<span class="nt">&lt;/message&gt;</span> </code></pre></div></div> <h2 id="support-for-pdf-20">Support for PDF 2.0</h2> <p>Since it’s been almost six years now since the <a href="https://www.iso.org/news/ref2199.html">launch</a> of PDF 2.0, I was curious to see to what extent it is supported by VeraPDF and JHOVE. To find out, I ran both on <a href="https://github.com/pdf-association/pdf20examples">this set of PDF 2.0 example files</a> by the PDF Association.</p> <p>Out of the 7 files in this dataset, 6 resulted in the following JHOVE error (full output <a href="https://github.com/KBNLresearch/pdf-characterisation/blob/main/output/pdf20examples/Simple%20PDF%202.0%20file-jhove.xml">here</a>):</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;message</span> <span class="na">offset=</span><span class="s">"0"</span> <span class="na">severity=</span><span class="s">"error"</span> <span class="na">id=</span><span class="s">"PDF-HUL-137"</span><span class="nt">&gt;</span>No PDF header<span class="nt">&lt;/message&gt;</span> </code></pre></div></div> <p>These files were subsequently not parsed at all. The only exception here is this <a href="https://github.com/pdf-association/pdf20examples/blob/master/PDF%202.0%20via%20incremental%20save.pdf">“PDF 2.0 via incremental save.pdf”</a> file, which has “1.7” as its designated version number in the PDF header.</p> <p>By contrast, VeraPDF was able to read all PDF 2.0 files in this dataset without any problems.</p> <h2 id="discussion">Discussion</h2> <p>The combined results of the above analyses of the Horror Corpus, the Adobe Acrobat Engineering dataset and the PDF 2.0 example files allow us to draw a number of conclusions.</p> <h3 id="jhove-9">JHOVE</h3> <p>First of all, even though both VeraPDF and JHOVE are able to detect many PDF features that are associated with known preservation risks, JHOVE has several limitations that make it less appealing for this purpose:</p> <ul> <li>JHOVE’s output doesn’t include any information about <strong>actions</strong>, which means it cannot be used for detecting e.g. JavaScript or Launch actions (which are often indicative of external dependencies).</li> <li>Many PDF features that are preservation risks are associated with <strong>annotations</strong>. Even though JHOVE is able to report most of these, the comparison against VeraPDF’s output shows that JHOVE’s reporting of annotations is often incomplete.</li> <li>JHOVE’s reporting on <strong>encryption and security-related restrictions</strong> could be more informative. Most importantly, it doesn’t allow one to single out files that require an <strong>open password</strong> (which present one of the most serious PDF preservation risks).</li> <li>Moreover, JHOVE makes the detection of <strong>user access restrictions</strong> (print, copy and text access) unnecessarily difficult by only reporting these features if they are <em>not</em> restricted. This is impractical, because it means we have to check for the <em>absence</em> of these features in JHOVE’s output. It also assumes prior knowledge on the user’s behalf of properties that are completely undocumented.</li> <li>Even though JHOVE does report whether <strong>fonts are embedded</strong>, this information is encoded in a way that is needlessly complicated, because it involves checking the output for the <em>absence</em> (again!) of three properties that are undocumented.</li> <li><strong>No documentation exists of any of JHOVE’s reported properties</strong>. This can make their interpretation difficult. The aforementioned font embedding issue is a good example of this: if we want to know if a particular font is embedded or not, we need to check three separate properties (<em>FontFile</em>, <em>FontFile2</em> and <em>FontFile3</em>), which are all undocumented. I was only able to figure this out by digging into JHOVE’s source code, and consulting the ISO 32000-1 PDF specification.</li> <li>Another (but related) problem is JHOVE’s rather <strong>clumsy and convoluted XML output format</strong>, which is made up by a labyrinthine assortment of nested “property” elements. This makes the format difficult to read, either by a human or a machine<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">8</a></sup>. It’s not possible to address the reported properties directly using, for example, standard <a href="https://en.wikipedia.org/wiki/XPath">xpath</a> expressions. This is because all properties are encoded as identically-named <em>property</em> elements, where the name of each individual property is the text value of its <em>name</em> child element. Of course this doesn’t preclude parsing the format altogether, but it does make working with JHOVE’s output considerably harder than most modern, well-designed XML formats.</li> <li>Finally, the tests showed that JHOVE <strong>lacks support for PDF 2.0</strong>. On encountering a document with a PDF 2.0 header, JHOVE simply reports an error and stops parsing the file. As a result, JHOVE’s output on PDF 2.0 documents does not contain any meaningful information.</li> </ul> <h3 id="verapdf-9">VeraPDF</h3> <p>VeraPDF was able to detect all features that are associated with known preservation risks in the “Horror Corpus” files. Unlike JHOVE, VeraPDF also reports on actions. VeraPDF’s coverage of annotations is more comprehensive, and it also provides detailed information about security-related restrictions. I was unable to find any detailed documentation of VeraPDF’s reported properties, but the documentation does provide a <a href="https://docs.verapdf.org/cli/config/#features.xml">general overview of the reported feature categories</a>. Also, VeraPDF’s XML output format is much simpler, better structured and easier to navigate than the one used by JHOVE. Each reported property can be addressed directly by the name of its corresponding XML element, and the property names are mostly self-explanatory.</p> <p>For convenience, the following table summarises the elements in VeraPDF’s output that turned out to be the most interesting ones for the current analysis (asterisks indicates repeatable elements):</p> <table> <thead> <tr> <th style="text-align: left">Element (relative to <em>job</em> element)</th> <th style="text-align: left">Significance</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">taskResult</td> <td style="text-align: left">Parse status, rough validity proxy, may be used to identify malformed files and files that require open password</td> </tr> <tr> <td style="text-align: left">featuresReport/documentSecurity</td> <td style="text-align: left">Child elements indicate security-related restrictions (e.g. copy, printing and text access passwords)</td> </tr> <tr> <td style="text-align: left">featuresReport/actions/action*</td> <td style="text-align: left">Some actions imply a preservation risk</td> </tr> <tr> <td style="text-align: left">featuresReport/annotations/annotation*</td> <td style="text-align: left">Some annotations imply a preservation risk</td> </tr> <tr> <td style="text-align: left">featuresReport/documentResources/fonts/font/fontDescriptor/embedded*</td> <td style="text-align: left">Value indicates whether font is embedded or not</td> </tr> <tr> <td style="text-align: left">featuresReport/embeddedFiles/embeddedFile*</td> <td style="text-align: left">Indicates presence of file attachment</td> </tr> </tbody> </table> <p>In order to report these, at minimum the following feature types must be enabled in VeraPDF’s configuration file:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">&lt;?xml version="1.0" encoding="UTF-8" standalone="yes"?&gt;</span> <span class="nt">&lt;featuresConfig&gt;</span> <span class="nt">&lt;enabledFeatures&gt;</span> <span class="nt">&lt;feature&gt;</span>ACTION<span class="nt">&lt;/feature&gt;</span> <span class="nt">&lt;feature&gt;</span>ANNOTATION<span class="nt">&lt;/feature&gt;</span> <span class="nt">&lt;feature&gt;</span>DOCUMENT_SECURITY<span class="nt">&lt;/feature&gt;</span> <span class="nt">&lt;feature&gt;</span>EMBEDDED_FILE<span class="nt">&lt;/feature&gt;</span> <span class="nt">&lt;feature&gt;</span>FONT<span class="nt">&lt;/feature&gt;</span> <span class="nt">&lt;/enabledFeatures&gt;</span> <span class="nt">&lt;/featuresConfig&gt;</span> </code></pre></div></div> <h2 id="conclusion">Conclusion</h2> <p>The tests with the Horror Corpus, the Acrobat Engineering files and the PDF 2.0 example files show that between VeraPDF and JHOVE, VeraPDF is the better choice for detecting specific PDF features that are associated with preservation risks. VeraPDF is able to report on a wider range of features, its output is far easier to interpret and process, and it is less likely to “give up” on files. A limitation of VeraPDF is that, unlike JHOVE, it is not capable of validating PDF’s lower-level data structures (full PDF validation). Such deviations from the published specifications can result in a whole separate class of preservation risks, but these are outside the scope of this work. VeraPDF’s parse status may be used as a rough validity proxy to identify malformed files. Depending on what preservation risks are considered important (which in turn depends on institutional contexts and preservation policies), both VeraPDF and JHOVE should be seen as complementary to each other.</p> <h2 id="acknowledgements">Acknowledgements</h2> <p>Thanks are due to Sam Alloing (KB) and Tyler Thorsted (Brigham Young University) for their feedback to an earlier draft of this post, and for various (online) discussions related to this work.</p> <h2 id="further-resources">Further resources</h2> <ul> <li> <p><a href="https://github.com/KBNLresearch/pdf-characterisation">Github repo with analysis scripts and raw tool output</a></p> </li> <li> <p><a href="https://github.com/openpreserve/format-corpus">Open Preservation Foundation format corpus</a></p> </li> <li> <p><a href="https://github.com/pdf-association/pdf20examples">PDF 2.0 examples</a></p> </li> <li> <p><a href="https://docs.google.com/spreadsheets/d/1eW7R8yACBciNimr16Z2ptC7fs1FlmZMnzdtG_DHBuD4/edit?usp=sharing">PDF Significant Properties Spreadsheet by Tyler Thorsted</a></p> </li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:3"> <p>See the KB <a href="https://www.kb.nl/en/file-download/download/public/842">File Format Guidelines</a> for more details. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>See <a href="/2012/12/19/identification-pdf-preservation-risks-apache-preflight-first-impression">“Identification of PDF preservation risks with Apache Preflight: a first impression”</a>, <a href="/2013/07/25/identification-pdf-preservation-risks-sequel">“Identification of PDF preservation risks with Apache Preflight: the sequel”</a> and <a href="/2014/01/27/identification-pdf-preservation-risks-analysis-govdocs-selected-corpus">“Identification of PDF preservation risks: analysis of Govdocs selected corpus”</a>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:7"> <p>See e.g. <a href="https://zenodo.org/record/1228650">“A PDF Test-Set for Well-Formedness Validation in JHOVE - The Good, the Bad and the Ugly”</a> by Lindlar, Tunnat and Wilson. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>Several other tools PDF validators exist, but none of these, including JHOVE, have <a href="https://www.pdfa.org/wp-content/until2016_uploads/2015/12/iPres2014-CanonicalPDF-submission_20140827.pdf">gained widespread acceptance by industry and other stakeholders</a>. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:8"> <p>This is explained in VeraPDF’s <a href="https://docs.verapdf.org/cli/config/#features.xml">documentation</a>. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:9"> <p>I initially assumed the <em>OwnerString</em> and <em>UserString</em> properties might be related to the presence of an owner password, but these are also reported for PDFs with other security-related restrictions, like print and copy passwords. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:1"> <p>JavaScript detection wasn’t possible in <a href="/2017/06/01/policy-based-assessment-with-verapdf-a-first-impression">an earlier analysis I did in 2017</a>, because VeraPDF didn’t support actions at that time. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>For my analyses I had to re-run JHOVE with output set to TEXT on various occasions to make sense of the output. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2023/05/25/identification-of-pdf-preservation-risks-with-verapdf-and-jhove https://bitsgalore.org/2023/05/25/identification-of-pdf-preservation-risks-with-verapdf-and-jhove 2023-05-25T00:00:00+02:00 <figure class="image"> <img src="/images/2023/03/clockwork-extraction.jpg" alt="Street scene showing crowd gathered around an open carriage in which a dentist performs a tooth extraction on a patient. Next to the patient a man is banging on a large drum." /> <figcaption>Clockwork picture of an itinerant dentist performing an extraction in French rural scene, wood frame, metal workings, first half 19th century. <a href="https://wellcomecollection.org/works/hwpe3cxp">Science Museum, London</a>. <a href="https://creativecommons.org/licenses/by/4.0/">Attribution 4.0 International (CC BY 4.0)</a> (cropped from original).</figcaption> </figure> <p>This blog post provides a brief introduction to extracting unformatted text from EPUB files. The occasion for this work was a request by my Digital Humanities colleagues who are involved in the <a href="https://www.surf.nl/en/news/sane-secure-data-environment-for-social-sciences-and-humanities">SANE (Secure ANalysis Environment) project</a>. The work on this project includes a use case that will use the SANE environment to analyse text from novels in EPUB format. My colleagues were looking for some advice on how to implement the text extraction component, preferably using a Python-based solution.</p> <p>So, I started by making a shortlist of potentially suitable tools. For each tool, I wrote a minimal code snippet for processing one file. Based on this I then created some simple demo scripts that show how each tool is used within a processing workflow. Next, I applied these scripts to two data sets, and used the results to obtain a first impression of the performance of each of the tools.</p> <!-- more --> <h2 id="evaluated-tools">Evaluated tools</h2> <p>I evaluated the following tools:</p> <ol> <li> <p><a href="https://github.com/chrismattmann/tika-python"><strong>Tika-python</strong></a>. This is a Python wrapper for <a href="https://tika.apache.org/">Apache Tika</a> (which itself is a Java application). Apache Tika is a toolkit for text and metadata extraction from a wide range of file formats, including EPUB.</p> </li> <li> <p><a href="https://github.com/deanmalmgren/textract"><strong>Textract</strong></a>. This offers text extraction functionality that is similar to Tika, but unlike Tika, Textract is natively written in Python.</p> </li> <li> <p><a href="https://github.com/aerkalov/ebooklib"><strong>EbookLib</strong></a>. This is a Python library for reading and writing E-books in various formats, including EPUB (both EPUB 2 en EPUB 3). EbookLib is also the E-book library that is used by Textract.</p> </li> <li> <p><a href="https://github.com/pymupdf/PyMuPDF"><strong>PyMuPDF</strong></a>. This is a Python binding for <a href="https://mupdf.com/">MuPDF</a>. MuPDF is primarily a PDF library, but it also supports EPUB.</p> </li> </ol> <p>The following table shows the versions of these tools that I used in my tests:</p> <table> <thead> <tr> <th style="text-align: left">Software</th> <th style="text-align: left">Version</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">Tika-python</td> <td style="text-align: left">2.6.0</td> </tr> <tr> <td style="text-align: left">Textract</td> <td style="text-align: left">1.6.5</td> </tr> <tr> <td style="text-align: left">EbookLib</td> <td style="text-align: left">0.18</td> </tr> <tr> <td style="text-align: left">PyMuPDF</td> <td style="text-align: left">1.24.11</td> </tr> </tbody> </table> <h2 id="test-environment-and-data">Test environment and data</h2> <p>For all of my tests I used a simple desktop PC running Linux Mint 20.1 (Ulyssa), MATE edition, with Python 3.8.10.</p> <p>I used two data sets:</p> <ol> <li>A selection of 15 files in <a href="https://idpf.org/epub/201">EPUB 2.0.1</a> format from the KB’s <a href="https://www.dbnl.org">DBNL</a> (Digital Library for Dutch Literature) collection.</li> <li>A selection of 10 files in <a href="https://www.w3.org/publishing/epub3/epub-spec.html">EPUB 3.2</a> format from <a href="https://standardebooks.org/">Standard Ebooks</a>.</li> </ol> <p>All files in both data sets are structurally valid EPUB (2.0.1 / 3.2): validation with <a href="https://github.com/w3c/epubcheck">EPUBCheck</a> 4.2.6 didn’t result in any reported errors or warnings<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>.</p> <h2 id="tika-python">Tika-python</h2> <p>After installing Tika-python, as a first test I tried to write a minimal code snippet that extracts the text from one single EPUB, and then writes the result as UTF-8 encoded text to a file. Following Tika-python’s <a href="https://github.com/chrismattmann/tika-python/blob/master/README.md">README</a> (the example under “Parser Interface”), I started out with with this:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#! /usr/bin/env python3 </span> <span class="kn">import</span> <span class="n">tika</span> <span class="kn">from</span> <span class="n">tika</span> <span class="kn">import</span> <span class="n">parser</span> <span class="n">fileIn</span> <span class="o">=</span> <span class="sh">"</span><span class="s">berk011veel01_01.epub</span><span class="sh">"</span> <span class="n">fileOut</span> <span class="o">=</span> <span class="sh">"</span><span class="s">berk011veel01_01.txt</span><span class="sh">"</span> <span class="n">parsed</span> <span class="o">=</span> <span class="n">parser</span><span class="p">.</span><span class="nf">from_file</span><span class="p">(</span><span class="n">fileIn</span><span class="p">)</span> <span class="n">content</span> <span class="o">=</span> <span class="n">parsed</span><span class="p">[</span><span class="sh">"</span><span class="s">content</span><span class="sh">"</span><span class="p">]</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">fileOut</span><span class="p">,</span> <span class="sh">'</span><span class="s">w</span><span class="sh">'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">'</span><span class="s">utf-8</span><span class="sh">'</span><span class="p">)</span> <span class="k">as</span> <span class="n">fout</span><span class="p">:</span> <span class="n">fout</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="n">content</span><span class="p">)</span> </code></pre></div></div> <h3 id="metadata-strings-in-text-output">Metadata strings in text output</h3> <p>Inspection of the resulting output file showed a succession of text strings with the names of embedded fonts towards the end of the file. As an example:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Charis SIL Bold Italic :: :: Charis SIL Small Caps </code></pre></div></div> <p>When I ran Tika (the Java application) directly without using the Tika-python wrapper, results were as expected. A closer inspection of the Tika-python source code showed that Tika-python’s parsing of the Tika output doesn’t quite work the way it should, with the result that extracted metadata is erroneously included in the text output.</p> <h3 id="workaround-set-service-to-text">Workaround: set service to text</h3> <p>Fortunately there’s a simple workaround for this. In the parser function call, just add the “service” parameter and set its value to “text”, as shown here:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#! /usr/bin/env python3 </span> <span class="kn">import</span> <span class="n">tika</span> <span class="kn">from</span> <span class="n">tika</span> <span class="kn">import</span> <span class="n">parser</span> <span class="n">fileIn</span> <span class="o">=</span> <span class="sh">"</span><span class="s">berk011veel01_01.epub</span><span class="sh">"</span> <span class="n">fileOut</span> <span class="o">=</span> <span class="sh">"</span><span class="s">berk011veel01_01.txt</span><span class="sh">"</span> <span class="n">parsed</span> <span class="o">=</span> <span class="n">parser</span><span class="p">.</span><span class="nf">from_file</span><span class="p">(</span><span class="n">fileIn</span><span class="p">,</span> <span class="n">service</span><span class="o">=</span><span class="sh">'</span><span class="s">text</span><span class="sh">'</span><span class="p">)</span> <span class="n">content</span> <span class="o">=</span> <span class="n">parsed</span><span class="p">[</span><span class="sh">"</span><span class="s">content</span><span class="sh">"</span><span class="p">]</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">fileOut</span><span class="p">,</span> <span class="sh">'</span><span class="s">w</span><span class="sh">'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">'</span><span class="s">utf-8</span><span class="sh">'</span><span class="p">)</span> <span class="k">as</span> <span class="n">fout</span><span class="p">:</span> <span class="n">fout</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="n">content</span><span class="p">)</span> </code></pre></div></div> <p>With this change, the font-related text strings were no longer reported.</p> <h3 id="image-tags-and-alt-text-strings">Image tags and alt-text strings</h3> <p>Unfortunately, setting the “service” parameter in this way has the unexpected side-effect that the text output now includes tags with alt-text descriptions for any images in the file. For example:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[image: cover] Aster Berkhof Veel geluk, professor! [image: DBNL] </code></pre></div></div> <h3 id="different-behaviour-between-tika-app-and-tikaserver">Different behaviour between Tika app and TikaServer</h3> <p>I initially thought this was also a bug in Tika-python, but it turns out this isn’t the case. Using the Tika Java application directly:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java -jar ~/tika/tika-app-2.6.0.jar -t berk011veel01_01.epub &gt; berk011veel01_01-app.txt </code></pre></div></div> <p>This resulted in an output file with no alt-text strings. However, Tika-python doesn’t wrap around Tika-app, but instead around <a href="https://cwiki.apache.org/confluence/display/TIKA/TikaServer">TikaServer</a>. After starting TikaServer, I used the command below to processes the same EPUB:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl -T berk011veel01_01.epub http://localhost:9998/tika --header "Accept: text/plain" &gt; berk011veel01_01-server.txt </code></pre></div></div> <p>The resulting file also included the offending image tags and alt-text strings. So, the Tika application and TikaServer behave differently. After reporting <a href="https://issues.apache.org/jira/browse/TIKA-3969?filter=12326160">an issue</a> for this, I received a confirmation from Tika’s lead developer:</p> <blockquote> <p>There’s a subtle difference in the handlers used in tika-app and tika-server. We’re using the “RichTextContentHandler” in server but not in app. I think I’ve known about this for a while, but we’ll be breaking behaviour for whichever one we fix.</p> </blockquote> <p>I also created a <a href="https://github.com/chrismattmann/tika-python/issues/389">separate issue</a> at Tika-python for the inclusion of metadata in the text output. Unfortunately this issue is closely related to (and partly the result of) the upstream issue in TikaServer. So until that upstream issue is fixed, the current (slightly confusing) situation will most likely persist.</p> <h3 id="ocr-if-tesseract-is-installed">OCR if Tesseract is installed</h3> <p>By default, Tika applies optical character recognition (OCR) to any images in an EPUB if the <a href="https://github.com/tesseract-ocr/tesseract">Tesseract</a> software is installed, and includes the OCR output in the extracted text. In many cases (at least for ours!) this might not be the desired behaviour. I only found out about this weeks after doing the original tests that are described in this post. Re-running some of the tests suddenly resulted in slightly larger output files, with text output that wasn’t originally there. It turns out that the root cause was that I had installed some software that installs Tesseract as a dependency (but I wasn’t aware of this). It’s possible to <a href="https://cwiki.apache.org/confluence/display/TIKA/TikaOCR#TikaOCR-disable-ocr">disable OCR</a> in the Java application and TikaServer using a <a href="https://tika.apache.org/1.9/configuring.html#Using_a_Tika_Configuration_XML_file">command-line option</a> that points to a configuration file. I haven’t found a way to do this in Tika-python. The safest option might be to make sure that Teseract is not installed, or to rename Tesseract’s installation folder.</p> <h2 id="textract">Textract</h2> <p>As with Tika-python, as a first test I again created a minimal code snippet for processing one EPUB file:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#! /usr/bin/env python3 </span> <span class="kn">import</span> <span class="n">textract</span> <span class="n">fileIn</span> <span class="o">=</span> <span class="sh">"</span><span class="s">berk011veel01_01.epub</span><span class="sh">"</span> <span class="n">fileOut</span> <span class="o">=</span> <span class="sh">"</span><span class="s">berk011veel01_01.txt</span><span class="sh">"</span> <span class="n">content</span> <span class="o">=</span> <span class="n">textract</span><span class="p">.</span><span class="nf">process</span><span class="p">(</span><span class="n">fileIn</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">'</span><span class="s">utf-8</span><span class="sh">'</span><span class="p">).</span><span class="nf">decode</span><span class="p">()</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">fileOut</span><span class="p">,</span> <span class="sh">'</span><span class="s">w</span><span class="sh">'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">'</span><span class="s">utf-8</span><span class="sh">'</span><span class="p">)</span> <span class="k">as</span> <span class="n">fout</span><span class="p">:</span> <span class="n">fout</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="n">content</span><span class="p">)</span> </code></pre></div></div> <p>For the very first EPUB file (from the DBNL collection) this resulted in an empty output file. Results were similar for most other DBNL EPUBs, and Textract only managed to extract a handful of words at most. Results were considerably better for the “Standard Ebooks” files, with output that was similar to Tika-python in most cases. I <a href="https://github.com/deanmalmgren/textract/issues/455">reported</a> this issue with the developers.</p> <h2 id="ebooklib">EbookLib</h2> <p>I mainly included EbookLib, because Textract uses it “under the hood” for EPUB, and I was curious if using it directly would give me similar results as Textract. Based on its <a href="https://docs.sourcefabric.org/projects/ebooklib/en/latest/tutorial.html#reading-epubdocs.sourcefabric.org/projects/ebooklib/en/latest/tutorial.html#reading-epub">documentation</a> I created the following minimal code snippet:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#! /usr/bin/env python3 </span> <span class="kn">from</span> <span class="n">html.parser</span> <span class="kn">import</span> <span class="n">HTMLParser</span> <span class="kn">import</span> <span class="n">ebooklib</span> <span class="kn">from</span> <span class="n">ebooklib</span> <span class="kn">import</span> <span class="n">epub</span> <span class="n">fileIn</span> <span class="o">=</span> <span class="sh">"</span><span class="s">berk011veel01_01.epub</span><span class="sh">"</span> <span class="n">fileOut</span> <span class="o">=</span> <span class="sh">"</span><span class="s">berk011veel01_01.txt</span><span class="sh">"</span> <span class="n">book</span> <span class="o">=</span> <span class="n">epub</span><span class="p">.</span><span class="nf">read_epub</span><span class="p">(</span><span class="n">fileIn</span><span class="p">)</span> <span class="n">content</span> <span class="o">=</span> <span class="sh">""</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">book</span><span class="p">.</span><span class="nf">get_items</span><span class="p">():</span> <span class="k">if</span> <span class="n">item</span><span class="p">.</span><span class="nf">get_type</span><span class="p">()</span> <span class="o">==</span> <span class="n">ebooklib</span><span class="p">.</span><span class="n">ITEM_DOCUMENT</span><span class="p">:</span> <span class="n">bodyContent</span> <span class="o">=</span> <span class="n">item</span><span class="p">.</span><span class="nf">get_body_content</span><span class="p">().</span><span class="nf">decode</span><span class="p">()</span> <span class="n">f</span> <span class="o">=</span> <span class="nc">HTMLFilter</span><span class="p">()</span> <span class="n">f</span><span class="p">.</span><span class="nf">feed</span><span class="p">(</span><span class="n">bodyContent</span><span class="p">)</span> <span class="n">content</span> <span class="o">+=</span> <span class="n">f</span><span class="p">.</span><span class="n">text</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">fileOut</span><span class="p">,</span> <span class="sh">'</span><span class="s">w</span><span class="sh">'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">'</span><span class="s">utf-8</span><span class="sh">'</span><span class="p">)</span> <span class="k">as</span> <span class="n">fout</span><span class="p">:</span> <span class="n">fout</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="n">content</span><span class="p">)</span> </code></pre></div></div> <p>Compared to Tika-python and Textract, the EbookLib script is a bit more involved, as EbookLib doesn’t provide any high-level text extraction functions. Instead, the user must iterate over all document items, extract the (X)HTML, and then convert that to unformatted text. At first glance, tests with the DBNL and Standard Ebooks EPUBs didn’t result in any issues, and the results were similar to Tika-python.</p> <h2 id="pymupdf">PyMuPDF</h2> <p>For PyMuPDF, I created the following minimal code snippet:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#! /usr/bin/env python3 </span><span class="kn">import</span> <span class="n">pymupdf</span> <span class="n">fileIn</span> <span class="o">=</span> <span class="sh">"</span><span class="s">berk011veel01_01.epub</span><span class="sh">"</span> <span class="n">fileOut</span> <span class="o">=</span> <span class="sh">"</span><span class="s">berk011veel01_01.txt</span><span class="sh">"</span> <span class="k">with</span> <span class="n">pymupdf</span><span class="p">.</span><span class="nf">open</span><span class="p">(</span><span class="n">fileIn</span><span class="p">)</span> <span class="k">as</span> <span class="n">doc</span><span class="p">:</span> <span class="n">content</span> <span class="o">=</span> <span class="sh">""</span> <span class="n">noChapters</span> <span class="o">=</span> <span class="n">doc</span><span class="p">.</span><span class="n">chapter_count</span> <span class="c1"># Iterate over chapters </span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nf">range</span><span class="p">(</span><span class="n">noChapters</span><span class="p">):</span> <span class="n">chapter_page_count</span> <span class="o">=</span> <span class="n">doc</span><span class="p">.</span><span class="nf">chapter_page_count</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="n">chapter_text</span> <span class="o">=</span> <span class="sh">""</span> <span class="c1"># Iterate over pages in chapter </span> <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nf">range</span><span class="p">(</span><span class="n">chapter_page_count</span><span class="p">):</span> <span class="n">page</span> <span class="o">=</span> <span class="n">doc</span><span class="p">[(</span><span class="n">i</span><span class="p">,</span> <span class="n">j</span><span class="p">)]</span> <span class="n">chapter_text</span> <span class="o">+=</span> <span class="n">page</span><span class="p">.</span><span class="nf">get_text</span><span class="p">()</span> <span class="n">content</span> <span class="o">+=</span> <span class="n">chapter_text</span> <span class="c1"># Add linebreak to mark end of chapter </span> <span class="n">content</span> <span class="o">+=</span> <span class="sh">"</span><span class="se">\n</span><span class="sh">"</span> <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">fileOut</span><span class="p">,</span> <span class="sh">'</span><span class="s">w</span><span class="sh">'</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="sh">'</span><span class="s">utf-8</span><span class="sh">'</span><span class="p">)</span> <span class="k">as</span> <span class="n">fout</span><span class="p">:</span> <span class="n">fout</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="n">content</span><span class="p">)</span> </code></pre></div></div> <p>There are a couple of things to note here.</p> <h3 id="chapters-and-pages">Chapters and pages</h3> <p>First, like EbookLib, we need to explicitly iterate over all chapters in the EPUB. Second, PyMuPDF’s document model is built on <em>pages</em>, which probably reflects its origins as a PDF library. However, the EPUB format doesn’t really have any notion of “pages” at all. Nevertheless, since PyMuPDF’s text extraction function only works at the page level, we still need to iterate over the “pages” of each chapter, even though it’s not entirely clear to me how PyMuPDF defines them. Since by default the chapter texts are simply concatenated, I explicitly added a linebreak to more clearly delineate the end of each chapter.</p> <h3 id="wrapping-and-linebreaks">Wrapping and linebreaks</h3> <p>Unlike the other tools tested here, PyMuPDF wraps the extracted text to a fixed page width, and inserts linebreaks at the wrapping boundaries. As an example, look at the following sentence in the source XHTML, which is encoded as one single line:</p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;div</span> <span class="na">class=</span><span class="s">"plat"</span><span class="nt">&gt;</span>Het was wonderbaar. Pierre ademde diep, en de lucht was zo dun en zo ijl, dat zijn hoofd er duizelig van werd.<span class="nt">&lt;/div&gt;</span> </code></pre></div></div> <p>In the corresponding PyMuPDF output, the text is split across two separate lines:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Het was wonderbaar. Pierre ademde diep, en de lucht was zo dun en zo ijl, dat zijn hoofd er duizelig van werd. </code></pre></div></div> <p>Whether this is actually a problem will probably depend on the use case, but it’s good to be aware that this happens.</p> <p>Finally, for the EPUBs in the KB’s DBNL dataset, PyMuPDF reported multiple instances of the following error:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>MuPDF error: syntax error: css syntax error: unexpected token (OEBPS/template.css:1) ( &gt;@&lt;font-face {font-family: "sc...) </code></pre></div></div> <p>The error is related to a style sheet, and doesn’t appear to affect text extraction.</p> <h2 id="demonstration-scripts">Demonstration scripts</h2> <p>Based on the above minimal code snippets, I created <a href="https://github.com/KBNLresearch/textExtractDemo">four simple demonstration scripts</a> for Python-tika, Textract, Ebooklib and PyMuPDF. Each of these scripts extracts the text of each EPUB file in a user-defined input directory. The extracted text is then written to a user-defined output directory. Each script also writes a file with word counts for the extraction results, which is useful for a rough comparison of the different tools.</p> <p>I ran each script twice, using the DBNL and Standard Ebooks data sets as input, respectively.</p> <h2 id="word-counts">Word counts</h2> <p>The table below shows the resulting word counts for the books in the DBNL data set:</p> <table> <thead> <tr> <th style="text-align: left">File name</th> <th style="text-align: left">Words (Tika)</th> <th style="text-align: left">Words (Textract)</th> <th style="text-align: left">Words (EbookLib)</th> <th style="text-align: left">Words (PyMuPDF)</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">eern001lief01_01.epub</td> <td style="text-align: left">25450</td> <td style="text-align: left">1</td> <td style="text-align: left">25446</td> <td style="text-align: left">25451</td> </tr> <tr> <td style="text-align: left">spro002mure01_01.epub</td> <td style="text-align: left">50553</td> <td style="text-align: left">0</td> <td style="text-align: left">50549</td> <td style="text-align: left">50554</td> </tr> <tr> <td style="text-align: left">berk011veel01_01.epub</td> <td style="text-align: left">67978</td> <td style="text-align: left">0</td> <td style="text-align: left">67974</td> <td style="text-align: left">67978</td> </tr> <tr> <td style="text-align: left">sche034drie01_01.epub</td> <td style="text-align: left">203853</td> <td style="text-align: left">3</td> <td style="text-align: left">203352</td> <td style="text-align: left">203864</td> </tr> <tr> <td style="text-align: left">jous010supe01_01.epub</td> <td style="text-align: left">202495</td> <td style="text-align: left">0</td> <td style="text-align: left">202491</td> <td style="text-align: left">202494</td> </tr> <tr> <td style="text-align: left">dele035wegv01_01.epub</td> <td style="text-align: left">76536</td> <td style="text-align: left">0</td> <td style="text-align: left">76530</td> <td style="text-align: left">76530</td> </tr> <tr> <td style="text-align: left">verv017eerl01_01.epub</td> <td style="text-align: left">33844</td> <td style="text-align: left">0</td> <td style="text-align: left">33840</td> <td style="text-align: left">33855</td> </tr> <tr> <td style="text-align: left">dhae007euro01_01.epub</td> <td style="text-align: left">394455</td> <td style="text-align: left">2</td> <td style="text-align: left">394400</td> <td style="text-align: left">394879</td> </tr> <tr> <td style="text-align: left">gomm002uurw01_01.epub</td> <td style="text-align: left">43754</td> <td style="text-align: left">0</td> <td style="text-align: left">43731</td> <td style="text-align: left">43748</td> </tr> <tr> <td style="text-align: left">gang009lalb01_01.epub</td> <td style="text-align: left">28453</td> <td style="text-align: left">4</td> <td style="text-align: left">28381</td> <td style="text-align: left">28390</td> </tr> <tr> <td style="text-align: left">geel005bloe01_01.epub</td> <td style="text-align: left">76316</td> <td style="text-align: left">0</td> <td style="text-align: left">76312</td> <td style="text-align: left">76313</td> </tr> <tr> <td style="text-align: left">hart008droo02_01.epub</td> <td style="text-align: left">77283</td> <td style="text-align: left">0</td> <td style="text-align: left">77279</td> <td style="text-align: left">77282</td> </tr> <tr> <td style="text-align: left">eede003vand04_01.epub</td> <td style="text-align: left">120481</td> <td style="text-align: left">6</td> <td style="text-align: left">120310</td> <td style="text-align: left">120553</td> </tr> <tr> <td style="text-align: left">meij031tuss02_01.epub</td> <td style="text-align: left">145678</td> <td style="text-align: left">4</td> <td style="text-align: left">145665</td> <td style="text-align: left">145692</td> </tr> <tr> <td style="text-align: left">maas013blau01_01.epub</td> <td style="text-align: left">55099</td> <td style="text-align: left">0</td> <td style="text-align: left">55093</td> <td style="text-align: left">55108</td> </tr> </tbody> </table> <p>Note the extreme (near zero) word counts for Textract. The results for Tika, EbookLib and PyMuPDF are roughly the same.</p> <p>Running the scripts on the Standard Ebooks EPUBs gave the following result:</p> <table> <thead> <tr> <th style="text-align: left">File name</th> <th style="text-align: left">Words (Tika)</th> <th style="text-align: left">Words (Textract)</th> <th style="text-align: left">Words (EbookLib)</th> <th style="text-align: left">Words (PyMuPDF)</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">william-shakespeare_king-lear.epub</td> <td style="text-align: left">28442</td> <td style="text-align: left">18621</td> <td style="text-align: left">28430</td> <td style="text-align: left">28357</td> </tr> <tr> <td style="text-align: left">david-garnett_lady-into-fox.epub</td> <td style="text-align: left">25240</td> <td style="text-align: left">25223</td> <td style="text-align: left">25228</td> <td style="text-align: left">25208</td> </tr> <tr> <td style="text-align: left">joseph-conrad_heart-of-darkness.epub</td> <td style="text-align: left">38717</td> <td style="text-align: left">38698</td> <td style="text-align: left">38705</td> <td style="text-align: left">38735</td> </tr> <tr> <td style="text-align: left">anthony-trollope_the-dukes-children.epub</td> <td style="text-align: left">223014</td> <td style="text-align: left">222995</td> <td style="text-align: left">223002</td> <td style="text-align: left">222712</td> </tr> <tr> <td style="text-align: left">agatha-christie_the-mysterious-affair-at-styles.epub</td> <td style="text-align: left">57401</td> <td style="text-align: left">57229</td> <td style="text-align: left">57271</td> <td style="text-align: left">57227</td> </tr> <tr> <td style="text-align: left">edgar-allan-poe_the-narrative-of-arthur-gordon-pym-of-nantucket.epub</td> <td style="text-align: left">71931</td> <td style="text-align: left">71837</td> <td style="text-align: left">71863</td> <td style="text-align: left">71844</td> </tr> <tr> <td style="text-align: left">p-g-wodehouse_short-fiction.epub</td> <td style="text-align: left">212224</td> <td style="text-align: left">212182</td> <td style="text-align: left">212212</td> <td style="text-align: left">212186</td> </tr> <tr> <td style="text-align: left">robert-louis-stevenson_the-strange-case-of-dr-jekyll-and-mr-hyde.epub</td> <td style="text-align: left">26370</td> <td style="text-align: left">26345</td> <td style="text-align: left">26358</td> <td style="text-align: left">26291</td> </tr> <tr> <td style="text-align: left">h-g-wells_the-time-machine.epub</td> <td style="text-align: left">33044</td> <td style="text-align: left">33024</td> <td style="text-align: left">33032</td> <td style="text-align: left">33018</td> </tr> <tr> <td style="text-align: left">thorstein-veblen_the-theory-of-the-leisure-class.epub</td> <td style="text-align: left">106537</td> <td style="text-align: left">106515</td> <td style="text-align: left">106525</td> <td style="text-align: left">106512</td> </tr> </tbody> </table> <p>In this case, all four tools resulted in similar word counts. The exception here is the “King Lear” EPUB, which for Textract gave a word count that was about 10 thousand lower than for the other tools. I haven’t looked in detail where this difference is coming from exactly, but it confirms that in its current state, Textract isn’t a suitable tool for our purposes.</p> <h2 id="table-of-contents">Table of Contents</h2> <p>Depending on the structure of the source EPUB, the extraction result may or may not contain a table of contents. In <a href="https://idpf.org/epub/20/spec/OPF_2.0.1_draft.htm">EPUB 2</a>, the table of contents is implemented as an XML-formatted “Navigation Control File” (NCX). The NCX was replaced by the <a href="https://www.w3.org/publishing/epub3/epub-overview.html#sec-nav-nav-doc">“Navigation Document”</a> (which is an XHTML file) in EPUB 3. Neither Tika nor EbookLib extract NCX resources, but both do extract Navigation Documents. Consequently, in most cases the extraction result only includes a table of contents for EPUB 3 files. Both Textract and PyMuPDF extract neither the NCX nor the Navigation Document.</p> <h2 id="conclusions">Conclusions</h2> <p>Based on these tests, Tika-python, EbookLib and PyMuPDF all look like potentially suitable Python-based tools for extracting unformatted text from EPUB files. Out of these, Tika-python provides the most straightforward interface. Tika also supports a wide range of other file formats, so any code based on Tika’s text extraction can be easily extended to other formats later.</p> <p>The inclusion of tags and alt-text descriptions for images in Tika’s output may be a problem though. As an example, imagine a researcher who uses Tika-python to analyse the emergence of certain words or phrases through time using EPUB versions of 19th century books. Any alt-text descriptions in such materials would most likely be contemporary, and as such they would “pollute” the original “signal” (19th century text) with modern language. So, prospective users of Tika-python should carefully review whether this behaviour is acceptable for their use case. The inclusion of optical character recognition output from embedded images in the extraction result can also result in some unexpected surprises, so it’s important that users are aware of Tika’s default behaviour in this regard.</p> <p>EbookLib doesn’t have these drawbacks, but the absence of a high-level text extraction interface does require some more work on the user’s side. Also, since EbookLib only supports a limited number of Ebook formats, extending any code based on it to other file formats will be less straightforward.</p> <p>Although PyMuPDF generally looks useful for EPUB text extraction, its built-in text wrapping with the addition of linebreaks might be unwanted for some use cases. PyMuPDF’s page-based document model also makes this library somewhat more involved to use, compared against the other tested tools.</p> <p>In its current form, Textract is not suitable for our use case.</p> <h2 id="limitations">Limitations</h2> <p>It’s important to highlight the limitations of this analysis. First, it is based on only two small, homogeneous data sets, both of which only contain structurally valid EPUB files. It’s unclear how well these results translate to more heterogeneous collections (which often contain files that violate the format specifications in various ways). Second, the main objective here was to obtain a broad impression of the behaviour of the tested tools. The scope didn’t include an in-depth analysis of the accuracy and completeness of the extraction results. Finally, I didn’t look into the computational performance of the tested tools. As the SANE use case will only involve processing a limited number of files, performance isn’t important here.</p> <h2 id="link-to-demo-scripts">Link to demo scripts</h2> <p>EPUB text extraction demo:</p> <p><a href="https://github.com/KBNLresearch/textExtractDemo">https://github.com/KBNLresearch/textExtractDemo</a></p> <h2 id="revision-history">Revision history</h2> <ul> <li>21 January 2025: added PyMuPDF analysis.</li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>For convenience I actually used the EPUBCheck Python wrapper: <a href="https://github.com/titusz/epubcheck/">https://github.com/titusz/epubcheck/</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2023/03/09/extracting-text-from-epub-files-in-python https://bitsgalore.org/2023/03/09/extracting-text-from-epub-files-in-python 2023-03-09T00:00:00+01:00 <figure class="image"> <img src="/images/2023/02/donkey-cart.png" alt="Icon of a donkey that is pulling a cart that has the words bitsgalore.org written on it. In the background the sun is shining." /> <figcaption>Donkey, cart and sun icons licensed from <a href="https://thenounproject.com/">the Noun Project</a>.</figcaption> </figure> <p>I recently moved the two Internet domains I own away from the UK-based domain registrar I’d been using since 2004 to a EU-based registrar. While the actual domain transfer was fairly simple, finding a registrar that suited my specific situation turned out more difficult than expected. Leaving my old registrar also resulted in a surprise. It’s unlikely that my situation is unique, so I thought it would be useful to share my experiences in this blog post, and point to some useful online resources that I found along the way. The move also allowed me to make my domains up to date with (mostly security-related) modern internet standards. I’ll briefly address this in the final sections of this post. This includes some suggestions on how to make these optimizations work with a GitHub Pages-hosted sites like this one.</p> <!-- more --> <h2 id="a-brief-history-of-my-domains">A brief history of my domains</h2> <p>I registered my first (<em>.net</em>) domain in early 2004. At the time, I was only looking for a “stable” e-mail address that was independent of any proprietary service or internet service provider. Having my own domain seemed the best way to realise this. I’ve been using this domain ever since, and my private e-mail address has remained unchanged throughout that 19 year period. By the end of 2013 I registered a second (<em>.org</em>) domain, which is linked to this very site<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. Both domains were registered with a UK-based registrar and domain hosting company, and I was also using their name servers. Both domains are linked to servers that are run by other providers for e-mail and web hosting. For example, this site uses <a href="https://pages.github.com/">GitHub Pages</a> for web hosting.</p> <h2 id="why-move">Why move?</h2> <p>After my registrar was bought out by another company in 2018, it seems the new owners weren’t all that interested in keeping up with the latest Internet standards. The Dutch Internet Standards Platform has an <a href="https://internet.nl/">online tool</a> that checks a domain for the use of modern (mostly security-related) Internet standards, and I used this to analyse this site some time ago. <a href="https://web.archive.org/web/20230210222235/https://internet.nl/site/www.bitsgalore.org/1921771/">This result</a>, which is from a re-run I did just a few days before the domain transfer, shows that the site’s performance in this regard was very poor indeed:</p> <figure class="image"> <img src="/images/2023/02/internetnl-10022023.png" alt="Screenshot of website test summary info for domain www.bitsgalore.org. It shows the site got an overall score of 55%, reporting problems related to IPv6 reachability, DNSSEC and HTTPS." /> </figure> <p>Without going too much into detail here, the main culprits were:</p> <ol> <li>My registrar’s domain servers weren’t reachable via <a href="https://en.wikipedia.org/wiki/IPv6">IPv6</a> addresses.</li> <li>My registrar didn’t offer <a href="https://en.wikipedia.org/wiki/Domain_Name_System_Security_Extensions">DNSSEC</a> (Domain Name System Security Extensions).</li> </ol> <p>These technical shortcomings aside, in this post-Brexit era I’m not keen on routing my e-mail traffic through domain servers that are located in a non-EU country. On top of that, after a series of price hikes, my registrar had become significantly more expensive than most of its competitors. This was made even worse by the introduction of substantial bank transfer fees after Brexit. In the end, I felt the premium pricing I was paying was out of balance with the comparatively poor service provided.</p> <h2 id="2030-a-domain-odyssey">2030: A Domain Odyssey</h2> <p>With the renewal date of my email domain approaching again, I started the search for a new registrar. This was complicated somewhat by something that happened in November 2019. At that time, <a href="https://en.wikipedia.org/wiki/Internet_Society">Internet Society</a> made an <a href="https://www.internetsociety.org/news/press-releases/2019/ethos-capital-to-acquire-public-interest-registry-from-the-internet-society/">announcement</a> that they were about to sell the <em>.org</em> top-level domain to a private equity firm. Just a few months earlier, <a href="https://en.wikipedia.org/wiki/ICANN">ICANN</a> had lifted the price caps on <em>.org</em> domains. This would give the new owner total freedom to raise prices on <em>.org</em> domains without any limitations<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>. As this would include renewals of existing domains, I immediately renewed the <em>bitsgalore.org</em> domain for the maximum period possible (ten years), while it was still relatively cheap. In the end my ten year renewal proved to be unnecessary, as the sale of the <em>.org</em> top-level domain was <a href="https://www.theregister.com/2020/05/01/icann_stops_dot_org_sale/">vetoed by ICANN</a> several months later. But by that time I already was the proud owner of a domain that wouldn’t expire until late 2030<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>.</p> <h2 id="what-happens-to-the-remaining-registration-period">What happens to the remaining registration period?</h2> <p>So what are the implications of transferring a domain that has already been paid for in advance? Does the new registrar take over the remaining contract period (until 2030) in that case? Or would the transfer reset the registration period, meaning I would lose the remaining 8 years at my previous registrar, and I’d have to pay for them again? My online research on this didn’t come up with any definitive answers, but most sources I found suggested that in most cases, any remaining years at the “old” registrar are automatically transferred to the “new” registrar. Additional payments are usually limited to either a transfer fee, or the renewal of the domain by one more year. See for example <a href="https://webmasters.stackexchange.com/questions/44258/domain-name-transfer-do-you-lose-years-left-at-current-registrar">this thread</a> on Webmasters Stack Exchange, and <a href="https://www.thesitewizard.com/domain/renewing-domain-name.shtml">this article</a>.</p> <p>I also came across <a href="https://www.icann.org/en/registry-agreements/multiple/revised-verisign-registry-agreements-appendix-c-16-4-2001-en">this document that is part of ICANN’s registry agreements</a>, which states (section 2.4 Transfer Grace Period):</p> <blockquote> <p>Transfer (other than ICANN-approved bulk transfer). If a domain is transferred within the Transfer Grace Period, there is no credit. The expiration date of the domain is extended by one year up to a maximum term of ten years.</p> </blockquote> <p>If I’m reading this correctly, this also suggests that transferring a domain to another registrar doesn’t result in the loss of the remaining registration period.</p> <h2 id="dutch-treat">Dutch treat</h2> <p>However, it seems not all registrars work that way. TransIP, which is a major Dutch domain registrar and hosting provider, <a href="https://www.transip.nl/knowledgebase/artikel/24-een-domeinnaam-naar-transip-verhuizen/">explicitly mention on their website</a> that transferring an existing domain to them will start a new contract period, and that they will not take over any contract periods with the existing registrar. Two other Dutch registrars I contacted by e-mail told me this as well, adding that the transfer would reset the existing renewal date. Both advised me to stick with my current registrar. I don’t know if this behaviour is unique to Dutch registrars. In any case, the responses I got did prompt me to widen the search net to registrars in other EU countries.</p> <h2 id="enter-the-franco-german-axis">Enter the Franco-German axis</h2> <p><a href="https://webmasters.stackexchange.com/a/44347">One of the replies</a> in the aforementioned Webmasters Stack Exchange thread mentions <a href="https://en.wikipedia.org/wiki/Gandi">Gandi</a>, a French registration and hosting company. The link in that post doesn’t work anymore, but a quick search on Gandi’s documentation site turned up <a href="https://docs.gandi.net/en/domain_names/transfer/transfer_table.html#ot">this table</a>. For a <em>.org</em> domain, this states that, after transfer, the “new expiration date will be 1 year after current expiration date”. While searching for some more information about Gandi, I stumbled across <a href="https://european-alternatives.eu/category/domain-name-registrar">this website</a>, which lists several EU-based domain name registrars. I’m not sure how independent the site is, and what its selection of registrars is exactly based on. With that said, I looked up the mentioned registrars, checked some online reviews, and this directed my attention to German domain and hosting company <a href="https://www.inwx.com/">INWX</a> as another potential candidate.</p> <p>I e-mailed both Gandi and INWX, and asked them what would be the status of my <em>bitsgalore.org</em> domain in case I transferred it to them. Both companies confirmed they would take over the remaining 8 years of the contract with my current registrar, and would only charge for renewing the domain by one more year (extending the registration until 2031). This was exactly the answer I was hoping for, as this would save me 8 years worth of renewal fees. I ultimately settled on INWX, although both companies look like good choices to me.</p> <h2 id="dealing-with-transfer-out-fees">Dealing with transfer-out fees</h2> <p>I then started the transfer out with my old registrar<sup id="fnref:7"><a href="#fn:7" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>, who turned out to charge a £10 + VAT administration fee for this on each domain. From what I understand, this practice is quite unusual, and in my case it would set me back for some £24. I don’t consider myself overly stingy, but paying money like that just to leave a company that’s not providing a great service to begin with seemed a bit much. A quick search turned up <a href="https://forums.digitalspy.com/discussion/comment/91500345/#Comment_91500345">this forum thread</a>. It links to ICANN’s <a href="https://www.icann.org/resources/pages/name-holder-faqs-2017-10-10-en">FAQs for Registrants: Transferring Your Domain Name</a>, which states (under “<em>My registrar is charging me a fee to transfer to a new registrar. Is this allowed?</em>”):</p> <blockquote> <p>[…] Registrars are allowed to set their own prices for this service so some may choose to charge a fee. However, a transfer cannot be denied due to non-payment of this transfer fee. There are other reasons your registrar can deny transfer request. See FAQ #8 above for more information.</p> </blockquote> <p>So, I contacted my old registrar, and asked them to waive the transfer fee, referring to the ICANN document. To their credit, they granted my request without any fuss, and promptly sent me the authorization codes!</p> <h2 id="starting-the-incoming-transfer">Starting the incoming transfer</h2> <p>With these authorization codes, I could now start the incoming end of the transfer at INWX. It’s worth pointing out it took five days for the transfer to take effect (I received a notification e-mail about this); from what I understand this is usually the case for <em>.org</em> and <em>.net</em> domains. However, I was able to edit the DNS records at INWX well before the actual transfer date. When the transfer happened, it didn’t result in any noticeable downtime on either domain.</p> <h2 id="modern-internet-standards">Modern Internet standards</h2> <p>Once I had verified that everything was working OK after the transfer, I activated <a href="https://en.wikipedia.org/wiki/Domain_Name_System_Security_Extensions">DNSSEC</a> on both of my domains. I then did a couple of tests with the <a href="https://internet.nl/">online tool</a> of The Dutch Internet Standards Platform<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>. A test on the <em>bitsgalore.org</em> apex (root) domain now <a href="https://web.archive.org/web/20230218140111/https://internet.nl/site/bitsgalore.org/1922728/">resulted</a> in a respectable 94% score, but repeating the test for the <em>www</em> subdomain <a href="https://web.archive.org/web/20230218140314/https://internet.nl/site/www.bitsgalore.org/1925412/">only yielded a score of 70%</a>. A closer look showed that the <em>www</em> subdomain was failing some of the DNSSEC tests.</p> <h2 id="making-dnssec-work-on-the-subdomain">Making DNSSEC work on the subdomain</h2> <p>The <a href="https://docs.github.com/en/pages/configuring-a-custom-domain-for-your-github-pages-site/managing-a-custom-domain-for-your-github-pages-site#configuring-an-apex-domain-and-the-www-subdomain-variant">documentation of GitHub Pages</a> on custom domains recommends to insert a <a href="https://en.wikipedia.org/wiki/CNAME_record">CNAME</a> record for <em>www</em> subdomains, and this is also how I originally set up this site. After checking with INWX’s support (which, by the way, is both excellent and fast) about my subdomain issue, they advised me to replace this CNAME record with a series of <a href="https://en.wikipedia.org/wiki/List_of_DNS_record_types#A">A</a> records for the www subdomain. In the end I went one step further, and also added corresponding <a href="https://en.wikipedia.org/wiki/List_of_DNS_record_types#AAAA">AAAA</a> records<sup id="fnref:6"><a href="#fn:6" class="footnote" rel="footnote" role="doc-noteref">6</a></sup>.</p> <p>To illustrate this further, I removed this record:</p> <table> <thead> <tr> <th style="text-align: left">Name</th> <th style="text-align: left">Type</th> <th style="text-align: left">TTL</th> <th style="text-align: left">Value</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">www</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">CNAME</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">86400</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">bitsgalore.github.io</code></td> </tr> </tbody> </table> <p>And replaced it with the records below:</p> <table> <thead> <tr> <th style="text-align: left">Name</th> <th style="text-align: left">Type</th> <th style="text-align: left">TTL</th> <th style="text-align: left">Value</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">www</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">A</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">86400</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">185.199.111.153</code></td> </tr> <tr> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">www</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">A</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">86400</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">185.199.110.153</code></td> </tr> <tr> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">www</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">A</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">86400</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">185.199.108.153</code></td> </tr> <tr> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">www</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">A</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">86400</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">185.199.109.153</code></td> </tr> <tr> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">www</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">AAAA</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">86400</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">2606:50c0:8001::153</code></td> </tr> <tr> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">www</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">AAAA</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">86400</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">2606:50c0:8000::153</code></td> </tr> <tr> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">www</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">AAAA</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">86400</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">2606:50c0:8003::153</code></td> </tr> <tr> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">www</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">AAAA</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">86400</code></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">2606:50c0:8002::153</code></td> </tr> </tbody> </table> <p>Here, the A records point to the <a href="https://en.wikipedia.org/wiki/Internet_Protocol_version_4">IPv4</a> addresses of GitHub’s servers, and the AAAA records point to the <a href="https://en.wikipedia.org/wiki/IPv6">IPv6</a> addresses of the same servers. I didn’t need to make any changes to the site’s GitHub repo.</p> <p>After this change, DNSSEC worked as intended for the <em>www</em> subdomain, and the Internet Standards tool now <a href="https://web.archive.org/web/20230217140008/https://internet.nl/site/www.bitsgalore.org/1928399/">resulted in a 97% score</a>:</p> <figure class="image"> <img src="/images/2023/02/internetnl-14022023.png" alt="Screenshot of website test summary info for domain www.bitsgalore.org. It shows the site got an overall score of 97%." /> </figure> <p>There are still a couple of minor things that could be improved (mainly related to GitHub’s servers it seems). Overall I’m happy with this result though, especially when keeping in mind that the site only scored a meagre <a href="https://web.archive.org/web/20230210222235/https://internet.nl/site/www.bitsgalore.org/1921771/">55%</a> just a week earlier.</p> <h2 id="email">Email</h2> <p>I also ran separate tests on <a href="https://internet.nl/test-mail/">email</a> for both my domains, and used the results to apply some optimizations. This is beyond the scope of this post (which is already getting too long), but I think it deserves a brief mention. For example, in November 2022 Google implemented a <a href="https://support.google.com/mail/answer/81126?hl=en&amp;ref_topic=7279058#auth-reqs">policy</a> that requires that the domains of incoming email messages are secured with either <a href="https://en.wikipedia.org/wiki/Sender_Policy_Framework">SPF</a> or <a href="https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail">DKIM</a>. If an incoming message doesn’t comply with this, Google’s servers will either mark it as spam, or even outright reject it. I learnt about this the hard way myself at the time, when all of a sudden I was unable to reach any of my email contacts with a Gmail address!</p> <h2 id="conclusion">Conclusion</h2> <p>The main thing I learnt about this domain move, is the importance of researching registrars’ terms and conditions for domains that have been paid for several years in advance. I was surprised at how different registrars handle this in unexpectedly different ways. The somewhat nebulous situation around transfer-out fees was another surprise, and it’s good to know one’s rights here. In the end, I estimate my online research in both areas saved me a total of around €150, which is quite substantial!</p> <p>This exercise also made me more aware of the importance of selecting a registrar that keeps up with modern Internet standards, by using name servers with IPv6 web addresses, and offering signed domain names (DNSSEC). Tools like <a href="https://internet.nl/">the one by the Dutch Internet Standards Platform</a> are hugely helpful for compliance testing against these standards. Overall, I’m very happy with the outcome. Both my domains are now largely compliant with modern Internet standards, while running at a lower cost compared to my previous registrar.</p> <h2 id="useful-links-and-resources">Useful links and resources</h2> <ul> <li> <p><a href="https://internet.nl/">Internet.nl</a> - online tool by the Dutch Internet Standards Platform that tests web and email domains for use of modern Internet standards.</p> </li> <li> <p><a href="https://european-alternatives.eu/category/domain-name-registrar">European domain name registrars</a> - lists some more EU-based domain registrars.</p> </li> <li> <p><a href="https://www.icann.org/resources/pages/name-holder-faqs-2017-10-10-en">FAQs for Registrants: Transferring Your Domain Name</a> - by ICANN.</p> </li> <li> <p><a href="https://www.icann.org/en/registry-agreements/multiple/revised-verisign-registry-agreements-appendix-c-16-4-2001-en">Revised VeriSign Registry Agreements: Appendix C</a> - part of ICANN’s <a href="https://en.wikipedia.org/wiki/Generic_top-level_domain">Generic Top-Level Domain</a> (gTLD) Registry Agreements.</p> </li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>This was originally meant to be a personal blog covering a variety of subjects. That never really worked out, and in 2019 I re-launched the site as a home for my digital preservation writings old and new. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>See The Register for details - <a href="https://www.theregister.com/2019/11/20/org_registry_sale_shambles/">“Internet world despairs as non-profit .org sold for $$$$ to private equity firm, price caps axed”</a> <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>My other domain, which I only use for e-mail, is a .net domain, and I renew this annually. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:7"> <p>Before starting a domain transfer, it’s a good idea to first copy your domain’s DNS settings (which you can find in your registrar’s admin interface) to a text file. This will enable you to enter the relevant DNS records at your new registrar later. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>After the transfer it took several hours for the tool to pick up the new name servers. This is probably caching-related, and something to keep in mind when using tools like these on freshly transferred domains. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:6"> <p>I’m not completely sure the AAAA records are really necessary here. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2023/02/20/moving-my-internet-domains https://bitsgalore.org/2023/02/20/moving-my-internet-domains 2023-02-20T00:00:00+01:00 <figure class="image"> <img src="/images/2023/01/ipmlab-laptop-1024.jpg" alt="Photo of a laptop running the Ipmlab software. In the foreground is a removable USB floppy drive with some 3.5 inch floppies lying on top of it. To the right of the laptop is a vintage floppy storage box that contains more floppies." /> </figure> <p>In 2017 I wrote <a href="/2017/06/19/image-and-rip-optical-media-like-a-boss">a blog post</a> on <a href="https://github.com/KBNLresearch/iromlab">Iromlab</a> (an acronym for “Image and Rip Optical Media Like A Boss”), a custom-built software tool that streamlines imaging and ripping of optical media using an Acronova Nimbie disc robot. The KB has been using Iromlab since 2019 as part of an <a href="https://www.kb.nl/over-ons/projecten/digitalisering-optische-dragers">ongoing effort</a> to preserve the information contained in its vast collection of legacy optical media. This project is expected to reach its completion later this year, but as demonstrated <a href="/2020/02/20/offline-digital-carriers-kb-deposit-collection">by this earlier inventory</a>, our deposit collection also contains various other types of legacy media that are under threat of becoming inaccessible. Out of these, 3.5 inch floppy disks are the most common data carriers (after optical media), so it made sense to focus on these as a next step.</p> <p>Using the existing Iromlab-based workflow as a starting point, I created a preliminary workflow tool that can be used for imaging our 3.5” floppies (and various other types portable media). In this post I’ll explain how this tool came about, and highlight some of the challenges I encountered during its development.</p> <!-- more --> <h2 id="the-plan">The plan</h2> <p>Since we already have a workflow in place for optical media, it seemed to make sense to use Iromlab (which uses IsoBuster as the main imaging application) as a starting point. This would allow us to re-use most of the software components of the existing optical media workflow with only minor modifications. It also implied that Windows (10) would be the target platform. So far for the theory, but as we’ll see below, things got slightly more complicated and messy along the way!</p> <h2 id="iromlab-to-ipmlab">Iromlab to Ipmlab</h2> <p>As a first step I forked the existing Iromlab code. Unlike optical media, the possibilities for automating the load and unload process are limited for floppies (although some hackers have successfully <a href="https://hackaday.com/2012/03/31/floppy-autoloader-takes-the-pain-out-of-archiving-5000-amiga-disks/">repurposed vintage floppy duplicators into DIY autoloaders</a>). So, I started by removing all code that controls the Nimbie disc robot, and adapted the overall worklow accordingly. Next I removed everything that is specific to optical media, and made some small changes to the IsoBuster call, in order to better accommodate 3.5 inch floppies. As all software needs a name, I settled on “Ipmlab” (Image Portable Media Like A Boss). In a twist of irony, the development of Ipmlab wasn’t exactly boss-like, as will become clear from the remainder of this post.</p> <h2 id="enter-aaru">Enter Aaru</h2> <p>Although the tests with an early prototype initially went well, I wasn’t entirely happy with IsoBuster’s behaviour in case of floppies with damaged sectors. Depending on the configuration settings, IsoBuster either requires manual intervention in this case, or alternatively it aborts the imaging process altogether (filling in the missing sectors with placeholder bytes). On Twitter, Robin François suggested the open-source, cross-platform <a href="https://www.aaru.app/">Aaru</a> software as a possible alternative imaging solution. This looked worthy of further exploration. Although I hadn’t used Aaru before, the results of some quick tests looked promising enough, so I added a simple Aaru wrapper module to Ipmlab.</p> <h2 id="write-blocker-woes">Write blocker woes</h2> <p>This all seemed to work fine at first, but once I connected any of my (external USB) floppy drives<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">1</a></sup> through a write blocker (Tableau T8u forensic USB Bridge), Aaru would sometimes throw device exceptions.</p> <p>It’s relevant here to mention that for testing I mostly used some old, DOS-formatted 3.5 inch floppies from my personal collection. These include both <a href="https://obsoletemedia.org/3-5-inch-microfloppy-high-density/">“high density”</a> disks with a capacity of 1.44 MB, as well as some older <a href="https://obsoletemedia.org/3-5-inch-microfloppy/">“double density”</a> disks, which can only hold 720 KB worth of data. The main pattern in the Aaru crashes was, that they typically happened whenever I tried to process a “high density” disk after having processed one or more “double density” ones, or vice versa. The crashes didn’t happen when the floppy drives were connected directly to my machine.</p> <h2 id="ddrescue-tests">Ddrescue tests</h2> <p>To get a better idea what was going on, I ran more tests using an alternative imaging application (<a href="https://www.gnu.org/software/ddrescue/">Ddrescue</a>), and repeated these tests on another machine with a different operating system (Linux Mint).</p> <p>I started by imaging one “double density” (720 KB) disk. Any subsequent floppies were typically imaged without problems, provided that these were also “double density” disks. These each resulted in a 737 KB disk image, that I could mount normally on my Linux machine. On the other hand, following up a “double density” disk with a “high density” disk resulted in the following output from Ddrescue:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Press Ctrl-C to interrupt ipos: 720896 B opos: 720896 B non-tried: 0 B rescued: 737280 B pct rescued: 100.00%, <span class="nb">read </span>errors: 0 </code></pre></div></div> <p>This may look normal at first, as ddrescue does not report any errors. On closer inspection though, we see that only 737 KB of data were extracted from the disk. This is unexpected, because a “high density” disk should really result in a 1.5 MB disk image. But the 737 KB value <em>does</em> correspond exactly to the expected size for a “double density” disk! So in spite of the lack of any error messages from Ddrescue, about half of the data on the disks are missing from the image file!</p> <p>As an additional test I switched off the write blocker, switched it on again, and then repeated the above experiment, but now starting with some “high density” disks. These were all imaged correctly, each resulting in 1.5 MB image files. Following this up with any of my “double density” disks invariably resulted in Ddrescue reporting read errors like this:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pct rescued: 48.88%, read errors: 7372 </code></pre></div></div> <p>So apparently Ddrescue expected 1.44 MB of data (the size of the “high density” disk I inserted first), instead of the 720 KB of a “double density” disk. The size of the resulting disk image in this case was 1.5 MB, with the first 737 KB containing the data from the disk, followed by null bytes for the remaining part of the file.</p> <p>Running Ddrescue on an <em>empty</em> floppy drive resulted in an image with only null bytes (all bad blocks), with an image size of either 737 KB or 1.5 MB, depending on whether a “double” or “high” density floppy had been inserted first.</p> <p>Repeating any of the above tests with Aaru mostly resulted in Aaru device exceptions. At first sight, this all suggested that the medium size that is exposed by the write blocker to the operating system somehow remains “stuck” to the size of the first floppy that was loaded after switching it on, and isn’t updated afterwards.</p> <h2 id="tableau-response">Tableau response</h2> <p>I contacted Tableau (the manufacturer of the write blocker) about this, who suggested that the behaviour might be caused by an inability of the write blocker to differentiate between the actual floppy and the USB adapter device. They didn’t envisage a short-term solution for this because of the complexities involved, and advised me to look for some alternative solution that doesn’t use this write blocker.</p> <h2 id="windows-writing-pesky-folders">Windows writing pesky folders</h2> <p>This created a bit of a dilemma. By default, Windows 10 (the target platform for the workflow) automatically tries to write a <a href="https://www.thewindowsclub.com/system-volume-information-folder">“System Volume Information” folder</a> to a newly detected floppy disk (or other storage medium for that matter):</p> <figure class="image"> <img src="/images/2023/01/svi-win10.png" alt="Screenshot of file manager showing the contents of a floppy disk. In addition to the original files, it also shows a System Volunme Information directory that was added by Windows 10 after inserting the floppy into the machine." /> <figcaption>Contents of a floppy. The System Volume Information directory was not originally part of the floppy, but was automatically added by Windows 10 after insertion into the floppy drive.</figcaption> </figure> <p>Of course we can prevent this by setting a floppy’s write-protect tab to the “protected” position (which is always a good idea). However, if an operator forgets to do this (and I think it’s reasonable to expect this will happen every once in a while), this would immediately result in changes to the source media. Reportedly <a href="https://superuser.com/questions/1199823/how-to-prevent-creation-of-system-volume-information-folder-in-windows-10-for">it is possible</a> to disable the automatic creation of “System Volume Information” folders, but this involves some rather ugly messing with the Windows registry and the services settings.</p> <h2 id="linux-to-the-rescue">Linux to the rescue</h2> <p>After some deliberation with my colleagues on the operational side, we decided to abandon Windows as a target platform, and switch to <a href="https://linuxmint.com/">Linux Mint</a> instead. Unlike Windows, Linux (Mint) doesn’t try to write anything to a floppy upon insertion. For additional protection we can <a href="https://github.com/KBNLresearch/ipmlab/blob/main/doc/setupGuide.md#disable-automatic-mounting-of-removable-media">disable automatic mounting of removable media</a>, which is fairly easy to set up. In combination with using a floppy’s write-protect tab, this provides a level of protection against accidental write actions that looks pretty reasonable to me<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>. Since much of the code was Linux-compatible from the onset, adapting Ipmlab was relatively straightforward.</p> <h2 id="return-of-ddrescue">Return of Ddrescue</h2> <p>As initial tests with the adapted development version of Ipmlab showed no problems, I moved to the final step of packaging the code. Or so I thought. When running the installed version of Ipmlab, Aaru now invariably failed with an unhandled exception. After submitting an <a href="https://github.com/aaru-dps/Aaru/issues/749">issue on this</a>, Aaru’s main developer Natalia Portillo suggested the issue was most likely caused by the console handler class that is used by the current stable (5.3) version of Aaru, and that the latest development version (which uses a different console handler) might give better results. I was able to confirm this with a test with the latest (6.0) development version, which indeed worked without any problems.</p> <p>However, as there’s still a lot of work to do before a stable 6.0 release will be ready, this again introduced a minor dilemma. In the end, I put my Aaru plans on hold (at least for now), and added a Ddrescue wrapper module to Ipmlab. I then made the imaging application a user-defined configuration variable, giving a user the choice between either Ddrescue or Aaru. This means that once Aaru 6.0 is ready for release, Ipmlab will support it<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>.</p> <h2 id="ipmlab-features">Ipmlab features</h2> <p>Like Iromlab, Ipmlab uses simple a batch structure, where each imaged medium is represented by a single directory that contains the disk image and all associated metadata. This includes:</p> <ul> <li>a report file in <a href="https://en.wikipedia.org/wiki/Digital_Forensics_XML">Digital Forensics XML format</a> (created using <a href="https://www.sleuthkit.org/sleuthkit/">Sleuth Kit</a>)</li> <li>a file with bibliographical metadata</li> <li>a <a href="https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html#Mapfile-structure">Ddrescue map file</a> (only if Ddrescue is used)</li> <li>various Aaru-specific metadata files (only if Aaru is used)</li> <li>a file with SHA-512 checksums of all of the above files.</li> </ul> <p>Each batch also contains:</p> <ul> <li>a batch manifest, which is a comma-delimited text file with all information that is needed to process the batch into ingest-ready Submission Information Packages further down the processing chain</li> <li>a detailed batch log</li> <li>an “end-of-batch” file that indicates that the batch was finalized</li> <li>a file that indicates the version of Ipmlab.</li> </ul> <p>Optionally, it’s possible to send some of the user input (PPN identifier or title string) to the corresponding Ipmlab entry widget from an external application through a <a href="https://en.wikipedia.org/wiki/Network_socket">socket connection</a>. This is useful for integrating Ipmlab with e.g. external administrative applications.</p> <figure class="image"> <img src="/images/2023/01/ipmPostSubmit.png" alt="Screenshot of Ipmlab user interface. Logging widget shows that Ddrescue is running in the background." /> <figcaption>Ipmlab interface while processing a floppy.</figcaption> </figure> <h2 id="limitations">Limitations</h2> <p>At the time I’m writing this post, Ipmlab is still undergoing some further tests by colleagues from our digital preservation department. I expect that the outcome of these will result in some further changes to the software. We don’t currently have any reliable information on the layouts of the 3.5 inch floppies in the KB collection, but I expect these will be mostly DOS/Windows formatted ones. There is still a possibility that further tests reveal a significant proportion of more exotic layouts, such as old Mac-formatted disks. In that case, various changes to the workflow would be needed, as these disk types would require the use of an external hardware controller, and possibly a flux-based imaging approach.</p> <h2 id="adapting-ipmlab">Adapting Ipmlab</h2> <p>Source code and documentation for Ipmlab are available <a href="https://github.com/KBNLresearch/ipmlab">here</a>. Like Iromlab, some features of Ipmlab are quite specific to the situation at the KB. The most obvious example is the use of unique PPN (Pica Production Number) identifiers for each medium, which are linked to records in the KB’s catalog. For a more generic user experience, it’s possible to disable the PPN lookup as a configuration option, which replaces the PPN entry field with a “title” field. The PPN lookup module could also be easily adapted to any other identifier or cataloguing system. Likewise, it’s fairly straightforward to implement alternative disk imaging applications.</p> <h2 id="acknowledgments">Acknowledgments</h2> <p>Thanks are due to Natalia Portillo and Robin François for their help and suggestions on Aaru.</p> <h2 id="additional-links-and-resources">Additional links and resources</h2> <ul> <li><a href="https://github.com/KBNLresearch/ipmlab">Ipmlab</a></li> <li><a href="https://github.com/KBNLresearch/ipmlab/tree/main/doc">Ipmlab documentation</a></li> <li><a href="/2020/02/20/offline-digital-carriers-kb-deposit-collection">Offline digital data carriers in the KB deposit collection</a></li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:2"> <p>These are refurbished “old” drives; this is significant, as modern USB floppy drives often cannot read “double density” disks. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:1"> <p>It’s important to stress that disabling auto-mount does not prevent against intentional write actions: a user can still mount a floppy manually, and write or delete files. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>Barring any major changes to Aaru’s command-line interface. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2023/01/23/writing-a-workflow-tool-for-imaging-portable-media https://bitsgalore.org/2023/01/23/writing-a-workflow-tool-for-imaging-portable-media 2023-01-23T00:00:00+01:00 <figure class="image"> <img src="/images/2022/11/fail_whale_by_ka_92-d5ra7vf.jpg" alt="Painting of a whale, which is lifted above the beach by a swarm of red Twitter birds. In the foreground, an elderly couple watches the scene." /> <figcaption><a href="https://web.archive.org/web/20130204212401/https://ka-92.deviantart.com/art/fail-whale-348157275">"Fail whale"</a> by <a href="https://web.archive.org/web/20130128112538/http://ka-92.deviantart.com/">Kuni (ka-92)</a> (license unknown), based on "Lifting a Dreamer" by <a href="http://www.yiyinglu.com/?portfolio=lifting-a-dreamer-aka-twitter-fail-whale">Yiying Lu</a>.</figcaption> </figure> <p>As a total collapse of Twitter is becoming more likely every day, many Twitter users have started to archive their personal data from the platform while it still exists. Twitter allows you to request and download your personal archive. Even though this works well, and the quality of the archive is surpringly good, it does have some shortcomings. The result of these shortcomings will be that some information in the archive (e.g. on followed accounts and followers) will be lost once Twitter ceases to exist. Some other information (in particular full, unshortened URLs) <em>is</em> included in the archive, but it is not easily accessible from the main HTML interface. The good news is, that some excellent tools exist to fix these shortcomings.</p> <p>In this post I outline the workflow I used to preserve my own Twitter archive, and while doing so I also provide some background information on the shortcomings of the Twitter archive. Since some of these steps may, at first sight, be a little daunting for less tech-savvy readers, I’ve tried to provide step-by-step instructions where possible.</p> <!-- more --> <h2 id="disclaimer">Disclaimer</h2> <p>First a little disclaimer: I don’t want to suggest that what follows is the “best”, “proper” or even a “good” way to do this. For most of the issues I’m addressing here, several alternatives exist, and some may be better than what I’m suggesting here. Also, an in-depth analysis of the Twitter archive, and a comparison of different tools and approaches are both beyond the scope of this post (besides I don’t have the time for this). Ultimately, all I’m describing here is a workflow that, for now, looks “good enough” for my own purposes. I’m sharing it here because I <em>think</em> it will be “good enough” for many others as well, or at least provide a reasonable starting point. With that said, I can’t give any guarantees here, so keep this in mind while reading what follows!</p> <h2 id="request-and-download-twitter-archive">Request and download Twitter archive</h2> <p>Most importantly, you need to request and download your Twitter data<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. The basic steps are:</p> <ol> <li>While logged in to Twitter, go to <a href="https://twitter.com/settings/account">your account’s settings</a>.</li> <li>Click on “Download an archive of your data”, and then simply follow the instructions.</li> <li>Twitter will issue a notification when the archive is ready: <img src="/images/2022/11/archive-notification.png" alt="" /> This may take a while. When I did this two weeks ago, I had to wait over 24 hours; various people have told me that currently it take several days. Once ready, you can download the archive as one large ZIP file.</li> <li>Once downloaded, unzip the file to a folder.</li> </ol> <h2 id="shortcomings-of-the-twitter-archive">Shortcomings of the Twitter archive</h2> <p>As several people have pointed out before, the archive data provided by Twitter have a number of shortcomings. Below a (probably incomplete) overview of the most obvious ones.</p> <h3 id="shortened-tco-links">Shortened t.co links</h3> <p>Perhaps most importantly, if you access your Twitter archive from your web browser by opening the “Your archive.html” file, any clickable hyperlinks you see are shortened t.co links that use <a href="https://web.archive.org/web/20221116161843/https://help.twitter.com/en/using-twitter/url-shortener">Twitter’s link shortener service</a>:</p> <p><img src="/images/2022/11/tweet-link.png" alt="" /></p> <p>These links will stop working once Twitter goes down. The original, unshortened links are actually stored in the underlying JSON data that are part of the archive (file “tweets.js” in the “data” folder). For example, <a href="https://gist.github.com/bitsgalore/cfdff3ce67f1ffa85f67e87c778a9e75">here’s the full data from the Tweet in that screenshot</a>. Let’s zoom in on its “urls” attribute:</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w"> </span><span class="nl">"urls"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"url"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"https://t.co/y2gpEVvjAd"</span><span class="p">,</span><span class="w"> </span><span class="nl">"expanded_url"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"https://youtu.be/C47ZCosJPAw"</span><span class="p">,</span><span class="w"> </span><span class="nl">"display_url"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"youtu.be/C47ZCosJPAw"</span><span class="p">,</span><span class="w"> </span><span class="nl">"indices"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"246"</span><span class="p">,</span><span class="w"> </span><span class="s2">"269"</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w"> </span></code></pre></div></div> <p>As you can see, the “urls” attribute contains both a shortened t.co link (“url”), the unshortened link (“expanded_url”), and a display link (“display_url”). So all the data are there, but the unshortened links just aren’t accessible from the archive’s web interface.</p> <h3 id="full-size-images">Full-size images</h3> <p>The Twitter archive only contains downscaled versions of posted images. Clicking on an image to expand it takes you to the live Twitter website. Again, this is something that will stop working once Twitter is gone.</p> <h3 id="twitter-network-data">Twitter network data</h3> <p>Although the Twitter archive does contain files with the accounts that you follow and the accounts that follow you, these are given as numerical identifiers that will most likely be meaningless if Twitter disappears. Here’s an example:</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"following"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"accountId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"216697909"</span><span class="p">,</span><span class="w"> </span><span class="nl">"userLink"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"https://twitter.com/intent/user?user_id=216697909"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <p>Here, the user link with account ID 216697909 (<a href="https://twitter.com/intent/user?user_id=216697909">https://twitter.com/intent/user?user_id=216697909</a>) resolves to <a href="https://twitter.com/openpreserve">the account of the Open Preservation Foundation</a>. Once Twitter is gone, this user link will stop resolving, which will make it very hard to figure out which actual person or organization was associated with it.</p> <p>Fortunately, some excellent solutions exist that can fix most of the above shortcomings, and they are pretty easy to use as well. Let’s start with addressing the network data issue, as this is something you can do immediately, without having to wait for your Twitter archive.</p> <h2 id="preserve-your-twitter-network-with-fedifinder">Preserve your Twitter network with FediFinder</h2> <p><a href="https://fedifinder.glitch.me/">FediFinder</a> is an online tool written by Luca Hammer. Its main purpose is to find Fediverse accounts that correspond to your Twitter connections. However, it also works great for tracking your entire Twitter network, including your followers, accounts that you follow, and list members.</p> <p>Just follow these steps:</p> <ol> <li> <p>While you’re logged into your Twitter account, go to <a href="https://fedifinder.glitch.me/">https://fedifinder.glitch.me/</a></p> </li> <li> <p>Click on the “Authorize to extract handles” button (this will give FediFinder read permissions to your Twitter account):</p> <p><img src="/images/2022/11/ff-authorize.png" alt="" /></p> </li> <li> <p>In the page that appears, click on “Scan followings”, “Scan followers” and “Load lists”. For each list, click on “Scan members”. If all goes well you’ll see something like this:</p> <p><img src="/images/2022/11/ff-scan-finished.png" alt="" /></p> </li> <li> <p>Scroll past the list of search results to the bottom of the page, and click on the small “accounts.csv” link<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>:</p> <p><img src="/images/2022/11/ff-accounts.png" alt="" /></p> </li> </ol> <p>Once downloaded, you can open the file in any spreadsheet software. For each account, it contains the Twitter user name, the real name, any lists of which the account is a member, associated Fediverse handles, the account’s location, and its profile description.</p> <p>Optionally, afterwards you may want to remove FediFinder from your Twitter account’s connected app list using <a href="https://twitter.com/settings/connected_apps/">this link</a>:</p> <p><img src="/images/2022/11/ff-revoke.png" alt="" /></p> <h2 id="improve-archive-with-twitter-archive-parser">Improve archive with Twitter archive parser</h2> <p>Tim Hutton has written <a href="https://github.com/timhutton/twitter-archive-parser">twitter-archive-parser</a>, which is a Python tool that fixes most of the remaining issues, and some other issues as well<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>. Most importantly, it creates both HTML and <a href="https://en.wikipedia.org/wiki/Markdown">Markdown</a> versions of the archive, with all shortened t.co URLs replaced with their original versions. Optionally, it can also be instructed to download full-size versions of images<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>.</p> <p>To use it, follow these steps:</p> <ol> <li> <p>Install Python 3 on your system if you don’t have it already. If you’re a Windows user and you’re not sure how to do this, check out <a href="https://github.com/pettarin/python-on-windows">Alberto Pettarin’s easy to follow instructions</a>.</p> </li> <li> <p>Then download the twitter-archive-parser script. For this, just right-click <a href="https://raw.githubusercontent.com/timhutton/twitter-archive-parser/main/parser.py">this link</a>, select “Save link as”, and save the file into the folder where you extracted the archive<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>.</p> </li> <li> <p>Start a command-prompt or terminal, and change the working directory to the folder where you extracted the archive. Windows users may want to have another look at <a href="https://github.com/pettarin/python-on-windows">Alberto Pettarin’s explainer</a>, in particular the “Using The Command Prompt” and “Changing The Working Directory” sections.</p> </li> <li> <p>Run the twitter-archive-parser script from the command-prompt or terminal, using the following command:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> python parser.py </code></pre></div> </div> <p>Depending on your operating system, you may need to replace “python” with “python3”:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> python3 parser.py </code></pre></div> </div> <p>The script will most likely ask you to confirm the installation of one or two Python modules (“requests” and “imagesize”); if this happens, just type “y” and press the Enter key.</p> </li> <li> <p>Finally, the script will ask if you want to download original-size images. Type “y” if you want this, or “n” if not. Note that downloading the images can take quite a bit of time (depending on the size of your Twitter archive, and the number of images and multimedia it contains).</p> <p>As an aside, I noticed that archive parser was unable to download some images. I don’t particularly care about this myself, but if images are crucially important to you, the “media” folder contains a download log (file “download_log.txt”) with full details of the download status of each image.</p> </li> </ol> <p>And that’s all there is to it!</p> <h2 id="alternative-approaches-and-variations">Alternative approaches and variations</h2> <p>As I already mentioned in the introduction, I’m not making any claims that the above steps are the “proper” way to do this, and various alternative approaches exist. In this section I’ll highlight a few of these.</p> <p>After I published the first draft of this post, I found <a href="https://wiert.me/2022/11/12/exporting-your-twitter-content-converting-to-markdown-and-getting-the-image-alt-texts-thanks-isotopp-hbeckpdx-for-the-info-and-kcgreenn-dreamjar-for-the-comic/">this earlier post by Jeroen Wiert Pluimers</a>. This describes an overall workflow that is similar to the one described here (it also uses archive parser), but adds the extraction of alt-text image descriptions, exporting of bookmarks, and archiving of t.co URL shortener links to the the <a href="https://web.archive.org/">Wayback Machine</a>.</p> <p>Ed Summers has written an <a href="https://github.com/docnow/twitter-archive-unshorten">alternative t.co unshortener</a>, which is explained <a href="https://inkdroid.org/2022/11/20/t-dot-co/">in this blog post</a>. Ed has also <a href="https://inkdroid.org/2022/11/16/bookmarks/">written this post on archiving Twitter bookmarks</a>, which are not included in the archive data. And <a href="https://gist.github.com/ryanfb/53f167feebde61ad262c4f09d879733e">here’s a Ruby script by Ryan Baumann</a> that exports your Twitter Bookmarks to JSON (note that the scripts deletes the original bookmarks to get around API limits). As I never use bookmarks, I’m not really interested in this myself, but this might be important to some users.</p> <p>Mike Hucka’s <a href="https://github.com/mhucka/taupe">Taupe tool</a> extracts URLs from tweets, retweets, replies, quote tweets, and “likes” from a personal Twitter archive, and writes these to a comma-delimited text file. This is especially useful if you want to preserve linked resources (e.g. by sending them to a web archive).</p> <p><a href="https://github.com/tweetback/tweetback/">Tweetback</a> is an open-source software project by Zach Leatherman that creates a static website from your Twitter archive. See <a href="https://www.zachleat.com/web/tweetback/">this blog post</a> for more information about it, as well as some links to static websites that were created with it. This looks like a really interesting option for those who want to publish their Twitter archive.</p> <p><a href="https://github.com/sqiouyilu/twitter-nest">Twitter-nest</a> is a set of tools by S. Qiouyi Lu that allow you to create a decentralized Twitter clone on WordPress.</p> <p>Internet Archive has also made it possible to upload your Twitter archive to its <a href="https://archive.org/web/">Wayback Machine</a>. See <a href="https://help.archive.org/help/how-to-archive-your-tweets-with-the-wayback-machine/">this article for instructions</a>.</p> <p>Finally <a href="https://techcrunch.com/2022/11/21/quit-twitter-better-with-these-free-tools-that-make-archiving-a-breeze/">this TechCrunch feature</a> mentions some more tools that might be worth perusing.</p> <h2 id="final-thoughts">Final thoughts</h2> <p>It’s good to keep in mind that the development of tools like archive parser currently <a href="https://digipres.club/web/@timhutton@mathstodon.xyz/109377490421206529">moves at a pretty fast pace</a>. Just as an example, when I ran archive parser only yesterday (19th of November), it wasn’t able to report Twitter followers and followings, whereas this functionality is included in the latest (20th of November) release. So I expect these tools will become even better over time (but don’t wait for it, as there’s a real chance that Twitter may be gone by then!).</p> <p>Please feel free to use the comment section to post links to alternative tools or methods, or if you spot any glaring errors in this post.</p> <h2 id="additional-links-and-resources">Additional links and resources</h2> <ul> <li> <p><a href="https://github.com/timhutton/twitter-archive-parser">Twitter archive parser</a> (“related tools” section lists some more more tools that might be useful)</p> </li> <li> <p><a href="https://fedifinder.glitch.me/">FediFinder</a></p> </li> <li> <p><a href="https://github.com/pettarin/python-on-windows">A step-by-step guide on installing Python and using the Command Prompt for Windows</a></p> </li> <li> <p><a href="https://inkdroid.org/2022/11/16/bookmarks/">Ed Summers on archiving Twitter bookmarks</a></p> </li> <li> <p><a href="https://ryanfb.github.io/etc/2022/11/21/exporting_as_many_of_your_twitter_bookmarks_as_possible.html">Ryan Baumann - Exporting As Many of Your Twitter Bookmarks As Possible</a></p> </li> <li> <p><a href="https://gist.github.com/ryanfb/53f167feebde61ad262c4f09d879733e">Ruby script by Ryan Baumann that exports Twitter bookmarks to JSON</a> (this also deletes the original bookmarks to get around API limits, so use with caution!)</p> </li> <li> <p><a href="https://inkdroid.org/2022/11/20/t-dot-co/">Alternative t.co unshorten approach by Ed Summers</a></p> </li> <li> <p><a href="https://github.com/docnow/twitter-archive-unshorten">twitter-archive-unshorten tool</a></p> </li> <li> <p><a href="https://wiert.me/2022/11/12/exporting-your-twitter-content-converting-to-markdown-and-getting-the-image-alt-texts-thanks-isotopp-hbeckpdx-for-the-info-and-kcgreenn-dreamjar-for-the-comic/">Jeroen Wiert Pluimers on exporting your Twitter content, converting to Markdown and getting the image alt-texts</a></p> </li> <li> <p><a href="https://github.com/mhucka/taupe">Taupe tool - extracts URLs from Twitter archive</a></p> </li> <li> <p><a href="https://techcrunch.com/2022/11/21/quit-twitter-better-with-these-free-tools-that-make-archiving-a-breeze/">Quit Twitter better with these free tools that make archiving a breeze (TechCrunch)</a></p> </li> <li> <p><a href="https://github.com/tweetback/tweetback/">Tweetback by Zach Leatherman</a></p> </li> <li> <p><a href="https://www.zachleat.com/web/tweetback/">Blog post on Tweetback</a></p> </li> <li> <p><a href="https://github.com/sqiouyilu/twitter-nest">Twitter-nest by S. Qiouyi Lu</a></p> </li> <li> <p><a href="https://help.archive.org/help/how-to-archive-your-tweets-with-the-wayback-machine/">How to archive your Tweets with the Wayback Machine</a></p> </li> </ul> <h2 id="revision-history">Revision history</h2> <ul> <li> <p>21 November 2022: updated info about Twitter’s notification when the archive download is ready.</p> </li> <li> <p>21 November 2022: added references to Ryan Baumann’s bookmarks export script.</p> </li> <li> <p>21 November 2022: added references to earlier blog post by Jeroen Wiert Pluimers.</p> </li> <li> <p>22 November 2022: added references to Taupe tool by Mike Hucka.</p> </li> <li> <p>23 November 2022: added references to TechCrunch feature.</p> </li> <li> <p>29 November 2022: added references to Tweetback.</p> </li> <li> <p>1 December 2022: added references to Twitter-nest.</p> </li> <li> <p>15 December 2022: added references to Wayback archiving; renamed and re-strucured final sections.</p> </li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:4"> <p>More details can be found <a href="https://help.twitter.com/en/managing-your-account/accessing-your-twitter-data">here</a>, although some of the info looks slightly out of date. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:1"> <p>The large “Export fedifinder_accounts.csv” link will give you a file that only includes Fediverse accounts. This can be useful for automating your follows on Mastodon, but if you want detailed information on <em>all</em> Twitter accounts you (also) need to use the small link! <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>Among other things, it also fixes some issues with Direct Messages, which by default don’t include user handles. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>The version of Archive parser I used wasn’t able to produce lists of followers and followings. More recent versions (20 November 2022 an onward) do have this functionality, but the output is less detailed than FediFinder. Also, it doesn’t distinguish between direct follows and follows through a list. So it’s probably a good idea to use FediFinder for this. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>The name of this folder looks something like this:<br /> “twitter-2022-11-07-2366bc80316…4e7b77”. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2022/11/20/how-to-preserve-your-personal-twitter-archive https://bitsgalore.org/2022/11/20/how-to-preserve-your-personal-twitter-archive 2022-11-20T00:00:00+01:00 <figure class="image"> <iframe width="750" height="422" src="https://www.youtube-nocookie.com/embed/C47ZCosJPAw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe> </figure> <p>Dutch electro outfit the Digital Dark Age Crew are one of the forgotten legends that used to be a mainstay of Rotterdam’s late 90s to mid-2000s underground electro scene. Their music was characterised by relentless electro beats, sparse synth lines, and lyrics that typically commented on the fragility and transience of digital media and digital information in general. In a twisted turn of events, this very theme would eventually define the Digital Dark Age Crew’s own history, ultimately leading to the group’s dramatic demise in 2007. After a fifteen year absence, the Digital Dark Age Crew have now made a long overdue comeback with their new track “Wheel Out the Digital Dark Age Klaxon”, which was released today on the occasion of <a href="https://www.dpconline.org/events/world-digital-preservation-day">World Digital Preservation Day 2022</a>. Time to take a look back at the history of the Digital Dark Age Crew, and their continued relevance today!</p> <!-- more --> <h2 id="early-history">Early history</h2> <p>Much is unclear about the exact origins of the Digital Dark Age Crew, and over the years, several of the group’s former members have provided different and often conflicting accounts on this. However, music historians agree that the project was founded in 1997 by Dutch producer Marinus Nullbyte, who ultimately became the sole constant member throughout a succession of many lineup changes. Seen invariably donning a space helmet during live performances and in rare photos of the group, Nullbyte’s true identity remains a secret to this day, and has been the subject of much online speculation among fans.</p> <h2 id="releases">Releases</h2> <p>The Digital Dark Age Crew’s output was released mainly on cheap, home-burned CD-recordables. These would often spontaneously disintegrate within a year after purchase, which somehow added to the group’s mystique. From 2005 onward, they also made their releases available through <a href="https://en.wikipedia.org/wiki/Myspace">Myspace</a>. Fourteen years later, it turned out that these were all lost as a result of Myspace’s infamous <a href="https://mashable.com/article/myspace-data-loss">server migration incident</a>.</p> <figure class="image"> <img src="/images/2022/11/logo-ddac-blog.png" alt="Human figure holding large floppy disk, with to its left the words Digital Dark Age Crew" /> <figcaption>Digital Dark Age Crew logo.</figcaption> </figure> <h2 id="burn-a-million-pdfs">Burn a Million PDFs</h2> <p>In their later years, the group shifted their focus to multimedia productions. Without doubt the most notorious of these was 2007’s “Watch the Digital Dark Age Crew Burn a Million PDFs”. While shooting video footage for this project just outside their headquarters, the group accidentally burned down their recording studio, in an unfortunate combination of misjudging the flammability of the source material they were working with, and a sudden change of wind direction. The fire, which resulted in the loss of all the master tapes of their output thus far, effectively marked the end of the Digital Dark Age Crew.</p> <h2 id="digital-dark-age-klaxon">Digital Dark Age Klaxon</h2> <p>Following the group’s dissolution, founding member Marinus Nullbyte turned his attention to a variety of other activities and projects. His fascination with the fragility of digital media continued, and was only further strengthened by the events that led to the downfall of his former group. Around 2016, Nullbyte first came up with the idea of a “digital dark age klaxon”, which was to be an intricate device designed to provide a warning signal in the event of impending digital doom. Details about the inner workings of this digital dark age klaxon remain somewhat nebulous, as Nullbyte has kept the project, which appears to be in a perpetual state of development, under close guard. He maintains though, as he has done for the past six years, that something tangible will be released “soon”.</p> <h2 id="legacy">Legacy</h2> <p>With much of their output gone forever, it’s tempting to think of the Digital Dark Age Crew as a group that is now largely forgotten, except perhaps by a few of their most determined old fans. But dig a little deeper, and it’s clear that their influence is still present today.</p> <p>For a start, it may be no coincidence that the term <a href="https://en.wikipedia.org/wiki/Digital_dark_age">“Digital Dark Age”</a> made its <a href="https://archive.ifla.org/IV/ifla63/63kuny1.pdf">first appearance in the information science literature</a> in 1997, the very year the Digital Dark Age Crew appeared on the scene. It’s not a long stretch to assume that the paper’s author took inspiration from both the message and the infectious electro sounds of the Digital Dark Age Crew’s early work. If true, the Digital Dark Age crew may have set in motion a chain of events that ultimately led to “father of the internet” Vint Cerf <a href="https://www.bbc.com/news/science-environment-31450389">warning of a coming “digital Dark Age”</a> in 2015.</p> <p>Moreover, Nullbyte’s post-Digital Dark Age Crew work on the digital dark age klaxon has been an ongoing source of puzzlement, speculation and inspiration among digital archivists and other information professionals the world over, and has even spawned <a href="https://twitter.com/hashtag/WheelOutTheDigitalDarkAgeKlaxon?src=hashtag_click&amp;f=live">its dedicated hashtag</a> on Twitter (which, ironically, could <a href="https://midrange.tedium.co/issues/elon-musk-social-media-cult-of-personality">itself</a> be he subject of <a href="https://davidgerard.co.uk/blockchain/2022/11/03/welcome-to-twitter-mr-musk-heres-your-accordion/">looming digital disaster</a> at the time of writing).</p> <figure class="image"> <img src="/images/2022/11/m-nullbyte-1014px.jpg" alt="Marinus Nullbyte, wearing a space helmet while holding a small synthesizer which has a microphone attached to it" /> <figcaption>Marinus Nullbyte of the Digital Dark Age Crew during a recording session for "Wheel Out the Digital Dark Age Klaxon".</figcaption> </figure> <h2 id="comeback">Comeback</h2> <p>Earlier this year, Nullbyte decided to once again put on his old space helmet, and revive the Digital Dark Age Crew project. The first result of this latest incarnation of the group, now essentially a Marinus Nullbyte solo venture, is the new track “Wheel Out The Digital Dark Age Klaxon”. The trademark Digital Dark Age Crew elements are instantly recognisable. Musically, the track offers a fresh take on the classic Digital Dark Age Crew sound. The lyrics intersperse the familiar subject of digital decay with key moments from the history of the group itself. Both themes are held together by a chorus that alludes to the digital dark age klaxon, but which ultimately leaves the listener none the wiser about the specifics of this puzzling device!</p> <p>It will be interesting to see whether the return of the Digital Dark Age Crew is a one-off, or perhaps the start of a new string of releases. Whatever the case, for now please enjoy … Wheel Out the Digital Dark Age Klaxon!</p> <figure class="image"> <iframe width="750" height="422" src="https://www.youtube-nocookie.com/embed/C47ZCosJPAw" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe> <figcaption>Digital Dark Age Crew - Wheel Out the Digital Dark Age Klaxon.</figcaption> </figure> <p>Direct link to video on YouTube, in case the embedded player doesn’t work:</p> <p><a href="https://youtu.be/C47ZCosJPAw">https://youtu.be/C47ZCosJPAw</a></p> <p>Video and audio (FLAC) files are available for download from Zenodo:</p> <p><a href="https://zenodo.org/record/7390809">https://zenodo.org/record/7390809</a></p> https://bitsgalore.org/2022/11/03/wheel-out-the-digital-dark-age-klaxon https://bitsgalore.org/2022/11/03/wheel-out-the-digital-dark-age-klaxon 2022-11-03T00:00:00+01:00 <figure class="image"> <img src="/images/2022/06/storage-media.jpg" alt="Still life of assorted storage media" /> </figure> <p>This blog post covers some techniques that can be used to identify storage media and storage devices using Python and the Windows API. This can be useful for distinguishing between different types of portable storage media, such as floppy disks and USB thumb drives. It also presents a demo script that integrates these techniques.</p> <!-- more --> <h2 id="preservation-of-portable-storage-media">Preservation of portable storage media</h2> <p>In 2019 the KB started a <a href="https://www.kb.nl/over-ons/projecten/digitalisering-optische-dragers">major initiative</a> to safeguard the information on optical data carriers in its collection, such as CD-ROMs, DVDs and audio CDs. The <a href="https://github.com/KBNLresearch/iromlab">Iromlab</a> software is a central component of the workflow we’re using for this. The optical media preservation project will most likely reach its completion near the end of this year. As the KB collection also contains many other types of portable digital storage media (see <a href="/2020/02/20/offline-digital-carriers-kb-deposit-collection">this blog post for an overview</a>), we’re currently looking into ways to preserve those as well. The first candidate will be 3.5 inch floppy disks. I’m currently working on a derivative of the <em>Iromlab</em> software that can be used for imaging floppies, but also other types of portable media, such as USB thumb drives. Like <em>Iromlab</em>, it wraps around <a href="https://www.isobuster.com/">IsoBuster</a> to do the actual imaging. Since I’d like the new software to be able to deal with a variety of physical storage media types and devices, it would be useful to have a way to automatically identify the medium type prior to the imaging stage. On Windows (which is the target environment for this work), this kind of hardware identification is possible through the <a href="https://docs.microsoft.com/en-us/windows/win32/api/">Win32 API</a>.</p> <h2 id="challenges">Challenges</h2> <p>In Python, parts of the Win32 API can be accessed using the <a href="https://github.com/mhammond/pywin32">pywin32</a> wrapper extensions, but using these extensions can be a bit of a challenge.There are a couple of reasons for this. First, the pywin32 documentation is incomplete and partially outdated. This means you’ll often have to rely on Microsoft’s documentation of the underlying C++ API. Using the API also involves some fairly low-level operations, such as creating file handles, defining output buffers and parsing binary output. Many of the (few) relevant code examples that can be found online are also outdated, which meant I had to combine examples and documentation from a variety of sources in order to get things working. Taken together, this all makes working with the Win32 API pretty daunting.</p> <h2 id="purpose-of-this-post">Purpose of this post</h2> <p>In this post I’ll try to document how I made basic media and device identification work for me. Given any device attached to a logical Windows drive (e.g. the “A drive”, “D drive”, etcetera), the objectives here are to identify:</p> <ol> <li>the storage media type (i.e. hard disk, floppy drive, etc.);</li> <li>the hardware device, and the storage media types that are supported by it.</li> </ol> <p>I’ll link to the relevant documentation throughout. At the end of this post I also present a simple demo script that ties everything together, and which could be used as a basis for your own code.</p> <h2 id="preparation">Preparation</h2> <p>In order to use the techniques described here, you’ll <a href="https://github.com/mhammond/pywin32">pywin32</a>, which you can install using:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip <span class="nb">install </span>pywin32 </code></pre></div></div> <p>Then create a new Python file, and add the following imports:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="n">sys</span> <span class="kn">import</span> <span class="n">struct</span> <span class="kn">import</span> <span class="n">argparse</span> <span class="kn">import</span> <span class="n">win32api</span> <span class="kn">import</span> <span class="n">win32file</span> <span class="kn">import</span> <span class="n">winioctlcon</span> </code></pre></div></div> <h2 id="device-name">Device name</h2> <p>First of all we need the device name that corresponds to the logical drive we’re interested in (e.g. drive A, C, D, and so on). For this we use (using the “A” drive as an example here):</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">drive</span><span class="o">=</span><span class="sh">"</span><span class="s">A</span><span class="sh">"</span> <span class="c1"># Low-level device name of device assigned to logical drive </span><span class="n">driveDevice</span> <span class="o">=</span> <span class="sh">"</span><span class="se">\\\\</span><span class="s">.</span><span class="se">\\</span><span class="sh">"</span> <span class="o">+</span> <span class="n">drive</span> <span class="o">+</span> <span class="sh">"</span><span class="s">:</span><span class="sh">"</span> </code></pre></div></div> <h2 id="create-file-handle">Create file handle</h2> <p>Next we create a file handle for this device, using the <a href="http://timgolden.me.uk/pywin32-docs/win32file__CreateFile_meth.html">win32file.CreateFile</a> method:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">handle</span> <span class="o">=</span> <span class="n">win32file</span><span class="p">.</span><span class="nc">CreateFile</span><span class="p">(</span><span class="n">driveDevice</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">win32file</span><span class="p">.</span><span class="n">FILE_SHARE_READ</span><span class="p">,</span> <span class="bp">None</span><span class="p">,</span> <span class="n">win32file</span><span class="p">.</span><span class="n">OPEN_EXISTING</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="bp">None</span><span class="p">)</span> </code></pre></div></div> <h2 id="retrieve-disk-geometry-info">Retrieve disk geometry info</h2> <p>We can now use the <a href="http://timgolden.me.uk/pywin32-docs/win32file__DeviceIoControl_meth.html">win32file.DeviceIoControl</a> function to obtain information about a physical disk:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">diskGeometry</span> <span class="o">=</span> <span class="n">win32file</span><span class="p">.</span><span class="nc">DeviceIoControl</span><span class="p">(</span><span class="n">handle</span><span class="p">,</span> <span class="n">winioctlcon</span><span class="p">.</span><span class="n">IOCTL_DISK_GET_DRIVE_GEOMETRY</span><span class="p">,</span> <span class="bp">None</span><span class="p">,</span> <span class="mi">24</span><span class="p">)</span> </code></pre></div></div> <p>In this case, the <em>DeviceIoControl</em> call contains four arguments:</p> <ul> <li>the file handle for the device;</li> <li>the control code <code class="language-plaintext highlighter-rouge">winioctlcon.IOCTL_DISK_GET_DRIVE_GEOMETRY</code> (documented <a href="https://docs.microsoft.com/en-us/windows/win32/api/winioctl/ni-winioctl-ioctl_disk_get_drive_geometry">here</a>), which tells <em>DeviceIoControl</em> to retrieves information about a physical disk’s geometry;</li> <li>an input buffer, which is set to <code class="language-plaintext highlighter-rouge">None</code> in this case because it is not used;</li> <li>the output buffer size in bytes.</li> </ul> <p>The output buffer size must be equal to (or larger than) the output that is returned by the function call. The output (a string of raw bytes) is defined by the <code class="language-plaintext highlighter-rouge">DISK_GEOMETRY</code> structure, which is <a href="https://docs.microsoft.com/en-us/windows/win32/api/winioctl/ns-winioctl-disk_geometry">documented here</a>. It contains 5 fields. The first field is a <a href="https://docs.microsoft.com/en-us/windows/win32/api/winnt/ns-winnt-large_integer-r1">large integer</a> (8 bytes); the remaining fields are all <a href="https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-dtyp/262627d8-3418-4627-9218-4ffe110850b2">4-byte unsigned integers</a>. This means the total size of the <code class="language-plaintext highlighter-rouge">DISK_GEOMETRY</code> structure is 24 bytes, so we use this as the buffer size.</p> <h3 id="parse-disk_geometry">Parse DISK_GEOMETRY</h3> <p>The <em>MediaType</em> field is the second item of the <code class="language-plaintext highlighter-rouge">DISK_GEOMETRY</code> structure. It is an unsigned integer (4 bytes) that starts at byte offset 8 of our <em>diskGeometry</em> variable. We can use Python’s <a href="https://docs.python.org/3/library/struct.html">struct module</a> to interpret the raw bytes into an integer value:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">offset</span> <span class="o">=</span> <span class="mi">8</span> <span class="n">mediaTypeCode</span> <span class="o">=</span> <span class="n">struct</span><span class="p">.</span><span class="nf">unpack</span><span class="p">(</span><span class="sh">"</span><span class="s">&lt;I</span><span class="sh">"</span><span class="p">,</span> <span class="n">diskGeometry</span><span class="p">[</span><span class="n">offset</span><span class="p">:</span><span class="n">offset</span> <span class="o">+</span> <span class="mi">4</span><span class="p">])[</span><span class="mi">0</span><span class="p">]</span> </code></pre></div></div> <p>Here the “<code class="language-plaintext highlighter-rouge">&lt;I</code>” format string informs unpack that the bytes represent a little-Endian unsigned integer. Note that <code class="language-plaintext highlighter-rouge">struct.unpack</code> always returns a tuple, hence the “<code class="language-plaintext highlighter-rouge">\[0\]</code>” index at the end.</p> <p>The resulting <em>mediaTypeCode</em> value is an integer number. These numbers can be mapped back to media type strings using the <em>MEDIA_TYPE</em> enumeration in <a href="https://github.com/mhammond/pywin32/blob/main/win32/Lib/winioctlcon.py">winioctlcon.py</a>. These strings are in turn documented <a href="https://docs.microsoft.com/en-us/windows/win32/api/winioctl/ne-winioctl-media_type">here</a>.</p> <h2 id="additional-device-info">Additional device info</h2> <p>Although the above method already allows us to identify, as an example, many types of floppy disks, it cannot distinguish between a USB thumb drive and a CD-ROM drive, both of which are simply identified as “RemovableMedia”. For some more granularity, we can call <a href="http://timgolden.me.uk/pywin32-docs/win32file__DeviceIoControl_meth.html">win32file.DeviceIoControl</a> with the <code class="language-plaintext highlighter-rouge">IOCTL_STORAGE_GET_MEDIA_TYPES_EX</code> control code (documented <a href="https://docs.microsoft.com/en-us/windows/win32/api/winioctl/ni-winioctl-ioctl_storage_get_media_types_ex">here</a>):</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">getMediaTypes</span> <span class="o">=</span> <span class="n">win32file</span><span class="p">.</span><span class="nc">DeviceIoControl</span><span class="p">(</span><span class="n">handle</span><span class="p">,</span> <span class="n">winioctlcon</span><span class="p">.</span><span class="n">IOCTL_STORAGE_GET_MEDIA_TYPES_EX</span><span class="p">,</span> <span class="bp">None</span><span class="p">,</span> <span class="mi">2048</span><span class="p">)</span> </code></pre></div></div> <p>The function arguments are largely identical to the earlier (disk geometry) call, but this time we use a 2048 byte output buffer. This is a somewhat arbitrary value. Unlike <code class="language-plaintext highlighter-rouge">IOCTL_DISK_GET_DRIVE_GEOMETRY</code>, the output size of <code class="language-plaintext highlighter-rouge">IOCTL_STORAGE_GET_MEDIA_TYPES_EX</code> is not fixed (see also the explanation that follows below), so I’m simply using a fairly large value to ensure the buffer will be large enough to fit the output under all circumstances. The output (again a string of raw bytes) is defined by the <code class="language-plaintext highlighter-rouge">GET_MEDIA_TYPES</code> structure, which is <a href="https://docs.microsoft.com/en-us/windows/win32/api/winioctl/ns-winioctl-get_media_types">documented here</a>.</p> <h3 id="parse-get_media_types">Parse GET_MEDIA_TYPES</h3> <p>The <code class="language-plaintext highlighter-rouge">GET_MEDIA_TYPES</code> structure is made up of the following fields:</p> <ol> <li>a 4-byte unsigned integer that represents a device type code;</li> <li>another 4-byte unsigned integer that represents the number of <code class="language-plaintext highlighter-rouge">DEVICE_MEDIA_INFO</code> structures</li> <li>a pointer to the first <code class="language-plaintext highlighter-rouge">DEVICE_MEDIA_INFO</code> structure.</li> </ol> <p>We can read the device type code and the number of <code class="language-plaintext highlighter-rouge">DEVICE_MEDIA_INFO</code> structures like this:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">deviceCode</span> <span class="o">=</span> <span class="n">struct</span><span class="p">.</span><span class="nf">unpack</span><span class="p">(</span><span class="sh">"</span><span class="s">&lt;I</span><span class="sh">"</span><span class="p">,</span> <span class="n">getMediaTypes</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">4</span><span class="p">])[</span><span class="mi">0</span><span class="p">]</span> <span class="n">mediaInfoCount</span> <span class="o">=</span> <span class="n">struct</span><span class="p">.</span><span class="nf">unpack</span><span class="p">(</span><span class="sh">"</span><span class="s">&lt;I</span><span class="sh">"</span><span class="p">,</span> <span class="n">getMediaTypes</span><span class="p">[</span><span class="mi">4</span><span class="p">:</span><span class="mi">8</span><span class="p">])[</span><span class="mi">0</span><span class="p">]</span> </code></pre></div></div> <p>The resulting <em>deviceCode</em> value is an integer number, which again can be mapped back to a device type string using an enumeration in <a href="https://github.com/mhammond/pywin32/blob/main/win32/Lib/winioctlcon.py">winioctlcon.py</a>. The value of <em>mediaInfoCount</em> tells us the number of <code class="language-plaintext highlighter-rouge">DEVICE_MEDIA_INFO</code> structures we need to parse.</p> <h3 id="parse-device_media_info">Parse DEVICE_MEDIA_INFO</h3> <p>The <code class="language-plaintext highlighter-rouge">DEVICE_MEDIA_INFO</code> structure itself is documented <a href="https://docs.microsoft.com/en-us/windows/win32/api/winioctl/ns-winioctl-device_media_info">here</a>. At first sight this may look a little intimidating, but essentially it just describes a “union” of 3 possible 32-byte structures. This means that each <code class="language-plaintext highlighter-rouge">DEVICE_MEDIA_INFO</code> instance is made up of either of those structures. For the purpose of this post, we’re only interested in the <em>MediaType</em> field here, and this field can be at an identical location (the second item of the <code class="language-plaintext highlighter-rouge">DEVICE_MEDIA_INFO</code> structure) for two of the three possible variants. Only in case of a tape device, <em>MediaType</em> is the first item. Tape devices can be identified from the value of <em>deviceCode</em>. Using this information, we can use the code below to iterate over all <code class="language-plaintext highlighter-rouge">DEVICE_MEDIA_INFO</code> structures, and extract their respective media type codes:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Start position in GET_MEDIA_TYPES structure # (remember we already read two 4-byte integers from it, # hence the 8 byte start offset!) </span><span class="n">offset</span> <span class="o">=</span> <span class="mi">8</span> <span class="c1"># Loop over DEVICE_MEDIA_INFO structures </span><span class="k">for</span> <span class="n">_</span> <span class="ow">in</span> <span class="nf">range</span><span class="p">(</span><span class="n">mediaInfoCount</span><span class="p">):</span> <span class="k">if</span> <span class="n">deviceCode</span> <span class="ow">in</span> <span class="p">[</span><span class="mi">31</span><span class="p">,</span> <span class="mi">32</span><span class="p">]:</span> <span class="c1"># Tape device, mediaTypeCode is first item </span> <span class="n">mediaTypeCode</span> <span class="o">=</span> <span class="n">struct</span><span class="p">.</span><span class="nf">unpack</span><span class="p">(</span><span class="sh">"</span><span class="s">&lt;I</span><span class="sh">"</span><span class="p">,</span> <span class="n">getMediaTypes</span><span class="p">[</span><span class="n">offset</span><span class="p">:</span><span class="n">offset</span> <span class="o">+</span> <span class="mi">4</span><span class="p">])[</span><span class="mi">0</span><span class="p">]</span> <span class="n">offset</span> <span class="o">+=</span><span class="mi">8</span> <span class="k">else</span><span class="p">:</span> <span class="c1"># Not a tape device, so skip 8 byte cylinders value </span> <span class="n">offset</span> <span class="o">+=</span> <span class="mi">8</span> <span class="n">mediaTypeCode</span> <span class="o">=</span> <span class="n">struct</span><span class="p">.</span><span class="nf">unpack</span><span class="p">(</span><span class="sh">"</span><span class="s">&lt;I</span><span class="sh">"</span><span class="p">,</span> <span class="n">getMediaTypes</span><span class="p">[</span><span class="n">offset</span><span class="p">:</span><span class="n">offset</span> <span class="o">+</span> <span class="mi">4</span><span class="p">])[</span><span class="mi">0</span><span class="p">]</span> <span class="c1"># Skip to position of next DEVICE_MEDIA_INFO structure </span> <span class="n">offset</span> <span class="o">+=</span> <span class="mi">24</span> </code></pre></div></div> <p>The resulting <em>mediaTypeCode</em> values are integer numbers that can be mapped back to media type strings using the <em>MEDIA_TYPE</em> and <em>STORAGE_MEDIA_TYPE</em> enumerations in <a href="https://github.com/mhammond/pywin32/blob/main/win32/Lib/winioctlcon.py">winioctlcon.py</a>. These strings are in turn documented <a href="https://docs.microsoft.com/en-us/windows/win32/api/winioctl/ne-winioctl-media_type">here</a> and <a href="https://docs.microsoft.com/en-us/windows/win32/api/winioctl/ne-winioctl-storage_media_type">here</a>.</p> <h2 id="putting-it-all-together">Putting it all together</h2> <p>I created a simple <a href="https://github.com/KBNLresearch/detectStorageMediaType/blob/main/detectStorageMediaType.py">demo script</a> that ties everything discussed here together. It also contains lookup functions that translate the media type and device codes into human-readable strings. You can run it from the command prompt with one or more logical drive names as command-line arguments. For example:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python detectStorageMediaType.py A D E </code></pre></div></div> <p>The script then tries to establish the media type (from the disk geometry), the device type and the media types that are supported by the device, and reports the results back like this:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Drive: A Media type: F3_1Pt44_512 Drive: D Media type: RemovableMedia Device type: FILE_DEVICE_CD_ROM Supported media types: CD_ROM RemovableMedia Drive: E Media type: RemovableMedia Device type: FILE_DEVICE_DISK Supported media types: RemovableMedia </code></pre></div></div> <p>You may note that the output is incomplete in some cases, which is because the API calls do not work for all devices. In the above example, the <code class="language-plaintext highlighter-rouge">IOCTL_STORAGE_GET_MEDIA_TYPES_EX</code> control code could not be used to access the floppy drive attached to logical drive <em>A</em>, so no device output is produced for that drive.</p> <p>This example also showcases the level of detail that is reported for floppy disks. As can be seen <a href="https://docs.microsoft.com/en-us/windows/win32/api/winioctl/ne-winioctl-media_type">here</a>, the code <code class="language-plaintext highlighter-rouge">F3_1Pt44_512</code> indicates a 3.5” floppy disk, with 1.44MB and 512 bytes per sector.</p> <p>Finally, the logical drive <em>D</em> in the above example was a virtual optical drive with an ISO image of a DVD-ROM attached to it. Despite that, the device is identified as “FILE_DEVICE_CD_ROM” (not “FILE_DEVICE_DVD”!), and DVDs are not listed as a supported media type. This could be a limitation of my test approach, which was based on Windows 10 running in VirtualBox inside a Linux host environment<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. This would need additional testing with physical hardware.</p> <h2 id="final-note">Final note</h2> <p>This post only scratches the surface of what’s possible with the Windows API, but hopefully it will be useful to others to start their own explorations. I’ve only tested the code presented here with a limited number of devices (both physical and virtual ones) attached to a virtual machine running Windows 10. If you run into any surprises, or have suggestions for improvements, feel free to leave a comment here, or open a pull request for the demo script.</p> <h2 id="further-resources">Further resources</h2> <ul> <li><a href="https://github.com/KBNLresearch/detectStorageMediaType">Demo script</a></li> <li><a href="http://timgolden.me.uk/pywin32-docs/">PyWin32 Documentation</a></li> <li><a href="https://docs.microsoft.com/en-us/windows/win32/api/">Programming reference for the Win32 API (Microsoft)</a></li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>For reasons unknown to me I was unable to attach a physical DVD drive to this virtual machine for my tests. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2022/06/14/identification-of-physical-storage-media-with-python-and-the-windows-api https://bitsgalore.org/2022/06/14/identification-of-physical-storage-media-with-python-and-the-windows-api 2022-06-14T00:00:00+02:00 <figure class="image"> <img src="/images/2022/04/cds.jpg" alt="Compact Discs still life" /> </figure> <p>It’s been a while since the last release of the <a href="https://github.com/KBNLresearch/isolyzer">Isolyzer</a> tool, but after four years of near-inactivity I just published <a href="https://github.com/KBNLresearch/isolyzer/releases/tag/1.4.0">Isolyzer 1.4</a>. In this post I provide some background information on how this release came about, and I briefly explain the main changes.</p> <!-- more --> <h2 id="history-of-isolyzer">History of Isolyzer</h2> <p>For those unfamiliar with the Isolyzer tool, here’s a brief recap. Isolyzer started its life in 2015. At that time I had just started working on optical media preservation and disc imaging. One of the problems I ran into, was that imaging optical media would occasionally result in incomplete (i.e. truncated) disc images. Worse, I found it near impossible to reliably identify these incomplete disc images with existing software tools. After a lot of digging into the specs, I figured out how to estimate expected file sizes using the block-level information in the ISO 9660 file system. I then applied this in a dedicated Python tool. Since many “ISO images” that exist in the wild are actually hybrids of different file systems (e.g. ISO 9660, Apple HFS or HFS+ and UDF), I gradually added support for those file systems as well. As a result, Isolyzer also became increasingly useful as a tool for extracting information about file systems inside ISO images. More details about Isolyzer’s history can be found <a href="/2017/01/13/detecting-broken-iso-images-introducing-isolyzer">here</a> and <a href="/2017/07/12/update-on-isolyzer-udf-hfs-and-more">here</a>.</p> <h2 id="apple-file-system-block-size-confusion">Apple file system block size confusion</h2> <p>The initial trigger that started the work on this release was an email from Tyler Thorsted. He had run into a number of problematic ISO images of CD-ROMs that had been created with Roxio Toast, a widely used CD-burning software application for Apple Macintosh. Although these images contained an Apple HFS file system with Apple Partition maps, the underlying file systems were not properly identified and parsed by Isolyzer (and some other tools as well). After some poking around with a Hex editor, it turned out that the file system in the offending images was arranged into 2048-byte blocks, instead of the 512-byte blocks that were expected by Isolyzer. This seemed easy enough to fix: instead of assuming a fixed 512-byte block size, I changed the code to use the block size value from the “zero block” structure instead, and then use this to iterate over the partition maps.</p> <h2 id="more-block-size-confusion-and-a-better-fix">More block size confusion (and a better fix)</h2> <p>Sadly, Tyler noticed that the fixed code resulted in missing partition map output for some images that were handled correctly by Isolyzer prior to my fix. For this particular case, the culprit was that the block size value in the “zero block” (which read 2048 bytes) didn’t correspond to the partition map data (which were arranged into 512 byte blocks). The “official” documentation that I could find provided some rather conflicting information on the correct implementation of partition maps, and their relation to the zero block. In the end I worked around this by using a simple iterative procedure that looks for partition maps at a number of pre-defined byte offsets into the image. The resulting match at the first (smallest) byte offset then gives the correct, actual block size. More details and links to sample files can be found in the (now closed) <a href="https://github.com/KBNLresearch/isolyzer/issues/22">issue on Github</a>.</p> <h2 id="high-sierra-file-system">High Sierra file system</h2> <p>Tyler also made a <a href="https://github.com/KBNLresearch/isolyzer/issues/26">feature request</a> for support of the <a href="https://web.archive.org/web/20220111023846/https://www.os2museum.com/files/docs/cdrom/CDROM_Working_Paper-1986.pdf">“High Sierra” file system</a>. This was the de facto standard CD-ROM file system for a few years during the late eighties, before it was made redundant by ISO 9660. The High Sierra format is very similar to ISO 9660, and shares the same data structures and fields (although the fields often appear in a different order). This made it relatively easy to add support for it in Isolyzer. As before, see the <a href="https://github.com/KBNLresearch/isolyzer/issues/26">Github issue</a> for more details and a link to a sample file.</p> <h2 id="xml-schema">XML schema</h2> <p>I also created an <a href="https://github.com/KBNLresearch/isolyzer/blob/main/xsd/isolyzer-v-1-0.xsd">XML schema</a> that makes it possible to validate Isolyzer’s output, and added a namespace definition. This should also make it easier to embed Isolyzer’s output into other XML files, if needed. I should stress that the schema has only had limited testing so far, so please get in contact (or <a href="https://github.com/KBNLresearch/isolyzer/issues/new/choose">report an issue</a>) if you come across unexpected behaviour.</p> <h2 id="native-wildcard-expansion-on-non-windows-platforms">Native wildcard expansion on non-Windows platforms</h2> <p>One long-standing annoyance (at least for me) has been Isolyzer’s handling of wildcard expressions at the command-line, which needed to be wrapped in quotation marks on Linux to work correctly. This is rooted in the slightly different ways wildcards are handled by Linux and Windows, respectively. This release fixes this by delegating wildcard expansion to the operating system for non-Windows operating systems. On Windows, Isolyzer still takes care of wildcard expansion (as it did before), since Windows does not do this natively.</p> <h2 id="python-2-no-longer-supported">Python 2 no longer supported</h2> <p>This release also no longer provides support for Python 2 (which is now obsolete), so you’ll have to use Python 3. Windows users can also use the stand-alone binaries, which don’t require Python at all.</p> <h2 id="other-changes">Other changes</h2> <p>In addition to the above changes, this release also includes several minor bug fixes. I have also expanded the set of test files, and set up some unit tests that use these tests files. These changes are all invisible to the user, but they make the Isolyzer release process more straightforward.</p> <h2 id="installation">Installation</h2> <p>For a fresh single-user install with pip use:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip <span class="nb">install </span>isolyzer <span class="nt">--user</span> </code></pre></div></div> <p>To upgrade an existing version of Isolyzer, use:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip <span class="nb">install </span>isolyzer <span class="nt">--upgrade</span> <span class="nt">--user</span> </code></pre></div></div> <p>Alternatively, Windows users can use the binaries that are available from <a href="https://github.com/KBNLresearch/isolyzer/releases/tag/1.4.0">the release page</a>. As always, these binaries are completely stand-alone, and don’t require Python on your machine.</p> <h2 id="feedback-welcome">Feedback welcome</h2> <p>As always, feedback on this release is very welcome. Please feel free to <a href="https://github.com/KBNLresearch/isolyzer/issues/new/choose">report an issue</a> if anything doesn’t work as expected!</p> <h2 id="acknowledgements">Acknowledgements</h2> <p>Thanks are due to Tyler Thorsted for the useful online discussions on the Apple and High Sierra issues, and for providing the necessary test files.</p> <h2 id="revision-history">Revision history</h2> <ul> <li>7 June 2022: changed all release candidate references to the final 1.4.0 release.</li> </ul> <h2 id="further-resources">Further resources</h2> <ul> <li><a href="https://github.com/KBNLresearch/isolyzer/releases/tag/1.4.0">Isolyzer 1.4.0 release page</a></li> <li><a href="https://github.com/KBNLresearch/isolyzer">Isolyzer on Github (with documentation)</a></li> </ul> https://bitsgalore.org/2022/04/20/introducing-isolyzer-1-4 https://bitsgalore.org/2022/04/20/introducing-isolyzer-1-4 2022-04-20T00:00:00+02:00 <figure class="image"> <img src="/images/2022/03/jm-cote-wiki.jpg" alt="Plumbers Tool Box" /> <figcaption><a href="https://commons.wikimedia.org/wiki/File:France_in_XXI_Century._Intencive_breeding.jpg">Intensive Breeding</a> by Jean Marc Cote, Public domain, via Wikimedia Commons.</figcaption> </figure> <p>At the KB we’ve been using JP2 (<a href="https://en.wikipedia.org/wiki/JPEG_2000">JPEG 2000</a> Part 1) as our primary image format for digitised newspapers, books and periodicals since 2007. The digitisation work is contracted out to external vendors, who supply the digitised pages as losslessly compressed preservation masters, as well as lossily compressed access images that are used within the <a href="https://www.delpher.nl/">Delpher</a> platform.</p> <p>Right now the KB is in the process of <a href="https://web.archive.org/web/20210215160819/https://www.kb.nl/en/news/2021/dutch-national-library-steps-into-a-new-future-of-digital-archiving">migrating its digital collections to a new preservation system</a>. This prompted the question whether it would be feasible to generate access JP2s from the preservation masters in-house at some point in the future, using software that runs inside the preservation system<sup id="fnref:6"><a href="#fn:6" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. As a first step towards answering that question, I created some simple proof of concept workflows, using three different JPEG 2000 codecs. I then tested these workflows with preservation master images from our collection. The main objective of this work was to find a workflow that both meets our current digitisation requirements, and is also sufficiently performant.</p> <!-- more --> <h2 id="master-and-access-requirements">Master and access requirements</h2> <p>The following table lists the requirements of our preservation master and access JP2s:</p> <table> <thead> <tr> <th style="text-align: left">Parameter</th> <th style="text-align: left">Value (master)</th> <th style="text-align: left">Value (access)</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">File format</td> <td style="text-align: left">JP2 (JPEG 2000 Part 1)</td> <td style="text-align: left">JP2 (JPEG 2000 Part 1)</td> </tr> <tr> <td style="text-align: left">Compression type</td> <td style="text-align: left">Reversible 5-3 wavelet filter</td> <td style="text-align: left">Irreversible 7-9 wavelet filter</td> </tr> <tr> <td style="text-align: left">Colour transform</td> <td style="text-align: left">Yes (only for colour images)</td> <td style="text-align: left">Yes (only for colour images)</td> </tr> <tr> <td style="text-align: left">Number of decomposition levels</td> <td style="text-align: left">5</td> <td style="text-align: left">5</td> </tr> <tr> <td style="text-align: left">Progression order</td> <td style="text-align: left">RPCL</td> <td style="text-align: left">RPCL</td> </tr> <tr> <td style="text-align: left">Tile size</td> <td style="text-align: left">1024 x 1024</td> <td style="text-align: left">1024 x 1024</td> </tr> <tr> <td style="text-align: left">Code block size</td> <td style="text-align: left">64 x 64 (2<sup>6</sup> x 2<sup>6</sup>)</td> <td style="text-align: left">64 x 64 (2<sup>6</sup> x 2<sup>6</sup>)</td> </tr> <tr> <td style="text-align: left">Precinct size</td> <td style="text-align: left">256 x 256 (2<sup>8</sup>) for 2 highest resolution levels; 128 x 128 (2<sup>7</sup>) for remaining resolution levels</td> <td style="text-align: left">256 x 256 (2<sup>8</sup>) for 2 highest resolution levels; 128 x 128 (2<sup>7</sup>) for remaining resolution levels</td> </tr> <tr> <td style="text-align: left">Number of quality layers</td> <td style="text-align: left">11</td> <td style="text-align: left">8</td> </tr> <tr> <td style="text-align: left">Target compression ratio layers</td> <td style="text-align: left">2560:1 [1] ; 1280:1 [2] ; 640:1 [3] ; 320:1 [4] ; 160:1 [5] ; 80:1 [6] ; 40:1 [7] ; 20:1 [8] ; 10:1 [9] ; 5:1 [10] ; - [11]</td> <td style="text-align: left">2560:1 [1] ; 1280:1 [2] ; 640:1 [3] ; 320:1 [4] ; 160:1 [5] ; 80:1 [6] ; 40:1 [7] ; 20:1 [8]</td> </tr> <tr> <td style="text-align: left">Error resilience</td> <td style="text-align: left">Start-of-packet headers; end-of-packet headers; segmentation symbols</td> <td style="text-align: left">Start-of-packet headers; end-of-packet headers; segmentation symbols</td> </tr> <tr> <td style="text-align: left">Sampling rate</td> <td style="text-align: left">Stored in “Capture Resolution” fields</td> <td style="text-align: left">Stored in “Capture Resolution” fields</td> </tr> <tr> <td style="text-align: left">Capture metadata</td> <td style="text-align: left">Embedded as XMP metadata in XML box</td> <td style="text-align: left">Embedded as XMP metadata in XML box</td> </tr> </tbody> </table> <p>As the table shows, most parameters are identical in both cases, except:</p> <ol> <li>The preservation masters are compressed losslessly, whereas for the access images irreversible (lossy) compression is used, using a fixed compression ratio of 20:1.</li> <li>The preservation masters contain 11 quality layers, whereas the access images only contain 8 quality layers.</li> </ol> <h2 id="deriving-the-access-image">Deriving the access image</h2> <p>In order to derive an access image from a preservation master, two approaches are possible:</p> <ol> <li>Create a “subset” from the preservation master that only contains the lower 8 quality layers (i.e. discarding the highest 3 layers).</li> <li>Do a full decode of the preservation master (e.g. to TIFF), and then re-compress the result to lossy JP2.</li> </ol> <p>The main advantage of the first (“subset”) approach is its computational efficiency: it only involves some simple reformatting of data from the source image’s codestream, without any need to decode or compress the image data. I explored this in <a href="/2013/08/19/optimising-archival-jp2s-derivation-access-copies">this 2013 blog post</a>, and at the time I was able to make it work with Kakadu’s “transcode” tool and Aware’s JPEG 20000 SDK<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>. However, its success depends largely on the correct implementation of the quality layers in the preservation masters. For example, if the 8th quality layer in a preservation master was accidentally compressed at some other compression ratio than the expected 20:1 value, the resulting access JP2 could be (much) smaller or larger than expected. Complicating things further, even though <a href="https://jpylyzer.openpreservation.org/">jpylyzer</a> will tell you both the overall compression ratio of a JP2 as well as the number of quality layers, it does not provide any information about the compression ratios of individual layers<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>.</p> <p>Because of this, I only explored the full decode + re-compress approach here. Although computationally less efficient than the “subset” approach, it has the advantage that the result is independent of the implementation of the quality layers in the preservation masters.</p> <h2 id="test-environment">Test environment</h2> <p>For my tests I used an ordinary desktop PC with 4 CPU cores, an Intel i5-6500 CPU (3.20GHz) processor and 12 GB RAM. The operating system was Linux Mint 20.1 Ulyssa (which is based on Ubuntu Focal Fossa 20.04).</p> <h2 id="codecs">Codecs</h2> <p>I initially planned to create a small proof of concept workflow based on <a href="https://kakadusoftware.com/">Kakadu</a>, as I already had some old test scripts for compressing TIFF images to JP2s that follow the KB’s master and access requirements. Then my colleague Sam Alloing suggested to have a look at the <a href="https://github.com/GrokImageCompression/grok">Grok codec</a>. Although I had been aware of Grok for some time, I had never got around to take it for a spin, mainly because I haven’t been working much on anyting related to JPEG 20000 for the past few years. Since Grok is a fork of <a href="https://www.openjpeg.org/">OpenJPEG</a>, which the KB already uses <a href="https://lab.kb.nl/about-us/blog/kb-national-library-netherlands-adopts-openjpeg-delpher-and-more-0">to decode JP2 images on the Delpher platform</a>, it then made sense to include OpenJPEG as well. So, in the end I used:</p> <ul> <li>OpenJPEG 2.4.2</li> <li>Grok 9.7.3</li> <li>Kakadu 7.9<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">4</a></sup></li> </ul> <p>I compiled both Grok and OpenJPEG from the source code. For Kakadu I used the pre-compiled demonstration binaries.</p> <h2 id="test-procedure">Test procedure</h2> <p>For each of the three codecs, I created a simple Bash script that takes an input and an output directory as its arguments. For each JP2 image in the input directory, the script goes through the following steps:</p> <ol> <li>Decode (uncompress) the JP2 to uncompressed TIFF.</li> <li>Compress the TIFF to lossy JP2, using (to the maximum extent posssible) the KB’s access JP2 requirements.</li> <li>Delete the TIFF file.</li> </ol> <p>Once all input JP2s have been processed, the script then runs the <a href="https://github.com/KBNLresearch/jprofile/">jprofile tool</a> on the output directory. Jprofile (which uses <a href="https://jpylyzer.openpreservation.org/">jpylyzer</a> under its hood) uses <a href="https://www.bitsgalore.org/2012/09/04/automated-assessment-jp2-against-technical-profile">Schematron rules</a> to verify to what extent the generated JP2s conform to the KB access requirements.</p> <p>The test scripts (which also contain the encoding parameter values for each codec) can be found here:</p> <table> <thead> <tr> <th style="text-align: left">Codec</th> <th style="text-align: left">Link to script</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">OpenJPEG</td> <td style="text-align: left"><a href="https://github.com/KBNLresearch/jp2totiff/blob/master/mastertoaccess-opj.sh">https://github.com/KBNLresearch/jp2totiff/blob/master/mastertoaccess-opj.sh</a></td> </tr> <tr> <td style="text-align: left">Grok</td> <td style="text-align: left"><a href="https://github.com/KBNLresearch/jp2totiff/blob/master/mastertoaccess-grok.sh">https://github.com/KBNLresearch/jp2totiff/blob/master/mastertoaccess-grok.sh</a></td> </tr> <tr> <td style="text-align: left">Kakadu</td> <td style="text-align: left"><a href="https://github.com/KBNLresearch/jp2totiff/blob/master/mastertoaccess-kdu.sh">https://github.com/KBNLresearch/jp2totiff/blob/master/mastertoaccess-kdu.sh</a></td> </tr> </tbody> </table> <h2 id="performance">Performance</h2> <p>I ran each of the scripts on a directory with 26 preservation master JP2s (144 MB) from the KB’s collection of digitised books. Before running any of the scripts, I used the following command to empty my machine’s cache memory:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>sysctl vm.drop_caches<span class="o">=</span>3 </code></pre></div></div> <p>I then used the operating system’s built-in <a href="https://linux.die.net/man/1/time">“time” tool</a> to measure the processing time needed by each of the scripts:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span><span class="nb">time</span> ~/kb/jp2totiff/mastertoaccess-grok.sh ./master-1 ./access-1-grok<span class="o">)</span> 2&gt; time-grok.txt </code></pre></div></div> <p>The main metrics provided by this command are:</p> <ul> <li>“real” - the actual amount of time passed between starting the script and its termination.</li> <li>“user” - The sum of the processing times of each of the individual processors.</li> </ul> <p>Below table shows the performance statistics for the three scripts:</p> <table> <thead> <tr> <th style="text-align: left">Codec</th> <th style="text-align: left">time (real)</th> <th style="text-align: left">time (user)</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">OpenJPEG</td> <td style="text-align: left">0m50.715s</td> <td style="text-align: left">1m20.497s</td> </tr> <tr> <td style="text-align: left">Grok</td> <td style="text-align: left">0m22.143s</td> <td style="text-align: left">1m1.308s</td> </tr> <tr> <td style="text-align: left">Kakadu</td> <td style="text-align: left">0m25.507s</td> <td style="text-align: left">0m48.990s</td> </tr> </tbody> </table> <p>It’s worth noting that each of these figures encompasses a full decode-encode cycle, with some additional overhead added by jprofile, and system commands that remove the temorary TIFF files. I was surprised to see that at 22 seconds, the Grok-based script was even (marginally) faster than the Kakadu-based one, which clocks in at 26 seconds<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>. The script that uses OpenJPEG is considerably slower at 51 seconds.</p> <h2 id="conformance-to-kb-access-requirements">Conformance to KB access requirements</h2> <p>The next table summarises the jprofile analysis, by listing the deviations from the KB acces requirements for each codec:</p> <table> <thead> <tr> <th style="text-align: left">Codec</th> <th style="text-align: left">Deviations from KB access requirements</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">OpenJPEG</td> <td style="text-align: left">XML box missing, resolution box missing, ICC profile missing</td> </tr> <tr> <td style="text-align: left">Grok</td> <td style="text-align: left">XML box missing</td> </tr> <tr> <td style="text-align: left">Kakadu</td> <td style="text-align: left">-</td> </tr> </tbody> </table> <p>The OpenJPEG JP2s fall short on three aspects. An XML box with XMP metadata, a resolution box, and an ICC profile are all missing. This is not surprising, as OpenJPEG simply doesn’t support these features at this stage. In the Grok JP2s, only the expected XML box is missing. This is because Grok wraps XMP metadata in a so-called “UUID box”. This behaviour is consistent with the <a href="https://en.wikipedia.org/wiki/ISO/IEC_base_media_file_format">ISO/IEC base media file format</a>, and is supported by e.g. Exiftool and jpylyzer. Only the Kakadu JP2s are 100% compliant with the requirements. However, since the exact location of XMP metadata doesn’t really matter for access, both the Kakadu and the Grok JP2s would be satisfactory for our purposes.</p> <h2 id="conclusions">Conclusions</h2> <p>Although based on only a small sample dataset, this proof of concept demonstrates that both Grok and Kakadu would be suitable for generating lossy access JP2s from our preservation masters. The performance of both codecs turned out to be comparable for the test data used. This means that with Grok we now have an open-source codec that is both sufficiently feature-rich and performant to be a viable alternative to commercial codecs like Kakadu. One potential hurdle for some users might be Grok’s build process, which can be slightly involved because it requires very recent versions of <a href="https://cmake.org/">CMake</a> and <a href="https://gcc.gnu.org/">gcc</a>. However, using <a href="https://github.com/KBNLresearch/jp2totiff/blob/master/doc/grok-installation.md">Grok’s documentation</a> and <a href="https://wiki.harvard.edu/confluence/display/DigitalImaging/Installing+OpenJPEG+on+Windows+10%2C+Linux%2C+and+MacOS">these useful additional instructions by Harvard’s Bill Comstock</a> I found the process easier than expected in the end. I’ve <a href="https://github.com/KBNLresearch/jp2totiff/blob/master/doc/grok-installation.md">documented the full build and installation process that worked for me here</a>.</p> <h2 id="acknowledgements">Acknowledgements</h2> <p>Thanks are due to Grok developer Aaron Boxer for fixing two small issues I ran into while running my Grok tests, and Sam Alloing for suggesting to look into Grok.</p> <h2 id="revision-history">Revision history</h2> <ul> <li>5 July 2022 - re-ran performance test with added <code class="language-plaintext highlighter-rouge">-threads</code> option for OpenJPEG, as suggested by Aaron Boxer in the comments.</li> </ul> <h2 id="further-resources">Further resources</h2> <ul> <li><a href="https://github.com/KBNLresearch/jp2totiff">Git repository with test scripts</a></li> <li><a href="https://github.com/KBNLresearch/jp2totiff/blob/master/doc/grok-installation.md">My Grok build and installation instructions</a></li> <li><a href="https://wiki.harvard.edu/confluence/display/DigitalImaging/Installing+OpenJPEG+on+Windows+10%2C+Linux%2C+and+MacOS">Bill Comstock, “Installing OpenJPEG (and Grok) on Windows 10, Linux, and MacOS”</a></li> <li><a href="/2013/08/19/optimising-archival-jp2s-derivation-access-copies">Optimising archival JP2s for the derivation of access copies</a></li> <li><a href="https://github.com/KBNLresearch/jprofile">Jprofile - Automated JP2 profiling for digitisation batches</a></li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:6"> <p>To be completely clear, at this stage this work is just an exploration of something we might do at some time in the future (or possibly not at all); there are no plans to actually implement this yet. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>As of 2022, Aware appears to have switched its focus to the development of biometrical software, and its website does not mention the JPEG 2000 SDK anymore. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:1"> <p>Adding this functionality to jpylyzer would require much more in-depth parsing of the codestream data than is currently the case. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>Note that this a pretty old version. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>These figures are not 100% comparable, because the Kakadu-based script includes an additional processing step to extract embedded metadata from the source file using ExifTool (Grok does this automatically at the codec level). <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2022/03/30/generating-lossy-access-jp2s-from-lossless-preservation-masters https://bitsgalore.org/2022/03/30/generating-lossy-access-jp2s-from-lossless-preservation-masters 2022-03-30T00:00:00+02:00 <figure class="image"> <img src="/images/2021/09/clippy-800.png" alt="Clippy saying It looks like you're migrating a spreadsheet to ... TIFF?!" /> </figure> <p>Earlier this month saw the publication of <a href="https://zenodo.org/record/5468116">The Significant Properties of Spreadsheets</a>. This is the final report of a six-year research effort by the Open Preservation Foundation’s Archives Interest Group (AIG), which is composed of participants from the National Archives of the Netherlands (NANETH), the National Archives of Estonia (NAE), the Danish National Archives (DNA), and Preservica. The report caught my attention for two reasons. First, there’s the subject matter of spreadsheets, on which I’ve written a few posts in the past<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. Second, it marks a surprising (at least to me!) return of “significant properties”, a concept that was omnipresent in the digital preservation world between, roughly, 2005 and 2010, but which has largely fallen into disuse since then. In this post I’m sharing some of my thoughts on the report.</p> <!-- more --> <h2 id="overall-aim">Overall aim</h2> <p>The authors describe the rationale behind their work as follows:</p> <blockquote> <p>Preserving files in spreadsheet formats is a priority for every member. We need to answer questions such as ‘should we migrate?’ and ‘how do we measure the success or quality of the migration?’. For the latter, we need to know what aspects of the file are important (significant), which led us to the decision to investigate the significant properties of spreadsheets.</p> </blockquote> <p>Various definitions of “significant properties” exist. The authors followed the 2007 <a href="https://significantproperties.kdl.kcl.ac.uk/wp22_significant_properties.pdf">InSPECT report</a> here, which defines them as:</p> <blockquote> <p>the characteristics of digital objects that must be preserved over time in order to ensure the continued accessibility, usability, and meaning of the objects, and their capacity to be accepted as evidence of what they purport to record.</p> </blockquote> <p>The authors describe the role of “significant properties” in the preservation process as follows:</p> <blockquote> <p>When the digital objects (e.g. files in a format) or the technology to use them (e.g. viewers) are at risk of becoming obsolete, preservation actions may be required (e.g. file format migration or viewer software emulation). Ensuring that the significant properties are reasonably preserved as a result of these preservation actions is then a means of validating these actions.</p> </blockquote> <p>So, by this view, the importance of “significant properties” lies in their utility to validate preservation actions.</p> <h2 id="simple-versus-complex-spreadsheets">Simple versus complex spreadsheets</h2> <p>Throughout the report, the authors differentiate between what they call “simple” (or “static”) and “complex” (or “dynamic”) spreadsheets. They define “simple” spreadsheets as:</p> <blockquote> <p>spreadsheets that are mainly used for (human) visualisation and contain static data values organised into tabular format on one or more worksheets.</p> </blockquote> <p>Irrespective of whether you agree with the “simple” versus “complex” distinction, the above definition is problematic because it mixes up actual spreadsheet characteristics (“contains static data”) with a subjective statement of how such spreadsheets are supposed to be used (“mainly used for human visualization”). Moreover, it’s not at all clear what the “human visualization” assessment is based on (some written intention statement by the original creators, an institutional policy, or something else?).</p> <p>By contrast, “complex” spreadsheets are defined as:</p> <blockquote> <p>spreadsheets that contain formulae, notes, macros, dates, links to external data sources or other functions or dynamic behaviour.</p> </blockquote> <h2 id="migrating-spreadsheets-to-tiff-and-pdfa">Migrating spreadsheets to TIFF and PDF/A</h2> <p>The authors explain that this distinction was motivated by a practical format migration use case. In particular, on page 10 they write that (emphasis added by me):</p> <blockquote> <p>simple spreadsheets would likely render more or less the same in most spreadsheet-rendering applications at every moment of time. <strong>One would lose no information when migrating to other, primarily rendering-oriented file formats, like the Tagged Image File Format (TIFF) currently accepted by DNA.</strong></p> </blockquote> <p>On page 11, they continue:</p> <blockquote> <p> In case of ‘simple/static’ spreadsheets with formatting (fonts, colours, styles, cell width, etc.), no significant loss of information would occur if the spreadsheets were migrated, to e.g. TIFF or PDF/A.</p> </blockquote> <p>And:</p> <blockquote> <p>In short, ‘simple/static’ spreadsheets can be migrated to non-spreadsheet specific file formats or formats that are not meant to preserve dynamic behaviour.</p> </blockquote> <p>These are truly mind-boggling statements, and I hardly know where to even start here. First, migrating a spreadsheet to an image format like TIFF would result in the loss of <em>most</em> of the information of the source document, including <em>all</em> coded text and numbers. This would break machine-readability, and would also make the data values inaccessible to visually impaired users. Most of the functionality of the source file would be lost as well, such as the ability to navigate and access individual cells, rows and columns, search and filter cell values, and copy content, to name but a few. These objections apply to PDF/A as well (albeit to a somewhat lesser degree). I already addressed this <a href="/2016/12/09/pdfa-as-a-preferred-sustainable-format-for-spreadsheets">in my 2016 post on PDF/A as a preferred, sustainable format for spreadsheets</a> (which, incidentally, is even cited in the report). Statements like these show a fundamental lack of understanding of digital information and file formats, and I’m surprised (and frankly a bit shocked) to find them in a contemporary report by digital preservation professionals<sup id="fnref:8"><a href="#fn:8" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>.</p> <p>On “complex” spreadsheets, the authors note that (page 11):</p> <blockquote> <p>Migrating to non-spreadsheet formats could cause severe information loss.</p> </blockquote> <p>While this is of course true, this largely applies to “simple” spreadsheets as well. Also, migrating to other spreadsheet formats could result in information loss as well.</p> <h2 id="identification-of-significant-properties">Identification of significant properties</h2> <p>The main outcome of the work is a categorised list of “significant properties” of spreadsheets. The authors arrived at this by adapting the <a href="https://significantproperties.kdl.kcl.ac.uk/inspect-finalreport.pdf">methodology from the InSPECT project</a>. I found the description of the InSPECT methodology and its application to spreadsheets in the AIG report quite hard to follow in places, so in this section I’ll attempt to provide a brief (and somewhat simplified) summary.</p> <p>The InSPECT methodology involves two main components. The first one is the “object analysis”. The overall objective here is the compilation of an extensive list of spreadsheet properties, based on sample files, characterisation tools and technical specifications. These were then categorised into “property groups”. Subsequently, the “property groups” were linked to expected “user behaviours” (e.g. “View data in cells”), which were classified into more generic “functions”. The associations between functions, behaviours and property groups are visualised in what the authors call the “spaghetti diagram”, which can be viewed <a href="https://zenodo.org/record/5468116/files/FBS%20diagram%20%28final%20report%29.png">here</a>.</p> <p>A particularly interesting strand of the object analysis work, was the development of a <a href="https://github.com/RvanVeenendaal/Spreadsheet-Complexity-Analyser">Spreadsheet Complexity Analyser</a>. This is a tool that classifies Microsoft Excel spreadsheets (both XLS and XLSX) as “simple” or “complex”, based on extracted document- and cell-level properties (which can be reported as well).</p> <p>The second component is a “stakeholder analysis”. Here, the authors presented the properties and property groups from the object analysis to various types of stakeholders (e.g. archivists, users), and asked them which properties they deemed “significant”.</p> <p>The authors combined the results from the “object analysis” and the “stakeholder analysis”, which resulted in <a href="https://zenodo.org/record/5468116/files/Combined%20%28relevant%20and%20significant%20properties%29.xlsx?download=1">this list</a>. Out of the 334 properties in the list, 105 were deemed “significant” by the stakeholders at the individual property level. At the property group level, only 15 out of the 38 property groups were deemed “significant”, which corresponds to 140 properties (i.e. all properties that are part of these 15 groups).</p> <p>In the following sections I will comment on some things that caught my attention.</p> <h2 id="specificity-of-properties">Specificity of properties</h2> <p>One thing that struck me while browsing the <a href="https://zenodo.org/record/5468116/files/Combined%20%28relevant%20and%20significant%20properties%29.xlsx?download=1">list of properties</a>, is that some of the identified properties are very general and lack specificity. As an example, properties such as “Database Functions”, “Engineering Functions” and “Date and Time Functions” only describe broad <em>categories</em> of spreadsheet functions, but not the actual functions themselves. I don’t really understand the value of these categories within the context of validating preservation actions (which, after all, is ultimately the larger aim of this work). As an example, imagine we migrate an XLSX spreadsheet to ODS, and for some weird reason the <a href="https://support.microsoft.com/en-us/office/weekday-function-60e44483-2ed1-439f-8bd0-e404c190949a"><em>WEEKDAY</em></a> function is changed to <a href="https://support.microsoft.com/en-us/office/year-function-c64f017a-1354-490d-981f-578e8ec8d3b9"><em>YEAR</em></a>. Both are “Date and Time Functions”, but the result of such a migration would still be nonsense. What we’d really like to know in this case, is whether <em>the exact functions</em> are preserved. Moreover, according to <a href="https://wiki.documentfoundation.org/Feature_Comparison:_LibreOffice_-_Microsoft_Office#Desktop_Spreadsheet_applications:_LibreOffice_Calc_vs._Microsoft_Excel">the Document Foundation Wiki</a> both Excel and LibreOffice Calc and have some functions that are unique to each application (29 and 22, respectively). This is a potential source for information loss when migrating between their respective formats. A significant properties approach would only be able to account for such information losses if the properties are defined at the level of individual spreadsheet functions<sup id="fnref:6"><a href="#fn:6" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>.Needless to say, this would increase the number of properties to take into account considerably, and in the absence of any way to automate this it would also make things even more time consuming.</p> <p>The properties related to macros<sup id="fnref:7"><a href="#fn:7" class="footnote" rel="footnote" role="doc-noteref">4</a></sup> provide another example. Microsoft Excel uses <a href="https://en.wikipedia.org/wiki/Visual_Basic_for_Applications">Visual Basic for Applications</a> (VBA) as the macro language for its XLS and XLSX formats. However, the Open Document Format does not dictate any specific macro or scripting language, and simply declares this as <a href="https://docs.oasis-open.org/office/OpenDocument/v1.3/os/part3-schema/OpenDocument-v1.3-os-part3-schema.html#attribute-script_language">implementation dependent</a>. By default, LibreOffice uses its own Basic implementation, which is <a href="https://help.libreoffice.org/latest/en-US/text/shared/guide/ms_user.html">largely incompatible with Microsoft’s VBA language</a>. In addition, it allows the use of other languages such as <a href="https://wiki.documentfoundation.org/Macros/Python_Guide/Introduction">Python</a> and <a href="https://webodf.org/blog/2012-04-13.html">JavaScript</a>. This variety of (mostly incompatible) macro languages introduces various preservation challenges. In spite of this, “macro language” or “scripting language” is not included in the “macros” property group. This raises some questions on how useful these properties actually are for validating preservation actions.</p> <h2 id="how-to-measure-the-properties">How to measure the properties</h2> <p>Related to this, the authors provide no explanation <em>how</em> any of the identified properties can be measured, and using what software. My best guess is that they used some spreadsheet application<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>. If this is indeed the case, how would one even start to do this at scale, with collections of thousands of spreadsheets? Again, the authors don’t even mention this. The Spreadsheet Complexity Analyser could be really useful here, and in fact, DNA tried this approach as part of the object analysis work (p. 19):</p> <blockquote> <p>The Danish National Archives ran the SCA against about 16,000 Microsoft Excel spreadsheets (both binary formats and OOXML) to investigate the possible information loss when converting Excel spreadsheets to ODS. (…) The test showed that the conversion from XLS and XLSX to ODS and back to XLS and XLSX resulted in minimal data loss. Yet, data loss for significant structures such as cell typographies, fonts and hyperlinks were encountered.</p> </blockquote> <p>However, the Spreadsheet Complexity Analyser is currently only able to extract 13 spreadsheet-specific properties<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">6</a></sup>, which is only a fraction of all the properties that were deemed “significant”. So while this is an admirable start, it doesn’t even come close to what would be needed for any real-world application.</p> <h2 id="where-are-the-examples-case-studies">Where are the examples, case studies?</h2> <p>The introductory chapter of the AIG report explicitly mentions how the work is aimed at validating preservation actions. However, it is remarkably vague on how the outcomes of the inSPECT methodology could help in solving real-world spreadsheet preservation challenges. The authors do mention application areas such as preservation actions (e.g. format migration and emulation) and choosing preferred formats. However, they don’t provide any examples or case studies that demonstrate how the identified properties, property groups and user behaviours would help here in practice. The closest thing to an actual example is given in the concluding chapter, where they write (p. 39):</p> <blockquote> <p>One example is that the Danish National Archives used the gained knowledge of spreadsheet properties in the decision to revise their accepted formats and adopt a spreadsheet-specific format, which probably will be the Open Document Spreadsheet format.</p> </blockquote> <p>This refers to DNA’s current practice of accepting TIFF(!) as a preservation format for spreadsheets. The authors then write:</p> <blockquote> <p>[O]ur work provided the lists, tools and insights DNA required to revise their accepted format policy and adopt a spreadsheet-specific format in their Preservation Policy.</p> </blockquote> <p>It’s not my intention to mock DNA’s odd choice of TIFF here<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">7</a></sup>, but that TIFF is a terrible target format for spreadsheets should be blindingly obvious to anyone with even the slightest understanding of spreadsheets or digital formats in general. You really don’t need to spend six years researching hundreds of “significant properties” to arrive at this conclusion<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">8</a></sup>!</p> <p>Because of the absence of any further examples or case studies, it remains unclear how the authors envisage the use of their work in any real-world format migration or emulation scenario. For a start, the sheer number of properties would make any manual, non-automated analysis extremely cumbersome and time consuming. Interestingly, this is confirmed by the following quote from the DNA stakeholder analysis (p. 18):</p> <blockquote> <p>Our experiences from the stakeholder interviews were that it can be extremely time and competency consuming to analyse every single property and behaviour for a complex content information type such as spreadsheets. In fact, for these kinds of analyses it can be counterproductive to conducting the interview if we do not try to stray away from the InSPECT approach and instead focus on facilitating a meaningful conversation with people and from this conversation try to deduce the behaviours necessary to preserve for future reuse of the data. The questions you instead can ask people are what they deem important to be able to do with the data and what data and associated functionality do they find important to preserve, if they were to reuse it in the future.</p> </blockquote> <p>I’m not surprised by this observation, as it is consistent with earlier work by Euan Cochrane. In his 2012 <a href="https://web.archive.org/web/20130218111126/http://archives.govt.nz/rendering-matters-report-results-research-digital-object-rendering">Rendering Matters report</a>, he showed that for a migration-based approach, at least 13.5 hours were needed to test only 100 Office files (a mixture of word-processing, spreadsheet, database and presentation formats) comprehensively. This doesn’t inspire much confidence in using such analyses at scale for any real-world applications.</p> <h2 id="significance-in-context">Significance in context</h2> <p>Some of the problems that I outlined in the previous sections could be remedied to some extent by introducing even more (and more detailed) properties, or by adding support for more properties to the Spreadsheet Complexity Analyser. However, this would make the analyses even more complex and time consuming. This shouldn’t come as a surprise, as Webb, Pearson &amp; Koerbin already pointed out some of these difficulties in their <a href="http://www.dlib.org/dlib/january13/webb/01webb.html">2013 D-Lib paper</a>:</p> <blockquote> <p>We have come to a tentative conclusion that recognising and taking action to maintain significant properties will be critical, but that the concept can be more of a stumbling block than a starting block, at least in the context of our own institution. We believe reference to significant properties in preservation planning requires some prior consideration of both the purposes for which digital content has been collected and the purposes of providing preservation attention. In effect, we are asking how can we know what attributes of digital materials we need to preserve if we haven’t articulated why we are preserving them?</p> </blockquote> <p>Specifically, they ran into two problems. The first one was the complexity of defining the measurable levels of properties that were required to fulfill a collection’s preservation intention. The second one was the realisation that potential changes to digital objects as a result of some preservation action can best be evaluated around the time the preservation action takes place, using the tools that are then available. This doesn’t mean they discarded the concept of significant properties altogether, but rather they decided that the definition of significant properties is informed by practical experience with (the tools used to perform) preservation actions.</p> <p>Owens is also skeptical about the significant properties approach in his 2018 book <a href="https://osf.io/preprints/lissa/5cpjt">The Theory and Craft of Digital Preservation</a>:</p> <blockquote> <p>At one point, many in the digital preservation field worked to try and identify context agnostic notions of the significant properties of various kinds of digital files and formats. While there is some merit to this approach, it has been largely abandoned in the face of the realization that significance is largely a matter of context. That is, significance isn’t an inherent property of forms of content, but an emergent feature of the context in which that content was created and the purpose it serves in an institutions’ collection.</p> </blockquote> <p>The over-arching problem of much of the work presented in the AIG report, is the almost total lack of any such context. The result is, that valuable time and effort are spent on the analysis of a myriad of atomic properties, many of which will only matter in certain specific contexts, and not at all in others. So, before embarking on any preservation action, I would like (at the absolute minimum) answers to questions such as:</p> <ul> <li>Who are the (present and future) users of the spreadsheet collections?</li> <li>What accessibility levels are required to meet these users needs (e.g. viewing in original environment, navigating, editing, accessibility for visually impaired users, machine-readability)?</li> <li>What preservation actions are needed to enable these accessibility requirements (e.g. emulation, migration to one or more access formats)?</li> </ul> <p>The answers to such questions largely fit into the “preservation intent” concept, which was (if I’m not mistaken) first introduced by Webb, Pearson &amp; Koerbin:</p> <blockquote> <p>The Library’s preservation intent methodology is simply to engage collection curators in making explicit statements about which collection materials, and which copies of collection materials, need to remain accessible for an extended period, and which ones can be discarded when no longer in use or when access to them becomes troublesome. Curators are also asked to make broad statements clarifying what ‘accessible’ means by stating the priority elements that need to be re-presented in any future access for each kind of digital object type in their collections.</p> </blockquote> <p>Owens considers preservation intent to be one of the foundations of any digital preservation work:</p> <blockquote> <p>In this model, preservation intent and collection development function are the foundation of your digital preservation work. All of the work one does to enable long-term access to digital information should be grounded in a stated purpose and intention that directly informs what is selected and what is retained.</p> </blockquote> <p>One important implication is that there is no “one size fits all” solution here. For example, the properties that are “significant” for an authentic rendering in an emulator will be quite different from those for a machine-readable access copy, and such nuances are not apparent from the AIG report, which seems to take the view that digital preservation is merely a matter of preserving the (contextless) “significant properties” (p. 41):</p> <blockquote> <p>[I]f you look closely at your digital preservation strategy, it boils down to finding the best way to preserve the significant properties of information.</p> </blockquote> <p>By contrast, taking the preservation intent as a starting point, a practical solution for the above case might look something like this:</p> <ul> <li>Always keep the original spreadsheets (in their original formats)</li> <li>Use emulation to ensure faithful rendering of these files in their original environment</li> <li>If needed, use migration to derive access copies in more convenient formats to suit specific use cases (e.g. text and data mining)<sup id="fnref:10"><a href="#fn:10" class="footnote" rel="footnote" role="doc-noteref">9</a></sup>.</li> </ul> <p>This would not require any complex and time-consuming analysis of significant properties, because the original data are preserved in their original formats. The Spreadsheet Complexity Analyser could possibly help validate the migration to the access formats (at least to some degree).</p> <h2 id="representativity-of-stakeholder-group">Representativity of stakeholder group</h2> <p>In the inSPECT methodology, the “significance” that is attributed to a property depends to a large degree on the background and knowledge level of the stakeholders. To their credit, the authors readily acknowledge this in the report, and remark that:</p> <blockquote> <p>A stakeholder that never makes use of more advanced features, such as formulas and macros, will not deem these significant.</p> </blockquote> <p>A <a href="https://planets-project.eu/docs/papers/Dappert_Significant_Characteristics_ECDL2009.pdf">2009 report by Dappert and Farquhar</a> already stressed how “significance” is largely in the eye of the stakeholder. So, one would expect that the stakeholders would be somewhat representative of the (future) users of the spreadsheet collections. However, the authors restricted their stakeholder analysis population to “individuals that are employed in the public sector”. They justify this by stating that:</p> <blockquote> <p>This is due to the fact that the organisations for which this study is carried out, the National Archives of various countries, preserve information from public institutes.</p> </blockquote> <p>I understand the logic behind limiting the size of the stakeholder group (if only for practical reasons), but the justification seems to imply that these National Archives see their depositing institutes as their only (or primary) users. This strikes me as an overly narrow view, as national archives also serve independent researchers, authors, (data) journalists, and members of the general public. It is not clear to what extent the actual (potential) users were represented in the current stakeholder analysis. For instance, a researcher or data journalist may attribute high importance to the ability to access spreadsheet collections through text and data-mining techniques. For a visually impaired user, accessibility through a screenreader application will be vitally important.</p> <h2 id="user-behaviours">User behaviours</h2> <p>In addition, one component of the inSPECT methodology’s stakeholder analysis is the collection of “actual behaviours”, which are defined as “activities that a specific category of stakeholder will likely perform when using the object”. These are then mapped against properties, which eventually determines to a large degree which properties are deemed “significant”. However, the authors decided that this would be too daunting a task (p. 15):</p> <blockquote> <p>With spreadsheets, however, this is an immense task. Stakeholders use spreadsheets for a wide range of activities with no established set of functions that has to be used every time. Furthermore, we felt this would be difficult to accomplish thoroughly during interviews with stakeholders, considering the size of the task. Therefore, these last steps were not performed by us during this research.</p> </blockquote> <p>Instead, they limited themselves to drafting a list of “expected behaviours”, which inSPECT defines as “the different types of activities that a user – any type of user – may wish to perform”. But even this turned out to be overly ambitious as well (p. 21, emphasis added by me):</p> <blockquote> <p>We soon realised that we would never establish an exhaustive list of all possible behaviours and <strong>chose to use those behaviours that we found most important from our perspective as archives preserving spreadsheets</strong>.</p> </blockquote> <h2 id="significant-to-future-users">Significant to (future) users?</h2> <p>The combined effects of the restricted stakeholder population, not taking into account “actual behaviours”, and limiting the “expected behaviours” to an archivist’s perspective raises some major concerns. Most importantly, it calls into question the degree to which the “significance” judgements truly reflect what is “significant” to actual and future users (which should be informed by the preservation intent). Ultimately, digital preservation is largely about preserving digital materials for future <em>use</em>. If accounting for actual user behaviours, needs and requirements is too gargantuan a task, shouldn’t this lead to the inevitable conclusion that, at least in this situation, the inSPECT methodology is not suitable as a basis for validating preservation actions<sup id="fnref:9"><a href="#fn:9" class="footnote" rel="footnote" role="doc-noteref">10</a></sup>?</p> <h2 id="conclusion">Conclusion</h2> <p>The AIG report on the significant properties of spreadsheets leaves me with some very mixed thoughts. On the one hand, it is clear that a lot of time, effort and dedication have gone into this work. The detailed breakdown of properties in the <a href="https://zenodo.org/record/5468116/files/List%20of%20properties%20and%20property%20groups%20%28blue%20sheet%29.ods?download=1">“blue sheet”</a> contains a wealth of information that, I expect, will be of interest to any digital preservationist working on spreadsheet preservation. I particularly like the <a href="https://github.com/RvanVeenendaal/Spreadsheet-Complexity-Analyser">Spreadsheet Complexity Analyser</a>, which looks like a genuinely useful tool.</p> <p>With that said, it’s not clear to me how the results of the main body of this work would support its own stated aim of validating preservation actions, and the authors provide no examples of this. It almost seems that they got lost in a labyrinthine web of properties, and lost sight of this overall aim along the way.</p> <p>I’m also concerned about the largely context-agnostic view of “significant properties” that the authors express throughout the report. On the one hand, this results in a scope that will be unnecessarily wide for most real-world preservation actions (meaning that the number of properties to consider becomes unmanageable quickly). Simultaneously, the authors made several decisions that, taken together, have the effect that the needs and requirements of (future) users of the spreadsheet collections are only minimally represented. I expect this will seriously limit the utility of this exercise as a basis for making preservation decisions.</p> <h2 id="link-to-report">Link to report</h2> <p><a href="https://zenodo.org/record/5468116">The Significant Properties of Spreadsheets (OPF AIG Final Report)</a></p> <h2 id="revision-history">Revision history</h2> <ul> <li>8 October 2021: added mention of Euan Cochrane’s Rendering Matters report to examples, case studies section.</li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>See my posts on <a href="/2014/10/29/quattro-pro-dos-obsolete-format-last">Quattro Pro for DOS</a> and <a href="/2016/12/09/pdfa-as-a-preferred-sustainable-format-for-spreadsheets">PDF/A as a preferred, sustainable format for spreadsheets?</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:8"> <p>The discussion of the Danish National Archives Stakeholder Analysis on page 29 provides a more realistic view, which is absolutely clear on why migrating to TIFF is a horrible idea for spreadsheets. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:6"> <p>Also, I cannot even imagine how one would measure these function categories, as to the best of my knowledge they are not even explicitly coded in the spreadsheet. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:7"> <p>I should add here that macros were not considered “significant” by the stakeholders. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>E.g. (some version of) Microsoft Excel, LibreOffice Calc, or perhaps something else? <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>The total number of extracted properties is actually 18, but 5 of these are filesystem-level level attributes (file name, size and time stamps) that are not specific to spreadsheets. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>I suspect this might have it roots in some ill-advised decision in some distant past. I think most memory institutions will have examples of similarly unfortunate historically-evolved practices, and DNA should be praised here for being open about this. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>It’s worth noting here that a footnote on page 31 mentions how DNA are working on revising their format policy because the TIFF format “will not support the migration of dynamic properties”. It also mentions that “[o]ur research has made clear how significant these are and why they must be preserved”. This suggests that despite spending six years of research on the significant properties of spreadsheets, DNA still see no problem with TIFF as a target format for “simple”/”static” spreadsheets. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:10"> <p>Even migration to some image format might be useful as supporting evidence of how a spreadsheet should be rendered in its original environment (but please use PNG, not TIFF). <a href="#fnref:10" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:9"> <p>Which, by the authors’ own account, is the underlying aim of this work. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2021/09/24/on-the-significant-properties-of-spreadsheets https://bitsgalore.org/2021/09/24/on-the-significant-properties-of-spreadsheets 2021-09-24T00:00:00+02:00 <figure class="image"> <img src="/images/2021/09/plumber-tools.jpg" alt="Photo of assortment of old plumbing tools." /> <figcaption><a href="https://www.flickr.com/photos/130648318@N06/42662053232">Plumbers Tool Box</a> by <a href="https://www.flickr.com/photos/130648318@N06/">pszz</a> on Flickr. Used under <a href="https://creativecommons.org/licenses/by-nc-sa/2.0/">CC BY-NC-SA 2.0</a>.</figcaption> </figure> <p>Over the years, I’ve been using a variety of open-source software tools for solving all sorts of issues with PDF documents. This post is an attempt to (finally) bring together my go-to PDF analysis and processing tools and commands for a variety of common tasks in one single place. It is largely based on a multitude of scattered lists, cheat-sheets and working notes that I made earlier. Starting with a brief overview of some general-purpose PDF toolkits, I then move on to a discussion of the following specific tasks:</p> <ul> <li>Validation and integrity testing</li> <li>PDF/A and PDF/UA compliance testing</li> <li>Document information and metadata extraction</li> <li>Policy/profile compliance testing</li> <li>Text extraction</li> <li>Link extraction</li> <li>Image extraction</li> <li>Conversion to other (graphics) formats</li> <li>Inspection of embedded image information</li> <li>Conversion of multiple images to PDF</li> <li>Cross-comparison of two PDFs</li> <li>Corrupted PDF repair</li> <li>File size reduction of PDF with hi-res graphics</li> <li>Inspection of low-level PDF structure</li> <li>View, search and extract low-level PDF objects</li> <li>Incremental updates and document versions: get information about the number of incremental updates, and restore previous versions</li> </ul> <!-- more --> <h2 id="how-this-selection-came-about">How this selection came about</h2> <p>Even though this post covers a lot of ground, the selection of tasks and tools presented here is by no means meant to be exhaustive. It was guided to a great degree by the PDF-related issues I’ve encountered myself in my day to day work. Some of these tasks could be done using other tools (including ones that are not mentioned here), and in some cases these other tools may well be better choices. So there’s probably a fair amount of selection bias here, and I don’t want to make any claims of presenting the “best” way to do any of these tasks here. Also, many of the example commands in this post can be further refined to particular needs (e.g. using additional options or alternative output formats), and they should probably best seen as (hopefully useful) starting points for the reader’s own explorations.</p> <p>All of the tools presented here are published as open-source, and most of them have a command-line interface. They all work under Linux (which is the main OS I’m using these days), but most of them are available for other platforms (including Windows) as well.</p> <h2 id="pdf-multi-tools">PDF multi-tools</h2> <p>Before diving into any specific tasks, let’s start with some general-purpose PDF tools and toolkits. Each of these are capable of a wide range of tasks (including some I won’t explicitly address here), and they can be seen as “Swiss army-knives” of PDF processing. Whenever I need to get some PDF processing or analysis done and I’m not sure what tool to use, these are usually my starting points. In the majority of cases, at least one of them turns out to have the functionality I’m looking for, so it’s a good idea to check them out if you’re not familiar with them already.</p> <h3 id="xpdfpoppler">Xpdf/Poppler</h3> <p><a href="https://www.xpdfreader.com/">Xpdf</a> and <a href="https://poppler.freedesktop.org/">Poppler</a> are both PDF viewers that include a collection of tools for processing and manipulating PDF files. Poppler is a fork of this software, which adds a number of unique tools that are not part of the original Xpdf package. The tools included with Poppler are:</p> <ul> <li><strong>pdfdetach</strong>: lists or extracts embedded files (attachments)</li> <li><strong>pdffonts</strong>: analyzes fonts</li> <li><strong>pdfimages</strong>: extracts images</li> <li><strong>pdfinfo</strong>: displays document information</li> <li><strong>pdfseparate</strong>: page extraction tool</li> <li><strong>pdfsig</strong>: verifies digital signatures</li> <li><strong>pdftocairo</strong>: converts PDF to PNG/JPEG/PDF/PS/EPS/SVG using the <a href="https://www.cairographics.org/">Cairo</a> graphics library</li> <li><strong>pdftohtml</strong>: converts PDF to HTML</li> <li><strong>pdftoppm</strong>: converts PDF to PPM/PNG/JPEG images</li> <li><strong>pdftops</strong>: converts PDF to PostScript (PS)</li> <li><strong>pdftotext</strong>: text extraction tool</li> <li><strong>pdfunite</strong>: document merging tool</li> </ul> <p>The tools in Xpdf are largely identical, but don’t include <em>pdfseparate</em>, <em>pdfsig</em>, <em>pdftocairo</em>, and <em>pdfunite</em>. Also, Xpdf has a separate <em>pdftopng</em> tool for converting PDF to PNG images (this functionality is covered by <em>pdftoppn</em> in the Poppler version). On Debian-based systems the Poppler tools are part of the package <em>poppler-utils</em>.</p> <h3 id="pdfcpu">Pdfcpu</h3> <p><a href="https://pdfcpu.io/">Pdfcpu</a> is a PDF processor that is written in the <em>Go</em> language. The documentation explicity mentions its main focus is strong support for batch processing and scripting via a rich command line. It supports all PDF versions up to PDF 1.7 (ISO-32000).</p> <h3 id="apache-pdfbox">Apache PDFBox</h3> <p><a href="https://pdfbox.apache.org/">Apache PDFBox</a> is an open source Java library for working with PDF documents. It includes a set of <a href="https://pdfbox.apache.org/2.0/commandline.html">command-line tools</a> for various PDF processing tasks. Binary distributions (as <a href="https://en.wikipedia.org/wiki/JAR_(file_format)">JAR</a> packages) are available <a href="https://pdfbox.apache.org/download.html">here</a> (you’ll need the “standalone” JARs).</p> <h3 id="qpdf">QPDF</h3> <p><a href="http://qpdf.sourceforge.net/">QPDF</a> is “a command-line program that does structural, content-preserving transformations on PDF files”.</p> <h3 id="mupdf">MuPDF</h3> <p><a href="https://www.mupdf.com/">MuPDF</a> is “a lightweight PDF, XPS, and E-book viewer”. It includes the <a href="https://www.mupdf.com/docs/index.html">mutool</a> utility, which can do a number of PDF processing tasks.</p> <h3 id="pdftk">PDFtk</h3> <p><a href="https://www.pdflabs.com/tools/pdftk-server/">PDFtk</a> (server edition) is a “command-line tool for working with PDFs” that is “commonly used for client-side scripting or server-side processing of PDFs”. More information can be found in the <a href="https://www.pdflabs.com/docs/pdftk-man-page/">documentation</a>, and the <a href="https://www.pdflabs.com/docs/pdftk-cli-examples/">command-line examples page</a>. For Ubuntu/Linux Mint users, the most straightforward installation option is the “pdftk-java” Debian package. This is a Java fork of PDFtk<sup id="fnref:7"><a href="#fn:7" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>.</p> <h3 id="ghostscript">Ghostscript</h3> <p><a href="https://www.ghostscript.com/">Ghostscript</a> is “an interpreter for the PostScript language and PDF files”. It provides rendering to a variety of raster and vector formats.</p> <p>The remaining sections of this post are dedicated to specific tasks. As you will see, many of these can be addressed using the multi-tools listed in this section.</p> <h2 id="validation-and-integrity-testing">Validation and integrity testing</h2> <p>PDFs that are damaged, structurally flawed or otherwise not conformant to the PDF format specification can result in a multitude of problems. A number of tools provide error checking and integrity testing functionality. This can range from limited structure checks, to full (claimed) validation against the filespec. It’s important to note that none of the tools mentioned here are perfect, and some faults that are picked up by one tool may be completely ignored by another one and vice versa. So it’s often a good idea to try multiple tools. A good example of this approach can be found in <a href="https://openpreservation.org/blogs/trouble-shooting-pdf-validation-errors-a-case-of-pdf-hul-38/">this blog post by Micky Lindlar</a>.</p> <h3 id="validate-with-pdfcpu">Validate with Pdfcpu</h3> <p>The Pdfcpu command-line tool has a <a href="https://pdfcpu.io/core/validate"><code class="language-plaintext highlighter-rouge">validate</code> command</a> that checks a file’s compliance against <a href="https://www.adobe.com/content/dam/acom/en/devnet/pdf/pdfs/PDF32000_2008.pdf">PDF 32000-1:2008</a> (i.e. the ISO version of PDF 1.7). It provides both a “strict” and a “relexed” validation mode, where the “relaxed” mode (which is the default!) ignores some common violations of the PDF specification. The command-line is:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pdfcpu validate whatever.pdf </code></pre></div></div> <p>The “strict” mode can be activated with the <code class="language-plaintext highlighter-rouge">-m</code> option:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pdfcpu validate <span class="nt">-m</span> strict whatever.pdf </code></pre></div></div> <h3 id="validate-with-jhove">Validate with JHOVE</h3> <p><a href="http://jhove.openpreservation.org/">JHOVE</a> is a is a file format identification, validation and characterisation tool that includes a module for PDF validation. It is widely used in the digital heritage (libraries, archives) sector. Here’s a typical command-line example (note that you explicitly need to invoke the <em>PDF-hul</em> module via the <code class="language-plaintext highlighter-rouge">-m</code> option; omitting this can give unexpected results):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jhove <span class="nt">-m</span> PDF-hul <span class="nt">-i</span> whatever.pdf </code></pre></div></div> <p>Check out the <a href="https://jhove.openpreservation.org/modules/pdf/">documentation</a> for more information about JHOVE’s PDF module, and its limitations.</p> <h3 id="validate-with-arlington-pdf-model-checker">Validate with Arlington PDF Model Checker</h3> <p>The <a href="https://openpreservation.org/news/arlington-pdf-model-checker-released/">Arlington PDF Model Checker</a> checks a PDF against the <a href="https://github.com/pdf-association/arlington-pdf-model">Arlington PDF Model</a>. The Arlington Model is a machine-readable representation of all object types that are defined by <a href="https://www.pdfa-inc.org/product/iso-32000-2-pdf-2-0-bundle-sponsored-access/">ISO 32000-2:2020</a> (PDF 2.0) and all earlier PDF versions. This does not offer full PDF validation<sup id="fnref:8"><a href="#fn:8" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>, but the coverage of PDF objects is vastly more comprehensive than JHOVE (or any other tool I’m aware of)<sup id="fnref:9"><a href="#fn:9" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>. The software is based on VeraPDF (which is discussed further on), and Java installers can be downloaded from <a href="https://software.verapdf.org/releases/arlington">VeraPDF’s releases section</a>.</p> <p>After installation, run the software like this:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>arlington-pdf-model-checker whatever.pdf <span class="o">&gt;</span> whatever.xml </code></pre></div></div> <p>By default, the Arlington PDF Model checker tries to automatically establish the PDF version, and then checks the file accordingly. Use the <code class="language-plaintext highlighter-rouge">-f</code> (alias: <code class="language-plaintext highlighter-rouge">--flavour</code>) option to force a specific version. As an example, the following command will result in validation against PDF 1.4:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>arlington-pdf-model-checker <span class="nt">-f</span> arlington1.4 whatever.pdf <span class="o">&gt;</span> whatever.xml </code></pre></div></div> <h3 id="check-integrity-with-qpdf">Check integrity with QPDF</h3> <p>The <code class="language-plaintext highlighter-rouge">--check</code> option of QPDF (see above) performs checks on a PDF’s overall file structure. QPDF does not provide full-fledged validation, and the <a href="http://qpdf.sourceforge.net/files/qpdf-manual.html">documentation</a> states that:</p> <blockquote> <p>A file for which –check reports no errors may still have errors in stream data content but should otherwise be structurally sound</p> </blockquote> <p>Nevertheless, QPDF is still useful for detecting various issues, especially in conjunction with the <code class="language-plaintext highlighter-rouge">--verbose</code> option. Here’s an example command-line:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>qpdf <span class="nt">--check</span> <span class="nt">--verbose</span> whatever.pdf </code></pre></div></div> <h3 id="check-for-ghostscript-rendering-errors">Check for Ghostscript rendering errors</h3> <p>Another useful technique is to process a PDF with Ghostscript (rendering the result to a “nullpage” device). For example:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gs <span class="nt">-dNOPAUSE</span> <span class="nt">-dBATCH</span> <span class="nt">-sDEVICE</span><span class="o">=</span>nullpage whatever.pdf </code></pre></div></div> <p>In case of any problems with the input file, Ghostscript will report quite detailed information. As an example, here’s the output for a PDF with a truncated document trailer:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> **** Error: An error occurred while reading an XREF table. **** The file has been damaged. This may have been caused **** by a problem while converting or transfering the file. **** Ghostscript will attempt to recover the data. **** However, the output may be incorrect. **** Warning: There are objects with matching object and generation **** numbers. The output may be incorrect. **** Error: Trailer dictionary not found. Output may be incorrect. No pages will be processed (FirstPage &gt; LastPage). **** This file had errors that were repaired or ignored. **** Please notify the author of the software that produced this **** file that it does not conform to Adobe's published PDF **** specification. </code></pre></div></div> <h3 id="check-for-errors-with-mutool-info-command">Check for errors with Mutool info command</h3> <p>Running Mutool (part of MuPDF, see above) with the <code class="language-plaintext highlighter-rouge">info</code> command returns information about internal pdf resources. In case of broken or malformed files the output includes error messages, which can be quite informative. Here’s an example command-line:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mutool info whatever.pdf </code></pre></div></div> <h3 id="check-for-errors-with-exiftool">Check for errors with ExifTool</h3> <p><a href="https://exiftool.org/">ExifTool</a> is designed for reading, writing and editing meta-information for a plethora of file formats, including PDF. Although it does not do full-fledged validation, it will report error and warning messages for various read issues, and these can be useful for identifying problematic PDFs. For example, here we use ExifTool on a PDF with some internal byte corruption:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>exiftool corrupted.pdf </code></pre></div></div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ExifTool Version Number : 11.88 File Name : corrupted.pdf Directory : . File Size : 87 kB File Modification Date/Time : 2022:02:07 14:36:47+01:00 File Access Date/Time : 2022:02:07 14:37:11+01:00 File Inode Change Date/Time : 2022:02:07 14:36:59+01:00 File Permissions : rw-rw-r-- File Type : PDF File Type Extension : pdf MIME Type : application/pdf PDF Version : 1.3 Linearized : No Warning : Invalid xref table </code></pre></div></div> <p>In this case the byte corruption results in an “Invalid xref table” warning. Many other errors and warnings are possible. Check out <a href="https://openpreservation.org/blogs/pdf-validation-with-exiftool-quick-and-not-so-dirty/">this blog post by Yvonne Tunnat</a> which discusses PDF “validation” with ExifTool in more detail.</p> <h3 id="other-options">Other options</h3> <ul> <li> <p><a href="https://verapdf.org/">VeraPDF</a> can provide useful information on damaged or invalid PDF documents. However, VeraPDF is primarily aimed at validation against <a href="https://en.wikipedia.org/wiki/PDF/A">PDF/A</a> and <a href="https://en.wikipedia.org/wiki/PDF/UA">PDF/UA</a> profiles, which are both subsets of <a href="https://en.wikipedia.org/wiki/PDF">ISO 32000</a> (which defines the PDF format’s full feature set). As a result, VeraPDF’s validation output can be somewhat difficult to interpret for “regular” PDFS (i.e. documents that are not PDF/A or PDF/UA). Nevertheless, experienced users may find VeraPDF useful for such files as well.</p> </li> <li> <p>Several online resources recommend the <em>pdfinfo</em> tool that is part of Xpdf and Poppler for integrity checking. However, while writing this post I ran a quick test of the tool on a PDF with a truncated document trailer<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">4</a></sup> (which is a very serious flaw), which was not flagged by <em>pdfinfo</em> at all.</p> </li> </ul> <h2 id="pdfa-and-pdfua-compliance-testing-with-verapdf">PDF/A and PDF/UA compliance testing with VeraPDF</h2> <p><a href="https://en.wikipedia.org/wiki/PDF/A">PDF/A</a> comprises a set of ISO-standardized profiles that are aimed at long-term preservation. <a href="https://en.wikipedia.org/wiki/PDF/UA">PDF/UA</a> is another ISO-standardized profile that ensures accessibility for people with disabilities. These are not separate file formats, but rather profiles within ISO 32000 that put some constraints on PDF’s full set of features. <a href="https://verapdf.org/">VeraPDF</a> was originally developed as an open source PDF/A validator that covers all parts of the PDF/A standards. Starting with version 1.18, it also added support for PDF/UA. The following command lists al available validation profiles:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>verapdf <span class="nt">-l</span> </code></pre></div></div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 1a - PDF/A-1A validation profile 1b - PDF/A-1B validation profile 2a - PDF/A-2A validation profile 2b - PDF/A-2B validation profile 2u - PDF/A-2U validation profile 3a - PDF/A-3A validation profile 3b - PDF/A-3B validation profile 3u - PDF/A-3U validation profile ua1 - PDF/UA-1 validation profile </code></pre></div></div> <p>When running VeraPDF, use the <code class="language-plaintext highlighter-rouge">-f</code> (flavour) option to set the desired validation profile. For example, for PDF/A-1A use something like this<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>verapdf <span class="nt">-f</span> 1a whatever.pdf <span class="o">&gt;</span> whatever-1a.xml </code></pre></div></div> <p>And for PDF/UA:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>verapdf <span class="nt">-f</span> ua1 whatever.pdf <span class="o">&gt;</span> whatever-ua.xml </code></pre></div></div> <p>The <a href="https://docs.verapdf.org/cli/validation/">documentation</a> provides more detailed instructions on how to use VeraPDF.</p> <h2 id="document-information-and-metadata-extraction">Document information and metadata extraction</h2> <p>A large number of tools are capable of displaying or extracting technical characteristics and various kinds of metadata, with varying degrees of detail. I’ll only highlight a few here.</p> <h3 id="extract-general-characteristics-with-pdfinfo">Extract general characteristics with pdfinfo</h3> <p>The <em>pdfinfo</em> tool that is part of Xpdf and Poppler is useful for a quick overview of a document’s general characteristics. The basic command line is:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pdfinfo whatever.pdf </code></pre></div></div> <p>Which gives the following result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Creator: PdfCompressor 3.1.32 Producer: CVISION Technologies CreationDate: Thu Sep 2 07:52:56 2021 CEST ModDate: Thu Sep 2 07:53:20 2021 CEST Tagged: no UserProperties: no Suspects: no Form: none JavaScript: no Pages: 1 Encrypted: no Page size: 439.2 x 637.92 pts Page rot: 0 File size: 24728 bytes Optimized: yes PDF version: 1.6 </code></pre></div></div> <h3 id="extract-metadata-with-apache-tika">Extract metadata with Apache Tika</h3> <p><a href="https://tika.apache.org/">Apache Tika</a> is a Java library that supports metadata and content extraction for a wide variety of file formats. For command-line use, download the <em>Tika-app</em> runnable JAR from <a href="https://tika.apache.org/download.html">here</a>. By default, Tika will extract both text and metadata, and report both in XHTML format. Tika has several command-line options that this behaviour. A basic metadata extraction command is (you may need to adapt the path and name of the JAR file)):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java <span class="nt">-jar</span> ~/tika/tika-app-2.1.0.jar <span class="nt">-m</span> whatever.pdf <span class="o">&gt;</span> whatever.txt </code></pre></div></div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Content-Length: 24728 Content-Type: application/pdf X-TIKA:Parsed-By: org.apache.tika.parser.DefaultParser X-TIKA:Parsed-By: org.apache.tika.parser.pdf.PDFParser access_permission:assemble_document: true access_permission:can_modify: true access_permission:can_print: true access_permission:can_print_degraded: true access_permission:extract_content: true access_permission:extract_for_accessibility: true access_permission:fill_in_form: true access_permission:modify_annotations: true dc:format: application/pdf; version=1.6 dcterms:created: 2021-09-02T05:52:56Z dcterms:modified: 2021-09-02T05:53:20Z pdf:PDFVersion: 1.6 pdf:charsPerPage: 0 pdf:docinfo:created: 2021-09-02T05:52:56Z pdf:docinfo:creator_tool: PdfCompressor 3.1.32 pdf:docinfo:modified: 2021-09-02T05:53:20Z pdf:docinfo:producer: CVISION Technologies pdf:encrypted: false pdf:hasMarkedContent: false pdf:hasXFA: false pdf:hasXMP: true pdf:producer: CVISION Technologies pdf:unmappedUnicodeCharsPerPage: 0 resourceName: whatever.pdf xmp:CreateDate: 2021-09-02T07:52:56Z xmp:CreatorTool: PdfCompressor 3.1.32 xmp:MetadataDate: 2021-09-02T07:53:20Z xmp:ModifyDate: 2021-09-02T07:53:20Z xmpMM:DocumentID: uuid:2ec84d65-f99d-49fe-9aac-bd0c1fff5e66 xmpTPg:NPages: 1 </code></pre></div></div> <p>Tika offers several options for alternative output formats (e.g. XMP and JSON); these are all <a href="https://tika.apache.org/2.1.0/gettingstarted.html">explained here</a> (section “Using Tika as a command line utility”).</p> <h3 id="extract-metadata-with-exiftool">Extract metadata with ExifTool</h3> <p><a href="https://exiftool.org/">ExifTool</a> is another good option for metadata extraction. Here’s an example:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>exiftool whatever.pdf </code></pre></div></div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ExifTool Version Number : 11.88 File Name : whatever.pdf Directory : . File Size : 24 kB File Modification Date/Time : 2021:09:02 12:23:32+02:00 File Access Date/Time : 2022:02:07 15:04:11+01:00 File Inode Change Date/Time : 2021:09:02 15:27:38+02:00 File Permissions : rw-rw-r-- File Type : PDF File Type Extension : pdf MIME Type : application/pdf PDF Version : 1.6 Linearized : Yes Create Date : 2021:09:02 07:52:56+02:00 Creator : PdfCompressor 3.1.32 Modify Date : 2021:09:02 07:53:20+02:00 XMP Toolkit : Adobe XMP Core 5.6-c017 91.164464, 2020/06/15-10:20:05 Metadata Date : 2021:09:02 07:53:20+02:00 Creator Tool : PdfCompressor 3.1.32 Format : application/pdf Document ID : uuid:2ec84d65-f99d-49fe-9aac-bd0c1fff5e66 Instance ID : uuid:28d0af59-9373-4358-88f2-c8c4db3915ed Producer : CVISION Technologies Page Count : 1 </code></pre></div></div> <p>ExifTool can also write the extracted metadata to a variety of output formats, which is explained in the documentation.</p> <h3 id="extract-metadata-from-embedded-documents">Extract metadata from embedded documents</h3> <p>One particularly useful feature of Tika is its ability to deal with embedded documents. As an example, <a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/digitally_signed_3D_Portfolio.pdf">this file</a> is a <a href="https://helpx.adobe.com/acrobat/using/overview-pdf-portfolios.html">PDF portfolio</a>, which can contain multiple files and file types. Invoking Tika with the <code class="language-plaintext highlighter-rouge">-J</code> (“output metadata and content from all embedded files”) option results in JSON-formatted output that contains metadata (and also extracted text) for all for all files that are embedded in this document:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java <span class="nt">-jar</span> ~/tika/tika-app-2.1.0.jar <span class="nt">-J</span> digitally_signed_3D_Portfolio.pdf <span class="o">&gt;</span> whatever.json </code></pre></div></div> <h3 id="elaborate-feature-extraction-with-verapdf">Elaborate feature extraction with VeraPDF</h3> <p>Although primarily aimed at PDF/A validation, <a href="https://verapdf.org/">VeraPDF</a> can also be used as a powerful metadata and feature extractor for any PDF file (including files that don’t follow the PDF/A or PDF/UA at all!). By default, VeraPDF is configured to only extract metadata from a PDF’s information dictionary, but this behaviour can be easily changed by modifying a configuration file, which is <a href="https://docs.verapdf.org/cli/config/#features.xml">explained in the documentation</a>. This enables you to obtain detailed information about things like Actions, Annotations, colour spaces, document security features (including encryption), embedded files, fonts, images, and much more. Then use a command line like<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">6</a></sup>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>verapdf --off --extract whatever.pdf &gt; whatever.xml </code></pre></div></div> <p>VeraPDF can also be used to recursively process all files with a .pdf extension in a directory tree, using the following command-line (here, <em>myDir</em> is the root of the directory tree):</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>verapdf --recurse --off --extract myDir &gt; whatever.xml </code></pre></div></div> <p>The <a href="https://docs.verapdf.org/cli/feature-extraction/">VeraPDF documentation</a> discusses the feature extraction functionality in more detail.</p> <h2 id="policy-or-profile-compliance-assessment-with-verapdf">Policy or profile compliance assessment with VeraPDF</h2> <p>The results of the feature extraction exercise described in the previous section can also be used as input for policy-based assessments. For instance, archival institutions may have policies that prohibit e.g. PDFs with encryption or fonts that are not embedded. This can also be done with VeraPDF. This requires that the rules that make up the policy are expressed as a machine-readable <a href="https://en.wikipedia.org/wiki/Schematron">Schematron</a> file. As an example, the Schematron file below is made up of two rules that each prohibit specific encryption-related features:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">&lt;?xml version="1.0"?&gt;</span> <span class="nt">&lt;sch:schema</span> <span class="na">xmlns:sch=</span><span class="s">"http://purl.oclc.org/dsdl/schematron"</span> <span class="na">queryBinding=</span><span class="s">"xslt"</span><span class="nt">&gt;</span> <span class="nt">&lt;sch:pattern</span> <span class="na">name=</span><span class="s">"Disallow encrypt in trailer dictionary"</span><span class="nt">&gt;</span> <span class="nt">&lt;sch:rule</span> <span class="na">context=</span><span class="s">"/report/jobs/job/featuresReport/documentSecurity"</span><span class="nt">&gt;</span> <span class="nt">&lt;sch:assert</span> <span class="na">test=</span><span class="s">"not(encryptMetadata = 'true')"</span><span class="nt">&gt;</span>Encrypt in trailer dictionary<span class="nt">&lt;/sch:assert&gt;</span> <span class="nt">&lt;/sch:rule&gt;</span> <span class="nt">&lt;/sch:pattern&gt;</span> <span class="nt">&lt;sch:pattern</span> <span class="na">name=</span><span class="s">"Disallow other forms of encryption (e.g. open password)"</span><span class="nt">&gt;</span> <span class="nt">&lt;sch:rule</span> <span class="na">context=</span><span class="s">"/report/jobs/job/taskResult/exceptionMessage"</span><span class="nt">&gt;</span> <span class="nt">&lt;sch:assert</span> <span class="na">test=</span><span class="s">"not(contains(.,'encrypted'))"</span><span class="nt">&gt;</span>Encrypted document<span class="nt">&lt;/sch:assert&gt;</span> <span class="nt">&lt;/sch:rule&gt;</span> <span class="nt">&lt;/sch:pattern&gt;</span> <span class="nt">&lt;/sch:schema&gt;</span> </code></pre></div></div> <p>A PDF can subsequently be tested against these rules (here in the file “policy.sch”) using the following basic command-line:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>verapdf <span class="nt">--extract</span> <span class="nt">--policyfile</span> policy.sch whatever.pdf <span class="o">&gt;</span> whatever.xml </code></pre></div></div> <p>The outcome of the policy-based assessment can be found in the output file’s <em>policyReport</em> element. In the example below, the PDF did not meet one of the rules:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;policyReport</span> <span class="na">passedChecks=</span><span class="s">"0"</span> <span class="na">failedChecks=</span><span class="s">"1"</span> <span class="na">xmlns:vera=</span><span class="s">"http://www.verapdf.org/MachineReadableReport"</span><span class="nt">&gt;</span> <span class="nt">&lt;passedChecks/&gt;</span> <span class="nt">&lt;failedChecks&gt;</span> <span class="nt">&lt;check</span> <span class="na">status=</span><span class="s">"failed"</span> <span class="na">test=</span><span class="s">"not(encryptMetadata = 'true')"</span> <span class="na">location=</span><span class="s">"/report/jobs/job/featuresReport/documentSecurity"</span><span class="nt">&gt;</span> <span class="nt">&lt;message&gt;</span>Encrypt in trailer dictionary<span class="nt">&lt;/message&gt;</span> <span class="nt">&lt;/check&gt;</span> <span class="nt">&lt;/failedChecks&gt;</span> <span class="nt">&lt;/policyReport&gt;</span> </code></pre></div></div> <p>More examples can be found in my 2017 post <a href="/2017/06/01/policy-based-assessment-with-verapdf-a-first-impression">Policy-based assessment with VeraPDF - a first impression</a>.</p> <h2 id="text-extraction">Text extraction</h2> <p>Text extraction from PDF documents is notoriously hard. <a href="https://filingdb.com/b/pdf-text-extraction">This post</a> gives a good overview of the main pitfalls. Tim Allison’s excellent <a href="https://irsg.bcs.org/informer/wp-content/uploads/OverviewOfTextExtractionFromPDFs.pdf">Brief Overview of the Portable Document Format (PDF) and Some Challenges for Text Extraction</a> provides a more in-depth discussion, and this really is a must-read for anyone seriously interested in this subject. With that said, quite a few tools are available, and below I list a few that are useful starting points.</p> <h3 id="extract-text-with-pdftotext">Extract text with pdftotext</h3> <p>The <em>pdftotext</em> tool that is part of Poppler and Xpdf is a good starting point. The basic command-line is:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pdftotext whatever.pdf whatever.txt </code></pre></div></div> <p>The tool has lots of options to fine-tune the default behaviour, so make sure to check those out if you’re looking for. Note that the available options vary somewhat between the Poppler and Xpdf versions. The <a href="https://manpages.debian.org/stretch/poppler-utils/pdftotext.1.en.html">documentation of the Poppler version is available here</a>, and <a href="https://www.xpdfreader.com/pdftotext-man.html">here is the Xpdf version</a>.</p> <h3 id="extract-text-with-pdfbox">Extract text with PDFBox</h3> <p>PDFBox is also a good choice for text extraction. Here’s an example command (you may need to adapt the path to the JAR file and its name according to the location and version on your system):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java <span class="nt">-jar</span> ~/pdfbox/pdfbox-app-2.0.24.jar ExtractText whatever.pdf whatever.txt </code></pre></div></div> <p>PDFBox also provides various options, which are <a href="https://pdfbox.apache.org/1.8/commandline.html#extracttext">documented here</a>.</p> <h3 id="extract-text-with-apache-tika">Extract text with Apache Tika</h3> <p>I already mentioned <a href="https://tika.apache.org/">Apache Tika</a> in the metadata extraction section. Tika is also a powerful text extraction tool, and it is particularly useful for situations where text extraction from multiple input formats is needed. For PDF it uses the PDF parser of PDFBox (see previous section). By default, Tika extracts both text and metadata, and reports both in XHTML format. If needed, you can change this behaviour with the <code class="language-plaintext highlighter-rouge">--text</code> option:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java <span class="nt">-jar</span> ~/tika/tika-app-2.1.0.jar <span class="nt">--text</span> whatever.pdf <span class="o">&gt;</span> whatever.txt </code></pre></div></div> <p>Again, an explanation of all available options is <a href="https://tika.apache.org/2.1.0/gettingstarted.html">available here</a> (section “Using Tika as a command line utility”).</p> <h3 id="batch-processing-with-tika">Batch processing with Tika</h3> <p>The above single-file command does not scale well for situations that require the processing of large volumes of PDFs<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">7</a></sup>. In such cases, it’s better to run Tika in batch mode. As an example, the command below will process all files in directory “myPDFs”, and store the results in output directory “tika-out”<sup id="fnref:6"><a href="#fn:6" class="footnote" rel="footnote" role="doc-noteref">8</a></sup>:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java <span class="nt">-jar</span> ~/tika/tika-app-2.1.0.jar <span class="nt">--text</span> <span class="nt">-i</span> ./myPDFs/ <span class="nt">-o</span> ./tika-out/ </code></pre></div></div> <p>Alternatively, you could use TikaServer. A <a href="https://tika.apache.org/download.html">runnable JAR is available here</a>. To use it, first start the server using:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java <span class="nt">-jar</span> ~/tika/tika-server-standard-2.1.0.jar </code></pre></div></div> <p>Once the server is running, use <a href="https://en.wikipedia.org/wiki/CURL">cURL</a> (from another terminal window) to submit text extraction requests:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-T</span> whatever.pdf http://localhost:9998/tika <span class="nt">--header</span> <span class="s2">"Accept: text/plain"</span> <span class="o">&gt;</span> whatever.txt </code></pre></div></div> <p>The full TikaServer documentation is <a href="https://cwiki.apache.org/confluence/display/TIKA/TikaServer">available here</a>.</p> <p>Yet another option is <a href="https://github.com/chrismattmann/tika-python">Tika-python</a>, which is a Python port of Tika that uses TikaServer under the hood (resulting in similar performance).</p> <h2 id="link-extraction">Link extraction</h2> <p>When extracting (hyper)links, it’s important to make a distinction between the following two cases:</p> <ol> <li>Links that are encoded as a “link annotation”, which is a data structure in PDF that results in a clickable link</li> <li>Non-clickable links/URLs that are just part of the body text.</li> </ol> <p>The automated extraction of the first case is straightforward, while the second case depends on some kind of lexical analysis of the body text (typically based on regular expressions). For most practical applications the extraction of both types is desired.</p> <h3 id="extract-links-with-pdfx">Extract links with pdfx</h3> <p>The <a href="https://www.metachris.com/pdfx/">pdfx</a> tool is designed to detect and extract external references, including URLs. Its URL detection uses lexical analysis, and is based on <a href="https://gist.github.com/gruber/8891611">RegEx patterns written by John Gruber</a>. The basic command line for URL extraction is:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pdfx <span class="nt">-v</span> whatever.pdf <span class="o">&gt;</span> whatever.txt </code></pre></div></div> <p>I did some limited testing with this tool in 2016. One issue I ran into is that pdfx <a href="https://github.com/metachris/pdfx/issues/21">truncates URLS that span more than one line</a>. As of 2021, this issue hasn’t been fixed so far, which seriously limits the usefulness of this (otherwise very interesting) tool. It’s worth mentioning that <em>pdfx</em> also provides functionality to automatically download all referenced PDFs from any PDF document. I haven’t tested this myself.</p> <h3 id="other-link-extraction-tools">Other link extraction tools</h3> <ul> <li> <p>Some years ago Ross Spencer wrote <a href="https://github.com/httpreserve/tikalinkextract">a link extraction tool that uses Apache Tika</a>. There’s more info in <a href="https://openpreservation.org/blogs/hyperlinks-in-your-files-how-to-get-them-out-using-tikalinkextract/">this blog post</a>.</p> </li> <li> <p>Around the same time I wrote <a href="https://gist.github.com/bitsgalore/aab680a9bccfc5496948b776ee06397c">this simple extraction script</a> that wraps around Apache Tika and the <a href="https://github.com/mvdan/xurls">xurl</a> tool. I used this to extract URLs from MS Word documents, but this should probably work for PDF too (I haven’t tested this though!).</p> </li> </ul> <h2 id="image-extraction-with-pdfimages">Image extraction with pdfimages</h2> <p>PDFs often contain embedded images, which can be extracted with <em>pdfimages</em> tool that is part of Xpdf/Poppler. At minimum, it takes as its arguments the name of the input PDF document, and the “image-root” which is actually just a text prefix that is used to generate the name of the output images. By default it writes its output to one of the <a href="https://en.wikipedia.org/wiki/Netpbm">Netpbm</a> file formats, but for convenience you might want to use the <code class="language-plaintext highlighter-rouge">-png</code> option, which uses the PNG format instead:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pdfimages <span class="nt">-png</span> whatever.pdf whatever </code></pre></div></div> <p>Output images are now written as “whatever-000.png”, “whatever-001.png”, “whatever-002.png”, and so on. The <code class="language-plaintext highlighter-rouge">-j</code>, <code class="language-plaintext highlighter-rouge">-jp2</code>, <code class="language-plaintext highlighter-rouge">-jbig2</code> and <code class="language-plaintext highlighter-rouge">-ccitt</code> switches can be used to store JPEG, JPEG2000, JBIG2 and CCITT images in their native formats, respectively (or use <code class="language-plaintext highlighter-rouge">-all</code>, which combines all of these options).</p> <!-- ### PDFBox ```bash java -jar ~/pdfbox/pdfbox-app-2.0.24.jar ExtractImages whatever.pdf ``` This gave me the following error for a PDF with an embedded JPEG 2000 image: ``` SEVERE: Cannot read JPEG2000 image: Java Advanced Imaging (JAI) Image I/O Tools are not installed ``` This is a [known bug](https://issues.apache.org/jira/browse/PDFBOX-4681) --> <h2 id="conversion-to-other-graphics-formats-with-pdftocairo">Conversion to other (graphics) formats with pdftocairo</h2> <p>The <em>pdftocairo</em> tool (Xpdf/Poppler ) can convert a PDF to a number of (mostly graphics) formats. The supported output formats are PNG, JPEG, TIFF, PostScript, Encapsulated PostScript, Scalable Vector Graphics and PDF. As an example, the following command will convert each page to a PNG image:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pdftocairo <span class="nt">-png</span> whatever.pdf </code></pre></div></div> <h2 id="list-embedded-image-information-with-pdfimages">List embedded image information with pdfimages</h2> <p>The <em>pdfimages</em> tool is also useful for getting an overview of all embedded images in a PDF, and their main characteristics (width, height, colour, encoding, resolution and size). just user the <code class="language-plaintext highlighter-rouge">-list</code> option as shown below:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pdfimages <span class="nt">-list</span> whatever.pdf </code></pre></div></div> <p>This results in a nice table like this:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>page num type width height color comp bpc enc interp ----------------------------------------------------------- 1 0 image 1830 2658 gray 1 1 jbig2 no 1 1 image 600 773 gray 1 8 jpx no page object ID x-ppi y-ppi size ratio ---------------------------------------- 1 16 0 301 301 99B 0.0% 1 17 0 300 300 17.9K 4.0% </code></pre></div></div> <h2 id="conversion-of-multiple-image-files-to-pdf">Conversion of multiple image files to PDF</h2> <h3 id="losslessly-convert-raster-images-to-pdf-with-img2pdf">Losslessly convert raster images to pdf with img2pdf</h3> <p>The <a href="https://gitlab.mister-muffin.de/josch/img2pdf">img2pdf</a> tool converts a list of image files to PDF. Unlike several other tools (such as ImageMagick), it does not re-encode the source images, but simply embeds them as PDF objects in their original formats. This means that the conversion is always lossless. The following example shows how to convert three <a href="http://fileformats.archiveteam.org/wiki/JP2">JP2 (JPEG 200 Part 1)</a> images:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>img2pdf image1.jp2 image2.jp2 image3.jp2 <span class="nt">-o</span> whatever.pdf </code></pre></div></div> <p>In the resulting PDF, each image is embedded as an image stream with the JPXDecode (JPEG 2000) filter.</p> <h2 id="pdf-comparison-with-comparepdf">PDF comparison with Comparepdf</h2> <p>The <a href="http://www.qtrac.eu/">Comparepdf</a><sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">9</a></sup> tool compares pairs of PDFs, based on either text or visual appearance. By default it uses the program exit code to store the result of the comparison. The tool’s command-line help text explains the possible outcomes:</p> <blockquote> <p>A return value of 0 means no differences detected; 1 or 2 signifies an error; 10 means they differ visually, 13 means they differ textually, and 15 means they have different page counts</p> </blockquote> <p>For clarity I used the <code class="language-plaintext highlighter-rouge">-v</code> switch in the examples below, which activates verbose output. To test if two PDFs contain the same text, use:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>comparepdf <span class="nt">-ct</span> <span class="nt">-v</span><span class="o">=</span>2 whatever.pdf wherever.pdf </code></pre></div></div> <p>If al goes well the output is either “No differences detected” or “Files have different texts”.</p> <p>To compare the visual appearance of two PDFs, use:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>comparepdf <span class="nt">-ca</span> <span class="nt">-v</span><span class="o">=</span>2 whatever.pdf wherever.pdf </code></pre></div></div> <p>In this case the output either shows “No differences detected” or “Files look different”.</p> <h2 id="repair-a-corrupted-pdf">Repair a corrupted PDF</h2> <p>Sometimes it is possible to recover the contents of corrupted or otherwise damaged PDF documents. <a href="https://superuser.com/questions/278562/how-can-i-fix-repair-a-corrupted-pdf-file">This thread on Super User</a> mentions two useful options.</p> <h3 id="repair-with-ghostscript">Repair with Ghostscript</h3> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gs <span class="nt">-o</span> whatever_repaired.pdf <span class="nt">-sDEVICE</span><span class="o">=</span>pdfwrite <span class="nt">-dPDFSETTINGS</span><span class="o">=</span>/prepress whatever_corrupted.pdf </code></pre></div></div> <h3 id="repair-with-pdftocairo">Repair with pdftocairo</h3> <p>A second option mentioned in the Super User thread is <em>pdftocairo</em>, which is part of Xpdf and Poppler:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pdftocairo <span class="nt">-pdf</span> whatever_corrupted.pdf whatever_repaired.pdf </code></pre></div></div> <p>It’s worth adding here that the success of any repair action largely depends on the nature and extent of the damage/corruption, so your mileage may very. Always make sure to carefully check the result, and keep a copy of the original file.</p> <h3 id="repair-with-pdftk">Repair with PDFtk</h3> <p>Finally, <em>pdftk</em> can, <a href="https://www.pdflabs.com/docs/pdftk-cli-examples/">according to its documentation</a>, “repair a PDF’s corrupted XREF table and stream lengths, if possible”. This uses the following command line:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pdftk whatever_corrupted.pdf output whatever_repaired.pdf </code></pre></div></div> <h2 id="reduce-size-of-pdf-with-hi-res-images-with-ghostscript">Reduce size of PDF with hi-res images with Ghostscript</h2> <p>The following Ghostscript command (source <a href="https://askubuntu.com/questions/113544/how-can-i-reduce-the-file-size-of-a-scanned-pdf-file/256449#256449">here</a> can be useful to reduce the size of a large PDF with high-resolution graphics (note that this will result in quality loss):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gs <span class="nt">-sDEVICE</span><span class="o">=</span>pdfwrite <span class="se">\</span> <span class="nt">-dCompatibilityLevel</span><span class="o">=</span>1.4 <span class="se">\</span> <span class="nt">-dPDFSETTINGS</span><span class="o">=</span>/ebook <span class="se">\</span> <span class="nt">-dNOPAUSE</span> <span class="nt">-dQUIET</span> <span class="nt">-dBATCH</span> <span class="se">\</span> <span class="nt">-sOutputFile</span><span class="o">=</span>whatever_small.pdf whatever_large.pdf </code></pre></div></div> <h2 id="reduce-size-of-pdf-with-hi-res-images-with-imagemagick">Reduce size of PDF with hi-res images with ImageMagick</h2> <p>As an alternative to the above Ghostscript command (which achieves a size reduction mainly by downsampling the images in the PDF to as lower resolution), you can also use <a href="https://imagemagick.org/">ImageMagick</a>’s <a href="https://imagemagick.org/script/convert.php"><em>convert</em> tool</a>. This allows you to reduce the file size by changing any combination of resolution (<a href="https://imagemagick.org/script/command-line-options.php#density"><code class="language-plaintext highlighter-rouge">-density</code></a> option), compression type (<a href="https://imagemagick.org/script/command-line-options.php#compress"><code class="language-plaintext highlighter-rouge">-compress</code></a> option) and compression quality (<a href="https://imagemagick.org/script/command-line-options.php#quality"><code class="language-plaintext highlighter-rouge">-quality</code></a> option).</p> <p>For example, the command below (source <a href="https://askubuntu.com/questions/113544/how-can-i-reduce-the-file-size-of-a-scanned-pdf-file/469255#469255">here</a>) reduces the size of a source PDF by re-encoding all images as JPEGs with 70% quality at 300 ppi resolution:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>convert <span class="nt">-density</span> 300 <span class="se">\</span> <span class="nt">-compress</span> jpeg <span class="se">\</span> <span class="nt">-quality</span> 70 <span class="se">\</span> whatever_large.pdf whatever_small.pdf </code></pre></div></div> <p>If the <code class="language-plaintext highlighter-rouge">-density</code> value is omitted, <em>convert</em> resamples all images to 72 ppi by default. If you don’t want that, make sure to set the <code class="language-plaintext highlighter-rouge">-density</code> value to the resolution of your source PDF (see the section “List embedded image information with pdfimages” on how to do that).</p> <p>Even though ImageMagick’s <em>convert</em> tool uses Ghostscript under the hood, it doesn’t preserve any text (and probably most other features) of the source PDF, so only use this if you’re only interested in the image data!</p> <h2 id="inspect-low-level-pdf-structure">Inspect low-level PDF structure</h2> <p>The following tools are useful for inspecting and browsing the internal (low-level object) structure of PDF files.</p> <h3 id="inspect-with-pdfbox-pdfdebugger">Inspect with PDFBox PDFDebugger</h3> <p>PDFBox includes a “PDF Debugger”, which you can start with the following command:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java <span class="nt">-jar</span> ~/pdfbox/pdfbox-app-2.0.24.jar PDFDebugger whatever.pdf </code></pre></div></div> <p>Subsequently a GUI window pops up that allows you to browse the PDF’s internal objects:</p> <figure class="image"> <img src="/images/2021/09/pdf-debugger.png" alt="PDF Debugger screenshot" /> <figcaption>Screenshot of PDFBOX PDFDebugger.</figcaption> </figure> <h3 id="inspect-with-itext-rups">Inspect with iText RUPS</h3> <p>The <a href="https://github.com/itext/i7j-rups">itext RUPS</a> viewer provides similar functionality to PDF Debugger. You can download a self-contained runnable JAR <a href="https://github.com/itext/i7j-rups/releases/latest">here</a> (select the “only-jars” ZIP file). Run it using:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java <span class="nt">-jar</span> ~/itext-rups/itext-rups-7.1.16.jar </code></pre></div></div> <p>Then open a PDF from the GUI, and browse your way through its internal structure:</p> <figure class="image"> <img src="/images/2021/09/itext-rups.png" alt="iText RUPS screenshot" /> <figcaption>Screenshot of iText RUPS.</figcaption> </figure> <h2 id="view-search-and-extract-pdf-objects-with-mutool-show">View, search and extract PDF objects with mutool show</h2> <p>Mutool’s <em>show</em> command allows you to print user-defined low-level PDF objects to stdout. A couple of things you can do with this:</p> <ul> <li>Print the document trailer: <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mutool show whatever.pdf trailer </code></pre></div> </div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>trailer &lt;&lt; /DecodeParms &lt;&lt; /Columns 3 /Predictor 12 &gt;&gt; /Filter /FlateDecode /ID [ &lt;500AB94E8F45C149808B2EEE98528B78&gt; &lt;431017E495216040A953126BB73D0CD4&gt; ] /Index [ 11 10 ] /Info 10 0 R /Length 47 /Prev 24426 /Root 12 0 R /Size 21 /Type /XRef /W [ 1 2 0 ] &gt;&gt; </code></pre></div> </div> </li> <li>Print the cross-reference table: <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mutool show whatever.pdf xref </code></pre></div> </div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>xref 0 21 00000: 0000000000 00000 f 00001: 0000019994 00000 n 00002: 0000020399 00000 n 00003: 0000020534 00000 n :: etc </code></pre></div> </div> </li> <li>Print an indirect object by its number: <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mutool show whatever.pdf 12 </code></pre></div> </div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>12 0 obj &lt;&lt; /Metadata 4 0 R /Pages 9 0 R /Type /Catalog &gt;&gt; endobj </code></pre></div> </div> </li> <li>Extract only stream contents as raw binary data and write to a new file: <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mutool show <span class="nt">-be</span> <span class="nt">-o</span> whatever.dat whatever.pdf 151 </code></pre></div> </div> <p>This command is particularly useful for extracting the raw data from a stream object (e.g. an image or multimedia file).</p> </li> </ul> <p>More advanced queries are possible as well. For example, the <a href="https://mupdf.com/docs/manual-mutool-show.html">mutool manual</a> gives the following example, which shows all JPEG compressed stream objects in a file:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mutool show whatever.pdf <span class="nb">grep</span> | <span class="nb">grep</span> <span class="s1">'/Filter/DCTDecode'</span> </code></pre></div></div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1 0 obj &lt;&lt;/BitsPerComponent 8/ColorSpace/DeviceRGB/Filter/DCTDecode/Height 516/Length 76403/Subtype/Image/Type/XObject/Width 1226&gt;&gt; stream 18 0 obj &lt;&lt;/BitsPerComponent 8/ColorSpace/DeviceRGB/Filter/DCTDecode/Height 676/Length 149186/Subtype/Image/Type/XObject/Width 1014&gt;&gt; stream 19 0 obj &lt;&lt;/BitsPerComponent 8/ColorSpace/DeviceRGB/Filter/DCTDecode/Height 676/Length 142232/Subtype/Image/Type/XObject/Width 1014&gt;&gt; stream 24 0 obj &lt;&lt;/BitsPerComponent 8/ColorSpace/DeviceRGB/Filter/DCTDecode/Height 676/Length 192073/Subtype/Image/Type/XObject/Width 1014&gt;&gt; stream 25 0 obj &lt;&lt;/BitsPerComponent 8/ColorSpace/DeviceRGB/Filter/DCTDecode/Height 676/Length 141081/Subtype/Image/Type/XObject/Width 1014&gt;&gt; stream </code></pre></div></div> <h2 id="incremental-updates-and-document-versions">Incremental updates and document versions</h2> <p>The <a href="https://github.com/enferex/pdfresurrect"><em>pdfresurrect</em></a> tool<sup id="fnref:10"><a href="#fn:10" class="footnote" rel="footnote" role="doc-noteref">10</a></sup> allows you to inspect a PDF for incremental updates, and restore previous versions of the document.</p> <h3 id="show-the-number-of-incremental-updatesversions">Show the number of incremental updates/versions</h3> <p>To show the number of incremental updates, use:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pdfresurrect <span class="nt">-q</span> whatever.pdf </code></pre></div></div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>whatever.pdf: 8 </code></pre></div></div> <p>So, in this case the PDF contains 8 versions, which means that after its initial creation it was updated 7 times.</p> <h3 id="restore-all-versions">Restore all versions</h3> <p>To restore all versions of a PDF, use:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pdfresurrect <span class="nt">-w</span> whatever.pdf </code></pre></div></div> <p>This creates a directory “whatever-versions” with all versions of the file, as well as a text file with a summary of the changes between the versions:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./whatever-versions/ ├── whatever-version-1.pdf ├── whatever-version-2.pdf ├── whatever-version-3.pdf ├── whatever-version-4.pdf ├── whatever-version-5.pdf ├── whatever-version-6.pdf ├── whatever-version-7.pdf ├── whatever-version-8.pdf └── whatever-versions.summary </code></pre></div></div> <h3 id="remove-information-from-previous-versions">Remove information from previous versions</h3> <p><em>Pdfresurrect</em> also allows you to remove (“scrub”) the information from previous versions altogether, using the <code class="language-plaintext highlighter-rouge">-s</code> switch. However, the <a href="https://github.com/enferex/pdfresurrect">documentation</a> warns that this functionality is experimental, and that it “should not be trusted for any serious security uses”. It also mentions that “currently this feature will likely not render a working pdf”. I did a quick test on a PDF with 8 document versions. For the resulting “scrubbed” PDF, running <em>pdfresurrect</em> with the <code class="language-plaintext highlighter-rouge">-q</code> switch still resulted in 8 reported versions, and running it with the <code class="language-plaintext highlighter-rouge">-w</code> switch resulted in 8 restored versions (which oddly were not identical to the versions retored from the original file). In its current state I wouldn’t recommend using this feature.</p> <h2 id="final-remarks">Final remarks</h2> <p>I intend to make this post a “living” document, and will add more PDF “recipes” over time. Feel free to leave a comment in case you spot any errors or omissions!</p> <h2 id="update-on-hacker-news-topic">Update on Hacker News topic</h2> <p>Someone created a <a href="https://news.ycombinator.com/item?id=33145498">Hacker News topic on this post</a>. The comments mention some additional tool suggestions that look useful. I might add some of these to a future revision.</p> <h2 id="further-resources">Further resources</h2> <ul> <li><a href="https://doi.org/10.46430/phen0088">Moritz Mähr, “Working with batches of PDF files”, The Programming Historian 9 (2020)</a></li> <li><a href="https://coptr.digipres.org/index.php/PDF">PDF tools in Community Owned Digital Preservation Tool Registry (COPTR)</a></li> <li><a href="/2017/06/01/policy-based-assessment-with-verapdf-a-first-impression">Policy-based assessment with VeraPDF - a first impression</a></li> <li><a href="https://filingdb.com/b/pdf-text-extraction">What’s so hard about PDF text extraction? ​</a></li> <li><a href="https://irsg.bcs.org/informer/wp-content/uploads/OverviewOfTextExtractionFromPDFs.pdf">Tim Allison, “Brief Overview of the Portable Document Format (PDF) and Some Challenges for Text Extraction”</a></li> <li><a href="https://openpreservation.org/blogs/pdf-validation-with-exiftool-quick-and-not-so-dirty/">Yvonne Tunnat, “PDF Validation with ExifTool – quick and not so dirty”</a></li> <li><a href="https://openpreservation.org/blogs/trouble-shooting-pdf-validation-errors-a-case-of-pdf-hul-38/">Micky Lindlar, “Trouble-shooting PDF validation errors – a case of PDF-HUL-38”</a></li> <li><a href="https://news.ycombinator.com/item?id=33145498">Hacker News topic on this post</a></li> </ul> <h2 id="revision-history">Revision history</h2> <ul> <li>7 September 2021: added sections on metadata extraction and Tika batch processing, following suggestions by Tim Allison.</li> <li>8 September 2021: added section on inspecting low-level PDF structure with iText RUPS, as suggested by Mark Stephens; added sections on PDFtk as suggested by Tyler Thorsted; corrected errors in <em>pdftocairo</em> and <em>gs</em> examples.</li> <li>9 September 2021: added section on image to PDF conversion.</li> <li>27 January 2022: added reference to Tim Allison’s article on PDF text extraction.</li> <li>7 February 2022: added sections on Exiftool, and added reference to Yvonne Tunnat’s blog post on PDF validation with ExifTool.</li> <li>10 October 2022: added update on and link to Hacker News topic on this post.</li> <li>28 November 2022: added reference to Micky Lindlar’s blog post on trouble-shooting PDF validation errors.</li> <li>16 February 2023: added section on reducing PDF file size with ImageMagick’s <em>convert</em> tool.</li> <li>26 September 2024: corrected mutool stream content extraction example.</li> <li>18 December 2024: added section on Arlington PDF Model Checker.</li> <li>25 January 2025: added section on incremental updates and document versions.</li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:7"> <p>The Debian package of the “original” PDFtk software was <a href="https://www.joho.se/2020/10/01/pdftk-and-php-pdftk-on-ubuntu-18-04-without-using-snap/">removed from the Ubuntu repositories</a> around 2018 due to “dependency issues”. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:8"> <p>See the “Limitations” section in <a href="https://github.com/pdf-association/arlington-pdf-model">the Arlington Model readme</a>. Personally, I’d be really excited to see some future software tool that combines the Arlington Model logic with additional checks for aspects that are not covered by it, such as file structure and cross-reference tables. This could really be the ultimate PDF validator, that would make several other tools in this section obsolete. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:9"> <p>Because of of this near-complete coverage of PDF objects, it’s also likely to report more problems for any given file than other tools. Since PDF readers are generally quite forgiving of common deviations of the specifification, many of these problems won’t affect rendering. There’s not much to do about this, as a validator by defininition needs to be strict. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:1"> <p>Command line: <code class="language-plaintext highlighter-rouge">pdfinfo whatever.pdf</code> <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>In this example output is redirected to a file; this is generally a good idea because of the amount of XML output generated by VeraPDF. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>The <code class="language-plaintext highlighter-rouge">--off</code> switch disables PDF/A validation. Output is redirected to a file (recommended because, depending on the configuration used, VeraPDF can generate a <em>lot</em> of output). <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>This is because a new Java VM is started for each processed PDF, which will result in poor performance. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:6"> <p>Of course this also works for metadata extraction, and both text and metadata extraction can be combined in one single command. As an example, the following command will extract both text and metadata, including any embedded documents: <code class="language-plaintext highlighter-rouge">java -jar ~/tika/tika-app-2.1.0.jar -J --text -i ./myPDFs/ -o ./tika-out/</code> <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>On Debian-based systems you can install it using <code class="language-plaintext highlighter-rouge">sudo apt install comparepdf</code>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:10"> <p>On Debian-based systems you can install it using <code class="language-plaintext highlighter-rouge">sudo apt install pdfresurrect</code>. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2021/09/06/pdf-processing-and-analysis-with-open-source-tools https://bitsgalore.org/2021/09/06/pdf-processing-and-analysis-with-open-source-tools 2021-09-06T00:00:00+02:00 <figure class="image"> <img src="/images/2021/02/stewardess-phone.jpg" alt="Satellite image of Wadden Sea" /> <figcaption>Production photo from "2001: A Space Odyssey". ©Stanley Kubrick Archives/TASCHEN.</figcaption> </figure> <p>My <a href="/2021/02/09/four-android-emulators-two-apps">previous post</a> addressed the emulation of mobile Android apps. In this follow-up, I’ll explore some other aspects of mobile app preservation, with a focus on acquisition and ingest processes. The <a href="https://zenodo.org/record/3460450">2019 iPres paper on the Acquisition and Preservation of Mobile eBook Apps</a> by Maureen Pennock, Peter May and Michael Day again was the departure point. In its concluding section, they recommend:</p> <blockquote> <p>In terms of target formats for acquisition, we reach the undeniable conclusion that acquisition of the app in its packaged form (either an IPA file or an APK file) is optimal for ensuring organisations at least acquire a complete published object for preservation.</p> </blockquote> <p>And:</p> <blockquote> <p>[T]his form should at least also include sufficient metadata about inherent technical dependencies to understand what is needed to meet them.</p> </blockquote> <p>In practical terms, this means that the workflows that are used for acquisition and (pre-)ingest must include components that are able to deal with the following aspects:</p> <ol> <li>Acquisition of the app packages (either by direct deposit from the publisher, or using the app store).</li> <li>Identification of the package format (APK for Android, IPA for iOS).</li> <li>Identification of metadata about the app’s technical dependencies.</li> </ol> <p>The main objective of this post is to get an idea of what would be needed to implement these components. Is it possible to do all of this with existing tools? If not so, what are the gaps? The underlying assumption here is an emulation-based preservation strategy<sup id="fnref:14"><a href="#fn:14" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>.</p> <!-- more --> <h2 id="outline-of-this-post">Outline of this post</h2> <p>As for the acquisition component, Pennock, May and Day recommend direct publisher deposit, as this may avoid some potential problems related to digital rights management and dependencies on content that is hosted remotely. Since we’ve only started exploring mobile app preservation at this stage, it’s too early to make any assumptions about what acquisition route would work best in our case. Because of this, I started out by investigating to what extent it is possible to download APK and iOS packages from the Google Play Store and the Apple App Store, respectively.</p> <p>I then tried to do automatic format identification on some sample APK and IPA packages, using recent versions of <a href="https://www.itforarchivists.com/siegfried/">Siegfried</a>, <a href="http://darwinsys.com/file/">Unix File</a> and <a href="http://tika.apache.org/">Apache Tika</a>.</p> <p>Next, I looked at how the APK and IPA formats store metadata about an app’s technical dependencies, and how this information can be extracted.</p> <p>The following sections discuss these components for both the Android and iOS platforms. For convenience I included a summary of the main findings at the end of this post, followed by some observations on the value of additional documentation such as video recordings that show mobile apps in action.</p> <h2 id="downloading-android-packages">Downloading Android packages</h2> <p>Android apps are distributed as <a href="https://en.wikipedia.org/wiki/Android_application_package">Android Package (APK)</a> installer files through the <a href="https://play.google.com/store/apps?hl=en">Google Play Store</a>. However, the Play Store doesn’t allow you to download APK files on a non-Android device, which is a problem within a preservation workflow. Various third-party websites exist that offer the possibility to download APK installers, but it is often difficult to establish their trustworthiness. Some of these sites also re-package the original app data, which introduces various concerns related to security and authenticity. Because of this, I would strongly advise against using any of these services within a preservation workflow.</p> <h3 id="virtual-machine-method">Virtual machine method</h3> <p>A better (but somewhat cumbersome) solution would be to set up a virtual machine or emulator with Android<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>, and use that to install the app directly from the Play store. Once installed, it is possible to transfer the APK installer file from the virtual machine to the host machine using the <a href="https://developer.android.com/studio/command-line/adb">Android Debug Bridge (adb) tool</a>. This involves the following steps:</p> <ol> <li>Get a list of all installed packages on the virtual machine (redirecting output to a text file): <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>adb shell pm list packages <span class="o">&gt;</span> packages.txt </code></pre></div> </div> </li> <li>Look up the package id of the installed app in this file. Taking the <a href="https://play.google.com/store/apps/details?id=com.Triplee.TripleeSocial">ARize app</a> from my previous post as an example, the identifier is <code class="language-plaintext highlighter-rouge">com.Triplee.TripleeSocial</code>. We can then use the following command to find the full file path of the package on the VM: <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>adb shell pm path com.Triplee.TripleeSocial </code></pre></div> </div> <p>Result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>package:/data/app/com.Triplee.TripleeSocial-r8iVFUp1MOSAc6LmHA1MDQ==/base.apk </code></pre></div> </div> </li> <li>Use the above file path to download the package to the host machine: <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>adb pull /data/app/com.Triplee.TripleeSocial-r8iVFUp1MOSAc6LmHA1MDQ<span class="o">==</span>/base.apk </code></pre></div> </div> </li> </ol> <p>In this case, this results in a file “base.apk”.</p> <h3 id="gplaycli-method">Gplaycli method</h3> <p>As the above method is a bit clumsy, I started looking for tools that allow downloading packages from the Play Store directly. Several such open-source tools exist, but many of these are abondoned projects that no longer work. After trying out a few of them, I utimately had success with <a href="https://github.com/matlink/gplaycli">gplaycli</a>, which is “a command line tool to search, install, update Android applications from the Google Play Store”. It allows you to download an APK, using the App ID as an identifier. Taking the ARize app as an example again, we can download the APK with the following command<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gplaycli <span class="nt">-d</span> com.Triplee.TripleeSocial </code></pre></div></div> <p>This resulted in a file “com.Triplee.TripleeSocial.apk”. I verified the file by doing a bitwise comparison against the APK obtained from the “virtual machine method” described in the previous section<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>. This confirmed both files were identical. It’s worth mentioning that gplaycli is also <a href="https://github.com/matlink/gplaycli/issues/8">reported to work for downloading paid apps</a> (provided the proper login credentials are used), but I haven’t tested this.</p> <h2 id="android-package-identification">Android package identification</h2> <p>As most archival ingest workflows include a format identification component, I tried to identify the ARize and Immer Android packages from my previous post with 3 widely used format identification tools. The table below shows the results:</p> <table> <thead> <tr> <th style="text-align: left">Tool</th> <th style="text-align: left">Version</th> <th style="text-align: left">ID</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><a href="https://www.itforarchivists.com/siegfried/">Siegfried</a></td> <td style="text-align: left">1.9.1; DROID Signature File V97<sup id="fnref:7"><a href="#fn:7" class="footnote" rel="footnote" role="doc-noteref">5</a></sup></td> <td style="text-align: left">x-fmt/412 (Java Archive Format)<br />x-fmt/263 (ZIP Format)</td> </tr> <tr> <td style="text-align: left"><a href="http://darwinsys.com/file/">Unix File</a></td> <td style="text-align: left">5.32</td> <td style="text-align: left">application/zip</td> </tr> <tr> <td style="text-align: left"><a href="http://tika.apache.org/">Apache Tika</a></td> <td style="text-align: left">1.23</td> <td style="text-align: left">application/vnd.android.package-archive</td> </tr> </tbody> </table> <p>Apache Tika was the only tool that identified both files as Android packages. Siegfried (which uses the <a href="https://www.nationalarchives.gov.uk/PRONOM/Default.aspx">PRONOM</a> format signatures) identified one file as a regular ZIP file, and the other one as a <a href="https://en.wikipedia.org/wiki/JAR_(file_format)">Java Archive</a>. Since the Android package format is based on the Java Archive format (which is in turn a subset of the ZIP format) this result is not necessarily wrong, but it lacks specificity. At the time of writing, the <a href="https://www.nationalarchives.gov.uk/PRONOM/Default.aspx">PRONOM</a> technical registry does not have an entry for the Android package format<sup id="fnref:6"><a href="#fn:6" class="footnote" rel="footnote" role="doc-noteref">6</a></sup>, so this result is not surprising. A look at <a href="https://github.com/apache/tika/blob/618345263ee41108e1a225dbcdbb8db16b2aae28/tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml#L316">Tika’s Mimetype definition file</a> reveals that Tika only uses the file extension to differentiate between the Android packages and Java archives:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;mime-type</span> <span class="na">type=</span><span class="s">"application/vnd.android.package-archive"</span><span class="nt">&gt;</span> <span class="nt">&lt;sub-class-of</span> <span class="na">type=</span><span class="s">"application/java-archive"</span><span class="nt">/&gt;</span> <span class="nt">&lt;glob</span> <span class="na">pattern=</span><span class="s">"*.apk"</span><span class="nt">/&gt;</span> <span class="nt">&lt;/mime-type&gt;</span> </code></pre></div></div> <h2 id="android-package-metadata">Android package metadata</h2> <p>To ensure long-term access, it is vital that an archived app installer is accompanied by <a href="https://en.wikipedia.org/wiki/Preservation_metadata">preservation metadata</a> about the technical environment that is needed to render it. At the very minimum this would include details about the required Android version(s), hardware features, and shared software libraries. This information (and much more) is stored in an Android Package’s <a href="https://developer.android.com/guide/topics/manifest/manifest-intro">App Manifest</a>. The App Manifest is stored in a <a href="https://en.wikipedia.org/wiki/Binary_XML">binary XML</a> format for which <a href="https://reverseengineering.stackexchange.com/questions/21806/where-is-android-binary-xml-format-documented">no publicly available documentation exists</a>. This makes reading it somewhat challenging, although <a href="https://stackoverflow.com/q/4191762/1209004">various software solutions for decoding the App Manifest exist</a>.</p> <h2 id="extraction-of-app-manifest">Extraction of App Manifest</h2> <p><a href="https://developer.android.com/studio/command-line/apkanalyzer.html">Apkanalyzer</a>, which is part of <a href="https://developer.android.com/studio/">Android Studio</a>, is the tool that is officially supported by Google. However, running apkanalyzer only resulted in a sequence of Java exceptions for me<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">7</a></sup>. Besides, it’s not entirely clear if the <a href="https://developer.android.com/studio/terms.html">terms and conditions</a> of Android Studio permit is use in an archival workflow<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">8</a></sup>. I eventually found <a href="https://github.com/androguard/androguard">Androguard</a>, which is a Python-based tool that is primarily aimed at reverse-engineering Android apps. Using the command below, it will extract and decode an APK’s app manifest, resulting in a human-readable XML file:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>androguard axml com.Triplee.TripleeSocial.apk <span class="nt">-o</span> arize-android.xml </code></pre></div></div> <h2 id="interesting-app-manifest-elements">Interesting App Manifest elements</h2> <p>The decoded app manifest from the above example can be found in full <a href="https://github.com/KBNLresearch/mobile-apps/blob/main/sample-files/arize-androidManifest.xml">here</a>. A detailed discussion of the app manifest is beyond the scope of this post, but it’s worth highlighting a few elements that are particularly interesting:</p> <ul> <li>The <a href="https://developer.android.com/guide/topics/manifest/uses-sdk-element">uses-sdk</a> element contains information about the app’s compatibility with one or more Android versions: <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;uses-sdk</span> <span class="na">android:minSdkVersion=</span><span class="s">"24"</span> <span class="na">android:targetSdkVersion=</span><span class="s">"29"</span><span class="nt">/&gt;</span> </code></pre></div> </div> <p>Here, the (confusingly named) <code class="language-plaintext highlighter-rouge">minSdkVersion</code> and <code class="language-plaintext highlighter-rouge">targetSdkVersion</code> attributes define the minimum and target API levels of the app, respectively. In <a href="https://developer.android.com/guide/topics/manifest/uses-sdk-element#ApiLevels">the table here</a> we see that API level 24 (the minimum level) corresponds to Android 7.0, and level 29 (the target level) to Android 10.</p> </li> <li>The <a href="https://developer.android.com/guide/topics/manifest/uses-feature-element">uses-feature</a> element is used to declare hardware or software features that are used by the app: <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;uses-feature</span> <span class="na">android:name=</span><span class="s">"android.hardware.camera"</span> <span class="na">android:required=</span><span class="s">"true"</span><span class="nt">/&gt;</span> </code></pre></div> </div> <p>In the above example, it informs us that the app needs a camera.</p> </li> <li>The <a href="https://developer.android.com/guide/topics/manifest/uses-library-element">uses-library</a> element tells us about any shared libraries that the app depends on.</li> </ul> <p>The above information largely defines the (emulated) technical environment that is required to run the app. Even though I’ve only skimmed the surface of the App Manifest here, its importance as a source for deriving technical and preservation metadata about an Android app should be clear.</p> <h2 id="downloading-ios-packages">Downloading iOS packages</h2> <p>Apple iOS apps are distributed through the <a href="https://www.apple.com/app-store/">Apple App Store</a>. Installer packages are published in the <a href="https://en.wikipedia.org/wiki/.ipa">iOS App Store Package (IPA)</a> format. A good overview of the format can be found <a href="https://web.archive.org/web/20200714200020/https://blog.razb.me/pulling-apart-an-ios-app/">here</a>. Like Google’s Play Store, Apple doesn’t allow you to download the packages on anything but an Apple device. Unlike the Android situation, there don’t appear to be any tools that are able to get around this limitation, and this seriously limits the possibilities to incorporate downloading iOS packages as part of a preservation workflow. It might be possible to work around these limitations to some degree by installing the app on either a physical or virtual<sup id="fnref:10"><a href="#fn:10" class="footnote" rel="footnote" role="doc-noteref">9</a></sup> iOS device, and then transfer the app to another machine. The open-source <a href="https://libimobiledevice.org/">libimobiledevice</a> library appears to be capable of file transfers between iOS and other platforms. However, according to various online sources iOS doesn’t actually keep the original IPA files after installation<sup id="fnref:8"><a href="#fn:8" class="footnote" rel="footnote" role="doc-noteref">10</a></sup>. I’m unable to confirm this, as I don’t currently have an iOS device available for further testing.</p> <h2 id="ios-package-identification">iOS package identification</h2> <p>As I was unable to obtain any IPA installers from the Apple App Store, I downloaded some random IPA files from an <a href="https://en.wikipedia.org/wiki/IOS_jailbreaking">iOS jailbreaking</a> website<sup id="fnref:9"><a href="#fn:9" class="footnote" rel="footnote" role="doc-noteref">11</a></sup>, and ran them through Siegfried, Unix File and Apache Tika. The table below shows the results:</p> <table> <thead> <tr> <th style="text-align: left">Tool</th> <th style="text-align: left">Version</th> <th style="text-align: left">ID</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><a href="https://www.itforarchivists.com/siegfried/">Siegfried</a></td> <td style="text-align: left">1.9.1; DROID Signature File V97<sup id="fnref:7:1"><a href="#fn:7" class="footnote" rel="footnote" role="doc-noteref">5</a></sup></td> <td style="text-align: left">x-fmt/263 (ZIP Format)</td> </tr> <tr> <td style="text-align: left"><a href="http://darwinsys.com/file/">Unix File</a></td> <td style="text-align: left">5.32</td> <td style="text-align: left">application/zip</td> </tr> <tr> <td style="text-align: left"><a href="http://tika.apache.org/">Apache Tika</a></td> <td style="text-align: left">1.23</td> <td style="text-align: left">application/x-itunes-ipa</td> </tr> </tbody> </table> <p>These results are similar to the situation for Android packages. Only Apache Tika was able to identify these files as IPA packages. Both Siegfried and File could only detect the container format. An inspection of PRONOM confirmed that it doesn’t include the IPA format yet. Also, Tika’s specific result for this format is again <a href="https://github.com/apache/tika/blob/master/tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml#L3819">only based on a file extension pattern</a>:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;mime-type</span> <span class="na">type=</span><span class="s">"application/x-itunes-ipa"</span><span class="nt">&gt;</span> <span class="nt">&lt;sub-class-of</span> <span class="na">type=</span><span class="s">"application/zip"</span><span class="nt">/&gt;</span> <span class="nt">&lt;_comment&gt;</span>Apple iOS IPA AppStore file<span class="nt">&lt;/_comment&gt;</span> <span class="nt">&lt;glob</span> <span class="na">pattern=</span><span class="s">"*.ipa"</span><span class="nt">/&gt;</span> <span class="nt">&lt;/mime-type&gt;</span> </code></pre></div></div> <h2 id="ios-package-metadata">iOS package metadata</h2> <p>For iOS apps, the <a href="https://developer.apple.com/documentation/bundleresources/information_property_list">information property list file</a> (Info.plist) in the root of the bundle directory<sup id="fnref:11"><a href="#fn:11" class="footnote" rel="footnote" role="doc-noteref">12</a></sup> contains various metadata, including information about the required technical environment. Confusingly, <a href="https://en.wikipedia.org/wiki/Property_list">Apple property lists</a> can be implemented in both XML and binary formats. For both test files I analyzed, the format was XML, but I’m not entirely sure if this is always the case. The Apple developer’s site provides <a href="https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/PropertyLists/UnderstandXMLPlist/UnderstandXMLPlist.html#//apple_ref/doc/uid/10000048i-CH6-SW1">a brief explanation of the format</a>, and I’ve uploaded <a href="https://github.com/KBNLresearch/mobile-apps/blob/main/sample-files/Info.plist">an example file here</a>. Even though the format only defines simple key-value pairs, Apple’s implementation is unusual to say the least. Instead of using the XML hierarchy, values are defined by their position relative to the “key” elements. The following fragment illustrates this:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;key&gt;</span>MinimumOSVersion<span class="nt">&lt;/key&gt;</span> <span class="nt">&lt;string&gt;</span>7.0<span class="nt">&lt;/string&gt;</span> <span class="nt">&lt;key&gt;</span>UIDeviceFamily<span class="nt">&lt;/key&gt;</span> <span class="nt">&lt;array&gt;</span> <span class="nt">&lt;integer&gt;</span>1<span class="nt">&lt;/integer&gt;</span> <span class="nt">&lt;integer&gt;</span>2<span class="nt">&lt;/integer&gt;</span> <span class="nt">&lt;/array&gt;</span> </code></pre></div></div> <p>In this example, the value of <em>MinimumOSVersion</em> is defined by the <em>string</em> element that directly follows the <em>key</em> element; likewise, the value of <em>UIDeviceFamily</em> is defined by the <em>array</em> element. This unusual layout means that simple parsing of these files with an XML library is not enough to interpret them in a meaningful way!</p> <h2 id="extraction-of-information-property-list">Extraction of information property list</h2> <p>I was unable to find any tools that directly extract and process the information property list (similar to what <a href="https://github.com/androguard/androguard">Androguard</a> does for Android packages)<sup id="fnref:12"><a href="#fn:12" class="footnote" rel="footnote" role="doc-noteref">13</a></sup>. However, Python has a built-in <a href="https://docs.python.org/3/library/plistlib.html">plistlib</a> module that is able to read and write property lists in both binary and XML format. I did some tests with it, and at first sight it appears to work well: reading the XML property lists for each of my test apps resulted in a Python dictionary that accurately represented the key-value pairs<sup id="fnref:13"><a href="#fn:13" class="footnote" rel="footnote" role="doc-noteref">14</a></sup>. Using this module, it would be fairly straightforward to write a tool that extracts the property list items directly out of an IPA file, and transform them into a more manageable format.</p> <h2 id="interesting-property-list-elements">Interesting property list elements</h2> <p>As with the Android App Manifest before, I won’t go into a detailed discussion of all the items inside the information property list, but the following ones caught my attention:</p> <ul> <li>The <a href="https://developer.apple.com/library/archive/documentation/DeviceInformation/Reference/iOSDeviceCompatibility/DeviceCompatibilityMatrix/DeviceCompatibilityMatrix.html">UIRequiredDeviceCapabilities</a> key declares “the hardware or specific capabilities” that an app needs in order to run. This <a href="https://developer.apple.com/library/archive/documentation/General/Reference/InfoPlistKeyReference/Articles/iPhoneOSKeys.html#//apple_ref/doc/uid/TP40009252-SW3">includes</a>, among other things, access to networking features, a camera or a microphone.</li> <li>The <a href="https://developer.apple.com/documentation/bundleresources/information_property_list/minimumosversion">MinimumOSVersion</a> key defines the minimum operating system version required for the app to run.</li> </ul> <p>Both are directly relevant for emulation purposes.</p> <h2 id="summary-of-test-results">Summary of test results</h2> <p>As this is quite a lengthy post, here’s a brief summary of the main results of the above tests:</p> <ul> <li>Downloading Android APK packages from the Google Play store is possible, but it does require unofficial third-party tools like <a href="https://github.com/matlink/gplaycli">gplaycli</a>. However, such tools may stop functioning if Google applies changes to its Play Store API. This has already happened previously, leading to various unmaintained tools that no longer work.</li> <li>Access to the Apple App Store appears to be completely restricted from non-Apple devices. Since installing an app on iOS reportedly gets rid of the IPA container, workarounds that use a native iOS device as an intermediate medium most likely won’t be usable for preservation workflows.</li> <li>Out of the three file format identification tools tested, only Apache Tika was able to correctly identify both APK and IPA files. However, Tika’s specific results are solely based on file extension patterns. At the time of writing PRONOM doesn’t cover these formats at all, and any tools that use its database (Siegfried, but also DROID and FIDO) only identify them at the higher container levels (ZIP, JAR). This could be easily remedied by developing PRONOM signatures for both formats<sup id="fnref:16"><a href="#fn:16" class="footnote" rel="footnote" role="doc-noteref">15</a></sup>.</li> <li>Both the APK and IPA formats contain package-level metadata about an app’s technical dependencies, such as the minimal OS version and required hardware.</li> <li>For the APK format a software tool that extracts this information is readily available. For the IPA format no such tool exists, but it could be developed with a limited amount of effort.</li> <li>Based on the cursory look presented here, these package-level metadata appear to be adequate for establishing the emulated environment needed to run an app, but they do not expose any dependencies on remotely hosted content<sup id="fnref:15"><a href="#fn:15" class="footnote" rel="footnote" role="doc-noteref">16</a></sup>.</li> </ul> <h2 id="value-of-documentation">Value of documentation</h2> <p>Pennock, May and Day propose the use of “alternative solutions such as recording or documentation” in case the end access solution (e.g. the emulator) does not provide a sufficiently ‘authentic’ experience. In addition to this, Trevor Owens makes an argument for storing screenshots and video recordings of software in his book “The theory and craft of digital preservation”<sup id="fnref:18"><a href="#fn:18" class="footnote" rel="footnote" role="doc-noteref">17</a></sup>:</p> <blockquote> <p>[M]oving to an approach to virtualize or emulate old systems on new hardware will inevitably be a complex process and having even a reference image of what it looked like at a particular moment in time would likely be valuable as a way to evaluate the extent to which it is being authentically rendered. Here a general principle emerges. Documentation (like the screenshot) can be useful as both a means to preserve significance and also as a means to create reference material to triangulate an object’s or work’s significance.</p> </blockquote> <p>For both the ARize and Immer apps from my previous blog post, Youtube channels exist with videos that demonstrate how these apps work<sup id="fnref:19"><a href="#fn:19" class="footnote" rel="footnote" role="doc-noteref">18</a></sup>. I don’t know how common this situation is for other apps, but such videos could serve as a reference for (future) emulation efforts, and it might be worthwhile to collect them as part of the acquisition process, and store them alongside the app packages.</p> <h2 id="final-thoughts">Final thoughts</h2> <p>This post is only a first (and at this stage incomplete) attempt at piecing together candidate components for acquisition and (pre-)ingest workflows for Android and iOS apps. Please feel free to use the comment section in case you have any corrections or additions.</p> <h2 id="further-resources">Further resources</h2> <ul> <li><a href="https://zenodo.org/record/3460450">Considerations on the Acquisition and Preservation of Mobile eBook Apps</a></li> <li><a href="https://github.com/matlink/gplaycli">gplaycli</a></li> <li><a href="https://github.com/androguard/androguard">Androguard</a></li> <li><a href="https://web.archive.org/web/20200714200020/https://blog.razb.me/pulling-apart-an-ios-app/">Pulling apart an iOS App</a></li> <li><a href="https://developer.android.com/guide/topics/manifest/manifest-intro">Android App Manifest documentation</a></li> <li><a href="https://developer.apple.com/library/archive/documentation/General/Reference/InfoPlistKeyReference/Introduction/Introduction.html">Information Property List Key Reference</a></li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:14"> <p>I’m well aware that the possibilities for emulating iOS-based devices are still very limited (for both technical and legal reasons), but that may be the subject of another post. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:1"> <p>See my <a href="/2021/02/09/four-android-emulators-two-apps">previous post on Android emulation options</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>Note that the current (3.29) version of the tool has a small bug that results in a warning, which is <a href="https://github.com/matlink/gplaycli/issues/272">documented here</a>. Other than that it does work as expected. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>I used the <a href="https://linux.die.net/man/1/cmp">cmp tool</a> for this, with the command <code class="language-plaintext highlighter-rouge">cmp base.apk com.Triplee.TripleeSocial.apk</code>. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:7"> <p>DROID Container Signature File 20201001.xml <a href="#fnref:7" class="reversefootnote" role="doc-backlink">&#8617;</a> <a href="#fnref:7:1" class="reversefootnote" role="doc-backlink">&#8617;<sup>2</sup></a></p> </li> <li id="fn:6"> <p>PRONOM version at the time of writing: DROID_SignatureFile_V97.xml, 1st October 2020. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>Tried with Android Studio 4.1.2, running under Linux Mint 19.3. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>See also the “Android Emulator for long-term access” section in my <a href="/2021/02/09/four-android-emulators-two-apps">previous blog post</a>. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:10"> <p>For example using a service like <a href="https://corellium.com/">Corellium</a>. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:8"> <p>See e.g. <a href="https://stackoverflow.com/a/29743193/1209004">here</a>, <a href="https://www.reddit.com/r/jailbreak/comments/4dhbtb/question_ipa_location_in_ios_9/">here</a> and <a href="https://medium.com/@lucideus/extracting-the-ipa-file-and-local-data-storage-of-an-ios-application-be637745624d">here</a>. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:9"> <p>I used the <a href="https://iosninja.io/ipa-library">ioninja.io</a> site. I have no idea about the site’s legal status or the safety of the downloads on offer, so proceed with caution! I only used the downloaded IPAs for some simple technical tests without installing them. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:11"> <p>This is typically the <code class="language-plaintext highlighter-rouge">Payload/Application.app</code> folder (where “Application” is replaced with the app’s name). <a href="#fnref:11" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:12"> <p>There is <a href="https://github.com/matiassingers/ipa-metadata">Ipa-metadata</a>, which is a tool for extracting “metadata and provisioning info about an .ipa file”. Although I was able to install it, running it on any of my test files would just return a “Callback must be a function” error, and nothing else. <a href="#fnref:12" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:13"> <p>The demo script that I wrote for my tests is <a href="https://github.com/KBNLresearch/mobile-apps/blob/main/scripts/readplist.py">available here</a>. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:16"> <p>I’d be happy to have a go at this myself. <a href="#fnref:16" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:15"> <p>This needs further confirmation from a more in-depth look at the available documentation. <a href="#fnref:15" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:18"> <p>The theory and craft of digital preservation. Johns Hopkins University Press, 2018. Link: <a href="http://www.trevorowens.org/theory-and-craft-of-digital-preservation/">http://www.trevorowens.org/theory-and-craft-of-digital-preservation/</a> <a href="#fnref:18" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:19"> <p>ARize app Youtube channel: <a href="https://www.youtube.com/channel/UCPEDDkRVC7jegjC02-7VsUw/videos">https://www.youtube.com/channel/UCPEDDkRVC7jegjC02-7VsUw/videos</a>; Immer app Youtube channel: <a href="https://www.youtube.com/channel/UCnrnDrJ5MXccQJxaicEi7cA">https://www.youtube.com/channel/UCnrnDrJ5MXccQJxaicEi7cA</a>. <a href="#fnref:19" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2021/02/24/towards-a-preservation-workflow-for-mobile-apps https://bitsgalore.org/2021/02/24/towards-a-preservation-workflow-for-mobile-apps 2021-02-24T00:00:00+01:00 <figure class="image"> <img src="/images/2021/02/android-header.png" alt="Header image" /> <figcaption><a href="https://commons.wikimedia.org/wiki/File:Android_robot.svg">"Android Robot"</a> by Google Inc., used under <a href="https://creativecommons.org/licenses/by/3.0">CC BY 3.0</a>, via Wikimedia Commons.</figcaption> </figure> <p>So far the KB hasn’t actively pursued the preservation of mobile apps. However, born-digital publications in app-only form have become increasingly common, as well as “hybrid” publications, with apps that are supplemental to traditional (paper) books. At the request of our Digital Preservation department, I’ve started some exploratory investigations into how to preserve mobile apps in the near future. The <a href="https://zenodo.org/record/3460450">2019 iPres paper on the Acquisition and Preservation of Mobile eBook Apps</a> by the British Library’s Maureen Pennock, Peter May and Michael Day provides an excellent starting point on the subject, and it highlights many of the challenges involved.</p> <p>Before we can start archiving mobile apps ourselves, some additional aspects need to be addressed in more detail. One of these is the question of how to ensure long-term access. <a href="https://en.wikipedia.org/wiki/Emulator">Emulation</a> is the obvious strategy here, but I couldn’t find much information on the emulation of mobile platforms within a digital preservation context. In this blog post I present the results of some simple experiments, where I tried to emulate two selected apps. The main objective here was to explore the current state of emulation of mobile devices, and to get an initial impression of the suitability of some existing emulation solutions for long-term access.</p> <p>For practical reasons I’ve limited myself to the Android platform<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. Attentive readers may recall I <a href="/2014/10/23/running-archived-android-apps-pc-first-impressions">briefly touched on this subject back in 2014</a>. As much of the information in that blog post has now become outdated, this new post presents a more up-to date investigation. I should probably mention here that I don’t own or use any Android device, or any other kind of smartphone or tablet for that matter<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>. This probably makes me the worst possible person to evaluate Android emulation, but who’s going to stop me trying anyway? No one, that’s who!</p> <!-- more --> <h2 id="android-emulation-options">Android emulation options</h2> <p>The Emulation General Wiki gives <a href="https://emulation.gametechwiki.com/index.php/Android_emulators">a good overview of Android emulators</a>. Most of these are closed-source, and the Wiki warns that some may come with malicious apps pre-installed. For the purposes of long-term preservation and acccess, open-source solutions are much more relevant. The Wiki lists the following open-source emulators:</p> <ul> <li><a href="https://www.android-x86.org/">Android-x86</a> is a port of the <a href="https://source.android.com/">Android open source project</a> to the <a href="https://en.wikipedia.org/wiki/X86">x86</a> architecture. It is not an emulator, but rather an operating system that can be installed on either a physical device, or within a virtual machine (e.g. using VirtualBox or QEMU).</li> <li><a href="https://developer.android.com/studio/run/emulator">Android Emulator</a> is part of <a href="https://developer.android.com/studio/">Android Studio</a>, Google’s offial development environment for Android. Android Emulator “simulates Android devices on your computer so that you can test your application on a variety of devices and Android API levels without needing to have each physical device”.</li> <li><a href="https://github.com/anbox/anbox">Anbox</a> is “a container-based approach to boot a full Android system on a regular GNU/Linux system like Ubuntu”.</li> <li><a href="http://www.shashlik.io/">Shashlik</a> is another project for running Android apps on Linux.</li> </ul> <p>It’s worth pointing out that most of these solutions aren’t really “emulators” in a strict sense. Android-x86 is an operating system can be run inside a virtual machine (without the need for real hardware emulation) on x86-based platforms. Android Emulator can do full <a href="https://en.wikipedia.org/wiki/ARM_architecture">ARM hardware</a> emulation, but is usually run with x86 system images. Anbox is not an emulator at all, but rather a <a href="https://en.wikipedia.org/wiki/Compatibility_layer">compatibility layer</a>, similar to how <a href="https://www.winehq.org/">WINE</a> allows one to run Windows applications on Unix-like operating systems. Shashlik uses a hybrid approach, by pairing a stripped-down Android base that is run within a modified version of QEMU with graphics rendering on the host machine<sup id="fnref:14"><a href="#fn:14" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>. For the sake of simplicity, I will use the term “emulation” for all of the above in this post.</p> <h2 id="test-setup">Test setup</h2> <p>The experiments described in the remainder of this post focus on the emulation of two selected test apps:</p> <ol> <li><a href="https://arize.io/">ARize</a> is an “augmented reality” app. It is used, among other things, to add 3-D visualizations and animations to the Dutch-language children’s book “De avonturen van Max - Op zoek naar F…”.<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>.</li> <li><a href="https://immer.app/">Immer</a> is a book reading app that claims to “help[ing] you read more often, easily and enjoyably on your phone or tablet”.</li> </ol> <p>I tried to emulate these apps using the following emulated environments:</p> <ol> <li>Android-x86, running in VirtualBox</li> <li>Android-86, running in QEMU</li> <li>Anbox</li> <li>Android Emulator (Android Studio)</li> </ol> <p>I did not include Shashlik, as this project has been <a href="https://github.com/shashlik">inactive since 2016</a>. I evaluated each of the emulations by the following (admittedly crude and incomplete) criteria:</p> <ul> <li>Does the emulated environment work at all?</li> <li>Is it possible to install the ARize and Immer apps?</li> <li>Do the installed apps work?</li> </ul> <p>I ran all tests on a regular desktop PC with the following characteristics:</p> <ul> <li>Quad-core <a href="https://ark.intel.com/content/www/us/en/ark/compare.html?productIds=88184">Intel Core i5-6500 processor</a></li> <li>12 GB RAM</li> <li>Operating system: Linux Mint 19.3 (Tricia)</li> </ul> <p>Below follows a more detailed discussion of each of the emulated environments. I deliberately included quite a lot of detail here, mostly to make life easier for others who may want to do similar tests themselves. If you’re only interested in the main findings, you may want to skip these details and head right over to the “Summary and conclusions” section.</p> <h2 id="android-x86--virtualbox">Android-x86 + VirtualBox</h2> <h3 id="setup">Setup</h3> <p><a href="https://www.sjoerdlangkemper.nl/2020/05/06/testing-android-apps-on-a-virtual-machine/">This blog post by Sjoerd Langkemper</a> gives a good overview of how to set up a virtual machine running Android-x86 with VirtualBox, and I largely followed the instructions mentioned here. I used VirtualBox verion 6.0.24, r139119. I had some difficulty getting a functional virtual machine from the latest Android-86 ISO installer image, so in the end I settled for the pre-built VirtualBox image from <a href="https://www.osboxes.org/android-x86/">osboxes.org</a> (Android-x86 9.0-R2, 64-bit). Initially the virtual machine got stuck on a black screen at startup. I was able to fix this by setting the graphics controller (which can be found under “Display” in the VM settings) to “VBoxSVGA”. I also set the number of processors (“System” settings, “Processor” tab) to 2, as according to the <a href="https://www.android-x86.org/documentation/virtualbox.html">Android-86 documentation</a>:</p> <blockquote> <p>Processor(s) should be set above 1 if you have more than one virtual processor in your host system. Failure to do so means every single app (like Google Chrome) might crush if you try to use it.</p> </blockquote> <p>Here’s what the virtual machine looks like after it has booted:</p> <figure class="image"> <img src="/images/2021/02/vbox_android_startup.png" alt="Home app, Android-x86 on VirtualBox." /> <figcaption>Home app, Android-x86 on VirtualBox.</figcaption> </figure> <p>At a first glance, everything pretty much works, although I did run into a number of crashes of the Chrome browser app. These all occurred while entering text into the address bar. Strangely, I couldn’t replicate these crashes in a later session, so I’m not sure about the exact cause.</p> <h3 id="app-installation">App installation</h3> <p>To install apps on Android, one would normally use the <a href="https://play.google.com/store/apps">Google Play Store</a>. Within a typical preservation workflow, you’re more likely to have a local copy of the app’s <a href="https://en.wikipedia.org/wiki/Android_application_package">Android Package (APK)</a>. Because of this, I tried to “emulate” this by using locally downloaded APKs for my experiments. To achieve this, I first used the <a href="https://github.com/matlink/gplaycli">gplaycli</a> tool to download APK files of the test apps to my local (Linux) machine. Installing locally downloaded APKs on an emulated Android machine can be a bit tricky, as VirtualBox (and most other Android emulators) provides no easy way to set up shared folders between the host machine and the emulated device. The easiest method (which works for <em>all</em> of the environments covered by this post) uses the <a href="https://developer.android.com/studio/command-line/adb">Android Debug Bridge (adb)</a>, which is part of <a href="https://developer.android.com/studio/">Android Studio</a>. <a href="https://www.sjoerdlangkemper.nl/2020/05/06/testing-android-apps-on-a-virtual-machine/">Langkemper’s blog post</a> covers the use of the <em>adb</em> tool in detail, so for brevity I’ll only show the basic steps here.</p> <p>First, we need to find the IP address of our virtual Android machine, which in my particular case turned out to be “127.0.0.1” (localhost)<sup id="fnref:6"><a href="#fn:6" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>. Then we can use the Android Debug Bridge tool to connect to the virtual machine:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>adb connect 127.0.0.1 </code></pre></div></div> <p>If all goes well, you should see this:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>connected to 127.0.0.1:5555 </code></pre></div></div> <p>If you get a “Connection refused” error instead, you might need to set a port forwarding rule for the virtual machine. In VirtualBox, go to “Network” in your VM’s settings. Click on “Advanced”, followed by “Port Forwarding”. Here, click on the green “+” icon (top-right). This adds a port forwarding rule. Now change the values of both “Host Port” and “Guest Port” to 5555 (defaults for both are 0):</p> <figure class="image"> <img src="/images/2021/02/vbox_ptforward.png" alt="Setting port forwarding rules in VirtualBox." /> <figcaption>Setting port forwarding rules in VirtualBox.</figcaption> </figure> <p>Then try to connect again. Once the connection is established, you can install the local APK file with <em>adb</em> using its “install” subcommand, with the name of the package file as an argument:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>adb <span class="nb">install </span>com.Triplee.TripleeSocial.apk </code></pre></div></div> <h3 id="results-for-test-apps">Results for test apps</h3> <p>I was able to install the test apps without any problems, and their launcher icons showed up promptly after the installation:</p> <figure class="image"> <img src="/images/2021/02/vbox_android_apps.png" alt="App launchers after installation (note ARize and Immer icons)." /> <figcaption>App launchers after installation (note ARize and Immer icons).</figcaption> </figure> <p>However, the ARize app crashed immediately after being launched:</p> <figure class="image"> <img src="/images/2021/02/vbox_android_arize.png" alt="ARize crash message after repeated launch attempts." /> <figcaption>ARize crash message after repeated launch attempts.</figcaption> </figure> <p>By contrast, the Immer app worked without any problems. Below are some screenshots that show Immer in action:</p> <figure class="image"> <img src="/images/2021/02/vbox_android_immer.png" alt="Immer welcome screen (Android-x86 + VirtualBox)." /> <figcaption>Immer welcome screen (Android-x86 + VirtualBox).</figcaption> </figure> <figure class="image"> <img src="/images/2021/02/vbox_android_immer_2.png" alt="Immer book selection screen (Android-x86 + VirtualBox)." /> <figcaption>Immer book selection screen (Android-x86 + VirtualBox).</figcaption> </figure> <figure class="image"> <img src="/images/2021/02/vbox_android_immer_3.png" alt="Immer book reading interface (Android-x86 + VirtualBox)." /> <figcaption>Immer book reading interface (Android-x86 + VirtualBox).</figcaption> </figure> <h2 id="android-x86--qemu">Android-x86 + QEMU</h2> <h3 id="setup-1">Setup</h3> <p>Since the Debian packages of QEMU for the Linux Mint version I’m using are way out of date, I first compiled and installed the most recent (5.2) QEMU release from its source, using the instructions <a href="https://www.qemu.org/download/#source">here</a><sup id="fnref:7"><a href="#fn:7" class="footnote" rel="footnote" role="doc-noteref">6</a></sup>. For setting up Android-x86 within QEMU, I largely followed <a href="https://linuxhint.com/android_qemu_play_3d_games_linux/">this guide by Nitesh Kumar</a> (starting from the <em>Android-x86 QEMU Installation Walkthrough</em> section). However, these instructions initially failed for me, and I could trace this back to problems with QEMU’s OpenGL (which is related to graphics rendering) support<sup id="fnref:8"><a href="#fn:8" class="footnote" rel="footnote" role="doc-noteref">7</a></sup>. After some experimentation, I managed to make it work by removing all OpenGL-related command line arguments. I’ll briefly summarize the setup steps here:</p> <ol> <li>Download the Android-x86 ISO installer image from <a href="https://osdn.net/projects/android-x86/releases">here</a> (I picked the 64-bit 9.0 R2 release).</li> <li>Create a virtual hard disk (size: 8 GB): <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>qemu-img create <span class="nt">-f</span> qcow2 androidx86_9_hda.img 8G </code></pre></div> </div> </li> <li>Boot the Android-x86 live ISO image inside a virtual machine, attaching also the virtual hard disk: <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>qemu-system-x86_64 <span class="se">\</span> <span class="nt">-enable-kvm</span> <span class="se">\</span> <span class="nt">-m</span> 2048 <span class="se">\</span> <span class="nt">-smp</span> 2 <span class="se">\</span> <span class="nt">-cpu</span> host <span class="se">\</span> <span class="nt">-device</span> ES1370 <span class="nt">-device</span> virtio-mouse-pci <span class="nt">-device</span> virtio-keyboard-pci <span class="se">\</span> <span class="nt">-serial</span> mon:stdio <span class="se">\</span> <span class="nt">-boot</span> <span class="nv">menu</span><span class="o">=</span>on <span class="se">\</span> <span class="nt">-net</span> nic <span class="se">\</span> <span class="nt">-net</span> user,hostfwd<span class="o">=</span>tcp::4444-:5555 <span class="se">\</span> <span class="nt">-hda</span> androidx86_9_hda.img <span class="se">\</span> <span class="nt">-cdrom</span> android-x86_64-9.0-r2.iso </code></pre></div> </div> </li> <li>Follow <a href="https://linuxhint.com/android_qemu_play_3d_games_linux/">Kumar’s guide</a> (starting from the first screenshot) to install Android on the virtual machine.</li> <li>After the installation process is completed, close down the virtual machine (you can do this by simply closing the QEMU window), and then re-start it using (same command as before, but omitting the <code class="language-plaintext highlighter-rouge">-cdrom</code> argument): <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>qemu-system-x86_64 <span class="se">\</span> <span class="nt">-enable-kvm</span> <span class="se">\</span> <span class="nt">-m</span> 2048 <span class="se">\</span> <span class="nt">-smp</span> 2 <span class="se">\</span> <span class="nt">-cpu</span> host <span class="se">\</span> <span class="nt">-device</span> ES1370 <span class="nt">-device</span> virtio-mouse-pci <span class="nt">-device</span> virtio-keyboard-pci <span class="se">\</span> <span class="nt">-serial</span> mon:stdio <span class="se">\</span> <span class="nt">-boot</span> <span class="nv">menu</span><span class="o">=</span>on <span class="se">\</span> <span class="nt">-net</span> nic <span class="se">\</span> <span class="nt">-net</span> user,hostfwd<span class="o">=</span>tcp::4444-:5555 <span class="se">\</span> <span class="nt">-hda</span> androidx86_9_hda.img </code></pre></div> </div> </li> </ol> <p>And here’s what this looks like after start-up::</p> <figure class="image"> <img src="/images/2021/02/qemu_android_startup.png" alt="Home app, Android-x86 on QEMU." /> <figcaption>Home app, Android-x86 on QEMU.</figcaption> </figure> <p>One oddity is that the rendering of colours doesn’t look quite right, with reds shown as shades of blue (this is even more apparent when you open some web pages with the Chrome app). Perhaps this could be remedied by using better device settings in QEMU, but I haven’t looked into this any further.</p> <h3 id="app-installation-1">App installation</h3> <p>Because of the slightly different network configuration, I had to add a reference to the <code class="language-plaintext highlighter-rouge">4444</code> network port make to make the <em>adb</em> connection to the QEMU machine:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>adb connect 127.0.0.1:4444 </code></pre></div></div> <p>After this, the package install procedure is identical to the one I showed for VirtualBox<sup id="fnref:9"><a href="#fn:9" class="footnote" rel="footnote" role="doc-noteref">8</a></sup>.</p> <h3 id="results-for-test-apps-1">Results for test apps</h3> <p>I was able to install both test apps without problems on the QEMU machine. As with VirtualBox, the ARize app consistently crashed after launch. The Immer app worked without any issues.</p> <h2 id="anbox">Anbox</h2> <h3 id="setup-2">Setup</h3> <p>I installed Anbox (version 4-56c25f1) by following the <a href="https://github.com/anbox/anbox/blob/master/docs/install.md">official Anbox documentation</a>. After the installation, fire up the “Anbox Application Manager” from (depending on your Linux desktop) the desktop menu or launch bar<sup id="fnref:10"><a href="#fn:10" class="footnote" rel="footnote" role="doc-noteref">9</a></sup>:</p> <figure class="image"> <img src="/images/2021/02/anbox_launch.png" alt="Anbox launcher." /> <figcaption>Anbox launcher in Linux Mint desktop menu.</figcaption> </figure> <p>Unlike the other platforms covered in this post, Anbox doesn’t try to “emulate” a single device, but rather provides a compatibility layer that allows you to run Android apps from the Application Manager. Each app is launched in its own window, as shown below:</p> <figure class="image"> <img src="/images/2021/02/anbox_desktop.png" alt="Anbox Application Manager with calculator, file manager and clock apps." /> <figcaption>Anbox Application Manager with calculator, file manager and clock apps.</figcaption> </figure> <p>Looking at the information under Settings, the current Anbox release is based on Android 7.1.1, which is quite an old version. A quick test with the pre-installed Webview Browser showed Anbox couldn’t connect to the internet, which apparently is a <a href="https://github.com/anbox/anbox/issues/1724">known issue</a>. After some searching I found <a href="https://wiki.archlinux.org/index.php/Anbox#Via_NetworkManager">a workaround here</a>. I simply ran the following command:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nmcli con add <span class="nb">type </span>bridge ifname anbox0 <span class="nt">--</span> connection.id <span class="se">\</span> anbox-net ipv4.method shared ipv4.addresses 192.168.250.1/24 </code></pre></div></div> <p>I then closed and re-started the Anbox Application Manager, after which internet connectivity was working properly.</p> <h3 id="app-installation-2">App installation</h3> <p>The Anbox Application Manager automatically launches an ADB server process, so you don’t need to manually run <code class="language-plaintext highlighter-rouge">adb connect</code>. Other than that, you can use the regular <code class="language-plaintext highlighter-rouge">adb install</code> commands to install the APK files.</p> <h3 id="results-for-test-apps-2">Results for test apps</h3> <p>My attempt to install the ARize app failed with this error message:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>adb: failed to install com.Triplee.TripleeSocial.apk: Failure [INSTALL_FAILED_NO_MATCHING_ABIS: Failed to extract native libraries, res=-113] </code></pre></div></div> <p>According to <a href="https://stackoverflow.com/a/24572239">this StackOverflow answer</a>, this error can occur if an app uses native (e.g. ARM) libraries that are not compatible with the architecture of the (virtual) destination machine.</p> <p>The Immer app installed without any problems, and I was also able to launch it. However, the text in the app is partially rendered outside the app window:</p> <figure class="image"> <img src="/images/2021/02/anbox_immer.png" alt="Immer welcome screen, Anbox." /> <figcaption>Immer welcome screen, Anbox.</figcaption> </figure> <p>After I tried to re-size or maximize the app window, all text disappeared altogether. I was also unable to get the core book selection and reading functionality working (I just ended up with an empty screen), although clicking on the icon at the bottom did allow me to open and edit the app’s user profile.</p> <h2 id="android-emulator-android-studio">Android Emulator (Android Studio)</h2> <h3 id="setup-3">Setup</h3> <p>I installed Android Studio 4.1.2, which includes version 30.3.5.0 of Android Emulator. The emulator can be accessed from the “Tools” menu in the main Android Studio application. Here, the “AVD Manager” allows you to set up one or more <a href="https://developer.android.com/studio/run/managing-avds">Android Virtual Devices</a>, each based on a user-specified “hardware profile, system image, storage area, skin, and other properties”. The setup process uses a wizard-like interface, which is pretty straightforward to use:</p> <figure class="image"> <img src="/images/2021/02/android_emulator_avd.png" alt="First step of Android Virtual Device setup process." /> <figcaption>First step of Android Virtual Device setup process.</figcaption> </figure> <p>Nevertheless, it offers many customization options that allow you to mimic very specific device configurations. Both x86 and ARM system images are available for a variety of Android versions. The documentation advises against full ARM emulation because of the better performance of the x86 images. According the <a href="https://developer.android.com/studio/releases/emulator#support_for_arm_binaries_on_android_9_and_11_system_images">Emulator 30.0.0 release notes</a>:</p> <blockquote> <p>If you were previously unable to use the Android Emulator because your app depended on ARM binaries, you can now use the Android 9 x86 system image or any Android 11 system image to run your app – it is no longer necessary to download a specific system image to run ARM binaries. These Android 9 and Android 11 system images support ARM by default and provide dramatically improved performance when compared to those with full ARM emulation.</p> </blockquote> <p>Because of this recommendation, I set up a device with Android 9 (for better comparison with the Android-86 tests) using the x86 image. The emulator can then be launched from either Android Studio, or <a href="https://developer.android.com/studio/run/emulator-commandline">using the command line</a>. On startup, the emulated device looks like this:</p> <figure class="image"> <img src="/images/2021/02/android_emulator_startup.png" alt="Home app, Android 9 on Android Emulator." /> <figcaption>Home app, Android 9 on Android Emulator.</figcaption> </figure> <p>Unlike the previous Android-x86 emulations, which are based on the Android Open Source Project, Android Emulator uses the official system images by Google. As a result, the “look and feel” of the Android Emulator virtual devices is quite different, and they also come with a larger number of pre-installed apps. Because of this, Android Emulator most likely approximates the Android experience on a physical device more closely<sup id="fnref:11"><a href="#fn:11" class="footnote" rel="footnote" role="doc-noteref">10</a></sup>.</p> <h3 id="app-installation-3">App installation</h3> <p>Like Anbox, Android Emulator automatically launches an ADB server process, so there’s no need to manually run <code class="language-plaintext highlighter-rouge">adb connect</code>. Installing APK files involves the usual <code class="language-plaintext highlighter-rouge">adb install</code> commands.</p> <h3 id="results-for-test-apps-3">Results for test apps</h3> <p>Both the ARize and Immer apps installed without any problems. Unlike all other emulators in this test, I was able to run the ARize app, although with some limitations. The screenshot below shows the app’s “gallery” screen:</p> <figure class="image"> <img src="/images/2021/02/arize_gallery.png" alt="ARize app gallery." /> <figcaption>ARize app gallery.</figcaption> </figure> <p>Clicking on an item in the gallery opens up a 3-D model, that can be manipulated by the user. As an example, here’s a model of a necklace:</p> <figure class="image"> <img src="/images/2021/02/arize_necklace.png" alt="3-D visualization of necklace in ARize app." /> <figcaption>3-D visualization of necklace in ARize app.</figcaption> </figure> <p>If I understand it correctly, clicking the “View AR” button should switch the app to “augmented reality” mode, where the 3-D model is combined with video from the phone’s camera. Although Android Emulator allows one to attach external cameras (in my case a webcam), this didn’t quite work for me, with the emulator reporting a warning<sup id="fnref:13"><a href="#fn:13" class="footnote" rel="footnote" role="doc-noteref">11</a></sup>. Whenever I tried to use the camera, the app showed a blank screen, after which the emulated device became unresponsive.</p> <p>The Immer app worked without any issues, as shown in the screenshot below:</p> <figure class="image"> <img src="/images/2021/02/android_emulator_immer.png" alt="Immer book reading interface (Android Emulator)." /> <figcaption>Immer book reading interface (Android Emulator).</figcaption> </figure> <p>Although far from perfect, these Android Emulator results still look promising, bearing in mind that none of the other tested emulators could even run the ARize app at all.</p> <h2 id="summary-and-discussion-of-results">Summary and discussion of results</h2> <p>The table below summarizes the main results of the emulation tests:</p> <table> <thead> <tr> <th style="text-align: left"> </th> <th style="text-align: left">Android-x86, VirtualBox</th> <th style="text-align: left">Android-x86, QEMU</th> <th style="text-align: left">Anbox</th> <th style="text-align: left">Android Studio</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><strong>Emulator version</strong></td> <td style="text-align: left">6.0.24, r139119</td> <td style="text-align: left">5.2</td> <td style="text-align: left">4-56c25f1</td> <td style="text-align: left">30.3.5.0</td> </tr> <tr> <td style="text-align: left"><strong>Android version</strong></td> <td style="text-align: left">Android-x86 9.0-R2 (64-bit)</td> <td style="text-align: left">Android-x86 9.0-R2 (64-bit)</td> <td style="text-align: left">Customized system image based on v. 7.1.1 of Android Open Source Project</td> <td style="text-align: left">Android 9.0 x86 system image (Google), API level 28</td> </tr> <tr> <td style="text-align: left"><strong>Emulation approach</strong></td> <td style="text-align: left">Virtualization</td> <td style="text-align: left">Virtualization</td> <td style="text-align: left">Compatibility layer</td> <td style="text-align: left">Virtualization (full ARM emulation optional, depending on system image)</td> </tr> <tr> <td style="text-align: left"><strong>ARize app installs</strong></td> <td style="text-align: left">Yes</td> <td style="text-align: left">Yes</td> <td style="text-align: left">No</td> <td style="text-align: left">Yes</td> </tr> <tr> <td style="text-align: left"><strong>ARize app works</strong></td> <td style="text-align: left">No</td> <td style="text-align: left">No</td> <td style="text-align: left">-</td> <td style="text-align: left">Partially (camera device not recognised; emulator unresponsive after using camera)</td> </tr> <tr> <td style="text-align: left"><strong>Immer app installs</strong></td> <td style="text-align: left">Yes</td> <td style="text-align: left">Yes</td> <td style="text-align: left">Yes</td> <td style="text-align: left">Yes</td> </tr> <tr> <td style="text-align: left"><strong>Immer app works</strong></td> <td style="text-align: left">Yes</td> <td style="text-align: left">Yes</td> <td style="text-align: left">Partially (rendering and navigation issues)</td> <td style="text-align: left">Yes</td> </tr> </tbody> </table> <p>It is important to stress that the tests presented here are limited in scope and size, and should not be interpreted as representative of Android apps in general. With that in mind, it is possible to draw some tentative conclusions.</p> <h3 id="android-x86-limitations">Android-x86 limitations</h3> <p>First of all, the results suggest that emulation approaches based on Android-x86 may have some serious limitations. Going by various reports I found on sites like StackOverflow, the ARize app crashing on startup might be indicative of a more widespread problem. Sjoerd Langkemper mentions in his <a href="https://www.sjoerdlangkemper.nl/2020/05/06/testing-android-apps-on-a-virtual-machine/">blog post</a> that:</p> <blockquote> <p>Testing on a virtual machine (VM) has some disadvantages. Testing on an actual Android phone is more reliable. Android is meant to run on ARM phones and not on x86 virtual machines, so things may randomly break when using a VM. Apps that ship with native libraries may not run at all in the VM, or they may run perfectly but don’t show up in the Play store.</p> </blockquote> <p>A similar explanation, citing the use of native ARM libraries that are not supported by Android-x86, is given <a href="https://stackoverflow.com/a/60148570">here</a>. I’m not sure this is the culprit here, but if correct, this would seriously limit the usefulness of Android-x86 for long-term access.</p> <h3 id="anbox-1">Anbox</h3> <p>In its current form, Anbox seems of limited value for long-term access. With that said, I quite like its approach to providing access to Android apps, which is very different to the other emulators covered by this post. The project has an active developer community, and I’m curious how it will will develop in the future.</p> <h3 id="android-emulator-for-long-term-access">Android Emulator for long-term access</h3> <p>By contrast, Android Emulator (from Android Studio) could be a very interesting solution for emulating Android apps. It is the only emulator that was able to run both test apps (although with some issues in case of the ARize app). It also has an overall look and feel that is more faithful to a physical Android device. It’s worth pointing out here that under the hood, Android Emulator uses (a modified version of) QEMU<sup id="fnref:12"><a href="#fn:12" class="footnote" rel="footnote" role="doc-noteref">12</a></sup>. Right now I’m unable to judge whether Android Emulator would be truly suitable as a solution for long-term access. Some concerns:</p> <ul> <li> <p>Google has developed it for the sole purpose of allowing Android developers to test their apps on a variety of (virtual) devices. It’s unlikely that Google will keep developing or maintaining it beyond Android’s lifetime. For long-term access, this implies that some organization should take over the maintenance of (a fork of) the software from that point onwards (assuming this is allowed under its licensing conditions, but see below).</p> </li> <li> <p>The <a href="https://developer.android.com/studio/terms.html">Terms and conditions</a> of the Android Software Development Kit (of which Android Emulator is a part) state that:</p> <blockquote> <p>3.1 Subject to the terms of the License Agreement, Google grants you a limited, worldwide, royalty-free, non-assignable, non-exclusive, and non-sublicensable license to use the SDK solely to develop applications for compatible implementations of Android.</p> </blockquote> <p>and:</p> <blockquote> <p>3.2 You may not use this SDK to develop applications for other platforms (including non-compatible implementations of Android) or to develop another SDK. You are of course free to develop applications for other platforms, including non-compatible implementations of Android, provided that this SDK is not used for that purpose.</p> </blockquote> <p>To my legally untrained eyes, this appears to rule out the use of any of the components of the Android SDK for preservation and long-term access.</p> </li> <li> <p>Also, Android Studio’s licensing information mentions that it includes “proprietary code subject to [a] separate license”. It’s not clear to me if this affects the Emulator component. The emulator’s subdirectory contains an additional +3000-line file with licensing information that applies specifically to the emulator component. I haven’t gone through it in detail (and am not planning to do so), but the licensing situation does look somewhat complex.</p> </li> </ul> <p>I can’t really assess to what extent this may impact the (future) use of Android Emulator as a long-term access solution, but I’d be interested in the opinion of any legal experts who may be reading this.</p> <h3 id="external-dependencies">External dependencies</h3> <p>On a final note, it’s important to stress that the ability to run an app in an emulated environment is only one part of the preservation puzzle, and by itself it doesn’t guarantee its accessibility over time. As Pennock, May &amp; Day write:</p> <blockquote> <p>If the app is to be acquired in the most robust and complete form possible then we must find some way to deal with apps which have an inherent reliance on content hosted externally to the app. These are likely to lose their integrity over time, particularly as linkage to archived web content does not yet (if at all) appear to have become standard practice in apps.</p> </blockquote> <p>This also applies to both the ARize and Immer apps, both of which rely on externally hosted content. To illustrate this, after disabling the internet connection on my PC, the ARize app showed this on start-up:</p> <figure class="image"> <img src="/images/2021/02/arize_noconnection.png" alt="Startup screen of ARize app after disabling internet connection." /> <figcaption>Startup screen of ARize app after disabling internet connection.</figcaption> </figure> <p>Immer simply started up with a blank screen. So, without (access to) the externally hosted resources, both apps are essentially useless.</p> <h2 id="further-resources">Further resources</h2> <ul> <li> <p><a href="https://zenodo.org/record/3460450">Considerations on the Acquisition and Preservation of Mobile eBook Apps</a></p> </li> <li> <p><a href="https://www.sjoerdlangkemper.nl/2020/05/06/testing-android-apps-on-a-virtual-machine/">Testing Android apps on a virtual machine</a></p> </li> <li> <p><a href="https://linuxhint.com/android_qemu_play_3d_games_linux/">How to Run Android in QEMU to Play 3D Android Games on Linux</a></p> </li> <li> <p><a href="https://astr0baby.wordpress.com/2019/07/09/android-8-1-in-qemu-and-burp-suite-ssl-interception/">Android 8.1 in qemu and Burp Suite SSL interception</a></p> </li> <li> <p><a href="https://developer.android.com/studio/run/managing-avds">Create and manage virtual devices in Android Emulator</a></p> </li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>The proprietary nature of iOS severely constrains any emulation options; I may address this in a future blog post. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>As a matter of fact I’m still using <a href="https://en.wikipedia.org/wiki/Motorola_C139">this basic dumb phone</a>, which I bough back in 2006. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:14"> <p>See <a href="http://www.shashlik.io/what-is/">http://www.shashlik.io/what-is/</a>. <a href="#fnref:14" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>This instruction video shows how this works <a href="https://youtu.be/h4syCHftyCs">https://youtu.be/h4syCHftyCs</a>. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:6"> <p>Finding the correct IP address can be a bit tricky. Langkemper’s blog suggests to either look at Android’s Wi-Fi preferences, or to run <code class="language-plaintext highlighter-rouge">ip a</code> or <code class="language-plaintext highlighter-rouge">ifconfig</code> in the Android terminal emulator app. However, in my case the value value shown in the Wi-Fi preferences is “10.0.2.15”, which is not recognised by <em>adb</em>. The <code class="language-plaintext highlighter-rouge">ifconfig</code> command reports 3 different entries (“wlan0”, “wifi_eth” and “lo”); eventually I found the value of the “lo” (“local loopback”) entry (“127.0.0.1”) did the trick. So you might need to experiment a bit to make things work. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:7"> <p>Compilation of QEMU requires Python <a href="https://ninja-build.org/">Ninja package</a>, so install this first by running <code class="language-plaintext highlighter-rouge">python3 -m pip install --user ninja</code>. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:8"> <p>E.g. see <a href="https://forums.opensuse.org/showthread.php/539026-Can-t-enable-opengl-on-the-qemu-machine">here</a> and <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1867343">here</a>. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:9"> <p>If you are connected to multiple devices at the same time, you’ll need to add the <code class="language-plaintext highlighter-rouge">-s</code> switch to your <em>adb</em> calls to specify the target device. See <a href="https://developer.android.com/studio/command-line/adb#directingcommands">the documentation</a> for details. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:10"> <p>This might seem obvious, but it’s not really clear from the documentation, so I thought I’d just mention it. <a href="#fnref:10" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:11"> <p>But bear in mind I didn’t have any physical Android devices available while doing these tests. <a href="#fnref:11" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:13"> <p>“Camera name ‘webcam0’ is not found in the list of connected cameras”. <a href="#fnref:13" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:12"> <p>This <a href="https://wiki.diebin.at/Under_the_hood_of_Android_Emulator_(appcert).html">“Under the hood of Android Emulator”</a> Wiki entry shows how QEMU is used within the emulator (note that it hasn’t been updated since 2011, so it may be well out of date). <a href="#fnref:12" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2021/02/09/four-android-emulators-two-apps https://bitsgalore.org/2021/02/09/four-android-emulators-two-apps 2021-02-09T00:00:00+01:00 <figure class="image"> <img src="/images/2020/09/wwwspace.png" alt="Satellite image of Wadden Sea" /> <figcaption><a href="https://commons.wikimedia.org/wiki/File:World_Wide_Web_-_Digital_Preservation.png">“World Wide Web - Digital Preservation”</a> by <a href="https://www.wikidata.org/wiki/Q55754361">Jørgen Stamp</a>, used under <a href="https://creativecommons.org/licenses/by/2.5/dk/deed.en">CC BY 2.5 DK</a>; <a href="https://commons.wikimedia.org/wiki/File:Wadden_Sea.jpg">“Wadden Sea”</a> by Envisat satellite, used under <a href="https://creativecommons.org/licenses/by-sa/3.0-igo">CC BY-SA 3.0-IGO</a>.</figcaption> </figure> <p>Earlier this year I wrote <a href="/2020/02/11/web-domain-geolocation-and-spatial-analysis">a blog post</a> about geo-locating web domains, and the subsequent analysis of the resulting data in <em>QGIS</em>. At the time, this work was meant as a proof of concept, and I had only tried it out on a small set of test data. We have now applied this methodology to the whole of the Dutch (<em>.nl</em>) web domain, and this follow-up post presents the results of this exercise.</p> <!-- more --> <h2 id="identification-of-frisian-sites-in-the-nl-domain">Identification of Frisian sites in the .nl domain</h2> <p>Briefly after I wrote my earlier blog post on geo-locating web domains, my web archiving colleagues expressed an interest in applying the procedure to the Dutch top-level <em>.nl</em> domain in its entirety. The immediate occasion for this was an ongoing <a href="https://newyear.isoc.nl/2019/pres/2019-01-017-03-KeesTeszelsky-FRL.pdf">initiative</a> to map and harvest the Frisian web domain. <a href="https://en.wikipedia.org/wiki/Friesland">Friesland</a> (Fryslân) is a province in the northern part the Netherlands. It has <a href="https://opendata.cbs.nl/statline/#/CBS/nl/dataset/37230ned/table?ts=1599490582090">about 650 thousand inhabitants</a>, over half of which speak <a href="https://en.wikipedia.org/wiki/West_Frisian_language">West Frisian</a> as their native language. In 2014 a dedicated Frisian <em>.frl</em> web domain was established, and in 2019 this domain spanned about 15,000 top-level domain sites, with mostly Frisian-language content<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. As many Frisian web sites are still part of other top-level domains such as the “regular” <em>.nl</em> domain, the identification of such sites could provide valuable additional information about the Frisian web. The reasoning behind this is, that if an <em>.nl</em> domain is hosted somewhere in Friesland, chances are it corresponds to a web site that either uses the Frisian-language, or is related to Friesland in some other way. Since we already had access to a list of all <em>.nl</em> domains (provided to us by registry operator <a href="https://www.sidn.nl/en/theme/about-sidn">Stichting Internet Domeinregistratie Nederland</a>), it seemed viable to extend the earlier geo-location exercise to the entire <em>.nl</em> top-level domain.</p> <h2 id="geolocation-of-web-domains">Geolocation of web domains</h2> <p>Geo-locating a web domain involves two steps<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>:</p> <ol> <li> <p><strong>IP lookup</strong>: for each domain, we need to establish its corresponding IP address. There are various ways (and tools) to do this, but for the current analysis I used the Unix <a href="https://linux.die.net/man/1/host"><em>host</em> tool</a>.</p> </li> <li> <p><strong>Lookup geolocation data</strong>: in this step we look up the geolocation attributes (latitude, longitude, but also country and city names) from each identified IP address. Here I used the <a href="https://github.com/maxmind/GeoIP2-python"><em>GeoIP2</em></a> Python module, with the <a href="https://dev.maxmind.com/geoip/geoip2/geolite2"><em>GeoLite2</em> City database</a> (freely available after registration). This database is widely used; for example, the British Library used it for geo-locating their 2014 domain crawl<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>.</p> </li> </ol> <h2 id="processing-millions-of-domains-in-the-era-of-covid-19">Processing millions of domains in the era of COVID-19</h2> <p>I first tried to run the geolocation script that I also used in my earlier blog post, using the full list of 5.86 million <em>.nl</em> domains as an input file. I started the script on the 2nd of March on a local machine in my office at the KB, and by extrapolating the throughput after one day, I estimated it would need about 10 days to finish. Unfortunately, by that time the KB had closed down due to the COVID-19 outbreak, which meant I was unable to ever retrieve the results any time soon. I briefly considered to re-run the script from home, but decided against it for practical reasons<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>. Then some months later my colleague Kees Teszelszky (who initiated the activities on the Frisian web domain) suggested we might use the services of <a href="https://www.surf.nl/en/services-and-support/purchasing-services-from-surfsara">SURFsara</a>, which hosts several cloud computing facilities that can be used free of charge by Dutch research institutions. After getting in touch with them, SURsara gave us access to its <a href="https://doc.hpccloud.surfsara.nl/">HPC Cloud</a> platform. Here I created a simple virtual machine running Ubuntu Server, and then installed the dependencies needed by the geolocation script.</p> <h2 id="parallelization">Parallelization</h2> <p>To improve the performance of the script itself, I also parallelized its IP lookup component (which would take up most of its running time). Tests on an early version of the parallelized script resulted in a nasty memory leak when run with the full domains list. Even though I was able to fix this (albeit in a rather ugly way), tests with the <a href="https://github.com/KBNLresearch/geolocatedomains/blob/master/scripts/geolocatedomains.py">improved script</a> still showed a slow increase of memory consumption over time. I was not sure how this increase would develop over the course of processing 5.86 million domains, so to be on the safe side I simply split the domains list into 6 smaller files of 1 million domains each, and then ran the script consecutively on all these files.</p> <h2 id="geolocation-results">Geolocation results</h2> <p>The processing of all 6 files completed without problems in about 54 hours. The main result of this is a set of 6 comma-delimited text files, with the following fields:</p> <ul> <li><strong>domain</strong>: name of the web domain</li> <li><strong>hasValidIP</strong>: Boolean flag that indicates whether the domain could be mapped to an IP address</li> <li><strong>countryIsoCode</strong>: country ISO code (if available)</li> <li><strong>cityName</strong>: city name (if available)</li> <li><strong>latitude</strong> / <strong>longitude</strong>: latitude and longitude in decimal degrees (if available)</li> <li><strong>accuracyRadius</strong>: indicator of the accuracy of the reported latitude / longitude pair in kilometers (if available)</li> </ul> <p>Note that this does not tell us yet in which province each web domain is hosted. However, it’s pretty easy to add this information, as I will show below.</p> <h2 id="adding-the-province-information">Adding the province information</h2> <p>In order to establish the corresponding province of each domain, I used <a href="https://www.qgis.org/">QGIS</a>, an open-source geographical information system, and a freely available <a href="https://www.nationaalgeoregister.nl/geonetwork/srv/dut/catalog.search#/metadata/e73b01f6-28c7-4bb7-a782-e877e8113e2c">vector layer of Dutch province boundaries</a> in <a href="https://en.wikipedia.org/wiki/Shapefile">Shapefile</a> format. In this Shapefile each province is represented as a polygon <sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>. After opening the shapefile, I imported the first geolocation CSV file. I subsequently used the <em>Join Attributes by Location …</em> function to add, for each domain point location, the value of the corresponding Province field from the Shapefile (if available). I then exported the result to a new CSV file (see my <a href="/2020/02/11/web-domain-geolocation-and-spatial-analysis">earlier post</a> for details). I repeated this for each of the 6 CSV files. Finally I combined the 6 resulting output CSV files into one single file.</p> <p>After importing this file into QGIS again (which is slow) I was able to create the following visualization of all domain hosting locations in the immediate vicinity of the Netherlands:</p> <figure class="image"> <img src="/images/2020/09/domains-map.png" alt="Map of identified server locations" /> <figcaption>Map of identified hosting locations (locations outside the immediate vicinity of the Netherlands not shown).</figcaption> </figure> <p>Note that one hosting location may represent many domains, so the number of visible dots does not reflect the actual number of domains. Also, I should perhaps stress here that these are all <em>hosting</em> locations (i.e. the location of the service provider that hosts a domain), <em>not</em> locations of individual domain owners!</p> <h2 id="spatial-distribution-of-the-dutch-web-domain">Spatial distribution of the Dutch web domain</h2> <p>The results of this geolocation exercise give some interesting insights into the spatial distribution of the Dutch web domain. Below chart shows the distribution of hosting countries (based on the country ISO codes from the GeoLite2 database):</p> <figure class="image"> <img src="/images/2020/09/countries.png" alt="Pie chart of active domain counts by country" /> <figcaption>Active domain counts by country.</figcaption> </figure> <p>Here are the same data as a table:</p> <table> <thead> <tr> <th style="text-align: left">Country</th> <th style="text-align: right">Count</th> <th style="text-align: right">% of all active domains</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">NL</td> <td style="text-align: right">3.45617e+06</td> <td style="text-align: right">75.03</td> </tr> <tr> <td style="text-align: left">DE</td> <td style="text-align: right">419623</td> <td style="text-align: right">9.11</td> </tr> <tr> <td style="text-align: left">US</td> <td style="text-align: right">235962</td> <td style="text-align: right">5.12</td> </tr> <tr> <td style="text-align: left">IE</td> <td style="text-align: right">180379</td> <td style="text-align: right">3.92</td> </tr> <tr> <td style="text-align: left">FR</td> <td style="text-align: right">78023</td> <td style="text-align: right">1.69</td> </tr> <tr> <td style="text-align: left">DK</td> <td style="text-align: right">68584</td> <td style="text-align: right">1.49</td> </tr> <tr> <td style="text-align: left">BE</td> <td style="text-align: right">42333</td> <td style="text-align: right">0.92</td> </tr> <tr> <td style="text-align: left">GB</td> <td style="text-align: right">26748</td> <td style="text-align: right">0.58</td> </tr> <tr> <td style="text-align: left">LU</td> <td style="text-align: right">15569</td> <td style="text-align: right">0.34</td> </tr> <tr> <td style="text-align: left">Other</td> <td style="text-align: right">56489</td> <td style="text-align: right">1.17</td> </tr> </tbody> </table> <p>This shows that almost 25% of all active <em>.nl</em> domains are hosted from locations outside the Netherlands (Germany, United States and Ireland are the most popular hosting countries). Zooming in on the Netherlands only, the following chart shows the distribution of provinces from which domains are hosted:</p> <figure class="image"> <img src="/images/2020/09/provinces.png" alt="Pie chart of active domain counts by province" /> <figcaption>Active domain counts by province (% of NL-hosted domains).</figcaption> </figure> <p>And again in table form:</p> <table> <thead> <tr> <th style="text-align: left">Province</th> <th style="text-align: right">Count</th> <th style="text-align: right">% of all NL-hosted domains</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">Noord-Holland</td> <td style="text-align: right">2.60368e+06</td> <td style="text-align: right">75.34</td> </tr> <tr> <td style="text-align: left">Zuid-Holland</td> <td style="text-align: right">270357</td> <td style="text-align: right">7.82</td> </tr> <tr> <td style="text-align: left">Gelderland</td> <td style="text-align: right">221120</td> <td style="text-align: right">6.4</td> </tr> <tr> <td style="text-align: left">Noord-Brabant</td> <td style="text-align: right">133056</td> <td style="text-align: right">3.85</td> </tr> <tr> <td style="text-align: left">Utrecht</td> <td style="text-align: right">62215</td> <td style="text-align: right">1.8</td> </tr> <tr> <td style="text-align: left">Flevoland</td> <td style="text-align: right">53595</td> <td style="text-align: right">1.55</td> </tr> <tr> <td style="text-align: left">Overijssel</td> <td style="text-align: right">33781</td> <td style="text-align: right">0.98</td> </tr> <tr> <td style="text-align: left">Groningen</td> <td style="text-align: right">28392</td> <td style="text-align: right">0.82</td> </tr> <tr> <td style="text-align: left">Fryslân</td> <td style="text-align: right">20184</td> <td style="text-align: right">0.58</td> </tr> <tr> <td style="text-align: left">Drenthe</td> <td style="text-align: right">14971</td> <td style="text-align: right">0.43</td> </tr> <tr> <td style="text-align: left">Limburg</td> <td style="text-align: right">11154</td> <td style="text-align: right">0.32</td> </tr> <tr> <td style="text-align: left">Zeeland</td> <td style="text-align: right">3325</td> <td style="text-align: right">0.1</td> </tr> </tbody> </table> <p>It’s remarkable to see that over 75% of all NL-hosted domains are hosted from the province of Noord-Holland (North Holland) alone. This might simply reflect the concentration of internet hosting companies in especially the Amsterdam area. At only about 20 thousand domains, Friesland (Fryslân) is one of the “smallest” provinces in terms of this analysis.</p> <h2 id="discussion">Discussion</h2> <p>In this blog post I have outlined the geo-location and data processing procedure, and I have also shown how the results of this exercise can reveal spatial characteristics of a top-level web domain. However, it’s important to be aware of the limitations of the source data used here, as well as the methodology as such.</p> <p>First of all there’s the accuracy of the geolocation results. The latitude and longitude values in the GeoLite2 database are only approximate, and refer to larger spatial entities, ranging from cities to even whole countries. As a measure of the approximate accuracy at any location, it reports an “accuracy radius” value, which is <a href="https://www.maxmind.com/en/geoip2-precision-insights">defined as</a>:</p> <blockquote> <p>The approximate accuracy radius, in kilometers, around the latitude and longitude for the geographical entity (country, subdivision, city or postal code) associated with the IP address.</p> </blockquote> <p>The figure below shows the distribution of accuracy radius values for all active domains that are part of the <em>.nl</em> domain:</p> <figure class="image"> <img src="/images/2020/09/accuracyradius.png" alt="Bar chart of distribution of accuracy radius values for active domains" /> <figcaption>Distribution of accuracy radius values for active domains.</figcaption> </figure> <p>The underlying data show an accuracy radius value of 100 km or more for 96% of all active domains. If these values reflect the true accuracy of the database, the results of the current analysis should be interpreted with a good deal of caution, especially for a small country like the Netherlands. It might be able to get better results with MaxMind’s (the company behind the GeoLite2 database) higher-precision databases, which are available commercially.</p> <p>Another thing to keep in mind, is that not all active domains have an associated web site. For example, some domains are only used for e-mail (I actually own such an email-only domain myself). For this analysis I haven’t investigated this any further.</p> <h2 id="acknowledgements">Acknowledgements</h2> <p><a href="https://www.sidn.nl/en/theme/about-sidn">Stichting Internet Domeinregistratie Nederland</a> provided us the top-level <em>.nl</em> domain data on which this analysis is based. <a href="https://www.surf.nl/en/services-and-support/purchasing-services-from-surfsara">SURFsara</a> (and in particular Nuno Ferreira) is thanked for allowing us to use their HPC Cloud computing facilities, and for their help in getting things running. Thanks are also due to Kees Teszelszky for his help with this work.</p> <h2 id="further-resources">Further resources</h2> <ul> <li> <p><a href="https://github.com/KBNLresearch/geolocatedomains/blob/master/scripts/geolocatedomains.py">Geolocation script</a></p> </li> <li> <p><a href="https://github.com/KBNLresearch/geolocatedomains/blob/master/scripts/analyze.py">Analysis script for QGIS-produced CSV file</a> - This script automatically creates a report with most of the table data presented in this blog. It also exports the Frisian records to a subset.</p> </li> <li> <p><a href="https://github.com/KBNLresearch/geolocatedomains/blob/master/notes/notes.md">Unedited working notes</a></p> </li> <li> <p><a href="/2020/02/11/web-domain-geolocation-and-spatial-analysis">Web domain geolocation and spatial analysis with QGIS</a><sup id="fnref:6"><a href="#fn:6" class="footnote" rel="footnote" role="doc-noteref">6</a></sup></p> </li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>See Kees Teszelszky - <a href="https://doi.org/10.34894/PO7PZD">‘Alle Begjin Is Swier’: The Use Of The Frisian Web Domain Web Data For Digital Humanities Research</a>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>See my <a href="/2020/02/11/web-domain-geolocation-and-spatial-analysis">earlier blog post</a> for a more detailed discussion. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>Geo-location in the 2014 UK Domain Crawl: <a href="https://blogs.bl.uk/webarchive/2015/07/geo-location-in-the-2014-uk-domain-crawl.html">https://blogs.bl.uk/webarchive/2015/07/geo-location-in-the-2014-uk-domain-crawl.html</a> <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>Mostly related to network stability, and the fact that I needed the two machines I have available at home for other things that would most likely interfere with the geolocation procedure. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>According to its <a href="https://www.nationaalgeoregister.nl/geonetwork/srv/dut/catalog.search#/metadata/e73b01f6-28c7-4bb7-a782-e877e8113e2c?tab=general">description</a>, the provinces file is suitable for map scales ranging from 1:750,000 to 1:1,000,000, which suggests a (very approximate) spatial accuracy of about 1 km, which is adequate for this purpose. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:6"> <p>Note that some of the content of this blog post is outdated, because it is based on an older version of QGIS. For example, with the current (3.14) version I did not need to perform the (somewhat tedious) coordinate system transformation step. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2020/09/09/mapping-the-dutch-web-domain https://bitsgalore.org/2020/09/09/mapping-the-dutch-web-domain 2020-09-09T00:00:00+02:00 <figure class="image"> <img src="/images/2020/06/liesbet-door.png" alt="Liesbet door" /> <figcaption>Original artwork copyright &copy;Liesbet Zikkenheimer.</figcaption> </figure> <p>In 2019, Dutch telecommunications company <a href="https://en.wikipedia.org/wiki/KPN">KPN</a> announced its plans to phase out its subsidiary <a href="https://en.wikipedia.org/wiki/XS4ALL">XS4ALL</a>, which is one of the oldest internet service providers in the Netherlands. With this decision, thousands of homepages and personal web sites that are hosted under the XS4ALL domain are at risk of disappearing forever. The web archiving team of the National Library of the Netherlands (KB) has started an initiative to rescue a selection of these homepages, which includes some of the oldest born-digital publications of the Dutch web. This blog post describes an attempt to rescue and restore one of the oldest and most unique homepages from this collection: Liesbet’s Virtual Home (Liesbet’s Atelier), the personal web site of Dutch Internet pioneer Liesbet Zikkenheimer, which has a history that goes back to 1995. First I give some background information about XS4ALL, and the KB-led rescue initiative. Then I move on to the various (mostly technical) aspects of restoring Liesbet’s Virtual Home. Finally, I address the challenges of capturing the restored site to an ingest-ready <a href="https://en.wikipedia.org/wiki/Web_ARChive">WARC</a> file.</p> <!-- more --> <h2 id="xs4all">XS4ALL</h2> <p><a href="https://en.wikipedia.org/wiki/XS4ALL">XS4ALL</a> is one of the oldest internet service providers in the Netherlands, and even one of the oldest providers in the whole world. The company was founded in 1993, and has its roots in the Dutch hacker scene. Since its inception, many pioneers of the Dutch internet hosted their homepages under the XS4ALL web domain. Some of these homepages have been online for more than 25 years, and as such they rank among the oldest born-digital publications of the Dutch web. In 2019, parent company KPN (which bought XS4ALL in 1998) announced their intention to phase out the XS4ALL brand. Eventually, all of the company’s services will continue under the KPN brand. This poses an acute threat to much of the (often unique) digital heritage that is hosted under the XS4ALL domain, as in the past a similar situation with provider <a href="https://nl.wikipedia.org/wiki/EuroNet">Euronet</a> resulted in <a href="https://www.tandfonline.com/doi/full/10.1080/24701475.2019.1603951">the loss of thousands of early Dutch homepages</a>.</p> <h2 id="rescuing-the-xs4all-homepages">Rescuing the XS4ALL homepages</h2> <p>So, the KB’s <a href="https://www.kb.nl/en/organisation/research-expertise/long-term-usability-of-digital-resources/web-archiving">web archiving team</a> started an initiative to rescue a selection of the XS4ALL homepages, and add them to the web archive in a special XS4ALL collection. This initiative is supported financially by the <a href="https://www.sidnfonds.nl/excerpt">SIDN Fund</a> and <a href="https://www.stichtinginternet4all.nl/">stichting Internet4all</a>.</p> <h2 id="digital-treasures">Digital treasures</h2> <p>As of May 2020, the KB has selected <a href="https://www.kb.nl/blogs/duurzame-toegang/bewaren-voor-iedereen-de-opbouw-van-de-webcollectie-xs4all-homepages">3370 homepages</a> for inclusion in XS4ALL collection. Out of the homepages that have been archived so far, 404 are marked as “digital treasures”. These homepages are remarkable because of their age, or because of characteristics that are either unique, or, on the other hand, typical of a particular era or trend.</p> <h2 id="liesbets-virtual-home">Liesbet’s Virtual Home</h2> <p>One of these “treasures” is <a href="https://ziklies.home.xs4all.nl/">Liesbet’s Virtual Home</a> (in Dutch: Liesbet’s Atelier). This is the old homepage of <a href="http://zicnet.nl/">Liesbet Zikkenheimer</a>, a Dutch Internet pioneer with a background in industrial and graphic design. Her homepage is a “treasure” for several reasons. First of all, it has a history that goes back to 1995, which makes it one of the oldest Dutch homepages that are still available today. Second, Zikkenheimer is an important figure in the history of the Dutch internet. To mention a few examples, in 1997 she developed and published <a href="https://web.archive.org/web/19980526231446/http://www.libelle.nl/libelle/dezeweek/dezeweek.html">the online version of popular women’s magazine Libelle</a>. She also created web sites for the Margriet and Viva magazines, and developed, published and managed several well-known web portals, most of which were primarily targeted at women. Finally, Liesbet’s Virtual Home is unique because of its structure and design. The site is literally structured like a physical house. Each page represents a particular room, and to get from, say, the living room to the loft, one needs to navigate through a hallway and two flights of stairs. It also has some interactive features that were quite unique at the time of is creation. So, the site meets every possible “digital treasure” criterion.</p> <h2 id="problems-with-the-live-site">Problems with the live site</h2> <p>Even though Liesbet’s Virtual Home <a href="https://ziklies.home.xs4all.nl/">is still online</a>, several features of the site are no longer working. In particular:</p> <ul> <li> <p>The navigation on several pages (including the landing page) is broken, because it is based on server-side <a href="https://en.wikipedia.org/wiki/Image_map">image maps</a> that are no longer available.</p> </li> <li> <p>Some interactive features like <a href="https://ziklies.home.xs4all.nl/slaapk/e-slaap1.html">this bedroom mirror</a> don’t work anymore, because the underlying scripts are missing.</p> </li> </ul> <p>This raised the question whether it would be possible to create a “restored” version of Liesbet’s Virtual Home that has these features working again.</p> <h2 id="local-copy-of-site-data">Local copy of site data</h2> <p>After my colleague Kees Teszelszky got in contact with Zikkenheimer, she sent us a ZIP file with a locally stored copy of the site’s directory structure. However, that local copy had several issues as well, and it quickly became obvious it couldn’t be used as a basis for a restored version of the site. However, the ZIP file did contain both the image map files as well as the scripts that are missing from the live site.</p> <h2 id="crawling-the-live-site">Crawling the live site</h2> <p>So, I decided to take the current live site as a starting point. I first tried to crawl and capture the site with <a href="https://github.com/KBNLresearch/xs4all-resources/blob/master/scripts/scrapesite.sh">this simple Bash script</a> that uses the <a href="https://www.gnu.org/software/wget/">wget</a> tool. At first sight this seemed to work reasonably well, but a closer inspection revealed that various components of the site were missing. A few examples:</p> <ul> <li> <p>The <a href="https://ziklies.home.xs4all.nl/e-toilet.html">toilet</a> pages are not referenced from their <a href="https://ziklies.home.xs4all.nl/e-start.html">parent pages</a>. This has the result that wget never finds them.</p> </li> <li> <p>Some components are only referenced through Javascript. For example, <a href="https://ziklies.home.xs4all.nl/woonk/woon03.html">this page</a> contains the following code that opens a video in a popup window:</p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;A</span> <span class="na">HREF=</span><span class="s">"javascript:openit('tvplus.mov')"</span><span class="nt">&gt;</span> </code></pre></div> </div> <p>Such items are not picked up by wget, which means they end up missing from the crawl.</p> </li> </ul> <h2 id="improved-crawl">Improved crawl</h2> <p>After some experimentation, I was able to improve the crawl by using multiple seed URLs. This means that instead of traversing the site from its index page only, I included both the unreferenced “toilet” pages, as well as a list of all the site’s visible directories and sub-directories (which I could identify from the initial crawl). First I put all of these in a text file (seed-urls.txt):</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>https://ziklies.home.xs4all.nl/ https://ziklies.home.xs4all.nl/toilet.html https://ziklies.home.xs4all.nl/e-toilet.html https://ziklies.home.xs4all.nl/atelier/ https://ziklies.home.xs4all.nl/bad/ https://ziklies.home.xs4all.nl/cas/ https://ziklies.home.xs4all.nl/gambia/ https://ziklies.home.xs4all.nl/keuken/ https://ziklies.home.xs4all.nl/slaapk/ https://ziklies.home.xs4all.nl/slaapk/gspot/ https://ziklies.home.xs4all.nl/toilet/ https://ziklies.home.xs4all.nl/woonk/ https://ziklies.home.xs4all.nl/woonk/agenda/ https://ziklies.home.xs4all.nl/zolder/ </code></pre></div></div> <p>I then ran <a href="https://github.com/KBNLresearch/xs4all-resources/blob/master/scripts/scrape-seeds.sh">this Bash script</a>, using the above text file as a command-line argument:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>scrape-seeds.sh seed-urls.txt </code></pre></div></div> <p>As a check I subsequently did a recursive diff on the output directories of both the original crawl and the improved crawl:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>diff <span class="nt">-r</span> ./wget-original/ziklies.home.xs4all.nl/ ./wget-improved/ziklies.home.xs4all.nl/ <span class="o">&gt;</span> diff-site-toilet.txt </code></pre></div></div> <p>This showed that the improved crawl contained over 60 files that were not in the original crawl. So, I used the result of this “improved” crawl as a basis for all subsequent restoration steps. In the following sections I will go through the whole restoration process. More details are available in a separate, minimally edited <a href="https://github.com/KBNLresearch/xs4all-resources/blob/master/doc/liesbets-atelier-restoration-notes.md">Restoration notes</a> document.</p> <h2 id="links-to-old-website-domain">Links to old website domain</h2> <p>Like all old XS4ALL homepages, Liesbet’s Virtual Home was originally hosted as a directory under XS4ALL’s root domain (<a href="http://www.xs4all.nl/~ziklies/">http://www.xs4all.nl/~ziklies/</a>). At some point XS4ALL gave its customers their own sub-domain (in this case the current address at <a href="https://ziklies.home.xs4all.nl/">https://ziklies.home.xs4all.nl/</a>), and redirected any URLs pointing to the “old” location to this sub-domain. Internally, Liesbet’s Virtual Home uses a mixture of relative URLs and absolute ones that still refer to the old location. This causes several issues if the site is hosted locally on a web server. Although it may be possible to remedy these issues using some clever server configuration, I couldn’t quite get this working. I ended up writing a <a href="https://github.com/KBNLresearch/xs4all-resources/blob/master/scripts/rewriteurls.sh">simple Bash script</a> that replaces all references to the “old” location with relative links (which always work, irrespective of the domain).</p> <h2 id="audit-trail">Audit trail</h2> <p>Since a restoration like this involves making changes to a unique digital heritage work, it’s a good idea to record these changes in a verifiable audit trail. To achieve this I simply set up the directory with the crawl as a <a href="https://en.wikipedia.org/wiki/Git">Git</a> repository, and then created a snapshot (Git commit) for each change. The following screenshot (from the <a href="https://git-scm.com/docs/gitk">gitk</a> Git repository browser) illustrates this:</p> <figure class="image"> <img src="/images/2020/06/liesbet-gitk.png" alt="Gitk screenshot" /> <figcaption>Screenshot of Gitk Git repository browser.</figcaption> </figure> <ul> <li> <p>The upper-left pane lists all snapshots/commits (newest at the top, oldest at the bottom).</p> </li> <li> <p>The lower-left pane shows all changes between the currently selected snapshot and the previous one. In this example, we can see that in file “kaart01.htm” three absolute page references (highlighted in red) were replaced by relative references (highlighted in green).</p> </li> <li> <p>The lower-right pane lists all files that were changed in this snapshot.</p> </li> </ul> <p>This way, the commit history provides a complete audit trail of all changes. It also has the advantage that any changes can be easily undone, if needed. For example, the URL rewriting operation discussed in the previous section had an unintentional side-effect for the <a href="https://ziklies.home.xs4all.nl/statistics.html">statistics page</a>. I could easily revert to the original URLs for this page with 1 single Git command.</p> <h2 id="missing-links-to-toilet">Missing links to toilet</h2> <p>As mentioned before, the <a href="https://ziklies.home.xs4all.nl/start.html">start.html</a> and <a href="https://ziklies.home.xs4all.nl/e-start.html">e-start.html</a> pages contain erroneous links to the “toilet” pages. Looking at the English-language page:</p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;B&gt;&lt;A</span> <span class="na">HREF=</span><span class="s">"http://imagine.xs4all.nl/ziklies/"</span><span class="nt">&gt;</span> Go to the toilet<span class="nt">&lt;/a&gt;&lt;/B&gt;</span> </code></pre></div></div> <p>Here, the URL points to an external domain that doesn’t exist anymore. So, I changed this to:</p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;B&gt;&lt;A</span> <span class="na">HREF=</span><span class="s">"e-toilet.html"</span><span class="nt">&gt;</span> Go to the toilet<span class="nt">&lt;/a&gt;&lt;/B&gt;</span> </code></pre></div></div> <p>I applied a similar fix to the Dutch-language page.</p> <h2 id="image-maps">Image maps</h2> <p>A number of pages on the site use HTML <a href="https://en.wikipedia.org/wiki/Image_map">image maps</a>. An example is the door image on the <a href="https://ziklies.home.xs4all.nl/">front page</a>. This is the corresponding HTML source:</p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;A</span> <span class="na">HREF=</span><span class="s">"/cgi-bin/imagemap/~ziklies/deurtje1.map"</span><span class="nt">&gt;&lt;img</span> <span class="na">src=</span><span class="s">"deurtje1.gif"</span> <span class="na">Border=</span><span class="s">0</span> <span class="na">ISMAP</span><span class="nt">&gt;&lt;/A&gt;</span> </code></pre></div></div> <p>This is a <em>server-side</em> image map, where the URL points to an external file (deurtje1.map). However, this file is not available anymore on the live site, which results in a <a href="https://en.wikipedia.org/wiki/HTTP_404">Page Not Found</a> error. Fortunately, I was able to find this file in the ZIP archive provided by Zikkenheimer. Here’s what it looks like:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>default http://www.xs4all.nl/~ziklies/start.html poly http://www.xs4all.nl/~ziklies/start.html 0,56 76,40 91,43 89,67 76,62 6,77 0,60 1,55 poly http://www.xs4all.nl/~ziklies/e-start.html 53,72 80,76 81,81 91,85 89,107 76,106 69,137 42,130 51,77 </code></pre></div></div> <p>This shows that the file simply defines areas within the image that are linked to URLs.</p> <h2 id="from-server-side-to-client-side-image-maps">From server-side to client-side image maps</h2> <p>Since <em>server-side</em> image maps <a href="https://eager.io/blog/a-quick-history-of-image-maps/">come with some caveats</a>, I took the liberty of re-implementing the server-side image map with a <em>client-side</em> image map. Both are functionally identical, but client-side image maps are simpler to implement and less likely to break<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. Instead of using an external file, a client-side image map is simply an embedded element inside the page, which means we can replace the <code class="language-plaintext highlighter-rouge">&lt;A&gt;</code> element in the previous HTML snippet by this:</p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;img</span> <span class="na">src=</span><span class="s">"deurtje1.gif"</span> <span class="na">usemap=</span><span class="s">"#deurtje1Map"</span> <span class="na">alt=</span><span class="s">"deurtje 1"</span> <span class="na">border=</span><span class="s">"0"</span><span class="nt">&gt;</span> <span class="nt">&lt;map</span> <span class="na">name=</span><span class="s">"deurtje1Map"</span><span class="nt">&gt;</span> <span class="nt">&lt;area</span> <span class="na">shape=</span><span class="s">"poly"</span> <span class="na">coords=</span><span class="s">"0,56 76,40 91,43 89,67 76,62 6,77 0,60 1,55"</span> <span class="na">href=</span><span class="s">"http://www.xs4all.nl/~ziklies/start.html"</span><span class="nt">&gt;</span> <span class="nt">&lt;area</span> <span class="na">shape=</span><span class="s">"poly"</span> <span class="na">coords=</span><span class="s">"53,72 80,76 81,81 91,85 89,107 76,106 69,137 42,130 51,77"</span> <span class="na">href=</span><span class="s">"http://www.xs4all.nl/~ziklies/e-start.html"</span><span class="nt">&gt;</span> <span class="nt">&lt;area</span> <span class="na">shape=</span><span class="s">"default"</span> <span class="na">href=</span><span class="s">"http://www.xs4all.nl/~ziklies/start.html"</span><span class="nt">&gt;</span> <span class="nt">&lt;/map&gt;</span> </code></pre></div></div> <p>Note that the values of the “coords” attributes are identical to the area definitions in the server-side image map. Below is a short video that shows how the restored image map works. Ringing the upper doorbell leads to the Dutch version of the site, whereas the lower doorbell opens the English version.</p> <figure class="image"> <video width="100%" height="100%" title="Image map demo" controls=""> <source src="/images/2020/06/imagemap.mp4" type="video/mp4" /> Your browser does not support the video tag. </video> <figcaption>Demonstration of image map.</figcaption> </figure> <p>The site contains 4 more broken server-side image maps. I replaced all of these with client-side image maps in the restored version. For <a href="https://ziklies.home.xs4all.nl/start.html">one page</a> the corresponding image map from the ZIP file contained some odd errors, so here I took the image map of the page’s <a href="https://ziklies.home.xs4all.nl/e-start.html">English-language counterpart</a>, and then updated all links accordingly. After these changes the image map navigation is fully functional again.</p> <h2 id="interactive-bedroom-mirror">Interactive bedroom mirror</h2> <p>The <a href="https://ziklies.home.xs4all.nl/slaapk/e-slaap1.html">bedroom</a> of Liesbet’s Virtual Home features an “interactive mirror”. It is a web form where the visitor can select combinations of clothing, hairstyle and earrings. After clicking on the “have a look into the mirror” button, the selected combination is shown as an image<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>. However, as the underlying scripts are missing from the live site, it now gives a <a href="https://en.wikipedia.org/wiki/HTTP_404">Page Not Found</a> error. As with the image maps before, the missing (Perl) scripts could be recovered from the ZIP file provided by Zikkenheimer. I added these to a (newly created) “cgi-bin” directory. I also had to make the scripts executable<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>, and adjust their <a href="https://en.wikipedia.org/wiki/Shebang_(Unix)">Shebang</a> strings to a valid interpreter location on my local machine (which, in my case, was different from the location used by the original web server).</p> <p>The main challenge was then to make the scripts play nicely with a web server. This is beyond the scope of this post, but I created an <a href="https://github.com/KBNLresearch/xs4all-resources/blob/master/doc/liesbets-atelier-apache-notes.md">Apache setup notes</a> document that describes how I made this all work with a local instance of the Apache web server. Amazingly, the script, which is nearly 25 years old, still works perfectly with a modern version of Perl (here Perl 5). The following video gives a brief glimpse of the restored interactive mirror in action:</p> <figure class="image"> <video width="100%" height="100%" title="Interactive bedroom mirror demo" controls=""> <source src="/images/2020/06/mirror.mp4" type="video/mp4" /> Your browser does not support the video tag. </video> <figcaption>Demonstration of interactive bedroom mirror.</figcaption> </figure> <h2 id="interactive-toilet-door">Interactive toilet door</h2> <p>The <a href="https://ziklies.home.xs4all.nl/e-toilet.html">toilet page</a> has an interactive feature where visitors can scratch a message onto a virtual toilet door. Zikkenheimer wrote to us that she initially updated the toilet door image by hand (presumably using submissions sent by e-mail), but that she later replaced this with a Python script that automatically updates the image. This script is no longer available from the live site, but it is included in the ZIP file. From a text string, it maps each <a href="https://en.wikipedia.org/wiki/Glyph">glyph</a> to a corresponding GIF image, and then pastes these GIF images onto the pre-existing version of the toilet door image, using randomly selected image co-ordinates.</p> <figure class="image"> <img src="/images/2020/06/toilet-door.png" alt="Toilet door" /> <figcaption>Interactive toilet door.</figcaption> </figure> <p>Interestingly, the source of the current “live version” references another script that simply sends the entered message by e-mail<sup id="fnref:6"><a href="#fn:6" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>. However, the Python script from the ZIP file shows that the toilet pages were originally hosted at a different server location, and <a href="https://web.archive.org/web/19981205234146/http://prima12.xs4all.nl/liesbet/toilet.html">an archived snapshot from the Internet Archive</a> still exists. The source code of that snapshot also contains a link to the Python script.</p> <h2 id="restoring-the-toilet-door">Restoring the toilet door</h2> <p>I tried to make the interactive toilet door work again, but while doing this I ran into several problems. First of all, the underlying Python script is based on Python 1.4, which was one of the earliest Python releases, and its code is not compatible with modern Python interpreters. I tried to upgrade it to Python-3-compatible code. Overall this was pretty easy, but it uses <a href="https://github.com/Solomoriah/gdmodule">gdmodule</a>, a Python module that is <a href="https://github.com/Solomoriah/gdmodule/issues/3">currently unsupported and not compatible</a> with Python 3. I was able to make the script work with Python 2.7, but since Python 2 <a href="https://www.python.org/doc/sunset-python-2/">is no longer maintained</a> this is not a sustainable solution.</p> <p>Also, the set of glyph images in the ZIP file appeared to be incomplete, only including lowercase characters (with an image for the “u” glyph strangely missing), and no uppercase characters or punctuation marks. If the user enters any of these missing characters, this results in an “Internal Server Error”. In addition, the script often places text outside of the toilet door image canvas. For both of these issues, it is unclear whether they are specific to the restored version, or simply reflect the state of the old “live” site. Because of these issues, I would not consider the restoration of this part of the site a success.</p> <h2 id="unsupported-file-formats">Unsupported file formats</h2> <p>The site also uses a number of file formats that are not supported by modern browsers. Some examples:</p> <ul> <li> <p>The <a href="https://ziklies.home.xs4all.nl/woonk/e-woon03.html">living room</a> features a clickable TV-set that is supposed to open a <a href="http://fileformats.archiveteam.org/wiki/Quicktime">Quicktime video</a> in a pop-up window. In the latest (77.0.1) version of the Firefox browser, it triggers the following message:</p> <figure class="image"> <img src="/images/2020/06/quicktime-ff.png" alt="No video with supported format and MIME type found" /> <figcaption>Firefox error message on Quicktime video.</figcaption> </figure> <p>In Chromium (83.0.4103.61), the pop-up window is empty, but it does download the file, so it can be played with external media player software.</p> </li> <li> <p>The clickable stereo on the same page links to a <a href="http://fileformats.archiveteam.org/wiki/AU">Sun Audio</a> file. Depending on the browser used, clicking the link either activates a prompt to play the file in a external player (Firefox), or it is simply downloaded (Chromium)<sup id="fnref:8"><a href="#fn:8" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>.</p> </li> <li> <p><a href="https://ziklies.home.xs4all.nl/slaapk/e-slaapa.html">This page</a> features an embedded alarm clock in MIDI format, which is also not natively supported by modern web browsers. Chromium does download the file, whereas Firefox appears to ignore it altogether.</p> </li> <li> <p>Last but not least, the bedroom page links to <a href="https://ziklies.home.xs4all.nl/slaapk/gspot/index.html">this</a>, which in turn <a href="https://ziklies.home.xs4all.nl/slaapk/gspot/gspot.dcr">links</a> to an embedded <a href="https://en.wikipedia.org/wiki/Adobe_Shockwave">Adobe Shockwave file</a>. Adobe <a href="https://helpx.adobe.com/shockwave/shockwave-end-of-life-faq.html">stopped supporting this format in 2019</a>. As a result, the format is no longer supported in web browsers. With no alternative rendering software available, the format is now functionally obsolete.</p> </li> </ul> <p>I haven’t addressed any of these issues in the current restoration attempt. A possible solution would be to use emulation or virtualization to view the site in a late-‘90s web browser<sup id="fnref:9"><a href="#fn:9" class="footnote" rel="footnote" role="doc-noteref">6</a></sup>. This may be worth further investigation.</p> <h2 id="serving-the-site">Serving the site</h2> <p>Throughout the restoration process I mostly used Python’s built-in <a href="https://docs.python.org/3/library/http.server.html">http.server</a> to test any changes I made. This is a lightweight web server that doesn’t require any elaborate configuration, with no need to copy files to reserved locations on the file system. It does have some limitations that make it unsuitable for production use, so for serving the “completed” site I set up and configured an <a href="https://httpd.apache.org/">Apache</a> web server instance. This allowed me to have the restored version of Liesbet’s Virtual Home running on my local machine, accessible from its original URL:</p> <figure class="image"> <img src="/images/2020/06/atelier-local-apache.png" alt="Liesbet's Virtual Home screenshot" /> <figcaption>Screenshot of Liesbet's Virtual Home, served with local Apache instance.</figcaption> </figure> <p>The installation and configuration process I followed is described in detail in a separate <a href="https://github.com/KBNLresearch/xs4all-resources/blob/master/doc/liesbets-atelier-apache-notes.md">Apache setup notes</a> document.</p> <h2 id="warc-capture">WARC capture</h2> <p>Like most web archives, the KB uses the <a href="https://en.wikipedia.org/wiki/Web_ARChive">WARC</a> format for storing archived web sites. Since capturing offline web content is a subject I’d been <a href="/2018/07/11/crawling-offline-web-content-the-nl-menu-case">working on earlier as part of the NL-menu rescue operation</a>, I started with <a href="https://github.com/KBNLresearch/xs4all-resources/blob/master/scripts/scrape-local-site.sh">this wget-based script</a>, which is a modified version of the script I used for NL-menu. However, the wget crawl didn’t adequately capture the form (and script) behind the <a href="https://ziklies.home.xs4all.nl/slaapk/e-slaap1.html">interactive bedroom mirror</a>. Some tests with the <a href="https://github.com/webrecorder/webrecorder-desktop">Webrecorder Desktop App</a> showed that Webrecorder was able to capture individual input combinations, but this required manual input for each combination. With 512 possible combinations, this was not a viable solution, so I needed some way to automate this.</p> <h2 id="warcio-to-the-rescue">Warcio to the rescue</h2> <p>Happily, several people responded to <a href="https://twitter.com/bitsgalore/status/1275405890947108866">my request for help on Twitter</a>. Webrecorder author Ilya Kreymer responded I might want to have a look at the <a href="https://github.com/webrecorder/warcio">warcio library</a> (of which is he is also the lead developer), and, even better, he <a href="https://twitter.com/IlyaKreymer/status/1275440674687471617">provided some example code</a> that showed how to do this. For a brief explanation of how this works, have a look at the input form below (for brevity I edited out some of the input choices):</p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;FORM</span> <span class="na">METHOD=</span><span class="s">"POST"</span> <span class="na">ACTION=</span><span class="s">"/cgi-bin/barbie1.cgi"</span><span class="nt">&gt;</span> <span class="nt">&lt;DL&gt;</span> <span class="nt">&lt;DD&gt;&lt;b&gt;</span> First: wich pants or skirt? <span class="nt">&lt;/b&gt;&lt;br&gt;</span> <span class="nt">&lt;INPUT</span> <span class="na">TYPE=</span><span class="s">"radio"</span> <span class="na">NAME=</span><span class="s">"onder"</span> <span class="na">VALUE=</span><span class="s">"1a"</span> <span class="nt">&gt;</span> A1 <span class="nt">&lt;INPUT</span> <span class="na">TYPE=</span><span class="s">"radio"</span> <span class="na">NAME=</span><span class="s">"onder"</span> <span class="na">VALUE=</span><span class="s">"2a"</span> <span class="nt">&gt;</span> A2 :: :: <span class="nt">&lt;INPUT</span> <span class="na">TYPE=</span><span class="s">"radio"</span> <span class="na">NAME=</span><span class="s">"onder"</span> <span class="na">VALUE=</span><span class="s">"7a"</span> <span class="nt">&gt;</span> A7<span class="nt">&lt;br&gt;</span> <span class="nt">&lt;/DL&gt;</span> <span class="nt">&lt;br&gt;</span> <span class="nt">&lt;DL&gt;</span> <span class="nt">&lt;DD&gt;&lt;b&gt;</span>Second: wich top, sweater or blouse fits? <span class="nt">&lt;/b&gt;&lt;br&gt;</span> <span class="nt">&lt;INPUT</span> <span class="na">TYPE=</span><span class="s">"radio"</span> <span class="na">NAME=</span><span class="s">"midden"</span> <span class="na">VALUE=</span><span class="s">"1b"</span> <span class="nt">&gt;</span> B1 <span class="nt">&lt;INPUT</span> <span class="na">TYPE=</span><span class="s">"radio"</span> <span class="na">NAME=</span><span class="s">"midden"</span> <span class="na">VALUE=</span><span class="s">"2b"</span> <span class="nt">&gt;</span> B2 :: :: <span class="nt">&lt;INPUT</span> <span class="na">TYPE=</span><span class="s">"radio"</span> <span class="na">NAME=</span><span class="s">"midden"</span> <span class="na">VALUE=</span><span class="s">"7b"</span> <span class="nt">&gt;</span> B7<span class="nt">&lt;br&gt;</span> <span class="nt">&lt;/DL&gt;</span> <span class="nt">&lt;br&gt;</span> <span class="nt">&lt;DL&gt;</span> <span class="nt">&lt;DD&gt;&lt;b&gt;</span>Last: wich earrings and what to do with her hair? <span class="nt">&lt;/b&gt;&lt;br&gt;</span> <span class="nt">&lt;INPUT</span> <span class="na">TYPE=</span><span class="s">"radio"</span> <span class="na">NAME=</span><span class="s">"top"</span> <span class="na">VALUE=</span><span class="s">"1c"</span> <span class="nt">&gt;</span> C1 <span class="nt">&lt;INPUT</span> <span class="na">TYPE=</span><span class="s">"radio"</span> <span class="na">NAME=</span><span class="s">"top"</span> <span class="na">VALUE=</span><span class="s">"2c"</span> <span class="nt">&gt;</span> C2 :: :: <span class="nt">&lt;INPUT</span> <span class="na">TYPE=</span><span class="s">"radio"</span> <span class="na">NAME=</span><span class="s">"top"</span> <span class="na">VALUE=</span><span class="s">"7c"</span> <span class="nt">&gt;</span> C7<span class="nt">&lt;br&gt;</span> <span class="nt">&lt;/DL&gt;&lt;br&gt;</span> <span class="nt">&lt;center&gt;</span> <span class="nt">&lt;INPUT</span> <span class="na">TYPE=</span><span class="s">"submit"</span> <span class="na">VALUE=</span><span class="s">"have a look into the mirror"</span><span class="nt">&gt;</span> <span class="nt">&lt;br&gt;&lt;/center&gt;</span> <span class="nt">&lt;/FORM&gt;</span> </code></pre></div></div> <p>Some key points:</p> <ul> <li> <p>The form sets three variables: “onder”, “midden” and “top”. Each of these can have 7 pre-defined values (with ranges [1a, 7a], [1b, 7b] and [1c, 7c], respectively).</p> </li> <li> <p>The data that are entered in the form are sent to the server using a <a href="https://en.wikipedia.org/wiki/POST_(HTTP)">POST</a> request.</p> </li> </ul> <p>With warcio, an individual set of input combinations can be captured like this:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="n">warcio.capture_http</span> <span class="kn">import</span> <span class="n">capture_http</span> <span class="kn">import</span> <span class="n">requests</span> <span class="n">url</span> <span class="o">=</span> <span class="sh">'</span><span class="s">http://ziklies.home.xs4all.nl</span><span class="sh">'</span> <span class="n">warcFile</span> <span class="o">=</span> <span class="sh">'</span><span class="s">ziklies.home.xs4all.nl.gz</span><span class="sh">'</span> <span class="k">with</span> <span class="nf">capture_http</span><span class="p">(</span><span class="n">warcFile</span><span class="p">):</span> <span class="n">requests</span><span class="p">.</span><span class="nf">post</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="p">{</span><span class="sh">'</span><span class="s">onder</span><span class="sh">'</span><span class="p">:</span> <span class="sh">'</span><span class="s">1a</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">midden</span><span class="sh">'</span><span class="p">:</span> <span class="sh">'</span><span class="s">1b</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">top</span><span class="sh">'</span><span class="p">:</span> <span class="sh">'</span><span class="s">1c</span><span class="sh">'</span><span class="p">})</span> </code></pre></div></div> <p>Note how the <em>data</em> parameter holds a dictionary with the three variables and their associated values. To capture the form’s full behavior, we can simply iterate over all input combinations, and then capture each of them. Having confirmed that this worked, I re-wrote my existing wget-based Bash script into a Python script that only uses warcio. The script is <a href="https://github.com/KBNLresearch/xs4all-resources/blob/master/scripts/scrape-ziklies-local.py">available here</a>. This approach doesn’t work for the interactive toilet door, since, unlike the bedroom mirror, it processes free text, which means that the number of possible inputs is infinite.</p> <h2 id="rendering-the-warc-with-pywb">Rendering the WARC with Pywb</h2> <p>I finally verified the WARC capture by importing it in <a href="https://github.com/webrecorder/pywb">Pywb</a>. More details on this can be found in my separate <a href="https://github.com/KBNLresearch/xs4all-resources/blob/master/doc/liesbets-atelier-warc-notes.md">WARC capture and rendering notes</a>. Rendering the WARC did not result in any problems, and to illustrate this, below screenshot shows one output combination of the interactive bedroom mirror:</p> <figure class="image"> <img src="/images/2020/06/mirror-pywb.png" alt="Mirror script screenshot" /> <figcaption>Output of interactive bedroom mirror script, rendered from WARC capture.</figcaption> </figure> <h2 id="conclusions">Conclusions</h2> <p>Although none of the procedures described in this blog post are particularly complex, the restoration of Liesbet’s Virtual Home involved a lot of trial and error, which ultimately made it a fairly laborious process. Nevertheless, this effort is justified given the value and historical significance of this homepage. It also involved some decisions that, from an authenticity point of view, are open to criticism. An example is the replacement of the missing server-side image maps by the more modern client-side variety. In any case, the experience gained from this project will also be useful for an upcoming attempt to restore a collection of old corporate websites from source data <a href="/2019/09/09/recovering-90s-data-tapes-experiences-kb-web-archaeology">that we recovered from data tapes last year</a>.</p> <h2 id="acknowledgements">Acknowledgements</h2> <p>I would like to thank Liesbet Zikkenheimer for making the offline data of Liesbet’s Virtual Home available to us for this project. Jak Boumans is thanked for alerting us to this unique homepage, and for establishing the contact with its creator. Ilya Kreymer is thanked for his suggestion on warcio, and more generally for creating the Webrecorder software suite, without which much of this work would simply be impossible. Thanks are also due to Kees Teszelszky. Finally, this work was financially supported by the <a href="https://www.sidn.nl/en">SIDN Fund</a> and <a href="https://www.stichtinginternet4all.nl/">Stichting Internet4all</a>.</p> <h2 id="additional-resources">Additional resources</h2> <h3 id="detailed-working-notes">Detailed working notes</h3> <p>The following notes provide more details on the restoration steps, the Apache server setup and the WARC capture process, respectively:</p> <ul> <li> <p><a href="https://github.com/KBNLresearch/xs4all-resources/blob/master/doc/liesbets-atelier-restoration-notes.md">Liesbet’s atelier restoration notes</a></p> </li> <li> <p><a href="https://github.com/KBNLresearch/xs4all-resources/blob/master/doc/liesbets-atelier-apache-notes.md">Liesbet’s Atelier Apache setup notes</a></p> </li> <li> <p><a href="https://github.com/KBNLresearch/xs4all-resources/blob/master/doc/liesbets-atelier-warc-notes.md">Liesbet’s atelier WARC capture and rendering notes</a></p> </li> </ul> <h3 id="processing-scripts">Processing scripts</h3> <p>All processing scripts used as part of this work are <a href="https://github.com/KBNLresearch/xs4all-resources/tree/master/scripts">available here</a>.</p> <h3 id="more-information-about-the-xs4all-homepages-rescue-initiative">More information about the XS4ALL homepages rescue initiative</h3> <p>Below posts (both in Dutch) give some additional background information about the XS4ALL homepages rescue initiative:</p> <ul> <li> <p><a href="https://www.kb.nl/blogs/duurzame-toegang/bewaren-voor-iedereen-de-opbouw-van-de-webcollectie-xs4all-homepages">Bewaren voor iedereen: de opbouw van de webcollectie XS4ALL-homepages</a></p> </li> <li> <p><a href="https://www.sidnfonds.nl/nieuws/redden-wat-van-waarde-is-webarchivering-homepages-xs4all">Redden wat van waarde is: webarchivering homepages XS4ALL</a></p> </li> </ul> <h2 id="revision-history">Revision history</h2> <ul> <li>8 July 2020: added sections on toilet door restoration attempt</li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:2"> <p>Some purists may consider this a technological anachronism. Client-side image maps were first introduced in HTML 3.2, which was published in 1997, whereas Liesbet’s <a href="https://ziklies.home.xs4all.nl/new.html">“what’s new” page</a> shows that most of the site’s development activity took place between early 1995 and late 1997. This could be a problem for users who want to view the site in a period browser (e.g. inside an emulated environment), which may not support client-side image maps. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>Actually as a composite of 3 images. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>Under Linux this is simply a matter of issuing a command like <code class="language-plaintext highlighter-rouge">chmod 755 barbie.cgi</code>. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:6"> <p>Documented here: <a href="https://www.xs4all.nl/service/installeren/hosting/mail-a-form-toevoegen/">https://www.xs4all.nl/service/installeren/hosting/mail-a-form-toevoegen/</a>. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:8"> <p>I’m not actually sure if this behavior was any different on late-‘90s browsers. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:9"> <p>I did try to open the page using <a href="http://oldweb.today/">oldweb.today</a>, but none of the browser environments I tried had the necessary Shockwave plugin installed. <a href="#fnref:9" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2020/06/30/restoring-liesbets-virtual-home https://bitsgalore.org/2020/06/30/restoring-liesbets-virtual-home 2020-06-30T00:00:00+02:00 <figure class="image"> <img src="/images/2020/04/scream.jpg" alt="Scream" /> <figcaption><a href="https://commons.wikimedia.org/wiki/File:%27The_Scream%27,_undated_drawing_Edvard_Munch,_Bergen_Kunstmuseum.JPG">“The Scream”</a>, undated drawing by Edvard Munch, Bergen Kunstmuseum, Public domain.</figcaption> </figure> <p>Earlier this week Library of Congress added <a href="https://www.loc.gov/preservation/digital/formats/fdd/fdd000519.shtml">a new entry on the standard “Digital publishing — EPUB3 preservation” (ISO/IEC TS 22424)</a> to its excellent Digital Formats web site. This standard was developed by the <a href="https://www.iso.org/committee/45374.html">ISO Technical Committee on Document description and processing languages</a>, and was published in January this year (2020).</p> <p>According to its authors, “the ISO/IEC TS 22424 series supports long-term preservation of EPUB publications via a dual strategy”. The standard is made up of 2 parts, which are sold as separate documents on the ISO website:</p> <ol> <li> <p><a href="https://www.iso.org/standard/73163.html">Part 1: Principles (ISO/IEC TS 22424-1:2020)</a></p> </li> <li> <p><a href="https://www.iso.org/standard/73169.html">Part 2: Metadata requirements ISO/IEC TS 22424-2:2020</a></p> </li> </ol> <p>In this blog post I will take a closer look at both parts of the standard. What do they purport, what is their scope, and to what degree do they live up to their stated promises? Readers who are only interested in the most important findings may want to jump to the “Summary and discussion” section at the end of this post.</p> <!-- more --> <p>As I didn’t have access to the final as-published versions of the documents, this review is based on a combination of the (non-paywalled) preview documents on the ISO website, and a pre-publication “Final text” from September 2019. The latter may be marginally different from the published standard, but I don’t expect these differences will affect the general conclusions of this review.</p> <h2 id="part-1---principles">Part 1 - Principles</h2> <p>Even though the full standard documents are behind a paywall, a <a href="https://www.iso.org/obp/ui/#iso:std:iso-iec:ts:22424:-1:ed-1:v1:en">free preview of Part 1 is available here</a>. It contains the introductory chapter, as well as the full table of contents.</p> <h3 id="purpose-and-scope">Purpose and scope</h3> <p>The introductory chapter clearly defines the purpose of this part of the standard:</p> <blockquote> <p>This document facilitates the long-term preservation of EPUB publications by specifying in general level EPUB features which are mandatory for long-term preservation (such as font embedding) and features which should be avoided if possible.</p> <p>This document can be seen as a stepping stone towards a detailed specification which would be related to EPUB in the same way as PDF/A, specified in ISO 19005-1 to ISO 19005-3, is related to the Portable Document Format (PDF).</p> </blockquote> <p>This suggests that the purpose of Part 1 of the standard is make some first steps towards a “preservation-friendly” EPUB profile, analogous to the <a href="https://en.wikipedia.org/wiki/PDF/A">PDF/A</a> profiles for the PDF format.</p> <p>According to the authors, long-term preservation requires:</p> <ol> <li>making the object such as [sic] EPUB publication fit for preservation, including features to be used and features to avoid;</li> <li>packaging the object (and any metadata related to it) together with any additional data such as other versions of the object and other documentation into an Open Archival Information System (OAIS) submission information package (SIP).</li> </ol> <p>They then write that “packaging is covered in ISO/IEC TS 22424-2” (Part 2 of the standard).</p> <h3 id="part-1-is-actually-about-packaging-well-mostly">Part 1 is actually about packaging (well, mostly)</h3> <p>Imagine my surprise then when I found the two main chapters of Part 1 to be titled “Packaging standards” (with a discussion of <a href="https://wiki.dpconline.org/index.php?title=2.2.3_INFORMATION_PACKAGE_VARIANTS">OAIS Information Packages</a>), and “Construction of OAIS information packages”. The latter chapter contains an exhaustive list of requirements and recommendations on how EPUB documents should be packaged in an OAIS Submission Information Package (SIP). So, on the surface it looks like the content that is supposed to be in Part 2 of the standard (packaging) somehow ended up in Part 1 instead!</p> <p>However, a more in-depth look reveals that the generic packaging requirements (which are unrelated to EPUB) are mixed with requirements that actually <em>are</em> specific to EPUB. For example, section 6.2.1 (“EPUB publications SHALL be sent to a repository system as well‐formed and complete Submission Information Packages”) lists requirements on validity against the EPUB specification, font embedding, multimedia content, remote resources and encryption. Confusingly, these are interspersed with other requirements on descriptive metadata and submission agreements (which are not EPUB-specific). Moreover, most of the packaging-related requirements and recommendations mentioned here are totally unrelated to EPUB, and would apply generically to any given format. This makes me wonder why these are part of this standard at all.</p> <h3 id="some-requirements-already-covered-by-epub-specification">Some requirements already covered by EPUB specification</h3> <p>In addition, some of the EPUB-specific requirements are already covered by the EPUB format specification. For example, section 6.4 (“Structure of information packages”) lists requirements on the presence of a manifest file inside an EPUB, the use of the EPUB Open Container Format, the presence of a “container.xml” file, and metadata inside the EPUB package and navigation documents. Any EPUB file that is valid against the EPUB format specification (which is required as per section 6.2.1) already satisfies these requirements, so their inclusion here is unnecessary.</p> <h2 id="part-2---metadata-requirements">Part 2 - Metadata requirements</h2> <p>For Part 2, there is again a <a href="https://www.iso.org/obp/ui/#iso:std:73169:en">free preview</a> with an introductory chapter and the full table of contents.</p> <h3 id="purpose-and-scope-1">Purpose and scope</h3> <p>The introduction defines the purpose of this part of the standard as follows:</p> <blockquote> <p>This document facilitates the long-term preservation of EPUB publications by specifying metadata elements which are required or recommended for long-term preservation (such as identifiers) and the ways in which the EPUB publication and related metadata can be packaged. EPUB versions 3 and 3.0.1 are covered; if necessary, the EPUB version applicable is specified.</p> </blockquote> <p>So, this suggests this is all about metadata and packaging. In the “Scope” section the authors also say this:</p> <blockquote> <p>This document makes EPUB compliant with current practices of Open Archival Information Systems (OAIS) archives and technical requirements of repository systems. The former tend to rely on OAIS in their operations; the latter prefer to ingest electronic documents only in containers conforming to standards such as METS (Metadata Encoding and Transmission Standard).</p> </blockquote> <p>This statement is remarkable, since OAIS does not include any requirements about file formats, so the concept of “OAIS-compliant” file formats is meaningless!</p> <h3 id="sip-level-metadata-vs-epub-level-metadata">SIP-level metadata vs EPUB-level metadata</h3> <p>The main body of Part 2 contains requirements related to various types of metadata. For example, Chapter 6 covers “Packaging metadata”, which the authors describe as follows:</p> <blockquote> <p>This chapter covers mainly metadata about the SIP (container) which is usually submitted using METS elements and attributes.</p> </blockquote> <p>However, it turns out that some sections within this chapter are actually about metadata <em>within</em> EPUB files. Examples are section 6.6 (“Core media type resource identifiers”), which defines requirements on identifiers used in the EPUB manifest (which is part of the EPUB file), and section 6.7 (“Foreign resource identifiers”).</p> <p>Like Part 1, some of the EPUB-specific requirements that are listed here are already covered by the EPUB specification. For example, section 7.2 (“Technical metadata”) states:</p> <blockquote> <p>EPUB version used SHALL be specified in the package element of the EPUB publication’s content.opf file.</p> </blockquote> <p>Since ‘version’ is a mandatory attribute of an EPUB 3 Package Document, the inclusion of this requirement is unnecessary. Similarly, section 7.4 (“Structural metadata”) states:</p> <blockquote> <p>EPUB Open Container Format (OCF) SHALL be used to describe the structure of [sic] EPUB publication.</p> </blockquote> <p>Again, this is true for <em>any</em> valid EPUB file, so why is this included in this standard?</p> <h3 id="oais-packages-again">OAIS packages (again!)</h3> <p>The final two chapters of Part 2 are about the structure and content of OAIS Submission Information Packages. But OAIS information packages are already discussed at length in Part 1 (even though that discussion probably shouldn’t be there to begin with!), so why is this subject revisited here?</p> <h2 id="summary-and-discussion">Summary and discussion</h2> <p>Since this blog post turned out a lot lengthier than I originally intended, this section summarizes and then discusses the most important findings. Overall, I was really surprised to find so many serious issues in a standard that has been authored over a 3-year period by an ISO-coordinated committee, especially given that most of these issues are obvious from even a cursory inspection of the standards documents!</p> <h3 id="lack-of-structure">Lack of structure</h3> <p>Based on the abstracts on the ISO web site and the introductory sections in the standards documents I expected the following:</p> <ul> <li>Part 1: requirements and recommendations that are a first step in the direction of a preservation-friendly EPUB profile, analogous to PDF/A for the PDF format.</li> <li>Part 2: requirements and recommendations on the packaging of publications in EPUB format into OAIS Submission Information Packages (SIPs), along with accompanying metadata.</li> </ul> <p>Instead, Part 1 of the standard is mostly about packaging, but elements of a “preservation-friendly” EPUB profile are buried within sections with package requirements and recommendations. On the other hand, Part 2 (which is supposed to be about packaging and metadata) also contains additional EPUB-specific requirements that one would expect to be part of the EPUB profile (that is, Part 1). Moreover, some of the EPUB-specific requirements are already covered by the EPUB format specification, which makes their inclusion in ISO/IEC TS 22424 unnecessary.</p> <p>The overall effect of this is that elements of the “dual strategy” that underlies ISO/IEC TS 22424 are now scattered across both parts of the standard in a seemingly arbitrary manner. This makes using and navigating the standard unnecessarily difficult. Since both parts of the standard are only available upon payment of CHF 118 and CHF 138, respectively, it also means that buyers of any of these documents will receive a product that is substantially different from its description on the ISO web site!</p> <h3 id="path-towards-a-preservation-friendly-epub-profile">Path towards a preservation-friendly EPUB profile</h3> <p>The introduction of Part 1 of the standard mentions that the EPUB-specific requirements here are only a “stepping stone” towards a more detailed specification (analogous to PDF/A), that could be developed at a later stage by the EPUB community. Even though the authors should be commended for their modesty in this regard, it does make me wonder if such an early-stages effort merits an ISO-published standard. I’m also not convinced of the effectiveness of a “stepping stone” document that is hidden behind a paywall. I do think a preservation-friendly EPUB profile would be useful, and I also think that various people from the digital libraries and archives sector would be willing to contribute to it. But only if this is done in an open way, and not by some arcane ISO-guided process that first keeps the progress hidden from its potential users for 3 years, and then buries the results behind a paywall (which appears to have been the case for the current standard).</p> <h3 id="should-packaging-even-be-part-of-this-standard">Should packaging even be part of this standard?</h3> <p>Although I’ve only given the packaging and package-level metadata requirements and recommendations a cursory look, on the surface many of them are generally applicable to any digital content, and are not specific to publications in EPUB format. At the same time, package-level requirements are typically strongly tied to institutional policies. For instance, Part 2 of the standard requires that the SIP-level METS and PREMIS metadata describe all resources that are inside an EPUB container, which seems a bit excessive to me. This all makes me wonder if OAIS-packaging should be part of this standard at all.</p> <h3 id="no-epub-32">No EPUB 3.2!</h3> <p>It is also worth noting that the current standard only covers EPUB 3 and EPUB 3.0.1 (which was published in 2014). It does not cover the current EPUB 3.2, but it does mention that this version will be covered in the next edition of the standard<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. This is surprising, given that EPUB 3.2 was already published in May 2019. It also means that the standard was already outdated at its date of publication.</p> <h2 id="additional-resources">Additional resources</h2> <ul> <li> <p><a href="https://www.iso.org/standard/73163.html">Digital publishing — EPUB3 preservation — Part 1: Principles</a></p> </li> <li> <p><a href="https://www.iso.org/standard/73169.html">Digital publishing — EPUB3 preservation — Part 2: Metadata requirements</a></p> </li> <li> <p><a href="https://www.loc.gov/preservation/digital/formats/fdd/fdd000519.shtml">EPUB (Electronic publication) Version 3 Preservation. ISO/IEC TS 22424:2020 (Library of Congress Sustainability of Digital Formats)</a></p> </li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>EPUB 3.1 is not covered either, but this is understandable, since this version was never supported by any reader or validation software. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2020/04/30/iso-iec-ts-22424-standard-on-epub3-preservation https://bitsgalore.org/2020/04/30/iso-iec-ts-22424-standard-on-epub3-preservation 2020-04-30T00:00:00+02:00 <figure class="image"> <img src="/images/2020/03/broken-zip.jpg" alt="Broken zip £12.50" /> <figcaption><a href="https://www.flickr.com/photos/dichohecho/3810699621/">“Broken zip £12.50”</a> by <a href="https://www.flickr.com/photos/dichohecho/">dichoecho</a>, used under <a href="https://creativecommons.org/licenses/by/2.0/">CC BY</a> / Cropped from original.</figcaption> </figure> <p>We recently started using <a href="https://en.wikipedia.org/wiki/OneDrive">Microsoft OneDrive</a> at work. The other day a colleague used OneDrive to share a folder with a large number of ISO images with me. Since I wanted to work with these files on my Linux machine at home, and no official OneDrive client for Linux exists a this point, I used OneDrive’s web client to download the contents of the folder. Doing so resulted in a 6 GB <a href="https://en.wikipedia.org/wiki/Zip_(file_format)">ZIP</a> archive. When I tried to extract this ZIP file with my operating system’s (Linux Mint 19.3 MATE) archive manager, this resulted in an error dialog, saying that “An error occurred while loading the archive”:</p> <p><img src="/images/2020/03/archive-manager-onedrive.png" alt="" /></p> <p>The output from the underlying extraction tool (<a href="https://en.wikipedia.org/wiki/7-Zip"><em>7-zip</em></a>) reported a “Headers Error”, with an “Unconfirmed start of archive”. It also reported a warning that “There are data after the end of archive”. No actual data were extracted whatsoever. This all looked a bit worrying, so I decided to have a more in-depth look at this problem.</p> <!-- more --> <h2 id="extracting-with-unzip">Extracting with Unzip</h2> <p>As a first test I tried to extract the file from the terminal using <em>unzip</em> (v. 6.0) using the following command:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>unzip kb-4d8a2f9a-5e0b-11ea-9376-40b0341fbf5f.zip </code></pre></div></div> <p>This resulted in the following output:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Archive: kb-4d8a2f9a-5e0b-11ea-9376-40b0341fbf5f.zip warning [kb-4d8a2f9a-5e0b-11ea-9376-40b0341fbf5f.zip]: 1859568605 extra bytes at beginning or within zipfile (attempting to process anyway) error [kb-4d8a2f9a-5e0b-11ea-9376-40b0341fbf5f.zip]: start of central directory not found; zipfile corrupt. (please check that you have transferred or created the zipfile in the appropriate BINARY mode and that you have compiled UnZip properly) </code></pre></div></div> <p>So, according to <em>unzip</em> the file is simply corrupt. <em>Unzip</em> wasn’t able to extract any actual data.</p> <h2 id="extracting-with-7-zip">Extracting with 7-zip</h2> <p>Next I tried to to extract the file with <em>7-zip</em> (v. 16.02) using this command:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>7z x kb-4d8a2f9a-5e0b-11ea-9376-40b0341fbf5f.zip </code></pre></div></div> <p>This resulted in the following (lengthy) output:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21 p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,4 CPUs Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz (506E3),ASM,AES-NI) Scanning the drive for archives: 1 file, 6154566547 bytes (5870 MiB) Extracting archive: kb-4d8a2f9a-5e0b-11ea-9376-40b0341fbf5f.zip ERRORS: Headers Error Unconfirmed start of archive WARNINGS: There are data after the end of archive -- Path = kb-4d8a2f9a-5e0b-11ea-9376-40b0341fbf5f.zip Type = zip ERRORS: Headers Error Unconfirmed start of archive WARNINGS: There are data after the end of archive Physical Size = 4330182775 Tail Size = 1824383772 ERROR: CRC Failed : kb-4d8a2f9a-5e0b-11ea-9376-40b0341fbf5f/afd3f61a-5e0e-11ea-ab97-40b0341fbf5f/08.wav Sub items Errors: 1 Archives with Errors: 1 Warnings: 1 Open Errors: 1 Sub items Errors: 1 </code></pre></div></div> <p>Here we see the familiar “Headers Error” and “Unconfirmed start of archive” errors, as well as a warning about a <a href="https://en.wikipedia.org/wiki/Cyclic_redundancy_check">cyclic redundancy check</a> that failed on an extracted file. Unlike <em>unzip</em>, <em>7-zip</em> does succeed in extracting some of the data, but seeing that the size of the extracted folder is only 4.3 GB, the extraction is incomplete (the size of the ZIP file is 6 GB!).</p> <h2 id="4-gib-size-limit-and-zip64">4 GiB size limit and ZIP64</h2> <p>At this point I started wondering if these issues could be related to the <em>size</em> of this particular ZIP file, especially since I have been able to process zipped OneDrive folders before without any problems. The <a href="https://en.wikipedia.org/wiki/Zip_(file_format)#ZIP64">Wikipedia entry on <em>ZIP</em></a> states that originally the format had a 4 GiB limit on the total size of the archive (as well as both the uncompressed and compressed size of a file). To overcome these limitations, a “ZIP64” extension was added to the format in <a href="https://web.archive.org/web/20011203085830/http://www.pkware.com/support/appnote.txt">version 4.5 of the ZIP specification</a> (which was published in 2001). To be sure, I verified that both <em>unzip</em> and <em>7-zip</em> on my machine support ZIP64<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>.</p> <h2 id="small-onedrive-zip-and-home-rolled-large-zip">Small OneDrive ZIP and home-rolled large ZIP</h2> <p>I did some additional tests to verify if my problem could be a ZIP64-related issue. First I downloaded a smaller (&lt;4 GB) folder from OneDrive, and tried to extract the resulting ZIP file with <em>unzip</em> and <em>7-zip</em>. Both were able to extract the file without any issues. Next I created two 8 GB ZIP files from data on my local machine with both the <em>zip</em> and <em>7-zip</em> tools. I then tried to extract both files with both <em>unzip</em> and <em>7-zip</em> (i.e. I extracted each file with both tools). Again, both extracted these files without any problems. Since these tests demonstrate that both <em>unzip</em> and <em>7-zip</em> are able to handle both large ZIP files (which by definition use ZIP64) as well as smaller OneDrive ZIP files, this suggests that something odd is going on with OneDrive’s implementation of ZIP64.</p> <h2 id="testing-the-zip-file-integrity">Testing the ZIP file integrity</h2> <p>The <em>zip</em> tool has a switch that can be used to test the integrity of a ZIP file. I ran it on the problematic file like this:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>zip <span class="nt">-T</span> kb-4d8a2f9a-5e0b-11ea-9376-40b0341fbf5f.zip </code></pre></div></div> <p>Here’s the result:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Could not find: kb-4d8a2f9a-5e0b-11ea-9376-40b0341fbf5f.z01 Hit c (change path to where this split file is) q (abort archive - quit) or ENTER (try reading this split again): </code></pre></div></div> <p>So, apparently the <em>zip</em> utility thinks this is a multi-volume archive (which it isn’t). Running this command on any of my other test files (the small OneDrive file, and the large files created by <em>zip</em> and <em>7-zip</em>) didn’t result in any errors.</p> <h2 id="tests-with-pythons-zipfile-module">Tests with Python’s zipfile module</h2> <p>The Python programming language by default includes a <a href="https://docs.python.org/3/library/zipfile.html"><em>zipfile</em></a> module, which has tools for reading and writing ZIP files. So, I wrote the following script, which opens the ZIP file in read mode, and then reads its contents (I used Python 3.6.9 for this):</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="n">zipfile</span> <span class="c1"># Open ZIP file </span><span class="n">myZip</span> <span class="o">=</span> <span class="n">zipfile</span><span class="p">.</span><span class="nc">ZipFile</span><span class="p">(</span><span class="sh">"</span><span class="s">kb-4d8a2f9a-5e0b-11ea-9376-40b0341fbf5f.zip</span><span class="sh">"</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="sh">'</span><span class="s">r</span><span class="sh">'</span><span class="p">)</span> <span class="c1"># Read all files in archive and check their CRCs and file headers. </span><span class="n">myZip</span><span class="p">.</span><span class="nf">testzip</span><span class="p">()</span> <span class="c1"># Close the ZIP file </span><span class="n">myZip</span><span class="p">.</span><span class="nf">close</span><span class="p">()</span> </code></pre></div></div> <p>Running the script raised the following error:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>zipfile.BadZipFile: zipfiles that span multiple disks are not supported </code></pre></div></div> <p>This looks somewhat related to the outcome of <em>zip</em>’s integrity test, which reported a multi-volume archive. Looking at the <a href="https://github.com/python/cpython/blob/d3af92ecc2f41d920e9a66211e2ab631fc473163/Lib/zipfile.py#L232">source code of the zipfile module</a> shows that this particular error is raised if a check on 2 data fields from the “zip64 end of central dir locator” fails:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="n">diskno</span> <span class="o">!=</span> <span class="mi">0</span> <span class="ow">or</span> <span class="n">disks</span> <span class="o">&gt;</span> <span class="mi">1</span><span class="p">:</span> <span class="k">raise</span> <span class="nc">BadZipFile</span><span class="p">(</span><span class="sh">"</span><span class="s">zipfiles that span multiple disks are not supported</span><span class="sh">"</span><span class="p">)</span> </code></pre></div></div> <p>Here’s the description of this data structure in the <a href="https://pkware.cachefly.net/webdocs/APPNOTE/APPNOTE-6.3.6.TXT">format specification</a>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>zip64 end of central dir locator signature 4 bytes (0x07064b50) number of the disk with the start of the zip64 end of central directory 4 bytes relative offset of the zip64 end of central directory record 8 bytes total number of disks 4 bytes </code></pre></div></div> <p>Python’s <em>zipfile</em> raises the error if either the value of the “number of the disk with the start of the zip64 end of central directory” field (variable <em>diskno</em>) isn’t equal to 0, or the “total number of disks” (variable <em>disks</em>) is larger than 1. So, I opened the file in a Hex editor, and zoomed in on the “zip64 end of central dir locator”:</p> <p><img src="/images/2020/03/onedrive-hex.png" alt="" /></p> <p>Here, the highlighted bytes (<code class="language-plaintext highlighter-rouge">0x504b0607</code>) make up the signature of the “zip64 end of central dir locator”<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>. The 4 bytes inside the blue rectangle contain the “number of the disk” value. Here, its value is 0, which is the correct and expected value. The 4 bytes inside the red rectangle contain the “total number of disks” value, which is also 0. But this is really odd, since neither value should trigger the “zipfiles that span multiple disks are not supported” error! Also, a check on the 8 GB ZIP files that I had created myself with <em>zip</em> and <em>7-zip</em> showed both to have a value of 1 for this field. So what’s going on here?</p> <h2 id="digging-into-zipfiles-history">Digging into zipfile’s history</h2> <p>The most likely explanation I could think of, was some difference between my local version of the Python <em>zipfile</em> module and the latest published version on Github. Using Github’s <a href="https://help.github.com/en/github/managing-files-in-a-repository/tracking-changes-in-a-file">blame view</a>, I inspected the revision history of the part of the check that raises the error. This revealed <a href="https://github.com/python/cpython/commit/ab0716ed1ea2957396054730afbb80c1825f9786">a recent change to <em>zipfile</em></a>: prior to a patch that was submitted in May 2019, the offending check was done slightly differently:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="n">diskno</span> <span class="o">!=</span> <span class="mi">0</span> <span class="ow">or</span> <span class="n">disks</span> <span class="o">!=</span> <span class="mi">1</span><span class="p">:</span> <span class="k">raise</span> <span class="nc">BadZipFile</span><span class="p">(</span><span class="sh">"</span><span class="s">zipfiles that span multiple disks are not supported</span><span class="sh">"</span><span class="p">)</span> </code></pre></div></div> <p>Note that in the old situation the test would fail if <em>disks</em> was any value other than 1, whereas in the new situation it only fails if <em>disks</em> is greater than 1. Given that for our OneDrive file the value is 0, this explains why the old version results in the error. The Git commit of the patch also includes the following note:</p> <blockquote> <p>Added support for ZIP files with disks set to 0. Such files are commonly created by builtin tools on Windows when use ZIP64 extension.</p> </blockquote> <p>So could this be the vital clue we need to solve this little file format mystery? Re-running my Python test script with the latest version of the <em>zipfile</em> module did not result in any reported errors, so this looked hopeful for a start. But is the 0 value of “total number of disks” also the thing that makes unzip and 7-zip choke?</p> <h2 id="hacking-into-the-onedrive-zip-file">Hacking into the OneDrive ZIP file</h2> <p>To put this to the test, I first made a copy of the OneDrive ZIP file. I opened this file in a Hex editor, and did a search on the hexadecimal string <code class="language-plaintext highlighter-rouge">0x504b0607</code>, which is the signature that indicates the start of the “zip64 end of central dir locator”<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>. I then changed the first byte of the “total number of disks” value (this is the 13th byte after the signature, indicated by the red rectangle in the screenshot) from <code class="language-plaintext highlighter-rouge">0x00</code> to <code class="language-plaintext highlighter-rouge">0x01</code>:</p> <p><img src="/images/2020/03/onedrive-hex-fixed.png" alt="" /></p> <p>This effectively sets the “total number of disks” value to 1 (unsigned little-endian 32-bit value). After saving the file, I repeated all my previous tests with <em>unzip</em>, <em>7-zip</em>, as well as <em>zip</em>’s integrity check. The modified ZIP file passed all these tests without any problems! The contents of the file could be extracted normally, and the extraction is also complete. The file can also be opened normally in Linux Mint’s archive manager, as this screenshot shows:</p> <p><img src="/images/2020/03/onedrive-fixed-archive-manager.png" alt="" /></p> <p>So, it turns out that the cause of the problem is the value of one field in the “zip64 end of central dir locator”, which can be provisionally fixed by nothing more than changing one single bit!</p> <h2 id="other-reports-on-this-problem">Other reports on this problem</h2> <p>If a widely-used platform by a major vendor such as Microsoft produces ZIP archives with major interoperability issues, I would expect others to have run into this before. <a href="https://askubuntu.com/questions/1115238/unable-to-extract-onedrive-zip-file-on-ubuntu-18-10">This question on <em>Ask Ubuntu</em></a> appears to describe the same problem, and <a href="https://onedrive.uservoice.com/forums/913528-onedrive-on-the-web/suggestions/35321278-fix-the-corrupted-zip-download-or-be-honest-about">here’s another report on corrupted large ZIP files</a> on a OneDrive-related forum. On Twitter Tyler Thorsted <a href="https://twitter.com/CHLThor/status/1237416651995283457">confirmed</a> my results for a 5 GB ZIP file downloaded from OneDrive, adding that the Mac OS X archive utility didn’t like the file either. Still, I’m surprised I couldn’t find much else on this issue.</p> <h2 id="similarity-to-apple-archive-utility-problem">Similarity to Apple Archive utility problem</h2> <p>The problem looks superficially similar to an older issue with Apple’s Archive utility, which would write corrupt ZIP archives for cases where the ZIP64 extension is needed. From the <a href="https://en.wikipedia.org/wiki/Zip_(file_format)#ZIP64">WikiPedia entry on <em>ZIP</em></a>:</p> <blockquote> <p>Mac OS Sierra’s Archive Utility notably does not support ZIP64, and can create corrupt archives when ZIP64 would be required.</p> </blockquote> <p>More details about this are available <a href="https://web.archive.org/web/20140331005235/http://www.springyarchiver.com/blog/topic/topic/203">here</a>, <a href="https://apple.stackexchange.com/questions/221020/large-zip-files-created-in-os-x-cannot-be-opened-in-windows">here</a> and <a href="https://github.com/thejoshwolfe/yauzl/issues/69">here</a>. Interestingly a <a href="https://twitter.com/kieranjol/status/1200118816686137344">Twitter thread by Kieran O’Leary</a> put me on track of this issue (which I hadn’t heard of before). It’s not clear to me if the OneDrive problem is identical or even related, but because of the similarities I thought it was at least worth a mention here.</p> <h2 id="conclusion">Conclusion</h2> <p>The tests presented here demonstrate how large ZIP files exported from the Microsoft OneDrive web client cannot be read by widely-used tools such as <em>unzip</em> and <em>7-zip</em>. The problem only occurs for large (&gt; 4 GiB) files that use the ZIP64 extension. The cause of this interoperability problem is the value of the “total number of disks” field in the “zip64 end of central dir locator”. In the OneDrive files, this value is set to 0 (zero), whereas most reader tools expect a value of 1. It is debatable whether the OneDrive files violate the <a href="https://pkware.cachefly.net/webdocs/APPNOTE/APPNOTE-6.3.6.TXT">ZIP format specification</a>, since the spec doesn’t say anything about the permitted values of this field. Affected files can be provisionally “fixed” by changing the first byte of the “total number of disks” field in a hex editor. However, to ensure that existing files that are affected by this issue remain accessible in the long term, we need a more structural and sustainable solution. It is probably fairly trivial to modify existing ZIP reader tools and libraries such as <em>unzip</em> and <em>7-zip</em> to deal with these files. I’ll try to get in touch with the developers of some of these tools about this issue. Ideally things should also be fixed on Microsoft’s end. If any readers have contacts there, please bring this post to their attention!</p> <h2 id="test-file">Test file</h2> <p>I’ve created an openly-licensed test file that demonstrates the problem. It is available here:</p> <p><a href="https://zenodo.org/record/3715394">https://zenodo.org/record/3715394</a></p> <h2 id="update-17-march-2020">Update (17 March 2020)</h2> <p>For <em>unzip</em> I found <a href="https://sourceforge.net/p/infozip/bugs/42/">this ticket</a> on the <em>Info-Zip</em> issue tracker, which looks identical to the problem discussed in this post. The ticket was already created in 2013, but its current status is not entirely clear.</p> <p>For <em>7-zip</em>, things are slightly complicated by the fact that for Unix <a href="https://sourceforge.net/projects/p7zip/">a separate <em>p7zip</em> port</a> exists, which currently is 3 major releases behind the <a href="https://sourceforge.net/projects/sevenzip/">main 7-zip</a> project. In any case, I’ve just opened <a href="https://sourceforge.net/p/p7zip/feature-requests/46/">this feature request</a> in the <em>p7zip</em> issue tracker.</p> <p>Meanwhile Andy Jackson has been trying to <a href="https://twitter.com/anjacks0n/status/1238852027045883904">get this issue to the attention of Microsoft</a>, so let’s see what happens from here.</p> <h2 id="fix-onedrive-zip-script-update-8-june-2020">Fix-OneDrive-Zip script (update 8 June 2020)</h2> <p>In the comments section, Paul Marquess posted a link to a small Perl script he wrote that automatically updates the “total number of disks” field of a problematic OneDrive ZIP file. The script is available here:</p> <p><a href="https://github.com/pmqs/Fix-OneDrive-Zip">https://github.com/pmqs/Fix-OneDrive-Zip</a></p> <p>I ran a quick test with my openly-licensed test file, using the following command:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>fix-onedrive-zip onedrive-zip-test-zeros.zip </code></pre></div></div> <p>After running the script, the file was indeed perfectly readable. Thanks Paul!</p> <h2 id="revision-history">Revision history</h2> <ul> <li>14 March 2020: added analysis with Python <em>zipfile</em>, and updated conclusions accordingly.</li> <li>17 March 2020: added update with links to Info-Zip and p7zip issue trackers.</li> <li>18 March 2020: added link to test file.</li> <li>8 June 2020: added reference to <em>Fix-OneDrive-Zip</em> script by Paul Marquess.</li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>For unzip you can check this this by running it with the <code class="language-plaintext highlighter-rouge">--version</code> switch. If the output includes <code class="language-plaintext highlighter-rouge">ZIP64_SUPPORT</code> this means ZIP64 is supported. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>Note that this is the big-endian representation of the signature, whereas the ZIP formation specification uses little-endian representations. See more on endianness <a href="https://en.wikipedia.org/wiki/Endianness">here</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>Since the “zip64 end of central dir locator” is located near the end of the file, the quickest way to find it is to scroll to the very end of the file in the Hex editor, and then do a reverse search (“Find Previous”) from there. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2020/03/11/does-microsoft-onedrive-export-large-ZIP-files-that-are-corrupt https://bitsgalore.org/2020/03/11/does-microsoft-onedrive-export-large-ZIP-files-that-are-corrupt 2020-03-11T00:00:00+01:00 <figure class="image"> <img src="/images/2020/02/carriers-stillife.jpg" alt="Still life of assorted data carriers" /> </figure> <p>Following earlier work on the preservation of optical media and data tapes, I recently got a request to make an inventory of offline digital data carriers in the KB’s deposit collection. The goal was to obtain approximate figures on the various carrier types in the collection. This was partially prompted by <a href="https://www.netwerkdigitaalerfgoed.nl/activiteiten/digitaal-erfgoed-houdbaar/bedreigd-digitaal-erfgoed-op-fysieke-dragers/">a project on at-risk digital heritage on physical carriers</a> by the <a href="https://www.netwerkdigitaalerfgoed.nl/en/">Dutch Digital Heritage Network</a> (NDE) that the KB is participating in. This blog post presents the results.</p> <!-- more --> <h2 id="data-carriers-in-the-kb-catalogue">Data carriers in the KB catalogue</h2> <p>The KB catalogue is the primary source of information about carriers in our collection. It is fully searchable using the <a href="https://www.loc.gov/standards/sru/">SRU protocol</a>.</p> <h2 id="carrier-type-codes-not-based-on-controlled-vocabulary">Carrier type codes not based on controlled vocabulary</h2> <p>The main piece of information to go by here is the value of the Dublin Core <em>extent</em> field<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. However, in our catalogue the values of this field are not set from a controlled vocabulary, which means that one particular carrier type can be represented by different values. For example, here are some variations I found for CD-ROMs:</p> <ul> <li>cdrom</li> <li>cd-rom</li> <li>1 cd-rom</li> <li>2 cd-roms</li> </ul> <p>For 5.25” floppy disks I came across:</p> <ul> <li>1 diskette 5.25”</li> <li>1 floppydisk 5.25”</li> <li>1 floppydisk 5.25” in doos</li> </ul> <p>This lack of consistency makes searching for specific carrier types rather difficult, and as a result most of the queries I used for this inventory are the result of trial and error.</p> <h2 id="catalogue-record-can-represent-multiple-carriers">Catalogue record can represent multiple carriers</h2> <p>In the examples above you may have noticed how in some cases the value represents multiple carriers (e.g. “2 cd-roms”). This is because our catalogue provides access at the level of publications, but not at the level of individual carriers that are part of a publication! A typical example to illustrate this, is a physical book with 2 supplemental CD-ROMs. In this case, the catalogue record represents the whole publication, and the <em>extent</em> field will have a value like “2 cd-roms”. As a result, the catalogue cannot be readily used to get precise figures on carrier types. In most cases the number of carriers for a specific carrier type will be considerably greater than the number of matching records, since an individual record (publication) may include multiple carriers. This is not really a problem for the current exercise, as its objective is limited to getting approximate figures only.</p> <h2 id="results">Results</h2> <p>I summarised the results in the table below. Here I made a subdivision between optical, magnetic and electronic carrier types. The table contains 4 columns:</p> <ul> <li> <p><strong>Carrier type:</strong> the types listed here largely follow those proposed for an upcoming survey that will be launched as part of the NDE project on at-risk digital heritage on physical carriers (with some additions).</p> </li> <li> <p><strong>Category:</strong> this indicates the carrier type category (optical, magnetic or electronic).</p> </li> <li> <p><strong>Query:</strong> this is the <a href="https://www.loc.gov/standards/sru/cql/">SRU</a> query I used to estimate the number of catalogue records for this carrier type. Follow the underlying hyperlinks to run the query yourself. Note that this field is empty for carrier types that -to the best of our knowledge- are not present in our deposit collection.</p> </li> <li> <p><strong>Matching records:</strong> this is the number of matching records in the catalogue. As explained in the previous section, this does not necessarily reflect the actual number of carriers, which may well be greater by a factor 2 (or even more).</p> </li> </ul> <table> <thead> <tr> <th style="text-align: left">Carrier type</th> <th style="text-align: left">Category</th> <th style="text-align: left">Query</th> <th style="text-align: left">Matching records</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">CD-ROM</td> <td style="text-align: left">Optical</td> <td style="text-align: left"><a href="http://jsru.kb.nl/sru/sru?query=extent%20any%20&quot;cdrom*%20cd-rom*&amp;x-collection=GGC&amp;maximumRecords=10">extent any “cdrom* cd-rom*”</a></td> <td style="text-align: left">8109</td> </tr> <tr> <td style="text-align: left">DVD</td> <td style="text-align: left">Optical</td> <td style="text-align: left"><a href="http://jsru.kb.nl/sru/sru?query=extent%20any%20&quot;dvd*&quot;&amp;x-collection=GGC&amp;maximumRecords=10">extent any “dvd*”</a></td> <td style="text-align: left">711</td> </tr> <tr> <td style="text-align: left">Blu-Ray</td> <td style="text-align: left">Optical</td> <td style="text-align: left"><a href="http://jsru.kb.nl/sru/sru?query=extent%20any%20&quot;bluray*%20blu-ray*&quot;&amp;x-collection=GGC&amp;maximumRecords=10">extent any “bluray* blu-ray*”</a></td> <td style="text-align: left">4</td> </tr> <tr> <td style="text-align: left">Audio CD</td> <td style="text-align: left">Optical</td> <td style="text-align: left"><a href="http://jsru.kb.nl/sru/sru?query=type%20any%20&quot;geluidsdrager&quot;%20and%20extent%20any%20&quot;cd*%20compact&quot;&amp;x-collection=GGC&amp;maximumRecords=10">type any “geluidsdrager” and extent any “cd* compact”</a></td> <td style="text-align: left">4605</td> </tr> <tr> <td style="text-align: left">CD-i</td> <td style="text-align: left">Optical</td> <td style="text-align: left"><a href="http://jsru.kb.nl/sru/sru?query=extent%20any%20&quot;cdi*%20cd-i*&quot;&amp;x-collection=GGC&amp;maximumRecords=10">extent any “cdi* cd-i*”</a></td> <td style="text-align: left">44</td> </tr> <tr> <td style="text-align: left">Optical carrier, unspecified</td> <td style="text-align: left">Optical</td> <td style="text-align: left"><a href="http://jsru.kb.nl/sru/sru?query=extent%20any%20&quot;optisch*%20schijf&quot;&amp;x-collection=GGC&amp;maximumRecords=10">extent any “optisch* schijf”</a></td> <td style="text-align: left">23</td> </tr> <tr> <td style="text-align: left">Tape, unspecified</td> <td style="text-align: left">Magnetic</td> <td style="text-align: left"><a href="http://jsru.kb.nl/sru/sru?query=extent%20any%20&quot;tape*&quot;&amp;x-collection=GGC&amp;maximumRecords=10">extent any “tape*”</a></td> <td style="text-align: left">2</td> </tr> <tr> <td style="text-align: left">Compact cassette, data (datassette)</td> <td style="text-align: left">Magnetic</td> <td style="text-align: left"><a href="http://jsru.kb.nl/sru/sru?query=extent%20any%20%22cassetteband*%20datacassette*%22&amp;x-collection=GGC&amp;maximumRecords=10">extent any “cassetteband*”</a></td> <td style="text-align: left">6</td> </tr> <tr> <td style="text-align: left">Floppy disk, 8”</td> <td style="text-align: left">Magnetic</td> <td style="text-align: left">-</td> <td style="text-align: left">0</td> </tr> <tr> <td style="text-align: left">Floppy disk, 5.25”</td> <td style="text-align: left">Magnetic</td> <td style="text-align: left"><a href="http://jsru.kb.nl/sru/sru?query=extent%20any%20&quot;flop*%20diskette*&quot;%20and%20extent%20any%20&quot;5.25*%205,25*&quot;&amp;x-collection=GGC&amp;maximumRecords=10">extent any “flop* diskette*” and extent any “5.25* 5,25*”</a></td> <td style="text-align: left">184</td> </tr> <tr> <td style="text-align: left">Floppy disk, 3.5”</td> <td style="text-align: left">Magnetic</td> <td style="text-align: left"><a href="http://jsru.kb.nl/sru/sru?query=extent%20any%20&quot;flop*%20diskette*&quot;%20and%20extent%20any%20&quot;3.5*%203,5*&quot;&amp;x-collection=GGC&amp;maximumRecords=10">extent any “flop* diskette*” and extent any “3.5* 3,5*”</a></td> <td style="text-align: left">1194</td> </tr> <tr> <td style="text-align: left">Floppy disk, unspecified</td> <td style="text-align: left">Magnetic</td> <td style="text-align: left"><a href="http://jsru.kb.nl/sru/sru?query=extent%20any%20&quot;flop*%20diskette*&quot;%20not%20extent%20any%20&quot;5.25*%203.5*&quot;&amp;x-collection=GGC&amp;maximumRecords=10">extent any “flop* diskette*” not extent any “5.25* 3.5*”</a></td> <td style="text-align: left">822</td> </tr> <tr> <td style="text-align: left">Zip Disk</td> <td style="text-align: left">Magnetic</td> <td style="text-align: left"><a href="http://jsru.kb.nl/sru/sru?query=extent%20any%20&quot;zipdis*%20zip-dis*&quot;&amp;x-collection=GGC&amp;maximumRecords=10">extent any “zipdis* zip-dis*”</a></td> <td style="text-align: left">0</td> </tr> <tr> <td style="text-align: left">Hard disk</td> <td style="text-align: left">Magnetic</td> <td style="text-align: left"><a href="http://jsru.kb.nl/sru/sru?query=extent%20any%20&quot;hard*&quot;%20and%20extent%20any%20&quot;schijf%20schijv*%20disk*&quot;&amp;x-collection=GGC&amp;maximumRecords=10">extent any “hard*” and extent any “schijf schijv* disk*”</a></td> <td style="text-align: left">2</td> </tr> <tr> <td style="text-align: left">Compact Flash card</td> <td style="text-align: left">Electronic</td> <td style="text-align: left">-</td> <td style="text-align: left">0</td> </tr> <tr> <td style="text-align: left">Sony Memory stick</td> <td style="text-align: left">Electronic</td> <td style="text-align: left">-</td> <td style="text-align: left">0</td> </tr> <tr> <td style="text-align: left">SD card</td> <td style="text-align: left">Electronic</td> <td style="text-align: left">-</td> <td style="text-align: left">0</td> </tr> <tr> <td style="text-align: left">USB thumb drive</td> <td style="text-align: left">Electronic</td> <td style="text-align: left"><a href="http://jsru.kb.nl/sru/sru?query=extent%20any%20&quot;usb*&quot;&amp;x-collection=GGC&amp;maximumRecords=10">extent any “usb*”</a></td> <td style="text-align: left">44</td> </tr> <tr> <td style="text-align: left">Solid-State drive</td> <td style="text-align: left">Electronic</td> <td style="text-align: left">-</td> <td style="text-align: left">0</td> </tr> </tbody> </table> <p>In the following sections I will highlight some of the most interesting observations that can be made from the table.</p> <h2 id="optical-carriers">Optical carriers</h2> <p>Unsurprisingly, optical carriers make up the majority of offline digital carriers in the KB deposit collection. An effort to preserve the contents of these carriers using the <a href="/2017/06/19/image-and-rip-optical-media-like-a-boss"><em>Iromlab</em></a> software is currently ongoing, but this does not cover Blu-Ray discs. At the current numbers we could probably just image them manually.</p> <h2 id="development-of-optical-carriers-since-2015">Development of optical carriers since 2015</h2> <p>Since I had already queried the catalogue for optical carriers <a href="https://zenodo.org/record/292341">as part of an investigation I did in 2015</a>, I also compared the current figures against those in 2015. The following figure shows the result for the most prevalent optical carrier types:</p> <p><img src="/images/2020/02/optical-2015-2020.png" alt="&quot;Optical carrier types, 2015 vs 2020&quot;" /></p> <p>And here in table form:</p> <table> <thead> <tr> <th style="text-align: left">Carrier type</th> <th style="text-align: left">Matching records (2015)</th> <th style="text-align: left">Matching records (2020)</th> <th style="text-align: left">Increase</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">CD-ROM</td> <td style="text-align: left">7980</td> <td style="text-align: left">8109</td> <td style="text-align: left">129</td> </tr> <tr> <td style="text-align: left">Audio CD</td> <td style="text-align: left">4065</td> <td style="text-align: left">4605</td> <td style="text-align: left">540</td> </tr> <tr> <td style="text-align: left">DVD</td> <td style="text-align: left">554</td> <td style="text-align: left">711</td> <td style="text-align: left">157</td> </tr> </tbody> </table> <p>The increase of the number of DVD records (an 28% increase) is particularly noteworthy. In absolute terms the number of audio CD records has increased even more. The growth of the CD-ROM collection has clearly levelled off at an increase of only 129 records over 5 years.</p> <h2 id="magnetic-carriers">Magnetic carriers</h2> <p>In the magnetic carriers category the number of publications with floppy disks is noteworthy. Assuming each of these publications contains between 1 and 2 disks on average, the total number of floppy disks would be in the range 2200 - 4400. The majority of these are 3.5” disks, but for about 38% the exact type cannot be established from the catalogue alone.</p> <p>An interesting curiosity is a handful of compact cassettes with software for the <a href="https://en.wikipedia.org/wiki/ZX_Spectrum"><em>ZX Spectrum</em></a> home computer.</p> <p>We also appear to have 2 tapes of unknown format, and 2 hard disks. These need further investigation.</p> <h2 id="electronic-carriers">Electronic carriers</h2> <p>Here we have 44 USB thumb drives, and we should probably consider imaging them sooner rather than later. We don’t appear to have any of the other carrier types in this category.</p> <h2 id="final-remarks">Final remarks</h2> <p>Despite the obvious limitations of the methodology, the above inventory provides a tentative overview of offline digital carriers in the KB deposit collection that will probably be useful for preservation planning. It might also guide future efforts at saving information on “at risk” carriers.</p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>Oddly, this field is meant to record <a href="https://www.dublincore.org/specifications/dublin-core/dcmi-terms/terms/extent/">“The size or duration of the resource”</a> as per the Dublin Core specification. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2020/02/20/offline-digital-carriers-kb-deposit-collection https://bitsgalore.org/2020/02/20/offline-digital-carriers-kb-deposit-collection 2020-02-20T00:00:00+01:00 <figure class="image"> <img src="/images/2020/02/domains-provinces-map.png" alt="Map of domain locations" /> </figure> <p>A few weeks ago one of my web archiving colleagues approached me with an interesting question. From a list of Dutch web domains, he wanted to identify the (Dutch) province in which each domain is hosted. He was particularly interested in domains hosted in the province of <a href="https://en.wikipedia.org/wiki/Friesland">Friesland</a>. After some experimentation I was able to answer this question using a two-step procedure:</p> <ol> <li>Geo-locate the web domains using a custom Python script.</li> <li>Combine the results of the geolocation exercise with openly available geographical data using <a href="https://en.wikipedia.org/wiki/QGIS"><em>QGIS</em></a>, an open-source geographical information system (GIS).</li> </ol> <p>Even though the outcome of the analysis is not particularly interesting, I imagine both the geolocation methodology and the GIS analysis steps might be useful to others. So, this blog post is primarily intended as a tutorial that gives a walkthrough of the steps I followed.</p> <!-- more --> <h2 id="web-domain-geolocation">Web domain geolocation</h2> <p>In Python, the <a href="https://github.com/maxmind/GeoIP2-python"><em>GeoIP2</em></a> module can be used to get geolocation data (latitude, longitude, but also country and city names) for any IP address. IP addresses can be queried through either a web service, or using a local database. I went for the local database option, which requires you to download an up-to-date version of the <em>GeoLite2</em> City database, which can downloaded from <a href="https://dev.maxmind.com/geoip/geoip2/geolite2">the <em>MaxMind</em> developer site</a>. To download the database you first need to create an account (which is free). Note that this database is widely used; for example, the British Library used it for geo-locating their 2014 domain crawl<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>.</p> <h2 id="ip-lookup">IP lookup</h2> <p>Since the <em>GeoIP2</em> module expects IP addresses as input, we first need to establish the IP address that is associated with each web domain in our list. In theory it should be possible to do IP lookups natively in <em>Python</em> (e.g. see <a href="https://stackoverflow.com/questions/6422907/ip-address-by-domain-name">here</a>). However, after running into various problems, I ultimately wrote a simple wrapper function around the Unix <a href="https://linux.die.net/man/1/host"><em>host</em> command</a>. Unfortunately this does mean the script will only run in Unix/Linux environments.</p> <h2 id="geolocation-script">Geolocation script</h2> <p>The <a href="https://gist.github.com/bitsgalore/b05c73934aece90e5a1b2a53fcce6f5b">complete geolocation script is available here</a>. It takes three command-line arguments:</p> <ol> <li>An input file. This is a text file, with each line containing one web domain. It has the following format (note that these are not valid URLs because they don’t include a leading scheme/protocol component): <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kruisbandkliniek.nl anima-communicatie.nl nieuwebrabander.nl isisschuurman.nl wamail.nl kopenzonderklussen.nl muziekles-laren.nl </code></pre></div> </div> </li> <li>An output file.</li> <li>The location of the database file.</li> </ol> <p>The output file is a comma-delimited text file. For each domain, it reports:</p> <ol> <li>A <em>Boolean</em> flag that indicates whether the domain can be mapped to an IP address.</li> <li>A country ISO code (if available).</li> <li>A city name (if available).</li> <li>Latitude and longitude (if available).</li> </ol> <h2 id="qgis">QGIS</h2> <p>After I ran the script on the list of Dutch domain names, I noticed that city name values were often missing. However, latitude and longitude were always available (provided that a domain can be mapped to an IP address), so that’s what we’ll work with here. In order to answer the question that started this whole exercise (“which domains are hosted in the province of Friesland”), we need to do some additional spatial analysis. <a href="https://en.wikipedia.org/wiki/Geographic_information_system">Geographic Information Systems</a> (GIS) are a class of software that are specifically suited to this. So, I downloaded and installed the free and open-source <a href="https://qgis.org/en/site/">QGIS</a> software.</p> <p>A general discussion of QGIS is beyond the scope of this blog post; however the <em>Programming Historian</em> website has <a href="https://programminghistorian.org/en/lessons/qgis-layers">a good introduction to QGIS</a><sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>. For the analyses covered in this blog post I used QGIS 2.8.6 (which is the slightly outdated version that is in the default Ubuntu software repositories).</p> <h2 id="visualise-the-point-data">Visualise the point data</h2> <p>As a first step, let’s visualise the output of the geolocation script. Launch QGIS, then from the <em>Layer</em> menu select <em>Add Layer</em> / <em>Add Delimited Text Layer …</em>:</p> <p><img src="/images/2020/02/add-text-layer.png" alt="" /></p> <p>This opens up a dialog window where you can select the output file of the geolocation script. There are also some options you can set here, but in this case you can stick to the default values:</p> <p><img src="/images/2020/02/textlayer-dialog.png" alt="" /></p> <p>Under <em>Geometry definition</em> you can select the columns with the X- and Y-co-ordinates. As it turns out, QGIS is clever enough to figure these out by itself from the column headings in the file. Click <em>OK</em> to start importing the file. Note that if any records in the file don’t contain latitude/longitude values, you will see an error message like this:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Errors in file /home/johan/test-geolocatDomains/out_NL_geo_Johan.csv 594 records discarded due to missing geometry definitions </code></pre></div></div> <p>It is safe to ignore this, as any remaining records are imported correctly. After closing the error message, you need to specify the coordinate reference system in the following dialog<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>:</p> <p><img src="/images/2020/02/crs-points.png" alt="" /></p> <p>Since the point data file uses geographic coordinates (latitude/longitude pairs), select <em>WGS 84</em> (which is also the default) and press <em>OK</em>. If all goes well you will now see the imported points:</p> <p><img src="/images/2020/02/points-imported.png" alt="" /></p> <p>Note that you can zoom in to parts of the map view for a more detailed look. Before going any further, this is a good moment to save our QGIS project; to do so open the <em>Project</em> menu on the left, select <em>Save</em>, type a file name and the click on the <em>Save</em> button. Done!</p> <h2 id="add-a-provinces-layer">Add a provinces layer</h2> <p>The objective of this exercise was to link web domains to provinces. A free, generalised vector layer of Dutch province boundaries is available <a href="https://www.nationaalgeoregister.nl/geonetwork/srv/dut/catalog.search#/metadata/e73b01f6-28c7-4bb7-a782-e877e8113e2c">here</a> in a number of formats. I downloaded the <a href="https://en.wikipedia.org/wiki/GeoPackage">GeoPackage</a> version (which is essentially an SQLite database). After unzipping the file, go to the <em>Layers</em> menu and select <em>Add Vector Layer …</em>.:</p> <p><img src="/images/2020/02/add-vector.png" alt="" /></p> <p>Then open the downloaded GeoPackage<sup id="fnref:6"><a href="#fn:6" class="footnote" rel="footnote" role="doc-noteref">4</a></sup> file in the dialog that appears:</p> <p><img src="/images/2020/02/add-vector-2.png" alt="" /></p> <p>After opening the file, the map view looks like this:</p> <p><img src="/images/2020/02/provinces-after-import.png" alt="" /></p> <p>Note that the provinces layer is visible as a tiny purple blob at the center of the image! The reason for this is that the default zoom level of the map view covers the whole geographical extent of the domain points data, which apparently includes locations all over the world! So let’s change that. In the <em>Layers</em> panel, hover over the provinces layer, right-click and then select <em>Zoom to Layer</em>:</p> <p><img src="/images/2020/02/zoom-provinces.png" alt="" /></p> <p>Result:</p> <p><img src="/images/2020/02/zoomed-provinces.png" alt="" /></p> <p>As you see, provinces layer now obscures the point data. You can fix this by dragging the provinces layer to the bottom in the <em>Layers</em> panel:</p> <p><img src="/images/2020/02/points-top.png" alt="" /></p> <h2 id="customise-appearance-of-provinces-layer">Customise appearance of provinces layer</h2> <p>Now let’s improve the appearance of the provinces layer. In the <em>Layers</em> panel, hover over the provinces layer, right-click and then select <em>Properties</em>:</p> <p><img src="/images/2020/02/layer-properties.png" alt="" /></p> <p>In the dialog that appears, first click on <em>Style</em> in the left pane. Click on <em>Single Symbol</em> in the drop-down menu, change it to <em>Categorized</em>, and then click on the <em>Apply</em> button:</p> <p><img src="/images/2020/02/style-1.png" alt="" /></p> <p>Now click on the <em>Column</em> drop-down menu, and select the column (attribute/field) that represents the map classes. In this case I used the <em>PROV_CODE</em> column. Next, click on the <em>Classify</em> button, which automatically assigns colours and legend values:</p> <p><img src="/images/2020/02/categorize.png" alt="" /></p> <p>Let’s also add some labels. Click on <em>Labels</em> in the left pane, and tick the <em>Label this layer with</em> checkbox. Use the drop-down menu to select the column (attribute/field) that is used to assign the labels. In my case I used the <em>PROV_NAAM</em> (“province name”) column:</p> <p><img src="/images/2020/02/labels.png" alt="" /></p> <p>Click on the <em>Apply</em> button again, and then on <em>OK</em>. Our map view now looks like this:</p> <p><img src="/images/2020/02/provinces-points.png" alt="" /></p> <p>Which is pretty decent! You can use the <em>Save as Image …</em> item in the <em>Project</em> menu to export the map layout into a variety of image formats.</p> <h2 id="spatial-analysis">Spatial analysis</h2> <p>Having a pretty-looking map is nice, but in order to know which domains are hosted in which provinces we need to do some actual analysis. More specifically, for each web domain (point) we need to extract the corresponding attributes from the provinces vector layer. This is one of the most basic functionalities of QGIS. However, most of the analysis tools of QGIS require that all input layers have the same coordinate reference system. This is not the case here: the geographical coordinates of our point data are defined as latitude/longitude pairs, whereas the provinces vector layer uses the <a href="https://www.spatialreference.org/ref/epsg/amersfoort-rd-new/">Amersfoort / RD New</a> coordinate system (which is commonly used for maps of the Netherlands)!</p> <h2 id="transform-point-data-to-coordinate-system-of-provinces-layer">Transform point data to coordinate system of provinces layer</h2> <p>So, before we go any further we first need to transform our point data to the province layer’s coordinate system<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>. In the <em>Layers</em> panel, hover over the points layer and right-click <em>Save As …</em>:</p> <p><img src="/images/2020/02/points-save-as.png" alt="" /></p> <p>Set <em>Format</em> to <em>Geography Markup Language [GML]</em> and specify a file name and location. In the <em>CRS</em> dropdown menu, select the coordinate system of the provinces layer (here: <em>Amersfoort / RD New</em>). Leave all other options to their default values:</p> <p><img src="/images/2020/02/points-save-as-2.png" alt="" /></p> <p>After pressing <em>OK</em> the new layer is automatically added to the map view.</p> <p>While at it, let’s also change the project coordinate system to <em>Amersfoort / RD New</em><sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">6</a></sup>: in the <em>Project</em> menu, click on <em>Project Properties …</em>. In the dialog that now appears, click on <em>Amersfoort / RD New</em> in the <em>Recently used coordinate reference systems</em> pane, and then press <em>Apply</em>:</p> <p><img src="/images/2020/02/set-project-crs.png" alt="" /></p> <p>Now press <em>OK</em>, and you should view something like this:</p> <p><img src="/images/2020/02/mapview-amersfoort.png" alt="" /></p> <h2 id="join-attributes-of-point-and-province-layers">Join attributes of point and province layers</h2> <p>We’re now ready to combine the attributes of the point and vector layers. From the <em>Vector</em> menu, select <em>Data Management Tools</em>, and then <em>Join Attributes by Location …</em>:</p> <p><img src="/images/2020/02/join-attributes-1.png" alt="" /></p> <p>Next, set the <em>Target vector layer</em> to the layer with the (transformed) point data, and the <em>Join vector layer</em> to the provinces vector layer. Leave <em>Attribute Summary</em> to its default value (<em>Take attributes of first located feature</em>), specify an output file and set the <em>Output table</em> setting to <em>Keep all records</em>:</p> <p><img src="/images/2020/02/join-attributes-2.png" alt="" /></p> <p>Now press <em>OK</em>. After a while this dialog pops up:</p> <p><img src="/images/2020/02/layertotoc.png" alt="" /></p> <p>Click <em>Yes</em>, and then close the <em>Join Attributes</em> dialog window.</p> <h2 id="export-to-text-file">Export to text file</h2> <p>As a final step we export the layer that we created in the previous step to a delimited text file. In the <em>Layers</em> panel, hover over the new layer and right-click <em>Save As …</em>:</p> <p><img src="/images/2020/02/exportjoineddata.png" alt="" /></p> <p>In the dialog that now pops up, set <em>Format</em> to <em>Comma Separated Value</em>, specify a file name and press <em>OK</em>:</p> <p><img src="/images/2020/02/savevectorlayeras.png" alt="" /></p> <p>You can now import the resulting text file in your favourite spreadsheet application. Here’s what it looks like in LibreOffice Calc:</p> <p><img src="/images/2020/02/domains-provinces-spreadheet.png" alt="" /></p> <p>As you can see here, each domain record now contains a column with the corresponding province name (and some additional province-specific columns), and from this we can easily identify the domains that are hosted in Friesland (or any other province):</p> <p><img src="/images/2020/02/domains-fryslan.png" alt="" /></p> <h2 id="final-thoughts">Final thoughts</h2> <p>As I’m originally a geographer by training, this was a nice opportunity to go back back to my roots a bit and muck around with geographical data. The last time I did any serious work with geographic information systems was around 2007; back then even simple analyses like this one needed expensive proprietary software. I hadn’t used QGIS before, and even though I’ve only scratched the surface here it looks like a really useful addition to the toolbox of anyone working with geodata.</p> <h2 id="further-resources">Further resources</h2> <ul> <li> <p><a href="https://gist.github.com/bitsgalore/b05c73934aece90e5a1b2a53fcce6f5b">Python script for geolocation of web domains</a></p> </li> <li> <p><a href="https://programminghistorian.org/en/lessons/qgis-layers">Installing QGIS 2.0 and Adding Layers (the Programming Historian)</a></p> </li> <li> <p><a href="https://docs.qgis.org/3.4/en/docs/gentle_gis_introduction/coordinate_reference_systems.html">Coordinate Reference Systems (QGIS Wiki)</a></p> </li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:5"> <p>Geo-location in the 2014 UK Domain Crawl: <a href="https://blogs.bl.uk/webarchive/2015/07/geo-location-in-the-2014-uk-domain-crawl.html">https://blogs.bl.uk/webarchive/2015/07/geo-location-in-the-2014-uk-domain-crawl.html</a> <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:1"> <p>Note that this is based on a pretty old version of QGIS, so some things may have changed since then. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>A good introduction to coordinate reference systems can be found here: <a href="https://docs.qgis.org/3.4/en/docs/gentle_gis_introduction/coordinate_reference_systems.html">https://docs.qgis.org/3.4/en/docs/gentle_gis_introduction/coordinate_reference_systems.html</a> <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:6"> <p>The other formats will most likely work fine as well, but I just selected GeoPackage, as unlike the Shapefile format it’s an open standard. <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>Or the other way round. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>This step is not necessary for the analysis, but I just did this because I don’t really like the way the map view looks in the WGS 84 coordinate system. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2020/02/11/web-domain-geolocation-and-spatial-analysis https://bitsgalore.org/2020/02/11/web-domain-geolocation-and-spatial-analysis 2020-02-11T00:00:00+01:00 <figure class="image"> <img src="/images/2019/09/tapes-dds-dlt.jpg" alt="DDS-1 (left) and DLT-IV (right) tape" /> </figure> <p>Earlier this year I published <a href="/2019/01/31/roll-the-tape-recovering-90s-data-tapes-in-bitcurator">this blog post</a> on the recovery of data from ’90s data tapes. I will give a presentation on this during <a href="https://ipres2019.org/">the upcoming <em>iPres</em> 2019 conference</a>, and wrote a paper that discusses this work in more detail than my earlier blog post. The original paper (in PDF format) can be found <a href="https://ipres2019.org/static/pdf/iPres2019_paper_9.pdf">here</a>. The paper references a wealth of useful resources, but some of these are not easily accessible because the <a href="https://en.wikipedia.org/wiki/LaTeX"><em>LaTeX</em></a> template used does not handle hyperlinks well (this will be fixed in the final, post-conference version of the paper). Because of this I’ve created a web-friendly version of the paper below.</p> <!-- more --> <hr /> <h2 id="abstract">Abstract</h2> <p>The recovery of digital data from tape formats from the mid to late ’90s is not well covered by existing digital preservation and forensics literature. This paper addresses this knowledge gap with a discussion of the hardware and software that can be used to read such tapes. It introduces <em>tapeimgr</em>, a user-friendly software application that allows one to read tapes in a format-agnostic manner. It also presents workflows that integrate the discussed hardware and software components. It then shows how these workflows were used to recover the contents of a set of DDS-1, DDS-3 and DLT-IV tapes from the mid to late ’90s. These tapes contain the source data of a number of “lost” web sites that the National Library of the Netherlands (KB) is planning to reconstruct at a later stage as part of its ongoing Web Archaeology project. The paper also presents some first results of sites from 1995 that have already been reconstructed from these tapes.</p> <p><strong>Keywords</strong> – tapes, digital forensics, web archaeology</p> <p><strong>Conference Topics</strong> – The Cutting Edge: Technical Infrastructure and Implementation</p> <h2 id="introduction">Introduction</h2> <p>When the National Library of the Netherlands (hereafter: KB) launched its web archive in 2007, many sites from the “early” Dutch web had already gone offline. As a result, the time period between (roughly) 1992 and 2000 is under-represented in the web archive. To improve the coverage of web sites from this historically important era, the KB has started to investigate the use of tools and methods from the emerging field of “web archaeology” <a href="https://doi.org/10.1177/0955749017725930">[1]</a>. Analogous to how archaeologists study past cultures from excavated physical artefacts, web archaeology is about reconstructing “lost” web sites using data that are recovered from old (and often obsolete) physical carriers. It is worth noting that Ross and Gow introduced the concept of “digital archaeology” (of which web archaeology is a special case) as early as 1999 <a href="http://www.ukoln.ac.uk/services/elib/papers/supporting/pdf/p2.pdf">[2]</a>.</p> <p>Over the last year, the KB web archiving team has reached out to a number of creators of “early” Dutch web sites that are no longer online. Many of these creators still possess offline information carriers with the original source data of their sites. This would potentially allow us to reconstruct those sites, and then ingest them into the web archive. The overall approach would be similar to how we already reconstructed the first Dutch web index <em>NL-Menu</em> in 2018 <a href="https://www.bitsgalore.org/2018/04/24/resurrecting-the-first-dutch-web-index-nl-menu-revisited">[3]</a>, <a href="https://www.bitsgalore.org/2018/07/11/crawling-offline-web-content-the-nl-menu-case">[4]</a>.</p> <p>A few of these creators have already provided us with sample sets of carriers which, though limited in size, comprise a range of physical formats, such as CD-ROMs, floppy disks, ZIP disks, USB thumb drives, and (internal) hard disks. One sample set was provided to us by the former owners of <em>xxLINK</em>, a web development and hosting company that was founded in 1994. It was the first Dutch company that provided these services, and throughout the ’90s <em>xxLINK</em> created web sites for a large number of well-known Dutch companies and institutions<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. A particularly interesting feature of the <em>xxLINK</em> sample set is that it includes 33 data tapes.</p> <p>There is a relative wealth of digital preservation and digital forensics literature on the recovery of data from physical carriers. Examples include Ross and Gow <a href="http://www.ukoln.ac.uk/services/elib/papers/supporting/pdf/p2.pdf">[2]</a>, Elford et al. <a href="http://archive.ifla.org/IV/ifla74/papers/084-Webb-en.pdf">[5]</a>, Woods and Brown <a href="https://kamwoods.net/publications/woodsbrownarch09.pdf">[6]</a>, Woods et al. <a href="http://doi.acm.org/10.1145/1998076.1998088">[7]</a>, Lee et al. <a href="http://www.dlib.org/dlib/may12/lee/05lee.html">[8]</a>, John <a href="http://dx.doi.org/10.7207/twr12-03">[9]</a> and Pennock et al. <a href="https://doi.org/10.5281/zenodo.1321629">[10]</a>. For many carrier types published workflow descriptions are readily available (see e.g. Prael and Wickner <a href="https://practicaltechnologyforarchives.org/issue4_prael_wickner/">[11]</a>, Salo <a href="https://radd.dsalo.info/wp-content/uploads/2017/10/BuildDocumentation.pdf">[12]</a> and the workflows published by the <em>BitCurator Consortium</em> <a href="https://bitcuratorconsortium.org/workflows">[13]</a>, to name but a few). Even though these cover a wide range of physical carrier types, the existing literature provides surprisingly little information on how to recover data from legacy tape formats. One of the few exceptions are De Haan <a href="https://doi.org/10.5281/zenodo.1255965">[14]</a> and De Haan et al. <a href="https://hart.amsterdam/image/2017/11/17/20171116_freeze_diy_handboek.pdf">[15]</a>, who describe how they rescued 11 GB worth of data from three DLT tapes. However, they do not provide much detail about the hardware and software setup they used for this.</p> <p>Reading these legacy tape formats presents a number of challenges. First, it requires specific hardware that is now largely obsolete. This includes not only the actual tape readers, but also host adapters that are needed to connect a tape reader to a modern forensic workstation, cables and adapter plugs. Because of this, finding the “right” hardware setup is often not straightforward. Furthermore, since the original software that was used to write (and read) legacy data tapes may not be available anymore (if it is known at all), the tapes should be read in a format-agnostic way at the block device level. This can be done with existing software tools, but these tools are not very user-friendly, and the resulting workflows can be quite unwieldy. Also, the logical interpretation of data files that have been recovered from tape requires some additional work. Finally, even though there are still various online resources that cover reading these tapes<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>, the information they provide is often fragmentary, or geared towards specific backup software or hardware. This is especially true for older resources that date back to the time when these tape formats were in heavy use.</p> <p>Hence, there appears to be a knowledge gap. The overall aim of this paper is to fill this gap by discussing the hardware and software that can be used to read such tapes, and by presenting practical workflows that allowed us to recover the information from the <em>xxLINK</em> tapes. These workflows are largely based on current hard- and software. They are also fully open source, and can be easily integrated into Linux-based platforms, including the popular <em>BitCurator</em><sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">3</a></sup> environment.</p> <h2 id="outline">Outline</h2> <p>This paper starts with a brief overview of the tape formats in the <em>xxLINK</em> sample set. This is followed by a discussion of the hardware that is needed for accessing tapes like these. This section also provides some suggestions that will hopefully be useful to others who are starting similar tape-related work. It then suggests a format-agnostic procedure for reading the data on the tapes, and presents a new software application that was developed specifically for reading tapes in a simple and user-friendly manner. Next follows a discussion of how this hardware and software setup were integrated into workflows, and how these workflows were used to recover the data on the <em>xxLINK</em> tapes. This is followed by two sections that explain the further processing of the recovered data: the extraction of the resulting container files, and the subsequent reconstruction of any “lost” web sites whose underlying data are enclosed in them. This section also shows some first results of sites that were recovered from a 1995 tape. The closing section summarizes the main conclusions.</p> <h2 id="tape-formats">Tape formats</h2> <p>The majority (19) of the tapes in the <em>xxLINK</em> sample set are DDS tapes, most of which were written in 1995. DDS (Digital Data Storage) is a family of tape formats that are based on Digital Audio Tape (DAT). Using the product codes I was able to identify the majority of these DDS tapes as DDS-1, which was the first generation of DDS. DDS-1 was introduced in 1989, and has a maximum capacity of 2 GB (uncompressed), or 4 GB (compressed). Two tapes could be identified as DDS-3, a format which was introduced in 1996 with a maximum capacity of 12 GB (uncompressed), or 24 GB (compressed). A total of 7 DDS generations have been released, the final one being DAT-320 in 2009<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>. Backward read compatibility of DDS drives is typically limited to 2 or 3 generations<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>. The <em>xxLINK</em> set also contains 14 DLT-IV tapes which were mostly written in 1999. DLT-IV is a member of the Digital Linear Tape (DLT) family of tape formats, which dates back to 1984. DLT-IV was first introduced in 1994<sup id="fnref:6"><a href="#fn:6" class="footnote" rel="footnote" role="doc-noteref">6</a></sup>, and has a capacity of up to 40 GB (uncompressed), or 80 GB (compressed)<sup id="fnref:7"><a href="#fn:7" class="footnote" rel="footnote" role="doc-noteref">7</a></sup>. Figure 1 shows what these tapes look like.</p> <figure class="image"> <img src="/images/2019/09/tapes-dds-dlt.jpg" alt="DDS-1 (left) and DLT-IV (right) tape" /> <figcaption>Figure 1: DDS-1 (left) and DLT-IV (right) tape</figcaption> </figure> <h2 id="hardware">Hardware</h2> <p>For all data recovery workflows that are part of the web archaeology project I set up a dedicated forensic workstation that is running the <em>BitCurator</em> environment. Reading the vintage tape formats in the <em>xxLINK</em> sample set requires some specific additional hardware, most of which can be bought used online at a low to moderate cost. Luckily, it turned out our IT department was still in the possession of an old (DDS-1-compatible) DDS-2 drive, as well as a DLT-IV drive. Both drives are shown in Figure 2.</p> <figure class="image"> <img src="/images/2019/09//tapedrives-dds-dlt.jpg" alt="DLT-IV (bottom) and DDS-2 (top) tape drives" /> <figcaption>Figure 2: DLT-IV (bottom) and DDS-2 (top) tape drives</figcaption> </figure> <p>In order to read the DDS-3 tapes, I purchased an additional used DAT-72 drive that has backward-compatibility with DDS-3.</p> <h3 id="scsi-host-adapter">SCSI host adapter</h3> <p>As all three tape drives have parallel SCSI<sup id="fnref:8"><a href="#fn:8" class="footnote" rel="footnote" role="doc-noteref">8</a></sup> connectors, I needed a SCSI host adapter (“SCSI card”) to connect them to the forensic workstation. Used SCSI cards can be easily found online, and they are usually sold cheaply. Nevertheless, finding a model that was compatible with both our workstation and the tape drives turned out to be somewhat complicated. This is due to a number of reasons.</p> <p>First, SCSI cards often have interfaces that are not compatible with current hardware. Many older models have a conventional PCI interface<sup id="fnref:9"><a href="#fn:9" class="footnote" rel="footnote" role="doc-noteref">9</a></sup>, but PCI has been largely replaced by PCI Express<sup id="fnref:10"><a href="#fn:10" class="footnote" rel="footnote" role="doc-noteref">10</a></sup> on modern motherboards and desktop machines. Some cards have a 64-bit PCI interface, which is only compatible with enterprise servers.</p> <p>Even if the interface is compatible, the physical dimensions of the card can cause further complications. Older “full-height” PCI Express cards will not fit into a “low-profile” (also known as “half-height”) slot, and vice versa (most modern machines only support “low-profile” cards). Many cards were originally sold with both a “full-height” and a “low-profile” bracket, which allows one to easily change the bracket to fit the target machine. Buying second-hand, it is not uncommon to find that either of the original brackets are missing.</p> <p>Online sellers do not always explicitly mention characteristics like these, and even if they do this information is not necessarily accurate. A useful resource in this regard is the web site of the Microsemi company, which has the technical specifications of the full range of Adaptec SCSI adapters<sup id="fnref:11"><a href="#fn:11" class="footnote" rel="footnote" role="doc-noteref">11</a></sup>. Figure 3 shows the PCI Express host adapter that we are using in our workstation.</p> <figure class="image"> <img src="/images/2019/09/scsi-controller.jpg" alt="PCI Express SCSI host adapter" /> <figcaption>Figure 3: PCI Express SCSI host adapter</figcaption> </figure> <h3 id="scsi-connectors-and-terminators">SCSI connectors and terminators</h3> <p>Rather than being one well-defined standard, parallel SCSI is actually a family of related standards that comprise a host of different interfaces, not all of which are mutually compatible<sup id="fnref:12"><a href="#fn:12" class="footnote" rel="footnote" role="doc-noteref">12</a></sup>, <sup id="fnref:13"><a href="#fn:13" class="footnote" rel="footnote" role="doc-noteref">13</a></sup>. None of these standards specify what connectors should be used to interconnect SCSI devices. Over time, this has resulted in a myriad of connector types that have been developed by different manufacturers<sup id="fnref:14"><a href="#fn:14" class="footnote" rel="footnote" role="doc-noteref">14</a></sup>. These are typically identified by multiple names. As an example, the commonly used 68-pin “DB68” connector is also referred to as “MD68”, “High-Density”, “HD 68”, “Half-Pitch” and “HP68”, whereas the “50-contact, Centronics-type” connector is alternatively known as a “SCSI-1” or “Alternative 2, A-cable connector”. This complicates both identifying the connector type on a particular device, as well as finding suitable adapter plugs and cables. For identifying a connector, the web site of Paralan provides a useful illustrated overview of the most common types<sup id="fnref:15"><a href="#fn:15" class="footnote" rel="footnote" role="doc-noteref">15</a></sup>.</p> <p>If the tape reader is the last device at either end of the SCSI chain, it must be fitted with a “terminator”<sup id="fnref:16"><a href="#fn:16" class="footnote" rel="footnote" role="doc-noteref">16</a></sup>, which is a resistor circuit that prevents the electrical signal from reflecting back from the ends of the bus. Without a terminator, the tape drive will not work properly, or, more likely, it will not work at all. External SCSI devices like our tape drives use terminator plugs, as shown in Figure 4. For internal devices, termination is often achieved through jumper settings, or by physically removing the terminating resistors from their sockets<sup id="fnref:17"><a href="#fn:17" class="footnote" rel="footnote" role="doc-noteref">17</a></sup>.</p> <figure class="image"> <img src="/images/2019/09/terminator.jpg" alt="SCSI terminator attached to DLT-IV drive" /> <figcaption>Figure 4: SCSI terminator attached to DLT-IV drive</figcaption> </figure> <h3 id="cleaning-cartridges">Cleaning cartridges</h3> <p>Over time, the heads of a tape drive will get dirty due to a gradual accumulation of dust, and sometimes also residue from the tapes that are used. As this ultimately results in read errors, it is important to periodically clean the drive with a dedicated cleaning cartridge. Most drives have an indicator that lights up when cleaning is needed. The cleaning procedure is usually very simple, and involves nothing more than inserting the cleaning cartridge into the machine, which then automatically starts the cleaning cycle. A single cleaning cartridge can be used multiple (typically about 50) times. Although I was able to purchase cleaning cartridges for both the DDS and the DLT-IV drives online, it is unclear whether new cartridges are still manufactured today. Since it is not easy or even recommended to clean these drives manually (in fact this is likely to result in damage), the availability of cleaning cartridges could be a concern for keeping older tape formats like these accessible in the long run.</p> <h2 id="software">Software</h2> <p>Once the hardware is set up, a number of options are available for reading the data from the tapes. Often, tapes contain backup archives that were written by backup utilities such as <em>tar</em>, <em>cpio</em>, <em>dump</em> or <em>NTBackup</em>, to name but a few. One approach would be to restore the contents of each tape using the original software that was used to write it. Even though many of these utilities are still available today (especially the Unix-based ones), this approach is not a practical one. First of all, it would require prior knowledge of the tape’s archive format. Although we may sometimes have this knowledge (e.g., the writing on a tape’s label may indicate it was created with the <em>tar</em> utility), in practice we often simply don’t know how a tape was written at all. Also, this approach would complicate things, because each format would require its own custom workflow. Finally, it would not work with formats for which the original software is not readily available on the forensic workstation (e.g. the Microsoft Tape Format that was used by Windows <em>NTBackup</em>).</p> <p>A better approach is to use tools like <em>dd</em><sup id="fnref:18"><a href="#fn:18" class="footnote" rel="footnote" role="doc-noteref">18</a></sup> which are able to read data directly at the block device level. This way, tapes can be read in a format-agnostic manner. The general idea here is that we use <em>dd</em> to read all archive files on a tape, irrespective of their format. We then use format identification tools such as <em>file(1)</em><sup id="fnref:19"><a href="#fn:19" class="footnote" rel="footnote" role="doc-noteref">19</a></sup>, <em>Apache Tika</em><sup id="fnref:20"><a href="#fn:20" class="footnote" rel="footnote" role="doc-noteref">20</a></sup>, <em>Fido</em><sup id="fnref:21"><a href="#fn:21" class="footnote" rel="footnote" role="doc-noteref">21</a></sup> or <em>Siegfried</em><sup id="fnref:22"><a href="#fn:22" class="footnote" rel="footnote" role="doc-noteref">22</a></sup> to establish the format of each archive file, and subsequently use dedicated, format-specific utilities to extract their contents. This is similar to existing forensic workflows that are used for other carrier types in e.g. <em>BitCurator</em>.</p> <h3 id="reading-a-tape-with-dd-and-mt">Reading a tape with dd and mt</h3> <p>In the simplest case, reading data from a tape involves nothing more than a <em>dd</em> command line such as this one:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">dd </span><span class="k">if</span><span class="o">=</span>/dev/nst0 <span class="nv">of</span><span class="o">=</span>file0001.dd <span class="nv">bs</span><span class="o">=</span>16384 </code></pre></div></div> <p>Here, the “<code class="language-plaintext highlighter-rouge">if</code>” argument tells <em>dd</em> to read input from the non-rewind block device <code class="language-plaintext highlighter-rouge">/dev/nst0</code>, and the value of “<code class="language-plaintext highlighter-rouge">of</code>” defines the file where output is written. The “<code class="language-plaintext highlighter-rouge">bs</code>” argument defines a block size (here in bytes), and this is where the first complication arises. The actual value that must be used here depends on the software that was used to write the tape, and its settings. If <em>dd</em> is invoked with a value that is too small, it will fail with a “cannot allocate memory” error. After some experimentation I was able to establish the block size using the following iterative procedure:</p> <ol> <li> <p>Starting with a block size of 512 bytes, try to read one single record (and direct the output to the null device, as we don’t need it):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">dd </span><span class="k">if</span><span class="o">=</span>/dev/nst0 <span class="nv">of</span><span class="o">=</span>/dev/null <span class="nv">bs</span><span class="o">=</span>512 <span class="nv">count</span><span class="o">=</span>1 </code></pre></div> </div> </li> <li> <p>Position the tape 1 record backward using the <em>mt</em><sup id="fnref:23"><a href="#fn:23" class="footnote" rel="footnote" role="doc-noteref">23</a></sup> tool (this resets the read position to the start of the current session):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mt <span class="nt">-f</span> /dev/nst0 bsr 1 </code></pre></div> </div> </li> <li> <p>If step 1 raised an error in <em>dd</em>, increase the block size value by 512 bytes, and repeat from step 1.</p> </li> </ol> <p>Repeating these steps until <em>dd</em> exits without errors will yield the correct block size. Re-running the <em>dd</em> command at the start of this section with this value will recover the first session on the tape to a single output file. This leads to a second complication: a tape may contain additional sessions. We can test for this by positioning the tape 1 record forward with the <em>mt</em> tool:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mt <span class="nt">-f</span> /dev/nst0 fsr 1 </code></pre></div></div> <p>If the <em>mt</em> call doesn’t result in an error (i.e. <em>mt</em>’ s exit code equals zero), at least one additional session exists. In that case we use <em>mt</em> again to position the tape 1 record backward (the start of the second session). We then repeat the block-estimation procedure for the second session, and read the data with <em>dd</em>.</p> <p>All of the above steps are repeated until <em>mt</em>’s <code class="language-plaintext highlighter-rouge">fsr</code> command results in a non-zero exit code, which means no additional sessions exist. The end result is that for each session on the tape we now have a corresponding output file.</p> <h3 id="tapeimgr">Tapeimgr</h3> <p>Even though the above procedure is not particularly complicated, having to go through all these steps by hand would be very cumbersome. Moreover, <em>dd</em>’s ability to overwrite entire block devices with one single command introduces a high risk of accidental data loss (hence its “destroy disk” nickname). Also, it would be useful to have a more user-friendly method for reading data tapes. Because of these reasons, I developed <em>tapeimgr</em><sup id="fnref:24"><a href="#fn:24" class="footnote" rel="footnote" role="doc-noteref">24</a></sup>, which is a software application that allows one to read data tapes using a simple, easy-to-use graphical user interface. Written in Python, it was loosely inspired by the popular <em>Guymager</em> forensic imaging software<sup id="fnref:25"><a href="#fn:25" class="footnote" rel="footnote" role="doc-noteref">25</a></sup>. Internally, <em>tapeimgr</em> just wraps around <em>dd</em> and <em>mt</em>, but it completely hides the complexities of these tools from the user. The software runs on any Linux distribution, and can be installed with <em>pip</em>, Python’s default package manager. Its only dependencies are a recent version of <em>Python</em> (3.2 or more recent), the <em>TkInter</em> package, and <em>dd</em> and <em>mt</em>. All of these are present by default on most modern Linux distros.</p> <p>Figure 5 shows <em>tapeimgr</em>’s interface.</p> <figure class="image"> <img src="/images/2019/09/tapeimgr.png" alt="The tapeimgr interface" /> <figcaption>Figure 5: The tapeimgr interface</figcaption> </figure> <p>At the very minimum, a user must select a directory to which all output for a given tape is written. If necessary, the read process can be further customized using a number of options that are described in detail in the <em>tapeimgr</em> documentation. There are also entry fields for descriptive metadata, and the values that are entered here are written to a metadata file in JSON format. This file also contains some basic event and technical metadata, including SHA-512 checksums of each session (which is represented as a file) that is read from the tape. After the user presses the <em>Start</em> button, the progress of the tape reading procedure can be monitored from a widget at the bottom of the interface; the information displayed here is also written to a log file. When <em>tapeimgr</em> has finished reading a tape, it displays a prompt that tells the user whether the read process completed successfully. In case of any problems, the log file contains detailed information about all steps in the tape reading process. In addition to the graphical user interface, <em>tapeimgr</em> also has a command-line interface, which makes it possible to integrate it into other applications.</p> <h3 id="limitations-of-tapeimgr">Limitations of tapeimgr</h3> <p>At this stage, <em>tapeimgr</em> has two important limitations. First, it only supports tapes for which the block size is constant within each session. More recent tape drives are often capable of writing tapes in “variable block” mode, where the block size within a session varies. This is not supported, although a possible (but so far untested) workaround may be to set the Initial Block Size to some arbitrary large value that is equal to or larger than the overall largest block size on the tape<sup id="fnref:26"><a href="#fn:26" class="footnote" rel="footnote" role="doc-noteref">26</a></sup>. Variations in block size <em>between</em> sessions are no problem, and are fully supported. Also, <em>tapeimgr</em> is not able to recover data from tapes that were partially overwritten. As an example, suppose that a tape originally contained 3 sessions with a size of 200 MB each. If someone later overwrote part of the first session with a 10 MB session at the start of that tape, running the tape through <em>tapeimgr</em> will only recover the first 10 MB session, and ignore the remaining sessions. The reason for this is that each write action adds an “End Of Media” (EOM) marker just beyond the end of the written data, and once an EOM is written, any previously recorded data beyond that point are no longer accessible (reportedly workarounds exist, but these are specific to kernel drivers)<sup id="fnref:27"><a href="#fn:27" class="footnote" rel="footnote" role="doc-noteref">27</a></sup>.</p> <h2 id="reading-the-xxlink-tapes">Reading the xxLINK tapes</h2> <p>With all the hardware and software in place, I first experimented with reading some unimportant, disposable DDS-1 and DLT-IV test tapes. Based on these tests I designed processing workflows, which I then documented by creating detailed descriptions that cover all steps that have to be followed to read a tape<sup id="fnref:28"><a href="#fn:28" class="footnote" rel="footnote" role="doc-noteref">28</a></sup>. Once I was confident that the workflows were sufficiently robust, I applied them to the <em>xxLINK</em> tapes. All but one of the DDS-1 tapes could be read without problems. For one tape, the recovery resulted in a 10-kB file with only null bytes, which most likely means the tape is faulty. The two DDS-3 tapes could be read successfully as well. Most of these tapes contained multiple (up to 4) sessions. Of the 14 DLT-IV tapes, only 7 could be read without problems. For the remaining 7, the reading procedure resulted in a zero-length file, which means the tape drive interprets them as empty. A common characteristic of all “failed” DLT-IV tapes is that they were written at 40 GB capacity, whereas the other tapes were written at 35 GB capacity. This is odd, as our DLT-IV drive does in fact support 40 GB capacity tapes (this was confirmed by writing some data to a blank test tape at 40 GB capacity, which could subsequently be read without problems). Although the exact cause is unknown at this stage, it is possible that these tapes are simply faulty, or perhaps they were erased or overwritten at some point. Interestingly, the label on at least one of the problematic tapes contains some writing that suggests it was already faulty around the time it was written. Table 1 gives a brief summary of the above results.</p> <table> <thead> <tr> <th style="text-align: left"> </th> <th style="text-align: center">DDS-1</th> <th style="text-align: center">DDS-3</th> <th style="text-align: center">DLT-IV</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"># tapes</td> <td style="text-align: center">17</td> <td style="text-align: center">2</td> <td style="text-align: center">14</td> </tr> <tr> <td style="text-align: left"># read successfully</td> <td style="text-align: center">16</td> <td style="text-align: center">2</td> <td style="text-align: center">7</td> </tr> </tbody> </table> <p><em>Table 1: Summary of xxLINK tapes read results</em></p> <h2 id="extraction-of-recovered-container-files">Extraction of recovered container files</h2> <p>It is important to stress that the above <em>tapeimgr</em>-based workflow only recovers the contents of the tapes at the bit level: for each session on the tape it results in one bitstream (file). Additional steps are needed to interpret the recovered bitstreams in a meaningful way. At the very the minimum two additional steps are necessary. First we need to identify the format of the container files that were recovered from the tapes. Once the container format is known, we can (hopefully) find some format-specific software that is capable of extracting the content of the container files. For format identification I ran the Unix <em>file(1)</em> command (v. 5.32)<sup id="fnref:29"><a href="#fn:29" class="footnote" rel="footnote" role="doc-noteref">29</a></sup> on all recovered files. The results are summarized in Table 2.</p> <table> <thead> <tr> <th style="text-align: left">Format (file(1))</th> <th style="text-align: center">Number of files</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">new-fs dump file (big endian)</td> <td style="text-align: center">28</td> </tr> <tr> <td style="text-align: left">new-fs dump file (little endian)</td> <td style="text-align: center">8</td> </tr> <tr> <td style="text-align: left">tar archive</td> <td style="text-align: center">4</td> </tr> <tr> <td style="text-align: left">POSIX tar archive</td> <td style="text-align: center">2</td> </tr> <tr> <td style="text-align: left">POSIX tar archive (GNU)</td> <td style="text-align: center">5</td> </tr> </tbody> </table> <p><em>Table 2: Formats of recovered files according to file(1)</em></p> <p>Most files in the <em>xxLINK</em> data set are Unix <em>dump</em> archives<sup id="fnref:30"><a href="#fn:30" class="footnote" rel="footnote" role="doc-noteref">30</a></sup>. <em>Dump</em> is an old backup utility, and its archive files can be extracted using the <em>restore</em> tool<sup id="fnref:31"><a href="#fn:31" class="footnote" rel="footnote" role="doc-noteref">31</a></sup>. Even though <em>dump</em> and <em>restore</em> are largely obsolete today, the software is still available in the Debian repositories, and as a result these tools can be easily installed on most Linux-based platforms. A few words of caution: first, by default <em>restore</em> extracts the contents of a <em>dump</em> file to the system’s root directory, i.e. it tries to recover a full backup. For our purposes this behaviour is clearly unwanted, and could even wreak havoc on the forensic workstation’s file system. However, extraction to a user-defined directory is possible by running <em>restore</em> in “interactive” mode<sup id="fnref:32"><a href="#fn:32" class="footnote" rel="footnote" role="doc-noteref">32</a></sup>. A disadvantage of having to use the interactive mode is that it makes bulk processing of <em>dump</em> files virtually impossible. This could be a serious problem when one has to deal with very large numbers of these files. Second, it is important to check the file system of the disk to which the container file is extracted. I initially tried to extract the <em>dump</em> files to an <em>NTFS</em>-formatted<sup id="fnref:33"><a href="#fn:33" class="footnote" rel="footnote" role="doc-noteref">33</a></sup> external hard disk. However, it turned out that the names of some files and directories inside the archive were not compatible with <em>NTFS</em>, and as a result these files were not extracted. Since the <em>dump</em> archives were originally created from a Unix-based file system, this is not surprising. Also, any file attributes that are not supported by <em>NTFS</em> (e.g. access permissions and ownership) are lost when extracting to <em>NTFS</em>. Extraction to another disk that was formatted as <em>Ext4</em><sup id="fnref:34"><a href="#fn:34" class="footnote" rel="footnote" role="doc-noteref">34</a></sup> (which is the default file system for most Linux distributions) resolved this issue.</p> <p>The remaining files are all <em>tar</em> archives, a format that is still widely used today. These files can be extracted by simply running the <em>tar</em><sup id="fnref:35"><a href="#fn:35" class="footnote" rel="footnote" role="doc-noteref">35</a></sup> command like this:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">tar</span> <span class="nt">-xvf</span> /path/to/file0001.dd <span class="o">&gt;</span> /dev/null </code></pre></div></div> <p>The earlier observations on the file system of the disk to which the container is extracted also apply to <em>tar</em> files.</p> <p>Using <em>restore</em> and <em>tar</em> I was able to successfully extract the contents of all container files. For a small number of <em>dump</em> files, <em>restore</em> reported errors about missing files that could not be found. Although the exact cause is unknown at this stage, a possible explanation could be that in these cases a single <em>dump</em> was stored as two volumes on separate physical tapes. This will need further investigation. Nevertheless, overall the interpretation of the <em>xxLINK</em> tapes at the container level is quite straightforward.</p> <p>It is worth noting that the extraction may be more complicated for other container formats. For example, a number of Microsoft backup tools for the Windows platform (e.g. <em>NTBackup</em>) used to write data to tape using the <em>Microsoft Tape Format</em><sup id="fnref:36"><a href="#fn:36" class="footnote" rel="footnote" role="doc-noteref">36</a></sup>. This is a proprietary format that is only officially supported by the original creator software, which is not freely available, and only runs under Windows (however, a few open-source tools exist that claim to support the format). As the <em>xxLINK</em> data set does not include this format, it was not investigated as part of this work.</p> <h2 id="reconstruction-of-sites">Reconstruction of sites</h2> <p>Once the contents of the container files are extracted, we can start reconstructing the web sites. At this stage of the Web Archaeology project we have only just made a start with this; however, it is possible to present some first results. As a first step we need to inspect the contents of the data that were extracted from the container files, and identify any files and directories that contain web site data. This includes not only any directories with the sites’ source data, but also server configuration files, which contain valuable information about the sites. For instance, from the configuration files it is possible to see at which domains and URLs they were originally hosted, and how internal forwards were handled. With this information it is possible to host any of the old sites on a locally running web server at their original domains. A detailed discussion of the technical details is beyond the scope of this paper, but the general approach is similar to the one we used earlier to reconstruct the <em>NL-Menu</em> web index in 2018 <a href="https://www.bitsgalore.org/2018/04/24/resurrecting-the-first-dutch-web-index-nl-menu-revisited">[3]</a>, <a href="https://www.bitsgalore.org/2018/07/11/crawling-offline-web-content-the-nl-menu-case">[4]</a>. It comprises the following steps<sup id="fnref:37"><a href="#fn:37" class="footnote" rel="footnote" role="doc-noteref">37</a></sup>:</p> <ol> <li> <p>Set up a web server (typically <em>Apache</em>)<sup id="fnref:38"><a href="#fn:38" class="footnote" rel="footnote" role="doc-noteref">38</a></sup>, and restrict access to the server to <em>localhost</em> (this ensures that any hosted sites are only accessible locally on the machine on which the server is running).</p> </li> <li> <p>Copy the contents of the site (i.e. its directory tree) to the default root folder used by the web server (typically <code class="language-plaintext highlighter-rouge">/var/www</code>), and update the file permissions.</p> </li> <li> <p>Configure the site by creating a configuration file (or by adding an entry to an existing configuration file).</p> </li> <li> <p>Activate the configuration file.</p> </li> <li> <p>Add the site’s domain to the <em>hosts</em> file (<code class="language-plaintext highlighter-rouge">/etc/hosts</code>).</p> </li> <li> <p>Restart the web server.</p> </li> </ol> <p>After following these steps, the site is now locally accessible at its original URL, and it can be viewed in a browser, or archived with web crawler software such as <em>wget</em>. Since the directory structure of the web site data on the <em>xxLINK</em> tapes is quite uniform, it was possible to automate these steps to a large extent. Using this approach, I have so far reconstructed about 20 sites from one of the 1995 DDS-1 tapes by hosting them on an <em>Apache</em> web server instance. A few examples will illustrate the diversity of the sites in the <em>xxLINK</em> data set. Figure 6 shows the home page of <em>xxLINK</em>’s web site.</p> <figure class="image"> <img src="/images/2019/09/xxlink.png" alt="xxLINK home page" /> <figcaption>Figure 6: xxLINK home page</figcaption> </figure> <p>Figure 7 shows a snapshot of the home page of the web site of Schiphol Airport, which pre-dates the earliest snapshot of this site in Internet Archive<sup id="fnref:39"><a href="#fn:39" class="footnote" rel="footnote" role="doc-noteref">39</a></sup> by several months.</p> <figure class="image"> <img src="/images/2019/09/schiphol.png" alt="Home page of Schiphol Airport" /> <figcaption>Figure 7: Home page of Schiphol Airport</figcaption> </figure> <p>Figure 8 shows a report on drugs policy in the Netherlands, which was published as part of the site of the Dutch Ministry of Health, Welfare and Sport.</p> <figure class="image"> <img src="/images/2019/09/vwsdrugs.png" alt="Report on Drugs policy in the Netherlands on web site of Dutch Ministry of Health, Welfare and Sport" /> <figcaption>Figure 8: Report on Drugs policy in the Netherlands on web site of Dutch Ministry of Health, Welfare and Sport</figcaption> </figure> <p>Finally, Figure 9 shows a contest published on the site of Dutch publisher Database Publications. The objective of the contest was to correctly identify the web addresses of the home pages shown in the image; free copies of <em>CorelDraw 5.0</em> were available to five lucky winners.</p> <figure class="image"> <img src="/images/2019/09/prijsvraag.png" alt="Home page identification contest on web site of Database Publications" /> <figcaption>Figure 9: Home page identification contest on web site of Database Publications</figcaption> </figure> <p>The site reconstruction procedure will most likely need further refinement. For instance, most of the sites on the 1995 tape are relatively simple static HTML sites, but a few include forms that use CGI scripts, which currently do not work. Also, it is possible that the sites on the more recent (1999) tapes are more complex, but this needs further investigation. Once the analysis and processing of the data from the remaining tapes is complete, a more in-depth report on the reconstruction procedure will be published separately.</p> <h2 id="conclusions">Conclusions</h2> <p>In this paper I showed how old DDS and DLT-IV tapes from the ’90s can be read on a modern desktop workstation running Linux (in this case the Ubuntu-based <em>BitCurator</em> environment). I also explained how I created workflows that allow one to recover data from these tapes in a format-agnostic way, using the user-friendly <em>tapeimgr</em> software. Finally I discussed how I then extracted the contents of the resulting files, and how I used this to reconstruct a number of “lost” web sites from 1995. The workflow descriptions are available on Github<sup id="fnref:40"><a href="#fn:40" class="footnote" rel="footnote" role="doc-noteref">40</a></sup>, and they will most likely evolve further over time. They are published under a permissive license that allows anyone to adapt them and create derivatives. They describe all aspects of the tape reading process in detail, including the hardware components used and their characteristics, links to relevant documentation, instructions on how to handle the tapes (e.g. how to write-protect them), how to operate the tape readers, and how to use them in conjunction with <em>tapeimgr</em>. The level of detail provided should be sufficient to allow others to reproduce these workflows, and adapt them to their needs if necessary. Since the process of reading data tapes on Linux-based systems is quite standardized, other tape formats that are not covered by this paper can probably be processed in a similar way.</p> <h2 id="acknowledgments">Acknowledgments</h2> <p>First of all thanks are due to Wendy van Dijk and Elizabeth Mattijsen for making the <em>xxLINK</em> tapes available to us, and to Kees Teszelszky, Peter de Bode and Jasper Faase for initiating the Web Archaeology project, and establishing the contact with <em>xxLINK</em>. Peter Boel and René van Egdom are thanked for their help digging out the tape drives and other obscure hardware peripherals, and Willem Jan Faber for various helpful hardware-related suggestions. Finally, thanks are due to the anonymous reviewers who provided valuable feedback to an earlier draft of this paper.</p> <h2 id="references">References</h2> <p>[1] B. Sierman and K. Teszelszky, “How can we improve our web collection? an evaluation of webarchiving at the KB National Library of the Netherlands (2007-2017),” <em>Alexandria: The Journal of National and International Library and Information Issues</em>, Aug. 2017. DOI: 10.1177/0955749017725930. [Online]. Available: <a href="https://doi.org/10.1177/0955749017725930">https://doi.org/10.1177/0955749017725930</a>.</p> <p>[2] S. Ross and A. Gow, “Digital archaeology: Rescuing neglected and damaged data resources. a jisc/npo study within the electronic libraries (elib) programme on the preservation of electronic materials.,” University of Glasgow, 1999. [Online]. Available: <a href="http://www.ukoln.ac.uk/services/elib/papers/supporting/pdf/p2.pdf">http://www.ukoln.ac.uk/services/elib/papers/supporting/pdf/p2.pdf</a>.</p> <p>[3] J. van der Knijff, <em>Resurrecting the first Dutch web index: NL-menu revisited</em>, 2018. [Online]. Available: <a href="https://www.bitsgalore.org/2018/04/24/resurrecting-the-first-dutch-web-index-nl-menu-revisited">https://www.bitsgalore.org/2018/04/24/resurrecting-the-first-dutch-web-index-nl-menu-revisited</a>.</p> <p>[4] ——, <em>Crawling offline web content: The NL-menu case</em>, 2018. [Online]. Available: <a href="https://www.bitsgalore.org/2018/07/11/crawling-offline-web-content-the-nl-menu-case">https://www.bitsgalore.org/2018/07/11/crawling-offline-web-content-the-nl-menu-case</a>.</p> <p>[5] D. Elford, N. Del Pozo, S. Mihajlovic, D. Pearson, G. Clifton, and C. Webb, “Media matters: Developing processes for preserving digital objects on physical carriers at the national library of Australia,” in <em>Proceedings, 74th IFLA General Conference and Council</em>, 2008. [Online]. Available: <a href="http://archive.ifla.org/IV/ifla74/papers/084-Webb-en.pdf">http://archive.ifla.org/IV/ifla74/papers/084-Webb-en.pdf</a>.</p> <p>[6] K. Woods and G. Brown, “From imaging to access - effective preservation of legacy removable media,” in <em>Proceedings, Archiving 2009</em>, 2009. [Online]. Available: <a href="https://kamwoods.net/publications/woodsbrownarch09.pdf">https://kamwoods.net/publications/woodsbrownarch09.pdf</a>.</p> <p>[7] K. Woods, C. Lee, and S. Garfinkel, “Extending digital repository architectures to support disk image preservation and access,” in <em>Proceedings of the 11th Annual International ACM/IEEE Joint Conference on Digital Libraries</em>, ser. JCDL ‘11, Ottawa, Ontario, Canada: ACM, 2011, pp. 57-66, ISBN: 978-1-4503- 0744-4. DOI: 10.1145/1998076.1998088. [Online]. Available: <a href="http://doi.acm.org/10.1145/1998076.1998088">http://doi.acm.org/10.1145/1998076.1998088</a>.</p> <p>[8] C. Lee, M. Kirschenbaum, A. Chassanoff, P. Olsen, and K. Woods, “BitCurator: Tools and techniques for digital forensics in collecting institutions,” <em>D-Lib Magazine</em>, vol. 18, 5/6 2012. [Online]. Available: <a href="http://www.dlib.org/dlib/may12/lee/05lee.html">http://www.dlib.org/dlib/may12/lee/05lee.html</a>.</p> <p>[9] J. John, “Digital forensics and preservation,” Digital Preservation Coalition, 2012. [Online]. Available: <a href="http://dx.doi.org/10.7207/twr12-03">http://dx.doi.org/10.7207/twr12-03</a>.</p> <p>[10] M. Pennock, P. May, M. Day, K. Davies, S. Whibley, A. Kimura, and E. Halvarsson, “The flashback project: Rescuing disk-based content from the 1980’s to the current day,” in <em>Proceedings, 11th Digital Curation Conference</em>, 2016. [Online]. Available: <a href="https://doi.org/10.5281/zenodo.1321629">https://doi.org/10.5281/zenodo.1321629</a>.</p> <p>[11] A. Prael and A. Wickner, “Getting to know FRED: Introducing workflows for born-digital content,” <em>Practical Technology for Archives</em>, vol. 4, 2015. [Online]. Available: <a href="https://practicaltechnologyforarchives.org/issue4_prael_wickner/">https://practicaltechnologyforarchives.org/issue4_prael_wickner/</a>.</p> <p>[12] D. Salo, <em>Building audio, video and data-rescue kits</em>, 2017. [Online]. Available: <a href="https://radd.dsalo.info/wp-content/uploads/2017/10/BuildDocumentation.pdf">https://radd.dsalo.info/wp-content/uploads/2017/10/BuildDocumentation.pdf</a>.</p> <p>[13] BitCurator Consortium, <em>Workflows</em>. [Online]. Available: <a href="https://bitcuratorconsortium.org/workflows">https://bitcuratorconsortium.org/workflows</a>.</p> <p>[14] T. De Haan, “Project the digital city revives, a case study of web archaeology,” <em>Proceedings of the 13th International Conference on Digital Preservation</em>, 2016. [Online]. Available: <a href="https://doi.org/10.5281/zenodo.1255965">https://doi.org/10.5281/zenodo.1255965</a>.</p> <p>[15] T. De Haan, R. Jansma, and P. Vogel, <em>Do it yourself handboek voor webarcheologie</em>, 2017. [Online]. Available: <a href="https://hart.amsterdam/image/2017/11/17/20171116_freeze_diy_handboek.pdf">https://hart.amsterdam/image/2017/11/17/20171116_freeze_diy_handboek.pdf</a>.</p> <hr /> <p>Originally published in <a href="https://ipres2019.org/static/pdf/iPres2019_paper_9.pdf">Proceedings of the 16th International Conference on Digital Preservation, 2019</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>Elizabeth Mattijsen, old xxLINK-homepage: <a href="https://liz.nl/xxlink-homepage.htm">https://liz.nl/xxlink-homepage.htm</a> <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>See e.g. the links in the “Tapes” section at <a href="https://github.com/KBNLresearch/forensicImagingResources/blob/master/doc/df-resources.md">https://github.com/KBNLresearch/forensicImagingResources/blob/master/doc/df-resources.md</a> <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>BitCurator: <a href="https://bitcurator.net/">https://bitcurator.net/</a> <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>“Digital Data Storage”, Wikipedia: <a href="https://en.wikipedia.org/wiki/Digital_Data_Storage">https://en.wikipedia.org/wiki/Digital_Data_Storage</a> <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>“HP StorageWorks DDS/DAT Media - DDS/DAT Media Compatibility Matrix”, Hewlett Packard: <a href="https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-lpg50457">https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-lpg50457</a> <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:6"> <p>“Digital Linear Tape”, Wikipedia: <a href="https://en.wikipedia.org/wiki/Digital_Linear_Tape">https://en.wikipedia.org/wiki/Digital_Linear_Tape</a> <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:7"> <p>“DLT Drive Media and Cleaning Tape Compatibility Guide”, TapeandMedia.com: <a href="https://www.tapeandmedia.com/dlt_capacity_info.asp">https://www.tapeandmedia.com/dlt_capacity_info.asp</a> <a href="#fnref:7" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:8"> <p>“Parallel SCSI”, Wikipedia: <a href="https://en.wikipedia.org/wiki/Parallel_SCSI">https://en.wikipedia.org/wiki/Parallel_SCSI</a> <a href="#fnref:8" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:9"> <p>“Conventional PCI”, Wikipedia: <a href="https://en.wikipedia.org/wiki/Conventional_PCI">https://en.wikipedia.org/wiki/Conventional_PCI</a> <a href="#fnref:9" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:10"> <p>“PCI Express”, Wikipedia: <a href="https://en.wikipedia.org/wiki/PCI_Express">https://en.wikipedia.org/wiki/PCI_Express</a> <a href="#fnref:10" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:11"> <p>“Adaptec Support”, Microsemi: <a href="https://storage.microsemi.com/en-us/support/scsi/">https://storage.microsemi.com/en-us/support/scsi/</a> <a href="#fnref:11" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:12"> <p>“Parallel SCSI”, Wikipedia: <a href="https://en.wikipedia.org/wiki/Parallel_SCSI">https://en.wikipedia.org/wiki/Parallel_SCSI</a> <a href="#fnref:12" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:13"> <p>“LVD, SE, HVD, SCSI Compatibility - Or Not”, Paralan: <a href="http://www.paralan.com/scsiexpert.html">http://www.paralan.com/scsiexpert.html</a> <a href="#fnref:13" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:14"> <p>“SCSI connector”, Wikipedia: <a href="https://en.wikipedia.org/wiki/SCSI_connector">https://en.wikipedia.org/wiki/SCSI_connector</a> <a href="#fnref:14" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:15"> <p>“What kind of SCSI do I have?”, Paralan: <a href="http://www.paralan.com/sediff.html">http://www.paralan.com/sediff.html</a> <a href="#fnref:15" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:16"> <p>“Parallel SCSI”, Wikipedia: <a href="https://en.wikipedia.org/wiki/Parallel_SCSI">https://en.wikipedia.org/wiki/Parallel_SCSI</a> <a href="#fnref:16" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:17"> <p>“SCSI termination Q&amp;A”, Adaptec: <a href="https://storage.microsemi.com/en-us/support/scsi/3940/aha-3940uwd/hw_install/scsi_termination.htm">https://storage.microsemi.com/en-us/support/scsi/3940/aha-3940uwd/hw_install/scsi_termination.htm</a> <a href="#fnref:17" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:18"> <p>“dd (Unix)”, Wikipedia: <a href="https://en.wikipedia.org/wiki/Dd_%28Unix%29">https://en.wikipedia.org/wiki/Dd_%28Unix%29</a> <a href="#fnref:18" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:19"> <p>“file (command)”, Wikipedia: <a href="https://en.wikipedia.org/wiki/File_(command)">https://en.wikipedia.org/wiki/File_(command)</a> <a href="#fnref:19" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:20"> <p>Apache Tika: <a href="https://tika.apache.org/">https://tika.apache.org/</a> <a href="#fnref:20" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:21"> <p>Fido: <a href="http://fido.openpreservation.org/">http://fido.openpreservation.org/</a> <a href="#fnref:21" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:22"> <p>Siegfried: <a href="https://www.itforarchivists.com/siegfried/">https://www.itforarchivists.com/siegfried/</a> <a href="#fnref:22" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:23"> <p>“mt(1) - Linux man page”, die.net: <a href="https://linux.die.net/man/1/mt">https://linux.die.net/man/1/mt</a> <a href="#fnref:23" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:24"> <p>“Tapeimgr”: <a href="https://github.com/KBNLresearch/tapeimgr">https://github.com/KBNLresearch/tapeimgr</a> <a href="#fnref:24" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:25"> <p>“Guymager homepage”: <a href="https://guymager.sourceforge.io/">https://guymager.sourceforge.io/</a> <a href="#fnref:25" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:26"> <p>“ “Cannot allocate memory” when reading from SCSI tape”, Unix Stack Exchange: <a href="https://unix.stackexchange.com/a/366217">https://unix.stackexchange.com/a/366217</a> <a href="#fnref:26" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:27"> <p>“Tape Driver Semantics”, Amanda Wiki: <a href="https://wiki.zmanda.com/index.php/Tape_Driver_Semantics">https://wiki.zmanda.com/index.php/Tape_Driver_Semantics</a> <a href="#fnref:27" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:28"> <p>“KB Forensic Imaging Resources”: <a href="https://github.com/KBNLresearch/forensicImagingResources/tree/master/doc">https://github.com/KBNLresearch/forensicImagingResources/tree/master/doc</a> <a href="#fnref:28" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:29"> <p>“file (command)”, Wikipedia: <a href="https://en.wikipedia.org/wiki/File_(command)">https://en.wikipedia.org/wiki/File_(command)</a> <a href="#fnref:29" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:30"> <p>“Unix dump”, ArchiveTeam File Formats Wiki: <a href="http://fileformats.archiveteam.org/wiki/Unix_dump">http://fileformats.archiveteam.org/wiki/Unix_dump</a> <a href="#fnref:30" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:31"> <p>“restore(8) - Linux man page”, die.net: <a href="https://linux.die.net/man/8/restore">https://linux.die.net/man/8/restore</a> <a href="#fnref:31" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:32"> <p>A step-by-step description can be found here: <a href="https://github.com/KBNLresearch/forensicImagingResources/blob/master/doc/extract-dumpfile.md">https://github.com/KBNLresearch/forensicImagingResources/blob/master/doc/extract-dumpfile.md</a> <a href="#fnref:32" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:33"> <p>“NTFS”, Wikipedia: <a href="https://en.wikipedia.org/wiki/NTFS">https://en.wikipedia.org/wiki/NTFS</a> <a href="#fnref:33" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:34"> <p>“Ext4”, Wikipedia: <a href="https://en.wikipedia.org/wiki/Ext4">https://en.wikipedia.org/wiki/Ext4</a> <a href="#fnref:34" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:35"> <p>“tar(1) - Linux man page”, die.net: <a href="https://linux.die.net/man/1/tar">https://linux.die.net/man/1/tar</a> <a href="#fnref:35" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:36"> <p>Microsoft Tape Format Specification Version 1.00a: <a href="http://laytongraphics.com/mtf/MTF_100a.PDF">http://laytongraphics.com/mtf/MTF_100a.PDF</a> <a href="#fnref:36" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:37"> <p>These steps are described in more detail here: <a href="https://github.com/KBNLresearch/nl-menu-resources/blob/master/doc/serving-static-website-with-Apache.md">https://github.com/KBNLresearch/nl-menu-resources/blob/master/doc/serving-static-website-with-Apache.md</a> <a href="#fnref:37" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:38"> <p>The Apache HTTP Server Project: <a href="https://httpd.apache.org/">https://httpd.apache.org/</a> <a href="#fnref:38" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:39"> <p>Link: <a href="https://web.archive.org/web/19961018155616/http://www.schiphol.nl/">https://web.archive.org/web/19961018155616/http://www.schiphol.nl/</a> <a href="#fnref:39" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:40"> <p>“KB Forensic Imaging Resources”: <a href="https://github.com/KBNLresearch/forensicImagingResources/tree/master/doc">https://github.com/KBNLresearch/forensicImagingResources/tree/master/doc</a> <a href="#fnref:40" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2019/09/09/recovering-90s-data-tapes-experiences-kb-web-archaeology https://bitsgalore.org/2019/09/09/recovering-90s-data-tapes-experiences-kb-web-archaeology 2019-09-09T00:00:00+02:00 <figure class="image"> <img src="/images/2019/04/floppies.jpg" alt="Photograph of SATA hard disk, USB Flash drive and 3.5 floppy disks" /> <figcaption>SATA hard disk, USB Flash drive and 3.5" floppy disks</figcaption> </figure> <p>As I explained in the introduction of <a href="/2019/01/31/roll-the-tape-recovering-90s-data-tapes-in-bitcurator">this earlier blog post</a>, as part of our ongoing web archaeology project we are currently developing workflows for reading data from a variety of physical carrier formats. After the earlier work on <a href="/2019/01/31/roll-the-tape-recovering-90s-data-tapes-in-bitcurator">data tapes</a> and <a href="/2019/03/22/a-simple-workflow-tool-for-imaging-optical-media-using-readom-and-ddrescue">optical media</a>, the next job was to image a small box with 3.5” floppy disks. Easy enough, and my first thought was to fire up <a href="https://guymager.sourceforge.io/"><em>Guymager</em></a> and be done with it. This turned out to be less straightforward than expected, which led to the development of yet another workflow tool: <a href="https://github.com/KBNLresearch/diskimgr"><em>diskimgr</em></a>. In the remainder of this post I will first show the issues I ran into with <em>Guymager</em>, and then demonstrate how these issues are remedied by <em>diskimgr</em>.</p> <!-- more --> <h2 id="guymager-workflow">Guymager workflow</h2> <p>As I tried to image some floppies with <em>Guymager</em>, I quickly ran into several problems, all of which involved the handling of user-added descriptive metadata. <em>Guymager</em> does in fact accommodate for this, but the way it is implemented introduces some practical issues. To illustrate this, here’s <em>Guymager</em>’s default entry form:</p> <figure class="image"> <img src="/images/2019/04/guymager-entry.png" alt="Screenshot of Guymager entry form, Expert Witness format" /> <figcaption>Guymager entry form, Expert Witness format</figcaption> </figure> <p>The form provides various fields that can be used to enter descriptive metadata, but they are <em>only</em> available if one chooses to write the disk image in <a href="https://www.loc.gov/preservation/digital/formats/fdd/fdd000406.shtml"><em>Expert Witness Format</em></a>. Changing the format to <em>Linux dd raw image</em> (which simply generates a raw copy of the imaged medium’s bytestream) disables these fields:</p> <figure class="image"> <img src="/images/2019/04/guymager-dd.png" alt="Screenshot of Guymager entry form, dd format" /> <figcaption>Guymager entry form, dd format</figcaption> </figure> <h2 id="expert-witness-format">Expert Witness Format</h2> <p>One possible solution would be to image the floppies to <em>Expert Witness Format</em> (<em>EWF</em>), in which case the entered descriptor fields are embedded in the disk image. However, I really don’t want to do this. First, saving to <em>EWF</em> would make further processing of the disk image (e.g. mounting the file system, or attaching it to a virtual machine or emulator) more difficult, since it requires that the reading application (emulator, disk mount tool) supports not only the floppy’s native file system (typically <a href="https://forensicswiki.org/wiki/FAT"><em>FAT</em></a> for floppies that were written with MS-DOS or Windows), but also the added <em>EWF</em> layer. Also, while <em>EWF</em>’s support for data compression can be tremendously useful for imaging large hard disks, it is largely unnecessary for 1.5 MB floppies.</p> <h2 id="guymager-metadata-entry">Guymager metadata entry</h2> <p>Last but not least, <em>Guymager</em>’s interface introduces another practical hurdle. When <em>Guymager</em> is launched, it shows a list of available devices:</p> <figure class="image"> <img src="/images/2019/04/guymager-mainscreen.png" alt="Screenshot of Guymager startup screen with device list" /> <figcaption>Guymager startup screen with device list</figcaption> </figure> <p>The user then selects the device they want to image, after which the aforementioned data entry form pops up. However, for floppies the descriptive metadata that one would enter here is typically found on the label on the floppy itself, <em>which by this time is inaccessible because the floppy is inside the floppy drive</em>!</p> <figure class="image"> <img src="/images/2019/04/floppy-label.jpg" alt="Photograph of floppy with descriptive information on label" /> <figcaption>Floppy with descriptive information on label</figcaption> </figure> <p>Of course there are workarounds to this (e.g. you could copy the information from the labels beforehand, put it in a text file and then paste it into <em>Guymager</em>’s entry fields), but overall this would be pretty clumsy.</p> <h2 id="linux-dd-raw">Linux dd raw</h2> <p>An alternative solution would be to select the <em>Linux dd raw</em> option in <em>Guymager</em>, and then add the metadata by hand afterwards. Again (as I also argued in <a href="/2019/03/22/a-simple-workflow-tool-for-imaging-optical-media-using-readom-and-ddrescue">my previous blog post</a>), this is pretty cumbersome, and prone to all sorts of errors.</p> <h2 id="diskimgr">Diskimgr</h2> <p>As the <a href="https://github.com/KBNLresearch/omimgr"><em>omimgr</em></a> already solves the above problems for optical media, I simply used the code of <em>omimgr</em> as a starting point, and adapted it into <a href="https://github.com/KBNLresearch/diskimgr"><em>diskimgr</em></a>. <em>Diskimgr</em> is a general-purpose disk imaging tool that can be used for a wide variety of digital media, such a floppy disks, USB Flash drives and hard disks. It provides a simple graphical user interface for the entry of descriptive metadata, and all entered and generated metadata are written to a <em>JSON</em> file, along with the image file.</p> <figure class="image"> <img src="/images/2019/04/diskimgr-1.png" alt="Screenshot of diskimgr interface" /> <figcaption>Diskimgr interface</figcaption> </figure> <p>Internally <em>diskimgr</em> wraps around <a href="https://linux.die.net/man/1/dd"><em>Unix dd</em></a> and <a href="https://linux.die.net/man/1/ddrescue"><em>ddrescue</em></a>. The general workflow of <em>diskimgr</em> is similar to the one employed by <em>omimgr</em>: first it tries to read a user-defined medium with <em>dd</em>. If <em>dd</em> fails, it prompts the user to give it another try with <em>ddrescue</em>. If <em>ddrescue</em> was unable to recover all the data from the medium, additional <em>ddrescue</em> passes may be run to further improve the result. As an example, the screenshot below was taken after a <em>diskimgr</em> session with a damaged 3.5” floppy disk:</p> <figure class="image"> <img src="/images/2019/04/ddrescue-pass2.png" alt="Screenshot of diskimgr after two ddrescue passes" /> <figcaption>Diskimgr after two ddrescue passes</figcaption> </figure> <p>After an initial attempt to image this floppy with <em>dd</em> failed with errors, a first pass with <em>ddrescue</em> resulted in a 106 kB block of unreadable data. A second <em>ddrescue</em> pass with the <em>Direct disc mode</em> option switched on reduced the size of the unreadable block to a mere 512 bytes (one sector).</p> <h2 id="metadata">Metadata</h2> <p>Descriptive metadata, a SHA-512 checksum of the disk image, and a host of event metadata are written to a <em>JSON</em> file in a format that is largely identical to the one used by <em>omimgr</em>. Below is an example:</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"acquisitionEnd"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2019-04-09T13:10:40.503984+02:00"</span><span class="p">,</span><span class="w"> </span><span class="nl">"acquisitionStart"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2019-04-09T13:09:59.835833+02:00"</span><span class="p">,</span><span class="w"> </span><span class="nl">"autoRetry"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"> </span><span class="nl">"blockDevice"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/dev/sdc"</span><span class="p">,</span><span class="w"> </span><span class="nl">"checksumType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"SHA-512"</span><span class="p">,</span><span class="w"> </span><span class="nl">"checksums"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"ks.img"</span><span class="p">:</span><span class="w"> </span><span class="s2">"79a17d3fa536b8fa750257b01d05124dadb888f1171e9ca5cc3398a2c16de81b1687b52c70135b966409a723ef5f3960536a6e994847c5ebe7d5eaffefa62dc7"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"KS metingen origineel"</span><span class="p">,</span><span class="w"> </span><span class="nl">"diskimgrVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0.1.0b3"</span><span class="p">,</span><span class="w"> </span><span class="nl">"extension"</span><span class="p">:</span><span class="w"> </span><span class="s2">"img"</span><span class="p">,</span><span class="w"> </span><span class="nl">"identifier"</span><span class="p">:</span><span class="w"> </span><span class="s2">"cc630cda-5ab7-11e9-bc82-dc4a3e5f53bf"</span><span class="p">,</span><span class="w"> </span><span class="nl">"interruptedFlag"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"> </span><span class="nl">"maxRetries"</span><span class="p">:</span><span class="w"> </span><span class="s2">"4"</span><span class="p">,</span><span class="w"> </span><span class="nl">"notes"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span><span class="w"> </span><span class="nl">"prefix"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ks"</span><span class="p">,</span><span class="w"> </span><span class="nl">"readCommandLine"</span><span class="p">:</span><span class="w"> </span><span class="s2">"dd if=/dev/sdc of=/home/johan/test/6/ks.img bs=512 conv=notrunc"</span><span class="p">,</span><span class="w"> </span><span class="nl">"readMethod"</span><span class="p">:</span><span class="w"> </span><span class="s2">"dd"</span><span class="p">,</span><span class="w"> </span><span class="nl">"readMethodVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"dd (coreutils) 8.25"</span><span class="p">,</span><span class="w"> </span><span class="nl">"rescueDirectDiscMode"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"> </span><span class="nl">"successFlag"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <h2 id="main-uses">Main uses</h2> <p>While <em>diskimgr</em> can be used for virtually any kind of block device, including entire hard disks, it is not intended to be a full replacement for tools such as <em>Guymager</em>. For instance, for imaging a 500 GB hard disk I’d probably still prefer <em>Guymager</em>. However, for situations where one wants to image a large number of small-size media (such as floppies), <em>Guymager</em> is less than ideal, and <em>diskimgr</em> might be worth a try. It also provides a user-friendly alternative to using <em>dd</em> and <em>ddrescue</em> from the command-line.</p> <h2 id="final-remarks">Final remarks</h2> <p>Just like <em>tapeimgr</em> and <em>omimgr</em>, <em>diskimgr</em> only works on Linux-based systems. Again, this is an initial release which has had limited testing, so use at your own risk. If you run into any issues, feel free to <a href="https://github.com/KBNLresearch/diskimgr/issues">report them here</a>.</p> <h2 id="link-to-diskimgr">Link to diskimgr</h2> <p><em>Diskimgr</em> and its documentation can be found here:</p> <p><a href="https://github.com/KBNLresearch/diskimgr"><em>diskimgr - Simple workflow tool for imaging block devices</em></a></p> https://bitsgalore.org/2019/04/10/a-simple-disk-imaging-workflow-tool https://bitsgalore.org/2019/04/10/a-simple-disk-imaging-workflow-tool 2019-04-10T00:00:00+02:00 <p>In 2015 I wrote <a href="/2015/11/13/preserving-optical-media-from-the-command-line">a blog post on preserving optical media from the command-line</a>. Among other things, it suggested a rudimentary workflow for imaging CD-ROMs and DVDs using the <a href="http://linux.die.net/man/1/readom"><em>readom</em></a> and <a href="http://linux.die.net/man/1/ddrescue"><em>ddrescue</em></a> tools. Even though we now have a <a href="/2017/06/19/image-and-rip-optical-media-like-a-boss">highly automated workflow in place</a> for bulk processing optical media from our deposit collection, <em>readom</em> and <em>ddrescue</em> still prove to be useful for various special cases that don’t quite fit into this workflow. The materials that we are currently receiving as part of our web archaeology activities are a good example. These are typically small sets of recordable CD-ROMs that are often quite old, and such discs are highly likely to be in less than perfect condition. For these cases a highly automated, <a href="https://github.com/KBNLresearch/iromlab"><em>iromlab</em></a>-like workflow is unnecessary, and to some degree even impractical. Nevertheless, it would be useful to have <em>some</em> degree of automation, especially for things like the addition and packaging of associated metadata. This prompted the development of the <a href="https://github.com/KBNLresearch/omimgr"><em>omimgr</em></a> workflow tool. In the the remainder of this blog post I will give an overview of <em>omimgr</em>.</p> <!-- more --> <h2 id="the-command-line-workflow">The command-line workflow</h2> <p>In <a href="/2015/11/13/preserving-optical-media-from-the-command-line">my 2015 blog post</a> I reviewed a number of command-line imaging tools for optical media. For CD-ROMs and DVDs I recommended to first try imaging the disc with <em>readom</em>, and use <em>ddrescue</em> in case <em>readom</em> fails. The logic behind this is that <em>readom</em> was specifically designed for reading optical media, which makes it preferable over generic block device recovery tools such as <a href="https://guymager.sourceforge.io/"><em>Guymager</em></a>, <a href="http://linux.die.net/man/1/dd"><em>dd</em></a> or <em>ddrescue</em>. However, <em>readom</em> gives up rather easily on discs that are damaged or otherwise degraded, and for these cases <em>ddrescue</em> is often capable of recovering surprising amounts of data. For example, our earlier success at <a href="/2018/04/24/resurrecting-the-first-dutch-web-index-nl-menu-revisited">resurrecting the first Dutch web index</a> was largely thanks to <em>ddrescue</em>’s ability to work its magic on the degraded CD-recordable that contained the source data. So, a basic command-line workflow that is based on <em>readom</em> and <em>ddrescue</em> would look like this:</p> <ol> <li> <p>Unmount the disc:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>umount /dev/sr0 </code></pre></div> </div> </li> <li> <p>Try to create an ISO image of the disc with <em>readom</em>:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>readom <span class="nv">retries</span><span class="o">=</span>4 <span class="nv">dev</span><span class="o">=</span>/dev/sr0 <span class="nv">f</span><span class="o">=</span>disc.iso </code></pre></div> </div> </li> <li> <p>If <em>readom</em> fails, try to image the disc with <em>ddrescue</em>:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ddrescue <span class="nt">-b</span> 2048 <span class="nt">-r4</span> <span class="nt">-v</span> /dev/sr0 disc.iso disc.map </code></pre></div> </div> </li> <li> <p>If <em>ddrescue</em> was unable to recover all the data on the disc, try to improve the result by re-running <em>ddrescue</em> in direct disc mode:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ddrescue <span class="nt">-d</span> <span class="nt">-b</span> 2048 <span class="nt">-r4</span> <span class="nt">-v</span> /dev/sr0 disc.iso disc.map </code></pre></div> </div> </li> <li> <p>If there are still read errors after the above command, try to improve the result by re-running <em>ddrescue</em> with another optical drive (e.g. an external USB-drive):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ddrescue <span class="nt">-b</span> 2048 <span class="nt">-r4</span> <span class="nt">-v</span> /dev/sr1 disc.iso disc.map </code></pre></div> </div> <p>(Note that steps 4 and 5 can be repeated for mutiple optical drives, if needed).</p> </li> <li> <p>Check the extracted ISO image for completeness with <a href="https://github.com/KBNLresearch/isolyzer"><em>isolyzer</em></a>:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>isolyzer disc.iso </code></pre></div> </div> <p>If the value of the <em>smallerThanExpected</em> element equals <em>False</em>, this is an indication that the ISO image is probably intact.</p> </li> </ol> <p>However, this is only part of the story. In most cases we will also want to record various types of metadata about the created disc image. A pretty minimal set includes:</p> <ul> <li> <p>Fixity information: a SHA-512 checksum of the ISO image.</p> </li> <li> <p>Descriptive metadata: an identifier that is associated with the disc, a description (this may simply be copied from the writing on a disc or its inlay card), and a text annotation for recording anything else about the disc that is noteworthy (e.g. its condition, or the fact that the entered description was based on handwritten text that is not clearly legible).</p> <p><img src="/images/2019/03/cd-writing.jpg" alt="CD-ROM with handwritten text" /></p> </li> <li> <p>Event metadata about the image acquisition process: the imaging software (and its associated version) that was used, the options it was invoked with, the status of the imaging process, and the outcome of any quality checks on the generated ISO image.</p> </li> </ul> <p>Adding these metadata by hand is pretty cumbersome, especially when processing multiple discs. It is also prone to all sorts of errors. So, I wrapped all the imaging, quality checks and metadata generation into a user-friendly piece of software with a graphical interface, similar to the earlier <a href="https://github.com/KBNLresearch/tapeimgr"><em>tapeimgr</em></a> tool (which served as a template for <em>omimgr</em>).</p> <h2 id="omimgr-operation">Omimgr operation</h2> <p>Imaging optical media with <em>omimgr</em> is simple. On start-up, it shows the following entry form:</p> <p><img src="/images/2019/03/omimgr-1.png" alt="Screenshot of omimgr interface at startup" /></p> <p>The entry fields are largely self-explanatory (but they are all <a href="https://github.com/KBNLresearch/omimgr">documented here</a>). The user can select an output directory, specify a device path that points to the optical drive (by default it uses the internal drive at <em>/dev/sr0</em>), and a preferred read method (by default <em>omimgr</em> starts with <em>readom</em>)<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. There is also a set of entry fields for basic descriptive metadata, and it is possible to assign an identifier (or generate one automatically). Imaging starts after the user presses the <em>Start</em> button.</p> <h2 id="if-readom-fails">If readom fails</h2> <p>If the initial attempt to image a disc with <em>readom</em> resulted in any errors, <em>omimgr</em> shows the following dialog box:</p> <p><img src="/images/2019/03/error-readom.png" alt="Screenshot of omimgr interface after readom error" /></p> <p>After pressing <em>Yes</em>, <em>omimgr</em> tries to image the disc with <em>ddrescue</em><sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>. If the <em>ddrescue</em> run resulted in any errors as well, <em>omimgr</em> offers the possibility to re-run it with different settings. There is no limit on the number of successive <em>ddrescue</em> runs on a disc, and, importantly, new runs do not overwrite the existing ISO image, but instead improve upon it. This makes it possible to use multiple optical drives on a disc.</p> <h2 id="interrupting-and-resuming">Interrupting and resuming</h2> <p>Since <em>ddrescue</em> may need a <em>lot</em> of time to recover data from a faulty disc (12-24 hours is no exception), it is possible to interrupt ongoing imaging processes with the <em>Interrupt</em> button. Interrupted <em>ddrescue</em> sessions can be resumed at any later time by selecting the session’s output directory. The <em>Load existing metadata</em> button will then load any previously entered descriptive metadata.</p> <h2 id="metadata">Metadata</h2> <p>A the end of each session, <em>omimgr</em> writes a metadata file in <em>JSON</em> format. Here’s an example:</p> <div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w"> </span><span class="nl">"acquisitionEnd"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2019-03-22T13:38:51.969934+01:00"</span><span class="p">,</span><span class="w"> </span><span class="nl">"acquisitionStart"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2019-03-22T13:37:43.060185+01:00"</span><span class="p">,</span><span class="w"> </span><span class="nl">"autoRetry"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"> </span><span class="nl">"checksumType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"SHA-512"</span><span class="p">,</span><span class="w"> </span><span class="nl">"checksums"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"backupjrc.iso"</span><span class="p">:</span><span class="w"> </span><span class="s2">"714426e7f965e4f6b33571ae4d60d945928dbee8c06f74225a138eaaa4ea4b2b7442620227e94920a0bc7ac17a6c7096fb310746cfff2c04b5c3e778ae8998ce"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Backup JRC 31-03-2000"</span><span class="p">,</span><span class="w"> </span><span class="nl">"extension"</span><span class="p">:</span><span class="w"> </span><span class="s2">"iso"</span><span class="p">,</span><span class="w"> </span><span class="nl">"identifier"</span><span class="p">:</span><span class="w"> </span><span class="s2">"e23f9158-4c9e-11e9-bbfc-dc4a3e413173"</span><span class="p">,</span><span class="w"> </span><span class="nl">"imageTruncated"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"> </span><span class="nl">"interruptedFlag"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"> </span><span class="nl">"isolyzerSuccess"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"> </span><span class="nl">"maxRetries"</span><span class="p">:</span><span class="w"> </span><span class="s2">"4"</span><span class="p">,</span><span class="w"> </span><span class="nl">"notes"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Outer edge of CD shows signs of corrosion"</span><span class="p">,</span><span class="w"> </span><span class="nl">"omDevice"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/dev/sr1"</span><span class="p">,</span><span class="w"> </span><span class="nl">"omimgrVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0.1.0"</span><span class="p">,</span><span class="w"> </span><span class="nl">"prefix"</span><span class="p">:</span><span class="w"> </span><span class="s2">"backupjrc"</span><span class="p">,</span><span class="w"> </span><span class="nl">"readCommandLine"</span><span class="p">:</span><span class="w"> </span><span class="s2">"readom retries=4 dev=/dev/sr1 f=/home/johan/test/backupjrc.iso"</span><span class="p">,</span><span class="w"> </span><span class="nl">"readMethod"</span><span class="p">:</span><span class="w"> </span><span class="s2">"readom"</span><span class="p">,</span><span class="w"> </span><span class="nl">"readMethodVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"readom 1.1.11 (Linux)"</span><span class="p">,</span><span class="w"> </span><span class="nl">"rescueDirectDiscMode"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"> </span><span class="nl">"successFlag"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre></div></div> <p>Note that the metadata file contains all of the entered descriptive metadata, a SHA-512 checksum of the ISO image, and a host of event metadata.</p> <h2 id="limitations">Limitations</h2> <p>Currently, <em>omimgr</em> only works on Linux-based systems. It does not support audio CDs, and can only be used for CD-ROMs and DVDs. <a href="https://en.wikipedia.org/wiki/Blue_Book_(CD_standard)"><em>Blue Book / CD-Extra</em></a> discs aren’t supported either. I wouldn’t rule out that these types of discs may eventually be supported in some future version of <em>omimgr</em>, but for now they don’t have any priority (currently the main use case is the imaging of recordable CD-ROMs and DVDs).</p> <p>It is also worth noting that <em>omimgr</em> only exposes a limited subset of <em>readom</em>’s and <em>ddrescue</em>’s functionality to the user. If you are looking for a full graphical front-end to <em>ddrescue</em> that gives access to all of its options, you should probably check out <a href="https://launchpad.net/ddrescue-gui"><em>DDRescue-GUI</em></a> instead<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>. Finally, the current version of <em>omimgr</em> is an initial release, which so far has had limited testing, so use at your own risk (as always), and feel free to <a href="https://github.com/KBNLresearch/omimgr/issues">report any issues</a> that you may come across.</p> <h2 id="link-to-omimgr">Link to omimgr</h2> <p><em>Omimgr</em> and its documentation can be found here:</p> <p><a href="https://github.com/KBNLresearch/omimgr"><em>omimgr - Simple workflow tool for imaging optical media</em></a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:3"> <p>Note that most of the default values shown here can be easily changed by modifying a configuration file. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:1"> <p>Checking the <em>Auto-retry</em> option bypasses this dialog, in which case <em>omimgr</em> will automatically start <em>ddrescue</em> without any user intervention. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>Incidentally <em>omimgr</em> contains a few lines of code that I borrowed from <em>DDRescue-GUI</em>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2019/03/22/a-simple-workflow-tool-for-imaging-optical-media-using-readom-and-ddrescue https://bitsgalore.org/2019/03/22/a-simple-workflow-tool-for-imaging-optical-media-using-readom-and-ddrescue 2019-03-22T00:00:00+01:00 <p>When the <a href="https://www.kb.nl/en/organisation/research-expertise/long-term-usability-of-digital-resources/web-archiving">KB web archive</a> was launched in 2007, many sites from the “early” Dutch web had already gone offline. As a result, the time period between (roughly) 1992 and 2000 is seriously under-represented in our web archive. To improve the coverage of web sites from this historically important era, we are now looking into <a href="https://hart.amsterdam/image/2016/11/28/20160730_redds_tjardadehaan.pdf">Web Archaeology</a> tools and methods. Over the last year our web archiving team has reached out to creators of “early” Dutch web sites that are no longer online. It’s not uncommon to find that these creators still have boxes of offline carriers with the original source data of those sites. Using these data, we would (in many cases) be able to reconstruct the sites, similarly to how we <a href="/2018/04/24/resurrecting-the-first-dutch-web-index-nl-menu-revisited">reconstructed the first Dutch web index</a> last year. Once reconstructed, they could then be ingested into our web archive.</p> <!-- more --> <h2 id="physical-carrier-formats">Physical carrier formats</h2> <p>At this early stage of the web archaeology project, a few site creators have already lended us sample sets of offline carriers. Even though these sets are limited in size, they already contain quite a wide range of physical formats: CD-ROMs, floppy disks, ZIP disks, USB thumb drives, (internal) hard disks, and a variety of tape formats. For reading the data on these carriers, we’ve set up a desktop workstation running the <a href="https://bitcurator.net/"><em>BitCurator</em></a> environment.</p> <h2 id="tapes">Tapes</h2> <p>One of the first sample sets we received contains a collection of over 30 data tapes from the mid to late ’90s. Roughly half of these are <a href="https://en.wikipedia.org/wiki/Digital_Data_Storage"><em>DDS-1</em></a> tapes, a format based on <a href="https://en.wikipedia.org/wiki/Digital_Audio_Tape"><em>Digital Audio Tape</em></a>. The other half are <em>DLT-IV</em> tapes, a type of <a href="https://en.wikipedia.org/wiki/Digital_Linear_Tape"><em>Digital Linear Tape</em></a>. The remainder of this blog post explains how we set up a workflow for reading these tapes. It also highlights some of the particular challenges we encountered along the way, and gives some (hopefully useful) hints and suggestions for others who are interested in setting up a similar workflow.</p> <h2 id="tape-drives">Tape drives</h2> <p>Obviously, to read these vintage tape formats you first need tape drives that support them. Luckily, our IT department turned out to have a working <em>DDS-2</em> drive (which also reads <em>DDS-1</em> tapes), as well as a <em>DLT-IV</em> drive tucked away on a shelf.</p> <figure class="image"> <img src="/images/2019/01/dlt-insert.jpg" alt="Photograph of DLT-IV drive with tape" /> <figcaption>DLT-IV drive with tape</figcaption> </figure> <h2 id="scsi-madness">SCSI madness</h2> <p>Since tape drives typically use <a href="https://en.wikipedia.org/wiki/Parallel_SCSI">parallel <em>SCSI</em></a> connectors, hooking them up to a modern PC is not straightforward, and requires the installation of a <a href="https://en.wikipedia.org/wiki/SCSI_host_adapter"><em>SCSI</em> host adapter</a> (AKA <em>SCSI</em> controller). Although used ones are available cheap online, choosing the right one can be tricky. Many older models are <a href="https://en.wikipedia.org/wiki/Conventional_PCI"><em>conventional PCI cards</em></a>, which are not compatible with most modern motherboards (which these days are more likely to have <a href="https://en.wikipedia.org/wiki/PCI_Express"><em>PCI Express</em></a> slots). Also, beware that many <em>SCSI</em> adapters have a 64-bit <em>PCI</em> interface, which is only compatible with enterprise servers<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. Since online sellers often don’t mention the interface type, <a href="https://storage.microsemi.com/en-us/support/scsi/">this website with Adaptec <em>SCSI</em> Card Specifications</a> is a useful resource for checking potential compatibility issues.</p> <p>In addition, rather than being one well-defined standard, parallel <em>SCSI</em> is really a hot mess of different standards that use different interfaces and connector types. Not all of these interfaces and connectors are mutually compatible, and interface mismatches <a href="https://twitter.com/charles_forsyth/status/1004356758893154305">can result in actual hardware damage</a>. The Wikipedia entry on parallel <em>SCSI</em> <a href="https://en.wikipedia.org/wiki/Parallel_SCSI#Compatibility">contains some useful information on compatibility issues</a>; a more <a href="http://www.paralan.com/scsiexpert.html">in-depth discussion can be found here</a>. The same web site (a treasure trove of all things <em>SCSI</em>) also has this <a href="http://www.paralan.com/sediff.html">illustrated overview of the most common connector types</a>, which I found immensely helpful for identifying the connector types of our tape drives. Because of the myriad <em>SCSI</em> connector types, you may also need adapter plugs or cables to connect the tape drive to the <em>SCSI</em> controller (in our case we used <a href="https://web.archive.org/web/20181002103944/https://www.ramelectronics.net/sm-044-r.aspx">this adapter plug</a> to connect the 68-pin high-density cable of our <em>DDS</em> drive to <em>SCSI</em> controller’s <em>VHCDI</em> connector). Finding matching cables and adapters can be quite a challenge, not least because multiple names are used for most <em>SCSI</em> connector types. For instance, the commonly used 68-pin “DB68” connector is also known as “MD68”, “High-Density”, “HD 68”, “Half-Pitch” and “HP68”. Aaargh!</p> <p>Another thing to keep in mind is that any unused <em>SCSI</em> buses on the tape drive must be fitted with a <a href="https://en.wikipedia.org/wiki/Parallel_SCSI#Termination">terminator</a>, so if your drive doesn’t have one already you’ll have to track down a matching type.</p> <h2 id="tapeimgr-software">Tapeimgr software</h2> <p>With the <em>SCSI</em> controller inserted into our <em>BitCurator</em> workstation, I hooked up one of the tape drives, and tried to read some test tapes<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>. Since we want to read the tape data in a format-agnostic way that is independent of the software that was originally used to write the tapes, I used Unix <a href="https://en.wikipedia.org/wiki/Dd_%28Unix%29"><em>dd</em></a> (and the <a href="https://linux.die.net/man/1/mt"><em>mt</em></a> tool to issue tape transport commands). After some experimentation, I was able to write a simple Bash script that sequentially reads all files on the tape. I then rewrote the script into what was to become the <a href="https://github.com/KBNLresearch/tapeimgr"><em>tapeimgr</em></a> software. <em>Tapeimgr</em> (which was loosely inspired by the <a href="https://guymager.sourceforge.io/"><em>Guymager</em></a> software) allows one to read data from a tape using a simple and user-friendly graphical interface. Internally, <em>tapeimgr</em> just wraps around <em>dd</em> and <em>mt</em>, but the complexities of these tools are hidden from the user. For a given tape, the software reads all files; before reading a file, it first runs an iterative procedure to establish the block size that was used for writing it. Once the end of a tape is reached, <em>tapeimgr</em> computes <em>SHA512</em> checksums of all recovered files, and these are subsequently written to a .json file (alongside some basic descriptive and event metadata). A detailed extraction log with the full output of <em>dd</em> is also written for each tape.</p> <figure class="image"> <img src="/images/2019/01/tapeimgr-2.png" alt="Screenshot of Tapeimgr interface" /> <figcaption>Tapeimgr interface</figcaption> </figure> <h2 id="post-processing">Post-processing</h2> <p>Since <em>tapeimgr</em> is format-agnostic, it’s up to the user to figure out how to further process the recovered files. Identifying the file format is the first step, and this can be done using the usual suspects such as Unix <a href="https://linux.die.net/man/1/file"><em>file(1)</em></a>, <a href="https://www.itforarchivists.com/siegfried/"><em>Siegfied</em></a> (both accessible through right-click context menus in <em>BitCurator</em>). Once the format is known, format-specific tools (e.g. <a href="https://linux.die.net/man/1/tar"><em>tar</em></a>) can be used to extract the files’ contents.</p> <h2 id="recovering-the-first-sample-set">Recovering the first sample set</h2> <p>After we were confident that our tape processing workflow worked correctly, we used it to process the sample set of <em>DDS</em> and <em>DLT-IV</em> tapes that were lended to us. The majority of the 19 <em>DDS</em> tapes in the sample set could be read without problems. Only 3 tapes resulted in any issues. Two tapes could not be read at all; both of them turned out to be <em>DDS-3</em> tapes, which are not supported by the <em>DDS-2</em> tape drive we used. A <em>DDS-3</em> or <em>DDS-4</em> drive should be able to read these tapes. For one other tape the extraction resulted in a 10-kB file with only null bytes, which most likely means the tape is faulty. Of the 14 <em>DLT-IV</em> tapes, 7 could be read without problems. For the remaining 7, the reading procedure only resulted in a zero-length file. Interestingly, a common characteristic of all “failed” tapes is that they were written at 40.0 GB capacity, whereas the other tapes were written at 35.0 GB capacity. This is odd, as our <em>DLT-IV</em> drive <em>does</em> support 40.0 GB capacity tapes (which was confirmed by writing some data to a blank test tape at 40.0 GB capacity, which could subsequently be read without problems). This needs some further investigation.</p> <h2 id="next-steps">Next steps</h2> <p>The next step (which we haven’t started yet) is to extract the contents of the recovered files. A cursory look suggests that most recovered files in our sample set use the <a href="http://fileformats.archiveteam.org/wiki/Unix_dump">Unix <em>dump</em> format</a>, which can be opened and extracted relatively easily. There are also some <a href="https://en.wikipedia.org/wiki/Tar_(computing)"><em>tar</em></a> archives, which are even easier to extract. Once that is done, the real job of reconstructing the web sites that they contain can start, but that will be a different story altogether.</p> <h2 id="workflow-descriptions-and-other-resources">Workflow descriptions and other resources</h2> <ul> <li> <p>The workflow descriptions for <a href="https://github.com/KBNLresearch/forensicImagingResources/blob/master/doc/tape-dds.md"><em>DDS-1</em></a> and <a href="https://github.com/KBNLresearch/forensicImagingResources/blob/master/doc/tape-dlt.md"><em>DLT-IV</em></a> tapes are available on Github. They also include a detailed description of all hardware components we used, including links to original documentation (if available).</p> </li> <li> <p>The <a href="https://github.com/KBNLresearch/tapeimgr"><em>tapeimgr</em> software is available here</a>.</p> </li> <li> <p>Finally, we’re maintaining a categorised <a href="https://github.com/KBNLresearch/forensicImagingResources/blob/master/doc/df-resources.md">Digital forensics and web archaeology resources list</a>, which contains links to many additional tape-related resources.</p> </li> </ul> <h2 id="acknowledgements">Acknowledgements</h2> <p>Thanks are due to Peter Boel and René van Egdom for their help digging out the tape drives and other obscure hardware peripherals, and Willem Jan Faber for various helpful hardware-related suggestions.</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2019/01/31/roll-the-tape-recovering-90s-data-tapes-in-bitcurator/">Open Preservation Foundation blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>Also see <a href="https://upload.wikimedia.org/wikipedia/commons/1/15/PCI_Keying.svg">this useful diagram</a> that shows different PCI card types. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>Importantly, for the testing phase we only used some unimportant tapes that we still happened to have lying around. This was done to minimise any chance of accidental damage to the tapes that were lended to us (we did not know in advance whether the tape drives were still working correctly!). <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2019/01/31/roll-the-tape-recovering-90s-data-tapes-in-bitcurator https://bitsgalore.org/2019/01/31/roll-the-tape-recovering-90s-data-tapes-in-bitcurator 2019-01-31T00:00:00+01:00 <p>In a <a href="/2018/04/24/resurrecting-the-first-dutch-web-index-nl-menu-revisited">previous blog post</a> I showed how we resurrected <em>NL-menu</em>, the first Dutch web index. It explains how we recovered the site’s data from an old CD-ROM, and how we subsequently created a local copy of the site by <a href="https://github.com/KBNLresearch/nl-menu-resources/blob/master/doc/serving-static-website-with-Apache.md">serving the CD-ROM’s contents on the <em>Apache</em> web server</a>. This follow-up post covers the final step: crawling the resurrected site to a <a href="https://en.wikipedia.org/wiki/Web_ARChive">WARC</a> file that can be ingested into our web archive.</p> <!-- more --> <h2 id="previous-work">Previous work</h2> <p>A <a href="https://kia.pleio.nl/file/download/55806143/Report%20on%20web-archiving%20in%20the%20Dutch%20National%20Archives.pdf">2016 report by Jeroen van Luin</a> documents the web archiving workflows that are used by the National Archives of the Netherlands. Interestingly, it also covers the archiving of websites that are no longer online from local copies. Since this is similar to our <em>NL-menu</em> case, I took this as a starting point. The report mentions two general workflows for crawling from localhost:</p> <ol> <li>One based on the <a href="https://github.com/internetarchive/heritrix3"><em>Heritrix</em></a> crawler.</li> <li>Another one that uses the <a href="https://www.gnu.org/software/wget/"><em>wget</em></a> tool.</li> </ol> <h2 id="preparation-change-machine-system-date-disable-network-connection">Preparation: change machine system date, disable network connection</h2> <p>In most cases it will be desirable that the site snapshot will appear in the Wayback timeline around the year/date it was actually online. This can be achieved by setting the computer’s system date to that date. This has to be done before running the crawl. In the case of <em>NL-menu</em> I used the “last modified” time stamp of the files on the CD-ROM filesystem as an approximation. On a Linux-based system the following command will do the trick:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo date</span> <span class="nt">--set</span><span class="o">=</span><span class="s2">"2004-01-23 21:03:09.000"</span> </code></pre></div></div> <p>Also, to completely rule out anything from the “live” web leaking into the crawl, it may be prudent to disable the network connection at this point (both wired and wireless connections).</p> <h2 id="heritrix">Heritrix</h2> <p>I started out with some limited tests with <em>Heritrix</em> 3, but this resulted in several problems. Most importantly, <em>Heritrix</em> appeared to ignore the <em>hosts</em> file on my machine (this file maps the domain <em>www.nl-menu.nl</em> to the <em>localhost</em> IP-adress). The effect of this was, that with the network disabled the crawl job would run indefinitely without ever downloading any data. After enabling the network, <em>Heritrix</em> would crawl <a href="http://www.nl-menu.nl/">the “live” site at <em>www.nl-menu.nl</em></a> instead of the locally resurrected version<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. Because of this, I quickly gave up on <em>Heritrix</em>, and moved on to <em>wget</em>.</p> <h2 id="wget-first-attempt">wget: first attempt</h2> <p>After some experimentation, the following set of <em>wget</em> options appeared to work reasonably well<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget <span class="nt">--mirror</span> <span class="se">\</span> <span class="nt">--page-requisites</span> <span class="se">\</span> <span class="nt">--adjust-extension</span> <span class="se">\</span> <span class="nt">--warc-file</span><span class="o">=</span><span class="s2">"nl-menu"</span> <span class="se">\</span> <span class="nt">--warc-cdx</span> <span class="se">\</span> <span class="nt">--output-file</span><span class="o">=</span><span class="s2">"nl-menu.log"</span> <span class="se">\</span> http://www.nl-menu.nl/ </code></pre></div></div> <p>The above command results in a 200 MB compressed WARC file, a mirror of the crawled directory tree, a CDX index file (not really nececessary, but useful for quality checks), and a log file.</p> <h2 id="rendering-the-warc">Rendering the WARC</h2> <p>In order to test the rendering of the WARC locally, I installed <a href="https://github.com/webrecorder/pywb"><em>pywb</em></a> (which is part of the Webrecorder project). I then created a test archive using:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wb-manager init my-web-archive </code></pre></div></div> <p>and then added my newly created WARC using:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wb-manager add my-web-archive ~/NL-menu/warc-wget/nl-menu.warc.gz </code></pre></div></div> <p>I then started the server with the command:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wayback </code></pre></div></div> <p>After this the archive is available from <a href="http://localhost:8080/my-web-archive/">http://localhost:8080/my-web-archive/</a>. Below is a screenshot of one page:</p> <p><img src="/images/2018/07/NL-menu-pywb.png" alt="" /></p> <p>Note that the <em>Archived</em> date as shown by <em>pywb</em> corresponds to our modified system date (i.e. 23 january 2004).</p> <h2 id="completeness-checks">Completeness checks</h2> <p>An analysis of the completeness of the <em>wget</em> capture<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">3</a></sup> revealed that over 660 files from the <em>NL-menu</em> source directory tree were missing in the crawl. Most (90%) of these were missing because they are simply not referenced (by way of a hyperlink) by any of the website resources that are crawled from the site root. Of the remaining 64 missing files, 51 are referenced through JavaScript variables (which are understandably not picked up by wget’s crawl mechanism). Other, less common reasons were:</p> <ul> <li>A file is only referenced through a <em>value</em> attribute of an <em>input</em> element.</li> <li>A file is only referenced through a <em>src</em> attribute of a <em>frame</em> element.</li> </ul> <p>This raises the obvious question: can we force <em>wget</em> to crawl <em>all</em> files in the source directory? Of course we can!</p> <h2 id="wget-use-input-file-switch-with-url-list">Wget: use –input-file switch with URL list</h2> <p>The solution here is to use <em>wget</em>’s <code class="language-plaintext highlighter-rouge">--input-file</code> switch, which takes a list of URLs which are sequentially crawled. As a first step we need to create a directory listing of the source directory of the website, and then transform each file entry into a corresponding URL. I did this using the following command:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>find /var/www/www.nl-menu.nl <span class="nt">-type</span> f <span class="se">\</span> | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/\/var\/www\//http:\/\//g'</span> <span class="o">&gt;</span> urls.txt </code></pre></div></div> <p>I then ran <em>wget</em><sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget <span class="nt">--page-requisites</span> <span class="se">\</span> <span class="nt">--warc-file</span><span class="o">=</span><span class="s2">"nl-menu"</span> <span class="se">\</span> <span class="nt">--warc-cdx</span> <span class="se">\</span> <span class="nt">--output-file</span><span class="o">=</span><span class="s2">"nl-menu.log"</span> <span class="se">\</span> <span class="nt">--input-file</span><span class="o">=</span>urls.txt </code></pre></div></div> <p>This results in a WARC file that contains <em>all</em> files from the source directory. But it does introduce a different problem: when the WARC is accessed using <em>pywb</em>, it shows up as over 80 thousand individual captures (i.e. each file appears to be treated as an individual capture)! This makes rendering of the WARC near impossible (if only because loading the list of captures is extremely slow to begin with).</p> <p>After getting in touch with <em>pywb</em> author Ilya Kreymer, Ilya pointed out that <em>pywb</em>’s behaviour here is the combined effect of a bug in <em>pywb</em> and a peculiarity of the <em>NL-menu</em> directory tree: unlike most websites, it does not contain an index document (e.g. <em>index.html</em>) at the domain root level. Instead, the domain root contains 2 directories which hold the Dutch and English-language versions of the site, respectively:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/var/www/www.nl-menu.nl/ ├── nlmenu.en │   ├── index.html │   ├── ... │   ├── ... │   └── ... └── nlmenu.nl ├── index.html ├── ... ├── ... └── ... </code></pre></div></div> <p>The result of this is that the input URL list (which is derived from files in the directory tree) does not contain the site’s root URL (<em>http://www.nl-menu.nl</em>), which in turn ends up missing in the WARC. Ultimately this leads <em>pywb</em> to do a prefix query which in this case results in 80 thousand URLs!</p> <h2 id="improved-url-list">Improved URL list</h2> <p>Ilya suggested to avoid this problem by explicitly adding the domain root to the URL list:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s2">"http://www.nl-menu.nl/"</span> <span class="o">&gt;</span> urls.txt </code></pre></div></div> <p>Incidentally we also need to add entries for the root directories of the Dutch and English language sub-sites:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s2">"http://www.nl-menu.nl/nlmenu.nl/"</span> <span class="o">&gt;&gt;</span> urls.txt <span class="nb">echo</span> <span class="s2">"http://www.nl-menu.nl/nlmenu.en/"</span> <span class="o">&gt;&gt;</span> urls.txt </code></pre></div></div> <p>Then we can add the remaining files (and rewrite file paths as URLs) as before:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>find /var/www/www.nl-menu.nl <span class="nt">-type</span> f <span class="se">\</span> | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/\/var\/www\//http:\/\//g'</span> <span class="o">&gt;</span> urls.txt </code></pre></div></div> <p>Finally run <em>wget</em>:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget <span class="nt">--page-requisites</span> <span class="se">\</span> <span class="nt">--warc-file</span><span class="o">=</span><span class="s2">"nl-menu"</span> <span class="se">\</span> <span class="nt">--warc-cdx</span> <span class="se">\</span> <span class="nt">--output-file</span><span class="o">=</span><span class="s2">"nl-menu.log"</span> <span class="se">\</span> <span class="nt">--input-file</span><span class="o">=</span>urls.txt </code></pre></div></div> <p>This results in a WARC that is complete <em>and</em> renders in <em>pywb</em> as well. On a side note, although rendering and navigating the site works well overall, there are still some small issues. For one thing, coming from the Dutch or English-language home page, clicking on the link to the site’s <em>registration</em> page results in a <em>Url Not Found</em> error, even though this page loads without problems coming from any other page on the site. I don’t really understand why this happens, although I suspect a combination of JavaScript and early-2000s <a href="https://en.wikipedia.org/wiki/Framing_(World_Wide_Web)">frames</a> madness may be to blame here.</p> <h2 id="authenticity-and-provenance">Authenticity and provenance</h2> <p>My <a href="http://openpreservation.org/blog/2018/04/24/resurrecting-the-first-dutch-web-index-nl-menu-revisited/">previous blog post</a> contained the following observation on authenticity and provenance:</p> <blockquote> <p>For one thing, we need to record metadata that makes it absolutely clear that our archival snapshot was taken from a locally reconstructed copy, rather than the original site.</p> </blockquote> <p>So the more specific question is: does the WARC that we just created contain any metadata about the provenance of this snapshot? To find out, let’s use the <em>warcdump</em> tool that is part of the <a href="https://github.com/internetarchive/warctools"><em>warctools</em></a> toolkit:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>warcdump nl-menu.warc.gz <span class="o">&gt;</span> nl-menu-dump.txt </code></pre></div></div> <p>This results in a (huge) text file with metadata about all archived resources inside the WARC. Here is an example of one (request) record:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>archive record at nl-menu.warc.gz:75708014 Headers: WARC-Type:request WARC-Target-URI:&lt;http://www.nl-menu.nl/nlmenu.en/index.html&gt; Content-Type:application/http;msgtype=request WARC-Date:2004-01-23T20:04:54Z WARC-Record-ID:&lt;urn:uuid:4b05df6b-d408-4f2f-8efb-bbd38098cbdb&gt; WARC-IP-Address:127.0.0.1 WARC-Warcinfo-ID:&lt;urn:uuid:7dacf508-ab81-4244-ae9c-ce04a1e18123&gt; WARC-Block-Digest:sha1:MJEIKQUIEOMARJPUEYXZFT4TTTUKX2IE Content-Length:159 Content Headers: Content-Type : application/http;msgtype=request Content-Length : 159 Content: GET /nlmenu\x2Een/index\x2Ehtml HTTP/1\x2E1\xD\xAUser\x2DAgent\x3A Wget/1\x2E19 \x28linux\x2Dgnu\x29\xD\xAAccept\x3A \x2A/\x2A\xD\xAAccept\x2DEncoding\x3A identity\xD\xAHost\x3A www\x2Enl\x2Dmenu\x2Enl\xD\xAConnection\x3A Keep\x2DAlive\xD\xA\xD\xA ... </code></pre></div></div> <p>Note this line:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>WARC-IP-Address:127.0.0.1 </code></pre></div></div> <p>The field <em>WARC-IP-Address</em> is defined in the <a href="https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/#warc-ip-address">WARC specification</a> as:</p> <blockquote> <p>The WARC-IP-Address is the numeric Internet address contacted to retrieve any included content. An IPv4 address shall be written as a “dotted quad”; an IPv6 address shall be written as specified in [RFC4291]. For a HTTP retrieval, this will be the IP address used at retrieval time corresponding to the hostname in the record’s target-Uri.</p> </blockquote> <p>In this case, the value 127.0.0.1 (=localhost) unambiguously shows that this resource was crawled from a local copy, and not from the live web. So the required provenance metadata does indeed exist<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">5</a></sup></p> <h2 id="conclusion">Conclusion</h2> <p>The above steps complete the recovery and archiving of the <em>NL-menu</em> website. In the coming months the KB will start some more “web archaeology” activities, and it will be interesting to see to what extent these <em>wget</em> “recipes” will work for other offline web content. In any case they may be a useful starting point.</p> <h2 id="acknowledgements">Acknowledgements</h2> <p>Thanks are due to Ilya Kreymer, Raffaele Messuti and Andy Jackson for their helpful suggestions on <em>wget</em> and <em>pywb</em>, and René Voorburg for his suggestions on improving and quality-checking the crawl.</p> <h2 id="additional-resources">Additional resources</h2> <ul> <li> <p>Jeroen van Luin, <a href="https://kia.pleio.nl/file/download/55806143/Report%20on%20web-archiving%20in%20the%20Dutch%20National%20Archives.pdf">Experiences with web archiving in the Dutch National Archives</a>. Report, National Archives of the Netherlands</p> </li> <li> <p><a href="https://github.com/KBNLresearch/nl-menu-resources/blob/master/doc/qa-archived-site.md">Rough working notes on quality/completeness checks</a> - unedited notes on some of the tests I did to assess the crawl quality (mostly in terms of completeness)</p> </li> <li> <p><a href="https://github.com/webrecorder/pywb">pywb</a>, Core Python Web Archiving Toolkit for replay and recording of web archives</p> </li> <li> <p><a href="https://github.com/internetarchive/warctools">warctools</a></p> </li> </ul> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2018/07/11/crawling-offline-web-content-the-nl-menu-case/">Open Preservation Foundation blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>A site still exists at this domain, even though its contents are now largely unrelated to the original <em>NL-menu</em> <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>Compared with the examples in the NA documentation, this leaves out the <em>-w</em> (wait) switch (since we are crawling from a local machine it is safe to crawl at maximum speed), the <em>-k</em> (convert links) switch, and the <em>-E</em> (adjust extension) switch. It also adds the <em>–warc-cdx</em> command (which writes an index file) and the <em>–output-file</em> switch (which writes a log) <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>A detailed discussion of the methods that were used for this analysis is beyond the scope of this blog post; however some <a href="https://github.com/KBNLresearch/nl-menu-resources/blob/master/doc/qa-archived-site.md">rough working notes are available here</a>. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>Note that I removed the <code class="language-plaintext highlighter-rouge">--mirror</code> option here, as it seems that this causes <em>wget</em> to do a recursive crawl <em>for each single URL</em> in the list. The result of this is that <em>wget</em> keeps crawling for hours without any (new) data being added to the crawl result. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>How this particular type of provenance metadata is exposed to a user of the web archive is a different matter <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2018/07/11/crawling-offline-web-content-the-nl-menu-case https://bitsgalore.org/2018/07/11/crawling-offline-web-content-the-nl-menu-case 2018-07-11T00:00:00+02:00 <p><em>NL-menu</em> was the first Dutch web index. The site was originally founded by a consortium of <a href="https://en.wikipedia.org/wiki/SURFnet">SURFnet</a>, Dutch universities and the KB. From the mid-nineties onwards it was maintained solely by the KB. <em>NL-menu</em> was <a href="https://www.robcoers.nl/nl-menu-is-straks-niet-meer-leve-nl-menu/">discontinued in 2004</a>, after which the site was taken offline. In 2006 the domain name was sold to a private company that used it for hosting a web index that was partially based on the original <em>NL-menu</em> site.</p> <!-- more --> <p>Meanwile, the original <em>NL-menu</em> has been largely lost to the mists of time. Even though the Internet Archive’s Wayback Machine contains <a href="https://web.archive.org/web/*/www.nl-menu.nl">rather a lot of snapshots of the site</a>, these are incomplete, and don’t capture the original look and feel. For example, <a href="https://web.archive.org/web/20020603232609/http://www.nl-menu.nl:80/nlmenu.nl/fset/gz.html">this page</a> is a snapshot from June 2002:</p> <p><img src="/images/2018/04/wayback1.png" alt="" /></p> <p>However, this doesn’t look even remotely like the site as it was in 2002. Just for one thing, at the top-left we see a <a href="https://en.wikipedia.org/wiki/Bing_(search_engine)"><em>Bing</em></a> search box, but <em>Bing</em> didn’t even exist until 2009! An inspection of the crawl time stamps (these can be seen by clicking on the top-right <em>About this capture</em> button) reveals that this “snapshot” is really an amalgam of elements that were crawled at wildly varying dates, some as recently as 2018:</p> <p><img src="/images/2018/04/wayback-timestamps.png" alt="" /></p> <p><em>NL-menu</em> is not part of the <a href="https://www.kb.nl/en/organisation/research-expertise/long-term-usability-of-digital-resources/web-archiving">KB Web Archive</a>, as the KB only started its web archiving activities in 2007. The only remaining “complete” copies of <em>NL-menu</em> are three (recordable) CD-ROMs that were burned shortly before the site was taken offline in 2004.</p> <p>As <em>NL-menu</em> is a unique source of information about the (relatively) early history of the Dutch Internet, we made an attempt at reconstructing the site as it appeared in early 2004. This involved the following steps:</p> <ol> <li>Recover the data from the remaining CD-ROMs</li> <li>Set up a local copy of the site by serving the recovered data om a webserver</li> <li>Crawl the recovered site for inclusion in our web archive</li> </ol> <p>The remainder of this blog describes how we went about the first two steps.</p> <h2 id="recovering-the-data">Recovering the data</h2> <p>A first attempt at viewing the contents of the CD-ROMs in a file manager resulted in read errors for <em>all</em> discs. This is not surprising, given the instability of CD-Rs, and the fact these discs were burned in early 2004. So, we tried to recover the contents of the discs with the dedicated data-recovery tool <a href="https://www.gnu.org/software/ddrescue/"><em>ddrescue</em></a>. We used the following command line:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ddrescue <span class="nt">-d</span> <span class="nt">-b</span> 2048 <span class="nt">-r4</span> <span class="nt">-v</span> /dev/sr0 NL-menu-ddrescue.iso NL-menu-ddrescue.log </code></pre></div></div> <p>Here <em>-d</em> tells <em>ddrescue</em> to read the disc using direct disc access mode, <em>-b</em> sets the block size (which is 2048 bytes for a CD-ROM); <em>-r4</em> sets the maximum number of retries in case of bad sectors to 4, and <em>-v</em> activates verbose output mode. File <em>NL-menu-ddrescue.iso</em> is the image file with the recovered data; <em>NL-menu-ddrescue.log</em> is a so-called <a href="https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html#Mapfile-structure"><em>mapfile</em></a>, which holds information on the recovery status of blocks of data.</p> <p>One of the advantages of <em>ddrescue</em> is that it can be run multiple consecutive times for each disc, using different optical drives if necessary. This is extremely useful, as it is not uncommon to find that some sectors on a disc result in read errors on one drive, whereas those sectors are read without problems by another drive (and vice versa).</p> <h2 id="results-of-recovery-process">Results of recovery process</h2> <p>Out of the three CD-ROMs, only one copy could be fully recovered without any unreadable sectors. The recovery process required multiple passes with <em>ddrescue</em>, using two computers and four different optical drives (two internal drives, and two external USB drives).</p> <p>Only half of the second disc could be recovered after a 16-hour recovery pass with <em>ddrescue</em>. An inspection of the resulting ISO image in a hex editor showed the recovered sectors of this disc to be byte-identical to the first disc (which was recovered in full). Having established this, we didn’t do any further attempts at recovering more data from this disc (since it is simply a copy of the first disc).</p> <p>For the third disc, 99.8% of the data could be recovered after four rounds with <em>ddrescue</em> with four optical drives. Below image shows a visualisation of the recovery process (made with <a href="https://sourceforge.net/projects/ddrescueview/"><em>ddrescueview</em></a>):</p> <p><img src="/images/2018/04/ddrescue-cd3.png" alt="" /></p> <p>Here, each block respresents one 2048-byte sector, where a red block is a sector with read errors. In this case 468 sectors spread across the disc are unreadable. This means that any files or folder definitions that occupy any of those sectors will be damaged. The resulting ISO image turned out to be readable, but one of the top-level directories (which contains half of the files on the disc) is not shown when the image is mounted or opened in an archive manager. So, we discarded this ISO image from any further processing as well. Unfortunately this disc did <em>not</em> turn out to be merely a copy of the first disc.</p> <h2 id="inspecting-the-contents-of-the-iso-image">Inspecting the contents of the ISO image</h2> <p>After mounting the ISO image of the first disc (i.e. the one that was recovered without errors) on a Linux machine, the following directory structure appears:</p> <p><img src="/images/2018/04/caja-1.png" alt="" /></p> <p>The <em>nlmenu.nl</em> directory contains the Dutch-language version of the site, and <em>nlmenu.en</em> the English-language version (oddly, there’s no top-level index page!). Here are the contents of the <em>nlmenu.nl</em> directory:</p> <p><img src="/images/2018/04/caja-2.png" alt="" /></p> <p>If we open <em>index.html</em> in a browser (Firefox) we see this:</p> <p><img src="/images/2018/04/index-from-fs.png" alt="" /></p> <p>We can see here that several images are not rendered; also none of the (internal) hyperlinks work. This happens because all file paths in the underlying HTML are defined relative to the site’s root directory, and these don’t resolve properly on the local file system. In order to render the site correctly we have to serve it from a locally installed web server.</p> <h2 id="serving-the-cd-rom-contents-with-a-web-server">Serving the CD-ROM contents with a web server</h2> <p>So, we installed the <a href="https://en.wikipedia.org/wiki/Apache_HTTP_Server"><em>Apache web</em> server</a> on a Linux machine, and then configured it to serve the unpacked contents of the ISO image. More details on how we did this can be found in <a href="https://github.com/KBNLresearch/nl-menu-resources/blob/master/doc/serving-static-website-with-Apache.md">these technical notes</a>. Configuring the <em>hosts</em> file (as explained in the technical notes) allowed us to render the site on <a href="https://en.wikipedia.org/wiki/Localhost">localhost</a> from its original URL:</p> <p><img src="/images/2018/04/nl-menu-nl.png" alt="" /></p> <p>We could crawl the resurrected site with Heritrix for inclusion in our web archive, but before doing this a number of authenticity-related issues need to be sorted out first. For one thing, we need to record metadata that makes it absolutely clear that our archival snapshot was taken from a locally reconstructed copy, rather than the original site. Also, it’s not completely straightfoward how the archiving date should be defined (is this 2004 ore 2018?). We’d be very interested to hear how other web archives are dealing with these issues.</p> <h2 id="publicly-available-version-of-the-recovered-site">Publicly available version of the recovered site</h2> <p>The KB web archive is only accessible on-site in our reading rooms. Since the KB owns the rights to <em>NL-menu</em>, we decided to make the reconstructed site available on the KB Research website. In order to make this work, we applied a couple of small changes to the original files:</p> <ul> <li>Relative references to website resources were re-written to reflect the location of the site on the kbresearch domain.</li> <li>All references to the original <em>nl-menu.nl</em> domain were updated to the <em>kbrearch.nl</em> domain (most importantly this prevents JavaScript-triggered redirects to the original live <em>nl-menu.nl</em> domain).</li> <li>The Dutch index page was copied to the site root, so that it’s used as a top-level index (this was done because the CD-ROM has no top-level index page).</li> </ul> <p>The above changes were all made using this <a href="https://github.com/KBNLresearch/nl-menu-resources/blob/master/scripts/fixhtml.sh">script</a>.</p> <p>The result of all this is available here:</p> <p><a href="http://www.kbresearch.nl/nl-menu/nl-menu/">http://www.kbresearch.nl/nl-menu/nl-menu/</a></p> <p>This reflects the state of <em>NL-menu</em> briefly before it closed down in 2004.</p> <p>Although at first glance the reconstructed site appears to be of much better quality than any of the available Wayback snaphots, there are a couple of caveats:</p> <ul> <li>Links to categories that contain sub categories don’t work in Firefox. A workaround is to right-click on the link and open it in a new tab (or disable JavaScript). In Chrome/Chromium these links work normally.</li> <li>The site contains a number of forms that don’t work because the associated CGI scripts are missing (these scripts are not on the CD-ROM).</li> <li>Some pages show the URL <em>http://www.kbresearch.nl</em>, instead of the original <em>http://www.nl-menu.nl</em>. This is an unintended side-effect of the script that was used to update the links.</li> </ul> <p>There may be more issues; please feel free to contact us if you spot anything that doesn’t look quite right!</p> <h2 id="acknowledgements">Acknowledgements</h2> <p>Thanks are due to the folllowing people for their advice and suggestions: Annemarie Beunen, Willem Jan Faber, Kees Teszelszky, and Lammert Zwaagstra.</p> <h2 id="additional-resources">Additional resources</h2> <ul> <li><a href="http://www.kbresearch.nl/nl-menu/nl-menu/">NL-menu (2004 snapshot at kbresearch.nl)</a></li> <li><a href="https://github.com/KBNLresearch/nl-menu-resources/blob/master/doc/serving-static-website-with-Apache.md">Serving a static website with the Apache web server (technical notes)</a></li> <li><a href="https://zenodo.org/record/881109">How can we improve our web collection? An evaluation of web archiving at the KB National Library of the Netherlands (2007-2017)</a></li> </ul> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2018/04/24/resurrecting-the-first-dutch-web-index-nl-menu-revisited/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2018/04/24/resurrecting-the-first-dutch-web-index-nl-menu-revisited https://bitsgalore.org/2018/04/24/resurrecting-the-first-dutch-web-index-nl-menu-revisited 2018-04-24T00:00:00+02:00 <p>Earlier this year I <a href="/2017/01/13/detecting-broken-iso-images-introducing-isolyzer">blogged about <em>Isolyzer</em></a>, a tool designed to help the detection of broken ISO images. Today I released a shiny new <a href="https://github.com/KBNLresearch/isolyzer">beta version</a> that adds a significant amount of new functionality. Below is an overview of the main changes, followed by some warnings and caveats.</p> <!-- more --> <h2 id="support-of-more-file-systems">Support of more file systems</h2> <p>Where previous versions only supported disc images with an <a href="https://en.wikipedia.org/wiki/ISO_9660"><em>ISO 9660</em></a> file system (with limited support for hybrid <em>ISO 9660</em>/Apple <em>HFS</em> file systems), the new release can deal with a much broader range of file systems. In particular, it adds support for the <a href="https://en.wikipedia.org/wiki/Universal_Disk_Format"><em>Universal Disk Format</em></a> (<em>UDF</em>) and Apple’s <a href="https://en.wikipedia.org/wiki/HFS_Plus"><em>HFS+</em></a> file system. Unlike previous versions, <em>Isolyzer</em> can now also deal with Apple disc layouts that don’t contain a <a href="https://en.wikipedia.org/wiki/Apple_Partition_Map">partition map</a> (see also <a href="https://en.wikipedia.org/wiki/Hybrid_disc#Multiple_file_systems">here</a> for more details on Apple disc layouts). Crucially, all of the above are supported both as stand-alone file systems (e.g. a CD image with exclusively a <em>HFS+</em> file system) as well as in various hybrid configurations (e.g. <a href="http://www.afterdawn.com/glossary/term.cfm/udf_bridge"><em>UDF Bridge</em></a> format).</p> <p>Details on how <em>Isolyzer</em> performs the calculations for estimating the expected image size in each of these situations can be found <a href="https://github.com/KBNLresearch/isolyzer#calculation-of-the-expected-file-size">here</a>. As it turns out, for UDF this calculation is not as straightforward as one would expect. The result is that <em>Isolyzer</em> typically under-estimates the true image size by several sectors. In most cases this is unlikely to be a problem; nevertheless, it may be possible to improve this in future versions.</p> <h2 id="changes-to-the-output-format">Changes to the output format</h2> <p>The addition of multiple file system support required some changes to <em>Isolyzer</em>’s output format. The main change is the addition of a <em>fileSystems</em> element, which in turn holds one or more <em>fileSystem</em> elements that each contain information about a detected file system. The updated output format is <a href="https://github.com/KBNLresearch/isolyzer#isolyzer-output">documented here</a>, and some examples are <a href="https://github.com/KBNLresearch/isolyzer#examples">available here</a>. Note that these changes may break existing workflows that use <em>Isolyzer</em>. This is also the main reason for making this a major (1.x) release. It is still possible that the format will change somewhat in the final (stable) release; this will largely depend on any feedback we may get from users of the tool.</p> <h2 id="documentation-and-test-images">Documentation and test images</h2> <p>Finally <em>Isolyzer</em>’s documentation has been given a major overhaul, as you can see from <a href="https://github.com/KBNLresearch/isolyzer/">the main Github page</a>. The repo now includes a <a href="https://github.com/KBNLresearch/isolyzer/tree/master/testFiles">new set of small test images</a> that cover all of the currently supported file systems (with the exception of <em>HFS</em>/<em>HFS+</em> file systems with a partition map, for which I’ve been unable to create or find a sufficiently small sample).</p> <h2 id="installation">Installation</h2> <p>You can install <em>Isolyzer</em> with <a href="https://en.wikipedia.org/wiki/Pip_(package_manager)"><em>pip</em></a>, using the following command:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip <span class="nb">install </span>isolyzer <span class="nt">--user</span> </code></pre></div></div> <p>For Windows users <a href="https://github.com/KBNLresearch/isolyzer/releases/tag/1.0.0">64- and 32-bit binaries are available here</a>. These binaries are completely stand-alone and don’t require Python on your machine.</p> <h2 id="feedback-appreciated">Feedback appreciated</h2> <p>As always feedback on <em>Isolyzer</em> is highly appreciated, if only because any problem its users come across we’ll probably run into ourselves at some point! (Please note though that because of upcoming holidays I may be a little slow in following up any queries.)</p> <h2 id="links">Links</h2> <ul> <li><a href="https://github.com/KBNLresearch/isolyzer/"><em>Isolyzer</em> on Github</a></li> <li><a href="https://github.com/KBNLresearch/isolyzer/releases/tag/1.0.0">Windows binaries (64- and 32-bit)</a>)</li> </ul> <h2 id="postscript-2-february-2019">Postscript (2 February 2019)</h2> <p>Removed some outdated and possibly confusing information from the “Installation” section.</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2017/07/12/update-on-isolyzer-udf-hfs-and-more/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2017/07/12/update-on-isolyzer-udf-hfs-and-more https://bitsgalore.org/2017/07/12/update-on-isolyzer-udf-hfs-and-more 2017-07-12T00:00:00+02:00 <p>Over the last months we’ve been working on the development of a provisional workflow for preserving the content of optical media in our collection. The main result thus far is <a href="https://github.com/KBNLresearch/iromlab"><em>Iromlab</em></a>, a custom workflow application that streamlines the imaging and ripping process. This blogpost gives an overview of <em>Iromlab</em>, as well as the reasons why we created it in the first place.</p> <!-- more --> <h2 id="the-challenges">The challenges</h2> <p>The first challenge is the sheer size of the collection: somewhere between 15,000 and 20,000 discs according to our latest estimates. Processing such volumes efficiently requires a workflow that is highly automated. Secondly, whereas the majority of optical discs in our collection are CD-ROMs, about 25% are audio CDs, which need to be processed quite differently. From the onset it was clear that the workflow should be able to handle both types of discs (we also have a significant number of DVDs as well, but these are processed in a similar manner to CD-ROMs). Finally, it is crucial that the disc images remain linked (through metadata) to their respective records in our catalogue. A particular quirk here is that in most cases, a catalogue record does not describe an individual disc. Often the discs are supplemental to a physical (paper) book. In that case the catalogue record describes the book, with a metadata field indicating that the book contains, for example, a CD-ROM. Another common situation is that the catalogue record describes multiple discs. For example, it may describe a language course that is made up of one DVD, 2 CD-ROMs and 3 audio CDs. In all these cases, there is a <a href="https://en.wikipedia.org/wiki/One-to-many_(data_model)">one-to-many</a> relationship between the catalogue record and the entities (books, optical discs) that it describes. Because of this, it is important that the workflow retains a record of the sequential order of discs within one catalogue record.</p> <h2 id="disc-robot">Disc robot</h2> <p>The level of automation that is required for this job implied we would need to use some kind of disc robot. Based on <a href="https://arxiv.org/abs/1309.4932">earlier work by colleagues at the British Library</a>, we bought an <a href="http://www.acronova.com/product/auto-blu-ray-duplicator-publisher-ripper-nimbie-usb-nb21/9/review.html"><em>Acronova Nimbie</em></a> machine (NB21-DVD type) for evaluation. The first thing that became obvious was that the software bundled with the machine has many limitations, making it unsuitable for our needs. Somewhat disappointingly, the bundle does not include any tools or an API to operate the disc loading mechanism (load, unload or reject a disc). However, a quick test with the <a href="https://www.dbpoweramp.com/batch-ripper.htm"><em>dBpoweramp</em> batch ripper software</a> (part of <a href="https://www.dbpoweramp.com/"><em>dBpoweramp</em></a>, which had been on top of our shortlist of candidate tools for audio ripping) revealed that this software includes command-line driver tools for loading, unloading and rejecting. This makes it possible to operate the disc robot from a custom-built script (which wraps around these command-line tools). It also enables one to use the machine with pretty much <em>any</em> extraction or audio ripping software that has a command-line interface: after loading a disc, the disc robot’s optical drive can be accessed like any ordinary (internal or external) drive. This sparked the development of <a href="https://github.com/KBNLresearch/iromlab"><em>Iromlab</em></a> (Image and Rip Optical Media Like A Boss), a custom workflow application to streamline the imaging and ripping process.</p> <h2 id="main-workflow-components">Main workflow components</h2> <p>The workflow is built around a number of tried and tested software components. For the disc type identification it uses <a href="https://linux.die.net/man/1/cd-info"><em>cd-info</em></a>, which is part of <a href="https://www.gnu.org/software/libcdio/"><em>libcdio</em></a>, the “GNU Compact Disc Input and Control Library”. Extraction of data tracks from CD-ROMs and DVDs is done with <a href="https://www.isobuster.com/"><em>IsoBuster</em></a>. Audio ripping is done with <a href="https://www.dbpoweramp.com/"><em>dBpoweramp</em></a>. Since <em>dBpoweramp</em> only has a graphical interface, we contacted its author, and through a small development contract with the KB he wrote a <a href="https://github.com/KBNLresearch/iromlab/tree/master/dBpowerampconsolerip">command-line tool</a> for the core ripping software. This enabled us to to integrate <em>dBpoweramp</em> into the workflow as well. Ripped audio files are verified for completeness using either <a href="http://www.etree.org/shnutils/shntool/"><em>Shntool</em></a> (WAVE format) or <a href="https://xiph.org/flac/"><em>flac</em></a> (Flac format). ISO images are checked with <a href="/2017/01/13/detecting-broken-iso-images-introducing-isolyzer"><em>Isolyzer</em></a>. Finally, operation of the disc robot is done using the <em>dBpoweramp</em> driver tools.</p> <h2 id="architecture">Architecture</h2> <p>The workflow (which is written in Python) integrates the above components. The figure below gives an overview:</p> <p><img src="/images/2017/06/iromlabArchitectureSmall.png" alt="" /></p> <p>In short, <em>Iromlab</em> consists of a main module (<em>iromlab.pyw</em>) that defines a graphical interface that is used to enter data about each disc. For each disc the entered data are sent as job files to a file-based processing queue. These jobs are monitored by a worker module (<em>cdworker.py</em>), which orchestrates the main heavy-lifting (disc identification, imaging, etcetera). The jobs are processed in <a href="https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)">“First In First Out”</a> order. As the physical discs are inserted into the machine in the same order, this ensures that the workflow keeps track of the relationship between the discs and the entered data (in the absence of data entry errors!). Since the worker module runs as a subprocess (separate thread from the data entry interface), new discs can be loaded continuously while the imaging/ripping process is ongoing. The overall goal was to make the workflow as simple as possible to use for an operator, which meant that manual data entry is reduced to a minimum.</p> <h2 id="batch-structure">Batch structure</h2> <p>All output of an <em>Iromlab</em> session is written to a batch. Each disc is represented by a directory which contains the extracted data (ISO image, WAVE/Flac files), log files of the extraction software, <em>cd-info</em> output and a SHA-512 checksum file. In addition, each batch has a <em>batch manifest</em>, which is a comma-delimited file that contains all information that is needed to process the batch into ingest-ready <a href="https://documents.clockss.org/index.php?title=Definition_of_SIP">Submission Information Packages</a> (SIPs) further down the processing chain. This includes a simple “success” flag that indicates whether the imaging of a disc was successful. Finally, the batch contains a log file with detailed output about the individual processes in the workflow. For an example of the batch structure, check out this small <a href="https://github.com/KBNLresearch/iromlabDemobatch">online demo batch</a>. Note that for copyright and file size reasons, all ISO and FLAC files from the original batch are replaced by empty (zero-byte) files in this demo.</p> <h2 id="data-entry">Data entry</h2> <p>Upon startup, <em>Iromlab</em> launches a simple graphical interface that is used for data entry. An operator then presses a button to create a new batch. Once the batch is initialised, the following fields need to be entered for each disc:</p> <p><img src="/images/2017/06/iromAllesBestandsformaten.png" alt="" /></p> <p>Here, <em>PPN</em> is the identifier of the catalogue record that is associated with the disc. <em>Volume number</em> defines the sequential order in case multiple discs are associated with one record (e.g. a CD box set). Note that <em>Carrier type</em> does not influence the imaging or ripping process; it is only used to describe the disc at the metadata level. After pressing “Submit”, <em>Iromlab</em> looks up the entered <em>PPN</em> identifier in the KB catalogue using an <a href="https://en.wikipedia.org/wiki/Search/Retrieve_via_URL"><em>SRU</em></a> query, and then prompts the operator to confirm the publication title that was returned:</p> <p><img src="/images/2017/06/iromConfirmTitle.png" alt="" /></p> <p>After confirming this, <em>Iromlab</em> prompts the operator to insert the disc into the disc robot:</p> <p><img src="/images/2017/06/loadDisc.png" alt="" /></p> <p>This second prompt tries to enforce a fixed processing order (i.e. enter the data first, and then insert the disc), which reduces the risk of synchronisation errors between the jobs queue and the stack of physical discs in the disc robot. After pressing “OK”, the job is added to the processing queue.</p> <figure class="image"> <img src="/images/2017/06/workstationSmall.jpg" alt="Photograph of Iromlab with disc robot" /> <figcaption>Iromlab in action; disc robot on the left.</figcaption> </figure> <h2 id="processing-a-disc">Processing a disc</h2> <p>Meanwhile the worker module continuously scans the processing queue for job files. Once it picks up a job, it reads the information from the job file, and tries to load a new disc. The processing of each disc then involves the following steps:</p> <ul> <li>Analyse the disc with <em>cd-info</em>.</li> <li>Record information about the sector layout of the physical disc (this is needed to ensure ISO images extracted from <a href="/2017/04/25/imaging-cd-extra-blue-book-discs">CD-Extra / Bluebook</a> discs are accessible).</li> <li>Rip any audio tracks to WAVE or FLAC; extract data tracks to ISO images.</li> <li>Record information about the extraction process (logs, exit codes).</li> <li>Check the integrity of the extracted/ripped files.</li> <li>Record fixity information (SHA-512 checksums).</li> <li>Record any information that is necessary further down the processing chain for the creation of ingest-ready SIPs.</li> <li>Add an entry for the disc to the batch manifest.</li> </ul> <h2 id="further-processing-of-the-batch">Further processing of the batch</h2> <p>In order to create ingest-ready SIPs from a batch, some further processing is necessary. In particular:</p> <ul> <li>The integrity of the batch needs to be verified (i.e. is the batch complete, were all discs imaged successfully?).</li> <li>Any discs that have problems need to be removed from the batch. Since in our case we decided that each SIP will contain all discs that belong to a catalogue record, this also means that for each problematic disc, <em>all</em> discs that belong to its corresponding catalogue record are removed (i.e. moved to a separate “error batch”).</li> <li>If the batch passed the verification step, it can be transformed into ingest-ready SIPs.</li> </ul> <p>The above steps are outside of <em>Iromlab</em>’s scope, but they are covered by the separate <a href="https://github.com/KBNLresearch/omSipCreator"><em>omSipCreator</em></a> tool. This is a work in progress that still needs some refinements, and it might be the subject of a follow-up blog.</p> <h2 id="using-iromlab-outside-the-kb">Using Iromlab outside the KB</h2> <p><em>Iromlab</em> is tailored specifically to the situation at the KB, and it is not meant to be a general-purpose workflow solution. Nevertheless, others might find (parts of) the software useful. For instance, it would be quite straightforward to adapt those parts of the software that query the KB catalogue to other institutional catalogues or databases. Also, even though <em>Iromlab</em> is tailor-made to a very specific combination of hardware and software, most of the software dependencies are implemented using simple wrapper modules. Adding new ones for alternative (imaging or ripping) tools would be pretty easy.</p> <h2 id="documentation">Documentation</h2> <p>User documentation for <em>Iromlab</em> is <a href="https://github.com/KBNLresearch/iromlab/tree/master/doc">available here</a>. It includes a <a href="https://github.com/KBNLresearch/iromlab/blob/master/doc/userGuide.md">User Guide</a>, as well as detailed instructions on how to <a href="https://github.com/KBNLresearch/iromlab/blob/master/doc/setupIromlab.md">install the software</a>, and how to setup and configure <a href="https://github.com/KBNLresearch/iromlab/blob/master/doc/setupIsobuster.md"><em>IsoBuster</em></a> and <a href="https://github.com/KBNLresearch/iromlab/blob/master/doc/setupDbpoweramp.md"><em>dBpoweramp</em></a>.</p> <h2 id="acknowledgements">Acknowledgements</h2> <p>Thanks are due to <em>dBpoweramp</em> creator “Mr. Spoon” for his work on the <em>dBpoweramp</em> command-line tool, and for allowing us to re-distribute the binaries. <em>IsoBuster</em>’s author Peter van Hove is thanked for some modifications he made to <em>IsoBuster</em>’s command-line interface.</p> <h2 id="links">Links</h2> <ul> <li><a href="https://github.com/KBNLresearch/iromlab"><em>Iromlab</em> on Github</a></li> <li><a href="https://github.com/KBNLresearch/iromlab/tree/master/doc">Documentation</a></li> <li><a href="https://github.com/KBNLresearch/iromlabDemobatch"><em>Iromlab</em> demo batch</a></li> </ul> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2017/06/19/image-and-rip-optical-media-like-a-boss/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2017/06/19/image-and-rip-optical-media-like-a-boss https://bitsgalore.org/2017/06/19/image-and-rip-optical-media-like-a-boss 2017-06-19T00:00:00+02:00 <p>Some four years ago I wrote <a href="/2013/07/25/identification-pdf-preservation-risks-sequel">a blog post</a> that demonstrated how <em>Apache Preflight</em> (the PDF/A validator tool that is part of <a href="https://pdfbox.apache.org/"><em>Apache PDFBox</em></a>) can be used to detect features in a PDF that are potential preservation risks. A <a href="//2014/01/27/identification-pdf-preservation-risks-analysis-govdocs-selected-corpus">follow-up blog</a> applied <a href="https://en.wikipedia.org/wiki/Schematron"><em>Schematron</em></a> rules to the <em>Preflight</em> output in an attempt at doing policy-based assessments. The results of that work were quite promising, but dealing with Preflight’s multitude of (especially font-related) validation errors proved to be a challenge.</p> <p>The idea of using a <em>PDF/A</em> validor for policy-based assessments of “regular” <em>PDF</em> files (i.e. <em>PDF</em>s that are not necessarily <em>PDF/A</em>) was explicitly addressed as a use case for <a href="http://verapdf.org/"><em>veraPDF</em></a>. With <em>VeraPDF</em> now having entered its “final testing phase”, I thought this was a good time for a small test-drive of <em>veraPDF</em>’s capabilities in this area. All test results are based on <em>VeraPDF</em> 1.4.7.</p> <!-- more --> <h2 id="test-data">Test data</h2> <p>For this test I used <em>PDF</em>s from the <a href="https://web.archive.org/web/20130503115947/http://acroeng.adobe.com/wp/"><em>Adobe Acrobat Engineering</em> website</a> (sadly gone since 2015). As in my 2013 blog post, I limited the analysis to:</p> <ul> <li>all files in the <em>General</em> section of the <a href="https://web.archive.org/web/20150228065249/http://acroeng.adobe.com:80/wp/?page_id=101"><em>Font Testing</em></a> category;</li> <li>all files in the <em>Classic Multimedia</em> section of the <a href="https://web.archive.org/web/20150228104639/http://acroeng.adobe.com:80/wp/?page_id=61"><em>Multimedia &amp; 3D Tests</em></a> category.</li> </ul> <p>The dataset is quite small, but contains many complex and otherwise challenging <em>PDF</em>s, which make it an interesting dataset for testing.</p> <h2 id="policy">Policy</h2> <p>The policy is similar to the one used in <a href="/2014/01/27/identification-pdf-preservation-risks-analysis-govdocs-selected-corpus">my 2014 blog post</a>, and it is defined by the following objectives:</p> <ol> <li>No encryption / password protection</li> <li>All fonts are embedded</li> <li>No embedded files</li> <li>No file attachments</li> <li>No multimedia content (audio, video, 3-D objects)</li> <li>No PDFs that raise an exception or result in a processing error in <em>VeraPDF</em> (<em>PDF</em> validity proxy)</li> </ol> <p>(Note that the 2014 blog post also mentioned the absence of <em>JavaScript</em> as an additional objective. However, it turned out that the necessary output for this is not currently reported by <em>VeraPDF</em>.)</p> <p>Subsequently I ‘translated’ each of these objectives into <em>Schematron</em> rules. For a basic <em>how-to</em> see the <a href="http://docs.verapdf.org/policy/"><em>veraPDF Policy Checking</em> documentation</a>.</p> <p>The full <em>Schematron</em> file can be found <a href="https://github.com/KBNLresearch/pdfPolicyVeraPDF/blob/master/schemas/demo-policy.sch">here</a>.</p> <h2 id="verapdf-configuration">VeraPDF configuration</h2> <p>It is important to note that, unlike in my earlier <em>Apache Preflight</em> experiments, the <em>Schematron</em> rules do not rely on the <em>PDF/A</em> validation output! Instead, <em>VeraPDF</em> can be instructed to include a ‘features report’ in its output, which directly points to technical features such as font properties, annotation types, security features, and so on. Most of the features that are needed for a policy-based assessment are disabled by default. So, we first need to activate these in the configuration (file <em>features.xml</em> in <em>VeraPDF</em>’s <em>config</em> directory). I edited it as below:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">&lt;?xml version="1.0" encoding="UTF-8" standalone="yes"?&gt;</span> <span class="nt">&lt;featuresConfig&gt;</span> <span class="nt">&lt;enabledFeatures&gt;</span> <span class="nt">&lt;feature&gt;</span>ANNOTATION<span class="nt">&lt;/feature&gt;</span> <span class="nt">&lt;feature&gt;</span>DOCUMENT_SECURITY<span class="nt">&lt;/feature&gt;</span> <span class="nt">&lt;feature&gt;</span>EMBEDDED_FILE<span class="nt">&lt;/feature&gt;</span> <span class="nt">&lt;feature&gt;</span>FONT<span class="nt">&lt;/feature&gt;</span> <span class="nt">&lt;feature&gt;</span>INFORMATION_DICTIONARY<span class="nt">&lt;/feature&gt;</span> <span class="nt">&lt;/enabledFeatures&gt;</span> <span class="nt">&lt;/featuresConfig&gt;</span> </code></pre></div></div> <h2 id="basic-operation">Basic operation</h2> <p>Supposing that the <em>PDF</em>s we want to analyze are in directory <code class="language-plaintext highlighter-rouge">~/myPdfs</code>, and that the <em>Schematron</em> rules that represent our policy are in the file <code class="language-plaintext highlighter-rouge">demo-policy.sch</code>, we can do a policy-based validation of all these files with one single command:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>verapdf <span class="nt">-x</span> <span class="nt">--policyfile</span> demo-policy.sch ~/myPdfs/<span class="k">*</span> <span class="o">&gt;</span> myPdfsOut.xml </code></pre></div></div> <p>Here the <code class="language-plaintext highlighter-rouge">-x</code> switch activates feature extraction. The output file <code class="language-plaintext highlighter-rouge">myPdfsOut.xml</code> contains, for each <em>PDF</em>, an element with <em>PDF/A</em> validation output, an element with the features report, and an element with the policy report.</p> <h2 id="analysis-script">Analysis script</h2> <p>Typically the <em>VeraPDF</em> output is rather unwieldy. To facilitate things I wrote a <a href="https://github.com/KBNLresearch/pdfPolicyVeraPDF">custom analysis script</a>, which does the following things:</p> <ol> <li>It runs <em>VeraPDF</em></li> <li>It creates a <a href="https://github.com/KBNLresearch/pdfPolicyVeraPDF/blob/master/examples/fonts_san.xml">trimmed-down version of the output file</a> that only contains the policy report. Also, for each PDF, it removes duplicate instances of failed (policy) checks (e.g. if a check on font embedding fails for 10 different fonts, only one reference to the failed check is kept)</li> <li>It creates a <a href="https://github.com/KBNLresearch/pdfPolicyVeraPDF/blob/master/examples/fonts_summary.csv">comma-delimited summary file</a> which lists for each PDF its path/name, followed by the description of each unique failed validation rule (taken from the <em>message</em> element in <em>VeraPDF</em>’s output).</li> </ol> <h2 id="running-the-analysis">Running the analysis</h2> <p>For this analysis I ran the above script for both the <em>fonts</em> and <em>multimedia</em> files, using the following command line (here for the <em>fonts</em> files):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~/pdfPolicyVeraPDF/policyValidate.sh /home/johan/pdfAcrobatEngineering/fonts /home/johan/pdfPolicyVeraPDF/schemas/demo-policy.sch fonts </code></pre></div></div> <h2 id="results-fonts-category">Results, fonts category</h2> <p>The following table lists, for each <em>PDF</em> in the <em>fonts</em> category, the corresponding (unique) validation errors (taken from the summary CSV file). Note that the text strings in the right column correspond to text values in the <em>assert</em> elements of the policy file.</p> <table> <thead> <tr> <th style="text-align: left">Test file</th> <th style="text-align: left">Failed assert(s)</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">EmbeddedCmap.pdf</td> <td style="text-align: left">Font is not embedded</td> </tr> <tr> <td style="text-align: left">embedded_fonts.pdf</td> <td style="text-align: left">Font is not embedded</td> </tr> <tr> <td style="text-align: left">embedded_pm65.pdf</td> <td style="text-align: left"> </td> </tr> <tr> <td style="text-align: left">notembedded_pm65.pdf</td> <td style="text-align: left">Font is not embedded</td> </tr> <tr> <td style="text-align: left">printtestfont_nonopt.pdf</td> <td style="text-align: left"> </td> </tr> <tr> <td style="text-align: left">printtestfont_opt.pdf</td> <td style="text-align: left"> </td> </tr> <tr> <td style="text-align: left">substitution_fonts.pdf</td> <td style="text-align: left">Font is not embedded</td> </tr> <tr> <td style="text-align: left">text_images_pdf1.2.pdf</td> <td style="text-align: left">Font is not embedded</td> </tr> <tr> <td style="text-align: left">TEXT.pdf</td> <td style="text-align: left">Font is not embedded</td> </tr> <tr> <td style="text-align: left">Type3_WWW-HTML.PDF</td> <td style="text-align: left">Font is not embedded</td> </tr> </tbody> </table> <p>These results show that most of the <em>PDF</em>s fail our policy on the font embedding objective.</p> <h2 id="results-multimedia-category">Results, multimedia category</h2> <p>Similarly, below are the results for the <em>multimedia</em> category:</p> <table> <thead> <tr> <th style="text-align: left">Test file</th> <th style="text-align: left">Failed assert(s)</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">20020402_CALOS.pdf</td> <td style="text-align: left">Font is not embedded;Movie annotation</td> </tr> <tr> <td style="text-align: left">3-D_PDF.pdf</td> <td style="text-align: left">3D annotation</td> </tr> <tr> <td style="text-align: left">AdobeChassisDemo-commented.pdf</td> <td style="text-align: left">3D annotation</td> </tr> <tr> <td style="text-align: left">AdobeChassisDemo-commented_Review.pdf</td> <td style="text-align: left">3D annotation</td> </tr> <tr> <td style="text-align: left">AVI+Transitions Demo.pdf</td> <td style="text-align: left">Document not parsable</td> </tr> <tr> <td style="text-align: left">Binder_6-3DPages.pdf</td> <td style="text-align: left">3D annotation</td> </tr> <tr> <td style="text-align: left">Disney-Flash.pdf</td> <td style="text-align: left">Font is not embedded;Screen annotation</td> </tr> <tr> <td style="text-align: left">drape_raster_contour_sample.pdf</td> <td style="text-align: left">Font is not embedded;3D annotation</td> </tr> <tr> <td style="text-align: left">gXsummer2004-stream.pdf</td> <td style="text-align: left">Document not parsable</td> </tr> <tr> <td style="text-align: left">Jpeg_linked.pdf</td> <td style="text-align: left">Encrypted document;Document not parsable</td> </tr> <tr> <td style="text-align: left">LabelExample.pdf</td> <td style="text-align: left">Encrypted document;Document not parsable</td> </tr> <tr> <td style="text-align: left">movie_down1.pdf</td> <td style="text-align: left">Movie annotation</td> </tr> <tr> <td style="text-align: left">movie.pdf</td> <td style="text-align: left">Movie annotation</td> </tr> <tr> <td style="text-align: left">MultiMedia_Acro6.pdf</td> <td style="text-align: left">Encrypted document;Document not parsable</td> </tr> <tr> <td style="text-align: left">MusicalScore.pdf</td> <td style="text-align: left">Font is not embedded;Screen annotation</td> </tr> <tr> <td style="text-align: left">phlmapbeta7.pdf</td> <td style="text-align: left">Font is not embedded;Screen annotation</td> </tr> <tr> <td style="text-align: left">remotemovieurl.pdf</td> <td style="text-align: left">Font is not embedded;Movie annotation</td> </tr> <tr> <td style="text-align: left">ScriptEvents.pdf</td> <td style="text-align: left">Font is not embedded;Screen annotation</td> </tr> <tr> <td style="text-align: left">Service Form_media.pdf</td> <td style="text-align: left">Font is not embedded;Screen annotation</td> </tr> <tr> <td style="text-align: left">SVG-AnnotAnim.pdf</td> <td style="text-align: left">Font is not embedded</td> </tr> <tr> <td style="text-align: left">SVG.pdf</td> <td style="text-align: left">Font is not embedded</td> </tr> <tr> <td style="text-align: left">Trophy.pdf</td> <td style="text-align: left">Font is not embedded;Screen annotation</td> </tr> <tr> <td style="text-align: left">us_population.pdf</td> <td style="text-align: left"> </td> </tr> </tbody> </table> <p>Here the reasons for failing the policy are more diverse. Many of these <em>PDF</em>s contain <em>Screen</em>, <em>Movie</em> or <em>3D</em> annotations. Non-embedded fonts are common as well. Three <em>PDF</em>s were not parsable because of encryption. This turns out to be a <a href="https://github.com/veraPDF/veraPDF-apps/issues/202">a bug</a> that is fixed in newer versions of <em>VeraPDF</em>. Two files (<em>AVI+Transitions Demo.pdf</em> and <em>gXsummer2004-stream.pdf</em>) were not parsable at all. These files could not be opened in Adobe Acrobat either. Finally, one 49 MB file (which is not listed in the table) resulted in an out-of-memory error that crashed <em>VeraPDF</em> altogether. I <a href="https://github.com/veraPDF/veraPDF-apps/issues/195">reported this as a bug</a>.</p> <h2 id="general-observations">General observations</h2> <p>First of all I was impressed with the amount of detailed information that <em>VeraPDF</em> can provide of a <em>PDF</em> file. I was also pleasantly surprised at the relative ease of doing policy-based assessments. This is mainly thanks to <em>VeraPDF</em>’s features report, which allows one to address features such as specific annotation types directly. During my earlier attempts at policy-based assessment with <em>Apache Preflight</em>, the detection of non-embedded fonts was particularly difficult (have a look at the <a href="https://github.com/openpreserve/pdfPolicyValidate/blob/master/schemas/pdf_policy_preflight_test.sch#L55">Schematron file</a> to see what I mean). With <em>VeraPDF</em> this only needs <a href="https://github.com/KBNLresearch/pdfPolicyVeraPDF/blob/master/schemas/demo-policy.sch#L42">one single line</a> (though admittedly this probably means that errors related to damaged or malformed fonts won’t be reported). Thanks to <em>VeraPDF</em>’s built-in functionality to do the Schematron validation, it is no longer necessary to use an external Schematron validator (though this is still possible).</p> <h2 id="actions-missing-in-action">Actions missing in action?</h2> <p>One thing I missed is the reporting of <em>Actions</em>. Without this, it is not possible to identify <em>PDF</em>s that contain <em>JavaScript</em> (and some other features as well). An option to include <em>Actions</em> in the ‘Feature Report’ would make a welcome addition. As the <em>PDF/A</em> validation profiles already include checks on <em>Actions</em>, this is probably pretty straightforward (see also <a href="https://github.com/veraPDF/veraPDF-apps/issues/174">this issue</a>).</p> <h2 id="writing-a-policy-file">Writing a policy file</h2> <p>Not having worked on <em>PDF</em>-related things for a while myself, it took me some time to figure out how to put together the (Schematron) policy file. The <em>VeraPDF</em> <a href="http://docs.verapdf.org/policy/">documentation gives some guidance</a>, but I couldn’t find an exhaustive description of every possible feature in the features report. This meant I first had to run <em>VeraPDF</em> (with feature extraction enabled) on a number of files that I <em>knew</em> to contain certain features I wanted to include in my policy (e.g. embedded fonts, multimedia), inspect the <em>XML</em> output, and then write my Schematron rules based on that output. As I have a pretty good knowledge of the specific <em>PDF</em> data structures involved I was able to do this, but it did make me wonder about users who don’t have that technical knowledge. Possible solutions would be:</p> <ul> <li>Additional documentation of all possible output elements in the features report. This <a href="http://docs.verapdf.org/cli/feature-extraction/">seems to be in the works already</a> (though not complete yet)</li> <li>Inclusion of some example policy files. Actually <em>veraPDF</em>’s Github repo <a href="https://github.com/veraPDF/veraPDF-policy-docs/tree/master/Schemas">contains a number of these</a> already, but they are not (yet) referenced by the documentation, and I only found out about them after I ran my tests.</li> </ul> <p>It would also help if users of <em>veraPDF</em> would publish and share their policy files.</p> <p>Finally it just occurred to me this is a good occasion to give one more bump to <a href="https://doi.org/10.5281/zenodo.801661">this 2009 report I wrote on long-term preservation risks of PDF</a>. It explicitly lists the data structures (e.g. annotations, actions) that are associated with specific (risky) features, which might provide users some guidance as to what features are potentially interesting for inclusion in a policy.</p> <h2 id="links">Links</h2> <ul> <li><a href="https://github.com/KBNLresearch/pdfPolicyVeraPDF">PDF policy-based validation demo, veraPDF</a> - Github repo with scripts, Schematron policy file and all output files</li> <li><a href="http://verapdf.org/">VeraPDF</a></li> <li><a href="https://doi.org/10.5281/zenodo.801661">Adobe Portable Document Format - Inventory of long-term preservation risks</a></li> </ul> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2017/06/01/policy-based-assessment-with-verapdf-a-first-impression/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2017/06/01/policy-based-assessment-with-verapdf-a-first-impression https://bitsgalore.org/2017/06/01/policy-based-assessment-with-verapdf-a-first-impression 2017-06-01T00:00:00+02:00 <p>The development work on an imaging/ripping workflow for optical media is shaping up steadily, and you can expect a write-up with more information about our software and hardware setup here in the near future (you can get a sneak peek <a href="https://github.com/KBNLresearch/iromlab">here</a>). However, this blog is about a very specific problem that we ran into while testing the workflow with a selection of discs from our collection. This selection included a few discs that follow the <a href="https://en.wikipedia.org/wiki/Blue_Book_(CD_standard)"><em>Blue Book</em></a> standard (also known as <em>CD-Extra</em>). This standard defines a method for combining audio and data tracks on one disc. A <em>CD-Extra</em> disc contains two sessions, where the first session holds all audio tracks, and the second session holds a data track. <em>Blue Book</em> was (and still is) widely used for audio CD’s with bonus videos or software.</p> <!-- more --> <p>In our workflow, these discs are handled in the following way:</p> <ol> <li>Identify the audio and data sessions with the <a href="https://linux.die.net/man/1/cd-info"><em>cd-info</em></a> tool (part of the <a href="https://www.gnu.org/software/libcdio/"><em>libcdio</em></a> library).</li> <li>Rip the audio tracks in the first session to <em>WAVE</em> or <em>FLAC</em> files using <a href="https://www.dbpoweramp.com/"><em>dBpoweramp</em></a>.</li> <li>Verify the audio files for completeness with <a href="http://www.etree.org/shnutils/shntool/"><em>shntool</em></a> (<em>WAVE</em>) or <a href="https://xiph.org/flac/"><em>flac</em></a> (<em>FLAC</em>).</li> <li>Extract the data track in the second session to an ISO image with <a href="https://www.isobuster.com/"><em>IsoBuster</em></a>.</li> <li>Verify the ISO image for completeness with <a href="https://github.com/KBNLresearch/isolyzer"><em>isolyzer</em></a>.</li> </ol> <h2 id="size-of-iso-image-smaller-than-expected">Size of ISO image smaller than expected</h2> <p>All <em>Blue Book</em> discs in our test passed steps 1 through 4 without any issues, but failed the final <em>isolyzer</em> check. This <em>isolyzer</em> check involves a comparison of the file size of the ISO image against the <em>expected</em> size, as calculated from the image’s Primary Volume Descriptor fields (and Apple HFS blocks, if present). If the actual size is smaller than the expected size, this indicates the image is incomplete. For <em>all</em> ISO images that were extracted from a <em>Blue Book</em> disc, the actual file size was significantly smaller than expected. Moreover, the images couldn’t be mounted in Linux, or opened in file archiver software (e.g. 7-Zip). Below is an excerpt from the <em>isolyzer</em> output of one of the offending images:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;tests&gt;</span> <span class="nt">&lt;containsISO9660Signature&gt;</span>True<span class="nt">&lt;/containsISO9660Signature&gt;</span> <span class="nt">&lt;containsApplePartitionMap&gt;</span>True<span class="nt">&lt;/containsApplePartitionMap&gt;</span> <span class="nt">&lt;containsAppleHFSHeader&gt;</span>False<span class="nt">&lt;/containsAppleHFSHeader&gt;</span> <span class="nt">&lt;containsAppleMasterDirectoryBlock&gt;</span>False<span class="nt">&lt;/containsAppleMasterDirectoryBlock&gt;</span> <span class="nt">&lt;parsedAppleZeroBlock&gt;</span>True<span class="nt">&lt;/parsedAppleZeroBlock&gt;</span> <span class="nt">&lt;parsedPrimaryVolumeDescriptor&gt;</span>True<span class="nt">&lt;/parsedPrimaryVolumeDescriptor&gt;</span> <span class="nt">&lt;sizeExpected&gt;</span>609912832<span class="nt">&lt;/sizeExpected&gt;</span> <span class="nt">&lt;sizeActual&gt;</span>554373120<span class="nt">&lt;/sizeActual&gt;</span> <span class="nt">&lt;sizeDifference&gt;</span>-55539712<span class="nt">&lt;/sizeDifference&gt;</span> <span class="nt">&lt;sizeAsExpected&gt;</span>False<span class="nt">&lt;/sizeAsExpected&gt;</span> <span class="nt">&lt;smallerThanExpected&gt;</span>True<span class="nt">&lt;/smallerThanExpected&gt;</span> <span class="nt">&lt;/tests&gt;</span> </code></pre></div></div> <p>So, in this case the ISO image is about 56 MB smaller than expected, even though <em>IsoBuster</em> did not report any errors during the extraction process. One thing that caught my eye after running a few of these problematic images through <em>isolyzer</em>, was that the value of <em>sizeDifference</em> always roughly corresponded to the size of the uncompressed audio on the disc. The <em>sizeExpected</em> value is calculated from the <em>Volume Space Size</em> field (which defines the number of logical blocks in the ISO 9660 file system) in the ISO’s Primary Volume Descriptor. This made me wonder: does the <em>Volume Space Size</em> value really reflect the number of sectors occupied by the <em>data track</em> (which I was quietly assuming), or does it perhaps reflect <em>all sectors on the disc</em> (including those in the audio session)?</p> <p>Finding the answer to this question was more difficult than I expected, but my initial suspicions were confirmed by this entry on the Debian mailing list:</p> <p><a href="https://lists.debian.org/debian-user/2005/01/msg02339.html">Re: reading the raw iso from a CD-Extra (multisession CD)</a></p> <p>Which states:</p> <blockquote> <p>[T]he sector numbers in the file system refer[s] to sectors of the original CD rather than sectors of session2.iso.</p> </blockquote> <p>Similarly, from this thread on the <em>libcdio</em> mailing list:</p> <p><a href="https://lists.gnu.org/archive/html/libcdio-devel/2010-02/msg00048.html">[Libcdio-devel] Retrieving DATA session from multisession audio disc</a></p> <blockquote> <p>Remember, the path table and directory structure of the iso reflect the fact that the ISO filesystem starts on sector 222145 (49:23:70) of the CD. If it is burned to another CD at a different position, it won’t work. Likewise, any program that reads the iso will need to be able to compensate for the offset.</p> </blockquote> <p>This confirms that all references to blocks of data (sectors) in these ISO images are defined <em>relative to the start of the physical CD</em>, and <strong>not</strong> relative to the start of the ISO image! So the questions are:</p> <ol> <li>How can we verify that these images are complete?</li> <li>How can we access these images at all?</li> </ol> <p>Using the information from (mostly) the aforementioned <em>libdio</em> mailing list thread I was able to answer both questions. The steps below will work on most Linux-based systems (and possibly some Windows-based ones as well).</p> <h2 id="cd-sector-layout">Cd sector layout</h2> <p>As a first step we need some information on the sector layout of the physical disc, and in particular the start sector of the second session (which contains the data track). You can do this by running <em>cd-info</em> while the disc is in the drive:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd-info /dev/sr0 <span class="o">&gt;</span> leesleeuw_cdinfo.txt </code></pre></div></div> <p>In the resulting output, the start sesssion of the data session is listed twice. First look at the <em>CD-ROM Track List</em> section:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CD-ROM Track List (1 - 3) #: MSF LSN Type Green? Copy? Channels Premphasis? 1: 00:02:00 000000 audio false no 2 no 2: 01:25:24 006249 audio false no 2 no 3: 06:05:46 027271 data false no 170: 66:14:61 297961 leadout (668 MB raw, 668 MB formatted) Media Catalog Number (MCN): 0000000000000 TRACK 1 ISRC: 000000000000 TRACK 2 ISRC: 000000000000 TRACK 3 ISRC: 000000000000 Last CD Session LSN: 27271 </code></pre></div></div> <p>This tells us that the CD contains three tracks, where track 1 and track 2 are audio tracks, and track 3 is a data track. The second column (heading: LSN) shows the start sector of each track; for track 3 this is 027271. This value is repeated in the <em>Last CD Session LSN</em> (sector number of last session). Finally it can be found again in the <em>CD Analysis Report</em> at the bottom of the file:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>session #2 starts at track 3, LSN: 27271, ISO 9660 blocks: 297809 </code></pre></div></div> <p>So, in this case session 2 (which contains the data track) starts on sector 27271 of the disc.</p> <h2 id="verify-iso-image-for-completeness">Verify ISO image for completeness</h2> <p>Now that we know the start sector, we can use this to verify the ISO image. For this I added the new <code class="language-plaintext highlighter-rouge">--offset</code> option to <em>isolyzer</em>. This lets you specify a start sector offset, which is subtracted from the expected size estimate that is calculated from the Primary Volume Descriptor<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. So we call <em>isolyzer</em> like this:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>isolyzer leesleeuw.iso <span class="nt">--offset</span> 27271 </code></pre></div></div> <p>The <em>tests</em> elements now looks like this:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;tests&gt;</span> <span class="nt">&lt;containsISO9660Signature&gt;</span>True<span class="nt">&lt;/containsISO9660Signature&gt;</span> <span class="nt">&lt;containsApplePartitionMap&gt;</span>True<span class="nt">&lt;/containsApplePartitionMap&gt;</span> <span class="nt">&lt;containsAppleHFSHeader&gt;</span>False<span class="nt">&lt;/containsAppleHFSHeader&gt;</span> <span class="nt">&lt;containsAppleMasterDirectoryBlock&gt;</span>False<span class="nt">&lt;/containsAppleMasterDirectoryBlock&gt;</span> <span class="nt">&lt;parsedAppleZeroBlock&gt;</span>True<span class="nt">&lt;/parsedAppleZeroBlock&gt;</span> <span class="nt">&lt;parsedPrimaryVolumeDescriptor&gt;</span>True<span class="nt">&lt;/parsedPrimaryVolumeDescriptor&gt;</span> <span class="nt">&lt;sizeExpected&gt;</span>554061824<span class="nt">&lt;/sizeExpected&gt;</span> <span class="nt">&lt;sizeActual&gt;</span>554373120<span class="nt">&lt;/sizeActual&gt;</span> <span class="nt">&lt;sizeDifference&gt;</span>311296<span class="nt">&lt;/sizeDifference&gt;</span> <span class="nt">&lt;sizeAsExpected&gt;</span>False<span class="nt">&lt;/sizeAsExpected&gt;</span> <span class="nt">&lt;smallerThanExpected&gt;</span>False<span class="nt">&lt;/smallerThanExpected&gt;</span> <span class="nt">&lt;/tests&gt;</span> </code></pre></div></div> <p>The <em>isolyzer</em> output now shows that the size of the ISO image is about 311 KB (152 sectors) larger than the expected value; this is completely fine.</p> <h2 id="access-the-file-system">Access the file system</h2> <p>Now that we’re (reasonably) sure the ISO image is complete, the next step is to access its contents. If the image has a <a href="https://en.wikipedia.org/wiki/Hybrid_disc#Multiple_file_systems">hybrid file system</a>, it may be possible to mount it directly on some platforms. For instance, opening an image that contains an Apple partition with Linux Mint’s Disk Image Mounter will mount the Apple partition (but not the ISO 9660 file system, which may not necessarily point to the same files!). Fortunately, <a href="https://lists.gnu.org/archive/html/libcdio-devel/2010-02/msg00050.html">this message on the <em>libcdio</em> mailing list</a> by Thomas Schmitt (one of the authors of the <a href="https://en.wikipedia.org/wiki/Libburnia"><em>libburnia</em></a> library) explains how to mount these images. The trick here is to insert a block of data at the start of our ISO image that corresponds to the size of the sectors that are missing from the physical disc (i.e. the sectors that are part of the first session). The effect of this is that all sector references will again match the actual sector locations in the image.</p> <p>First we create a file that contains 27271 sectors that are filled with zero-bytes (note that the <em>seek</em> position equals <em>Session Start Sector - 1</em>):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">dd </span><span class="k">if</span><span class="o">=</span>/dev/zero <span class="nv">bs</span><span class="o">=</span>2K <span class="nv">count</span><span class="o">=</span>1 <span class="nv">seek</span><span class="o">=</span>27270 <span class="nv">of</span><span class="o">=</span>leesleeuw_tmp.iso </code></pre></div></div> <p>Next we append our ISO image to this file:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat </span>leesleeuw.iso <span class="o">&gt;&gt;</span>leesleeuw_tmp.iso </code></pre></div></div> <p>Finally we mount the image:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>mount <span class="nt">-t</span> iso9660 <span class="nt">-o</span> loop,sbsector<span class="o">=</span>27270 leesleeuw_tmp.iso /media/johan </code></pre></div></div> <p>We can now navigate the file system with the file manager.</p> <h2 id="create-access-iso">Create access ISO</h2> <p>In order to use the ISO with an emulator or virtual machine, we have to do some additional work. In theory, modifying all sector addresses in the ISO would do the trick, but as far as I’m aware there is no software tool that is capable of this. Instead, we’ll use the mounted file system from the previous step to create a completely new ISO image. We can do this with the following command:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>xorrisofs <span class="nt">-r</span> <span class="nt">-J</span> <span class="nt">-o</span> leesleeuw_new.iso /media/johan </code></pre></div></div> <p>Again, credits go to Thomas Schmitt who suggested this on the <a href="https://lists.gnu.org/archive/html/libcdio-devel/2010-02/msg00053.html"><em>libcdio</em> mailing list</a>.</p> <p>Finally we check the image with <em>isolyzer</em>:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;tests&gt;</span> <span class="nt">&lt;containsISO9660Signature&gt;</span>True<span class="nt">&lt;/containsISO9660Signature&gt;</span> <span class="nt">&lt;containsApplePartitionMap&gt;</span>False<span class="nt">&lt;/containsApplePartitionMap&gt;</span> <span class="nt">&lt;containsAppleHFSHeader&gt;</span>False<span class="nt">&lt;/containsAppleHFSHeader&gt;</span> <span class="nt">&lt;containsAppleMasterDirectoryBlock&gt;</span>False<span class="nt">&lt;/containsAppleMasterDirectoryBlock&gt;</span> <span class="nt">&lt;parsedPrimaryVolumeDescriptor&gt;</span>True<span class="nt">&lt;/parsedPrimaryVolumeDescriptor&gt;</span> <span class="nt">&lt;sizeExpected&gt;</span>508913664<span class="nt">&lt;/sizeExpected&gt;</span> <span class="nt">&lt;sizeActual&gt;</span>508913664<span class="nt">&lt;/sizeActual&gt;</span> <span class="nt">&lt;sizeDifference&gt;</span>0<span class="nt">&lt;/sizeDifference&gt;</span> <span class="nt">&lt;sizeAsExpected&gt;</span>True<span class="nt">&lt;/sizeAsExpected&gt;</span> <span class="nt">&lt;smallerThanExpected&gt;</span>False<span class="nt">&lt;/smallerThanExpected&gt;</span> <span class="nt">&lt;/tests&gt;</span> </code></pre></div></div> <p>I tried this approach for a couple of ISO images with old Windows installables. I mounted the images to a Windows 2000 machine running in VirtualBox, and in all cases I was able to access the images and install the software without problems.</p> <h2 id="recommendations-and-caveats">Recommendations and caveats</h2> <p>Based on some limited tests, the above approach looks useful for providing access to extracted data sessions from CD-Extra discs that would otherwise be inaccessible. There are some caveats though:</p> <p>First, the derived ISO images should <em>only</em> be treated as access images, not as preservation masters! There are several reasons for this. An important one is that if the original image contained an Apple partition, the reformatting procedure will get rid of it! Since Apple partitions may point to different data than the ISO 9660 file system, this may result in loss of data (files).</p> <p>Second, sometimes the data session may reference the audio session. A simple example would be an audio player application that lets a user play audio tracks. This functionality will be lost in a emulated environment. I’m not aware of any solutions for this, especially given the lack of (open) disc image formats that are able to capture all data on a multisession CD. However, it is not impossible that such a format will appear at some point in the future. If that happens, it may be possible to reformat the ripped audio and data tracks into that format, but this would only work if a full record of the sector layout of the physical disc is available.</p> <p>Since we need some of this information already for accessing the ISO images, the most important recommendation that follows from this work is to always keep a record of the sector layout of CD-Extra / Blue Book discs. This can be done by simple running <a href="https://linux.die.net/man/1/cd-info"><em>cd-info</em></a> on each disc, and storing its output as metadata with the ripped/imaged files.</p> <p>Finally, I was surpprised at the complete lack of any software tool that is capable of manipulating sector offsets in an ISO image. Having such a tool would enable us to create readable access ISOs in a more straightforward way (i.e. without first having to add padding bytes to the source source image and thren having to mount the resulting image). It might also be less error-prone. I wonder if there’s any interest from the wider community to invest in the development of such a tool?</p> <h2 id="links">Links</h2> <ul> <li><a href="https://github.com/KBNLresearch/isolyzer">Isolyzer</a></li> <li><a href="https://github.com/KBNLresearch/isolyzer/raw/master/testFiles/multisession.iso">Sample multisession ISO image</a> (6 MB download; start sector of the data session is 21917)</li> </ul> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2017/04/25/imaging-cd-extra-blue-book-discs/">Open Preservation Foundation blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>This is analogous to the <code class="language-plaintext highlighter-rouge">-N</code> option in <em>cdinfo</em>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2017/04/25/imaging-cd-extra-blue-book-discs https://bitsgalore.org/2017/04/25/imaging-cd-extra-blue-book-discs 2017-04-25T00:00:00+02:00 <p>In my <a href="/2017/01/04/breaking-waves-and-some-flacs">previous blog post</a> I addressed the detection of broken audio files in an automated workflow for ripping audio CDs. For (data) CD-ROMs and DVDs that are imaged to an ISO image, a similar problem exists: how can we be reasonably sure that the created image is complete? In this blog post I will discuss some possible ways of doing this using existing tools, along with their limitations. I then introduce <em>Isolyzer</em>, a new tool that might be a useful addition to the existing methods.</p> <!-- more --> <h2 id="checksums">Checksums</h2> <p>A number of techniques exist to verify a newly created ISO image. A seemingly obvious solution would be to do a checksum comparison on both the ISO image and the physical carrier. For instance, the following will work on any Linux system:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>md5sum myimage.iso md5sum /dev/sr0 </code></pre></div></div> <p>The first line computes an MD5 checksum from the ISO image; the second line repeats this for the physical carrier. This method is not completely fail-safe. In some tests I did over a year ago, I ran into a a very strange issue where my attempts to image a CD would sometimes <a href="http://qanda.digipres.org/1076/incomplete-image-after-imaging-rom-prevent-and-detect-this">result in incomplete reads</a>, and, as a result, truncated ISO images. The problem was most likely caused by faulty hardware (the machine on which I ran those tests more or less died shortly afterwards). Most worryingly, the machine would sometimes return incomplete data, both while creating the ISO image as well as during the subsequent checksum calculation on the physical carrier. The result of this was that the computed checksums were identical in both cases, <em>which meant that the image passed the checksum quality check, even though it was incomplete</em>!</p> <h2 id="isovfy">Isovfy</h2> <p>The popular <a href="https://en.wikipedia.org/wiki/Cdrtools"><em>cdrtools</em></a> library includes a tool called <a href="http://linux.die.net/man/8/isoinfo"><em>isovfy</em></a>. Its man page describes it as follows:</p> <blockquote> <p>isovfy is a utility to verify the integrity of an iso9660 image. Most of the tests in isovfy were added after bugs were discovered in early versions of mkisofs. It isn’t all that clear how useful this is anymore, but it doesn’t hurt to have this around.</p> </blockquote> <p>I already commented on this tool in an <a href="/2015/11/13/preserving-optical-media-from-the-command-line/">earlier blog post</a>:</p> <blockquote> <p>The documentation of the tool isn’t very clear about what specific checks it performs. In one of my tests I fed it an ISO image that had its last 50 MB missing (truncated). This did not result in any error or warning message! Most of the reported isovfy errors that I came across in my tests simply reflected the file system on the physical CD not conforming to ISO 9660 (this seems to be pretty common).</p> </blockquote> <p>You can try this yourself by running <em>isovfy</em> on the following two ISO images:</p> <ul> <li><a href="https://github.com/KBNLresearch/verifyISOSize/blob/master/testFiles/minimal.iso?raw=true">This is a small (350 kB) ISO image</a> that is intact</li> <li><a href="https://github.com/KBNLresearch/verifyISOSize/blob/master/testFiles/minimal_trunc.iso?raw=true">Here’s the same image with most of its data truncated</a> (the image is only 48 KB)</li> </ul> <p>I ran both images through <em>isovfy</em> (version 3.02a06); both resulted in the following output:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Root at extent 17, 2048 bytes [0,0] No errors found </code></pre></div></div> <p>This demonstrates that <em>isovfy</em> is not very useful for detecting truncated ISO files.</p> <h2 id="digging-into-the-specs">Digging into the specs</h2> <p>At this point I decided it was time to start digging into some specs. The <a href="http://wiki.osdev.org/ISO_9660">ISO 9660 page on the OSDev Wiki</a> gives a good explanation of the internal organisation of an ISO 9660 image. From this I learnt that the <a href="http://wiki.osdev.org/ISO_9660#The_Primary_Volume_Descriptor">Primary Volume Descriptor</a> (which is a data structure that is present on all ISO images) contains two interesting fields:</p> <ul> <li><em>Volume Space Size</em>, which is the “number of Logical Blocks in which the volume is recorded”;</li> <li><em>Logical Block Size</em>, which is “the size in bytes of a logical block”.</li> </ul> <p>In theory, multiplying both figures should give the expected size of the ISO image, and this would provide a useful way to check if data are missing. To test this, I wrote a Python script that parses an ISO’s Primary Volume Descriptor fields, calculates the expected file size and then compares this against the actual file size. Running the script against some 20 ISO images I had lying around showed that for 7 files the expected size was indeed identical to the actual file size. For most images, the actual size turned out to be marginally larger than expected (typically about 300-600 kB). For 3 images, the actual size was about twice the expected size. Digging deeper, I found out that these were hybrid images that contain an Apple partition on top of the ISO 9660 file system. According to <a href="https://en.wikipedia.org/wiki/Hybrid_disc#Multiple_file_systems">this Wikipedia article</a>, these hybrid discs come in two varieties:</p> <ol> <li>Hybrid discs that contain an Apple Partition Map (located at 512 bytes into the disc/image).</li> <li>Hybrid discs without a Partition Map. These contain a Master Directory Block (located at 1024 bytes into the disc/image).</li> </ol> <p>In my case all of the 3 hybrid images turned out to be of the first category. Using the information <a href="https://en.wikipedia.org/wiki/Apple_Partition_Map#Layout">here</a> and <a href="https://opensource.apple.com/source/IOStorageFamily/IOStorageFamily-116/IOApplePartitionScheme.h">here</a> I was able to add detection of such hybrid images to my code, as well as a simple parser for the ‘zero block’ structure that contains two fields that define the partition’s size: <em>Block Size</em> and <em>Block Count</em>. For my hybrid images, multiplying both figures resulted in a value that was close to (but again marginally smaller than) the actual file size.</p> <p>Finally, I also added detection of the second hybrid disc category (no Partition Map, but Master Directory Block). The Master Directory Block also contains Block Size and Block Count fields that allow one to calculate the size of the file system.</p> <h2 id="isolyzer">Isolyzer</h2> <p>I wrapped up the results of the above analyses into <a href="https://github.com/KBNLresearch/verifyISOSize"><em>Isolyzer</em></a>, which is a dedicated (Python) tool for checking the size of an ISO image. What it does is this:</p> <ol> <li>Locate the image’s Primary Volume Descriptor (PVD).</li> <li>From the PVD, read the Volume Space Size (number of sectors/blocks) and Logical Block Size (number of bytes for each block) fields.</li> <li>Calculate the expected file size as ( Volume Space Size x Logical Block Size ).</li> <li>If the image contains an Apple Partition Map, read the Block Size and Block Count fields from the ‘zero block’</li> <li>Calculate the expected file size as ( Block Size x Block Count )</li> <li>If the image contains an Apple Master Directory Block, read its Block Size and Block Count fields</li> <li>Calculate the expected file size as ( Block Size x Block Count )</li> <li>Calculate the final expected file size as the largest value out of any of the above 3 values</li> <li>Compare this against the actual size of the image files.</li> </ol> <p>In addition to this, Isolyzer also extracts and reports technical metadata from the Primary Volume Descriptor and the Zero Block.</p> <p>Currently the test results are reported in the following format (this may well change in upcoming releases):</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;tests&gt;</span> <span class="nt">&lt;containsISO9660Signature&gt;</span>True<span class="nt">&lt;/containsISO9660Signature&gt;</span> <span class="nt">&lt;containsApplePartitionMap&gt;</span>False<span class="nt">&lt;/containsApplePartitionMap&gt;</span> <span class="nt">&lt;containsAppleHFSHeader&gt;</span>False<span class="nt">&lt;/containsAppleHFSHeader&gt;</span> <span class="nt">&lt;containsAppleMasterDirectoryBlock&gt;</span>False<span class="nt">&lt;/containsAppleMasterDirectoryBlock&gt;</span> <span class="nt">&lt;parsedPrimaryVolumeDescriptor&gt;</span>True<span class="nt">&lt;/parsedPrimaryVolumeDescriptor&gt;</span> <span class="nt">&lt;sizeExpected&gt;</span>358400<span class="nt">&lt;/sizeExpected&gt;</span> <span class="nt">&lt;sizeActual&gt;</span>358400<span class="nt">&lt;/sizeActual&gt;</span> <span class="nt">&lt;sizeDifference&gt;</span>0<span class="nt">&lt;/sizeDifference&gt;</span> <span class="nt">&lt;sizeAsExpected&gt;</span>True<span class="nt">&lt;/sizeAsExpected&gt;</span> <span class="nt">&lt;smallerThanExpected&gt;</span>False<span class="nt">&lt;/smallerThanExpected&gt;</span> <span class="nt">&lt;/tests&gt;</span> </code></pre></div></div> <p>In the above example the <em>sizeExpected</em> field is the size as calculated from the ISO/Apple headers, and <em>sizeActual</em> is the actual size. In this case both are identical. Below some output for a truncated ISO:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;tests&gt;</span> <span class="nt">&lt;containsISO9660Signature&gt;</span>True<span class="nt">&lt;/containsISO9660Signature&gt;</span> <span class="nt">&lt;containsApplePartitionMap&gt;</span>False<span class="nt">&lt;/containsApplePartitionMap&gt;</span> <span class="nt">&lt;containsAppleHFSHeader&gt;</span>False<span class="nt">&lt;/containsAppleHFSHeader&gt;</span> <span class="nt">&lt;containsAppleMasterDirectoryBlock&gt;</span>False<span class="nt">&lt;/containsAppleMasterDirectoryBlock&gt;</span> <span class="nt">&lt;parsedPrimaryVolumeDescriptor&gt;</span>True<span class="nt">&lt;/parsedPrimaryVolumeDescriptor&gt;</span> <span class="nt">&lt;sizeExpected&gt;</span>358400<span class="nt">&lt;/sizeExpected&gt;</span> <span class="nt">&lt;sizeActual&gt;</span>49157<span class="nt">&lt;/sizeActual&gt;</span> <span class="nt">&lt;sizeDifference&gt;</span>-309243<span class="nt">&lt;/sizeDifference&gt;</span> <span class="nt">&lt;sizeAsExpected&gt;</span>False<span class="nt">&lt;/sizeAsExpected&gt;</span> <span class="nt">&lt;smallerThanExpected&gt;</span>True<span class="nt">&lt;/smallerThanExpected&gt;</span> <span class="nt">&lt;/tests&gt;</span> </code></pre></div></div> <p>So, in this case <em>sizeDifference</em> is negative, and flag <em>smallerThanExpected</em> equals ‘True’ (which indicates a damaged image).</p> <h2 id="feedback-wanted">Feedback wanted</h2> <p>At this stage Isolyzer is a bit experimental and pretty rough around the edges, and I wouldn’t recommend it for production use. Nevertheless I’m curious about any feedback on the tool. Do others find this useful? Are things missing (i.e. other hybrid disc types I’m not aware of), or did I get anything completely wrong?</p> <p>One thing that puzzles me a bit is that for the majority of ISO images I’ve come across, the expected size as calculated by Isolyzer is marginally smaller than the actual size. The difference is typically in the order of about 300-600 kB. I’m not quite sure what’s causing this, although <a href="http://twiki.org/cgi-bin/view/Wikilearn/CdromMd5sumsAfterBurning">this article</a> mentions that some CD writing software packages add padding bytes when writing a CD. I wasn’t able to verify if this, although <a href="http://superuser.com/a/220353">this SuperUser answer</a> on validating a burnt DVD suggests it as well. If anyone knows more about this, please let me know!</p> <h2 id="download-links">Download links</h2> <p>Isolyzer can be found <a href="https://github.com/KBNLresearch/verifyISOSize">here on Github</a>. It can be installed using <em>pip</em>; <a href="https://github.com/KBNLresearch/verifyISOSize#installation-with-pip">see the instructions here</a>. For Windows users who cannot/don’t want to install Python I also provided stand-alone Windows binaries, which are available for download <a href="https://github.com/KBNLresearch/verifyISOSize/releases">here</a>.</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2017/01/13/detecting-broken-iso-images-introducing-isolyzer/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2017/01/13/detecting-broken-iso-images-introducing-isolyzer https://bitsgalore.org/2017/01/13/detecting-broken-iso-images-introducing-isolyzer 2017-01-13T00:00:00+01:00 <p>At the KB we have a large collection of offline optical media. Most of these are CD-ROMs, but we also have a sizeable proportion of audio CDs. We’re currently in the process of designing a workflow for stabilising the contents of these materials using disk imaging. For audio CDs this involves ‘ripping’ the tracks to audio files. Since the workflow will be automated to a high degree, basic quality checks on the created audio files are needed. In particular, we want to be sure that the created audio files are complete, as it is possible that some hardware failure during the ripping process could result in truncated or otherwise incomplete files.</p> <p>To get a better idea of what software tool(s) are best suitable for this task, I created a small dataset of audio files which I deliberately damaged. I subsequently ran each of these files through a set of candidate tools, and then looked which tools were able to detect the faulty files. The first half of this blog post focuses on the <a href="http://fileformats.archiveteam.org/wiki/WAVE"><em>WAVE</em></a> format; the second half covers the <a href="http://fileformats.archiveteam.org/wiki/FLAC"><em>FLAC</em></a> format (at the moment we haven’t decided on which format to use yet).</p> <!-- more --> <h2 id="wave-dataset"><em>WAVE</em> dataset</h2> <p>For the <em>WAVE</em> dataset I started out with a <a href="https://github.com/KBNLresearch/detectDamagedAudio/blob/master/data/frogs-01.wav">small, intact <em>WAVE</em> file</a>. Using a Hex editor I then made the following derivatives of this file:</p> <ul> <li><a href="https://github.com/KBNLresearch/detectDamagedAudio/blob/master/data/frogs-01-last-byte-missing.wav">frogs-01-last-byte-missing.wav</a> - one byte is missing at the end of the file</li> <li><a href="https://github.com/KBNLresearch/detectDamagedAudio/blob/master/data/frogs-01-last-2032-bytes-missing.wav">frogs-01-last-2032-bytes-missing.wav</a> - a chunk of 2032 bytes is missing at the end of the file</li> <li><a href="https://github.com/KBNLresearch/detectDamagedAudio/blob/master/data/frogs-01-byte-missing-at-offset-811537.wav">frogs-01-byte-missing-at-offset-811537.wav</a> - one byte is missing at offset 811537</li> </ul> <h2 id="candidate-tools-wave">Candidate tools, <em>WAVE</em></h2> <p>The candidate tools I used to analyse the <em>WAVE</em> files are:</p> <ul> <li><a href="http://jhove.openpreservation.org/"><strong>jhove</strong></a> includes a <a href="http://jhove.openpreservation.org/modules/wave/"><em>WAVE</em> validation module</a>, which makes it an obvious choice. The tested version is 1.14.6, 2016-05-12.</li> <li><a href="http://www.etree.org/shnutils/shntool/"><strong>shntool</strong></a> is a “multi-purpose WAVE data processing and reporting utility”. It was first released in 2000. The tested version is 3.0.7.</li> <li><a href="https://ffmpeg.org/"><strong>ffmpeg</strong></a> is a popular conversion tool for audio and video formats. The tested version is 3.2.2.</li> <li><a href="https://mediaarea.net/en/MediaInfo"><strong>mediainfo</strong></a> is a widely-used feature extraction tool for audiovisual files. The tested version is v0.7.81.</li> </ul> <p>Note that of the above tools, only Jhove and Shntool are designed to detect problems in <em>WAVE</em> files. Both Ffmpeg and Mediainfo were primarily designed for other purposes (format conversion and technical metadata extraction), and they were <em>not</em> designed to detect defective files! I included these tools here mainly because they are widely used, and I was curious whether they would throw up anything interesting in case of defective files<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. I ran the tools with the following command-line arguments (replacing “foo.wav” with the actual file name):</p> <h3 id="jhove">Jhove</h3> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jhove <span class="nt">-m</span> WAVE-hul foo.wav </code></pre></div></div> <h3 id="shntool">Shntool</h3> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>shntool info foo.wav </code></pre></div></div> <h3 id="ffmpeg">Ffmpeg</h3> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ffmpeg <span class="nt">-v</span> error <span class="nt">-i</span> foo.wav <span class="nt">-f</span> null - </code></pre></div></div> <h3 id="mediainfo">Mediainfo</h3> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mediainfo foo.wav </code></pre></div></div> <p>I automated this using a simple <a href="https://github.com/KBNLresearch/detectDamagedAudio/blob/master/runtoolsWAV.sh">shell script</a> that runs each tool on all files, and then writes the output to a set of text files.</p> <h2 id="results-wave">Results, <em>WAVE</em></h2> <p>The full output results of each tool can be found <a href="https://github.com/KBNLresearch/detectDamagedAudio/tree/master/outputWAV">here</a>.</p> <h3 id="jhove-1">Jhove</h3> <p>The ‘Status’ field in Jhove’s output summarises the validation outcome. Here are the results for each file:</p> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">Result</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">frogs-01.wav</td> <td style="text-align: left">Status: Well-Formed and valid</td> </tr> <tr> <td style="text-align: left">frogs-01-last-byte-missing.wav</td> <td style="text-align: left">Status: Well-Formed and valid</td> </tr> <tr> <td style="text-align: left">frogs-01-last-2032-bytes-missing.wav</td> <td style="text-align: left">Status: Well-Formed and valid</td> </tr> <tr> <td style="text-align: left">frogs-01-byte-missing-at-offset-811537.wav</td> <td style="text-align: left">Status: Well-Formed and valid</td> </tr> </tbody> </table> <p>So, Jhove was unable to detect <em>any</em> of the damaged files at all!</p> <h3 id="shntool-1">Shntool</h3> <p>Shntool checks a <em>WAVE</em> on six criteria, which are listed in its output under ‘Possible problems’:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Possible problems: File contains ID3v2 tag: no Data chunk block-aligned: yes Inconsistent header: no File probably truncated: no Junk appended to file: no Odd data size has pad byte: n/a </code></pre></div></div> <p>The thing to watch here is the ‘File probably truncated’ item:</p> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">Result</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">frogs-01.wav</td> <td style="text-align: left">File probably truncated: no</td> </tr> <tr> <td style="text-align: left">frogs-01-last-byte-missing.wav</td> <td style="text-align: left">File probably truncated: yes (missing 1 byte)</td> </tr> <tr> <td style="text-align: left">frogs-01-last-2032-bytes-missing.wav</td> <td style="text-align: left">File probably truncated: yes (missing 2032 bytes</td> </tr> <tr> <td style="text-align: left">frogs-01-byte-missing-at-offset-811537.wav</td> <td style="text-align: left">File probably truncated: yes (missing 1 byte)</td> </tr> </tbody> </table> <p>So, Shntool was able to detect all damaged files.</p> <h3 id="ffmpeg-1">Ffmpeg</h3> <p>For our Ffmpeg call we monitor any errors that are sent to the standard error stream. The results:</p> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">result</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">frogs-01.wav</td> <td style="text-align: left">-</td> </tr> <tr> <td style="text-align: left">frogs-01-last-byte-missing.wav</td> <td style="text-align: left">[pcm_s16le @ 0x3545380] Invalid PCM packet, data has size 3 but at least a size of 4 was expected<br />Error while decoding stream #0:0: Invalid data found when processing input</td> </tr> <tr> <td style="text-align: left">frogs-01-last-2032-bytes-missing.wav</td> <td style="text-align: left">-</td> </tr> <tr> <td style="text-align: left">frogs-01-byte-missing-at-offset-811537.wav</td> <td style="text-align: left">[pcm_s16le @ 0x2768380] Invalid PCM packet, data has size 3 but at least a size of 4 was expected<br />Error while decoding stream #0:0: Invalid data found when processing input</td> </tr> </tbody> </table> <p>Interestingly, Ffmpeg reports an error for both files that have 1 byte missing, but it doesn’t for the file that has 2023 bytes missing. This suggests that Ffmpeg is <em>not</em> suitable for detecting broken <em>WAVE</em> files.</p> <h3 id="mediainfo-1">Mediainfo</h3> <p>Mediainfo didn’t report errors or warnings for any of these files. This is not surprising, but it does confirm that Mediainfo cannot be used for detecting broken <em>WAVE</em> files.</p> <h2 id="flac-dataset"><em>FLAC</em> dataset</h2> <p>Analogous to the <em>WAVE</em> dataset, I started out with a <a href="https://github.com/KBNLresearch/detectDamagedAudio/blob/master/data/frogs-01.flac">small, intact <em>FLAC</em> file</a>, which I then butchered into the following derivative files:</p> <ul> <li><a href="https://github.com/KBNLresearch/detectDamagedAudio/blob/master/data/frogs-01-last-byte-missing.flac">frogs-01-last-byte-missing.flac</a> - one byte is missing at the end of the file</li> <li><a href="https://github.com/KBNLresearch/detectDamagedAudio/blob/master/data/frogs-01-last-1000-bytes-missing.flac">frogs-01-last-1000-bytes-missing.flac</a> - a chunk of 1000 bytes is missing at the end of the file</li> <li><a href="https://github.com/KBNLresearch/detectDamagedAudio/blob/master/data/frogs-01-byte-missing-at-offset-651202.flac">frogs-01-byte-missing-at-offset-651202.flac</a> - one byte is missing at offset 651202</li> </ul> <h2 id="candidate-tools-flac">Candidate tools, <em>FLAC</em></h2> <p>The set of candidate tools is identical to the one used for the <em>WAVE</em> analysis, with two exceptions:</p> <ul> <li><a href="https://xiph.org/flac/"><strong>flac</strong></a> is the reference implementation of the <em>FLAC</em> format. The tested version is 1.3.0.</li> <li>Since Jhove does not include a <em>FLAC</em> module, it was not used.</li> </ul> <h3 id="flac">Flac</h3> <p>The Flac tool is able to encode audio to <em>FLAC</em>, and decode and analyze <em>FLAC</em> files. For this tests I ran it with the * -t* (or <em>–test</em>) option:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>flac <span class="nt">-t</span> foo.flac </code></pre></div></div> <p>This decodes a <em>FLAC</em> without writing the decoded data to a file. Any errors during the decoding process are reported to the standard error stream.</p> <h2 id="results-flac">Results, <em>FLAC</em></h2> <p>The full output results of each tool can be found <a href="https://github.com/KBNLresearch/detectDamagedAudio/tree/master/outputFLAC">here</a>.</p> <h3 id="shntool-2">Shntool</h3> <p>Even though Shntool supports <em>FLAC</em>, it was not able to detect the missing data in any of the files:</p> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">Result</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">frogs-01.flac</td> <td style="text-align: left">File probably truncated: no</td> </tr> <tr> <td style="text-align: left">frogs-01-last-byte-missing.flac</td> <td style="text-align: left">File probably truncated: no</td> </tr> <tr> <td style="text-align: left">frogs-01-last-1000-bytes-missing.flac</td> <td style="text-align: left">File probably truncated: no</td> </tr> <tr> <td style="text-align: left">frogs-01-byte-missing-at-offset-651202.flac</td> <td style="text-align: left">File probably truncated: no</td> </tr> </tbody> </table> <p>So, Shntool does not provide any meaningful information on whether a <em>FLAC</em> is damaged.</p> <h3 id="ffmpeg-2">Ffmpeg</h3> <p>Here are the results for Ffmpeg:</p> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">Result</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">frogs-01.flac</td> <td style="text-align: left">-</td> </tr> <tr> <td style="text-align: left">frogs-01-last-byte-missing.flac</td> <td style="text-align: left">[flac @ 0x294b860] overread: 1<br />Error while decoding stream #0:0: Invalid data found when processing input</td> </tr> <tr> <td style="text-align: left">frogs-01-last-1000-bytes-missing.flac</td> <td style="text-align: left">[flac @ 0x3c5d860] overread: 1<br />Error while decoding stream #0:0: Invalid data found when processing input</td> </tr> <tr> <td style="text-align: left">frogs-01-byte-missing-at-offset-651202.flac</td> <td style="text-align: left">[flac @ 0x279faa0] overread: 1<br />Error while decoding stream #0:0: Invalid data found when processing input</td> </tr> </tbody> </table> <p>So, Ffmpeg was able to identify all damaged <em>FLAC</em>s.</p> <h3 id="mediainfo-2">Mediainfo</h3> <p>Similar to the <em>WAVE</em> results, Mediainfo again didn’t report errors or warnings for any of these files.</p> <h3 id="flac-1">Flac</h3> <p>Finally the results for the Flac tool:</p> <table> <thead> <tr> <th style="text-align: left">File</th> <th style="text-align: left">Result</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">frogs-01.flac</td> <td style="text-align: left">-</td> </tr> <tr> <td style="text-align: left">frogs-01-last-byte-missing.flac</td> <td style="text-align: left">ERROR while decoding data<br />state = FLAC__STREAM_DECODER_END_OF_STREAM</td> </tr> <tr> <td style="text-align: left">frogs-01-last-1000-bytes-missing.flac</td> <td style="text-align: left">ERROR while decoding data<br />state = FLAC__STREAM_DECODER_END_OF_STREAM</td> </tr> <tr> <td style="text-align: left">frogs-01-byte-missing-at-offset-651202.flac</td> <td style="text-align: left">ERROR while decoding data<br />state = FLAC__STREAM_DECODER_READ_FRAME</td> </tr> </tbody> </table> <p>So, the Flac tool was able to identify all defective files<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>.</p> <h2 id="conclusion">Conclusion</h2> <p>Out of the candidate tools considered here, only Shntool was able to identify all damaged <em>WAVE</em> files in this experiment. As a result, this (ancient!) tool still appears to be the best choice for detecting damaged <em>WAVE</em> files. Surpringly, Jhove was unable to detect <em>any</em> of the damaged files at all, and is probably best avoided for this particular purpose. For <em>FLAC</em>, both the Flac tool (<em>FLAC</em> reference implementation) and Ffmpeg were able to detect all damaged files, and both appear to be suitable tools.</p> <h2 id="dataset-and-scripts">Dataset and scripts</h2> <p>All example files, scripts and raw tool output are available here:</p> <p><a href="https://github.com/KBNLresearch/detectDamagedAudio">https://github.com/KBNLresearch/detectDamagedAudio</a></p> <h2 id="post-scriptum-update-on-mediainfo-and-mediaconch">Post scriptum: update on MediaInfo and MediaConch</h2> <p>In response to this post the developers of MediaInfo <a href="https://github.com/MediaArea/MediaInfoLib/pull/352">added support for detecting truncated <em>WAVE</em> files</a>. This should cover all of the damaged <em>WAVE</em> files presented here. Moreover, their Twitter account announced that <a href="https://twitter.com/MediaArea_Net/status/817303297786867712">detection of <em>FLAC</em> flaws</a> is planned for the <a href="https://mediaarea.net/MediaConch/">MediaConch tool</a>, but that they are looking for sponsors for this.</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2017/01/04/breaking-waves-and-some-flacs/">Open Preservation Foundation blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:2"> <p>Also, <a href="http://superuser.com/a/100290/681049">this thread on <em>superuser.com</em></a> recommends Ffmpeg for checking the integrity of video files. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:1"> <p>On a side note, I noticed that the error stream of the Flac tool sometimes contained a sequence of 21 non-printable ‘0x08’ (backspace0 characters. This is probably a bug. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2017/01/04/breaking-waves-and-some-flacs https://bitsgalore.org/2017/01/04/breaking-waves-and-some-flacs 2017-01-04T00:00:00+01:00 <p>Earlier this week the National Archives of the Netherlands (NANeth) published a <a href="http://www.nationaalarchief.nl/sites/default/files/docs/na_rapport_voorkeursformaten-web_0.pdf">report on preferred file formats</a>. It gives an overview of NANeth’s ‘preferred’ and ‘acceptable’ formats for 9 content categories, and also explains the reasoning behind the selected formats. Even though in Dutch language only, the report is well worth a look. However, I found a few of the choices a little surprising, especially the ‘spreadsheet’ category for which it lists the following ‘preferred’ and ‘acceptable’ formats:</p> <table> <thead> <tr> <th style="text-align: left">Preferred</th> <th style="text-align: left">Acceptable</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">ODS, CSV, PDF/A</td> <td style="text-align: left">XLS, XLSX</td> </tr> </tbody> </table> <p>The report gives the following explanation on the ‘preferred’ formats (translated from Dutch):</p> <blockquote> <ul> <li>ODS - ODS is part of the OpenDocument standard (ODF, NEN-ISO/IEC 26300:2007), which is listed as the standard for office documents on the <a href="https://www.forumstandaardisatie.nl/lijst-open-standaarden/in_lijst/verplicht-pas-toe-leg-uit">‘act or explain’ list</a> of ‘Forum Standaardisatie’<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">1</a></sup></li> <li>CSV - for the storage of non-interactive information in cells, a comma-delimited (.csv) text file can be used instead of a spreadsheet</li> <li>PDF/A - PDF/A is a widely used open standard and a NEN/ISO standard (ISO:19005). PDF/A-1 and PDF/A-2 are part of the ‘act or explain’ list of ‘Forum Standaardisatie’. Note: some (interactive) functionality will not be available after conversion to PDF/A. If this functionality is deemed essential, this will be a reason for not choosing PDF/A</li> </ul> </blockquote> <p>In the remainder of this blog post I will pinpoint some problems of the choice of PDF/A and its justification.</p> <!-- more --> <h2 id="demo-spreadsheet">Demo spreadsheet</h2> <p>To illustrate my arguments, I created a <a href="https://github.com/bitsgalore/spreadsheetsPDF/raw/master/demoNumbersCalculations.xlsx">simple demo spreadsheet</a> in xlsx format (created in Microsoft Excel 2010). It contains two columns:</p> <ul> <li>Column A: random number between 0 and 100 (as static values)</li> <li> <p>Column B: formula that takes the value from Column A and adds its square root:</p> <pre><code class="language-data">=A2 + SQRT(A2) </code></pre> </li> </ul> <h2 id="displayed-precision-not-equal-to-stored-precision">Displayed precision not equal to stored precision</h2> <p>Without applying any special formatting, this is what the spreadsheet looks like in MS Excel 2010:</p> <p><img src="/images/2016/12/numbers2.png" alt="" /></p> <p>The first thing of interest here is that the displayed values in the cells are different from those that are actually stored! For example, the value that is shown in cell A2 is:</p> <pre><code class="language-data">52.06077146 </code></pre> <p>Note that 8 decimal places are shown. But by looking at the formula bar you can see a different value:</p> <pre><code class="language-data">52.0607714623856 </code></pre> <p>which contains 13 decimal places. Since Excel internally <a href="https://en.wikipedia.org/wiki/Numeric_precision_in_Microsoft_Excel">stores numbers at a precision of 15 significant figures</a>, only the latter corresponds to the actual (stored) value.</p> <h2 id="loss-of-precision-after-exporting-to-pdfa">Loss of precision after exporting to PDF/A</h2> <p>I exported the spreadsheet to PDF/A-1a using Acrobat PDFMaker. The result can be found <a href="https://github.com/bitsgalore/spreadsheetsPDF/raw/master/demoNumbersCalculations.pdf">here</a>. Below is what the PDF looks like when opened in Adobe Acrobat:</p> <p><img src="/images/2016/12/numbers2_pdfa.png" alt="" /></p> <p>So, the PDF only contains the values at Excel’s displayed precision (in this case typically 9-10 significant figures), and the remaining precision got lost in the conversion.</p> <p>In addition, unlike the source spreadsheet, the PDF <em>only</em> contains static numbers. This means that information about the relation between the values in Columns A and B (i.e. the formula) is completely lost.</p> <h2 id="loss-of-precision-after-exporting-to-csv">Loss of precision after exporting to CSV</h2> <p>Interestingly, exporting to a comma-delimited text file resulted in the same loss of precision! See <a href="https://github.com/bitsgalore/spreadsheetsPDF/blob/master/demoNumbersCalculations.csv">the exported CSV file here</a>. For brevity I won’t go into any further detail on CSV, but it’s important to be aware that this issue exists.</p> <h2 id="effects-of-cell-formatting">Effects of cell formatting</h2> <p>A possible way around the rounding issue would be to use Excel’s <em>Format Cells</em> dialog, which allows one to set a fixed number of decimal places to be used for display:</p> <p><img src="/images/2016/12/formatcells.png" alt="" /></p> <p>This is also less than ideal, if only for the reason that a fixed value will result in the display of non-significant figures. For example, applying a setting of 14 decimal places to the value in cell A1 results in:</p> <pre><code class="language-data">52.06077146238560 </code></pre> <p>which is different from the stored value:</p> <pre><code class="language-data">52.0607714623856 </code></pre> <p>Moreover, this approach gets extremely cumbersome for spreadsheets that contain numbers at different precisions (e.g. it is pretty common to have one column with integer values, and another one with floating-point numbers).</p> <p>In practice, Excel’s number formatting is often used to <em>reduce</em> the number of displayed digits (e.g. to make the columns more visually pleasing, or to avoid messy output when printing). <a href="https://github.com/bitsgalore/spreadsheetsPDF/raw/master/demoDisplay2DigitsOnly.xlsx">Here’s a version of the spreadsheet</a> where I adjusted the formatting to display two decimal places only, and <a href="https://github.com/bitsgalore/spreadsheetsPDF/raw/master/demoDisplay2DigitsOnly.pdf">here is the resulting PDF</a>. It looks like this:</p> <p><img src="/images/2016/12/formatcells_pdfa.png" alt="" /></p> <p>So in this case even more information is lost!</p> <h2 id="interactive-or-dynamic">Interactive or dynamic?</h2> <p>The preferred formats document does acknowledge that PDF/A may not always be suited for spreadsheets, using the following statement (in Dutch):</p> <blockquote> <p>Let wel: bepaalde (interactieve) functionaliteit zal na omzetting naar PDF/A formaat niet meer beschikbaar zijn. Als deze functionaliteit als essentieel wordt beschouwd, is dit een reden om niet voor PDF/A te kiezen</p> </blockquote> <p>Which translates in English as:</p> <blockquote> <p>Note: some (interactive) functionality will not be available after conversion to PDF/A. If this functionality is deemed essential, this will be a reason for not choosing PDF/A</p> </blockquote> <p>This statement is problematic for various reasons. First, whether functionality is deemed ‘essential’ largely depends on the context and intended user base. By stressing the <em>interactive</em> aspect, the authors imply (perhaps unintentionally?) that any spreadsheets that do not take any interaction with a user can be safely converted to PDF/A. But what does ‘interactive’ mean in this context? Taking my earlier <a href="https://github.com/bitsgalore/spreadsheetsPDF/raw/master/demoNumbersCalculations.xlsx">sample spreadsheet</a> as an example: a user may ‘interact’ with that spreadsheet by changing the values in Column A, after which all values in Column B are recalculated. Does that make it interactive? If yes, applying the ‘interactivity’ criterion like this would cover <em>any</em> spreadsheet for which the value in any cell is dependent on one or more values in other cells. This applies to most spreadsheets, apart from those that only contain static data. But in that case a distinction between ‘static’ and ‘dynamic’ spreadsheets might be more useful<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>.</p> <h2 id="reading-pdfa-spreadsheets">Reading PDF/A spreadsheets</h2> <p>Finally, I’m quite puzzled how a PDF/A representation of a spreadsheet is meant to be read. Who are the intended users? What is the target software? What is the context? Sure enough a PDF may be sufficient for on-screen viewing, but what if a (future) user wants to recover the original row and column values? Excel is not capable of this (in fact it cannot even import a PDF at all)? What if someone wants to use the data for some actual calculations? Data extraction from PDF is notoriously difficult (hence the phrase <a href="https://twitter.com/search?q=%22pdf%20is%20where%20data%20goes%20to%20die%22&amp;src=typd">“pdf is where data goes to die”</a>), which is mainly due to the lack of structure of the format<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>.</p> <h2 id="concluding-remarks">Concluding remarks</h2> <p>The above observations only scrape the surface of the perils of using PDF for spreadsheet data. To be clear: there may be situations where <em>PDF/A</em> is a good (and possibly even the best) choice. For example, spreadsheets are often used for printable forms, and having these as a PDF/A representation may be perfectly fine<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>. Nevertheless, NANeth’s recommendations on choosing between their ‘preferred formats’ appear to be suboptimal, because they do not take into account the purpose for which a spreadsheet was created, its content, its intended use and the intended (future) user(s). In particular, using ‘interactivity’ as the main criterion seems somewhat dangerous.</p> <h2 id="data">Data</h2> <p>The example files that are referred to in this blog post are all available here:</p> <p><a href="https://github.com/bitsgalore/spreadsheetsPDF">https://github.com/bitsgalore/spreadsheetsPDF</a></p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2016/12/09/pdfa-as-a-preferred-sustainable-format-for-spreadsheets/">Open Preservation Foundation blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:5"> <p>Forum Standaardisatie is a Dutch government body that promotes the use of open standards in the public sector. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>On a related note, it is well known that (formulae in) spreadsheets often contain errors, and that these can have major implications (there are numerous examples on the <a href="http://www.eusprig.org/horror-stories.htm"><em>Horror Stories</em> section</a> of the <a href="http://www.eusprig.org/">European Spreadsheet Risks Interest Group </a>). Once converted to PDF/A, such errors are impossible to detect. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>Andy Jackson once compared this to <a href="https://twitter.com/anjacks0n/status/471242447813898242">“reconstructing the cow from the burger”</a> <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>Incidentally, spreadsheet forms can be highly interactive (e.g. by letting a user enter data by selecting a value from a drop-down list); this is again an indication that interactivity may not be a good criterion for deciding on PDF/A as a target format <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2016/12/09/pdfa-as-a-preferred-sustainable-format-for-spreadsheets https://bitsgalore.org/2016/12/09/pdfa-as-a-preferred-sustainable-format-for-spreadsheets 2016-12-09T00:00:00+01:00 <p><a href="https://github.com/IDPF/epubcheck"><em>EpubCheck</em></a> is an invaluable tool for assessing the quality of <em>EPUB</em> files. Still, it is possible that <em>EPUB</em>s that are valid according to the format specification (and thus <em>EpubCheck</em>) are nevertheless inaccessible to some users. Some weeks ago a colleague sent me an <em>EPUB</em> 2 file that produced some really strange behaviour across a number of viewer applications. For a start, the text wouldn’t reflow properly after re-sizing the viewer window, and increasing the font size resulted in garbled text. Running the file through <em>EpubCheck</em> did return some validation errors, but none of these were related to the behaviour I was getting. Closer inspection revealed some very peculiar stylesheet and <em>HTML</em> use.</p> <!-- more --> <h2 id="crazy-fixed-layout">Crazy Fixed Layout</h2> <p>As I cannot share the original file for rights reasons, I fired up the <a href="https://sigil-ebook.com/"><em>Sigil</em></a> e-book editor and made a handcrafted <em>EPUB</em> that reproduces its behaviour. You can <a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/build/epub20_crazy_fixed_layout.epub?raw=true">download the file here</a>. If you open it in an e-book viewer, it will probably look perfectly normal at first sight. For example, here’s a screenshot I made using the <a href="https://calibre-ebook.com/"><em>Calibre</em></a> viewer:</p> <p><img src="/images/2016/04/calibre_normal.png" alt="" /></p> <p>Next I reduced the width of the viewer window. One would expect the text to re-flow to the new width. Instead this happened:</p> <p><img src="/images/2016/04/calibre_resized_screen.png" alt="" /></p> <p>After increasing the font size, I ended up with this:</p> <p><img src="/images/2016/04/calibre_largefont.png" alt="" /></p> <p>I got similar results in <a href="https://chrome.google.com/webstore/detail/readium/fepbnnnkkadjhjahcafoaglimekefifl"><em>Chome’s Readium</em> extension</a>. On my e-Ink reader, a Sony PRS-T2, the book rendered as follows:</p> <p><img src="/images/2016/04/sony_fixedlayout.png" alt="" /></p> <p>However, I wasn’t able to change the font size.</p> <h2 id="analysis">Analysis</h2> <p>The file <a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/epubcheckout/4.0.1/epub20_crazy_fixed_layout.xml">passes validation in <em>EpubCheck</em> 4.0.1</a> without errors. However, the output does contain a series of warnings about the use of absolute positions in a stylesheet:</p> <pre><code class="language-data">CSS-017, WARN, [CSS selector specifies absolute position.], OEBPS/Styles/styles.css (6-2) CSS-017, WARN, [CSS selector specifies absolute position.], OEBPS/Styles/styles.css (24-1) CSS-017, WARN, [CSS selector specifies absolute position.], OEBPS/Styles/styles.css (43-1) :: </code></pre> <p>To really understand what causes the problem, we need to look inside the file’s <em>HTML</em> and <em>CSS</em> resources. Here’s some of the <a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/content/epub20_crazy_fixed_layout/OEBPS/Text/Section0001.xhtml"><em>HTML</em> that underlies the text</a>:</p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;p</span> <span class="na">id=</span><span class="s">"p01"</span> <span class="na">class=</span><span class="s">"para"</span><span class="nt">&gt;</span>This is an <span class="nt">&lt;em&gt;</span>EPUB<span class="nt">&lt;/em&gt;</span> 2 file that uses a fixed layout.<span class="nt">&lt;/p&gt;</span> <span class="nt">&lt;p</span> <span class="na">id=</span><span class="s">"p02"</span> <span class="na">class=</span><span class="s">"para"</span><span class="nt">&gt;</span>This is achieved by placing each line inside a<span class="nt">&lt;/p&gt;</span> <span class="nt">&lt;p</span> <span class="na">id=</span><span class="s">"p03"</span> <span class="na">class=</span><span class="s">"para"</span><span class="nt">&gt;&lt;em&gt;</span>paragraph<span class="nt">&lt;/em&gt;</span> element. Each <span class="nt">&lt;em&gt;</span>paragraph<span class="nt">&lt;/em&gt;</span> element<span class="nt">&lt;/p&gt;</span> <span class="nt">&lt;p</span> <span class="na">id=</span><span class="s">"p04"</span> <span class="na">class=</span><span class="s">"para"</span><span class="nt">&gt;</span>is placed at a fixed position on the page. Even<span class="nt">&lt;/p&gt;</span> <span class="nt">&lt;p</span> <span class="na">id=</span><span class="s">"p05"</span> <span class="na">class=</span><span class="s">"para"</span><span class="nt">&gt;</span>though this file is valid <span class="nt">&lt;em&gt;</span>EPUB<span class="nt">&lt;/em&gt;</span>, this is a pretty<span class="nt">&lt;/p&gt;</span> <span class="nt">&lt;p</span> <span class="na">id=</span><span class="s">"p06"</span> <span class="na">class=</span><span class="s">"para"</span><span class="nt">&gt;</span> terrible idea, because in most readers the text<span class="nt">&lt;/p&gt;</span> <span class="nt">&lt;p</span> <span class="na">id=</span><span class="s">"p07"</span> <span class="na">class=</span><span class="s">"para"</span><span class="nt">&gt;</span>will not reflow after resizing the viewer window.<span class="nt">&lt;/p&gt;</span> </code></pre></div></div> <p>So, every line is wrapped inside a <em>paragraph</em> element, each of which has a unique <em>id</em> selector. These refer to style definitions in the <a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/content/epub20_crazy_fixed_layout/OEBPS/Styles/styles.css"><em>EPUB</em>’s stylesheet</a>. Here are the definitions for the first two lines:</p> <div class="language-css highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">#p01</span> <span class="p">{</span> <span class="nl">position</span><span class="p">:</span><span class="nb">absolute</span><span class="p">;</span> <span class="nl">left</span><span class="p">:</span><span class="m">40px</span><span class="p">;</span> <span class="nl">top</span><span class="p">:</span><span class="m">80px</span><span class="p">;</span> <span class="nl">letter-spacing</span><span class="p">:</span><span class="m">0.42px</span><span class="p">;</span> <span class="nl">word-spacing</span><span class="p">:</span><span class="m">0.1em</span><span class="p">;</span> <span class="p">}</span> <span class="nf">#p02</span> <span class="p">{</span> <span class="nl">position</span><span class="p">:</span><span class="nb">absolute</span><span class="p">;</span> <span class="nl">left</span><span class="p">:</span><span class="m">40px</span><span class="p">;</span> <span class="nl">top</span><span class="p">:</span><span class="m">120px</span><span class="p">;</span> <span class="nl">letter-spacing</span><span class="p">:</span><span class="m">0.42px</span><span class="p">;</span> <span class="nl">word-spacing</span><span class="p">:</span><span class="m">0.1em</span><span class="p">;</span> </code></pre></div></div> <p>Each style definition specifies a line’s position on the canvas (<em>left</em>, <em>top</em>); moreover, these co-ordinates are defined as <em>absolute</em> positions. This means that each line is placed at a fixed position, regardless of whether this makes any sense given the actual dimensions of the viewer window (or device), or the user’s preferred font size. It seems that the intention of the producer of the original <em>EPUB</em> (from which I derived my example) was to create some sort of <a href="http://www.idpf.org/epub/fxl/">“fixed layout”</a> document. However, this doesn’t make much sense for books with simple, text-only layouts (as in this case). Worse, depending on the viewing device and the user the file may be effectively inaccessible. For example, someone with a visual impairment may only be able to read an <em>EPUB</em> using very large font sizes, which in this case results in garbled text.</p> <h2 id="crazy-columns">Crazy Columns</h2> <p>Things can even get worse. I once came across an <em>EPUB</em> that used similar tricks to achieve a two-column layout. Again I’m not able to share the original file, so I created <a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/build/epub20_crazy_columns.epub?raw=true">another <em>EPUB</em> that mimicks its behavour</a>. In the <em>Calibre</em> viewer it looks like this:</p> <p><img src="/images/2016/04/calibre_columns.png" alt="" /></p> <p>As with the first example, the text doesn’t reflow after resizing the viewer window, and increasing the font resulted in this:</p> <p><img src="/images/2016/04/calibre_columns_largefont.png" alt="" /></p> <p>This is what I got when I opened the file in my Sony e-Ink reader:</p> <p><img src="/images/2016/04/sony_crazycolumns1.png" alt="" /></p> <p>After I increased the font size this happened:</p> <p><img src="/images/2016/04/sony_crazycolumns2.png" alt="" /></p> <p>Similarly, when I tried to copy the text in the file to the clipboard, and then pasted it in a text editor, I ended up with this:</p> <blockquote> <p>This is an EPUB filepage. Even though thisthat uses a two-columnfile is valid EPUB, there’slayout. For each column,no way to establish theevery line is placed atlogical reading order ofa fixed position on thethe text.</p> </blockquote> <p>Ouch!</p> <h2 id="analysis-1">Analysis</h2> <p>Again, throwing this file at <em>EpubCheck</em> 4 <a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/epubcheckout/4.0.1/epub20_crazy_columns.xml">doesn’t result in any validation errors</a>, although just like the previous file there are some warnings about the use of absolute positions in the stylesheet:</p> <pre><code class="language-data">CSS-017, WARN, [CSS selector specifies absolute position.], OEBPS/Styles/styles.css (13-1) </code></pre> <p>A peek <a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/content/epub20_crazy_columns/OEBPS/Text/Section0001.xhtml">inside the <em>HTML</em></a> reveals the true horrors of this <em>EPUB</em>. This is how the text is encoded:</p> <div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;div</span> <span class="na">class=</span><span class="s">"pos"</span> <span class="na">style=</span><span class="s">"left: 40px; top: 100px;"</span><span class="nt">&gt;</span>This is an <span class="nt">&lt;em&gt;</span>EPUB<span class="nt">&lt;/em&gt;</span> file<span class="nt">&lt;div&gt;</span> <span class="nt">&lt;div</span> <span class="na">class=</span><span class="s">"pos"</span> <span class="na">style=</span><span class="s">"left: 260px; top: 100px;"</span><span class="nt">&gt;</span>page. Even though this<span class="nt">&lt;/div&gt;</span> <span class="nt">&lt;div</span> <span class="na">class=</span><span class="s">"pos"</span> <span class="na">style=</span><span class="s">"left: 40px; top: 140px;"</span><span class="nt">&gt;</span>that uses a two-column<span class="nt">&lt;/div&gt;</span> <span class="nt">&lt;div</span> <span class="na">class=</span><span class="s">"pos"</span> <span class="na">style=</span><span class="s">"left: 260px; top: 140px;"</span><span class="nt">&gt;</span>file is valid <span class="nt">&lt;em&gt;</span>EPUB<span class="nt">&lt;/em&gt;</span>, there's<span class="nt">&lt;/div&gt;</span> </code></pre></div></div> <p>So, every line of each column is wrapped in a division element that has a fixed position. The class <em>pos</em> in the <a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/content/epub20_crazy_columns/OEBPS/Styles/styles.css">stylesheet</a> defines the general layout of each division element. In this case, it specifies that all positions are (again) absolute:</p> <div class="language-css highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">.pos</span> <span class="p">{</span><span class="nl">position</span><span class="p">:</span><span class="nb">absolute</span><span class="p">;</span> <span class="p">}</span> </code></pre></div></div> <p>Technically this is pretty similar to the first example. Note that the above <em>HTML</em> doesn’t contain any semantic information on the fact that there are two separate columns. Worse, the order of the text in the HTML doesn’t even follow the actual reading order! This also explains the results after copying and pasting. <a href="https://en.wikipedia.org/wiki/Screen_reader">Screen reader</a> applications will not be able to handle this either, which makes books like these inaccessible to many visually impaired users. All of this could have been avoided if the book’s producer had followed the <a href="https://www.w3.org/TR/css3-multicol/">W3C multi-column layout specification</a>.</p> <h2 id="conclusion">Conclusion</h2> <p>I don’t know how common (or rare) <em>EPUB</em>s like the above are. They may just be weird edge cases. Nevertheless, their existence indicates that checking for validity alone may not be sufficient to ensure accessibility for all users (in particular those with a visual impairment). In any case, files like these can be identified relatively easily by checking <em>EpubCheck</em>’s output for the presence of a <em>CSS-017</em> warning (“CSS selector specifies absolute position”)<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. These examples also underline the importance of guidelines and best practices. Several good resources for making accessible <em>EPUB</em> are available from the <a href="http://www.idpf.org/accessibility/guidelines/"><em>EPUB</em> 3 Accessibility Guidelines</a>, including a useful <a href="http://www.idpf.org/accessibility/guidelines/content/qa/qa-checklist.php">Accessibility QA Checklist</a>. I would also be interested in hearing other people’s experiences with “weird” <em>EPUB</em>s like these.</p> <h2 id="postscript">Postscript</h2> <p>Alberto Pettarin pointed me to his blog post <a href="http://www.albertopettarin.it/blog/2015/02/21/current-fixed-layout-ebooks-considered-harmful.html"><em>(Current) Fixed Layout eBooks Considered Harmful</em></a>. Written in 2015, it addresses the problems with current implementations of fixed layouts in <em>EPUB</em>, and if you found this blog post interesting, I would suggest to check out Alberto’s blog as well.</p> <p>Alberto’s <a href="https://twitter.com/acutebit/status/718031931221360640">Twitter feed</a> also drew my attention to an interesting <em>EPUB</em> with the program of the recent <em>EPUB</em> Summit in Bordeaux. You can <a href="http://edrlab.org/edrlab/wp-content/uploads/2016/04/EDRLabprogram_EN_HD_final.epub_.zip">download it here</a> (you need to unzip it first!). The file is interesting because:</p> <ol> <li>It does not pass validation by <em>EpubCheck</em> (the mimetype file entry is not the first file resource in the archive)</li> <li>It uses a fixed, multi-column layout that doesn’t scale in either <em>Readium</em> or <em>Calibre</em>’s viewer (changing the font size has no effect), and I’m wondering if it is usable at all on any handheld devices!</li> </ol> <p>There’s some irony in that this file was published by <a href="http://edrlab.org/edrlab/"><em>EDRLab</em></a>, an organisation that describes itself as “the European headquarter for IDPF and Readium Foundation”, and which mentions “support for people who have print disabilities” as a “key part”of its mission. Oh well …</p> <h2 id="link-to-dataset">Link to dataset</h2> <p>The <em>EPUB</em>s used for this blog post are part of the <a href="https://github.com/KBNLresearch/epubPolicyTests">EPUB KB policy testing repository</a>. This is an annotated set of openly licensed <em>EPUB</em> files that were specifically created for testing purposes.</p> <hr /> <p>Originally published at the <a href="http://blog.kbresearch.nl/2016/04/04/valid-but-not-accessible-epub-crazy-fixed-layouts/">KB Research blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>Note that <em>EpubCheck</em> 3 (now outdated) <a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/epubcheckout/3.0.1/epub20_crazy_columns.xml">does not report this warning</a>, so always use <em>EpubCheck</em> 4. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2016/04/04/valid-but-not-accessible-epub-crazy-fixed-layouts https://bitsgalore.org/2016/04/04/valid-but-not-accessible-epub-crazy-fixed-layouts 2016-04-04T00:00:00+02:00 <p>About a month ago the <a href="http://idpf.org/">International Digital Publishing Forum</a>, the standards body behind the <em>EPUB</em> format, published an <a href="http://www.idpf.org/epub/31/spec/epub-spec.html">Editor’s Draft of <em>EPUB</em> 3.1</a>. This is meant to be the successor of the <a href="http://idpf.org/epub/301">current 3.0.1 version</a>. IDPC has set up a <a href="http://idpf.org/news/first-editors-draft-of-epub-31-available-for-review">community review</a>, which allows interested parties to comment on the draft. The proposed changes relative to <em>EPUB</em> 3.0.1 are summarised <a href="http://www.idpf.org/epub/31/spec/epub-changes.html">in this document</a>. A note at the top states (emphasis added by me):</p> <blockquote> <p>The EPUB working group has opted for a <em>radical change approach</em> to the addition and deletion of features in the 3.1 revision to <em>move the standard aggressively forward</em> with the overarching goals of alignment with the Open Web Platform and simplification of the core specifications.</p> </blockquote> <p>As Gary McGath <a href="https://fileformats.wordpress.com/2016/02/05/epub-3-1/">pointed out earlier</a>, this is a pretty bold statement for what is essentially a minor version. The authors of the draft also mention that they expect it “will provoke strong reactions both for and against”, and that changes that raise “strong negative reactions” from the community “will be reviewed for future drafts”.</p> <p>This blog post is an attempt to identify the main implications of the current draft for libraries and archives: to what degree would the proposed changes affect (long-term) accessibility? Since the current draft is particularly notable for its aggressive <em>removal</em> of various existing <em>EPUB</em> features, I will focus on these. These observations are all based on the 30 January 2016 draft of the changes document.</p> <!-- more --> <h2 id="removed-support-for-epubcfi-for-linking"><a href="http://www.idpf.org/epub/31/spec/epub-changes.html#sec-epub31-cfi">Removed support for EPUBCFI for linking</a></h2> <p>The <em>EPUB</em> Canonical Fragment Identifier (<a href="http://www.idpf.org/epub/linking/cfi/epub-cfi.html">EPUBCFI</a>) “defines a standardized method for referencing arbitrary content within an EPUB Publication”. Until <em>EPUB</em> 3.0.1, Reading Systems were required to support EPUBCFI for hyperlinking within and between documents. This requirement is dropped in <em>EPUB</em> 3.1 (although it would still be possible to use EPUBCFI for annotations and bookmarks).</p> <p>In principle this change could result in problems if an <em>EPUB</em> that uses CFI for hyperlinks is opened in a 3.1 reading system: in that case the hyperlinks would not work. However, according to <em>EPUB</em> editor Matt Garrish, <a href="https://github.com/IDPF/epub-revision/issues/662#issuecomment-193793093">authors simply do not use CFI for hyperlinking</a>. He also mentions a check by Google on their corpus of millions of books, which only turned up a few instances of CFI use. One of these was a link in an <em>EPUB</em> best practices book, while the remaining ones were all part of the <em>EPUB</em> test suite documents. If these results are representative of all <em>EPUB</em>s “in the wild”, the implications of the change would be negligible.</p> <h2 id="reduced-set-of-metadata-elements-in-package-document"><a href="http://www.idpf.org/epub/31/spec/epub-changes.html#sec-pkg-metadata">Reduced set of metadata elements in Package Document</a></h2> <p><em>EPUB</em> 3.1 imposes restrictions on the metadata elements that can be embedded in the Package Document. Up to version 3.0.1, the full <a href="http://dublincore.org/documents/dces/">Dublin Core Metadata Element Set</a> was supported, whereas in 3.1 only the <code class="language-plaintext highlighter-rouge">dc:identifier</code>, <code class="language-plaintext highlighter-rouge">dc:title</code>, <code class="language-plaintext highlighter-rouge">dc:language</code>, <code class="language-plaintext highlighter-rouge">dc:creator</code>, <code class="language-plaintext highlighter-rouge">dc:publisher</code> and <code class="language-plaintext highlighter-rouge">dc:type</code> elements are allowed. Additional metadata <em>can</em> be included, but they need to be defined in a separate resource (file), which is referenced from the <code class="language-plaintext highlighter-rouge">metadata</code> element using the <a href="http://www.idpf.org/epub/31/spec/epub-packages.html#sec-link-elem"><code class="language-plaintext highlighter-rouge">link</code></a> element. Below is an example that uses a <a href="https://en.wikipedia.org/wiki/MARC_standards">MARC</a> file:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;link</span> <span class="na">rel=</span><span class="s">"record"</span> <span class="na">href=</span><span class="s">"meta/9780000000001.xml"</span> <span class="na">media-type=</span><span class="s">"application/marc"</span><span class="nt">/&gt;</span> </code></pre></div></div> <p>Complicating things further, the <a href="http://www.idpf.org/epub/31/spec/epub-packages.html#sec-link-elem">EPUB 3.1 Packages draft</a> says:</p> <blockquote> <p>Linked resources that are not Publication Resources are not subject to Core Media Type requirements [EPUB31] and may be located inside or outside [EPUB31] the EPUB Container. Retrieval of Remote Resources is optional.</p> </blockquote> <p>So, linked metadata resources can have <em>any</em> possible format, and they may not even be included in the <em>EPUB</em> container. Even though these changes would have no direct consequences for long-term accessibility, they would seriously complicate document processing (e.g. ingest) workflows that rely on the metadata in the Package Document. It would also affect end users who rely on these metadata fields to <a href="https://github.com/IDPF/epub-revision/issues/642#issuecomment-181450515">sort and find their ebooks</a><sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>.</p> <h2 id="removal-of-the-ncx"><a href="http://www.idpf.org/epub/31/spec/epub-changes.html#sec-pkg-ncx">Removal of the NCX</a></h2> <p><em>EPUB</em> 2 documents contain the <a href="http://www.idpf.org/epub/20/spec/OPF_2.0.1_draft.htm#Section2.4.1"><em>NCX</em></a> file (“Navigation Control file for XML”), which provides a mechanism to navigate a publication. It is essentially a hierarchical table of contents. The <em>NCX</em> was <a href="http://www.idpf.org/epub/301/spec/epub-publications.html#ncx-superseded">superseded</a> by the <a href="http://www.idpf.org/epub/301/spec/epub-contentdocs.html#sec-xhtml-nav"><em>Navigation Document</em></a> in <em>EPUB</em> 3.0.1. However, the <em>NCX</em> was allowed in <em>EPUB</em> 3.01 publications, which was useful for keeping <em>EPUB</em> 3 publications compatible with older (<em>EPUB</em> 2-based) reading systems<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>. The 3.1 draft forbids the <em>NCX</em> altogether, which means that such “hybrid” <em>EPUB</em>s are not possible without breaking the specification.</p> <p>The main consequence of this is that it would make <em>EPUB</em> 3.1 files incompatible with older reading systems. More specifically, basic navigation functionality such as direct access to a chapter from the table of contents would not work.</p> <p>To get an approximate idea of the impact of this, I had a look at the <a href="http://epubtest.org/"><em>EPUB</em> 3 support grid</a>, which gives detailed information about the support of specific <em>EPUB</em> 3 features for commonly used devices, apps, and reading systems. <a href="http://epubtest.org/testsuite/epub3/feature/toc-nav/">This link</a> shows support of the <a href="http://www.idpf.org/epub/301/spec/epub-contentdocs.html#sec-xhtml-nav-def-types-toc"><code class="language-plaintext highlighter-rouge">toc nav</code></a> element, which defines the primary navigational hierarchy in the Navigation Document. Only 55% (34 out of 62) of all tested reading systems fully support the <code class="language-plaintext highlighter-rouge">toc nav</code> element, with 37% (23 out of 62) not supporting it at all<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>. This may not be a big deal for users of software-based reading systems (which make up the majority of the support grid), but users of (older) E-ink readers often don’t have the option to upgrade their devices. A good example is this (now discontinued) <a href="http://epubtest.org/evaluation/52/">Sony e-Ink hardware reader</a>. Unfortunately, E-ink devices appear to be underrepresented in the support grid. For example, it contains no information whatsoever on any of the popular Kobo readers.</p> <p>The proposal to remove the <em>NCX</em> provoked strong reactions in the <a href="https://github.com/IDPF/epub-revision/issues/633">community review</a>, with one respondent stating it would lead to <a href="https://github.com/IDPF/epub-revision/issues/633#issuecomment-170398642">“dropping support for millions of eInk reading systems”</a>. It would also contradict <a href="http://www.idpf.org/epub/301/spec/epub-publications.html#ncx-superseded">this statement from the <em>EPUB</em> 3.0.1 specification</a> (emphasis added by me):</p> <blockquote> <p>The NCX feature defined in [OPF2] is superseded by the EPUB Navigation Document [ContentDocs301]. <em>EPUB 3 Publications</em> may include an NCX (as defined in OPF 2.0.1) for EPUB 2 Reading System forwards compatibility purposes, but EPUB 3 Reading Systems must ignore the NCX.</p> </blockquote> <p>The explicit reference to <em>EPUB 3 Publications</em> (<strong>not</strong> <em>EPUB 3.0.1 Publications</em>!!) implies that the statement applies to <em>EPUB</em> 3 in general. Removing the <em>NCX</em> in another <em>EPUB</em> 3 release would be at odds with this.</p> <h2 id="removal-of-the-guide-element"><a href="http://www.idpf.org/epub/31/spec/epub-changes.html#sec-pkg-guide">Removal of the guide Element</a></h2> <p>The <a href="http://www.idpf.org/epub/20/spec/OPF_2.0.1_draft.htm#Section2.6"><em>guide</em> element</a> was an optional data structure in <em>EPUB</em> 2 that provided “convenient access” to structural components of a publication. It was <a href="http://www.idpf.org/epub/301/spec/epub-publications.html#sec-guide-elem">deprecated</a> in <em>EPUB</em> 3.0.1. Without any data on the actual usage of this feature, it is difficult to say much about the impact of its complete removal (this was also <a href="https://github.com/IDPF/epub-revision/issues/644#issuecomment-191706305">pointed out by one respondent to the community review</a>).</p> <h2 id="removal-of-the-bindings-element"><a href="http://www.idpf.org/epub/31/spec/epub-changes.html#sec-pkg-bindings">Removal of the bindings Element</a></h2> <p>In <em>EPUB</em> 3.0.1 the <a href="http://www.idpf.org/epub/301/spec/epub-publications.html#sec-bindings-elem"><em>bindings</em></a> element could be used to define fallbacks for <a href="http://www.idpf.org/epub/301/spec/epub-publications.html#gloss-publication-resource-foreign">foreign resources</a>. <a href="https://github.com/IDPF/epub-revision/issues/639">According to <em>EPUB</em> editor Matt Garrish</a> “this feature is not widely used or supported”, and the impact on accessibility appears to be negligible.</p> <h2 id="removal-of-the-switch-element"><a href="http://www.idpf.org/epub/31/spec/epub-changes.html#sec-cdoc-switch">Removal of the switch Element</a></h2> <p>The <a href="http://www.idpf.org/epub/301/spec/epub-contentdocs.html#sec-xhtml-content-switch"><em>switch</em></a> element in <em>EPUB</em> 3.0.1 allows one to define alternative representations of XML fragments. Here’s an example:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;epub:switch</span> <span class="na">id=</span><span class="s">"cmlSwitch"</span><span class="nt">&gt;</span> <span class="nt">&lt;epub:case</span> <span class="na">required-namespace=</span><span class="s">"http://www.xml-cml.org/schema"</span><span class="nt">&gt;</span> <span class="nt">&lt;cml</span> <span class="na">xmlns=</span><span class="s">"http://www.xml-cml.org/schema"</span><span class="nt">&gt;</span> <span class="nt">&lt;molecule</span> <span class="na">id=</span><span class="s">"sulfuric-acid"</span><span class="nt">&gt;</span> <span class="nt">&lt;formula</span> <span class="na">id=</span><span class="s">"f1"</span> <span class="na">concise=</span><span class="s">"H 2 S 1 O 4"</span><span class="nt">/&gt;</span> <span class="nt">&lt;/molecule&gt;</span> <span class="nt">&lt;/cml&gt;</span> <span class="nt">&lt;/epub:case&gt;</span> <span class="nt">&lt;epub:default&gt;</span> <span class="nt">&lt;p&gt;</span>H<span class="nt">&lt;sub&gt;</span>2<span class="nt">&lt;/sub&gt;</span>SO<span class="nt">&lt;sub&gt;</span>4<span class="nt">&lt;/sub&gt;&lt;/p&gt;</span> <span class="nt">&lt;/epub:default&gt;</span> <span class="nt">&lt;/epub:switch&gt;</span> </code></pre></div></div> <p>Here, we have a chemical formula in <a href="https://en.wikipedia.org/wiki/Chemical_Markup_Language"><em>ChemML</em></a> format and in standard <em>HTML</em>. <em>ChemML</em> is not natively supported in <em>EPUB</em>, so by default a reader will display the <em>HTML</em> version. However, wrapping both in a <em>switch</em> element would allow a <em>ChemML</em>-capable reader to render that representation instead.</p> <p>I asked <em>EPUB</em> editor Matt Garrish how an <em>EPUB</em> 3.1-compliant reader would render content that is wrapped in a <em>switch</em> element. He <a href="https://github.com/IDPF/epub-revision/issues/637#issuecomment-193894732">replied</a> that by default <em>all</em> of the switch content would be rendered. So for the example above, a reader would try to render both the <em>HTML</em> and the <em>ChemML</em> versions (with the latter failing on most reading systems). Matt stressed the significance of the <em>switch</em> element, adding that people have been using it, “if not extensively”.</p> <h2 id="removal-of-the-trigger-element"><a href="http://www.idpf.org/epub/31/spec/epub-changes.html#sec-cdoc-trigger">Removal of the trigger Element</a></h2> <p>The <a href="http://www.idpf.org/epub/301/spec/epub-contentdocs.html#sec-xhtml-epub-trigger"><em>trigger</em></a> element in <em>EPUB</em> 3.0.1 is used to define simple user interfaces for multimedia content. Since this can be done natively in <em>HTML</em> 5, it is dropped from <em>EPUB</em> 3.1. <a href="https://github.com/IDPF/epub-revision/issues/638#issue-125825057">Here</a> editor Matt Garrish explains that the feature is both “sparsely used” (referring to a survey of publishers) and “poorly supported”.</p> <h2 id="miscellaneous-changes">Miscellaneous changes</h2> <p>Apart from the changes above (which all <em>remove</em> features from the existing specification), the <em>EPUB</em> 3.1 draft also adds a number of new features, and clarifies some existing ones. I won’t go over them in detail, but here’s a brief overview:</p> <ul> <li> <p><a href="http://www.idpf.org/epub/31/spec/epub-changes.html#sec-epub31-cmt">New Core Media Types</a> - adds support for <a href="https://www.w3.org/TR/WOFF2/"><em>WOFF</em> 2.0</a> and <a href="https://en.wikipedia.org/wiki/SFNT"><em>SFNT</em></a> fonts.</p> </li> <li> <p><a href="http://www.idpf.org/epub/31/spec/epub-changes.html#sec-cdoc-html">Addition of Support for HTML Syntax of HTML5</a> - adds support for the <em>HTML</em> Syntax of <em>HTML5</em> (currently only the <em>XHTML</em> syntax is supported<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>).</p> </li> <li> <p><a href="http://www.idpf.org/epub/31/spec/epub-changes.html#sec-cdoc-css">Replacement of EPUB Style Sheets with CSS References</a> - this replaces the current “<em>EPUB</em> Style Sheets profile” by the “official definition” of <em>CSS</em>.</p> </li> <li> <p><a href="http://www.idpf.org/epub/31/spec/epub-changes.html#sec-cdoc-wcag">Addition of WCAG Support</a> - adds the recommendation that all <em>HTML</em> Content Documents conform to the WCAG Guidelines to ensure they are accessible for people with disabilities.</p> </li> </ul> <p>Finally, the draft contains clarifications on <a href="http://www.idpf.org/epub/31/spec/epub-changes.html#sec-epub31-fallbacks">Foreign Resource Fallbacks</a> and <a href="http://www.idpf.org/epub/31/spec/epub-changes.html#sec-cdoc-scripting">Scripting Support</a>.</p> <h2 id="epub-31-or-epub-40"><em>EPUB</em> 3.1 or <em>EPUB</em> 4.0?</h2> <p>By now it should be clear that the aggressive removal of features in <em>EPUB</em> 3.1 would have some far-reaching consequences. This is particularly true for the removal of the <em>NCX</em>, which would make <em>EPUB</em> 3.1 files incompatible with many existing E-ink readers. It would do this by ruling out the option to make backward-compatible “hybrid” files. As Gary McGath <a href="https://fileformats.wordpress.com/2016/02/05/epub-3-1/">pointed out earlier</a>, introducing “radical changes” in what is essentially a minor version is pretty unusual practice for any standard. Nowadays, most software and file formats use some variation of <a href="http://semver.org/">semantic versioning</a>, with version numbers that follow the general form <em>MAJOR.MINOR.PATCH</em>. Here, each component of the version number has a well-defined meaning:</p> <ol> <li><em>MAJOR</em> version is increased in case of incompatible API changes,</li> <li><em>MINOR</em> version is increased when functionality is added in a backwards-compatible manner, and</li> <li><em>PATCH</em> version is increased in case of backwards-compatible bug fixes.</li> </ol> <p>Since the current draft includes multiple backward-incompatible changes, this makes me wonder why the editors didn’t name it <em>EPUB</em> 4.0 instead! Kovid Goyal, lead developer of the popular <a href="https://calibre-ebook.com/">Calibre</a> software, made <a href="https://github.com/IDPF/epub-revision/issues/642#issuecomment-182916894">the following comment on this</a>:</p> <blockquote> <p>[I]f you want to make backwards incompatible changes, please, dont do it in a point release. From glancing over your changes document, it seems to me that you want to make several breaking changes. That’s great, EPUB 3 could do with some serious breaking. But name it EPUB 4. I really dont want to have tell my users that calibre supports EPUB 3.1 but not EPUB 3.</p> </blockquote> <p>I agree with Kovid here. Having multiple sub-versions of <em>EPUB</em> 3, with <em>some</em> of them being backward-compatible with <em>EPUB</em> 2, while this backward compatibility is explicitly ruled out in another sub-version, is bound to create a situation that will be incomprehensible for most e-book buyers. Worse, it could even undermine overall confidence in the format. For memory institutions it would also make the management of <em>EPUB</em> 3 publications unnecessarily complicated. Not only would some <em>EPUB</em> 3.1 files not render correctly in an <em>EPUB</em> 3.0.1 reader, the opposite would be true as well.</p> <h2 id="flashback">Flashback</h2> <p>In my 2012 <a href="https://zenodo.org/record/839711">report on <em>EPUB</em> for archival preservation</a> I already mentioned the stability of the <em>EPUB</em> format as a concern:</p> <blockquote> <p>EPUB 3 shows quite major changes relative to version 2, which raises concerns about the format’s stability over time. These concerns are reinforced by the fact that EPUB 3 is heavily dependent on (X)HTML5 and CSS3, both of which are unfinished “works in progress”, which may undergo various changes before being finalised.</p> </blockquote> <p>These concerns are once more confirmed by the current <em>EPUB</em> 3.1 draft. However, it remains to be seen how many of these changes will make it to the final version. The community review process is ongoing at this moment, so if you’re getting a little uneasy after reading this blog post, there’s still time to get involved and make your voice heard!</p> <h2 id="acknowledgement">Acknowledgement</h2> <p>Thanks to Matt Garrish for his prompt replies to my questions on Github.</p> <hr /> <p>Originally published at the <a href="http://blog.kbresearch.nl/2016/03/10/the-future-of-epub-a-first-look-at-the-epub-3-1-editors-draft/">KB Research blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:5"> <p>The <a href="https://github.com/IDPF/epub-revision/issues/642">discussion thread on this topic in the issue tracker</a> is worth checking out, as it contains some excellent additional observations. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>See here how <a href="http://toc.oreilly.com/2013/02/oreillys-journey-to-epub-3.html">O’Reilly’s keeps their <em>EPUB</em> 3 books compatible with <em>EPUB</em> 2 readers</a> <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>This figure includes reading systems for which support is unknown <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>See the <a href="https://dev.w3.org/html5/html-author/#the-html-and-xhtml-syntax"><em>HTML5</em> Reference</a> for a discussion of the differences between both syntaxes <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2016/03/10/the-future-of-epub-a-first-look-at-the-epub-3-1-editors-draft https://bitsgalore.org/2016/03/10/the-future-of-epub-a-first-look-at-the-epub-3-1-editors-draft 2016-03-10T00:00:00+01:00 <p>Yesterday (7 December) we released <a href="http://jpylyzer.openpreservation.org//2015/12/07/Release-of-jpylyzer-1-16-0">version 1.16.0</a> of the <a href="http://jpylyzer.openpreservation.org/"><em>jpylyzer</em></a> tool, which is this year’s third release of the software (excluding bugfix releases). This blog post gives a brief overview of the main <em>jpylyzer</em> improvements that have been implemented over this year.</p> <!-- more --> <h2 id="changes-in-xml-output">Changes in XML output</h2> <p>The 1.14 release introduced two output improvements. Most importantly, an <a href="https://en.wikipedia.org/wiki/XML_Schema_%28W3C%29">XML Schema Definition</a> (XSD) was created. The schema formally defines the output format, and it also makes it possible to validate output files. In addition, a namespace declaration was added. These changes make the post-processing of <em>jpylyzer</em>’s output more straightforward.</p> <p>The 1.16 release added the <em>statusInfo</em> element, which tells you whether the validation completed without any internal errors. It contains the following sub-elements:</p> <ul> <li> <p><em>success</em>: a Boolean flag that indicates whether the validation attempt completed normally (“True”) or not (“False”). A value of “False” indicates an internal error that prevented <em>jpylyzer</em> from validating the file.</p> </li> <li> <p><em>failureMessage</em>: if the validation attempt failed (value of <em>success</em> equals “False”), this field gives further details about the reason of the failure.</p> </li> </ul> <p>This means that the general structure of the output now looks like this:</p> <p><img src="/images/2015/12/outputStructure.png" alt="Jplylyzer output structure" /></p> <h2 id="recursive-traversal-of-directory-trees">Recursive traversal of directory trees</h2> <p>Another feature that was introduced with the 1.14 release is the <code class="language-plaintext highlighter-rouge">--recurse</code> option. This allows one to recursively traverse a directory tree. The code for this feature was created by Adam Retter, Jaishree Davey and Laura Damian of The National Archives (UK).</p> <h2 id="memory-mapping">Memory mapping</h2> <p>The 1.15 release introduced the use of <a href="https://en.wikipedia.org/wiki/Memory-mapped_file">memory mapping</a> for reading input images. This results in better performance when processing (very) large files. Images that would cause a memory error in previous versions are now handled without any problem. Also, the processing of very large files can be significantly faster than in earlier releases, and is less prone to freezing other processes that are simultaneously running on the machine. This improvement was suggested by Stefan Weil of Mannheim University Library, and the changes are based on a patch he submitted.</p> <p>Two examples illustrate the benefits of this change:</p> <ul> <li> <p><a href="http://hirise-pds.lpl.arizona.edu/download/PDS/RDR/ESP/ORB_011200_011299/ESP_011265_1560/ESP_011265_1560_RED.JP2">This 2 GB image</a> resulted in a memory error with <em>jpylyzer</em> 1.14.2 on a Windows machine with 4 GB RAM. The latest versions process the file without problems.</p> </li> <li> <p>On a Linux Mint machine with 8 GB RAM, <a href="http://apollo.sese.asu.edu/data/pancam/AS16/jp2/AS16-P-4102.jp2">this 6.7 GB image</a> also resulted in a memory error. Again, the current version handles the file without any problem.</p> </li> </ul> <p>This doesn’t mean that memory errors are now a thing of the past entirely; they may still occur under some circumstances. For instance, a test with the 6.7 GB image failed on a Linux Mint machine with 4 GB RAM. So it seems prudent to make sure that the amount of available RAM always exceeds the maximum image size by a fairly wide safety margin. Also, chip architecture and operating system may put further constraints on the amount of memory than can be mapped at a time.</p> <h2 id="improved-exception-handling">Improved exception handling</h2> <p>Prior to release 1.16.0, an exception during the processing of an image could cause <em>jpylyzer</em> to crash. For example, an extremely large image can result in an internal memory error, and this would grind <em>jpylyzer</em> to a halt. This is particularly problematic when using the new <code class="language-plaintext highlighter-rouge">--recurse</code> option: in this case a single <em>jpylyzer</em> invocation may involve the processing of thousands of images at a time. One single (e.g. extremely large) image could then result in unusable output; moreover, it would be difficult to identify <em>which</em> image caused the crash in the first place! Release 1.16.0 introduces improved exception handling that allows <em>jpylyzer</em> to handle such situations more gracefully.</p> <h2 id="robustness">Robustness</h2> <p>The combined effect of the exception handling, memory mapping and status output should make <em>jpylyzer</em> releases from 1.16.0 onwards significantly more robust than previous versions. As an example, here’s some (simplified) output for a 6.5 GB JP2 that caused a memory error:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">&lt;?xml version='1.0' encoding='UTF-8'?&gt;</span> <span class="nt">&lt;jpylyzer&gt;</span> <span class="nt">&lt;toolInfo&gt;</span> <span class="nt">&lt;toolName&gt;</span>jpylyzer.py<span class="nt">&lt;/toolName&gt;</span> <span class="nt">&lt;toolVersion&gt;</span>1.16.0<span class="nt">&lt;/toolVersion&gt;</span> <span class="nt">&lt;/toolInfo&gt;</span> <span class="nt">&lt;fileInfo&gt;</span> <span class="nt">&lt;fileName&gt;</span>AS16-P-4102.jp2<span class="nt">&lt;/fileName&gt;</span> <span class="nt">&lt;filePath&gt;</span>/home/johan/testJpylyzer/AS16-P-4102.jp2<span class="nt">&lt;/filePath&gt;</span> <span class="nt">&lt;fileSizeInBytes&gt;</span>6745365021<span class="nt">&lt;/fileSizeInBytes&gt;</span> <span class="nt">&lt;fileLastModified&gt;</span>Wed Dec 2 20:05:29 2015<span class="nt">&lt;/fileLastModified&gt;</span> <span class="nt">&lt;/fileInfo&gt;</span> <span class="nt">&lt;statusInfo&gt;</span> <span class="nt">&lt;success&gt;</span>False<span class="nt">&lt;/success&gt;</span> <span class="nt">&lt;failureMessage&gt;</span>memory error (file size too large)<span class="nt">&lt;/failureMessage&gt;</span> <span class="nt">&lt;/statusInfo&gt;</span> <span class="nt">&lt;isValidJP2&gt;</span>False<span class="nt">&lt;/isValidJP2&gt;</span> <span class="nt">&lt;tests/&gt;</span> <span class="nt">&lt;properties/&gt;</span> <span class="nt">&lt;/jpylyzer&gt;</span> </code></pre></div></div> <p>Previous versions would simply crash in this situation. Now, automated workflows can simply check for the value of the <em>success</em> field to verify the status of the validation. More importantly, if the <em>jpylyzer</em> invocation involved multiple input files (e.g. through the <code class="language-plaintext highlighter-rouge">--recurse</code> option), errors like these will not stop the processing of the remaining files.</p> <h2 id="64-bit-windows-binaries">64-bit Windows binaries</h2> <p>Finally, from version 1.15.1 onwards we are now providing 64 bit Windows binaries of <em>jpylyzer</em> (previously only 32-bit binaries were available).</p> <h2 id="links">Links</h2> <p><a href="http://jpylyzer.openpreservation.org/">Jpylyzer website</a></p> <hr /> <p>Originally published at the <a href="http://blog.kbresearch.nl/2015/12/08/jpylyzer-2015-round-up/">KB Research blog</a></p> https://bitsgalore.org/2015/12/08/jpylyzer-2015-round-up https://bitsgalore.org/2015/12/08/jpylyzer-2015-round-up 2015-12-08T00:00:00+01:00 <p>The KB has quite a large collection of offline optical media, such as CD-ROMs, DVDs and audio CDs. We’re currently investigating how to stabilise the contents of these materials using disk imaging. During the initial phase of this work I did a number of tests with various open-source tools. It’s doubtful whether we’ll end up using these same tools in our actual workflows. The main reason for this is the sheer size of the collection, which we estimated at some 15,000 physical carriers; possibly even more. At those volumes we will need a solution that involves the use of a disk robot, and these often require dedicated software (we still need to investigate this more in-depth).</p> <p>Nevertheless, throughout the initial testing phase I was surprised at the number of useful tools that are available in the open source domain. Since this will probably be of interest to others as well, I decided to polish a selection from my <a href="https://gist.github.com/bitsgalore/1bea8f015eca21a706e7#file-notescdimaging-md">rough working notes</a> into a somewhat more digestible form (or so I hope!). I edited my original notes down to the following topics:</p> <ul> <li>How to figure out the device path of the CD drive</li> <li>How to create an ISO image from a CD-ROM or DVD</li> <li>How to check the integrity of the created ISO image</li> <li>How to extract audio from an audio CD</li> </ul> <p>In addition there’s a final section that covers my attempts at imaging a multisession / mixed mode CD. The result of this particular exercise wasn’t all that successful, but I included it anyway, as some may find it useful. All software mentioned here are open-source tools that are available for any modern Linux distribution (I’m using Linux Mint myself). Some can be used under Windows as well using <a href="https://www.cygwin.com/">Cygwin</a>.</p> <!-- more --> <h2 id="2024-update-on-imaging-and-ripping-tools">2024 update on imaging and ripping tools</h2> <p>The information in this post largely reflects the tool landscape at the time I originally wrote this in 2015. Since then, several new tools have emerged. For a more up to date (2024) discussion of imaging and ripping tools, my go-to reference would be Misty De Meo’s <a href="https://www.mistys-internet.website/blog/blog/2024/09/13/the-working-archivists-guide-to-enthusiast-cd-rom-archiving-tools/">The Working Archivist’s Guide to Enthusiast CD-ROM Archiving Tools</a>. It recommends the <a href="https://github.com/superg/redumper">redumper</a> tool as the first choice for archivists interested in a commandline tool. <a href="https://github.com/aaru-dps/Aaru">Aaru</a> is another recent imaging tool. I haven’t personally done any elaborate testing with either of these tools, but if I get round to this I may include them in a future update.</p> <h2 id="find-the-device-path-of-the-cd-drive-linux">Find the device path of the CD drive (Linux)</h2> <p>The majority of the tools covered by this blog post need the device path of the CD drive as a command-line argument. Under Linux you can usually find this by inspecting the output of the following command (run this while a CD or DVD is inserted in your drive):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mount|grep ^<span class="s1">'/dev'</span> </code></pre></div></div> <p>If all goes well, the result will look similar to this:</p> <pre><code class="language-data">/dev/sda1 on / type ext4 (rw,errors=remount-ro) /dev/sr0 on /media/johan/REBELS_0 type iso9660 (ro,nosuid,nodev,uid=1000,gid=1000,iocharset=utf8,mode=0400,dmode=0500,uhelper=udisks2) </code></pre> <p>So, in this case the path to the CD drive is <em>/dev/sr0</em> (if you have multiple optical drives you may also see <em>/dev/sr1</em>, and so on).</p> <h2 id="finding-the-device-path-on-windows-cygwin">Finding the device path on Windows (Cygwin)</h2> <p>For some reason the <em>mount</em> command doesn’t result in the printing of any device paths in CygWin. Instead, try this:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">ls</span> /dev/ </code></pre></div></div> <p>Which produces a list of all devices:</p> <pre><code class="language-data">clipboard dsp mqueue random sda2 sdc1 stdin ttyS2 conin fd null scd0 sdb shm stdout urandom conout full ptmx sda sdb1 sr0 tty windows console kmsg pty0 sda1 sdc stderr ttyS0 zero </code></pre> <p>In the above output both <em>sr0</em> and <em>scd0</em> point to the CD drive, and either the full paths <em>/dev/sr0</em> or <em>/dev/scd0</em> will work (again in case of multiple drives you may be looking for <em>/dev/sr1</em> or <em>/dev/scd1</em>).</p> <p>In all examples below I assumed that the device path is <em>/dev/sr0</em>; substitute your own path if necessary.</p> <h2 id="create-iso-image-of-a-cd-rom-or-dvd">Create ISO image of a CD-ROM or DVD</h2> <p>A number of tools allow you to create an (ISO<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>) image from a CD-ROM or DVD. Although generic Unix data copying and recovery tools like <a href="http://linux.die.net/man/1/dd">dd</a> and <a href="http://linux.die.net/man/1/ddrescue">ddrescue</a> are often used for this, various people have pointed out that the result may be unreliable because they only perform limited error checking. See for example the comments <a href="http://www.commandlinefu.com/commands/view/10957/rip-a-cddvd-to-iso-format.0">here</a> and <a href="https://pthree.org/2011/09/26/how-to-properly-create-and-burn-cddvd-iso-images-from-the-command-line/">here</a>; both recommend to use the <a href="http://linux.die.net/man/1/readom"><em>readom</em></a> tool, which is part of the <a href="https://en.wikipedia.org/wiki/Cdrkit"><em>cdrkit</em></a> library. My own experience with <em>readom</em> is that while it works great in most cases, it is less suitable for CD-ROMs that are damaged or otherwise degraded. In those cases <em>ddrescue</em> is often a better choice. So below I’ll first show how to use <em>readom</em>, followed by a <em>ddrescue</em> example that specifically addresses the recovery of a CD-ROM gives read errors in <em>readom</em>.</p> <h3 id="running-readom">Running readom</h3> <p>The <a href="http://linux.die.net/man/1/readom">documentation</a> recommends to always run <em>readom</em> as root. Also, before running <em>readom</em>, the CD or DVD must be unmounted<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>. So, after inserting the CD or DVD, first enter this:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>umount /dev/sr0 </code></pre></div></div> <p>Then run <em>readom</em> as root:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>readom <span class="nv">retries</span><span class="o">=</span>4 <span class="nv">dev</span><span class="o">=</span>/dev/sr0 <span class="nv">f</span><span class="o">=</span>mydisk.iso </code></pre></div></div> <p>Here the value of the <em>retries</em> parameter defines the number of attempts that <em>readom</em> will make at trying to recover unreadable sectors. The default value is 128, which can result in huge processing times for CDs that are seriously damaged. The <em>f</em> parameter sets the name of the image file that is created. If all goes well the following output is printed to the screen at the end of the imaging process:</p> <pre><code class="language-data">Read speed: 4234 kB/s (CD 24x, DVD 3x). Write speed: 0 kB/s (CD 0x, DVD 0x). Capacity: 309104 Blocks = 618208 kBytes = 603 MBytes = 633 prMB Sectorsize: 2048 Bytes Copy from SCSI (10,0,0) disk to file 'mydisk.iso' end: 309104 addr: 309104 cnt: 44 Time total: 259.287sec Read 618208.00 kB at 2384.3 kB/sec. </code></pre> <h3 id="when-read-errors-happen-try-ddrescue">When read errors happen: try ddrescue</h3> <p>If the source medium is in a bad condition or otherwise damaged, <em>readom</em> will most likely terminate prematurely with read errors. If this happens, you may get better results with <a href="http://linux.die.net/man/1/ddrescue"><em>ddrescue</em></a>. There are two reasons for this:</p> <ol> <li>Unlike <em>readom</em>, which usually gives up pretty soon after the first read error occurs, <em>ddrescue</em> was specifically designed to deal with source media that contain errors. Consequently, it is much more persistive in such cases.</li> <li>If you try to read a defective source medium using two different CD drives (let’s call them <em>A</em> and <em>B</em>), it is not uncommon to find that some sectors that result in read errors on drive <em>A</em> are read correctly by drive <em>B</em> (and vice versa). With <em>ddrescue</em> it is possible to take advantage of this.</li> </ol> <p>The <a href="https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html#Optical-media"><em>ddrescue</em> Manual</a> manual gives a (very concise) example of how this works. Based on this I created the following, more detailed example.</p> <p>First we run <em>ddrescue</em> with the following command line:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ddrescue <span class="nt">-b</span> 2048 <span class="nt">-r4</span> <span class="nt">-v</span> /dev/sr0 mydisk.iso mydisk.log </code></pre></div></div> <p>Here <code class="language-plaintext highlighter-rouge">-b</code> sets the block size (which is typically 2048 bytes for a CD-ROM); <code class="language-plaintext highlighter-rouge">-r4</code> sets the maximum number of retries in case of bad sectors to 4<sup id="fnref:7"><a href="#fn:7" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>, and <code class="language-plaintext highlighter-rouge">-v</code> activates verbose output mode. File <em>mydisk.log</em> is a so-called <a href="https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html#Mapfile-structure"><em>mapfile</em></a> (known as <em>logfile</em> in <em>ddrescue</em> versions prior to 1.20). The <em>mapfile</em> contains (among a few other things) information on the recovery status of blocks of data. After running the above command on a faulty CD-ROM, we end up with output that looks like this:</p> <pre><code class="language-data">GNU ddrescue 1.17 About to copy 624918 kBytes from /dev/sr0 to mydisk.iso Starting positions: infile = 0 B, outfile = 0 B Copy block size: 32 sectors Initial skip size: 32 sectors Sector size: 2048 Bytes Press Ctrl-C to interrupt rescued: 624871 kB, errsize: 47104 B, current rate: 0 B/s ipos: 508162 kB, errors: 3, average rate: 592 kB/s opos: 508162 kB, time since last successful read: 12.3 m Finished </code></pre> <p>From this we can see the following:</p> <ul> <li>The CD-ROM contains 624918 kBytes of data (2nd line from top).</li> <li>Only 624871 kBytes were extracted (‘rescued’ field)</li> <li>A total of 47104 bytes were <em>not</em> rescued (‘errorsize’ field)</li> <li>3 errors occurred while reading the CD (‘errors’ field)</li> </ul> <p>However, it is often possible to improve the result by additional runs of <em>ddrescue</em> using either different options, or other hardware. First we’ll see if we can improve things by re-running in <a href="https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html#Direct-disc-access"><em>direct disc access</em></a> mode (this does not work on some systems, in which case <em>ddrescue</em> will report a warning). So we use the following command<sup id="fnref:6"><a href="#fn:6" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ddrescue <span class="nt">-d</span> <span class="nt">-b</span> 2048 <span class="nt">-r1</span> <span class="nt">-v</span> /dev/sr0 mydisk.iso mydisk.log </code></pre></div></div> <p>Here the <code class="language-plaintext highlighter-rouge">-d</code> switch activates direct disc access, which bypasses the kernel cache (note that the number of retries is set to 1 in the above example). Running the command causes <em>ddrescue</em> to update both the ISO and the mapfile. The screen output now looks like this:</p> <pre><code class="language-data">GNU ddrescue 1.17 About to copy 624918 kBytes from /dev/sr0 to mydisk.iso Starting positions: infile = 0 B, outfile = 0 B Copy block size: 32 sectors Initial skip size: 32 sectors Sector size: 2048 Bytes Press Ctrl-C to interrupt Initial status (read from logfile) rescued: 624871 kB, errsize: 47104 B, errors: 3 Current status rescued: 624912 kB, errsize: 6144 B, current rate: 0 B/s ipos: 508162 kB, errors: 3, average rate: 1706 B/s opos: 508162 kB, time since last successful read: 7 s Finished </code></pre> <p>What we see here:</p> <ul> <li>624871 kBytes were extracted (previously this was 624871)</li> <li>Consequently ‘errsize’ has gone down from to 47104 to 6144 bytes</li> </ul> <p>So this is better, but still not perfect. So let’s try if we can improve the results by using a different CD-reader. At this point I hooked up an external USB CD-drive, and moved my faulty CD-ROM from the internal reader to the external one. In this case my external drive is mapped under device path <em>/dev/sr2</em> (re-run the aforementioned steps to find the device path if necessary). This gives the following command-line:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ddrescue <span class="nt">-d</span> <span class="nt">-b</span> 2048 <span class="nt">-r4</span> <span class="nt">-v</span> /dev/sr2 mydisk.iso mydisk.log </code></pre></div></div> <p>Now the output looks like this:</p> <pre><code class="language-data">GNU ddrescue 1.17 About to copy 624918 kBytes from /dev/sr2 to mydisk.iso Starting positions: infile = 0 B, outfile = 0 B Copy block size: 32 sectors Initial skip size: 32 sectors Sector size: 2048 Bytes Press Ctrl-C to interrupt Initial status (read from logfile) rescued: 624916 kB, errsize: 2048 B, errors: 1 Current status rescued: 624918 kB, errsize: 0 B, current rate: 682 B/s ipos: 106450 kB, errors: 0, average rate: 682 B/s opos: 106450 kB, time since last successful read: 0 s Finished </code></pre> <p>From the output we can see that after re-running <em>ddrescue</em> with the external drive, both ‘errsize’ and the number of errors have gone down to 0. In other words: all of the contents of the CD have been rescued without any errors. Yay!</p> <p>In the above example I used two different CD readers that were connected to the same machine, but you could use as many readers as you like. It also possible to do the first run on one machine, transfer the ISO image and the mapfile to another machine, and then re-run <em>ddrescue</em> there (this even works across OS platforms).</p> <h2 id="check-integrity-of-iso-image-against-physical-cd-rom-or-dvd">Check integrity of ISO image against physical CD-ROM or DVD</h2> <p>In theory you could use check the integrity of the created ISO image by computing a checksum on both the ISO file and the physical carrier, and then comparing both:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">md5sum </span>mydisk.iso <span class="nb">md5sum</span> /dev/sr0 </code></pre></div></div> <p>However, in practice this comparison is not all that useful. Using dedicated data recovery tools like <em>readom</em> and particularly <em>ddrescue</em> often results in a more accurate capture of the data on a disc than accessing the corresponding device directly. Because of this, computing a checksum on the device using something like <code class="language-plaintext highlighter-rouge">md5sum /dev/sr0</code> can give unreliable results, resulting in a checksum mismatch that does not indicate any fault of the ISO image. It is worth noting that the aforementioned <a href="https://pthree.org/2011/09/26/how-to-properly-create-and-burn-cddvd-iso-images-from-the-command-line/">Aaron Toponce article</a> claims that <em>readom</em> already does a checksum check. If true, the additional check would be overkill (especially given that computing a checksum on a physical CD or DVD is time consuming). However, I couldn’t find any confirmation of this in either <em>readom</em>’s documentation nor its source code (although I found the source hard to read, so I may have simply overlooked it).</p> <h2 id="verify-iso-image">Verify ISO image</h2> <p>In theory, there shouldn’t be any need for additional quality checks on an ISO image once its integrity against the physical carrier is confirmed by the checksum. However, since <em>cdrkit</em> includes an <a href="http://linux.die.net/man/8/isoinfo"><em>isovfy</em></a> tool that claims to “ verify the integrity of an iso9660 image”, I decided I might as well give it a try. It works by entering:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>isovfy mydisk.iso </code></pre></div></div> <p>Here’s some example output:</p> <pre><code class="language-data">Root at extent 13, 2048 bytes [0 0] No errors found </code></pre> <p>The documentation of the tool isn’t very clear about <em>what</em> specific checks it performs. In one of my tests I fed it an ISO image that had its last 50 MB missing (truncated). This did not result in any error or warning message! Most of the reported <em>isovfy</em> errors that I came across in my tests simply reflected the file system on the physical CD not conforming to ISO 9660 (this seems to be pretty common). Based on this it looks like <em>isovfy</em> isn’t very useful after all.</p> <h3 id="isolyzer">Isolyzer</h3> <p>In response to the problems I encountered with <em>isovfy</em>, I created the <a href="https://github.com/KBNLresearch/isolyzer"><em>isolyzer</em></a> tool. <em>Isolyzer</em> checks the file size of an ISO image against the size information in the file system headers. This can be used to identify damaged and incomplete ISO images. Currently supported file systems are ISO 9660, UDF, HFS, HFS+ and a number of hybrids of these file systems. More information on <em>Isolyzer</em> can be found <a href="/2017/01/13/detecting-broken-iso-images-introducing-isolyzer">here</a> and <a href="/2017/07/12/update-on-isolyzer-udf-hfs-and-more">here</a>.</p> <h2 id="get-information-about-an-iso-image">Get information about an ISO image</h2> <h3 id="isoinfo">Isoinfo</h3> <p>The <a href="http://wiki.osdev.org/ISO_9660#The_Primary_Volume_Descriptor"><em>Primary Volume Descriptor</em></a> (PVD) of an ISO 9660 file system contains general information about the CD or DVD. The <a href="http://linux.die.net/man/8/isoinfo"><em>isoinfo</em></a> tool (which is also part of <em>cdrkit</em>) is able to print the most important PVD fields to the screen:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>isoinfo <span class="nt">-d</span> <span class="nt">-i</span> mydisk.iso </code></pre></div></div> <p>Result:</p> <pre><code class="language-data">CD-ROM is in ISO 9660 format System id: Volume id: REBELS_0 Volume set id: Publisher id: Data preparer id: Application id: NERO - BURNING ROM Copyright File id: Abstract File id: Bibliographic File id: Volume set size is: 1 Volume set sequence number is: 1 Logical block size is: 2048 Volume size is: 333151 Joliet with UCS level 3 found NO Rock Ridge present </code></pre> <p>You can also run <em>isoinfo</em> directly on the physical carrier:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>isoinfo <span class="nt">-d</span> <span class="nt">-i</span> /dev/sr0 </code></pre></div></div> <p>To get a listing of all files and directories that are part of the filesystem, use this:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>isoinfo <span class="nt">-f</span> <span class="nt">-i</span> mydisk.iso </code></pre></div></div> <p>Result:</p> <pre><code class="language-data">/AUTORUN.EXE;1 /AUTORUN.INF;1 /DISK0 /LICENSE2.TXT;1 /LICENSEF.TXT;1 /LICENSEU.TXT;1 /SETUP.EXE;1 /DISK0/CONTROLS.CFG;1 /DISK0/DISK0;1 :: :: etc </code></pre> <p>It looks like all items that are followed by <em>;1</em> are files, and those that aren’t are directories. Also, the <code class="language-plaintext highlighter-rouge">-l</code> option can be used for a detailed list that includes additional file attributes (size, date, etc.).</p> <h3 id="disktype">Disktype</h3> <p>The <a href="http://disktype.sourceforge.net/"><em>disktype</em></a> tool is particularly useful for identifying <a href="https://en.wikipedia.org/wiki/Hybrid_disc">hybrid disc</a> images that combine multiple file systems. For example:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>disktype bewaarmachine.iso </code></pre></div></div> <p>This results in:</p> <pre><code class="language-data">--- bewaarmachine.iso Regular file, size 342.1 MiB (358727680 bytes) Apple partition map, 2 entries Partition 1: 1 KiB (1024 bytes, 2 sectors from 1) Type "Apple_partition_map" Partition 2: 172.4 MiB (180773376 bytes, 353073 sectors from 346957) Type "Apple_HFS" HFS file system Volume name "de bewaarmachine" Volume size 172.4 MiB (180764672 bytes, 44132 blocks of 4 KiB) ISO9660 file system Volume name "BEWAARMACHINE_PC" Application "TOAST ISO 9660 BUILDER COPYRIGHT (C) 1993-1996 MILES SOFTWARE GMBH - HAVE A NICE DAY" Data size 169.4 MiB (177641472 bytes, 86739 blocks of 2 KiB) </code></pre> <p>In this example we have an image that contains both an ISO 9660 and an Apple HFS filesystem. <em>Disktype</em> can also be run directly on the physical carrier, using:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>disktype /dev/sr0 </code></pre></div></div> <h3 id="isolyzer-1">Isolyzer</h3> <p>The <a href="https://github.com/KBNLresearch/isolyzer"><em>isolyzer</em></a> tool also gives detailed information about an ISO image, including the file systems it contains. As an example:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>isolyzer bewaarmachine.iso <span class="o">&gt;</span> bewaarmachine.xml </code></pre></div></div> <p>This results <a href="https://gist.github.com/bitsgalore/bf5f9fb8e936efb9c4bc06a04443ef4a">in this output file</a>.</p> <h2 id="rip-audio-cd-with-cdparanoia">Rip audio CD with cdparanoia</h2> <p>The data structure of an audio CD is fundamentally different from a CD-ROM or DVD, and because of this its content cannot be stored as an ISO image. The most widely-used approach is to extract (or “rip”) the audio tracks on a CD to separate <a href="http://fileformats.archiveteam.org/wiki/WAV"><em>WAVE</em></a> files. A complicating factor here is that the way audio is encoded on a CD tends to obscure (small) read errors during playback. As a result, a single linear read will not result in a reliable transfer of the audio data. More details can be found in <a href="http://journal.code4lib.org/articles/9581">this excellent article by Alexander Duryee</a>. Duryee recommends a number of extraction tools that overcome this problem using sophisticated verification and correction functionality. One of these tools is the <a href="http://linux.die.net/man/1/cdparanoia"><em>cdparanoia</em></a> ripper. As an example, the following command can be used to rip a CD in batch mode, where each track is stored as a separate <em>WAVE</em> file:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cdparanoia <span class="nt">-B</span> <span class="nt">-L</span> </code></pre></div></div> <p>or:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cdparanoia <span class="nt">-B</span> <span class="nt">-l</span> </code></pre></div></div> <p>The <code class="language-plaintext highlighter-rouge">-L</code> switch results in the generation of a detailed log file; <code class="language-plaintext highlighter-rouge">-l</code> produces a summary log (name: <em>cdparanoia.log</em>)<sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">5</a></sup>. File names are generated automatically like this:</p> <pre><code class="language-data">track01.cdda.wav track02.cdda.wav track03.cdda.wav </code></pre> <p><a href="https://gist.github.com/bitsgalore/1bea8f015eca21a706e7#file-cdparanoialogsummary-log">Here is a link to an example log file</a>. The output may look a little weird at first sight, which is because <em>cdparanoia</em> reports all status and progress information as symbols and smilies, respectively. Their meaning is explained in the <a href="http://linux.die.net/man/1/cdparanoia">documentation</a>.</p> <h2 id="extract-dataaudio-from-mixed-mode-and-enhanced-cds">Extract data/audio from mixed mode and ‘enhanced’ CDs</h2> <p>Some CDs combine data and audio tracks. There are essentially two ways to do this:</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Mixed_Mode_CD">Mixed Mode</a> CDs contain both audio and data, both of which are written into one single session. Mixed Mode was often used for ’90s video games.</li> <li>The <a href="https://en.wikipedia.org/wiki/Blue_Book_(CD_standard)">Blue Book</a> standard defines a way to combine audio and data tracks. Blue Book CDs contain two sessions, where the first one contains one or more audio tracks, and the second one a data track. Examples of such discs are “enhanced” audio CDs that include software or movies as bonus material. They are sometimes referred to a “CD-Extra” discs.</li> </ul> <p>Even though the <em>data</em> part of such CDs is typically compatible with an ISO 9660 (or HFS, HFS+) file system, the audio tracks are not. Since <a href="http://anjackson.net/keeping-codes/practice/developing-a-robust-migration-workflow-for-preserving-and-curating-handheld-media.html">there is no good, open and mature file format to describe the contents of a CD precisely</a>, such CDs pose a particular challenge. In addition, tools such as <em>readom</em> and <em>ddrescue</em> typically only recognise the first session on a multisession CD, which means that they are not suitable for handling this type of disc.</p> <p>Based on some (relatively limited) testing, the <a href="http://linux.die.net/man/1/cdrdao"><em>cdrdao</em></a> tool does a good job at imaging mixed mode CDs, and a reasonable (but less than ideal) job for Blue book discs. Below are some brief notes on how to recognise both types of disc, and how to image them.</p> <h3 id="identifying-mixed-mode-cds">Identifying mixed-mode CDs</h3> <p>We can identify mixed-mode discs by running the <em>cd-info</em> command, which is part of the <a href="https://www.gnu.org/software/libcdio/libcdio.html"><em>GNU libcdio</em></a> package:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd-info /dev/sr0 </code></pre></div></div> <p>Now pay attention to the “CD Analysis Report” at the bottom of <em>cd-info</em>’s output:</p> <pre><code class="language-data">CD Analysis Report CD-TEXT for Disc: CD-TEXT for Track 1: CD-TEXT for Track 2: mixed mode CD CD-ROM with ISO 9660 filesystem ISO 9660: 235494 blocks, label `TNT_ROM ' Application: TOAST ISO 9660 BUILDER COPYRIGHT (C) 1993 MILES SOFTWARE ENGINEERING - HAVE A NICE DAY Preparer : Publisher : System : APPLE COMPUTER, INC., TYPE: 0002 Volume : TNT_ROM Volume Set : mixed mode CD XA sectors session #2 starts at track 2, LSN: 235719, ISO 9660 blocks: 235494 ISO 9660: 235494 blocks, label `TNT_ROM </code></pre> <p>Both the 4th line from the top and the 3rd line from the bottom contain the text <code class="language-plaintext highlighter-rouge">mixed mode CD</code>.</p> <h3 id="imaging-mixed-mode-cds">Imaging mixed-mode CDs</h3> <p><a href="https://web.archive.org/web/20151202174511/http://linuxreviews.org/howtos/cdrecording/">This article on the Linux Reviews site</a> (archived link) contains instructions on how to rip a mixed-mode CD using <em>cdrdao</em>. It involves a number of steps.</p> <p>First we have to unmount the disc:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>umount /dev/sr0 </code></pre></div></div> <p>Then run <em>cdrdao</em> with the following arguments:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cdrdao read-cd <span class="nt">--read-raw</span> <span class="nt">--datafile</span> toolstales.bin <span class="nt">--device</span> /dev/sr0 <span class="nt">--driver</span> generic-mmc-raw toolstales.toc </code></pre></div></div> <p>The result of this is a disc image in <em>BIN/TOC</em> format. The <em>.toc</em> file looks like this:</p> <pre><code class="language-data">CD_ROM_XA CATALOG "0000000000000" // Track 1 TRACK MODE2_RAW NO COPY DATAFILE "toolstales.bin" 52:20:69 // length in bytes: 554058288 // Track 2 TRACK AUDIO NO COPY NO PRE_EMPHASIS TWO_CHANNEL_AUDIO SILENCE 00:02:00 FILE "toolstales.bin" #554058288 0 14:58:22 START 00:02:00 </code></pre> <p>The <em>BIN/TOC</em> format is not easily accessible, so we need to do some additional post-processing to convert the image into a more accessible format.</p> <p>First we convert the <em>.toc</em> file to the <em>.cue</em> format (as defined in Appendix A of the <a href="https://web.archive.org/web/20070614044112/http://www.goldenhawk.com/download/cdrwin.pdf"><em>CDRWIN</em> User Guide</a>). For this we use the <em>toc2cue</em> tool (which is part of <em>cdrdao</em>):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>toc2cue toolstales.toc toolstales.cue </code></pre></div></div> <p>We now have a <em>BIN/CUE</em> image. On Linux we can mount this image with a virtual drive controller such as <a href="https://cdemu.sourceforge.io/"><em>cdemu</em></a>. This way both the audio and the data are accessible in the same way they would be from the physical carrier.</p> <p>If needed it is possible to extract the data track of the <em>BIN/CUE</em> file to an ISO image, and any audio tracks to <em>WAVE</em> files. For this we need the <a href="http://linux.die.net/man/1/bchunk"><em>bchunk</em></a> tool. Now we invoke it with the following arguments:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bchunk <span class="nt">-s</span> <span class="nt">-w</span> toolstales.bin toolstales.cue toolstales </code></pre></div></div> <p>In the example above, the <code class="language-plaintext highlighter-rouge">-w</code> option tells <em>bchunk</em> to extract audio tracks to <em>WAVE</em> files, and the <code class="language-plaintext highlighter-rouge">-s</code> option does a byte swap on the audio samples<sup id="fnref:8"><a href="#fn:8" class="footnote" rel="footnote" role="doc-noteref">6</a></sup>. The last argument defines the base name for all created output files. In this case the command results in 2 files: <em>toolstales01.iso</em>, which is a mountable ISO image, and <em>toolstales02.wav</em>, which is the audio track.</p> <h3 id="identifying-enhanced-cds">Identifying enhanced CDs</h3> <p>To identify enhanced (Blue Book) CDs, we again use <em>cd-info</em>:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd-info /dev/sr0 </code></pre></div></div> <p>Here’s the corresponding “CD Analysis Report” at the bottom of <em>cd-info</em>’s output:</p> <pre><code class="language-data">CD Analysis Report CD-TEXT for Disc: CD-TEXT for Track 1: :: :: CD-TEXT for Track 18: CD-Plus/Extra session #2 starts at track 18, LSN: 163570, ISO 9660 blocks: 170006 ISO 9660: 170006 blocks, label `NO </code></pre> <p>Note <code class="language-plaintext highlighter-rouge">CD-Plus/Extra</code>, which indicates this is a multi-session CD.</p> <h3 id="imaging-enhanced-cds">Imaging enhanced CDs</h3> <p>The procedure for imaging enhanced CDs is largely identical to the one for mixed-mode CDs. However, a major limitation here is that <em>cdrdao</em> is not able to combine the data/audio from both sessions into one disc image: running <em>cdrdao</em> with the command-line arguments as shown in the previous section will only create an image of the first session! However, it is possible to image both sessions separately into two image files. As an example, below are the steps I followed in an attempt to make a copy of They Might Be Giants’ <a href="http://tmbw.net/wiki/No!"><em>“No”</em></a> album (which contains some video content). Again I first unmounted the disk:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>umount /dev/sr0 </code></pre></div></div> <p>Then I used the below command to create an image of the first session (note the <code class="language-plaintext highlighter-rouge">--session</code> option):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cdrdao read-cd <span class="nt">--read-raw</span> <span class="nt">--session</span> 1 <span class="nt">--datafile</span> no1.bin <span class="nt">--device</span> /dev/sr0 <span class="nt">--driver</span> generic-mmc-raw no1.toc </code></pre></div></div> <p>And then again for the second session:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cdrdao read-cd <span class="nt">--read-raw</span> <span class="nt">--session</span> 2 <span class="nt">--datafile</span> no2.bin <span class="nt">--device</span> /dev/sr0 <span class="nt">--driver</span> generic-mmc-raw no2.toc </code></pre></div></div> <p>Running <em>cdrdao</em> twice like this, I was able to create two separate images with the audio and file system data, respectively.</p> <p>As in the mixed-mode example, both sessions are extracted as <em>BIN/TOC</em> files, so again we use <em>bchunk</em> to convert to <em>BIN/CUE</em>:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>toc2cue no1.toc no1.cue toc2cue no2.toc no2.cue </code></pre></div></div> <p>And then use <em>bchunk</em> to extract ISO and <em>WAVE</em> files from the <em>BIN/CUE</em> images:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bchunk <span class="nt">-s</span> <span class="nt">-w</span> no1.bin no1.cue no1 bchunk <span class="nt">-s</span> <span class="nt">-w</span> no2.bin no2.cue no2 </code></pre></div></div> <p>One thing to watch out for is that in most cases ISO images from an enhanced CD cannot directly be accessed or mounted. The reason for this is that the sector offsets that point to the files in the image are defined <em>relative to the beginning of the physical disc</em>, and not <em>relative to the start of the image</em>! More details on this can be found <a href="/2017/04/25/imaging-cd-extra-blue-book-discs">in this blog post</a>, which also describes a workaround that allows one to access such images under Linux.</p> <h2 id="additional-material">Additional material</h2> <ul> <li> <p>The rough, unedited notes on which this blog post is based can be found <a href="https://gist.github.com/bitsgalore/1bea8f015eca21a706e7#file-notescdimaging-md">here</a> (they contain some additional material that I left out here for readability).</p> </li> <li> <p>The <a href="https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html#Optical-media">User Manual of <em>ddrescue</em></a> gives some useful additional examples of how this tool can be used to recover data from a faulty CD-ROM.</p> </li> <li> <p><a href="https://www.mistys-internet.website/blog/blog/2024/09/13/the-working-archivists-guide-to-enthusiast-cd-rom-archiving-tools/">The Working Archivist’s Guide to Enthusiast CD-ROM Archiving Tools</a> provides an up to date (2024) overview of the imaging and ripping tools landscape.</p> </li> </ul> <h2 id="revision-history">Revision history</h2> <h3 id="june-2019">June 2019</h3> <ul> <li>Revised section on checksum verification, and added explanation that in practice this is not very useful.</li> <li>Added references to <em>Isolyzer</em>.</li> <li>The section on mixed-mode and multisession discs confusingly mixed up both types of carriers. I have clarified the distinction between both types of carriers, the instructions for imaging them are now in separate sections.</li> </ul> <h3 id="november-2024">November 2024</h3> <ul> <li>Added references to The Working Archivist’s Guide to Enthusiast CD-ROM Archiving Tools, Redumper and Aaru.</li> </ul> <hr /> <p>Originally published at the <a href="http://blog.kbresearch.nl/2015/11/13/preserving-optical-media-from-the-command-line/">KB Research blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:2"> <p>Whether the resulting image will conform to ISO 9660 depends on the source medium, as the image is simply a byte-exact copy of the data on the physical carrier’s file system. So for a DVD that uses the <a href="https://en.wikipedia.org/wiki/Universal_Disk_Format">UDF</a> format, the ISO image will be UDF as well. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>If you don’t do this you will end up with this error: <em>Error trying to open /dev/sr0 exclusively (Device or resource busy)… retrying in 1 second.</em> <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:7"> <p>This is a pretty arbitrary value, and you can use whatever value you like. <a href="#fnref:7" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:6"> <p>It is important that the names of the ISO and mapping file are identical to those used in the previous <em>ddrescue</em> run. This allows the tool to process <em>only</em> the problematic sectors (and skip everything else). <a href="#fnref:6" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>Strangely, in my tests a parse error occurred when I specified user-defined file names here. Also, it appeared that the summary log file resulted in more detailed output than the detailed one. This needs a more in-depth look! <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:8"> <p>I initially omitted this, and ended up with <em>WAVE</em> files that all played as static noise! This is a bit odd, since according to its <a href="http://www-mmsp.ece.mcgill.ca/Documents/AudioFormats/WAVE/WAVE.html">specification</a> the <em>WAVE</em> format is little-Endian by definition. <a href="#fnref:8" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2015/11/13/preserving-optical-media-from-the-command-line https://bitsgalore.org/2015/11/13/preserving-optical-media-from-the-command-line 2015-11-13T00:00:00+01:00 <p>Today my attention was caught by <a href="https://www.townswebarchiving.com/2015/10/jpeg2000-and-digitisation-expert-round-table/">this report of an “Expert round table” on JPEG2000 and Digitisation</a>, which was published on the TownsWeb Archiving blog. Although the report as a whole is quite balanced, it’s unfortunate that it provides fuel to some long-running myths about JPEG 2000 not supporting fully lossless compression. Since I wasn’t able to leave a comment on the Townweb blog itself, I turned my response into this small blog post.</p> <!-- more --> <h2 id="visually-lossless-vs-mathematically-lossless">Visually lossless vs mathematically lossless</h2> <p>For a start, <strong>Dave Thompson</strong> says about JPEG 2000:</p> <blockquote> <p>Its wavelet based technology means that it can be used in a compressed format which is visually lossless.</p> </blockquote> <p>While this statement is true, it may confuse some people, since it doesn’t mention that <em>mathematically</em> lossless compression is supported as well (in which case decoding the image returns the <em>exact</em> pixel values as they were prior to compression). Also, “visually lossless” compression is just a lossy compression that results in compression errors that are not detectable to the eye (see also the definition <a href="http://www.digitizationguidelines.gov/term.php?term=compressionvisuallylossless">here</a>). This is not unique to JPEG 2000, and there’s nothing that stops you from implementing visually lossless compression with “ordinary” JPEG, even though this would be pretty inefficient when compared to JPEG 2000.</p> <h2 id="suitability-for-preservation">Suitability for preservation</h2> <p>More seriously, <strong>Paul Sugden</strong> questions JPEG 2000’s suitability as a preservation format. The reason behind his concerns is a conference talk he attended:</p> <blockquote> <p>I attended a conference last year where an expert seemed to demonstrate that it was not possible to convert a JPEG2000 image precisely back to the original lossless TIFF file from which it was created. His example showed that after the retro conversion the file size of the newly created TIFF was different to the original TIFF as captured, and there were also visible differences between the images (albeit minor differences).</p> </blockquote> <p>From this he concludes:</p> <blockquote> <p>In my opinion, a file format that does not offer accurate retro-conversion back to precisely the image that was originally captured certainly cannot be seen as a reliable preservation format.</p> </blockquote> <p>This is quite a bold statement, especially given that it is based on only <em>one</em> report (of which Sugden doesn’t provide any specific information). Having done some pretty extensive testing with different encoders and decoders, I suspect the <em>real</em> problem here is just some bug in a specific encoder. I’ve encountered similar problems myself (e.g. see <a href="https://github.com/bitsgalore/jpegToLosslessJP2">here</a>), but those are just software bugs, which say <em>nothing about the format itself</em>, nor about its general suitability as a preservation format.</p> <p>So, without any supporting evidence I don’t see much that justifies the sweeping generalisations that are made in the blog post (but I’m open to be proven wrong!).</p> <h2 id="verifying-lossless-image-migrations">Verifying lossless image migrations</h2> <p>To be clear: JPEG 2000 fully supports completely lossless compression. The “losslessness” is also easy to verify using pixel-wise comparisons between source and destinaton images (e.g. using ImageMagick’s “compare” tool, some examples <a href="https://github.com/bitsgalore/jpegToLosslessJP2">here</a>). Not all encoders handle things like embedded metadata and ICC profiles equally well. Also, JPEG 2000’s baseline JP2 format has some restrictions on the embedding of ICC profiles, which can be a problem in very specific cases (e.g. when the ICC profile of the source TIFF you want to convert is not supported by JP2).</p> https://bitsgalore.org/2015/10/19/Response-to-report-on-JPEG-2000-expert-round-table https://bitsgalore.org/2015/10/19/Response-to-report-on-JPEG-2000-expert-round-table 2015-10-19T00:00:00+02:00 <p>This is the second and final instalment of a 2-part blog on the use of PDF/A validators for identifying preservation risks in PDF. You can read the first part <a href="/2015/07/07/why-pdfa-validation-matters-even-if-you-dont-have-pdfa">here</a>. In Part 1 I showed how PDF/A validators can be used to identify preservation risks in a PDF. I illustrated this with an example that uses the PDF/A validator component of Adobe Acrobat’s Preflight tool. Needless to say, Acrobat is not scalabe to situations where you need to verify large volumes of PDFs. Luckily, several stand-alone PDF/A validators exist that are designed especially to do just that.</p> <!-- more --> <h2 id="apache-preflight">Apache Preflight</h2> <p>During the <a href="http://www.scape-project.eu/">SCAPE</a> project we did a number of experiments with the PDF/A validator that is part of the open-source <a href="https://pdfbox.apache.org/">Apache PDFBox</a> library (incidentally it is also called Preflight). Throwing the PDF of our last example at Apache Preflight results in the following output<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">&lt;?xml version="1.0" encoding="UTF-8" standalone="no"?&gt;</span> <span class="nt">&lt;preflight</span> <span class="na">name=</span><span class="s">"Jpeg_linked.pdf"</span><span class="nt">&gt;</span> <span class="nt">&lt;executionTimeMS&gt;</span>9792<span class="nt">&lt;/executionTimeMS&gt;</span> <span class="nt">&lt;isValid</span> <span class="na">type=</span><span class="s">"PDF/A1-b"</span><span class="nt">&gt;</span>false<span class="nt">&lt;/isValid&gt;</span> <span class="nt">&lt;errors</span> <span class="na">count=</span><span class="s">"96"</span><span class="nt">&gt;</span> <span class="nt">&lt;error</span> <span class="na">count=</span><span class="s">"1"</span><span class="nt">&gt;</span> <span class="nt">&lt;code&gt;</span>3.1.3<span class="nt">&lt;/code&gt;</span> <span class="nt">&lt;details&gt;</span>Invalid Font definition, CourierNewPSMT: FontFile entry is missing from FontDescriptor<span class="nt">&lt;/details&gt;</span> <span class="nt">&lt;/error&gt;</span> <span class="nt">&lt;error</span> <span class="na">count=</span><span class="s">"1"</span><span class="nt">&gt;</span> <span class="nt">&lt;code&gt;</span>7.11<span class="nt">&lt;/code&gt;</span> <span class="nt">&lt;details&gt;</span>Error on MetaData, PDF/A identification schema http://www.aiim.org/pdfa/ns/id/ is missing<span class="nt">&lt;/details&gt;</span> <span class="nt">&lt;/error&gt;</span> <span class="nt">&lt;error</span> <span class="na">count=</span><span class="s">"3"</span><span class="nt">&gt;</span> <span class="nt">&lt;code&gt;</span>6.2.1<span class="nt">&lt;/code&gt;</span> <span class="nt">&lt;details&gt;</span>Action is forbidden, GoToPage isn't authorized as named action<span class="nt">&lt;/details&gt;</span> <span class="nt">&lt;page&gt;</span>0<span class="nt">&lt;/page&gt;</span> <span class="nt">&lt;/error&gt;</span> :: :: <span class="nt">&lt;error</span> <span class="na">count=</span><span class="s">"1"</span><span class="nt">&gt;</span> <span class="nt">&lt;code&gt;</span>1.4.2<span class="nt">&lt;/code&gt;</span> <span class="nt">&lt;details&gt;</span>Trailer Syntax error, The trailer dictionary contains Encrypt<span class="nt">&lt;/details&gt;</span> <span class="nt">&lt;/error&gt;</span> <span class="nt">&lt;/errors&gt;</span> <span class="nt">&lt;/preflight&gt;</span> </code></pre></div></div> <h2 id="assessment-against-a-technical-profile--policy">Assessment against a technical profile / policy</h2> <p>By post-processing Preflight’s XML output further, it is possible to automatically evaluate PDFs against a user-defined set of features (i.e. a technical profile, equivalent to what was known as a <a href="http://openpreservation.org/blog/2013/09/04/control-policies-scape-project/"><em>control policy</em></a> in the SCAPE project). This is pretty straightforward if you express all features (or policy elements) as <a href="https://en.wikipedia.org/wiki/Schematron">Schematron</a> rules. Here’s an example of a Schematron rule that checks for encryption:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">&lt;?xml version="1.0"?&gt;</span> <span class="c">&lt;!-- Schematron rules for policy-based validation of PDF, based on output of Apache Preflight. --&gt;</span> <span class="nt">&lt;s:schema</span> <span class="na">xmlns:s=</span><span class="s">"http://purl.oclc.org/dsdl/schematron"</span><span class="nt">&gt;</span> <span class="nt">&lt;s:pattern</span> <span class="na">name=</span><span class="s">"Checks for encryption"</span><span class="nt">&gt;</span> <span class="nt">&lt;s:rule</span> <span class="na">context=</span><span class="s">"/preflight/errors/error"</span><span class="nt">&gt;</span> <span class="nt">&lt;s:assert</span> <span class="na">test=</span><span class="s">"not(code = '1.0' and contains(details,'password'))"</span><span class="nt">&gt;</span>Open password<span class="nt">&lt;/s:assert&gt;</span> <span class="nt">&lt;s:assert</span> <span class="na">test=</span><span class="s">"not(code = '1.4.2')"</span><span class="nt">&gt;</span>Encryption<span class="nt">&lt;/s:assert&gt;</span> <span class="nt">&lt;/s:rule&gt;</span> <span class="nt">&lt;/s:pattern&gt;</span> <span class="nt">&lt;/s:schema&gt;</span> </code></pre></div></div> <p>Rules can be defined for other features as well (e.g. multimedia, fonts), which makes it possible to test against custom policies. The figure below illustrates the general procedure:</p> <p><img src="/images/2015/07/preflightflow.png" alt="" /></p> <p>A simple demo (based on Shellscript) that implements the above workflow can be found <a href="https://github.com/openpreserve/pdfPolicyValidate">here</a>.</p> <h2 id="test-with-govdocs1-corpus">Test with Govdocs1 corpus</h2> <p>As part of the SCAPE work, we tested whether we could use Preflight in this way to assess a large set of PDFs. For this we used about 15,000 PDFs from the <a href="http://digitalcorpora.org/corpora/govdocs">Govdocs1 corpus</a>. We tried to assess these PDFs against a user-defined policy, which was made up of the following elements:</p> <ol> <li>No encryption or password protection</li> <li>Fonts must be embedded and complete</li> <li>No JavaScript</li> <li>No <a href="/2013/01/09/what-do-we-mean-embedded-files-pdf">embedded files</a> (i.e. file attachments)</li> <li>No multimedia content (audio, video, 3-D objects)</li> <li>File should be valid PDF<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>.</li> </ol> <p>The somewhat disappointing result of this exercise was that only 26% of all PDFs in the dataset satisfied all criteria in our test policy! Closer inspection of the Preflight output showed the majority of validation errors that caused this to be related to fonts. Preflight is able to report on many different font-related errors, but their exact meaning is not always clear, and neither is their impact on the rendering process. This made it difficult to establish whether the results reflected the quality of the PDFs, or perhaps our assessment was too strict on font errors.</p> <h2 id="way-forward-verapdf">Way forward: VeraPDF</h2> <p>The <a href="/2014/01/27/identification-pdf-preservation-risks-analysis-govdocs-selected-corpus">original report on the Govdocs1 analysis</a> ended with the following conclusions:</p> <blockquote> <p>These preliminary results show that policy-based assessment of PDF is possible using a combination of Apache Preflight and Schematron. However, dealing with <strong>font issues</strong> appears to be a particular challenge. Also, the lack of reliable tools to test for <strong>overall conformity to PDF (e.g. ISO 32000)</strong> is still a major limitation. Another limitation of this analysis is the lack of <strong>ground truth</strong>, which makes it difficult to assess the accuracy of the results.</p> </blockquote> <p>Earlier this year work started on <a href="http://verapdf.org/">VeraPDF</a>, an open-source PDF/A validator that -like Preflight- will be part of the PDFBox library. Its development is funded by the EU <a href="http://www.preforma-project.eu/">PREFORMA</a> project. The consortium that is behind the software includes the <a href="http://www.pdfa.org/">PDF Association</a>, whose member base covers a wide spectrum of vendors that already implement PDF technology. Although still in its early stages, it’s interesting to see how the VeraPDF work could help in solving the issues that we identified as part of the SCAPE work.</p> <h3 id="font-issues">Font issues</h3> <p>As I’m writing this, font checks haven’t been implemented yet in the VeraPDF code. Nevertheless, <a href="https://github.com/veraPDF/veraPDF-validation-profiles">validation profiles</a> already exist for a number of aspects of PDF/A. These profiles contain one or more validation rules, and each rule explicitly references its corresponding clause in the PDF/A standard. For example, have a look at <a href="https://github.com/veraPDF/veraPDF-validation-profiles/blob/master/PDF_A/1b/6.2%20Graphics/6.2.4%20Images/verapdf-profile-6-2-4-t01.xml">this rule on images</a>:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">&lt;?xml version="1.0" encoding="UTF-8"?&gt;</span> <span class="nt">&lt;profile</span> <span class="na">xmlns=</span><span class="s">"http://www.verapdf.org/ValidationProfile"</span> <span class="na">model=</span><span class="s">"org.verapdf.model.PDFA1a"</span><span class="nt">&gt;</span> <span class="nt">&lt;name&gt;</span>ISO 19005-1:2005 - 6.2.4 Images - Alternates<span class="nt">&lt;/name&gt;</span> <span class="nt">&lt;description&gt;&lt;/description&gt;</span> <span class="nt">&lt;creator&gt;</span>veraPDF Consortium<span class="nt">&lt;/creator&gt;</span> <span class="nt">&lt;created&gt;</span>2015-06-16T22:22:45Z<span class="nt">&lt;/created&gt;</span> <span class="nt">&lt;hash&gt;</span>sha-1 hash code<span class="nt">&lt;/hash&gt;</span> <span class="nt">&lt;rules&gt;</span> <span class="nt">&lt;rule</span> <span class="na">id=</span><span class="s">"6-2-4-t01"</span> <span class="na">object=</span><span class="s">"PDXImage"</span><span class="nt">&gt;</span> <span class="nt">&lt;description&gt;</span>An Image dictionary shall not contain the Alternates key<span class="nt">&lt;/description&gt;</span> <span class="nt">&lt;test&gt;</span>Alternates_size == 0<span class="nt">&lt;/test&gt;</span> <span class="nt">&lt;error&gt;</span> <span class="nt">&lt;message&gt;</span>Alternates key is present in the Image dictionary(<span class="nt">&lt;/message&gt;</span> <span class="nt">&lt;/error&gt;</span> <span class="nt">&lt;reference&gt;</span> <span class="nt">&lt;specification&gt;</span>ISO19005-1<span class="nt">&lt;/specification&gt;</span> <span class="nt">&lt;clause&gt;</span>6.2.4<span class="nt">&lt;/clause&gt;</span> <span class="nt">&lt;/reference&gt;</span> <span class="nt">&lt;/rule&gt;</span> <span class="nt">&lt;/rules&gt;</span> <span class="nt">&lt;/profile&gt;</span> </code></pre></div></div> <p>Here, the fields in the <em>clause</em> field in the <em>reference</em> element refers to a specific clause in the PDF/A-1 (ISO 19005-1) specification. This makes the errors much easier to interpret, since they are directly linked to the standard. I expect that this will make the interpretation of font-related errors much clearer as well.</p> <h3 id="conformance-to-canonical-pdf">Conformance to canonical PDF</h3> <p>A PDF may satisfy all requirements of PDF/A, and still be broken. An example is file <a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/veraPDFHiResWrongObjectID.pdf?raw=true">veraPDFHiResWrongObjectID.pdf</a>. If you open it in Acrobat you will see this:</p> <p><img src="/images/2015/07/wrongobjectid.png" alt="" /></p> <p>Neverheless, Apache Preflight considers this to be “valid” PDF/A:</p> <p><img src="/images/2015/07/wrongobjectidpreflight.png" alt="" /></p> <p>The reason for this is that the structure of this file is broken at a deeper level than the (relatively high-level) PDF/A profiles<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>. Although the current funding for VeraPDF only addresses PDF/A, the <a href="http://www.openpreservation.org/documents/public/veraPDF_FunctionalTechnicalSpecification_v1.0.pdf">veraPDF Technical and Functional Specification</a> stresses that its validation model is extensible, and this would ultimately allow more elaborate validation. From p. 16 of the document:</p> <blockquote> <p>The veraPDF model encourages plug-ins for parsing not only PDF/A-related third-party data structures (…), but also for other features in ISO 32000, other ISO standards for PDF such as PDF/E or PRC, images, and for embedded content such as rich media or attachments (…)</p> </blockquote> <p>This suggests that ultimately, VeraPDF has the potential to develop into a full-fledged canonical (ISO 32000) PDF validator. Obviously this would be a huge task that would require substantial additional effort and funding, but it’s encouraging to see that the overall design already allows for such a move.</p> <h3 id="ground-truth">Ground truth</h3> <p>During the SCAPE project we often struggled to find suitable openly licensed test files. In fact, for much of the policy-based assessment work we relied on files on the <a href="http://acroeng.adobe.com">Adobe Acrobat Engineering website</a>, which is a true treasure trove of PDFs with exotic features. Or rather <em>was</em>, as the site’s been offline for at least several weeks now, and it’s unclear when (if?) it will be back<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>. Back in 2013, the BL’s Andy Jackson already <a href="https://forums.adobe.com/thread/1262403?tstart=0">inquired about the license terms of those files</a>, and Adobe’s response was that although the files were free to use, redistribution was not allowed. Fast-forward two years, and the files are gone! Internet Archive has <a href="https://web.archive.org/web/*/http://acroeng.adobe.com">several snapshots of the site</a>, but they are incomplete and do not include all sample files.</p> <p>This poignantly illustrates the importance of test data that are available under a sufficiently open license that allows redistribution. I’m happy to see that the VeraPDF initiative includes work on the production a number of openly-licensed test corpora (see also sections CE 3.2 and TS 6.2 of the <a href="http://www.openpreservation.org/documents/public/veraPDF_FunctionalTechnicalSpecification_v1.0.pdf">Technical and Functional Specification</a>).</p> <h2 id="conclusion">Conclusion</h2> <p>In this blog series I’ve given a brief overview of some preservation risks of the PDF format, and I showed how PDF/A validators can be used to identify such risks, <em>even in files that are not really PDF/A</em>. I also explained the main problems we encountered while trying to use the open-source Apache Preflight PDF/A validator to identify preservation risks in a large collection of PDFs. The new VeraPDF initiative is still in its early stages, but it appears to be addressing most of these issues. Therefore it would be interesting to apply it to some of the datasets that we used for SCAPE, once the software is more fully developed.</p> <h2 id="further-resources">Further resources</h2> <ul> <li> <p><a href="/2015/07/07/why-pdfa-validation-matters-even-if-you-dont-have-pdfa">Why PDF/A validation matters, even if you don’t have PDF/A (Part 1)</a></p> </li> <li> <p><a href="http://wiki.opf-labs.org/display/TR/Portable+Document+Format">Portable Document Format in the <em>OPF File Format Risk Registry</em></a></p> </li> <li> <p><a href="https://github.com/openpreserve/format-corpus/tree/master/pdfCabinetOfHorrors">The Archivist’s PDF Cabinet of Horrors (test corpus)</a></p> </li> </ul> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2015/07/08/why-pdfa-validation-matters-part-2/">Open Preservation Foundation blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>This is only an extract from the complete output file, which is much larger. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>Preflight does not perform canonical PDF validation, but it does do some additional checks beyond PDF/A, hence the “should” rather than “must”. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>More precisely, I deliberately changed the <a href="https://raw.githubusercontent.com/corkami/pics/master/PDF.png">object reference</a> to an image to a nonsense value. Incidentally, Acrobat Preflight <em>does</em> detect this error, which means that it checks at least <em>some</em> aspects of canonical PDF. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>Adobe’s web team <a href="https://twitter.com/bitsgalore/status/615485792375468032">are aware of the issue</a>, but it’s not clear when the site will be back (if at all) <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2015/07/08/why-pdfa-validation-matters-part-2 https://bitsgalore.org/2015/07/08/why-pdfa-validation-matters-part-2 2015-07-08T00:00:00+02:00 <p>This is the first instalment of a 2-part blog. It was prompted by the upcoming Digital Preservation Coalition briefing <a href="http://www.dpconline.org/events/details/95-preserving-pdfs-jul15"><em>When is a PDF not a PDF?</em></a>, for which I was asked to prepare a presentation. My initial idea was to give an overview of the work we did on PDF preservation risk assessment using a PDF/A validator in the <a href="http://www.scape-project.eu/">SCAPE</a> project. Most of this has already been <a href="/2012/12/19/identification-pdf-preservation-risks-apache-preflight-first-impression">covered</a> by a <a href="/2013/07/25/identification-pdf-preservation-risks-sequel">series</a> of <a href="/2014/01/27/identification-pdf-preservation-risks-analysis-govdocs-selected-corpus">earlier blog posts</a>. Those blogs very much represent different stages of a work in progress, and I think this makes them somewhat challenging for readers who are new to the subject.</p> <!-- more --> <p>The purpose of this 2-part blog is twofold: first it is an attempt to give an accessible overview of the earlier work on PDF preservation risks, stressing the importance of PDF/A validator tools in detecting these risks. Second, it provides some tentative suggestions of how the ongoing work on the new <a href="http://verapdf.org/">VeraPDF</a> PDF/A validator could close some of the gaps and limitations of the SCAPE work.</p> <h2 id="preservation-risks-of-pdf">Preservation risks of PDF</h2> <p>The PDF format has a number of features that don’t sit well with the aims of long-term preservation and accessibility. This includes encryption and password protection, external dependencies (e.g. fonts that are not embedded in a document), and reliance on external software. More details can be found in the <a href="http://wiki.opf-labs.org/display/TR/Portable+Document+Format">PDF entry of the OPF File Format Risk Registry</a>. Below are some examples; I included download links, so you can try them out for yourself.</p> <h3 id="document-open-password">Document Open password</h3> <p>If you try to open file <a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/encryption_openpassword.pdf?raw=true"><em>encryption_openpassword.pdf</em></a> in Adobe Acrobat, you end up with this dialog:</p> <p><img src="/images/2015/07/openpassword.png" alt="" /></p> <p>Without the password, the file cannot be opened at all.</p> <h3 id="print-password">Print password</h3> <p>File <a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/encryption_noprinting.pdf?raw=true"><em>encryption_noprinting.pdf</em></a> can be opened normally, but you cannot print it:</p> <p><img src="/images/2015/07/printpassword.png" alt="" /></p> <h3 id="embedded-quicktime-movie">Embedded Quicktime movie</h3> <p>File <a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/embedded_video_quicktime.pdf?raw=true"><em>embedded_video_quicktime.pdf</em></a> contains multimedia content in <a href="http://fileformats.archiveteam.org/wiki/Quicktime">Quicktime</a> format. Acrobat cannot render this format natively, and relies on an external player. This is what happened when I opened the file on my PC:</p> <p><img src="/images/2015/07/embeddquicktime.png" alt="" /></p> <p>After I clicked on <em>Get Media Player</em>, I was taken <a href="http://cgi.adobe.com/special/acrobat/mediaplayerfinder/mediaplayerfinder.cgi?">here</a>:</p> <p><img src="/images/2015/07/embeddquicktime2.png" alt="" /></p> <p>I wasn’t able to configure Acrobat to use a media player that supports Quicktime <sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>.</p> <h3 id="external-reference-to-multimedia-file">External reference to multimedia file</h3> <p>The file <a href="https://web.archive.org/web/20100714002808/http://acroeng.adobe.com/Test_Files/movie/movie.pdf"><em>movie.pdf</em></a> contains references to external multimedia files. If you click on any of them you get an error like this one:</p> <p><img src="/images/2015/07/movieexternal.png" alt="" /></p> <h3 id="font-not-embedded">Font not embedded</h3> <p>File <a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/calistoMTNoFontsEmbedded.pdf?raw=true"><em>calistoMTNoFontsEmbedded.pdf</em></a> uses <em>Calisto MT</em>, but the font is not embedded. Since <em>Calisto MT</em> is a Windows system font, the file looks fine on my Windows PC:</p> <p><img src="/images/2015/07/fontsorig.png" alt="" /></p> <p>The font does not come pre-installed with common Linux distros, and as a result the file looks quite a bit different on my Linux machine:</p> <p><img src="/images/2015/07/fontslinux.png" alt="" /></p> <h3 id="3d-content">3D content</h3> <p>The file <a href="https://github.com/openpreserve/format-corpus/blob/master/pdfCabinetOfHorrors/digitally_signed_3D_Portfolio.pdf?raw=true"><em>digitally_signed_3D_Portfolio.pdf</em></a> contains 3D artwork. Acrobat correctly renders the 3D content, which can be manipulated interactively by the user:</p> <p><img src="/images/2015/07/3dacrobat.png" alt="" /></p> <p>However, Acrobat aside, the majority of PDF readers don’t support 3D content, with the result that in other readers you may end up with something like this:</p> <p><img src="/images/2015/07/3dsumatra.png" alt="" /></p> <h2 id="detecting-risky-features">Detecting risky features</h2> <p>Archives or libraries may want to check their PDFs for one or more features like those shown above. Reasons for doing so include:</p> <ul> <li> <p>Pre-ingest checks against an institutional policy (e.g. an archive may not accept PDFs that are password protected)</p> </li> <li> <p>Profiling of existing collections for preservation risks (e.g. embedded multimedia content in hard-to-render formats)</p> </li> </ul> <p>For this quite a few useful software tools are already available. For example, <a href="https://github.com/qpdf/qpdf">qpdf</a> gives detailed information about encryption and password protection:</p> <p><img src="/images/2015/07/encryptqpdf.png" alt="" /></p> <p>Similarly, the <a href="http://www.linuxcommand.org/man_pages/pdffonts1.html">pdffonts</a> tool that is part of <a href="http://www.foolabs.com/xpdf/">xpdf</a> is useful for checking whether fonts in a PDF are embedded:</p> <p><img src="/images/2015/07/fontsxpdf.png" alt="" /></p> <p>As the number of features you want to check for increases, this approach becomes increasingly cumbersome: most of tools only cover <em>some</em> features, so you rapidly end up having to deal with a multitude of software tools and output formats. So you may ask yourself if there’s a way to do this more efficiently.</p> <h2 id="pdfa-validation">PDF/A validation</h2> <p>This is where <a href="https://en.wikipedia.org/wiki/PDF/A">PDF/A</a> enters the picture. The PDF/A standards are nothing more than a set of profiles that impose some restrictions on a PDF, ruling out features that are not well-suited to long-term accessibility. Unsurprisingly, these include the very same features that we are interested in here, such as encryption, non-embedded fonts, multimedia content, and so on. Several tools exist that compare a PDF against PDF/A and report any deviations. These PDF/A validators are typically used to verify “true” PDF/A files; however, they can also be used to detect user-specified risky features in regular PDFs.</p> <p>The professional version of Adobe Acrobat has a PDF/A validator built into its <a href="http://help.adobe.com/en_US/acrobat/X/pro/using/WS58a04a822e3e50102bd615109794195ff-7b82.w.html">Preflight</a> tool. After opening a PDF in Acrobat, it allows you to verify its compliance with a number of profiles, including PDF/A (currently A-1, 2 and 3):</p> <p><img src="/images/2015/07/acrobatpreflight1.png" alt="" /></p> <p>This results in output as shown here:</p> <p><img src="/images/2015/07/acrobatpreflight2.png" alt="" /></p> <p><a href="http://acroeng.adobe.com/Test_Files/classic_multimedia//Jpeg_linked.pdf">This PDF</a><sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup> (which isn’t a PDF/A) violates the PDF/A-1a profile in several ways, but supposing we’re only interested in encryption and non-embedded fonts, the relevant information can be extracted from Preflight’s output quite easily. This example demonstrates the overall feasibility of identifying preservation risks with a PDF/A validator, but it is not scalabe to situations where you need to verify large volumes of PDFs. This will be the main focus of the <a href="/2015/07/08/why-pdfa-validation-matters-part-2">second part</a> of this blog series.</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2015/07/07/why-pdfa-validation-matters-even-if-you-dont-have-pdfa/">Open Preservation Foundation blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>Acrobat’s <em>Preferences</em> do include some options for configuring behavior with multimedia content (explained <a href="https://helpx.adobe.com/acrobat/using/playing-video-audio-multimedia-formats.html">here</a>), but the list of media players in the <em>Preferred Media Player</em> dropdown list only included Windows Media Player and Adobe Flash Player. Neither of these support Quicktime. VLC Media player <em>does</em> support Quicktime, but it is not included in the dropdown list, leaving me no way to configure it. Bummer! <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>At the time of writing the Acrobat Engineering site was down, and this particular PDF is not included in any Wayback crawls either. Bummer again! <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2015/07/07/why-pdfa-validation-matters-even-if-you-dont-have-pdfa https://bitsgalore.org/2015/07/07/why-pdfa-validation-matters-even-if-you-dont-have-pdfa 2015-07-07T00:00:00+02:00 <p>The current version of the KB’s digital repository system (<a href="https://www.kb.nl/en/organisation/research-expertise/long-term-usability-of-digital-resources/the-e-depot-project-cycle">e-Depot</a>) doesn’t include any tools for automated <a href="http://www.forensicswiki.org/wiki/File_Format_Identification">file format identification</a> yet. Our previous <a href="https://www.kb.nl/en/organisation/research-expertise/long-term-usability-of-digital-resources/history-the-kb-and-digital-preservation">DIAS</a> system didn’t have identification functionality either. As a result, information on file formats in digital our collections is largely based on publisher metadata and file extensions. Neither are necessarily correct. Moreover, previous analyses revealed a number of prevalent file extensions that could not be easily linked to a specific format. One result of this situation was that we couldn’t even reliably tell to what extent patrons were able to view e-Depot content on the PCs in our reading rooms (the obviously common formats aside).</p> <p>To get a better view of the formats in our collection, we did an analysis of the “top 50” most prevalent file extensions in our e-Depot: what are the corresponding formats, can these formats be automatically identified, and can we render them in our reading rooms? This blog post summarises the main findings of this work.</p> <!-- more --> <h2 id="extension-counts">Extension counts</h2> <p>As a first step, we compiled a list with file counts for every unique file extension in our e-Depot. Importantly, we did this for <em>all</em> files on the file system, including main files, supplemental content and (original) metadata files. The following chart shows the number of files for every extension, sorted in descending order (note that the vertical axis has a logarithmic scale):</p> <p><img src="/images/2015/04/distributionFormats.png" alt="Distribution of file formats" /></p> <p>The total number of of unique extensions is no less than 1163. Somewhat surprisingly, <em>.gif</em> turned out to be the most prevalent extension at 34 million files<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. Altogether, the 10 most common extensions make up 99% of al files in the e-Depot. There is a long tail of extensions of which less than 10 file objects exist, and these make up for over half of all unique extensions. In the remainder of this blog we will take a closer look at the “top 50” of all file extensions. The full list is too large to include in this blog post, but you can view it as as a table at the following link:</p> <p><a href="https://gist.github.com/bitsgalore/21028de28b7f05066585#file-extensionskbdm-md">50 most prevalent formats in KB e-Depot by file extension</a></p> <h2 id="analysis-of-sample-dataset">Analysis of sample dataset</h2> <p>For each extension we extracted about 20 sample files<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>. We then tried to identify each file with <a href="https://tika.apache.org/">Apache Tika</a> (version 1.4) in <a href="https://tika.apache.org/1.8/detection.html">detector</a> mode. The third column of our <a href="https://gist.github.com/bitsgalore/21028de28b7f05066585#file-extensionskbdm-md">table</a> shows the results for each extension. A manual inspection of selected samples revealed some further details, which are listed in the fourth column of the <a href="https://gist.github.com/bitsgalore/21028de28b7f05066585#file-extensionskbdm-md">table</a> (you may need to use the scrollbar at the bottom to view it). One interesting finding was that <a href="http://fileformats.archiveteam.org/wiki/Matlab_figure">Matlab Figure</a> files were misidentified by Apache Tika as either <code class="language-plaintext highlighter-rouge">application/x-xfig</code> or <code class="language-plaintext highlighter-rouge">image/jpeg</code>.</p> <p>Further analysis of the contents of the 22 ZIP files in our test dataset yielded some additional formats:</p> <table> <thead> <tr> <th style="text-align: left">Extension</th> <th style="text-align: left">ID Tika</th> <th style="text-align: left">Remarks</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">cif</td> <td style="text-align: left">text/plain</td> <td style="text-align: left"><a href="http://fileformats.archiveteam.org/wiki/CIF">Crystallographic Information File</a></td> </tr> <tr> <td style="text-align: left">csv</td> <td style="text-align: left">text/csv</td> <td style="text-align: left"><a href="http://fileformats.archiveteam.org/wiki/CSV">Comma-separated values</a></td> </tr> <tr> <td style="text-align: left">mol</td> <td style="text-align: left">text/plain</td> <td style="text-align: left"><a href="http://fileformats.archiveteam.org/wiki/MOL">MDL Molfile</a></td> </tr> <tr> <td style="text-align: left">tdb</td> <td style="text-align: left">text/plain</td> <td style="text-align: left"><a href="http://fileformats.archiveteam.org/wiki/TDB">Thermo-Calc Database Format</a></td> </tr> <tr> <td style="text-align: left">r</td> <td style="text-align: left">text/x-rsrc</td> <td style="text-align: left"><a href="http://fileformats.archiveteam.org/wiki/R">R source code</a></td> </tr> <tr> <td style="text-align: left">m</td> <td style="text-align: left">text/x-objcsrc</td> <td style="text-align: left"><a href="http://fileformats.archiveteam.org/wiki/Objective-C">Objective-C source code</a></td> </tr> </tbody> </table> <p>Because of the small sample size (and also the fact that the ZIP files were taken from similar batches), these results cannot be taken as representative. Nevertheless, it does show that the identification of scientific text-based data formats such as <em>mol</em> or <em>cif</em> often isn’t very informative. Automatic identification of such formats is difficult anyway, because they typically don’t have unique patterns or header fields.</p> <h2 id="accessibility-in-reading-rooms">Accessibility in reading rooms</h2> <p>Finally we wanted to know to what extent the PCs in our reading rooms support our most common formats. To find out, we simply plugged a thumb drive with our test dataset into one of these PCs, and tried to open sample files for each extension in our “top 50” (and those found in the ZIP files as well). To make the results of this exercise easier to digest, we grouped all extensions into <a href="https://gist.github.com/bitsgalore/7a758505c0bbbae3db4e#file-formatcategories-md">12 format categories</a>. The table below shows the main results for each category:</p> <table> <thead> <tr> <th style="text-align: left">Category</th> <th style="text-align: left">Rendering software in reading rooms</th> <th style="text-align: left">Formats accessible in reading rooms?</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">Image formats</td> <td style="text-align: left">MS Paint, Windows Photoviewer</td> <td style="text-align: left">Yes</td> </tr> <tr> <td style="text-align: left">PDF</td> <td style="text-align: left">Adobe Acrobat</td> <td style="text-align: left">Yes</td> </tr> <tr> <td style="text-align: left">Web formats</td> <td style="text-align: left">Internet Explorer, Google Chrome</td> <td style="text-align: left">Yes</td> </tr> <tr> <td style="text-align: left">Office formats</td> <td style="text-align: left">Microsoft Office</td> <td style="text-align: left">Yes (support for old Office formats presently not clear)</td> </tr> <tr> <td style="text-align: left">Audio</td> <td style="text-align: left">Windows Media Player, VLC Media Player</td> <td style="text-align: left">No (hardware in reading rooms doesn’t support audio)</td> </tr> <tr> <td style="text-align: left">Video</td> <td style="text-align: left">Windows Media Player, VLC Media Player</td> <td style="text-align: left">Partially (hardware in reading rooms doesn’t support audio)</td> </tr> <tr> <td style="text-align: left">Metadata</td> <td style="text-align: left">Internet Explorer, Notepad, Wordpad</td> <td style="text-align: left">Yes</td> </tr> <tr> <td style="text-align: left">Executables, installers, system files</td> <td style="text-align: left">Not applicable</td> <td style="text-align: left">No</td> </tr> <tr> <td style="text-align: left">Containers</td> <td style="text-align: left">Windows Explorer, 7-Zip</td> <td style="text-align: left">Yes</td> </tr> <tr> <td style="text-align: left">Source code / scripts</td> <td style="text-align: left">Notepad, Wordpad</td> <td style="text-align: left">Partially: available software doesn’t support syntax highlighting</td> </tr> <tr> <td style="text-align: left">(Scientific) text-based data formats</td> <td style="text-align: left">Notepad, Wordpad</td> <td style="text-align: left">Partially: available software doesn’t support syntax highlighting; CSV files are not imported correctly by MS Excel</td> </tr> <tr> <td style="text-align: left">(Scientific) binary data formats</td> <td style="text-align: left">-</td> <td style="text-align: left">No</td> </tr> </tbody> </table> <p>The main conclusion is that most formats in our “Top 50” are sufficiently accessible. Nevertheless, there is some room for improvement:</p> <ul> <li> <p>The currently installed version of Microsoft Office (2010) does not support all previous versions of some of the Office formats. According to <a href="https://technet.microsoft.com/en-us/library/dd797428%28v=office.14%29.aspx">Microsoft’s documentation</a> there’s no support for Powerpoint 95 presentations, and the documentation is not clear on Word 95 and earlier either. From the current analysis we cannot establish whether we have these old formats in our collection, so this may need further work in the future.</p> </li> <li> <p>Comma-delimited text files are not read correctly by Excel. This is caused by region-specific settings of the PCs in the reading rooms, which cause Excel to expect a semicolon as a separator instead of a comma (the comma is used as a decimal separator in Dutch!). This could be improved by changing the configuration of the PCs (but a side-effect would be that semicolon-separated files would then go wrong instead!).</p> </li> <li> <p>The applications that are currently available for the “plain” text formats are not that great for scripts, large data files and files that have non-Windows <a href="http://en.wikipedia.org/wiki/Newline">line endings</a>. This could be easily solved by installing a more sophisticated text editor such as <a href="http://notepad-plus-plus.org/">Notepad++</a>.</p> </li> <li> <p>As part of the scientific binary data category, we came across some 1800 <a href="http://fileformats.archiveteam.org/wiki/Matlab_figure">Matlab Figure</a> files. This is a proprietary format that requires the <a href="http://fileformats.archiveteam.org/wiki/Matlab">Matlab</a> software, which is not available in our reading rooms. So, essentially these files are not accessible to our users. Whether we will take any action on this is a different matter, since Matlab licences are expensive and the number of files is relatively small anyway.</p> </li> </ul> <h2 id="links">Links</h2> <ul> <li> <p><a href="https://gist.github.com/bitsgalore/21028de28b7f05066585#file-extensionskbdm-md">50 most prevalent formats in KB e-Depot by file extension</a></p> </li> <li> <p><a href="https://gist.github.com/bitsgalore/4326300f185eec3d6d48#file-edepotfextentions_v3-md">All file extensions in e-Depot and corresponding file counts</a></p> </li> </ul> <h2 id="acknowledgements">Acknowledgements</h2> <p>Victor van der Wolf prepared the file extension counts; Danny Stephan prepared the database queries for the sample dataset. Barbara Sierman came up with the initial idea of a “file formats top 50”.</p> <hr /> <p>Originally published at the <a href="http://blog.kbresearch.nl/2015/04/29/top-50-file-formats-in-the-kb-e-depot/">KB Research blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>Most of these are tiny images that are part of XML representations of scientific papers (mostly mathematical equations). <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>The dataset is not representative of the collection as a whole because of its limited size, and the sub-optimal sampling procedure that was used. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2015/04/29/top-50-file-formats-in-the-kb-e-depot https://bitsgalore.org/2015/04/29/top-50-file-formats-in-the-kb-e-depot 2015-04-29T00:00:00+02:00 <p>Back in 2012 the KB conducted a first <a href="https://zenodo.org/record/839711">investigation</a> of the suitability of the <em>EPUB</em> format for long-term preservation. The KB will soon start receiving publications in this format, and in anticipation of this, our Collection Care department has formulated a policy on the minimum requirements an <em>EPUB</em> must meet to ensure long-term accessibility. The policy largely follows the recommendations from the 2012 report. This blog explores to what extent it is possible to automatically assess the <em>EPUB</em>s that we receive against our policy using a combination of the <a href="https://github.com/idpf/epubcheck"><em>Epubcheck</em></a> tool and <a href="http://www.schematron.com/"><em>Schematron</em></a> rules.</p> <!-- more --> <h2 id="kb-epub-policy">KB <em>EPUB</em> policy</h2> <p>The KB’s policy on <em>EPUB</em> is made up of the following objectives:</p> <ol> <li> <p>File must be valid <em>EPUB</em> (either version 2 or 3)</p> <p><em>Rationale</em>: this minimises the risk of interoperability problems.</p> </li> <li> <p>File may not contain DRM or encryption</p> <p><em>Rationale</em>: this minimises the risk that files become inaccessible. An edge case here is <a href="http://www.idpf.org/epub/30/spec/epub30-ocf.html#font-obfuscation">font obfuscation</a>, which mangles some leading bytes in embedded fonts. This technology is merely meant as a stumbling block to discourage third parties from re-using embedded fonts, and it doesn pose a serious threat to long-term accessibility.</p> </li> <li> <p>File may not contain foreign resources</p> <p><em>Rationale</em>: the <a href="http://www.idpf.org/epub/301/spec/epub-publications.html#sec-core-media-types">Core Media Types</a> define a set of file formats that must be supported by all conforming <em>EPUB</em> readers. <a href="http://www.idpf.org/epub/301/spec/epub-publications.html#gloss-publication-resource-foreign">Foreign resources</a> are resources that are not part of this set, and the KB’s policy is to not accept them. This requirement minimises the risk of accepting files that contain content that may not be rendered correctly by some readers.</p> </li> <li> <p>File may not contain <em>DTBook</em> content</p> <p><em>Rationale</em>: <em>EPUB</em> 2 offered the option to use the <a href="http://www.niso.org/workrooms/daisy/Z39-86-2005.html"><em>DTBook</em></a> (<em>DAISY</em> Digital Talking Book) format as an alternative to <em>XHTML</em> 1.1. Support for <em>DTBook</em> was <a href="http://www.idpf.org/epub/30/spec/epub30-changes.html#sec-removals-dtbook">dropped in <em>EPUB</em> 3</a>. Support is already limited with current <em>EPUB</em> reading software: both the popular <a href="http://calibre-ebook.com/"><em>Calibre</em></a> and <a href="http://readium.org/"><em>Readium</em></a> viewers are unable to process <em>EPUBS</em> with <em>DTBook</em> content (although my <a href="https://en.wikipedia.org/wiki/Sony_Reader#2012_Model_.28Discontinued_late_2013.29">Sony Reader device</a> handles them without problems). This does not bode well for the future.</p> </li> </ol> <h2 id="automated-conformance-checking">Automated conformance checking</h2> <p>To check if an <em>EPUB</em> conforms to the above policy, we need to:</p> <ol> <li>test for validity against the format’s standard;</li> <li>extract technical information that tells us something about DRM and file resources inside the <em>EPUB</em>;</li> <li>assess the results of steps 1 and 2 against our policy.</li> </ol> <p>The <a href="https://github.com/idpf/epubcheck"><em>Epubcheck</em></a> validator is the obvious candidate for steps 1 and 2. Since <em>Epubcheck</em> is capable of reporting its results in XML format, we can use <a href="http://www.schematron.com/"><em>Schematron</em></a> rules for the final assessment step. The general approach is similar to earlier work on the <a href="/2012/09/04/automated-assessment-jp2-against-technical-profile/">JP2</a> and <a href="/2014/01/27/identification-pdf-preservation-risks-analysis-govdocs-selected-corpus/">PDF</a> formats, as well as the British Library’s <a href="https://github.com/openpreserve/flint">Flint</a> tool.</p> <h2 id="test-data">Test data</h2> <p>For testing, we first need a corpus of files that are known violate one or more objectives of our policy. As this turned out to be more difficult than expected, I created a small <a href="https://github.com/KBNLresearch/epubPolicyTests">set of test files</a>. Some of the files in this dataset were created from scratch; others were taken directly or adapted from existing openly licensed datasets. The following table lists the main characteristics of the files in the dataset<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>:</p> <table> <thead> <tr> <th style="text-align: left">Test</th> <th style="text-align: left">Epub version</th> <th style="text-align: left">Description</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/build/epub20_minimal.epub?raw=true">Minimal</a></td> <td style="text-align: left">2</td> <td style="text-align: left">Basic file with one text resource and one image</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/build/epub20_minimal_encryption.epub?raw=true">Encryption</a></td> <td style="text-align: left">2</td> <td style="text-align: left">Fake encrypted file that includes <em>encryption.xml</em> resource in <code class="language-plaintext highlighter-rouge">META-INF</code>, indicating that main text resource is encrypted<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup></td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/build/epub30_font_obfuscation.epub?raw=true">Font obfuscation</a></td> <td style="text-align: left">3</td> <td style="text-align: left">Includes fonts that are obfuscated (which results in <em>hasEncryption</em> in epubcheck). Taken from <a href="https://code.google.com/p/epub-samples/">EPUB 3 Sample Documents</a> (<a href="https://code.google.com/p/epub-samples/downloads/detail?name=wasteland-otf-obf-20120118.epub&amp;can=2&amp;q="><em>wasteland with OTF fonts, obfuscated</em></a>).</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/build/epub20_foreign_resource_no_fallback.epub?raw=true">Foreign resource without fallback</a></td> <td style="text-align: left">2</td> <td style="text-align: left">Includes JP2 image, which is a format that is not on the list of Core Media Types</td> </tr> <tr> <td style="text-align: left"><a href="build/epub20_foreign_resource_with_fallback.epub?raw=true">Foreign resource with fallback 1</a></td> <td style="text-align: left">2</td> <td style="text-align: left">Includes JP2 image, which is a format that is not on the list of Core Media Types; fallback defined in manifest, identifier in content document</td> </tr> <tr> <td style="text-align: left"><a href="build/epub20_foreign_resource_with_fallback_noID.epub?raw=true">Foreign resource with fallback 2</a></td> <td style="text-align: left">2</td> <td style="text-align: left">Includes JP2 image, which is a format that is not on the list of Core Media Types; fallback defined in manifest, no identifier in content document</td> </tr> <tr> <td style="text-align: left"><a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/build/epub20_dtbook.epub?raw=true">DTBook</a></td> <td style="text-align: left">2</td> <td style="text-align: left">Includes Digital Talking Book content. Adapted from <a href="https://code.google.com/p/threepress/source/browse/branches/bookworm-caching/library/test-data/data/hauy.epub?r=583">threepress</a>, published under <a href="http://opensource.org/licenses/BSD-3-Clause">BSD 3</a> license.</td> </tr> </tbody> </table> <p>Apart from the above files, the dataset also includes:</p> <ul> <li>the <a href="https://github.com/KBNLresearch/epubPolicyTests/tree/master/content">full source</a> of each test file (as a directory structure);</li> <li>a <a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/build.sh">bash script</a> that automatically builds (zips) all directories to <em>EPUB</em>s;</li> <li><a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/analyse.sh">another bash script</a> that analyses all <em>EPUB</em>s with version 3 and 4 of <em>Epubcheck</em>;</li> <li>the <a href="https://github.com/KBNLresearch/epubPolicyTests/tree/master/epubcheckout">full <em>Epubcheck</em> output</a> of each file.</li> </ul> <p>All files are openly licensed, and by adapting the existing tests it is pretty straightforward to add new ones.</p> <h2 id="analysis-with-epubcheck">Analysis with Epubcheck</h2> <p>The first question that we need to answer here is whether <em>Epubcheck</em>’s output is sufficiently detailed for our needs. So, as a first step I analysed all files in the dataset with <a href="https://github.com/idpf/epubcheck"><em>Epubcheck</em></a>. I did this using both <a href="https://github.com/IDPF/epubcheck/releases/tag/v3.0.1"><em>Epubcheck</em> 3.0.1</a> (the current stable version) and <a href="https://github.com/IDPF/epubcheck/releases/tag/v4.0.0-alpha11">the alpha 11 release of <em>Epubcheck</em> 4.0.0</a>. The full output can be found <a href="https://github.com/KBNLresearch/epubPolicyTests/tree/master/epubcheckout">here</a>. In the following sections I will address each of the objectives of the KB policy.</p> <h2 id="encryption-objective">Encryption objective</h2> <p>For the ‘fake’ encrypted file <em>Epubcheck</em>’s output contains a <em>hasEncryption</em> property. Moreover, the <em>messages</em> element in the output contains an error message that refers to the encrypted resource. In <em>Epubcheck</em> 3 this is:</p> <pre><code class="language-data">ERROR: : OPS/XHTML file OEBPS/Text/pdfMigration.html cannot be decrypted </code></pre> <p>A double-check with a ‘real’ encrypted <em>EPUB</em> (which is proprietary and could not be included in the dataset) confirmed that each encrypted resource produces an error message of the general form:</p> <pre><code class="language-data">ERROR: : $fileType file $fileName cannot be decrypted </code></pre> <p>Here, <code class="language-plaintext highlighter-rouge">$fileType</code> and <code class="language-plaintext highlighter-rouge">$fileName</code> refer to the file type and name of the affected resource. The ‘fake’ encrypted file also resulted in some additional error messages about undefined fragment identifiers, but these look like secondary errors that result from <em>Epubcheck</em> ‘s inability to decrypt the encrypted resource.</p> <p>The behaviour of <em>Epubcheck</em> 4 is similar, although the error message is slightly different:</p> <pre><code class="language-data">RSC-004, ERROR, [File 'OEBPS/Text/pdfMigration.html' could not be decrypted.],epub20_minimal_encryption.epub </code></pre> <p>The file with the obfuscated fonts also results in a <em>hasEncryption</em> entry in <em>Epubcheck</em>’s output. <em>Epubcheck</em> (both versions 3 and 4) doesn’t provide any direct clue that the encryption in this file is limited to some obfuscated fonts. For our policy-based assessment we can therefore ignore the <em>hasEncryption</em> entry, and simply check for the presence of “cannot be decrypted” error messages (see above).</p> <h2 id="dtbook-objective">DTBook objective</h2> <p><em>Epubcheck</em>’s output does not give any explicit clue to the presence of <em>DTBook</em> content. However, <em>Epubcheck</em> 3 does report a read error on the corresponding file resource:</p> <pre><code class="language-data">ERROR: : I/O error reading OEBPS/hauy-2005-1.xml: Stream closed </code></pre> <p><em>Epubcheck</em> 4 does not report this error. A check of the <em>DTBook</em> resource confirmed that it is valid against <a href="http://www.daisy.org/z3986/2005/dtbook-2005-2.dtd">version 2 of the <em>DTBook</em> Document Type Definition</a> (I checked this using both <a href="http://jhove.sourceforge.net/">JHOVE</a> and an <a href="http://www.validome.org/xml/validate/">online XML validator</a>). This suggests that <em>Epubcheck</em> 3 doesn’t properly recognise (cannot parse?) <em>DTBook</em> content, and incorrectly flags <em>EPUB</em>s that hold this as “Not well-formed”. The behaviour of <em>Epubcheck</em> 4 is correct (see also <a href="https://github.com/IDPF/epubcheck/issues/518">this issue report</a>).</p> <h2 id="foreign-resources-objective">Foreign resources objective</h2> <p>The test dataset contains 3 files with foreign resources (resources that are not on the list of Core Media Types). In the first one I simply replaced a <em>PNG</em> image by a <em>JP2</em> (and updated the manifest and the reference in the text accordingly). This results in the following validation error (<em>Epubcheck</em> 3):</p> <pre><code class="language-data">ERROR: /OEBPS/Text/pdfMigration.html(20): non-standard image resource 'OEBPS/Images/pdfVenn.jp2' of type 'image/jp2' </code></pre> <p>And in <em>Epubcheck</em> 4;</p> <pre><code class="language-data">MED-003, ERROR, [Non-standard image resource of type image/jp2 found.], OEBPS/Text/pdfMigration.html (20-63) </code></pre> <p>This error also causes the validation to fail. The <em>EPUB</em> specification allows the use of foreign resources, but <a href="http://www.idpf.org/epub/20/spec/OPF_2.0.1_draft.htm#Section2.3.1.1">only if they have a Core Media fallback</a>. I created two additional test files that use the original <em>PNG</em> image as a fallback<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>; I then updated the manifest of these files accordingly. <em>Epubcheck</em> validates both files as “Well-formed”, but gives no information whatsoever on the presence of foreign resources. Somewhat alarmingly, both <em>Calibre</em> and <em>Readium</em> failed to read <a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/build/epub20_foreign_resource_with_fallback.epub?raw=true">either</a> of these <a href="https://github.com/KBNLresearch/epubPolicyTests/blob/master/build/epub20_foreign_resource_with_fallback_noID.epub?raw=true">files</a> correctly: the (fall back) image was not shown in both cases. As it turns out, <a href="https://github.com/IDPF/epubcheck/issues/511">very few <em>EPUB</em> readers support manifest fallbacks</a>, even though this feature has been part of the <em>EPUB</em> specification for a long time (at least since <em>EPUB</em> 2).</p> <h2 id="translating-the-policy-to-schematron-rules">Translating the policy to Schematron rules</h2> <p>If <em>Epubcheck</em> were able to address all apects of the KB’s policy, it would be possible to translate each of its objectives into a <em>Schematron</em> rule. As <em>Epubcheck</em> doesn’t yet provide the required information on foreign resources and <em>DTBook</em> content, for now we can only do this for the validity and encryption objectives. The <em>Schematron</em> rule for validity is:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;s:pattern</span> <span class="na">name=</span><span class="s">"wellFormed"</span><span class="nt">&gt;</span> <span class="nt">&lt;s:rule</span> <span class="na">context=</span><span class="s">"/jh:jhove/jh:repInfo"</span><span class="nt">&gt;</span> <span class="nt">&lt;s:assert</span> <span class="na">test=</span><span class="s">"(jh:status = 'Well-formed')"</span><span class="nt">&gt;</span>Not well-formed epub<span class="nt">&lt;/s:assert&gt;</span> <span class="nt">&lt;/s:rule&gt;</span> <span class="nt">&lt;/s:pattern&gt;</span> </code></pre></div></div> <p>For the encryption objective we have this:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">&lt;!-- This rule rules out encrypted content, but permits font obfuscation--&gt;</span> <span class="nt">&lt;s:pattern</span> <span class="na">name=</span><span class="s">"encryptedResources"</span><span class="nt">&gt;</span> <span class="nt">&lt;s:rule</span> <span class="na">context=</span><span class="s">"/jh:jhove/jh:repInfo/jh:messages"</span><span class="nt">&gt;</span> <span class="nt">&lt;s:assert</span> <span class="na">test=</span><span class="s">"count(jh:message[contains(.,'cannot be decrypted')]) = 0"</span><span class="nt">&gt;</span>Contains encrypted resources<span class="nt">&lt;/s:assert&gt;</span> <span class="nt">&lt;/s:rule&gt;</span> <span class="nt">&lt;/s:pattern&gt;</span> </code></pre></div></div> <p>Alternatively, we could have created a rule that uses the <em>hasEncryption</em> property here, but that would cause any files with obfuscated fonts to fail the assessment. The corresponding schema (adapted from the BL’s <a href="https://github.com/openpreserve/flint"><em>Flint</em></a> tool) can be found <a href="https://github.com/KBNLresearch/epubPolicyValidate/blob/master/schemas/kbPolicy.sch">here</a>. It is designed to work with <em>Epubcheck</em> 3 only <sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>.</p> <h2 id="demo">Demo</h2> <p>I created a simple <a href="https://github.com/KBNLresearch/epubPolicyValidate"><em>EPUB</em> policy-based validation demo</a>. It is a shell script that validates all <em>EPUB</em> files in a user-defined directory with <em>Epubcheck</em>, and subsequently assesses <em>Epubcheck</em>’s output against a user-defined schema. Note that the purpose of the script is just to demonstrate the general procedure; it is not recommended for operational use.</p> <h2 id="possible-epubcheck-enhancements">Possible Epubcheck enhancements</h2> <p>The above tests demonstrate that currently <em>Epubcheck</em> is able to cover two aspects of the KB’s policy on <em>EPUB</em>: validity and encryption. However, its output doesn’t provide the information we need on the presence of foreign resources and <em>DTBook</em> content. It would be useful if <em>Epubcheck</em> could be extended with an option that reports all resources in an <em>EPUB</em> with their corresponding media types. This information can be extracted from the <em>manifest</em> element of an <em>EPUB</em>’s <a href="http://www.idpf.org/epub/301/spec/epub-publications.html#sec-package-documents">Package Document</a>. For a file with both <em>DTBook</em> content and a foreign resource (a <em>JP2</em> file) this looks something like this:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;manifest&gt;</span> <span class="nt">&lt;item</span> <span class="na">href=</span><span class="s">"toc.ncx"</span> <span class="na">id=</span><span class="s">"ncx"</span> <span class="na">media-type=</span><span class="s">"application/x-dtbncx+xml"</span> <span class="nt">/&gt;</span> <span class="nt">&lt;item</span> <span class="na">href=</span><span class="s">"Images/pdfVenn.jp2"</span> <span class="na">id=</span><span class="s">"pdfVennJP2"</span> <span class="na">media-type=</span><span class="s">"image/jp2"</span> <span class="na">fallback=</span><span class="s">"pdfVennPNG"</span> <span class="nt">/&gt;</span> <span class="nt">&lt;item</span> <span class="na">href=</span><span class="s">"Images/pdfVenn.png"</span> <span class="na">id=</span><span class="s">"pdfVennPNG"</span> <span class="na">media-type=</span><span class="s">"image/png"</span> <span class="nt">/&gt;</span> <span class="nt">&lt;item</span> <span class="na">href=</span><span class="s">"hauy-2005-1.xml"</span> <span class="na">id=</span><span class="s">"opf3"</span> <span class="na">media-type=</span><span class="s">"application/x-dtbook+xml"</span> <span class="nt">/&gt;</span> <span class="nt">&lt;/manifest&gt;</span> </code></pre></div></div> <p>Some simple <em>Schematron</em> rules on the <em>media-type</em> attribute would make it possible to filter this for the presence of <em>DTBook</em> content (where <em>media-type</em> is <code class="language-plaintext highlighter-rouge">application/x-dtbook+xml</code>) or foreign resources (which have a <em>media-type</em> value that is not on the Core Media Types list). Another solution would be to add properties like <code class="language-plaintext highlighter-rouge">hasDTBook</code> and <code class="language-plaintext highlighter-rouge">hasForeignResources</code> to <em>Epubcheck</em>’s output. This solution is less generic, but possibly more user-friendly.</p> <p>Finally, the presence of <em>DTBook</em> resources <sup id="fnref:5"><a href="#fn:5" class="footnote" rel="footnote" role="doc-noteref">5</a></sup> incorrectly causes the validation to fail in <em>Epubcheck</em> 3; this has been fixed in <em>Epubcheck</em> 4.</p> <h2 id="links">Links</h2> <ul> <li><a href="https://github.com/idpf/epubcheck">Epubcheck</a></li> <li><a href="https://github.com/KBNLresearch/epubPolicyTests">EPUB KB policy testing dataset</a>, includes full source, build and analysis scripts.</li> <li><a href="https://github.com/KBNLresearch/epubPolicyValidate">Policy-based validation demo based on Epubcheck</a></li> <li><a href="http://epubtest.org/compare/">EPUB 3 Support Grid - comparison of support of EPUB 3.0 features by different reading systems</a></li> </ul> <hr /> <p>Originally published at the <a href="http://blog.kbresearch.nl/2015/03/13/policy-based-assessment-of-epub-with-epubcheck/">KB Research blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>These files are all released under the Creative Commons 3.0 BY-SA license, unless stated otherwise. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>The ‘encryption’ in this file is actually fake: I merely replace the original text resource with a <a href="http://linux.die.net/man/1/base64">base64</a> encoded representation of that file. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>They only differ in the way the JP2 image is referenced in the text, as the <em>EPUB</em> specification is not completely clear on this. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>Doesn’t yet work with <em>Epubcheck</em> 4 because it uses slightly different output messages (could be easily adapted). <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:5"> <p>These are, by the way, pretty rare. <a href="#fnref:5" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2015/03/13/policy-based-assessment-of-epub-with-epubcheck https://bitsgalore.org/2015/03/13/policy-based-assessment-of-epub-with-epubcheck 2015-03-13T00:00:00+01:00 <p>Shortly before Christmas, Dutch daily newspaper <em>Trouw</em> <a href="http://www.nrc.nl/nieuws/2014/12/20/trouw-trekt-126-artikelen-van-perdiep-ramesar-in/">removed 126 articles</a> from its website. These articles were all authored by Perdiep Ramesar, a former journalist of the newspaper. Ramesar had been fired by <em>Trouw</em> in November, after it turned out that many of the sources that are cited in his articles were <a href="http://static1.trouw.nl/static/asset/2014/Onderzoeksrapport_bronnengebruik_Trouw_19122014_7707.pdf">fabricated</a>. The most notorious example was a series of pieces about the so-called “Sharia Triangle”, a neighbourhood in the city of The Hague, which Ramesar claimed was being ruled by Sharia law. As it turned out, this story was largely based on fabricated sources. Nevertheless, it was taken at face value by most major Dutch news outlets at the time, and even prompted a <a href="http://www.tweedekamer.nl/kamerstukken/detail?id=2013D34540&amp;did=2013D34540">parliamentary debate</a>.</p> <p><em>Trouw</em>’s decision to remove the 126 articles overnight was met with considerable criticism. For example, historian Jan Dirk Snel <a href="http://jandirksnel.wordpress.com/2014/12/24/geschiedvervalsing-het-echte-schandaal-bij-trouw-is-nu-pas-begonnen/">noted</a> that the removal of these articles makes it impossible to check <em>what</em> was wrong with them in the first place. Various other critics accused <em>Trouw</em> of trying to <a href="http://www.journalismlab.nl/2014/12/perdiep-gewist-gaan-trouw-en-ad-gaan-voor-geschiedvervalsing/">rewrite history</a>.</p> <!-- more --> <h2 id="internet-archive-to-the-rescue">Internet Archive to the rescue?</h2> <p>A quick check on a handful of Ramesar’s articles revealed that quite a few were still accessible from the <a href="http://archive.org/web/">Internet Archive’s Wayback Machine</a>. This got me curious how many out of the 126 deleted articles would still be available there. Answering this question isn’t completely straightforward, because the Wayback Machine isn’t easily searchable. In order to locate any of the deleted articles, one first needs to know its original URL (i.e. the one at <em>Trouw</em>’s website). A <a href="http://static3.trouw.nl/static/asset/2014/Artikelen_met_niet_verifieerbare_bronnen_Ramesar_2007_2014_7708.pdf">list of all deleted articles</a> does exist, but this only provides each article’s <em>title</em>, without listing the full URL.</p> <p>However, by entering each title into a search engine (I used a combination of <em>Google</em> and <em>DuckDuckGo</em><sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>), I was able to recover the original URL of every article in the list. In many cases the URLs were still present in the cache of the search engine. In other cases URLs could be recovered from linking pages on the <em>Trouw</em> website. I then wrote a simple <a href="https://github.com/bitsgalore/trouwRamesarWayback/blob/master/scripts/checkLinksInWayback.py">script</a> to check the availability of each URL in Internet Archive’s Wayback Machine. The script is just a wrapper around Wayback’s <a href="https://archive.org/help/wayback_api.php">Availability JSON API</a>, which is insanely handy (and really easy to use as well!). This yielded a <a href="/images/2015/01/ramesarTrouwURLSWayback.csv">list</a> with -for each article- its status in Wayback (i.e. has it been archived), and, if so, the URL to the most recent capture.</p> <h2 id="result">Result</h2> <p>The results of the above exercise are summarised in <a href="/images/2015/01/tabelRamesar.html">this table</a>. As it turned out, 53 out of the 126 deleted articles are still accessible from the Internet Archive. These are mostly pieces that were written from 2010 onward, and include the notorious “Sharia Triangle” ones. From the time period 2007-2009 very few articles could be found.</p> <h2 id="possibly-more">Possibly more?</h2> <p>It may be possible that more removed articles are hidden in the Internet Archive. This is because of the way the <em>Trouw</em> website handles news items. If I understand things correctly, articles in <em>Trouw</em> are often first published under a <em>news</em> URL; subsequently it is moved to the <em>archive</em> section of the website, where it is published under a different URL. By way of illustration, a <em>DuckDuckGo</em> search of the article <em>Ik kan mezelf niet veranderen in een witte man</em> yielded the following URL:</p> <p><a href="http://www.trouw.nl/tr/nl/5009/Archief/archief/article/detail/3287592/2012/07/17/Ik-kan-mezelf-niet-veranderen-in-een-witte-man.dhtml">http://www.trouw.nl/tr/nl/5009/Archief/archief/article/detail/3287592/2012/07/17/Ik-kan-mezelf-niet-veranderen-in-een-witte-man.dhtml</a></p> <p>This <em>archive</em> link (recognisable from the word <em>archief</em> in the URL) cannot be found anywhere in Internet Archive. By chance I encountered a different link to the same article on the website of historian Jan Dirk Snel:</p> <p><a href="http://www.trouw.nl/tr/nl/4504/Economie/article/detail/3287689/2012/07/17/Ik-kan-mezelf-niet-veranderen-in-een-witte-man.dhtml">http://www.trouw.nl/tr/nl/4504/Economie/article/detail/3287689/2012/07/17/Ik-kan-mezelf-niet-veranderen-in-een-witte-man.dhtml</a></p> <p>This is the <em>news</em> link under which the article was first published, and a snapshot of it exists in the Internet Archive:</p> <p><a href="http://web.archive.org/web/20141224185904/http://www.trouw.nl/tr/nl/4504/Economie/article/detail/3287689/2012/07/17/Ik-kan-mezelf-niet-veranderen-in-een-witte-man.dhtml">http://web.archive.org/web/20141224185904/http://www.trouw.nl/tr/nl/4504/Economie/article/detail/3287689/2012/07/17/Ik-kan-mezelf-niet-veranderen-in-een-witte-man.dhtml</a></p> <p>Likewise, I expect that some articles may have slipped through the net in a similar way. Nevertheless, I think the above results are pretty good as they are!</p> <p><strong><a href="/images/2015/01/tabelRamesar.html">Click here for the full list of removed articles, includes both original URLs and URLs in Internet Archive (if available)</a></strong></p> <h2 id="links">Links</h2> <ul> <li><a href="/images/2015/01/ramesarTrouwURLSWayback.csv">Data as comma-separated text file (UTF-8)</a></li> <li><a href="https://github.com/bitsgalore/trouwRamesarWayback">Github repo with scripts and raw data files</a></li> <li><a href="https://github.com/bitsgalore/trouwRamesarWayback/archive/master.zip">Github repo (as single ZIP file)</a></li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>In order to get unscrambled links from Google, I used the following FireFox add-on: <a href="https://palant.de/2011/11/28/google-yandex-search-link-fix">https://palant.de/2011/11/28/google-yandex-search-link-fix</a> <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2015/01/06/Dutch-newspaper-wipes-articles-Internet-Archive-rescue https://bitsgalore.org/2015/01/06/Dutch-newspaper-wipes-articles-Internet-Archive-rescue 2015-01-06T00:00:00+01:00 <p>Eerder deze week <a href="http://www.nrc.nl/nieuws/2014/12/20/trouw-trekt-126-artikelen-van-perdiep-ramesar-in/">verwijderde dagblad <em>Trouw</em> 126</a> artikelen van haar website die geschreven waren door ontslagen journalist Perdiep Ramesar. Aanleiding hiervoor was het <a href="http://static1.trouw.nl/static/asset/2014/Onderzoeksrapport_bronnengebruik_Trouw_19122014_7707.pdf">onderzoek</a> naar door Ramesar opgevoerde “niet traceerbare” bronnen. De beslissing van <em>Trouw</em> om de onbetrouwbare artikelen van de site af te halen stuitte op nogal wat kritiek. Sommigen noemden het <a href="http://www.journalismlab.nl/2014/12/perdiep-gewist-gaan-trouw-en-ad-gaan-voor-geschiedvervalsing/">geschiedvervalsing</a>. Historicus Jan Dirk Snel <a href="http://jandirksnel.wordpress.com/2014/12/24/geschiedvervalsing-het-echte-schandaal-bij-trouw-is-nu-pas-begonnen/">merkte terecht op</a> dat nu de stukken zijn verwijderd, niemand meer kan controleren wat er eventueel wel of niet aan deugt.</p> <!-- more --> <h2 id="vindbaarheid-in-internet-archive">Vindbaarheid in Internet Archive</h2> <p>Uit nieuwsgierigheid heb ik van een aantal van de verwijderde artikelen gekeken of ze nog te vinden waren in het <a href="https://archive.org/">Internet Archive</a>. Voor sommige stukken bleek dit inderdaad het geval. Vervolgens werd ik benieuwd hoeveel van de 126 verwijderde artikelen nog vindbaar zouden zijn. Het probleem hierbij is alleen dat het Internet Archive niet echt makkelijk doorzoekbaar is. Om een artikel op het spoor te komen, heb je eigenlijk de originele URL (dus op de <em>Trouw</em> website) nodig. <em>Trouw</em> heeft wel een <a href="http://static3.trouw.nl/static/asset/2014/Artikelen_met_niet_verifieerbare_bronnen_Ramesar_2007_2014_7708.pdf">lijst met de verwijderde artikelen</a> gepubliceerd, maar hierin wordt van elk artikel alleen de <em>titel</em> vermeld, en niet de volledige link.</p> <p>Omdat de artikelen nog maar recent zijn verwijderd, zitten de URLs nog wel in de cache van de meeste zoekmachines. Door de titels uit de lijst met verwijderde artikelen in te voeren in <em>Google</em> en <em>DuckDuckGo</em>, lukte het me om van alle 126 artikelen de originele URLs te achterhalen<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. Met behulp van een zelfgeschreven <a href="https://github.com/bitsgalore/trouwRamesarWayback/blob/master/scripts/checkLinksInWayback.py">scriptje</a> heb ik vervolgens elke URL opgezocht in het Internet Archive. Dit leverde me een <a href="/images/2014/12/ramesarTrouwURLSWayback.csv">lijst</a> op met -voor elk artikel- de status in Internet Archive (is het gearchiveerd of niet), en, indien aanwezig, de meest recent gearchiveerde versie.</p> <h2 id="resultaat">Resultaat</h2> <p>Het resultaat van de hierboven beschreven analyse heb ik samengevat in <a href="/images/2014/12/tabelRamesar.html">deze tabel</a>. Van de 126 verwijderde artikelen zijn er 53 nog opvraagbaar in het Internet Archive. Het gaat hierbij vooral om artikelen uit 2010 en later; uit de periode 2007-2009 is nog maar weinig te vinden.</p> <h2 id="nog-meer-te-vinden">Nog meer te vinden?</h2> <p>Overigens verwacht ik dat met goed zoeken nog wel meer te vinden valt: voor zover ik het het goed begrijp, wordt een artikel op de <em>Trouw</em> website eerst onder een <em>nieuwslink</em> gepubliceerd; vervolgens verhuist het naar het archief, waarna het onder een <em>archieflink</em> beschikbaar is. Een voorbeeld is het artikel <em>Ik kan mezelf niet veranderen in een witte man</em>. Een zoekactie met <em>DuckDuckGo</em> leverde me hiervan de volgende link op:</p> <p><a href="http://www.trouw.nl/tr/nl/5009/Archief/archief/article/detail/3287592/2012/07/17/Ik-kan-mezelf-niet-veranderen-in-een-witte-man.dhtml">http://www.trouw.nl/tr/nl/5009/Archief/archief/article/detail/3287592/2012/07/17/Ik-kan-mezelf-niet-veranderen-in-een-witte-man.dhtml</a></p> <p>Deze <em>archieflink</em> is niet te vinden in Internet Archive. Op de site van Jan Dirk Snel kwam ik van hetzelfde artikel de <em>nieuwslink</em> tegen:</p> <p><a href="http://www.trouw.nl/tr/nl/4504/Economie/article/detail/3287689/2012/07/17/Ik-kan-mezelf-niet-veranderen-in-een-witte-man.dhtml">http://www.trouw.nl/tr/nl/4504/Economie/article/detail/3287689/2012/07/17/Ik-kan-mezelf-niet-veranderen-in-een-witte-man.dhtml</a></p> <p>En die zit wel in Internet Archive:</p> <p><a href="http://web.archive.org/web/20141224185904/http://www.trouw.nl/tr/nl/4504/Economie/article/detail/3287689/2012/07/17/Ik-kan-mezelf-niet-veranderen-in-een-witte-man.dhtml">http://web.archive.org/web/20141224185904/http://www.trouw.nl/tr/nl/4504/Economie/article/detail/3287689/2012/07/17/Ik-kan-mezelf-niet-veranderen-in-een-witte-man.dhtml</a></p> <p>Er zullen dus nog wel meer artikelen op vergelijkbare wijze door het net zijn geglipt. Als lezers nog aanvullingen of correcties hebben dan hoor ik dat natuurlijk graag!</p> <h2 id="ad">AD</h2> <p>Het <em>AD</em> is nog veel verder gegaan dan <em>Trouw</em>, en heeft gelijk <em>alle</em> artikelen waarvan Ramesar auteur is verwijderd. Van veel van deze stukken <a href="https://www.google.nl/?q=%22Perdiep+Ramesar%22+site:ad.nl#q=%22Perdiep+Ramesar%22+site:ad.nl">zijn de originele URLs nog te achterhalen via de <em>Google</em> cache</a>. Maar niet lang meer! Omdat het hier om honderden artikelen gaat, is het geen doen om de URLs allemaal handmatig op te vragen. <a href="https://developers.google.com/custom-search/json-api/v1/overview"><em>Google</em> biedt een <em>Search API</em></a> aan, en daarmee zou het mogelijk moeten zijn om dit grotendeels te automatiseren. Die URLs kun je vervolgens weer proberen terug te zoeken in Internet Archive, net zoals ik dat voor de <em>Trouw</em> artikelen heb gedaan. Ik ga daar zelf nu geen tijd in steken, maar misschien heeft iemand anders zin om hiermee aan de slag te gaan. Enige haast is hierbij wel geboden, want binnen een paar weken zullen de links uit <em>Google</em>’s cache verdwenen zijn, en het is maar de vraag of je het dan nog ooit terug kunt vinden.</p> <p><strong><a href="/images/2014/12/tabelRamesar.html">Klik hier voor de volledige lijst artikelen, inclusief originele URLs en URLs in Internet Archive (voor zover beschikbaar)</a></strong></p> <h2 id="links">Links</h2> <ul> <li><a href="/images/2014/12/ramesarTrouwURLSWayback.csv">Data als kommagescheiden tekstbestand (UTF-8)</a></li> <li><a href="https://github.com/bitsgalore/trouwRamesarWayback">Github repository met gebruikte scripts en databestanden</a></li> <li><a href="https://github.com/bitsgalore/trouwRamesarWayback/archive/master.zip">ZIP bestand met alle scripts en databestanden</a></li> </ul> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>Google heeft hierbij de vervelende gewoonte om niet de directe links naar de zoekresultaten te geven. Om dit te omzeilen heb ik de volgende FireFox add-on gebruikt: <a href="https://palant.de/2011/11/28/google-yandex-search-link-fix">https://palant.de/2011/11/28/google-yandex-search-link-fix</a> <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2014/12/28/Perdiep-Ramesar-Internet-Archive https://bitsgalore.org/2014/12/28/Perdiep-Ramesar-Internet-Archive 2014-12-28T00:00:00+01:00 <p>Back in 2006, Dutch weblog <a href="http://sargasso.nl/">Sargasso</a> started following the activity of about 260 Dutch blogs that were active at the time, mainly by looking at the frequency of new postings. <!-- more --> Earlier this week Sargasso <a href="http://sargasso.nl/battle-blogs-war-lost/">published an update of the the status of these blogs in 2014</a>. They summarised the results of their analysis with this fascinating visualisation<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>:</p> <iframe width="415" scrolling="no" height="620" frameborder="0" src="https://sargasso.nl/wp-content/uploads/2014/11/botb2014b.htm"></iframe> <p>Each block in the figure represents one blog, where the color and the number of dots indicate how often new postings appear. Here’s the legend:</p> <table> <thead> <tr> <th style="text-align: left">Symbol</th> <th style="text-align: left">Posting frequency</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><img src="http://www.sargasso.nl/wp-content/uploads/2006/03/bf1k1.png" alt="" /></td> <td style="text-align: left">High</td> </tr> <tr> <td style="text-align: left"><img src="http://www.sargasso.nl/wp-content/uploads/2006/03/bf2k1.png" alt="" /></td> <td style="text-align: left">Occasional gaps</td> </tr> <tr> <td style="text-align: left"><img src="http://www.sargasso.nl/wp-content/uploads/2006/03/bf3k1.png" alt="" /></td> <td style="text-align: left">Gaps of one or more days</td> </tr> <tr> <td style="text-align: left"><img src="http://www.sargasso.nl/wp-content/uploads/2006/03/bf4k1.png" alt="" /></td> <td style="text-align: left">Lots of large gaps</td> </tr> <tr> <td style="text-align: left"><img src="http://www.sargasso.nl/wp-content/uploads/2006/03/bf5k1.png" alt="" /></td> <td style="text-align: left">Almost dead</td> </tr> <tr> <td style="text-align: left"><img src="http://www.sargasso.nl/wp-content/uploads/2006/03/bf6k1.png" alt="" /></td> <td style="text-align: left">Casualty of war</td> </tr> </tbody> </table> <p>The really outstanding feature here is that most of these blogs are now in the “casualty of war” category. They’re simply <em>gone</em>; the URLs either don’t work, or they link to a completely different website. A pretty dramatic demonstration of the importance of web archiving if you ask me.</p> <h2 id="link-to-original-feature-on-sargasso">Link to original feature on Sargasso</h2> <p><a href="http://sargasso.nl/battle-blogs-war-lost/">Battle of the Blogs - The war is lost</a> (in Dutch)</p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>If the visualisation doesn’t show, you may have a browser extension installed that blocks the “sargasso.nl” domain from which it is loaded. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2014/11/13/Demise-Of-Dutch-Blogosphere https://bitsgalore.org/2014/11/13/Demise-Of-Dutch-Blogosphere 2014-11-13T00:00:00+01:00 <p>While browsing <a href="http://fileformats.archiveteam.org/wiki/Main_Page">ArchiveTeam’s File Formats Wiki</a> earlier this week, I came across some entries I created there on <a href="http://fileformats.archiveteam.org/wiki/Quattro_Pro">Quattro Pro spreadsheets</a> two years ago. At the time I had also contributed some old Quattro Pro for DOS spreadsheets (<a href="http://opf-labs.org/format-corpus/office/spreadsheet/wq1/">here</a> and <a href="http://opf-labs.org/format-corpus/office/spreadsheet/wq2/">here</a>) from my personal archives to the <a href="https://github.com/openplanets/format-corpus">OPF format corpus</a>. Seeing those files again, I decided to spend an afternoon trying to access them using modern-day software. This turned out to be more challenging than expected. It even made me wonder whether, at long last, I had finally run into a case of the much discussed (but rarely observed) phenomenon of <a href="https://openpreservation.org/blog/2010/12/22/obsolescence-overrated/">format obsolescence</a>. Yes, big words indeed, and if anyone would like to prove me wrong, the comments section below is your friend!</p> <!-- more --> <h2 id="what-its-all-about">What it’s all about</h2> <p><a href="http://en.wikipedia.org/wiki/Quattro_Pro">Quattro Pro</a> is a spreadsheet program that was first released in 1988. It’s still around today as part of the <a href="http://www.wordperfect.com/gb/product/office-suite/">WordPerfect Office suite</a>. <a href="http://fileformats.archiveteam.org/wiki/Quattro_Pro">A number of file formats</a> have been associated with the software. This blog post covers the old Quattro Pro for DOS formats:</p> <ul> <li><a href="http://fileformats.archiveteam.org/wiki/WQ1">Quattro Pro for DOS, versions 1-4 (WQ1)</a></li> <li><a href="http://fileformats.archiveteam.org/wiki/WQ2">Quattro Pro for DOS, versions 5.0 and 5.5 (WQ2)</a></li> </ul> <p>First of all, let’s have a look to what extent contemporary spreadsheet software can handle these formats.</p> <h2 id="ms-excel">MS Excel</h2> <p>Support for Quattro Pro spreadsheets (including recent versions of the format!) was removed altogether from more recent versions of Excel, as shown by this <a href="http://office.microsoft.com/en-us/excel-help/file-formats-that-are-supported-in-excel-HP010352464.aspx#BMunsupportedformats">overview of file formats that are not supported in Excel 2010</a>. On a side note, this list also includes all versions of <a href="http://fileformats.archiveteam.org/wiki/Lotus_1-2-3">Lotus 1-2-3</a> (which was once widely used). Older versions of Excel did offer support for the format. According to Microsoft, <a href="http://office.microsoft.com/en-us/excel-help/about-opening-and-saving-files-from-other-programs-HP005253843.aspx">Excel 2003 supports Quattro Pro spreadsheets</a> (albeit only after installing some converter add-ons from the Microsoft Office Web site<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>). The website explicitly mentions Quattro Pro for DOS files, although it also says “there are some limitations to opening the worksheets”.</p> <h2 id="libreoffice--openoffice">LibreOffice / OpenOffice</h2> <p>According to <a href="https://wiki.documentfoundation.org/Feature_Comparison:_LibreOffice_-_Microsoft_Office">this overview</a>, LibreOffice offers support for <a href="http://fileformats.archiveteam.org/wiki/WB2">Quattro Pro 6 (WB2)</a> spreadsheets, but it cannot handle the older Quattro Pro for DOS formats. It doesn’t mention <a href="http://fileformats.archiveteam.org/wiki/Quattro_Pro">newer versions</a> of the format either. The situation is the same for <a href="https://wiki.openoffice.org/wiki/Documentation/OOo3_User_Guides/Getting_Started/File_formats">OpenOffice</a>.</p> <h2 id="tests-with-quattro-pro-x7">Tests with Quattro Pro X7</h2> <p>With neither Excel, LibreOffice or OpenOffice being able to open my spreadsheets, I went over to the WordPerfect website and <a href="http://www.wordperfect.com/gb/free-trials/">grabbed a trial version of the WordPerfect Office suite</a> (which includes Quattro Pro X7). I then tried opening some files, all of which are available from the <a href="https://github.com/openplanets/format-corpus/tree/master/office/spreadsheet">spreadsheet</a> section of the OPF Format Corpus.</p> <h3 id="simple-numerical--text-data">Simple numerical / text data</h3> <p>I started out by opening two versions of a spreadsheet that contains simple numerical and text data. The <a href="https://github.com/openpreserve/format-corpus/blob/master/office/spreadsheet/wq1/KSBASE.WQ1">first version</a> has the <a href="http://fileformats.archiveteam.org/wiki/WQ1">WQ1 (Quattro Pro for DOS version 1-4)</a> format. The file opens without problems:</p> <p><img src="/images/2014/10/ksbase_wq1.png" alt="" /></p> <p>I also had <a href="https://github.com/openpreserve/format-corpus/blob/master/office/spreadsheet/wq2/KSBASE.WQ2">another version of that spreadsheet</a> in <a href="http://fileformats.archiveteam.org/wiki/WQ2">WQ2 (Quattro Pro for DOS version 5)</a> format. Opening this file produced the following result:</p> <p><img src="/images/2014/10/ksbase_wq2.png" alt="" /></p> <p>For some reason the numbers in some columns (<em>A</em>, <em>C</em>, <em>D</em>, <em>G</em>, <em>H</em>, <em>I</em>) aren’t displayed, but clicking on any of those calls reveals they are actually still there. Changing the formatting properties also makes them visible again, as shown below:</p> <p><img src="/images/2014/10/ksbase_wq2_fixedformatting.png" alt="" /></p> <p>So it looks like this is only a formatting issue.</p> <h3 id="simple-formulas-charts">Simple formulas, charts</h3> <p>Next I moved on to two other spreadsheets, which are a bit more interesting because they do some simple calculations<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup> and also contain charts. First I opened <a href="https://github.com/openpreserve/format-corpus/blob/master/office/spreadsheet/wq2/KS4001.WQ2">KS4001.WQ2</a>; the screenshot below shows how it is rendered by Quattro Pro:</p> <p><img src="/images/2014/10/ks4001_wq2.png" alt="" /></p> <p>The main calculation results are in cells <em>H17</em> and <em>H18</em>. The blue arrows highlight the cells from which these values are calculated. The calculated results are also correct. As before, two columns <em>appear</em> to be empty, but again, clicking these cells reveals that the underlying data (numbers in column <em>C</em>, and a calculation result in column <em>D</em>) are still present. I really don’t remember what the chart originally looked like, but I was pleasantly surprised to see it’s still displayed at all!</p> <h3 id="external-links">External links</h3> <p>Things got really interesting when I tried opening <a href="https://github.com/openpreserve/format-corpus/blob/master/office/spreadsheet/wq2/KS4000.WQ2">this WQ2 spreadsheet</a>. Upon opening, Quattro Pro comes up with a preview of the file, and a <em>Hotlinks</em> dialogue box:</p> <p><img src="/images/2014/10/ks4000_wq2_hotlinks.png" alt="" /></p> <p>I highlighted some areas in red; we’ll get back to that in a second. I first selected <em>Open Supporting</em> in the dialogue box, and pressed <em>OK</em>. The result was that both this spreadsheet and <a href="https://github.com/openpreserve/format-corpus/blob/master/office/spreadsheet/wq2/KSBASE.WQ2">another one (our earlier <em>KSBASE.WQ2</em>)</a> were loaded, so apparently it contains a link to that file. After loading, the spreadsheet displays as follows:</p> <p><img src="/images/2014/10/ks4000_wq2_opensupporting.png" alt="" /></p> <p>Now, pay special attention to the highlighted cells and compare them against the initial preview. This reveals some pretty dramatic changes: some of the preview values in rows 4 and 5 are replaced by <em>Evaluator Stack Error</em> after the file is fully loaded. Clicking on those cells also results in odd sequences of Unicode characters in the formula bar:</p> <p><img src="/images/2014/10/stackError.png" alt="" /></p> <p>My best guess is that these cells contain external references that -for whatever reason- aren’t resolved correctly. This in turn also influences the calculation results in cells <em>H17</em> and <em>H18</em>. I also tried opening the file using the <em>Update References</em> and <em>None</em> options; in both cases the results are the same as described above.</p> <p>With no access to the software that originally created the files, it is impossible to tell why the external references aren’t working. It could be a bug of Quattro Pro, but I’m not ruling out that the spreadsheet may simply be faulty (e.g. perhaps the original referenced spreadsheet got replaced by an identically-named file at some point). Nevertheless, the fact that the correct cell values are displayed in the preview, shows that the original data <em>are</em> present, and it’s rather worrying that Quattro Pro doesn’t offer an option to fully load the files without updating/overruling them.</p> <h2 id="implications-for-long-term-access">Implications for long-term access</h2> <p>Apart from Quattro Pro, modern spreadsheet programs offer no support for Quattro Pro for DOS spreadsheets. The most recent version of Quattro Pro still reads both DOS era formats, although there are some problems. Some of these are formatting-related (e.g. cells that contain data showing up as blank), and can be easily remedied. The behaviour of one spreadsheet with an external dependency is much more problematic, especially because Quattro Pro updates the original values (which are stored in the file) after fully loading the spreadsheet. Migrating this spreadsheet to another format would result in the loss of some of the original data. So, based on this (admittedly cursory) analysis it looks like no modern-day software is able to correctly handle the Quattro Pro for DOS formats. Add to this that the Quattro Pro for DOS formats are proprietary with (as far as I’m aware) no publicly available specifications, and I think we have a pretty strong candidate for a format that may be (nearly) obsolete.</p> <h2 id="solutions">Solutions</h2> <p>Although I haven’t explored any concrete solutions for accessing Quattro Pro for DOS spreadsheets, some obvious routes would be:</p> <ul> <li>Run an old copy of Quattro Pro for DOS (e.g. in a virtual machine) and export the spreadsheet to e.g. the <a href="http://fileformats.archiveteam.org/wiki/Lotus_1-2-3">Lotus 1-2-3</a> format (which is still reasonably well supported today).</li> <li>Run an old version of MS Excel (2003 or earlier) and export the spreadsheet to the <a href="http://fileformats.archiveteam.org/wiki/XLS">XLS</a> format.</li> </ul> <p>If anyone decides to have a go at this, I’d be very interested to see the results!</p> <h2 id="update-analysis-by-euan-cochrane-lotus-1-2-3-problematic-as-well">Update: analysis by Euan Cochrane; Lotus 1-2-3 problematic as well?</h2> <p>In response to this blog post, Euan Cochrane has done <a href="https://www.webarchive.org.uk/wayback/en/archive/20160105203022/http://openpreservation.org/blog/2014/10/29/opening-johans-quattro-pro-files-quattro-pro-6-win-311/">some additional tests with my Quattro Pro files</a> using Quattro Pro 6 running in an emulated environment. Euan’s analysis is highly recommended for any readers of this post. Moreover, trying to open a Lotus 1-2-3 file that Euan created as part of his analysis made me realise that Lotus 1-2-3 spreadsheets may also be more problematic than I initially thought. See my <a href="https://www.webarchive.org.uk/wayback/en/archive/20160105203022mp_/http://openpreservation.org/blog/2014/10/29/opening-johans-quattro-pro-files-quattro-pro-6-win-311/#comment-354">comment</a> under Euan’s blog post.</p> <h2 id="post-script-february-2019">Post script February 2019</h2> <p>In the folder from which I recovered this blog post, I also found a note with what looks like a comment to either this post, or Euan’s follow-up post (as these comments have disappeared from the OPF site there’s no way to tell). As it contains some relevant additional information on the Lotus 1-2-3, I’ve included it below:</p> <blockquote> <p>After I posted the above comment I also found out that IBM officially ceased its support for Lotus 123 on 30 September 2014. See the announcement here:</p> <p><a href="http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&amp;infotype=an&amp;appname=iSource&amp;supplier=897&amp;letternum=ENUS913-091">http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&amp;infotype=an&amp;appname=iSource&amp;supplier=897&amp;letternum=ENUS913-091</a></p> <p>Here’s a recent feature on this from the Register:</p> <p><a href="https://www.theregister.co.uk/2014/10/02/so_long_lotus_123_ibm_ceases_support_after_over_30_years_of_code/">https://www.theregister.co.uk/2014/10/02/so_long_lotus_123_ibm_ceases_support_after_over_30_years_of_code/</a></p> <p>This also applies to the Lotus SmartSuite and Organizer products (I’m not familiar with those products, so I have no idea if this has any additional format-related implications).</p> <h3 id="what-will-happen-to-the-lotus-1-2-3-codebase">What will happen to the Lotus 1-2-3 codebase?</h3> <p>This really makes me wonder what will happen to the old Lotus 1-2-3 codebase. Interestingly, in 2012 IBM discontinued its <a href="http://en.wikipedia.org/wiki/IBM_Lotus_Symphony">IBM Lotus Symphony</a> suite, after which they donated the codebase of that product to the Apache Software Foundation, who then merged it into Apache OpenOffice.</p> <p>I’m not aware of any efforts to save the Lotus 1-2-3 codebase, but I think this would be immensely helpful to keep those old spreadsheet formats accessible. I don’t know if this is something IBM would be willing to do (e.g. by releasing it as open source). This could also be interesting to of initiatives like the <a href="http://www.documentliberation.org/">Document Liberation Project</a>. If anyone has any info/additional thoughts on this please leave a comment!</p> </blockquote> <h2 id="post-script-may-2025">Post script May 2025</h2> <p>Over ten years after I wrote this post, I just <a href="/2025/05/28/quattro-pro-for-dos-revisited-an-obsolete-format-no-more">posted this follow-up</a>. This shows that LibreOffice Calc is now able to read my old Quattro Pro for DOS spreadsheets, although there are some issues.</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2014/10/29/quattro-pro-dos-obsolete-format-last/">Open Preservation Foundation blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>I wasn’t able to locate these converters anywhere on Microsoft’s website. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>Just in case anyone’s wondering: these spreadsheets calculate a soil’s saturated <a href="http://en.wikipedia.org/wiki/Hydraulic_conductivity">hydraulic conductivity</a> from field measurement data using the <a href="http://www.samsamwater.com/library/DETERMINING_HYDRAULIC_CONDUCTIVITY_WITH_THE_INVERSED_AUGER_HOLE_AND_INFILTROMETER_METHODS.pdf">inverse auger hole method</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2014/10/29/quattro-pro-dos-obsolete-format-last https://bitsgalore.org/2014/10/29/quattro-pro-dos-obsolete-format-last 2014-10-29T00:00:00+01:00 <p>Earlier this week I had a discussion with some colleagues about the archiving of mobile phone and tablet apps (iPhone/Android), and, equally important, ways to provide long-term access. The immediate incentive for this was an announcement by a Dutch publisher, who recently published a children’s book that is accompanied by its own app. Also, there are already several examples of Ebooks that are published exclusively as mobile apps. So, even though we’re not receiving any apps in our collections yet, we’ll have to address this at some point, and it’s useful to have an initial idea of the challenges that may lie ahead.</p> <!-- more --> <p>The scope of this blog is <em>not</em> to provide any in-depth coverage of the long-term preservation of mobile apps. Instead, I was just curious about two specific aspects:</p> <ul> <li>Is it possible to run a phone app on a regular PC, and, if yes, how?</li> <li>How can you use this to run an archived copy of an app?</li> </ul> <p>I spent a few afternoons running some preliminary tests. Since I’m pretty sure other institutions must be looking into this as well, I thought I might as well share the results, as well as some useful resources I came across along the way. For now I limited myself to the <a href="http://www.android.com/">Android</a> platform (<a href="http://en.wikipedia.org/wiki/IOS">iOS</a> presents additional challenges because of its restrictive license).</p> <h2 id="the-android-app-format">The Android app format</h2> <p>First of all it is helpful to know a bit more about the Android app format. Basically, an app (<em>.apk</em>) file is just a ZIP archive with a specific file and directory structure. It is based on Java’s <a href="http://fileformats.archiveteam.org/wiki/Jar">Jar</a> format. For more information see this entry on Archive Team’s file format wiki:</p> <p><a href="http://fileformats.archiveteam.org/wiki/APK">http://fileformats.archiveteam.org/wiki/APK</a></p> <p>Here you can also find how to download local copies of Android app files (e.g. to a PC), which is something that is not possible directly from the <a href="https://play.google.com/store/apps?hl=en">Google Play store</a>.</p> <h2 id="running-android-on-a-pc">Running Android on a PC</h2> <p>If you want to run Android on a regular (Linux or Windows) PC, several options exist. <a href="http://www.extremetech.com/computing/83812-run-android-apps-on-your-windows-pc-2">This article</a> gives a good general overview. The “best” solution according to its author is a <a href="http://www.bluestacks.com/app-player.html">third-party developed app player</a>. However, that player only works under Windows, it is proprietary, and non-free: to use it, you either pay a monthly fee, or put up with “sponsored apps”. Google’s Android SDK also includes an <a href="http://developer.android.com/tools/help/emulator.html">emulator</a>, which is mainly targeted at app developers. I didn’t look into that now, mainly because of its alleged poor performance. Instead, I went for a third option.</p> <h2 id="android-on-virtualbox">Android on VirtualBox</h2> <p>The <a href="http://www.android-x86.org/">Android-x86 Project</a> has created a port of the <a href="http://source.android.com/">Android Open Source Project</a> that runs on <a href="http://en.wikipedia.org/wiki/X86">X86-based</a> architectures. This opens op the possibility to run Android on an ordinary PC, either as the main operating system, or in a virtual machine. So, I decided to take the latter route and installed Android on a virtual machine using <a href="https://www.virtualbox.org/">VirtualBox</a>. This is relatively straightforward, and several excellent step-by-step descriptions on how to do this exist, for instance:</p> <ul> <li> <p><a href="http://www.howtogeek.com/164570/how-to-install-android-in-virtualbox/">How to Install Android in VirtualBox</a> (<a href="http://web.archive.org/web/20141023111241/http://www.howtogeek.com/164570/how-to-install-android-in-virtualbox/">archived link</a>); and</p> </li> <li> <p><a href="http://www.fixedbyvonnie.com/2014/02/install-android-4-4-kitkat-windows-using-virtualbox/">How to install Android 4.4 KitKat in Windows using VirtualBox</a> (<a href="http://web.archive.org/web/20141023111321/http://www.fixedbyvonnie.com/2014/02/install-android-4-4-kitkat-windows-using-virtualbox/">archived link</a>).</p> </li> </ul> <p>The latest ISO images from Android-x86 can be found here (for this test I used version 4.4):</p> <p><a href="http://sourceforge.net/projects/android-x86/files/">http://sourceforge.net/projects/android-x86/files/</a></p> <h2 id="running-and-waking-up-android">Running and waking up Android</h2> <p>Setting up the virtual machine was easy enough, and Android appeared to work well straight away. One thing that can be confusing for first-time users is the way VirtualBox deals with mouse input: once you click your mouse in the screen area that is occupied by the virtual machine, VirtualBox shows a dialog asking whether it should “capture” the mouse. Once you click <em>Capture</em>, the mouse can only be used inside the virtual machine (and not for any other applications that are running on the host machine). You can <em>uncapture</em> the mouse at any time by pressing the right-hand <em>Ctrl</em> key. Another thing that initially puzzled me, is that Android enters sleep mode after several minutes of inactivity, resulting in a black screen. Once in sleep mode, it is not very obvious how to wake it up again (even rebooting the VM didn’t do the trick). After some searching I found that <a href="http://www.sysads.co.uk/2014/01/install-android-4-3-virtualbox-screenshots">the solution</a> here is to press the <a href="http://en.wikipedia.org/wiki/Menu_key">Menu key</a> on the keyboard (located next to the right-hand <em>Ctrl</em> key on most keyboards), which instantly brings the machine back to life.</p> <h2 id="moving-an-app-to-the-virtual-machine">Moving an app to the virtual machine</h2> <p>To install an app in Android, you would normally go to the <a href="https://play.google.com/store/apps?hl=en">Google Play store</a>. In an archival setting it is more likely that you already have an archived copy stored somewhere, so what we need here is the ability to install from a local <a href="http://fileformats.archiveteam.org/wiki/APK">APK file</a>. This is also known as “side loading”, and <a href="http://www.cnet.com/how-to/how-to-install-apps-outside-of-google-play/">this article</a> gives general instructions on how to do this with a physical device. Since we’re running Android on a virtual machine here, things are a bit different, and ideally we should be able to share a folder between the host machine and the (virtual) guest device. In theory this is all possible in VirtualBox, but as it turns out <a href="http://superuser.com/questions/665696/shared-folder-in-virtualbox-with-android-not-working">it doesn’t work</a> because <a href="http://stackoverflow.com/questions/8235165/getting-vbox-guest-addtions-for-android-x86">Android-86 doesn’t support VirtualBox Guest Additions</a>. As a workaround, I ended up uploading my the APK to DropBox, and then opened DropBox in Android’s web browser to download the file.</p> <h2 id="installing-the-app">Installing the app</h2> <p>The downloaded APK is now located in the <em>Download</em> folder, which is accessible using Android’s file browser. After clicking on it, the following security warning popped up:</p> <p><img src="/images/2014/10/installBlocked.png" alt="" /></p> <p>This is because, by default, apps from unknown sources (i.e. other than the <a href="https://play.google.com/store/apps?hl=en">Google Play store</a>) are blocked by Android. The solution here is to click on <em>Settings</em>, which opens up the security settings dialog:</p> <p><img src="/images/2014/10/tickUnknownSources.png" alt="" /></p> <p>Here, check the <em>Unknown sources</em> option <sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. Then go back to the <em>Downloads</em> folder and click on the APK file again. It will now install the app.</p> <h2 id="final-thoughts">Final thoughts</h2> <p>In this blog I provided some basic information about Android’s APK format, how to run Android in VirtualBox, and how to install an archived app. I tested this myself with a handful of apps. One thing I noticed was that some apps didn’t quite work as expected on my virtual Android machine, but as I didn’t have access to a ‘real’ (physical) device it’s impossible to tell whether this was due to the virtualisation or just a shortcoming of those apps. This would obviously need more work. Nevertheless, considering that I only spent a few odd afternoons on this, this approach looks quite promising.</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2014/10/23/running-archived-android-apps-pc-first-impressions/">Open Preservation Foundation blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>Note that this will also enable you to install apps that are possibly harmful; use with care! <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2014/10/23/running-archived-android-apps-pc-first-impressions https://bitsgalore.org/2014/10/23/running-archived-android-apps-pc-first-impressions 2014-10-23T00:00:00+02:00 <p>Some time ago Will Palmer, Peter May and Peter Cliff of the British Library published a really interesting <a href="http://www.scape-project.eu/publication/palmer-ipres2013">paper that investigated three different JPEG 2000 codecs</a>, and their effects on image quality in response to lossy compression. Most remarkably, their analysis revealed differences not only in the way these codecs <em>encode</em> (compress) an image, but also in the <em>decoding</em> phase. In other words: reading the same lossy JP2 produced different results depending on which implementation was used to decode it.</p> <p>A limitation of the paper’s methodology is that it obscures the individual effects of the encoding and decoding components, since both are essentially lumped in the analysis. Thus, it’s not clear how much of the observed degradation in image quality is caused by the compression, and how much by the decoding. This made me wonder how similar the <em>decode</em> results of different codecs really are.</p> <!-- more --> <h2 id="an-experiment">An experiment</h2> <p>To find out, I ran a simple experiment:</p> <ol> <li>Encode a TIFF image to JP2.</li> <li>Decode the JP2 back to TIFF using different decoders.</li> <li>Compare the decode results using some similarity measure.</li> </ol> <h2 id="codecs-used">Codecs used</h2> <p>I used the following codecs:</p> <ul> <li><a href="http://www.kakadusoftware.com/">Kakadu</a> v7.2.2 (kakadu)</li> <li><a href="http://www.openjpeg.org/">OpenJPEG</a> 2.0 (opj20)</li> <li><a href="http://www.imagemagick.org/">ImageMagick</a> 6.8.9-8 (im)</li> <li><a href="http://www.graphicsmagick.org/">GraphicsMagick</a> 1.3.18 (gm)</li> <li><a href="http://www.irfanview.com/">IrfanView</a> 4.35 with JPEG2000 plugin 4.33 (irfan)</li> </ul> <p>Note that GraphicsMagick still uses the <a href="http://www.ece.uvic.ca/~frodo/jasper/">JasPer</a> library for JPEG 2000. ImageMagick now uses OpenJPEG (older versions used JasPer). IrfanViews’s JPEG 2000 plugin is made by <a href="http://www.luratech.com/en/products/luratech-jp2-irfanview-plug-in/">LuraTech</a>.</p> <h2 id="creating-the-jp2">Creating the JP2</h2> <p>First I compressed my source TIFF (a grayscale newspaper page) to a lossy JP2 with a compression ratio about about 4:1. For this example I used OpenJPEG, with the following command line:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>opj_compress <span class="nt">-i</span> krant.tif <span class="nt">-o</span> krant_oj_4.jp2 <span class="nt">-r</span> 4 <span class="nt">-I</span> <span class="nt">-p</span> RPCL <span class="nt">-n</span> 7 <span class="nt">-c</span> <span class="o">[</span>256,256],[256,256],[256,256],[256,256],[256,256],[256,256],[256,256] <span class="nt">-b</span> 64,64 </code></pre></div></div> <h2 id="decoding-the-jp2">Decoding the JP2</h2> <p>Next I decoded this image back to TIFF using the aforementioned codecs. I used the following command lines:</p> <table> <thead> <tr> <th style="text-align: left">Codec</th> <th style="text-align: left">Command line</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><strong>opj20</strong></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">opj_decompress -i krant_oj_4.jp2 -o krant_oj_4_oj.tif</code></td> </tr> <tr> <td style="text-align: left"><strong>kakadu</strong></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">kdu_expand -i krant_oj_4.jp2 -o krant_oj_4_kdu.tif</code></td> </tr> <tr> <td style="text-align: left"><strong>kakadu-precise</strong></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">kdu_expand -i krant_oj_4.jp2 -o krant_oj_4_kdu_precise.tif -precise</code></td> </tr> <tr> <td style="text-align: left"><strong>irfan</strong></td> <td style="text-align: left">Used GUI</td> </tr> <tr> <td style="text-align: left"><strong>im</strong></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">convert krant_oj_4.jp2 krant_oj_4_im.tif</code></td> </tr> <tr> <td style="text-align: left"><strong>gm</strong></td> <td style="text-align: left"><code class="language-plaintext highlighter-rouge">gm convert krant_oj_4.jp2 krant_oj_4_gm.tif</code></td> </tr> </tbody> </table> <p>This resulted in 6 images. Note that I ran Kakadu twice: once using the default settings, and also with the <em>-precise</em> switch, which “forces the use of 32-bit representations”.</p> <h2 id="overall-image-quality">Overall image quality</h2> <p>As a first analysis step I computed the overall peak signal to noise ratio (PSNR) for each decoded image, relative to the source TIFF:</p> <table> <thead> <tr> <th style="text-align: left">Decoder</th> <th style="text-align: left">PSNR</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><strong>opj20</strong></td> <td style="text-align: left">48.08</td> </tr> <tr> <td style="text-align: left"><strong>kakadu</strong></td> <td style="text-align: left">48.01</td> </tr> <tr> <td style="text-align: left"><strong>kakadu-precise</strong></td> <td style="text-align: left">48.08</td> </tr> <tr> <td style="text-align: left"><strong>irfan</strong></td> <td style="text-align: left">48.08</td> </tr> <tr> <td style="text-align: left"><strong>im</strong></td> <td style="text-align: left">48.08</td> </tr> <tr> <td style="text-align: left"><strong>gm</strong></td> <td style="text-align: left">48.07</td> </tr> </tbody> </table> <p>So <em>relative to the source image</em> these results are only marginally different.</p> <h2 id="similarity-of-decoded-images">Similarity of decoded images</h2> <p>But let’s have a closer look at how similar the different decoded images are. I did this by computing PSNR values of all possible decoder pairs. This produced the following matrix:</p> <table> <thead> <tr> <th style="text-align: left">Decoder</th> <th style="text-align: left">opj20</th> <th style="text-align: left">kakadu</th> <th style="text-align: left">kakadu-precise</th> <th style="text-align: left">irfan</th> <th>im</th> <th>gm</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><strong>opj20</strong></td> <td style="text-align: left">-</td> <td style="text-align: left">57.52</td> <td style="text-align: left">78.53</td> <td style="text-align: left">79.17</td> <td>96.35</td> <td>64.43</td> </tr> <tr> <td style="text-align: left"><strong>kakadu</strong></td> <td style="text-align: left">57.52</td> <td style="text-align: left">-</td> <td style="text-align: left">57.51</td> <td style="text-align: left">57.52</td> <td>57.52</td> <td>57.23</td> </tr> <tr> <td style="text-align: left"><strong>kakadu-precise</strong></td> <td style="text-align: left">78.53</td> <td style="text-align: left">57.51</td> <td style="text-align: left">-</td> <td style="text-align: left">79.00</td> <td>78.53</td> <td>64.52</td> </tr> <tr> <td style="text-align: left"><strong>irfan</strong></td> <td style="text-align: left">79.17</td> <td style="text-align: left">57.52</td> <td style="text-align: left">79.00</td> <td style="text-align: left">-</td> <td>79.18</td> <td>64.44</td> </tr> <tr> <td style="text-align: left"><strong>im</strong></td> <td style="text-align: left">96.35</td> <td style="text-align: left">57.52</td> <td style="text-align: left">78.53</td> <td style="text-align: left">79.18</td> <td>-</td> <td>64.43</td> </tr> <tr> <td style="text-align: left"><strong>gm</strong></td> <td style="text-align: left">64.43</td> <td style="text-align: left">57.23</td> <td style="text-align: left">64.52</td> <td style="text-align: left">64.44</td> <td>64.43</td> <td>-</td> </tr> </tbody> </table> <p>Note that, unlike the table in the previous section, these PSNR values are only a measure of the <em>similarity</em> between the different decoder results. They don’t directly say anything about <em>quality</em> (since we’re not comparing against the source image). Interestingly, the PSNR values in the matrix show two clear groups:</p> <ul> <li><strong>Group A</strong>: all combinations of OpenJPEG, Irfanview, ImageMagick and Kakadu in <em>precise</em> mode, all with a PSNR of &gt; 78 dB.</li> <li><strong>Group B</strong>: all remaining decoder combinations, with a PSNR of &lt; 64 dB.</li> </ul> <p>What this means is that OpenJPEG, Irfanview, ImageMagick and Kakadu in <em>precise</em> mode all decode the image in a similar way, whereas Kakadu (default mode) and GraphicsMagick behave differently. Another way of looking at this is to count the pixels that have different values for each combination. This yields up to 2 % different pixels for all combinations in group <em>A</em>, and about 12 % in group <em>B</em>. Finally, we can look at the peak absolute error value (PAE) of each combination, which is the maximum value difference for any pixel in the image. This figure was 1 pixel level (0.4 % of the full range) in both groups.</p> <p>I also repeated the above procedure for a small RGB image. In this case I used Kakadu as the encoder. The decoding results of that experiment showed the same overall pattern, although the differences between groups <em>A</em> and <em>B</em> were even more pronounced, with PAE values in group <em>B</em> reaching up to 3 pixel values (1.2 % of full range) for some decoder combinations.</p> <h2 id="what-does-this-say-about-decoding-quality">What does this say about decoding quality?</h2> <p>It would be tempting to conclude from this that the codecs that make up group <em>A</em> provide better quality decoding than the others (GraphicsMagick, Kakadu in default mode). If this were true, one would expect that the overall PSNR values <em>relative to the source TIFF</em> (see previous table) would be higher for those codecs. But the values in the table are only marginally different. Also, in the test on the small RGB image, running Kakadu in <em>precise</em> mode <em>lowered</em> the overall PSNR value (although by a tiny amount). Such small effects could be due to chance, and for a conclusive answer one would need to repeat the experiment for a large number of images, and test the PSNR differences for statistical significance (as was done in the BL analysis).</p> <p>I’m still somewhat surprised that even in group <em>A</em> the decoding results aren’t <em>identical</em>, but I suspect this has something to do with small rounding errors that arise during the decode process (maybe someone with a better understanding of the mathematical intricacies of JPEG 2000 decoding can comment on this). Overall, these results suggest that the errors that are introduced by the decode step are very small when compared against the encode errors.</p> <h2 id="conclusions">Conclusions</h2> <p>OpenJPEG, (recent versions of) ImageMagick, IrfanView and Kakadu in <em>precise</em> mode all produce similar results when decoding lossily compressed JP2s, whereas Kakadu in default mode and GraphicsMagick (which uses the JasPer library) behave differently. These differences are very small when compared to the errors that are introduced by the encoding step, but for critical decode applications (migrate lossy JP2 to something else) they may still be significant. As both ImageMagick and GraphicsMagick are often used for calculating image (quality) statistics, the observed differences also affect the outcome of such analyses: calculating PSNR for a JP2 with ImageMagick and GraphicsMagick results in two different outcomes!</p> <p>For <em>losslessy</em> compressed JP2s, the decode results for all tested codecs are 100% identical<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>.</p> <p>This tentative analysis does not support any conclusions on which decoders are ‘better’. That would need additional tests with more images. I don’t have time for that myself, but I’d be happy to see others have a go at this!</p> <h2 id="link">Link</h2> <p><a href="http://www.scape-project.eu/publication/palmer-ipres2013">William Palmer, Peter May and Peter Cliff: An Analysis of Contemporary JPEG2000 Codecs for Image Format Migration (Proceedings, iPres 2013)</a></p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2014/09/26/six-ways-decode-lossy-jp2/">Open Preservation Foundation blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:2"> <p>Identical in terms of pixel values; for this analysis I didn’t look at things such as embedded ICC profiles, <a href="http://wiki.opf-labs.org/display/TR/Handling+of+ICC+profiles">which not all encoders/decoders handle well</a>. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2014/09/26/six-ways-decode-lossy-jp2 https://bitsgalore.org/2014/09/26/six-ways-decode-lossy-jp2 2014-09-26T00:00:00+02:00 <p>Vandaag maakte de Britse <a href="http://www.dpconline.org/about"><em>Digital Preservation Coalition</em></a> de finalisten bekend die in de race zijn voor de <a href="http://www.dpconline.org/advocacy/awards/digital-preservation-awards-2014"><em>Digital Preservation Awards 2014</em></a>. Deze prijs is in 2004 in het leven geroepen om aandacht te vestigen op initiatieven die een belangrijke bijdrage leveren aan het toegankelijk houden van digitaal erfgoed.</p> <!-- more --> <p>In de categorie <a href="http://www.dpconline.org/newsroom/latest-news/1272-dpa-2014-award-researchandinnovation-finalists"><em>Research and Innovation</em></a> is een op de KB door de afdeling Onderzoek ontwikkelde softwaretool genomineerd: <a href="http://jpylyzer.openpreservation.org/"><em>jpylyzer</em></a>. Met <em>jpylyzer</em> kun je op een eenvoudige manier controleren of <em>JP2</em> (JPEG 2000) beeldbestanden technisch in orde zijn. Binnen de KB wordt de tool onder meer ingezet bij de kwaliteitscontrole van gedigitaliseerde boeken, kranten en tijdschriften. <em>Jpylyzer</em> wordt ook gebruikt door diverse internationale collega-instellingen.</p> <p><em>Jpylyzer</em> is deels ontwikkeld binnen het Europese project <a href="http://www.scape-project.eu/"><em>SCAPE</em></a>, waarin de KB projectpartner is. De uiteindelijke winnaars worden op 17 november bekendgemaakt.</p> <p>Meer informatie over de nominatie van <em>jpylyzer</em> is te vinden op de website van de <em>Digital Preservation Coalition</em>:</p> <p><a href="http://www.dpconline.org/newsroom/latest-news/1271-dpa-2014finalists">http://www.dpconline.org/newsroom/latest-news/1271-dpa-2014finalists</a></p> <p>Het volgende artikel is interessant voor wie meer wil weten over <em>jpylyzer</em>, en waarom we zo’n tool eigenlijk nodig hebben:</p> <p><a href="https://www.kb.nl/organisatie/onderzoek-expertise/onderzoek-digitalisering-en-digitale-duurzaamheid/afgesloten-projecten/jpylyzer-jp2-validator-and-extractor">Miljoenen digitale bestanden controleer je niet even met de hand</a></p> <p>Ten slotte is hier de <em>jpylyzer</em> homepage:</p> <p><a href="http://jpylyzer.openpreservation.org/">http://jpylyzer.openpreservation.org/</a></p> <hr /> <p>Originally published at the <a href="http://blog.kbresearch.nl/2014/09/11/jpylyzer-software-finalist-voor-digitale-duurzaamheidsprijs/">KB Research blog</a></p> https://bitsgalore.org/2014/09/11/jpylyzer-software-finalist-voor-digitale-duurzaamheidsprijs https://bitsgalore.org/2014/09/11/jpylyzer-software-finalist-voor-digitale-duurzaamheidsprijs 2014-09-11T00:00:00+02:00 <p>It is well-known that PDF documents can contain features that are preservation risks (e.g. see <a href="https://web.archive.org/web/20130515073645/http://libraries.stackexchange.com/questions/964/what-preservation-risks-are-associated-with-the-pdf-file-format">here</a> and <a href="http://wiki.opf-labs.org/display/TR/Portable+Document+Format">here</a>). Migration of existing <em>PDF</em>s to <em>PDF/A</em> is sometimes advocated as a strategy for mitigating these risks. However, the benefits of this approach are often questionable, and the migration process can also be quite risky in itself. As I often get questions on this subject, I thought it might be worthwhile to do a short write-up on this.</p> <!-- more --> <h2 id="pdfa-is-a-profile"><em>PDF/A</em> is a profile</h2> <p>First, it’s important to stress that each of the <em>PDF/A</em> standards (<em>A-1</em>, <em>A-2</em> and <em>A-3</em>) are really just <em>profiles</em> within the <em>PDF</em> format. More specifically, <em>PDF/A-1</em> offers a subset of <a href="http://acroeng.adobe.com/PDFReference/PDF_1.4/PDF%20Reference%201.4.pdf"><em>PDF 1.4</em></a>, whereas <em>PDF/A-2</em> and <em>PDF/A-3</em> are based on <a href="http://acroeng.adobe.com/PDFReference/ISO32000/PDF32000-Adobe.pdf">the ISO 32000 version of <em>PDF 1.7</em></a>. What these profiles have in common, is that they prohibit some features (e.g. multimedia, encryption, interactive content) that are allowed in ‘regular’ <em>PDF</em>. Also, they narrow down the way other features are implemented, for example by requiring that all fonts are embedded in the document. This can be illustrated with the following simple Venn diagram below, which shows the feature sets of the aforementioned <em>PDF</em> flavours:</p> <p><img src="/images/2014/08/pdfVenn.png" alt="PDF Venn diagram" /></p> <p>Here we see how <em>PDF/A-1</em> is a subset of <em>PDF 1.4</em>, which in turn is a subset of <em>PDF 1.7</em>. <em>PDF A/2</em> and <em>PDF A/3</em> (aggregated here as one entity for the sake of readability) are subsets of <em>PDF 1.7</em>, and include all the features of <em>PDF A/1</em>.</p> <p>Keeping this in mind, it’s easy to see that migrating an arbitrary <em>PDF</em> to <em>PDF/A</em> can result in problems.</p> <h2 id="loss-alteration-during-migration">Loss, alteration during migration</h2> <p>Suppose, as an example, that we have a <em>PDF</em> that contains a movie. This is prohibited in <em>PDF/A</em>, so migrating to <em>PDF/A</em> will simply result in the loss of the multimedia content. Another example are fonts: all fonts in a <em>PDF/A</em> document must be embedded. But what happens if the source <em>PDF</em> uses non-embedded fonts that are not available on the machine on which the migration is run? Will the migration tool exit with a warning, or will it silently use some alternative, perhaps similar font? And how do you check for this?</p> <h2 id="complexity-and-effect-of-errors">Complexity and effect of errors</h2> <p>Also, migrations like these typically involve a complete re-processing of the <em>PDF</em>’s internal structure. The format’s complexity implies that there’s a lot of potential for things to go wrong in this process. This is particularly true if the source <em>PDF</em> contains subtle errors, in which case the risk of losing information is very real (even though the original document may be perfectly readable in a viewer). Since we don’t really have any tools for detecting such errors (i.e. a <a href="http://duff-johnson.com/wp-content/uploads/2014/01/PDFValidationDreamOrYawn.pdf">sufficiently reliable <em>PDF</em> validator</a>), these cases can be difficult to deal with. Some further considerations can be found <a href="http://web.archive.org/web/20130605142355/http://libraries.stackexchange.com/questions/1117/converting-invalid-pdfs-or-not-for-digital-preservation">here</a> (the context there is slightly different, but the risks are similar).</p> <h2 id="digitised-vs-born-digital">Digitised vs born-digital</h2> <p>The origin of the source <em>PDF</em>s may be another thing to take into account. If <em>PDF</em>s were originally created as part of a digitisation project (e.g. scanned books), the <em>PDF</em> is usually little more than a wrapper around a bunch of images, perhaps augmented by an OCR layer. Migrating such <em>PDF</em>s to <em>PDF/A</em> is pretty straightforward, since the source files are unlikely to contain any features that are not allowed in <em>PDF/A</em>. At the same time, this also means that the benefits of migrating such files to <em>PDF/A</em> are pretty limited, since the source <em>PDF</em>s weren’t problematic to begin with!</p> <p>The potential benefits <em>PDF/A</em> may be more obvious for a lot of born-digital content; however, for the reasons listed in the previous section, the migration is more complex, and there’s just a lot more that can go wrong (see also <a href="http://qanda.digipres.org/19/what-are-the-benefits-and-risks-of-using-the-pdf-a-file-format?show=21#a21">here</a> for some additional considerations).</p> <h2 id="conclusions">Conclusions</h2> <p>Although migrating <em>PDF</em> documents to <em>PDF/A</em> may look superficially attractive, it is actually quite risky in practice, and it may easily result in unintentional data loss. Moreover, the risks increase with the number of preservation-unfriendly features, meaning that the migration is most likely to be successful for source <em>PDF</em>s that weren’t problematic to begin with, which belies the very purpose of migrating to <em>PDF/A</em>. For specific cases, migration to <em>PDF/A</em> may still be a sensible approach, but the expected benefits should be weighed carefully against the risks. In the absence of stable, generally accepted tools for assessing the quality of <em>PDF</em>s (both source <em>and</em> destination!), it would also seem prudent to always keep the originals.</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2014/08/27/when-not-migrate-pdf-pdfa/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2014/08/27/when-not-migrate-pdf-pdfa https://bitsgalore.org/2014/08/27/when-not-migrate-pdf-pdfa 2014-08-27T00:00:00+02:00 <p>This short tutorial shows how to take a snapshot of a web page, and save it to the Internet Archive’s <a href="http://en.wikipedia.org/wiki/Wayback_Machine">Wayback Machine</a>.</p> <!-- more --> <h2 id="method-1-web-interface">Method 1: web interface</h2> <ol> <li>Go to the Wayback website: <a href="https://archive.org/web/">https://archive.org/web/</a></li> <li>Paste the URL of the page you want to archive into the <em>Save Page Now</em> box (at the bottom-right).</li> <li>Click on the <em>Save Page</em> button (or press <em>enter</em>).</li> <li>Wait while the page is being crawled. Once the archiving process is complete, the URL of the archived page appears.</li> </ol> <h2 id="method-2-bookmarklet">Method 2: bookmarklet</h2> <p>This method is faster than using the web interface, but you will first need to install a <a href="http://en.wikipedia.org/wiki/Bookmarklet">bookmarklet</a> (which is just a browser bookmark that contains some JavaScript).</p> <h3 id="installation">Installation</h3> <ol> <li> <p>Go to the <em>Save Page to Wayback Machine Bookmarklet</em> link here: <a href="http://marklets.com/Save%20Page%20to%20Wayback%20Machine.aspx">http://marklets.com/Save%20Page%20to%20Wayback%20Machine.aspx</a></p> </li> <li> <p>Click at the left-hand site of the URL bar, and drag it to the bookmarks toolbar of your browser. The figure below shows how this works in FireFox:</p> <p><img src="/images/2014/08/wbmarkletInstall.png" alt="Installation of bookmarklet" /></p> <p>Alternatively you can also use <em>Add Bookmark</em> in the <em>Bookmarks</em> menu.</p> </li> </ol> <h3 id="using-the-bookmarklet">Using the bookmarklet</h3> <ol> <li>Open the web page that you want to save in your browser.</li> <li>Click on <em>Save Page to Wayback Machine</em> in the bookmarks toolbar.</li> <li>Wait while the page is being crawled. Once the archiving process is complete, the URL of the archived page appears.</li> </ol> <h2 id="method-3-chrome-extension">Method 3: Chrome extension</h2> <p>If you’re using the Google Chrome browser, you may want to check out Jimmy Lin’s “Save a Page” extension. Once installed, it allows you to save a page by simply right-clicking on it. The extension can be found here:</p> <p><a href="https://github.com/lintool/chrome-archive-this-page">https://github.com/lintool/chrome-archive-this-page</a></p> <p>Just follow the installation instructions on that page.</p> <h2 id="limitations">Limitations</h2> <ul> <li>Webmasters can use <a href="http://en.wikipedia.org/wiki/Robots_exclusion_standard"><em>robots.txt</em></a> to prevent web crawlers from crawling/saving anything on their website.</li> <li>If a webmaster decides to change the <em>robots.txt</em> permissions at some point in the future, a saved page may be removed from the Wayback Machine. For details see: <a href="https://archive.org/about/exclude.php">https://archive.org/about/exclude.php</a>.</li> </ul> <h2 id="acknowledgement">Acknowledgement</h2> <p>This tutorial partially draws from a <a href="http://searchengineland.com/save-urls-wayback-machine-demand-191150">blog post</a> by Gary Price on <a href="http://searchengineland.com/"><em>Search Engine Land</em></a>.</p> https://bitsgalore.org/2014/08/02/How-to-save-a-web-page-to-the-Internet-Archive https://bitsgalore.org/2014/08/02/How-to-save-a-web-page-to-the-Internet-Archive 2014-08-02T00:00:00+02:00 <p>One of my first blogs here covered an <a href="/2011/09/21/evaluation-identification-tools-first-results-scape">evaluation of a number of format identification tools</a>. One of the more surprising results of that work was that out of the five tools that were tested, no less than four of them (<em>FITS</em>, <em>DROID</em>, <em>Fido</em> and <em>JHOVE2</em>) failed to even <em>run</em> when executed with their associated launcher script. In many cases the <em>Windows</em> launcher scripts (batch files) only worked when executed from the installation folder. Apart from making things unnecessarily difficult for the user, this also completely flies in the face of all existing conventions on command-line interface design. Around the time of this work (summer 2011) I had been in contact with the developers of all the evaluated tools, and until last week I thought those issues were a thing of the past. Well, was I wrong!</p> <!-- more --> <h2 id="fits-08"><a href="http://projects.iq.harvard.edu/files/fits/files/fits-0.8.0.zip">FITS 0.8</a></h2> <p>Fast-forward 2.5 years: this week I saw the announcement of the latest <a href="http://projects.iq.harvard.edu/fits">FITS</a> release. This got me curious, also because of the recent work on this tool as part of the <a href="http://www.openplanetsfoundation.org/blogs/2013-11-06-fits-blitz">FITS Blitz</a>. So I downloaded <a href="http://projects.iq.harvard.edu/files/fits/files/fits-0.8.0.zip">FITS 0.8</a>, installed it in a directory called <em>c:\fits\</em>on my <em>Windows</em> PC, and then typed (while being in directory <em>f:\myData\</em>):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>f:<span class="se">\m</span>yData&gt;c:<span class="se">\f</span>its<span class="se">\f</span>its </code></pre></div></div> <p>Instead of the expected helper message I ended up with this:</p> <pre><code class="language-data">The system cannot find the path specified. Error: Could not find or load main class edu.harvard.hul.ois.fits.Fits </code></pre> <p>Hang on, I’ve seen this before … don’t tell me this is the same bug that I already reported 2.5 years ago ? Well, turns out <a href="https://github.com/harvard-lts/fits/issues/10">it is</a> after all!</p> <p>This got me curious about the status of the other tools that had similar problems in 2011, so I started downloading the latest versions of <a href="http://www.nationalarchives.gov.uk/information-management/our-services/dc-file-profiling-tool.htm"><em>DROID</em></a>, <a href="https://bitbucket.org/jhove2/main/wiki/Home"><em>JHOVE2</em></a> and <a href="https://github.com/openplanets/fido"><em>Fido</em></a>. As I was on a roll anyway, I gave <a href="http://jhove.sourceforge.net/">JHOVE</a> a try as well (even though it was not part of the 2011 evaluation). The objective of the test was simply to <em>run</em> each tool and get some screen output (e.g. a help message), nothing more. I did these tests on a PC running <em>Windows</em> 7 with <em>Java</em> version 1.7.0_25. Here are the results.</p> <h2 id="droid-613"><a href="http://www.nationalarchives.gov.uk/documents/information-management/droid-binary-6.1.3-bin.zip">DROID 6.1.3</a></h2> <p>First I installed <em>DROID</em> in a directory <em>C:\droid\</em>. Then I executed it using:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>f:<span class="se">\m</span>yData&gt;c:<span class="se">\d</span>roid<span class="se">\d</span>roid </code></pre></div></div> <p>This started up a <em>Java Virtual Machine Launcher</em> that showed this message box:</p> <p><img src="/images/2014/01/droidError.png" alt="" /></p> <p>The <em>Running DROID</em> text document that comes with <em>DROID</em> says:</p> <blockquote> <p>To run DROID on Windows, use the “droid.bat” file. You can either double-click on this file, or run it from the command-line console, by typing “droid” <strong>when you are in the droid installation folder</strong>.</p> </blockquote> <p>So, no progress on this for <em>DROID</em> either, then. I <em>was</em> able to get <em>DROID</em> running by circumventing the launcher script like this:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java <span class="nt">-jar</span> c:<span class="se">\d</span>roid<span class="se">\d</span>roid-command-line-6.1.3.jar </code></pre></div></div> <p>This resulted in the following output:</p> <pre><code class="language-data">No command line options specified </code></pre> <p>This isn’t particularly helpful. There <em>is</em> a helper message, for which you have to give the <em>-h</em> flag on the command line. But you don’t get to see this until you give the <em>-h</em> flag on the command line. Catch 22 anyone?</p> <h2 id="jhove2-210"><a href="http://bitbucket.org/jhove2/main/downloads/jhove2-2.1.0.zip">JHOVE2-2.1.0</a></h2> <p>After installing <em>JHOVE2</em> in <em>c:\jhove2\</em>, I typed:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>f:<span class="se">\m</span>yData&gt;c:<span class="se">\j</span>hove2<span class="se">\j</span>hove2 </code></pre></div></div> <p>This gave me <strong>1393</strong> (yes, you read that right: 1393!) <em>Java</em> deprecation warnings, each along the lines of:</p> <pre><code class="language-data">16:51:02,702 [main] WARN TypeConverterDelegate : PropertyEditor [com.sun.beans.editors.EnumEditor] found through deprecated global PropertyEditorManager fallback - consider using a more isolated form of registration, e.g. on the BeanWrapper/BeanFactory! </code></pre> <p>This was eventually followed by the (expected) <em>JHOVE2</em> help message, and a quick test on some actual files confirmed that <em>JHOVE2</em> <em>does</em> actually work. Nevertheless, by the time the tsunami of warning messages is over, many first-time users will have started running for the bunkers!</p> <h2 id="fido-131"><a href="https://github.com/openplanets/fido/releases/tag/1.3.1-70">Fido 1.3.1</a></h2> <p><em>Fido</em> doesn’t make use of any launcher scripts any more, and the default way to run it is to use the <em>Python</em> script directly. After installing in <em>c:\fido\</em> I typed:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>f:<span class="se">\m</span>yData&gt;c:<span class="se">\f</span>ido<span class="se">\f</span>ido.py </code></pre></div></div> <p>Which resulted in ….. (drum roll) … a nicely formatted <em>Fido</em> help message, which is exactly what I was hoping for. Beautiful!</p> <h2 id="jhove-111"><a href="http://sourceforge.net/projects/jhove/files/latest/download">JHOVE 1.11</a></h2> <p>I installed <em>JHOVE</em> in <em>c:\jhove\</em> and then typed:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>f:<span class="se">\m</span>yData&gt;c:<span class="se">\j</span>hove<span class="se">\j</span>hove </code></pre></div></div> <p>Which resulted in this:</p> <pre><code class="language-data">Exception in thread "main" java.lang.NoClassDefFoundError: edu/harvard/hul/ois/j hove/viewer/ConfigWindow at edu.harvard.hul.ois.jhove.DefaultConfigurationBuilder.writeDefaultCon figFile(Unknown Source) at edu.harvard.hul.ois.jhove.JhoveBase.init(Unknown Source) at Jhove.main(Unknown Source) Caused by: java.lang.ClassNotFoundException: edu.harvard.hul.ois.jhove.viewer.Co nfigWindow at java.net.URLClassLoader$1.run(Unknown Source) at java.net.URLClassLoader$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) ... 3 more </code></pre> <p>Ouch!</p> <h2 id="final-remarks">Final remarks</h2> <p>I limited my tests to a <em>Windows</em> environment only, and results may well be better under <em>Linux</em> for some of these tools. Nevertheless, I find it nothing less than astounding that so many of these (often widely cited) preservation tools fail to even <em>execute</em> on today’s <a href="http://en.wikipedia.org/wiki/Usage_share_of_operating_systems">most widespread operating system</a>. Granted, in some cases there are workarounds, such as tweaking the launcher scripts, or circumventing them altogether. However, this is not an option for less tech-savvy users, who will simply conclude “<em>Hey, this tool doesn’t work</em>”, give up, and move on to other things. Moreover, this means that much of the (often huge) amounts of development effort that went into these tools will simply fail to reach its potential audience, and I think this is a tremendous waste. I’m also wondering why there’s been so little progress on this over the past 2.5 years. Is it really that difficult to develop preservation tools with command-line interfaces that follow basic design conventions that have been ubiquitous elsewhere for more than 30 years? Tools that <em>just work</em>?</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2014/01/31/why-cant-we-have-digital-preservation-tools-just-work/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2014/01/31/why-cant-we-have-digital-preservation-tools-just-work https://bitsgalore.org/2014/01/31/why-cant-we-have-digital-preservation-tools-just-work 2014-01-31T00:00:00+01:00 <p>This blog follows up on three earlier posts about detecting preservation risks in <em>PDF</em> files. In <a href="/2012/12/19/identification-pdf-preservation-risks-apache-preflight-first-impression">part 1</a> I explored to what extent the <a href="http://pdfbox.apache.org/cookbook/pdfavalidation.html"><em>Preflight</em></a> component of the <a href="http://pdfbox.apache.org/"><em>Apache PDFBox</em></a> library can be used to detect specific preservation risks in <em>PDF</em> documents. This was followed up by some work during the <em>SPRUCE</em> Hackathon in Leeds, which is covered by <a href="http://www.openplanetsfoundation.org/blogs/2013-03-15-pdf-eh-another-hackathon-tale">this blog post by Peter Cliff</a>. Then last summer I did a series of <a href="/2013/07/25/identification-pdf-preservation-risks-sequel">additional tests</a> using files from the <a href="http://acroeng.adobe.com/wp/"><em>Adobe Acrobat Engineering</em> website</a>. The main outcome of this more recent work was that, although showing great promise, <em>Preflight</em> was struggling with many more complex <em>PDF</em>s. Fast-forward another six months and, thanks to the excellent response of the <em>Preflight</em> developers to our bug reports, the most serious of these problems are now largely solved<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. So, time to move on to the next step!</p> <!-- more --> <h2 id="govdocs-selected">Govdocs Selected</h2> <p>Ultimately, the aim of this work is to be able to profile large <em>PDF</em> collections for specific preservation risks, or to verify that a <em>PDF</em> conforms to an institute-specific policy before ingest. To get a better idea of how that might work in practice, I decided to do some tests with the <a href="http://www.openplanetsfoundation.org/blogs/2012-07-26-1-million-21000-reducing-govdocs-significantly"><em>Govdocs Selected</em></a> dataset, which is a subset of the <a href="http://digitalcorpora.org/corpora/files">Govdocs1</a> corpus. As a first step I ran the latest version of <em>Preflight</em> on every <em>PDF</em> in the corpus (about 15 thousand)<sup id="fnref:2"><a href="#fn:2" class="footnote" rel="footnote" role="doc-noteref">2</a></sup>.</p> <h2 id="validation-errors">Validation errors</h2> <p>As I was curious about the most common validation errors (or, more correctly, violations of the <em>PDF/A-1b</em> profile), I ran a little post-processing script on the output files to calculate error occurrences. The following table lists the results. For each <em>Preflight</em> error (which is represented as an error code), the table shows the number of <em>PDF</em>s for which the error was reported (expressed as a percentage)<sup id="fnref:3"><a href="#fn:3" class="footnote" rel="footnote" role="doc-noteref">3</a></sup>.</p> <table> <thead> <tr> <th style="text-align: left">Error code</th> <th style="text-align: left">% PDFs reported</th> <th style="text-align: left">Description (from <a href="http://svn.apache.org/repos/asf/pdfbox/trunk/preflight/src/main/java/org/apache/pdfbox/preflight/PreflightConstants.java">Preflight source code</a>)</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">2.4.3</td> <td style="text-align: left">79.5</td> <td style="text-align: left">color space used in the PDF file but the DestOutputProfile is missing</td> </tr> <tr> <td style="text-align: left">7.1</td> <td style="text-align: left">52.5</td> <td style="text-align: left">Invalid metadata found</td> </tr> <tr> <td style="text-align: left">2.4.1</td> <td style="text-align: left">39.1</td> <td style="text-align: left">RGB color space used in the PDF file but the DestOutputProfile isn’t RGB</td> </tr> <tr> <td style="text-align: left">1.2.1</td> <td style="text-align: left">38.8</td> <td style="text-align: left">Error on the object delimiters (obj / endobj)</td> </tr> <tr> <td style="text-align: left">1.4.6</td> <td style="text-align: left">34.3</td> <td style="text-align: left">ID in 1st trailer and the last is different</td> </tr> <tr> <td style="text-align: left">1.2.5</td> <td style="text-align: left">32.1</td> <td style="text-align: left">The length of the stream dictionary and the stream length is inconsistent</td> </tr> <tr> <td style="text-align: left">7.11</td> <td style="text-align: left">31.9</td> <td style="text-align: left">PDF/A Identification Schema not found</td> </tr> <tr> <td style="text-align: left">3.1.2</td> <td style="text-align: left">31.6</td> <td style="text-align: left">Some mandatory fields are missing from the FONT Descriptor Dictionary</td> </tr> <tr> <td style="text-align: left">3.1.3</td> <td style="text-align: left">29.4</td> <td style="text-align: left">Error on the “Font File x” in the Font Descriptor <em>(ed.:font not embedded?)</em></td> </tr> <tr> <td style="text-align: left">3.1.1</td> <td style="text-align: left">27.2</td> <td style="text-align: left">Some mandatory fields are missing from the FONT Dictionary</td> </tr> <tr> <td style="text-align: left">3.1.6</td> <td style="text-align: left">17.1</td> <td style="text-align: left">Width array and Font program Width are inconsistent</td> </tr> <tr> <td style="text-align: left">5.2.2</td> <td style="text-align: left">13</td> <td style="text-align: left">The annotation uses a flag which is forbidden</td> </tr> <tr> <td style="text-align: left">2.4.2</td> <td style="text-align: left">12.8</td> <td style="text-align: left">CMYK color space used in the PDF file but the DestOutputProfile isn’t CMYK</td> </tr> <tr> <td style="text-align: left">1.2.2</td> <td style="text-align: left">12</td> <td style="text-align: left">Error on the stream delimiters (stream / endstream)</td> </tr> <tr> <td style="text-align: left">1.2.12</td> <td style="text-align: left">9.5</td> <td style="text-align: left">The stream uses a filter which isn’t defined in the PDF Reference document</td> </tr> <tr> <td style="text-align: left">1.4.1</td> <td style="text-align: left">9.3</td> <td style="text-align: left">ID is missing from the trailer</td> </tr> <tr> <td style="text-align: left">3.1.11</td> <td style="text-align: left">8.4</td> <td style="text-align: left">The CIDSet entry i mandatory from a subset of composite font</td> </tr> <tr> <td style="text-align: left">1.1</td> <td style="text-align: left">8.3</td> <td style="text-align: left">Header syntax error</td> </tr> <tr> <td style="text-align: left">1.2.7</td> <td style="text-align: left">7.5</td> <td style="text-align: left">The stream uses an invalid filter (The LZW)</td> </tr> <tr> <td style="text-align: left">3.1.5</td> <td style="text-align: left">7.3</td> <td style="text-align: left">Encoding is inconsistent with the Font</td> </tr> <tr> <td style="text-align: left">2.3</td> <td style="text-align: left">6.7</td> <td style="text-align: left">A XObject has an unexpected key defined</td> </tr> <tr> <td style="text-align: left">Exception</td> <td style="text-align: left">6.6</td> <td style="text-align: left">Preflight raised an exception</td> </tr> <tr> <td style="text-align: left">3.1.9</td> <td style="text-align: left">6.1</td> <td style="text-align: left">The CIDToGID is invalid</td> </tr> <tr> <td style="text-align: left">3.1.4</td> <td style="text-align: left">5.7</td> <td style="text-align: left">Charset declaration is missing in a Type 1 Subset</td> </tr> <tr> <td style="text-align: left">7.2</td> <td style="text-align: left">5</td> <td style="text-align: left">Metadata mismatch between PDF Dictionnary and xmp</td> </tr> <tr> <td style="text-align: left">7.3</td> <td style="text-align: left">4.3</td> <td style="text-align: left">Description schema required not embedded</td> </tr> <tr> <td style="text-align: left">2.3.2</td> <td style="text-align: left">4.2</td> <td style="text-align: left">A XObject has an unexpected value for a defined key</td> </tr> <tr> <td style="text-align: left">7.1.1</td> <td style="text-align: left">3.3</td> <td style="text-align: left">Unknown metadata</td> </tr> <tr> <td style="text-align: left">3.3.1</td> <td style="text-align: left">3.1</td> <td style="text-align: left">a glyph is missing</td> </tr> <tr> <td style="text-align: left">1.4.8</td> <td style="text-align: left">2.6</td> <td style="text-align: left">Optional content is forbidden</td> </tr> <tr> <td style="text-align: left">2.2.2</td> <td style="text-align: left">2.4</td> <td style="text-align: left">A XObject SMask value isn’t None</td> </tr> <tr> <td style="text-align: left">1.0.14</td> <td style="text-align: left">2.1</td> <td style="text-align: left">An object has an invalid offset</td> </tr> <tr> <td style="text-align: left">1.4.10</td> <td style="text-align: left">1.6</td> <td style="text-align: left">Last %%EOF sequence is followed by data</td> </tr> <tr> <td style="text-align: left">2.2.1</td> <td style="text-align: left">1.6</td> <td style="text-align: left">A Group entry with S = Transparency is used or the S = Null</td> </tr> <tr> <td style="text-align: left">1</td> <td style="text-align: left">1.6</td> <td style="text-align: left">Syntax error</td> </tr> <tr> <td style="text-align: left">5.2.3</td> <td style="text-align: left">1.5</td> <td style="text-align: left">Annotation uses a Color profile which isn’t the same than the profile contained by the OutputIntent</td> </tr> <tr> <td style="text-align: left">1.0.6</td> <td style="text-align: left">1.2</td> <td style="text-align: left">The number is out of Range</td> </tr> <tr> <td style="text-align: left">5.3.1</td> <td style="text-align: left">1.1</td> <td style="text-align: left">The AP dictionary of the annotation contains forbidden/invalid entries (only the N entry is authorized)</td> </tr> <tr> <td style="text-align: left">6.2.5</td> <td style="text-align: left">1</td> <td style="text-align: left">An explicitly forbidden action is used in the PDF file</td> </tr> <tr> <td style="text-align: left">1.4.7</td> <td style="text-align: left">1</td> <td style="text-align: left">EmbeddedFile entry is present in the Names dictionary</td> </tr> </tbody> </table> <p>This table does look a bit intimidating (but see <a href="http://wiki.opf-labs.org/display/TR/Summary+of+Apache+Preflight+errors">this summary of <em>Preflight</em> errors</a>); nevertheless it is useful to point out a couple of general observations:</p> <ul> <li>Some errors are <em>really</em> common; for instance, error <em>2.4.3</em> is reported for nearly 80% of all <em>PDF</em>s in the corpus!</li> <li>Errors related to color spaces, metadata and fonts are particularly common.</li> <li>File structure errors (1.x range) are reported quite a lot as well. Although I haven’t looked at this in any detail, I expect that for some files these errors truly reflect a deviation from the <em>PDF/A-1</em> profile, whereas in other cases these files may simply not be valid <em>PDF</em> (which would be more serious).</li> <li>About 6.5% of all analysed files raised an exception in <em>Preflight</em>, which could either mean that something is seriously wrong with them, or alternatively it may point to bugs in <em>Preflight</em>.</li> </ul> <h2 id="policy-based-assessment">Policy-based assessment</h2> <p>Although it’s easy to get overwhelmed by the <em>Preflight</em> output above, we should keep in mind here that the ultimate aim of this work is <em>not</em> to validate against <em>PDF/A-1</em>, but to assess arbitrary <em>PDF</em>s against a pre-defined technical profile. This profile may reflect an institution’s low-level preservation policies on the requirements a <em>PDF</em> must meet to be deemed suitable for long-term preservation. In <em>SCAPE</em> such low-level policies are called <em>control policies</em>, and you can find more information on them <a href="http://www.openplanetsfoundation.org/blogs/2013-07-29-scape-creating-machine-understandable-policy-human-readable-policy">here</a> and <a href="http://www.openplanetsfoundation.org/blogs/2013-09-04-control-policies-scape-project">here</a>.</p> <p>To illustrate this, I’ll be using a hypothetical control policy for <em>PDF</em> that is defined by the following objectives:</p> <ol> <li>File must not be encrypted or password protected</li> <li>Fonts must be embedded and complete</li> <li>File must not contain <em>JavaScript</em></li> <li>File must not contain embedded files (i.e. file attachments)</li> <li>File must not contain multimedia content (audio, video, 3-D objects)</li> <li>File should be valid <em>PDF</em></li> </ol> <p><em>Preflight</em>’s output contains all the information that is needed to establish whether each objective is met (except objective 6, which would need a <a href="http://duff-johnson.com/2014/01/24/are-your-documents-readable-how-would-you-know/">full-fledged <em>PDF</em> validator</a>). By translating the above objectives into a set of <a href="http://en.wikipedia.org/wiki/Schematron"><em>Schematron</em></a> rules, it is pretty straightforward to assess each <em>PDF</em> in our dataset against the control policy. If that sounds familiar: this is the same approach that we used earlier for <a href="/2012/09/04/automated-assessment-jp2-against-technical-profile">assessing <em>JP2</em> images against a technical profile</a>. A schema that represents our control policy can be found <a href="https://github.com/openplanets/pdfPolicyValidate/blob/master/schemas/pdf_policy_preflight_test.sch">here</a>. Note that this is only a first attempt, and it may well need some further fine-tuning (more about that later).</p> <h2 id="results-of-assessment">Results of assessment</h2> <p>As a first step I validated all <em>Preflight</em> output files against <a href="https://github.com/openplanets/pdfPolicyValidate/blob/master/schemas/pdf_policy_preflight_test.sch">this schema</a>. The result is rather disappointing:</p> <table> <thead> <tr> <th style="text-align: left">Outcome</th> <th style="text-align: left">Number of files</th> <th style="text-align: left">%</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">Pass</td> <td style="text-align: left">3973</td> <td style="text-align: left">26</td> </tr> <tr> <td style="text-align: left">Fail</td> <td style="text-align: left">11120</td> <td style="text-align: left">74</td> </tr> </tbody> </table> <p>So, only 26% of all <em>PDF</em>s in <em>Govdocs Selected</em> meet the requirements of our control policy! The figure below gives us some further clues as to why this is happening:</p> <p><img src="/images/2014/01/failedAssertions_small.png" alt="Failed assertions graph" /></p> <p>Here each bar represents the occurrences of individual failed tests in our <a href="https://github.com/openplanets/pdfPolicyValidate/blob/master/schemas/pdf_policy_preflight_test.sch">schema</a>.</p> <h2 id="font-errors-galore">Font errors galore</h2> <p>What is clear here is that the majority of failed tests is font-related. The <em>Schematron</em> rules that I used for the assessment currently includes <em>all</em> font errors that are reported by <em>Preflight</em>. Perhaps this is too strict on objective 2 (“<em>Fonts must be embedded and complete</em>”). A particular difficulty here is that it is often hard to envisage the impact of particular font errors on the rendering process. On the other hand, the results are consistent with the outcome of a 2013 survey by the <a href="http://www.pdfa.org/">PDF Association</a>, which showed that its members see fonts as the most challenging aspect of <em>PDF</em>, both for processing and writing (source: <a href="http://duff-johnson.com/wp-content/uploads/2014/01/PDFValidationDreamOrYawn.pdf">this presentation</a> by Duff Johnson). So, the assessment results may simply reflect that font problems are widespread<sup id="fnref:4"><a href="#fn:4" class="footnote" rel="footnote" role="doc-noteref">4</a></sup>. One should also keep in mind that <em>Govdocs selected</em> was created by selecting on unique combinations of file properties from files in <a href="http://digitalcorpora.org/corpora/files">Govdocs1</a>. As a result, one would expect this dataset to be more heterogeneous than most ‘typical’ <em>PDF</em> collections, and this would also influence the results. For instance, the <em>Creating Program</em> selection property could result in a relative over-representation of files that were produced by some crappy creation tool. Whether this is really the case could be easily tested by repeating this analysis for other collections.</p> <h2 id="other-errors">Other errors</h2> <p>Only a small small number of <em>PDF</em>s with encryption, <em>JavaScript</em>, embedded files and multimedia content were detected. I should add here that the occurrence of <em>JavaScript</em> is probably underestimated due to a <a href="https://issues.apache.org/jira/browse/PDFBOX-1754">pending <em>Preflight</em> bug</a>. A major limitation is that there are currently no reliable tools that are able to test overall conformity to <em>PDF</em>. This problem (and a hint at a solution) is also the subject of a recent <a href="http://duff-johnson.com/2014/01/24/are-your-documents-readable-how-would-you-know/">blog post by Duff Johnson</a>. In the current assessment I’ve taken the occurrence of <em>Preflight</em> exceptions (and general processing errors) as an indicator for non-validity. This is a pretty crude approximation, because some of these exceptions may simply indicate a bug in <em>Preflight</em> (rather than a faulty <em>PDF</em>). One of the next steps will therefore be a more in-depth look at some of the <em>PDF</em>s that caused an exception.</p> <h2 id="conclusions">Conclusions</h2> <p>These preliminary results show that policy-based assessment of <em>PDF</em> is possible using a combination of <em>Apache Preflight</em> and <em>Schematron</em>. However, dealing with font issues appears to be a particular challenge. Also, the lack of reliable tools to test for overall conformity to <em>PDF</em> (e.g. <a href="http://acroeng.adobe.com/PDFReference/ISO32000/PDF32000-Adobe.pdf">ISO 32000</a>) is still a major limitation. Another limitation of this analysis is the lack of ground truth, which makes it difficult to assess the accuracy of the results.</p> <h2 id="demo-script-and-data-downloads">Demo script and data downloads</h2> <p>For those who want to have a go at the analyses that I’ve presented here, I’ve created a simple <a href="https://github.com/openplanets/pdfPolicyValidate">demo script here</a>. The raw output data of the <em>Govdocs selected</em> corpus can be found <a href="https://github.com/openplanets/preflightGovdocsSelected">here</a>. This includes all <em>Preflight</em> files, the <em>Schematron</em> output and the error counts. A download link for the <em>Govdocs selected</em> corpus can be found at the bottom of <a href="http://www.openplanetsfoundation.org/blogs/2012-07-26-1-million-21000-reducing-govdocs-significantly">this blog post</a>.</p> <h2 id="acknowledgements">Acknowledgements</h2> <p><em>Apache Preflight</em> developers Eric Leleu, Andreas Lehmkühler and Guillaume Bailleul are thanked for their support and prompt response to my questions and bug reports.</p> <h2 id="related-blog-posts">Related blog posts</h2> <ul> <li><a href="/2012/12/19/identification-pdf-preservation-risks-apache-preflight-first-impression">Identification of PDF preservation risks with Apache Preflight: a first impression</a></li> <li><a href="/2013/07/25/identification-pdf-preservation-risks-sequel">Identification of PDF preservation risks: the sequel</a></li> <li><a href="http://duff-johnson.com/2014/01/24/are-your-documents-readable-how-would-you-know/">Are your documents readable? How would you know? (Duff Johnson)</a></li> <li><a href="http://www.openplanetsfoundation.org/blogs/2012-07-26-1-million-21000-reducing-govdocs-significantly">From 1 Million to 21,000: Reducing Govdocs Significantly (Dave Tarrant)</a></li> <li><a href="http://www.openplanetsfoundation.org/blogs/2013-07-29-scape-creating-machine-understandable-policy-human-readable-policy">Creating machine understandable policy from human readable policy (Catherine Jones)</a></li> <li><a href="http://www.openplanetsfoundation.org/blogs/2013-09-04-control-policies-scape-project">Control Policies in the SCAPE Project (Sean Bechhofer)</a></li> <li><a href="http://wiki.opf-labs.org/display/TR/Portable+Document+Format">PDF on the OPF File Format Risk Registry</a></li> </ul> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2014/01/27/identification-pdf-preservation-risks-analysis-govdocs-selected-corpus/">Open Preservation Foundation blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>This was already suggested by <a href="http://wiki.opf-labs.org/display/TR/Analysis+of+Acrobat+Engineering+PDFs+with+Acrobat+Preflight+and+Apache+Preflight">this re-analysis of the <em>Acrobat Engineering</em> files </a> that I did in November. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2"> <p>This selection was only based on file extension, which introduces the possibility that some of these files aren’t really <em>PDF</em>s. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:3"> <p>Errors that were reported for less than 1% of all analysed <em>PDF</em>s are not included in the table. <a href="#fnref:3" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:4"> <p>In addition to this, it seems that <em>Preflight</em> <a href="https://issues.apache.org/jira/browse/PDFBOX-1864">sometimes fails to detect fonts that are not embedded</a>, so the number of <em>PDF</em>s with font issues may be even greater than this test suggests. <a href="#fnref:4" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2014/01/27/identification-pdf-preservation-risks-analysis-govdocs-selected-corpus https://bitsgalore.org/2014/01/27/identification-pdf-preservation-risks-analysis-govdocs-selected-corpus 2014-01-27T00:00:00+01:00 <p>My previous blog <a href="/2013/09/30/assessing-file-format-risks-searching-bigfoot"><em>Assessing file format risks: searching for Bigfoot?</em></a> resulted in some interesting feedback from a number of people. There was a particularly elaborate response from Ross Spencer, and I originally wanted to reply to that directly using the comment fields. However, my reply turned out to be a bit more lengthy than I meant to, so I decided to turn it into a separate blog entry.</p> <!-- more --> <h2 id="numbers-first">Numbers first?</h2> <p>Ross overall point is that <a href="http://www.openplanetsfoundation.org/comment/511#comment-511">we need the numbers first</a>; he makes a plea for collecting more format-related data, and adding numbers to these. Although these data do not directly translate into risks, Ross argues that it might be possible to use them to address format risks at a later stage. This may look like a sensible approach at first glance, but on closer inspection there’s a pretty fundamental problem, which I’ll try to explain below. To avoid any confusion here, I will be speaking of “format risk” here in the sense used by <a href="http://purl.pt/24107/1/iPres2013_PDF/A%20Risk%20Analysis%20of%20File%20Formats%20for%20Preservation%20Planning.pdf">Graf &amp; Gordea</a>, which follows from the idea of “institutional obsolescence” (which is probably worth a blog post by itself, but I won’t go into this here).</p> <h2 id="the-risk-model">The risk model</h2> <p>Graf &amp; Gordea define institutional obsolescence in terms of “the additional effort required to render a file beyond the capability of a regular PC setup in particular institution”. Let’s call this effort <em>E</em>. Now the aim is to arrive at an index that has some predictive power of <em>E</em>. Let’s call this index <em>R_E</em>. For the sake of the argument it doesn’t matter how <em>R_E</em> is defined precisely, but it’s reasonable to assume it will be proportional to <em>E</em> (i.e. as the effort to render a file increases, so does the risk):</p> <p><em>R_E</em> ∝ <em>E</em></p> <p>The next step is to find a way to estimate <em>R_E</em> (the dependent variable) as a function of a set of potential predictor variables:</p> <p><em>R_E</em> = <em>f</em>(<em>S</em>, <em>P</em>, <em>C</em>, … )</p> <p>where <em>S</em> = software count, <em>P</em> = popularity, <em>C</em> = complexity, and so on. To establish the predictor function we have two possibilities:</p> <ol> <li>use a statistical approach (e.g. multiple regression or something more sophisticated);</li> <li>use a conceptual model that is based on prior knowledge of how the predictor variables affect <em>R_E</em>.</li> </ol> <p>The first case (statistical approach) is only feasible if we have actual data on <em>E</em>. For the second case we also need observations on <em>E</em>, if only to be able to say anything about the model’s ability to predict <em>R_E</em> (verification).</p> <h2 id="no-observed-data-on-e">No observed data on <em>E</em>!</h2> <p>Either way, the problem here is that there’s an almost complete lack of any data on <em>E</em>. Although we may have a handful of isolated ‘war stories’, these don’t even come close to the amount of data that would be needed to support any risk model, no matter whether it is purely statistical or based on an underlying conceptual model<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>. So how are we going to model a quantity for which we do not have any observed data in the first place? Or am I overlooking something here?</p> <p>Looking at Ross’s suggestions for collecting more data, all of the examples he provides fall into the <em>potential</em> (!) predictor variables category. For instance, prompted by my observation on compression in <em>PDF</em>, Ross suggests to start analysing large collections of <em>PDF</em>s to establish patterns on the occurrence of various types of compression (and other features), and attach numbers to them. Ross acknowledges that such numbers by themselves don’t tell you if <em>PDF</em> is “riskier” than another format, but he argues that:</p> <blockquote> <p>once we’ve got them [the numbers], subject matter experts and maybe some of those mathematical types with far greater statistics capability than my own might be able to work with us to do something just a little bit clever with them.</p> </blockquote> <p>Aside from the fact that it’s debatable whether, in practical terms, the use of compression is really a risk (is there any evidence to back up this claim?), there’s a more fundamental issue here. Bearing in mind that, ultimately, the thing we’re <em>really</em> interested in here is <em>E</em>, how could collecting more data on potential predictor variables of <em>E</em> ever help here <em>in the near absence of any actual data</em> on <em>E</em>? No amount of clever maths or statistics can compensate for that! Meanwhile, ongoing work on the prediction of <em>E</em> mainly seems to be focused on the collection, aggregation and analysis of potential predictor variables (which is also illustrated by Ross’s suggestions), even though the purpose of these efforts remains largely unclear.</p> <p>Within this context I was quite intrigued by the grant proposal mentioned by <a href="http://www.openplanetsfoundation.org/comment/513#comment-513">Andrea Goethals</a> which, from the description, looks like an actual (and quite possibly the first) attempt at the systematic collection of data on <em>E</em> (although like <a href="http://www.openplanetsfoundation.org/comment/513#comment-513">Andy Jackson said here</a> I’m also wondering whether this may be too ambitious).</p> <h2 id="obsolescence-related-risks-versus-format-instance-risks">Obsolescence-related risks versus format instance risks</h2> <p>On a final note, Ross makes the following remark about the role of tools:</p> <blockquote> <p>[W]ith tools such as Jpylyzer we have such powerful ways of measuring formats - and more and more should appear over time.</p> </blockquote> <p>This is true to some extent, but a tool like <em>jpylyzer</em> only provides information on format <em>instances</em> (i.e. features of <em>individual files</em>); it doesn’t say anything about preservation risks of the JP2 format <em>in general</em>. The same applies to tools that are are able to <a href="/2013/07/25/identification-pdf-preservation-risks-sequel">detect features in individual PDF files</a> that are risky from a long-term preservation point of view. Such risks affect file instances of <em>current</em> formats, and this is an area that is covered by the <a href="http://wiki.opf-labs.org/display/TR/OPF+File+Format+Risk+Registry">OPF File Format Risk Registry</a> that is being developed within SCAPE (it only covers a limited number of formats). They are largely unrelated to (institutional) format obsolescence, which is the domain that is being addressed by <a href="http://ffma.ait.ac.at:8080/preservation-riskmanagement/"><em>FFMA</em></a>. This distinction is important, because both types of risks need to be tackled in fundamentally different ways, using different tools, methods and data. Also, by not being clear about which risks are being addressed, we may end up not using our data in the best possible way. For example, Ross’s suggestion on compression in <em>PDF</em> entails (if I’m understanding him correctly) the analysis of large volumes of <em>PDF</em>s in order to gather statistics on the use of different compression types. Since such statistics say little about individual file instances, a more practically useful approach might be to profile individual files instances for ‘risky’ features.</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2013/10/08/measuring-bigfoot/">Open Preservation Foundation blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>On a side note even conceptual models often need to be fine-tuned against observed data, which can make them pretty similar to statistically-derived models. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2013/10/08/measuring-bigfoot https://bitsgalore.org/2013/10/08/measuring-bigfoot 2013-10-08T00:00:00+02:00 <p>Last week someone pointed my attention to a recent <em>iPres</em> paper by Roman Graf and Sergiu Gordea titled “<a href="http://purl.pt/24107/1/iPres2013_PDF/A%20Risk%20Analysis%20of%20File%20Formats%20for%20Preservation%20Planning.pdf">A Risk Analysis of File Formats for Preservation Planning</a>”. The authors propose a methodology for assessing preservation risks for file formats using information in publicly available information sources. In short, their approach involves two stages:</p> <ol> <li> <p>Collect and aggregate information on file formats from data sources such as <a href="http://www.nationalarchives.gov.uk/PRONOM">PRONOM</a>, <a href="http://www.freebase.com/">Freebase</a> and <a href="http://dbpedia.org">DBPedia</a></p> </li> <li> <p>Use this information to compute scores for a number of pre-defined risk factors (e.g. the number of software applications that support the format, the format’s complexity, its popularity, and so on). A weighted average of these individual scores then gives an overall risk score.</p> </li> </ol> <p>This has resulted in the “File Format Metadata Aggregator” (<em>FFMA</em>), which is an expert system aimed at establishing a “<em>well structured knowledge base with defined rules and scored metrics that is intended to provide decision making support for preservation experts</em>”.</p> <!-- more --> <p>The paper caught my attention for two reasons: first, a number of years ago some colleagues at the <em>KB</em> developed a <a href="http://www.kb.nl/hrd/dd/dd_links_en_publicaties/publicaties/KB_file_format_evaluation_method_27022008.pdf">method for evaluating file formats</a> that is based on a similar way of looking at preservation risks. Second, just a few weeks ago I found out that the University of North Carolina is also working on <a href="http://www.ils.unc.edu/digccurr/ct_poster/Ryan.pdf">a method for assessing “File Format Endangerment”</a> which seems to be following a similar approach. Now let me start by saying that I’m extremely uneasy about assessing preservation risks in this way. To a large extent this is based on experiences with the <em>KB</em>-developed method, which is similar to the assessment method behind <em>FFMA</em>. I will use the remainder of this blog post to explain my reservations.</p> <h2 id="criteria-are-largely-theoretical">Criteria are largely theoretical</h2> <p><em>FFMA</em> implicitly assumes that it is possible to assess format-specific preservation risks by evaluating formats against a list of pre-defined criteria. In this regard it is similar to (and builds on) the logic behind, to name but two examples, <a href="http://www.digitalpreservation.gov/formats/sustain/sustain.shtml">Library of Congress’ Sustainability Factors</a> and <a href="http://www.nationalarchives.gov.uk/documents/selecting-file-formats.pdf">UK National Archives’ format selection criteria</a>. However, these criteria are largely based on theoretical considerations, without being backed up by any empirical data. As a result, their predictive value is largely unknown.</p> <h2 id="appropriateness-of-measures">Appropriateness of measures</h2> <p>Even if we agree that criteria such as software support and the existence of migration paths to some alternative format are important, how exactly do we measure this? It is pretty straightforward to simply count the number of supporting software products or migration paths, but this says nothing about their <em>quality</em> or suitability for a specific task. For example, <em>PDF</em> is supported by a plethora of software tools, yet it is well known that few of them support <em>every</em> feature of the format (possibly even none, with the exception of Adobe’s implementation). Here’s another example: quite a few (open-source) software tools support the <em>JP2</em> format, but for this many of them (including <em>ImageMagick</em> and <em>GraphicsMagick</em>) rely on <a href="http://www.ece.uvic.ca/~frodo/jasper/"><em>JasPer</em></a>, a JPEG 2000 library that is notorious for its poor performance and stability. So even if a format is supported by lots of tools, this will be of little use if the quality of those tool are poor.</p> <h2 id="risk-model-and-weighting-of-scores">Risk model and weighting of scores</h2> <p>Just as the employed criteria are largely theoretical, so is the computation of the risk scores, the weights that are assigned to each risk factor, and they way the individual scores are aggregated into an overall score. The latter is computed as the weighted sum of all individual scores, which means that a poor score on, for example, <em>Software Count</em> can be compensated by a high score on other factors. This doesn’t strike me as very realistic, and it is also at odds with e.g. <a href="http://blog.dshr.org/2009/01/are-format-specifications-important-for.html">David Rosenthal’s view</a> of formats with open source renderers being immune from format obsolescence.</p> <h2 id="accuracy-of-underlying-data">Accuracy of underlying data</h2> <p>A cursory look at the <a href="http://ffma.ait.ac.at:8080/preservation-riskmanagement/">web service implementation of <em>FFMA</em></a> revealed some results that make me wonder about the data that are used for the risk assessment. According to <em>FFMA</em>:</p> <ul> <li><a href="http://ffma.ait.ac.at:8080/preservation-riskmanagement/rest/loddataanalysis/html/riskscorereport?name=png&amp;configName=&amp;classificationName="><em>PNG</em></a>, <a href="http://ffma.ait.ac.at:8080/preservation-riskmanagement/rest/loddataanalysis/html/riskscorereport?name=jpg&amp;configName=&amp;classificationName="><em>JPG</em></a> and <a href="http://ffma.ait.ac.at:8080/preservation-riskmanagement/rest/loddataanalysis/html/riskscorereport?name=gif&amp;configName=&amp;classificationName=">GIF</a> are uncompressed formats (they’re not!);</li> <li><a href="http://ffma.ait.ac.at:8080/preservation-riskmanagement/rest/loddataanalysis/html/riskscorereport?name=pdf&amp;configName=&amp;classificationName="><em>PDF</em></a> is <em>not</em> a compressed format (in reality text in <em>PDF</em> nearly always uses <a href="http://en.wikipedia.org/wiki/DEFLATE">Flate compression</a>, whereas a <a href="http://www.prepressure.com/pdf/basics/compression">whole array of compression methods</a> may be used for images);</li> <li><a href="http://ffma.ait.ac.at:8080/preservation-riskmanagement/rest/loddataanalysis/html/riskscorereport?name=jp2&amp;configName=&amp;classificationName="><em>JP2</em></a> is not supported by <em>any</em> software (Software Count=0!), it doesn’t have a <em>MIME</em> type, it is frequently used, and it is supported by web browsers (all wrong, although arguably <em>some</em> browser support exists if you account for external plugins);</li> <li><a href="http://ffma.ait.ac.at:8080/preservation-riskmanagement/rest/loddataanalysis/html/riskscorereport?name=jpf&amp;configName=&amp;classificationName="><em>JPX</em></a> is <em>not</em> a compressed format and it is less complex than <em>JP2</em> (in reality it is an extension of <em>JP2</em> with added complexity).</li> </ul> <p>To some extent this may also explain the peculiar ranking of formats in Figure 6 of the paper, which marks down <em>PDF</em> and <em>MS Word</em> (!) as formats with a lower risk than <em>TIFF</em> (<em>GIF</em> has the overall lowest score).</p> <h2 id="what-risks">What risks?</h2> <p>It is important that the concept of ‘preservation risk’ as addressed by <em>FFMA</em> is closely related to (and has its origins in) the idea of formats becoming obsolete over time. This idea is controversial, and the authors do acknowledge this by defining preservation risks in terms of the “<em>additional effort required to render a file beyond the capability of a regular PC setup in [a] particular institution</em>”. However, in its current form <em>FFMA</em> only provides generalized information about formats, without addressing specific risks <em>within</em> formats. A good example of this is <em>PDF</em>, which may contain various features that are <a href="/2012/07/26/pdf-inventory-long-term-preservation-risks">problematic</a> for long-term preservation. Also note how <em>PDF</em> is marked as a <a href="http://ffma.ait.ac.at:8080/preservation-riskmanagement/rest/loddataanalysis/html/riskscorereport?name=pdf&amp;configName=&amp;classificationName="><em>low-risk</em> format</a>, despite the fact that it can be a container for <em>JP2</em> which is considered <a href="http://ffma.ait.ac.at:8080/preservation-riskmanagement/rest/loddataanalysis/html/riskscorereport?name=jp2&amp;configName=&amp;classificationName="><em>high-risk</em></a>. So doesn’t that imply that a <em>PDF</em> that contains <em>JPEG 2000</em> compressed images is at a higher risk?</p> <h2 id="encyclopedia-replacing-expertise">Encyclopedia replacing expertise?</h2> <p>A possible response to the objections above would be to refine <em>FFMA</em>: adjust the criteria, modify the way the individual risk scores are computed, tweak the weights, change the way the overall score is computed from the individual scores, and improve the underlying data. Even though I’m sure this could lead to some improvement, I’m eerily reminded here of <a href="http://www.openplanetsfoundation.org/blogs/2013-09-13-registries-we-need">this recent <strike>rant</strike> blog post</a> by Andy Jackson, in which he shares his concerns about the archival community’s preoccupation with format, software, and hardware registries. Apart from the question whether the existing registries are actually helpful in solving real-world problems, Jackson suggests that:</p> <blockquote> <p>Maybe we don’t know what information we need?</p> </blockquote> <p>and:</p> <blockquote> <p>Maybe we don’t even know who or what we are building registries for?</p> </blockquote> <p>He also wonders:</p> <blockquote> <p>Are we trying to replace imagination and expertise with an encyclopedia?</p> </blockquote> <p>I think these comments apply equally well to the recurring attempts at reducing format-specific preservation risks to numerical risk factors, scores and indices. This approach simply doesn’t do justice to the subtleties of practical digital preservation. Worse still, I see a potential danger of non-experts taking the results from such expert systems at face value, which can easily lead to ill-judged decisions. Here’s an example.</p> <h2 id="kb-example">KB example</h2> <p>About five years some colleagues at the <em>KB</em> developed a “quantifiable file format risk assessment method”, which is described in <a href="http://www.kb.nl/hrd/dd/dd_links_en_publicaties/publicaties/KB_file_format_evaluation_method_27022008.pdf">this report</a>. This method was <a href="http://www.kb.nl/hrd/dd/dd_links_en_publicaties/publicaties/Alternative_File_Formats_for_Storing_Masters_2_1.pdf">applied</a> to decide which still image format was the best candidate to replace the then-current format for digitisation masters. The outcome of this was used to justify a change from uncompressed <em>TIFF</em> to <em>JP2</em>. It was only much later that we found out about a host of practical and standard-related problems with the format, some of which are discussed <a href="http://jpeg2000wellcomelibrary.blogspot.nl/2010/12/guest-post-ensuring-suitability-of-jpeg.html">here</a> and <a href="http://www.dlib.org/dlib/may11/vanderknijff/05vanderknijff.html">here</a>. <em>None</em> of these problems were accounted for by the earlier risk assessment method (and I have a hard time seeing how they ever could be)! The risk factor approach of <em>GGMA</em> is covering similar ground, and this adds to my scepticism about addressing preservation risks in this manner.</p> <h2 id="final-thoughts">Final thoughts</h2> <p>Taking into account the problems mentioned in this blog post, I have a hard time seeing how scoring models such as the one used by <em>FFMA</em> would help in solving practical digital preservation issues. It also makes me wonder why this idea keeps on being revisited over and over again. Similar to the format registry situation, is this perhaps another manifestation of the “<em>trying to replace imagination and expertise with an encyclopedia</em> phenomenon? What exactly is the point of classifying or ranking formats according to perceived preservation “risks” if these “risks” are largely based on theoretical considerations, and are so general that they say next to nothing about individual file (format) instances? Isn’t this all a bit like <a href="http://www.searchingforbigfoot.com/">searching for Bigfoot</a>? Wouldn’t the time and effort involved in these activities be better spent on trying to solve, document and publish concrete format-related problems and their solutions? Some examples can be found <a href="http://unsustainableideas.wordpress.com/2012/10/15/ppt-4-adventure-learning/">here</a> (accessing old Powerpoint 4 files), <a href="http://notepad.benfinoradin.info/2013/09/12/it-takes-a-village-to-save-a-hard-drive/">here</a> (recovering the contents of an old Commodore Amiga hard disk), <a href="http://anjackson.github.io/keeping-codes/experiments/BBC%20Micro%20Data%20Recovery.html">here</a> (BBC Micro Data Recovery), or even <a href="http://wiki.opf-labs.org/display/TR/OPF+File+Format+Risk+Registry">here</a> (problems with contemporary formats)?</p> <p>I think there could also be a valuable role here for some of the <em>FFMA</em>-related work in all this: the aggregation component of <em>FFMA</em> looks really useful for the automatic discovery of, for example, software applications that are able to read a specific format, and this could be could be hugely helpful in solving real-world preservation problems.</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2013/09/30/assessing-file-format-risks-searching-bigfoot/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2013/09/30/assessing-file-format-risks-searching-bigfoot https://bitsgalore.org/2013/09/30/assessing-file-format-risks-searching-bigfoot 2013-09-30T00:00:00+02:00 <p>Like many other organisations that are using JPEG 2000, the KB produces two representations of most of its digitised content (newspapers, books, periodicals):</p> <ul> <li>a high-quality, losslessly compressed JP2 that is the archival master;</li> <li>a lesser-quality, lossily compressed JP2 that is used as an access image (this is used for e.g. our <a href="https://resolver.kb.nl/resolve?urn=ddd:010620674:mpeg21:a0201">newspapers website</a>).</li> </ul> <p>The majority of our digitisation work is contracted out to external suppliers, and both master and access images are typically derived from from a parent (TIFF) image, which is converted to JP2 using the settings for master and access images, respectively. This means that we’re not currently using the archival masters for producing derived images. However, there may be a need for this at some point in the future. For instance, we may need higher quality access images, or access images that give better performance in our access environment. Because of this, I was asked to take a further look into ways to derive access JP2s directly from our archival masters.</p> <p>In this blog post I’ll be sharing some preliminary findings of this work, which may be of interest to other JPEG 2000 practitioners as well. All images and test results that I’ll be showing along the way are available from this <a href="https://github.com/bitsgalore/JP2AccessGeneration">Github repository</a>, so you can have a go at these data yourself, if you’re so inclined.</p> <!-- more --> <h2 id="masters-vs-access-images">Masters vs access images</h2> <p>To better understand the remainder of this blog post it is helpful to outline the differences between our masters and our access images. The tables below list the encoding-related specifications of both.</p> <h3 id="specifications-master">Specifications master</h3> <table> <thead> <tr> <th style="text-align: left">Parameter</th> <th style="text-align: left">Value</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">File format</td> <td style="text-align: left">JP2 (JPEG 2000 Part 1)</td> </tr> <tr> <td style="text-align: left">Compression type</td> <td style="text-align: left">Lossless (reversible 5-3 wavelet filter)</td> </tr> <tr> <td style="text-align: left">Colour transform</td> <td style="text-align: left">Yes (only for colour images)</td> </tr> <tr> <td style="text-align: left">Number of decomposition levels</td> <td style="text-align: left">5</td> </tr> <tr> <td style="text-align: left">Progression order</td> <td style="text-align: left">RPCL</td> </tr> <tr> <td style="text-align: left">Tile size</td> <td style="text-align: left">1024 x 1024</td> </tr> <tr> <td style="text-align: left">Code block size</td> <td style="text-align: left">64 x 64 (2⁶ x 2⁶)</td> </tr> <tr> <td style="text-align: left">Number of quality layers</td> <td style="text-align: left">1</td> </tr> <tr> <td style="text-align: left">Error resilience</td> <td style="text-align: left">Start-of-packet headers; end-of-packet headers; segmentation symbols</td> </tr> </tbody> </table> <h3 id="specifications-access">Specifications access</h3> <table> <thead> <tr> <th style="text-align: left">Parameter</th> <th style="text-align: left">Value</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">File format</td> <td style="text-align: left">JP2 (JPEG 2000 Part 1)</td> </tr> <tr> <td style="text-align: left">Compression type</td> <td style="text-align: left">Lossy (irreversible 9-7 wavelet filter)</td> </tr> <tr> <td style="text-align: left">Colour transform</td> <td style="text-align: left">Yes (only for colour images)</td> </tr> <tr> <td style="text-align: left">Number of decomposition levels</td> <td style="text-align: left">5</td> </tr> <tr> <td style="text-align: left">Progression order</td> <td style="text-align: left">RPCL</td> </tr> <tr> <td style="text-align: left">Tile size</td> <td style="text-align: left">1024 x 1024</td> </tr> <tr> <td style="text-align: left">Code block size</td> <td style="text-align: left">64 x 64 (2⁶ x 2⁶)</td> </tr> <tr> <td style="text-align: left">Precinct size</td> <td style="text-align: left">256 x 256 (2⁸) for 2 highest resolution levels; 128 x 128 (2⁷) for remaining resolution levels</td> </tr> <tr> <td style="text-align: left">Number of quality layers</td> <td style="text-align: left">8</td> </tr> <tr> <td style="text-align: left">Target compression ratio layers</td> <td style="text-align: left">2560:1 [1] ; 1280:1 [2] ; 640:1 [3] ; 320:1 [4] ; 160:1 [5] ; 80:1 [6] ; 40:1 [7] ; 20:1 [8]</td> </tr> <tr> <td style="text-align: left">Error resilience</td> <td style="text-align: left">Start-of-packet headers; end-of-packet headers; segmentation symbols</td> </tr> </tbody> </table> <p>The main differences between the two are:</p> <ol> <li>access images are compressed lossily (reduced file size), whereas lossless compression is used for the masters;</li> <li>access images contain quality layers (enables progressive decoding), whereas the masters don’t;</li> <li>access images use precincts (optimises performance while panning across zoomed-in regions), which aren’t used in the masters either.</li> </ol> <h2 id="methods">Methods</h2> <p>So, the central question here is: if we have an image that was encoded according to the master specifications, how can we derive an image from this that conforms to our access specifications? To find out, I did a number of tests on the image <em>balloon_master.jp2</em>, which was created according to the KB’s master specifications. It looks like this (surprise, surprise!):</p> <p><img src="/images/2013/08/balloon_lossless.png" alt="Balloon, master" /></p> <p>I tried to derive an access image from this master using 2 popular JPEG 2000 software toolkits:</p> <ul> <li><a href="http://www.kakadusoftware.com/">Kakadu</a> (v 7.2.2)</li> <li><a href="http://www.aware.com/imaging/jpeg2000sdk.html">Aware JPEG 2000 SDK</a> (v 3.19.0.0)</li> </ul> <p>For both software packages I limited myself to using only the pre-compiled binaries (i.e. the <em>kdu_..</em> demo tools for Kakadu, and the <em>j2kdriver</em> command-line tool for Aware).</p> <h2 id="kakadu">Kakadu</h2> <p>Kakadu’s <em>kdu_compress</em> tool doesn’t accept any of the JPEG 2000 formats as <em>input</em>; however, it does include a <em>kdu_transcode</em> tool which is capable of a wide array of reformatting operations. I should add here that <em>kdu_transcode</em> is primarily intended as a demo tool that showcases Kakadu’s codestream reformatting capabilities, and it doesn’t produce output in the JP2 format (for a detailed explanation by Kakadu’s author look <a href="http://tech.groups.yahoo.com/group/kakadu_jpeg2000/message/6777">here</a>)<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>.</p> <p>However, <em>kdu_transcode</em> is capable of wrapping output in a <em>JPX</em> container (which can be made <em>JP2</em>-compatible), so this is what I used for these tests. To keep things simple, I started out by instructing the tool to create an output image with a 20:1 compression ratio (ignoring any of the layer / precinct requirements). For an RGB image with 8 bits/component this corresponds to an equivalent bitrate of 1.2, so I ended up with the following command line:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kdu_transcode <span class="nt">-i</span> balloon_master.jp2 <span class="se">\</span> <span class="nt">-o</span> balloon_access_kdu.jpf <span class="se">\</span> <span class="nt">-jpx_layers</span> sRGB,0,1,2 <span class="se">\</span> <span class="nv">Sprofile</span><span class="o">=</span>PROFILE2 <span class="nt">-rate</span> 1.2 </code></pre></div></div> <p>The resulting output image did have the expected size, but opening it in an image viewer revealed a problem:</p> <p><img src="/images/2013/08/balloon_access_kdu.png" alt="Balloon, transcoded by Kakadu" /></p> <p>Compared to the source image, most of the colour information has gone, resulting in a representation that is largely grayscale. The reason behind this seemingly unexpected result is fairly simple: when <em>kdu_transcode</em> creates the derived (lower quality) image, it does so by discarding some of the information that makes up the source image. In other words, it doesn’t decode and recompress the image, but instead re-arranges the compressed image data (which is a very fast process). For a source image with multiple quality layers, the result would be largely equivalent to discarding some of the highest quality layers. However, our source image only has one single quality layer, so this isn’t possible here. Instead, we end up with a result in which most of the colour information is missing (my guess is that the exact behaviour in such cases also depends on the progression order that was used for encoding the source image). Importantly, this is <em>not</em> a flaw of the tool, but simply a consequence of the way the source image was formatted upon its creation.</p> <h2 id="aware">Aware</h2> <p>Aware’s <em>j2kdriver</em> tool supports encoding, decoding and reformating of JP2 images. I used the following command-line in an attempt to create a lossy access image (note that he <em>-w</em> switch sets the transformation to irreversible 9-7 wavelet, and the <em>-R</em> switch sets the target compression ratio to 20:1):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>j2kdriver <span class="nt">-i</span> balloon_master.jp2 <span class="se">\</span> <span class="nt">-R</span> 20 <span class="nt">-w</span> <span class="se">\</span> I97 <span class="se">\</span> <span class="nt">-t</span> JP2 <span class="se">\</span> <span class="nt">-o</span> balloon_access_aw.jp2 </code></pre></div></div> <p>This produced an output image that has the same size as the master! Similar to Kakadu’s <em>kdu_transcode</em> tool, <em>j2kdriver</em> makes no attempt at decoding and recompressing the source image in this case. However, the Aware tool does have a number of reformatting options, including one that allows you to discard quality layers. Needless to say, as the source image contains only one quality layer, this isn’t of much use in this case.</p> <h2 id="optimising-the-archival-masters-for-access-generation">Optimising the archival masters for access generation</h2> <p>In order to produce access images from our current archival masters, we would need to fully decode the source images and then recompress them. Even though this is perfectly possible (e.g. we could simply convert each JP2 to TIFF and then compress that back to lossy JP2), this is both awkward and computationally expensive. A more elegant approach would be to take advantage of JPEG 2000’s ability to include multiple quality layers. We’re already using quality layers in our existing access images, but this is mainly to optimise performance for access. However, we can also define quality layers in the preservation masters, and we can do this in such a way that a subset of all the quality layers in the master become equivalent to the access image. Access images can then be generated by simply discarding one or more quality layers in the preservation master, without any need for re-compressing the whole image. Visually, this results in the following situation:</p> <p><img src="/images/2013/08/layers.png" alt="Layers, master vs access" /></p> <p>This is also the approach that Rob Buckley suggested in this <a href="http://wellcomelibrary.org/assets/wtx056572.pdf">2009 report for the Wellcome Library</a>. In this case we have a losslessly compressed master with 11 quality layers. Access images at a 20:1 compression ratio can then be derived by simply discarding the highest 3 quality layers.</p> <h2 id="making-it-work">Making it work</h2> <p>To make this all work, I first optimised the specifications of the preservation masters by incorporating the quality layer definitions from our access specifications, adding 3 further quality layers to accommodate for the higher quality that is produced by lossless compression. I also added precinct definitions, since we’re using those for access as well. This resulted in the following profile:</p> <table> <thead> <tr> <th style="text-align: left">Parameter</th> <th style="text-align: left">Value</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">File format</td> <td style="text-align: left">JP2 (JPEG 2000 Part 1)</td> </tr> <tr> <td style="text-align: left">Compression type</td> <td style="text-align: left">Lossless (reversible 5-3 wavelet filter)</td> </tr> <tr> <td style="text-align: left">Colour transform</td> <td style="text-align: left">Yes (only for colour images)</td> </tr> <tr> <td style="text-align: left">Number of decomposition levels</td> <td style="text-align: left">5</td> </tr> <tr> <td style="text-align: left">Progression order</td> <td style="text-align: left">RPCL</td> </tr> <tr> <td style="text-align: left">Tile size</td> <td style="text-align: left">1024 x 1024</td> </tr> <tr> <td style="text-align: left">Code block size</td> <td style="text-align: left">64 x 64 (2⁶ x 2⁶)</td> </tr> <tr> <td style="text-align: left">Precinct size</td> <td style="text-align: left">256 x 256 (2⁸) for 2 highest resolution levels; 128 x 128 (2⁷) for remaining resolution levels</td> </tr> <tr> <td style="text-align: left">Number of quality layers</td> <td style="text-align: left">11</td> </tr> <tr> <td style="text-align: left">Target compression ratio layers</td> <td style="text-align: left">2560:1 [1] ; 1280:1 [2] ; 640:1 [3] ; 320:1 [4] ; 160:1 [5] ; 80:1 [6] ; 40:1 [7] ; 20:1 [8] ; 10:1 [9] ; 5:1 [10] ; 2.5:1 [11]*</td> </tr> <tr> <td style="text-align: left">Error resilience</td> <td style="text-align: left">Start-of-packet headers; end-of-packet headers; segmentation symbols</td> </tr> </tbody> </table> <p>Then I went back to that dreaded balloon image TIFF, and created a new lossless master that follows the optimised specifications (<em>balloon_master_layers_precincts.jp2</em>).</p> <h2 id="generating-the-access-image">Generating the access image</h2> <p>Kakadu’s <em>kdu_transcode</em> doesn’t allow you to explicitly discard quality layers, but the <code class="language-plaintext highlighter-rouge">-rate</code> switch can be used to select an output bitrate, which has pretty much the same effect. So can can simply set all parameters to identical values as in our earlier example (remember that the 1.2 bitrate is equivalent to a compression ratio of 20:1 for an RGB image):</p> <h3 id="kakadu-1">Kakadu</h3> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kdu_transcode <span class="nt">-i</span> balloon_master_layers_precincts.jp2 <span class="se">\</span> <span class="nt">-o</span> balloon_access_precincts_kdu.jpf <span class="se">\</span> <span class="nt">-jpx_layers</span> sRGB,0,1,2 <span class="se">\</span> <span class="nv">Sprofile</span><span class="o">=</span>PROFILE2 <span class="se">\</span> <span class="nt">-rate</span> 1.2 </code></pre></div></div> <p>In contrast to our earlier test, the resulting image has a very good quality. Note that using <em>kdu_transcode</em> in this way produces output images that have the same number of quality layers as the source image (here: 11). However, in this case 3 are actually empty (i.e. the 4 highest quality layers are effectively identical). This is not a problem at all, it just means that progressive decoding of the image will result in an improved quality up to (and including) layer 8, with layers 9, 10 and 11 not adding anything on top of that.</p> <h3 id="aware-1">Aware</h3> <p>Aware works differently in that it allows you to define explicitly which quality layers must be included in the output image. To get an access image with 20:1 compression ratio, we need to include the 4th best quality layer and anything below it (i.e. discard the 3 highest quality layers), which is done with the following command:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>j2kdriver <span class="nt">-i</span> balloon_master_layers_precincts.jp2 <span class="se">\</span> <span class="nt">-ql</span> 4 <span class="se">\</span> <span class="nt">-t</span> JP2 <span class="se">\</span> <span class="nt">-o</span> balloon_access_layers_precincts_aw.jp2 </code></pre></div></div> <p>Note that instead of decoding images by quality layer, it is also possible to do this by resolution level. This can be useful if derived images at a lower resolution are needed. Both Kakadu’s <em>kdu_transcode</em> and Aware’s <em>j2kdriver</em> application are capable of this, provided that the master images contain a sufficient number of resolution levels (which is controlled by the number of decomposition levels at the encoding stage).</p> <h2 id="conclusions">Conclusions</h2> <p>Careful selection of how JP2 preservation masters are generated can greatly facilitate the derivation of access images at a later stage. Tests with images that follow the KB’s current master specifications showed that lossy access images could only be derived by fully decoding and re-compressing them. Though not necessarily a problem, a more efficient approach would be to make better use of quality layers. This allows access images to be derived by simply extracting a subset of the master, without the need to decode or re-compress the source data. Tests with two widely used JPEG 2000 software toolkits (Kakadu and Aware) show that using this approach the process of deriving access images is both simple and efficient.</p> <h2 id="acknowledgements">Acknowledgements</h2> <p>Thanks go out to David Taubman, whose <a href="http://tech.groups.yahoo.com/group/kakadu_jpeg2000/message/6777">reply to some of my questions on Kakadu’s transcode tool</a> was largely the impetus for this blog post.</p> <h2 id="useful-links">Useful links</h2> <ul> <li><a href="https://github.com/bitsgalore/JP2AccessGeneration">Dataset with all test images (Github link)</a></li> <li><a href="http://wellcomelibrary.org/assets/wtx056572.pdf">Buckley &amp; Tanner (2009): JPEG 2000 as a Preservation and Access Format for the Wellcome Trust Digital Library</a></li> </ul> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2013/08/19/optimising-archival-jp2s-derivation-access-copies/">Open Preservation Foundation blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>For most operational uses you would need to create a custom application using the full SDK. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2013/08/19/optimising-archival-jp2s-derivation-access-copies https://bitsgalore.org/2013/08/19/optimising-archival-jp2s-derivation-access-copies 2013-08-19T00:00:00+02:00 <p>Last winter I started a first attempt at <a href="/2012/12/19/identification-pdf-preservation-risks-apache-preflight-first-impression">identifying preservation risks</a> in PDF files using the <em>Apache Preflight</em> PDF/A validator. This work was later followed up by others in two SPRUCE hackathons in Leeds (see <a href="http://www.openplanetsfoundation.org/blogs/2013-03-15-pdf-eh-another-hackathon-tale">this blog post</a> by Peter Cliff) and London (described <a href="http://wiki.opf-labs.org/display/SPR/PDFA+Validation+tools+give+different+results">here</a>). Much of this later work tacitly assumes that <em>Apache Preflight</em> is able to successfully identify features in PDF that are a potential risk for long-term access. This <a href="http://wiki.opf-labs.org/display/SPR/PDFBox+Preflight+2++-+Uses+and+Abuses">Wiki page on uses and abuses of Preflight</a> (created as part of the final SPRUCE hackathon) even goes as far as stating that “<em>Preflight is thorough and unforgiving (as it should be)</em>”. But what evidence do we have to support such claims? The only evidence that I’m aware of, are the results obtained from a <a href="https://github.com/openplanets/format-corpus/tree/master/pdfCabinetOfHorrors">small test corpus of custom-created PDFs</a>. Each PDF in this corpus was created in such a way that it includes only one specific feature that is a potential preservation risk (e.g. encryption, non-embedded fonts, and so on). However, PDFs that exist ‘in the wild’ are usually more complex. Also, the PDF specification often allows you to implement similar features in subtly different ways. For these reasons, it is essential to obtain additional evidence of <em>Preflight</em>’s ability to detect ‘risky’ features before relying on this tool in any operational setting.</p> <!-- more --> <h2 id="adobe-acrobat-engineering-test-files">Adobe Acrobat Engineering test files</h2> <p>Shortly after I completed my initial tests, Adobe released the <a href="http://acroeng.adobe.com/wp/">Acrobat Engineering website</a>, which contains a large volume of test documents that are used by Adobe for testing their products. Although the test documents are not fully annotated, they are subdivided into categories such as <em>Multimedia &amp; 3D Tests</em> and <em>Font tests</em>. This makes these files particularly useful for additional tests on <em>Preflight</em>.</p> <h2 id="methodology">Methodology</h2> <p>The general methodology I used to analyse these files is identical to what I did in my <a href="https://zenodo.org/record/2556637">2012 report</a>: first, each PDF was validated using <em>Apache Preflight</em>. As a control I also validated the PDFs with the <em>Preflight</em> component of Adobe Acrobat, using the <em>PDF/A-1b</em> profile. The table below lists the software versions used:</p> <table> <thead> <tr> <th style="text-align: left">Software</th> <th style="text-align: left">Version</th> </tr> </thead> <tbody> <tr> <td style="text-align: left">Apache Preflight</td> <td style="text-align: left">2.0.0</td> </tr> <tr> <td style="text-align: left">Adobe Acrobat</td> <td style="text-align: left">10.14</td> </tr> <tr> <td style="text-align: left">Acrobat Preflight</td> <td style="text-align: left">10.1.3 (090)</td> </tr> </tbody> </table> <h2 id="re-analysis-of-pdf-cabinet-of-horrors-corpus">Re-analysis of PDF Cabinet of Horrors corpus</h2> <p>Because the current analysis is based on a more recent version of <em>Apache Preflight</em> than the one used in the 2012 report (which was 1.8.0), I first re-ran the analysis of the PDFs in the <a href="https://github.com/openplanets/format-corpus/tree/master/pdfCabinetOfHorrors">PDF Cabinet of Horrors corpus</a>. The main results are reproduced <a href="http://wiki.opf-labs.org/display/TR/Portable+Document+Format">here</a>. The main differences with respect to that earlier version are:</p> <ol> <li><em>Apache Preflight</em> now has an option to produce output in <em>XML</em> format (as <a href="https://issues.apache.org/jira/browse/PDFBOX-1540">suggested by William Palmer</a> following the Leeds SPRUCE hackathon)</li> <li>Better reporting of non-embedded fonts (see also <a href="https://issues.apache.org/jira/browse/PDFBOX-1449">this issue</a>)</li> <li>Unlike the earlier version, <em>Preflight</em> 2.0.0 does not give any meaningful output in case of encrypted and password-protected PDFs! This is probably a bug, for which I submitted a report <a href="https://issues.apache.org/jira/browse/PDFBOX-1659">here</a>.</li> </ol> <h2 id="analysis-acrobat-engineering-pdfs">Analysis Acrobat Engineering PDFs</h2> <p>Since the Acrobat Engineering site hosts a <em>lot</em> of PDFs, I only focused on a limited subset for the current analysis:</p> <ol> <li>all files in the <em>General</em> section of the <a href="http://acroeng.adobe.com/wp/?page_id=101">Font Testing</a> category;</li> <li>all files in the <em>Classic Multimedia</em> section of the <a href="http://acroeng.adobe.com/wp/?page_id=61">Multimedia &amp; 3D Tests</a> category.</li> </ol> <p>The results are summarized in two tables (see next sections). For each analysed PDF, the table lists:</p> <ul> <li>the error(s) reported by Adobe Acrobat Preflight;</li> <li>the error code(s) reported by Apache Preflight (see Preflight’s <a href="http://svn.apache.org/repos/asf/pdfbox/trunk/preflight/src/main/java/org/apache/pdfbox/preflight/PreflightConstants.java">source code</a> for a listing of all possible error codes);</li> <li>the error description(s) reported by Apache Preflight in the <em>details</em> output element.</li> </ul> <p>For the sake of readability, the tables only list those error messages/codes that are directly related to font problems, multimedia, encryption and JavaScript. The full output for all tested files can be found <a href="https://github.com/bitsgalore/apachePreflightAcroEng">here</a>.</p> <h2 id="fonts">Fonts</h2> <p>The table below summarizes the results of the PDFs in the <a href="http://acroeng.adobe.com/wp/?page_id=101">Font Testing</a> category:</p> <table> <thead> <tr> <th style="text-align: left">Test file</th> <th style="text-align: left">Acrobat Preflight error(s)</th> <th style="text-align: left">Apache Preflight Error Code(s)</th> <th style="text-align: left">Apache Preflight Details</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/fonts//EmbeddedCmap.pdf">EmbeddedCmap.pdf</a></td> <td style="text-align: left">Font not embedded (and text rendering mode not 3) ; Glyphs missing in embedded font</td> <td style="text-align: left">3.1.3</td> <td style="text-align: left">Invalid Font definition, FontFile entry is missing from FontDescriptor for HeiseiKakuGo-W5</td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/fonts//TEXT.pdf">TEXT.pdf</a></td> <td style="text-align: left">Font not embedded (and text rendering mode not 3); Glyphs missing in embedded font ; TrueType font has differences to standard encodings but is not a symbolic font; Wrong encoding for non-symbolic TrueType font</td> <td style="text-align: left">3.1.5; 3.1.1; 3.1.2; 3.1.3; 3.2.4</td> <td style="text-align: left">Invalid Font definition, The Encoding is invalid for the NonSymbolic TTF; Invalid Font definition, Some required fields are missing from the Font dictionary; Invalid Font definition, FontDescriptor is null or is a AFM Descriptor; Invalid Font definition, FontFile entry is missing from FontDescriptor for Arial,Italic <em>(repeated for other fonts)</em>; Font damaged, The CharProcs references an element which can’t be read</td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/fonts//Type3_WWW-HTML.PDF">Type3_WWW-HTML.PDF</a></td> <td style="text-align: left">-</td> <td style="text-align: left">3.1.6</td> <td style="text-align: left">Invalid Font definition, The character with CID”58” should have a width equals to 15.56599 <em>(repeated for other fonts)</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/fonts//embedded_fonts.pdf">embedded_fonts.pdf</a></td> <td style="text-align: left">Font not embedded (and text rendering mode not 3); Type 2 CID font: CIDToGIDMap invalid or missing</td> <td style="text-align: left">3.1.9; 3.1.11</td> <td style="text-align: left">Invalid Font definition; Invalid Font definition, The CIDSet entry is missing for the Composite Subset</td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/fonts//embedded_pm65.pdf">embedded_pm65.pdf</a></td> <td style="text-align: left">-</td> <td style="text-align: left">3.1.6</td> <td style="text-align: left">Invalid Font definition, Width of the character “110” in the font program “HKPLIB+AdobeCorpID-MyriadRg”is inconsistent with the width in the PDF dictionary <em>(repeated for other font)</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/fonts//notembedded_pm65.pdf">notembedded_pm65.pdf</a></td> <td style="text-align: left">Font not embedded (and text rendering mode not 3); Glyphs missing in embedded font</td> <td style="text-align: left">3.1.3</td> <td style="text-align: left">Invalid Font definition, FontFile entry is missing from FontDescriptor for TimesNewRoman <em>(repeated for other fonts)</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/fonts//printtestfont_nonopt.pdf">printtestfont_nonopt.pdf</a>*</td> <td style="text-align: left">ICC profile is not valid; ICC profile is version 4.0 or newer; ICC profile uses invalid color space;ICC profile uses invalid type</td> <td style="text-align: left">-</td> <td style="text-align: left"><em>Preflight throws exception (exceptionThrown), exits with message ‘Invalid ICC Profile Data’</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/fonts//printtestfont_opt.pdf">printtestfont_opt.pdf</a>*</td> <td style="text-align: left">ICC profile is not valid; ICC profile is version 4.0 or newer; ICC profile uses invalid color space; ICC profile uses invalid type</td> <td style="text-align: left">-</td> <td style="text-align: left"><em>Preflight throws exception (exceptionThrown), exits with message ‘Invalid ICC Profile Data’</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/fonts//substitution_fonts.pdf">substitution_fonts.pdf</a></td> <td style="text-align: left">Font not embedded (and text rendering mode not 3)</td> <td style="text-align: left">3.1.1; 3.1.2; 3.1.3</td> <td style="text-align: left">Invalid Font definition, Some required fields are missing from the Font dictionary; Invalid Font definition, FontDescriptor is null or is a AFM Descriptor; Invalid Font definition, FontFile entry is missing from FontDescriptor for Souvenir-Light <em>(repeated for other fonts)</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/fonts//text_images_pdf1.2.pdf">text_images_pdf1.2.pdf</a></td> <td style="text-align: left">Font not embedded (and text rendering mode not 3); Glyphs missing in embedded font; Width information for rendered glyphs is inconsistent</td> <td style="text-align: left">3.1.1; 3.1.2</td> <td style="text-align: left">Invalid Font definition, Some required fields are missing from the Font dictionary; Invalid Font definition, FontDescriptor is null or is a AFM Descriptor</td> </tr> </tbody> </table> <p>* <em>As this document doesn’t appear to have any font-related issues it’s unclear why it is in the Font Testing category. Errors related to ICC profiles reproduced here because of relevance to Apache Preflight exception.</em></p> <h2 id="general-observations">General observations</h2> <p>An intercomparison between the results of Acrobat Preflight and Apache Preflight shows that Apache Preflight’s output may vary in case of non-embedded fonts. In most cases it produces error code 3.1.3 (as was the case with the <em>PDF Cabinet of Horrors</em> dataset), but other errors in the 3.1.x range may occur as well. The 3.1.6 “character width” error is something that was also encountered during the <a href="http://wiki.opf-labs.org/display/SPR/PDFA+Validation+tools+give+different+results">London SPRUCE Hackathon</a>, and according to the information <a href="https://groups.google.com/forum/#!topic/pdfnet-sdk/L2osfwaap98">here</a> this is most likely the result of the PDF/A specification not being particularly clear. So, this looks like a non-serious error that can be safely ignored in most cases.</p> <h2 id="multimedia">Multimedia</h2> <p>The next table shows the results for <a href="http://acroeng.adobe.com/wp/?page_id=61">Multimedia &amp; 3D Tests</a> category:</p> <table> <thead> <tr> <th style="text-align: left">Test file</th> <th style="text-align: left">Acrobat Preflight error(s)</th> <th style="text-align: left">Apache Preflight Error Code(s)</th> <th style="text-align: left">Apache Preflight Details</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/classic_multimedia//20020402_CALOS.pdf">20020402_CALOS.pdf</a></td> <td style="text-align: left">-</td> <td style="text-align: left">1.0; 1.2.1</td> <td style="text-align: left"><em>No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/classic_multimedia//Disney-Flash.pdf">Disney-Flash.pdf</a></td> <td style="text-align: left">Contains action of type JavaScript; Document contains JavaScripts; Document contains additional actions (AA); Font not embedded (and text rendering mode not 3); Form field does not have appearance dict; Form field has actions; Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry</td> <td style="text-align: left">1.0; 1.2.1</td> <td style="text-align: left"><em>No multimedia-related errors; Preflight did report syntax and body syntax error</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/classic_multimedia//Jpeg_linked.pdf">Jpeg_linked.pdf</a></td> <td style="text-align: left">Document is encrypted; Encrypt key present in file trailer; Named action with a value other than standard page navigation used; Incorrect annotation type used (not allowed in PDF/A); Font not embedded (and text rendering mode not 3)</td> <td style="text-align: left">1.0; 1.2.1</td> <td style="text-align: left"><em>No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/classic_multimedia//MultiMedia_Acro6.pdf">MultiMedia_Acro6.pdf</a></td> <td style="text-align: left">Document is encrypted; EmbeddedFiles entry in Names dictionary; Encrypt key present in file trailer; PDF contains EF (embedded file) entry; Incorrect annotation type used (not allowed in PDF/A)</td> <td style="text-align: left">1.0; 1.2.1</td> <td style="text-align: left"><em>No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/classic_multimedia//MusicalScore.pdf">MusicalScore.pdf</a></td> <td style="text-align: left">CIDset in subset font is incomplete; CIDset in subset font missing; Contains action of type JavaScript; Document contains JavaScripts; Document contains additional actions (AA); Font not embedded (and text rendering mode not 3); Form field has actions; Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry; Type 2 CID font: CIDToGIDMap invalid or missing</td> <td style="text-align: left">1.0; 1.2.1</td> <td style="text-align: left"><em>No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/classic_multimedia//SVG-AnnotAnim.pdf">SVG-AnnotAnim.pdf</a></td> <td style="text-align: left">Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry</td> <td style="text-align: left">5.2.1; 1.2.9</td> <td style="text-align: left">Forbidden field in an annotation definition, The subtype isn’t authorized : SVG; Body Syntax error, EmbeddedFile entry is present in a FileSpecification dictionary</td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/classic_multimedia//SVG.pdf">SVG.pdf</a></td> <td style="text-align: left">Contains action of type JavaScript; Document contains JavaScripts; Font not embedded (and text rendering mode not 3); Form field has actions; PDF contains EF (embedded file) entry</td> <td style="text-align: left">1.0; 1.2.1</td> <td style="text-align: left"><em>No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/classic_multimedia//ScriptEvents.pdf">ScriptEvents.pdf</a></td> <td style="text-align: left">Contains action of type JavaScript; Document contains JavaScripts; Font not embedded (and text rendering mode not 3); Form field has actions; Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry</td> <td style="text-align: left">1.0; 1.2.1</td> <td style="text-align: left"><em>No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/classic_multimedia//Service%20Form_media.pdf">Service Form_media.pdf</a></td> <td style="text-align: left">Contains action of type JavaScript; Contains action of type ResetForm; Document contains JavaScripts; Document contains additional actions (AA); Font not embedded (and text rendering mode not 3); Glyphs missing in embedded font; Incorrect annotation type used (not allowed in PDF/A); Named action with a value other than standard page navigation used; PDF contains EF (embedded file) entry</td> <td style="text-align: left">1.0; 1.2.1</td> <td style="text-align: left"><em>No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/classic_multimedia//Trophy.pdf">Trophy.pdf</a></td> <td style="text-align: left">Contains action of type JavaScript; Document contains JavaScripts; Font not embedded (and text rendering mode not 3); Form field has actions; Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry</td> <td style="text-align: left">1.0; 1.2.1</td> <td style="text-align: left"><em>No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/classic_multimedia//VolvoS40V50-Full.pdf">VolvoS40V50-Full.pdf</a></td> <td style="text-align: left">Preflight exits with: “An error occurred while parsing a contents stream. Unable to analyze the PDF file”</td> <td style="text-align: left">1.0; 1.2.1</td> <td style="text-align: left"><em>No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/classic_multimedia//gXsummer2004-stream.pdf">gXsummer2004-stream.pdf</a></td> <td style="text-align: left">File cannot be loaded in Acrobat (<em>damaged file)</em></td> <td style="text-align: left">1.0; 1.1</td> <td style="text-align: left"><em>No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/classic_multimedia//phlmapbeta7.pdf">phlmapbeta7.pdf</a></td> <td style="text-align: left">Document contains additional actions (AA); Font not embedded (and text rendering mode not 3); Incorrect annotation type used (not allowed in PDF/A); PDF contains EF (embedded file) entry</td> <td style="text-align: left">1.0; 1.2.1</td> <td style="text-align: left"><em>No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/classic_multimedia//us_population.pdf">us_population.pdf</a></td> <td style="text-align: left">Preflight exits with: “An error occurred while parsing a contents stream. Unable to analyze the PDF file”</td> <td style="text-align: left">1.0; 1.2.1</td> <td style="text-align: left"><em>No multimedia, font or encryption-related errors; Preflight did report syntax and body syntax error</em></td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/movie//movie.pdf">movie.pdf</a></td> <td style="text-align: left">Incorrect annotation type used (not allowed in PDF/A)</td> <td style="text-align: left">5.2.1</td> <td style="text-align: left">Forbidden field in an annotation definition, The subtype isn’t authorized : Movie</td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/movie//movie_down1.pdf">movie_down1.pdf</a></td> <td style="text-align: left">Incorrect annotation type used (not allowed in PDF/A)</td> <td style="text-align: left">5.2.1</td> <td style="text-align: left">Forbidden field in an annotation definition, The subtype isn’t authorized : Movie</td> </tr> <tr> <td style="text-align: left"><a href="http://acroeng.adobe.com/Test_Files/movie//remotemovieurl.pdf">remotemovieurl.pdf</a></td> <td style="text-align: left">Font not embedded (and text rendering mode not 3); Incorrect annotation type used (not allowed in PDF/A)</td> <td style="text-align: left">5.2.1; 3.1.1; 3.1.2; 3.1.3</td> <td style="text-align: left">Forbidden field in an annotation definition, The subtype isn’t authorized : Movie; Invalid Font definition, Some required fields are missing from the Font dictionary; Invalid Font definition, FontDescriptor is null or is a AFM Descriptor; Invalid Font definition, FontFile entry is missing from FontDescriptor for Arial</td> </tr> </tbody> </table> <h2 id="general-observations-1">General observations</h2> <p>The results from the <em>Multimedia</em> PDFs are interesting for several reasons. First of all, these files include a wide variety of ‘risky’ features, such as multimedia content, embedded files, JavaScript, non-embedded fonts and encryption. These were successfully identified by <em>Acrobat Preflight</em> in most cases. <em>Apache Preflight</em>, on the other hand, only reported non-specific and fairly uninformative errors (1.0 + 1.2.1) for 12 out of 17 files. Even though <em>Preflight</em> was correct in establishing that these files were not valid PDF/A-1b, it wasn’t able to drill down to the level of specific features for the majority of these files.</p> <p>Looking more into detail at those 1.0 and 1.2.1 errors, the detailed description of most of them is:</p> <pre><code class="language-data"> Syntax error, Expected pattern 'obj but missed at character 'o' </code></pre> <p>To me it looks like <em>Preflight</em> doesn’t correctly parse the binary structure of the PDF. Opening a few of the problematic PDFs revealed that the object identifiers in these files were followed <em>immediately</em> by the object contents, e.g:</p> <pre><code class="language-data">32 0 obj&lt;&lt;/Kids[33 0 R]&gt;&gt; endobj </code></pre> <p>whereas more commonly they are separated by a line terminator, like this:</p> <pre><code class="language-data">32 0 obj &lt;&lt;/Kids[33 0 R]&gt;&gt; endobj </code></pre> <p>As far as I’m aware neither the PDF specification nor PDF/A have anything to say about line endings in this case, so my best guess is that this is simply a bug that results in the file not being fully parsed. I submitted a bug report for this issue <a href="https://issues.apache.org/jira/browse/PDFBOX-1674">here</a>.</p> <h2 id="summary-and-conclusions">Summary and conclusions</h2> <p>The re-analysis of the PDF Cabinet of Horrors corpus, and the subsequent analysis of a sub-set of the Adobe Acrobat Engineering PDFs shows a number of things. First, <em>Apache Preflight</em> 2.0.0 does not properly identify encryption and password-protection. This looks like a bug that is probably easily fixed. Second, the analysis of the <em>Font Testing</em> PDFs from the Acrobat Engineering site revealed that non-embedded fonts may result in a variety of error codes in <em>Apache Preflight</em> (assuming here that the <em>Acrobat Preflight</em> results are accurate). So, when using <em>Apache Preflight</em> to check font embedding, it’s probably a good idea to treat all font-related errors (perhaps with the exception of character width errors) as a potential risk. The more complex PDFs in the <em>Multimedia</em> category proved to be quite challenging to <em>Apache Preflight</em>: for most files here, it was not able to identify <em>specific features</em> such as multimedia content, embedded files, JavaScript and non-embedded fonts. A cursory analysis of some of the failed files suggests that this is probably a bug that results in <em>Apache Preflight</em> not being able to parse the file structure correctly. Keeping in mind that he specificity of <em>Preflight</em>’s validation output already improved considerably since version 1.8.0, a fix of both this issue and the encryption problem would probably result in another significant improvement. In the meantime, it’s important to keep the expectations about the tool’s capabilities realistic, in order to avoid some potential unintended misuses.</p> <h2 id="links">Links</h2> <ul> <li> <p><a href="https://github.com/bitsgalore/apachePreflightAcroEng">Full Acrobat Preflight and Apache Preflight output for all tested files (Github)</a></p> </li> <li> <p><a href="http://wiki.opf-labs.org/display/TR/Portable+Document+Format">Portable Document Format on OPF File Format Risk Registry</a></p> </li> <li> <p><a href="http://acroeng.adobe.com/wp/">Adobe Acrobat Engineering website</a></p> </li> <li> <p><a href="https://zenodo.org/record/2556637">Identification of PDF preservation risks with Apache Preflight: a first impression</a></p> </li> <li> <p><a href="http://wiki.opf-labs.org/display/SPR/PDFBox+Preflight+2++-+Uses+and+Abuses">PDFBox Preflight 2 - Uses and Abuses (OPF Wiki)</a></p> </li> </ul> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2013/07/25/identification-pdf-preservation-risks-sequel/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2013/07/25/identification-pdf-preservation-risks-sequel https://bitsgalore.org/2013/07/25/identification-pdf-preservation-risks-sequel 2013-07-25T00:00:00+02:00 <p>It’s been more than two years now since I wrote my D-Lib paper <a href="http://www.dlib.org/dlib/may11/vanderknijff/05vanderknijff.html"><em>JPEG 2000 for Long-term Preservation: JP2 as a Preservation Format</em></a>. From time to time people ask me about the status of the issues that are mentioned in that paper, so here’s a long overdue update.</p> <!-- more --> <h2 id="issues-addressed-in-the-2011-paper">Issues addressed in the 2011 paper</h2> <p>The <a href="http://www.dlib.org/dlib/may11/vanderknijff/05vanderknijff.html">D-Lib paper</a> mainly focused on two problems with the (then-current version of the) <a href="http://www.jpeg.org/public/15444-1annexi.pdf">JP2 format specification</a>:</p> <ol> <li>The specification was overly restrictive on the embedding of ICC profiles. By only allowing <em>input</em> profiles, this ruled out the use of <em>display</em> profiles. In practice this meant that widely-used working colour spaces such as <a href="http://www.adobe.com/digitalimag/pdfs/AdobeRGB1998.pdf"><em>Adobe RGB</em></a> and <a href="http://www.eci.org/doku.php?id=en:colourstandards:workingcolorspaces"><em>eciRGB 2</em></a> could not be used in <em>JP2</em> without violating the standard.</li> <li><em>JP2</em> makes a distinction between <em>capture</em> resolution and <em>default display</em> resolution, which are stored in two designated sets of header fields. However, the specification was not clear in which case either set of fields should be used.</li> </ol> <p>This lead to the situation that not all software products were interpreting the specification in the same way. For instance, some encoders would (silently) produce files in <a href="http://fileformats.archiveteam.org/wiki/JPX"><em>JPX</em></a> format whenever they encountered an input image with an embedded <em>display</em> ICC profile. Other encoders would embed the profile, changing the profile class in the process, whereas others yet would ignore the limitation altogether and embed the profile without complaining. Similarly, some software products would only write (and read) the <em>capture</em> resolution fields (while ignoring any <em>default display</em> ones), whereas the opposite was true for other products. This in turn raised various interoperability issues, many of which are potential risks in a long-term preservation context.</p> <p>The 2011 paper concluded that:</p> <blockquote> <p>[t]hese issues could be remedied by some small adjustments of JP2’s format specification, which would create minimal backward compatibility problems, if any at all.</p> </blockquote> <h2 id="amendment-to-the-standard">Amendment to the standard</h2> <p>So, enter the <a href="http://www.iso.org/iso/home/store/catalogue_tc/catalogue_detail.htm?csnumber=59863">amendment “<em>Updated ICC profile support and resolution clarification</em>”</a>, which was published by ISO earlier this year. This amendment remedies the above issues by applying the following changes to the <a href="http://www.jpeg.org/public/15444-1annexi.pdf">existing JP2 format specification</a>:</p> <ol> <li>The <em>Restricted ICC profile</em> method now permits the use of <em>display</em> profiles (previously only <em>input</em> profiles were allowed). The other restrictions (e.g. that ICC profiles should be of either the Monochrome or the Three-Component Matrix-Based type) remain unchanged.</li> <li>It is more specific about the intended uses of the <em>capture</em> and the <em>default display</em> resolution boxes. Of particular interest here is that <em>capture</em> resolution now reflects the resolution at which the image samples were “captured or created”. Previously the word “digitized” was used, which ruled out the case of born-digital materials. The use of the <em>default display</em> resolution box is also further clarified. In practice this means that the <em>capture</em> resolution is pretty much equivalent to the <em>XResolution</em> /<em>YResolution</em> fields in TIFF.</li> </ol> <p>Note that the full amendment text is only available after purchase at ISO (previously an earlier draft was available for free, but apparently it was taken down recently).</p> <h2 id="implementation-of-changes">Implementation of changes</h2> <p>Since a standard isn’t worth much unless it is used, let’s have a quick look at the three most popular JPEG 2000 implementations (a more elaborate overview is available <a href="http://wiki.opf-labs.org/display/TR/Handling+of+ICC+profiles">here</a> and <a href="http://wiki.opf-labs.org/display/TR/Resolution+not+in+expected+header+fields">here</a> at the <a href="http://wiki.opf-labs.org/display/TR/OPF+File+Format+Risk+Registry">OPF File Format Risk Registry</a>).</p> <p>Recent versions of <strong>Kakadu</strong>’s <em>kdu_compress</em> are now able to correctly handle <em>ICC</em> profiles. Compressing a <em>TIFF</em> that contains an <em>ICC</em> profile that meets the (updated) <em>restricted ICC profile</em> definition now produces a <em>JP2</em> with the profile correctly embedded. Moreover, <em>kdu_compress</em> now uses the <em>capture</em> resolution fields to store the image’s resolution (as derived from the <em>TIFF</em>). Previously, the <em>Kakadu</em> demo applications were using the <em>default display</em> fields instead, which resulted in various interoperability issues, because most other decoders/encoders were using the <em>capture</em> fields. This is all solved in the latest version.</p> <p>By default <strong>Aware</strong> and <strong>Luratech</strong> already used the <em>capture</em> resolution fields back in 2011, and this behaviour is now consistent with the updated standard. As for <em>ICC</em> profiles, <em>Aware</em> accepted <em>display</em> profiles without complaining in my 2011 tests, and with the amendment in effect, these images are now also valid <em>JP2</em>. <em>Luratech</em> used to handle <em>display</em> profiles by changing the profile class field tom <em>input</em>. That was in 2011, and I don’t know if anything has changed since then, but then again this behaviour never caused by problems in the first place.</p> <h2 id="round-up-and-conclusions">Round-up and conclusions</h2> <p>The amendment to JP2 fixes the previous shortcomings that were mentioned in my 2011 D-Lib paper. Moreover, the behaviour of the three most popular (commercial) JPEG 2000 implementations now closely follows the updated specification, which should minimise any interoperability problems related to ICC profiles and resolution.</p> <h2 id="links">Links</h2> <p><a href="http://www.dlib.org/dlib/may11/vanderknijff/05vanderknijff.html">JPEG 2000 for Long-term Preservation: JP2 as a Preservation Format (D-Lib)</a></p> <p><a href="http://www.jpeg.org/public/15444-1annexi.pdf">JP2 format specification (2004 version)</a></p> <p><a href="http://www.iso.org/iso/home/store/catalogue_tc/catalogue_detail.htm?csnumber=59863">ISO/IEC 15444-1:2004/Amd 6:2013. Updated ICC profile support and resolution clarification</a></p> <p><a href="http://wiki.opf-labs.org/display/TR/Handling+of+ICC+profiles">Handling of ICC Profiles by JPEG 2000 encoders (OPF File Format Risk Registry)</a></p> <p><a href="http://wiki.opf-labs.org/display/TR/Resolution+not+in+expected+header+fields">Handling of resolution fields by JPEG 2000 encoders (OPF File Format Risk Registry)</a></p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2013/07/01/icc-profiles-and-resolution-jp2-update-2011-d-lib-paper/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2013/07/01/icc-profiles-and-resolution-jp2-update-2011-d-lib-paper https://bitsgalore.org/2013/07/01/icc-profiles-and-resolution-jp2-update-2011-d-lib-paper 2013-07-01T00:00:00+02:00 <h2 id="introduction">Introduction</h2> <p>Last year (2012) the KB released a <a href="https://zenodo.org/record/839711">report on the suitability of the <em>EPUB</em> format for archival preservation</a>. A substantial number of <em>EPUB</em>-related developments have happened since then, and as a result some of the report’s findings and conclusions have become outdated. This applies in particular to the observations on <em>EPUB</em> 3, and the support of <em>EPUB</em> by characterisation tools. This blog post provides an update to those findings. It addresses the following topics in particular:</p> <ul> <li>Use of <em>EPUB</em> in scholarly publishing</li> <li>Adoption and use of <em>EPUB</em> 3</li> <li><em>EPUB</em> 3 reader support</li> <li>Support of <em>EPUB</em> by characterisation tools</li> </ul> <p>In the following sections I will briefly summarise the main developments in each of these areas, after which I will wrap up things in a concluding section.</p> <!-- more --> <h2 id="use-of-epub-in-scholarly-publishing">Use of <em>EPUB</em> in scholarly publishing</h2> <p>Although scholarly publishing is still dominated by <em>PDF</em>, the use of <em>EPUB</em> in this sector is on the rise. <a href="http://scholarlykitchen.sspnet.org/2013/03/19/is-it-time-for-scholarly-journal-publishers-to-begin-distributing-articles-using-epub-3/">This blog post by Todd Carpenter</a> gives the following examples:</p> <ul> <li>BioMed Central is one of the first publishers that started offering their scholarly publications in <em>EPUB</em> format (as explained in this <a href="http://blogs.biomedcentral.com/bmcblog/2012/12/11/biomed-central-now-publishes-in-epub-format/">December 2012 blog post</a>). One of the journals that are available in <em>EPUB</em> format is the <a href="http://www.jneuroinflammation.com/content">Journal of Neuroinflammation</a>.</li> <li><a href="http://www.hindawi.com/epub/">Hindawi Publishing Corporation recently added <em>EPUB</em></a> as one of the available formats for all of their journal and book publications.</li> <li>Lippincott Williams &amp; Wilkins (which is part of Kluwer) also offers <a href="http://journals.lww.com/pages/results.aspx?txtKeywords=epub">some of their titles</a> in <em>EPUB</em> format.</li> </ul> <p>At the time of writing, the above publishers are all using <em>EPUB</em> 2.</p> <h2 id="adoption-and-use-of-epub-3">Adoption and use of <em>EPUB</em> 3</h2> <p>Over the last year a number of organisations that are representing the publishing industry have expressed their support of <em>EPUB</em> 3.</p> <p>The <a href="http://www.bisg.org">Book Industry Study Group (BISG)</a> is a trade association for <a href="http://www.bisg.org/directory/">companies in the publishing industry</a>. Last year (August 2012) BISG released a <a href="http://www.bisg.org/what-we-do-4-155-pol-1201-endorsement-of-epub-3.php">policy statement</a> in which it endorsed “<em>EPUB 3 as the accepted and preferred standard for representing, packaging, and encoding structured and semantically enhanced Web content — including XHTML, CSS, SVG, images, and other resources — for distribution in a single-file format</em>”.</p> <p>Early this year (March 2013) the <a href="http://www.internationalpublishers.org/">International Publishers Association (IPA)</a> issued a <a href="http://www.internationalpublishers.org/images/stories/PR/2013/epub3pr_final.pdf">press release</a> that also endorsed <em>EPUB</em> 3 as a “<em>preferred standard format for representing HTML and other web content for distribution as single-file publications</em>”. IPA represents over 60 national publishing organisations from more than 50 countries.</p> <p>Finally, the <a href="http://www.europeanbooksellers.eu/">European Booksellers Federation</a> recently released a <a href="http://www.europeanbooksellers.eu/positionpaper/interoperability-e-books-formats">report on the interoperability of eBook Formats</a>. Its authors did a comparison of the features and functionality provided by <em>EPUB</em> 3, Amazon’s <a href="https://en.wikipedia.org/wiki/Amazon_Kindle#Proprietary_formats_.28AZW.2C_KF8.29">KF8 (Kindle)</a> and Apple’s e-book formats. They concluded that <em>EPUB</em> 3 “<em>clearly covers the superset of the expressive abilities of all the formats</em>”, and that there is “<em>no technical or functional reason not to use and establish EPUB 3 as an/the interoperable (open) ebook format standard</em>”. This all suggests that <em>EPUB</em> 3 is widely supported by the publishing industry.</p> <p>Having said that, the actual use of <em>EPUB</em> 3 is still limited at this stage, even though some publishers have already started using the format. Earlier this year technical publisher O’Reilly started releasing <a href="http://toc.oreilly.com/2013/02/oreillys-journey-to-epub-3.html">all their new eBook bundles in <em>EPUB</em> 3 format</a>. The announcement mentions that their backlist will be updated as well. Interestingly, they decided to create “hybrid” <em>EPUB</em>s that are backward-compatible with <em>EPUB</em> 2. In November 2012 publisher Hachette also <a href="http://www.digitalbookworld.com/2012/hachette-launches-epub3-program-committed-to-the-format/">announced the launch of their <em>EPUB 3</em> program</a>.</p> <h2 id="epub-3-reader-support"><em>EPUB</em> 3 reader support</h2> <p>At this time reader support for <em>EPUB</em> 3 is still limited, but there have been a number of significant developments since the second half of 2012:</p> <ul> <li><a href="http://www.apple.com/apps/ibooks/">Apple iBooks</a> started support of <em>EPUB</em> 3 during the course of 2012, which means that the format can be read on Apple mobile devices (e.g. <em>iPad</em>).</li> <li><a href="http://azardi.infogridpacific.com/">Azardi Desktop</a> is a viewer that supports most features of <em>EPUB</em> 3. It is available for Windows, Mac and Linux.</li> <li>Helicon is an ebook production house that <a href="http://www.digitalbookworld.com/2012/helicon-books-claims-first-e-reader-for-android-with-full-epub3-support/">claimed to have developed the first reading application for Android that fully supports <em>EPUB</em> 3</a> in December 2012.</li> <li><a href="http://readium.org/">Readium</a> is an open-source set of libraries for viewing and creating <em>EPUB</em> 2 and <em>EPUB</em> 3 content. The project has already produced a <a href="https://chrome.google.com/webstore/detail/empty-title/fepbnnnkkadjhjahcafoaglimekefifl?hl=en">Google Chrome extension</a> that allows you to view <em>EPUB</em> content in the browser. Since March 2013, Readium is backed by the <a href="http://readium.org/readium-foundation-announced">Readium Foundation</a>, which is a consortium of (currently 27) <a href="http://readium.org/membership">companies</a> that are active in digital publishing. Current work focuses on the development of <a href="http://readium.org/projects/readium-sdk">Readium SDK</a>, a high-performance Software Development Kit that is optimised for mobile devices.</li> <li>E-Reader manufacturer Kobo <a href="http://www.digitalbookworld.com/2012/kobo-to-fully-support-epub-3-by-third-quarter-2013/">announced</a> that it aims to offer full support of <em>EPUB</em> 3 by the third quarter of 2013.</li> <li>According to <a href="http://goodereader.com/blog/electronic-readers/the-conundrum-of-digital-publishing-html5-or-epub-3/">this piece</a>, Barnes and Noble also said they will support <em>EPUB</em> 3 sometime in 2013.</li> <li>In February Sony <a href="http://goodereader.com/blog/electronic-readers/sony-reader-for-android-updated-to-support-epub-3/">added support for <em>EPUB</em> 3 to its Reader app for Google Android</a>. Since Sony is a member of the <a href="http://readium.org/readium-foundation-announced">Readium Foundation</a>, it is to be expected that <em>EPUB</em> 3 support for their e-readers will follow as well (although the company hasn’t made any official statement on this).</li> </ul> <h2 id="support-of-epub-by-characterisation-tools">Support of <em>EPUB</em> by characterisation tools</h2> <p>The 2012 report concluded that <em>EPUB</em> was not optimally supported by characterisation tools. This situation has improved quite a lot since that time.</p> <h3 id="identification">Identification</h3> <p><em>EPUB</em> is now <a href="http://www.nationalarchives.gov.uk/PRONOM/Format/proFormatSearch.aspx?status=detailReport&amp;id=1270">included in <em>PRONOM</em></a>, and has a corresponding <a href="http://www.nationalarchives.gov.uk/information-management/projects-and-work/droid.htm">DROID</a> signature. This means that <a href="http://fido.openpreservation.org/"><em>Fido</em></a> should now be able to identify the format as well. On a side note, PRONOM doesn’t differentiate between <em>EPUB</em> 2 and 3, and it appears that the current record (which is only an outline record anyway) either combines both versions, or only refers to <em>EPUB</em> 2. PRONOM should probably be more specific on this.</p> <h3 id="validation-and-feature-extraction">Validation and feature extraction</h3> <p>The 2012 report included tests of 2 <em>EPUB</em> validator tools: <a href="http://code.google.com/p/epubcheck/"><em>epubcheck</em></a> and <a href="http://code.google.com/p/flightcrew/">flightcrew</a>. While testing <em>epubcheck</em> in 2012, I was’t entirely happy with the rather unstructured output that the tool produced. Also, I couldn’t find <em>any</em> tool that was capable of extracting technical meta-information about an <em>EPUB</em>, like the presence of encryption or other digital rights management technology (feature extraction). Happily, starting with version 3.0 <em>epubcheck</em> is capable of extracting this kind of information. Moreover, it added an option to <a href="http://code.google.com/p/epubcheck/wiki/Extraction">report its output in structured <em>XML</em> format</a> that follows the <a href="http://sourceforge.net/projects/jhove/"><em>JHOVE</em></a> schema. I haven’t done any elaborate testing, but a quick run on some of <a href="http://code.google.com/p/epub-samples/">these <em>EPUB</em> 3 samples</a> showed that <em>epubcheck</em> was able to identify font obfuscation, in which case a property <em>hasEncryption</em> (value <em>true</em>) is reported. I wasn’t able to find any <em>EPUB</em> files with <em>DRM</em>, so I cannot confirm if <em>epubcheck</em> detects this as well.</p> <h3 id="flightcrew">Flightcrew</h3> <p>As for <em>flightcrew</em>, no new versions of that tool have been released since August 2011, and it looks like it is not under any active development.</p> <h2 id="discussion-and-conclusions">Discussion and conclusions</h2> <p>Since the release of the <a href="https://zenodo.org/record/839711">KB report on the suitability of <em>EPUB</em> for archival preservation</a> the <em>EPUB</em> landscape has changed rather a lot. First, a number of academic publishers have started to offer scholarly content in this format. Although <em>EPUB</em> 3 is still in its early stages, various organisations representing the publishing industry have explicitly expressed their support of <em>EPUB</em> 3. A number of software applications now exist that are able to read the format, and work on a high-performance open source <em>EPUB</em> 3 Software Development Kit is backed by major players in the digital publishing industry (including e-reader manufacturers such as Kobo and Sony). <em>EPUB</em> support by characterisation tools has improved as well, mostly thanks to a number of recent enhancements of <em>epubcheck</em>. So, overall, <em>EPUB</em>’s credentials as a preservation format appear to have improved quite a bit over the last year. In the case of <em>EPUB</em> 3 it’s still too early to say anything about actual adoption, but the conditions for adoption to happen look pretty favourable. This is something I will get back to in my next update, perhaps in another year from now.</p> <h2 id="useful-links">Useful links</h2> <ul> <li> <p><a href="http://idpf.org/">International Digital Publishing Forum </a></p> </li> <li> <p><a href="http://code.google.com/p/epub-samples/">EPUB 3 Sample Documents</a></p> </li> <li> <p><a href="http://azardi.infogridpacific.com/resources.html">Fixed Layout (FLO) Demonstration ePub3 Documents by Azardi</a></p> </li> <li> <p><a href="http://code.google.com/p/epubcheck/">epubcheck</a></p> </li> <li> <p><a href="http://readium.org/">Readium Foundation</a></p> </li> <li> <p><a href="https://zenodo.org/record/839711">EPUB for archival preservation (2012 report)</a></p> </li> <li> <p><a href="http://www.europeanbooksellers.eu/positionpaper/interoperability-e-books-formats">On the Interoperability of eBook Formats - report by the European Booksellers Federation</a></p> </li> </ul> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2013/05/23/epub-archival-preservation-update/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2013/05/23/epub-archival-preservation-update https://bitsgalore.org/2013/05/23/epub-archival-preservation-update 2013-05-23T00:00:00+02:00 <p>About a year ago, work started on <a href="http://www.openplanetsfoundation.org/blogs/2012-02-20-summary-outputs-and-roadmap-feb-2012">packaging SCAPE tools</a>. <em>Jpylyzer</em> was the first SCAPE tool that was <a href="http://www.openplanetsfoundation.org/blogs/2012-02-15-sustainability-and-adoption-preservation-tools">turned into a Debian package</a>. Some time later, the OPF set up a couple of machine images at Amazon Web Services, which can be used to <a href="http://www.openplanetsfoundation.org/blogs/2012-03-08-turning-github-code-debian-packages-opf-way">create packages repeatedly using a virtual machine</a>. Even though I’ve used the Amazon service a couple of times myself, I really know next to nothing about Debian packages, and it’s safe to say that the underlying build process has been more or less a complete mystery to me.</p> <p>To get a better understanding of the process for building Debian packages, I had a try at packaging <em>jpylyzer</em> on my local machine (which runs on <a href="http://blog.linuxmint.com/?p=2216">Linux Mint 14</a>). Some time ago Dave Tarrant and Rui Castro wrote <a href="http://wiki.opf-labs.org/display/SP/Building+Your+Debian+Package">a nice step-by-step guide on building Debian packages</a> on the OPF Wiki, so I tried to follow the instructions there. While working on this, I made some notes, mainly to remind myself of what I was doing. Then I realised that some of this might be useful to others as well, so I decided to turn it into a blog post.</p> <!-- more --> <h2 id="objectives">Objectives</h2> <p>The objectives of this exercise were:</p> <ul> <li>to get more more familiar with the packaging process myself;</li> <li>to provide some input on how useful the <a href="http://wiki.opf-labs.org/display/SP/Building+Your+Debian+Package">guide</a> on the OPF Wiki is from the perspective of someone who is largely ignorant of the packaging procedure;</li> <li>to identify any problems in <em>jpylyzer</em>’s packaging procedure.</li> </ul> <p>I did two experiments: first, I did a <em>very</em> limited test where I tried to create a template directory structure using <em>debhelper</em>, which would be the first step when starting from scratch. Since for <em>jpylyzer</em> all the files in the <em>debian</em> directory already exist, I then moved on to building <em>jpylyzer</em> using the existing files.</p> <h2 id="test-1-creating-the-directory-structure-from-scratch">Test 1: creating the directory structure from scratch</h2> <p>For this, I first installed all the required packages listed in the <em>Pre-Requisites</em> section of the <a href="http://wiki.opf-labs.org/display/SP/Building+Your+Debian+Package">guide</a> using:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt-get <span class="nb">install </span>build-essential dh-make devscripts debhelper lintian </code></pre></div></div> <p>Subsequently I followed the instructions in the <em>Getting Started</em> section. For this I simply created an empty directory:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir </span>debtest_1.0.0 </code></pre></div></div> <p>And then:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd </span>debtest_1.0.0 </code></pre></div></div> <p>Then I ran <em>dh_make</em>:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dh_make </code></pre></div></div> <p>This resulted in an error message, telling me that the package name and its version number should be separated by a dash (‘-‘) instead of an underscore (‘_’), or, alternatively, that the -p flag should be used. So I changed the directory name:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mv </span>debtest_1.0.0 debtest-1.0.0 </code></pre></div></div> <p>Re-running <em>dh_make</em>, it now accepted the directory name, but it complained about a missing tarball (which I purposefully didn’t make in this test). However, as <em>dh_make</em> offered the suggestion to use the <code class="language-plaintext highlighter-rouge">--createorig</code> option (which creates a tarball) I tried this:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dh_make <span class="nt">--createorig</span> </code></pre></div></div> <p>This resulted in the creation of a <em>debian</em> directory with file templates, and an (empty) tarball <em>debtest_1.0.0.orig.tar.gz</em> which was created in the parent (<em>debtest</em>) directory.</p> <p>So, apart from the dash/underscore mix-up this is all pretty straightforward.</p> <h2 id="test-2-building-jpylyzer">Test 2: building <em>jpylyzer</em></h2> <p>In this second test I tried to build <em>jpylyzer</em> using the <a href="https://github.com/openplanets/jpylyzer/tree/master/debian">already existing files in the <em>debian</em> folder of <em>jpylyzer</em>’s Git repository</a>. First I cloned the repository to my local machine:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/openplanets/jpylyzer.git </code></pre></div></div> <p>Then I went into the <em>jpylyzer</em> directory:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd </span>jpylyzer </code></pre></div></div> <p>From there I tried to build <em>jpylyzer</em> directly, using the command given in the <a href="http://wiki.opf-labs.org/display/SP/Building+Your+Debian+Package">guide’s</a> <em>Building your package</em> section:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dpkg-buildpackage <span class="nt">-tc</span> </code></pre></div></div> <h2 id="missing-changelog">Missing changelog</h2> <p>The above command resulted in an error message about a missing <em>changelog</em> file in the <em>debian</em> folder. The <em>changelog</em> section in the <a href="http://wiki.opf-labs.org/display/SP/Building+Your+Debian+Package">OPF guide</a> does mention an OPF-hosted GitHub 2 Changelog service, which is supposed to be callable from the rules file. But I don’t see any reference to it in jpylyzer’s <a href="https://github.com/openplanets/jpylyzer/blob/master/debian/rules"><em>rules</em></a> file, so I don’t really know how this is supposed to work! To to keep going I simply grabbed the default changelog that was created by <em>debhelper</em> in an earlier experiment. After this I ran the command again.</p> <h2 id="unknown-commands-in-makefile">Unknown commands in makefile</h2> <p>This time, <em>dpkg-buildpackage</em> exited with the following errors:</p> <pre><code class="language-data">pymakespec --onefile jpylyzer.py make[1]: pymakespec: Command not found make[1]: *** [build] Error 127 make[1]: Leaving directory `/home/johan/debtest/jpylyzer' make: *** [build] Error 2 dpkg-buildpackage: error: debian/rules build gave error exit status </code></pre> <p>These errors arise from the following lines in <em>jpylyzer</em>’s <a href="https://github.com/openplanets/jpylyzer/blob/master/Makefile">makefile</a>:</p> <pre><code class="language-data">build: pymakespec --onefile jpylyzer.py pyinstaller jpylyzer.spec @echo "Built in dist/jpylyzer" </code></pre> <p>The <em>pymakespec</em> and <em>pyinstaller</em> commands above are most likely shell scripts that launch the <em>Makespec.py</em> and <em>pyinstaller.py</em> scripts that are both part of <a href="http://www.pyinstaller.org/"><em>PyInstaller</em></a> (these are used for building an executable from the source code). However, neither the shell scripts nor any references to them are included in <em>jpylyzer</em>’s repository (my best guess is that they exist only on a specific machine instance - perhaps the Amazon virtual machines?), so the makefile simply won’t work.</p> <p>I was able to fix this by changing the references to the shell scripts to this (using <em>PyInstaller 1.5</em>):</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python /home/johan/pyinstall1.5/Makespec.py <span class="nt">--onefile</span> jpylyzer.py python home/johan/pyinstall1.5/pyinstaller.py jpylyzer.spec </code></pre></div></div> <p>For <em>PyInstaller 2</em> these two lines should be substituted by:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python /home/johan/pyinstall/pyinstaller.py <span class="nt">--onefile</span> jpylyzer.py </code></pre></div></div> <p>Note here that <em>PyInstaller</em> has no default installation location, and the file paths will vary from machine to machine!</p> <p>After making these changes I was able to run <em>dpkg-buildpackage</em> without any problems:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dpkg-buildpackage <span class="nt">-tc</span> </code></pre></div></div> <p>Result: the following files were created in the repo’s parent directory:</p> <ul> <li>jpylyzer_1.9.0_amd64.changes</li> <li>jpylyzer_1.9.0_amd64.deb</li> <li>jpylyzer_1.9.0.dsc</li> <li>jpylyzer_1.9.0.tar.gz</li> </ul> <h2 id="tarball-schmarball">Tarball schmarball</h2> <p>One thing that confused me at first: the <em>Getting Started</em> section in the <a href="http://wiki.opf-labs.org/display/SP/Building+Your+Debian+Package">OPF guide</a> mentions the need for building a <em>native</em> package before starting the Debian packaging:</p> <blockquote> <p>If you have got here and you don’t have any already packaged code (a tar ball with makefile etc) then you will need to build a native package.</p> </blockquote> <p>So, I initially thought I would need to create a tarball of my repo first. As it turns out this is not the case: the tarball is created automatically once you run <em>dpkg-buildpackage</em>. So this is one thing less to worry about!</p> <h2 id="verifying-the-package-with-lintian">Verifying the package with <em>lintian</em></h2> <p>As a final step I used <em>lintian</em> to verify my package:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lintian jpylyzer_1.9.0_amd64.deb </code></pre></div></div> <p>This resulted in the following output (using <em>PyInstaller</em> 1.5):</p> <pre><code class="language-data">E: jpylyzer: unstripped-binary-or-object usr/bin/jpylyzer W: jpylyzer: hardening-no-fortify-functions usr/bin/jpylyzer W: jpylyzer: wrong-bug-number-in-closes l3:#nnnn E: jpylyzer: debian-changelog-file-contains-invalid-email-address johan@unknown E: jpylyzer: helper-templates-in-copyright </code></pre> <p>With <em>PyInstaller 2</em> I got this additional warning:</p> <pre><code class="language-data">W: jpylyzer: hardening-no-relro usr/bin/jpylyzer </code></pre> <p>I still need to give these errors and warnings an in-depth look. At least one error is related to the bogus <em>changelog</em> file I used. Some others (e.g. <em>unstripped-binary-or-object</em>) appear to be related to the build process of the binaries.</p> <h2 id="conclusions">Conclusions</h2> <p>Using the <a href="http://wiki.opf-labs.org/display/SP/Building+Your+Debian+Package">Building Your Debian Package</a> guide on the OPF Wiki I was able to create a rudimentary skeleton structure for Debian packaging. I was also able to build a Debian package for <em>jpylyzer</em>. The exercise revealed some problems with the Debian setup for <em>jpylyzer</em>. The most important ones are:</p> <ul> <li>It’s unclear how <em>jpylyzer</em>’s <em>changelog</em> file is supposed to be generated. Perhaps there’s a dependency on some external service (the OPF Github 2 Changelog service?), but I cannot find any documentation on how to make this work!</li> <li>The makefile calls <em>PyInstaller</em> in a non-standard an undocumented way. This is easy to fix locally if you are familiar with <em>PyInstaller</em>, but not so otherwise. Also, the interfaces of versions 1.5 and 2 of <em>PyInstaller</em> are different, and depending of what version you are running this may require additional changes to the makefile.</li> <li>Even though I was able to build a Debian package for <em>jpylyzer</em>, it still ended up with some <em>lintian</em> errors.</li> </ul> <p>I also came across a few minor errors in the OPF guide. I left a short comment on this <a href="http://wiki.opf-labs.org/display/SP/Building+Your+Debian+Package">here</a> (scroll to bottom). Overall, I found the guide really helpful, and it provides an accessible and relatively painless introduction to the packaging process.</p> <h2 id="reference">Reference</h2> <p><a href="http://wiki.opf-labs.org/display/SP/Building+Your+Debian+Package">Building Your Debian Package (OPF Wiki)</a></p> <h2 id="post-scriptum">Post scriptum</h2> <p>Proof again that it’s always a bad idea to come up with a clever title for a blog post without <a href="https://www.google.com/#hl=en&amp;gs_rn=11&amp;gs_ri=psy-ab&amp;cp=2&amp;gs_id=zz&amp;xhr=t&amp;q=%22adventures+in+Debian+Packaging%22&amp;es_nrs=true&amp;pf=p&amp;sclient=psy-ab&amp;oq=%22adventures+in+Debian+Packaging%22&amp;gs_l=&amp;pbx=1&amp;bav=on.2,or.r_qf.&amp;bvm=bv.45580626,d.d2k&amp;fp=6b211d71a5d752ed&amp;biw=1140&amp;bih=553">Googling</a> it first: after writing this post I found out that the <a href="http://mhvlug.org/">Mid Hudson Valley Linux and Open Source Users Group</a> will be organising a meeting called <a href="http://mhvlug.org/meetings/2013/adventures-in-debian-packaging"><em>Adventures in Debian Packaging</em></a> later this year in Poughkeepsie, NY. Completely unrelated to this blog, of course, but it’s only fair to give it a mention. Well, there you go.</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2013/04/23/adventures-debian-packaging/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2013/04/23/adventures-debian-packaging https://bitsgalore.org/2013/04/23/adventures-debian-packaging 2013-04-23T00:00:00+02:00 <p>The most important new feature of the recently released <a href="http://www.iso.org/iso/catalogue_detail.htm?csnumber=57229">PDF/A-3</a> standard is that, unlike <em>PDF/A-2</em> and <em>PDF/A-1</em>, it allows you to embed <em>any</em> file you like. Whether this is a good thing or not is the subject of some <a href="http://blogs.loc.gov/digitalpreservation/2012/11/all-in-embedded-files-in-pdfa/">heated on-line discussions</a>. But what do we actually mean by <em>embedded files</em>? As it turns out, the answer to this question isn’t as straightforward as you might think. One of the reasons for this is that in colloquial use we often talk about “embedded files” to describe the inclusion of <em>any</em> “non-text” element in a <em>PDF</em> (e.g. an image, a video or a file attachment). On the other hand, the word “embedded files” in the <em>PDF</em> standards (including <em>PDF/A</em>) refers to something much more specific, which is closely tied to <em>PDF</em>’s internal structure.</p> <!-- more --> <h2 id="embedded-files-and-embedded-file-streams">Embedded files and embedded file streams</h2> <p>When the <em>PDF</em> standard mentions “embedded files”, what it really refers to is a specific data structure. <em>PDF</em> has a <em>File Specification Dictionary</em> object, which in its simplest form is a table that contains a reference to some external file. <em>PDF 1.3</em> extended this, making it possible to embed the contents of referenced files directly within the body of the <em>PDF</em> using <em>Embedded File Streams</em>. They are described in detail in Section 7.11.4 of the <a href="http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/PDF32000_2008.pdf"><em>PDF</em> Specification (ISO 32000)</a>. A <em>File Specification Dictionary</em> that refers to an embedded file can be identified by the presence of an <em>EF</em> entry.</p> <p>Here’s an example (source: <a href="http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/PDF32000_2008.pdf">ISO 32000</a>). First, here’s a file specification dictionary:</p> <pre><code class="language-data">31 0 obj &lt;&lt;/Type /Filespec /F (mysvg.svg) /EF &lt;&lt;/F 32 0 R&gt;&gt; &gt;&gt; endobj </code></pre> <p>Note the <em>EF</em> entry, which references another <em>PDF</em> object. This is the actual embedded file stream. Here it is:</p> <pre><code class="language-data">32 0 obj &lt;&lt;/Type /EmbeddedFile /Subtype /image#2Fsvg+xml /Length 72&gt;&gt; stream …SVG Data… endstream endobj </code></pre> <p>Note that the part between the <em>stream</em> and <em>endstream</em> keywords holds the actual file data, here an <em>SVG</em> image, but this could really be anything!</p> <p>So, in short, when the <em>PDF</em> standard mentions “embedded files”, this really means <em>Embedded File Streams</em>.</p> <h2 id="so-what-about-embedded-images">So what about “embedded” images?</h2> <p>Here’s the first source of confusion: if a <em>PDF</em> contains images, we often colloquially call these “embedded”. However, internally they are not represented as <em>Embedded File Streams</em>, but as so-called <em>Image XObjects</em>. (In fact the <em>PDF</em> standard also includes yet another structure called <em>inline images</em>, but let’s forget about those just to avoid making things even more complicated.)</p> <p>Here’s an example of an <em>Image XObject</em> (again taken from <a href="http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/PDF32000_2008.pdf">ISO 32000</a>):</p> <pre><code class="language-data">10 0 obj % Image XObject &lt;&lt; /Type /XObject /Subtype /Image /Width 100 /Height 200 /ColorSpace /DeviceGray /BitsPerComponent 8 /Length 2167 /Filter /DCTDecode &gt;&gt; stream …Image data… endstream endobj </code></pre> <p>Similar to embedded filestreams, the part between the <em>stream</em> and <em>endstream</em> keywords holds the actual image data. The difference is that only a limited set of pre-defined formats are allowed. These are defined by the <em>Filter</em> entry (see Section 7.4 in <a href="http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/PDF32000_2008.pdf">ISO 32000</a>). In the example above, the value of <em>Filter</em> is <em>DCTDecode</em>, which means we are dealing with <em>JPEG</em> encoded image data.</p> <h2 id="embedded-file-streams-and-file-attachments">Embedded file streams and file attachments</h2> <p>Going back to embedded file streams, you may now start wondering what they are used for. According to Section 7.11.4.1 of <a href="http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/PDF32000_2008.pdf">ISO 32000</a>, they are primarily intended as a mechanism to ensure that external references in a <em>PDF</em> (i.e. references to other files) remain valid. It also states:</p> <blockquote> <p>The embedded files are included purely for convenience and need not be directly processed by any conforming reader.</p> </blockquote> <p>This suggests that the usage of embedded file streams is simply restricted to file attachments (through a <em>File Attachment Annotation</em> or an <em>EmbeddedFiles</em> entry in the document’s name dictionary).</p> <p>Here’s a sample file (created in <em>Adobe Acrobat</em> 9) that illustrates this:</p> <p><a href="http://www.opf-labs.org/format-corpus/pdfCabinetOfHorrors/fileAttachment.pdf">http://www.opf-labs.org/format-corpus/pdfCabinetOfHorrors/fileAttachment.pdf</a></p> <p>Looking at the underlying code we can see the <em>File Specification Dictionary</em>:</p> <pre><code class="language-data">37 0 obj &lt;&lt; /Desc() /EF&lt;&lt;/F 38 0 R&gt;&gt; /F(KSBASE.WQ2) /Type/Filespec/UF(KSBASE.WQ2)&gt;&gt; endobj </code></pre> <p>Note the <em>/EF</em> entry, which means the referenced file is embedded (the actual file data are in a separate stream object).</p> <p>Further digging also reveals an <em>EmbeddedFiles</em> entry:</p> <pre><code class="language-data">33 0 obj &lt;&lt; /EmbeddedFiles 34 0 R /JavaScript 35 0 R &gt;&gt; endobj </code></pre> <p>However, careful inspection of <a href="http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/PDF32000_2008.pdf">ISO 32000</a> reveals that embedded file streams can also be used for multimedia! We’ll have a look at that in the next section…</p> <h2 id="embedded-file-streams-and-multimedia">Embedded file streams and multimedia</h2> <p>Section 13.2.1 (Multimedia) of the <a href="http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/PDF32000_2008.pdf">PDF Specification (ISO 32000)</a> describes how multimedia content is represented in <em>PDF</em> (emphases added by me):</p> <blockquote> <ul> <li> <p><strong>Rendition actions</strong> (…) shall be used to begin the playing of multimedia content.</p> </li> <li>A rendition action associates a <strong>screen annotation</strong> (…) with a <strong>rendition</strong> (…)</li> <li><strong>Renditions</strong> are of two varieties: <strong>media renditions</strong> (…) that define the characteristics of the media to be played, and selector renditions (…) that enables choosing which of a set of media renditions should be played.</li> <li><strong>Media renditions</strong> contain entries that specify what should be played (…), how it should be played (…), and where it should be played (…)</li> </ul> </blockquote> <p>The actual data for a media object are defined by <em>Media Clip Objects</em>, and more specifically by the <em>media clip data dictionary</em>. Its description (Section 13.2.4.2) contains a note, saying that this dictionary “may reference a URL to a streaming video presentation or a movie <em>embedded in the PDF file</em>”. The description of the media clip data dictionary (Table 274) also states that the actual media data are “either a full file specification or a form XObject”.</p> <p>In plain English, this means that multimedia content in <em>PDF</em> (e.g. movies that are meant to be rendered by the viewer) may be represented internally as an embedded file stream.</p> <p>The following sample file illustrates this:</p> <p><a href="http://www.opf-labs.org/format-corpus/pdfCabinetOfHorrors/embedded_video_quicktime.pdf">http://www.opf-labs.org/format-corpus/pdfCabinetOfHorrors/embedded_video_quicktime.pdf</a></p> <p>This <em>PDF</em> 1.7 file was created in <em>Acrobat</em> 9, and if you open it you will see a short <em>Quicktime</em> movie that plays upon clicking on it.</p> <p>Digging through the underlying <em>PDF</em> code reveals a <em>Screen Annotation</em>, a <em>Rendition Action</em> and a <em>Media clip data dictionary</em>. The latter looks like this:</p> <pre><code class="language-data">41 0 obj &lt;&lt; /CT(video/quicktime) /D 42 0 R /N(Media clip from animation.mov) /P&lt;&lt;/TF(TEMPACCESS)&gt;&gt; /S/MCD &gt;&gt; endobj </code></pre> <p>It contains a reference to another object (42 0), which turns out to be a <em>File Specification Dictionary</em>:</p> <pre><code class="language-data">42 0 obj &lt;&lt; /EF&lt;&lt;/F 43 0 R&gt;&gt; /F(&lt;embedded file&gt;) /Type/Filespec /UF(&lt;embedded file&gt;) &gt;&gt; endobj </code></pre> <p>What’s particularly interesting here is the <em>/EF</em> entry, which means we’re dealing with an embedded file stream here. (The actual movie data are in a stream object (43 0) that is referenced by the file specification dictionary.)</p> <p>So, the analysis of this sample file confirms that embedded filestreams are actually used by <em>Adobe Acrobat</em> for multimedia content.</p> <h2 id="what-does-pdfa-say-on-embedded-file-streams">What does <em>PDF/A</em> say on embedded file streams?</h2> <p>In <a href="http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=38920"><strong>PDF/A-1</strong></a>, embedded file streams are not allowed at all:</p> <blockquote> <p>A file specification dictionary (…) shall not contain the <strong>EF</strong> key. A file’s name dictionary shall not contain the <strong>EmbeddedFiles</strong> key</p> </blockquote> <p>In <a href="http://www.iso.org/iso/home/store/catalogue_tc/catalogue_detail.htm?csnumber=50655"><strong>PDF/A-2</strong></a>, embedded file streams <em>are</em> allowed, but only if the embedded file itself is <em>PDF/A</em> (1 or 2) as well:</p> <blockquote> <p>A file specification dictionary, as defined in ISO 32000-1:2008, 7.11.3, may contain the <strong>EF</strong> key, provided that the embedded file is compliant with either ISO 19005-1 or this part of ISO 19005.</p> </blockquote> <p>Finally, in <a href="http://www.iso.org/iso/home/store/catalogue_tc/catalogue_detail.htm?csnumber=57229"><strong>PDF/A-3</strong></a> this last limitation was dropped, which means that <em>any</em> file may be embedded<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>.</p> <h2 id="does-this-mean-pdfa-3-supports-multimedia">Does this mean <em>PDF/A-3</em> supports multimedia?</h2> <p><strong>No, not at all!</strong> Even though nothing stops you from embedding multimedia content (e.g. a <em>Quicktime</em> movie), you wouldn’t be able to use it as a renderable object inside a <em>PDF/A-3</em> document. The reason is that the <em>annotations</em> and <em>actions</em> that are needed for this (e.g. <em>Screen</em> annotations and <em>Rendition</em> actions, to name but a few) are not allowed in <em>PDF/A-3</em>. So effectively you are only able to use embedded file streams as attachments.</p> <h2 id="adobe-adding-to-the-confusion">Adobe adding to the confusion</h2> <p>A few weeks ago the embedding issue came up again in a <a href="http://fileformats.wordpress.com/2012/12/22/not-pdf/">blog post by Gary McGath</a>. One of the comments there is from Adobe’s Leonord Rosenthol (who is also the Project Leader for <em>PDF/A</em>). After correctly pointing out some mistakes in both the original blog post and in an earlier a comment by me, he nevertheless added to the confusion by stating that objects that are are rendered by the viewer (movies, etc.) all use <em>Annotations</em>, and that embedded files (which he apparently uses a a synonym to attachments) are handled in a completely different manner. This doesn’t appear to be completely accurate: at least one class of renderable objects (screen annotations/rendition actions) may be using embedded filestreams. Also, embedded files that are used as attachments may be associated with a <em>File Attachment Annotation</em>, which means that “under the hood” both cases are actually more similar than first meets the eye (which is confirmed by the analysis of the 2 sample files in the preceding sections). Contributing to this confusion is also the fact that Section 7.11.4 of <a href="http://www.adobe.com/content/dam/Adobe/en/devnet/acrobat/pdfs/PDF32000_2008.pdf">ISO 32000</a> erroneously states that embedded file streams are only used for non-renderable objects like file attachments, which is contradicted by their allowed use for multimedia content.</p> <h2 id="does-any-of-this-matter-really">Does any of this matter, really?</h2> <p>Some might argue that the above discussion is nothing but semantic nitpicking. However, details like these <em>do</em> matter if we want to do a proper assessment of preservation risks in <em>PDF</em> documents. As an example, <a href="/2012/12/19/identification-pdf-preservation-risks-apache-preflight-first-impression">in this previous blog post</a> I demonstrated how a <em>PDF/A</em> validator tool can be used to profile <em>PDF</em>s for “risky” features. Such tools typically give you a list of features. It is then largely up to the user to further interpret this information.</p> <p>Now suppose we have a pre-ingest workflow that is meant to accept <em>PDF</em>s with multimedia content, while at the same time rejecting file attachments. By only using the presence of an embedded file stream (reported by both <em>Apache</em>’s and <em>Acrobat</em>’s <em>Preflight</em> tools) as a rejection criterion, we could end up unjustly rejecting files with multimedia content as well. To avoid this, we also need to take into account what the embedded file stream is used for, and for this we need to look at what annotation types are used, and the presence of any <em>EmbeddedFiles</em> entry in the document’s name dictionary. However, if we don’t know precisely <em>which</em> features we are looking for, we may well arrive at the wrong conclusions!</p> <p>This is made all the worse by the fact that preservation issues are often formulated in vague and non-specific ways. An example is <a href="http://wiki.opf-labs.org/display/AQuA/Embedded+objects+in+PDFs">this issue on the OPF Wiki on the detection of “embedded objects”</a>. The issue’s description suggests that images and tables are the main concern (both of which aren’t strictly speaking embedded objects). The <a href="http://wiki.opf-labs.org/display/AQuA/Detect%2C+extract+and+analyse+embedded+objects+in+PDFs">corresponding solution page</a> subsequently complicates things further by also throwing file attachments in the mix. In order to solve issues like these, it is helpful to know that images are (mostly) represented as <em>Image XObjects</em> in <em>PDF</em>. The solution should then be a method for detecting <em>Image XObjects</em>. However, without some background knowledge of <em>PDF</em>’s internal data structure, solving issues like these becomes a daunting, if not impossible task.</p> <h2 id="final-note">Final note</h2> <p>In this blog post I have tried to shed some light on a number of common misconceptions about embedded content in <em>PDF</em>. I might have inadvertently created some new ones in the process, so feel free to contribute any corrections or additions using the comment fields below.</p> <p>The <em>PDF</em> specification is vast and complex, and I have only addressed a limited number of its features here. For instance, one might argue that a discussion of embedding-related features should also include fonts, metadata, <em>ICC</em> profiles, and so on. The coverage of multimedia features here is also incomplete, as I didn’t include <em>Movie Annotations</em> or <em>Sound Annotations</em> (which preceded the <em>Screen Annotations</em>, which are now more commonly used). These things were all left out here because of time and space constraints. This also means that further surprises may well be lurking ahead!</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2013/01/09/what-do-we-mean-embedded-files-pdf/">Open Preservation Foundation blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>Source: <a href="http://www.pdfa.org/2012/10/pdf-association-newsletter-issue-26/">this unofficial newsletter item</a>, as at this moment I don’t have access to the full specification of <em>PDF/A-3</em>. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2013/01/09/what-do-we-mean-embedded-files-pdf https://bitsgalore.org/2013/01/09/what-do-we-mean-embedded-files-pdf 2013-01-09T00:00:00+01:00 <p>The <em>PDF</em> format contains various features that may make it difficult to access content that is stored in this format in the long term. Examples include (but are not limited to):</p> <ul> <li>Encryption features, which may either restrict some functionality (copying, printing) or make files inaccessible altogether.</li> <li>Multimedia features (embedded multimedia objects may be subject to format obsolescence)</li> <li>Reliance on external features (e.g. non-embedded fonts, or references to external documents)</li> </ul> <!-- more --> <p>A more exhaustive overview is given here:</p> <p><a href="https://zenodo.org/record/801661">Adobe Portable Document Format - Inventory of long-term preservation risks</a></p> <p>and also here:</p> <p><a href="https://web.archive.org/web/20130515073645/http://libraries.stackexchange.com/questions/964/what-preservation-risks-are-associated-with-the-pdf-file-format">https://web.archive.org/web/20130515073645/http://libraries.stackexchange.com/questions/964/what-preservation-risks-are-associated-with-the-pdf-file-format</a></p> <p>When <em>creating</em> a <em>PDF</em>, it is possible to minimise these risks by using one of the <em>PDF/A</em> standards, which delineate a number of <em>PDF</em> feature profiles that are unlikely to result in any long-term accessibility problems. However, the simple fact is that most <em>PDF</em>s that are out there are not <em>PDF/A</em>.</p> <h2 id="pdf-profiling">PDF Profiling</h2> <p>For assessing risks in existing collections, it would be helpful to be able to screen or profile <em>PDF</em>s for specific ‘risky’ features, such as encryption or font embedding. Since <em>PDF/A</em> was specifically designed to eliminate these ‘risky’ features, one would expect that <em>PDF/A</em> validators (i.e. software tools that check the conformance of a <em>PDF</em> file against the <em>PDF/A</em> specification) would be able to provide some useful information on this.</p> <p>In a first attempt to test whether this approach is feasible at all, I did some tests with <em>Apache Preflight</em>, an open-source <em>PDF/A-1</em> validator that is part of the <a href="http://pdfbox.apache.org/"><em>Apache PDFBox</em></a> library.The specific objectives of this work were:</p> <ul> <li>To get a first impression of the <em>Apache Preflight</em> (part of <em>PDFBox</em>) <em>PDF/A-1b</em> validator.</li> <li>To investigate if <em>Apache Preflight</em> is able to detect unwanted (from a preservation point of view) features in <em>PDF</em> files (i.e. <em>PDF</em>s that are not necessarily of the <em>PDF/A</em> sub-type) such as password protection, encryption and non-embedded fonts.</li> <li>To provide a comparison with the <em>Preflight</em> module of <em>Adobe Acrobat</em> 9.5.</li> <li>To decide if doing more work on <em>Apache Preflight</em> (more elaborate testing, possible involvement in its development) are worthwhile.</li> </ul> <p>The results can be found in the report <a href="https://zenodo.org/record/2556637">Identification of preservation risks in PDF with Apache Preflight: a first impression</a></p> <h2 id="the-archivists-pdf-cabinet-of-horrors">The Archivist’s PDF Cabinet of Horrors</h2> <p>The report’s findings are to a large extent based on a suite of small, simple test files that were created especialy for this work. Each file contains one ‘risky’ feature, with focus on the following feature classes:</p> <ul> <li>Encryption</li> <li>Multimedia</li> <li>Scripts</li> <li>Fonts</li> <li>File attachments</li> <li>External references</li> <li>Byte corruption</li> </ul> <p>The dataset can be found here:</p> <p><a href="http://www.opf-labs.org/format-corpus/pdfCabinetOfHorrors/">http://www.opf-labs.org/format-corpus/pdfCabinetOfHorrors/</a></p> <h2 id="link-to-full-report">Link to full report</h2> <p><a href="https://zenodo.org/record/2556637">Identification of preservation risks in PDF with Apache Preflight: a first impression</a></p> <h2 id="update-january-2013">Update (January 2013)</h2> <p>Since the report was published, a number of improvements have been made to Apache Preflight which should fix some of the reported issues. I haven’t tested the latest version yet, but will try doing this some time soon.</p> <h2 id="update-march-2013">Update (March 2013)</h2> <p>During the SPRUCE <a href="http://wiki.opf-labs.org/display/SPR/SPRUCE+Hackathon+Leeds%2C+Unified+Characterisation">hackathon on unified characterisation</a> (Leeds, 11-12 March, 2013) additional work was done on this. See this <a href="https://openpreservation.org/blogs/2013-03-15-pdf-eh-another-hackathon-tale">blog post by Pete Cliff</a> for more details. Importantly, the tests done during the hackathon showed that <em>Apache Preflight</em>’s ability to identify ‘risky’ features has improved significantly since I published my report back in December, and the issues that are mentioned in the report appear to have been largely resolved!</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2012/12/19/identification-pdf-preservation-risks-apache-preflight-first-impression/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2012/12/19/identification-pdf-preservation-risks-apache-preflight-first-impression https://bitsgalore.org/2012/12/19/identification-pdf-preservation-risks-apache-preflight-first-impression 2012-12-19T00:00:00+01:00 <p>I’ve already written a number of blog posts on format validation of <em>JP2</em> files. Format validation is only a one aspect of a quality assessment workflow. Digitisation guidelines typically impose various constraints on the technical characteristics of preservation and access images. For example, they may state that a preservation master must be losslessly compressed, and that its progression order must be <em>RPCL</em>. A <em>format profile</em> is a set of such technical constraints. The process that compares the technical characteristics of a file against a format profile is sometimes called <a href="http://wiki.opf-labs.org/pages/viewpage.action?pageId=6062098"><em>Policy Driven Validation</em></a>. This corresponds to what <em>JHOVE2</em> refers to as <a href="https://bitbucket.org/jhove2/main/wiki/Glossary"><em>Assessment</em></a> (which I think is a better description).</p> <p>This blog post describes a simple method for doing a rule-based assessment of <em>JP2</em> images. It uses <a href="http://en.wikipedia.org/wiki/Schematron">Schematron</a>, which is a rule-based validation language, to ‘validate’ the output of <em>jpylyzer</em> against a profile. Before getting into any technical details, let’s first have a look at an example of a format profile.</p> <!-- more --> <h2 id="example-format-profile">Example format profile</h2> <p>The table below shows the format profile that we’ll be using throughout this blog post, which is a typical ‘access’-oriented profile using lossy compression. Note that it is provided here for illustrative purposes only!</p> <table> <thead> <tr> <th><strong>Parameter</strong></th> <th><strong>Value</strong></th> </tr> </thead> <tbody> <tr> <td>File format</td> <td>JP2 (JPEG 2000 Part 1)</td> </tr> <tr> <td>Compression type</td> <td>Lossy (irreversible 9-7 wavelet filter)</td> </tr> <tr> <td>Colour transform</td> <td>Yes (only for colour images)</td> </tr> <tr> <td>Number of decomposition levels</td> <td>5</td> </tr> <tr> <td>Progression order</td> <td>RPCL</td> </tr> <tr> <td>Tile size</td> <td>1024 x 1024</td> </tr> <tr> <td>Code block size</td> <td>64 x 64</td> </tr> <tr> <td>Precinct size</td> <td>256 x 256 for 2 highest resolution levels; 128 x 128 for remaining resolution levels</td> </tr> <tr> <td>Number of quality layers</td> <td>8</td> </tr> <tr> <td>Target compression ratio</td> <td>20:1</td> </tr> <tr> <td>Error resilience</td> <td>Start-of-packet headers; end-of-packet headers; segmentation symbols</td> </tr> <tr> <td>Grid resolution</td> <td>Stored in “Capture Resolution” fields</td> </tr> <tr> <td>ICC profiles</td> <td>Embedded using “Restricted ICC” method</td> </tr> <tr> <td>Capture metadata</td> <td>Embedded in XML box</td> </tr> </tbody> </table> <h2 id="corresponding-properties-in-jpylyzer-output">Corresponding properties in <em>jpylyzer</em> output</h2> <p><em>Jpylyzer</em> provides information on <em>all</em> of the technical characteristics that are listed in the table. You can check this yourself by running <em>jpylyzer</em> on any <em>JP2</em> file and looking at the resulting output. A few examples:</p> <ul> <li> <p>Compression type - value of <em>transformation</em> field:</p> <pre><code class="language-data">/jpylyzer/properties/contiguousCodestreamBox/cod/transformation </code></pre> </li> <li> <p>Progression order - value of <em>order</em> field:</p> <pre><code class="language-data">/jpylyzer/properties/contiguousCodestreamBox/cod/order </code></pre> </li> <li> <p>ICC profiles - value of <em>meth</em> field:</p> <pre><code class="language-data">/jpylyzer/properties/jp2HeaderBox/colourSpecificationBox/meth </code></pre> </li> <li> <p>Grid resolution - presence of <em>captureResolutionBox</em> element:</p> <pre><code class="language-data">/jpylyzer/properties/jp2HeaderBox/resolutionBox </code></pre> </li> </ul> <h2 id="expressing-the-profile-as-a-set-of-assessable-rules">Expressing the profile as a set of assessable rules</h2> <p>In order to assess <em>jpylyzer</em>’s output against the profile, we first need to translate the profile to a set of assessable rules. This is where Schematron comes in. Look at parameter ‘Compression type’ in the table. In the previous section we saw that it corresponds to the <em>transformation</em> field in <em>jpylyzer</em>’s output. Below is a Schematron rule that asserts if <em>transformation</em> has the required value:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;s:rule</span> <span class="na">context=</span><span class="s">"/jpylyzer/properties/contiguousCodestreamBox/cod"</span><span class="nt">&gt;</span> <span class="nt">&lt;s:assert</span> <span class="na">test=</span><span class="s">"transformation = '9-7 irreversible'"</span><span class="nt">&gt;</span>wrong transformation<span class="nt">&lt;/s:assert&gt;</span> <span class="nt">&lt;/s:rule&gt;</span> </code></pre></div></div> <p>In words, the rule asserts that the value of <em>transformation</em> (which is located in <em>/jpylyzer/properties/contiguousCodestreamBox/</em>) equals <em>9-7 irreversible</em>. If the rule fails, this will result in the error message “wrong transformation”.</p> <p>Both the location (<em>context</em>) and the test statement are expressed using <a href="http://www.w3schools.com/xpath/xpath_syntax.asp">XPath syntax</a>, which allows more complex tests as well.</p> <h2 id="check-that-value-doesnt-exceed-threshold">Check that value doesn’t exceed threshold</h2> <p>The following rule checks if the compression ratio doesn’t exceed a threshold value (this is actually a bit tricky, as for images that don’t contain much information very high compression ratios may be obtained without losing quality):</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;s:rule</span> <span class="na">context=</span><span class="s">"/jpylyzer/properties"</span><span class="nt">&gt;</span> <span class="nt">&lt;s:assert</span> <span class="na">test=</span><span class="s">"compressionRatio &amp;lt; 35"</span><span class="nt">&gt;</span>Too much compression<span class="nt">&lt;/s:assert&gt;</span> <span class="nt">&lt;/s:rule&gt;</span> </code></pre></div></div> <p>(Note that the character reference “&amp;lt” represents “&lt;”, which isn’t allowed in <em>XML</em>.)</p> <h2 id="check-if-element-exists">Check if element exists</h2> <p>The following Schematron rule checks if the <em>captureResolutionBox</em> element exists:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;s:rule</span> <span class="na">context=</span><span class="s">"/jpylyzer/properties/jp2HeaderBox/resolutionBox"</span><span class="nt">&gt;</span> <span class="nt">&lt;s:assert</span> <span class="na">test=</span><span class="s">"captureResolutionBox"</span><span class="nt">&gt;</span>no capture resolution box<span class="nt">&lt;/s:assert&gt;</span> <span class="nt">&lt;/s:rule&gt;</span> </code></pre></div></div> <h2 id="outcome-depends-on-values-of-multiple-elements">Outcome depends on values of multiple elements</h2> <p>Here’s a more complex rule that checks whether a colour transformation (<em>multipleComponentTransformation</em>) was used while creating the image. A colour transformation is only possible for colour images, so in order to make this work for grayscale images as well, the rule must take into account that <em>multipleComponentTransformation</em> will be ‘no’ in that case (<em>nC</em> represents the number of image components):</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;s:rule</span> <span class="na">context=</span><span class="s">"/jpylyzer/properties/contiguousCodestreamBox/cod"</span><span class="nt">&gt;</span> <span class="nt">&lt;s:assert</span> <span class="na">test=</span> <span class="s">"(multipleComponentTransformation = 'yes') and (../../jp2HeaderBox/imageHeaderBox/nC = '3') or (multipleComponentTransformation = 'no') and (../../jp2HeaderBox/imageHeaderBox/nC = '1')"</span><span class="nt">&gt;</span> no colour transformation<span class="nt">&lt;/s:assert&gt;</span> <span class="nt">&lt;/s:rule&gt;</span> </code></pre></div></div> <h2 id="multiple-element-instances">Multiple element instances</h2> <p>Our profile states that the precinct size must be 256 x 256 for the 2 highest resolution levels, and 128 x 128 for the remaining ones. These occur as multiple instances of the <em>precinctSize</em> and <em>precinctSizeY</em> element in <em>jpylyzer</em>’s output, which we can handle as follows (note: for 5 decomposition levels we will have 6 resolution levels):</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;s:rule</span> <span class="na">context=</span><span class="s">"/jpylyzer/properties/contiguousCodestreamBox/cod"</span><span class="nt">&gt;</span> <span class="nt">&lt;s:assert</span> <span class="na">test=</span><span class="s">"precinctSizeY[1] = '128'"</span><span class="nt">&gt;</span>precinctSizeY doesn't match profile<span class="nt">&lt;/s:assert&gt;</span> <span class="nt">&lt;s:assert</span> <span class="na">test=</span><span class="s">"precinctSizeY[2] = '128'"</span><span class="nt">&gt;</span>precinctSizeY doesn't match profile<span class="nt">&lt;/s:assert&gt;</span> <span class="nt">&lt;s:assert</span> <span class="na">test=</span><span class="s">"precinctSizeY[3] = '128'"</span><span class="nt">&gt;</span>precinctSizeY doesn't match profile<span class="nt">&lt;/s:assert&gt;</span> <span class="nt">&lt;s:assert</span> <span class="na">test=</span><span class="s">"precinctSizeY[4] = '128'"</span><span class="nt">&gt;</span>precinctSizeY doesn't match profile<span class="nt">&lt;/s:assert&gt;</span> <span class="nt">&lt;s:assert</span> <span class="na">test=</span><span class="s">"precinctSizeY[5] = '256'"</span><span class="nt">&gt;</span>precinctSizeY doesn't match profile<span class="nt">&lt;/s:assert&gt;</span> <span class="nt">&lt;s:assert</span> <span class="na">test=</span><span class="s">"precinctSizeY[6] = '256'"</span><span class="nt">&gt;</span>precinctSizeY doesn't match profile<span class="nt">&lt;/s:assert&gt;</span> <span class="nt">&lt;/s:rule&gt;</span> </code></pre></div></div> <h2 id="the-full-profile-as-a-schema">The full profile as a schema</h2> <p>A sample schema that covers all aspects of the example format profile is available <a href="https://github.com/bitsgalore/jpylyzerProfileDemo/blob/master/demoAccessLossy.sch">here</a>.</p> <h2 id="assessment-of-jpylyzer-output-against-the-schema">Assessment of <em>jpylyzer</em> output against the schema</h2> <p>For the actual assessment (or validation) of <em>jpylyzer</em> output against the schema a couple of options exist. Probably the most widely-used one is the <a href="http://www.schematron.com/implementation.html">ISO Schematron reference implementation</a>. Validation using that software involves a number of successive <em>XSLT</em> stylesheet transformations. A more accessible (but probaby less performant) alternative is the <a href="http://www.probatron.org/probatron4j.html"><em>Probatron</em> command-line executable</a>. Using <em>Probatron</em>, asssessment of a <em>JP2</em> would typically involve the following two steps:</p> <h3 id="1-run-jpylyzer">1. Run <em>jpylyzer</em></h3> <p>For example:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jpylyzer balloon.jp2 <span class="o">&gt;</span> balloon_jp2.xml </code></pre></div></div> <h3 id="2-validate-jpylyzers-output-against-the-schema">2. Validate <em>jpylyzer</em>’s output against the schema</h3> <p>Example:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java <span class="nt">-jar</span> probatron.jar balloon<span class="se">\_</span>jp2.xml profile.sch <span class="o">&gt;</span> balloon_jp2_assessment.xml </code></pre></div></div> <h2 id="example-output">Example output</h2> <p>The above procedure produces an <em>XML</em> file that contains a <em>failed assert</em> element for each test that failed. For example, the output below is generated if the number of layers is wrong:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;svrl:failed-assert</span> <span class="na">test=</span><span class="s">"layers = '8'"</span> <span class="na">location=</span><span class="s">"/jpylyzer[1]/properties[1]/contiguousCodestreamBox[1]/cod[1]"</span> <span class="na">line=</span><span class="s">"45"</span> <span class="na">col=</span><span class="s">"550"</span><span class="nt">&gt;</span> <span class="nt">&lt;svrl:text&gt;</span>wrong number of layers<span class="nt">&lt;/svrl:text&gt;</span> <span class="nt">&lt;/svrl:failed-assert&gt;</span> </code></pre></div></div> <h2 id="demo">Demo</h2> <p>I created a small <a href="https://github.com/bitsgalore/jpylyzerProfileDemo">demo</a> that illustrates the assessment procedure. It includes two <em>JP2</em> images, the full schema of the example profile of this blog post, and a Windows batch file. For the moment it is located in my personal Github, but the schemas will probably be included in upcoming <em>jpylyzer</em> releases. To use the demo, just <a href="https://github.com/bitsgalore/jpylyzerProfileDemo/zipball/master">download the ZIP file</a>, unzip it, open the batch file in a text editor and follow the instructions at the top of the file.</p> <h2 id="final-note">Final note</h2> <p>Although this blog post only covers the assessment of <em>JP2</em> images using <em>jpylyzer</em>, the same procedure can be used for other formats and tools (provided that the tools are capable of producing <em>XML</em> output). Second, knowing that a <em>JP2</em> is valid and conforms to a technical profile is certainly important, but it doesn’t say anything about the (quality of the) actual image content. So in an operational setting this will often require additional checks (e.g. a pixel-wise comparison between source and destination images).</p> <h2 id="acknowledgement">Acknowledgement</h2> <p>Big thanks go out to Adam Retter (The National Archives) for his suggestion to use <em>Schematron</em>, <a href="https://twitter.com/bitsgalore/status/232829808608948224">just as I was struggling to make this work in <em>XSD</em></a>. Adam also shared some of his own <em>Schematron</em> schemas with me, which were a starting point for the work presented here.</p> <h2 id="useful-links">Useful links</h2> <ul> <li><a href="http://www.schematron.com/">Schematron</a></li> <li><a href="http://www.w3schools.com/xpath/xpath_syntax.asp">XPath syntax</a></li> <li><a href="http://jpylyzer.openpreservation.org/">Jpylyzer</a></li> <li><a href="http://www.probatron.org/probatron4j.html">Probatron</a></li> <li><a href="https://github.com/bitsgalore/jpylyzerProfileDemo">Demo: check if JP2 file matches a technical profile</a></li> <li><a href="https://github.com/bitsgalore/jpylyzerProfileDemo/zipball/master">Demo (download as ZIP)</a></li> </ul> <h2 id="post-script-february-2019">Post script, February 2019</h2> <p>Since this post was originally published, <em>jpylyzer</em>’s output format has changed slightly: from version 1.14.0 onward, all output elements have an associated namespace. This means that the Schematron rules must be adapted accordingly. A <a href="https://github.com/KBNLresearch/jprofile/tree/master/jprofile/schemas">set of example Schematron defintions that work with current versions of <em>jpylyzer</em> can be found here</a>. They are part of <a href="https://github.com/KBNLresearch/jprofile"><em>jprofile</em></a>, a simple tool that we use at the KB to assess JP2s from external suppliers. The source code of <em>jprofile</em> also demonstrates how to do this type of assessment in Python.</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2012/09/04/automated-assessment-jp2-against-technical-profile/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2012/09/04/automated-assessment-jp2-against-technical-profile https://bitsgalore.org/2012/09/04/automated-assessment-jp2-against-technical-profile 2012-09-04T00:00:00+02:00 <p>The purpose of this post is to give a brief introduction to creating, editing and submitting format signatures (or ‘<em>magic</em>’ entries) for the well-known <em>File</em> tool. The occasion for this was some work I did last week on improving <em>File</em>’s identification of the <em>JPEG 2000</em> formats. I had some difficulty finding any easy-to-follow documentation that describes how to do this. The information is all out there, but it’s pretty fragmented. So, I wrote this brief tutorial, which is intended as an accessible introduction to <em>magic</em> editing. It only covers the very basics, but hopefully this is enough to overcome some initial stumbling blocks.</p> <!-- more --> <h2 id="how-to-get-file">How to get File</h2> <p><em>File</em> is part of most <em>Unix</em>/<em>Linux</em> distributions. If you’re a <em>Windows</em> user I would recommend to get <a href="http://www.cygwin.com/">Cygwin</a>, which includes the latest version of <em>File</em>. A stand-alone <em>Windows</em> port of <em>File</em> does exist, but apparently it is not maintained anymore.</p> <h2 id="how-it-works">How it works</h2> <p>Just like tools such as <em>DROID</em>, <em>Fido</em> and <em>Apache Tika</em>, <em>File</em>’s identification is based on format signatures (which in the case of <em>File</em> are usually called <em>magic numbers</em>. These are stored in a <em>magic</em> file, which is located in the <em>magic</em> directory. Typical locations of this <em>magic</em> directory are:</p> <pre><code class="language-data">/usr/share/misc </code></pre> <p>Or:</p> <pre><code class="language-data">/usr/share/file </code></pre> <p>Or, on CygWin:</p> <pre><code class="language-data">/cygwin/usr/share/misc/ </code></pre> <h2 id="files-in-the-magic-directory-compiled-versus-source">Files in the magic directory: compiled versus source</h2> <p>Inside the <em>magic</em> directory you will see a file called <em>magic.mgc</em>. This is a <em>compiled</em> <em>magic</em> file. This is the one that <em>File</em> uses by default (although you can override this behaviour, which we’ll see in a moment).</p> <p>Depending on your system, you may also see another file that is simply called <em>magic</em>. This is the uncompiled (i.e. human-readable) source of the <em>magic</em> file.</p> <p>Note that in the source distribution of <em>File</em>, the source <em>magic</em> is organised in a directory structure. You can see this in the <a href="https://github.com/glensc/file/tree/master/magic/Magdir">Github mirror of File’s source repository</a>.</p> <h2 id="compiling-a-magic-source-file">Compiling a magic source file</h2> <p>So how do we get from a source file to a compiled <em>magic</em> file? Easy: just run <em>File</em> with the <code class="language-plaintext highlighter-rouge">-C</code> (compile) switch, and use the <code class="language-plaintext highlighter-rouge">-m</code> switch to specify the source file. For example, supposing we have a source file called <em>myMagic</em>, compile it using:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>file <span class="nt">-C</span> <span class="nt">-m</span> myMagic </code></pre></div></div> <p>This produces the compiled <em>magic</em> file <em>myMagic.mgc</em>. You can then use the compiled file by specifying its name using (again) the <code class="language-plaintext highlighter-rouge">-m</code> switch, e,.g.:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>file <span class="nt">-i</span> <span class="nt">-m</span> myMagic <span class="k">*</span> </code></pre></div></div> <p>Note that in the above case <em>File</em> actually uses <em>myMagic.mgc</em> (apparently it expects this extension); the following command line:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>file <span class="nt">-i</span> <span class="nt">-m</span> myMagic.mgc <span class="k">*</span> </code></pre></div></div> <p>produces identical results.</p> <h2 id="format-of-the-magic-source-file">Format of the magic source file</h2> <p>So what does the <em>magic</em> source file look like? I will only stick to the basics here, as an exhaustive description of the <em>magic</em> source format is given in the <a href="http://manpages.ubuntu.com/manpages/precise/en/man5/magic.5.html">man pages</a> of <em>magic</em>.</p> <p>The most important thing to remember is that each line in the file specifies a test. Each test is made up of 4 items, which are separated by one or more whitespace characters (usually tabs, but spaces appear to work as well):</p> <ul> <li><em>offset</em> - specifies the offset, in bytes, into the file of the data which is to be tested.</li> <li><em>type</em> - the type of the data to be tested (see <a href="http://manpages.ubuntu.com/manpages/precise/en/man5/magic.5.html">here</a> for a list of all possible values) .</li> <li><em>test</em> - the value to be compared with the value from the file.</li> <li><em>message</em> - the message to be printed if the comparison succeeds</li> </ul> <p>Here’s an example (actually this is the <em>JPEG 2000</em> <em>magic</em> that is currently used in <em>File</em> 5.11):</p> <pre><code class="language-data">0 string \x00\x00\x00\x0C\x6A\x50\x20\x20\x0D\x0A\x87\x0A JPEG 2000 </code></pre> <p>Here we have a test that compares the start of a file object (byte offset 0) against a 12-character string (here represented as hexadecimal codes). If the pattern is found, the text <em>JPEG 2000</em> will be printed to screen. Note that for most formats we will need further <em>sublevel tests</em>, which I will illustrate in the next section. Also, even though this introduction only decribes tests at fixed byte-positions, more sophisticated tests are possible as well. See the <a href="http://manpages.ubuntu.com/manpages/precise/en/man5/magic.5.html">man pages</a> for details on this.</p> <h2 id="improving-the-jpeg-2000-magic">Improving the JPEG 2000 magic</h2> <p>The main problem of the above <em>JPEG 2000</em> entry is that it only checks for the first 12 bytes of a file. This is enough for establishing that a file is part of the <em>JPEG 2000</em> ‘family’ of file formats, but it doesn’t tell you the exact sub-format, which can be any of the following:</p> <ul> <li><em>JP2</em> (basic still image format)</li> <li><em>JPX</em> (extended still image format)</li> <li><em>JPM</em> (compound format)</li> <li><em>MJ2</em> (Motion <em>JPEG 2000</em>)</li> </ul> <p>In other words: although <em>File</em> may be telling us that a file object matches <em>JPEG 2000</em>, this gives us zero information on whether it contains simple image data (<em>JP2</em>) or video content (<em>MJ2</em>)! Fortunately, the headers of each of the above formats include a <em>Brand</em> field, which is a 4-byte string of characters that uniquely identifies each format. This string starts at byte 20, and for <em>JP2</em> it equals ‘\x6a\x70\x32\x20’, so we can simply add this as a second test to the existing <em>magic</em> entry:</p> <pre><code class="language-data">&gt;20 string \x6a\x70\x32\x20 Part 1 (JP2) </code></pre> <p>Note the “<code class="language-plaintext highlighter-rouge">&gt;</code>” character, which indicates that this is a higher-level test on top of the previous one.</p> <h2 id="adding-mimetype-information">Adding mimetype information</h2> <p>Optionally, tests may also be associated with a <em>mimetype</em>. In that case the line that represents the test is followed by a second line, which contains the following items:</p> <ul> <li><em>!:mime</em> - indicates that this line is a <em>mimetype</em> declaration</li> <li><em>MIME Type</em> - the actual <em>mimetype</em></li> </ul> <p>For <em>JP2</em> this is:</p> <pre><code class="language-data">!:mime image/jp2 </code></pre> <h2 id="adding-the-other-formats">Adding the other formats</h2> <p>Repeating the above procedure for the full set of <em>JPEG 2000</em> formats we end up with this:</p> <pre><code class="language-data">0 string \x00\x00\x00\x0C\x6A\x50\x20\x20\x0D\x0A\x87\x0A JPEG 2000 &gt;20 string \x6a\x70\x32\x20 Part 1 (JP2) !:mime image/jp2 &gt;20 string \x6a\x70\x78\x20 Part 2 (JPX) !:mime image/jpx &gt;20 string \x6a\x70\x6d\x20 Part 6 (JPM) !:mime image/jpm &gt;20 string \x6d\x6a\x70\x32 Part 3 (MJ2) !:mime video/mj2 </code></pre> <h2 id="compiling-the-file">Compiling the file</h2> <p>To create the compiled file, follow the simple steps below::</p> <ol> <li> <p>Save your <em>magic</em> entry to a file (e.g. <em>jpeg2000Magic</em>). See also the <a href="https://github.com/bitsgalore/jp2kMagic/blob/master/magic/jpeg2000Magic">complete entry on my personal Github repo</a>.</p> </li> <li> <p>If you’re using <em>Windows</em>, you may need to convert <em>Windows</em>-style linebreaks to <em>Unix</em> linebreaks, for which you can use the <em>dos2unix</em> tool (which is included in <em>Cygwin</em>):</p> </li> </ol> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dos2unix jpeg2000Magic </code></pre></div></div> <ol> <li>Now compile the file:</li> </ol> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>file <span class="nt">-C</span> <span class="nt">-m</span> jpeg2000Magic </code></pre></div></div> <p><em>Et voilà</em>, our compiled <em>magic</em> file is ready for use!</p> <h2 id="testing-it">Testing it</h2> <p>For testing, run <em>File</em> on any number of files, e.g.:</p> <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>file <span class="nt">-i</span> <span class="nt">-m</span> jpeg2000Magic <span class="k">*</span> </code></pre></div></div> <p>Here’s an example of the output you may get:</p> <pre><code class="language-data">balloon.jp2: image/jp2; charset=binary balloon.jpf: image/jpx; charset=binary balloon.jpm: image/jpm; charset=binary Speedway.mj2: video/mj2; charset=binary </code></pre> <h2 id="submitting-magic">Submitting magic</h2> <p>File’s <a href="https://github.com/glensc/file">source repository on Github</a> gives a number of guidelines for submitting <em>magic</em>. <em>Don’t</em> use the Github repo for submitting stuff or commenting, as it is just a read-only mirror! Instead, make your submits through the <a href="http://mx.gw.com/mailman/listinfo/file"><em>File</em> mailing list</a>, or use the <a href="http://bugs.gw.com/my_view_page.php">bug tracker</a>.</p> <h2 id="useful-links">Useful links</h2> <ul> <li><a href="http://www.darwinsys.com/file/">Fine Free File Command</a></li> <li><a href="http://manpages.ubuntu.com/manpages/precise/en/man5/magic.5.html">Documentation of the <em>magic</em> file</a></li> <li><a href="https://github.com/glensc/file">Read-only mirror of <em>File</em> CVS repository, updated nightly</a></li> <li><a href="http://mx.gw.com/mailman/listinfo/file"><em>File</em> mailing list</a></li> <li><a href="http://www.cygwin.com/">Cygwin</a></li> <li><a href="http://www.iana.org/assignments/media-types/index.html">MIME Media Types at the Internet Assigned Numbers Authority</a></li> <li><a href="https://github.com/bitsgalore/jp2kMagic/blob/master/magic/jpeg2000Magic">Link to updated <em>JPEG 2000</em> <em>magic</em></a></li> </ul> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2012/08/09/magic-editing-and-creation-primer/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2012/08/09/magic-editing-and-creation-primer https://bitsgalore.org/2012/08/09/magic-editing-and-creation-primer 2012-08-09T00:00:00+02:00 <p>In this blog post I’ll be dusting off some old stuff for a change. The occasion for this is the following question, posted by Paul Wheatley on the <a href="http://libraries.stackexchange.com/">Libraries and Information Science Stack Exchange website</a> a few days ago:</p> <blockquote> <p><a href="http://libraries.stackexchange.com/questions/964/what-preservation-risks-are-associated-with-the-pdf-file-format">What preservation risks are associated with the PDF file format?</a></p> </blockquote> <!-- more --> <h2 id="report">Report</h2> <p>This reminded me of a <a href="https://zenodo.org/record/801661">report I wrote on this very subject</a> back in 2009. (Incidentally this was my very first foray into the wacky world of digital preservation, but that’s another story.) Originally this document was intended for internal use at the <em>KB</em>, but looking at it again, I think it may be of interest to a wider audience. It also aligns quite nicely with the upcoming work on a knowledge base of file-format related risks that will be done as part of the <a href="http://www.scape-project.eu/">SCAPE</a> project. The main idea here is to take a file format, identify its main (preservation-related) risks, and describe how “risky” features can be detected by existing (characterisation) tools. In fact I was envisaging something along these lines when I wrote <em>PDF</em> report in 2009, but other things got in the way, and I never got round to the final step. The SCAPE work should finally make this happen.</p> <p>Although the work on the knowledge base is still in its early stages, some very first results can be found <a href="http://wiki.opf-labs.org/display/TR/Formats">here</a>. The initial focus will be on <em>JPEG 2000</em> (<em>JP2</em>/<em>JPX</em>) and <em>PDF</em>.</p> <p>As for the report, I should add that some of it is a little rough around the edges, and you may note some gaps and not-quite-finished bits. This is also why we never released this first time around. Also, one aspect that is not well covered is <em>PDF</em>’s potential for transmitting viruses and other malware. Nevertheless, as a general introduction to the format and an overview of its main risks I think it’s not too shabby, but I’ll let you be the judge of that! As always, feel free to use the comment fields for you feedback and suggestions.</p> <h2 id="link-to-report">Link to report</h2> <p><a href="https://zenodo.org/record/801661"><em>Adobe Portable Document Format - Inventory of long-term preservation risks</em>, KB/ National Library of the Netherlands</a></p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2012/07/26/pdf-inventory-long-term-preservation-risks/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2012/07/26/pdf-inventory-long-term-preservation-risks https://bitsgalore.org/2012/07/26/pdf-inventory-long-term-preservation-risks 2012-07-26T00:00:00+02:00 <p>Over the last few years, the <a href="http://idpf.org/epub"><em>EPUB</em></a> format has gained widespread popularity in the consumer market. The <a href="http://www.kb.nl/index-en.html">KB</a> has been approached by a number of publishers that wish to use <em>EPUB</em> for delivering some of their electronic publications. Surprisingly little information is available on the format’s suitability for archival preservation, apart from Library of Congress’ <a href="http://www.digitalpreservation.gov/formats/"><em>Sustainability of Digital Formats</em></a> web pages, which contain entries on <a href="http://www.digitalpreservation.gov/formats/fdd/fdd000278.shtml"><em>EPUB</em> 2</a> and <a href="http://www.digitalpreservation.gov/formats/fdd/fdd000308.shtml"><em>EPUB</em> 3</a>.</p> <p>So, the KB’s Departments of Collection and Collection Care requested a more detailed investigation of <em>EPUB</em>’s preservation credentials. More specifically, answers were needed to the following questions:</p> <ul> <li> <p>What are the main characteristics of EPUB?</p> </li> <li> <p>What functionality does EPUB provide, and is this sufficient for representing e.g. content with sophisticated layout and typography requirements?</p> </li> <li> <p>How well is the <em>EPUB</em> supported by software tools that are used in (pre-)ingest workflows?</p> </li> <li> <p>How suitable is <em>EPUB</em> for archival preservation? What are the main risks?</p> </li> </ul> <!-- more --> <h2 id="epub-for-archival-preservation">EPUB for archival preservation</h2> <p>The <a href="https://zenodo.org/record/839711">report <em>EPUB for archival preservation</em></a> is a first attempt at answering these questions as well as possible. It starts out with a simple example that illustrates the general structure of an EPUB file, followed by a more in-depth discussion on of specific aspects of the format. It then covers functionality-related aspects such as layout, appearance and multimedia support, and the main differences between <em>EPUB</em> 2 and <em>EPUB</em> 3.</p> <p>Support by characterisation tools is important for processing <em>EPUB</em> files in an operational workflow, so a brief review (and some preliminary tests) of relevant identification, validation and feature extraction tools is included as well.</p> <p>To assess the overall suitability of <em>EPUB</em> for preservation, the format was evaluated against a set of widely used criteria (mainly from The National Archives and Library of Congress). The final chapter wraps up the main conclusions, and suggests a number of recommendations.</p> <h2 id="community-input">Community input</h2> <p>Since it appears that not much has been published on <em>EPUB</em> within an archival preservation context so far, we would really appreciate to hear your thoughts on the report. Is anything important missing? Did I overlook any relevant tools? Is there anything in particular that you strongly disagree with? Please use the comment fields below to let us know!</p> <p>In addition, the final chapter contains two subsections with <em>Community Recommendations</em> and <em>Tool Recommendations</em>. These are all things we can do as a community to simplify the use of <em>EPUB</em> in archival settings. Please consider getting involved if you feel you could make a contribution.</p> <h2 id="link-to-report">Link to report</h2> <p><a href="https://zenodo.org/record/839711"><em>EPUB for archival preservation</em>, KB/ National Library of the Netherlands</a></p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2012/06/18/epub-archival-preservation/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2012/06/18/epub-archival-preservation https://bitsgalore.org/2012/06/18/epub-archival-preservation 2012-06-18T00:00:00+02:00 <h2 id="introduction">Introduction</h2> <p>In this blog post I will give a brief update of the latest <a href="http://jpylyzer.openpreservation.org/"><em>jpylyzer</em></a> developments. <em>Jpylyzer</em> is a validation and feature extraction tool for the <a href="http://www.jpeg.org/public/15444-1annexi.pdf">JP2 (JPEG 2000 Part 1)</a> still image format.</p> <!-- more --> <h2 id="history-of-jpylyzer">History of jpylyzer</h2> <p>Around mid-summer 2011, the <a href="http://www.kb.nl/index-en.html">KB</a> started initial preparations for migrating 146 TB of TIFF images from the Dutch <a href="http://www.metamorfoze.nl/english/home">Metamorfoze</a> program to JP2. We realised that the possibility of hardware failure (e.g. short network interruptions) during the migration process would imply a major risk for the creation of malformed and damaged files. Around the same time, we received some rather worrying reports from the <a href="http://www.bl.uk/">British Library</a>, who were confronted with JP2 images that contained damage that couldn’t be detected with existing tools such as <a href="http://hul.harvard.edu/jhove/">JHOVE</a>.</p> <p>This prompted me to have a go at writing a rudimentary software tool that was able to detect some simple forms of file corruption in JP2. A <a href="/2011/09/01/simple-jp2-file-structure-checker">blog post</a> I wrote on this resulted in quite a bit of feedback, and several people asked about the possibility to extend the tool’s functionality to a full-fledged JP2 validator and feature extractor. Since this fitted in nicely with some <a href="http://www.scape-project.eu/">SCAPE</a> work that was envisaged on quality assurance in imaging workflows, I started work on a <a href="/2011/12/14/prototype-jp2-validator-and-properties-extractor">first prototype of <em>jpylyzer</em></a>, which saw the light of day in December.</p> <p>In the remainder of this blog post I will outline the main developments that have happened since then.</p> <h2 id="refactoring-of-existing-code">Refactoring of existing code</h2> <p>Shortly after the release of the first prototype, my KB colleague René van der Ark spontaneously offered to do a refactoring job on my original code, which was clumsy and unnecessarily lengthy in places. This has resulted in a code that is more modular, and which adheres more closely to established programming practices. As a result, the refactored code is significantly more maintainable than the original one, which makes it easier for other programmers to contribute to <em>jpylyzer</em>. This should also contribute to the long-term sustainability of the software.</p> <h2 id="new-features">New features</h2> <p>Since the first prototype the following functionality was added to <em>jpylyzer</em>:</p> <ul> <li>New validator functions were added for <em>XML</em> boxes, <em>UUID</em> boxes, <em>UUID Info</em> boxes, <em>Palette</em> boxes and <em>Component Mapping</em> boxes.</li> <li>A check was added that verifies whether the number of tiles in an image matches the image- and tile size information in the codestream header.</li> <li>Another check was added that verifies whether all tile-parts within each tile exist.</li> <li>The ICC profile feature extraction function has been given an overhaul, and it now extracts all ICC header items. In addition, the output is now reported in a more user-friendly format.</li> <li>For codestream comments, all characters from the Latin character set are now supported.</li> <li>The reporting of the validation results has been made more concise. By default, <em>jpylyzer</em> now only reports the results of validation tests that <em>failed</em> (previously <em>all</em> test results were reported). This behaviour can be overruled with a new <em>–verbose</em> switch.</li> </ul> <p>In addition to the above, various bugs and minor issues have been addressed as well.</p> <h2 id="debian-packages">Debian packages</h2> <p>During the SCAPE Braga meeting in February, work started on the creation of <a href="http://en.wikipedia.org/wiki/Deb_%28file_format%29">Debian packages</a> for <em>jpylyzer</em>. The availability of Debian packages greatly simplifies <em>jpylyzer</em>’s installation on Linux-based systems. <a href="http://www.openplanetsfoundation.org/blogs/2012-02-15-sustainability-and-adoption-preservation-tools">This work</a> was done by Dave Tarrant <a href="http://www.southampton.ac.uk/">(University of Southampton)</a>, Miguel Ferreira, Rui Castro, Hélder Silva <a href="http://www.keep.pt/en">(KEEP Solutions)</a> and Rainer Schmidt <a href="http://www.ait.ac.at/">(AIT)</a>.</p> <h2 id="jpylyzer-now-hosted-by-opf">Jpylyzer now hosted by OPF</h2> <p>In order to make a tool sustainable, it is important that its maintenance and development are not solely dependent on one single institution or person. Because of this, <em>jpylyzer</em> is now hosted by the Open Planets Foundation, which ensures the involvement of a wider community. <em>Jpylyzer</em> also has <a href="http://jpylyzer.openpreservation.org/">its own home page on the OPF site</a>. It contains links to the source code, Windows executables, Debian packages and the User Manual.</p> <h2 id="jpylyzer-home-page">Jpylyzer home page</h2> <p><a href="http://jpylyzer.openpreservation.org/">http://jpylyzer.openpreservation.org/</a></p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2012/04/23/update-jpylyzer/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2012/04/23/update-jpylyzer https://bitsgalore.org/2012/04/23/update-jpylyzer 2012-04-23T00:00:00+02:00 <p>This will be my shortest blog post ever. Following up on <a href="/2011/12/14/prototype-jp2-validator-and-properties-extractor">my previous blog post on a prototype JP2 validator and properties extractor</a> (jpylyzer), there is now a comprehensive User Manual of the tool. Just follow the link below:</p> <p><a href="http://jpylyzer.openpreservation.org/userManual.html">http://jpylyzer.openpreservation.org/userManual.html</a></p> <p>Link to <em>jpylyzer</em> home page:</p> <p><a href="http://jpylyzer.openpreservation.org/">http://jpylyzer.openpreservation.org/</a></p> <p>Meanwhile work on <em>jpylyzer</em> remains ongoing, so watch this space for any updates on this.</p> <p><strong>Update February 2019</strong>: updated links in original blog post</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2012/01/10/jpylyzer-documentation/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2012/01/10/jpylyzer-documentation https://bitsgalore.org/2012/01/10/jpylyzer-documentation 2012-01-10T00:00:00+01:00 <p>A few months ago I wrote a <a href="/2011/09/01/simple-jp2-file-structure-checker">blog post</a> on a simple JP2 file structure checker. This led to some interesting online discussions on JP2 validation. Some people asked me about the feasibility of expanding the tool to a full-fledged JP2 validator. Despite some initial reservations, I eventually decided to dedicate a couple of weeks to writing a rough prototype. The first results of this work are now ready in the form of the <em>jpylyzer</em> tool. Although I initially intended to limit its functionality to validation (i.e. verification against the format specifications), I quickly realised that since validation would require the tool to extract and verify all header properties anyway, it would make little sense not to include this information in its output. As a result, <em>jpylyzer</em> is both a validator and a properties extractor.</p> <!-- more --> <h2 id="validation-in-a-nutshell">Validation in a nutshell</h2> <p>It is beyond the scope of this blog post to provide an in-depth description of how <em>jpylyzer</em> validates a JP2 file. This will all be covered in detail by a comprehensive user manual, which I will try to write over the following weeks. For now I will restrict myself to a very brief overview. First of all, it is helpful here to know that internally a JP2 file is made up of a number of building blocks that are called ‘boxes’. This is illustrated by the figure below:</p> <p><img src="/images/2011/12/jp2Boxes.png" alt="JP2 Boxes diagram" /></p> <p>Some of these boxes are required (indicated by solid lines in the figure), whereas others (depicted with dashed lines) are optional. Some boxes are ‘superboxes’ that contain other boxes. A number of boxes can have multiple instances, whereas others are always unique. In addition, the order in which the boxes may appear in a JP2 file is subject to certain restrictions. This is all defined by the <a href="http://www.jpeg.org/public/15444-1annexi.pdf">standard</a>. At the highest level, <em>jpylyzer</em> parses the box structure of a file and checks whether it follows the standard. At a lower level, the information that is contained within the boxes is often subject to restrictions as well. For instance, the header field that defines how the colour space of an image is specified only has two legal values; any other value is meaningless and would therefore invalidate the file. Finally, there are a number of interdependencies between property values. For instance, if the value of the ‘Bits Per Component’ field of an image equals 255, this implies that the JP2 Header box contains a ‘Bits Per Component’ box. There are numerous other examples; the important thing here is that I have tried to make <em>jpylyzer</em> as exhaustive as possible in this regard.</p> <p>It is also worth pointing out that <em>jpylyzer</em> checks whether any embedded ICC profiles are actually allowed, as JP2 has a number of restrictions in this regard. There is a slight (intentional) deviation from the standard here, as an <a href="http://jpeg2000wellcomelibrary.blogspot.com/2011/04/guest-post-color-in-jp2.html">amendment</a> to the standard is currently in preparation that will allow the use of “display device” profiles in JP2. The current version of <em>jpylyzer</em> is already anticipating this change, and will consider JP2s that contain such ICC profiles valid (provided that they do not contain any other errors of course).</p> <h2 id="whats-not-included-yet">What’s not included yet</h2> <p>As this is a first prototype, <em>jpylyzer</em> is still a work in progress. Although most aspects of the JP2 file format are covered, a few things are still missing at this stage:</p> <ul> <li>Support of the Palette and Component Mapping boxes (which are optional sub-boxes in the JP2 Header Box) is not included yet. The current version of <em>jpylyzer</em> recognises these boxes, but doesn’t perform any analyses on them. This will change in upcoming versions.</li> <li>The IPR, XML, UUID and UUID Info boxes are not yet supported either.</li> <li>The analysis and validation of the image codestream is still somewhat limited. Currently <em>jpylyzer</em> reads and validates the required parts of the main codestream header (for those who are in the know on this: the SIZ, COD and QCD markers). It also checks if the information in the codestream header is consistent with the JP2 image header (the information in both headers is partially redundant). Finally, it loops through all tile parts in an image, and checks if the length (in bytes) of each tile-part is consistent with the markers that delineate the start and end of each tile-part in the codestream. This is particularly useful for detecting certain types of image corruption where one or more bytes are missing from the codestream (either at the end or in the middle).</li> <li>For now only codestream comments that consist solely of ASCII characters are reported. As the standard permits the use of non-ASCII characters of the Latin (ISO/IEC 8859-15) character set, this means that codestream comments that contain e.g. accent characters are currently not reported by <em>jpylyzer</em>. This will change in upcoming versions.</li> </ul> <h2 id="downloads">Downloads</h2> <p>You can download the source code of <em>jpylyzer</em> from the following location<sup id="fnref:1"><a href="#fn:1" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>:</p> <p><a href="https://github.com/bitsgalore/jpylyzer/">https://github.com/bitsgalore/jpylyzer/</a></p> <p>This requires <a href="http://www.python.org/download/releases/2.7.2/">Python 2.7</a>, or <a href="http://www.python.org/getit/releases/3.2/">Python 3.2</a> or more recent. A word of warning though: due to a number of reasons the source code ended up somewhat clumsy and unnecessarily verbose (one colleague even remarked that looking at it induced nostalgic memories of good old GW-BASIC!). With some major refactoring the overall length of the code could probably be reduced to half its current size, and I may have a go at this at some later point. For now it’ll have to do as it is!</p> <p>I also created some Windows binaries for those who do not want to install Python. Just follow the link below, download the ZIP file and extract it to an empty directory. Then simply use ‘jpylyzer.exe’ directly on the command line:</p> <p><a href="https://github.com/bitsgalore/jpylyzer/downloads">https://github.com/bitsgalore/jpylyzer/downloads</a></p> <h2 id="a-final-word">A final word</h2> <p>Keep in mind that the current version of <em>jpylyzer</em> is still a prototype. There may (and probably will) be unresolved bugs, and it really shouldn’t be used in any operational workflows at this stage. So far I have tested it with a range of JPEG 2000 images (mostly JP2, but also some JPX), including some images that I deliberately corrupted. No matter how corrupt an image is, this should never cause <em>jpylyzer</em> to crash. Therefore, it would be extremely useful if people could test the tool on their worst and weirdest images, and report back in case of any unexpected results. For early 2012 I’m also planning to write a comprehensive user guide that gives some more details on the validation process, as well as an explanation on the reported properties. Support of the JP2 boxes that are currently missing will also follow around that time. Meanwhile, any feedback on the current prototype is highly appreciated!</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2011/12/14/prototype-jp2-validator-and-properties-extractor/">Open Preservation Foundation blog</a></p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1"> <p>Postscript, February 2019 - all links in this section point to an insanely outdated version of <em>jpylyzer</em>! Don’t use them, but go to the <em>jpylyzer</em> web site at <a href="http://jpylyzer.openpreservation.org/">http://jpylyzer.openpreservation.org/</a> instead (unless you’re some future software historian with an unlikely interest in the history of <em>jpylyzer</em>). <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div> https://bitsgalore.org/2011/12/14/prototype-jp2-validator-and-properties-extractor https://bitsgalore.org/2011/12/14/prototype-jp2-validator-and-properties-extractor 2011-12-14T00:00:00+01:00 <p>As I already briefly mentioned in a <a href="/2011/07/11/improved-identification-xml-python-experiment">previous blog post</a>, one of the objectives of the <a href="http://www.scape-project.eu">SCAPE</a> project is to develop an architecture that will enable large scale characterisation of digital file objects. As a first step, we are evaluating existing characterisation tools. The overall aim of this work is twofold. First, we want to establish which tools are suitable candidates for inclusion in the SCAPE architecture. As the enhancement of existing tools is another goal of SCAPE, the evaluation is also aimed at getting a better idea of the specific strengths and weaknesses of each individual tool. The outcome of this will be helpful for deciding what modifications and improvements are needed. Also, many of these tools are widely used outside of the SCAPE project, which means that the results will most likely be relevant to a wider audience (including the original tool developers).</p> <!-- more --> <h2 id="evaluation-of-identification-tools">Evaluation of identification tools</h2> <p>Over the last months, work on this has focused on format identification tools. This has resulted in a <a href="https://zenodo.org/record/840345">report</a> which is attached with this blog post. We have evaluated the following tools:</p> <ul> <li><a href="http://sourceforge.net/apps/mediawiki/droid/index.php?title=Main_Page">DROID</a> 6.0</li> <li><a href="https://github.com/openplanets/fido">FIDO</a> 0.9</li> <li><a href="http://www.darwinsys.com/file/">Unix File Utility</a></li> <li><a href="http://code.google.com/p/fits/">FITS</a> 0.5</li> <li><a href="https://bitbucket.org/jhove2/main/wiki/Home">JHOVE2</a></li> </ul> <p>All tools were evaluated against a set of 22 criteria. Extensive testing using real data has been a key part of the work. One area which, I think, we haven’t been able to tackle sufficiently so far is the accuracy of the tools. This is problematic, since it would require a test corpus where the format of each file object is known <em>a priori</em>. In most large data sets this information will be derived from the very same tools that we are trying to test, so we need to see if we can say anything meaningful about this in a follow-up.</p> <h2 id="involvement-of-tool-developers">Involvement of tool developers</h2> <p>Over the previous months we’ve been sending out earlier drafts of this document to the developers of DROID, FIDO, FITS and JHOVE2, and we have received a lot of feedback to this. In the case of FIDO, a new version is underway, and this should correct most (if not all) of the problems that are mentioned in the report. For the other tools we have also received confirmation that some of the found issues will be fixed in upcoming releases.</p> <h2 id="status-of-the-report-and-future-work">Status of the report and future work</h2> <p>The attached report should be seen as a living document. There will probably be one or more updates at some later point, and we may decide to include more tests using additional data. Meanwhile, as always, we appreciate any of your feedback on this!</p> <h2 id="link-to-report">Link to report</h2> <p><a href="https://zenodo.org/record/840345">Evaluation of characterisation tools – Part 1: Identification</a></p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2011/09/21/evaluation-identification-tools-first-results-scape/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2011/09/21/evaluation-identification-tools-first-results-scape https://bitsgalore.org/2011/09/21/evaluation-identification-tools-first-results-scape 2011-09-21T00:00:00+02:00 <p>Over the last few weeks I’ve been working on the design of a workflow that the KB is planning to use for the migration of a collection of (mostly old) TIFF images to JP2. One major risk of such a migration is that hardware failures during the migration process may result in corrupted images. For instance, one could imagine a brief network or power interruption that occurs while an image is being written to disk. In that case data may be missing from the written file. Ideally we would be able to detect such errors using format validation tools such as <a href="http://hul.harvard.edu/jhove/">JHOVE</a>. Some time ago Paul Wheatley reported that the BL at some point were dealing with corrupted, incomplete JP2 files that were nevertheless deemed “well-formed and valid” by JHOVE. So I started doing some experiments in which I deliberately butchered up some images, and subsequently checked to what extent existing tools would detect this.</p> <!-- more --> <p>I started out with removing some trailing bytes from a lossily compressed JP2 image. As it turned out, I could remove most of the image code stream (reducing the original 2 MB image to a mere 4 kilobytes!), but JHOVE would still say the file was “well-formed and valid”. I was also able to open and render these files with viewer applications such as Adobe Photoshop, Kakadu’s viewer and Irfanview. The behaviour of the viewer apps isn’t really a surprise, since the ability to render an image without having to load the entire code stream is actually one of the features that make JPEG 2000 so interesting for many access applications. JHOVE’s behaviour was a bit more surprising, and perhaps slightly worrying.</p> <h2 id="jp2structcheck-tool">jp2StructCheck tool</h2> <p>This made me wonder about a way to detect incomplete code streams in JP2 files. A quick glance at the <a href="http://www.jpeg.org/public/15444-1annexi.pdf">standard</a> revealed that image code streams should always be terminated by a two-byte ‘end of codestream marker’. As this is something that is straightforward to check, I fired up <a href="http://www.python.org/">Python</a> and ended up writing a very simple <a href="https://github.com/bitsgalore/jp2StructCheck">JP2 file structure checker</a>. Since the image code stream in JP2 does not have to be located at the end of the file (even though it usually is), it is necessary to do a superficial parsing of JP2’s ‘box’ structure (which is documented <a href="http://www.jpeg.org/public/15444-1annexi.pdf">here</a>). So I thought I might as well include an additional check that verifies if the JP2 contains all required boxes.</p> <p>In brief, when <em>jp2StructCheck</em> analyses a file, it first parses the top-level box structure, and collects the unique identifiers (or marker codes) of all boxes. If it encounters the box that contains the code stream, it checks if the code stream is terminated by a valid end-of-codestream marker. Finally, it checks if the file contains all the compulsory/required top-level boxes. These are:</p> <ul> <li>JPEG 2000 signature box</li> <li>File Type box</li> <li>JP2 Header box</li> <li>Contiguous Codestream box</li> </ul> <p>In order to test the box checking mechanism I did some additional image butchering, where I deliberately changed the tags of existing boxes so that they wouldn’t be recognised. When I subsequently ran these images through JHOVE, this revealed some additional surprises. For instance, after changing the markers of the Contiguous Codestream box or even the JP2 Header box (which effectively makes them unrecognisable), JHOVE would still report these images as “well-formed and valid” (although in the case of the missing JP2 Header box JHOVE did report an error).</p> <h2 id="limitations-of-jp2structcheck">Limitations of jp2StructCheck</h2> <p>It is important to note here that <em>jp2StructCheck</em> only checks the top-level boxes. In case of a superbox (which is a box that contains child boxes), it does not recurse into its child boxes. For example, it does not check if a JP2 Header box (which is a superbox) contains a Bits Per Component Box (which is required by the standard). So the scope of the tool is limited to a rather superficial check of the general file structure. It is <em>not</em> a JP2 validator, and it is certainly not a replacement for JHOVE (which performs a more in-depth analysis)! The main scope is to be able to detect certain types of file corruption that may occur as a result of hardware failure (e.g. network interruptions) during the creation of an image.</p> <p>In addition, the fact that a code stream is terminated by and end-of-codestream marker is no guarantee that the code stream is complete. For instance, if due to some hardware failure some part of the middle of the codestream is not written, <em>jp2StructCheck</em> will not detect this! It may be possible to improve the level of error detection by including additional codestream markers. This is something I might have a look at at some later point.</p> <h2 id="downloads">Downloads</h2> <p>I created a <a href="https://github.com/bitsgalore/jp2StructCheck">Github repository</a> that contains the source code of <em>jp2StructCheck</em>, some documentation, and a small data set with some test images.</p> <p>As some people may not want to install Python on their system, I also created a <a href="https://github.com/downloads/bitsgalore/jp2StructCheck/jp2StructCheck31082011distWin32.zip">binary distribution that should work on most Windows systems</a>.</p> <p>The documentation (in PDF format) is <a href="https://github.com/downloads/bitsgalore/jp2StructCheck/jp2StructCheck.pdf">here</a>.</p> <p>Finally, use <a href="https://github.com/downloads/bitsgalore/jp2StructCheck/testImages.zip">this link</a> to download the test images.</p> <h2 id="final-notes">Final notes</h2> <p>I’m curious to hear if anyone finds <em>jp2StructCheck</em> useful at all, so please feel free to use the comment fields below for your feedback (including reports on any bugs that may exist).</p> <h2 id="post-script-february-2019">Post script, February 2019</h2> <p>The <em>jp2StructCheck</em> tool is superseded by <a href="http://jpylyzer.openpreservation.org/"><em>jpylyzer</em></a> (of which <em>jp2StructCheck</em> was an early precursor). Unlike <em>jp2StructCheck</em>, <em>jpylyzer</em> is a full-fledged validator for the <em>JP2</em> format.</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2011/09/01/simple-jp2-file-structure-checker/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2011/09/01/simple-jp2-file-structure-checker https://bitsgalore.org/2011/09/01/simple-jp2-file-structure-checker 2011-09-01T00:00:00+02:00 <p>As a part of the <a href="http://www.scape-project.eu">SCAPE</a> project, I’m currently heavily involved in the evaluation of various file format identification tools. The overall aim of this work is to determine which tools are suitable candidates for inclusion in the SCAPE architecture. In addition, we’re also trying to get a better idea of each tool’s specific strengths and weaknesses, which will hopefully serve as useful input to the developers community. We’re actually planning to publish the first results of this work on the OPF blog some time soon, so you may want to keep your eyes peeled for that.</p> <!-- more --> <h2 id="identification-using-byte-signatures">Identification using byte signatures</h2> <p>In this blog entry I will focus on one particular area in which most identification tools appear to be struggling: the identification of XML files. Most identification tools try to establish a file’s format by looking for characteristic byte sequences, or ‘signatures’. Examples of tools that use this approach are <a href="http://sourceforge.net/projects/droid/">DROID</a>, <a href="https://github.com/openplanets/fido">Fido</a> and the <a href="http://darwinsys.com/file/">Unix File tool</a>. Signature-based identification works well for most binary formats, but for text-based formats the results are often less reliable. This also applies to XML. Signature-based tools typically identify XML by the presence of an XML declaration, which, in its simplest form, looks like this:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">&lt;?xml version="1.0"?&gt;</span> </code></pre></div></div> <p>The problem is that not all XML files actually contain an XML declaration. Also, the use of an XML declaration is not mandatory. The<a href="http://www.w3.org/TR/xml/">XML specification</a> states that for a file to qualify as “valid” XML it <em>should</em> contain the declaration. This merely means that the use of the declaration is recommended (which follows from the use of the word <em>“should”</em> and not <em>“must”</em>). XML files that don’t contain the declaration are by definition not “valid”, but they may still be “well-formed”.</p> <p>However, if (part of) the declaration is used as a signature, this means that any files that don’t have the declaration will not be identified as XML by any of the above tools. This is exactly what happened in our tests for DROID, Fido and the Unix File tool. DROID and Fido simply leave such files unidentified, whereas the Unix File tool identifies them as ‘plain text’ (which, of course, is correct at a lower level, but not very helpful). Unfortunately, such files are pretty common in practice.</p> <h2 id="using-an-xml-parser-to-identify-xml">Using an XML parser to identify XML</h2> <p>A different approach to identify these files would be to run them through an XML parser. If a parser can make sense of a file’s contents this means it is well-formed (but not necessarily valid!) XML. In all other cases, it’s something else.</p> <p>I ended up writing some <a href="http://www.python.org/">Python</a> code to see how this would work in practice. I first created two re-usable Python functions that check any given file for well-formedness using Python’s highly performant ‘expat’ parser (based on <a href="http://code.activestate.com/recipes/52256-check-xml-well-formedness/">original code by Farhad Fouladi</a>). I then wrote a simple command-line application around it, which is called “isXMLDemo.py”. The demo can be used to analyse one file at a time, or, alternatively, all files in a directory tree. The output is a formatted text file that contains, for each analysed file, the identification result (which is either “isXML” or “noXML”).</p> <p>I was surprised at how fast the XML parsing actually is. To give an indication, I used “isXMLDemo.py” to analyse a 1.15 GB dataset that contains 11,892 file objects. I ran this experiment under Microsoft Windows XP Professional using a PC with a 3 Ghz GenuineIntel processor and 1 GB RAM. The total time needed to analyse all files was about 90 seconds, which corresponds to an average throughput of about 131 files per second.</p> <h2 id="xml-parsing-in-fido">XML parsing in Fido?</h2> <p>Since the core functions that do the actual XML parsing are completely reusable, it would probably be fairly easy to incorporate this kind of identification into Fido. This would obviously have some impact on Fido’s performance, but not by very much. XML parsing could also be offered as an option. In that case, the decision on whether to parse or not to parse is up to the user.</p> <p>An obvious limitation of this approach is that it will not identify XML that is not well-formed. Also, it makes the line between identification and validation somewhat blurry, but in practical terms that shouldn’t be a real problem. Finally, one could argue that knowing that a file contains XML is not very informative at all, since it is merely a container for something else.  This was the subject of an <a href="http://www.openplanetsfoundation.org/blogs/2011-02-17-new-direction-file-characterisation">earlier blog post by Asger Blekinge</a>. However, even then, identifying the container is a necessary first step, and one that the current tools don’t seem to be too good at yet.</p> <h2 id="demo">Demo</h2> <p>For those who want to do some tests for themselves, I have <strike>attached the demo script to this post</strike> <a href="https://github.com/bitsgalore/isXMLDemo">uploaded the demo script to GitHub</a>. The <strike>ZIP file</strike> repository contains the Python script with its  documentation in PDF format. If you end up with any interesting results, or if you have any other thoughts on this: please report back in the comments!</p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2011/07/11/improved-identification-xml-python-experiment/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2011/07/11/improved-identification-xml-python-experiment https://bitsgalore.org/2011/07/11/improved-identification-xml-python-experiment 2011-07-11T00:00:00+02:00 <p>The JPEG 2000 compression standard is steadily becoming more and more popular in the archival community. Several large (national) libraries are now using the <a href="http://www.jpeg.org/public/15444-1annexi.pdf">JP2 format</a> (which corresponds to Part 1 of the standard) as the master format in mass digitisation projects. However, some aspects of the JP2 file format are defined in ways that are open to multiple interpretations. This applies to the embedding of <a href="http://en.wikipedia.org/wiki/ICC_profile">ICC profiles</a> (which are used to define colour space information), and the definition of grid resolution. This situation has lead to a number of interoperability issues that are potential risks for long-term preservation.</p> <!-- more --> <h2 id="paper">Paper</h2> <p>I recently addressed this in a <a href="http://www.dlib.org/dlib/may11/vanderknijff/05vanderknijff.html">paper</a> that has just been published in <a href="http://www.dlib.org/">D-Lib Magazine</a>. An earlier version of the paper was used as a ‘defect report’ by the <a href="http://www.jpeg.org/">JPEG committee</a>. The paper gives a detailed description of the problems, and shows to what extent the most widely-used JPEG 2000 encoders are affected by these issues.</p> <h2 id="solutions">Solutions</h2> <p>The paper also suggests some possible solutions. Importantly, none of the found problems require any changes to the actual file format; rather, some features should simply be defined slightly differently. In the case of the ICC profile issue this boils down to allowing a widely used class of ICC profiles that are currently prohibited in JPEG 2000. The resolution issue could be fixed by a more specific definition of the existing resolution fields.</p> <h2 id="amendment-to-the-standard">Amendment to the standard</h2> <p>Both issues will be addressed in an amendment to the standard. Rob Buckley provides more details on this (along with some interesting background information on colour space support in JP2) in a recent <a href="http://jpeg2000wellcomelibrary.blogspot.com/2011/04/guest-post-color-in-jp2.html">blog entry on the Wellcome Library’s JPEG 2000 blog</a>. As Rob puts it:</p> <blockquote> <p>The final outcome of all this will be a JP2 file format standard that aligns with current practice; supports RGB spaces such as Adobe RGB 1998, ProPhoto RGB and eci RGB v2; and provides a smooth migration path from TIFF masters as JP2 increasingly becomes used as an image preservation format.</p> </blockquote> <p>So, some relatively small adjustments to the standard could result in a significant improvement of the suitability of JP2 for preservation purposes.</p> <p>Since various institutions are using JPEG 2000 <em>now</em>, the paper also provides some practical recommendations that may help in mitigating the risks for existing collections.</p> <h2 id="link-to-paper">Link to paper</h2> <p>  <a href="http://www.dlib.org/dlib/may11/vanderknijff/05vanderknijff.html">JPEG 2000 for Long-term Preservation: JP2 as a Preservation Format</a></p> <hr /> <p>Originally published at the <a href="https://openpreservation.org/blog/2011/06/06/paper-jpeg-2000-preservation-9/">Open Preservation Foundation blog</a></p> https://bitsgalore.org/2011/06/06/paper-jpeg-2000-preservation-9 https://bitsgalore.org/2011/06/06/paper-jpeg-2000-preservation-9 2011-06-06T00:00:00+02:00 <p>In my <a href="http://www.dpconline.org/component/docman/doc_download/526-jp2knov2010vanderkniff">presentation</a> during the <a href="http://blog.wellcomelibrary.org/2010/11/wellcome-trust-hosts-jpeg-2000-seminar/">Wellcome Trust’s JPEG 2000 seminar</a> I discussed the suitability of JPEG 2000 (and more specifically its JP2 format) for long-term preservation. I highlighted the erroneous restriction in the JP2 (and JPX) format specification that only allows ICC profiles of the ‘input’ class to be used. This effectively prohibits the use of all working colour spaces such as Adobe RGB, which are defined using ‘display device’ profiles. I also showed how different software vendors interpret the format specification in subtly different ways, and how such issues can create problems in the long term, such as the loss of colour space and resolution information after some future migration.</p> <!-- more --> <p>This leads us to the question to which extent we can predict a specific file format’s suitability for long-term preservation. The answer is not that straightforward. The Library of Congress assesses file formats against 7 <a href="http://www.digitalpreservation.gov/formats/sustain/sustain.shtml">‘sustainability factors’</a>, whereas the National Archives have formulated <a href="http://www.nationalarchives.gov.uk/documents/selecting-file-formats.pdf">a list of 12 criteria</a>. It is beyond the scope of this blog post to present a detailed analysis of the extent to which JP2 lives up to either set of criteria. However, it is interesting to have a look at whether these criteria could have been helpful in identifying the issues covered by my presentation.</p> <h2 id="format-specifications">Format specifications</h2> <p>First, both the LoC’s ‘sustainability factors’ and the TNA criteria acknowledge the importance of having published specifications of a file format. The LoC uses a ‘Disclosure’ factor, which refers to “the existence of complete documentation, preferably subject to external expert evaluation”. TNA take this one step further by also defining a ‘Documentation Quality’ criterion, which expresses the degree to which documentation is comprehensive, accurate and comprehensible. This last criterion largely covers the JPEG 2000 ICC issue, although it’s questionable how useful this would have been to identify it <em>a priori</em>. A problem with errors and ambiguities in format specifications is that they can be incredibly easy to overlook, and you may only become aware of them after discovering that different software products interpret the specifications in slightly different ways.</p> <h2 id="adoption">Adoption</h2> <p>Formats that are widely used are typically well supported by an array of software tools, and such formats are unlikely to disappear into obsolescence. TNA expresses this through a ‘Ubiquity’ criterion, which essentially reflects a file format’s overall popularity. The definition of the LoC’s ‘Adoption’ factor includes a list of criteria that can be used as “evidence of adoption”. The first set of criteria here includes “bundling of tools with personal computers, native support in Web browsers or market-leading content creation tools, and the existence of many competing products for creation, manipulation, or rendering of digital objects in the format”. Note that JP2 isn’t doing particularly well when measured against any of these criteria. However, the LoC list adds that “a format that has been reviewed by other archival institutions and accepted as a preferred or supported archival format also provides evidence of adoption”. This certainly seems to be the case for JP2. But how relevant is this, really? Going back to the ICC profiles issue: the JP2 file format has been around for about 10 years now, and its acceptance by the archival community has been growing steadily over the last 5 years or so. Yet, this whole issue seems to have gone unnoticed in the archival community for all those years, and I think this is slightly worrying.</p> <p>Now let’s imagine for a moment that JP2 would have been picked up by the digital photography and graphic design communities. For such uses the ability to do proper colour management is a basic prerequisite, and limiting the support of ICC profiles to the ‘input’ class would have made the format virtually useless to these user communities. My guess is that in this -entirely fictional- scenario, the format specification would have either improved quickly (based on feedback from the user community), or the respective user communities would have simply stopped using the format altogether. The problem here seems to be that very few people in the archiving community are even aware of such things as colour spaces and colour management, let alone their importance within the context of preservation. With more established formats such as TIFF this may not be as much of a problem, if only because TIFF has been ‘road tested’ for decades by the photography and graphic design communities. As an archiving community we cannot fall back to any similar ‘road testing’ in the case of JP2. And this brings me to my next point.</p> <h2 id="importance-of-hands-on-experience">Importance of hands-on experience</h2> <p>Preservation criteria such as those of the LoC or TNA are invaluable for assessing the suitability of a format for preservation, but I believe it is equally important to have actual hands-on experience with the tools that are used for creating, modifying, and reading the format. For instance, the TNA criteria use the <em>number</em> of software tools that support a given format as an indicator for the extent of current software support of that format. But knowing the <em>number</em> of tools says nothing about how good or useful these tools actually are! In the case of JP2, quite a large number of (mostly free or open-source) tools exist that, under the hood, are using the open <a href="http://www.ece.uvic.ca/~mdadams/jasper/">JasPer</a> library. JasPer is known to have performance and stability issues that make it unsuitable for most professional applications (for which, I should emphasise, it was never developed in the first place!). These issues affect <em>all</em> software tools that are using JasPer. So, only counting the number of available tools may be simply missing the point without incorporating any additional quality criteria. But how would you define these?</p> <p>Part of the answer, I think, is that assessing a format’s suitability for long-term preservation is not a purely top-down process. Most of the software-related issues that I showed in my presentation were found by simply experimenting with actual files, encoders and characterisation tools: convert a TIFF to JP2; convert it back to TIFF; use existing metadata-extraction and characterisation tools such as <a href="http://www.sno.phy.queensu.ca/~phil/exiftool/">ExifTool</a> and <a href="http://hul.harvard.edu/jhove/">JHOVE</a> to analyse the in- and output files; try to understand the output of these tools; compare the output before and after the conversion, and so on. Such experiments are extremely useful for getting a feel for the strengths and weaknesses of specific software tools, and they can reveal problems that are not readily captured by pre-defined criteria. In some cases, their results may be used to refine existing criteria, or even add new ones.</p> <h2 id="final-notes-on-preservation-criteria">Final notes on preservation criteria</h2> <p>Although I wouldn’t downplay the importance of preservation criteria such as those used by the LoC or TNA, I think it’s important to realise that such criteria are largely based on theoretical considerations. In most cases they are not based on any empirical data, and as a result their predictive value is largely unknown. For example, <a href="http://blog.dshr.org/2009/01/are-format-specifications-important-for.html">an interesting blog post by David Rosenthal</a> argues that preserving the specifications of a file format doesn’t contribute anything to practical digital preservation. According to Rosenthal, the availability of working open-source rendering software is much more important, and he explains how “formats with open source renderers are, for all practical purposes, immune from format obsolescence”.</p> <p>This takes us directly to the lack of JPEG 2000-related activity in the open source community, which I also referred to in my presentation. Perhaps the best way to ensure sustainability of JPEG 2000 and the JP2 format would be to invest in a truly open JP2 software library, and release this under a free software license. This could either take the form of the development of a completely new library, or investing in the improvement and further development of an existing one, such as <a href="http://www.openjpeg.org/">OpenJPEG</a>. This would require an investment from the archival community, but the payoff may be well worth it.</p> <h2 id="acknowledgements">Acknowledgements</h2> <p>This blog entry was largely inspired by an e-mail discussion that was started by Richard Clark, and in particular by a contribution to this discussion by William Kilbride.</p> <hr /> <p>Originally published at the <a href="http://blog.wellcomelibrary.org/2010/12/guest-post-ensuring-the-suitability-of-jpeg-2000-for-preservation/">Wellcome Library Blog</a></p> https://bitsgalore.org/2010/12/02/ensuring-suitability-of-jpeg https://bitsgalore.org/2010/12/02/ensuring-suitability-of-jpeg 2010-12-02T00:00:00+01:00