ARCW National Forum, Aberystwyth

On Monday, June 9th - International Archives Day, I had the privilege to attend and speak at the Archives and Records Council Wales (ARCW) National Forum in Aberystwyth. The day included presentations on conservation, digital mapping, environmental policies; representing international students within university archives, reflecting narratives of resilience from Ugandan Indian migrants to Wales. Together, the day reflected the sheer breath of working being conducted by Welsh archives and the researchers they support.

A mosaic on the castle wall

It was a great day and I left feeling that the wider sector could learn a lot from the self-advocacy and coordination of archives in Wales. In a few months, all Welsh enclosures will be digitally mapped, georeferenced and made accessible by the Royal Commission on the Ancient and Historical Monuments of Wales (RCAHMW). In two years, the Commission will follow by doing the same with all 19th century OS maps having coordinated the initial digitisation with every regional archive in the country.

Far less impressive, I was asked to speak on the theme of AI in archives. I wanted to ensure that I framed Automated Text Recognition (ATR) and my experiments, mostly conducted with the National Library of Scotland, as a window onto broader discussions of embedding AI-enabled systems within places renowned by the public as trustworthy keepers of heritage. I wanted to show that archives utilising AI is a specific issue due to the very nature of archives as repositories of historical material. While we may take direction from wider AI assessments and other fields - health etc., this responsibility and the sectoral values surrounding it cannot be lost.

Despite conflating archives and libraries at points - my apologies, I received interesting questions - some practical about the efficacy of ATR on texts with illustrations. I suggested that these materials pose greater issue, alongside maps and music notation, but with the introduction of transformer LLMs - image recognition could likely be made interoperable with ATR. Others wanted to know how ATR accuracy compared to crowdsourcing and volunteers, with a need to quantify potential savings and find a mixed approach whereby human labour is best utilised. We also discussed archives' appetite for error, with no system achieving immaculate outputs free of manual correction.

A blowy day in Aberystwyth

Below is the presentation speech. You can find the relevant slides here via Zenodo: Critical AI Approaches at the National Library of Scotland

Brutalism worthy of 'The Brutalist' on Aberystwyth University campus near the conference venue

Hi everyone, Prynhawn da,

Thank you for inviting me to talk about the National Library of Scotland’s work on critical AI transcription. With newfound AI affordances for cultural heritage, libraries are encountering growing opportunities, demands and potential risks in delivering their digital collections. This has led the library sector to begin reasserting intellectual leadership in a space dominated by private actors with often superior resources - through community initiatives, library labs and critical infrastructure theorisation and development. 

As a small part of this critical turn, I’ll speak about Automated Text Recognition (ATR), the computational approach used in recognising and analysing handwriting and complex print, with equal surety, speed and accuracy as printed material using AI non-sequential processing. Some of you may be more familiar with Handwritten Text Recognition (HTR), however ATR is an increasingly established overarching term, which encompasses work on complex print. 

My hope is to move beyond an ATR tutorial or a ‘how-to’ presentation, instead - drawing on the NLS’s experiments in AI transcription - I’ll attempt to show how this work is transferable to other critical AI approaches in archives and libraries. What are the main lessons from this NLS experimentation? What potential affordances and limitations does ATR hold? 

I’ll begin with a general overview of ATR and its developmental history, including its main uses, before looking at our NLS experiments and my current work on eXplainable AI transcription. 

What is ATR? 

To begin answering whether ATR holds relevance for wider AI adoption in libraries, we first need to situate ourselves in the technology’s past development. This shows ATR to be complex and avoids relegating AI to ‘chat’ or chat-based interfaces, think ChatGPT or Gemini, instead demonstrating that such transcription technology is the product of accumulated solutions and stresses - beginning in the 19th century. 

ATR is preceded by the more familiar Optical Character Recognition (OCR), which made strides in electronically translating images-of-text into machine-readable outputs. OCR attempts to emulate human ways of reading, looking for contrast between written ink and document backgrounds and variations between character classes, dividing pages into further smaller elements -> text blocks, tables, words, and finally isolated characters. Schantz (1981) provides the only comprehensive history of OCR development, although he suggests its development came from ‘men of vision’ – the title of his opening chapter, neglecting the predominantly female workforce behind its development, which Hicks (2017) attempts to reinsert - a good read.

Despite this, Schantz’s general narrative is correct – OCR first emerged from disability studies with readers for the blind in the 1800s, where cursor movements produced distinctive sounds for the visually-impaired user. By the 1970s, after its use in mid-century military processes – deciphering correspondence and intelligence, OCR accuracy improved enough to be rolled-out commercially. However, OCR’s unsuitability for handwritten text and complex print was long-known. In 1969 Jacob Rabinow, a pioneer in developing OCR processes to be used in the US Postal Service, stated: 

“[T]he more control one puts into a document, the simpler and less costly the reading machine … the answer is ‘standardise’. Standardise the type of paper … standardise the quality of printing, standardise the format, and standardise the font” (Rabinow, 1969:  38-42). 

This, of course, leads to errorful outputs and ‘dirty OCR’ - shown here on an NLS sample of the Scottish Courant (1710). Broadly, the redder the square, the less confident the OCR is at recognising the text - of course this remains a guide, but highlights the potential for inaccurate results, impacting search systems and catalogues.

Why use ATR? 

As the final part of our overview, ATR has a number of uses for libraries - 

Beyond providing more accurate digital transcriptions, we found ATR to align with the NLS Reaching People strategy to make it “easier to discover the Library’s special and hidden collections through our programme of online listing, cataloguing and discovery work”. At first glance, then, HTR integration seems highly compatible with strategic values around access and democratisation, for both the NLS and the wider library profession. ATR can also ensure that outputs from archives are multivocal, more accessible and representative.

ATR also enables the cleaning of existing transcriptions; greater access to endangered language material, with ATR models trained for languages like Vietnamese ChuNom, displaced by modern Vietnamese with presently only has 100 scholars able to read the language; enhance layout analysis of varying directional scripts and a virtuous cycle of model training. Of course, Human-in-the-Loop correction is needed to achieve any one of these aims, otherwise the adage of ‘garbage in/garbage out’ applies. ATR, therefore, is a tool to alleviate non-critical manual labour and can be a trusted companion to transcription work. As we see in terms of NLS experimentation, it is not a means of replacing library labour, and at points increases the roles needed within the sector.  

Multiple ATRs exist, but we’ll focus on Transkribus, due to its being the largest consumer-level tool, as well as its unique governance structure as an academic cooperative of c. 230 co-owners across 30 countries. Transkribus regularly achieves 95% character accuracy on unpredictable handwritten scripts, after the manual transcription of c. 15,000 words/50 pgs., and 99% on complex print (with the layout not text being the main obstacle) with less training, c. 20 pgs. The cooperative supports a system that has transcribed 90 million historical text images, with more than 235,000 registered users. 

NLS experimentation

We now turn to our ATR experiments at the NLS. In 2021, we transcribed and made available the 1810-1811 diary of Marjory Fleming (1803-1811), a Scottish child author born in Kirkcaldy, Fife, who became posthumously famous for her diaries. The diaries provide an authoritative account of her life, schoolwork, and death, and were given a great deal of public attention in the late Victorian period. The Fleming diaries are the first NLS dataset created using HTR technology, with comparable accuracy - 89%, and the underlying data made available via the NLS Data Foundry online repository. This work also led to experiments in using HTR to form automated critical editions, with persons/places/speech/dates tagged and the aligned images and transcription made available via Transkribus Sites. The following slides will consider what other possibilities exist.

Transkribus was then applied across a range of heterogeneous NLS collections in the autumn of 2022, without tailoring to show any potential out-of-the-box savings - 

In the case of the Exam Papers and the Scottish newspapers, Transkribus was found to outperform current OCR solutions (ABBYY FineReader). These results show that pre-trained models can be applied to printed text collections, with HTR able to outperform OCR in cases where aggravating factors such as complex layout and font variation can negatively affect OCR accuracy. Default Transkribus models provided less accuracy than training a bespoke model in the manner demonstrated on the Fleming Diaries, but require far less resourcing due to there being no requirement for manual transcription. The only collection where Transkribus returned lower accuracy than OCR was for the Chapman and Myllar prints; this is because the text was manually corrected between 1996-1996 as part of a transcription project. 

We also found that Transkribus provides possibilities for extraction from a variety of media. For instance, recognising and extracting metadata from maps. The NLS has a large online collection of maps, which have in recent years been the subject of exploratory studies to analyse the benefits of Open Source tools and web-mapping applications. In order to understand the potential of ATR, we attempted to extract metadata relating to print codes and map tiles. Since the first revisions of Ordnance Survey maps in the mid-1800s, there have been markings on each map sheet that indicate the date of revision or printing. These are generally found in small print beneath or beside the area of the map plate. As such, they provide important contextual and provenance data for researchers. However, these printing codes can be immensely time consuming to extract manually. We found that it was possible to easily extract printing codes from maps in our test sample, utilising Transkribus, with the print codes appearing in a standardised way and the technology prompted to ignore the main map text. The left image shows an OS map with the Layout Analysis model applied and shows how specific aspects of the map can be identified.

Alongside this, we experimented with recognising the presence of music notation, conducted on a page from the National Library of Scotland’s (NLS) 19th century Glen Collection of Printed Music. There is clear potential here, with other pilot projects underway at the Austrian Academy of Science and University of Alicante, with work aiming to automatically recognise mediaeval rubrics and chant texts. 

Although there are considerable savings in time and resources in using ATR to generate transcripts of historical materials, it is not a panacea. If it is to be successfully used to increase access to information, it needs to be embedded into both digitisation workflows within libraries as well as public-facing digital library infrastructures. 

For instance, ATR requires NLS changes to image processing. Its OCR is not routinely applied to handwritten collections such as medieval manuscripts, so images captured during digitisation are not routinely deskewed. However, deskewing images has long been considered good practice for ATR; as it allows a documented page to be accurately segmented and recognised sequentially using Layout Analysis, with batch processing leads to quality control issues. Furthermore, the NLS digitisation team seeks to retain the material sense of the items they capture. They are therefore wary of manipulating items too heavily, as this will negatively impact upon the material authenticity of their scans. Documents uploaded to the Library’s Digital Object Database (DOD) include covers and blank pages, allowing readers to better understand the material aspects of the digitised items, but leading to additional processing.

The NLS also uses an Automated Ingest Tool (AIT), which splits the workflow of digitising, converting and transcribing the collection image into various stages that run on virtual machines. This process automatically converts images to PDFs and JPEGs, OCRing the latter with ABBYY FineReader. While our experiments showed Transkribus outperformed the accuracy of this combined tool, it is not already part of an established workflow - requires going in and out of different environments, which leads to additional overheads: training, time, cost.

In terms of cost, we presented the NLS Library Leadership Team with several options based on processing need.  We recommended Option 3 in 2022-2023, allowing Transkribus to be used on an ad-hoc basis for individual projects and digitisation proposals by curators. This builds in existing curatorial skills, and the increased awareness and understanding of Transkribus resulting from the placement. Option 3 would then allow staff to become more familiar with Transkribus, and constrain costs while a portfolio of small-scale projects test the real-world impact upon digital production workflows, gradually moving to Option 2 where Transkribus might be used to create core data models to support transcription of priority collections for the Library. This is where the NLS currently sits in its ATR approach.

Further work into eXplainable ATR approaches

Lastly, I want to mention my current work - which considers how ATR might support developing sector-wide ethical frameworks for AI adoption. CILIP provides an ethical framework that outlines library values such as transparency, confidentiality, social responsibility, and the public good. Reflections on AI adoption require a careful analysis of whether technologies challenge or enhance each of these values, and particularly the shared observation that libraries are an essential public good and fundamental to democratic societies. 

Through the project Recognising Hands, Recognising Process, we are developing the means to ascertain whether ATR can be aligned with eXplainable AI principles, which set out measures to ensure sufficient information is provided to nontechnical users to comprehend a model’s internal behaviour. To what level do libraries require XAI to utilise AI transcription at scale? XAI is seen as a way to scrutinise the technology’s transparency as well as anticipating its potential ethical implications when scaled. We have also challenged the perceived trade-off between XAI and model performance, which features heavily in scholarly literature and, in some cases, appears to forego attempts to prioritise building intelligible systems. This research also finds that a lack of consistent XAI around ATR leaves room for overhyping tool accuracy and creates potential for AI-driven approaches to disrupt social norms between libraries as trustworthy stewards of historical texts and end-users. This project, which uses a mixed-method of reviewing literature, interviewing NLS staff and experimenting with multiple ATR approaches across the same collection hopes to provide real-world recommendations for how to select, implement and test XAI approaches within libraries, as well as upskill staff in applying such principles.

Concluding remarks 

We’ve moved from looking at ATR as an isolated tool to detailing experiments with it against a broader backdrop of library systems (bureaucratic, cultural and digital), to finally asking whether ATR can provide insight into wider AI adoption within libraries. ATR, while bringing affordances, is not a panacea and outstanding issues remain - calculating the environmental cost, encouraging open data practices, thinking of its impact on legalistic frameworks and intellectual property, as well as the technical issues mentioned. In gathering authors from around the world to contribute to our Critical Approaches to ATR, due to be published next year by facet, we are hoping to address some of these challenges. The role of librarians in articulating benefits and drawbacks to this technology is paramount, with continued dialogue needed to ensure the views of those working closest with collections are heard. 

Thank you very much, I welcome any questions.