Text to Multimodal - Reflections on DH24 Conference, Washington D.C.

Last week, I had the privilege to speak at ADHO’s (Alliance of Digital Humanities Organisations) Digital Humanities (DH) Conference, at George Mason University near Washington D.C., although I presented online and so interacted with proceedings mostly through Whova – the conference app – from my desk.

Chaired by the British Library’s Mia Ridge, our session looked at methods for making text derived from archives “multimodal” for users. I spoke alongside Yuchen Yang, who displayed a tool for enabling users to prompt and return archived videos, in turn constructing a more intimate - personal story out of digital holdings. Yuchen stated that the tool was to “provide a simple interface for heritage institutions to engage with their visitors” and “step away from traditional search bars and filters”, fitting into Whitelaw’s 2015 call for more “generous” interfaces.

What interested me, alongside the storytelling elements of Yuchen’s work, was the notion of collections as data telling “histories as daily happenings without set context.” Although, in research, context is often key - the untethering of resources appears a decisive way to remind ourselves as library practitioners that, as Wilson (2006: 665) states, we should move beyond the purely technical study of systems toward understanding the role of technology in users’ everyday lives and interactions with collections. Both presentations therefore emphasised the need for collections to be digitally accessible and malleable to further develop library audiences. I thank everyone who spoke throughout the conference and the local organising committee at George Mason’s Roy Rosenzweig Centre.

Below is a truncated version of the DH talk:

Introduction - This paper presents a Digital Engagement Framework (DEF) for expanding AI-enabled Handwritten Text Recognition (HTR) within libraries, based on interviews conducted while on placement at the National Library of Scotland (NLS) between 2022-2023. With HTR’s clear potential to make digitised holdings of handwriting more accessible for audiences (Mühlberger et al., 2019), this paper shows how the technology can also enable libraries to put traditional values of preservation, collections’ development, collections’ awareness, and research support into further action. As such, the presentation culminates in an adapted version of Visser and Richardson’s (2013: 60) DEF framework, used for GLAM (galleries, libraries, archives, and museums) to map their own digital engagement activities onto.

Method - As Helsper and Eynon (2013: 696) discuss, DEFs emerged from the need to test for specific pathways to inclusion: accounting for sociodemographic factors, digital skills, and literacy, while holding potential in making complex technological workflows understandable for public audiences (Visser & Richardson, 2013: 6). The NLS formed a strong case study due to its involvement with our existing HTR research, as well as long record of audience engagement activities as a national institution, with the Library’s 2020-2025 strategy calling for “outstanding digital engagement” and focused on enabling creative collection usage (Reaching People, 2020: 13). We began by cataloguing extant and reviewing works on audience development in the library sector. This informed an Action Research (AR) method, analysing interventions into practice on a local level (Ebbutt, 1985: 156) through participative orientated interviews of the NLS staff currently undertaking such work (Bradbury, 2015: 1). These research activities were covered by University of Edinburgh’s ethics approval, with interviewees giving permission to be named in this paper.

Findings - Coupling HTR with user engagement appears as a creative and intentional way of ameliorating access issues to collections. Shah (2023), current National Librarian of Scotland, stressed HTR as holding the “possibility of opening up a number of different histories” which “can offer an opportunity for democratising the production of historical knowledge”, in turn enabling AI literacy to emerge in a meaningful way: opposed to fuelling pure “boosterism” from a perceived need to be seen as technologically adept. In this way, as Gooding (2023) states, libraries can re-emphasise their sectoral values, such as their focus on transparent data creation and intellectual freedom. A focus on creative but cautious HTR usage, adapted for certain collections, also dismantles problematic notions of universal digital engagement within libraries. As Shah (2023) continues to say, libraries, by chasing universal standardised approaches, often go “full throttle into something that we think is going to be great for the future, but then the future comes and what we have is actually not fit for purpose.”

HTR was also seen by Shah (2023) as providing user communities the ability to advocate for collections in fresh and modern ways, through co-creating digital resources via volunteer scanning. This benefit was seen in our trialling HTR-aligned scanners within the NLS Special Collections Reading Room. The Transkribus ScanTent, specifically, was seen to benefit community outreach, with the technology automatically uploading digitised materials to Transkribus (a predominate HTR platform) for further transcription processing. We performed staff drop-ins to form a base understanding of the technology and answer questions directly and approached the Library Leadership Team (LLT) for their support – demonstrating digital engagement’s reliance on senior management buy-in (Visser and Richardson, 2013: 5). The current ScanTent trial began in June 2023, and is ongoing. Through this trial we met fears surrounding the ScanTent allowing digitisation on an industrial scale within existing NLS copyright management and self-service photography guidelines.

However, the introduction of these scanners was not without issue. McCarron (2023), Head of NLS Reader Services, wondered whether high intensity scanning would disrupt other users and impact the Library as an inclusive “third space” with low entry thresholds. Goodnight and Jeitner (2017: 217) suggest that the great focus on libraries as interactive makerspaces has neglected the importance of libraries as spaces of quiet contemplation.  Assessing HTR’s possible disruption therefore is paramount.

After focusing on some specific aspects of the DEF, the below image shows the entire resource now available to the NLS. The DEF maps the roles of NLS staff interviewed onto user needs to expand its HTR provision. Although not comprehensive, it describes several existing NLS engagement activities as “assets” due to their proven success in public outreach. In particular, the NLS’s social media and areas of public engagement (such as workshops, exhibitions, and symposiums) are included. This panel is corresponded to “audiences in need of reaching.” Using preliminary audience development analysis, completed by The Audience Agency (2023a: 35) for the NLS: the DEF includes those with additional support requirements, as well as audiences outside of Scotland’s Central Belt (Edinburgh and Glasgow). Between “assets” and “audiences” are a set of ideas for “engagement” activities, decided on through action research interviewing, such as the co-creation of HTR models with users through crowdsourcing methods, especially Scottish Gaelic models. The “engagement” panel provides a clear multistage approach for NLS staff to follow in expanding HTR services: beginning on introductory blog posts, user onboarding, and interactive training sessions. “Reach” denotes how the NLS could use HTR processing to further publicise its collections and technological approaches, through social media (outlined in the “channels” panel) and involvement in the READ-COOP: the academic cooperative responsible for the development and maintenance of Transkribus. The two other areas of the DEF in need of explanation are the “guidelines” panel, which shows how HTR coincides with other internal NLS procedures, like the self-service photography guidelines mentioned, as well as newly constructed AI ethics frameworks. Lastly, to enable a more general focus in relating HTR to digital engagement, the “trends” panel shows current discussions within relevant fields from which activities can be performed against including crowdsourcing, post-custodial libraries, and the financial strains currently on the UK cultural heritage sector: which impact staff training, tool development and purchasing.

Resulting Digital Engagement Framework for expanding HTR at the National Library of Scotland

Conclusion - In conclusion, although a work-in-progress, this DEF offers a succinct way for libraries to plan future and ongoing HTR work to develop their audience outreach. This is to be adapted after receiving user feedback or for institutional contexts. In the NLS’s case, after further clarifying their relevant user communities and resolving potential issues surrounding HTR’s broader provision, such as with greater scanning, this DEF will aid in fully harnessing the technology: supported by robust metrics and user data.

References -

Bradbury, H. (2015). Introduction: How to Situate and Define Action Research. In: Hiliary Bradbury (ed.). The SAGE Handbook of Action Research. 3rd Edition. Thousand Oaks, CA.: Sage, pp. 1-14.

Ebbutt, D. (1985). Educational action research: some general concerns and specific quibbles. In R.G. Burgess (ed.). Issues in Educational Research: Qualitative methods. London: The Falmer Press, pp. 152-174.

Gooding, P. (2023). Recording. Collaboration, Transparency, and Technology: AI as a community challenge for Libraries. In: NLS AI Symposium, Edinburgh, 25 April. Available at: https://www.youtube.com/watch?v=l98O8GXLa-Q.

Goodnight, J., Jeitner, E. (2017). ‘Sending Out an SOS: Being Mindful of Students’ Need for Quiet Study Spaces.’ In: Smanatha Schmehl Hines, Kathryn Moore Crowe (eds). The Future of Library Space. Bingley: Emerald Publishing, pp. 217-213.

Helsper, E.J., Eynon, R. (2013). Distinct skill pathways to digital engagement. European Journal of Communication. 28(6):696-713. doi:  10.1177/0267323113499113

McCarron, L. (2023). Interview by Joseph Nockels [Microsoft Teams]. 27 April.

Mühlberger, G., Seaward, L., Terras, M., Ares Oliveira, S., Bosch, V., Bryan, M., Colutto, S., Déjean, H., Diem, M., Fiel, S., Gatos, B., Greinoecker, A., Grüning, T., Hackl, G., Haukkovaara, V., Heyer, G., Hirvonen, L., Hodel, T., Jokinen, M., Kahle, P., Kallio, M., Kaplan, F., Kleber, F., Labahn, R., Lang, E.M., Laube, S., Leifert, G., Louloudis, G., McNicholl, R., Meunier, J.-L., Michael, J., Mühlbauer, E., Philipp, N., Pratikakis, I., Puigcerver Pérez, J., Putz, H., Retsinas, G., Romero, V., Sablatnig, R., Sánchez, J.A., Schofield, P., Sfikas, G., Sieber, C., Stamatopoulos, N., Strauß, T., Terbul, T., Toselli, A.H., Ulreich, B., Villegas, M., Vidal, E., Walcher, J., Weidemann, M., Wurster, H., and Zagoris, K. (2019). Transforming scholarship in the archives through handwriting text recognition, Transkribus as a case study. emerald publishing. 75(50): 965-967. doi: 10.1108/JD-07-2018-0114/full/html

National Library of Scotland (2020) Reaching People: Library Strategy 2020-2025, https://www.nls.uk/media/43mla4h3/2020-2025-library-strategy.pdf 

Shah, A. (2023). Interview by Joseph Nockels [Microsoft Teams]. 29 June.

The Audience Agency (2023a). Internal Documentation. National Library of Scotland Audience Development Plan 2023-2024.

Transkribus. (2024) https://www.transkribus.org/

Visser, J., Richardson, J. (2013) Digital Engagement in Culture, Heritage and the Arts, https://digitalengagementframework.com/wp-content/uploads/2022/08/Digital_engagement_in_culture_heritage_and_the_arts-2.pdf

Whitelaw, M. (2015) Generous Interfaces for Digital Cultural Collections, DHQ, 9(1), https://www.digitalhumanities.org/dhq/vol/9/1/000205/000205.htm

Wilson, T.D. (2006) On user studies and information needs. Journal of Documentation. 62(6):658-670. doi:10.1108/eb026702

Yang, Y. (2024) ‘From Videos to Vectors: Holistic Encoding and New Interface for Exploration and Discovery.’ August 8, Alliance of Digital Humanities Organisations, George Mason University, Washington D.C.