Exploring National Library of Scotland datasets with Jupyter Notebooks

From digitisation to Data Foundry

The ‘One Third Digital’ activity at the National Library of Scotland situates the Library well in relation to the more recent computational turn in cultural heritage: digitising collections at scale ensures a rapid release of digital material. Defining ‘digital scholarship’ as ‘the use of computational methods, with National Library of Scotland collections, to enable new forms of research’ (Ames, 2020), the Library established a new Digital Scholarship Service in September 2019, with five objectives:

Encourage, enable & support use of computational research methods with the collections.

As part of this first programme of work – making collections available as data – the Library began publishing datasets on its Data Foundry website in September 2019. The initial focus for this work has been on providing digitised material as datasets, to align with and exploit the existing digitisation programme, with future plans to publish metadata, maps-as-data, audiovisual material, web archive data and organisational data. After some adjustments to the digitisation and ingest processes, to ensure consistent file formats were produced to enable digitised material to be packed up as ‘datasets’ and to embed provenance information about how and why items have been digitised in the metadata, the Library committed to releasing a dataset each month from September 2019 to the end of 2020.

The design and launch of the Data Foundry have been strongly influenced by the recent Collections as Data movement, and particularly the Always Already Computational project (Padilla, Allen, et al., 2019) and its Mellon-funded successor, Collections as Data: Part to Whole (Padilla, Kettler et al., 2019), as well as the broader OpenGlam (2020) movement. These projects have advocated for the value of presenting cultural heritage collections in open and reusable formats as machine-readable data, and the role that this plays in encouraging new uses of the collections. Aligning with these values, and with three core principles of openness, transparency and practicality, the Data Foundry is designed to be easy to access and use, providing ‘no-nonsense’ data with clear rights information, straightforward downloads, dataset trials and plain text-only options (National Library of Scotland, 2019a). The Data Foundry is also home to the Library’s Open Data Publication Plan, which details the formats the Library’s datasets use, the rights statements and licenses they are made available under, and the standards they adhere to (three-star open data); it also lists the datasets published in this way (National Library of Scotland, 2019c).

Notebooks as a COVID-19 response

With the Digital Scholarship Service launched six months before the UK lockdown began, workflows and planned outputs were established and able to continue remotely, despite COVID-19. Furthermore, with a 29% increase in unique page views in March 2020 compared to February 2020, it became clear that digital resources such as the Data Foundry were becoming increasingly important during the pandemic. Yet while the Library’s datasets are rich pickings for those who have computer-programming skills, those without the ability to code are left unable to make the most of these collections. Jupyter Notebooks, bringing these collections to those who are unable to code, were already a part of the vision for the Data Foundry, and with the increasing need for digital collections for newly remote audiences, this goal to enable all users and skill sets to access cultural heritage datasets in an easy and convenient way gained more traction.

Jupyter Notebook is a web application which allows users to write and interact with live code; it is often used in a learning and teaching environment. However, the value of Jupyter Notebooks for library services and collections data has also been demonstrated in recent years by the GLAM Workbench established by Tim Sherratt (2020), and through work by Gustavo Candela et al. (2020) at the University of Alicante. These ‘workbenches’ make library data more accessible by using Jupyter Notebooks to analyse and explore the collections – meaning that the contents of large cultural heritage datasets can be easily explored by anyone, including those without coding skills. Inspired by this work, the Library invested in a Digital Research Intern position – the first remote-working post recruited by the Library – with a remit to create Jupyter Notebooks exploring five Data Foundry collections through text analysis.

As Rule et al. (2018) explain, ‘Jupyter Notebooks…were designed to support reproducible research by enabling scientists to craft easily shared computational narratives that mix code, results, and text’. Furthermore, Havens (2020) notes that Jupyter Notebooks align with the FAIR data principles of findability, accessibility, interoperability and reuse. By highlighting aspects of the Library’s datasets through Jupyter Notebooks, the Library therefore not only conforms to research values such as reproducibility, but also brings its collections to new audiences – a key component of the 2020–2025 Library strategy, ‘Reaching People’ (National Library of Scotland, 2020g). Furthermore, with the COVID-19-induced acceleration of the ‘digital shift’ in libraries and the closure of many in-person services (Greenhall, 2020), users were now increasingly moving to online services and resources; adding new ways to explore and analyse the collections online, for use by multiple audiences and skill sets, would strengthen the offerings of the Library’s burgeoning Digital Scholarship Service.

Collections data as Notebooks

The Jupyter Notebooks were created to give all library users an opportunity to explore the Library’s collections as data, even if they have never programmed or conducted data analysis. To represent the range of the collections data the Library has made available for analysis, the datasets chosen for analysis in Jupyter Notebooks are diverse in size, topic and format. The datasets are:

A Medical History of British India: digitised and manually corrected text of 468 papers from 1850–1950 covering topics related to public health, disease mapping, vaccination, veterinary practice, the military and colonial relationships (National Library of Scotland, 2020a).

At the beginning of the Notebooks, we included contextual information about the collections. This information summarises the contents and significance of the collection, which was written in partnership with the collections’ curators, to guide the development of research questions or larger projects that could benefit from analysis of the collections’ data. The contextual information also includes the data source (a web page on the Library’s Data Foundry) so that library users can quickly find more information about the collection represented in the dataset. Additionally, the Notebook’s contextual information states the data format, which shapes how the dataset can be programmatically explored, and the data creation process, which indicates how accurately a dataset reflects its associated physical collection items.

In all sections of the Jupyter Notebooks, explanatory text accompanies code, explaining what the code does and, where not self-evident, the results of the code. To explore the text datasets (1 to 4), we used the Natural Language Toolkit (NLTK) – a library of code for conducting text analysis with the programming language Python (Bird and Loper, 2004). For datasets 2 and 3, we also used Altair – a data visualisation library that facilitates the creation of charts with the programming language Python (VanderPlas et al., 2018). NLTK provides methods and functions for standardisation, information extraction, classification and machine learning. To explore the MARCXML metadata (dataset 5), we used ElementTree – an application programming interface (API) provided with Python (Python Software Foundation, 2020) – and Pandas – a library of code with methods and functions for data science work in Python (McKinney, 2011).

Each Notebook was structured around the same high-level sections: ‘Preparation’, ‘Data Cleaning and Standardisation’, ‘Summary Statistics’ and, for all but the fifth dataset, ‘Exploratory Analysis’. In ‘Preparation’, we load the dataset into the Notebook, estimate the size of the dataset (i.e. total words and total sentences), and group the dataset into subsets. For datasets of digitised text (datasets 1 to 4), we grouped the dataset by the individual works it contained – for example, we assigned each file in the fourth dataset to its associated book title. For the dataset of metadata, which the Library provides as a single file, we created a subset of the metadata based on eight MARC fields to extract author names, titles, language(s) of publication, publication dates, publication places and topics. In ‘Data Cleaning and Standardisation’, the dataset is normalised in preparation for different types of data analysis, which includes identifying the words and sentences in the running text, reducing words to their root form, lower-casing words, and labelling parts of speech in the sentences. In ‘Summary Statistics’, we calculate the frequency of words, or total times words occur in a dataset, and visualise the results for a selection of the most frequently occurring words. We also analyse vocabulary, calculating the diversity of word choice for subsets of a dataset and comparing them using the lexical diversity metric – the ratio of the total number of unique words to the total number of words.

The ‘Exploratory Analysis’ section varies more than the previous sections across the Notebooks. In the Notebook exploring the first dataset, the ‘Exploratory Analysis’ section contains questions developed in discussion with the dataset’s curator to suggest potential research directions to library patrons. The Notebook exploring the second dataset analyses the occurrence of select words by decade of publication. Exploratory analysis of the third dataset focuses on identifying women named in the collection using a natural language processing method called named entity recognition. In the exploratory analysis of the fourth dataset, the Notebook calculates and visualises changes in the lexical diversity of each book and each publication year included in the dataset. The fifth dataset’s Notebook does not contain an ‘Exploratory Analysis’ section because we chose to dedicate more space to explanations of the data formats MARC, XML and MARCXML in the ‘Preparation’ section.

In addition to the data format, differences in the Notebooks arose due to the size of the datasets. Datasets 1 and 2 have plain-text datasets of a significantly larger size than datasets 3 and 4. To ensure that users could run the code in the Notebooks in a reasonable amount of time (ideally, each block of code in a Jupyter Notebook cell would take seconds to output an answer), the analysis applied to the larger collections differed from the analysis applied to the smaller collections. For example, although tagging each word in a sentence with its corresponding part of speech is a standard task that can prepare a user for more complex text analysis tasks, part-of-speech tagging took several minutes to run on the datasets for datasets 1 and 2. We decided to exclude part-of-speech tagging from these collections’ Notebooks, as well as the more complex text analysis tasks that depended on part-of-speech tagging. Part-of-speech tagging is included in one of the Notebooks, however, because it is a fundamental text analysis task: dataset 3’s Notebook includes part-of-speech tagging, as well as a complex text analysis task dependent upon it, named entity recognition.

The size of the collections datasets and their Notebooks also introduced challenges for their online publication. On testing the Jupyter Notebooks in an interactive online environment provided by the Binder (2020) service, we discovered that the standard Binder did not provide adequate memory space to run the Notebooks completely. The Library’s interactive Notebooks were moved instead to a different Binder service called GESIS Notebooks, which provides larger memory space (GESIS, 2018). The Notebooks for datasets 1 and 2 were also revised to reduce their memory requirements, removing code that was similar to other Notebooks. The Notebook for dataset 5 cannot be run entirely in a GESIS Notebook, however; its interactivity on that platform is limited to its second and third sections only (excluding the ‘Preparation’ section). Users may interact with the entire Notebook for dataset 5 by running the Notebook locally on their computer, downloading the Jupyter Notebook software and the Notebook file from GitHub. The next section details the options we provide to users for accessing all five Notebooks.

Publication and impact

The final step of the project was to publish the Notebooks on the Data Foundry website in a clear and consistent way, aligning with the Data Foundry’s three principles of openness, transparency and practicality. Given the range of users who could be accessing the Notebooks, the aim was to ensure that they were clearly explained and framed. As a result, each Notebook page on the Data Foundry website summarises what the Notebook will enable the user to explore, and also suggests that users consult the introduction to Jupyter Notebooks on Sherratt’s (2020) GLAM Workbench. Importantly, each page also contains ‘A Note on the Data’, which explains the problematic nature of working with historical data – such as issues with the accuracy of optical character recognition, and antiquated language and attitudes which may be present in the data. Each Notebook is then made available through three different routes to ensure a variety of choice for different skill sets or uses. These are:

A static HyperText Markup Language (HTML) view of the Notebook;

A Digital Object Identifier (DOI) is added to each page as a persistent identifier for the data analysis work and to ensure that any references to the Notebooks always resolve.

The Notebooks were published on the Data Foundry and publicised on social media on 24 September 2020, with extremely positive feedback. On Twitter, the tweet by the National Library of Scotland account announcing the launch of the Jupyter Notebooks received over 100 retweets, including retweets not only in English, but also in Russian, Portuguese, French and German. This ‘launch’ tweet had a reach of over 60,000 Twitter users and over 2,000 engagements (clicks on the link within the tweet to the Notebooks or to the profile page of the Twitter account, for example), as well as positive feedback from the international community (National Library of Scotland, 2020h). Meanwhile, Google Analytics shows that 40% of the traffic to the Data Foundry on the launch date visited the Jupyter Notebooks page – significantly higher than the page with the next highest share of the traffic, at 9%. Furthermore, between 24 September and 22 October 2020, the Jupyter Notebooks page remained the highest visited page on the Data Foundry. Over the same time period, the Data Foundry also saw a 99.6% increase in overall page views compared to the previous one-month period. The most popular Notebook on the website is ‘Exploring A Medical History of British India’, with the remaining Notebooks sharing evenly in the increased traffic to the website (National Library of Scotland, 2020b).

There are, however, some limitations to these measurements. The particular popularity of ‘Exploring A Medical History of British India’ could be the result of its position on the web page – the top left – which studies have shown draws most of readers’ attention on screens (Nielson, 2010). Furthermore, Google Analytics only shows us which pages are accessed, and not which Notebooks are viewed, downloaded or used. Currently, the Library does not have access to these kinds of usage statistics, but we will continue to seek further ways to gather this information – as well as demonstrating impact through other means, such as anecdotal use of the Notebooks on university courses and promoting the Notebooks during engagement activities.

Beyond the Data Foundry, we can track engagement activity with the Notebooks through the software development platform GitHub, where we created an online repository for the Jupyter Notebooks (National Library of Scotland, 2020f). A GitHub repository displays information about who has added to software, downloaded software, or requested changes to software in the repository. To date, we have seen two requested changes – ‘merge requests’ – to the repository, indicating that two people found ways to improve the repository’s Jupyter Notebooks when interacting with them, made the changes, and requested that their changes be integrated into the repository.

Internally, the Jupyter Notebooks were launched in a Digital Scholarship Staff Seminar on 23 September 2020, which was attended by those working in curatorial, metadata, digital preservation, digitisation and developer roles. The presentation introduced Jupyter Notebooks as a tool for exploring the Library’s collections and provided highlights from each Notebook to demonstrate the output of the Digital Research Intern position. The feedback after the presentation emphasised the impact that the Notebooks are having on the Library’s culture, helping to facilitate a shift towards an environment that integrates code and data as a complement to physical and digitised collection items. Furthermore, the lasting impact of the internship (developing a series of tools to enable the Library’s diverse audiences to explore collections in more detail) and the value to the broader library community (as an example of how libraries can support users of collections as data) demonstrates the relevance of such roles.

Looking ahead: from collections as data to library users as data analysts

The Jupyter Notebooks project has been a success for the National Library of Scotland – both breaking new ground for the Library as an online remote-working project amidst a challenging global context and opening up new ways of exploring Data Foundry collections as increasing numbers of library users seek to make use of online resources. Given the positive feedback and the strong indications of engagement from Google Analytics and GitHub, the value of producing Notebooks providing exploratory analysis of cultural heritage datasets appears clear. Following this positive start, the Google Analytics for the Data Foundry website and its Jupyter Notebooks pages, as well as the GitHub repository, will continue to be monitored to assess where interest lies. High usage of the Notebooks will, we hope, help to make the case to use further resources in this area, and to begin to create a Notebook ‘as standard’ for each dataset the Library publishes.

Furthermore, this project has demonstrated that there are opportunities for libraries and software developers to work together at the intersection of library data services and online tools, and develop resources which are designed specifically for library data. There are a number of use cases around libraries and library users working with collections data which would benefit from further software and tool development to enable exploration of large datasets using library standards. For example, expanding the memory capacity of interactive platforms such as MyBinder (Binder, 2020) and GESIS Notebooks (GESIS, 2018) would remove limitations on the data that can be loaded into and produced within online Jupyter Notebooks. Here, we hope, there are opportunities for future collaborative development work.

Following the intensive activity around the release of collections as data during 2019 and 2020, the Library’s Digital Scholarship Service now needs to ensure that its users are able to access and explore these collections – and particularly as new users turn to online materials. With cultural heritage organisations finding new use cases for software such as Jupyter Notebook, the Digital Scholarship Service seeks to put itself at the forefront of creating opportunities for users to carry out large-scale exploration of cultural heritage collections as data.