Transforming historical research practices – a digital infrastructure for the VOC archives (GLOBALISE)

Introduction

The archives of the Dutch East India Company (VOC, in business from 1602 until 1800) comprise one of the most complete and extensive bodies of sources on early modern world history. They provide a unique lens on the history of countries and cultures of Europe, Africa, Asia and Australia. The VOC was the largest trading company active in Asia until the 1750s, but was more than just a ‘merchant’; it also took on roles as ‘ruler’, ‘military’ and ‘producer’. ² From the early seventeenth century onwards, the VOC established an empire that combined the establishment of trading posts in a vast region stretching from Persia to Japan with the conquest of strategic ports in places likes Malacca and Macassar, and extensive territorial control in Taiwan (Formosa), the Moluccas, Banda, Java, Sri Lanka (Ceylon), South-India (especially Malabar) and South-Africa.

The different organisational and governing bodies of the VOC produced some 25 million documents that are now kept in archives across the globe and are included in the UNESCO Memory of the World Register. In recent years, these archives have become increasingly digitally available. The Dutch National Archives aims to make its entire VOC collections available in digital form and now holds some 10 million scans. The Sejarah Nusantara project funded by the Corts Foundation scanned almost 3 million folios of the VOC archives in the Arsip Nasional Republik Indonesia in Jakarta, and hundreds of thousands of folios of VOC records have been scanned in the Sri Lanka National Archives in Colombo and Tamil Nadu State Archive in Chennai. More scanning activity is to be expected, for example for the VOC archives in Cape Town.

Because of their huge potential, the rich archival sources of the VOC have attracted continued scholarly interest around the world from the nineteenth century onwards. This has resulted in large bodies of published source editions, research aids, indexes, and more recently, datasets and various single-use tools. Together, these form a valuable, yet highly scattered research infrastructure. The recent and ongoing large-scale digitisation of archives and research aids has increased the accessibility of source material; the efforts of especially the National Archives have been invaluable in this respect. However, it remains a daunting task to do large-scale and in-depth research with the VOC archives. The large corpus of research data and aids is highly fragmented in accessibility and reach, and there are no transcriptions available for most scans, so the documents cannot be searched for keywords. This means that navigating through the contents of digitised archival material remains limited to ‘traditional’ instruments, such as inventories and indexes, and that the scanned material still has to be read page for page by the human eye to extract its contents.

The large-scale digitisation has thus created a revolution in the accessibility of source material, but further steps are needed to improve its searchability, let alone its researchability. As a first step towards unlocking the full potential of the material for research, the National Archives produced automatic transcriptions for around 1 million scans of the VOC Archives in a trial project in 2019. ³ Researchers can now search for keywords in this selected part of the archive, and find instances of slave trade or sugar production, for example, or references to names of locations or people. These can then be further analysed through close reading of text fragments (e.g. contextualising or comparing the retrieved information) or by processing the information of series of results into datasets (often still manually checking and transferring the data).

The increased searchability notwithstanding, much of the research process still remains almost completely manual and highly labour intensive. Answers to more complex questions that for instance examine the relations between certain developments (e.g. sugar production and slave trade) depend on combining several of these cycles of time-consuming repetitive manual processing. In short, despite the availability of transcriptions, doing research on this vast collection of documents remains challenging, because several crucial constraints remain. Firstly, the sources contain an abundance of entities and events, originally recorded in early modern Dutch with substantial variations in spelling. Secondly, for references to entities and events to become meaningful, and thus applicable for research, you need to know the context in which they are mentioned, and often look up historical background information in a secondary source.

GLOBALISE: unlocking the VOC archives for in-depth research

GLOBALISE aims to tackle some of these key constraints on research and facilitate its acceleration by making the digitised VOC Archives more researchable. The project focuses on the ‘Overgekomen Brieven en Papieren’ (‘Letters and papers received’, OBP) series. This is one of the most interesting and important document series in the VOC archives. It consists of more than 5 million handwritten pages of local records of the Asian VOC settlements that were copied in Batavia (present-day Jakarta) and sent to the Dutch Republic. These documents from the seventeenth and eighteenth centuries not only provide a view of the organisation of the VOC and the colonised societies under its rule, but are also brimming with unique data about the peoples and regions with which it came into contact. It thus offers a unique view on interactions between European and non-European actors throughout the Indian Ocean and Indonesian Archipelago worlds in the seventeenth and eighteenth centuries, as well as on the major processes of colonisation and globalisation that to a large degree shaped the modern world. The entire series was recently scanned by the Dutch National Archives.

Within the OBP, the ‘Generale Missiven’ (‘General Letters’) deserve special mention. This is a series of reports from the central administration (Council of India) in Batavia to the board (Gentlemen Seventeen) in the Dutch Republic. These approximately 105,000 folios contain a running account of all things current for the VOC world over almost two centuries. They provide a detailed and structured overview of historic events, as well as social, political, economic and ecological developments in and around the regions where the VOC was active. The General Letters are based on the wider corpus of documents contained in the OBP, and provide an ideal content-based key to this wider series.

We have set up our own state-of-the-art handwritten text recognition (HTR) pipeline to process the scans of the OBP series, converting the handwritten documents into computer-readable text. Subsequently, we apply an innovative combination of historical and semantic contextualisation methods on the source texts. Entities, events and their contexts will be recognised, identified, contextualised and semantically modelled. This will then allow for complex querying and in-depth forms of analysis of the archival corpus.

The historical and semantic contextualisation methods are intimately connected and interwoven, but it is important to distinguish between the two methods and to see them as independent interacting processes. On the one hand, we create a historical contextualisation layer – a ‘digital historical encyclopaedia’, if you will. This entails identifying historical entities from the source texts by linking them to a large collection of reference data, which we compile from the abundant existing data and research aids on the VOC. We invite scholars and citizen scientists to help us enhance this body of historical contextual knowledge with regard to relevant entity categories (e.g. places, polities, people, commodities, measurements) over the course of the GLOBALISE. The valuable collection of historical, validated reference data collected and made digitally available in this project will be relevant to a very broad research domain: VOC history, the history of Asia, Africa and Australia, and encounters between European and non-Europeans. It can be used beyond the scope of this project to enrich and analyse other sources.

On the other hand, the project develops a semantic contextualisation layer. We train natural language processing (NLP) models to first recognise entities (such as people, places, goods and ships), events (such as diplomatic missions, ship voyages, wars and rebellions) and dates in the text. Subsequently, we also try to derive the contexts in which the entities and events are mentioned. So, for instance, we train our models to not only recognise the mention of a person in the text, but also extract what the person was doing, and where and when this happened. We develop a dedicated formal ontology to structure the data yielded by the NLP processes, enabling human and computational reasoning over the vast amount of complex information extracted from the documents. Finally, we link the many millions of mentions of entities and events to the ‘digital encyclopaedia’, thus adding historical context.

By creating and connecting the historical and semantic contextualisation layers, the GLOBALISE project develops a research infrastructure that facilitates complex and in-depth querying of the VOC material. It will, for example, be possible, thanks to the semantic contextualisation layer, to find references to events such as the production or trade of a commodity at specific moments in time and place with a single query (Figure 1, III). The results will be accompanied by validated reference data from the historical contextualisation layer (Figure 1, II). This adds meaning to the mentions from the text; the query results are relevant historical observations. A user-friendly interface on top of the research infrastructure makes the two layers accessible. It allows researchers to explore, query, visualise and analyse the data (Figure 1, IV).

OPEN IN VIEWER

Figure 1.

The GLOBALISE process from scan to handwritten text recognition (I), and onwards through a process of historical contextualisation (II) interacting with semantic contextualisation (III) to a user-friendly infrastructure for complex querying (IV).

To operationalise this, we work with an interdisciplinary team. The key to success of this project lies, we think, in the cooperation between historians and computational linguists. We have gathered the necessary expertise within the project team (see box), and to ensure that the various expertises reinforce each other, and keep reinforing each other throughout the project, we work in small iterations, which produce results in which each team member sees their contribution. For a historian, it is great to see how a computational linguist can extract large amounts of mentions of entities and events from the VOC material. Conversely, it stimulates a computational linguist to hear from a historian what those data mean, and to see how additional manual annotations improve the model further and thus lead to better results.

Research practices

Let us elaborate on the GLOBALISE method, and its potential implications for research practices, with an example. The Generale Missiven series within the OBP provides extensive reports on the size of production and the flows of trade of commodities that played a vital role in the early modern world economy. An example can be found in the letter dated 28 December 1636:

Op Formosa off Packan neempt de suyckerplantagie seer toe; desen jaere hebben 12,042 cattij witte ende 110,461 catty swarte suycker gelevert, die nae Japan is versonden. Anno 1637 is ’t apparent ende hebben de bouwlieden belooft 3 a 400 duysent catty witte suyckeren te leveren.

In this fragment, the Council of India reports to the board of the VOC that the plantation of sugar on Formosa (Taiwan) was on the rise, and that specific amounts of sugar had been shipped to Japan. For next year, a further increase in sugar production was promised. Semantic contextualisation entails recognising the mentions of entities (Formosa/Packan, Japan; black/white sugar) and events (plantation of sugar, sugar was delivered, sugar was promised, sugar was sent). Our method also allows the extraction of the way in which the different entities and events are interrelated (production on Formosa/Packan and shipping to Japan), as well as the commodities involved, their quantities and the units of measurement in which these are noted (cattij in this example). In our process of historical contextualisation, we identify the entities by linking them to our validated reference data. The references to Packan and Formosa will be linked to Taiwan, and will also be identified as related to the VOC between 1624 and 1662. The cattij that is used as a measurement for white and black sugar will be identified as a unit of 0.625 kg. This leads to the historical observation that the sugar production in 1636 in Taiwan (part of the VOC empire) was around 76,565 kg. The GLOBALISE facility will allow researchers to extract and resolve such references semi-automatically on a large-scale, leading in this example to data that show a spectacular growth in annual sugar production from 310,000 kg in 1641 to 1.5 million kg by 1657 (Figures 2 and 3).

OPEN IN VIEWER

Figure 2.

Scan of the original document.

OPEN IN VIEWER

Figure 3.

Mock-up of the GLOBALISE query builder. In this example, a query is built for the question ‘How much sugar was produced in Taiwan in the period when it was ruled by the VOC?’

The query builder of the research facility will allow researchers to develop their own complex searches based on specific relationships between entities and events in an intuitive manner. This allows the quick retrieval of large amounts of meaningful observations, which would have been very time-consuming to obtain using current research practices. Similar to the example of (sugar) production, the facility will allow complex queries to be created for trade patterns, human mobility (migration, slave trade), diplomacy, political conflicts and much more.

Using the GLOBALISE facility, researchers can ask and answer new and complex research questions, and test a wide range of hypotheses. For example, with regard to the seventeenth-century sugar boom in Taiwan, it is known that the establishment of this industry was related to the settlement of Chinese planters and migrants. At the same time, much less is known about the sugar industry itself, such as the labour relations in the plantations. Since VOC officials reported the need for slave labour, might it be possible that both Chinese migrants and enslaved workers from other parts of Asia were employed in the sugar industry? The new opportunities to query observations on the development of sugar production, Chinese migration and slave imports would allow researchers to analyse their possible relations. The results would have important implications for debates on plantation societies and slave labour regimes, or the impact and legacy of colonialism.

The project will not be able to create contextualisation layers for all the many and diverse domains that could possibly be interesting to researchers. We therefore strive to create the infrastructure in such a way that researchers and citizen scientists will be able to add their own contextualisation layers and annotations. This would allow a reverse use of complex contextualised querying that could deeply impact research practices. The historical contextualisation of, for example, places, facilitates historicised (temporal) distinctions between places that were or were not part of the VOC empire, and their relationship with the VOC. This can be applied as a contextual layer for queries about events for which user-created layers were added, for instance on the occurrence of disasters (famines or droughts) or religious conversions, or even on references to human–animal encounters that might signal shifting patterns of habitation (e.g. elephant or tiger attacks). It could then be tested, for instance, whether a correlation exists between famines and the characteristics of the places and regions involved, such as whether these were administered by the VOC. Additional characteristics of the places and regions can easily be included, such as whether they were marked by specific agricultural production (e.g. sugar-producing regions vs. other regions), conflict (e.g. uprisings and warfare) or successions of rulers. The GLOBALISE research facility thus aims to develop research opportunities for the large-scale, structured and comprehensive analysis of complex and changing patterns of intercultural interaction during early globalisation and colonial expansion, and their impact on societies in Europe, Africa, Australia and Asia.

To conclude

In the process of creating this GLOBALISE research facility, the project develops standalone infrastructure components that in themselves are already crucial steps forward for the research field. These are in particular: (a) full transcriptions of the Overgekomen Brieven en Papieren series, including the Generale Missiven; (b) the historical reference data collection, with data on e.g. polities, ships, people and commodities; (c) the models and ontologies used to create the semantic contextualisation layer; and (d) the backend infrastructure and user interfaces to collect, create, store and publish data and annotations. All of these results of the GLOBALISE project will be made open and publicly available.

The ‘digital encyclopaedia’, a VOC thesaurus (with classifications for e.g. ship types, commodities and polities), and the ontology that describes the relationships between entities and events will all be in English. This will make it possible for researchers with limited knowledge of Dutch to track down the data relevant to their work. The infrastructure will thus be relevant to researchers across the globe who are interested in the processes of early globalisation and colonialism and their formative influence on both Europe, in particular the Netherlands, and large areas of the Indian Ocean and Indonesian Archipelago worlds.

Finally, we would like to stress that even though we are developing the GLOBALISE research infrastructure to facilitate research with VOC material, initially but not exclusively focused on the OBP, our infrastructure and method are more widely applicable. Because we make all components publicly and open-source available (the HTR and NLP software and models, as well as the backend infrastructure components and the user interface code), other projects can relatively easily set up a similar research infrastructure. Depending on the type of documents loaded into it, more or less training material for the HTR and NLP processes will be needed to produce meaningful results. For early modern Dutch documents, handwriting and entity recognition will go reasonably well without the need for large amounts of additional training data. For the more in-depth semantic contextualisation process, it will be necessary to manually annotate a large number of sentences per topic and then use these to retrain the NLP model.