DM2E: A Linked Data source of Digitised Manuscripts for the Digital Humanities

1. Introduction

The project “Digitised Manuscripts to Europeana” (DM2E)1

The Linked Data representation of the metadata as described in this paper can be accessed online3

The rights statements for the described digitized objects are individually assigned by the content providers who have to choose an appropriate statement from the options6

One major aspect of the DM2E project was publishing metadata about a number of international high profile collections both as Linked Data and through Europeana. Despite its name, DM2E is not restricted to manuscripts but contains also other historical resources like letters, books, images, journal articles, or archival items. Table 1 shows an overview on the content available as Linked Data, broken down by provider and collection name, metadata and content language, type of content, instance count, used reference authorities and metadata source format. The stated counts represent the respective number of instances for which the property dm2e:displayLevel is set to true (see Section 3). As can be seen, this dataset is based on the integration of a variety of source metadata formats, reflecting the heterogeneity of the underlying materials, their international character and the flexibility of the DM2E model to effectively represent such diverse content.

OPEN IN VIEWER

Fig. 1.

Example of the DM2E model in use: representation snippet of a Wittgenstein manuscript.

Accordingly, there is no common workflow for providers to map their data to the DM2E model. Starting with the tools and the documentation provided by DM2E, providers therefore developed their own metadata transformations mainly based on XSLT, although some chose to directly implement export routines into their collection management systems. Initial consistency checks for mapped data revealed that despite the very detailed specification of the DM2E model some providers showed great creativity in individual interpretations of specific model features, most notably regarding the representation of hierarchies. In order to maintain a homogeneous data representation, specific mapping rules have been established for such cases and distributed amongst the providers in form of a recommendation document. In some cases, transformations created by one provider could be reused or adapted for other providers. This especially proved to be effective for the highly standardized library metadata formats such as MARC, METS/MODS and MAB2. The mapping recommendations and the resulting metadata crosswalks are documented on the DM2E wiki,7

The DM2E model is an application profile of EDM, i.e., an application-specific specialization for the representation of manuscripts and similar historical content like old prints, posters, books and old journals [6]. EDM itself is very generic to represent resources provided by museums, libraries, archives and galleries all over Europe. It is based on top-level ontologies like OAI-ORE, Dublin Core and SKOS. Core classes are edm:ProvidedCHO for the described cultural heritage object (CHO), ore:Aggregation for the metadata record provided for the described CHO and edm:WebResource for views of the described CHO, such as images. CHOs can be further qualified by links to contextual resources being instances of edm:Agent, edm:TimeSpan, edm:Place, or skos:Concept.

The example of a manuscript by the philosopher Ludwig Wittgenstein shown in Fig. 1 illustrates how DM2E data is built-up. Bold resources have been added in the DM2E model, others are part of the underlying EDM. ore:Aggregation shows, for example, who has created and mapped the metadata and where the CHO is shown on the Web, while edm:ProvidedCHO is about the physical object that is described. The DM2E model allows that CHOs have multiple hierarchical layers. The CHO here has two layers: the manuscript and paragraphs within the manuscript. The type of the CHO is given via dc:type in edm:ProvidedCHO. Agents are divided into organizations and persons and can be further described. dm2e:levelOfHierarchy “1” says that the manuscript is the highest hierarchical level of this object. Hierarchies are very collection-specific; usually the provider knows best which level is most relevant for scholars. Therefore, the property dm2e:displayLevel is used to give applications a hint, if this CHO for example should appear in a result list of a search interface or if it should show up in Europeana. Users searching for Wittgenstein usually do not want to see every paragraph of a Wittgenstein manuscript in the search results.

The DM2E model adds mostly subclasses and – properties for the domain of manuscripts to existing elements of the EDM. Main additions have been made for person roles, e.g. dm2e:composer, pro:author and more properties that specialize dc:creator, classes that are used to indicate the CHO’s type like bibo:Letter, dm2e:Manuscript or fabio:Article or properties to describe specifics of manuscripts or other document types of the model, e.g. dm2e:incipit for the opening words of a manuscript or dm2e:receivedOn for a date on which a letter was received. An hierarchical object is described on every level using edm:ProvidedCHO and ore:Aggregation as the matadata varies between pages or even paragraphs. Additionally, this allows to refer to every CHO as an independent object if desired. The model was created and refined in an iterative, agile process taking into account several mapping workshops and constant feedback by the data providers and application developers. This feedback led to model changes where properties or classes have been added or dropped, or property ranges have been adapted. After the intellectual collection of requirements and resulting initial versions of the DM2E model, an evaluation of the mapped data [2] has been conducted to get information about the actual usage of the model. As a result, many properties and classes of the model have been removed despite originally been asked for by providers as they have not been used. In Table 3, we provide an overview on the usage of the metadata fields for the class edm:ProvidedCHO across the datasets.

Whenever possible, established vocabularies have been reused, precisely: BIBFRAME, BIBO, CIDOC-CRM, FABIO, PRO, rdaGr2, VIVO and VoID. DM2E-specific usage guidelines for each reused element are provided via dm2e:scopeNote. On the five stars scale for LOD vocabulary use proposed by [12], the model gets four stars, as it is dereferencable and machine-readable, linked to other vocabularies, has metadata about it but is not (yet) linked to by other vocabularies. As of January 20, 2015, the DM2E model contains 65 additional properties and 23 additional classes. The DM2E dataset currently includes descriptions for 2,670,996 cultural heritage objects, 2,478,765 of which representing single annotatable pages, while 182,259 have displayLevel set. Regarding contextual resources, 33,080 objects of type skos:Concept are available, 37,772 are typed as edm:TimeSpan, 21,304 as edm:Agent, 3,751 as foaf:Organization, 104,779 as foaf:Person and 27,266 as edm:Place. Compared to the EDM data that is available via the Europeana LOD pilot [11], the specialized DM2E data forms a smaller, complementary dataset with RDF statements at a very detailed level. The namespace URI for the DM2E model schema is http://onto.dm2e.eu/schemas/dm2e/ and http://data.dm2e.eu/data/ for instances. The full documentation of the model, including detailed changelogs between model versions, can be accessed via the DM2E wiki.

4. Application

Two applications consuming the DM2E Linked Data have been implemented in the DM2E project to provide the scholarly research platform. The first one is a faceted browser that allows scholars to make sense of the DM2E collections and navigate them along several dimensions, iteratively restricting the search results by language, author, publishing institutions, and other metadata fields. Facets are derived from a SOLR8

The API response in a JSON array, from which the dereferenciable URL of Linked Data resources are easily retrievable by looking at the “id” slot. SOLR does not provide a direct way to get all the facets used in the index, which would facilitate the developer in writing queries. However such a list can be retrieved with a work-around with the following query: q=*:*&wt=csv&rows=0&facet.11

Based on the SOLR API, a end-user faceted browser was developed by customising Ajax-SOLR. Each resource shown in the faceted browser can be opened in its provider’s own digital library (following the EDM property edm:isShownAt) and its Linked Data representation can be reached by clicking on “see RDF data” that points to its dereferenciable URL. The DM2E faceted search is available online12

Furthermore, for datasets containing links to annotatable digital objects, users are provided with “Annotate with Pundit” links that direct them to the DM2E semantic annotation environment. The latter constitutes the second Web application built on top of the data and is based on Pundit and Feed, two software components developed as part of the DM2E project.

Pundit14

On the other hand, the Feed REST API provides access to Pundit “as a service” by using the URL of a Web page to be annotated as a call parameter. An extension to the Feed API has been developed in DM2E, allowing this call parameter to also be a dereferenceable URL of an RDF description of a digitized object. Feed parses this RDF description to create a customized annotation environment. While the faceted search application only works on resources at display level (e.g. entire books), the annotation environment allows users to go deeper into the hierarchy. The dcterms:isPartOf and edm:isNextInSequence properties are used to provide basic navigation functionalities such as reaching a specific page of a manuscript or going to the next/previous pages. The actual digital contents that users can annotate are retrieved by following dm2e:hasAnnotatableVersionAt links or, alternatively, the edm:object links. There can be multiple annotatable objects associated with a resource as, for example, the facsimile image and its HTML transcription. In this case, both contents are shown and made annotatable.17

By annotating digital objects with Pundit, users in fact create additional RDF knowledge. This could be, for example, links connecting a geographical map of Metz, depicted in a manuscript page, to the city of Metz in DBpedia, or links from a sentence of a manuscript transcription to a DBpedia entity that is mentioned in such a sentence. Such RDF data, in turn, can be indexed to enrich the faceted search interface, thus improving search and discovery. Demonstrative examples of such end-user information enrichments can be seen in an online screencast.19

As of September 30, 2014, about 6,600 annotations for about 900 digital objects from the DM2E dataset have been created by scholars using our research platform.

Linking our datasets to external sources like GND,20

Altogether, about 24,000 links have been automatically generated. With a manual analysis of 150 random links from agents to DBpedia and 150 random links from places to Linked Geodata, we evaluated the quality. For agents, 125 correct links have been detected which results in a precision of 0.83. Since DBpedia covers several labels, we can for example correctly link “Jakobä” to the DBpedia agent “Jacqueline Countess of Hainaut.” The incorrect links result either from ambiguous names, e.g. “Heinrich Fischer” who refers to a Swiss rower in DBpedia and not to an author, or from incomplete information, e.g. if only the first name or surname is given.

For places, 128 correct links can be detected, resulting in a precision of 0.85 which is similar to the precision for agents. Since Linked Geodata includes labels in various languages, even places with a German label such as “München” can be linked to “Munich”. The reasons for incorrect links, however, are the same to the ones for agents, e.g. the German city “Heidelberg” is mapped to the city “Heidelberg” located in South Africa due to identical labels.

Across all datasets, about 18% of all agents and 60% of all places are linked on average. With a different linkage rule it is possible to detect more links but with the risk to reduce the precision. Further, the amount of detected links as well as their quality highly depend on the popularity and currency of the resources. Since more than one linkset can be available in our system and the user can track their provenance, more liberal linkage rules can also be applied and the user can be informed about its quality.

At the core of DM2E’s infrastructure is a Jena TDB24

Several aspects make the DM2E dataset an interesting and unique source of information. First of all – following the goals of the DM2E project – it contains data from many, carefully selected collections of not only manuscripts, but also old prints, posters, books and old journals with historic value. The data model was developed specifically for this domain where no suitable comprehensive data models existed yet. The DM2E model is also an example of an application profile, an application-specific specialization of the EDM. As such, the data blends well with the huge amount of EDM data available through Europeana. In contrast to many other Linked Datasets, the model and the API have both been tailored to the original data as well as to consuming applications. From a technical point of view, the use of multiple resource representations, versioning and the provision of a full provenance chain have to be mentioned, particularly the proper separation of original, curated metadata from data enrichments generated by automated processes with varying quality. The main short-coming is arguably the lack of a publicly available SPARQL endpoint, mainly due to performance considerations. Fast response times for the scholarly research platform have higher priority. The SOLR-based search and browse interface, however, is provided as convenient entry point to the data and provides a RESTful search API sufficient for most use cases. The data itself also has some shortcomings due to the heterogeneity of the original data. The quality of the metadata ranges from rich descriptions with unambiguous identifiers from authority files for agents, places and subjects to sparse descriptions with few information hidden in free-text fields. It is insofar a dilemma that the contextualization works best for the better data and particularly the poor data is hard to improve. A remedy might be the feedback of data from the annotations provided by the scholars. We plan to investigate this as part of our future work, when more annotations will hopefully be available.