Topical tags vs non-topical tags: Towards a bipartite classification?

3.1. A simple model

The classification models introduced in Section 2 are wildly different in their structure, aim and content, but they seem to be in agreement on one aspect: even with different names (i.e. subject, dc:subject, ‘what the resource is about’, content-based tags or descriptive tags) this category appears in all models, and appears to subsume the idea of aboutness [16], that is, what the document is about, one of the most important concepts in information science. While we may all agree on an informal, approximate idea of the aboutness of a document, of course the definition is subtle and is not completely overlapping in at least some of the categorizations described above. To avoid confusion, we will refer to this category with yet another term, topicality, also used in information science although with different nuances, and will call topical tag any term that associates the resource with a topic (e.g. a knowledge domain) that is appropriate to its content.

Even more diverse is the list of the other categories, which span over intentionality, subjectivity, quality and context of the tagging. There is little or no agreement over these categories and most probably no agreement is actually possible, so we will not attempt to impose one, and will refer to these tags with a negative term, as non-topical tags, to refer to terms that are not identifying a knowledge domain appropriate for the content of the resource. This is a very simple taxonomy of two elements only, even narrower than the one in Strohmaier et al. [11], since categorizing tags (i.e. such tags associated with a resource by a categorizer) do include, to a certain extent, even topical terms, although of a very general type.

Yet we can assert some intuitive characteristics of topical vs non-topical tags:

Of course, variability exists in how tags are used: while some tags (e.g. chocolate) are most evidently of a topical nature and others (e.g. toread) are hardly imaginable as topics, many can (and in fact do) position themselves in an intermediate position, being used sometimes as topics (e.g. kids in the context of an article about young humans) and sometimes not (e.g. kids in the context of a web page describing a toy or an amusement park for young humans).

As introduced in Section 1, our intuition is that it could be interesting to provide an automatic approach to identifying with an acceptable confidence the non-topical tags, that is, those that are commonly used with the intention of asserting facts not about the content of the resource. Intuitively, non-topical tags will be present identically in many different domains, and therefore have patterns of contiguity completely different from those of a topical tag.

3.2. Towards the creation of lightweight non-topical ontologies

The identification of non-topical tags as deriving from the use that tag authors made of them, rather than from intrinsic qualities of the tag, suggests the existence of emerging ontologies implicitly used by the community. In the domain of Semantic Web research, an important step forward about folksonomies and emergent ontologies was proposed by Mika [18]. In his work, Mika proposes a tripartite model for the analysis of ontologies so as to take into account the social dimension, besides the description of concepts and instances. In addition, he shows how it is possible to identify emergent ontologies from folksonomies with emergent semantics approaches.

While on the one hand the tags grouped according to their co-occurrence on items, as suggested by Mika [18] and Specia and Motta [19], show emerging clusters that seem to describe topical characteristics of the tag space, thus implicitly define sort of domain ontologies, on the other hand the identification of non-topical tags seems to let a different kind of lightweight ontology emerge.

For instance, the tag kids– when considered as a non-topical tag – may imply the existence of a particular class describing the intended audience for the related items, while the tag newyorktimes may refer to the particular source from which the described resource is derived. Lightweight non-topical ontologies (i.e. ontologies that do not talk about topical attributes of resources) emerge from the identification of non-topical tags according to a number of particular purposes, and can be organized according to the non-topical characteristics of many of the classifications described in Section 2 (including the one described in Dublin Core Metadata Initiative DCMI Metadata Terms [20]) or other subject headings (e.g. along the line of Yi and Chan [21]), without the need to choose one ontology over the other.

This open-ended approach suggests to us also that we evaluate non-topicality as either time-independent or dependent (i.e. rigid or anti-rigid, according to the OntoClean methodology [22]), since users can decide to apply them permanently or only temporarily. ² For instance, the tag toread (usually referring to a document that should be read later on) may subsequently (e.g. after the document has in fact been read) become pointless or wrong. In these cases, ontology evolution [23] approaches such as Zablith et al. [24] may be applied to folksonomies so as to study how emerging non-topical ontologies evolve over time.

4.1. Collecting tags

In order to follow through with our intent to design and test an algorithm that can determine the topicality or non-topicality of folksonomic tags, we collected a portion of a folksonomy using tags extracted from Delicious. A script was run to read Delicious news feeds between August 2010 and September 2010. The script ran for about 6 weeks, gathering information about slightly less that 1.3 million documents. The first pass of the crawling process collected a sequence of tagging events of the type ‘user assigns label to URL’, independently from their topic. Note that this process is different from a keyword-based search, but rather it is the systematic capture of the continuous flux of activity on Delicious. The news feeds contain only data about the first time a user tags a document, so as a second step we subsequently took every document described in the first batch and we queried Delicious about it, accessing all of its tags along with the user name of the tagger and the timestamps of the tagging. No manual selection or post-processing was performed on the tag set. This process took about a week using 10 different machines simultaneously.

Overall, we gathered information about 1,280,686 documents, for a total of 21,408,652 tags (16.7 tags per document on average), 1,205,958 unique tags, 491,702 users and 7,034,524 tagging events. ³ The distribution of the number of documents per number of tags is shown (on a logarithmic scale) in Figure 1. The dataset was stored as an XML file (950 MB) according to the following format:

<tags t=“1229008773” u=“[username]” href=“005cb474bfc10f41036b543f042ae791”>

<t>jquery</t>

<t>webdesign</t>

<t>navigation</t></tags>

OPEN IN VIEWER

Figure 1.

The distribution of documents per tag in our dataset. It shows the number of tags in our collection (axis y) associated with exactly x documents.

Every element tags represents a tagging event, that is, the activity of a user who tagged a document with zero or more tags (specified through the element t). The three attributes of the element tags are:

The procedure described here aims at collecting a large dataset with high coverage over topics, that is, containing tags relative to as many domains as possible. As such, should the need ever arise to investigate folksonomic datasets in a different manner, for example looking for domain-specific or language-specific subsets, our crawling algorithm will have to be adapted.

4.2. NTDS: Non-Topicality by Distributional Semantics

We base our reasoning on the idea that the non-topical tags of a given folksonomy can emerge from the analysis of the topological organization of the folksonomy itself. Roughly speaking, the idea is to organize the tag space in clusters where tags are grouped according to how many times they are used together to annotate resources, which then become connected through hubs, that is, tags that connect several clusters between them. Our intuition is that the hubs represent non-topical tags of the folksonomy, since they connect many different contexts. The actual approach we used is introduced as follows.

Our method is grounded on the latent semantic analysis (LSA) first introduced by Deerwester et al. [25] and largely used by the NLP and information-retrieval communities. The occurrence of each word is represented as a vector of words co-occurring with it in the same context (different contexts can be used, e.g. some neighbouring words, a sentence or even an entire document). All vectors are thus collected in a [word × word in context] matrix A. This is a huge and sparse matrix, because each original vector spans through the entire lexicon of the target language. Applying a Singular Value Decomposition technique to A we map each vector into a subspace, called the word-space, with reduced dimensionality k, keeping most of the original distributional information and making evident high-order latent relationships between words. A good value for k (=300) has been empirically determined in the LSA literature to allow for the extraction of latent high-order relationships between words.

Each word is then mapped into a k-dimensional vector in order to efficiently compare it with other word-vectors to find similarities using the cosine distance:

Sim ( w 1 , w 2 ) = w 1 * w 2 ∥ w 1 ∥ · ∥ w 2 ∥

(1)

where w1*w2 is the Euclidean scalar product and ||w|| is the norm of the vector w. The closer the vectors in the k-dimensional space are, the more the distributional behaviours of the two words are similar. So, appropriate clustering methods can identify sets of similar words.

Various other word-space models can be used instead of LSA, for example Hyperspace Analogue to Language [26] and Latent Dirichlet Allocation [27]. The interested reader can look at the review on word-spaces from Turney and Pantel [28]. We chose LSA for our experiments mainly for its simplicity and because of the high availability of this model. The global results, as well as the conclusions, should not be heavily affected by a different choice of the word-space model.

Starting from this standard technique, Widdows [29] studied the properties of this word-spaces, and the networks of words generated by connecting each word to its most similar words.

We applied this idea by building a word-space considering the content of the tagging events (i.e. all the tags associated by a single user to a specific document) as if it constituted a ‘text-sentence’ in the standard word-space models. Thus the context for building the distributional model became this set of tags. Then, for each tag T we extracted the 200 most similar tags through the word-space model. Each tag T′ in the set of immediate neighbours, NT, is connected with T with an edge weighted using the similarity between the two tags Sim(T,T′). We then iterated the same process considering each tag T′ in NT enriching the graph GT with many more edges.

Given the network of tag GT we can observe, in line with the considerations of, among many, Widdows [29] and Heyer et al. [30], that tags used only in few specific contexts tend to be placed in a highly connected subgraph, while tags used in different contexts tend to be hubs between different highly connected subgraphs. In other words, GT exhibits the property to be a small-world graph in the sense defined by Watts and Strogatz [31].

Widdows [29] and Dorow et al. [32] introduce a measure for unweighted graphs, the node curvature (also called clustering coefficient [31]), to quantify the tags’ behaviour and measure the cohesiveness of the tags’ neighbourhoods The curvature of a node (tag, in our case) t, NC(t), is defined by:

NC ( t ) = # ( triangles t participates in ) # ( triangles t could participate in ) = 2 · ∑ Triangle ( t , ta , tb ) 1 nn · ( nn − 1 )

(2)

where Triangle (ta,tb,tc) marks a triangle between the nodes ta, tb and tc. The node curvature assumes values between 0 and 1. A value of 0 occurs if there is no link between any of the node’s neighbours, and a node has a curvature of 1 if all of its neighbours are linked (see Figure 2 for some artificial examples). Thus a node that exhibits a high curvature is part of a strongly interconnected subnet, that is, a small world, while a node exhibiting small curvature should be a hub between different subnets (i.e. different small worlds).

OPEN IN VIEWER

Figure 2.

Different node curvature, as a function of neighbouring nodes, as defined in Widdows [29].

In NTDS, we extended this measure for weighted graphs as follows: instead of counting 1 for each triangle in which t takes part, we add to the sum the minimum edge weight among the edges that compose the considered triangle. Thus, if the node t participates in a triangle with nodes ta and tb, the Weighted Node Curvature (WNC) is defined as:

WNC ( t ) = 2 · ∑ Triangle ( t , ta , tb ) min ( w t − ta , w t − tb , w ta − tb ) nn · ( nn − 1 )

(3)

where w_t–t′ is the weight between node t and t′ and nn is the number of neighbours of t. Extending the computation of node curvature considering weighted edges representing node similarity results in a more precise definition of node/neighbourhood cohesiveness and allows for a better small-world identification.

In our hypothesis, if the WNC of a tag is small, the tag connects different domains (i.e. it shows a non-topical characterization), as shown in Figure 3, while WNC scores closer to 1 should be hints of topical tags.

OPEN IN VIEWER

Figure 3.

A snapshot of the tag graph produced by NTDS, where the tag ‘online’ has been recognized as non-topical tag since it links several tag clusters.

Once we have computed the word-space model for all our documents, a rather slow process, the proposed algorithm can compute WNC quite quickly, processing a large bunch of tags in few seconds.

5.1. The first test

In a first test we compared the output of the NTDS algorithm against a reference evaluation provided by three human cataloguers. Three individuals, with some experience in Delicious and a computer science background, were asked to evaluate the quality of the output of the NTDS algorithm by examining a random selection of 10 of the 1000 most frequently used elements of our collection of 1.2 million tags. For each of them, we selected 20 documents associated with the tag and asked the cataloguers to evaluate whether the tag was or was not topical for the document. We gave no strict definition of topicality, and relied instead on the common interpretation of the term as presented in Section 3.1.

The tags considered in the experiment were books, environment, free, game, healthcare, howto, online, philosophy, python and vegetarian. The WNC scores returned by the NTDS algorithm are shown in Table 2. As specified, a low score indicates a non-topical characterization of the tag, and a higher score, on the contrary, a characterization of the tag as topical.

OPEN IN VIEWER

Table 2.

WNC scores returned by the NTDS algorithm for the 10 tags used in the first experiment.

Tag	WNC score
books	0.32
environment	0.42
free	0.09
game	0.40
healthcare	0.59
howto	0.09
online	0.05
philosophy	0.56
python	0.41
vegetarian	0.72

We then asked cataloguers to classify each of the 20 occurrences of a particular tag (one per document) as either topical or non-topical. Given these classifications, we calculated two measures, score and match, defined as follows:

score ( u , tag ) = | { doc k | topical ( tag , doc k , u ) = true } | | { doc k | exists ( tag , doc k ) = true } |

(4)

match ( u 1 , u 2 , tag ) = | { doc k | topical ( tag , doc k , u 1 ) = topical ( tag , doc k , u 2 ) } | | { doc k | exists ( tag , d oc k ) = true } |

(5)

The supporting function topical(tag,doc,user) returns true if tag was classified as topical in document doc by user, and exists(tag,doc) returns true if doc was annotated with tag. The score measure is the mean of the annotations a user u made on tag– here, too, lower values imply non-topicality, and higher values imply more topicality. Finally, the match measure describes how similarly users u1 and u2 categorized tag– producing higher values when they categorized it in a similar way, and lower values otherwise. The score and match values are shown in Tables 3 and 4, respectively.

OPEN IN VIEWER

Table 3.

The scores resulting from cataloguers’ annotations of the first experiment. Lower values imply non-topicality and higher values imply more topicality.

Tag	cataloguer1	cataloguer2	cataloguer3
books	0.32	0.88	0.66
environment	0.34	0.62	0.98
free	0.00	0.02	0.04
game	0.86	0.62	0.76
healthcare	0.68	0.52	0.98
howto	0.00	0.12	0.08
online	0.02	0.00	0.08
philosophy	0.64	0.52	0.96
python	0.58	0.38	0.74
vegetarian	0.20	0.08	1.00

OPEN IN VIEWER

Table 4.

The matches among cataloguers in the first experiment. Higher values imply that the two users categorized a tag in a similar way, and lower values otherwise.

Tag	cataloguer1 vs cataloguer3	cataloguer1 vs cataloguer2	cataloguer2 vs cataloguer3
books	0.58	0.44	0.66
environment	0.32	0.60	0.60
free	0.96	0.98	0.94
game	0.62	0.68	0.78
healthcare	0.70	0.72	0.54
howto	0.92	0.88	0.92
online	0.90	0.98	0.92
philosophy	0.68	0.68	0.52
python	0.76	0.68	0.52
vegetarian	0.20	0.84	0.08

Our expectation was that there would be similarities between the topicality as expressed by the human score and by the WNC score returned by NTDS. In Figure 4 we show the results of the first experiment. All tags are presented so that their x coordinate represents the WNC score and the y coordinate represents the value of the mean of the human scores. Additionally, the size of the circle represents the value variability between humans – the smaller the circle, the greater the agreement – calculated as the value of the mean of the human matches for a particular tag. ⁵

OPEN IN VIEWER

Figure 4.

The tags of the first test ordered by WNC score (x-axis) and the mean of human score values (y-axis). The size of the circles marks the spread in agreement (i.e. the mean of human match values) between cataloguers (the larger the area, the smaller the agreement).

Evaluating the result of the test, we noticed that the algorithm performed well within the boundaries of the differences between humans, and in a few cases (i.e. howto, free, online, philosophy and healthcare) it was exactly overlapping some of their opinions. However, while the human agreement for tags with lower scores was very strong (i.e. the bottom-left cluster), there still were some differences in opinion between humans for those tags perceived as more topical (the top-right cluster), which made the results hard to interpret.

Wrap-up discussions and interviews showed substantial disagreements between the human cataloguers about what constituted a topical tag: for instance, no agreement was reached about whether ‘game’ is a topical tag for a web page that is a game, but does not talk about this one game or games in general (e.g. a web page containing a Flash online game and nothing else). We decided therefore that the common interpretation of ‘topical’ could be too imprecise or not sufficiently shared, and decided to run a second test with a stricter definition of what constitutes a topical tag. Furthermore, the random selection did not create a balanced distribution of tags according to their WNC scores and we decided to look for a more evenly distributed selection by choosing test tags explicitly.

5.2. The second test

The second test was performed again against a reference evaluation provided by three human cataloguers. This time 20 of the 1000 most common tags (except the 10 tags already used in the first experiment) were selected so as to provide a reasonable distribution in their WNC ranking. Then 50 documents were selected for each of them, and we asked three cataloguers to provide their own reference evaluation of the tags based on the selected documents. The effort for such test is heavy, as it requires 1000 web pages to be accessed and, at the very least, cursorily scanned to determine the justification for the use of the corresponding tag, and so average run time for each of our testers was over 18 hours, which explains the actual number of humans we were able to enrol.

In order to help users to better discriminate the use of the tags, we provided them with a precise guideline in the form of a human-executable algorithm.

Given a document D and one of its tags X, we say that X is a topical tag in D if and only if:

If X was not identified as topical according to any of the previous rules, then X was a non-topical tag in D.

For instance, on a page with a discussion about some powerups of Call Of Duty (a first-person shooter of some fame), all of the following would be considered topical tags: callofduty, powerup (for rule 1), FPS (for rule 2), game (for rule 3). ⁶ Of course these guidelines are still vague enough to allow for subjective interpretation of a tag. In particular, the meaning of ‘talks about’ was left to users, as well as the handling of rule 3 for terms that are not actually present in Wordnet (e.g. ‘webdev’).

The list of the tags considered in the experiment was: architecture, awesome, bible, blog, business, chocolate, copyright, gardening, geometry, guitar, inspiration, iphone, linkedin, logo, resource, science, university, upload, webcomic and webdev. The WNC scores returned by the NTDS algorithm are shown in Table 5.

OPEN IN VIEWER

Table 5.

WNC scores returned by the NTDS algorithm for the 20 tags used in the second experiment.

Tag	WNC score
architecture	0.45
awesome	0.14
bible	0.69
blog	0.13
business	0.21
chocolate	0.80
copyright	0.47
gardening	0.62
geometry	0.53
guitar	0.57
inspiration	0.19
iphone	0.28
linkedin	0.13
logo	0.31
resource	0.07
science	0.19
university	0.53
upload	0.28
webcomic	0.48
webdev	0.09

As before, we asked cataloguers to classify each of the 50 occurrences (one per document) of a particular tag as either topical or non-topical and we then calculated again the related score and match values, shown in Tables 6 and 7, respectively.

OPEN IN VIEWER

Table 6.

The scores resulting from cataloguers’ annotations of the second experiment. Lower values imply non-topicality, and higher values imply more topicality.

Tag	cataloguer1	cataloguer2	cataloguer3
architecture	0.60	0.64	0.90
awesome	0.00	0.02	0.02
bible	0.42	0.56	0.66
blog	0.08	0.08	0.08
business	0.40	0.66	0.52
chocolate	0.26	0.18	0.94
copyright	0.74	0.76	0.86
gardening	0.68	0.50	0.94
geometry	0.40	0.86	0.88
guitar	0.40	0.48	0.68
inspiration	0.02	0.00	0.02
iphone	0.12	0.24	0.80
linkedin	0.66	0.68	0.62
logo	0.38	0.70	0.56
resource	0.00	0.50	0.18
science	0.56	0.52	0.78
university	0.54	0.28	0.36
upload	0.52	0.74	0.44
webcomic	0.02	0.20	0.04
webdev	0.80	0.66	0.76

OPEN IN VIEWER

Table 7.

The matches among cataloguers in the second experiment. Higher values imply that the two users categorized a tag in a similar way, and lower values otherwise.

Tag	cataloguer1 vs cataloguer3	cataloguer1 vs cataloguer2	cataloguer2 vs cataloguer3
Architecture	0.66	0.60	0.62
awesome	0.98	0.96	0.94
bible	0.58	0.82	0.64
blog	0.92	0.94	0.94
business	0.50	0.64	0.52
chocolate	0.28	0.72	0.18
copyright	0.88	0.82	0.78
gardening	0.96	0.98	0.94
geometry	0.74	0.66	0.52
guitar	0.48	0.54	0.82
inspiration	0.72	0.80	0.72
iphone	0.92	0.88	0.92
linkedin	0.96	0.98	0.98
logo	0.32	0.84	0.44
resource	0.82	0.90	0.84
science	0.64	0.60	0.76
university	0.90	0.98	0.92
upload	0.82	0.46	0.52
webcomic	0.64	0.56	0.48
webdev	0.62	0.72	0.74

We noticed that the algorithm did perform worse than the previous test when comparing humans’ scores. Contrary to what we expected, the new guidelines did not noticeably increase the inter-human agreement, and rather created a strange situation in the comparison against the WNC scores by the NTSD algorithm. As shown in Figure 5, in fact, we see three main clusters of tags: the bottom-left and top-right ones represent a good similarity between the cataloguers and NTDS, while the top-left one shows a rather radical difference, where humans classified the terms as more or less topical while the algorithm classified them as clearly non-topical. A further analysis and an interview with the cataloguers helped to shed light on the occurrence: these are all terms that in many cases the cataloguers considered as topical because of a literal interpretation of rule 3, and that were felt as very general and unspecific for the document: ⁷ for instance, the homepage of the Queensland museum ⁸ that was interpreted as ‘science’ or a page about Adobe Creative Suite 5.5 ⁹ that was interpreted as ‘webdev’.

OPEN IN VIEWER

Figure 5.

The tags of the second test ordered by WNC score (x-axis) and the mean of human scores (y-axis). The size of the circles marks the spread in agreement (i.e. the mean of human matches) between cataloguers (the larger the area, the smaller the agreement).

These terms were interpreted as topical in the classification of the human testers because of the excessively generic rule 3, and were considered non-topical in the NSTD algorithm – probably because they were used as introductory terms in many different specialized contexts, so that they became hubs of separate clusters because of their very generality. We call these terms umbrella tags, and we postulate that in traditional classification science the problem has rarely if ever occurred because of the constant effort of its practitioners to use the most specific term available in the thesaurus, which most often constituted a leaf in the tree of the available terms. Distinguishing umbrella tags from non-topical tags becomes therefore an open topic of discussion within our research framework. In addition, as a side note, in principle the algorithmic approach used in NTDS could also confuse topical tags that are very general, such as science, with unpopular non-topical tags. In particular, unpopular non-topical tags may not connect several clusters owing to their rare usage, and they may be potentially confused with general tags connecting a similar number of clusters. A further analysis in this direction should be addressed in future studies as well.

5.3. The third experiment

Testing the agreement between humans and NTDS in the classification of a tag is a way to test, basically, whether the algorithm can guess the opinion of the author of the tagging about the tags themselves. This task has proven to be long, fatiguing and for practical reasons only applicable to a rather limited set of tags.

Yet, one of the fundamental assumptions of folksonomies can help us in designing another test on a larger set of tags that does not involve a comparable amount of work as the previous ones. It has been demonstrated (e.g. in Halpin et al. [34]) that sufficiently large folksonomies tend to become stable, that is, a coherent categorization scheme emerges from collaborative tagging so that tags organize themselves in a power law that shows a statistically reliable agreement in the evaluation of the resource, in particular between taggers and searchers. In practice this means that searchers’ expectation of the meaning of a tag is a reliable indicator of the intended meaning of the taggers.

Therefore, for the third test we decided to test the quality of the output of the NTDS algorithm against the expectation of topicality or non-topicality that searchers have when shown tags that were used to describe documents, instead of considering the original characterization of the tags in the context of specific documents. Since this requires no access to the documents – that is, only a first-glance evaluation of the meaning and characteristics of the tags is actually required – a quick and low-impact test could be conducted.

We selected 60 different tags within the 1000 most used tags (excluding those already used in the previous tests), 20 in the group with the lowest score (WNC < 0.2, i.e. putatively, clearly non-topical), 20 in the group with the highest score (WNC > 0.8, i.e. putatively, clearly topical) and 20 close to the median score (WNC between 0.4 and 0.6, i.e. putatively, intrinsically ambiguous). The WNC scores returned by the NTDS algorithm are shown in Table 8.

OPEN IN VIEWER

Table 8.

WNC scores returned by the NTDS algorithm for the 60 tags used in the third experiment.

Tag	WNC score
art	0.28
arthureames	0.89
arthurmerlin	0.90
aspnet	0.50
australia	0.52
blogs	0.18
brendonryan	0.92
community	0.24
comparison	0.18
configuration	0.15
deancastiel	0.85
dessert	0.79
directory	0.16
editing	0.12
engine	0.17
examples	0.14
extensions	0.19
fandomsherlock	0.92
fashion	0.40
geo	0.40
graphic	0.39
graphics	0.19
harrydraco	0.93
harrypotter	0.89
hiking	0.55
hp	0.85
inception	0.89
internet	0.30
ipod	0.19
jaredjensen	0.87
javascript	0.18
kirkspock	0.91
kurtblaine	0.94
make	0.12
mckaysheppard	0.88
merlin	0.90
merlinarthur	0.90
new	0.17
pairingarthureames	0.89
portal	0.13
power	0.60
programming	0.11
read	0.22
safety	0.29
science	0.19
share	0.31
sherlock	0.92
sherlockjohn	0.92
social	0.19
socialmedia	0.36
ssl	0.36
startrek	0.91
stevedanny	0.82
support	0.13
teachers	0.30
twitter	0.40
urban	0.32
useful	0.06
viapackratius	0.13
website	0.20

After having sorted them in a random order, they were submitted to four potential searchers who were asked to evaluate how much they expected the tag to have been used by taggers as topical or non-topical according to the definitions provided in Section 3.1.

Values were disposed on a five-point Likert scale, with 1 representing ‘mostly or always non-topical’, 2 being ‘more non-topical than topical’, 3 being ‘similarly non-topical and topical’, 4 being ‘more topical than non-topical’ and 5 being ‘mostly or always topical’. The results obtained from the users were then linearized between 0 and 1 (1 being 0.0, 2 being 0.25, 3 being 0.5, 4 being 0.75, and 5 being 1.0) – we assumed that the linearized values represent a reasonable approximation of the score assigned.

Contrary to the previous experiments, here we used the following alternative implementation of the score and match functions to calculate directly the mean of these values for each tag:

score ( Annots tag k ) = ∑ user i ∈ Annots tag k getValue ( user i , Annots tag k ) | { user i | user i ∈ Annots tag k } |

(4)

match ( Annots tag k ) = | LikertCategories | − | getCategoriesUsedIn ( Annots tag k ) | | LikertCategories | − 1

(5)

The formula score takes as input the annotations made by humans for a particular tag tag_K and returns a score from 0 to 1 measuring the how much tag_K was annotated as non-topical (close to 0) or topical (close to 1). The formula match takes as input the annotations made by humans for tag_K and returns a score from 0 to 1 measuring the agreement between experts for that particular tag_K. LikertCategories is the set of the five categories used in the experiment (labelled from 1 to 5); the function getValue returns the converted the Likert category specified by a user for tag_K in the appropriate 0–1 value (as shown in Table 9), while the function getCategoriesUsedIn returns the set of all Likert categories used by humans when annotating tag_K. ¹⁰

OPEN IN VIEWER

Table 9.

The conversion of the cataloguers’ annotations of the third experiment, given according to Likert categories, in the appropriate 0–1 values. Lower values imply non-topicality, and higher values imply more topicality.

Tag	cataloguer1	cataloguer2	cataloguer3	cataloguer4
art	0.75	0.75	0.75	0.75
arthureames	1	0.75	1	1
arthurmerlin	1	1	1	1
aspnet	0.75	0.25	0.5	0.75
australia	1	0.5	1	0.5
blogs	0.25	0.5	0.25	0.25
brendonryan	1	1	1	1
community	0.25	0.75	0.5	0
comparison	0	0.75	0	0
configuration	0.75	0	0.25	0
deancastiel	1	1	1	1
dessert	1	1	1	0.5
directory	0	0.25	0.5	0.5
editing	1	0.25	0.75	0.25
engine	0.75	0.25	0.5	0.25
examples	0	0	0	0
extensions	0.75	0.25	0.25	0.25
fandomsherlock	1	1	1	1
fashion	1	0.5	0.75	0.75
geo	0.5	0.5	0.75	0.75
graphic	0.25	0.5	0.5	0.25
graphics	0.75	0.5	0.5	0.25
harrydraco	1	1	1	1
harrypotter	1	1	1	1
hiking	1	0.75	0.75	1
hp	1	0.75	1	0.5
inception	1	1	1	1
internet	0.5	0.5	0.5	0.5
ipod	1	0.5	1	0.75
jaredjensen	1	1	1	1
javascript	0.5	0.25	0.75	0.5
kirkspock	1	1	1	1
kurtblaine	1	1	1	1
make	0	0	0	0
mckaysheppard	1	1	1	1
merlin	1	1	1	1
merlinarthur	1	1	1	1
new	0	0	0	0
pairingarthureames	1	0.75	1	1
portal	1	0	0.25	0.25
power	1	0.5	0.5	0.25
programming	0.75	0.25	0.75	0.5
read	0	0	0	0
safety	1	0.75	0.5	0.25
science	1	0.5	0.75	0.5
share	0	0	0	0
sherlock	1	1	1	1
sherlockjohn	1	1	1	1
social	0.25	0.5	0.25	0.25
socialmedia	0.25	0.75	0.75	0.5
ssl	0.25	0.75	0.5	1
startrek	1	1	1	1
stevedanny	1	1	1	1
support	0.25	0.5	0.25	0.25
teachers	0.25	0.25	0.5	0.25
twitter	0.25	0.75	0.5	0.75
urban	0.5	0.25	0.25	0.5
useful	0	0	0	0
viapackratius	0	0	0	1
website	0	0.25	0.25	0

The graph in Figure 6 shows how the intuitive characterization of tags given by humans compares with the one proposed by NTDS. The small size of most circles proves that the agreement between the humans is rather larger than before. Also, although the distribution of tags still shows the presence of umbrella terms, the proximity to the diagonal (representing total match between humans and algorithm) is much higher than before, proving that the human perception of the tags can be rather close to the NTDS cataloguing.

OPEN IN VIEWER

Figure 6.

The tags of the third test ordered by the WNC score (x-axis) and the human scores (y-axis), calculated linearizing the human’s values expressed through the Likert scale between 0 and 1 and then calculating the mean. The size of the circles displays the spread in agreement between cataloguers (the larger the area, the smaller the agreement).