Abstract
Enterprise search is now an essential tool for employees. In this article, I introduce 12 factors critical to the successful implementation and sustainability of quality search applications. Ranging from content quality, the position of search in an integrated information management strategy and adequate support for users, perhaps the most important is recognizing that the process of ensuring that search is meeting user requirements never comes to an end. That few organizations have made a considered commitment to ensuring that employees can find the information they need when they need it is highlighted by large-scale surveys of the state of enterprise search. The result is confusion and inefficiency whilst ideally users want to find information that they can trust to make decisions that count. This can be removed if the introduction of a search tool is planned within an organizational information management strategy and with a clear understanding of user needs.
Keywords
Introduction
The term ‘enterprise search’ does not have a concise definition. Often it is regarded as the implementation of a single search application that can index all information repositories inside an organization and provide highly relevant information to any and all employees. Even quite small organizations already have more than one search application as well as other instances of search inside document management, enterprise resource planning and other enterprise applications. Enterprise search is better regarded as a strategic approach to ensuring that employees can find and use all the information that the organization has created and collected, subject to security permissions.
The technology of enterprise search dates back to the early 1960s and arguably came of age with the launch of the IBM STAIRS Storage and Information Retrieval System in 1973 (Bourne and Hahn, 2003). Over the last 15 years, the benefits of enterprise search have gradually begun to be recognized, especially in providing an effective way of finding information on corporate intranets with perhaps millions of documents. Two technology developments have catalysed recent interest in enterprise search. The first has been the provision of a powerful search application within Microsoft SP2013. The second is the availability of open-source search applications based on Apache Lucene (https://lucene.apache.org) which can be downloaded at no license cost. From my practical experience and intensive reading, 12 critical success factors have emerged if enterprise search is to thrive: 1. Content quality is essential for quality search
Good search technology will quickly reveal poor content. There should be guidelines for content and metadata quality. It is of little benefit to the organization if a search lists 20 algorithmically relevant documents with a content quality that renders them unfit to be trusted. 2. Invest in a search support team
Before you do anything else ensure that a search support team with the skills, enthusiasm, organizational knowledge and networks can be established, ideally as a search centre of excellence across all search and search-based applications. 3. Get the best out of the current investment in search
There is usually much that can be done to improve the current search applications once the search team and the search vendor work together on options and priorities. The information gained from search log files is a very important element of defining search requirements and setting benchmarks for any new search application. 4. Recognize that enterprise search is an approach and not a technology
Enterprise search is about creating a managed search environment that ensures employees find the information they need to achieve organizational and/or personal objectives. Be aware that the promise of federated search across multiple applications and repositories has yet to be fulfilled. 5. Set search within an information management strategy
Search can benefit every employee in helping them make better decisions. As with all business critical applications, a strategy can be helpful in defining objectives and resources. However, this strategy needs to be grounded in an organizational commitment to information management which recognizes that information is a business asset. 6. Understand user requirements and monitor user satisfaction
Search logs will indicate the queries that have been used but not the information that was being sought. It is important to understand the business and information context of users and to monitor user satisfaction with search on a continuous basis. 7. Recognize that information discovery involves searching, browsing and monitoring
Users need to be able to search when needed, browse when needed and monitor as needed. These three processes need to be linked together to provide an effective information discovery environment. 8. Assess the business impact of search
Go beyond search log analysis and user satisfaction surveys and understand where search is making an impact on business performance. This is the only way that it is possible to make a case for additional investment. 9. Train and support your users
Search is not intuitive. It is far more than entering words into the search query box. Make sure that there is a range of online and face-to-face advice available. The process of training will highlight areas for improvement for other users. 10. Remember that search is a dialogue
Aiming to get the most relevant documents at the top of the first page of results is a waste of effort. In an enterprise, environment users will have complex and often ill-defined queries that require them to be able to refine their query and re-evaluate the results with the minimum of effort. 11. Procure value not functionality
When the time comes to invest in a new search application specify requirements in terms of what your users expect the search application to deliver and not on what features you would like to have supplied. 12. Regard achieving search excellence as a journey and not as a project
The process of ensuring that search is meeting user requirements never comes to an end. Every day there are new employees, new business challenges, new business opportunities and new developments in search technology. Search should never be a ‘project’ but instead be a way of working.
Most importantly, be proud of what you are doing. Tell stories about how search is making an impact on business performance, listen out for opportunities and talk to users every day. They need to feel that search is owned by people who understand the business.
It is only since the advent of the annual Global Intranet Trends survey (http://www.netjmc.com) in 2006 that it started to become clear that users were often very dissatisfied with the quality of the search applications in their organizations. Subsequent editions of the survey (now the Digital Workplace Trends survey) have indicated that this situation has not changed significantly. The aim of this article is to summarize recent research on the extent to which enterprise search applications are meeting user requirements and then to expand on the 12 critical success factors for enterprise search based on this research and on consulting projects undertaken by the author since 2001.
Enterprise versus Web search
It is important to appreciate some differences between enterprise search and the Web search services provided by Google, Microsoft and many other search service companies. McGovern (http://www.gerrymcgovern.com/new-thinking/fixing-appalling-intranet-search) has pointed out that in the case of the Web, the entire purpose of having setting up a website, or even just a Twitter account, is to publish information as widely as possible. Very considerable budgets are allocated by organizations to search engine optimization and just as much money is allocated by Google and Bing to ensure that the results are not biased by the adroit use of metadata. All the information indexed by these Web search engines is by definition public, and the intention is that it should be found easily. It is also important to note that the purpose of these search services is to gain revenue from advertising sales, which is the source of the very considerable research budgets that these companies have available to enhance their search services.
Inside the organization, there is no such imperative to publish. Indeed, employees may have no awareness of whether or not the document that they have consigned to a file share is in fact being indexed by a corporate search application. For the public Web, the ability to rank relevance through citation linking (a concept dating back to 1965) is of very significant benefit for search, but in an enterprise environment all of this contextual information (including prior searches and geospatial context) is not directly available. Moreover, security management in organizations means that there are limits (often poorly defined) to the information that individual employees are able to access and share.
The final difference is that with public information, there are multiple routes of access. Airline flight times can be tracked through the websites of airports, the airlines and multiple third-party sites. Moreover, it is probably unlikely that major decisions will be based on the outcomes of search on just a single website or search site. In the enterprise situation, information that cannot be found in effect does not exist. A major new research and development project might be set-up just because the outcomes of a similar project remain hidden behind a security firewall set-up by a manager who then left the organization without transferring access to a colleague.
The state of enterprise search
In 2014, two large-scale surveys of the state of enterprise search were published by AIIM (2014) and by Findwise (2014). The majority of the respondents to the AIIM survey were from North America, with the Findwise survey providing a wider coverage of Europe. There was a strong consensus in their main findings.
Each of these surveys contains a considerable amount of data. Below is a summary of the outcomes taken across both surveys but without any attempt to normalize the survey data. Although typically 75 per cent of respondents agree that access to information is very important to the organization, in only around 15 per cent of cases are employees totally satisfied with the performance of the search applications. Respondents indicate a requirement to search a very wide range of repositories and content formats. Paying attention to content quality, metadata and taxonomies results in an improvement of search performance of at least 10 per cent points. In most companies, even those with up to 5000 employees, there is rarely more than one person with specific responsibility for search. The budget for search is held by IT even though the skills required to support search are in library and information science, user experience and computational linguistics, all of which skills are rarely found in an IT department. Only a minority of respondents have an enterprise search strategy. Many organizations have multiple search applications that are managed on an individual basis with no central coordination and support. Little attention is paid to assessing the performance of the search application or the level of user satisfaction with search. Maintaining the security of information, perhaps even at a document level, is a very major concern to IT managers.
Search is a decision support application
Looking at the list above, it is clear that few organizations have made a considered commitment to ensuring that employees can find the information they need when they need it. Staff will only use the search application to find information when they need to make a decision. No one, even when there is a very good search application available, will search just for the purposes of making a search. A substantial amount of research has been undertaken to understand the complex behaviours of searching for information. Russell Rose and Tate provide an overview of what are often referred to as information-seeking models and present a novel approach to defining enterprise search requirements (Russell-Rose and Tate, 2012).
Ideally, users want to find information that they can trust to make decisions that support the achievement of business objectives and also to support their own career development. If decisions are being made without the best available information, then the organization is at risk. Indeed, when making a business case for search, working with the risk manager to see how a risk score could be reduced through an investment in search can be a very effective way of making a business case.
The governance structure of all organizations has the objective of reducing business risks to a manageable level. The importance of managing risk is obvious from the way in which US SEC 10-K filings require a detailed assessment of corporate risks to be placed at the beginning of the Annual Filing. A review of these risks usually shows that they could arise from an inability to find all relevant information held by the company. It could be argued that a failure to ensure that this information is available to staff through search is a breach of the responsibilities of the board of directors to shareholders.
The second implication is that the alternative to using a broken search application is to email or call a colleague asking whether they have the information or know where it might be found. They may well have the information, but how confident can the requestor be that this information is in fact the most recent and most reliable? A second issue is that only now, and mainly through the work of Mark, http://www.ics.uci.edu/˜gmark/Home_page/Welcome.html, are we recognizing the impact of interruptions. Many organizations have an ethos of every employee being available to support the work of others. What is not appreciated that once interrupted employees tend to re-prioritize their work and may not return to the work in hand prior to the interruption for a period of time. Finally, this approach penalizes the people who do maintain an awareness of where information is stored and how it can be accessed. If search is poor, the individual and the organization become less efficient.
Critical success factors for search
This article sets out 12 critical success factors for enterprise search. These are based on the presentations to the Enterprise Search Summit and Enterprise Search Europe (http://www.enterprisesearcheurope.com/2015/) conferences, the surveys referred to above and experience gained from undertaking a range of enterprise search consulting assignments over the last 15 years.
1. Content quality is essential for quality search
Good search technology will reveal good content and poor content with equal ease. The fundamental metric of search performance is that the user is able to trust that the information presented in the search results satisfies two criteria. The first is that the search application has presented all the relevant content that the enterprise has created. Immediately, there are two issues. The user does not know how much relevant content the enterprise holds and therefore has to trust that the search application is capable of finding it. The user also does not know where the content is stored and has to trust that the application is able to search across all possible servers and applications.
It takes very little to breach trust in the application. Even if the users cannot know the totality of the relevant information, they will usually know a few items that they expect to find on the first couple of pages. If one search fails this test, then the application may be given a second chance. A second failure will almost inevitably mean that the user gives up on the search application, using it only when all other options have been exhausted.
The reason why search seems to have failed is that the quality of the content is not adequate, especially regarding the title and the date. The title is important because search engines tend to bias their ranking of content towards the words in the title. The user will also be dependent on the title when browsing through a list of results. Titles such as ‘Q2 Sales Meeting, Toronto’ will not reveal that the main subject of the presentation was a detailed review of a major new customer in Germany.
The date is important because in an enterprise environment users will often need to find either the most recent version of a policy or a client pitch, or the earliest. The earliest is a reassurance that there is no relevant information before a certain date, narrowing the scope of the search.
There is a second element of trust and that is whether the content can be relied upon to make a decision. A user will be looking for an indication of an author that they know they can rely on or that the report has been through an internal review process. This is where a content management strategy in which all content is reviewed on a regular basis (usually annual) is so important. If the author has left the organization, then someone else needs to be designated as the content owner. Dale (2014) has provided a very good analysis of the issues around content curation.
2. Invest in a search support team
A search application, especially on an intranet, will be used across the organization by probably most members of staff to find information that they will use in making decisions that should be of benefit to the organization and to the member of staff. Despite the importance of the search application, there is rarely even one full-time search manager. Because search is often owned by IT, their focus is usually on making sure that the uptime is acceptable and that the content crawls and indexing are carried out on schedule. The reality is that making search work effectively involves a group of people carrying out a range of tasks, including: Ensuring that user requirements are clearly identified, developing a search strategy, setting performance indicators and taking overall responsibility for the management of search. Analysing the search logs on a regular basis with a knowledge of the business of the organization so that the significance of the frequency of the occurrence of terms can be assessed. Taking action to improve the ranking of particular topics and documents and in particular exploring queries, resulting in a low or zero number of hits. Training users and support staff, providing a help desk, working through the feedback from forms on the search user interface and also securing feedback from participating (even if virtually) in meetings of groups of users with common search requirements.
If the organization has a search user base of 5000 or more people, then it is likely that fulfilling each of these responsibilities need a full-time member of staff, and this excludes the IT support. The team members need a background in information science, information retrieval, computational linguistics and user research as well as a good knowledge of how the business operates.
Many organizations are now establishing a virtual Search Centre of Excellence, with staff from around the organization with the appropriate skills (including appropriate language skills) working together in a virtual team. The value and importance of this team is such that any investment in technology will be wasted if a team along these lines cannot be supported.
3. Get the best out of the current investment in search
When search does not work, the first reaction is that clearly a new search application needs to be procured. However, there is usually much that can be done to improve the current search applications once the search team, the IT department and the search vendor work together on options and priorities. The information gained from search log files is a very important element of defining search requirements and setting benchmarks for any new search application, but these have to be set in the context of the business objectives of the organization.
It may seem counter-intuitive but deciding not to crawl and index certain types of content may result in a better search performance. Indexing a decade of content may create so much noise that more recent and more trustworthy content is pushed down the results list. A visible example of this is provided by the corporate website search of Unilever (http://www.unilever.com). The results page for a search for ‘ice cream’ starts with two items from 2007, with other results in the first 10 dating from 2003, 2009 and 2010, with just one from 2013 and one from 2014.
This is why having a search strategy is so important (White, 2012). Often ad hoc decisions are made about the scope of the content to be indexed or to boost certain types of content, and the result is that the overall performance of the application is decreased. A search strategy should not only set out what can be delivered but also state what cannot be delivered, and it may run to around 40 or so topics, all of which need careful consideration.
4. Recognize that enterprise search is an approach and not a technology
Enterprise search is about creating a managed search environment that ensures employees find the information they need to achieve organizational and/or personal objectives. Implementing a single enterprise search application runs some very high risks. IT departments in particular see federated search as the obvious solution, using one master application to search across a number of specialized applications. If federated search was easy why does Google offer Google Scholar?
In principle, it is possible to crawl and index any number of individual search applications, or business applications with a search component, and create a single index. That is not difficult. What is difficult is creating a ranking list of results that make any sort of sense to the user, including presenting them in a consistent way. The results lists are also likely to be quite long and delivering results with high precision is very difficult.
Another approach is to manage search with one search application; this then sends out the query to other search applications. Results from all the applications are then either integrated or more usually presented in a number of different sections of the search results page.
Both options require the use of ‘connectors’, which are pieces of software that either convert the query into a format applicable to each application and then return the result to the master search application or deliver content from multiple applications to be indexed.
Connectors are challenging to write and maintain. A small change in configuration in one of the queried applications may end up disconnecting the connector. When a connector between two search applications fails (though often they just fail to perform as expected) then there is always an interesting discussion between the vendors concerning which end of the connector has failed. Connectors will also manage security protocols, either through early or late binding. Almost inevitably matching the security models in each application will introduce some latency into the delivery of results, and this needs to be carefully managed.
5. Set search within an information management strategy
Search can benefit every employee in helping them make better decisions. As with all business critical applications, a strategy can be helpful in defining objectives and resources. However, this strategy needs to be grounded in an organizational commitment to information management which recognizes that information is a business asset.
Figure 1 sets out an information life cycle model.
Information life cycle.
Although a substantial amount of attention has been given to developing knowledge management (KM) strategies and procedures, this seems not to have been matched by a similar level of concern for information management, though there is no reliable survey evidence for this assumption. There are now signs of a growing realization of the importance of information management. In the UK, Chartered Institute of Library and Information Professionals (the CILIP; http://www.cilip.org.uk/cilip/advocacy-campaigns-awards/advocacy-campaigns/information-management/information-management-1) has been in the forefront of highlighting the benefits of implementing good practice in information management, as has AIIM and the Information Governance Institute (http://iginitiative.com/) in the US. The reason why the process of discovery is so important is that without it information cannot be used or shared, so the investment in creating and storing content is wasted. In effect, the line from creation to archiving is broken.
6. Recognize that information discovery involves searching, browsing and monitoring
This article has focused on search, but this is only one element in the information discovery process. Users also want to browse through a well-structured hierarchy and to be alerted to new content that could be of interest to them. This is why it is so important to review search and Web logs together. A few years ago, I was working for a global company with around 80,000 employees. At an initial meeting, the latest search logs were tabled, and it did not take long to realize that the combination of searches for ‘conference call’, ‘conference number’ and ‘teleconference number’ made this query the most popular in the month. The search team was very pleased about this, as it showed that staff used search to find the location-specific gateway number for the corporate audio conference network.
A search on these terms revealed that it was in fact very difficult to find the actual number, which is perhaps why so many alternate queries were being posted. It transpired that the number had been posted on the intranet home page as a drop-down list, but it was decided that drop-down lists should not appear on the home page and it was removed. The time that must have been wasted and the frustration of users can only be imagined. Putting the drop-down list back on the home page pushed the search queries down to a handful within a couple of months.
A good illustration of the benefits of an integrated approach to searching, browsing and monitoring is provided by the IBM Streamz application (https://www.research.ibm.com/haifa/dept/imt/papers/guyCIKM12.pdf).
7. Assess the business impact of search
Search logs will indicate the queries that have been used but not the information that was being sought. It is important to understand the business and information context of users and to monitor user satisfaction with search on a continuous basis. It is important to go beyond search log analysis and user satisfaction surveys and understand where search is making an impact on business performance. This is the only way that it is possible to make a case for additional investment in the search team, and if needed additional technology. There are many aspects of search assessment, and in summary these include: Technical – Crawl and index performance, query and result latency and security management Query – Dwell time, click position, click-through, session length, filter and facet use Discovery – Using standard queries and test collections to assess ranking performance and aligning search with browse Satisfaction – Confidence in results, ease of use, low/zero hit analysis, help desk and feedback from users Impact – Decision speed and confidence, customer satisfaction, innovation and fewer interruptions.
Rosenfeld (2011) has written an exceptionally useful book on search metrics that also covers website search, and the upcoming edition of Information Architecture for the World Wide Web (Morville and Rosenfeld, in press) also contains a very useful chapter on website search implementation.
Evaluating search performance in terms of time saved is not a sensible approach, except perhaps in professional services organizations that bill by the hour. Even then the requirement from the user is that they want search that they can trust to deliver information of value. This may result in a search taking longer because the quality of the results list is such that it is taking more time than in the past to review the substantially longer list of results. There is also an interesting paper by Microsoft (Teevan et al., 2014) on the merits of slow search!
8. Remember that search is a dialogue
Search is not intuitive. When the call goes out that search must be as intuitive as Google, it is worth pointing out that there are books published on how to get the best out of Google. It is not as intuitive as it looks. If someone looking for a birthday present walks into Selfridges, the largest department store in London, and asks where the birthday present department is located, they will be told that there is no such department. However, the store staff will begin a dialogue with the visitor to ask for information about the person about to celebrate their birthday and what their interests are.
The element of dialogue is often lost on Web and intranet designers working on user interface design. The information architecture is usually extensively tested through wireframes and then usability tests. Any attempt by a user to use search is regarded as a failure of the information architecture and a revised version is then produced with the same result. It is of course very difficult to test search until the content is loaded and indexed, but it is quite possible to work through tasks that users undertake on a regular basis to work out when search is either the primary option or will be used after an initial review of the architecture.
9. Pay attention to searching for people and expertise
With all the focus on finding relevant documents, it is easy to overlook the importance of searching for people and expertise. The task of ‘finding people’ conceals a wide range of situations where finding an individual person is important. These typically include: finding information about someone with a specific name; finding information about someone the spelling of whose name may not be known for certain; finding who in an office has responsibility for a specific task (e.g. billing management); find someone who has specific expertise; and find someone who may be able to help me with advice, information and/or knowledge.
Finding information about a specific person is relatively easy, if there are good staff profiles. Things become more difficult when finding someone when there is no definitive spelling of their name. Is it Kristian with a K or Christian with a C? This problem becomes more acute when the name does not use the Roman alphabet and has been transliterated from (say) Arabic into English. (http://www.basistech.com/whitepapers/the-name-matching-you-need-a-comparison-of-name-matching-technologies/)
It is not unusual in a multinational organization to find a wide range of job titles and also people with the same job title having different roles and responsibilities. This is especially the case in smaller offices where staff may have multiple roles. Searching for the person responsible for health and safety in a smaller office may be quite a challenge, as the work might be undertaken by almost anyone in the office that has the relevant training.
Finding people with specific expertise is much more difficult than searching a profile. Dave Snowden, one of the leading KM practitioners, has observed (http://www.elsua.net/2006/08/10/knowledge-management-rules-by-dave-snowden/) that: knowledge will only ever be volunteered – it cannot be conscripted; we only know what we know when we need to know it; and we always know more than we can tell, and we will always tell more than we can write down.
These observations call into question what the potential return on investment would be from creating extensive profiles. This approach works well when the expertise requirement can be reduced to one or two keywords.
10. Procure value not functionality
When the time comes to invest in a new search application, it is very important to distinguish between fitness to specification and fitness to purpose. Ideally, these should be the same, but it is not uncommon for the procuring department (usually IT) to be mesmerized by a long list of features that may have little direct impact on search satisfaction. In early 2015, two members of the development team at Elastic (formerly Elasticsearch) wrote a book that sets out every single feature of this open-source search application (Gormley and Tong, 2015). The book runs to over 710 pages, and often there are two features per page. Understanding which of these features will result in effective search across the content of the organization needs the skills listed above in Critical Success Factor No. 2, and without these skills, the chances of writing the optimum functional specification to pass onto either internal or external developments are close to zero.
The issues around procurement are not helped by the way in which commercial search vendors (and that includes some vendors of products based on open-source code) market their functional abilities. There is often a reference to ‘blazing fast search’ without defining what that means in absolute terms and also a claim that the search application will deliver accurate results. That is impossible. Accurate is a relative adjective. It is always interesting to ask ‘How accurate?’
11. Regard achieving search excellence as a journey and not as a project
The process of ensuring that search is meeting user requirements never comes to an end. Every day there are new employees, new business challenges, new business opportunities and new developments in search technology. Search should never be a project but instead be a way of working. All too often the team that was involved in search development disappears when the project is over. This is often the case with SP2013, where the capabilities of managing the user interface are very considerable, but the value of these capabilities only becomes visible when the project has come to an end.
Another reason why search is a journey is that every day new content is added, perhaps in a big dump through the acquisition of a business or because a server has now been declared to be open to all employees. Users are then faced with some radical changes to the ranking, especially where they have been used to content being boosted to appear on the first page of results.
Search excellence comes back to having a search team in place before any decisions are made about the future direction of search.
12. Stay aware of what search can offer
With all the challenges of maintaining an enterprise search application with inadequate senior management support, it is easy to work on a day-to-day basis and not appreciate the rapid development of both search technology and the way in which search is now increasingly being used to deliver business-critical information. The SP2013 application from Microsoft is in effect a search-based application in which much of the functionality is driven by search. In May 2015, Microsoft announced that the Delve graph search application would be introduced into SP2013 by the end of 2015 (https://products.office.com/en-us/business/explore-office-delve), just one example of the rapid development of search technology.
Summary
Enterprise search applications play a very important role in ensuring that staff throughout an organization can gain access to the wealth of information and knowledge that has been created over many years by current and past employees. If this information cannot be found, then it cannot be used and shared to support decision-making. Worse, the information may have to be recreated at considerable cost. Where enterprise search applications fail to meet user requirements, the reason is rarely a technology failure. The problems are related primarily to content quality and to an inadequate level of skilled support for the application.
Footnotes
Author’s Note
A comprehensive list of books on search technology and implementation can be found at http://www.intranetfocus.com/enterprise-search/books-and-reports and a list of blogs on search can be found at 
.