Characterizing the intelligence analysis process through a longitudinal field study: Implications for visual analytics

Participants

We recruited three groups of students, one team of four undergraduate students (Team A) and two teams of five graduate students (Teams B and C), from the Department of Intelligence Studies at Mercyhurst College. ¹⁷ Mercyhurst’s Intelligence Program, started in 1992, provides education for students who want to pursue a career as an intelligence analyst. It is recognized as one of the top programs for intelligence studies in the United States, offering a broad range of classes and degrees for students seeking a career as an analyst in national security, law enforcement, or the private sector.

We recruited students who were taking the courses named “Strategic Intelligence” (undergraduate) and “Managing Strategic Intelligence” (graduate), in which teams are required to conduct an analysis project over a 10-week term. The two courses are very similar with respect to the projects. The students all were close to graduation, with past internship experience, and most of them had already received job offers.

While these student teams clearly are not practicing professional analysts, there was not a significant difference between the way the students worked and the way real analysts work, according to the instructor. The analysis process used in the class was modeled directly after the process employed by the US National Intelligence Council to produce its strategic reports, the National Intelligence Estimates. ¹⁸ The instructor also intentionally stayed relatively detached from the students, acting as a mentor and limiting his supervision, so that the teams could autonomously work on the project. The teams were diverse in expertise on the subject matter, which is common for teams in the intelligence community (IC). One key difference from real-world practice was the relative absence of administrative and bureaucratic overhead affecting the student teams, as well as issues relating to security clearances. They operated in a much more “sanitary” environment than the real world.

Task

Different types of intelligence questions exist—we focused on one of the most common types, strategic intelligence. Strategic intelligence is “intelligence that is required for the formulation of strategy, policy, and military plans and operations at national and theater levels.” ¹⁹ In other words, strategic intelligence is the intelligence necessary to create and implement a strategy, typically a grand strategy. It aims to provide ways to accommodate and/or coordinate a variety of variables. Strategic intelligence is exploratory and long term in nature.

The requirement for tasks within the class was that “the questions should be relevant and relatively important to the client’s success or failure but outside their control.” We served as a client/decision maker for Team A in order to observe the process even closer, whereas Teams B and C worked with external organizations. The specific issues each team addressed were the following.

Study protocol and procedures

The analyst teams conducted the project for 10 weeks—from the week of 1 September through the week of 10 November 2010 for Team A and from the week of 1 December to the week of 14 February 2011 for Teams B and C. Normally, strategic intelligence projects range from a couple of months to years; 10 weeks is short but within normal limits for strategic intelligence.

Before the project began, the external clients formulated a draft of their initial intelligence problem. In the first week of the project, the clients conducted a conference call with the analyst team to discuss the scope and requirements of the problem. During the next 2 weeks, the analysts refined the problem and wrote a formal statement of the intelligence question, which they call “Terms of Reference (TOR).” Upon approval from the decision makers, the teams began working on the problem, which took another 7 weeks.

The wiki platform was used as a workspace for analysts to document their process and findings, and we were able to monitor the wiki’s status throughout the project period. The final reports of the projects were also documented on the wikis.

During the project period, we conducted two face-to-face meetings with each team—one in week 7 and the other in week 10. In the meetings, we interviewed each team as a group and the class instructor in order to learn more details about the project’s status, process, difficulties, and future steps. Each interview took approximately an hour. While the interview was semistructured, we followed an interview guide containing several key topics, ^20,21 including the following:

We also observed two team meetings firsthand, which took about 3 h in total.

Data collection and analysis

Most of the process descriptions and produced artifacts were stored digitally. The teams reported methodologies, tools used, sources, as well as the findings on their own website (wiki). To further understand the process, we analyzed interview notes and audio recordings from the focus group interviews. We used the artifacts produced by the analysts, such as drawings, wiki pages, tables, and slides, as further data. Additionally, we had access to history logs of wiki page changes.

We transcribed each interview’s audio recording and then coded the transcripts based on the grounded theory approach. ²² We began by identifying major themes and categories from the text. One emergent theme focused on the analysis process, including methodology and challenges encountered. Another theme was collaboration, focusing on how and to what degree the analysts collaborated and what types of collaboration existed. Throughout the coding process, we iteratively refined the categories. We then elaborated on supporting evidence from the data for each category through a deductive approach.

Overall analysis process

Through the project, we found that four component processes were essential to the overall analysis: constructing a conceptual model, collection, analysis, and production. While the four processes did not occur linearly, this section describes the importance of each and how the analyst teams worked on each.

Phase 1: constructing a conceptual model

Once the teams and clients/decision makers finalized the requirements of the intelligence question, the teams started to build a conceptual model, which is a map of issues and concepts that the team will be investigating to address the problem. The conceptual model illustrates the areas the analysts need to research by helping them to visualize the question at hand. The question is placed in the center and then several high-level components of the question surround the question (Figure 1). Each component branches out and creates a bigger map, from which the team gains an idea of the areas with less/more information that they need to research. This allows the team to focus on collecting a set of data with an appropriate scope.

OPEN IN VIEWER

Figure 1.

A conceptual model.

While the significance of the conceptual model differs depending on the question and the team, it plays a key role for the team to understand the domain area and determine the direction of research. We were told that analysts often construct this conceptual model implicitly, rather than externalizing it, which we found quite interesting. The instructor commented:

In most cases, it’s implicit. People don’t write it down. But it’s the way they are actually doing it. There’s a model in people’s heads, and that’s far more important than the data. There’s research that says analysts’ judgments are far more driven by the way they think about the problem than the data itself. So making the way you think about the problem explicit would allow analysts to identify whether they disagree about how to think about this model, and to merge their best thinking about this model. So the process happens, it’s just the degree of to which it’s made explicit, that is unusual.

Phase 4: production

Once individual collection and analysis were almost finished, the teams met and tried to synthesize findings from each part, which led to the “key findings”—the major product of the analysis. Production was an intensive reading/writing process in which the team collaborated tightly with each other. This stage was more to prepare a presentation for the decision makers. Team members repeatedly checked their sources and findings to make sure that they were consistent and logical.

Reiterating, while we separated these four components for the sake of clarification, the process was not simple, and it was not clear which phase the team was in throughout the project. Instead, the characteristics of the question and the analytical method chosen most influenced the process. In our study, we observed two different styles of intelligence analysis process. The difference in approaches resulted from the type of the intelligence question.

Intuitive analysis—Team A

Team A addressed potentially influential factors to online distance education in the near future. Because the requirements were rather broad and intuitive, the team decided to take a top-down approach, investigating meta-information sources such as research that forecasts future education trends.

Instead of using a specific analytical method, this team depended considerably on the conceptual model and used it as a guide throughout the entire project. They put significant time and effort into constructing it, revising the conceptual model until the seventh week of the project. Because of time constraints, they were not able to cover all the topics in the model. Through discussions, they chose a number of concepts they felt most worth exploring and divided up the concepts for each member.

After collecting and reading information for their designated topic, each analyst wrote a short format analytical report that synthesized the information. Most of the analysis simply involved reading. For a few topics that required careful weighing of alternative explanations, the team employed analysis of competing hypothesis (ACH). ¹⁴ While documenting results, everyone was able to review and edit others’ drafts on the wiki page, and team members frequently discussed others’ analysis (short write-ups) both online and face-to-face. Therefore, everyone was responsible for the reporting of each topic.

After working on the individual topics, the team met to write key findings together. This team invested considerable efforts in synthesizing their findings because their narrative was extremely important for their intuitive type of analysis.

Structured analysis—Teams B and C

Teams B and C used structured analysis with quantified information because their research questions tended to be more specific and required rank ordering of entities (e.g. top x indicators, key people/companies). Both teams built their conceptual model in the beginning as a base model. For these teams, however, the model was more of a collection plan rather than an actual conceptual model. Although they used the model to collect information and divide up the work, they did not refer to it for the remainder of the project. Instead, they started building a matrix in a spreadsheet to collect and analyze data from diverse sources. The matrix was rather a reinterpretation of the conceptual model, and each cell in the matrix indicated a collection requirement.

The purpose of the matrix was to evaluate each entity based on the criteria chosen and to identify the most influential ones, those of most interest to the decision maker. Team B, which was asked to identify key people, technologies, and companies that might affect IC products, created a matrix and chose criteria while collecting information (Figure 2). They identified 180 entities and graded each based on the criteria, noting the ones with highest scores. Team C, which was asked to identify indicators displayed by potential insider threats to a defense department, analyzed data from the 117 case studies about crimes using a matrix (Figure 3). They used it to compare the relationship between crimes and motivations, as well as crimes and indicators.

OPEN IN VIEWER

Figure 2.

MCIM of people, technologies, and companies.

OPEN IN VIEWER

Figure 3.

Case study matrix of crimes.

In both the teams, the matrix captured the conceptual model and how each team was thinking about the question. Filling in the cells was a time-consuming part as analysts needed to read and analyze each case/source to fill in one cell, addressing “the devil in the details.”

However, this type of analysis required additional efforts in the production phase. Initially, the teams converted qualitative information from sources into quantitative information for rank ordering. Once they had completed the matrix, the teams needed to transform its data into a story, so that it could be made useful to decision makers.

Upon the completion of the projects, the instructor evaluated the teams’ performances as being in the “top 10% of the projects over 8 years.” He commented that all three teams performed the analysis well, and in one case, the decision maker briefed the head of his organization with the team’s results.

Tools and methods used

The teams used various software tools and analytical methods to develop hypotheses, arrive at analytical estimates, and create written reports and multimedia products.

Wiki usage in intelligence analysis

Since wikis were extensively used as a workspace for analysts in our study, we further examined their wiki usage throughout the process. In addition to the three wikis we observed in the study, we also examined five other wikis as a reference. This section details how and to what extent wikis were used in their intelligence process.

How analysts used wikis

As shown in the statistics, the analysts heavily used a wiki for the project. The teams’ purpose for using the wikis was interesting as well.

In the projects we observed, the use of wiki sites started from the very beginning of the project. Once the teams met with clients and clarified the questions and requirements, they set up their own wiki site and uploaded TOR to a page. In this stage, the teams also invited clients to the wiki, so that the teams could communicate with their clients more effectively. Throughout the process, the teams used the discussion feature available in the wikis, enabling communication between analysts about specific tasks and the overall project.

In the early stages of a project, a wiki was primarily a repository for information. When the teams were actually working on a conceptual model, they used another tool (i.e. Mindmeister) and had more face-to-face meetings for discussion. Once they created an initial conceptual model, they put the model into the wiki (Figure 4). As the project evolved, the model helped analysts identify intelligence gaps, discover new information, and update current information, thus aiding in the evolution of the project. Wikis, therefore, helped the team members stay familiar with the conceptual model throughout the project period.

OPEN IN VIEWER

Figure 4.

A conceptual model added to a wiki.

Once the teams decided what to collect based on the conceptual model and started collection, the wiki served as a platform for rapid gathering of information from different sources. Although the teams used more sophisticated tools such as Zotero or Google spreadsheets for the majority of the collection effort, they still uploaded a selected number of highly useful resources to the wikis. Because wikis preserve information and make it easily findable with a simple search function, they seemed to be a good repository for resources. For most of the resources, analysts referenced the sources by hyperlinking to outside or internal sources, which is easily supported by wikis.

While the teams were collecting information, analysts started working on the analysis part. For intuitive analysis, a wiki was the main workspace for the analysis. Analysts created pages for their assigned topics and directly started writing into the pages. In the analysis phase, wikis provided transparency to both the teams and the clients. While working on their own pages, analysts could review others’ work—each contribution and each modification in the process. Although a wiki alone did not lead to specific insights or findings, it was a medium for a better collaborative output. For structured analysis, the analysts primarily worked on Excel spreadsheets because they had to create matrices and fill out each cell. The analytical gains derived from using wikis were relatively small.

The production phase is all about wrapping up the project and putting on final touches to create a polished product. All the teams extensively used wikis in the production phase because they chose a wiki platform as a final deliverable. We observed many structural changes at the end of the project. Teams reorganized numerous pages they created, merged or deleted pages, and modified links to make the wiki easier to navigate without significant changes in the content. Because wikis are flexible and can easily adapt when requirements change or analysts prefer a different structure, wikis seemed to be ideal for production. Analysts also did a lot of formatting and editing on the wiki to polish the output.

Finally, the teams were able to create a final report on the wiki platform, documenting all of the analysis process, methods, resources, and key findings. Since a well-organized wiki format helped clients navigate the output, the teams created a table of contents for more effective navigation (Figure 5).

OPEN IN VIEWER

Figure 5.

Table of contents created for clients to navigate.

Wiki example: “National Intelligence Estimate”

As mentioned earlier, we examined wikis that had been created in the past class projects in addition to the three wikis used in the projects we observed. To better illustrate how analysts organized the wikis and what kind of contents they put into the pages, we provide a detailed example of a wiki used for a “National Intelligence Estimate on infectious and chronic disease.” While the wiki was created 5 years ago, it still has about 350 unique visitors per day (as of 22 February 2012).

In the project, a team of 26 graduate student analysts collaboratively worked on estimating important impacts/threats to US national interests resulting from infectious and chronic disease originating outside the United States over the next 10–15 years. The home page of the wiki contains a short description of the project, a navigational tutorial including a video, and special features such as terms and icons for readers to better understand the reports (Figure 6).

OPEN IN VIEWER

Figure 6.

Home page containing “About the project,” “Navigating the wiki,” and “Understanding the reports.”

In the left pane, the default top menu has general actions related to the wiki such as “Pages and Files” and “Recent Changes,” as well as the search box. The navigation pane on the bottom left lists important pages that the clients and other potential readers should visit, including “Home,” “Terms of Reference,” “Final Estimates,” “National Interest Matrix,” “Methods and Process,” “Resources,” and “Contact Information.”

In Terms of Reference, readers can find the key estimative question and secondary estimative questions of the projects, which define the scope of the analysis in detail.

The Final Estimates page (Figure 7) contains the answer to the Key Estimative Question at the global, regional, national interest, and country levels. Each estimate page contains detailed explanation including an executive summary and discussion, ranging from one to five pages. The Global Estimate Video is also included in this page.

National Interest Matrix is a page where the team embedded a matrix created in Excel spreadsheet. The matrix quantifies the impact of disease on US interests in each country and region round the world on a 5-point scale. As the team took a structured analysis approach, this matrix gives a good overview of how the team derived the Final Estimates.

The Methods and Process page provides background information detailing the analytical processes that the team followed. It describes how the team developed the project to best answer the Key Estimative and Secondary Questions, allowing readers to understand the team’s approach and procedures throughout the project period.

The Resources page is a library of the works collected by the team throughout the project, including reports, journal articles, books, websites, and videos. The resources are also organized by geographic regions, so that readers can easily look up to the references.

Finally, Contact Information lists all 26 analysts’ email addresses and their roles in this project, so that readers could contact them with relevant questions and comments.

OPEN IN VIEWER

Figure 7.

Final Estimates at the global, regional, and country level.

All the pages are connected through hyperlinks, allowing easy navigation between the pages. This particular wiki would have been over 1000 pages if it were a paper document. Using the wiki, the team seems to have effectively organized the contents of the report, so that readers can access the information they need more easily.

Intelligence analysis is about finding an answer to a problem via a sequential process

Some existing models of the intelligence analysis view it as an answer-finding process with a sequential flow, as noted in several models of the intelligence analysis process. ^12,15,27 This perception presumes that the process is linear, sequential, and discrete by step. Pirolli and Card’s ⁷ sensemaking model includes the notion of iterations and revisions between steps, but the fundamental assumption is that separate stages exist throughout the process and that analysts transition between stages.

However, this model was not the intelligence process we observed. Instead, the process appeared to be more parallel and organic, as one analyst described:

Intelligence analysis is not about getting from point A to point B along the route, but it is better associated with basic research where you don’t necessarily know where you are going to go. You’re cutting a path through the jungle that’s never been explored. That’s what you’re doing in most intelligence analysis projects. It’s not a mechanical process in a sense that an assembly line is. It’s a very exploratory activity by nature. You have to expect that some of the stuff you do, some of the things you think, you have to be willing to discard them. Because at the end, they will be rarely relevant, or you find something better that contradicts to it, and that’s just the nature of it.

The key part of the intelligence process is the analysis of a specific set of data

Visual analytics systems often manipulate preprocessed data for the analysis. A primary misconception about intelligence analysis is that the data analysis process, in which investigators analyze a set of collected data, is the most difficult part and takes the most time. This belief assumes that the analysis occurs after investigators collect all data required for the analysis.

This view, however, needs to be changed. Although analysis is important, we observed that the process of “constructing a frame,” as described in the data–frame theory, ²⁸ is more important. In other words, intelligence is about determining how to answer a question, what to research, what to collect, and what criteria to use. This process becomes part of the analysis—analysis implicitly occurs during the process of the construction. Analysts also explore different sets of analytical techniques to address a problem. Deciding which method to use is important, but it often changes during analysis as the way that analysts think about the problem evolves, depending on information that constantly flows in.

Understanding that collection and analysis are integrated together in the process of building a frame is extremely important. Systems are not likely to be successful in supporting intelligence without acknowledging that fact. One analyst commented:

Intell analysis is not like that you have a set of data in hand and run a program. It’s like a conundrum from the very beginning. You have to learn how to learn, how to frame the question, and how to answer it through collecting and evaluating sources.

Analysts do not often collaborate

One common perception of intelligence views analysts as isolated individuals who prefer to work alone, struggling with pieces of information, rather than as collaborative teams. ⁵ However, a faculty member at Mercyhurst countered this perception:

Collaboration is almost all intelligence analysts have done in the context of the team. In the CIA or DIA, working as a team is pretty normal. While working on a particular topic within an agency is typical, also typical is working on an interagency team that consists of analysts from different agencies such as state department teams, DIA teams, and NSA teams.

Analysts are normally organized by function or geographical region. These typically operate as loose teams. Strategic projects almost always involve a team as do crisis projects (for example I am sure there are multiple Libya teams that did not exist a month ago). In short, teamwork is the norm although the teams differ in the degree of formality and to the degree that there is a designated leader.

During our study, we also observed many collaborative elements of intelligence analysis. Collaboration is commonplace in intelligence analysis, and understanding how that occurs is important because it influences one’s whole notion of the process. The IC itself has recognized the importance of improved collaboration since 9-11. ²⁹ Although collaborative tools have been built and they are pushing users into tighter collaboration, it is still important to understand where tighter collaboration will be beneficial and where it may not help much.

We found that multiple layers of collaboration exist in intelligence analysis and that the degree of collaboration differs depending on the type of task and the group dynamics. We observed that analysts usually do not collaborate tightly on data and content—the actual collection and analysis. Although the teams had meetings frequently—twice or three times per week—the main purpose was to discuss their status, issues, and the next steps, as two analysts said:

We come up with an agenda before meeting, a list of what we’re supposed to talk about—what we did, what we want to do, what the questions we need to solve as a group. We didn’t really plan that way, but it just happened. It’s the way it is.

There was no detailed, content-based work during the meeting although sometimes we had discussions on controversial issues. We basically do work in our own time, get preliminary ideas of it, and get together and discuss what issues we had.

The teams often worked together in the same laboratory, but it was rare that they worked on the same document or content. They worked on their own part, but they often talked to each other about issues and questions. Essentially, they took advantage of being in the same place at the same time. To better support analysts’ work, we need to understand the collaborative aspects of the intelligence process and where technology can leverage collaboration. Some tasks are inherently done better by an individual.

We can help intelligence analysts by developing sophisticated analytical tools that assist their thinking process

Visual analytics researchers often seek to help intelligence analysts by developing technologically advanced analytical tools, thereby assisting their cognitive processes. The tools support specific types of analysis, specific analytical methods, and specific stages of the process. Such tools certainly can be helpful, especially to assist analysts to handle a flood of information.

However, our study revealed that analysts want something more than that. Currently, more than 50 analytical methods exist in the IC, ³⁰ and analysts try many different kinds of techniques depending on the problem. Consequently, their dependency on a specific analytical technique is relatively low. Instead, the ability to manage the intelligence process effectively and employ various analytical methods and tools quickly is more important, as the instructor and an analyst said:

Everything is fragmented. I’ve got Mindmeister here, Mindmeister doesn’t interface with my search technology and Google reader. I’ve got to manually go out and figure out what all those bullets are. No help from a computer … But there isn’t any set that ties all these, the pieces are there, Mindmeister, Zotero, RSS Google reader, MSWord, the wiki, are here, but nothing links all that in one seamless thing so I can go from the requirement to a product in a single package, in a single way.

Doesn’t need to be perfect at all. Needs to be able to jump back and forth and if somebody says to me, “Oh no, your product due tomorrow!” I’ve got to be able to take whatever system. I’m only in 2/3 of the process and I’ve got to be able to jump to the end of process and write the final report, get it done by tomorrow. That’s the way intelligence works.

If the processes of collection and analysis are integrated in a single system, this helps analysts apply structured analytical methods such as ACH, social network analysis, geospatial mapping, and decision matrix. In our interviews, two teams mentioned that if they had more time, they would have tried other analytical techniques. Analysts always want to push their findings and triage, aggressively reshuffling their analysis. One of the most effective ways to do this is to employ multiple analytical methods and compare and contrast findings from each. The ability to try various techniques with the data can help analysts find effective ways for addressing questions and strengthening their analysis.

Linear versus parallel

One might believe that the way intelligence analysts work is quite simple and straightforward. First, they specify requirements, build a conceptual model of what to research, then collect information, analyze data using various techniques, and finally write a report. This belief is a common misconception about intelligence as mentioned in the previous section. The reality is quite different. Rather than working linearly, analysts work on everything during almost the entire project. That is, analysts do not hold writing until enough information is collected; they keep revising analysis and writing as new information flows in. Analysts do not decide what to research and move on to collecting information; they start searching for information even when they are not sure what to research. Analysts do not produce final products after they are done with analysis; they already have an idea or a structure of final products in the very beginning, although it may be rough.

This “parallelism” is portrayed well in Wheaton’s model of the intelligence process (Figure 8). In each phase, one of the core processes is emphasized most but all other functions operate in parallel. Wheaton ³¹ argues that “All four functions begin almost immediately, but through the course of the project, the amount of time spent focused on each function will change, with each function dominating the overall process at some point.”

OPEN IN VIEWER

Figure 8.

Wheaton’s multiphasic model of the intelligence process. ²⁸

Although several distinct elements exist in the analysis process, all are very closely coupled and the connection is very organic. One can easily observe an analyst working on collecting new information while analyzing and checking the credibility of previously collected sources at the same time. In our study, we observed that a team’s conceptual model changed drastically in the middle of the process, that a new information source was added 10 days before the deadline, and that a previous analysis report was discarded and new analysis began at a later stage. The matrices also kept changing as new information arrived. While the teams were working on the matrix, they were collecting information at the same time to make sure that they were familiar with the area. Several quotes better explain this:

But it isn’t as rigidly isolated as it’s on that (traditional) cycle because you can’t build a good conceptual model without knowing what’s out there. So there’s little bit of collection as you’re building the model and we refined it.

Our conceptual model is changing. It doesn’t get set in phase 1 and we drive it, that’s the difference between this process and an outline. An outline drives your production. But we are using it differently. As it changes, we’re changing our analytic focus, we’re making decisions about production, who’s going to write something, who’s going to do the analysis, based on how it’s changing and that’s being informed by new information that comes in.

It’s the most updated version. There’s never final. We are constantly revising as we go along. You get to the point where it’s ready to be in the final report, but then if you get new information that contradicts that, then that paper either has to be drastically edited or has to be thrown out entirely.

Pirolli and Card’s sensemaking model

How does this new way of thinking about the intelligence process relate to Pirolli and Card’s ⁷ sensemaking model? Because it is the most widely used model in the visual analytics domain, we were curious about how well their model explains real-world intelligence analysis processes.

Pirolli and Card’s model provides new insights into the intelligence process, suggests leverage points for analysis tools, and has guided the design of many visual analytics systems. However, we argue that the model still implies sequential, discrete stages of the intelligence process although it acknowledges that analysts can move either top-down or bottom-up or jump to different stages. For example, the model does not explain why analysts so frequently jump from one state to another state that is not adjacent. Many visual analytics tools thus support specific states only (e.g. shoebox and evidence file, evidence marshaling, and foraging), and often they do not blend into the entire process of intelligence analysis.

More importantly, the model describes how information transforms and how data flow, rather than how analysts work and how they transition. It gives a very insightful illustration of how the form of information evolves from raw data to reportable results. However, it does not quite fit analysts’ mental model of their work process because they do not work as information is transformed. Rather, information is transformed by how analysts proceed. Similarly, all different states of the model can exist at any point during the process. Analysts may have polished reports on certain subtopics, drafts of analysis, structured matrices, and a collection of documents at a time.

The Pirolli–Card model identified various leverage points for visual analytics tools, but the linearity of the model could give researchers an inaccurate impression of the process. While models are inherently abstract and stage based, it is important to understand the context and the purpose of the model. We would characterize their model as more of an information-processing process rather than intelligence analysis process. Pirolli–Card explicitly state that the model was suggested as a starting point to investigate the domain. While it has contributed to visual analytics researchers understanding of the domain, now we need to change our assumptions to build systems that better help intelligence analysts with their work.

Collaboration throughout the process

Throughout the project, the teams worked tightly together although the degree to which they collaborated differed depending on the phase of analysis. Once the project started, the team set up weekly meetings. The first thing they had to decide was to specify requirements of the problem and then they collaboratively worked on building the conceptual model. Whether the team kept using this model or changed to a matrix, it played a role as a representation of their “group thinking,” as an analyst described:

You want to say that this is the way I’m thinking about this problem. These are some of things I need to think about. And what we’ve done by building the conceptual model is to have that sort of group interaction, which is not necessarily harmonious action. There can be disagreements about how we should be thinking about this. And if there’s shifting, moving it around, that represents an evolution of the way of our thinking.

Once the team had an idea of the areas to explore, they divided up the work and assigned concepts to each analyst. While each one worked on different concepts, they collaborated in collecting information by using a group library. Although this seems to be loose collaboration, the benefit the team gained was invaluable because it could significantly save time and effort in collection. An analyst explained how they worked in collection using Zotero:

Zotero is a good example of one way we collaborate. Each person creates a group library on the Zotero server. If I find a website that I think is useful, whether for my topic or someone else’s topic, I add it to our group collection, and then other members can see it before they go searching the Internet for something. And if she doesn’t find that in Zotero, then she might go out Google. So … try Zotero first, you might already have it.

While working on and analyzing their own topics, team members often met with each other to check status and discuss issues. When not all team members were available, they used typewith.me, ³² a web-based collaboration tool for writing. When most of the areas they had planned to explore were covered and analyzed, they collectively wrote the key findings—the crux of the analysis project. Very tight collaboration occurred in this work. They met together and spent significant time to synthesize findings from all the topics and write the key findings.

Sharing versus content versus function

We found that three different types of collaboration exist when analysts discuss the topic: sharing, content, and function.

Sharing is a way to collaborate by sharing information. In our study, analysts shared sources to better assist their search process and understanding of the topics. At a higher level, however, this can be the sharing of analytical products as well as information sources. When people mention the importance of collaboration in the IC, they primarily refer to the sharing of information sources and analytical results. ³³ This type of collaboration can be significantly supported by technology.

Collaboration also occurs at the content level. This type of collaboration, in which analysts work together to create analytical products, can be seen more often in a small-sized team. Examples in this project include constructing a conceptual model together, dividing concepts and assigning to each analyst, commenting on each other’s analysis, working on ACH together, and writing the key findings together. However, in our study, once work was divided, then each part was done individually. The degree of tightness in this type of collaboration may directly affect the quality of analysis. The more closely the team works together, the more that output is coherent and logical. However, in reality, it is difficult to collaborate on content because of efficiency. This type of collaboration is also difficult to facilitate via technology because so many subtle issues—such as social dynamics, politics, teamwork, and motivations—are involved.

Functional collaboration is needed to execute practical tasks for completing the project, such as editing, creating a matrix structure, specialized analysis on a specific topic, and polishing deliverables. While analysts work on the same thing and divide up the analytical product in the content level, functional collaboration naturally emerges at the later stage of the process as the team begins to think about allocating multiple functions. In this type of collaboration, analysts reinforce their strength. For example, if one is a good editor and has a detailed eye, then that person would do the editing, as one analyst explained:

There was a lot of collaboration. A spent a lot of time working on wiki stuff. B spent time doing the ACH stuff. C did a lot of technical stuff. So each of us spent extra amount of time doing something specific. Whichever parts of this you want to take on, those are the parts that get divided up. As different as I do this (analysis), I do this concept, I do this concept.

Olson et al. ³⁴ similarly characterized collaborative activities of groups by analyzing design meetings from four software teams. Focusing on the time spent in meeting activities, they found similar patterns across design teams. In meetings, teams spent 40% of the time in direct discussions of design, 30% of meeting time was spent taking stock of their progress, and coordination activities consumed approximately 20% of the time. Clarification of ideas across these activities took one-third of the time, indicating that participants spent a large amount of time sharing and explaining expertise.

Low initial barriers, easy to use, and compatible

The primary reason that the analysts liked to use wikis is that they are easy to use and work well with other software. Analysts are often pressed for time. While a number of sophisticated systems exist, analysts do not have plenty of time to learn about and get familiar with a system. If a system has a high learning curve, the chances are rare it will be adopted by analysts. Analysis also occurs across several tools. Results and information snippets often need to be integrated into a single system, and wikis seem to do a reasonable job in embedding outputs from other applications such as YouTube videos or Google docs. In our study, the teams noted:

We chose Wikispaces as our preferred wiki platform because it is relatively easy to use and it works well with various external applications. For example, it allows for the embedding of You Tube videos or Google Documents. It is easy for new users to learn rules that allow for the basic editing of pages. More advanced users can use the text editor which is part of the wiki, or the widget function called “Other Code” to change or add code in order to add additional sophisticated functionality to wiki pages.

Even though wikis have relatively low initial barriers, analysts who are not familiar with such applications would have adoption issues and stick to traditional approaches. Once they overcome the initial barrier, however, the productivity gains seem to be significant, as shown in the following quote:

When they choose to continue more traditional style work flow, that has everything to do with the adoption problems—classic problems in wikis. Two factors that are important in that are “how easy is wiki to use” and “how reliable it is.” Once analysts overcome the problems and adopt it early, the productivity gains are huge.

All-in-one-place nature, from requirements to production

The analyst teams used the wiki platform as a main tool throughout the entire project—from requirements to production. In the beginning, the wiki served as a structural, organizational space for the remaining analysis. Then it became a living document for work in progress, ultimately resulting in a final deliverable. Although the analysts employed other tools at some point when they needed a specific analytical functionality, the wiki stayed as the major place of their work.

This all-in-one-place nature allowed the analysts to shift their attention easily between the four functions (phases) of the process. By capturing most of the process up to date, the wiki serves as an effective platform for starting collection efforts, analyses, and even finished products.

The ability to support production seemed to be attractive to both analysts and decision makers. They can export the wiki as a printable format such as pdf and use either an electronic or a printed version of the product.

Focus on the process as well as the result

Since the team members (and possibly decision makers) consistently reviewed and monitored the wiki pages throughout the project, they were able to focus on details in every stage of the process. That is, wikis reflect the reality of the intelligence process, allowing the teams to focus on the process as well as the result of the intelligence. ³⁵ This is the particular advantage the wikis provided compared to other systems.

Transparency in process

Wikis provide transparency in the analysis process. For example, everyone can access the history of edits including edits on pages, files, and tags. Being able to see the history of edits increases accountability for the analysis process. The instructor commented on this feature:

This team made about 2,000 edits so far. Go and find, and take a look at how many edits were made in the key findings section. I like to watch the history of important documents like key findings. You can look at the initial version, you can look at the middle version, you can look at the final version, and you can watch the evolution of changes, much as same as Wikipedia. Same kind of thing happens here. An important document gets seen or edited a lot; you can actually watch the team think. That’s pretty cool.

Editors and readers are often interested in the history of edits and like to compare the changes made, particularly for important pages, similarly as done for Wikipedia.

Platform for asynchronous, calm collaboration

A single analyst working on a project without colleagues may see little benefit in using a wiki, other than the fact that it is accessible anywhere. However, as described in the previous sections, almost all analytical processes are collaborative among many people.

The benefit of wikis is maximized when the team size is relatively large. Most of the analyst teams we studied had about four or five members, which might have been manageable without wikis. Still, the teams told us that wikis were very effective for collaboration by making the collaboration status—who is working on what and how much progress has been made—visible.

Wikis also encouraged discussion and communication of ideas and questions among analysts. Since analysis should be objective and incorporate different perspectives, these collaborative features of wikis helped the analysts accommodate varying viewpoints. In addition, wikis allowed each individual to easily review and edit others’ work. This peer-review process provided an opportunity for analysts to challenge assumptions, double-check sources, and identify areas that need further research. Ultimately, with repetitive peer reviews to the same article, the bias and inaccuracies could be minimized. One analyst commented about this collaborative support of wikis:

Wikis don’t make me faster, they make us faster.

Flexibility in structure: easy to reorient anything

It is hard to structure a site without knowing its contents and flow. Since analysis is an ongoing process and requirements and collection needs often change, analysts cannot predict and arbitrarily create pages in wikis. That is, the table of contents and organization of pages keep changing until the very end of the analysis.

Wikis have a very important characteristic that can help with this matter; they provide the analyst teams with enormous flexibility. Using traditional approaches, it is quite difficult and cumbersome to reorient everything, but wikis support this very efficiently.

Analysts in our study often changed the structure of the wiki pages, even at the end of the process, but they did not find it difficult. Because all the information and writings the team had created were stored as separate pages by topics, it was relatively easy for them to add/remove, change structures, and modify links. The flexibility inherent in the wiki seemed to allow for easy reorganization of the pages.

Externalize the thinking process—help analysts continuously build a conceptual model

Good analytic practices encourage continuous improvement upon the conceptual model throughout research, which continues through the end of the project.

The analysts in our study told us that the process of making sense of a problem and building a conceptual structure is one of the most important parts of intelligence analysis as it decides the direction of analysis. In most cases, analysts encounter a situation in which they need to learn about new subject matter, but it takes time and effort until they become familiar with the domain. Because they cannot build a good mental model of the problem without knowing what information is available, they struggle to know more about the domain until the later stage of analysis.

Using the power of representation, visual analytics systems can help analysts build a conceptual model or a structure of the problem and domain. For example, the system can take the main question the analyst has and suggest a number of possibly related concepts and keywords based on online encyclopedias, table of contents of books, and tagging services. The system should allow the analyst to refine the concepts, so that it can repeat the search and suggest other relevant concepts. By connecting, grouping, and organizing concepts, analysts can continuously build up their conceptual model or structure of the area throughout the process. One analyst cited experience:

Ok, I got to model something, I’ve got to do a report on Ghana, I don’t know anything all about Ghana, where’s the tool that if I hit the button, it gives me a picture of what the relationship is, the model how to think about Ghana? It gives me 60-70% of the solution. But it gives me the ability to input and tweak and change those. Because I want to have a role in that, I can’t allow the computers to do all my thinking, you know.

Support for this externalization should occur throughout the analysis process because as analysts learn more about the domain, they alter their way of thinking and refine their visual model. Externalizing the thinking process can also assist analysts when they review their analysis after the project terminates. Supporting this activity would be especially useful because it will inform how the analysts could have done better and the areas that need to be examined if they did a similar project, as the instructor said:

The other thing this model helps you do is at the end of the project you can look back and go, “What did we not have time to do? And how does that impact our company, our estimates?” Because whatever reason we didn’t get to it, this was important, we thought this define the space … We can sit back and go, ok, how confident are we on our estimates, knowing that our analysis is always at some level incomplete? And it’s always incomplete, but how does it impact our confidence in our product? That’s another way to use this representation.

Support source management—enable managing both pushed and pulled information and organizing sources meaningfully

One prominent characteristic of how analysts think about sources is that they have to be always vigilant of new sources. They often search for the same keywords again to see if any new materials have been added regarding the topic (pulled sources). They also receive news articles through RSS feeds every day and check if they have received interesting information (pushed sources).

This process of searching sources takes more time than one may think, and systems should allow analysts to manage both pushed and pulled information associated with concepts they have identified. For example, a system could populate several concepts chosen by the analyst and store all the pulled sources in a database such as Zotero. Based on sources already found, the system also could recommend push resources such as blogs and news articles. For each source collected, the analyst could express if it is a useful source or not. Analysts commented on this functionality:

Sources are what we have to get, but where is the tool where I can integrate them? My RSS feeds dump into me every morning. But then I do searches as well. Where’s the tool that allows me to integrate all data, the information that is useful for me?

If that kind of system exists, I have the ability to go back and find all my sources. Automatically, this (keywords, phrases) gets populated. And every point, I have the ability to say no or yes, no or yes to a source. But the actual extraction or the pulling, and the organization of that are automatic from that.

Then the list of sources can be organized in a meaningful way—for example, by keyword queries, by tags the analyst annotated, or by date the source was added. The system could also provide several ways of representing source results such as summary and tag clouds. Further support for analysis or visualization of collected sources as a group would be extremely beneficial.

All these technical capabilities currently exist in visual analytics systems. Now, it is important that they be integrated together appropriately.

Support analysis with constantly changing information—integrate collection and analysis in a single system and help analysts use structured methods during collection

As described in the previous section, collection and analysis are not separate but highly integrated processes. Analysts do not wait until all the data are gathered; rather, they start analysis even when they have only a few pieces of information. Through the repeated process of collection and analysis, they revise a frame and use the collected data as supporting evidence for the frame.

One of the reasons why wikis were successfully used in our study was because of its flexibility in structure—it is relatively easy to reorient anything in wikis. While requirements and collection kept changing throughout the process, Wikis provided the analyst teams with enormous flexibility and allowed them to easily restructure the content of the pages.

Currently, many systems provide analytical support assuming that processed data are available. If a system does not support a seamless transition between collection and analysis, it is likely to be less successful in assisting the analysis. Analysts collect during analysis, and they analyze during collection. This differs from the statistical analysis, in which a structure or a frame about how to analyze the data is clearly defined and analysis is done with clean dataset. An analyst mentioned:

If they had more reliable, structured data, I’d use statistical analysis. But intelligence data is unstructured and dirty. You don’t know what the best way to analyze it is until the middle of the process, or even the end of the process.

Multiple visual analytics systems provide analytical capabilities. By supporting more flexible data manipulation, so that analysts can easily import and remove data from the analysis pool, these systems will be more usable, with better integration into the analysis process.

If the processes of collection and analysis are integrated in a single system, this helps analysts apply structured analytical methods such as ACH, social network analysis, geospatial mapping, and decision matrix. In our interviews, two teams mentioned that if they had more time, they would have tried other analytical techniques. Analysts always want to push their findings and triage, aggressively reshuffling their analysis. One of the most effective ways to do this is to employ multiple analytical methods and compare and contrast findings from each. The ability to try various techniques with the data can help analysts find effective ways for addressing questions and strengthening their analysis.

We had this time crunch. We pretty much got rid of the process of re-evaluating our hypothesis, finding what’s the most important to make it perfect, and hitting on that, and going back to the stuff that we didn’t deem as important. If we had time, we would fill that in.

Help analysts create convincing production—support insight provenance and sanity checks of analytical products

Production is what differentiates intelligence analysis from general sensemaking, which does not necessarily entail external representation. Even when analysts finish their analysis, they need to convert the results into a concise format, so that decision makers can understand their findings. This can be a tedious and time-consuming part of the intelligence process.

When asked about the most difficult part of their project, two teams mentioned production. Interestingly, this difficulty comes from sanity checking and insight provenance, not simply from formatting and writing issues. The sanity check, or qualitative double-check, takes time because data and findings are derived from many sources and analysts have meshed them through the process of collection and analysis. Analysts need to return to original sources and provide a rationale by which their statements are made. They also have to add references to their statements, for which they have to revisit original sources. The following quote from an analyst illustrates those difficulties:

Most difficult part … basically going back through all the sources we used to grade these technologies, people, and companies, then taking basic pieces from those and making a narrative out of it. So explaining why we thought they are the keys and then relating it to the rest of the other findings.

A system that promotes simple insight provenance during analysis could help analysts save their time in production.

Support asynchronous collaboration rather than synchronous collaboration for exploratory analysis

We discussed three different layers of collaboration in the intelligence process and that the degree to which technology can contribute varies. In particular, visual analytics systems seem to have the potential to help collaboration in “sharing” and “content.”

From our study, we found that these types of collaboration tend to occur asynchronously, rather than synchronously. When meeting face-to-face, analysts did not work on actual tasks but spent time checking their status, coordinating next steps, and discussing issues. Even when they worked in the same laboratory for several hours, team members took their own computer and worked individually. Although they often talked to each other, it was for simple coordination issues or specific questions about the content. One analyst stated about his perception on collaboration:

We discussed how each of us interprets the data. We’re very group-oriented when it comes to discussing to a consensus. Other than that, we prefer to work individually especially for the actual analysis. Of course we collaborate even when we work on our own parts, but there’s no one who really knows about those concepts or entities like you do.

In our study, wikis were effectively used for asynchronous, calm collaboration—by making collaboration status visible, encouraging discussion and communication of ideas, and allowing individuals easily to review and edit others’ work.

In a nutshell, analysts collaborate cognitively. Rather than trying to build a system that allows analysts to work at the same time in the same workspace, providing a system that not only promotes individual workspaces but also provides asynchronous collaborative features—such as the ability to share sources and data, view and comment on others’ work, and merge individual work together—would appear to be more beneficial.

Note that our findings are based on strategic intelligence. In other types of intelligence such as tactical and operational intelligence, which form the basis for immediate action, real-time collaboration is also important because such intelligence must be shared and used quickly.

Unifying the pieces

Because their typical processes of requirements gathering, collection, analysis, and production are so intertwined, and it takes considerable time to coordinate between different software systems, it appeared to us that analysts want an all-in-one system that can streamline the analysis process and save their time. In our study, one of the most important reasons for using wikis in the intelligence analysis was because the all-in-one-place nature of wikis allowed the analysts to shift their attention easily between the four functions (phases) of the process. By capturing most of the process up to date, the wiki serves as an effective platform for starting collection efforts, analyses, and even finished products. When asked about their “dream” system, a few analysts answered:

If I had to go back to the beginning and start all the way over, I should be able to jump back and forth seamlessly between all of these processes. We need a tool that compensates for that.

It should be one program. We spend more time to make it work together. Nothing’s compatible with others. We want a program that syncs all the documents. Help us do our visualization with the documents. A program that is compatible with Excel spreadsheet. Don’t want to open 20 different programs.

Thus, a hypothetical tool that simplifies the intelligence analysis process would function as follows:

Thus, analysts could flexibly move between conceptual model, collection, analysis, and production. The system accompanies the analyst from requirements to product in a single platform, speeding up the process, as expressed in one analyst’s comment:

If I had something like that, I’d be blazingly fast. I mean I would be able to do this 10-week project in three weeks.

Interestingly, our suggestions reiterate the findings of other researchers who identified the importance of unifying disparate tools in a different domain. In an observational study of the scientific data analysis process, Springmeyer et al. ⁴⁸ concluded that “an effective data analysis environment should provide an integrated set of tools which supports not only visualization, but some of the additional functionality” such as capturing the context of analysis and linking materials from different stages of analysis.