Tracing Decision Processes in Complex , Ambiguous , Information-rich Environments

In order to understand analytic processes in organisations, better tools are required to trace decision processes. An impediment to progress in this area has been the tendency to over-simplify inherently complex information environments or the data that they produce. We address this issue by providing worked examples from the Analysis Simulation Project (ASP). ASP methodology consists of a computer-administered information grid of 64 cells of 50-130 words each. The information accessing activity of participants is recorded. Our results present novel approaches to maintaining optimal complexity in environmental representation and data analysis. Specifically we demonstrate how computer-mediated process tracing methods can simulate the cognitive experience of complexity and how the data can be used to examine behaviour at increasing levels of complexity. We conclude that computer-mediated process-tracing tools provide an opportunity to comprehensively model www.ccsenet.org/ijps International Journal of Psychological Studies Vol. 4, No. 1; March 2012 Published by Canadian Center of Science and Education 159 complex information processing behaviour and therefore allow improved insight into phenomena with similar outcomes but distinct processes.


Introduction
Employees are frequently required to make strategic decisions within complex, ambiguous, and incomplete information environments.These can pertain to organisational direction or analytic products such as risk assessments and market forecasts.The increasing availability of information means that organisations will gain strategic advantage through the ability to integrate data to infer accurate predictions and diagnoses, rather than through superior access to information.However, there is increasing evidence that human judgments are systematically biased when selecting and analysing information in complex, ambiguous environments such as criminal cases (O'Brien, 2009;Wastell, Weeks, Wearing, & Duncan, under review), military intelligence (Lehner, Adelman, Cheikes, & Brown, 2008;Wastell, Weeks, Wearing, Duncan, & Ebrahimi, In press), and mineral exploration (Wastell, Etheridge, McMahon, Lucas, & Hartley, 2011).If organisational decisions are subject to systematic bias we need to understand the underlying decision processes because they may indicate when human error is likely to be most problematic and how best to counteract it.
Methodological advancements are required to achieve this.Specifically, we need the ability to track information search and interpretation behaviour in complex, ambiguous environments.Process tracing is a broad term referring to any technique that measures dynamic changes in cognitive or affective processes (Kuhberger, Schulte-Mecklenbeck, & Ranyard, 2010).Process tracing techniques have been used in experiments ranging from simplified pseudo-diagnostic tasks with four items of information available (Covey & Lovie, 1998), through to the analysis of decision processes in field studies (see Patrick & James, 2004; for a review see Riedl, Brandstatter, & Roithmayr, 2008).However, experimental process-tracing techniques have been largely constrained to option comparison tasks with limited ambiguity and depth of information.In cases where a more complex information environment has been used, procedures have generally been limited to a single process, for example information selection or interpretation, but rarely both (For exceptions see O 'Brien, 2009;Slowiaczek, Klayman, Sherman, & Skov, 1992;Wastell et al., under review).
There are a number of barriers that have prevented the expansion of experimental process-tracing techniques into complex, ambiguous information environments.Firstly, decision outcomes are frequently compared to a normative performance standard, often computed using complex Bayesian methods (Fischhoff & Beyth-Marom, 1983).When the context is expanded to ambiguous information environments, Bayesian calculations can become prohibitively complex because they depend on subjective probabilities.Secondly, the amount and complexity of the data gathered can be overwhelming (Kuhberger et al., 2010).Finally, there is a lack of standardisation to process tracing techniques (Kuhberger et al., 2010), which may hinder acceptance of the methodological approach.
The complexity of the information environments experienced in most applied analytic tasks often prompts over simplification at either the task representation level or the data analysis level.This paper demonstrates how an expansion of process-tracing techniques into complex, ambiguous environments can achieve optimal complexity.The steps we follow are demonstrated using a specific case but are widely applicable.These principles are not intended to be a comprehensive manual for process tracing techniques, but rather a framework that we hope will be refined and expanded upon through multidisciplinary methodological discussion.The principles are outlined below and expanded with worked examples throughout the paper.This paper has two broad aims.Firstly, to demonstrate the application of these principles to the Analysis Simulation Project (ASP) and secondly, to highlight the advantages process-tracing methodology has for the examination of decision processes in complex information environments.We begin by using the creation of ASP to demonstrate the simulation construction considerations: cognition, environment, simplification and experience.Experiment 1 will then be introduced and used to demonstrate theory-based measure identification including research question, measures and assumptions.Experiment 2 will provide demonstrations of temporal specificity and impact.Finally, Experiment 3 will apply theory-based measure identification to a new context, with a focus on demonstrating spatial specificity.

Simulation Construction
The Analysis Simulation Project (ASP) was developed to examine reasoning and judgment processes in the National Security domain.Recently western governments have had cause to question the quality of work from their intelligence agencies.A number of official inquiries (Butler, 2004;WMD, 2005) have questioned the methods that analysts have used in producing intelligence estimates.As a result some authors have begun to call for action to change analyst behaviour based on the findings of research into reasoning and judgment (See Bar-Joseph & McDermont, 2008;Wastell, 2010).ASP was developed to address this need in the National Security domain, but the approach taken is applicable across a wide variety of fields, and has already been expanded into security and business domains.All stages of simulation construction were conducted in collaboration and discussion with subject matter experts.

Cognition
ASP differs from many simulation tasks in that it was developed to examine complex decision-making processes.The focus is therefore on simulating the intellectual experience and cognitive demands, rather than the physical space (See Omodei & Wearing, 1995 for a discussion).One major consideration was the cognitive processes, resources, and limitations that impact performance on decision tasks.Specifically we wanted to ensure that participants could access all typical resources available in analysis tasks including a task description, information sources, spaces to record notes, and the final analysis product.These considerations directly informed the structure of the ASP analysis environment.
All identified resources were included in the program for participant use.The first screen of the program presents the 'tasking' that provides participants with instructions on how to use the program, a scenario, and a brief description of their task, which can be referred to throughout the experiment.The second screen contains the "information resources", "workspace" and the "product space" (see Figure 1).The workspace and product space take the form of two large text boxes to the right and below the information grid where participants are able to keep notes and write their report, respectively.Participants are also able to copy and paste or drag and drop information from the information resources into the workspace or report space.
We also wanted to simulate the cognitive demands of analysis tasks where there is generally more information than can be processed given the time available and memory constraints.This informed the design of the "information resources", which are provided in the form of an 8 (categories) by 8 (entries) information grid of 64 cells as shown in Figure 1.Each cell of the information grid contains substantial amounts of information, ranging from 50 to 130 words in length, and only 40 minutes was given for the investigation.Finally, we wanted task outputs to be similar enough to analysis tasks to produce the appropriate cognitive focus and workload.Tasks were modelled around typical analyst output requirements and request a brief report and a decision (e.g., the most likely perpetrator of a crime) or recommendation (e.g., level of threat).

Environment
The second major consideration was elements of the information environment likely to interact with analysis behaviour and cognitive experience.Elements identified as being particularly crucial were access to domain-specific knowledge, whether the task was predictive or diagnostic, and the complexity of the information.These will be addressed individually.
Access to domain specific knowledge is an influential element in decision-making tasks.For example, participants choose to examine different information (Griggs & Cox, 1982;McKenzie & Mikkelsen, 2000) and make different inferences (Lopez, Atran, Coley, Medin, & Smith, 1997;Proffitt, Coley, & Medin, 2000) if they are able to draw on domain-specific memory.We therefore chose content that is familiar to a range of general-population audiences, which allowed testing on non-experts who still had some degree of domain-relevant knowledge through popular mass media and experience.
Analytic inferences are also affected by whether the task is predictive or diagnostic (Fernbach, Darlow, & Sloman, 2011;Friedrich, 1993).Predictive tasks involve forecasting a future event such that the correct answer cannot be known for certain but probable outcomes can be inferred using the information available.An example is risk assessment.In contrast, diagnostic tasks, such as medical diagnosis or criminal conviction, have a correct answer but the information provided may or may not be adequate to detect it.In order to cover both types of tasks, two variants were created.In the predictive task, ASP -SINTELLA -Country Study (Note 1), participants were asked to utilise information on a fictitious country called Parette to develop a threat assessment within a security context.In the detection task, ASP -SINTELLA -Murder Investigation, participants were asked to investigate a murder and decide who was the most probable perpetrator.The tasks were also chosen to represent important analytical activities.
Finally, whether the information environment is seen as deterministic (with definitive, all or none outcomes) or probabilistic also impacts behaviour (McKenzie & Amin, 2002).We attempted to maintain the structure of the analyst's information environment with minimal simplification.The information environment experienced in analytic work is complex and dynamic with the following characteristics (Omodei & Wearing, 1995); 1) Large number of interrelated variables 2) Ambiguity and uncertainty in key variables 3) Multiple decision and interpretation alternatives 4) Dynamic variability where one's own actions alter the situation These characteristics were incorporated in the development of the ASP content.For example, in the Murder Investigation scenario the content of the cells is often ambiguous, incomplete, interrelated and in some cases contradicts other information.There is no "silver bullet" or item of absolute proof that any particular suspect is the perpetrator.There are seven possible suspects, each with potential motives to have committed the murder allowing for multiple interpretations.Finally, although the content itself does not change dynamically yet, the interpretation of ambiguous cells can be swayed by other information accessed.For example alibi claims can be interpreted as evidence of innocence or deception depending on whether corroborating or conflicting evidence has been accessed.

Simplification
In order to maintain experimental control over elements of the task and to make ASP participation feasible, some aspects had to be simplified.Firstly the time allowed for analysis tasks would usually be days, rather than minutes, and the information available would be significantly greater.Our aim was to design a training and assessment tool that was efficient to use, while maintaining cognitive fidelity.Therefore, we attempted to replicate the cognitive experience of searching extensive information in limited time, instead of the absolute time frames and information quantity typical of analysis tasks.Secondly, analysts would rarely be presented with a matrix of available information at the very start of an investigation.This was a necessary simplification in order to examine behavioural differences within a standardised information display.It is also not entirely unrealistic.Internet searches and database queries will commonly return a selection of information that analysts can choose to access or not.Results from this simulation are therefore applicable to understanding information selection behaviour in search tasks, but cannot extend to the explanation of query generation paradigms.

Experience
Having defined the essential elements of the simulation, the next step was to check that they were adequately demonstrated in participant's behaviour and responses.To do this we identified what we would expect to find if the simulation was experienced as intended.
If the resources included were indeed useful, we would expect them to be utilised.The word limit for reports was 250.We would expect on average at least half capacity reports (125 words) and more for workspace usage.
Cognition H1: On average, participants will revisit the 'tasking' at least once and type or copy at least 125 words into the workspace and report space.
If there is more information than could be processed in the time given, we would expect the proportion of cells opened to be normally distributed, rather than skewed towards 100%.However this could also indicate that not all information was considered useful.Therefore we would also expect people to report feeling time pressured.
Cognition H2: The proportion of cells opened will be normally distributed and the majority of participants will claim to need more time.
The main environmental prediction that we wanted to measure was the complexity of the information environment.Most aspects of complex information environments were built into the design of ASP.However, it was deemed important to check the ambiguity of the information environment.In the Murder Investigation task, we would be satisfied that ambiguity was sufficient if fewer than 50% of participants choose the correct perpetrator out of seven potential suspects.
Environment H1: A minority of participants will select the nominally correct perpetrator (Murder Investigation).
In the Country Study task, we would be satisfied that ambiguity was sufficient if threat assessments show a wide range encompassing at least half of the scale range including low and high threat assessment values.
Environment H2: Threat assessments will range from 25% or lower to 75% or higher.

Theory-based Measure Identification
Process tracing simulations have the capacity to measure a lot of data.For example ASP measures all cell-related and text-space related activity, including time per activity (to the millisecond) and sequence information.The detailed data is valuable for delving into the specifics of the process under examination, as demonstrated in Sections 4.3.1 and 5.3.1.However, it is very important at all levels of analysis that theory is used to define what measures are examined.

Research Question
The Murder Investigation was used to examine a cognitive bias called tunnel vision (Snook & Cullen, 2008).Tunnel vision is the tendency to focus on a single option to the exclusion of others, and has been implicated in a number of broader cognitive biases including confirmation bias (Klayman, 1995;Lehner et al., 2008;Oswald & Grosjean, 2004) and pseudo-diagnosticity (Evans, Venn, & Feeney, 2002).We predicted that participants would demonstrate tunnel vision in the Murder Investigation environment by giving disproportionally high attention to the chosen suspect compared to other suspects.

Measures
In order to measure this construct we need to define the behavioural indicators of "disproportionate attention" and "suspect-relevant information".Attention could not be gauged by first-time cell openings because cell headings on the information grid did not reliably indicate which, if any, suspects that cell had implication for.Therefore, measures had to reflect interest in the cell content rather than the cell heading.We chose to operationalise attention to cell content as reopening cells or using the workspace immediately after closing a cell.The proportion of cell-revisits and workspace visits per category of cells was chosen as the best measure because it was not skewed by high activity for a single cell.Suspect-relevant information was determined by coding each cell of the information board for whether or not it had implication for the guilt or innocence of each of the seven suspects.These ratings were then used to create categories of cells: chosen-suspect implication (cells with implication for the chosen suspect), other-suspect implication (cells with implication for non-chosen suspects) or no-implication (cells without implication for any suspects).This resulted in the following hypothesis: Tunnel Vision H1: On average participants will revisit, and follow with workspace activity, a greater proportion of chosen-suspect implication cells than other-suspect implication cells.

Assumptions
Reducing a high level construct such as tunnel vision, down to behavioural predictions such as cell revisits, cannot be done without making assumptions.We believe it is important to acknowledge and assess these assumptions where possible.We assume that cell revisits and workspace activity indicate intentional attention to those cells.This assumption would be undermined if participants predominantly returned to cells because they had forgotten the content or used the workspace indiscriminately.If we did not check this assumption, and found no evidence of tunnel vision, we could falsely reject a hypothesis due to insensitive measures.Regardless of whether tunnel vision is observed, we would expect participants to pay more attention to cells with implication for any suspect, than to cells with none, given their task of identifying the most probable perpetrator.Assumption H1: On average participants will revisit, and follow with workspace activity, a greater proportion of suspect-implication cells than no-implication cells.

Participants
A sample of 50 participants (60% female) from a large Australian University were recruited and paid AUS$20 for one hour of their time.Participants came from all major faculties of the university including Arts (24%), Business/Economics (22%), Science (6%) and Human Sciences (Psychology 16%, Other 16%), with 16% not specifying a faculty.

Procedure
Participants were seated at individual computers that were shielded from each other.After filling in an information and consent form, participants completed a series of demographic questions and questionnaires on the computer.They were then presented with the "tasking" that provided instructions on how to use the information resources, workspace and product spaces, and details of the scenario and task.The scenario provided limited details about the crime scene, the seven suspects, and their connections to each other and the victim.The task was to decide on the most probable perpetrator.
Participants then continued to the screen with the information resources, workspace and product space described in detail in section 2.1.The information grid provided information on activities and people associated with the victim in categories such as "Persons of Interest" and "Police Documentation".Participants had to click on information cells to open and close the cell before opening another cell.Participants were advised of the time elapsed at the bottom of this page.An automated message warned them when there was 20% (8 minutes) of the time left, and a second message asked them to make their decision and click 'next' when time (40 minutes) was up.Participants were then taken to a short questionnaire asking about their decision, reasons, and perceptions of the scenario, followed by some questions about program usability.

Results
For expedience, the effectiveness of the simulation construction will be examined across Experiments 1 and 2 in Section 4.3.3.For Experiment 1 we will focus on the results for the assumption and tunnel vision hypotheses.
The Assumption Hypothesis was supported, as shown in Table 1 participants revisited or immediately followed with workspace activity a greater proportion of the cells with implication for suspects, compared to those with no implication.The effect sizes were small to moderate for repeat cell openings and moderate to large for workspace activity.Tunnel vision was examined in the same way, by using within subject comparisons between chosen-suspect implication and other-suspect implication cells and confirmed the tunnel vision hypothesis (Wastell et al., under review).

Experiment 2: Murder Investigation B
Experiment 1 effectively replicated findings of tunnel vision in this new information environment.However, behaviour was still generalised across the entire task time and thus provided no insight into the process by which tunnel vision occurs.Some authors assume that tunnel vision is triggered as soon as a single option is favoured.For example Kruglanski and Webster (1996) suggested that people are motivated to seize and freeze on an option.Other theorists (e.g., Simon, Snow, & Read, 2004) assume a more gradual process whereby preference and interpretation assert a bi-directional influence.This approach would suggest that tunnel vision does not begin at a certain point but rather increases as one becomes more certain about their position.These approaches only differ in process, not in outcome, and therefore require process-tracing techniques to distinguish them.Experiment 2 examines whether tunnel vision emerges only after the chosen-suspect is selected for investigation.

Temporal Specificity
In Experiment 1 we could not determine when participants began to investigate their finally chosen suspect with any certainty.To overcome this, in Experiment 2 we incorporated "interruptions".Participants were asked to select a prime suspect (or unsure) and secondary suspects (or none) before beginning the information search and after every 10 cells opened.We defined the "settle point" as the interruption at which participants selected their chosen suspect as the prime suspect and then as prime or secondary suspect for every subsequent interruption.If tunnel vision occurs only after the chosen suspect is selected for investigation, we would expect evidence of tunnel vision to be significantly greater after the settle point.
Tunnel vision H2: There will be a significant interaction between time and cell activity (repeat visits and workspace activity).The difference between cell activity related to chosen-suspect implication cells and other-suspect implication cells will be greater after the settle point than before.

Impact
A major concern raised against process-tracing techniques is that the very process of measuring or eliciting responses over time changes those responses (Kuhberger et al., 2010).This is a valid concern and where possible it is advisable to compare behaviour with and without the process-tracing intervention (as demonstrated in Section 4.3.2;for another example see Holyoak & Simon, 1999).
The inclusion of interruptions in Experiment 2, but not in Experiment 1, allows for two important examinations.Firstly we could check that interruptions did not fundamentally alter measures of tunnel vision.We chose to use self-report questionnaires presented after every ten-cell openings.This maintained separation between meta-cognitive reporting on the investigation, and the investigation itself.This was particularly important in our case because we wanted to simulate the cognitive demands of the investigation task as faithfully as possible, which would have been compromised by concurrent protocols such as think aloud techniques.
Impact H1: The proportion of cells revisited or followed by workspace activity will not be significantly different between Experiments 1 and 2.
Secondly, we could compare the self-reported investigation decisions throughout the task and retrospectively.Concurrent self-report would be expected to be more accurate than retrospective self-report, the question of methodological interest is how big this difference is.Given the common method, reporting on the same behaviour, a high correlation would be expected.
Impact H2: Retrospective reports on the number of suspects investigated will show a strong correlation with the number of suspects named as prime or secondary suspects during the investigation.

Participants
A sample of 44 participants (54% female) from a large Australian University were recruited and paid AUS$20 for one hour of their time.Participants came from all major faculties of the university including Arts (34%), Business/Economics (25%), Science (18%) and Human Sciences (Psychology 12%, Others 11%).

Procedure
The procedure and tasking was exactly the same as in Experiment 1, with one exception.After the "tasking" page and after each 10 cell-openings, participants were asked to select one of seven prime suspect or "unsure" and up to seven secondary suspects.

Temporal Specificity
The "settle point" was normally distributed.On average people settled on their chosen suspect by the 7 th interruption after the first 50 to 60 cell openings (M = 55.29,SD = 31.36,N = 34).The number of cells opened post settle (M = 27.64,SD = 25.58,N = 34) showed a slight right skew suggesting that participants would often finish the activity soon after settling.A quarter of the participants (n = 10) never settled on their chosen suspect.Despite a replication of Tunnel vision H1 and Assumption H1, Tunnel vision H2 was rejected.The difference between chosen-suspect and other-suspect implication cells remained constant pre and post settle as indicated by a non significant interaction effect that explained no variance in the case of repeat cell openings, and only 3% of the variance in the case of workspace usage.This pattern of results is more consistent with a gradual rather than a sudden evolution of tunnel vision.
Impact H2 was rejected.The number of suspects selected in the retrospective question asking participants which suspects they investigated, showed no correlation with the number of suspects named as prime suspect (r = -.01,N = 41, p = .970),or named at all as prime or secondary suspect (r = .24,N = 41, p = .133)throughout the task.On average people said they investigated fewer suspects (M = 2.66, SD = 1.33) than they named as prime or secondary suspect throughout the task (M = 4.09, SD = 1.71), t (40) = 4.52, p < .0005,d = .88.

Simulation Construction
Recall that the main features we aimed to replicate were cognitive resources and limitations and elements of the information environment.We first examined whether the resources provided were utilised.Cognition H1 was accepted.As shown in Table 2 average report content was over half capacity, workspace content was about full report capacity (250 words), and of the 80% of participants who revisited the tasking, they did so on average about four times.Despite no obligation to use the workspace, more than 90% of participants typed or copied content into the workspace across both Murder Investigation experiments as shown in Table 2.These results suggest that the resources provided were useful.In post-task qualitative comments only one resource was consistently mentioned as lacking, which was the ability to draw diagrams and mind-maps.
Cognition H2 was partially accepted.As shown in Table 2, the proportion of cells opened in Experiment 1 and Experiment 2 showed slight negative skew, indicating a tendency to open the majority of cells.In Experiment 1, 24% of participants opened all 64 cells, but in Experiment 2 only 2% did.However, there was no difference between participants who opened all cells and those who did not in terms of accuracy of decision or perceived time pressure.The majority of participants (73% and 74%) said that they did not have enough time.This indicates that although for some participants, there was enough time to open all cells, the majority of participants experienced time pressure, and were unable to successfully integrate all the information received.
Finally, Environment H1 predicted that a minority of participants would select the nominally correct perpetrator.This hypothesis was also supported.As shown in Figure 2, the most probable murderer (suspect 3 in the current version) was chosen by 39% of participants on average across the two samples.The proportion of participants who chose each suspect also roughly followed the pattern of likelihoods calculated by the researchers using the Analysis of Competing Hypotheses (ACH; PARC & Heuer, 2005) software program (see Figure 2).Confidence did not differ by decision accuracy, suggesting a good degree of ambiguity.

Experiment 3: Country Study
Unlike the Murder Investigation task, the Country Study task has no correct answer.The tasking is predictive and requires participants to investigate a fictitious country called "Parette" to produce a report and an assessment.

Research Question
The Country Study task was used to examine a spatially specific analysis technique particularly valuable for framing questions.Framing is the phenomena whereby logically equivalent descriptions of a task produce different responses (Levin, Schneider, & Gaeth, 1998).For example meat that is "90% lean" is rated more positively than meat that is "10% fat" (Levin, Johnson, Russo, & Deldin, 1985).Experimental examinations of framing effects usually focus on the outcome or decision.However framing theories differ on the process by which the decision occurs.For example, the associative hypothesis states that framing induces a negative, or positive, focus that flows over to the selection and interpretation of information (Levin et al., 1998).In contrast the reference point hypothesis states that the frame chosen by the communicator conveys implicit information (Sher & McKenzie, 2006).This information is used in decisions but does not guide information search behaviour.The ability to detect not only differences in decision outcome, but also in information search pattern, would be valuable for the examination of the associative hypothesis in particular.
In this case, we are not comparing logically equivalent framing manipulations, but rather testing that a) information search behaviour is different for different tasks even given an identical information environment and b) that the analysis techniques used are sensitive to these differences.

Spatial Specificity
We begin with general analyses based on typical framing questions.The exploratory questions of interest are: Country Study RQ1: Do different tasks prompt different threat estimates?
Threat estimates are the equivalent to a decision outcome measure in traditional framing tasks and are provided with the participant's report after the investigation.Since we are interested in the pattern of cell openings, we first investigate cell opening behaviour at the general level.We are interested in how tasks direct information search, rather than reactions to information; therefore in this case we focus on first cell openings excluding repeat cell openings.
Country Study RQ2: Are different tasks associated with a different proportion of the 64 cells opened?
Finally, the question we are most interested in is whether we can detect different patterns of cell openings.
Country Study RQ3: Are different tasks associated with different degrees of attention to particular groups of cells?Groups of cells are not predefined here as they were in the Murder Investigation.Rather, we examine individual cells by looking at the column by row by task interaction, and attention to columns using the column by task interaction term.As columns are labelled, whereas rows are not, attention to certain columns is more likely and more interpretable than attention to certain rows.

Participants
As shown in Table 3, the participant qualities in the two Country Study samples differ.The first sample was given credit for participation, whereas the second sample was paid.This resulted in a higher concentration of business and economics students in the first sample.This does not undermine the integrity of these studies for examining the effect of the simulation or providing worked examples of analysis approaches, as is our purpose.

Procedure
The procedure was the same as in Experiment 1. However the information content, scenario, and task differed.The information grid provided information on a fictitious country called "Parette", with category headings such as "Crime, Police and Security" and "Places of Interest".The scenario provided some context around the request for a report.The task was either a Travel Advisory, where participants were asked to advise potential tourists about Parette, or a Country Brief, where participants were asked to provide an overview of Parette for a member of university staff.

Spatial Specificity
In answer to the first research question, at a general level the two samples given different tasks did produce quite different estimates.The threat assessment average fell just above the midpoint towards extreme threat for the Travel Advisory condition, but on the midpoint for the Country Brief as shown in Table 4, t (75) = 2.08, p = .041,d = 0.48.On average participants were almost 20 percentage points more confident on the Travel Advisory than they were on the Country Brief, t (75) = 3.72, p < .0005,d = 0.83.This provides preliminary indication that the two samples differ in their perception of the information space across task types.In answer to the second research question, at a general level of analysis there is no difference in the percentage of cells opened t (75) = 1.61, p = .113,d = 0.37.The time on task, repeat cell openings, and words in the report show no significant differences across tasks either (Table 4).
In order to investigate Research Question 3, a visual demonstration was produced examining the most commonly opened cells (excluding repeat openings) across the two Country Study samples shown in Table 5.In both tasks there seemed to be a higher concentration of cells opened toward the top left corner of the information board suggesting some impact from layout.However, there was much less consensus on the Country Brief task than there was on the Travel Advisory task, as demonstrated by the greater number of cells opened by a small proportion of the participants.There also appeared to be more interest in the "Crime, police and security" and "International relations" columns in the Travel Advisory task compared to the Country Brief task.These visual observations were confirmed using a series of random effects logistic regressions, as implemented in Stata 11 (StataCorp., 2009).This method allows for the fact that each participant contributed more than one observation, and that the number of observations varied over individuals.The model can be seen as having two levels.The participants are at level 2 and are assumed to be independent of each other, while the observations they contribute are at level 1 and are assumed to be correlated within individuals.The three-way interaction between rows, columns, and group was not significant, suggesting no significant differences in individual cells opened.However, columns showed a significant two-way interaction with group.Adjusting the acceptable alpha level to .05/8 to account for the multiple comparisons, activity in columns 5 and 8 differed significantly by group.Column 5, "Crime, police and security", was nearly eight times more likely to be visited in the Travel Advisory task compared to the Country Brief task  2 (1, N = 77) = 20.50,p < .00005,Odds Ratio = 7.82.Column 8, "International relations", was five times more likely to be visited in the Travel Advisory task compared to the Country Brief task  2 (1, N = 77) = 12.89, p = .0003,Odds Ratio = 4.99.

Simulation Construction
As previously demonstrated in the Murder Investigation task, we checked that the cognitive experience of the simulation was as intended.Cognition H1 was accepted.As shown in Table 4 average report content was over half capacity, workspace content was over full report capacity (250 words), and of the 62% of participants who revisited the tasking, they did so on average about twice.These results suggest that the resources provided were useful.
In this case Cognition H2 was not accepted.The proportion of cells opened in Sample 1 and Sample 2 were normally distributed as shown in Table 4.However, only a single participant indicated that they didn't have enough time in the Travel Advisory task (3%) and one third indicated that they didn't have enough time in the Country Brief task (31%).In both samples about a quarter indicated that they were given too much time (22% and 27%).This may suggest that the time limit was too generous to adequately apply a sense of time pressure.
Finally, Environment H2 proposed that if ambiguity were sufficient, the range of threat assessments should encompass the 25-75% range.As shown in Table 4 ambiguity was acceptable.In the Travel Advisory and Country Brief tasks, participants produced various interpretations of the standardised content as evidenced by threat-assessment values that ranged from almost no threat, to close to extreme threat and confidence in these assessments that ranged from very uncertain to extremely confident as shown in Table 4.

General Discussion
The framework introduced in Section 1 and applied to ASP methodology yielded valuable results.Firstly, by defining and measuring the experience that we wanted to produce, we were able to ascertain that in many respects the simulation was successful at producing the target experience.This activity also provided concrete feedback on potential improvements.Specifically, the time provided will be shortened for ASP-SINTELLA-Country Study tasks, and we will look into providing the capacity to draw diagrams.
Secondly, the application of the framework demonstrated how the richness and depth of data could be maintained, without excessively complicated analyses by moving from a general to a specific level of analysis.The tunnel vision examination showed that participants paid more attention to their chosen suspect than to all other 6 suspects combined.However, this tendency was no greater after they settled on their chosen suspect than it was before.This seems to contradict the seize and freeze model (Kruglanski & Webster, 1996) that suggests a sudden emergence of tunnel vision behaviour, but may still be consistent with the gradual evolution of tunnel vision.It could be argued that our measure of the "settle point" was too crude an estimation of the point at which behaviour changes.To address this critique, or other research questions requiring more refined analysis of temporally evolving behaviour, similar techniques could be used to compare behaviour in different phases of activity or map behaviour across numerous points in time.Alternatively individual cell behaviour can be modelled and compared using binary logistic regressions.These advances in data collection and analysis techniques make theoretical process differences accessible to research.
In addition to the temporal specificity demonstrated by the Murder Investigation tasks, spatial specificity was demonstrated by the Country Study tasks.These analyses demonstrate that by using a standardised information space, we can detect and measure the impact of tasking or population on behaviour in a complex information space.In this case tasking and population are confounded.This does not weaken the proof of concept provided here, but accepted methods of randomised control need to be employed to strengthen any theoretical conclusions drawn.With adequate randomised control, this methodology could be used to examine the process behind framing effects, compare information focus across tasks or task presentation methods, across different work teams, across experts and novices, or across professions.As presented here, the content of cells is confounded with cell location.This is not a problem for comparing behaviour across different tasks or populations because presentation order and content are constant.Therefore any differences can be safely attributed to the manipulated variable.
In both of these cases the process-tracing technique, which allowed for examination of temporally and spatially specific questions, provided greater insight into the processes involved.In both tunnel vision and framing, there is a common outcome, but theoretical differences in process.Process tracing methodology can help us to distinguish these theories, which is valuable for theoretical and practical advancement.For example, whether tunnel vision begins at a certain point, or whether it evolves gradually has important implications for bias remediation.The first would promote delaying preliminary decisions whereas the latter would target the conviction with which preliminary decisions are held.The provision of appropriate recommendations in this case, and others, relies on sophisticated process tracing abilities.
This research also provides insight into the value of concurrent, as apposed to retrospective self-report.These results demonstrate that participant's self reported investigation during the task showed very little correspondence with their retrospective assessment of the suspects they had investigated.One could argue that naming prime suspects and secondary suspects does not indicate those suspects were actually investigated.However, we would expect these measures to be at least moderately correlated, especially for prime suspects who are presumably likely to be investigated.This provides some indication that concurrent and retrospective self-report measures can produce very different results.Therefore it is important to balance the risk of influencing behaviour through the use of concurrent self-report against the advantages to be gained through more appropriate temporal positioning of questions.
In conclusion, process-tracing techniques have a lot to offer.We have provided principles for the expansion of process tracing techniques into complex, ambiguous information environments.These advances open up opportunities to observe and measure the process of analysis within realistic organisational scenarios.Possible applications include, but are not limited to, assessment of differences in focal information between experts and trainees or across collaborating work groups; measurement of behaviour pre and post training; presentation of individual behaviour for use in training and development programs; and the study of behavioural changes over time.We hope that this paper will promote consideration of process-tracing techniques as a valuable tool for research in complex, ambiguous environments.Wastell, C. A., Weeks, N. J., Wearing, A. J., Duncan, P., & Ebrahimi, W. (In press).The impact of closed mindedness on the assessment of threat: An empirical study.Intelligence and National Security.
WMD. (2005).The commission on the intelligence capabilities of the United States regarding weapons of mass destruction, Report to the President of the United States.Washington D.C.

Note
Note 1. SINTELLA stands for Simulation of INTELLigence Analysis and was originally developed within the security and defence domain though now refers to the more general processes of the analysis of complex, dynamic, uncertain information environments.
Define the essential intellectual experiences and cognitive demands to simulate.b.Environment: Identify environmental elements likely to impact on these processes.c.Simplification: Acknowledge necessary simplifications and resulting restrictions in generalisation.d.Experience: Measure how well the simulation imitates the essential experiences and demands using behavioural and subjective measures.2) Theory-based measure identification a. Research Question: Define the question of interest.b.Measures: Operationalise measures based on theory.c.Assumptions: Use behavioural data to check assumptions.d.Specificity: Move from general analyses to more spatially or temporally specific analyses driven by the question of interest.e. Impact: Measure the impact of methodology where possible.

Figure 1 .
Figure 1.Layout of ASP -SINTELLA main screen Each of the 64 cells opens to reveal 50 to 130 words of information on the topic indicated by the heading.

Figure 1 .
Figure 1.Participant choice of suspect compared to researcher rated probability of being the perpetrator for the two murder investigation experiments

Table 1 .
Measures of attention across cells with and without implication for any suspect

Table 2 .
Participant behaviour on the two murder investigation experiments

Table 5 .
Percentage of participants who opened each cell (at least once) across the two prediction tasks